记一次企微接口返回非utf-8字符导致json反序列化失败问题

背景：

我这边负责的基于企微开发的scrm产品，最近租户反馈，客户数量和企微后台的客户数量对不上。尝试手动触发同步外部联系人数据，发现有异常，导致同步中断了。

异常日志如下：

feign.codec.DecodeException: Error while extracting response for type [class xxx.remote.response.ExternalContactUserDetailResponse] and content type [application/json;charset=UTF-8]; nested exception is org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Invalid UTF-8 start byte 0x8d; nested exception is com.fasterxml.jackson.databind.JsonMappingException: Invalid UTF-8 start byte 0x8d
 at [Source: (ByteArrayInputStream); line: 1, column: 135] (through reference chain: xxx.remote.response.ExternalContactUserDetailResponse["external_contact"]->xxx.remote.response.externaluser.ExternalContactUser["avatar"])

   at feign.SynchronousMethodHandler.decode(SynchronousMethodHandler.java:174)
   at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:134)
   at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:77)
   at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:102)
   at com.sun.proxy.$Proxy217.externalContactGet(Unknown Source)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
   at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)
   at xxx.aspect.AbstractLogAspect.doAround(AbstractLogAspect.java:91)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
   at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
   at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
   at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
   at com.sun.proxy.$Proxy218.externalContactGet(Unknown Source)
   at xxx.external.workwx.service.impl.WwContactUserServiceImpl.get(WwContactUserServiceImpl.java:90)
   at xxx.user.service.impl.ExternalUsertTest.testDetail(ExternalUsertTest.java:92)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
   at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
   at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
   at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
   at org.springframework.test.context.junit4.statements.RunBeforeTestExecutionCallbacks.evaluate(RunBeforeTestExecutionCallbacks.java:73)
   at org.springframework.test.context.junit4.statements.RunAfterTestExecutionCallbacks.evaluate(RunAfterTestExecutionCallbacks.java:83)
   at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
   at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
   at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
   at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:251)
   at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
   at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
   at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
   at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
   at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
   at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
   at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
   at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
   at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: org.springframework.web.client.RestClientException: Error while extracting response for type [class xxx.remote.response.ExternalContactUserDetailResponse] and content type [application/json;charset=UTF-8]; nested exception is org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Invalid UTF-8 start byte 0x8d; nested exception is com.fasterxml.jackson.databind.JsonMappingException: Invalid UTF-8 start byte 0x8d
 at [Source: (ByteArrayInputStream); line: 1, column: 135] (through reference chain: xxx.remote.response.ExternalContactUserDetailResponse["external_contact"]->xxx.remote.response.externaluser.ExternalContactUser["avatar"])
   at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:115)
   at xxx.feign.config.UnwrapResponseDecoder.decodeInternal(UnwrapResponseDecoder.java:123)
   at xxx.feign.config.UnwrapResponseDecoder.decode(UnwrapResponseDecoder.java:91)
   at org.springframework.cloud.openfeign.support.ResponseEntityDecoder.decode(ResponseEntityDecoder.java:45)
   at feign.optionals.OptionalDecoder.decode(OptionalDecoder.java:36)
   at feign.SynchronousMethodHandler.decode(SynchronousMethodHandler.java:170)
   ... 57 more
Caused by: org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Invalid UTF-8 start byte 0x8d; nested exception is com.fasterxml.jackson.databind.JsonMappingException: Invalid UTF-8 start byte 0x8d
 at [Source: (ByteArrayInputStream); line: 1, column: 135] (through reference chain: xxx.remote.response.ExternalContactUserDetailResponse["external_contact"]->xxx.remote.response.externaluser.ExternalContactUser["avatar"])
   at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:243)
   at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(AbstractJackson2HttpMessageConverter.java:225)
   at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:100)
   ... 62 more
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Invalid UTF-8 start byte 0x8d
 at [Source: (ByteArrayInputStream); line: 1, column: 135] (through reference chain: xxx.remote.response.ExternalContactUserDetailResponse["external_contact"]->xxx.remote.response.externaluser.ExternalContactUser["avatar"])
   at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:394)
   at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:353)
   at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1711)
   at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:371)
   at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159)
   at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:127)
   at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:369)
   at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159)
   at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4013)
   at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3084)
   at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:237)
   ... 64 more
Caused by: com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0x8d
 at [Source: (ByteArrayInputStream); line: 1, column: 139]
   at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804)
   at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:669)
   at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidInitial(UTF8StreamJsonParser.java:3537)
   at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidChar(UTF8StreamJsonParser.java:3533)
   at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2479)
   at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2405)
   at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:269)
   at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:35)
   at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:10)
   at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:127)
   at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:369)
   ... 71 more

这日志，显然就是企微接口的响应的字符格式有问题，找企微客服沟通，让他们把接口或者数据处理一下。

和企微客服的沟通结论：

这答复非常有意思，我自己企业的客户我都没办法引导他重新上传微信头像。

更何况我是服务商，叫租户去叫他们的客户去重新上传微信头像。还是这异常导致同步失败，同步都失败了，哪还知道是谁的头像错了？企微客服给的建议显然不靠谱，只能看看怎么样通过代码兼容一下它这个特殊符号了。

解决方案：

feign配置自定义decoder，把特殊字符去掉。

把特殊字符去掉的代码示例:

InputStream inputStream = response.body().asInputStream();
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
IOUtils.copy(inputStream, outputStream);
CharBuffer buffer = StandardCharsets.UTF_8.newDecoder().onMalformedInput(CodingErrorAction.IGNORE).decode(ByteBuffer.wrap(outputStream.toByteArray()));
byte[] array = StandardCharsets.UTF_8.encode(buffer).array();
outputStream = new ByteArrayOutputStream(array.length);
outputStream.write(array, 0, array.length);