java性能調(diào)優(yōu)記錄(線程阻塞)

AndroidTraveler 發(fā)布于2019-08-16 14:31 / 1573人閱讀

摘要：其中內(nèi)容如下網(wǎng)關(guān)的邏輯是讀取請求中的值，根據(jù)字段去內(nèi)存的路由表中匹配路由，然后轉(zhuǎn)發(fā)請求到對應(yīng)的微服務(wù)中去。排查測試接口本身的性能發(fā)現(xiàn)接口的可以達到。解決改寫代碼以實現(xiàn)同樣的功能之后進行測試，發(fā)現(xiàn)從提升到了，問題解決。

1. 問題

spring-cloud-gateway 作為統(tǒng)一的請求入口，負(fù)責(zé)轉(zhuǎn)發(fā)請求到相應(yīng)的微服務(wù)中去。

采用的 Spring Cloud 的版本為 Finchley SR2。

測試一個接口的性能，發(fā)現(xiàn) tps 只有 1000 req/s 左右就上不去了。

[root@hystrix-dashboard wrk]# wrk -t 10 -c 200 -d 30s --latency -s post-test.lua "http://10.201.0.28:8888/api/v1/json"
Running 30s test @ http://10.201.0.28:8888/api/v1/json
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   188.34ms  110.13ms   2.00s    78.43%
    Req/Sec   106.95     37.19   333.00     77.38%
  Latency Distribution
     50%  165.43ms
     75%  243.48ms
     90%  319.47ms
     99%  472.64ms
  30717 requests in 30.04s, 7.00MB read
  Socket errors: connect 0, read 0, write 0, timeout 75
Requests/sec:   1022.62
Transfer/sec:    238.68KB

其中 post-test.lua 內(nèi)容如下：

request = function()
    local headers = {}
    headers["Content-Type"] = "application/json"
    local body = [[{
        "biz_code": "1109000001",
        "channel": "7",
        "param": {
            "custom_id": "ABCD",
            "type": "test",
            "animals": ["cat", "dog", "lion"],
            "retcode": "0"
        }
    }]]
    return wrk.format("POST", nil, headers, body)
end

網(wǎng)關(guān)的邏輯是讀取請求中 body 的值，根據(jù) biz_code 字段去內(nèi)存的路由表中匹配路由，然后轉(zhuǎn)發(fā)請求到對應(yīng)的微服務(wù)中去。

2. 排查

測試接口本身的性能：

[root@hystrix-dashboard wrk]# wrk -t 10 -c 200 -d 30s --latency -s post-test.lua "http://10.201.0.32:8776/eeams-service/api/v1/json"
Running 30s test @ http://10.201.0.32:8776/eeams-service/api/v1/json
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    26.72ms    8.59ms 260.23ms   89.66%
    Req/Sec   752.18    101.46     0.94k    78.67%
  Latency Distribution
     50%   23.52ms
     75%   28.02ms
     90%   35.58ms
     99%   58.25ms
  224693 requests in 30.02s, 50.83MB read
Requests/sec:   7483.88
Transfer/sec:      1.69MB

發(fā)現(xiàn)接口的 tps 可以達到 7000+。

通過 spring-boot-admin 查看網(wǎng)關(guān)的 cpu、內(nèi)存等占用情況，發(fā)現(xiàn)都沒有用滿；查看線程狀況，發(fā)現(xiàn) reactor-http-nio 線程組存在阻塞情況。對于響應(yīng)式編程來說，reactor-http-nio 線程出現(xiàn)阻塞結(jié)果是災(zāi)難性的。

通過 jstack 命令分析線程狀態(tài)，定位阻塞的代碼（第 19 行）：

"reactor-http-nio-4" #19 daemon prio=5 os_prio=0 tid=0x00007fb784d7f240 nid=0x80b waiting for monitor entry [0x00007fb71befc000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
    - waiting to lock <0x000000008b0cec30> (a java.lang.Object)
    at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:93)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.springframework.util.ClassUtils.forName(ClassUtils.java:282)
    at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.registerWellKnownModulesIfAvailable(Jackson2ObjectMapperBuilder.java:753)
    at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.configure(Jackson2ObjectMapperBuilder.java:624)
    at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.build(Jackson2ObjectMapperBuilder.java:608)
    at org.springframework.http.codec.json.Jackson2JsonEncoder.(Jackson2JsonEncoder.java:54)
    at org.springframework.http.codec.support.AbstractCodecConfigurer$AbstractDefaultCodecs.getJackson2JsonEncoder(AbstractCodecConfigurer.java:177)
    at org.springframework.http.codec.support.DefaultServerCodecConfigurer$ServerDefaultCodecsImpl.getSseEncoder(DefaultServerCodecConfigurer.java:99)
    at org.springframework.http.codec.support.DefaultServerCodecConfigurer$ServerDefaultCodecsImpl.getObjectWriters(DefaultServerCodecConfigurer.java:90)
    at org.springframework.http.codec.support.AbstractCodecConfigurer.getWriters(AbstractCodecConfigurer.java:121)
    at org.springframework.http.codec.support.DefaultServerCodecConfigurer.getWriters(DefaultServerCodecConfigurer.java:39)
    at org.springframework.web.reactive.function.server.DefaultHandlerStrategiesBuilder.build(DefaultHandlerStrategiesBuilder.java:103)
    at org.springframework.web.reactive.function.server.HandlerStrategies.withDefaults(HandlerStrategies.java:90)
    at org.springframework.cloud.gateway.support.DefaultServerRequest.(DefaultServerRequest.java:81)
    at com.glsc.imf.dbg.route.RouteForJsonFilter.filter(RouteForJsonFilter.java:34)
    at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.lambda$filter$0(FilteringWebHandler.java:115)
    at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain$$Lambda$800/1871561393.get(Unknown Source)
    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:44)

最終定位到問題代碼為：

DefaultServerRequest req = new DefaultServerRequest(exchange);    // 這行代碼存在性能問題
return req.bodyToMono(JSONObject.class).flatMap(body -> {
    ...
});

這里的邏輯是我需要讀取請求中 body 的值，并轉(zhuǎn)化為 json，之后根據(jù)其中的特定字段去匹配路由，然后進行轉(zhuǎn)發(fā)。這里選擇了先把 exchange 轉(zhuǎn)化為 DefaultServerRequest，目的是為了使用該類的 bodyToMono 方法，可以方便的進行轉(zhuǎn)換。

3. 解決

改寫代碼以實現(xiàn)同樣的功能：

return exchange.getRequest().getBody().collectList()
        .map(dataBuffers -> {
            ByteBuf byteBuf = Unpooled.buffer();
            dataBuffers.forEach(buffer -> {
                try {
                    byteBuf.writeBytes(IOUtils.toByteArray(buffer.asInputStream()));
                } catch (IOException e) {
                    e.printStackTrace();
                }
            });
            return JSON.parseObject(new String(byteBuf.array()));
        })
    .flatMap(body -> {
        ...
    });

之后進行測試，

[root@hystrix-dashboard wrk]# wrk -t 10 -c 200 -d 30s --latency -s post-test.lua "http://10.201.0.28:8888/api/v1/json"
Running 30s test @ http://10.201.0.28:8888/api/v1/json
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    48.47ms   45.85ms 325.87ms   88.55%
    Req/Sec   548.13    202.55   760.00     80.01%
  Latency Distribution
     50%   31.18ms
     75%   39.44ms
     90%  112.18ms
     99%  227.19ms
  157593 requests in 30.02s, 35.94MB read
Requests/sec:   5249.27
Transfer/sec:      1.20MB

發(fā)現(xiàn) tps 從 1000 提升到了 5000+，問題解決。