日志

一次JVM分析记录

root@db:/home/db# jps
3328 Logstash
3780 CliToolLauncher
6020 Elasticsearch
2743821 cx-bi-low-code.jar
3632722 cx-bi-auth.jar
3335 nacos-server.jar
3154845 cx-bi-msg.jar
458025 kafka-ui-api.jar
858527 cx-bi-oss.jar
1340231 Jps
7283 ServerApplication
862619 cx-bi-gateway.jar
2568606 cx-bi-flow.jar
2575130 cx-bi-log.jar
3282348 cx-bi-callback.jar
3674535 cx-bi-system.jar
root@db:/home/db# jstat -gc 3674535 1 1
    S0C         S1C         S0U         S1U          EC           EU           OC           OU          MC         MU       CCSC      CCSU     YGC     YGCT     FGC    FGCT     CGC    CGCT       GCT
        0.0      1024.0         0.0       496.0     329728.0     228352.0     193536.0     123580.5   135808.0   134873.8   16064.0   15577.1    308     2.638     0     0.000    10     0.027     2.665
root@db:/home/db# echo $(((193536+329728+1024)/1024))'M(堆大小)'
512M(堆大小)
root@db:/home/db# jstat -gccause 3674535 1 1
  S0     S1     E      O      M     CCS    YGC     YGCT     FGC    FGCT     CGC    CGCT       GCT    LGCC                 GCC
  0.00  48.43  75.16  63.85  99.31  96.97    308     2.638     0     0.000    10     0.027     2.665 G1 Evacuation Pause  No GC
root@db:/home/db# jstat -gccause 3674535 1 1
  S0     S1     E      O      M     CCS    YGC     YGCT     FGC    FGCT     CGC    CGCT       GCT    LGCC                 GCC
  0.00  48.43  76.71  63.85  99.31  96.97    308     2.638     0     0.000    10     0.027     2.665 G1 Evacuation Pause  No GC
root@db:/home/db# jmap -histo 3674535 |head -50
 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:         42884      102900224  [I (java.base@17.0.17)
   2:        578091       81506880  [B (java.base@17.0.17)
   3:         13833       62494472  [C (java.base@17.0.17)
   4:        178011       13676664  [Ljava.lang.Object; (java.base@17.0.17)
   5:        144180       11534400  com.alibaba.csp.sentinel.node.metric.MetricNode
   6:        128675       11323400  java.lang.reflect.Method (java.base@17.0.17)
   7:        435868       10460832  java.lang.String (java.base@17.0.17)
   8:        209572        8382880  java.util.LinkedHashMap$Entry (java.base@17.0.17)
   9:        448181        7170896  org.apache.seata.core.rpc.netty.AbstractNettyRemotingClient$MergedSendRunnable$$Lambda$835/0x00007fd4b4803c60
  10:        203623        6515936  java.util.concurrent.ConcurrentHashMap$Node (java.base@17.0.17)
  11:         50773        4102632  [Ljava.util.HashMap$Node; (java.base@17.0.17)
  12:         67145        3222960  org.aspectj.weaver.reflect.ShadowMatchImpl
  13:         25294        3038536  java.lang.Class (java.base@17.0.17)
  14:         48195        2698920  java.util.LinkedHashMap (java.base@17.0.17)
  15:         89621        2150904  java.util.ArrayList (java.base@17.0.17)
  16:         67145        2148640  org.aspectj.weaver.patterns.ExposedState
  17:         66495        2127840  java.util.HashMap$Node (java.base@17.0.17)
  18:         26756        1926432  java.lang.reflect.Field (java.base@17.0.17)
  19:          1763        1725552  [Ljava.util.concurrent.ConcurrentHashMap$Node; (java.base@17.0.17)
  20:         66343        1630208  [Ljava.lang.Class; (java.base@17.0.17)
  21:         45344        1451008  java.io.File (java.base@17.0.17)
  22:         27127        1302096  org.springframework.util.ConcurrentReferenceHashMap$SoftEntryReference
  23:         79259        1268144  java.lang.Object (java.base@17.0.17)
  24:          8103        1263672  [J (java.base@17.0.17)
  25:          2968        1092224  java.lang.Thread (java.base@17.0.17)
  26:         31968        1079200  [Ljava.lang.String; (java.base@17.0.17)
  27:         21034        1009632  java.util.HashMap (java.base@17.0.17)
  28:         39068         937632  java.lang.StringBuilder (java.base@17.0.17)
  29:         36171         868104  org.springframework.core.MethodClassKey
  30:          1441         841544  com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.internal.shaded.org.jctools.queues.MpscChunkedArrayQueue
  31:         17411         696440  java.util.TreeMap$Entry (java.base@17.0.17)
  32:         11997         671832  java.nio.HeapCharBuffer (java.base@17.0.17)
  33:         17762         568384  java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode (java.base@17.0.17)
  34:          7839         564408  java.lang.reflect.Constructor (java.base@17.0.17)
  35:          1862         551152  io.swagger.v3.oas.models.media.StringSchema
  36:         17183         549856  java.lang.ref.WeakReference (java.base@17.0.17)
  37:         11205         537840  com.fasterxml.jackson.core.json.JsonWriteContext
  38:          8265         528960  java.util.stream.IntPipeline$10 (java.base@17.0.17)
  39:          6589         500904  [Ljava.lang.reflect.Method; (java.base@17.0.17)
  40:         20385         489240  java.util.Collections$UnmodifiableRandomAccessList (java.base@17.0.17)
  41:          8701         487256  java.nio.HeapByteBuffer (java.base@17.0.17)
  42:         20288         486912  org.springframework.util.ConcurrentReferenceHashMap$Entry
  43:          8265         462840  java.util.stream.IntPipeline$Head (java.base@17.0.17)
  44:          7999         447944  java.util.concurrent.ConcurrentHashMap$KeyIterator (java.base@17.0.17)
  45:         17876         429024  sun.reflect.generics.tree.SimpleClassTypeSignature (java.base@17.0.17)
  46:         17757         426168  java.util.concurrent.ConcurrentLinkedQueue$Node (java.base@17.0.17)
  47:          6536         418304  java.util.concurrent.ConcurrentHashMap (java.base@17.0.17)
  48:          8508         408384  java.util.TreeMap (java.base@17.0.17)



root@db:/home/db# jstat -gccause 3674535 1 1
  S0     S1     E      O      M     CCS    YGC     YGCT     FGC    FGCT     CGC    CGCT       GCT    LGCC                 GCC
  0.00  48.15  34.47  63.86  99.31  96.97    309     2.641     0     0.000    10     0.027     2.667 G1 Evacuation Pause  No GC
root@db:/home/db# jstat -gccause 3674535 1 1
  S0     S1     E      O      M     CCS    YGC     YGCT     FGC    FGCT     CGC    CGCT       GCT    LGCC                 GCC
  0.00  48.15  34.78  63.86  99.31  96.97    309     2.641     0     0.000    10     0.027     2.667 G1 Evacuation Pause  No GC
root@db:/home/db# jstat -gccause 3674535 1 1
  S0     S1     E      O      M     CCS    YGC     YGCT     FGC    FGCT     CGC    CGCT       GCT    LGCC                 GCC
  0.00  48.15  34.78  63.86  99.31  96.97    309     2.641     0     0.000    10     0.027     2.667 G1 Evacuation Pause  No GC
root@db:/home/db# jstat -gccause 3674535 1 1
  S0     S1     E      O      M     CCS    YGC     YGCT     FGC    FGCT     CGC    CGCT       GCT    LGCC                 GCC
  0.00  48.15  35.09  63.86  99.31  96.97    309     2.641     0     0.000    10     0.027     2.667 G1 Evacuation Pause  No GC
root@db:/home/db# jstat -gccause 3674535 1 1
  S0     S1     E      O      M     CCS    YGC     YGCT     FGC    FGCT     CGC    CGCT       GCT    LGCC                 GCC
  0.00  48.15  35.40  63.86  99.31  96.97    309     2.641     0     0.000    10     0.027     2.667 G1 Evacuation Pause  No GC
root@db:/home/db#
关键命令:
jstat -gc 3674535 1 1
jstat -gccause 3674535 1 1
jmap -histo 3674535 |head -50

本来觉得system服务有大量的用户部门权限相关接口,目前jar配置的-Xmx512m -Xms512m可能不够用,分析了一下运行时的内存情况好像没什么问题。
背景:
1.G1默认S(幸存区)不怎么用,E(伊甸园)比O(老年代)为3比2左右(329728.0:193536.0)
2.G1基本由CGC代替FGC,避免StopTheWorld
3.JDK8开始M(元空间原永久代)移至堆外,可以看到S1C+EC+OC=512M
分析:
无负载情况下基本堆占用在329728.0*35+193536.0*0.63,基本小于百分50,有时:
0.2(E)+0.63(O)
0.7(E)+0.63(O)
E会快速回收O未超过70%使用率,情况良好
服务无高并发需求,目前够用docker内存分配需要至少1G(堆+元空间+系统+线程+其他)以上

jmap情况
类占用内存情况基本没问题,都是基础类型
com.alibaba.csp.sentinel.node.metric.MetricNode类有点超纲,不过持续观察下问题不大,增长快但是回收也快,性能也没什么影响cpu占用基本5%以下,之前system服务基本cpu在15%以上,当时没时间看,只看了下jstack大概知道是seata的问题,后面又调整了一下maven依赖,现在变正常了,没机会确认那个问题了

查看堆与堆外内存占用情况

# JVM 启动参数
-XX:NativeMemoryTracking=summary -XX:+UnlockDiagnosticVMOptions
# 查看占用
jcmd <pid> VM.native_memory summary
结果:
Native Memory Tracking:

Total: 1048576000 bytes (1000 MB)
JVM: 512000000
Class: 100000000
Thread: 400000000
Code: 50000000
GC: 20000000
Compiler: 10000000
Internal: 10000000
Symbol: 5000000
Arena Chunk: 50000000

分析:
Heap (Xmx/Xms) 对象实例,G1 分代 512MB
Metaspace (M / CCS) 类元数据、反射信息 50–200MB 甚至更多
Thread stacks 每个线程栈 (默认 1MB) 线程数 * 1MB,比如 200 线程 → 200MB
DirectByteBuffer / NIO NIO buffer、netty buffer 可能几十到几百 MB
JIT 编译 / Code cache HotSpot 编译后的机器码 20–100MB
Native libraries JNI 调用 不定
GC internals / humongous overhead G1 内部管理 几十 MB

发表回复

您的电子邮箱地址不会被公开。 必填项已用 * 标注