Linux服务器CPU使用率过高排查与解决思路



日常工作中有时候会遇到Linux服务器上出现CPU负载过高的情况，影响程序的运行，一般的排除思路如下：

方法一：
使用top命令

然后按shift+p按照CPU排序，找到占用CPU过高的进程的pid

root@zph-IdeaCentre-GeekPro-14IOB:/home/git# top

top - 15:18:21 up 10 days, 6:07, 1 user, load average: 1.36, 1.55, 1.32
任务: 546 total, 1 running, 544 sleeping, 0 stopped, 1 zombie
%Cpu(s): 4.1 us, 1.4 sy, 0.0 ni, 94.4 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st
MiB Mem : 15783.1 total, 225.3 free, 9583.5 used, 5974.3 buff/cache
MiB Swap: 2048.0 total, 923.0 free, 1125.0 used. 5403.6 avail Mem

进程号 USER PR NI VIRT RES SHR %CPU %MEM TIME+ COMMAND
1194120 zph 20 0 2675864 163884 27852 S 37.7 1.0 801:42.88 resp
1187764 zph 20 0 10.4g 247872 85600 S 11.3 1.5 204:42.53 ynote-desktop
4005 zph 20 0 7685196 742284 231580 S 9.9 4.6 998:07.78 gnome-shell
1187524 zph 20 0 16.6g 195948 157956 S 8.3 1.2 24:24.30 chrome
1187183 zph 20 0 17.0g 343324 187812 S 4.6 2.1 28:38.91 chrome
3803 root 20 0 864388 558572 261488 S 4.3 3.5 193:00.09 Xorg
1850081 zph 20 0 2736288 83572 27296 S 3.6 0.5 9:02.04 ShiYeLine.exe
1662680 zph 20 0 3778160 323532 79952 S 2.3 2.0 39:23.77 sunloginclient
1723352 zph 20 0 14916 13216 1492 S 2.0 0.1 32:03.67 wineserver64
2.使用top -H -p [进程id]

找到进程中消耗资源最高的线程的id

root@zph-IdeaCentre-GeekPro-14IOB:/home/git# top -H -p 1194120

top - 15:21:30 up 10 days, 6:10, 1 user, load average: 2.09, 1.58, 1.36
Threads: 18 total, 1 running, 17 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.6 us, 0.6 sy, 0.0 ni, 96.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 15783.1 total, 253.0 free, 9591.9 used, 5938.3 buff/cache
MiB Swap: 2048.0 total, 923.0 free, 1125.0 used. 5411.6 avail Mem

进程号 USER PR NI VIRT RES SHR %CPU %MEM TIME+ COMMAND
1885928 zph 20 0 2675864 163892 27852 R 8.3 1.0 0:00.25 qredisclient::t
1194120 zph 20 0 2675864 163892 27852 S 0.3 1.0 9:04.88 resp
1194259 zph 20 0 2675864 163892 27852 S 0.0 1.0 0:16.59 QXcbEventQueue
1194260 zph 20 0 2675864 163892 27852 S 0.0 1.0 0:00.00 QDBusConnection
1194261 zph 20 0 2675864 163892 27852 S 0.0 1.0 0:00.03 QQmlThread
1194278 zph 20 0 2675864 163892 27852 S 0.0 1.0 0:00.00 Qt bearer threa
1194279 zph 20 0 2675864 163892 27852 S 0.0 1.0 0:00.25 QPythonWorker
1194280 zph 20 0 2675864 163892 27852 S 0.0 1.0 0:00.01 QQuickPixmapRea
1199359 zph 20 0 2675864 163892 27852 S 0.0 1.0 0:00.00 qredisclient::t
1294047 zph 20 0 2675864 163892 27852 S 0.0 1.0 0:15.49 qredisclient::t
1420987 zph 20 0 2675864 163892 27852 S 0.0 1.0 0:00.00 qredisclient::t
1425577 zph 20 0 2675864 163892 27852 S 0.0 1.0 0:00.01 qredisclient::t
1425582 zph 20 0 2675864 163892 27852 S 0.0 1.0 0:00.00 qredisclient::t
1686855 zph 20 0 2675864 163892 27852 S 0.0 1.0 0:00.00 qredisclient::t
1701340 zph 20 0 2675864 163892 27852 S 0.0 1.0 0:00.01 qredisclient::t
1701429 zph 20 0 2675864 163892 27852 S 0.0 1.0 0:00.00 qredisclient::t
1839931 zph 20 0 2675864 163892 27852 S 0.0 1.0 0:00.95 qredisclient::t
1885932 zph 20 0 2675864 163892 27852 S 0.0 1.0 0:00.00 qredisclient::t
3.使用echo ‘obase=16;[线程id]’ | bc或者printf “%x\n” [线程id]

将需要的线程ID转换为16进制格式

bc是linux的计算器命令

root@zph-IdeaCentre-GeekPro-14IOB:/home/git# printf “%x\n” 1194120
“123888n”
4.执行jstack [进程id] |grep -A 30 [线程id的16进制]”

查看线程状态信息
grep -A
30:
-A #：显示匹配到的字符串所在的行以及其后 # 行(After)
-B #：显示匹配到的字符串所在的行以及前 # 行(Before)
-C #：显示匹配到的字符串所在的行以及前后各 # 行(Contextt)

jstack 1194120|grep '3eb7' -A 10
方法二：
使用top命令

然后按shift+p按照CPU排序，找到占用CPU过高的进程的pid

root@zph-IdeaCentre-GeekPro-14IOB:/home/git# top

top - 15:18:21 up 10 days, 6:07, 1 user, load average: 1.36, 1.55, 1.32
任务: 546 total, 1 running, 544 sleeping, 0 stopped, 1 zombie
%Cpu(s): 4.1 us, 1.4 sy, 0.0 ni, 94.4 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st
MiB Mem : 15783.1 total, 225.3 free, 9583.5 used, 5974.3 buff/cache
MiB Swap: 2048.0 total, 923.0 free, 1125.0 used. 5403.6 avail Mem

进程号 USER PR NI VIRT RES SHR %CPU %MEM TIME+ COMMAND
1194120 zph 20 0 2675864 163884 27852 S 37.7 1.0 801:42.88 resp
1187764 zph 20 0 10.4g 247872 85600 S 11.3 1.5 204:42.53 ynote-desktop
4005 zph 20 0 7685196 742284 231580 S 9.9 4.6 998:07.78 gnome-shell
1187524 zph 20 0 16.6g 195948 157956 S 8.3 1.2 24:24.30 chrome
1187183 zph 20 0 17.0g 343324 187812 S 4.6 2.1 28:38.91 chrome
3803 root 20 0 864388 558572 261488 S 4.3 3.5 193:00.09 Xorg
1850081 zph 20 0 2736288 83572 27296 S 3.6 0.5 9:02.04 ShiYeLine.exe
1662680 zph 20 0 3778160 323532 79952 S 2.3 2.0 39:23.77 sunloginclient
1723352 zph 20 0 14916 13216 1492 S 2.0 0.1 32:03.67 wineserver64
2.使用ps -mp pid -o THREAD,tid,time | sort -rn

获取线程信息，并找到占用CPU高的线程

root@zph-IdeaCentre-GeekPro-14IOB:/home/git# ps -mp 1187524 -o THREAD,tid,time | sort -rn
zph 0.5 19 - do_sys - - 1187547 00:24:17
zph 0.5 - - - - - - 00:27:44
zph 0.0 19 - futex_ - - 1208102 00:00:00
zph 0.0 19 - futex_ - - 1187544 00:00:00
zph 0.0 19 - futex_ - - 1187542 00:00:00
zph 0.0 19 - futex_ - - 1187541 00:00:00
zph 0.0 19 - futex_ - - 1187540 00:00:00
zph 0.0 19 - futex_ - - 1187539 00:00:00
zph 0.0 19 - futex_ - - 1187538 00:00:00
zph 0.0 19 - futex_ - - 1187537 00:00:00
zph 0.0 19 - futex_ - - 1187536 00:00:00
zph 0.0 19 - futex_ - - 1187535 00:00:00
zph 0.0 19 - futex_ - - 1187534 00:00:00
zph 0.0 19 - futex_ - - 1187533 00:00:00
zph 0.0 19 - futex_ - - 1187532 00:00:00
zph 0.0 19 - futex_ - - 1187531 00:00:00
zph 0.0 19 - futex_ - - 1187530 00:00:00
zph 0.0 19 - futex_ - - 1187529 00:00:00
zph 0.0 19 - futex_ - - 1187528 00:00:00
zph 0.0 19 - futex_ - - 1187527 00:00:00
zph 0.0 19 - futex_ - - 1187526 00:00:00
zph 0.0 19 - ep_pol - - 1187545 00:03:12
zph 0.0 19 - ep_pol - - 1187543 00:00:00
zph 0.0 19 - do_sys - - 1187524 00:00:12

3.使用echo ‘obase=16;[线程id]’ | bc或者printf “%x\n” [线程id]

将需要的线程ID转换为16进制格式

root@zph-IdeaCentre-GeekPro-14IOB:/home/git# printf “%x\n” 1194120
“123888n”
4.执行jstack [进程id] |grep -A 30 [线程id的16进制]”

查看线程状态信息
grep -A
30:
-A #：显示匹配到的字符串所在的行以及其后 # 行(After)
-B #：显示匹配到的字符串所在的行以及前 # 行(Before)
-C #：显示匹配到的字符串所在的行以及前后各 # 行(Contextt)

jstack 1194120|grep '3eb7' -A 10