The high CPU usage of a node due to dockerd-current could be due to corrupt docker log files. The steps below outline how to clear the corrupted log files and reduce the CPU usage.
Follow the steps to reduce high cpu usage
Create a step-by-step guide
Step 1: SSH into the node
ssh <node name>
Step 2: Change user to sudo
sudo -i
Step 3: To check what process have a high CPU usage
top
Step 4: To find the corrupt docker file logs
- Run the command below:
- All corrupt docker logs will be shown.
find /var/lib/docker/containers/ -name *-json.log -exec bash -c 'jq '.' {} > /dev/null 2>&1 || echo "{}"' \;
Step 5: Store corrupt docker files in a file
- Run the command below:
find /var/lib/docker/containers/ -name *-json.log -exec bash -c 'jq '.' {} > /dev/null 2>&1 || echo "{}"' \; > /tmp/rm.txt
Step 6: Remove the docker logs stored in the file
- Run the command below:
- To check if all the logs have been removed, run the command in step 4 again. There should be no logs shown.
for i in $(cat /tmp/rm.txt) ; do rm -rf $i ; done
Step 7: Restart docker
- Before restarting docker, the live-restore needs to be enabled for docker. Use the command below to check:
- NOTE: If live-restore is not activated and you restart docker, all running containers will be stopped and not restarted. Live-restore keeps containers running while the docker daemon is unavailable.
- Here is a link to enable live-restore if not enabled: https://docs.docker.com/config/containers/live-restore/
docker info
Use the command below to restart docker
systemctl restart docker
Check the status of docker to make sure it has restarted
systemctl status docker
Check status of CPU usage using the top command again. It will take some time for the CPU to decrease but it will eventually.
top
Source: https://forums.docker.com/t/dockerd-using-100-cpu/94962