The high CPU usage of a node due to dockerd-current could be due to corrupt docker log files. The steps below outline how to clear the corrupted log files and reduce the CPU usage.


Follow the steps to reduce high cpu usage


Create a step-by-step guide


Step 1: SSH into the node

ssh <node name>


Step 2: Change user to sudo

sudo -i


Step 3: To check what process have a high CPU usage

  • The command below list the process on the node that have a high CPU usage. Dockerd-current CPU will be very high, maybe above 100%. This issue could be due to corrupt files in the docker logs.

top

Step 4: To find the corrupt docker file logs

  • Run the command below:
  • All corrupt docker logs will be shown.
find /var/lib/docker/containers/ -name *-json.log -exec bash -c 'jq '.' {} > /dev/null 2>&1 || echo "{}"' \;


Step 5: Store corrupt docker files in a file

  • Run the command below:
find /var/lib/docker/containers/ -name *-json.log -exec bash -c 'jq '.' {} > /dev/null 2>&1 || echo "{}"' \; > /tmp/rm.txt

Step 6: Remove the docker logs stored in the file

  • Run the command below:
  • To check if all the logs have been removed, run the command in step 4 again. There should be no logs shown.
for i in $(cat /tmp/rm.txt) ; do rm -rf  $i ; done

Step 7: Restart docker

  • Before restarting docker, the live-restore needs to be enabled for docker. Use the command below to check:
  • NOTE: If live-restore is not activated and you restart docker, all running containers will be stopped and not restarted. Live-restore keeps containers running while the docker daemon is unavailable.
  • Here is a link to enable live-restore if not enabled: https://docs.docker.com/config/containers/live-restore/
docker info
  • Use the command below to restart docker

systemctl restart docker
  • Check the status of docker to make sure it has restarted

systemctl status docker
  • Check status of CPU usage using the top command again. It will take some time for the CPU to decrease but it will eventually.

top

Source: https://forums.docker.com/t/dockerd-using-100-cpu/94962