How to change the log retention period for Prometheus to address alerts such as PersistentVolumeUsageNearFull.


Before changing the long retention period, first check with the client their required log retention period for reporting.


Reducing the retention period.


Step 1: Select the project name

[lsd@prd-ochelper2 ~]$ oc project lsdobserve
Now using project "lsdobserve" on

Step 2: Identify pod of interest (lsdobserve-prometheus-server) 

[lsd@prd-ochelper2 ~]$ oc describe pvc storage-volume-lsdobserve-prometheus-server-0 | grep "Used By"
Used By:       lsdobserve-prometheus-server-0

Step 3: RSH into the pod and df -h to view the filesystem

[lsd@prd-ochelper2 ~]$ oc rsh -c prometheus-server lsdobserve-prometheus-server-0
/prometheus $ df -h
Filesystem                Size      Used Available Use% Mounted on
...
/dev/rbd0                49.0G     35.9G     13.1G  78% /data
...

Step 4: Edit the prometheus statefulset

[lsd@prd-ochelper2 ~]$ oc edit statefulset.apps/lsdobserve-prometheus-server

Look at the pod command args and change the value specified for --storage.tsdb.retention.time=

- args:
  - --storage.tsdb.retention.time=10d
  - --config.file=/etc/config/prometheus.yml
  - --storage.tsdb.path=/data
  - --web.console.libraries=/etc/prometheus/console_libraries
  - --web.console.templates=/etc/prometheus/consoles
  - --web.enable-lifecycle

Adjust the retention as needed


Step 5: Validate the pod is starting up and db is being reconciled

oc logs -c prometheus-server lsdobserve-prometheus-server-0

exec into pod and validate the space has  been reduced

[lsd@prd-ochelper2 ~]$ oc rsh -c prometheus-server lsdobserve-prometheus-server-0
/prometheus $ df -h

you can also check the pod to ensure the retention period changes have carried through to the pod