Add persistent storage to prometheus

- add healthcheck to ensure prometheus service is running correctly and
restarts it if it becomes unhealthy
- set the data retention time to 30d
- update readme

Issue-ID: CPS-2466

Change-Id: I0f6186f8577e8e0113831ed040298b823e33d624
Signed-off-by: halil.cakal <halil.cakal@est.tech>

3 files changed, 42 insertions, 4 deletions

diff --git a/docker-compose/README.md b/docker-compose/README.md
index de1fbe296c..ea5a2a7610 100644
--- a/docker-compose/README.md
+++ b/docker-compose/README.md
@@ -1,7 +1,7 @@
 <!--
   ============LICENSE_START=======================================================
    Copyright (C) 2020 Pantheon.tech
-   Modifications Copyright (C) 2020-2021 Nordix Foundation.
+   Modifications Copyright (C) 2020-2024 Nordix Foundation.
    Modifications Copyright (C) 2021 Bell Canada.
    Modifications Copyright (C) 2022 TechMahindra Ltd.
   ================================================================================
@@ -67,6 +67,34 @@ or
 VERSION=<version> DB_USERNAME=cps DB_PASSWORD=cps docker-compose up -d
 ```
 
+## Running Docker containers with profile: monitoring
+
+Run docker-compose with profile, monitoring, then it will start monitoring services:
+* prometheus
+* grafana
+* kafka-ui
+
+```bash
+docker-compose --profile monitoring up -d
+```
+
+### prometheus service
+It collects and stores metrics as time series data, recording information with a timestamp.
+
+The environment variable, PROMETHEUS_RETENTION_TIME, is used to set the retention time for the metrics
+in the prometheus database. The default value is 15d, but can be changed to any value.
+
+To be able to use the historical data, the prometheus container should not be removed.
+Instead, it can be stopped and started using the following commands:
+
+```bash
+docker-compose start prometheus
+```
+
+```bash
+docker-compose stop prometheus
+```
+
 ## Running or debugging Java built code
 
 Before running CPS, a Postgres database instance needs to be started. This can be done with following
diff --git a/docker-compose/docker-compose.yml b/docker-compose/docker-compose.yml
index 604cfe91a2..d2eff8da2b 100644
--- a/docker-compose/docker-compose.yml
+++ b/docker-compose/docker-compose.yml
@@ -178,12 +178,19 @@ services:
     restart: always
     volumes:
       - ./config/prometheus.yml:/etc/prometheus/prometheus.yml
+      - prometheus_data:/prometheus
+    environment:
+      - PROMETHEUS_RETENTION_TIME=${PROMETHEUS_RETENTION_TIME:-30d}
+    healthcheck:
+      test: [ "CMD-SHELL", "wget --spider --quiet --tries=1 --timeout=10 http://localhost:9090/-/healthy || exit 1" ]
+      interval: 30s
+      timeout: 10s
+      retries: 3
     profiles:
       - monitoring
 
   grafana:
     image: grafana/grafana-oss:latest
-    user: ""
     container_name: ${GRAFANA_CONTAINER_NAME:-grafana}
     depends_on:
       prometheus:
@@ -196,7 +203,7 @@ services:
       - grafana:/var/lib/grafana
     environment:
       - GF_SECURITY_ADMIN_PASSWORD=admin
-      - GF_SERVER_DOMAIN:localhost
+      - GF_SERVER_DOMAIN=localhost
     profiles:
       - monitoring
 
@@ -223,3 +230,6 @@ services:
 
 volumes:
   grafana:
+    driver: local
+  prometheus_data:
+    driver: local
diff --git a/k6-tests/run-k6-tests.sh b/k6-tests/run-k6-tests.sh
index 8c4048ba8c..74021ec28b 100755
--- a/k6-tests/run-k6-tests.sh
+++ b/k6-tests/run-k6-tests.sh
@@ -20,7 +20,7 @@ set -o nounset  # Disallow expansion of unset variables
 set -o pipefail # Use last non-zero exit code in a pipeline
 #set -o xtrace   # Uncomment for debugging
 
-# default is empty string, which means performance tests
+# Default test profile is kpi.
 testProfile=${1:-kpi}
 
 on_exit() {