diff options
| -rw-r--r-- | docs/development/devtools/testing/s3p/clamp-s3p-results/Stability_after_stats.png | bin | 123032 -> 52736 bytes | |||
| -rw-r--r-- | docs/development/devtools/testing/s3p/clamp-s3p-results/acm_performance_jmeter.png | bin | 229066 -> 80792 bytes | |||
| -rw-r--r-- | docs/development/devtools/testing/s3p/clamp-s3p-results/acm_stability_jmeter.png | bin | 218877 -> 93200 bytes | |||
| -rw-r--r-- | docs/development/devtools/testing/s3p/clamp-s3p-results/acm_stability_table.png | bin | 454197 -> 213035 bytes | |||
| -rw-r--r-- | docs/development/devtools/testing/s3p/clamp-s3p.rst | 208 |
5 files changed, 80 insertions, 128 deletions
diff --git a/docs/development/devtools/testing/s3p/clamp-s3p-results/Stability_after_stats.png b/docs/development/devtools/testing/s3p/clamp-s3p-results/Stability_after_stats.png Binary files differindex 38242866..e53641f5 100644 --- a/docs/development/devtools/testing/s3p/clamp-s3p-results/Stability_after_stats.png +++ b/docs/development/devtools/testing/s3p/clamp-s3p-results/Stability_after_stats.png diff --git a/docs/development/devtools/testing/s3p/clamp-s3p-results/acm_performance_jmeter.png b/docs/development/devtools/testing/s3p/clamp-s3p-results/acm_performance_jmeter.png Binary files differindex bad1cf71..ae3308d1 100644 --- a/docs/development/devtools/testing/s3p/clamp-s3p-results/acm_performance_jmeter.png +++ b/docs/development/devtools/testing/s3p/clamp-s3p-results/acm_performance_jmeter.png diff --git a/docs/development/devtools/testing/s3p/clamp-s3p-results/acm_stability_jmeter.png b/docs/development/devtools/testing/s3p/clamp-s3p-results/acm_stability_jmeter.png Binary files differindex 2f576505..3721435e 100644 --- a/docs/development/devtools/testing/s3p/clamp-s3p-results/acm_stability_jmeter.png +++ b/docs/development/devtools/testing/s3p/clamp-s3p-results/acm_stability_jmeter.png diff --git a/docs/development/devtools/testing/s3p/clamp-s3p-results/acm_stability_table.png b/docs/development/devtools/testing/s3p/clamp-s3p-results/acm_stability_table.png Binary files differindex 28942eff..6401eb1d 100644 --- a/docs/development/devtools/testing/s3p/clamp-s3p-results/acm_stability_table.png +++ b/docs/development/devtools/testing/s3p/clamp-s3p-results/acm_stability_table.png diff --git a/docs/development/devtools/testing/s3p/clamp-s3p.rst b/docs/development/devtools/testing/s3p/clamp-s3p.rst index eb17d894..dc0b5998 100644 --- a/docs/development/devtools/testing/s3p/clamp-s3p.rst +++ b/docs/development/devtools/testing/s3p/clamp-s3p.rst @@ -11,63 +11,35 @@ Policy Clamp Automation Composition ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Both the Performance and the Stability tests were executed by performing requests -against acm components installed as docker images in local environment. +against acm components installed as docker images in local environment. These tests we all +performed on a Ubuntu VM with 32GB of memory, 16 CPU and 50GB of disk space. ACM Deployment ++++++++++++++ -The docker containers can be deployed via Policy CSIT script. -Clone the Policy/docker repo to the local vm +In an effort to allow the execution of the s3p tests to be as close to automatic as possible, +a script will be executed that will perform the following: -.. code-block:: bash - - git clone "https://gerrit.onap.org/r/policy/docker" - -Set the following environment variables on the system before deploying the containers. - -.. code-block:: bash - - export CONTAINER_LOCATION=nexus3.onap.org:10001/ - export PROJECT=clamp - -Invoke the following script from the ~/docker/csit folder. - -.. code-block:: bash +- Install of a microk8s kubernetes environment +- Bring up the policy components +- Checks that the components are successfully up and running before proceeding +- Install Java 17 +- Install Jmeter locally and configure it +- Specify whether you want to run stability or performance tests - ./start-all.sh -This script installs the docker containers of ACM and Policy components required for running the tests. +The remainder of this document outlines how to run the tests and the test results - -Jmeter setup +Common Setup ++++++++++++ +The common setup for performance and stability tests is now automated - being carried out by a script in- **testsuites/run-s3p-test.sh**. -Apache jmeter tool is installed either on the same virtual machine or on a different virtual machine. +Clone the policy-clamp repo to access the test scripts .. code-block:: bash - # Install required packages - sudo apt install -y wget unzip - - # Install JMeter - mkdir -p jmeter - cd jmeter - wget https://dlcdn.apache.org//jmeter/binaries/apache-jmeter-5.5.zip # check if valid version - unzip -q apache-jmeter-5.5.zip - rm apache-jmeter-5.5.zip - - -Setup Verification -++++++++++++++++++ -Ensure the following components are up and running before executing the test. - -- acm runtime component docker image is started and running. -- Participant docker images policy-clamp-cl-pf-ppnt, policy-clamp-cl-http-ppnt, policy-clamp-cl-k8s-ppnt are started and running. -- Dmaap simulator for communication between components. -- mariadb docker container for policy and clampacm database. -- policy-api for communication between policy participant and policy-framework -- Both tests were run via jMeter, which was installed on a separate VM. + git clone https://gerrit.onap.org/r/policy/clamp Stability Test of acm components ++++++++++++++++++++++++++++++++ @@ -76,46 +48,34 @@ Test Plan --------- The 72 hours stability test ran the following steps sequentially in a single threaded loop. -- **Create Policy defaultDomain** - creates an operational policy using policy/api component -- **Delete Policy sampleDomain** - deletes the operational policy sampleDomain using policy/api component -- **Commission AC definition** - commissions the acm definition in runtime -- **Instantiate acm** - Instantiate the acm towards participants -- **Check acm state** - check the current state of acm -- **Change State to PASSIVE** - change the state of the acm to PASSIVE -- **Check acm state** - check the current state of acm -- **Change State to UNINITIALISED** - change the state of the ACM to UNINITIALISED -- **Check acm state** - check the current state of acm -- **Delete instantiated acm** - delete the instantiated acm from all participants -- **Delete ACM Definition** - delete the acm definition on runtime - -The following parameters can be configured on the JMX file for the test. - -- **HTTP Authorization Manager** - used to store user/password authentication details. -- **HTTP Header Manager** - used to store headers which will be used for making HTTP requests. -- **User Defined Variables** - used to store following user defined parameters. - -============================= ======================================================================== - **Name** **Description** -============================= ======================================================================== - RUNTIME_HOST IP Address or host name of acm runtime component - RUNTIME_PORT Port number of acm runtime components for making REST API calls - POLICY_PARTICIPANT_HOST IP Address or host name of policy participant - POLICY_PARTICIPANT_HOST_PORT Port number of policy participant -============================= ======================================================================== - -Download the ACM stability.jmx and performance.jmx files from the Policy-Clamp repo. - -Stability jmx file +- **Commission Automation Composition Definitions** - Commissions the ACM Definitions +- **Register Participants** - Registers the presence of participants in the acm database +- **Prime AC definition** - Primes the AC Definition to the participants +- **Instantiate acm** - Instantiate the acm instance +- **DEPLOY the ACM instance** - change the state of the acm to DEPLOYED +- **Check instance state** - check the current state of instance and that it is DEPLOYED +- **UNDEPLOY the ACM instance** - change the state of the ACM to UNDEPLOYED +- **Check instance state** - check the current state of instance and that it is UNDEPLOYED +- **Delete instance** - delete the instance from all participants and ACM db +- **DEPRIME ACM definitions** - DEPRIME ACM definitions from participants +- **Delete ACM Definition** - delete the ACM definition on runtime + +This runs for 72 hours. Test results are present in the **testsuites/automated-performance/s3pTestResults.jtl** +directory. Logs are present for jmeter in **testsuites/automated-performance/jmeter.log** and +**testsuites/automated-performance/nohup.out** -.. code-block:: bash - - ~/clamp/testsuites/stability/src/main/resources/testplans/stability.jmx +Run Test +-------- -The test was run in the background via "nohup", to prevent it from being interrupted: +The code in the setup section also serves to run the tests. Just one execution needed to do it all. .. code-block:: bash - nohup ./jmeter/apache-jmeter-5.5/bin/jmeter -n -t stability.jmx -l testresults.jtl + bash run-s3p-test.sh run stability + +Once the test execution is completed, the results are present in the **automate-performance/s3pTestResults.jtl** file. + +This file can be imported into the Jmeter GUI for visualization. The below results are tabulated from the GUI. Test Results ------------ @@ -124,37 +84,31 @@ Test Results Stability test plan was triggered for 72 hours. -.. Note:: - - .. container:: paragraph - - The assertions of state changes are not completely taken care of, as the stability is ran with acm components - alone, and not including complete policy framework deployment, which makes it difficult for actual state changes from - PASSIVE to RUNNING etc to happen. - **Test Statistics** ======================= ================= ================== ================================== **Total # of requests** **Success %** **Error %** **Average time taken per request** ======================= ================= ================== ================================== -97916 100.00 % 0.00 % 246 ms +260590 100.00 % 0.00 % 997 ms ======================= ================= ================== ================================== **ACM component Setup** -================ ============================================================ =========================================== ========================= -**CONTAINER ID** **IMAGE** **PORT** **NAME** -================ ============================================================ =========================================== ========================= - a9cb0cd103cf nexus3.onap.org:10001/onap/policy-clamp-runtime-acm:latest 6969/tcp policy-clamp-runtime-acm - 886e572b8438 nexus3.onap.org:10001/onap/policy-clamp-ac-pf-ppnt:latest 6969/tcp policy-clamp-ac-pf-ppnt - 035707b1b95f nexus3.onap.org:10001/onap/policy-api:latest 6969/tcp policy-api - d34204f95ff3 nexus3.onap.org:10001/onap/policy-clamp-ac-http-ppnt:latest 6969/tcp policy-clamp-ac-http-ppnt - 4470e608c9a8 nexus3.onap.org:10001/onap/policy-clamp-ac-k8s-ppnt:latest 6969/tcp policy-clamp-ac-k8s-ppnt - 62229d46b79c nexus3.onap.org:10001/onap/policy-models-simulator:latest 3905/tcp, 6666/tcp, 6668-6670/tcp, 6680/tcp simulator - efaf0ca5e1f0 nexus3.onap.org:10001/mariadb:10.5.8 3306/tcp mariadb - e84cf17db2a4 nexus3.onap.org:10001/onap/policy-pap:latest 6969/tcp policy-pap - 0a16eecd13c9 nexus3.onap.org:10001/onap/policy-apex-pdp:latest 6969/tcp policy-apex-pdp -================ ============================================================ =========================================== ========================= +============================================== ============================================================ =========================================== +**NAME** **IMAGE** **PORT** +============================================== ============================================================ =========================================== + policy-clamp-runtime-acm-5c6d8fbfb-jz8rb nexus3.onap.org:10001/onap/policy-clamp-runtime-acm:latest 30007/tcp + policy-clamp-ac-pf-ppnt-55c4cb99f4-spvng nexus3.onap.org:10001/onap/policy-clamp-ac-pf-ppnt:latest 30008/tcp + policy-api-58cb45fc9b-ff5md nexus3.onap.org:10001/onap/policy-api:latest 30002/tcp + policy-clamp-ac-http-ppnt-7b99cbfbf8-d4w9v nexus3.onap.org:10001/onap/policy-clamp-ac-http-ppnt:latest 30009/tcp + policy-clamp-ac-k8s-ppnt-6d854cc8b6-twdkh nexus3.onap.org:10001/onap/policy-clamp-ac-k8s-ppnt:latest 30010/tcp + policy-models-simulator-bcd494d87-bfg6g nexus3.onap.org:10001/onap/policy-models-simulator:latest 30904/tcp + mariadb-galera-0 nexus3.onap.org:10001/mariadb:10.5.8 3306/tcp + policy-pap-847d89997d-x9h99 nexus3.onap.org:10001/onap/policy-pap:latest 30003/tcp + policy-apex-pdp-0 nexus3.onap.org:10001/onap/policy-apex-pdp:latest 6969/tcp +============================================== ============================================================ =========================================== + + .. Note:: @@ -190,8 +144,15 @@ Performance test of acm components has the goal of testing the min/avg/max proce Setup Details ------------- -The performance test is performed on a similar setup as Stability test. The JMeter VM will be sending a large number of REST requests to the runtime component and collecting the statistics. +We can setup the environment and execute the tests like this from the **clamp/testsuites** directory +.. code-block:: bash + + bash run-s3p-test.sh run performance + +This runs for 2 hours. Test results are present in the **testsuites/automate-performance/s3pTestResults.jtl** +directory. Logs are present for jmeter in **testsuites/automate-performance/jmeter.log** and +**testsuites/automated-performance/nohup.out** Test Plan --------- @@ -204,24 +165,15 @@ Performance test plan is the same as the stability test plan above except for th Run Test -------- -Performance jmx file - -.. code-block:: bash - - ~/clamp/testsuites/performance/src/main/resources/testplans/performance.jmx - -Running/Triggering the performance test will be the same as the stability test. That is, launch JMeter pointing to corresponding *.jmx* test plan. The *RUNTIME_HOST*, *RUNTIME_PORT*, *POLICY_PARTICIPANT_HOST*, *POLICY_PARTICIPANT_HOST_PORT* are already set up in *.jmx* +The code in the setup section also serves to run the tests. Just one execution needed to do it all. .. code-block:: bash - nohup ./jmeter/apache-jmeter-5.5/bin/jmeter -n -t performance.jmx -l testresults.jtl + bash run-s3p-test.sh run performance -Once the test execution is completed, execute the below script to get the statistics: - -.. code-block:: bash +Once the test execution is completed, the results are present in the **automate-performance/s3pTestResults.jtl** file. - $ cd ./clamp/testsuites/performance/src/main/resources/testplans - $ ./results.sh resultTree.log +This file can be imported into the Jmeter GUI for visualization. The below results are tabulated from the Jmeter GUI. Test Results ------------ @@ -233,24 +185,24 @@ Test results are shown as below. ======================= ================= ================== ================================== **Total # of requests** **Success %** **Error %** **Average time taken per request** ======================= ================= ================== ================================== -13591 100 % 0.00 % 249 ms +15520 100 % 0.00 % 464 ms ======================= ================= ================== ================================== **ACM component Setup** -================ ============================================================ =========================================== ========================= -**CONTAINER ID** **IMAGE** **PORT** **NAME** -================ ============================================================ =========================================== ========================= - a9cb0cd103cf nexus3.onap.org:10001/onap/policy-clamp-runtime-acm:latest 6969/tcp policy-clamp-runtime-acm - 886e572b8438 nexus3.onap.org:10001/onap/policy-clamp-ac-pf-ppnt:latest 6969/tcp policy-clamp-ac-pf-ppnt - 035707b1b95f nexus3.onap.org:10001/onap/policy-api:latest 6969/tcp policy-api - d34204f95ff3 nexus3.onap.org:10001/onap/policy-clamp-ac-http-ppnt:latest 6969/tcp policy-clamp-ac-http-ppnt - 4470e608c9a8 nexus3.onap.org:10001/onap/policy-clamp-ac-k8s-ppnt:latest 6969/tcp policy-clamp-ac-k8s-ppnt - 62229d46b79c nexus3.onap.org:10001/onap/policy-models-simulator:latest 3905/tcp, 6666/tcp, 6668-6670/tcp, 6680/tcp simulator - efaf0ca5e1f0 nexus3.onap.org:10001/mariadb:10.5.8 3306/tcp mariadb - e84cf17db2a4 nexus3.onap.org:10001/onap/policy-pap:latest 6969/tcp policy-pap - 0a16eecd13c9 nexus3.onap.org:10001/onap/policy-apex-pdp:latest 6969/tcp policy-apex-pdp -================ ============================================================ =========================================== ========================= +============================================== ============================================================ =========================================== +**NAME** **IMAGE** **PORT** +============================================== ============================================================ =========================================== + policy-clamp-runtime-acm-5c6d8fbfb-jz8rb nexus3.onap.org:10001/onap/policy-clamp-runtime-acm:latest 30007/tcp + policy-clamp-ac-pf-ppnt-55c4cb99f4-spvng nexus3.onap.org:10001/onap/policy-clamp-ac-pf-ppnt:latest 30008/tcp + policy-api-58cb45fc9b-ff5md nexus3.onap.org:10001/onap/policy-api:latest 30002/tcp + policy-clamp-ac-http-ppnt-7b99cbfbf8-d4w9v nexus3.onap.org:10001/onap/policy-clamp-ac-http-ppnt:latest 30009/tcp + policy-clamp-ac-k8s-ppnt-6d854cc8b6-twdkh nexus3.onap.org:10001/onap/policy-clamp-ac-k8s-ppnt:latest 30010/tcp + policy-models-simulator-bcd494d87-bfg6g nexus3.onap.org:10001/onap/policy-models-simulator:latest 30904/tcp + mariadb-galera-0 nexus3.onap.org:10001/mariadb:10.5.8 3306/tcp + policy-pap-847d89997d-x9h99 nexus3.onap.org:10001/onap/policy-pap:latest 30003/tcp + policy-apex-pdp-0 nexus3.onap.org:10001/onap/policy-apex-pdp:latest 6969/tcp +============================================== ============================================================ =========================================== **JMeter Screenshot** |
