diff options
21 files changed, 115 insertions, 53 deletions
diff --git a/docs/development/devtools/apex-s3p-results/apex_perf_jmeter_results.png b/docs/development/devtools/apex-s3p-results/apex_perf_jmeter_results.png Binary files differindex c17d3a6c..0fa35c0b 100644 --- a/docs/development/devtools/apex-s3p-results/apex_perf_jmeter_results.png +++ b/docs/development/devtools/apex-s3p-results/apex_perf_jmeter_results.png diff --git a/docs/development/devtools/apex-s3p-results/apex_stability_jmeter_results.png b/docs/development/devtools/apex-s3p-results/apex_stability_jmeter_results.png Binary files differindex e092d77c..585f99c5 100644 --- a/docs/development/devtools/apex-s3p-results/apex_stability_jmeter_results.png +++ b/docs/development/devtools/apex-s3p-results/apex_stability_jmeter_results.png diff --git a/docs/development/devtools/apex-s3p.rst b/docs/development/devtools/apex-s3p.rst index 9575277c..6a3be847 100644 --- a/docs/development/devtools/apex-s3p.rst +++ b/docs/development/devtools/apex-s3p.rst @@ -24,11 +24,13 @@ At a minimum, the following ONAP components are needed: policy, mariadb-galera, Before deploying, the values.yaml files are changed to use NodePort instead of ClusterIP for policy-api, policy-pap, and policy-apex-pdp, so that they are accessible from jmeter:: - policy-apex-pdp NodePort 10.43.70.176 <none> 6969:30237/TCP - policy-api NodePort 10.43.2.99 <none> 6969:30240/TCP - policy-pap NodePort 10.43.203.178 <none> 6969:30442/TCP + policy-apex-pdp NodePort 10.43.131.43 <none> 6969:31739/TCP + policy-api NodePort 10.43.67.153 <none> 6969:30430/TCP + policy-pap NodePort 10.43.200.57 <none> 6969:30585/TCP -The node ports (30237, 30240 and 30442 above) are used in JMeter. The HOSTNAME for JMeter is set to the IP returned by running kubectl cluster-info. +The node ports (31739, 30430 and 30585 above) are used in JMeter. +The HOSTNAMEs for JMeter are set to the IPs returned by running "kubectl get node -o wide" +and to find the applications for each node by running "kubectl describe node <node-name>". Set up policy-models-simulator ------------------------------ @@ -41,7 +43,7 @@ It is run as a docker image from a node accessible to the kubernetes cluster:: docker run -d --rm --publish 6680:6680 --publish 31054:3905 \ --volume "apex-pdp/testsuites/apex-pdp-stability/src/main/resources/simulatorConfig:/opt/app/policy/simulators/etc/mounted" \ - nexus3.onap.org:10001/onap/policy-models-simulator:2.6-SNAPSHOT-latest + nexus3.onap.org:10001/onap/policy-models-simulator:2.7-SNAPSHOT-latest The published ports 6680 and 31054 are used in JMeter for CDS and DMaaP simulators. @@ -173,7 +175,7 @@ Stability test plan was triggered for 72 hours. There were no failures during th ======================= ================= ================== ================================== **Total # of requests** **Success %** **Error %** **Average time taken per request** ======================= ================= ================== ================================== -430331 100 % 0.00 % 152 ms +430397 100 % 0.00 % 151.694 ms ======================= ================= ================== ================================== .. Note:: @@ -242,7 +244,7 @@ Test results are shown as below. ======================= ================= ================== ================================== **Total # of requests** **Success %** **Error %** **Average time taken per request** ======================= ================= ================== ================================== -47586 100 % 0.00 % 163 ms +47567 100 % 0.00 % 163.841 ms ======================= ================= ================== ================================== **JMeter Screenshot** diff --git a/docs/development/devtools/api-s3p-results/api-response-time-distribution_J.png b/docs/development/devtools/api-s3p-results/api-response-time-distribution_J.png Binary files differindex f9a1ba43..6b62b2b2 100644 --- a/docs/development/devtools/api-s3p-results/api-response-time-distribution_J.png +++ b/docs/development/devtools/api-s3p-results/api-response-time-distribution_J.png diff --git a/docs/development/devtools/api-s3p-results/api-response-time-distribution_performance_J.png b/docs/development/devtools/api-s3p-results/api-response-time-distribution_performance_J.png Binary files differindex 3669bc95..60476027 100644 --- a/docs/development/devtools/api-s3p-results/api-response-time-distribution_performance_J.png +++ b/docs/development/devtools/api-s3p-results/api-response-time-distribution_performance_J.png diff --git a/docs/development/devtools/api-s3p-results/api-response-time-overtime_J.png b/docs/development/devtools/api-s3p-results/api-response-time-overtime_J.png Binary files differindex 7d9f8cc0..b32ff6ae 100644 --- a/docs/development/devtools/api-s3p-results/api-response-time-overtime_J.png +++ b/docs/development/devtools/api-s3p-results/api-response-time-overtime_J.png diff --git a/docs/development/devtools/api-s3p-results/api-response-time-overtime_performance_J.png b/docs/development/devtools/api-s3p-results/api-response-time-overtime_performance_J.png Binary files differindex f16030f7..82a0b8ae 100644 --- a/docs/development/devtools/api-s3p-results/api-response-time-overtime_performance_J.png +++ b/docs/development/devtools/api-s3p-results/api-response-time-overtime_performance_J.png diff --git a/docs/development/devtools/api-s3p-results/api-s3p-jm-1_J.png b/docs/development/devtools/api-s3p-results/api-s3p-jm-1_J.png Binary files differindex a4f57635..c219a63c 100644 --- a/docs/development/devtools/api-s3p-results/api-s3p-jm-1_J.png +++ b/docs/development/devtools/api-s3p-results/api-s3p-jm-1_J.png diff --git a/docs/development/devtools/api-s3p-results/api-s3p-jm-2_J.png b/docs/development/devtools/api-s3p-results/api-s3p-jm-2_J.png Binary files differindex 9c54bc71..0083f3ca 100644 --- a/docs/development/devtools/api-s3p-results/api-s3p-jm-2_J.png +++ b/docs/development/devtools/api-s3p-results/api-s3p-jm-2_J.png diff --git a/docs/development/devtools/api-s3p-results/api_top_after_72h.jpg b/docs/development/devtools/api-s3p-results/api_top_after_72h.jpg Binary files differdeleted file mode 100644 index 048d61f3..00000000 --- a/docs/development/devtools/api-s3p-results/api_top_after_72h.jpg +++ /dev/null diff --git a/docs/development/devtools/api-s3p-results/api_top_after_72h.png b/docs/development/devtools/api-s3p-results/api_top_after_72h.png Binary files differnew file mode 100644 index 00000000..de4c4553 --- /dev/null +++ b/docs/development/devtools/api-s3p-results/api_top_after_72h.png diff --git a/docs/development/devtools/api-s3p-results/api_top_before_72h.jpg b/docs/development/devtools/api-s3p-results/api_top_before_72h.jpg Binary files differdeleted file mode 100644 index c4a499b8..00000000 --- a/docs/development/devtools/api-s3p-results/api_top_before_72h.jpg +++ /dev/null diff --git a/docs/development/devtools/api-s3p-results/api_top_before_72h.png b/docs/development/devtools/api-s3p-results/api_top_before_72h.png Binary files differnew file mode 100644 index 00000000..2b334377 --- /dev/null +++ b/docs/development/devtools/api-s3p-results/api_top_before_72h.png diff --git a/docs/development/devtools/api-s3p.rst b/docs/development/devtools/api-s3p.rst index c19e4d67..12c3a516 100644 --- a/docs/development/devtools/api-s3p.rst +++ b/docs/development/devtools/api-s3p.rst @@ -25,7 +25,7 @@ All the transaction flows are initiated from a test client server running JMeter Setup Details ------------- -The stability test was performed on a default ONAP OOM installation in the University of New Hampshire (UNH) Lab environment. +The stability test was performed on a default ONAP OOM installation in the Nordix Lab environment. JMeter was installed on a separate VM to inject the traffic defined in the `API stability script <https://git.onap.org/policy/api/tree/testsuites/stability/src/main/resources/testplans/policy_api_stability.jmx>`_ @@ -33,9 +33,9 @@ with the following command: .. code-block:: bash - nohup ./apache-jmeter-5.4.1/bin/jmeter.sh -n -t policy_api_stability.jmx -l stabilityTestResultsPolicyApi.jtl + nohup apache-jmeter-5.5/bin/jmeter -n -t policy_api_stability.jmx -l stabilityTestResultsPolicyApi.jtl & -The test was run in the background via “nohup”, to prevent it from being interrupted. +The test was run in the background via “nohup” and “&”, to prevent it from being interrupted. Test Plan --------- @@ -127,7 +127,7 @@ The load was performed against a non-tweaked ONAP OOM installation. ======================= ============= =========== =============================== =============================== =============================== **Total # of requests** **Success %** **TPS** **Avg. time taken per request** **Min. time taken per request** **Max. time taken per request** ======================= ============= =========== =============================== =============================== =============================== - 914881 100% 3.5 1413 ms 100 ms 22201 ms + 950839 100% 3.67 1351 ms 126 ms 16324 ms ======================= ============= =========== =============================== =============================== =============================== .. image:: api-s3p-results/api-s3p-jm-1_J.png @@ -147,11 +147,11 @@ A snapshot is taken before and after test execution to monitor the changes in re Memory and CPU usage before test execution: -.. image:: api-s3p-results/api_top_before_72h.jpg +.. image:: api-s3p-results/api_top_before_72h.png Memory and CPU usage after test execution: -.. image:: api-s3p-results/api_top_after_72h.jpg +.. image:: api-s3p-results/api_top_after_72h.png Performance Test of Policy API @@ -165,7 +165,7 @@ Performance test of policy-api has the goal of testing the min/avg/max processin Setup Details ------------- -The performance test was performed on a default ONAP OOM installation in the University of New Hampshire (UNH) Lab environment. +The performance test was performed on a default ONAP OOM installation in the Nordix Lab environment. JMeter was installed on a separate VM to inject the traffic defined in the `API performance script <https://git.onap.org/policy/api/tree/testsuites/performance/src/main/resources/testplans/policy_api_performance.jmx>`_ @@ -173,9 +173,9 @@ with the following command: .. code-block:: bash - nohup ./apache-jmeter-5.4.1/bin/jmeter.sh -n -t policy_api_performance.jmx -l performanceTestResultsPolicyApi.jtl + nohup apache-jmeter-5.5/bin/jmeter -n -t policy_api_performance.jmx -l performanceTestResultsPolicyApi.jtl & -The test was run in the background via “nohup”, to prevent it from being interrupted. +The test was run in the background via “nohup” and “&”, to prevent it from being interrupted. Test Plan --------- @@ -193,7 +193,7 @@ Running/Triggering performance test will be the same as stability test. That is, ======================= ============= =========== =============================== =============================== =============================== **Total # of requests** **Success %** **TPS** **Avg. time taken per request** **Min. time taken per request** **Max. time taken per request** ======================= ============= =========== =============================== =============================== =============================== - 10417 100% 1.1 17340 ms 100 ms 365982 ms + 16212 100% 1.8 11109 ms 162 ms 237265 ms ======================= ============= =========== =============================== =============================== =============================== .. image:: api-s3p-results/api-s3p-jm-2_J.png diff --git a/docs/development/devtools/clamp-s3p-results/Stability_after_stats.png b/docs/development/devtools/clamp-s3p-results/Stability_after_stats.png Binary files differindex 8370f0df..38242866 100644 --- a/docs/development/devtools/clamp-s3p-results/Stability_after_stats.png +++ b/docs/development/devtools/clamp-s3p-results/Stability_after_stats.png diff --git a/docs/development/devtools/clamp-s3p-results/Stability_before_stats.png b/docs/development/devtools/clamp-s3p-results/Stability_before_stats.png Binary files differdeleted file mode 100644 index 46eaddf6..00000000 --- a/docs/development/devtools/clamp-s3p-results/Stability_before_stats.png +++ /dev/null diff --git a/docs/development/devtools/clamp-s3p-results/acm_performance_jmeter.png b/docs/development/devtools/clamp-s3p-results/acm_performance_jmeter.png Binary files differnew file mode 100644 index 00000000..bad1cf71 --- /dev/null +++ b/docs/development/devtools/clamp-s3p-results/acm_performance_jmeter.png diff --git a/docs/development/devtools/clamp-s3p-results/acm_stability_jmeter.png b/docs/development/devtools/clamp-s3p-results/acm_stability_jmeter.png Binary files differindex abe15373..2f576505 100644 --- a/docs/development/devtools/clamp-s3p-results/acm_stability_jmeter.png +++ b/docs/development/devtools/clamp-s3p-results/acm_stability_jmeter.png diff --git a/docs/development/devtools/clamp-s3p-results/acm_stability_table.png b/docs/development/devtools/clamp-s3p-results/acm_stability_table.png Binary files differindex 70c14feb..28942eff 100644 --- a/docs/development/devtools/clamp-s3p-results/acm_stability_table.png +++ b/docs/development/devtools/clamp-s3p-results/acm_stability_table.png diff --git a/docs/development/devtools/clamp-s3p-results/cl-s3p-performance-result-jmeter.png b/docs/development/devtools/clamp-s3p-results/cl-s3p-performance-result-jmeter.png Binary files differdeleted file mode 100644 index a9a8e01d..00000000 --- a/docs/development/devtools/clamp-s3p-results/cl-s3p-performance-result-jmeter.png +++ /dev/null diff --git a/docs/development/devtools/clamp-s3p.rst b/docs/development/devtools/clamp-s3p.rst index 639b295d..eb17d894 100644 --- a/docs/development/devtools/clamp-s3p.rst +++ b/docs/development/devtools/clamp-s3p.rst @@ -13,8 +13,54 @@ Policy Clamp Automation Composition Both the Performance and the Stability tests were executed by performing requests against acm components installed as docker images in local environment. -Setup Details -+++++++++++++ + +ACM Deployment +++++++++++++++ + +The docker containers can be deployed via Policy CSIT script. +Clone the Policy/docker repo to the local vm + +.. code-block:: bash + + git clone "https://gerrit.onap.org/r/policy/docker" + +Set the following environment variables on the system before deploying the containers. + +.. code-block:: bash + + export CONTAINER_LOCATION=nexus3.onap.org:10001/ + export PROJECT=clamp + +Invoke the following script from the ~/docker/csit folder. + +.. code-block:: bash + + ./start-all.sh + +This script installs the docker containers of ACM and Policy components required for running the tests. + + +Jmeter setup +++++++++++++ + +Apache jmeter tool is installed either on the same virtual machine or on a different virtual machine. + +.. code-block:: bash + + # Install required packages + sudo apt install -y wget unzip + + # Install JMeter + mkdir -p jmeter + cd jmeter + wget https://dlcdn.apache.org//jmeter/binaries/apache-jmeter-5.5.zip # check if valid version + unzip -q apache-jmeter-5.5.zip + rm apache-jmeter-5.5.zip + + +Setup Verification +++++++++++++++++++ +Ensure the following components are up and running before executing the test. - acm runtime component docker image is started and running. - Participant docker images policy-clamp-cl-pf-ppnt, policy-clamp-cl-http-ppnt, policy-clamp-cl-k8s-ppnt are started and running. @@ -32,7 +78,7 @@ The 72 hours stability test ran the following steps sequentially in a single thr - **Create Policy defaultDomain** - creates an operational policy using policy/api component - **Delete Policy sampleDomain** - deletes the operational policy sampleDomain using policy/api component -- **Commission Contorlloop definition** - commissions the acm definition in runtime +- **Commission AC definition** - commissions the acm definition in runtime - **Instantiate acm** - Instantiate the acm towards participants - **Check acm state** - check the current state of acm - **Change State to PASSIVE** - change the state of the acm to PASSIVE @@ -42,7 +88,7 @@ The 72 hours stability test ran the following steps sequentially in a single thr - **Delete instantiated acm** - delete the instantiated acm from all participants - **Delete ACM Definition** - delete the acm definition on runtime -The following steps can be used to configure the parameters of test plan. +The following parameters can be configured on the JMX file for the test. - **HTTP Authorization Manager** - used to store user/password authentication details. - **HTTP Header Manager** - used to store headers which will be used for making HTTP requests. @@ -57,11 +103,19 @@ The following steps can be used to configure the parameters of test plan. POLICY_PARTICIPANT_HOST_PORT Port number of policy participant ============================= ======================================================================== +Download the ACM stability.jmx and performance.jmx files from the Policy-Clamp repo. + +Stability jmx file + +.. code-block:: bash + + ~/clamp/testsuites/stability/src/main/resources/testplans/stability.jmx + The test was run in the background via "nohup", to prevent it from being interrupted: .. code-block:: bash - nohup ./jMeter/apache-jmeter-5.2.1/bin/jmeter -n -t stability.jmx -l testresults.jtl + nohup ./jmeter/apache-jmeter-5.5/bin/jmeter -n -t stability.jmx -l testresults.jtl Test Results ------------ @@ -83,22 +137,24 @@ Stability test plan was triggered for 72 hours. ======================= ================= ================== ================================== **Total # of requests** **Success %** **Error %** **Average time taken per request** ======================= ================= ================== ================================== -99992 100.00 % 0.00 % 192 ms +97916 100.00 % 0.00 % 246 ms ======================= ================= ================== ================================== **ACM component Setup** -================ ========================================================= =========================================== ========================= -**CONTAINER ID** **IMAGE** **PORTS** **NAMES** -================ ========================================================= =========================================== ========================= - a9cb0cd103cf onap/policy-clamp-cl-runtime:latest 6969/tcp policy-clamp-cl-runtime - 886e572b8438 onap/policy-clamp-cl-pf-ppnt:latest 6973/tcp policy-clamp-cl-pf-ppnt - 035707b1b95f nexus3.onap.org:10001/onap/policy-api:latest 6969/tcp policy-api - d34204f95ff3 onap/policy-clamp-cl-http-ppnt:latest 6971/tcp policy-clamp-cl-http-ppnt - 4470e608c9a8 onap/policy-clamp-cl-k8s-ppnt:latest 6972/tcp, 8083/tcp policy-clamp-cl-k8s-ppnt - 62229d46b79c nexus3.onap.org:10001/onap/policy-models-simulator:latest 3905/tcp, 6666/tcp, 6668-6670/tcp, 6680/tcp simulator - efaf0ca5e1f0 nexus3.onap.org:10001/mariadb:10.5.8 3306/tcp mariadb -================ ========================================================= =========================================== ========================= +================ ============================================================ =========================================== ========================= +**CONTAINER ID** **IMAGE** **PORT** **NAME** +================ ============================================================ =========================================== ========================= + a9cb0cd103cf nexus3.onap.org:10001/onap/policy-clamp-runtime-acm:latest 6969/tcp policy-clamp-runtime-acm + 886e572b8438 nexus3.onap.org:10001/onap/policy-clamp-ac-pf-ppnt:latest 6969/tcp policy-clamp-ac-pf-ppnt + 035707b1b95f nexus3.onap.org:10001/onap/policy-api:latest 6969/tcp policy-api + d34204f95ff3 nexus3.onap.org:10001/onap/policy-clamp-ac-http-ppnt:latest 6969/tcp policy-clamp-ac-http-ppnt + 4470e608c9a8 nexus3.onap.org:10001/onap/policy-clamp-ac-k8s-ppnt:latest 6969/tcp policy-clamp-ac-k8s-ppnt + 62229d46b79c nexus3.onap.org:10001/onap/policy-models-simulator:latest 3905/tcp, 6666/tcp, 6668-6670/tcp, 6680/tcp simulator + efaf0ca5e1f0 nexus3.onap.org:10001/mariadb:10.5.8 3306/tcp mariadb + e84cf17db2a4 nexus3.onap.org:10001/onap/policy-pap:latest 6969/tcp policy-pap + 0a16eecd13c9 nexus3.onap.org:10001/onap/policy-apex-pdp:latest 6969/tcp policy-apex-pdp +================ ============================================================ =========================================== ========================= .. Note:: @@ -116,11 +172,7 @@ Stability test plan was triggered for 72 hours. **Memory and CPU usage** -The memory and CPU usage can be monitored by running "docker stats" command. A snapshot is taken before and after test execution to monitor the changes in resource utilization. - -Memory and CPU usage before test execution: - -.. image:: clamp-s3p-results/Stability_before_stats.png +The memory and CPU usage can be monitored by running "docker stats" command. Memory and CPU usage after test execution: @@ -152,11 +204,17 @@ Performance test plan is the same as the stability test plan above except for th Run Test -------- +Performance jmx file + +.. code-block:: bash + + ~/clamp/testsuites/performance/src/main/resources/testplans/performance.jmx + Running/Triggering the performance test will be the same as the stability test. That is, launch JMeter pointing to corresponding *.jmx* test plan. The *RUNTIME_HOST*, *RUNTIME_PORT*, *POLICY_PARTICIPANT_HOST*, *POLICY_PARTICIPANT_HOST_PORT* are already set up in *.jmx* .. code-block:: bash - nohup ./jMeter/apache-jmeter-5.2.1/bin/jmeter -n -t performance.jmx -l testresults.jtl + nohup ./jmeter/apache-jmeter-5.5/bin/jmeter -n -t performance.jmx -l testresults.jtl Once the test execution is completed, execute the below script to get the statistics: @@ -175,23 +233,25 @@ Test results are shown as below. ======================= ================= ================== ================================== **Total # of requests** **Success %** **Error %** **Average time taken per request** ======================= ================= ================== ================================== -13809 100 % 0.00 % 206 ms +13591 100 % 0.00 % 249 ms ======================= ================= ================== ================================== **ACM component Setup** -================ ========================================================= =========================================== ========================= -**CONTAINER ID** **IMAGE** **PORTS** **NAMES** -================ ========================================================= =========================================== ========================= - a9cb0cd103cf onap/policy-clamp-cl-runtime:latest 6969/tcp policy-clamp-cl-runtime - 886e572b8438 onap/policy-clamp-cl-pf-ppnt:latest 6973/tcp policy-clamp-cl-pf-ppnt - 035707b1b95f nexus3.onap.org:10001/onap/policy-api:latest 6969/tcp policy-api - d34204f95ff3 onap/policy-clamp-cl-http-ppnt:latest 6971/tcp policy-clamp-cl-http-ppnt - 4470e608c9a8 onap/policy-clamp-cl-k8s-ppnt:latest 6972/tcp, 8083/tcp policy-clamp-cl-k8s-ppnt - 62229d46b79c nexus3.onap.org:10001/onap/policy-models-simulator:latest 3905/tcp, 6666/tcp, 6668-6670/tcp, 6680/tcp simulator - efaf0ca5e1f0 nexus3.onap.org:10001/mariadb:10.5.8 3306/tcp mariadb -================ ========================================================= =========================================== ========================= +================ ============================================================ =========================================== ========================= +**CONTAINER ID** **IMAGE** **PORT** **NAME** +================ ============================================================ =========================================== ========================= + a9cb0cd103cf nexus3.onap.org:10001/onap/policy-clamp-runtime-acm:latest 6969/tcp policy-clamp-runtime-acm + 886e572b8438 nexus3.onap.org:10001/onap/policy-clamp-ac-pf-ppnt:latest 6969/tcp policy-clamp-ac-pf-ppnt + 035707b1b95f nexus3.onap.org:10001/onap/policy-api:latest 6969/tcp policy-api + d34204f95ff3 nexus3.onap.org:10001/onap/policy-clamp-ac-http-ppnt:latest 6969/tcp policy-clamp-ac-http-ppnt + 4470e608c9a8 nexus3.onap.org:10001/onap/policy-clamp-ac-k8s-ppnt:latest 6969/tcp policy-clamp-ac-k8s-ppnt + 62229d46b79c nexus3.onap.org:10001/onap/policy-models-simulator:latest 3905/tcp, 6666/tcp, 6668-6670/tcp, 6680/tcp simulator + efaf0ca5e1f0 nexus3.onap.org:10001/mariadb:10.5.8 3306/tcp mariadb + e84cf17db2a4 nexus3.onap.org:10001/onap/policy-pap:latest 6969/tcp policy-pap + 0a16eecd13c9 nexus3.onap.org:10001/onap/policy-apex-pdp:latest 6969/tcp policy-apex-pdp +================ ============================================================ =========================================== ========================= **JMeter Screenshot** -.. image:: clamp-s3p-results/cl-s3p-performance-result-jmeter.png +.. image:: clamp-s3p-results/acm_performance_jmeter.png |
