diff options
| -rwxr-xr-x[-rw-r--r--] | docs/development/devtools/testing/s3p/xacml-s3p-results/s3p-perf-xacml.png | bin | 71291 -> 67454 bytes | |||
| -rwxr-xr-x | docs/development/devtools/testing/s3p/xacml-s3p-results/s3p-stability-xacml.png | bin | 0 -> 47141 bytes | |||
| -rw-r--r-- | docs/development/devtools/testing/s3p/xacml-s3p.rst | 192 |
3 files changed, 126 insertions, 66 deletions
diff --git a/docs/development/devtools/testing/s3p/xacml-s3p-results/s3p-perf-xacml.png b/docs/development/devtools/testing/s3p/xacml-s3p-results/s3p-perf-xacml.png Binary files differindex 2c27967f..b5ef5794 100644..100755 --- a/docs/development/devtools/testing/s3p/xacml-s3p-results/s3p-perf-xacml.png +++ b/docs/development/devtools/testing/s3p/xacml-s3p-results/s3p-perf-xacml.png diff --git a/docs/development/devtools/testing/s3p/xacml-s3p-results/s3p-stability-xacml.png b/docs/development/devtools/testing/s3p/xacml-s3p-results/s3p-stability-xacml.png Binary files differnew file mode 100755 index 00000000..d45aeee4 --- /dev/null +++ b/docs/development/devtools/testing/s3p/xacml-s3p-results/s3p-stability-xacml.png diff --git a/docs/development/devtools/testing/s3p/xacml-s3p.rst b/docs/development/devtools/testing/s3p/xacml-s3p.rst index 5ea2e287..2ca0a533 100644 --- a/docs/development/devtools/testing/s3p/xacml-s3p.rst +++ b/docs/development/devtools/testing/s3p/xacml-s3p.rst @@ -9,39 +9,48 @@ ########################## -Performance Test of Policy XACML PDP (Jakarta) -********************************************** +Policy XACML PDP S3P Tests +~~~~~~~~~~~~~~~~~~~~~~~~~~ -The Performance test was executed by performing requests -against the Policy RESTful APIs. +Both the Performance and the Stability tests were executed by performing requests +against Policy components installed in Kubernetes environment. These tests were all +performed on a Ubuntu VM with 32GB of memory, 16 CPU and 100GB of disk space. -A default ONAP installation in the Policy tenant in UNH was used to run the tests. +Policy XACML PDP Deployment ++++++++++++++++++++++++++++ -The Agent VMs in this lab have the following configuration: +In an effort to allow the execution of the s3p tests to be as close to automatic as possible, +a script will be executed that will perform the following: -- 16GB RAM -- 8 VCPU +- Install of a microk8s kubernetes environment +- Bring up the policy components +- Checks that the components are successfully up and running before proceeding +- Install Java 17 +- Install Jmeter locally and configure it +- Specify whether you want to run stability or performance tests -Summary -======= -The Performance test was executed, and the result analysed, via: +The remainder of this document outlines how to run the tests and the test results -.. code-block:: bash +Common Setup +++++++++++++ +The common setup for performance and stability tests is now automated - being carried out by a script in- **testsuites/run-s3p-test.sh**. - jmeter -Jduration=1200 -Jusers=10 \ - -Jxacml_ip=$ip -Jpap_ip=$ip -Japi_ip=$ip \ - -Jxacml_port=30111 -Jpap_port=30197 -Japi_port=30664 \ - -n -t perf.jmx -l testresults.jtl +Clone the policy-xacml-pdp repo to access the test scripts -Note: the ports listed above correspond to port 6969 of the respective components. +.. code-block:: bash -The performance tests runs the following, all in parallel: + git clone https://gerrit.onap.org/r/policy/xacml-pdp xacml-pdp -- Healthcheck, 10 simultaneous threads -- Statistics, 10 simultaneous threads -- Decisions, 10 simultaneous threads, each running the following in sequence: +Stability Test of Policy XACML PDP +++++++++++++++++++++++++++++++++++ +Test Plan +--------- +The 24 hours stability test ran the following steps. + +- Healthcheck, 2 simultaneous threads +- Decisions, 2 simultaneous threads, each running the following tasks in sequence: - Monitoring Decision - Monitoring Decision, abbreviated - Naming Decision @@ -50,62 +59,97 @@ The performance tests runs the following, all in parallel: - Frequency Limiter Guard Decision - Min/Max Guard Decision -When the script starts up, it uses policy-api to create, and policy-pap to deploy, -the policies that are needed by the test. It assumes that the "naming" policy has -already been created and deployed. Once the test completes, it undeploys and deletes -the policies that it previously created. +This runs for 24 hours. Test results are present in the **testsuites/automated-performance/s3pTestResults.jtl** +file and in **/tmp/** directory. Logs are present for jmeter in **testsuites/automated-performance/jmeter.log** and +**testsuites/automated-performance/nohup.out** -Results -======= +Run Test +-------- -The test was run for 20 minutes with 10 users (i.e., threads), with the following results: +The code in the setup section also serves to run the tests. Just one execution needed to do it all. -.. csv-table:: - :header: "Number of Users", "Throughput (requests/second)", "Average Latency (ms)" +.. code-block:: bash - 10, 4603, 2 + bash run-s3p-test.sh run stability -.. image:: xacml-s3p-results/s3p-perf-xacml.png +Once the test execution is completed, the results are present in the **automate-performance/s3pTestResults.jtl** file. +This file can be imported into the Jmeter GUI for visualization. The below results are tabulated from the GUI. -Stability Test of Policy XACML PDP -********************************** +Test Results +------------ -This test was run using jmeter on a default -ONAP installation in the Policy tenant in UNH. +**Summary** -The Agent VMs in this lab have the following configuration: +Stability test plan was triggered for 24 hours. -- 16GB RAM -- 8 VCPU +**Test Statistics** -Summary -======= +======================= ================= ======================== ========================= +**Total # of requests** **Error %** **Average Latency (ms)** **Measured requests/sec** +======================= ================= ======================== ========================= + 63618103 0.00 % 5 ms 736.064 ms +======================= ================= ======================== ========================= -The stability test was performed on a default ONAP OOM installation in the Policy tenant of the UNH lab. -JMeter injected the traffic defined in the -`XACML PDP stability script -<https://git.onap.org/policy/xacml-pdp/tree/testsuites/stability/src/main/resources/testplans/stability.jmx>`_ -with the following command: +**JMeter Results** -.. code-block:: bash +.. image:: xacml-s3p-results/s3p-stability-xacml.png - jmeter.sh -Jduration=259200 -Jusers=2 -Jxacml_ip=$ip -Jpap_ip=$ip -Japi_ip=$ip \ - -Jxacml_port=30111 -Jpap_port=30197 -Japi_port=30664 --nongui --testfile stability.jmx +**Policy component Setup** -Note: the ports listed above correspond to port 6969 of the respective components. +============================================== ==================================================================== =========================================== +**NAME** **IMAGE** **PORT** +============================================== ==================================================================== =========================================== +mariadb-galera-0 docker.io/bitnami/mariadb-galera:10.5.8 3306/TCP +policy-models-simulator-bcd494d87-cldfr nexus3.onap.org:10001/onap/policy-models-simulator:3.0.2-SNAPSHOT 30904/TCP +policy-apex-pdp-0 nexus3.onap.org:10001/onap/policy-apex-pdp:3.0.2-SNAPSHOT 30001/TCP +prometheus-f66f97b6-kpg5c nexus3.onap.org:10001/prom/prometheus:latest 30909/TCP +policy-distribution-86777bdd78-95zjx nexus3.onap.org:10001/onap/policy-distribution:3.0.2-SNAPSHOT 30006/TCP +policy-clamp-ac-http-ppnt-7b99cbfbf8-d5zjk nexus3.onap.org:10001/onap/policy-clamp-ac-http-ppnt:7.0.3-SNAPSHOT +policy-clamp-ac-pf-ppnt-55c4cb99f4-pp699 nexus3.onap.org:10001/onap/policy-clamp-ac-pf-ppnt:7.0.3-SNAPSHOT 30008/TCP +policy-clamp-ac-k8s-ppnt-6d854cc8b6-nzlgh nexus3.onap.org:10001/onap/policy-clamp-ac-k8s-ppnt:7.0.3-SNAPSHOT +policy-clamp-runtime-acm-5c6d8fbfb-zww5j nexus3.onap.org:10001/onap/policy-clamp-runtime-acm:7.0.3-SNAPSHOT 30007/TCP +policy-pap-847d89997d-gljzh nexus3.onap.org:10001/onap/policy-pap:3.0.2-SNAPSHOT 30003/TCP +policy-xacml-pdp-656d57d578-9h6fx nexus3.onap.org:10001/onap/policy-xacml-pdp:3.0.2-SNAPSHOT 30004/TCP +policy-api-58cb45fc9b-9qcx6 nexus3.onap.org:10001/onap/policy-api:3.0.2-SNAPSHOT 30002/tcp +============================================== ==================================================================== =========================================== -The default log level of the root and org.eclipse.jetty.server.RequestLog loggers in the logback.xml -of the XACML PDP -(om/kubernetes/policy/components/policy-xacml-pdp/resources/config/logback.xml) -was set to WARN since the OOM installation did have log rotation enabled of the -container logs in the kubernetes worker nodes. +.. Note:: -The stability test, stability.jmx, runs the following, all in parallel: + .. container:: paragraph + + There were no failures during the 24 hours test. + +The XACML PDP offered very good performance with JMeter for the traffic mix described above. +The average transaction time is insignificant. + + +Performance Test of Policy XACML PDP +++++++++++++++++++++++++++++++++++++ + +Introduction +------------ + +Performance test of acm components has the goal of testing the min/avg/max processing time and rest call throughput for all the requests with multiple requests at the same time. + +Setup Details +------------- + +We can setup the environment and execute the tests like this from the **xacml-pdp/testsuites** directory + +Test Plan +--------- + +Performance test plan is the same as the stability test plan above except for the few differences listed below. + +- Increase the number of threads up to 10 (simulating 10 users' behaviours at the same time). +- Reduce the test time to 20 minutes. + +The performance tests runs the following, all in parallel: + +- Healthcheck, 10 simultaneous threads +- Decisions, 10 simultaneous threads, each running the following in sequence: -- Healthcheck, 2 simultaneous threads -- Statistics, 2 simultaneous threads -- Decisions, 2 simultaneous threads, each running the following tasks in sequence: - Monitoring Decision - Monitoring Decision, abbreviated - Naming Decision @@ -119,16 +163,32 @@ the policies that are needed by the test. It assumes that the "naming" policy h already been created and deployed. Once the test completes, it undeploys and deletes the policies that it previously created. -Results -======= +Run Test +-------- -The stability summary results were reported by JMeter with the following summary line: +The code in the setup section also serves to run the tests. Just one execution needed to do it all. .. code-block:: bash - summary = 941639699 in 71:59:36 = 3633.2/s Avg: 1 Min: 0 Max: 842 Err: 0 (0.00%) + bash run-s3p-test.sh run performance -The XACML PDP offered very good performance with JMeter for the traffic mix described above. -The average transaction time is insignificant. The maximum transaction time of 842 ms. -There was a Drools stability test running in parallel, hence the actual load was higher. +Once the test execution is completed, the results are present in the **automate-performance/s3pTestResults.jtl** file and in **/tmp/** directory. + +This file can be imported into the Jmeter GUI for visualization. The below results are tabulated from the GUI. + +Test Results +------------ + +**Summary** +The test was run for 20 minutes with 10 users (i.e., threads), with the following results: + +**Test Statistics** + +======================= ================= ======================== ========================= +**Total # of requests** **Error %** **Average Latency (ms)** **Measured requests/sec** +======================= ================= ======================== ========================= + 912141 0.00 % 25.5666 ms 741.917 ms +======================= ================= ======================== ========================= + +.. image:: xacml-s3p-results/s3p-perf-xacml.png |
