blob: 2cf3e236d90b9b47c79cfe413c970bf7dccba778 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
|
.. This work is licensed under a
.. Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
.. _acm-s3p-label:
.. toctree::
:maxdepth: 2
Policy Clamp Automation Composition
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Both the Performance and the Stability tests were executed by performing requests
against acm components installed as docker images in local environment. These tests we all
performed on a Ubuntu VM with 32GB of memory, 16 CPU and 50GB of disk space.
ACM Deployment
++++++++++++++
In an effort to allow the execution of the s3p tests to be as close to automatic as possible,
a script will be executed that will perform the following:
- Install of a microk8s kubernetes environment
- Bring up the policy components
- Checks that the components are successfully up and running before proceeding
- Install Java 17
- Install Jmeter locally and configure it
- Specify whether you want to run stability or performance tests
The remainder of this document outlines how to run the tests and the test results
Common Setup
++++++++++++
The common setup for performance and stability tests is now automated - being carried out by a script in- **testsuites/run-s3p-test.sh**.
Clone the policy-clamp repo to access the test scripts
.. code-block:: sh
git clone https://gerrit.onap.org/r/policy/clamp
Stability Test of acm components
++++++++++++++++++++++++++++++++
Test Plan
---------
The 72 hours stability test ran the following steps sequentially in a single threaded loop.
- **Commission Automation Composition Definitions** - Commissions the ACM Definitions
- **Register Participants** - Registers the presence of participants in the acm database
- **Prime AC definition** - Primes the AC Definition to the participants
- **Instantiate acm** - Instantiate the acm instance
- **DEPLOY the ACM instance** - change the state of the acm to DEPLOYED
- **Check instance state** - check the current state of instance and that it is DEPLOYED
- **UNDEPLOY the ACM instance** - change the state of the ACM to UNDEPLOYED
- **Check instance state** - check the current state of instance and that it is UNDEPLOYED
- **Delete instance** - delete the instance from all participants and ACM db
- **DEPRIME ACM definitions** - DEPRIME ACM definitions from participants
- **Delete ACM Definition** - delete the ACM definition on runtime
This runs for 72 hours. Test results are present in the **testsuites/automated-performance/s3pTestResults.jtl**
directory. Logs are present for jmeter in **testsuites/automated-performance/jmeter.log** and
**testsuites/automated-performance/nohup.out**
Run Test
--------
The code in the setup section also serves to run the tests. Just one execution needed to do it all.
.. code-block:: sh
./run-s3p-test.sh run stability
Once the test execution is completed, the results are present in the **automate-performance/s3pTestResults.jtl** file.
This file can be imported into the Jmeter GUI for visualization. The below results are tabulated from the GUI.
Test Results
------------
**Summary**
Stability test plan was triggered for 72 hours.
**Test Statistics**
======================= ================= ================== ==================================
**Total # of requests** **Success %** **Error %** **Average time taken per request**
======================= ================= ================== ==================================
261852 100.00 % 0.00 % 387.126 ms
======================= ================= ================== ==================================
**ACM component Setup**
============================================== ================================================================== ====================
**NAME** **IMAGE** **PORT**
============================================== ================================================================== ====================
zookeeper-deployment-7ff87c7fcc-ptkwv confluentinc/cp-zookeeper:latest 2181/TCP
kafka-deployment-5c87d497b-2jv27 confluentinc/cp-kafka:latest 9092/TCP
policy-models-simulator-6947667bdc-v4czs nexus3.onap.org:10001/onap/policy-models-simulator:latest 3904:30904/TCP
prometheus-f66f97b6-rknvp nexus3.onap.org:10001/prom/prometheus:latest 9090:30909/TCP
mariadb-galera-0 nexus3.onap.org:10001/bitnami/mariadb-galera:10.5.8 3306/TCP
policy-apex-pdp-0 nexus3.onap.org:10001/onap/policy-apex-pdp:3.1.3-SNAPSHOT 6969:30001/TCP
policy-clamp-ac-http-ppnt-7d747b5d98-4phjf nexus3.onap.org:10001/onap/policy-clamp-ac-http-ppnt7.1.3-SNAPSHOT 8084/TCP
policy-clamp-ac-sim-ppnt-97f487577-4p7ks nexus3.onap.org:10001/onap/policy-clamp-ac-sim-ppnt7.1.3-SNAPSHOT 6969/TCP
policy-clamp-ac-k8s-ppnt-6bbd86bbc6-csknn nexus3.onap.org:10001/onap/policy-clamp-ac-k8s-ppnt7.1.3-SNAPSHOT 8083:30443/TCP
policy-clamp-ac-pf-ppnt-5fcbbcdb6c-twkxw nexus3.onap.org:10001/onap/policy-clamp-ac-pf-ppnt7.1.3-SNAPSHOT 6969:30008/TCP
policy-clamp-runtime-acm-66b5d6b64-4gnth nexus3.onap.org:10001/onap/policy-clamp-runtime-acm7.1.3-SNAPSHOT 6969:30007/TCP
policy-pap-f7899d4cd-7m898 nexus3.onap.org:10001/onap/policy-pap:3.1.3-SNAPSHOT 6969:30003/TCP
policy-api-7f7d995b4-ckb84 nexus3.onap.org:10001/onap/policy-api:3.1.3-SNAPSHOT 6969:30002/TCP
============================================== ================================================================== ====================
.. Note::
.. container:: paragraph
There were no failures during the 72 hours test.
**JMeter Screenshot**
.. image:: clamp-s3p-results/acm_stability_jmeter.png
**JMeter Screenshot**
.. image:: clamp-s3p-results/acm_stability_table.png
Performance Test of acm components
++++++++++++++++++++++++++++++++++
Introduction
------------
Performance test of acm components has the goal of testing the min/avg/max processing time and rest call throughput for all the requests with multiple requests at the same time.
Setup Details
-------------
We can setup the environment and execute the tests like this from the **clamp/testsuites** directory
.. code-block:: sh
./run-s3p-test.sh run performance
Test results are present in the **testsuites/automate-performance/s3pTestResults.jtl**
directory. Logs are present for jmeter in **testsuites/automate-performance/jmeter.log** and
**testsuites/automated-performance/nohup.out**
Test Plan
---------
The Performance test ran the following steps sequentially by 5 threaded users. Any user will create 100 compositions/instances.
- **SetUp** - SetUp Thread Group
- **Register Participants** - Registers the presence of participants in the acm database
- **AutomationComposition Test Flow** - flow by 5 threaded users.
- **Creation and Deploy** - Creates 100 Compositions and Instances
- **Commission Automation Composition Definitions** - Commissions the ACM Definitions
- **Prime AC definition** - Primes the AC Definition to the participants
- **Instantiate acm** - Instantiate the acm instance
- **DEPLOY the ACM instance** - change the state of the acm to DEPLOYED
- **Check instance state** - check the current state of instance and that it is DEPLOYED
- **Get participants** - fetch all participants
- **Get compositions** - fetch all compositions
- **Undeploy and Delete** - Deletes instances and Compositions created before
- **UNDEPLOY the ACM instance** - change the state of the ACM to UNDEPLOYED
- **Check instance state** - check the current state of instance and that it is UNDEPLOYED
- **Delete instance** - delete the instance from all participants and ACM db
- **DEPRIME ACM definitions** - DEPRIME ACM definitions from participants
- **Delete ACM Definition** - delete the ACM definition on runtime
Run Test
--------
The code in the setup section also serves to run the tests. Just one execution needed to do it all.
.. code-block:: sh
./run-s3p-test.sh run performance
Once the test execution is completed, the results are present in the **automate-performance/s3pTestResults.jtl** file.
This file can be imported into the Jmeter GUI for visualization. The below results are tabulated from the Jmeter GUI.
Test Results
------------
Test results are shown as below.
**Test Statistics**
======================= ================= ================== ==================================
**Total # of requests** **Success %** **Error %** **Average time taken per request**
======================= ================= ================== ==================================
8624 100 % 0.00 % 1296.8 ms
======================= ================= ================== ==================================
**ACM component Setup**
============================================== ================================================================== ====================
**NAME** **IMAGE** **PORT**
============================================== ================================================================== ====================
zookeeper-deployment-7ff87c7fcc-5svgw confluentinc/cp-zookeeper:latest 2181/TCP
kafka-deployment-5c87d497b-hmbhc confluentinc/cp-kafka:latest 9092/TCP
policy-models-simulator-6947667bdc-crcwq nexus3.onap.org:10001/onap/policy-models-simulator:latest 3904:30904/TCP
prometheus-f66f97b6-24dvx nexus3.onap.org:10001/prom/prometheus:latest 9090:30909/TCP
mariadb-galera-0 nexus3.onap.org:10001/bitnami/mariadb-galera:10.5.8 3306/TCP
policy-apex-pdp-0 nexus3.onap.org:10001/onap/policy-apex-pdp:3.1.3-SNAPSHOT 6969:30001/TCP
policy-clamp-ac-sim-ppnt-97f487577-pn56t nexus3.onap.org:10001/onap/policy-clamp-ac-sim-ppnt7.1.3-SNAPSHOT 6969/TCP
policy-clamp-ac-http-ppnt-7d747b5d98-qjjlv nexus3.onap.org:10001/onap/policy-clamp-ac-http-ppnt7.1.3-SNAPSHOT 8084/TCP
policy-clamp-ac-k8s-ppnt-6bbd86bbc6-ffbz2 nexus3.onap.org:10001/onap/policy-clamp-ac-k8s-ppnt7.1.3-SNAPSHOT 8083:30443/TCP
policy-clamp-ac-pf-ppnt-5fcbbcdb6c-vmsnv nexus3.onap.org:10001/onap/policy-clamp-ac-pf-ppnt7.1.3-SNAPSHOT 6969:30008/TCP
policy-clamp-runtime-acm-66b5d6b64-6vjl5 nexus3.onap.org:10001/onap/policy-clamp-runtime-acm7.1.3-SNAPSHOT 6969:30007/TCP
policy-pap-f7899d4cd-8sjk9 nexus3.onap.org:10001/onap/policy-pap:3.1.3-SNAPSHOT 6969:30003/TCP
policy-api-7f7d995b4-dktdw nexus3.onap.org:10001/onap/policy-api:3.1.3-SNAPSHOT 6969:30002/TCP
============================================== ================================================================== ====================
**JMeter Screenshot**
.. image:: clamp-s3p-results/acm_performance_jmeter.png
|
