10 files changed, 113 insertions, 100 deletions
diff --git a/docs/development/devtools/api-s3p-results/api-response-time-distribution_I.png b/docs/development/devtools/api-s3p-results/api-response-time-distribution_I.png
new file mode 100644
index 00000000..b53da6ba
--- /dev/null
+++ b/docs/development/devtools/api-s3p-results/api-response-time-distribution_I.png
diff --git a/docs/development/devtools/api-s3p-results/api-response-time-distribution_performance_I.png b/docs/development/devtools/api-s3p-results/api-response-time-distribution_performance_I.png
new file mode 100644
index 00000000..e65cc151
--- /dev/null
+++ b/docs/development/devtools/api-s3p-results/api-response-time-distribution_performance_I.png
diff --git a/docs/development/devtools/api-s3p-results/api-response-time-overtime_I.png b/docs/development/devtools/api-s3p-results/api-response-time-overtime_I.png
new file mode 100644
index 00000000..ad2f62e9
--- /dev/null
+++ b/docs/development/devtools/api-s3p-results/api-response-time-overtime_I.png
diff --git a/docs/development/devtools/api-s3p-results/api-response-time-overtime_performance_I.png b/docs/development/devtools/api-s3p-results/api-response-time-overtime_performance_I.png
new file mode 100644
index 00000000..91d291bd
--- /dev/null
+++ b/docs/development/devtools/api-s3p-results/api-response-time-overtime_performance_I.png
diff --git a/docs/development/devtools/api-s3p-results/api-s3p-jm-1_I.png b/docs/development/devtools/api-s3p-results/api-s3p-jm-1_I.png
new file mode 100644
index 00000000..d38df8dc
--- /dev/null
+++ b/docs/development/devtools/api-s3p-results/api-s3p-jm-1_I.png
diff --git a/docs/development/devtools/api-s3p-results/api-s3p-jm-2_I.png b/docs/development/devtools/api-s3p-results/api-s3p-jm-2_I.png
new file mode 100644
index 00000000..1c85d949
--- /dev/null
+++ b/docs/development/devtools/api-s3p-results/api-s3p-jm-2_I.png
diff --git a/docs/development/devtools/api-s3p-results/api_top_after_72h.JPG b/docs/development/devtools/api-s3p-results/api_top_after_72h.JPG
new file mode 100644
index 00000000..c69b2d0f
--- /dev/null
+++ b/docs/development/devtools/api-s3p-results/api_top_after_72h.JPG
diff --git a/docs/development/devtools/api-s3p-results/api_top_before_72h.JPG b/docs/development/devtools/api-s3p-results/api_top_before_72h.JPG
new file mode 100644
index 00000000..c2d17afa
--- /dev/null
+++ b/docs/development/devtools/api-s3p-results/api_top_before_72h.JPG
diff --git a/docs/development/devtools/api-s3p.rst b/docs/development/devtools/api-s3p.rst
index 55867b44..1c66297b 100644
--- a/docs/development/devtools/api-s3p.rst
+++ b/docs/development/devtools/api-s3p.rst
@@ -33,8 +33,9 @@ with the following command:
 
 .. code-block:: bash
 
-    jmeter.sh --nongui --testfile policy_api_stability.jmx --logfile result.jtl
+    nohup ./apache-jmeter-5.4.1/bin/jmeter.sh -n -t policy_api_stability.jmx -l stabilityTestResultsPolicyApi.jtl
 
+The test was run in the background via “nohup”, to prevent it from being interrupted.
 
 Test Plan
 ---------
@@ -54,6 +55,7 @@ of each entity is set to the running thread number.
 
 **API Test Flow (5 threads running the same steps in the same loop)**
 
+- Get Policy Metrics
 - Create a new Monitoring Policy Type with Version 6.0.#
 - Create a new Monitoring Policy Type with Version 7.0.#
 - Create a new Optimization Policy Type with Version 6.0.#
@@ -118,18 +120,31 @@ The load was performed against a non-tweaked ONAP OOM installation.
 =======================  =============  ===========  ===============================  ===============================  ===============================
 **Total # of requests**  **Success %**    **TPS**    **Avg. time taken per request**  **Min. time taken per request**  **Max. time taken per request**
 =======================  =============  ===========  ===============================  ===============================  ===============================
-    627746                    100%         2.42              2058 ms                               26 ms                          72809 ms
+    242277                    100%         0.935              5340 ms                               1 ms                          736976 ms
 =======================  =============  ===========  ===============================  ===============================  ===============================
 
-.. image:: images/api-s3p-jm-1_H.png
+.. image:: api-s3p-results/api-s3p-jm-1_I.png
 
 **JMeter Results**
 
 The following graphs show the response time distributions.   The "Get Policy Types" API calls are the most expensive calls that
-average a 10 seconds plus response time.
+average a 7 seconds plus response time.
 
-.. image:: images/api-response-time-distribution_H.png
-.. image:: images/api-response-time-overtime_H.png
+.. image:: api-s3p-results/api-response-time-distribution_I.png
+.. image:: api-s3p-results/api-response-time-overtime_I.png
+
+**Memory and CPU usage**
+
+The memory and CPU usage can be monitored by running "top" command in the policy-api pod.
+A snapshot is taken before and after test execution to monitor the changes in resource utilization.
+
+Memory and CPU usage before test execution:
+
+.. image:: api-s3p-results/api_top_before_72h.JPG
+
+Memory and CPU usage after test execution:
+
+.. image:: api-s3p-results/api_top_after_72h.JPG
 
 
 Performance Test of Policy API
@@ -151,8 +166,9 @@ with the following command:
 
 .. code-block:: bash
 
-    jmeter.sh --nongui --testfile policy_api_performance.jmx --logfile result.jtl
+    nohup ./apache-jmeter-5.4.1/bin/jmeter.sh -n -t policy_api_performance.jmx -l performanceTestResultsPolicyApi.jtl
 
+The test was run in the background via “nohup”, to prevent it from being interrupted.
 
 Test Plan
 ---------
@@ -170,18 +186,18 @@ Running/Triggering performance test will be the same as stability test. That is,
 =======================  =============  ===========  ===============================  ===============================  ===============================
 **Total # of requests**  **Success %**    **TPS**    **Avg. time taken per request**  **Min. time taken per request**  **Max. time taken per request**
 =======================  =============  ===========  ===============================  ===============================  ===============================
-    4082                     100%           0.45              1297 ms                              4 ms                          63612 ms
+    2822                     100%           0.31              63794 ms                              2 ms                          1183376 ms
 =======================  =============  ===========  ===============================  ===============================  ===============================
 
-.. image:: images/api-s3p-jm-2_H.png
+.. image:: api-s3p-results/api-s3p-jm-2_I.png
 
 Test Results
 ------------
 
 The following graphs show the response time distributions.
 
-.. image:: images/api-response-time-distribution_performance_H.png
-.. image:: images/api-response-time-overtime_performance_H.png
+.. image:: api-s3p-results/api-response-time-distribution_performance_I.png
+.. image:: api-s3p-results/api-response-time-overtime_performance_I.png
 
 
 
diff --git a/docs/development/devtools/drools-s3p.rst b/docs/development/devtools/drools-s3p.rst
index 303a6beb..1586379e 100644
--- a/docs/development/devtools/drools-s3p.rst
+++ b/docs/development/devtools/drools-s3p.rst
@@ -10,31 +10,34 @@
 Policy Drools PDP component
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Both the Performance and the Stability tests were executed against a default ONAP installation in the policy-k8s tenant in the windriver lab, from an independent VM running the jmeter tool to inject the load.
+Both the Performance and the Stability tests were executed against an ONAP installation in the policy-k8s tenant
+in the windriver lab, from an independent VM running the jmeter tool to inject the load.
 
 General Setup
 *************
 
-The kubernetes installation allocated all policy components in the same worker node VM and some additional ones.
-The worker VM hosting the policy components has the following spec:
+The installation runs the following components in a single VM:
 
-- 16GB RAM
-- 8 VCPU
-- 160GB Ephemeral Disk
+- AAF
+- AAI
+- DMAAP
+- POLICY
 
-The standalone VM designated to run jmeter has the same configuration.  The jmeter JVM
-was instantiated with a max heap configuration of 12G.
+The VM has the following hardware spec:
 
-The drools-pdp container uses the default JVM memory settings from a default OOM installation:
+- 126GB RAM
+- 12 VCPUs
+- 155GB Ephemeral Disk
 
-.. code-block:: bash
+Jmeter is run from a different VM with the following configuration:
 
-    VM settings:
-        Max. Heap Size (Estimated): 989.88M
-        Using VM: OpenJDK 64-Bit Server VM
+- 16GB RAM
+- 8 VCPUs
+- 155GB Ephemeral Disk
 
+The drools-pdp container uses the JVM memory settings from a default OOM installation.
 
-Other ONAP components used during the stability tests are:
+Other ONAP components exercised during the stability tests were:
 
 - Policy XACML PDP to process guard queries for each transaction.
 - DMaaP to carry PDP-D and jmeter initiated traffic to complete transactions.
@@ -48,19 +51,14 @@ The following components are simulated during the tests.
 - APPC responses for the vCPE and vFW use cases.
 - AAI to answer queries for the use cases under test.
 
-In order to avoid interferences with the APPC component while running the tests,
-the APPC component was disabled.
-
 SO, and AAI actors were simulated within the PDP-D JVM by enabling the
 feature-controlloop-utils before running the tests.
 
 PDP-D Setup
 ***********
 
-The kubernetes charts were modified previous to the installation with
-the changes below.
-
-The feature-controlloop-utils was started by adding the following script:
+The kubernetes charts were modified previous to the installation
+to add the following script that enables the controlloop-utils feature:
 
 .. code-block:: bash
 
@@ -75,7 +73,7 @@ Stability Test of Policy PDP-D
 PDP-D performance
 =================
 
-The test set focused on the following use cases:
+The tests focused on the following use cases:
 
 - vCPE
 - vDNS
@@ -86,23 +84,20 @@ For 72 hours the following 5 scenarios ran in parallel:
 - vCPE success scenario
 - vCPE failure scenario (failure returned by simulated APPC recipient through DMaaP).
 - vDNS success scenario.
-- vDNS failure scenario.
+- vDNS failure scenario (failure by introducing in the DCAE ONSET a non-existant vserver-name reference).
 - vFirewall success scenario.
 
-Five threads ran in parallel, one for each scenario.   The transactions were initiated
+Five threads ran in parallel, one for each scenario, back to back with no pauses.   The transactions were initiated
 by each jmeter thread group.   Each thread initiated a transaction, monitored the transaction, and
-as soon as the transaction ending was detected, it initiated the next one, so back to back with no
-pauses.
+as soon as the transaction ending was detected, it initiated the next one.
 
-All transactions completed successfully as it was expected in each scenario, with no failures.
-
-The command executed was
+JMeter was run in a docker container with the following command:
 
 .. code-block:: bash
 
-    ./jmeter -n -t /home/ubuntu/drools-applications/testsuites/stability/src/main/resources/s3p.jmx  -l /home/ubuntu/jmeter_result/jmeter.jtl -e -o /home/ubuntu/jmeter_result > /dev/null 2>&1
+    docker run --interactive --tty --name jmeter --rm --volume $PWD:/jmeter -e VERBOSE_GC="" egaillardon/jmeter-plugins --nongui --testfile s3p.jmx --loglevel WARN
 
-The results were computed by monitoring the statistics REST endpoint accessible through the telemetry shell or APIs.
+The results were accessed by using the telemetry API to gather statistics:
 
 
 vCPE Success scenario
@@ -114,20 +109,11 @@ ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e:
 
     # Times are in milliseconds
 
-    # Previous to the run, there was 1 failure as a consequence of testing
-    # the flows before the stability load was initiated.   There was
-    # an additional failure encountered during the execution.
-
-    "ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e": {
-        "policyExecutedCount": 161328,
-        "policyExecutedSuccessCount": 161326,
-        "totalElapsedTime": 44932780,
-        "averageExecutionTime": 278.5181741545175,
-        "birthTime": 1616092087842,
-        "lastStart": 1616356511841,
-        "lastExecutionTime": 1616356541972,
-        "policyExecutedFailCount": 2
-    }
+    Control Loop Name: ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e
+    Number of Transactions Executed: 114007
+    Number of Successful Transactions: 112727
+    Number of Failure Transactions: 1280
+    Average Execution Time: 434.9942021103967 ms.
 
 
 vCPE Failure scenario
@@ -139,16 +125,12 @@ ControlLoop-vCPE-Fail:
 
     # Times are in milliseconds
 
-    "ControlLoop-vCPE-Fail": {
-        "policyExecutedCount": 250172,
-        "policyExecutedSuccessCount": 0,
-        "totalElapsedTime": 63258856,
-        "averageExecutionTime": 252.8614553187407,
-        "birthTime": 1616092143137,
-        "lastStart": 1616440688824,
-        "lastExecutionTime": 1616440689010,
-        "policyExecutedFailCount": 250172
-    }
+    Control Loop Name: ControlLoop-vCPE-Fail
+    Number of Transactions Executed: 114367
+    Number of Successful Transactions: 114367 (failure transactions are expected)
+    Number of Failure Transactions: 0         (success transactions are not expected)
+    Average Execution Time: 433.61750330077734 ms.
+
 
 vDNS Success scenario
 =====================
@@ -159,16 +141,12 @@ ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3:
 
     # Times are in milliseconds
 
-    "ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3": {
-        "policyExecutedCount": 235438,
-        "policyExecutedSuccessCount": 235438,
-        "totalElapsedTime": 37564263,
-        "averageExecutionTime": 159.550552587093,
-        "birthTime": 1616092578063,
-        "lastStart": 1616356511253,
-        "lastExecutionTime": 1616356511653,
-        "policyExecutedFailCount": 0
-    }
+    Control Loop Name: ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3
+    Number of Transactions Executed: 237512
+    Number of Successful Transactions: 229532
+    Number of Failure Transactions: 7980
+    Average Execution Time: 268.028794334602 ms.
+
 
 vDNS Failure scenario
 =====================
@@ -179,16 +157,11 @@ ControlLoop-vDNS-Fail:
 
     # Times are in milliseconds
 
-    "ControlLoop-vDNS-Fail": {
-        "policyExecutedCount": 2754574,
-        "policyExecutedSuccessCount": 0,
-        "totalElapsedTime": 14396495,
-        "averageExecutionTime": 5.22639616869977,
-        "birthTime": 1616092659237,
-        "lastStart": 1616440696444,
-        "lastExecutionTime": 1616440696444,
-        "policyExecutedFailCount": 2754574
-    }
+    Control Loop Name: ControlLoop-vDNS-Fail
+    Number of Transactions Executed: 1957987
+    Number of Successful Transactions: 1957987 (failure transactions are expected)
+    Number of Failure Transactions: 0         (success transactions are not expected)
+    Average Execution Time: 39.369322166081794
 
 
 vFirewall Success scenario
@@ -200,17 +173,41 @@ ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a:
 
     # Times are in milliseconds
 
-    # Previous to the run, there were 2 failures as a consequence of testing
-    # the flows before the stability load was initiated.   There was
-    # an additional failure encountered during the execution.
-
-    "ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a": {
-        "policyExecutedCount": 145197,
-        "policyExecutedSuccessCount": 145194,
-        "totalElapsedTime": 33100249,
-        "averageExecutionTime": 227.96785746261975,
-        "birthTime": 1616092985229,
-        "lastStart": 1616356511732,
-        "lastExecutionTime": 1616356541972,
-        "policyExecutedFailCount": 3
-    }
+    Control Loop Name: ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a
+    Number of Transactions Executed: 120308
+    Number of Successful Transactions: 118895
+    Number of Failure Transactions: 1413
+    Average Execution Time: 394.8609236293513 ms.
+
+
+Commentary
+==========
+
+There has been a degradation of performance observed in this release
+when compared with the previous one.
+Approximately 1% of transactions were not completed as expected for
+some use cases.   Average Execution Times are extended as well.
+The unexpected results seem to point in the direction of the
+interactions of the distributed locking feature with the database.
+These areas as well as the conditions for the test need to be investigated
+further.
+
+.. code-block:: bash
+
+    # Common pattern in the audit.log for unexpected transaction completions
+
+    a8d637fc-a2d5-49f9-868b-5b39f7befe25||ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a|
+    policy:usecases:[org.onap.policy.drools-applications.controlloop.common:controller-usecases:1.9.0:usecases]|
+    2021-10-12T19:48:02.052+00:00|2021-10-12T19:48:02.052+00:00|0|
+    null:operational.modifyconfig.EVENT.MANAGER.FINAL:1.0.0|dev-policy-drools-pdp-0|
+    ERROR|400|Target Lock was lost|||VNF.generic-vnf.vnf-name||dev-policy-drools-pdp-0||
+    dev-policy-drools-pdp-0|microservice.stringmatcher|
+    {vserver.prov-status=ACTIVE, vserver.is-closed-loop-disabled=false,
+    generic-vnf.vnf-name=fw0002vm002fw002, vserver.vserver-name=OzVServer}||||
+    INFO|Session org.onap.policy.drools-applications.controlloop.common:controller-usecases:1.9.0:usecases|
+
+    # The "Target Lock was lost" is a common message error in the unexpected results.
+
+
+END-OF-DOCUMENT
+