8 files changed, 145 insertions, 152 deletions
diff --git a/docs/development/devtools/devtools.rst b/docs/development/devtools/devtools.rst
index 5b75d328..5c84fd02 100644
--- a/docs/development/devtools/devtools.rst
+++ b/docs/development/devtools/devtools.rst
@@ -297,15 +297,8 @@ familiar with the Policy Framework components and test any local changes.
    api-smoke.rst
    pap-smoke.rst
    apex-smoke.rst
-   distribution-smoke.rst
-
-..
    drools-smoke.rst
-
-..
    xacml-smoke.rst
-
-..
    distribution-smoke.rst
 
 
diff --git a/docs/development/devtools/drools-s3p.rst b/docs/development/devtools/drools-s3p.rst
index 22c1b47d..571e09a3 100644
--- a/docs/development/devtools/drools-s3p.rst
+++ b/docs/development/devtools/drools-s3p.rst
@@ -10,32 +10,20 @@
 Policy Drools PDP component
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Both the Performance and the Stability tests were executed against an ONAP installation in the policy-k8s tenant
-in the windriver lab, from an independent VM running the jmeter tool to inject the load.
+Both the Performance and the Stability tests were executed against an ONAP installation in the Policy tenant
+in the UNH lab, from the admin VM running the jmeter tool to inject the load.
 
 General Setup
 *************
 
-The installation runs the following components in a single VM:
-
-- AAF
-- AAI
-- DMAAP
-- POLICY
-
-The VM has the following hardware spec:
-
-- 126GB RAM
-- 12 VCPUs
-- 155GB Ephemeral Disk
-
-Jmeter is run from a different VM with the following configuration:
+Agent VMs in this lab have the following configuration:
 
 - 16GB RAM
-- 8 VCPUs
-- 155GB Ephemeral Disk
+- 8 VCPU
 
-The drools-pdp container uses the JVM memory settings from a default OOM installation.
+Jmeter is run from the admin VM.
+
+The drools-pdp container uses the JVM memory and CPU settings from the default OOM installation.
 
 Other ONAP components exercised during the stability tests were:
 
@@ -51,22 +39,6 @@ The following components are simulated during the tests.
 - APPC responses for the vCPE and vFW use cases.
 - AAI to answer queries for the use cases under test.
 
-SO, and AAI actors were simulated within the PDP-D JVM by enabling the
-feature-controlloop-utils before running the tests.
-
-PDP-D Setup
-***********
-
-The kubernetes charts were modified previous to the installation
-to add the following script that enables the controlloop-utils feature:
-
-.. code-block:: bash
-
-    oom/kubernetes/policy/charts/drools/resources/configmaps/features.pre.sh:
-
-    #!/bin/sh
-    sh -c "features enable controlloop-utils"
-
 Stability Test of Policy PDP-D
 ******************************
 
@@ -82,132 +54,38 @@ The tests focused on the following use cases:
 For 72 hours the following 5 scenarios ran in parallel:
 
 - vCPE success scenario
-- vCPE failure scenario (failure returned by simulated APPC recipient through DMaaP).
 - vDNS success scenario.
-- vDNS failure scenario (failure by introducing in the DCAE ONSET a non-existent vserver-name reference).
 - vFirewall success scenario.
+- vCPE failure scenario (simulates a failure scenario returned by simulated APPC recipient through DMaaP).
+- vDNS failure scenario (simulates a failure by introducing in the DCAE ONSET a non-existent vserver-name reference).
 
 Five threads ran in parallel, one for each scenario, back to back with no pauses.   The transactions were initiated
 by each jmeter thread group.   Each thread initiated a transaction, monitored the transaction, and
 as soon as the transaction ending was detected, it initiated the next one.
 
-JMeter was run in a docker container with the following command:
-
-.. code-block:: bash
-
-    docker run --interactive --tty --name jmeter --rm --volume $PWD:/jmeter -e VERBOSE_GC="" egaillardon/jmeter-plugins --nongui --testfile s3p.jmx --loglevel WARN
-
-The results were accessed by using the telemetry API to gather statistics:
-
-
-vCPE Success scenario
-=====================
-
-ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e:
-
-.. code-block:: bash
-
-    # Times are in milliseconds
-
-    Control Loop Name: ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e
-    Number of Transactions Executed: 114007
-    Number of Successful Transactions: 112727
-    Number of Failure Transactions: 1280
-    Average Execution Time: 434.9942021103967 ms.
-
-
-vCPE Failure scenario
-=====================
-
-ControlLoop-vCPE-Fail:
+The results are illustrated on the following graphs:
 
-.. code-block:: bash
-
-    # Times are in milliseconds
-
-    Control Loop Name: ControlLoop-vCPE-Fail
-    Number of Transactions Executed: 114367
-    Number of Successful Transactions: 114367 (failure transactions are expected)
-    Number of Failure Transactions: 0         (success transactions are not expected)
-    Average Execution Time: 433.61750330077734 ms.
-
-
-vDNS Success scenario
-=====================
-
-ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3:
-
-.. code-block:: bash
-
-    # Times are in milliseconds
-
-    Control Loop Name: ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3
-    Number of Transactions Executed: 237512
-    Number of Successful Transactions: 229532
-    Number of Failure Transactions: 7980
-    Average Execution Time: 268.028794334602 ms.
-
-
-vDNS Failure scenario
-=====================
-
-ControlLoop-vDNS-Fail:
-
-.. code-block:: bash
-
-    # Times are in milliseconds
-
-    Control Loop Name: ControlLoop-vDNS-Fail
-    Number of Transactions Executed: 1957987
-    Number of Successful Transactions: 1957987 (failure transactions are expected)
-    Number of Failure Transactions: 0         (success transactions are not expected)
-    Average Execution Time: 39.369322166081794
-
-
-vFirewall Success scenario
-==========================
-
-ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a:
-
-.. code-block:: bash
-
-    # Times are in milliseconds
-
-    Control Loop Name: ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a
-    Number of Transactions Executed: 120308
-    Number of Successful Transactions: 118895
-    Number of Failure Transactions: 1413
-    Average Execution Time: 394.8609236293513 ms.
+.. image:: images/s3p-drools-1.png
+.. image:: images/s3p-drools-2.png
+.. image:: images/s3p-drools-3.png
+.. image:: images/s3p-drools-4.png
 
 
 Commentary
 ==========
 
-There has been a degradation of performance observed in this release
-when compared with the previous one.
-Approximately 1% of transactions were not completed as expected for
-some use cases.   Average Execution Times are extended as well.
-The unexpected results seem to point in the direction of the
-interactions of the distributed locking feature with the database.
-These areas as well as the conditions for the test need to be investigated
-further.
+There is around 1% unexpected failures during the 72-hour run.   This can also be seen in the
+final output of jmeter:
 
 .. code-block:: bash
 
-    # Common pattern in the audit.log for unexpected transaction completions
-
-    a8d637fc-a2d5-49f9-868b-5b39f7befe25||ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a|
-    policy:usecases:[org.onap.policy.drools-applications.controlloop.common:controller-usecases:1.9.0:usecases]|
-    2021-10-12T19:48:02.052+00:00|2021-10-12T19:48:02.052+00:00|0|
-    null:operational.modifyconfig.EVENT.MANAGER.FINAL:1.0.0|dev-policy-drools-pdp-0|
-    ERROR|400|Target Lock was lost|||VNF.generic-vnf.vnf-name||dev-policy-drools-pdp-0||
-    dev-policy-drools-pdp-0|microservice.stringmatcher|
-    {vserver.prov-status=ACTIVE, vserver.is-closed-loop-disabled=false,
-    generic-vnf.vnf-name=fw0002vm002fw002, vserver.vserver-name=OzVServer}||||
-    INFO|Session org.onap.policy.drools-applications.controlloop.common:controller-usecases:1.9.0:usecases|
-
-    # The "Target Lock was lost" is a common message error in the unexpected results.
+    summary = 37705505 in 72:00:56 =  145.4/s Avg:    30 Min:     0 Max: 20345 Err: 360852 (0.96%)
 
+The 1% errors were found to be related to the nature of the run, where each one of the 5 use case
+threads run without pauses starting one after the other a new round of their assigned control loop.
+It has been found that at times, the release time of the lock (which requires DB operations) outruns
+the initiation of the next control loop (using the same resource), therefore the newly initiated control
+loop fails.  In reality, this scenario with the same resource being used back to back in consecutive control
+loop rounds will be unlikely.
 
-END-OF-DOCUMENT
 
diff --git a/docs/development/devtools/drools-smoke.rst b/docs/development/devtools/drools-smoke.rst
new file mode 100644
index 00000000..07c812c4
--- /dev/null
+++ b/docs/development/devtools/drools-smoke.rst
@@ -0,0 +1,62 @@
+.. This work is licensed under a
+.. Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+.. _policy-api-smoke-testing-label:
+
+.. toctree::
+   :maxdepth: 2
+
+Policy Drools PDP and Applications Smoke Test
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The policy-drools-pdp smoke testing can be executed against a kubernetes based ONAP installation,
+and/or a docker-compose set up similar to the one executed by CSIT tests.
+
+General Setup
+*************
+
+ONAP OOM kubernetes
+-------------------
+
+For installation instructions, please refer to the following documentation:
+
+`OOM Quick Start Guide <https://docs.onap.org/projects/onap-oom/en/latest/oom_quickstart_guide.html>`_
+
+At a minimum policy needs the following components installed:
+
+- onap base charts
+- AAF for certificate generation
+- DMaaP message-router for communication among policy components.
+
+AAI, SO and other components can be simulated by installing the simulator charts:
+
+`Policy Simulator Helm Chart <https://git.onap.org/policy/docker/tree/helm/policy-models-simulator>`_
+
+docker-compose based
+--------------------
+
+A smaller testing environment can be put together by replicating the CSIT test environment:
+
+`Policy CSIT Test infrastructure <https://git.onap.org/policy/docker/tree/csit>`_
+
+Testing procedures
+******************
+
+The smoke tests should be focused on verifying the proper workings of drools
+and dependent components.   The following scenarios should be considered:
+
+- PDP-D registration with PAP.
+- PDP-D restarts and re-registration with PAP.
+- Proper workings of telemetry tool.
+- Exploration of correct PDP-D states with the telemetry tool.
+- Statistics and prometheus metrics.
+- Verify correct states of API, PAP, and controllers using the PDP-D healthchecks.
+- Verify distributed locking capability and proper use of the database.
+- Verify vCPE, vDNS, and vFirewall use cases and recorded metrics.
+
+The following testsuites contain everything necessary for the previous verifications:
+
+- `CSIT Robot framework <https://git.onap.org/policy/docker/tree/csit/drools-applications/tests/drools-applications-test.robot>`_
+- `JMeter S3P <https://git.onap.org/policy/drools-applications/tree/testsuites/stability/src/main/resources/s3p.jmx>`_
+
diff --git a/docs/development/devtools/images/s3p-drools-1.png b/docs/development/devtools/images/s3p-drools-1.png
new file mode 100644
index 00000000..5dc70c57
--- /dev/null
+++ b/docs/development/devtools/images/s3p-drools-1.png
diff --git a/docs/development/devtools/images/s3p-drools-2.png b/docs/development/devtools/images/s3p-drools-2.png
new file mode 100644
index 00000000..e985a712
--- /dev/null
+++ b/docs/development/devtools/images/s3p-drools-2.png
diff --git a/docs/development/devtools/images/s3p-drools-3.png b/docs/development/devtools/images/s3p-drools-3.png
new file mode 100644
index 00000000..8f2a1d4c
--- /dev/null
+++ b/docs/development/devtools/images/s3p-drools-3.png
diff --git a/docs/development/devtools/images/s3p-drools-4.png b/docs/development/devtools/images/s3p-drools-4.png
new file mode 100644
index 00000000..369d1f33
--- /dev/null
+++ b/docs/development/devtools/images/s3p-drools-4.png
diff --git a/docs/development/devtools/xacml-smoke.rst b/docs/development/devtools/xacml-smoke.rst
new file mode 100644
index 00000000..5b7eb8ea
--- /dev/null
+++ b/docs/development/devtools/xacml-smoke.rst
@@ -0,0 +1,60 @@
+.. This work is licensed under a
+.. Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+.. _policy-api-smoke-testing-label:
+
+.. toctree::
+   :maxdepth: 2
+
+XACML PDP Smoke Test
+~~~~~~~~~~~~~~~~~~~~
+
+The policy-xacml-pdp smoke testing can be executed against a kubernetes based ONAP installation,
+and/or a docker-compose set up similar to the one executed by CSIT tests.
+
+General Setup
+*************
+
+ONAP OOM kubernetes
+-------------------
+
+For installation instructions, please refer to the following documentation:
+
+`OOM Quick Start Guide <https://docs.onap.org/projects/onap-oom/en/latest/oom_quickstart_guide.html>`_
+
+At a minimum policy needs the following components installed:
+
+- onap base charts
+- AAF for certificate generation
+- DMaaP message-router for communication among policy components.
+
+docker-compose based
+--------------------
+
+A smaller testing environment can be put together by replicating the CSIT test environment:
+
+`Policy CSIT Test infrastructure <https://git.onap.org/policy/docker/tree/csit>`_
+
+Testing procedures
+******************
+
+The smoke tests should be focused on verifying the proper workings of the xacml
+PDP and dependent components.   The following scenarios should be considered:
+
+- PDP-X registration with PAP.
+- PDP-X restarts and re-registration with PAP.
+- Healtchecks
+- Statistics and Prometheus metrics.
+- Verify decision with monitoring policies.
+- Verify decision with optimization policy.
+- Verify decision with min/max policy.
+- Verify decision with frequency limiter policy.
+- Verify decision with default guard policy.
+- Verify decision with naming policy.
+
+The following testsuites contain everything necessary for the previous verifications:
+
+- `CSIT Robot framework <https://git.onap.org/policy/docker/tree/csit/xacml-pdp/tests/xacml-pdp-test.robot>`_
+- `JMeter S3P <https://git.onap.org/policy/xacml-pdp/tree/testsuites/stability/src/main/resources/testplans/stability.jmx>`_
+