pdp-d stability documentation

Issue-ID: POLICY-2356
Signed-off-by: jhh <jorge.hernandez-herrero@att.com>
Change-Id: I390c43cb89145f16a35ca1a65a4e6efb0d185852
Signed-off-by: jhh <jorge.hernandez-herrero@att.com>

6 files changed, 299 insertions, 0 deletions

diff --git a/docs/development/devtools/drools-s3p.rst b/docs/development/devtools/drools-s3p.rst
index 3082732f..429186b6 100644
--- a/docs/development/devtools/drools-s3p.rst
+++ b/docs/development/devtools/drools-s3p.rst
@@ -10,3 +10,302 @@
 Policy Drools PDP component
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
+Both the Performance and the Stability tests were executed against a default ONAP installation in the PFPP tenant, from an independent VM running the jmeter tool to inject the load.
+
+General Setup
+*************
+
+The kubernetes installation allocated all policy components in the same worker node VM and some additional ones.   The worker VM hosting the policy components has the
+following spec:
+
+- 16GB RAM 
+- 8 VCPU
+- 160GB Ephemeral Disk
+
+The standalone VM designated to run jmeter has the same configuration and was only
+used to run this tool allocating 12G of heap memory to the jmeter tool.
+
+Other ONAP components used during the estability tests are:
+
+- Policy XACML PDP to process guard queries for each transaction.
+- DMaaP to carry PDP-D and jmeter initiated traffic to complete transactions.
+- Policy API to create (and delete at the end of the tests) policies for each
+  scenario under test.
+- Policy PAP to deploy (and undeploy at the end of the tests) policies for each scenario under test.
+
+The following components are simulated during the tests.
+
+- SO actor for the vDNS use case.
+- APPC responses for the vCPE and vFW use cases.
+- AAI to answer queries for the usecases under test.
+
+In order to restrict APPC responses to just the jmeter too driving all transactions,
+the APPC component was disabled.
+
+SO, and AAI actors were simulated internally within the PDP-D by enabling the
+feature-controlloop-utils previous to run the tests.
+
+PDP-D Setup
+***********
+
+The kubernetes charts were modified previous to the installation with
+the changes below.
+
+The oom/kubernetes/policy/charts/drools/resources/configmaps/base.conf was
+modified:
+
+.. code-block:: bash
+
+    --- a/kubernetes/policy/charts/drools/resources/configmaps/base.conf
+    +++ b/kubernetes/policy/charts/drools/resources/configmaps/base.conf
+    @@ -85,27 +85,27 @@ DMAAP_SERVERS=message-router
+
+    # AAI
+
+    -AAI_HOST=aai.{{.Release.Namespace}}
+    -AAI_PORT=8443
+    +AAI_HOST=localhost
+    +AAI_PORT=6666
+    AAI_CONTEXT_URI=
+
+    # MSO
+
+    -SO_HOST=so.{{.Release.Namespace}}
+    -SO_PORT=8080
+    -SO_CONTEXT_URI=onap/so/infra/
+    -SO_URL=https://so.{{.Release.Namespace}}:8080/onap/so/infra
+    +SO_HOST=localhost
+    +SO_PORT=6667
+    +SO_CONTEXT_URI=
+    +SO_URL=https://localhost:6667/
+
+    # VFC
+
+    -VFC_HOST=
+    -VFC_PORT=
+    +VFC_HOST=localhost
+    +VFC_PORT=6668
+    VFC_CONTEXT_URI=api/nslcm/v1/
+
+    # SDNC
+
+    -SDNC_HOST=sdnc.{{.Release.Namespace}}
+    -SDNC_PORT=8282
+    +SDNC_HOST=localhost
+    +SDNC_PORT=6670
+    SDNC_CONTEXT_URI=restconf/operations/
+
+The AAI actor had to be modified to disable https to talk to the AAI simulator.
+
+.. code-block:: bash
+
+    ~/oom/kubernetes/policy/charts/drools/resources/configmaps/AAI-http-client.properties
+
+    http.client.services=AAI
+
+    http.client.services.AAI.managed=true
+    http.client.services.AAI.https=false
+    http.client.services.AAI.host=${envd:AAI_HOST}
+    http.client.services.AAI.port=${envd:AAI_PORT}
+    http.client.services.AAI.userName=${envd:AAI_USERNAME}
+    http.client.services.AAI.password=${envd:AAI_PASSWORD}
+    http.client.services.AAI.contextUriPath=${envd:AAI_CONTEXT_URI}
+
+The SO actor had to be modified similarly.
+
+.. code-block:: bash
+
+    oom/kubernetes/policy/charts/drools/resources/configmaps/SO-http-client.properties:
+
+    http.client.services=SO
+
+    http.client.services.SO.managed=true
+    http.client.services.SO.https=false
+    http.client.services.SO.host=${envd:SO_HOST}
+    http.client.services.SO.port=${envd:SO_PORT}
+    http.client.services.SO.userName=${envd:SO_USERNAME}
+    http.client.services.SO.password=${envd:SO_PASSWORD}
+    http.client.services.SO.contextUriPath=${envd:SO_CONTEXT_URI}
+
+The feature-controlloop-utils was started by adding the following script:
+
+.. code-block:: bash
+
+    oom/kubernetes/policy/charts/drools/resources/configmaps/features.pre.sh:
+
+    #!/bin/bash
+    bash -c "features enable controlloop-utils"
+
+The PDP-D uses a small configuration:
+
+
+Stability Test of Policy PDP-D
+******************************
+
+The 72 hour stability test happened in parallel with the estability run of the API component.
+
+.. code-block:: bash
+
+  small:
+    limits:
+      cpu: 1
+      memory: 4Gi
+    requests:
+      cpu: 100m
+      memory: 1Gi
+
+Approximately 3.75G heap was allocated to the PDP-D JVM at initialization.
+
+Worker Node performance
+=======================
+
+The VM named onap-k8s-07 was monitored for the duration of the two parallel
+stability runs.  The table below show the usage ranges:
+
+.. code-block:: bash
+
+    NAME          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
+    onap-k8s-07   <=1374m      <=20%  <=10643Mi       <=66%
+
+PDP-D performance
+=================
+
+The PDP-D was monitored during the run an stayed below the following ranges:
+
+.. code-block:: bash
+
+    NAME           CPU(cores)   MEMORY(bytes)
+    dev-drools-0   <=142m         684Mi
+
+Garbagge collection was monitored without detecting any major spike.
+
+The following use cases were tested:
+
+- vCPE
+- vDNS
+- vFirewall
+
+For 72 hours the following 5 scenarios were run in parallel:
+
+- vCPE success scenario
+- vCPE failure scenario (failure returned by simulated APPC recipient through DMaaP).
+- vDNS success scenario.
+- vDNS failure scenario.
+- vFirewall success scenario.
+
+Five threads, one for each scenario described above, push the traffic back to back
+with no pauses.
+
+All transactions completed successfully as expected in each scenario.
+
+The command executed was
+
+.. code-block:: bash
+
+    jmeter -n -t /home/ubuntu/jhh/s3p.jmx > /dev/null 2>&1
+
+The results were computed by taking the ellapsed time from the audit.log
+(this log reports all end to end transactions, marking the start, end, and
+ellapsed times).
+
+The count reflects the number of successful transactions as expected in the
+use case, as well as the average, standard deviation, and max/min.   An histogram
+of the response times have been added as a visual indication on the most common transaction times.
+
+vCPE Success scenario
+=====================
+
+ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e:
+
+.. code-block:: bash
+
+    count    155246.000000
+    mean        269.894226
+    std          64.556282
+    min         133.000000
+    50%         276.000000
+    max        1125.000000
+
+
+Transaction Times histogram:
+
+.. image:: images/ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e.png
+
+
+vCPE Failure scenario
+=====================
+
+ControlLoop-vCPE-Fail:
+
+.. code-block:: bash
+
+    ControlLoop-vCPE-Fail :
+    count    149621.000000
+    mean        280.483522
+    std          67.226550
+    min         134.000000
+    50%         279.000000
+    max        5394.000000
+
+
+Transaction Times histogram:
+
+.. image:: images/ControlLoop-vCPE-Fail.png
+
+vDNS Success scenario
+=====================
+
+ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3:
+
+.. code-block:: bash
+
+    count    293000.000000
+    mean         21.961792
+    std           7.921396
+    min          15.000000
+    50%          20.000000
+    max         672.000000
+
+Transaction Times histogram:
+
+.. image:: images/ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3.png
+
+vDNS Failure scenario
+=====================
+
+ControlLoop-vDNS-Fail:
+
+.. code-block:: bash
+
+    count    59357.000000
+    mean      3010.261267
+    std         76.599948
+    min          0.000000
+    50%       3010.000000
+    max       3602.000000
+
+Transaction Times histogram:
+
+.. image:: images/ControlLoop-vDNS-Fail.png
+
+vFirewall Failure scenario
+==========================
+
+ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a:
+
+.. code-block:: bash
+
+    count    175401.000000
+    mean        184.581251
+    std          35.619075
+    min         136.000000
+    50%         181.000000
+    max        3972.000000
+
+Transaction Times histogram:
+
+.. image:: images/ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a.png
+
+
+
+
diff --git a/docs/development/devtools/images/ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e.png b/docs/development/devtools/images/ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e.png
new file mode 100644
index 00000000..788e2313
--- /dev/null
+++ b/docs/development/devtools/images/ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e.png
diff --git a/docs/development/devtools/images/ControlLoop-vCPE-Fail.png b/docs/development/devtools/images/ControlLoop-vCPE-Fail.png
new file mode 100644
index 00000000..16fc9836
--- /dev/null
+++ b/docs/development/devtools/images/ControlLoop-vCPE-Fail.png
diff --git a/docs/development/devtools/images/ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3.png b/docs/development/devtools/images/ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3.png
new file mode 100644
index 00000000..92f82eb6
--- /dev/null
+++ b/docs/development/devtools/images/ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3.png
diff --git a/docs/development/devtools/images/ControlLoop-vDNS-Fail.png b/docs/development/devtools/images/ControlLoop-vDNS-Fail.png
new file mode 100644
index 00000000..e5f4ce3b
--- /dev/null
+++ b/docs/development/devtools/images/ControlLoop-vDNS-Fail.png
diff --git a/docs/development/devtools/images/ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a.png b/docs/development/devtools/images/ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a.png
new file mode 100644
index 00000000..345ea7d0
--- /dev/null
+++ b/docs/development/devtools/images/ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a.png