Updated docs and results for S3P / API Frankfurt

Issue-ID: POLICY-2453
Signed-off-by: Chris Ramstad <cramstad@ciena.com>
Change-Id: I21506a89eff6409d1905218f962161c3b077865a

2 files changed, 153 insertions, 154 deletions

diff --git a/docs/development/devtools/api-s3p.rst b/docs/development/devtools/api-s3p.rst
index 77205008..982571ba 100644
--- a/docs/development/devtools/api-s3p.rst
+++ b/docs/development/devtools/api-s3p.rst
@@ -17,8 +17,8 @@ Policy API S3P Tests
 Introduction
 ------------
 
-The 72 hour stability test of policy API has the goal of verifying the stability of running policy design API REST service by 
-ingesting a steady flow of transactions of policy design API calls in a multi-thread fashion to simulate multiple clients' behaviors. 
+The 72 hour stability test of policy API has the goal of verifying the stability of running policy design API REST service by
+ingesting a steady flow of transactions of policy design API calls in a multi-thread fashion to simulate multiple clients' behaviors.
 All the transaction flows are initiated from a test client server running JMeter for the duration of 72+ hours.
 
 Setup Details
@@ -33,7 +33,7 @@ VM2 will be running API REST service and visualVM.
 
 **Lab Environment**
 
-Intel ONAP Integration and Deployment Labs 
+Intel ONAP Integration and Deployment Labs
 `Physical Labs <https://wiki.onap.org/display/DW/Physical+Labs>`_,
 `Wind River <https://www.windriver.com/>`_
 
@@ -76,49 +76,55 @@ JMeter: 5.1.1
 Make the etc/hosts entries
 
 .. code-block:: bash
-   
+
     $ echo $(hostname -I | cut -d\  -f1) $(hostname) | sudo tee -a /etc/hosts
-    
+
 Update the Ubuntu software installer
 
 .. code-block:: bash
-   
+
     $ sudo apt-get update
-    
+
 Check and install Java
 
 .. code-block:: bash
-   
+
     $ sudo apt-get install -y openjdk-8-jdk
     $ java -version
-    
+
 Ensure that the Java version executing is OpenJDK version 8
-    
+
 Check and install docker
 
 .. code-block:: bash
-    
+
     $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
     $ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
     $ sudo apt-get update
     $ sudo apt-cache policy docker-ce
-    $ sudo apt-get install -y docker-ce
+    $ sudo apt-get install -y unzip docker-ce
     $ systemctl status docker
     $ docker ps
 
 Change the permissions of the Docker socket file
 
 .. code-block:: bash
-   
+
     $ sudo chmod 777 /var/run/docker.sock
 
+Or add the current user to the docker group
+
+.. code-block:: bash
+
+    $ sudo usermod -aG docker $USER
+
 Check the status of the Docker service and ensure it is running correctly
 
 .. code-block:: bash
-   
+
     $ service docker status
     $ docker ps
-    
+
 **VM1 in lab**
 
 **Install JMeter**
@@ -126,27 +132,27 @@ Check the status of the Docker service and ensure it is running correctly
 Download & install JMeter
 
 .. code-block:: bash
-   
+
     $ mkdir jMeter
     $ cd jMeter
-    $ wget http://mirrors.whoishostingthis.com/apache//jmeter/binaries/apache-jmeter-5.1.1.zip
-    $ unzip apache-jmeter-5.1.1.zip
-    
+    $ wget http://mirrors.whoishostingthis.com/apache//jmeter/binaries/apache-jmeter-5.2.1.zip
+    $ unzip apache-jmeter-5.2.1.zip
+
 **Install other necessary components**
 
 Pull api code & run setup components script
 
 .. code-block:: bash
-   
+
     $ cd ~
     $ git clone https://git.onap.org/policy/api
     $ cd api/testsuites/stability/src/main/resources/simulatorsetup
-    $ ./setup_components.sh
-    
+    $ . ./setup_components.sh
+
 After installation, make sure the following mariadb container is up and running
 
 .. code-block:: bash
-   
+
     ubuntu@test:~/api/testsuites/stability/src/main/resources/simulatorsetup$ docker ps
     CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
     3849ce44b86d        mariadb:10.2.14     "docker-entrypoint.s…"   11 days ago         Up 11 days          0.0.0.0:3306->3306/tcp   mariadb
@@ -158,16 +164,16 @@ After installation, make sure the following mariadb container is up and running
 Pull api code & run setup api script
 
 .. code-block:: bash
-   
+
     $ cd ~
     $ git clone https://git.onap.org/policy/api
     $ cd api/testsuites/stability/src/main/resources/apisetup
-    $ ./setup_api.sh <host ip running api> <host ip running mariadb>
+    $ . ./setup_api.sh <host ip running api> <host ip running mariadb>
 
 After installation, make sure the following api container is up and running
 
 .. code-block:: bash
-   
+
     ubuntu@tools-2:~/api/testsuites/stability/src/main/resources/apisetup$ docker ps
     CONTAINER ID        IMAGE                                                  COMMAND                  CREATED             STATUS              PORTS                                          NAMES
     4f08f9972e55        nexus3.onap.org:10001/onap/policy-api:2.1.1-SNAPSHOT   "bash ./policy-api.sh"   11 days ago         Up 11 days          0.0.0.0:6969->6969/tcp, 0.0.0.0:9090->9090/tcp   policy-api
@@ -179,22 +185,22 @@ VisualVM needs to be installed in the virtual machine having API up and running.
 Install visualVM
 
 .. code-block:: bash
-   
+
     $ sudo apt-get install visualvm
-    
+
 Run few commands to configure permissions
 
 .. code-block:: bash
-   
+
     $ cd /usr/lib/jvm/java-8-openjdk-amd64/bin/
     $ sudo touch visualvm.policy
     $ sudo chmod 777 visualvm.policy
-      
+
     $ vi visualvm.policy
-      
+
     Add the following in visualvm.policy
-      
-      
+
+
     grant codebase "file:/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar" {
        permission java.security.AllPermission;
     };
@@ -202,10 +208,10 @@ Run few commands to configure permissions
 Run following commands to start jstatd using port 1111
 
 .. code-block:: bash
-   
+
     $ cd /usr/lib/jvm/java-8-openjdk-amd64/bin/
     $ ./jstatd -p 1111 -J-Djava.security.policy=visualvm.policy  &
-    
+
 **Local Machine**
 
 **Run & configure visualVM**
@@ -213,9 +219,9 @@ Run following commands to start jstatd using port 1111
 Run visualVM by typing
 
 .. code-block:: bash
-   
+
     $ jvisualvm
-    
+
 Connect to jstatd & remote policy-api JVM
 
     1. Right click on "Remote" in the left panel of the screen and select "Add Remote Host..."
@@ -228,115 +234,6 @@ Sample Screenshot of visualVM
 
 .. image:: images/results-5.png
 
-Test Plan
----------
-
-The 72+ hours stability test will be running the following steps sequentially in multi-threaded loops.
-Thread number is set to 5 to simulate 5 API clients' behaviors (they can be calling the same policy CRUD API simultaneously).
-
-**Setup Thread (will be running only once)**
-    
-- Get policy-api Healthcheck
-- Get API Counter Statistics
-- Get Preloaded Policy Types
-
-**API Test Flow (5 threads running the same steps in the same loop)**
-
-- Create a new TCA Policy Type with Version 1.0.0
-- Create a new TCA Policy Type with Version 2.0.0
-- Create a new TCA Policy Type with Version 3.0.0
-- Create a new TCA Policy Type with Version 4.0.0
-- Create a new TCA Policy Type with Version 5.0.0
-- Create a new TCA Policy Type with Version 6.0.0
-- Create a new TCA Policy Type with Version 7.0.0
-- Create a new TCA Policy Type with Version 8.0.0
-- Create a new TCA Policy Type with Version 9.0.0
-- Create a new TCA Policy Type with Version 10.0.0
-- Create a new TCA Policy Type with Version 11.0.0
-- A 10 sec timer
-- Get All Existing Policy Types
-- Get All Existing Versions of the New TCA Policy Type
-- Get Version 1.0.0 of the New TCA Policy Type
-- Get Version 2.0.0 of the New TCA Policy Type
-- Get Version 3.0.0 of the New TCA Policy Type
-- Get Version 4.0.0 of the New TCA Policy Type
-- Get Version 5.0.0 of the New TCA Policy Type
-- Get Version 6.0.0 of the New TCA Policy Type
-- Get Version 7.0.0 of the New TCA Policy Type
-- Get Version 8.0.0 of the New TCA Policy Type
-- Get Version 9.0.0 of the New TCA Policy Type
-- Get Version 10.0.0 of the New TCA Policy Type
-- Get Version 11.0.0 of the New TCA Policy Type
-- Get the Latest Version of the New TCA Policy Type
-- A 10 sec timer
-- Create a New TCA Policy with Version 1.0.0 over the New TCA Policy Type Version 2.0.0
-- Create a New TCA Policy with Version 2.0.0 over the New TCA Policy Type Version 2.0.0
-- Create a New TCA Policy with Version 3.0.0 over the New TCA Policy Type Version 2.0.0
-- Create a New TCA Policy with Version 4.0.0 over the New TCA Policy Type Version 2.0.0
-- Create a New TCA Policy with Version 5.0.0 over the New TCA Policy Type Version 2.0.0
-- Create a New TCA Policy with Version 6.0.0 over the New TCA Policy Type Version 2.0.0
-- Create a New TCA Policy with Version 7.0.0 over the New TCA Policy Type Version 2.0.0
-- Create a New TCA Policy with Version 8.0.0 over the New TCA Policy Type Version 2.0.0
-- Create a New TCA Policy with Version 9.0.0 over the New TCA Policy Type Version 2.0.0
-- Create a New TCA Policy with Version 10.0.0 over the New TCA Policy Type Version 2.0.0
-- Create a New TCA Policy with Version 11.0.0 over the New TCA Policy Type Version 2.0.0
-- A 10 sec Timer
-- Get All Existing TCA Policies
-- Get All Existing Versions of TCA Policies
-- Get Version 1.0.0 of the New TCA Policy
-- Get Version 2.0.0 of the New TCA Policy
-- Get Version 3.0.0 of the New TCA Policy
-- Get Version 4.0.0 of the New TCA Policy
-- Get Version 5.0.0 of the New TCA Policy
-- Get Version 6.0.0 of the New TCA Policy
-- Get Version 7.0.0 of the New TCA Policy
-- Get Version 8.0.0 of the New TCA Policy
-- Get Version 9.0.0 of the New TCA Policy
-- Get Version 10.0.0 of the New TCA Policy
-- Get Version 11.0.0 of the New TCA Policy
-- Get the Latest Version of the New TCA Policy
-- A 10 sec Timer
-- Create a New Guard Policy with Version 1
-- Create a New Guard Policy with Version 5
-- Create a New Guard Policy with Version 9
-- Create a New Guard Policy with Version 12
-- A 10 sec Timer
-- Get Version 1 of the New Guard Policy
-- Get Version 5 of the New Guard Policy
-- Get Version 9 of the New Guard Policy
-- Get Version 12 of the New Guard Policy
-- Get the Latest Version of the New Guard Policy
-- A 10 sec Timer
-
-**TearDown Thread (will only be running after API Test Flow is completed)**
-
-- Delete Version 2.0.0 of the New TCA Policy Type (suppose to return 409-Conflict)
-- Delete Version 3.0.0 of the New TCA Policy Type
-- Delete Version 4.0.0 of the New TCA Policy Type
-- Delete Version 5.0.0 of the New TCA Policy Type
-- Delete Version 6.0.0 of the New TCA Policy Type
-- Delete Version 7.0.0 of the New TCA Policy Type
-- Delete Version 8.0.0 of the New TCA Policy Type
-- Delete Version 9.0.0 of the New TCA Policy Type
-- Delete Version 10.0.0 of the New TCA Policy Type
-- Delete Version 11.0.0 of the New TCA Policy Type
-- Delete Version 1.0.0 of the New TCA Policy
-- Delete Version 2.0.0 of the New TCA Policy
-- Delete Version 3.0.0 of the New TCA Policy
-- Delete Version 4.0.0 of the New TCA Policy
-- Delete Version 5.0.0 of the New TCA Policy
-- Delete Version 6.0.0 of the New TCA Policy
-- Delete Version 7.0.0 of the New TCA Policy
-- Delete Version 8.0.0 of the New TCA Policy
-- Delete Version 9.0.0 of the New TCA Policy
-- Delete Version 10.0.0 of the New TCA Policy
-- Delete Version 11.0.0 of the New TCA Policy
-- Re-Delete Version 2.0.0 of the New TCA Policy Type (will return 200 now since all TCA policies created over have been deleted)
-- Delete Version 1 of the new Guard Policy
-- Delete Version 5 of the new Guard Policy
-- Delete Version 9 of the new Guard Policy
-- Delete Version 12 of the new Guard Policy
-
 Run Test
 --------
 
@@ -345,9 +242,9 @@ Run Test
 Connect to lab VPN
 
 .. code-block:: bash
-    
+
     $ sudo openvpn --config <path to lab ovpn key file>
-    
+
 SSH into JMeter VM (VM1)
 
 .. code-block:: bash
@@ -357,9 +254,9 @@ SSH into JMeter VM (VM1)
 Run JMeter test in background for 72+ hours
 
 .. code-block:: bash
-  
+
     $ mkdir s3p
-    $ nohup ./jMeter/apache-jmeter-5.1.1/bin/jmeter.sh -n -t ~/api/testsuites/stability/src/main/resources/testplans/policy_api_stability.jmx &
+    $ nohup ./jMeter/apache-jmeter-5.2.1/bin/jmeter.sh -n -t ~/api/testsuites/stability/src/main/resources/testplans/policy_api_stability.jmx &
 
 (Optional) Monitor JMeter test that is running in background (anytime after re-logging into JMeter VM - VM1)
 
@@ -367,9 +264,77 @@ Run JMeter test in background for 72+ hours
 
     $ tail -f s3p/stability.log nohup.out
 
+Test Plan
+---------
 
-Test Results
-------------
+The 72+ hours stability test will be running the following steps sequentially
+in multi-threaded loops. Thread number is set to 5 to simulate 5 API clients'
+behaviors (they can be calling the same policy CRUD API simultaneously).
+Each thread creates a different version of the policy types and policies to not
+interfere with one another while operating simultaneously.  The point version
+of each entity is set to the running thread number.
+
+**Setup Thread (will be running only once)**
+
+- Get policy-api Healthcheck
+- Get API Counter Statistics
+- Get Preloaded Policy Types
+
+**API Test Flow (5 threads running the same steps in the same loop)**
+
+- Create a new Monitoring Policy Type with Version 6.0.#
+- Create a new Monitoring Policy Type with Version 7.0.#
+- Create a new Optimization Policy Type with Version 6.0.#
+- Create a new Guard Policy Type with Version 6.0.#
+- Create a new Native APEX Policy Type with Version 6.0.#
+- Create a new Native Drools Policy Type with Version 6.0.#
+- Create a new Native XACML Policy Type with Version 6.0.#
+- Get All Policy Types
+- Get All Versions of the new Monitoring Policy Type
+- Get Version 6.0.# of the new Monitoring Policy Type
+- Get Version 6.0.# of the new Optimzation Policy Type
+- Get Version 6.0.# of the new Guard Policy Type
+- Get Version 6.0.# of the new Native APEX Policy Type
+- Get Version 6.0.# of the new Native Drools Policy Type
+- Get Version 6.0.# of the new Native XACML Policy Type
+- Get the Latest Version of the New Monitoring Policy Type
+- Create Monitoring Policy Ver 6.0.# w/Monitoring Policy Type Ver 6.0.#
+- Create Monitoring Policy Ver 7.0.# w/Monitoring Policy Type Ver 7.0.#
+- Create Optimization Policy Ver 6.0.# w/Optimization Policy Type Ver 6.0.#
+- Create Guard Policy Ver 6.0.# w/Guard Policy Type Ver 6.0.#
+- Create Native APEX Policy Ver 6.0.# w/Native APEX Policy Type Ver 6.0.#
+- Create Native Drools Policy Ver 6.0.# w/Native Drools Policy Type Ver 6.0.#
+- Create Native XACML Policy Ver 6.0.# w/Native XACML Policy Type Ver 6.0.#
+- Get Version 6.0.# of the new Monitoring Policy
+- Get Version 6.0.# of the new Optimzation Policy
+- Get Version 6.0.# of the new Guard Policy
+- Get Version 6.0.# of the new Native APEX Policy
+- Get Version 6.0.# of the new Native Drools Policy
+- Get Version 6.0.# of the new Native XACML Policy
+- Get the Latest Version of the new Monitoring Policy
+- Delete Version 6.0.# of the new Monitoring Policy
+- Delete Version 7.0.# of the new Monitoring Policy
+- Delete Version 6.0.# of the new Optimzation Policy
+- Delete Version 6.0.# of the new Guard Policy
+- Delete Version 6.0.# of the new Native APEX Policy
+- Delete Version 6.0.# of the new Native Drools Policy
+- Delete Version 6.0.# of the new Native XACML Policy
+- Delete Monitoring Policy Type with Version 6.0.#
+- Delete Monitoring Policy Type with Version 7.0.#
+- Delete Optimization Policy Type with Version 6.0.#
+- Delete Guard Policy Type with Version 6.0.#
+- Delete Native APEX Policy Type with Version 6.0.#
+- Delete Native Drools Policy Type with Version 6.0.#
+- Delete Native XACML Policy Type with Version 6.0.#
+
+**TearDown Thread (will only be running after API Test Flow is completed)**
+
+- Get policy-api Healthcheck
+- Get Preloaded Policy Types
+
+
+Test Results El-Alto
+--------------------
 
 **Summary**
 
@@ -396,6 +361,40 @@ Policy API stability test plan was triggered and running for 72+ hours without a
 .. image:: images/results-4.png
 
 
+Test Results Frankfurt
+----------------------
+
+PFPP ONAP Windriver lab
+
+**Summary**
+
+Policy API stability test plan was triggered and running for 72+ hours without
+any real errors occurring.  The single failure was on teardown and was due to
+simultaneous test plans running concurrently on the lab system.
+
+Compared to El-Alto, 10x the number of API calls were made in the 72 hour run.
+However, the latency increased (most likely due to the synchronization added
+from
+`POLICY-2533 <https://jira.onap.org/browse/POLICY-2533>`_.
+This will be addressed in the next release.
+
+**Test Statistics**
+
+=======================  =============  ===========  ===============================  ===============================  ===============================
+**Total # of requests**  **Success %**  **Error %**  **Avg. time taken per request**  **Min. time taken per request**  **Max. time taken per request**
+=======================  =============  ===========  ===============================  ===============================  ===============================
+    514953                    100%           0%              2510 ms                               336 ms                          15034 ms
+=======================  =============  ===========  ===============================  ===============================  ===============================
+
+**VisualVM Results**
+
+VisualVM results were not captured as this was run in the PFPP ONAP Windriver
+lab.
+
+**JMeter Results**
+
+.. image:: images/api-s3p-jm-1_F.png
+
 
 Performance Test of Policy API
 ++++++++++++++++++++++++++++++
@@ -403,7 +402,7 @@ Performance Test of Policy API
 Introduction
 ------------
 
-Performance test of policy-api has the goal of testing the min/avg/max processing time and rest call throughput for all the requests when the number of requests are large enough to saturate the resource and find the bottleneck. 
+Performance test of policy-api has the goal of testing the min/avg/max processing time and rest call throughput for all the requests when the number of requests are large enough to saturate the resource and find the bottleneck.
 
 Setup Details
 -------------
@@ -417,7 +416,7 @@ Test Plan
 ---------
 
 Performance test plan is the same as stability test plan above.
-Only differences are, in performance test, we increase the number of threads up to 20 (simulating 20 users' behaviors at the same time) whereas reducing the test time down to 1 hour. 
+Only differences are, in performance test, we increase the number of threads up to 20 (simulating 20 users' behaviors at the same time) whereas reducing the test time down to 1 hour.
 
 Run Test
 --------
diff --git a/docs/development/devtools/images/api-s3p-jm-1_F.png b/docs/development/devtools/images/api-s3p-jm-1_F.png
new file mode 100644
index 00000000..48190165
--- /dev/null
+++ b/docs/development/devtools/images/api-s3p-jm-1_F.png