Update stability test documentation

+ fix remaining rst errors, broken links,..

Issue-ID: INT-1551

Signed-off-by: mrichomme <morgan.richomme@orange.com>
Change-Id: I30a480722b34128c997dcfd4f9ae7c1143e6d87c
Signed-off-by: mrichomme <morgan.richomme@orange.com>

9 files changed, 44 insertions, 227 deletions

diff --git a/docs/automated-usecases.csv b/docs/automated-usecases.csv
new file mode 100644
index 000000000..af8c34b82
--- /dev/null
+++ b/docs/automated-usecases.csv
@@ -0,0 +1,7 @@
+Use Case;Description;Test Framework
+basic_vm;Onboard, distribute and instantiate a simple Ubuntu VM based on its heat template.;`onap-tests <https://gitlab.com/Orange-OpenSource/lfn/onap/onap-tests/-/blob/master/onap_tests/xtesting/basic_vm.py>`__
+clearwater_ims;Onboard, distribute and instantiate a clearwater vIMS (6 VMs).;`onap-tests <https://gitlab.com/Orange-OpenSource/lfn/onap/onap-tests/-/blob/master/onap_tests/xtesting/clearwater_ims.py>`__
+freeradius_nbi;Onboard, distribute and instantiate a simple VM including freeradius, the instantiation is realized through the external API module (not direct call to SO);`onap-tests <https://gitlab.com/Orange-OpenSource/lfn/onap/onap-tests/-/blob/master/onap_tests/xtesting/freeradius_nbi.py>`__
+hv-ves;Automation of 5G Realtime PM and High Volume Stream Data Collection;`robot <https://git.onap.org/testsuite/tree/robot/testsuites/hvves-ci.robot>`__
+5GBulkPm;Automation of 5G Bulk PM;`robot <https://git.onap.org/testsuite/tree/robot/testsuites/usecases/5gbulkpm.robot>`__
+pnf-registrate;Automation of 5G - PNF PnP;`robot <https://git.onap.org/testsuite/tree/robot/testsuites/pnf-registration.robot>`__
diff --git a/docs/docs_5G_oof_pci.rst b/docs/docs_5G_oof_pci.rst
index 3fbeffe46..423ec328d 100644
--- a/docs/docs_5G_oof_pci.rst
+++ b/docs/docs_5G_oof_pci.rst
@@ -1,12 +1,8 @@
 .. This work is licensed under a Creative Commons Attribution 4.0
-
    International License. http://creativecommons.org/licenses/by/4.0
 
-
 .. contents::
-
    :depth: 3
-
 ..
 
 .. _docs_5G_oof_pci:
diff --git a/docs/docs_5g_pnf_pnp.rst b/docs/docs_5g_pnf_pnp.rst
index b32087b96..c2fcc8548 100644
--- a/docs/docs_5g_pnf_pnp.rst
+++ b/docs/docs_5g_pnf_pnp.rst
@@ -22,6 +22,7 @@ The PNF Plug and Play is a procedure that is executed between a PNF and ONAP. In
 - `5G - PNF Plug and Play - Integration Test Cases <https://wiki.onap.org/display/DW/5G+-+PNF+PnP+-+Integration+Test+Cases>`_
 - `Instruction how to setup and use PnP PNF Simulator <https://gerrit.onap.org/r/gitweb?p=integration/simulators/pnf-simulator.git;a=blob_plain;f=pnfsimulator/README.md;hb=refs/heads/master>`_
 - `PnP PNF Simulator git repository <https://gerrit.onap.org/r/gitweb?p=integration/simulators/pnf-simulator.git;a=summary>`_
+
 How to Use
 ~~~~~~~~~~
 
diff --git a/docs/docs_CCVPN.rst b/docs/docs_CCVPN.rst
index 94bd4e09b..11fb83525 100644
--- a/docs/docs_CCVPN.rst
+++ b/docs/docs_CCVPN.rst
@@ -231,17 +231,3 @@ Copy csar files, which got from so-sdc-controller, to so-bpmn-infra:
 - check the /app/ASDC deretory, if doesn't exist, create it ( eg: mkdir /app/ASDC -p )
 - exit from the so-bpmn-infra ( eg: exit )
 - copy all csar files to so-bpmn-infra ( eg: kubectl.exe cp service-Siteservice-csar.csar onap/dev-so-so-bpmn-infra-54db5cd955-h7f5s:/app/ASDC/1/service-Siteservice-csar.csar )
-
-5) Manual steps in closed loop Scenario
-
-Following steps were undertaken for the closed loop testing.
-
-a. Give controller ip, username and password, trust store and key store file in restconf collector collector.properties
-b. Updated DMAAP ip in cambria.hosts in DmaapConfig.json in restconf collector and run restconf collector
-c. Followed the steps provided in this link(https://wiki.onap.org/display/DW/Holmes+User+Guide+-+Casablanca) to push CCVPN rules to holmes
-d. Followed the steps provided in this link(https://wiki.onap.org/display/DW/ONAP+Policy+Framework%3A+Installation+of+Amsterdam+Controller+and+vCPE+Policy) as reference to push CCVPN policies to policy module and updated sdnc.url, username and password in environment(/opt/app/policy/config/controlloop.properties.environment)
-
-As per wiki (Policy on OOM), push-policied.sh script is used to install policies. but I observed that CCVPN policy is not added in this script. So merged CCVPN policy using POLICY-1356 JIRA ticket. but policy is pushed by using push-policy_casablanca.sh script during integration test.
-It is found that the changes made were overwritten and hence had to patch the DG manually. This will be tracked by the JIRA SDNC-540.
-
-all above manual steps can be found  https://wiki.onap.org/display/DW/Manual+steps+for+CCVPN+Integration+Testing
diff --git a/docs/docs_pnf_onboarding_preonboarding.rst b/docs/docs_pnf_onboarding_preonboarding.rst
index deb7c3e36..bf38ca569 100644
--- a/docs/docs_pnf_onboarding_preonboarding.rst
+++ b/docs/docs_pnf_onboarding_preonboarding.rst
@@ -4,7 +4,7 @@
 .. _docs_pnf_onboarding_preonboarding:
 
 5G - PNF Pre-Onboarding & Onboarding
-----------------------
+------------------------------------
 
 Description
 ~~~~~~~~~~~
diff --git a/docs/docs_postman.rst b/docs/docs_postman.rst
deleted file mode 100644
index 46f4eec5d..000000000
--- a/docs/docs_postman.rst
+++ /dev/null
@@ -1,190 +0,0 @@
-.. This work is licensed under a Creative Commons Attribution 4.0
-.. International License.  http://creativecommons.org/licenses/by/4.0
-.. Copyright 2017 AT&T Intellectual Property.  All rights reserved.
-.. _docs_postman:
-.. _postman-guides:
-
-Test ONAP API with Postman
-==========================
-
-Postman
--------
-
-Postman is a tool that allows a user to run REST API.
-
-The user defines the API requests and has the possibility to group them
-in files called a "Collections".
-
-The user can then run each individual API request or run a complete collection.
-
-Postman includes the possibility to define "environment" variables.
-
-An API request can then get or set a value from/to that "environment" variable.
-
-Get and install Postman tool on your own machine (Linux/windows).
-
-Postman is available here: https://www.getpostman.com/
-
-
-ONAP Postman collections
-------------------------
-
-ONAP Integration project provides several Postman collections with two
-environment files.
-
-Those Postman Collections will allow a Developer to experiment various ONAP API
-on various ONAP components (SDC, NBI, SO, AAI, SDNC)
-
-- declare a vendor
-- declare a VSP
-- upload a package
-- declare a VF based on the VSP
-- declare a Service composed of the VF and a Virtual Link
-- distribute all those informations
-- declare a customer, a service subscription
-- declare OwningEntity, Platform...
-- declare a Complex, Cloud Region, Tenant
-- associate customer/service/tenant
-- declare a service instance via a serviceOrder
-- declare a vnf
-- declare a vf-module
-- declare a network
-
-A collection is also provided to delete objects
-(reminder: it is not possible to delete object in SDC)
-
-They have been tested with Onap ElAlto (they are not all compatible with
-Dublin, and there is not guaranty about ONAP "master" as API definition
-can change)
-
-
-Download ONAP Postman collections
----------------------------------
-
-From your machine, git clone the ONAP Integration project.
-
-::
-
-  git clone "https://gerrit.onap.org/r/integration"
-
-
-Import ONAP Postman collections
--------------------------------
-
-ONAP Postman collection are in the repository integration/test/postman
-
-Launch Postman tool
-
-import all ONAP Collections into Postman
-
-.. figure:: files/postman/import.png
-   :align: center
-
-And you should see all the collections into Postman
-
-.. figure:: files/postman/collections.png
-   :align: center
-
-Each collection is made of several API operations
-
-.. figure:: files/postman/collection-detail.png
-   :align: center
-
-
-Running ONAP Postman collections
---------------------------------
-
-Running all those collections, in the order, from 1 to 10 will create a lot of
-objects in ONAP components :
-
-- SDC : vendor, VSP, zip file upload, VF from VSP, Service, add VF to Service
-- VID : OwningEntity, LineOfBusiness, Project, Platform
-- AAI : customer, subscription, cloud region, tenant
-- NBI : serviceOrder to add a service instance, serviceOrder to delete
-  a service instance
-
-The order is very important because a lot of API requests will need the API
-response from the previous operation to get and set some variable values.
-
-.. figure:: files/postman/collection-detail-test.png
-   :align: center
-
-It is possible to run the complete collection using Postman
-
-.. figure:: files/postman/run.png
-   :align: center
-
-You need, a zip file that contains Heat files for a VNF.
-
-Collection 3 is about uploading that file into ONAP SDC.
-
-.. figure:: files/postman/zipfile.png
-   :align: center
-
-Before running those collections, once in Postman, you need to have a look
-at "globals" environment parameters.
-
-.. figure:: files/postman/globals.png
-   :align: center
-
-All variables that begin by "auto" must NOT be changed (they will be modified
-using API response).
-
-All other variables must be adapted to your needs.
-
-In particular, you need to put your own values for cloud_region_id, tenant_name
-and tenant_id to fit with the place where you will instantiate the VNF.
-
-
-::
-
-  service:freeradius
-  vf_name:integration_test_VF_freeradius
-  vsp_name:integration_test_VSP
-  vendor_name:onap_integration_vendor
-  owning_entity:integration_test_OE
-  platform:integration_test_platform
-  project:integration_test_project
-  lineofbusiness:integration_test_LOB
-  customer_name:generic
-  cloud_owner_name:OPNFV
-  cloud_region_id:RegionOne
-  tenant_name:openlab-vnfs
-  tenant_id:234a9a2dc4b643be9812915b214cdbbb
-  externalId:integration_test_BSS-order-001
-  service_instance_name:integration_test_freeradius_instance_001
-
-
-Using Newman
-------------
-
-Newman is a tool that allow to run postman collections via command-line
-
-On a linux server, with Docker installed on it, run those lines:
-
-::
-
-  git clone https://gitlab.com/Orange-OpenSource/lfn/onap/onap-tests.git
-  cd onap-tests/postman
-  sudo apt-get -y install zip
-  USECASE=$'ubuntu16'
-  zip -j $USECASE.zip ../onap_tests/templates/heat_files/$USECASE/*
-  TAB=$'\t\t\t\t\t\t\t'
-  sed -i -e "s/.*src.*/$TAB\"src\": \"$USECASE.zip\"/" 03_Onboard_VSP_part2.postman_collection.json
-  docker pull postman/newman:alpine
-  docker run --network="host" --volume="/home/debian/rene/onap-tests/postman:/etc/newman" postman/newman:alpine run 01_Onboard_Vendor.postman_collection.json --environment integration_test_urls.postman_environment.json --globals globals.postman_globals.json --export-environment integration_test_urls.postman_environment.json --reporters cli,json --reporter-cli-no-assertions --reporter-cli-no-console
-  docker run --network="host" --volume="/home/debian/rene/onap-tests/postman:/etc/newman" postman/newman:alpine run 02_Onboard_VSP_part1.postman_collection.json --environment integration_test_urls.postman_environment.json --globals globals.postman_globals.json --export-environment integration_test_urls.postman_environment.json
-  docker run --network="host" --volume="/home/debian/rene/onap-tests/postman:/etc/newman" postman/newman:alpine run 03_Onboard_VSP_part2.postman_collection.json --environment integration_test_urls.postman_environment.json --globals globals.postman_globals.json --export-environment integration_test_urls.postman_environment.json
-  docker run --network="host" --volume="/home/debian/rene/onap-tests/postman:/etc/newman" postman/newman:alpine run 04_Onboard_VSP_part3.postman_collection.json --environment integration_test_urls.postman_environment.json --globals globals.postman_globals.json --export-environment integration_test_urls.postman_environment.json
-  docker run --network="host" --volume="/home/debian/rene/onap-tests/postman:/etc/newman" postman/newman:alpine run 05_Onboard_VF.postman_collection.json --environment integration_test_urls.postman_environment.json --globals globals.postman_globals.json --export-environment integration_test_urls.postman_environment.json
-  docker run --network="host" --volume="/home/debian/rene/onap-tests/postman:/etc/newman" postman/newman:alpine run 06_Onboard_Service.postman_collection.json --environment integration_test_urls.postman_environment.json --globals globals.postman_globals.json --export-environment integration_test_urls.postman_environment.json
-  docker run --network="host" --volume="/home/debian/rene/onap-tests/postman:/etc/newman" postman/newman:alpine run 07_Declare_owningEntity_LineOfBusiness_project_platform.postman_collection.json --environment integration_test_urls.postman_environment.json --globals globals.postman_globals.json --export-environment integration_test_urls.postman_environment.json
-  docker run --network="host" --volume="/home/debian/rene/onap-tests/postman:/etc/newman" postman/newman:alpine run 08_Declare_Customer_Service_Subscription_Cloud.postman_collection.json --insecure --environment integration_test_urls.postman_environment.json --globals globals.postman_globals.json --export-environment integration_test_urls.postman_environment.json
-  docker run --network="host" --volume="/home/debian/rene/onap-tests/postman:/etc/newman" postman/newman:alpine run 10_instantiate_service_vnf_vfmodule.postman_collection.json --environment integration_test_urls.postman_environment.json --globals globals.postman_globals.json --export-environment integration_test_urls.postman_environment.json --reporters cli,json --reporter-cli-no-assertions --reporter-cli-no-console
-
-All collections are processed, then you can see results and you will
-also obtain result json files in the onap-tests/postman/newman directory
-
-Of course you can adapt globals variables in globals.postman_globals.json
-or change the USECASE=$'ubuntu16' value to onboard any heat template located
-in onap_tests/templates/heat_files directory
diff --git a/docs/docs_usecases.rst b/docs/docs_usecases.rst
index bdb39cedb..ff5e18bff 100644
--- a/docs/docs_usecases.rst
+++ b/docs/docs_usecases.rst
@@ -42,7 +42,6 @@ Functional Requirements
     :widths: 60,20,20
     :header-rows: 1
 
-
 .. csv-table:: 5G functional requirements table
     :file: functional-requirements-5g.csv
     :widths: 60,20,20
@@ -58,11 +57,20 @@ Some use cases have been integrated in ONAP gates. It means the tests are run on
 each daily or gating CI chain. The goal is to detect any regression as soon as
 possible and demonstrate the ability to automate the use cases.
 
-The frankfurt use cases integrated in gates are:
+.. csv-table:: automated use cases table
+    :file: automated-usecases.csv
+    :widths: 10,80,10
+    :delim: ;
+    :header-rows: 1
+
+The robot scripts can be found in ONAP testsuite repository, an execution
+run-time is provided through the robot pod.
 
-- pnf-regitrate
-- 5gbulkpm
-- hv-ves
+The python onap_tests framework is hosted on
+https://gitlab.com/Orange-OpenSource/lfn/onap/onap-tests. Please not that this
+framework is valid up to Frankfurk and will be deprecated in Guilin. It will
+be replaced by scenarios leveraging python-onapsdk
+https://gitlab.com/Orange-OpenSource/lfn/onap/python-onapsdk.
 
 Deprecated Use Cases
 ~~~~~~~~~~~~~~~~~~~~
diff --git a/docs/docs_vFW_CNF_CDS.rst b/docs/docs_vFW_CNF_CDS.rst
index d7c93e033..77b618e5b 100644
--- a/docs/docs_vFW_CNF_CDS.rst
+++ b/docs/docs_vFW_CNF_CDS.rst
@@ -545,7 +545,7 @@ where my_cba.zip is the cba model of this use case and input_uat.yml is followin
 
 
 .. note::  This call will run all the calls (given in input_uat.yml) towards CDS and records the functionality, so there needs to be working environment (SDNC, AAI, Naming, Netbox, etc.) to record valid final uat.yml.
-As an output of this call final uat.yml content is received. Final uat.yml in this use case looks like this:
+           As an output of this call final uat.yml content is received. Final uat.yml in this use case looks like this:
 
 ::
 
@@ -2009,14 +2009,16 @@ Service Creation with SDC
 Service Creation in SDC is composed of the same steps that are performed by most other use-cases. For reference, you can relate to `vLB use-case`_
 
 Onboard VSP
-    - Remember during VSP onboard to choose "Network Package" Onboarding procedure
+
+- Remember during VSP onboard to choose "Network Package" Onboarding procedure
 
 Create VF and Service
 Service -> Properties Assignment -> Choose VF (at right box):
-    - skip_post_instantiation_configuration - True
-    - sdnc_artifact_name - vnf
-    - sdnc_model_name - vFW_CNF_CDS
-    - sdnc_model_version - K8s 1.0.45
+
+- skip_post_instantiation_configuration - True
+- sdnc_artifact_name - vnf
+- sdnc_model_name - vFW_CNF_CDS
+- sdnc_model_version - K8s 1.0.45
 
 Distribution Of Service
 .......................
diff --git a/docs/integration-s3p.rst b/docs/integration-s3p.rst
index 835c08e1b..49c67850f 100644
--- a/docs/integration-s3p.rst
+++ b/docs/integration-s3p.rst
@@ -28,13 +28,15 @@ functional after running for an extended amounts of time.
 This is done by repeated running tests against an ONAP instance for a period of
 72 hours.
 
-The 72 hour stability run result was **PASS**.
+::
+
+  **The 72 hour stability run result was PASS**
 
-The onboard and instantiate tests ran for over 115 hours before environment
+The onboard and instantiate tests ran for over **115 hours** before environment
 issues stopped the test. There were errors due to both tooling and environment
 errors.
 
-The overall memory utilization only grew about 2% on the work nodes despite
+The overall memory utilization only grew about **2%** on the work nodes despite
 the environment issues. Interestingly the kubernetes ochestration node memory
 grew more which could mean we are over driving the API's in some fashion.
 
@@ -84,10 +86,15 @@ The jenkins job needs a couple of updates for each release:
 The path for robot logs on dockerdata-nfs  changed in Frankfurt so the
 /dev-robot/ becomes /dev/robot
 
-The stability tests used robot container image  **1.6.1-STAGING-20200519T201214Z**.
-
-robot container updates: API_TYPE was set to GRA_API since we have deprecated
-VNF_API.
+.. note::
+   For Frankfurt release, the  stability test has been executed on an
+   kubernetes infrastructure based on El Alto recommendations. The kubernetes
+   version was 1.15.3 (frankfurt 1.15.11) and the helm version was 2.14.2
+   (frankfurt 2.16.6). However the ONAP dockers were updated to Frankfurt RC2
+   candidate versions. The results are informative and can be compared with
+   previous campaigns. The stability tests used robot container image
+   **1.6.1-STAGING-20200519T201214Z**. Robot container was patched to use GRA_API
+   since VNF_API has been deprecated.
 
 Shakedown consists of creating some temporary tags for stability72hrvLB,
 stability72hrvVG,stability72hrVFWCL to make sure each sub test ran successfully