diff options
Diffstat (limited to 'docs')
18 files changed, 127 insertions, 63 deletions
diff --git a/docs/apex/APEX-DecisionMakerExample.rst b/docs/apex/APEX-DecisionMakerExample.rst index 57791ad6..05aeefa1 100644 --- a/docs/apex/APEX-DecisionMakerExample.rst +++ b/docs/apex/APEX-DecisionMakerExample.rst @@ -39,7 +39,7 @@ APEX Examples Decision Maker .. |File > New to create a new Policy Model| image:: images/mfp/MyFirstPolicy_P1_newPolicyModel1.png .. |Create a new Policy Model| image:: images/mfp/MyFirstPolicy_P1_newPolicyModel2.png -.. |ONAP| image:: ../../../images/logos.png +.. |ONAP| image:: ../_static/logo_onap_2017.png :class: builtBy :target: http://www.onap.org/ .. |apexConfigRESTClient_link| raw:: html diff --git a/docs/apex/APEX-User-Manual.rst b/docs/apex/APEX-User-Manual.rst index 175f539d..c48583a8 100644 --- a/docs/apex/APEX-User-Manual.rst +++ b/docs/apex/APEX-User-Manual.rst @@ -8,8 +8,8 @@ APEX User Manual .. contents:: :depth: 3 -Installation -^^^^^^^^^^^^ +Installation of Apex +^^^^^^^^^^^^^^^^^^^^ Requirements ------------ @@ -226,13 +226,14 @@ Build APEX +----------------------------------------------------------------------------------------------------------------+ | Unix, Cygwin | +================================================================================================================+ -| .. container:: | +| .. container:: content | | | | .. container:: listingblock | | | | .. container:: content | | | | .. code:: | +| | | :number-lines: | | | | -rwxrwx---+ 1 esvevan Domain Users 772 Sep 3 11:55 apex-pdp-package-full_2.0.0~SNAPSHOT_all.changes* | diff --git a/docs/architecture/architecture.rst b/docs/architecture/architecture.rst index 4c8b3f7b..7edf9e07 100644 --- a/docs/architecture/architecture.rst +++ b/docs/architecture/architecture.rst @@ -393,7 +393,7 @@ this information to automatically generate a policy. Note that SDC provides a wrapper for the SDC API as a Java Client and also provides a TOSCA parser. See the documentation for the `Policy Distribution Component -<https://docs.onap.org/en/latest/submodules/policy/distribution.git/docs/index.html>`__. +<https://docs.onap.org/projects/onap-policy-parent/en/latest/distribution/distribution.html>`__. In Step 4 above, the \ *PolicyDesign* must download the CSAR file. If the policy is to be composed from the TOSCA definition, it must also parse the TOSCA definition. diff --git a/docs/development/devtools/api-s3p.rst b/docs/development/devtools/api-s3p.rst index 439719be..55867b44 100644 --- a/docs/development/devtools/api-s3p.rst +++ b/docs/development/devtools/api-s3p.rst @@ -135,14 +135,54 @@ average a 10 seconds plus response time. Performance Test of Policy API ++++++++++++++++++++++++++++++ -A specific performance test was omitted in Honululu (as in Guilin). The JMeter script used in the stability run injected -back to back traffic with 5 parallel threads with no pauses between requests. Since the JMeter threads operate -in synchronous mode (waiting for a request's response before sending the next request), JMeter injection rates autoregulate -because of the backpressure imposed by the response times. Even though the response times are high, the -"Response over Time" graph above indicates that they remain constant at large, throughout the duration of the test. -This together with the absence of notorious spikes in the kubernetes node CPU utilization suggests that the API -component is not strained. A more enlightning set of tests, would plot jmeter threads (increasing load) -against response times. These tests have not been performed in this release. +Introduction +------------ + +Performance test of policy-api has the goal of testing the min/avg/max processing time and rest call throughput for all the requests when the number of requests are large enough to saturate the resource and find the bottleneck. + +Setup Details +------------- + +The performance test was performed on a default ONAP OOM installation in the Intel Wind River Lab environment. +JMeter was installed on a separate VM to inject the traffic defined in the +`API performace script +<https://git.onap.org/policy/api/tree/testsuites/performance/src/main/resources/testplans/policy_api_performance.jmx>`_ +with the following command: + +.. code-block:: bash + + jmeter.sh --nongui --testfile policy_api_performance.jmx --logfile result.jtl + + +Test Plan +--------- + +Performance test plan is the same as stability test plan above. +Only differences are, in performance test, we increase the number of threads up to 20 (simulating 20 users' behaviors at the same time) whereas reducing the test time down to 2.5 hours. + +Run Test +-------- + +Running/Triggering performance test will be the same as stability test. That is, launch JMeter pointing to corresponding *.jmx* test plan. The *API_HOST* and *API_PORT* are already set up in *.jmx*. + +**Test Statistics** + +======================= ============= =========== =============================== =============================== =============================== +**Total # of requests** **Success %** **TPS** **Avg. time taken per request** **Min. time taken per request** **Max. time taken per request** +======================= ============= =========== =============================== =============================== =============================== + 4082 100% 0.45 1297 ms 4 ms 63612 ms +======================= ============= =========== =============================== =============================== =============================== + +.. image:: images/api-s3p-jm-2_H.png + +Test Results +------------ + +The following graphs show the response time distributions. + +.. image:: images/api-response-time-distribution_performance_H.png +.. image:: images/api-response-time-overtime_performance_H.png + diff --git a/docs/development/devtools/distribution-s3p.rst b/docs/development/devtools/distribution-s3p.rst index 1db411c6..13c47924 100644 --- a/docs/development/devtools/distribution-s3p.rst +++ b/docs/development/devtools/distribution-s3p.rst @@ -121,7 +121,7 @@ Run the setup_components.sh script to start the test support components: .. code-block:: bash - ~/distribution/testsuites/stability/src/main/resources/distributionsetup/setup_distribution.sh + ~/distribution/testsuites/stability/src/main/resources/simulatorsetup/setup_components.sh After installation, ensure the following docker containers are up and running: @@ -182,7 +182,7 @@ Download and install JMeter VM2 Only: Install & configure visualVM -------------------------------------- -VisualVM needs to be installed in the virtual machine running Distrbution (VM2). It will be used to monitor CPU, Memory and GC for Distribution while the stability tests are running. +VisualVM needs to be installed in the virtual machine running Distribution (VM2). It will be used to monitor CPU, Memory and GC for Distribution while the stability tests are running. .. code-block:: bash @@ -222,7 +222,7 @@ This will load up the visualVM GUI Connect to Distribution JMX Port. 1. Right click on "Local" in the left panel of the screen and select "Add JMX Connection" - 2. Enter the Port 9090. this is the JMX port exposed by the dsitribution container + 2. Enter the Port 9090. this is the JMX port exposed by the distribution container 3. Double click on the newly added nodes under "Local" to start monitoring CPU, Memory & GC. Example Screenshot of visualVM @@ -236,9 +236,9 @@ Stability Test of Policy Distribution Introduction ------------ -The 72 hour Stability Test for policy distribution has the goal of introducing a steady flow of transactions initiated from a test client server running JMeter. The policy distribution is configured with a special FileSystemReception plugin to monitor a local directory for newly added csar files to be processed by itself. The input CSAR will be added/removed by the test client(JMeter) and the result will be pulled from the backend(PAP and PolicyAPI) by the test client (JMeter). +The 72 hour Stability Test for policy distribution has the goal of introducing a steady flow of transactions initiated from a test client server running JMeter. The policy distribution is configured with a special FileSystemReception plugin to monitor a local directory for newly added csar files to be processed by itself. The input CSAR will be added/removed by the test client(JMeter) and the result will be pulled from the backend (PAP and PolicyAPI) by the test client (JMeter). -The test will be performed in an environment where Jmeter will continuously add/remove a test csar into the special directory where policy distribuion is monitoring and will then get the processed results from PAP and PolicyAPI to verify the successful deployment of the policy. The policy will then be undeployed and the test will loop continuously until 72 hours have elapsed. +The test will be performed in an environment where Jmeter will continuously add/remove a test csar into the special directory where policy distribution is monitoring and will then get the processed results from PAP and PolicyAPI to verify the successful deployment of the policy. The policy will then be undeployed and the test will loop continuously until 72 hours have elapsed. Test Plan Sequence @@ -291,7 +291,7 @@ From the apache JMeter folder run the test for 72h, pointing it towards the stab .. code-block:: bash - ~/jmeter/apache-jmeter-5.1.1/bin/jmeter -n -t ~/distribution/testsuites/stability/src/main/resources/testplans/stability.jmx -Jduration=259200 -l ~/20201016-1715-distr-stability.jtl & + ~/jmeter/apache-jmeter-5.1.1/bin/jmeter -n -t ~/distribution/testsuites/stability/src/main/resources/testplans/stability.jmx -Jduration=259200 -l ~/distr-stability.jtl & Test Results @@ -304,18 +304,13 @@ Test Results **Test Statistics** -.. csv-table:: Stability Results - Summary Report - :file: csv/20201016-1715-distr-stability-summary.csv - :header-rows: 1 - -.. csv-table:: Stability Results - Aggregate Report - :file: csv/20201016-1715-distr-stability-aggregate.csv - :header-rows: 1 +.. image:: images/dist_stability_statistics.PNG +.. image:: images/dist_stability_threshold.PNG **VisualVM Screenshots** -.. image:: images/20201016-1715-distr-stability-20201018T2040-monitor.png -.. image:: images/20201016-1715-distr-stability-20201018T2040-threads.png +.. image:: images/dist_stability_monitor.PNG +.. image:: images/dist_stability_threads.PNG Performance Test of Policy Distribution @@ -342,7 +337,7 @@ Performance test plan is different from the stability test plan. - Instead of handling one policy csar at a time, multiple csar's are deployed within the watched folder at the exact same time. - We expect all policies from these csar's to be deployed within 30 seconds. -- There are also multithreaded tests running towards the healtchcheck and statistics endpoints of the distribution service. +- There are also multithreaded tests running towards the healthcheck and statistics endpoints of the distribution service. Running the Test Plan @@ -359,7 +354,7 @@ From the apache JMeter folder run the test for 4h, pointing it towards the perfo .. code-block:: bash - ~/jmeter/apache-jmeter-5.1.1/bin/jmeter -n -t ~/distribution/testsuites/performance/src/main/resources/testplans/performance.jmx -Jduration=14400 -l ~/20201020-1730-distr-performance.jtl & + ~/jmeter/apache-jmeter-5.1.1/bin/jmeter -n -t ~/distribution/testsuites/performance/src/main/resources/testplans/performance.jmx -Jduration=14400 -l ~/distr-performance.jtl & Test Results ------------ @@ -371,13 +366,8 @@ Test Results **Test Statistics** -.. csv-table:: Performance Results - Summary Report - :file: csv/20201020-1730-distr-performance-summary.csv - :header-rows: 1 - -.. csv-table:: Performance Results - Aggregate Report - :file: csv/20201020-1730-distr-performance-aggregate.csv - :header-rows: 1 +.. image:: images/dist_perf_statistics.PNG +.. image:: images/dist_perf_threshold.PNG **VisualVM Screenshots** diff --git a/docs/development/devtools/images/20201016-1715-distr-stability-20201018T2040-monitor.png b/docs/development/devtools/images/20201016-1715-distr-stability-20201018T2040-monitor.png Binary files differdeleted file mode 100755 index abdba921..00000000 --- a/docs/development/devtools/images/20201016-1715-distr-stability-20201018T2040-monitor.png +++ /dev/null diff --git a/docs/development/devtools/images/20201016-1715-distr-stability-20201018T2040-threads.png b/docs/development/devtools/images/20201016-1715-distr-stability-20201018T2040-threads.png Binary files differdeleted file mode 100755 index 2a9745ae..00000000 --- a/docs/development/devtools/images/20201016-1715-distr-stability-20201018T2040-threads.png +++ /dev/null diff --git a/docs/development/devtools/images/api-response-time-distribution_performance_H.png b/docs/development/devtools/images/api-response-time-distribution_performance_H.png Binary files differnew file mode 100644 index 00000000..68a5c79e --- /dev/null +++ b/docs/development/devtools/images/api-response-time-distribution_performance_H.png diff --git a/docs/development/devtools/images/api-response-time-overtime_performance_H.png b/docs/development/devtools/images/api-response-time-overtime_performance_H.png Binary files differnew file mode 100644 index 00000000..63fc859a --- /dev/null +++ b/docs/development/devtools/images/api-response-time-overtime_performance_H.png diff --git a/docs/development/devtools/images/api-s3p-jm-2_H.png b/docs/development/devtools/images/api-s3p-jm-2_H.png Binary files differnew file mode 100644 index 00000000..da9d1e92 --- /dev/null +++ b/docs/development/devtools/images/api-s3p-jm-2_H.png diff --git a/docs/development/devtools/images/dist_perf_statistics.PNG b/docs/development/devtools/images/dist_perf_statistics.PNG Binary files differnew file mode 100644 index 00000000..eeefeeee --- /dev/null +++ b/docs/development/devtools/images/dist_perf_statistics.PNG diff --git a/docs/development/devtools/images/dist_perf_threshold.PNG b/docs/development/devtools/images/dist_perf_threshold.PNG Binary files differnew file mode 100644 index 00000000..58fbffd1 --- /dev/null +++ b/docs/development/devtools/images/dist_perf_threshold.PNG diff --git a/docs/development/devtools/images/dist_stability_monitor.PNG b/docs/development/devtools/images/dist_stability_monitor.PNG Binary files differnew file mode 100644 index 00000000..83eae8cc --- /dev/null +++ b/docs/development/devtools/images/dist_stability_monitor.PNG diff --git a/docs/development/devtools/images/dist_stability_statistics.PNG b/docs/development/devtools/images/dist_stability_statistics.PNG Binary files differnew file mode 100644 index 00000000..dce9b7cc --- /dev/null +++ b/docs/development/devtools/images/dist_stability_statistics.PNG diff --git a/docs/development/devtools/images/dist_stability_threads.PNG b/docs/development/devtools/images/dist_stability_threads.PNG Binary files differnew file mode 100644 index 00000000..13e27c99 --- /dev/null +++ b/docs/development/devtools/images/dist_stability_threads.PNG diff --git a/docs/development/devtools/images/dist_stability_threshold.PNG b/docs/development/devtools/images/dist_stability_threshold.PNG Binary files differnew file mode 100644 index 00000000..d65e8cc3 --- /dev/null +++ b/docs/development/devtools/images/dist_stability_threshold.PNG diff --git a/docs/development/devtools/xacml-s3p.rst b/docs/development/devtools/xacml-s3p.rst index 74369fc2..1411f90b 100644 --- a/docs/development/devtools/xacml-s3p.rst +++ b/docs/development/devtools/xacml-s3p.rst @@ -7,22 +7,20 @@ .. toctree:: :maxdepth: 2 -Policy XACML PDP component ########################## -Both the Performance and the Stability tests were executed by performing requests -against the Policy RESTful APIs residing on the XACML PDP installed in the windriver -lab. This was running on a kubernetes pod having the following configuration: +Performance Test of Policy XACML PDP +************************************ + +The Performance test was executed by performing requests +against the Policy RESTful APIs residing on the XACML PDP installed on a Cloud based Virtual Machine. +VM Configuration: - 16GB RAM - 8 VCPU -- 160GB Disk - -Both tests were run via jmeter, which was installed on a separate VM so-as not -to impact the performance of the XACML-PDP being tested. +- 1TB Disk -Performance Test of Policy XACML PDP -************************************ +ONAP was deployed using a K8s Configuration on a separate VM. Summary ======= @@ -34,9 +32,7 @@ The Performance test was executed, and the result analyzed, via: jmeter -Jduration=1200 -Jusers=10 \ -Jxacml_ip=$ip -Jpap_ip=$ip -Japi_ip=$ip \ -Jxacml_port=31104 -Jpap_port=32425 -Japi_port=30709 \ - -n -t perf.jmx - - ./result.sh + -n -t perf.jmx -l testresults.jtl Note: the ports listed above correspond to port 6969 of the respective components. @@ -68,15 +64,26 @@ threads), with the following results: .. csv-table:: :header: "Number of Users", "Throughput (requests/second)", "Average Latency (ms)" - 10, 6064, 4.1 - 20, 6495, 7.2 - 40, 6457, 12.2 - 80, 5803, 21.3 + 10, 8929, 3.10 + 20, 10827, 5.05 + 40, 11800, 9.35 + 80, 11750, 18.62 Stability Test of Policy XACML PDP ************************************ +The stability test was executed by performing requests +against the Policy RESTful APIs residing on the XACML PDP installed in the windriver +lab. This was running on a kubernetes pod having the following configuration: + +- 16GB RAM +- 8 VCPU +- 160GB Disk + +The test was run via jmeter, which was installed on a separate VM so-as not +to impact the performance of the XACML-PDP being tested. + Summary ======= @@ -91,21 +98,42 @@ with the following command: jmeter.sh -Jduration=259200 -Jusers=2 -Jxacml_ip=$ip -Jpap_ip=$ip -Japi_ip=$ip \ -Jxacml_port=31104 -Jpap_port=32425 -Japi_port=30709 --nongui --testfile stability.jmx +Note: the ports listed above correspond to port 6969 of the respective components. + The default log level of the root and org.eclipse.jetty.server.RequestLog loggers in the logback.xml of the XACML PDP (om/kubernetes/policy/components/policy-xacml-pdp/resources/config/logback.xml) was set to ERROR since the OOM installation did not have log rotation enabled of the container logs in the kubernetes worker nodes. +The stability test, stability.jmx, runs the following, all in parallel: + +- Healthcheck, 2 simultaneous threads +- Statistics, 2 simultaneous threads +- Decisions, 2 simultaneous threads, each running the following tasks in sequence: + - Monitoring Decision + - Monitoring Decision, abbreviated + - Naming Decision + - Optimization Decision + - Default Guard Decision (always "Permit") + - Frequency Limiter Guard Decision + - Min/Max Guard Decision + +When the script starts up, it uses policy-api to create, and policy-pap to deploy +the policies that are needed by the test. It assumes that the "naming" policy has +already been created and deployed. Once the test completes, it undeploys and deletes +the policies that it previously created. + Results ======= -The stability summary results were reported by JMeter with the following line: +The stability summary results were reported by JMeter with the following summary line: .. code-block:: bash - 2020-10-23 19:44:31,515 INFO o.a.j.r.Summariser: summary = 1061746369 in 72:00:16 = 4096.0/s Avg: 0 Min: 0 Max: 2584 Err: 0 (0.00%) + summary = 207771010 in 72:00:01 = 801.6/s Avg: 6 Min: 0 Max: 411 Err: 0 (0.00%) -The XACML PDP offered good performance with JMeter for the traffic mix described above, creating 4096 threads per second +The XACML PDP offered good performance with JMeter for the traffic mix described above, using 801 threads per second to inject the traffic load. No errors were encountered, and no significant CPU spikes were noted. +The average transaction time was 6ms. with a maximum of 411ms. diff --git a/docs/release-notes.rst b/docs/release-notes.rst index 6d7a5401..852498d2 100644 --- a/docs/release-notes.rst +++ b/docs/release-notes.rst @@ -89,7 +89,7 @@ System Limitations The policy API component requires a fresh new database when migrating to the honolulu release. Therefore, upgrades require a fresh new database installation. Please see the -`Installing or Upgrading Policy <https://onap.readthedocs.io/en/latest/submodules/policy/parent.git/docs/installation/oom.html#installing-or-upgrading-policy>`__ section for appropriate procedures. +`Installing or Upgrading Policy <https://docs.onap.org/projects/onap-policy-parent/en/honolulu/installation/oom.html#installing-or-upgrading-policy>`__ section for appropriate procedures. Known Vulnerabilities ~~~~~~~~~~~~~~~~~~~~~ @@ -97,6 +97,8 @@ Known Vulnerabilities Workarounds ~~~~~~~~~~~ +* `POLICY-2998 <https://jira.onap.org/browse/POLICY-2998>`_ - Provide a script to periodically purge the statistics table + Security Notes ============== @@ -117,6 +119,9 @@ Security Notes - Upgrade io.cucumber to 6.9.1 - Upgrade org.apache.commons:commons-lang3 to 3.11 - Upgrade commons-io to 2.8.0 +* `POLICY-2943 <https://jira.onap.org/browse/POLICY-2943>`_ - Review license scan issues + - Upgrade com.hazelcast to 4.1.1 + - Upgrade io.netty to 4.1.58.Final * `POLICY-2936 <https://jira.onap.org/browse/POLICY-2936>`_ - Upgrade to latest version of CDS API - Upgrade io.grpc to 1.35.0 - Upgrade com.google.protobuf to 3.14.0 @@ -220,7 +225,7 @@ System Limitations The policy API component requires a fresh new database when migrating to the guilin release. Therefore, upgrades require a fresh new database installation. Please see the -`Installing or Upgrading Policy <https://onap.readthedocs.io/en/latest/submodules/policy/parent.git/docs/installation/oom.html#installing-or-upgrading-policy>`__ section for appropriate procedures. +`Installing or Upgrading Policy <https://docs.onap.org/projects/onap-policy-parent/en/guilin/installation/oom.html#installing-or-upgrading-policy>`__ section for appropriate procedures. Known Vulnerabilities ~~~~~~~~~~~~~~~~~~~~~ @@ -471,14 +476,14 @@ POLICY-APEX-PDP * Passing parameters from ApexConfig to policy logic. - TaskParameters can be used to pass parameters from ApexConfig to the policy logic. Consider a scenario where from CLAMP, serviceId or closedLoopId has to be passed to the policy, and this should be available to perform some logic or action within the policy. In the CLAMP UI, while configuring the APEX Policy, specifying taskParameters with these will enable this. - - More information about the usage of Task Parameters can be found here: https://onap.readthedocs.io/en/latest/submodules/policy/parent.git/docs/apex/APEX-User-Manual.html#configure-task-parameters + - More information about the usage of Task Parameters can be found here: https://docs.onap.org/projects/onap-policy-parent/en/frankfurt/apex/APEX-User-Manual.html#configure-task-parameters - In the taskLogic, taskParameters can be accessed as executor.parameters.get("ParameterKey1")) - - More information can be found here: https://onap.readthedocs.io/en/latest/submodules/policy/parent.git/docs/apex/APEX-Policy-Guide.html#accessing-taskparameters + - More information can be found here: https://docs.onap.org/projects/onap-policy-parent/en/frankfurt/apex/APEX-Policy-Guide.html#accessing-taskparameters * GRPC support for APEX-CDS interaction. - - APEX-PDP now supports interaction with CDS over gRPC. Up through El Alto, CDS interaction was possible over REST only. A new plugin was developed in APEX for this feature. Refer the link for more details. https://onap.readthedocs.io/en/latest/submodules/policy/parent.git/docs/apex/APEX-User-Manual.html#grpc-io + - APEX-PDP now supports interaction with CDS over gRPC. Up through El Alto, CDS interaction was possible over REST only. A new plugin was developed in APEX for this feature. Refer the link for more details. https://docs.onap.org/projects/onap-policy-parent/en/frankfurt/apex/APEX-User-Manual.html#grpc-io POLICY-XACML-PDP ================ @@ -525,7 +530,7 @@ System Limitations The policy API component requires a fresh new database when migrating to the frankfurt release. Therefore, upgrades require a fresh new database installation. Please see the -`Installing or Upgrading Policy <https://onap.readthedocs.io/en/frankfurt/submodules/policy/parent.git/docs/installation/oom.html#installing-or-upgrading-policy>`__ section for appropriate procedures. +`Installing or Upgrading Policy <https://docs.onap.org/projects/onap-policy-parent/en/frankfurt/installation/oom.html#installing-or-upgrading-policy>`__ section for appropriate procedures. Known Vulnerabilities ~~~~~~~~~~~~~~~~~~~~~ |
