diff options
| -rw-r--r-- | docs/development/devtools/drools-s3p.rst | 175 |
1 files changed, 86 insertions, 89 deletions
diff --git a/docs/development/devtools/drools-s3p.rst b/docs/development/devtools/drools-s3p.rst index 303a6beb..1586379e 100644 --- a/docs/development/devtools/drools-s3p.rst +++ b/docs/development/devtools/drools-s3p.rst @@ -10,31 +10,34 @@ Policy Drools PDP component ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Both the Performance and the Stability tests were executed against a default ONAP installation in the policy-k8s tenant in the windriver lab, from an independent VM running the jmeter tool to inject the load. +Both the Performance and the Stability tests were executed against an ONAP installation in the policy-k8s tenant +in the windriver lab, from an independent VM running the jmeter tool to inject the load. General Setup ************* -The kubernetes installation allocated all policy components in the same worker node VM and some additional ones. -The worker VM hosting the policy components has the following spec: +The installation runs the following components in a single VM: -- 16GB RAM -- 8 VCPU -- 160GB Ephemeral Disk +- AAF +- AAI +- DMAAP +- POLICY -The standalone VM designated to run jmeter has the same configuration. The jmeter JVM -was instantiated with a max heap configuration of 12G. +The VM has the following hardware spec: -The drools-pdp container uses the default JVM memory settings from a default OOM installation: +- 126GB RAM +- 12 VCPUs +- 155GB Ephemeral Disk -.. code-block:: bash +Jmeter is run from a different VM with the following configuration: - VM settings: - Max. Heap Size (Estimated): 989.88M - Using VM: OpenJDK 64-Bit Server VM +- 16GB RAM +- 8 VCPUs +- 155GB Ephemeral Disk +The drools-pdp container uses the JVM memory settings from a default OOM installation. -Other ONAP components used during the stability tests are: +Other ONAP components exercised during the stability tests were: - Policy XACML PDP to process guard queries for each transaction. - DMaaP to carry PDP-D and jmeter initiated traffic to complete transactions. @@ -48,19 +51,14 @@ The following components are simulated during the tests. - APPC responses for the vCPE and vFW use cases. - AAI to answer queries for the use cases under test. -In order to avoid interferences with the APPC component while running the tests, -the APPC component was disabled. - SO, and AAI actors were simulated within the PDP-D JVM by enabling the feature-controlloop-utils before running the tests. PDP-D Setup *********** -The kubernetes charts were modified previous to the installation with -the changes below. - -The feature-controlloop-utils was started by adding the following script: +The kubernetes charts were modified previous to the installation +to add the following script that enables the controlloop-utils feature: .. code-block:: bash @@ -75,7 +73,7 @@ Stability Test of Policy PDP-D PDP-D performance ================= -The test set focused on the following use cases: +The tests focused on the following use cases: - vCPE - vDNS @@ -86,23 +84,20 @@ For 72 hours the following 5 scenarios ran in parallel: - vCPE success scenario - vCPE failure scenario (failure returned by simulated APPC recipient through DMaaP). - vDNS success scenario. -- vDNS failure scenario. +- vDNS failure scenario (failure by introducing in the DCAE ONSET a non-existant vserver-name reference). - vFirewall success scenario. -Five threads ran in parallel, one for each scenario. The transactions were initiated +Five threads ran in parallel, one for each scenario, back to back with no pauses. The transactions were initiated by each jmeter thread group. Each thread initiated a transaction, monitored the transaction, and -as soon as the transaction ending was detected, it initiated the next one, so back to back with no -pauses. +as soon as the transaction ending was detected, it initiated the next one. -All transactions completed successfully as it was expected in each scenario, with no failures. - -The command executed was +JMeter was run in a docker container with the following command: .. code-block:: bash - ./jmeter -n -t /home/ubuntu/drools-applications/testsuites/stability/src/main/resources/s3p.jmx -l /home/ubuntu/jmeter_result/jmeter.jtl -e -o /home/ubuntu/jmeter_result > /dev/null 2>&1 + docker run --interactive --tty --name jmeter --rm --volume $PWD:/jmeter -e VERBOSE_GC="" egaillardon/jmeter-plugins --nongui --testfile s3p.jmx --loglevel WARN -The results were computed by monitoring the statistics REST endpoint accessible through the telemetry shell or APIs. +The results were accessed by using the telemetry API to gather statistics: vCPE Success scenario @@ -114,20 +109,11 @@ ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e: # Times are in milliseconds - # Previous to the run, there was 1 failure as a consequence of testing - # the flows before the stability load was initiated. There was - # an additional failure encountered during the execution. - - "ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e": { - "policyExecutedCount": 161328, - "policyExecutedSuccessCount": 161326, - "totalElapsedTime": 44932780, - "averageExecutionTime": 278.5181741545175, - "birthTime": 1616092087842, - "lastStart": 1616356511841, - "lastExecutionTime": 1616356541972, - "policyExecutedFailCount": 2 - } + Control Loop Name: ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e + Number of Transactions Executed: 114007 + Number of Successful Transactions: 112727 + Number of Failure Transactions: 1280 + Average Execution Time: 434.9942021103967 ms. vCPE Failure scenario @@ -139,16 +125,12 @@ ControlLoop-vCPE-Fail: # Times are in milliseconds - "ControlLoop-vCPE-Fail": { - "policyExecutedCount": 250172, - "policyExecutedSuccessCount": 0, - "totalElapsedTime": 63258856, - "averageExecutionTime": 252.8614553187407, - "birthTime": 1616092143137, - "lastStart": 1616440688824, - "lastExecutionTime": 1616440689010, - "policyExecutedFailCount": 250172 - } + Control Loop Name: ControlLoop-vCPE-Fail + Number of Transactions Executed: 114367 + Number of Successful Transactions: 114367 (failure transactions are expected) + Number of Failure Transactions: 0 (success transactions are not expected) + Average Execution Time: 433.61750330077734 ms. + vDNS Success scenario ===================== @@ -159,16 +141,12 @@ ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3: # Times are in milliseconds - "ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3": { - "policyExecutedCount": 235438, - "policyExecutedSuccessCount": 235438, - "totalElapsedTime": 37564263, - "averageExecutionTime": 159.550552587093, - "birthTime": 1616092578063, - "lastStart": 1616356511253, - "lastExecutionTime": 1616356511653, - "policyExecutedFailCount": 0 - } + Control Loop Name: ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3 + Number of Transactions Executed: 237512 + Number of Successful Transactions: 229532 + Number of Failure Transactions: 7980 + Average Execution Time: 268.028794334602 ms. + vDNS Failure scenario ===================== @@ -179,16 +157,11 @@ ControlLoop-vDNS-Fail: # Times are in milliseconds - "ControlLoop-vDNS-Fail": { - "policyExecutedCount": 2754574, - "policyExecutedSuccessCount": 0, - "totalElapsedTime": 14396495, - "averageExecutionTime": 5.22639616869977, - "birthTime": 1616092659237, - "lastStart": 1616440696444, - "lastExecutionTime": 1616440696444, - "policyExecutedFailCount": 2754574 - } + Control Loop Name: ControlLoop-vDNS-Fail + Number of Transactions Executed: 1957987 + Number of Successful Transactions: 1957987 (failure transactions are expected) + Number of Failure Transactions: 0 (success transactions are not expected) + Average Execution Time: 39.369322166081794 vFirewall Success scenario @@ -200,17 +173,41 @@ ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a: # Times are in milliseconds - # Previous to the run, there were 2 failures as a consequence of testing - # the flows before the stability load was initiated. There was - # an additional failure encountered during the execution. - - "ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a": { - "policyExecutedCount": 145197, - "policyExecutedSuccessCount": 145194, - "totalElapsedTime": 33100249, - "averageExecutionTime": 227.96785746261975, - "birthTime": 1616092985229, - "lastStart": 1616356511732, - "lastExecutionTime": 1616356541972, - "policyExecutedFailCount": 3 - } + Control Loop Name: ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a + Number of Transactions Executed: 120308 + Number of Successful Transactions: 118895 + Number of Failure Transactions: 1413 + Average Execution Time: 394.8609236293513 ms. + + +Commentary +========== + +There has been a degradation of performance observed in this release +when compared with the previous one. +Approximately 1% of transactions were not completed as expected for +some use cases. Average Execution Times are extended as well. +The unexpected results seem to point in the direction of the +interactions of the distributed locking feature with the database. +These areas as well as the conditions for the test need to be investigated +further. + +.. code-block:: bash + + # Common pattern in the audit.log for unexpected transaction completions + + a8d637fc-a2d5-49f9-868b-5b39f7befe25||ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a| + policy:usecases:[org.onap.policy.drools-applications.controlloop.common:controller-usecases:1.9.0:usecases]| + 2021-10-12T19:48:02.052+00:00|2021-10-12T19:48:02.052+00:00|0| + null:operational.modifyconfig.EVENT.MANAGER.FINAL:1.0.0|dev-policy-drools-pdp-0| + ERROR|400|Target Lock was lost|||VNF.generic-vnf.vnf-name||dev-policy-drools-pdp-0|| + dev-policy-drools-pdp-0|microservice.stringmatcher| + {vserver.prov-status=ACTIVE, vserver.is-closed-loop-disabled=false, + generic-vnf.vnf-name=fw0002vm002fw002, vserver.vserver-name=OzVServer}|||| + INFO|Session org.onap.policy.drools-applications.controlloop.common:controller-usecases:1.9.0:usecases| + + # The "Target Lock was lost" is a common message error in the unexpected results. + + +END-OF-DOCUMENT + |