SRM, Locking, Pooling documentation

SRM, Locking, Pooling  documentation
-------------------------------------------------------------

Change-Id: Id66463fc89f2c2c466ad92936816be878cfd77b7
Issue-ID: POLICY-536
Signed-off-by: Saryu Shah <ss3917@att.com>

1 files changed, 495 insertions, 0 deletions

diff --git a/docs/platform/swarch_srm.rst b/docs/platform/swarch_srm.rst
new file mode 100644
index 000000000..2de6baebc
--- /dev/null
+++ b/docs/platform/swarch_srm.rst
@@ -0,0 +1,495 @@
+
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+*****************************************
+Scalability, Resiliency and Manageability 
+*****************************************
+
+.. contents::
+    :depth: 3
+
+The new Beijing release scalability, resiliency, and manageablity are described here.   These capabilities apply to the OOM/Kubernetes installation.
+
+Installation
+^^^^^^^^^^^^
+Follow the OOM installation instructions at http://onap.readthedocs.io/en/latest/submodules/oom.git/docs/index.html
+
+Overview of the running system
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Upon initialization, you should see pools of 4 PDP-Ds and 2 PDP-Xs:
+
+.. code-block:: bash
+   :caption: verify pods
+
+    kubectl get pods --all-namespaces -o=wid
+     
+    onap    dev-brmsgw-5dbc4c8dc4-llk5s        1/1       Running   0     18m     10.42.120.43    k8sx
+    onap    dev-drools-0                       1/1       Running   0     18m     10.42.60.27     k8sx
+    onap    dev-drools-1                       1/1       Running   0     16m     10.42.105.190   k8sx
+    onap    dev-drools-2                       1/1       Running   0     15m     10.42.139.82    k8sx
+    onap    dev-drools-3                       1/1       Running   0     15m     10.42.128.4     k8sx
+    onap    dev-nexus-7d96568f5f-qp5td         1/1       Running   0     18m     10.42.172.8     k8sx
+    onap    dev-pap-8587696769-vwj6k           2/2       Running   0     18m     10.42.19.137    k8sx
+    onap    dev-pdp-0                          2/2       Running   0     18m     10.42.144.218   k8sx
+    onap    dev-pdp-1                          2/2       Running   0     15m     10.42.233.111   k8sx
+    onap    dev-policydb-587d55bdff-4f5dz      1/1       Running   0     18m     10.42.12.242    k8sx
+
+
+and a service for every component:
+
+.. code-block:: bash
+   :caption: verify services
+
+    kubectl get services --all-namespaces
+     
+    onap    brmsgw         NodePort    10.43.209.173   <none>     9989:30216/TCP                  24m
+    onap    drools         NodePort    10.43.27.92     <none>     6969:30217/TCP,9696:30221/TCP   24m
+    onap    nexus          NodePort    10.43.19.171    <none>     8081:30236/TCP                  24m
+    onap    pap            NodePort    10.43.9.166     <none>     8443:30219/TCP,9091:30218/TCP   24m
+    onap    pdp            ClusterIP   None            <none>     8081/TCP                        24m
+    onap    policydb       ClusterIP   None            <none>     3306/TCP                        24m
+
+Config and Decision policy requests will be distributed across PDP-Xs through the *pdp* service.    PDP-X clients (such as DCAE) should configure their URLs to go through the *pdp* service.   Their requests will be distributed across the available PDP-X replicas.    The PDP-Xs can be also accessed individually (dev-pdp-0 and dev-pdp-1 above), but is preferable to that external clients use the service.
+
+PDP-Ds are also accessible on a group fashion by using the service IP.   Nevertheless, as DMaaP is the main means of communication with other ONAP components, the service interface is not used heavily.
+
+
+Healthchecks
+^^^^^^^^^^^^
+
+Verify that the policy healtcheck passes by the robot framework:
+
+.. code-block:: bash
+   :caption: ~/oom/kubernetes/robot/ete-k8s.sh onap health 2> /dev/null | grep PASS
+
+    Basic Policy Health Check                                             | PASS |
+
+A policy healthcheck (with more detailed output) can be done directly to the drools service in the policy VM.
+
+.. code-block:: none
+   :caption: Healtcheck on the PDP-D service
+
+    curl --silent --user '<username>:<password> -X GET http://localhost:30217/healthcheck | python -m json.tool
+     
+    {
+        "details": [
+            {
+                "code": 200,
+                "healthy": true,
+                "message": "alive",
+                "name": "PDP-D",
+                "url": "self"
+            },
+            {
+                "code": 200,
+                "healthy": true,
+                "message": "",
+                "name": "PAP",
+                "url": "http://pap:9091/pap/test"
+            },
+            {
+                "code": 200,
+                "healthy": true,
+                "message": "",
+                "name": "PDP",
+                "url": "http://pdp:8081/pdp/test"
+            }
+        ],
+        "healthy": true
+    }
+
+
+PDP-X active/active pool
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+The policy engine UI (console container in the pap pod) can be used to check that the the 2 individual PDP-Xs are synchronized.
+The console URL is accessible at  ``http://<oom-vm>:30219/onap/login.htm``.   Select the PDP tab.
+
+    .. image:: srmPdpxPdpMgmt.png
+
+After initialization, there will be no policies loaded into the policy subsystem.    You can verify it, by accessing the Editor tab in the UI.
+
+
+PDP-D Active/Active Pool
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+The PDP-Ds replicas will come up with the amsterdam controller installed in brainless mode (no maven coordinates) since the controller has not been associated with a set of drools rules to run (control loop rules).
+
+The following command can be issued on each of the PDP-D replicas IPs:
+
+.. code-block:: bash
+   :caption: Querying the rules association for a PDP-D replica 
+
+    curl --silent --user '<username>:<password>' -X GET http://<drools-replica-ip>:9696/policy/pdp/engine/controllers/amsterdam/drools | python -m json.tool
+    
+    {
+        "alive": false,
+        "artifactId": "NO-ARTIFACT-ID",
+        "brained": false,
+        "canonicalSessionNames": [],
+        "container": null,
+        "groupId": "NO-GROUP-ID",
+        "locked": false,
+        "recentSinkEvents": [],
+        "recentSourceEvents": [],
+        "sessionNames": [],
+        "version": "NO-VERSION"
+    }
+
+Installing Policies
+^^^^^^^^^^^^^^^^^^^
+
+The OOM default installation will come with no policies pre-configured.  There is a sample script used by integration teams to load policies to support all 4 use cases at:   /tmp/policy-install/config/push-policies.sh in the pap container within the pap pod.   This script can be modified for your own particular installation, for example if only interested in vCPE use cases, remove those vCPE related API REST calls.   For the vFW use case, you may want to edit the encoded operational policy to point to the proper resourceID in your installation.
+
+The above mentioned push-policies.sh script can be executed as follows:
+
+.. code-block:: bash
+   :caption: Installing the default policies
+
+    kubectl exec -it dev-pap-8587696769-vwj6k -c pap -n onap -- bash -c "export PRELOAD_POLICIES=true; /tmp/policy-install/config/push-policies.sh"
+     
+     
+    ..
+    Create BRMSParam Operational Policies
+    ..
+    Create BRMSParamvFirewall Policy
+    ..
+    Transaction ID: ef08cc65-9950-4478-a4ab-0f3bc2519f60 --Policy with the name com.Config_BRMS_Param_BRMSParamvFirewall.1.xml was successfully created.Create BRMSParamvDNS Policy
+    ..
+    Transaction ID: 52e33efe-ba66-47de-b404-8d441107d8a9 --Policy with the name com.Config_BRMS_Param_BRMSParamvDNS.1.xml was successfully created.Create BRMSParamVOLTE Policy
+    ..
+    Transaction ID: f13072b7-6258-4c16-99da-f908d29363ec --Policy with the name com.Config_BRMS_Param_BRMSParamVOLTE.1.xml was successfully created.Create BRMSParamvCPE Policy
+    ..
+    Transaction ID: 616f970a-b45e-40f7-88cd-d63000d22cca --Policy with the name com.Config_BRMS_Param_BRMSParamvCPE.1.xml was successfully created.Create MicroService Config Policies
+    Create MicroServicevFirewall Policy
+    ..
+    Transaction ID: 4c143a15-20af-408a-9285-bc7940261829 --Policy with the name com.Config_MS_MicroServicevFirewall.1.xml was successfully created.Create MicroServicevDNS Policy
+    ..
+    Transaction ID: 1e54ae73-509b-490e-bf62-1fea7989fd5f --Policy with the name com.Config_MS_MicroServicevDNS.1.xml was successfully created.Create MicroServicevCPE Policy
+    ..
+    Transaction ID: 32239868-bab2-4e12-9fd9-81a0ed4a6b1c --Policy with the name com.Config_MS_MicroServicevCPE.1.xml was successfully created.Creating Decision Guard policy
+    ..
+    Transaction ID: b43cb9d5-42c7-4654-aacf-d4898c4d13bb --Policy with the name com.Decision_AllPermitGuard.1.xml was successfully created.Push Decision policy
+    ..
+    Transaction ID: 3c1e4ae6-6991-415b-9f2d-c665a8c5a026 --Policy 'com.Decision_AllPermitGuard.1.xml' was successfully pushed to the PDP group 'default'.Pushing BRMSParam Operational policies
+    ..
+    Transaction ID: 58d26d03-b5b8-4fd3-b2df-1411a1c36420 --Policy 'com.Config_BRMS_Param_BRMSParamvFirewall.1.xml' was successfully pushed to the PDP group 'default'.pushPolicy : PUT : com.BRMSParamvDNS
+    ..
+    Transaction ID: 0854e54a-504b-4f06-bc2f-30f491cb9f5a --Policy 'com.Config_BRMS_Param_BRMSParamvDNS.1.xml' was successfully pushed to the PDP group 'default'.pushPolicy : PUT : com.BRMSParamVOLTE
+    ..
+    Transaction ID: d33c7dde-5c99-4dab-b4ff-9988473cd88d --Policy 'com.Config_BRMS_Param_BRMSParamVOLTE.1.xml' was successfully pushed to the PDP group 'default'.pushPolicy : PUT : com.BRMSParamvCPE
+    ..
+    Transaction ID: e8c8a73e-127c-4318-9e59-3cae9dcbe011 --Policy 'com.Config_BRMS_Param_BRMSParamvCPE.1.xml' was successfully pushed to the PDP group 'default'.Pushing MicroService Config policies
+    ..
+    Transaction ID: ec0429d7-e35f-4978-8a6c-40d2b5b3be61 --Policy 'com.Config_MS_MicroServicevFirewall.1.xml' was successfully pushed to the PDP group 'default'.pushPolicy : PUT : com.MicroServicevDNS
+    ..
+    Transaction ID: f7072f05-7b74-45b5-9bd3-99b7f8023e3e --Policy 'com.Config_MS_MicroServicevDNS.1.xml' was successfully pushed to the PDP group 'default'.pushPolicy : PUT : com.MicroServicevCPE
+    ..
+    Transaction ID: 6d47db63-7956-4f5f-ab34-aeb5a124a90d --Policy 'com.Config_MS_MicroServicevCPE.1.xml' was successfully pushed to the PDP group 'default'.
+
+
+The policies pushed can be viewed through the Policy UI:
+
+    .. image:: srmEditor.png
+
+As a consequence of pushing the policies, the brmsgw component will compose drools rules artifacts and publish them to the nexus respository at ``http://<oom-vm>:30236/nexus/``
+
+    .. image:: srmNexus.png
+
+At the same time each replica of the PDP-Ds will receive notifications for each new version of the policies to run for the amsterdam controller.   You can run the following command to see how the amsterdam controller is associated with the latest rules version.    The following command can be used for verification for each replica:
+
+
+.. code-block:: none
+   :caption: Querying the rules association of a PDP-D replica
+
+    curl --silent --user '<username><password> -X GET http://<replica-ip>:9696/policy/pdp/engine/controllers/amsterdam/drools | python -m json.tool
+    {
+        "alive": true,
+        "artifactId": "policy-amsterdam-rules",
+        "brained": true,
+        "groupId": "org.onap.policy-engine.drools.amsterdam",
+        "locked": false,
+        "modelClassLoaderHash": 1223551265,
+        "recentSinkEvents": [],
+        "recentSourceEvents": [],
+        "sessionCoordinates": [
+            "org.onap.policy-engine.drools.amsterdam:policy-amsterdam-rules:0.4.0:closedloop-amsterdam"
+        ],
+        "sessions": [
+            "closedloop-amsterdam"
+        ],
+        "version": "0.4.0"
+    }
+
+Likewise, for verification purposes, each PDP-X replica can be queried directly to retrieve policy information.   The following commands can be used to query a policy through the pdp service:
+
+
+.. code-block:: bash
+   :caption: Querying the "pdp" service for the vFirewal policy
+
+    ubuntu@k8sx:~$ kubectl exec -it dev-pap-8587696769-vwj6k -c pap -n onap bash
+    policy@dev-pap-8587696769-vwj6k:/tmp/policy-install$ curl --silent -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'ClientAuth: cHl0aG9uOnRlc3Q=' --header 'Authorization: Basic dGVzdHBkcDphbHBoYTEyMw==' --header 'Environment: TEST' -d '{"policyName": ".*vFirewall.*"}' http://pdp:8081/pdp/api/getConfig | python -m json.tool
+    [
+        {
+            "config": "{\"service\":\"tca_policy\",\"location\":\"SampleServiceLocation\",\"uuid\":\"test\",\"policyName\":\"MicroServicevFirewall\",\"description\":\"MicroService vFirewall Policy\",\"configName\":\"SampleConfigName\",\"templateVersion\":\"OpenSource.version.1\",\"version\":\"1.1.0\",\"priority\":\"1\",\"policyScope\":\"resource=SampleResource,service=SampleService,type=SampleType,closedLoopControlName=ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\",\"riskType\":\"SampleRiskType\",\"riskLevel\":\"1\",\"guard\":\"False\",\"content\":{\"tca_policy\":{\"domain\":\"measurementsForVfScaling\",\"metricsPerEventName\":[{\"eventName\":\"vFirewallBroadcastPackets\",\"controlLoopSchemaType\":\"VNF\",\"policyScope\":\"DCAE\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyVersion\":\"v0.0.1\",\"thresholds\":[{\"closedLoopControlName\":\"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.vNicUsageArray[*].receivedTotalPacketsDelta\",\"thresholdValue\":300,\"direction\":\"LESS_OR_EQUAL\",\"severity\":\"MAJOR\",\"closedLoopEventStatus\":\"ONSET\"},{\"closedLoopControlName\":\"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.vNicUsageArray[*].receivedTotalPacketsDelta\",\"thresholdValue\":700,\"direction\":\"GREATER_OR_EQUAL\",\"severity\":\"CRITICAL\",\"closedLoopEventStatus\":\"ONSET\"}]}]}}}",
+            "matchingConditions": {
+                "ConfigName": "SampleConfigName",
+                "Location": "SampleServiceLocation",
+                "ONAPName": "DCAE",
+                "service": "tca_policy",
+                "uuid": "test"
+            },
+            "policyConfigMessage": "Config Retrieved! ",
+            "policyConfigStatus": "CONFIG_RETRIEVED",
+            "policyName": "com.Config_MS_MicroServicevFirewall.1.xml",
+            "policyType": "MicroService",
+            "policyVersion": "1",
+            "property": null,
+            "responseAttributes": {},
+            "type": "JSON"
+        },
+        {
+            "config":  ..... 
+            "matchingConditions": {
+                "ConfigName": "BRMS_PARAM_RULE",
+                "ONAPName": "DROOLS"
+            },
+            "policyConfigMessage": "Config Retrieved! ",
+            "policyConfigStatus": "CONFIG_RETRIEVED",
+            "policyName": "com.Config_BRMS_Param_BRMSParamvFirewall.1.xml",
+            "policyType": "BRMS_PARAM",
+            "policyVersion": "1",
+            "property": null,
+            "responseAttributes": {
+                "controller": "amsterdam"
+            },
+            "type": "OTHER"
+        }
+    ]
+    
+
+while the following commands could be used to query an specific PDP-X replica:
+
+
+.. code-block:: bash
+   :caption: Querying PDP-X 0 for the vCPE policy
+
+    curl --silent -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'ClientAuth: cHl0aG9uOnRlc3Q=' --header 'Authorization: Basic dGVzdHBkcDphbHBoYTEyMw==' --header 'Environment: TEST' -d '{"policyName": ".*vCPE.*"}' http://10.42.144.218:8081/pdp/api/getConfig | python -m json.tool
+    [
+        {
+            "config": ...,
+            "matchingConditions": {
+                "ConfigName": "BRMS_PARAM_RULE",
+                "ONAPName": "DROOLS"
+            },
+            "policyConfigMessage": "Config Retrieved! ",
+            "policyConfigStatus": "CONFIG_RETRIEVED",
+            "policyName": "com.Config_BRMS_Param_BRMSParamvCPE.1.xml",
+            "policyType": "BRMS_PARAM",
+            "policyVersion": "1",
+            "property": null,
+            "responseAttributes": {
+                "controller": "amsterdam"
+            },
+            "type": "OTHER"
+        },
+        {
+            "config": "{\"service\":\"tca_policy\",\"location\":\"SampleServiceLocation\",\"uuid\":\"test\",\"policyName\":\"MicroServicevCPE\",\"description\":\"MicroService vCPE Policy\",\"configName\":\"SampleConfigName\",\"templateVersion\":\"OpenSource.version.1\",\"version\":\"1.1.0\",\"priority\":\"1\",\"policyScope\":\"resource=SampleResource,service=SampleService,type=SampleType,closedLoopControlName=ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"riskType\":\"SampleRiskType\",\"riskLevel\":\"1\",\"guard\":\"False\",\"content\":{\"tca_policy\":{\"domain\":\"measurementsForVfScaling\",\"metricsPerEventName\":[{\"eventName\":\"Measurement_vGMUX\",\"controlLoopSchemaType\":\"VNF\",\"policyScope\":\"DCAE\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyVersion\":\"v0.0.1\",\"thresholds\":[{\"closedLoopControlName\":\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",\"thresholdValue\":0,\"direction\":\"EQUAL\",\"severity\":\"MAJOR\",\"closedLoopEventStatus\":\"ABATED\"},{\"closedLoopControlName\":\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",\"thresholdValue\":0,\"direction\":\"GREATER\",\"severity\":\"CRITICAL\",\"closedLoopEventStatus\":\"ONSET\"}]}]}}}",
+            "matchingConditions": {
+                "ConfigName": "SampleConfigName",
+                "Location": "SampleServiceLocation",
+                "ONAPName": "DCAE",
+                "service": "tca_policy",
+                "uuid": "test"
+            },
+            "policyConfigMessage": "Config Retrieved! ",
+            "policyConfigStatus": "CONFIG_RETRIEVED",
+            "policyName": "com.Config_MS_MicroServicevCPE.1.xml",
+            "policyType": "MicroService",
+            "policyVersion": "1",
+            "property": null,
+            "responseAttributes": {},
+            "type": "JSON"
+        }
+    ]
+    
+PDP-X Resiliency
+^^^^^^^^^^^^^^^^
+
+A PDP-X container failure can be simulated by performing a"policy.sh stop" operation within the PDP-X container, this in fact will shutdown the PDP-X service.    The kubernetes liveness operation will detect that the ports are down, inferring there's a problem with the service, and in turn, will restart the container.   In the following example will cause PDP-X 1 to fail.
+
+.. code-block:: bash
+   :caption: Causing PDP-X 1 service to fail
+
+    ubuntu@k8sx:~$ kubectl exec -it dev-pdp-1 --container pdp -n onap -- bash -c "source /opt/app/policy/etc/profile.d/env.sh; policy.sh stop;"
+        pdplp: STOPPING ..
+        pdp: STOPPING ..
+
+Upon detection of the service being down through the liveness check, the container will be restarted.   Note the restart count when querying the status of the pods:
+
+.. code-block:: bash
+   :caption: Checking PDP-X 1 restart count
+
+    ubuntu@k8sx:~$ kubectl get pods --all-namespaces -o=wide
+     
+    NAMESPACE  NAME                             READY     STATUS    RESTARTS   AGE     IP              NODE
+
+    onap       dev-brmsgw-5dbc4c8dc4-llk5s      1/1       Running   0          3d      10.42.120.43    k8sx
+    onap       dev-drools-0                     1/1       Running   0          3d      10.42.60.27     k8sx
+    onap       dev-drools-1                     1/1       Running   0          3d      10.42.105.190   k8sx
+    onap       dev-drools-2                     1/1       Running   0          3d      10.42.139.82    k8sx
+    onap       dev-drools-3                     1/1       Running   0          3d      10.42.128.4     k8sx
+    onap       dev-nexus-7d96568f5f-qp5td       1/1       Running   0          3d      10.42.172.8     k8sx
+    onap       dev-pap-8587696769-vwj6k         2/2       Running   0          3d      10.42.19.137    k8sx
+    onap       dev-pdp-0                        2/2       Running   0          3d      10.42.144.218   k8sx
+    onap       dev-pdp-1                        2/2       Running   1          3d      10.42.233.111   k8sx    <--- **
+    onap       dev-policydb-587d55bdff-4f5dz    1/1       Running   0          3d      10.42.12.242    k8sx
+    
+
+During the restart process, the PAP component, will detect that PDP-X 1 is down and therefore its state being reflected in the PDP-X screen:
+
+    .. image:: srmPdpxResiliencyPdpMgmt1.png
+
+This screen will be updated to reflect PDP-X 1 is back alive, after PDP-X 1 synchronizes itself with the PAP. 
+
+    .. image:: srmPdpxResiliencyPdpMgmt2.png
+
+At that point, PDP-X is usable either directly or through the service to query for policies.
+
+
+.. code-block:: bash
+   :caption: Query PDP-X 1 for vCPE policy
+
+    ubuntu@k8sx:~$ curl --silent -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'ClientAuth: cHl0aG9uOnRlc3Q=' --header 'Authorization: Basic dGVzdHBkcDphbHBoYTEyMw==' --header 'Environment: TEST' -d '{"policyName": ".*vCPE.*"}' http://10.42.233.111:8081/pdp/api/getConfig | python -m json.tool
+    [
+        {
+            "config": "..",
+            "matchingConditions": {
+                "ConfigName": "BRMS_PARAM_RULE",
+                "ONAPName": "DROOLS"
+            },
+            "policyConfigMessage": "Config Retrieved! ",
+            "policyConfigStatus": "CONFIG_RETRIEVED",
+            "policyName": "com.Config_BRMS_Param_BRMSParamvCPE.1.xml",
+            "policyType": "BRMS_PARAM",
+            "policyVersion": "1",
+            "property": null,
+            "responseAttributes": {
+                "controller": "amsterdam"
+            },
+            "type": "OTHER"
+        },
+        {
+            "config": "{\"service\":\"tca_policy\",\"location\":\"SampleServiceLocation\",\"uuid\":\"test\",\"policyName\":\"MicroServicevCPE\",\"description\":\"MicroService vCPE Policy\",\"configName\":\"SampleConfigName\",\"templateVersion\":\"OpenSource.version.1\",\"version\":\"1.1.0\",\"priority\":\"1\",\"policyScope\":\"resource=SampleResource,service=SampleService,type=SampleType,closedLoopControlName=ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"riskType\":\"SampleRiskType\",\"riskLevel\":\"1\",\"guard\":\"False\",\"content\":{\"tca_policy\":{\"domain\":\"measurementsForVfScaling\",\"metricsPerEventName\":[{\"eventName\":\"Measurement_vGMUX\",\"controlLoopSchemaType\":\"VNF\",\"policyScope\":\"DCAE\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyVersion\":\"v0.0.1\",\"thresholds\":[{\"closedLoopControlName\":\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",\"thresholdValue\":0,\"direction\":\"EQUAL\",\"severity\":\"MAJOR\",\"closedLoopEventStatus\":\"ABATED\"},{\"closedLoopControlName\":\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",\"thresholdValue\":0,\"direction\":\"GREATER\",\"severity\":\"CRITICAL\",\"closedLoopEventStatus\":\"ONSET\"}]}]}}}",
+            "matchingConditions": {
+                "ConfigName": "SampleConfigName",
+                "Location": "SampleServiceLocation",
+                "ONAPName": "DCAE",
+                "service": "tca_policy",
+                "uuid": "test"
+            },
+            "policyConfigMessage": "Config Retrieved! ",
+            "policyConfigStatus": "CONFIG_RETRIEVED",
+            "policyName": "com.Config_MS_MicroServicevCPE.1.xml",
+            "policyType": "MicroService",
+            "policyVersion": "1",
+            "property": null,
+            "responseAttributes": {},
+            "type": "JSON"
+        }
+    ]
+
+PDP-D Resiliency
+^^^^^^^^^^^^^^^^
+
+A PDP-D container failure can be simulated by performing a"policy stop" operation within the PDP-D container, this in fact will shutdown the PDP-D service.    The kubernetes liveness operation will detect that the ports are down, inferring there's a problem with the service, and in turn, will restart the container.   In the following example will cause PDP-D 3 to fail.
+
+.. code-block:: bash
+   :caption: Causing PDP-D 3 to fail
+
+    ubuntu@k8sx:~/oom/kubernetes$ kubectl exec -it dev-drools-3 --container drools -n onap -- bash -c "source /opt/app/policy/etc/profile.d/env.sh; policy stop"
+    [drools-pdp-controllers]
+    L []: Stopping Policy Management... Policy Management (pid=3284) is stopping... Policy Management has stopped.
+
+
+Upon detection of the service being down through the liveness check, the container will be restarted.   Note the restart count when querying the status of the pods:
+
+.. code-block:: bash
+   :caption: Checking PDP-D 3 restart count
+
+    ubuntu@k8sx:~/oom/kubernetes$ kubectl get pods --all-namespaces -o=wide
+    
+    NAMESPACE  NAME                             READY     STATUS    RESTARTS   AGE     IP              NODE
+
+    onap       dev-brmsgw-5549d99466-7989k      1/1       Running   0          1h      10.42.252.245   k8sx
+    onap       dev-drools-0                     1/1       Running   0          1h      10.42.30.52     k8sx
+    onap       dev-drools-1                     1/1       Running   0          1h      10.42.9.245     k8sx
+    onap       dev-drools-2                     1/1       Running   0          1h      10.42.95.0      k8sx
+    onap       dev-drools-3                     1/1       Running   1          1h      10.42.224.52    k8sx
+    onap       dev-nexus-6558979c95-xlxcc       1/1       Running   0          1h      10.42.142.36    k8sx
+    onap       dev-pap-64b67f66b9-lc8vl         2/2       Running   0          1h      10.42.187.255   k8sx
+    onap       dev-pdp-0                        2/2       Running   0          1h      10.42.164.57    k8sx
+    onap       dev-pdp-1                        2/2       Running   0          1h      10.42.155.145   k8sx
+    onap       dev-policydb-7d4b75869-qd8n5     1/1       Running   0          1h      10.42.148.37    k8sx
+   
+
+PDP-X Scaling
+^^^^^^^^^^^^^
+
+To scale a new PDP-X, set the replica count appropriately.   In our scenario below, we are going to scale the PDP-X with a new replica, PDP-X 2, to have a pool of 3 PDP-X.
+
+.. code-block:: bash
+   :caption: Scaling a PDP-X
+
+    helm upgrade -i dev local/onap --namespace onap --set policy.pdp.replicaCount=3
+     
+    Release "dev" has been upgraded. Happy Helming!
+    LAST DEPLOYED: Mon May 14 01:37:03 2018
+    NAMESPACE: onap
+    STATUS: DEPLOYED
+    ..
+     
+    kubectl get pods --all-namespaces -o=wide
+     
+    NAMESPACE  NAME                             READY     STATUS    RESTARTS   AGE     IP              NODE
+    ..
+    onap       dev-pdp-0                        2/2       Running   0          1h      10.42.164.57    k8sx
+    onap       dev-pdp-1                        2/2       Running   0          1h      10.42.155.145   k8sx
+    onap       dev-pdp-2                        2/2       Running   0          1m      10.42.47.58     k8sx
+    ..
+
+
+PDP-D Scaling
+^^^^^^^^^^^^^
+
+To scale a new PDP-D, set the replica count appropriately.   In our scenario below, we are going to scale the PDP-D service with a new replica, PDP-D 4, to have a pool of 5 PDP-D.
+
+.. code-block:: bash
+   :caption: Scaling a PDP-D
+
+    helm upgrade -i dev local/onap --namespace onap --set policy.drools.replicaCount=5
+    Release "dev" has been upgraded. Happy Helming!
+    LAST DEPLOYED: Mon May 14 01:45:19 2018
+    NAMESPACE: onap
+    STATUS: DEPLOYED
+     
+    ubuntu@k8sx:~/oom/kubernetes$ kubectl get pods --all-namespaces -o=wide
+    NAMESPACE  NAME                             READY     STATUS    RESTARTS   AGE     IP              NODE
+    ..
+    onap       dev-drools-0                     1/1       Running   0          1h      10.42.30.52     k8sx
+    onap       dev-drools-1                     1/1       Running   0          1h      10.42.9.245     k8sx
+    onap       dev-drools-2                     1/1       Running   0          1h      10.42.95.0      k8sx
+    onap       dev-drools-3                     1/1       Running   1          1h      10.42.224.52    k8sx
+    onap       dev-drools-4                     1/1       Running   0          1m      10.42.237.251   k8sx
+    ..
+    
+        
+
+
+End of Document
+
+.. SSNote: Wiki page ref. https://wiki.onap.org/display/DW/Scalability%2C+Resiliency+and+Manageability
+
+