5.1.0 policy-handler - policy-updates from new PDP5.1.0

DCAEGEN2-1851:
- policy-handler now supports the policy-update notification
  from the new policy-engine thru DMaaP MR
  = no policy-filters - only policy-id values
- see README for discoverable config settings of dmaap_mr client
  = DMaaP MR client has the same flexibility as policy_engine
  = set the query.timeout to high value like 15000 (default)
- requests to DMaaP MR go through a single blocking connection
- first catch-up only after draining the policy-updates from DMaaP MR
  on the first loop
- safe parsing of messages from DMaaP MR
- policy-engine changed the data type for policy-version field
  from int to string that is expected to have the semver value
- related change to deployment-handler (DCAEGEN2-2085) has to be
  deployed to handle the non-numeric policyVersion
- on new PDP API: http /policy_latest and policy-updates
  return the new data from the new PDP API with the following fields
  added/renamed by the policy-handler to keep other policy related parts
  intact in R4-R6 (see pdp_api/policy_utils.py)
  * policyName = policy_id + "." + policyVersion.replace(".","-")
                                 + ".xml"
  * policyVersion = str(metadata["policy-version"])
  * "config" - is the renamed "properties" from the new PDP API response
- enabled the /catch_up and the periodic auto-catch-up for the new PDP
  API
- enabled GET /policies_latest - returns the latest policies for the
  deployed components
- POST /policies_latest - still disabled since no support for the
  policy-filters is provided for the new PDP API
- fixed hiding the Authorization value on comparing the configs
- logging of secrets is now sha256 to see whether they changed
- added X-ONAP-RequestID to headers the same way as X-ECOMP-RequestID
- on policy-update process the removal first, then addition
- changed the pool_connections=1 (number of pools) on PDP and DH sides
  == only a single destination is expected for each
- log the exception as fatal into error.log
- other minor fixes and refactoring
- unit-test coverage 74%
- integration testing is requested

DCAEGEN2-1976:
- policy-handler is enhanced to get user/password from env vars
  for PDP and DMaaP MR clients and overwriting the Authorization field
  in https headers received from the discoverable config
  = to override the Authorization value on policy_engine,
    set the environment vars $PDP_USER and $PDP_PWD in policy-handler
    container
  = to override the Authorization value on dmaap_mr,
    if using https and user-password authentication,
    set the environment vars $DMAAP_MR_USER and $DMAAP_MR_PWD in
    policy-handler container

Change-Id: Iad8eab9e20e615a0e0d2822f4735dc64c50aa55c
Signed-off-by: Alex Shatov <alexs@att.com>
Issue-ID: DCAEGEN2-1851
Issue-ID: DCAEGEN2-1976

1 files changed, 59 insertions, 2 deletions

diff --git a/README.md b/README.md
index 3266b2f..4427324 100644
--- a/README.md
+++ b/README.md
@@ -143,9 +143,9 @@ make sure that both of the following settings are set properly
 }
 ```
 
-#### point the discovarable config of the policy-handler to point to the **new PDP API**
+#### the discovarable config of the policy-handler to point to the **new PDP API**
 
-In short: keep the consul-kv record for he policy-handler as before R4 Dublin.
+In short: keep the consul-kv record for the policy-handler as before R4 Dublin.
 
 Here is a sample config from consul-kv.  Please replace the {{ ... }} with real setup values
 
@@ -201,6 +201,19 @@ Here is a sample config from consul-kv.  Please replace the {{ ... }} with real
       "tls_ca_mode": "cert_directory",
       "timeout_in_secs": 60
     },
+    "dmaap_mr" : {
+        "url" : "http://{{ YOUR_DMAAP_MR_URL }}/events/{{ POLICY_UPDATE_TOPICNAME }}/{{ POLICY_UPDATE_CONSUMEGROUP }}/{{ POLICY_UPDATE_CONSUMERID }}",
+        "query": {
+          "timeout": 15000
+        },
+        "headers" : {
+            "Content-Type" : "application/json",
+            "Authorization": "Basic {{ YOUR_DMAAP_MR_SUBSCRIBER_AUTHORIZATION }}"
+        },
+        "target_entity" : "dmaap_mr",
+        "tls_ca_mode" : "cert_directory",
+        "timeout_in_secs": 60
+    },
     "deploy_handler": {
       "target_entity": "deployment_handler",
       "url": "http://deployment_handler:8188",
@@ -272,9 +285,16 @@ Here is a sample config from consul-kv.  Please replace the {{ ... }} with real
         Accept : "application/json"
         "Content-Type" : "application/json"
         ClientAuth : "Basic {{ YOUR_POLICY_ENGINE_CLIENT_AUTH }}"
+
+        # to override the Authorization value,
+        #    set the environment vars $PDP_USER and $PDP_PWD in policy-handler
         Authorization : "Basic {{ YOUR_POLICY_ENGINE_AUTHORIZATION }}"
+
         Environment : "{{ YOUR_POLICY_ENGINE_ENVIRONMENT }}"
+
+      # target_entity name that is used for logging
       target_entity : "policy_engine"
+
       # optional tls_ca_mode specifies where to find the cacert.pem for tls
       #   can be one of these:
       #       "cert_directory" - use the cacert.pem stored locally in cert_directory.
@@ -288,6 +308,43 @@ Here is a sample config from consul-kv.  Please replace the {{ ... }} with real
       # optional timeout_in_secs specifies the timeout for the http requests
       timeout_in_secs: 60
 
+
+    # DMaaP MR subscriber config
+    # These are the url of and the auth for the external system, namely the policy-engine (PDP).
+    # We obtain that info manually from PDP and DMaaP folks at the moment.
+    dmaap_mr :
+      url: "http://{{ YOUR_DMAAP_MR_URL }}/events/{{ POLICY_UPDATE_TOPICNAME }}/{{ POLICY_UPDATE_CONSUMEGROUP }}/{{ POLICY_UPDATE_CONSUMERID }}"
+
+      query:
+        # The number of milliseconds for DMaaP MR to wait for messages if none are immediately available.
+        # This should normally be used, and set at 15000 or higher.
+        # This is referred to as long-polling timeout
+        # ?timeout=15000 passed to DMaaP MR in the query
+        timeout: 15000
+
+      headers:
+        "Content-Type": "application/json"
+        # provide Authorization for the subscriber if using https and user-password authentication
+        # to override the Authorization value,
+        #    set the environment vars $DMAAP_MR_USER and $DMAAP_MR_PWD in policy-handler
+        Authorization: "Basic {{ YOUR_DMAAP_MR_SUBSCRIBER_AUTHORIZATION }}"
+
+      # target_entity name that is used for logging
+      target_entity: "dmaap_mr"
+      # optional tls_ca_mode specifies where to find the cacert.pem for tls
+      #   can be one of these:
+      #       "cert_directory" - use the cacert.pem stored locally in cert_directory.
+      #                          this is the default if cacert.pem file is found
+      #
+      #       "os_ca_bundle"     - use the public ca_bundle provided by linux system.
+      #                          this is the default if cacert.pem file not found
+      #
+      #       "do_not_verify"  - special hack to turn off the verification by cacert and hostname
+      tls_ca_mode: "cert_directory"
+      # optional timeout_in_secs specifies the timeout for the http requests
+      timeout_in_secs: 60
+
+
     # deploy_handler config
     #    changed from string "deployment_handler" in 2.3.1 to structure in 2.4.0
     deploy_handler :