5.1.0 policy-handler - policy-updates from new PDP5.1.0

DCAEGEN2-1851:
- policy-handler now supports the policy-update notification
  from the new policy-engine thru DMaaP MR
  = no policy-filters - only policy-id values
- see README for discoverable config settings of dmaap_mr client
  = DMaaP MR client has the same flexibility as policy_engine
  = set the query.timeout to high value like 15000 (default)
- requests to DMaaP MR go through a single blocking connection
- first catch-up only after draining the policy-updates from DMaaP MR
  on the first loop
- safe parsing of messages from DMaaP MR
- policy-engine changed the data type for policy-version field
  from int to string that is expected to have the semver value
- related change to deployment-handler (DCAEGEN2-2085) has to be
  deployed to handle the non-numeric policyVersion
- on new PDP API: http /policy_latest and policy-updates
  return the new data from the new PDP API with the following fields
  added/renamed by the policy-handler to keep other policy related parts
  intact in R4-R6 (see pdp_api/policy_utils.py)
  * policyName = policy_id + "." + policyVersion.replace(".","-")
                                 + ".xml"
  * policyVersion = str(metadata["policy-version"])
  * "config" - is the renamed "properties" from the new PDP API response
- enabled the /catch_up and the periodic auto-catch-up for the new PDP
  API
- enabled GET /policies_latest - returns the latest policies for the
  deployed components
- POST /policies_latest - still disabled since no support for the
  policy-filters is provided for the new PDP API
- fixed hiding the Authorization value on comparing the configs
- logging of secrets is now sha256 to see whether they changed
- added X-ONAP-RequestID to headers the same way as X-ECOMP-RequestID
- on policy-update process the removal first, then addition
- changed the pool_connections=1 (number of pools) on PDP and DH sides
  == only a single destination is expected for each
- log the exception as fatal into error.log
- other minor fixes and refactoring
- unit-test coverage 74%
- integration testing is requested

DCAEGEN2-1976:
- policy-handler is enhanced to get user/password from env vars
  for PDP and DMaaP MR clients and overwriting the Authorization field
  in https headers received from the discoverable config
  = to override the Authorization value on policy_engine,
    set the environment vars $PDP_USER and $PDP_PWD in policy-handler
    container
  = to override the Authorization value on dmaap_mr,
    if using https and user-password authentication,
    set the environment vars $DMAAP_MR_USER and $DMAAP_MR_PWD in
    policy-handler container

Change-Id: Iad8eab9e20e615a0e0d2822f4735dc64c50aa55c
Signed-off-by: Alex Shatov <alexs@att.com>
Issue-ID: DCAEGEN2-1851
Issue-ID: DCAEGEN2-1976

1 files changed, 19 insertions, 13 deletions

diff --git a/policyhandler/pdp_api/policy_utils.py b/policyhandler/pdp_api/policy_utils.py
index 1d06d14..f2ed522 100644
--- a/policyhandler/pdp_api/policy_utils.py
+++ b/policyhandler/pdp_api/policy_utils.py
@@ -1,5 +1,5 @@
 # ================================================================================
-# Copyright (c) 2019 AT&T Intellectual Property. All rights reserved.
+# Copyright (c) 2019-2020 AT&T Intellectual Property. All rights reserved.
 # ================================================================================
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -30,14 +30,19 @@ class PolicyUtils(object):
     """policy-client utils"""
 
     @staticmethod
-    def gen_req_to_pdp(policy_id):
-        """request to get a single policy from pdp by policy_id"""
+    def gen_req_to_pdp(policy_ids):
+        """request to get policies from pdp by policy_id list or a single value"""
+        if not policy_ids:
+            policy_ids = []
+        elif not isinstance(policy_ids, list):
+            policy_ids = [policy_ids]
+
         return {
             PDP_REQ_ONAP_NAME: "DCAE",
             PDP_REQ_ONAP_COMPONENT: Audit.service_name,
             PDP_REQ_ONAP_INSTANCE: Audit.SERVICE_INSTANCE_UUID,
             "action": "configure",
-            PDP_REQ_RESOURCE: {PDP_POLICY_ID: [policy_id]}
+            PDP_REQ_RESOURCE: {PDP_POLICY_ID: policy_ids}
         }
 
     @staticmethod
@@ -52,7 +57,7 @@ class PolicyUtils(object):
             "version": "1.0.0",
             "metadata": {
                 "policy-id": "onap.scaleout.tca",
-                "policy-version": 1,
+                "policy-version": "1.2.3",
                 "description": "The scaleout policy for vDNS"
             },
             "properties": {
@@ -73,13 +78,13 @@ class PolicyUtils(object):
         {
             "policy_id": "onap.scaleout.tca",
             "policy_body": {
-                "policyName": "onap.scaleout.tca.1.xml",
-                "policyVersion": 1,
+                "policyName": "onap.scaleout.tca.1-2-3.xml",
+                "policyVersion": "1.2.3",
                 "type": "onap.policies.monitoring.cdap.tca.hi.lo.app",
                 "version": "1.0.0",
                 "metadata": {
                     "policy-id": "onap.scaleout.tca",
-                    "policy-version": 1,
+                    "policy-version": "1.2.3",
                     "description": "The scaleout policy for vDNS"
                 },
                 "config": {
@@ -97,19 +102,20 @@ class PolicyUtils(object):
             }
         }
         """
-        if not policy_body or not policy_body.get(PDP_PROPERTIES):
+        if not policy_body:
             return None
 
         pdp_metadata = policy_body.get(PDP_METADATA, {})
         policy_id = pdp_metadata.get(PDP_POLICY_ID)
         policy_version = pdp_metadata.get(PDP_POLICY_VERSION)
-        if not policy_id or not policy_version:
+        if not policy_id or policy_version is None:
             return None
 
-        policy_body[POLICY_NAME] = "{}.{}.xml".format(policy_id, policy_version)
+        policy_body[POLICY_NAME] = "{}.{}.xml".format(policy_id, policy_version.replace(".", "-"))
         policy_body[POLICY_VERSION] = str(policy_version)
-        policy_body[POLICY_CONFIG] = policy_body[PDP_PROPERTIES]
-        del policy_body[PDP_PROPERTIES]
+        if PDP_PROPERTIES in policy_body:
+            policy_body[POLICY_CONFIG] = policy_body[PDP_PROPERTIES]
+            del policy_body[PDP_PROPERTIES]
 
         return {POLICY_ID:policy_id, POLICY_BODY:policy_body}