5.1.0 policy-handler - policy-updates from new PDP5.1.0

DCAEGEN2-1851:
- policy-handler now supports the policy-update notification
  from the new policy-engine thru DMaaP MR
  = no policy-filters - only policy-id values
- see README for discoverable config settings of dmaap_mr client
  = DMaaP MR client has the same flexibility as policy_engine
  = set the query.timeout to high value like 15000 (default)
- requests to DMaaP MR go through a single blocking connection
- first catch-up only after draining the policy-updates from DMaaP MR
  on the first loop
- safe parsing of messages from DMaaP MR
- policy-engine changed the data type for policy-version field
  from int to string that is expected to have the semver value
- related change to deployment-handler (DCAEGEN2-2085) has to be
  deployed to handle the non-numeric policyVersion
- on new PDP API: http /policy_latest and policy-updates
  return the new data from the new PDP API with the following fields
  added/renamed by the policy-handler to keep other policy related parts
  intact in R4-R6 (see pdp_api/policy_utils.py)
  * policyName = policy_id + "." + policyVersion.replace(".","-")
                                 + ".xml"
  * policyVersion = str(metadata["policy-version"])
  * "config" - is the renamed "properties" from the new PDP API response
- enabled the /catch_up and the periodic auto-catch-up for the new PDP
  API
- enabled GET /policies_latest - returns the latest policies for the
  deployed components
- POST /policies_latest - still disabled since no support for the
  policy-filters is provided for the new PDP API
- fixed hiding the Authorization value on comparing the configs
- logging of secrets is now sha256 to see whether they changed
- added X-ONAP-RequestID to headers the same way as X-ECOMP-RequestID
- on policy-update process the removal first, then addition
- changed the pool_connections=1 (number of pools) on PDP and DH sides
  == only a single destination is expected for each
- log the exception as fatal into error.log
- other minor fixes and refactoring
- unit-test coverage 74%
- integration testing is requested

DCAEGEN2-1976:
- policy-handler is enhanced to get user/password from env vars
  for PDP and DMaaP MR clients and overwriting the Authorization field
  in https headers received from the discoverable config
  = to override the Authorization value on policy_engine,
    set the environment vars $PDP_USER and $PDP_PWD in policy-handler
    container
  = to override the Authorization value on dmaap_mr,
    if using https and user-password authentication,
    set the environment vars $DMAAP_MR_USER and $DMAAP_MR_PWD in
    policy-handler container

Change-Id: Iad8eab9e20e615a0e0d2822f4735dc64c50aa55c
Signed-off-by: Alex Shatov <alexs@att.com>
Issue-ID: DCAEGEN2-1851
Issue-ID: DCAEGEN2-1976

5 files changed, 36 insertions, 57 deletions

diff --git a/policyhandler/pdp_api_v0/policy_listener.py b/policyhandler/pdp_api_v0/policy_listener.py
index 67e4c49..7525e4d 100644
--- a/policyhandler/pdp_api_v0/policy_listener.py
+++ b/policyhandler/pdp_api_v0/policy_listener.py
@@ -1,5 +1,5 @@
 # ================================================================================
-# Copyright (c) 2018-2019 AT&T Intellectual Property. All rights reserved.
+# Copyright (c) 2018-2020 AT&T Intellectual Property. All rights reserved.
 # ================================================================================
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -16,7 +16,7 @@
 #
 
 """
-policy-listener communicates with policy-engine
+policy_listener communicates with policy-engine
 thru web-socket to receive push notifications
 on updates and removal of policies.
 
@@ -298,8 +298,8 @@ class PolicyListener(Thread):
 
 
     def shutdown(self, audit):
-        """Shutdown the policy-listener"""
-        _LOGGER.info(audit.info("shutdown policy-listener"))
+        """Shutdown the policy_listener"""
+        _LOGGER.info(audit.info("shutdown policy_listener"))
         with self._lock:
             self._keep_running = False
 
diff --git a/policyhandler/pdp_api_v0/policy_matcher.py b/policyhandler/pdp_api_v0/policy_matcher.py
index 357af49..deb6619 100644
--- a/policyhandler/pdp_api_v0/policy_matcher.py
+++ b/policyhandler/pdp_api_v0/policy_matcher.py
@@ -1,5 +1,5 @@
 # ================================================================================
-# Copyright (c) 2018-2019 AT&T Intellectual Property. All rights reserved.
+# Copyright (c) 2018-2020 AT&T Intellectual Property. All rights reserved.
 # ================================================================================
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -38,24 +38,6 @@ class PolicyMatcher(object):
     PDP_API_FOLDER = os.path.basename(os.path.dirname(os.path.realpath(__file__)))
 
     @staticmethod
-    def get_deployed_policies(audit):
-        """get the deployed policies and policy-filters"""
-        deployed_policies, deployed_policy_filters = DeployHandler.get_deployed_policies(audit)
-
-        if audit.is_not_found():
-            warning_txt = "got no deployed policies or policy-filters"
-            _LOGGER.warning(warning_txt)
-            return {"warning": warning_txt}, None, None
-
-        if not audit.is_success() or (not deployed_policies and not deployed_policy_filters):
-            error_txt = "failed to retrieve policies from deployment-handler"
-            _LOGGER.error(error_txt)
-            return {"error": error_txt}, None, None
-
-        return None, deployed_policies, deployed_policy_filters
-
-
-    @staticmethod
     def build_catch_up_message(audit, deployed_policies, deployed_policy_filters):
         """
         find the latest policies from policy-engine for the deployed policies and policy-filters
@@ -135,7 +117,7 @@ class PolicyMatcher(object):
     @staticmethod
     def match_to_deployed_policies(audit, policies_updated, policies_removed):
         """match the policies_updated, policies_removed versus deployed policies"""
-        deployed_policies, deployed_policy_filters = DeployHandler.get_deployed_policies(audit)
+        _, deployed_policies, deployed_policy_filters = DeployHandler.get_deployed_policies(audit)
         if not audit.is_success():
             return {}, {}, {}
 
diff --git a/policyhandler/pdp_api_v0/policy_rest.py b/policyhandler/pdp_api_v0/policy_rest.py
index c59625e..30fc043 100644
--- a/policyhandler/pdp_api_v0/policy_rest.py
+++ b/policyhandler/pdp_api_v0/policy_rest.py
@@ -1,5 +1,5 @@
 # ================================================================================
-# Copyright (c) 2017-2019 AT&T Intellectual Property. All rights reserved.
+# Copyright (c) 2017-2020 AT&T Intellectual Property. All rights reserved.
 # ================================================================================
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -28,8 +28,7 @@ from threading import Lock
 import requests
 
 from ..config import Config, Settings
-from ..onap.audit import (REQUEST_X_ECOMP_REQUESTID, AuditHttpCode,
-                          AuditResponseCode, Metrics)
+from ..onap.audit import AuditHttpCode, AuditResponseCode, Metrics
 from ..policy_consts import (ERRORED_POLICIES, LATEST_POLICIES, POLICY_BODY,
                              POLICY_FILTER, POLICY_FILTERS, POLICY_ID,
                              POLICY_NAMES)
@@ -84,10 +83,10 @@ class PolicyRest(object):
         changed, pool_size = PolicyRest._settings.get_by_key(Config.POOL_CONNECTIONS, 20)
         if changed:
             PolicyRest._requests_session.mount(
-                'https://', requests.adapters.HTTPAdapter(pool_connections=pool_size,
+                'https://', requests.adapters.HTTPAdapter(pool_connections=1,
                                                           pool_maxsize=pool_size))
             PolicyRest._requests_session.mount(
-                'http://', requests.adapters.HTTPAdapter(pool_connections=pool_size,
+                'http://', requests.adapters.HTTPAdapter(pool_connections=1,
                                                          pool_maxsize=pool_size))
 
         _, config = PolicyRest._settings.get_by_key(Config.FIELD_POLICY_ENGINE)
@@ -159,7 +158,7 @@ class PolicyRest(object):
             _LOGGER.error(
                 audit.error("no url for PDP", error_code=AuditResponseCode.AVAILABILITY_ERROR))
             audit.set_http_status_code(AuditHttpCode.SERVER_INTERNAL_ERROR.value)
-            return None
+            return None, None
 
         with PolicyRest._lock:
             session = PolicyRest._requests_session
@@ -171,7 +170,7 @@ class PolicyRest(object):
 
         metrics = Metrics(aud_parent=audit, targetEntity=target_entity, targetServiceName=url)
 
-        headers[REQUEST_X_ECOMP_REQUESTID] = metrics.request_id
+        headers = metrics.put_request_id_into_headers(headers)
 
         log_action = "post to {} at {}".format(target_entity, url)
         log_data = "msg={} headers={}, custom_kwargs({}) timeout_in_secs({})".format(
@@ -191,7 +190,7 @@ class PolicyRest(object):
                           else AuditHttpCode.SERVER_INTERNAL_ERROR.value)
             error_msg = ("failed {}: {} to {}".format(type(ex).__name__, str(ex), log_line))
 
-            _LOGGER.exception(error_msg)
+            _LOGGER.exception(metrics.fatal(error_msg))
             metrics.set_http_status_code(error_code)
             audit.set_http_status_code(error_code)
             metrics.metrics(error_msg)
@@ -412,7 +411,7 @@ class PolicyRest(object):
 
         policies_to_find = {}
         for (policy_id, policy_version) in policies_updated:
-            if not policy_id or not policy_version or not policy_version.isdigit():
+            if not policy_id or policy_version is None or not policy_version.isdigit():
                 continue
             policy = policies_to_find.get(policy_id)
             if not policy:
diff --git a/policyhandler/pdp_api_v0/policy_updates.py b/policyhandler/pdp_api_v0/policy_updates.py
index eafdca2..ac68f4a 100644
--- a/policyhandler/pdp_api_v0/policy_updates.py
+++ b/policyhandler/pdp_api_v0/policy_updates.py
@@ -1,5 +1,5 @@
 # ================================================================================
-# Copyright (c) 2018-2019 AT&T Intellectual Property. All rights reserved.
+# Copyright (c) 2018-2020 AT&T Intellectual Property. All rights reserved.
 # ================================================================================
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -40,7 +40,7 @@ class PolicyUpdates(object):
         self._policies_removed = {}
 
     def reset(self):
-        """resets the state"""
+        """resets the state - removes the pending policy-updates"""
         self.__init__()
 
     def pop_policy_updates(self):
@@ -62,25 +62,12 @@ class PolicyUpdates(object):
 
     def push_policy_updates(self, policies_updated, policies_removed):
         """consolidate the new policies_updated, policies_removed to existing ones"""
-        for policy_body in policies_updated:
-            policy_name = policy_body.get(POLICY_NAME)
-            policy = PolicyUtils.convert_to_policy(policy_body)
-            if not policy:
-                continue
-            policy_id = policy.get(POLICY_ID)
-
-            self._policies_updated[policy_id] = policy
-
-            rm_policy_names = self._policies_removed.get(policy_id, {}).get(POLICY_NAMES)
-            if rm_policy_names and policy_name in rm_policy_names:
-                del rm_policy_names[policy_name]
-
         for policy_body in policies_removed:
-            policy_name = policy_body.get(POLICY_NAME)
             policy = PolicyUtils.convert_to_policy(policy_body)
             if not policy:
                 continue
             policy_id = policy.get(POLICY_ID)
+            policy_name = policy_body.get(POLICY_NAME)
 
             if policy_id in self._policies_removed:
                 policy = self._policies_removed[policy_id]
@@ -90,16 +77,27 @@ class PolicyUpdates(object):
             policy[POLICY_NAMES][policy_name] = True
             self._policies_removed[policy_id] = policy
 
+        for policy_body in policies_updated:
+            policy = PolicyUtils.convert_to_policy(policy_body)
+            if not policy:
+                continue
+            policy_id = policy.get(POLICY_ID)
+            policy_name = policy_body.get(POLICY_NAME)
+
+            self._policies_updated[policy_id] = policy
+
+            rm_policy_names = self._policies_removed.get(policy_id, {}).get(POLICY_NAMES)
+            if rm_policy_names and policy_name in rm_policy_names:
+                del rm_policy_names[policy_name]
+
         req_message = ("policy-update notification - updated[{0}], removed[{1}]"
                        .format(len(self._policies_updated),
                                len(self._policies_removed)))
 
         if not self._audit:
-            self._audit = Audit(job_name="policy_update",
-                                req_message=req_message,
+            self._audit = Audit(job_name="policy_update", req_message=req_message,
                                 retry_get_config=True)
-        else:
-            self._audit.req_message = req_message
+        self._audit.req_message = req_message
 
         _LOGGER.info(
             "pending(%s) for %s policies_updated %s policies_removed %s",
diff --git a/policyhandler/pdp_api_v0/policy_utils.py b/policyhandler/pdp_api_v0/policy_utils.py
index d337665..2cbb22c 100644
--- a/policyhandler/pdp_api_v0/policy_utils.py
+++ b/policyhandler/pdp_api_v0/policy_utils.py
@@ -1,5 +1,5 @@
 # ================================================================================
-# Copyright (c) 2018-2019 AT&T Intellectual Property. All rights reserved.
+# Copyright (c) 2018-2020 AT&T Intellectual Property. All rights reserved.
 # ================================================================================
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -63,7 +63,7 @@ class PolicyUtils(object):
             return None
         policy_name = policy_body.get(POLICY_NAME)
         policy_version = policy_body.get(POLICY_VERSION)
-        if not policy_name or not policy_version:
+        if not policy_name or policy_version is None:
             return None
         policy_id = PolicyUtils.extract_policy_id(policy_name)
         if not policy_id:
@@ -81,7 +81,7 @@ class PolicyUtils(object):
         for policy_body in policy_bodies:
             policy_name = policy_body.get(POLICY_NAME)
             policy_version = policy_body.get(POLICY_VERSION)
-            if not policy_name or not policy_version or not policy_version.isdigit():
+            if not policy_name or policy_version is None or not policy_version.isdigit():
                 continue
             if expected_versions and policy_version not in expected_versions:
                 continue
@@ -108,7 +108,7 @@ class PolicyUtils(object):
                 continue
             policy_id = policy.get(POLICY_ID)
             policy_version = policy.get(POLICY_BODY, {}).get(POLICY_VERSION)
-            if not policy_id or not policy_version or not policy_version.isdigit():
+            if not policy_id or policy_version is None or not policy_version.isdigit():
                 continue
             if (policy_id not in policies
                     or int(policy_version) > int(policies[policy_id][POLICY_BODY][POLICY_VERSION])):