5.1.0 policy-handler - policy-updates from new PDP5.1.0

DCAEGEN2-1851:
- policy-handler now supports the policy-update notification
  from the new policy-engine thru DMaaP MR
  = no policy-filters - only policy-id values
- see README for discoverable config settings of dmaap_mr client
  = DMaaP MR client has the same flexibility as policy_engine
  = set the query.timeout to high value like 15000 (default)
- requests to DMaaP MR go through a single blocking connection
- first catch-up only after draining the policy-updates from DMaaP MR
  on the first loop
- safe parsing of messages from DMaaP MR
- policy-engine changed the data type for policy-version field
  from int to string that is expected to have the semver value
- related change to deployment-handler (DCAEGEN2-2085) has to be
  deployed to handle the non-numeric policyVersion
- on new PDP API: http /policy_latest and policy-updates
  return the new data from the new PDP API with the following fields
  added/renamed by the policy-handler to keep other policy related parts
  intact in R4-R6 (see pdp_api/policy_utils.py)
  * policyName = policy_id + "." + policyVersion.replace(".","-")
                                 + ".xml"
  * policyVersion = str(metadata["policy-version"])
  * "config" - is the renamed "properties" from the new PDP API response
- enabled the /catch_up and the periodic auto-catch-up for the new PDP
  API
- enabled GET /policies_latest - returns the latest policies for the
  deployed components
- POST /policies_latest - still disabled since no support for the
  policy-filters is provided for the new PDP API
- fixed hiding the Authorization value on comparing the configs
- logging of secrets is now sha256 to see whether they changed
- added X-ONAP-RequestID to headers the same way as X-ECOMP-RequestID
- on policy-update process the removal first, then addition
- changed the pool_connections=1 (number of pools) on PDP and DH sides
  == only a single destination is expected for each
- log the exception as fatal into error.log
- other minor fixes and refactoring
- unit-test coverage 74%
- integration testing is requested

DCAEGEN2-1976:
- policy-handler is enhanced to get user/password from env vars
  for PDP and DMaaP MR clients and overwriting the Authorization field
  in https headers received from the discoverable config
  = to override the Authorization value on policy_engine,
    set the environment vars $PDP_USER and $PDP_PWD in policy-handler
    container
  = to override the Authorization value on dmaap_mr,
    if using https and user-password authentication,
    set the environment vars $DMAAP_MR_USER and $DMAAP_MR_PWD in
    policy-handler container

Change-Id: Iad8eab9e20e615a0e0d2822f4735dc64c50aa55c
Signed-off-by: Alex Shatov <alexs@att.com>
Issue-ID: DCAEGEN2-1851
Issue-ID: DCAEGEN2-1976

1 files changed, 16 insertions, 17 deletions

diff --git a/policyhandler/policy_updater.py b/policyhandler/policy_updater.py
index 3fcde40..42c5ee6 100644
--- a/policyhandler/policy_updater.py
+++ b/policyhandler/policy_updater.py
@@ -1,5 +1,5 @@
 # ================================================================================
-# Copyright (c) 2017-2019 AT&T Intellectual Property. All rights reserved.
+# Copyright (c) 2017-2020 AT&T Intellectual Property. All rights reserved.
 # ================================================================================
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -40,7 +40,7 @@ class PolicyUpdater(Thread):
         self._reconfigure_receiver = on_reconfigure_receiver
 
         self._lock = Lock()
-        self._run = Event()
+        self._run_event = Event()
         self._settings = Settings(CATCH_UP, Config.RECONFIGURE)
 
         self._catch_up_timer = None
@@ -74,11 +74,11 @@ class PolicyUpdater(Thread):
         self._settings.commit_change()
         return True
 
-    def policy_update(self, policies_updated, policies_removed):
+    def policy_update(self, *args):
         """enqueue the policy-updates"""
         with self._lock:
-            self._policy_updates.push_policy_updates(policies_updated, policies_removed)
-            self._run.set()
+            self._policy_updates.push_policy_updates(*args)
+            self._run_event.set()
 
     def catch_up(self, audit=None):
         """need to bring the latest policies to DCAE-Controller"""
@@ -89,7 +89,7 @@ class PolicyUpdater(Thread):
                     "catch_up %s request_id %s",
                     self._aud_catch_up.req_message, self._aud_catch_up.request_id
                 )
-            self._run.set()
+            self._run_event.set()
 
     def reconfigure(self, audit=None):
         """job to check for and bring in the updated config for policy-handler"""
@@ -100,7 +100,7 @@ class PolicyUpdater(Thread):
                     "%s request_id %s",
                     self._aud_reconfigure.req_message, self._aud_reconfigure.request_id
                 )
-            self._run.set()
+            self._run_event.set()
 
     def run(self):
         """wait and run the policy-update in thread"""
@@ -108,10 +108,10 @@ class PolicyUpdater(Thread):
         self._run_reconfigure_timer()
         while True:
             _LOGGER.info("waiting for policy-updates...")
-            self._run.wait()
+            self._run_event.wait()
 
             with self._lock:
-                self._run.clear()
+                self._run_event.clear()
 
             if not self._keep_running():
                 break
@@ -126,7 +126,7 @@ class PolicyUpdater(Thread):
 
             self._on_policy_update()
 
-        _LOGGER.info("exit policy-updater")
+        _LOGGER.info("exit policy_updater")
 
     def _keep_running(self):
         """thread-safe check whether to continue running"""
@@ -235,7 +235,7 @@ class PolicyUpdater(Thread):
 
                 if self._reconfigure_receiver(aud_reconfigure):
                     need_to_catch_up = True
-                    changed_configs.append("web-socket")
+                    changed_configs.append("policy-receiver")
 
                 reconfigure_result = " -- config changed on {} changes: {}".format(
                     json.dumps(changed_configs), Config.discovered_config)
@@ -246,7 +246,7 @@ class PolicyUpdater(Thread):
 
             Config.discovered_config.commit_change()
             aud_reconfigure.audit_done(result=reconfigure_result)
-            _LOGGER.info(log_line + reconfigure_result)
+            _LOGGER.info("%s%s", log_line, reconfigure_result)
 
             if need_to_catch_up:
                 self._pause_catch_up_timer()
@@ -300,8 +300,7 @@ class PolicyUpdater(Thread):
             _LOGGER.info(log_line)
             self._pause_catch_up_timer()
 
-            (_, policies,
-             policy_filters) = pdp_client.PolicyMatcher.get_deployed_policies(aud_catch_up)
+            (_, policies, policy_filters) = DeployHandler.get_deployed_policies(aud_catch_up)
 
             catch_up_message = None
             if aud_catch_up.is_not_found():
@@ -422,11 +421,11 @@ class PolicyUpdater(Thread):
 
 
     def shutdown(self, audit):
-        """Shutdown the policy-updater"""
-        _LOGGER.info("shutdown policy-updater")
+        """Shutdown the policy_updater"""
+        _LOGGER.info("shutdown policy_updater")
         with self._lock:
             self._aud_shutdown = audit
-            self._run.set()
+            self._run_event.set()
 
         self._stop_timers()