5.1.0 policy-handler - policy-updates from new PDP5.1.0

DCAEGEN2-1851:
- policy-handler now supports the policy-update notification
  from the new policy-engine thru DMaaP MR
  = no policy-filters - only policy-id values
- see README for discoverable config settings of dmaap_mr client
  = DMaaP MR client has the same flexibility as policy_engine
  = set the query.timeout to high value like 15000 (default)
- requests to DMaaP MR go through a single blocking connection
- first catch-up only after draining the policy-updates from DMaaP MR
  on the first loop
- safe parsing of messages from DMaaP MR
- policy-engine changed the data type for policy-version field
  from int to string that is expected to have the semver value
- related change to deployment-handler (DCAEGEN2-2085) has to be
  deployed to handle the non-numeric policyVersion
- on new PDP API: http /policy_latest and policy-updates
  return the new data from the new PDP API with the following fields
  added/renamed by the policy-handler to keep other policy related parts
  intact in R4-R6 (see pdp_api/policy_utils.py)
  * policyName = policy_id + "." + policyVersion.replace(".","-")
                                 + ".xml"
  * policyVersion = str(metadata["policy-version"])
  * "config" - is the renamed "properties" from the new PDP API response
- enabled the /catch_up and the periodic auto-catch-up for the new PDP
  API
- enabled GET /policies_latest - returns the latest policies for the
  deployed components
- POST /policies_latest - still disabled since no support for the
  policy-filters is provided for the new PDP API
- fixed hiding the Authorization value on comparing the configs
- logging of secrets is now sha256 to see whether they changed
- added X-ONAP-RequestID to headers the same way as X-ECOMP-RequestID
- on policy-update process the removal first, then addition
- changed the pool_connections=1 (number of pools) on PDP and DH sides
  == only a single destination is expected for each
- log the exception as fatal into error.log
- other minor fixes and refactoring
- unit-test coverage 74%
- integration testing is requested

DCAEGEN2-1976:
- policy-handler is enhanced to get user/password from env vars
  for PDP and DMaaP MR clients and overwriting the Authorization field
  in https headers received from the discoverable config
  = to override the Authorization value on policy_engine,
    set the environment vars $PDP_USER and $PDP_PWD in policy-handler
    container
  = to override the Authorization value on dmaap_mr,
    if using https and user-password authentication,
    set the environment vars $DMAAP_MR_USER and $DMAAP_MR_PWD in
    policy-handler container

Change-Id: Iad8eab9e20e615a0e0d2822f4735dc64c50aa55c
Signed-off-by: Alex Shatov <alexs@att.com>
Issue-ID: DCAEGEN2-1851
Issue-ID: DCAEGEN2-1976

1 files changed, 33 insertions, 18 deletions

diff --git a/policyhandler/onap/audit.py b/policyhandler/onap/audit.py
index 3c09c16..269dfd8 100644
--- a/policyhandler/onap/audit.py
+++ b/policyhandler/onap/audit.py
@@ -1,5 +1,5 @@
 # ================================================================================
-# Copyright (c) 2017-2019 AT&T Intellectual Property. All rights reserved.
+# Copyright (c) 2017-2020 AT&T Intellectual Property. All rights reserved.
 # ================================================================================
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -25,6 +25,7 @@
 """
 
 import copy
+import hashlib
 import json
 import os
 import re
@@ -41,6 +42,7 @@ from .health import Health
 from .process_info import ProcessInfo
 
 REQUEST_X_ECOMP_REQUESTID = "X-ECOMP-RequestID"
+REQUEST_X_ONAP_REQUESTID = "X-ONAP-RequestID"
 REQUEST_REMOTE_ADDR = "Remote-Addr"
 REQUEST_HOST = "Host"
 HOSTNAME = "HOSTNAME"
@@ -118,7 +120,7 @@ class AuditResponseCode(Enum):
 class _Audit(object):
     """put the audit object on stack per each initiating request in the system
 
-    :request_id: is the X-ECOMP-RequestID for tracing
+    :request_id: is the X-ONAP-RequestID or X-ECOMP-RequestID for tracing
 
     :req_message: is the request message string for logging
 
@@ -172,7 +174,7 @@ class _Audit(object):
         """create audit object per each request in the system
 
         :job_name: is the name of the audit job for health stats
-        :request_id: is the X-ECOMP-RequestID for tracing
+        :request_id: is the X-ONAP-RequestID or X-ECOMP-RequestID for tracing
         :req_message: is the request message string for logging
         :kwargs: - put any request related params into kwargs
         """
@@ -184,6 +186,12 @@ class _Audit(object):
         self.max_http_status_code = 0
         self._lock = threading.Lock()
 
+    def put_request_id_into_headers(self, headers=None):
+        """when sending message out - put the request_id into headers"""
+        headers = headers or {}
+        headers[REQUEST_X_ONAP_REQUESTID]  = self.request_id
+        headers[REQUEST_X_ECOMP_REQUESTID] = self.request_id
+        return headers
 
     @staticmethod
     def register_item_health(health_name, health_getter=None):
@@ -241,6 +249,8 @@ class _Audit(object):
 
     def set_http_status_code(self, http_status_code):
         """accumulate the highest(worst) http status code"""
+        if http_status_code is None:
+            http_status_code = AuditHttpCode.SERVER_INTERNAL_ERROR.value
         with self._lock:
             if self.max_http_status_code < AuditHttpCode.SERVER_INTERNAL_ERROR.value:
                 self.max_http_status_code = max(http_status_code, self.max_http_status_code)
@@ -308,9 +318,7 @@ class _Audit(object):
         """debug+error - the warn level of logging"""
         all_kwargs = self.merge_all_kwargs(**kwargs)
 
-        if error_code and isinstance(error_code, AuditResponseCode):
-            all_kwargs[ERROR_CODE] = error_code.value
-            all_kwargs[ERROR_DESCRIPTION] = AuditResponseCode.get_human_text(error_code)
+        self._set_error_code_in_kwargs(error_code, all_kwargs)
 
         _Audit._logger_debug.warn(log_line, **all_kwargs)
         _Audit._logger_error.warn(log_line, **all_kwargs)
@@ -320,9 +328,7 @@ class _Audit(object):
         """debug+error - the error level of logging"""
         all_kwargs = self.merge_all_kwargs(**kwargs)
 
-        if error_code and isinstance(error_code, AuditResponseCode):
-            all_kwargs[ERROR_CODE] = error_code.value
-            all_kwargs[ERROR_DESCRIPTION] = AuditResponseCode.get_human_text(error_code)
+        self._set_error_code_in_kwargs(error_code, all_kwargs)
 
         _Audit._logger_debug.error(log_line, **all_kwargs)
         _Audit._logger_error.error(log_line, **all_kwargs)
@@ -332,25 +338,32 @@ class _Audit(object):
         """debug+error - the fatal level of logging"""
         all_kwargs = self.merge_all_kwargs(**kwargs)
 
-        if error_code and isinstance(error_code, AuditResponseCode):
-            all_kwargs[ERROR_CODE] = error_code.value
-            all_kwargs[ERROR_DESCRIPTION] = AuditResponseCode.get_human_text(error_code)
+        self._set_error_code_in_kwargs(error_code, all_kwargs)
 
         _Audit._logger_debug.fatal(log_line, **all_kwargs)
         _Audit._logger_error.fatal(log_line, **all_kwargs)
         return log_line
 
+    def _set_error_code_in_kwargs(self, error_code, all_kwargs):
+        """set the error code and description in kwargs for logging"""
+        if not error_code or not isinstance(error_code, AuditResponseCode):
+            error_code = AuditResponseCode.UNKNOWN_ERROR
+        all_kwargs[ERROR_CODE] = error_code.value
+        all_kwargs[ERROR_DESCRIPTION] = AuditResponseCode.get_human_text(error_code)
+
     @staticmethod
     def hide_secrets(obj):
         """hides the known secret field values of the dictionary"""
         if not isinstance(obj, dict):
             return obj
 
-        for key in obj:
+        for key, val in obj.items():
             if key.lower() in [HEADER_CLIENTAUTH, HEADER_AUTHORIZATION]:
-                obj[key] = "*"
-            elif isinstance(obj[key], dict):
-                obj[key] = _Audit.hide_secrets(obj[key])
+                hval = hashlib.sha256()
+                hval.update(val.encode())
+                obj[key] = "***({})***".format(hval.hexdigest())
+            elif isinstance(val, dict):
+                obj[key] = _Audit.hide_secrets(val)
 
         return obj
 
@@ -375,7 +388,7 @@ class Audit(_Audit):
         """create audit object per each request in the system
 
         :job_name: is the name of the audit job for health stats
-        :request_id: is the X-ECOMP-RequestID for tracing
+        :request_id: is the X-ONAP-RequestID or X-ECOMP-RequestID for tracing
         :req_message: is the request message string for logging
         :aud_parent: is the parent Audit - used for sub-query metrics to other systems
         :kwargs: - put any request related params into kwargs
@@ -388,7 +401,9 @@ class Audit(_Audit):
         headers = self.kwargs.get("headers", {})
         if headers:
             if not self.request_id:
-                self.request_id = headers.get(REQUEST_X_ECOMP_REQUESTID)
+                self.request_id = headers.get(REQUEST_X_ONAP_REQUESTID,
+                                              headers.get(REQUEST_X_ECOMP_REQUESTID))
+
             self.kwargs.setdefault(AUDIT_IPADDRESS, headers.get(REQUEST_REMOTE_ADDR))
             self.kwargs.setdefault(AUDIT_SERVER, headers.get(REQUEST_HOST))