5.1.0 policy-handler - policy-updates from new PDP5.1.0

DCAEGEN2-1851:
- policy-handler now supports the policy-update notification
  from the new policy-engine thru DMaaP MR
  = no policy-filters - only policy-id values
- see README for discoverable config settings of dmaap_mr client
  = DMaaP MR client has the same flexibility as policy_engine
  = set the query.timeout to high value like 15000 (default)
- requests to DMaaP MR go through a single blocking connection
- first catch-up only after draining the policy-updates from DMaaP MR
  on the first loop
- safe parsing of messages from DMaaP MR
- policy-engine changed the data type for policy-version field
  from int to string that is expected to have the semver value
- related change to deployment-handler (DCAEGEN2-2085) has to be
  deployed to handle the non-numeric policyVersion
- on new PDP API: http /policy_latest and policy-updates
  return the new data from the new PDP API with the following fields
  added/renamed by the policy-handler to keep other policy related parts
  intact in R4-R6 (see pdp_api/policy_utils.py)
  * policyName = policy_id + "." + policyVersion.replace(".","-")
                                 + ".xml"
  * policyVersion = str(metadata["policy-version"])
  * "config" - is the renamed "properties" from the new PDP API response
- enabled the /catch_up and the periodic auto-catch-up for the new PDP
  API
- enabled GET /policies_latest - returns the latest policies for the
  deployed components
- POST /policies_latest - still disabled since no support for the
  policy-filters is provided for the new PDP API
- fixed hiding the Authorization value on comparing the configs
- logging of secrets is now sha256 to see whether they changed
- added X-ONAP-RequestID to headers the same way as X-ECOMP-RequestID
- on policy-update process the removal first, then addition
- changed the pool_connections=1 (number of pools) on PDP and DH sides
  == only a single destination is expected for each
- log the exception as fatal into error.log
- other minor fixes and refactoring
- unit-test coverage 74%
- integration testing is requested

DCAEGEN2-1976:
- policy-handler is enhanced to get user/password from env vars
  for PDP and DMaaP MR clients and overwriting the Authorization field
  in https headers received from the discoverable config
  = to override the Authorization value on policy_engine,
    set the environment vars $PDP_USER and $PDP_PWD in policy-handler
    container
  = to override the Authorization value on dmaap_mr,
    if using https and user-password authentication,
    set the environment vars $DMAAP_MR_USER and $DMAAP_MR_PWD in
    policy-handler container

Change-Id: Iad8eab9e20e615a0e0d2822f4735dc64c50aa55c
Signed-off-by: Alex Shatov <alexs@att.com>
Issue-ID: DCAEGEN2-1851
Issue-ID: DCAEGEN2-1976

1 files changed, 60 insertions, 17 deletions

diff --git a/tests/main/test_policyhandler.py b/tests/main/test_policyhandler.py
index 73ab9ca..c474e29 100644
--- a/tests/main/test_policyhandler.py
+++ b/tests/main/test_policyhandler.py
@@ -1,5 +1,5 @@
 # ============LICENSE_START=======================================================
-# Copyright (c) 2017-2019 AT&T Intellectual Property. All rights reserved.
+# Copyright (c) 2017-2020 AT&T Intellectual Property. All rights reserved.
 # ================================================================================
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -21,20 +21,17 @@ import json
 import time
 import uuid
 
-import cherrypy
 import pytest
-from cherrypy.test.helper import CPWebCase
+import cherrypy
 
-from policyhandler.config import Config
-from policyhandler.onap.audit import REQUEST_X_ECOMP_REQUESTID, Audit
-from policyhandler.pdp_api.pdp_consts import POLICY_NAME
-from policyhandler.policy_consts import LATEST_POLICIES
+from cherrypy.test.helper import CPWebCase
+from policyhandler.onap.audit import (REQUEST_X_ECOMP_REQUESTID,
+                                      REQUEST_X_ONAP_REQUESTID, Audit)
 from policyhandler.policy_receiver import PolicyReceiver
 from policyhandler.utils import Utils
 from policyhandler.web_server import _PolicyWeb
 
 from ..mock_tracker import Tracker
-from .mock_policy_engine import MockPolicyEngine
 
 _LOGGER = Utils.get_logger(__file__)
 
@@ -59,7 +56,27 @@ class WebServerTest(CPWebCase):
 
     def test_web_policy_latest(self):
         """test /policy_latest/<policy-id>"""
-        policy_id, expected_policy = MockPolicyEngine.gen_policy_latest(3)
+        policy_id = "test_scope_prefix.pdp_decision_sit"
+        expected_policy = {
+            "policy_id": "test_scope_prefix.pdp_decision_sit",
+            "policy_body": {
+                "type": "unit.test.type.policies",
+                "version": "1.0.0",
+                "metadata": {
+                    "policy-id": "test_scope_prefix.pdp_decision_sit",
+                    "policy-version": "4.4.4",
+                    "description": "description for test_scope_prefix.pdp_decision_sit"
+                },
+                "policyName": "test_scope_prefix.pdp_decision_sit.4-4-4.xml",
+                "policyVersion": "4.4.4",
+                "config": {
+                    "policy_updated_from_ver": 3,
+                    "policy_updated_to_ver": 4,
+                    "policy_hello": "world!",
+                    "updated_policy_id": "test_scope_prefix.pdp_decision_sit"
+                }
+            }
+        }
 
         self.getPage("/policy_latest/{0}".format(policy_id or ""))
         self.assertStatus('200 OK')
@@ -83,15 +100,17 @@ class WebServerTest(CPWebCase):
         _LOGGER.info("result: %s", result)
         _LOGGER.info("body: %s", self.body)
 
-        self.assertStatus('404 Not Found')
+        self.assertStatus('200 OK')
 
     def test_web_policies_latest(self):
         """test POST /policies_latest with policyName"""
         body = json.dumps({"junk": "to-be-developed"})
+        request_id = str(uuid.uuid4())
         result = self.getPage("/policies_latest", method='POST',
                               body=body,
                               headers=[
-                                  (REQUEST_X_ECOMP_REQUESTID, str(uuid.uuid4())),
+                                  (REQUEST_X_ECOMP_REQUESTID, request_id),
+                                  (REQUEST_X_ONAP_REQUESTID, request_id),
                                   ("Content-Type", "application/json"),
                                   ('Content-Length', str(len(body)))
                               ])
@@ -102,17 +121,20 @@ class WebServerTest(CPWebCase):
 
     @pytest.mark.usefixtures(
         "fix_deploy_handler",
+        "fix_dmaap_mr",
         "fix_cherrypy_engine_exit")
     def test_zzzzz_shutdown(self):
         """test shutdown"""
-        _LOGGER.info("start shutdown")
+        _LOGGER.info("testing the shutdown")
         assert not PolicyReceiver.is_running()
-        audit = Audit(job_name="test_zzzzz_shutdown", req_message="start shutdown")
+        audit = Audit(job_name="test_zzzzz_shutdown", req_message="testing the shutdown")
         PolicyReceiver.run(audit)
 
         result = self.getPage("/healthcheck")
         _LOGGER.info("healthcheck result: %s", result)
 
+        time.sleep(1)
+
         WebServerTest.do_gc_test = False
         _LOGGER.info("shutdown...")
         audit.audit_done("shutdown")
@@ -120,11 +142,32 @@ class WebServerTest(CPWebCase):
         _LOGGER.info("shutdown result: %s", result)
         self.assertStatus('200 OK')
         _LOGGER.info("got shutdown: %s", self.body)
-        time.sleep(1)
+        time.sleep(5)
         assert not PolicyReceiver.is_running()
 
-        if Config.is_pdp_api_default():
-            _LOGGER.info("passive for new PDP API")
-            return
+        Tracker.validate()
+
+    @pytest.mark.usefixtures(
+        "fix_deploy_handler",
+        "fix_dmaap_mr",
+        "fix_cherrypy_engine_exit")
+    def test_zzz_policy_updates_and_catch_ups(self):
+        """test run policy handler with policy updates and catchups"""
+        _LOGGER.info("start policy_updates_and_catch_ups")
+        assert not PolicyReceiver.is_running()
+
+        audit = Audit(job_name="test_zzz_policy_updates_and_catch_ups",
+                      req_message="start policy_updates_and_catch_ups")
+        PolicyReceiver.run(audit)
+
+        _LOGGER.info("sleep 20 before shutdown...")
+        time.sleep(20)
+
+        result = self.getPage("/healthcheck")
+        _LOGGER.info("healthcheck result: %s", result)
+
+        PolicyReceiver.shutdown(audit)
+        time.sleep(5)
+        assert not PolicyReceiver.is_running()
 
         Tracker.validate()