5.0.0 policy-handler - new PDP API or old PDP API4.0.0-ONAPdublin

- in R4 Dublin the policy-engine introduced a totally new API
- policy-handler now has a startup option to either use the new PDP API
  or the old PDP API that was created-updated before the end of 2018
- see README.md and README_pdp_api_v0.md for instructions on how to
  setup the policy-handler running either with the new PDP API
  or the old (pdp_api_v0) PDP API
- this is a massive refactoring that changed almost all the source files,
  but kept the old logic when using the old (pdp_api_v0) PDP API
- all the code related to PDP API version is split into two subfolders
  = pdp_api/ contains the new PDP API source code
  = pdp_api_v0/ contains the old (2018) PDP API source code
  = pdp_client.py imports from either pdp_api or pdp_api_v0
  = the rest of the code is only affected when it needs to branch
    the logic
- logging to policy_handler.log now shows the path of the source file to
  allow tracing which PDP API is actually used
- when the new PDP API is used, the policy-update flow is disabled
  = passive mode of operation
  = no web-socket
  = no periodic catch_up
  = no policy-filters
  = reduced web-API - only a single /policy_latest endpoint is available
    /policies_latest returns 404
    /catch_up request is accepted, but ignored
- on new PDP API: http /policy_latest returns the new data from the
  new PDP API with the following fields added by the policy-handler
  to keep other policy related parts intact in R4
  (see pdp_api/policy_utils.py)
  = "policyName" = policy_id + "." + "policyVersion" + ".xml"
  = "policyVersion" = str("metadata"."policy-version")
  = "config" - is the renamed "properties" from the new PDP API response
- unit tests are split into two subfolders as well
  = main/ for the new PDP API testing
  = pdp_api_v0/ for the old (2018) PDP API
- removed the following line from the license text of changed files
  ECOMP is a trademark and service mark of AT&T Intellectual Property.

- the new PDP API is expected to be extended and redesigned in R5 El Alto
- on retiring the old PDP API - the intention is to be able to remove
  the pdp_api_v0/ subfolder and minimal related cleanup of the code
  that imports that as well as the cleanup of the config.py, etc.

Change-Id: Ief9a2ae4541300308caaf97377f4ed051535dbe4
Signed-off-by: Alex Shatov <alexs@att.com>
Issue-ID: DCAEGEN2-1128

1 files changed, 255 insertions, 1 deletions

diff --git a/README.md b/README.md
index 65c2eba..3266b2f 100644
--- a/README.md
+++ b/README.md
@@ -5,7 +5,7 @@ See [wiki for DCAE gen2 architecture of policy-handling by DCAE-controller](http
 ## web-service for policies to be used by DCAE-Controller
 
 - GET `/policy_latest/<policy_id>` -- get the latest policy from policy-engine that is identified by `policy_id`
-- POST `/policies_latest` -- gets the latest policies that match to the **policy-filter** provided in the body of the request.  The policy-filter mimics the body of the /getConfig on policy-engine.
+- POST `/policies_latest` -- *only for the old(pdp_api_v0) PDP API* -- gets the latest policies that match to the **policy-filter** provided in the body of the request.  The policy-filter mimics the body of the /getConfig on policy-engine.
 
     sample request - policy-filter
 
@@ -66,6 +66,8 @@ local config file `policy_handler/etc/config.json` contains:
 {
   "wservice_port" : 25577,
   "consul_url" : "http://consul:8500",
+  "consul_timeout_in_secs" : 60,
+  "pdp_api_version" : null,
   "policy_handler" : {
     "system" : "policy_handler",
     "tls" : {
@@ -84,6 +86,10 @@ Field descriptions
 
 - `wservice_port` - port of the policy-hanlder web-service
 - `consul_url` - optional url for the consul agent
+- `consul_timeout_in_secs` - optional timeout in seconds to wait for the response from consul agent
+- `pdp_api_version` - optional value for PDP_API_VERSION.
+  - The default PDP API is used when this field is null or absent.  The new PDP as of 2019 is the default
+  - To use the old PDP API that was created before the end of 2018, put any value like pdp_api_v0 into this field.  Alternatlively, provide a non-empty environment variable $PDP_API_VERSION=pdp_api_v0 on the run of the docker container
 - `policy_handler` - local config for policy-handler application
   - `system` - general system name of the policy-handler
   - `tls` - tls settings for the https clients and server - required to enable tls
@@ -96,6 +102,254 @@ Field descriptions
 
 ----------
 
+## discoverable configure from consul-kv for the **new PDP API**
+
+on the start of the policy-handler, it will get the discoverable part of the configure from consul-kv for the key taken from the `local-config.policy_handler.system`
+
+See [README_pdp_api_v0.md](Readme_pdp_api_v0.md) for instructions on how to set up the policy-handler to work with the **old PDP API** that was created not later than **2018**
+
+### using the new PDP API
+
+As of **R4 Dublin** release, the PDP API is totally redesigned.  The policy-handler is changed to have a startup option to either using the new PDP API, or the old PDP API (pdp_api_v0).
+
+By **default**, the policy-handler will startup configured to use only the **new PDP API**.
+
+#### service_mode in healthcheck
+
+**R4 Dublin**: when the polcy-handler runs against the default **new PDP API**, the /healthchek response should contain the following values under service_mode element
+
+```json
+{
+  ...
+  "service_mode": {
+    "is_active_mode_of_operation": false,
+    "is_pdp_api_default": true
+  }
+}
+
+```
+
+#### make sure that the start up of the policy-handler is configured to use the **new PDP API**
+
+make sure that both of the following settings are set properly
+
+- make sure that the environment variable `$PDP_API_VERSION` is either **empty** or **not set** at all on the run of the docker container of the policy-handler
+
+- make sure that the `pdp_api_version` param is either `null` or not present in the startup config of the policy-handler at `etc/config.json`
+
+```json
+{
+    "pdp_api_version" : null,
+}
+```
+
+#### point the discovarable config of the policy-handler to point to the **new PDP API**
+
+In short: keep the consul-kv record for he policy-handler as before R4 Dublin.
+
+Here is a sample config from consul-kv.  Please replace the {{ ... }} with real setup values
+
+```json
+{
+  ...
+  "policy_engine": {
+    "url": "https://{{ policy_ip_addr }}:{{ policy_ip_port }}",
+    "path_decision": "/decision/v1",
+    "tls_ca_mode": "cert_directory",
+    "timeout_in_secs": 60,
+    "target_entity": "policy_engine",
+    "headers": {
+      "Accept": "application/json",
+      "Content-Type": "application/json",
+      "Authorization": "Basic {{ YOUR_POLICY_ENGINE_AUTHORIZATION }}",
+      "ClientAuth": "Basic {{ YOUR_POLICY_ENGINE_CLIENT_AUTH }}",
+      "Environment": "{{ YOUR_POLICY_ENGINE_ENVIRONMENT }}"
+    }
+  }
+}
+```
+
+----------
+
+## full discoverable configure from consul-kv
+
+```json
+{
+  "policy_handler": {
+    "thread_pool_size": 4,
+    "pool_connections": 20,
+    "policy_retry_count": 5,
+    "policy_retry_sleep": 5,
+    "mode_of_operation": "active",
+    "catch_up": {
+      "interval": 1200
+    },
+    "reconfigure": {
+      "interval": 600
+    },
+    "policy_engine": {
+      "url": "{{ YOUR_POLICY_ENGINE_URL }}",
+      "path_decision": "/decision/v1",
+      "headers": {
+        "Accept": "application/json",
+        "Content-Type": "application/json",
+        "ClientAuth": "Basic {{ YOUR_POLICY_ENGINE_CLIENT_AUTH }}",
+        "Authorization": "Basic {{ YOUR_POLICY_ENGINE_AUTHORIZATION }}",
+        "Environment": "{{ YOUR_POLICY_ENGINE_ENVIRONMENT }}"
+      },
+      "target_entity": "policy_engine",
+      "tls_ca_mode": "cert_directory",
+      "timeout_in_secs": 60
+    },
+    "deploy_handler": {
+      "target_entity": "deployment_handler",
+      "url": "http://deployment_handler:8188",
+      "max_msg_length_mb": 5,
+      "query": {
+        "cfy_tenant_name": "default_tenant"
+      },
+      "tls_ca_mode": "cert_directory",
+      "timeout_in_secs": 60
+    },
+    "service_activator": {
+      "target_entity": "service_activator",
+      "url": "http://service_activator:123",
+      "path_register": "/register",
+      "tls_ca_mode": "cert_directory",
+      "timeout_in_secs": 20,
+      "post_register": {
+        "component_name": "policy_handler",
+        "reconfigure_path": "/reconfigure",
+        "http_protocol": "http"
+      }
+    }
+  }
+}
+```
+
+### field description in yaml format that is equivalent to the actual json structure of the full discoverable config
+
+```yaml
+  policy_handler :
+    # parallelize the getConfig queries to policy-engine on each policy-update notification
+    thread_pool_size : 4
+
+    # parallelize requests to policy-engine and keep them alive
+    pool_connections : 20
+
+    # retry to getConfig from policy-engine on policy-update notification
+    policy_retry_count : 5
+    policy_retry_sleep : 5
+
+    # mode of operation for the policy-handler
+    # either active or passive
+    # in passive mode the policy-hanlder will not listen to
+    #                 and will not bring the policy-updates from policy-engine
+    mode_of_operation : "active"
+
+    # config of automatic catch_up for resiliency
+    catch_up :
+      # interval in seconds on how often to call automatic catch_up
+      # example: 1200 is 20*60 seconds that is 20 minutes
+      interval : 1200
+
+    # config of periodic reconfigure-rediscover for adaptability
+    reconfigure:
+      # interval in seconds on how often to call automatic reconfigure
+      # example: 600 is 10*60 seconds that is 10 minutes
+      interval : 600
+
+    # PDP (policy-engine) config
+    # These are the url of and the auth for the external system, namely the policy-engine (PDP).
+    # We obtain that info manually from PDP folks at the moment.
+    # In long run we should figure out a way of bringing that info into consul record
+    #    related to policy-engine itself.
+    policy_engine :
+      url : "{{ YOUR_POLICY_ENGINE_URL }}"
+      # path to decision on the new PDP API as of 2019
+      path_decision : "/decision/v1"
+      headers :
+        Accept : "application/json"
+        "Content-Type" : "application/json"
+        ClientAuth : "Basic {{ YOUR_POLICY_ENGINE_CLIENT_AUTH }}"
+        Authorization : "Basic {{ YOUR_POLICY_ENGINE_AUTHORIZATION }}"
+        Environment : "{{ YOUR_POLICY_ENGINE_ENVIRONMENT }}"
+      target_entity : "policy_engine"
+      # optional tls_ca_mode specifies where to find the cacert.pem for tls
+      #   can be one of these:
+      #       "cert_directory" - use the cacert.pem stored locally in cert_directory.
+      #                          this is the default if cacert.pem file is found
+      #
+      #       "os_ca_bundle"     - use the public ca_bundle provided by linux system.
+      #                          this is the default if cacert.pem file not found
+      #
+      #       "do_not_verify"  - special hack to turn off the verification by cacert and hostname
+      tls_ca_mode : "cert_directory"
+      # optional timeout_in_secs specifies the timeout for the http requests
+      timeout_in_secs: 60
+
+    # deploy_handler config
+    #    changed from string "deployment_handler" in 2.3.1 to structure in 2.4.0
+    deploy_handler :
+      # name of deployment-handler service used by policy-handler for logging
+      target_entity : "deployment_handler"
+      # url of the deployment-handler service for policy-handler to direct the policy-updates to
+      #   - expecting dns to resolve the name deployment_handler to ip address
+      url : "http://deployment_handler:8188"
+      # limit the size of a single data segment for policy-update messages
+      #       from policy-handler to deployment-handler in megabytes
+      max_msg_length_mb : 5
+      query :
+        # optionally specify the tenant name for the cloudify under deployment-handler
+        #    if not specified the "default_tenant" is used by the deployment-handler
+        cfy_tenant_name : "default_tenant"
+      # optional tls_ca_mode specifies where to find the cacert.pem or skip tls verification
+      #   can be one of these:
+      #       "cert_directory" - use the cacert.pem stored locally in cert_directory.
+      #                          this is the default if cacert.pem file is found
+      #
+      #       "os_ca_bundle"     - use the public ca_bundle provided by linux system.
+      #                          this is the default if cacert.pem file not found
+      #
+      #       "do_not_verify"  - special hack to turn off the verification by cacert and hostname
+      tls_ca_mode : "cert_directory"
+      # optional timeout_in_secs specifies the timeout for the http requests
+      timeout_in_secs: 60
+
+    # optional service_activator config
+    #    is used to report the active-passive mode_of_operation of the DCAE-C cluster
+    service_activator :
+      # name of service_activator service used by policy-handler for logging
+      target_entity : "service_activator"
+      # url of the service_activator service for policy-handler to detect the mode-of-operation
+      url : "http://service_activator:123"
+      # path-endpoint to posting the registration to get the mode_of_operation
+      path_register : "/register"
+      # optional tls_ca_mode specifies where to find the cacert.pem or skip tls verification
+      #   can be one of these:
+      #       "cert_directory" - use the cacert.pem stored locally in cert_directory.
+      #                          this is the default if cacert.pem file is found
+      #
+      #       "os_ca_bundle"     - use the public ca_bundle provided by linux system.
+      #                          this is the default if cacert.pem file not found
+      #
+      #       "do_not_verify"  - special hack to turn off the verification by cacert and hostname
+      tls_ca_mode : "cert_directory"
+      # optional timeout_in_secs specifies the timeout for the http requests
+      timeout_in_secs : 20
+      # /register request message to post to the service_activator
+      # put anything that service_activator expects for the registration of the policy-handler
+      post_register :
+        # discoverable component name
+        component_name : "policy_handler"
+        # endpoint on policy-handler that will receive the POST on reconfigure event
+        reconfigure_path : "/reconfigure"
+        # protocol for the /reconfigure event
+        http_protocol : "http"
+```
+
+----------
+
 ## run
 
 in folder `policy_handler`: