| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
DCAEGEN2-1851:
- policy-handler now supports the policy-update notification
from the new policy-engine thru DMaaP MR
= no policy-filters - only policy-id values
- see README for discoverable config settings of dmaap_mr client
= DMaaP MR client has the same flexibility as policy_engine
= set the query.timeout to high value like 15000 (default)
- requests to DMaaP MR go through a single blocking connection
- first catch-up only after draining the policy-updates from DMaaP MR
on the first loop
- safe parsing of messages from DMaaP MR
- policy-engine changed the data type for policy-version field
from int to string that is expected to have the semver value
- related change to deployment-handler (DCAEGEN2-2085) has to be
deployed to handle the non-numeric policyVersion
- on new PDP API: http /policy_latest and policy-updates
return the new data from the new PDP API with the following fields
added/renamed by the policy-handler to keep other policy related parts
intact in R4-R6 (see pdp_api/policy_utils.py)
* policyName = policy_id + "." + policyVersion.replace(".","-")
+ ".xml"
* policyVersion = str(metadata["policy-version"])
* "config" - is the renamed "properties" from the new PDP API response
- enabled the /catch_up and the periodic auto-catch-up for the new PDP
API
- enabled GET /policies_latest - returns the latest policies for the
deployed components
- POST /policies_latest - still disabled since no support for the
policy-filters is provided for the new PDP API
- fixed hiding the Authorization value on comparing the configs
- logging of secrets is now sha256 to see whether they changed
- added X-ONAP-RequestID to headers the same way as X-ECOMP-RequestID
- on policy-update process the removal first, then addition
- changed the pool_connections=1 (number of pools) on PDP and DH sides
== only a single destination is expected for each
- log the exception as fatal into error.log
- other minor fixes and refactoring
- unit-test coverage 74%
- integration testing is requested
DCAEGEN2-1976:
- policy-handler is enhanced to get user/password from env vars
for PDP and DMaaP MR clients and overwriting the Authorization field
in https headers received from the discoverable config
= to override the Authorization value on policy_engine,
set the environment vars $PDP_USER and $PDP_PWD in policy-handler
container
= to override the Authorization value on dmaap_mr,
if using https and user-password authentication,
set the environment vars $DMAAP_MR_USER and $DMAAP_MR_PWD in
policy-handler container
Change-Id: Iad8eab9e20e615a0e0d2822f4735dc64c50aa55c
Signed-off-by: Alex Shatov <alexs@att.com>
Issue-ID: DCAEGEN2-1851
Issue-ID: DCAEGEN2-1976
|
|
Change-Id: I7859dde9460620e18edca887f5dfc611639b268c
Issue-ID: DCAEGEN2-1919
Signed-off-by: Schmalzried, Terry (ts862m) <ts862m@att.com>
|
|
- in R4 Dublin the policy-engine introduced a totally new API
- policy-handler now has a startup option to either use the new PDP API
or the old PDP API that was created-updated before the end of 2018
- see README.md and README_pdp_api_v0.md for instructions on how to
setup the policy-handler running either with the new PDP API
or the old (pdp_api_v0) PDP API
- this is a massive refactoring that changed almost all the source files,
but kept the old logic when using the old (pdp_api_v0) PDP API
- all the code related to PDP API version is split into two subfolders
= pdp_api/ contains the new PDP API source code
= pdp_api_v0/ contains the old (2018) PDP API source code
= pdp_client.py imports from either pdp_api or pdp_api_v0
= the rest of the code is only affected when it needs to branch
the logic
- logging to policy_handler.log now shows the path of the source file to
allow tracing which PDP API is actually used
- when the new PDP API is used, the policy-update flow is disabled
= passive mode of operation
= no web-socket
= no periodic catch_up
= no policy-filters
= reduced web-API - only a single /policy_latest endpoint is available
/policies_latest returns 404
/catch_up request is accepted, but ignored
- on new PDP API: http /policy_latest returns the new data from the
new PDP API with the following fields added by the policy-handler
to keep other policy related parts intact in R4
(see pdp_api/policy_utils.py)
= "policyName" = policy_id + "." + "policyVersion" + ".xml"
= "policyVersion" = str("metadata"."policy-version")
= "config" - is the renamed "properties" from the new PDP API response
- unit tests are split into two subfolders as well
= main/ for the new PDP API testing
= pdp_api_v0/ for the old (2018) PDP API
- removed the following line from the license text of changed files
ECOMP is a trademark and service mark of AT&T Intellectual Property.
- the new PDP API is expected to be extended and redesigned in R5 El Alto
- on retiring the old PDP API - the intention is to be able to remove
the pdp_api_v0/ subfolder and minimal related cleanup of the code
that imports that as well as the cleanup of the config.py, etc.
Change-Id: Ief9a2ae4541300308caaf97377f4ed051535dbe4
Signed-off-by: Alex Shatov <alexs@att.com>
Issue-ID: DCAEGEN2-1128
|
|
DCAEGEN2-931:
- exposed POST /reconfigure endpoint on the web-server
that initiates the reconfigure process right away
DCAEGEN2-932:
- mode_of_operation: active or passive
= active is as before this change
= in passive mode the policy-handler
* closes the web-socket to PDP
* skips the periodic catch_ups
* still periodically checks for reconfigure
* still allows usig the web-server to retrieve policies from PDP
- default is active
- when mode_of_operation changes from passive to active,
the policy-handler invokes the catch_up right away
- config-kv contains the optional override field mode_of_operation
= changing the mode_of_operation in config-kv and invoking
POST /reconfigure will bring the new value and change the
mode of operation of the policy-handler if no service_activator
section is provided in consul-kv record
- if config-kv contains the service_activator section,
= the policy-handler registers with service_activator - untested
= and receives the mode_of_operation - untested
= service_activator can POST-notify the policy-handler to
initiate the /reconfigure
- reduced the default web-socket ping interval from 180 to 30
seconds because PDP changed its default timeout on the web-socket
from 400 seconds to 50 seconds
Change-Id: If7dd21c008d9906aca97939be65dfa9c2f007535
Signed-off-by: Alex Shatov <alexs@att.com>
Issue-ID: DCAEGEN2-931
Issue-ID: DCAEGEN2-932
|
|
DCAEGEN2-853:
- stop reporting the absence of policies or updates
as error - this is an expected result == INFO or WARNING
DCAEGEN2-903: preparation for TLS on the web-server of policy-handler
DCAEGEN2-930:
- configurable timeouts for http requests from policy-handler
- added configurable pinging on the web-socket to PDP
- added healthcheck info on the web-socket
- upgraded websocket-client lib to 0.53.0
DCAEGEN2-1017: fixed a bug on policy-filter matching
by filter_config_name
- refactored and enhanced the unit-tests
Change-Id: I111ddc57bb978554ef376cbf916965b6667dad9b
Signed-off-by: Alex Shatov <alexs@att.com>
Issue-ID: DCAEGEN2-853
Issue-ID: DCAEGEN2-903
Issue-ID: DCAEGEN2-930
Issue-ID: DCAEGEN2-1017
|
|
- changed API and functionality - new dataflow
- new dataflow between policy-handler and deployment-handler
on policy-update and catchup
= GETting policy_ids+versions and policy-filters from
deployment-handler
= PUTting policy-update and catchup in the new message format
= data segmenting the policy-update/catchup messages to
deployment-handler to avoid 413 on deployment-handler side
= matching policies from policy-engine to policies
and policy-filters from deployment-handler
= coarsening the policyName filter received from deployment-handler
to reduce the number messages passed to policy-engine on catchup
= consolidating sequential policy-updates into a single request
when the policy-update is busy
- removed policy scope-prefixes from config and logic -
it is not needed anymore because
= the policy matching happens directly to policies and
policy-filters received from deployment-handler
= on catchup - the policy scope-prefix equivalents are calculated
based on the data received from deployment-handler
- API - GET /policies_latest now returns the info on deployed
policy_ids+versions and policy-filters, rather than policies
of the scope-prefixes previously found in config (obsolete)
- not sending an empty catch_up message to deployment-handler
when nothing changed
- send policy-removed to deployment-handler when getting
404-not found from PDP on removal of policy
- config change: removed catch_up.max_skips - obsolete
- brought the latest CommonLogger.py
- minor refactoring - improved naming of variables
Change-Id: I36b3412eefd439088cb693703a6e5f18f4238b00
Signed-off-by: Alex Shatov <alexs@att.com>
Issue-ID: DCAEGEN2-492
|
|
- no change of functionality or API
- removed the unused enum34>=1.1.6 from requirements.txt and setup.py
- refactored run_policy.sh to redirect the stdout+stderr only once
- refactoring to remove smells+vulnerability reported by sonar
-- renamed Config.config to Config.settings
-- removed the commented out code in customizer.py
-- renamed StepTimer.NEXT to StepTimer.STATE_NEXT to avoid the
naming confusion with the method StepTimer.next.
Also renamed the related StepTimer.STATE_* constants
-- refactored several functions by extracting methods to eliminate
4 out of 5 "brain-overload" smells reported by sonar
-- moved the literal string for the socket_host "0.0.0.0" to a
constant on the web-server to avoid the reported vulnerability
Change-Id: I4c7d47d41c6ecd7cb28f6704f5dad2053c1ca7d6
Signed-off-by: Alex Shatov <alexs@att.com>
Issue-ID: DCAEGEN2-515
|
|
- in search of the memory leak that is falsely reported
by docker stats, the following runtime logging was added
= process_memory - rss and other memory of the current process
= virtual_memory - the memory info of the whole system
= thread_stacks - the active threads with the full stack on each
Change-Id: I5f5ab3a477bfba3aecc5963547aa82da6269670b
Signed-off-by: Alex Shatov <alexs@att.com>
Issue-ID: DCAEGEN2-514
|
|
- added try-except for top level Exception into all threads
of policy-handler to avoid losing the thread and tracking
the unexpected crashes
- rediscover the deployment-handler if not found before
and after each catchup
- refactored audit - separated metrics from audit
- added more stats and runtime info to healthcheck
= gc counts and garbage info if any detected
= memory usage - to detect the potential memory leaks
= request_id to all stats
= stats of active requests
- avoid reallocating the whole Queue of policy-updates after catchup
= clear of the internal queue under proper lock
Change-Id: I3fabcaac70419a68bd070ff7d591a75942f37663
Signed-off-by: Alex Shatov <alexs@att.com>
Issue-ID: DCAEGEN2-483
|
|
- fixed the bug of unpredictably stopping of the periodic catch-up
step-timer due to thread race condition in policy-handler
= added critical sections under the reentrant lock on every group
of local var change in step-timer
- added more stats for healthcheck to track each type of
job-operation separately
= that helps narrowing down identifying the potential problems
- unit test coverage 76%
Change-Id: I92ddf6c92a3d225d9b87427e3edfb7f80669501a
Signed-off-by: Alex Shatov <alexs@att.com>
Issue-ID: DCAEGEN2-472
|
|
- added etc_customize/ folder and customize.sh script
= customize.sh script is expected to be overridden by company
to customize Docker image build
= the whole etc_customize/ folder is copied into docker image
= it is up to the company what to put into that folder - any files
- added customize/ folder with CustomizeBase and Customize classes
= CustomizeBase defines the interface and the default=ONAP behavior
= CustomizeBase is owned by ONAP and should not be changed
by the company
= Customize inherits CustomizeBase
= policy-handler instantiates Customize
to get the customized behavior
= Customize is owned by the company and should be changed
by the company = ONAP is not going to change Customize
= the methods of Customize are expected to be overridden
by the company to change the behavior of the policy-handler
= sample Customize class can be found in README.md
= Company is allowed to add more files to customize/ folder
if that is required for better structuring of their code
as soon as it is invoked by the methods of Customize
Change-Id: I46f8170afaaa48e1005e4398a768a781db0a0e6c
Signed-off-by: Alex Shatov <alexs@att.com>
Issue-ID: DCAEGEN2-379
|
|
- removed #org.onap.dcae from license text
Change-Id: I07f11e60c4677109ccb826c4e969b47acb4c498a
Signed-off-by: Alex Shatov <alexs@att.com>
Issue-ID: DCAEGEN2-347
|
|
Change-Id: I685c63758b7ce22766885d399f06e9ba14ca59f2
Issue-ID: DCAEGEN2-249
Signed-off-by: Alex Shatov <alexs@att.com>
|
|
* added errored_scopes and scope_prefixes to the message
to deployment-handler - to prevent erroneous
removal of policies
* hardcoded condition for scope not found 404 at policy-engine
to separate it from error on the scope retrieval 400
* adjusting the web API message in sync with notification
to deployment-handler
* unit test coverage 74%
Change-Id: Ie736a1b7aee0631b6785669c6b765bd240dd77b8
Issue-ID: DCAEGEN2-249
Signed-off-by: Alex Shatov <alexs@att.com>
|
|
* json of what is returned from policy-engine instead of 404 html
Change-Id: I44fb7e113e23f68cac44e6ad21c1bc9b37cf3d58
Issue-ID: DCAEGEN2-249
Signed-off-by: Alex Shatov <alexs@att.com>
|
|
Change-Id: I2a3628cb67d15ab2828f6818764d111df13e795a
Issue-ID: DCAEGEN2-249
Signed-off-by: Alex Shatov <alexs@att.com>
|
|
* new feature variable collection of policies per component in DCAE
* massive refactoring
* dissolved the external PolicyEngine.py into policy_receiver.py
- kept only the web-socket communication to PolicyEngine
* new /healthcheck - shows some stats of service running
* Unit Test coverage 75%
Change-Id: I816b7d5713ae0dd88fa73d3656f272b4f3e7946e
Issue-ID: DCAEGEN2-249
Signed-off-by: Alex Shatov <alexs@att.com>
|
|
Change-Id: I35cd80b6e082f4b84740bab752774e8abc40ca35
Issue-Id: DCAEGEN2-46
Signed-off-by: Alex Shatov <alexs@att.com>
|
Alex Shatov
Schmalzried, Terry (ts862m)