DCAE R10 ARC updates

Change-Id: Ie9a063aff6b5ab07f99849cf158b99da4fc133f6
Signed-off-by: Vijay Venkatesh Kumar <vv770d@att.com>
Issue-ID: DCAEGEN2-3140

3 files changed, 36 insertions, 40 deletions

diff --git a/docs/sections/architecture.rst b/docs/sections/architecture.rst
index 31164d62..04d41daa 100644
--- a/docs/sections/architecture.rst
+++ b/docs/sections/architecture.rst
@@ -6,46 +6,26 @@
 Architecture

 ============

 

-Data Collection Analytics and Events (DCAE) is the primary data collection and analysis system of ONAP. DCAE architecture comprises of DCAE Platform and 

-DCAE Service components making DCAE flexible, elastic, and expansive enough for supporting the potentially infinite number of ways of constructing intelligent 

-and automated control loops on distributed and heterogeneous infrastructure. 

+DCAE project provides intelligence for ONAP to support automation (via open-loop and CL) by performing network data collections, analytics & correlation and trigger actionable rootcause events.

 

-DCAE Platform supports the functions to deploy, host and perform LCM applications of Service components. DCAE Platform components enable model driven deployment of 

-service components and middleware infrastructures that service components depend upon, such as special storage and computation platforms.  When triggered by an 

-invocation call (such as CLAMP or via DCAE Dashboard),  DCAE Platform follows the TOSCA model of the control loop that is specified by the triggering call, 

-interacts with the underlying networking and computing infrastructure such as OpenSatck installations and Kubernetes clusters to deploy and configure the virtual 

-apparatus (i.e. the collectors, the analytics, and auxiliary microservices) that are needed to form the control loop, at locations that requested.  

-DCAE Platform also provisions DMaaP topics and manages the distribution scopes of the topics following the prescription of the control loop model by interacting 

-with controlling function of DMaaP.

+Prior to Jakara release, DCAE architecture comprised of DCAE Platform and DCAE Service components; DCAE Platform supported the functions to deploy, host and perform LCM applications of Service components.

 

-DCAE Service components are the  functional entities that realize the collection and analytics needs of ONAP control loops.  They include the collectors for various 

-data collection needs, event processors for data standardization,  analytics that assess collected data, and various auxiliary microservices that assist data 

-collection and analytics, and support other ONAP functions.  Service components and DMaaP buses form the "data plane" for DCAE, where DCAE collected data is 

-transported among different DCAE service components.

+With Jakarta release, DCAE Platform component centered around Cloudify has been deprecated. All Microservice orchestration and lifecycle management will be handled through Helm/Kubernetes. 

 

-DCAE use Consul's distributed K-V store service to manage component configurations where each key is based on the unique identity of a DCAE component (identified by ServiceComponentName), and the value is the configuration for the corresponding component. The K-V store for each service components is created during deployment. DCAE platform creates and updates the K-V pairs based on information provided as part of the control loop blueprint deployment, or through a notification/trigger received from other ONAP components such as Policy Framework and CLAMP.  Either through periodically polling or proactive pushing, the DCAE  components get the configuration updates in realtime and apply the configuration updates.  DCAE Platform also offers dynamic template resolution for configuration parameters that are dynamic and only known by the DCAE platform, such as dynamically provisioned DMaaP topics. This approach standardizes component deployment and configuration management for DCAE service components in multi-site deployment.

+The DCAE services components includes all the microservices - collectors, analytics and event processor which supports active data-flow  and processing as required by ONAP usecases. These Service components are the functional entities that realize the various 

+data collection needs, event processors for data standardization, analytics that assess collected data, and various auxiliary microservices that assist automated closed loop flows.

+

+The architecture of DCAE with Helm transformation is more flexible, microsservice oriented and supports model based component design and deployment through DCAE-MOD. Also with migration to helm, DCAE miroservice deployments can be handled independently, dependencies are captured under its helm charts.

+

+Prior to Jakarta release, DCAE components relied on Consul's distributed K-V to manage and store component configuration. However with Jakarta release, Consul dependency has been removed completely across all DCAE service components. 

+All Microservice configuration are resolved through files mounted via Configmap created part of 

+dcae-services helm chart deployment. 

 

 

 DCAE Components

 ---------------

 

-The following lists the components included in ONAP DCAE .  All DCAE components are offered as Docker containers.  Following ONAP level deployment methods, these components can be deployed as Kubernetes Deployments and Services.  

-

-- DCAE Platform

-    - Core Platform

-        - Cloudify Manager: TOSCA model executor.  Materializes TOSCA models of control loop, or Blueprints, into properly configured and managed virtual DCAE functional components.

-        - Plugins (K8S, Dmaap, Policy, Clamp, Postgres)

-    - Extended Platform

-        - Configuration Binding Service: Agent for service component configuration fetching; providing configuration parameter resolution.

-        - Deployment Handler: API for triggering control loop deployment based on control loop's TOSCA model.

-        - Policy Handler: Handler for fetching policy updates from Policy engine; and updating the configuration policies of KV entries in Consul cluster KV store for DCAE components.

-        - Service Change Handler: Handler for interfacing with SDC; receiving new TOSCA models; and storing them in DCAE's own inventory.

-        - DCAE Inventory-API: API for DCAE's TOSCA model store.

-        - VES OpenApi Manager: Optional validator of VES_EVENT type artifacts executed during Service distributions.

-    - Platform services

-        - Consul: Distributed service discovery service and KV store.

-        - Postgres Database: DCAE's TOSCA model store.

-        - Redis Database: DCAE's transactional state store, used by TCA for supporting persistence and seamless scaling.

+The following lists the components included in ONAP DCAE.  All DCAE components are offered as Docker containers.  Following ONAP level deployment methods, these components can be deployed as Kubernetes Deployments and Services.  

 

 - DCAE Services

     - Collectors

@@ -68,6 +48,8 @@ The following lists the components included in ONAP DCAE .  All DCAE components
         - BBS-EventProcessor Service

         - PM Subcription Handler

         - DataLake Handlers (DL-Admin, DL-Feeder, DES)

+    - Misc Services

+        - VES OpenApi Manager (Optional validator of VES_EVENT type artifacts executed during Service distributions)

         

 

 The figure below shows the DCAE architecture and how the components work with each other.  The components on the right constitute the Platform/controller components which are statically deployed. The components on the right represent the services which can be both deployed statically or dynamically (via CLAMP)

@@ -76,20 +58,29 @@ The figure below shows the DCAE architecture and how the components work with ea
   The following diagram has been created on https://app.diagrams.net/. There is an editable version of the diagram

   in repository under path docs/sections/images/architecture_diagram. Import this file to mentioned page to edit diagram.

 

-.. image:: images/R8_architecture_diagram.png

+.. image:: images/R10_architecture_diagram.png

  

 

 Deployment Scenarios

 --------------------

 

-Because DCAE service components are deployed on-demand following the control loop needs for managing ONAP deployed services, DCAE must support dynamic and on-demand deployment of service components based on ONAP control loop demands.  This is why all other ONAP components are launched from the ONAP level method, DCAE only deploys a subset of its components during this ONAP deployment process and rest of DCAE components will be deployed on-demand based on usecase needs triggered by control loop request originated from CLAMP, or even by operator manually invoking DCAE's deployment API call.

+Because DCAE service components are deployed on-demand following the control loop needs for managing ONAP deployed services, DCAE must

+support dynamic and on-demand deployment of service components based on ONAP control loop demands.  

 

-ONAP supports deployment through OOM Helm Chart currently (Heat deployment support is discontinued since R3). Hence all DCAE Platform components are deployed via Helm charts - this includes Cloudify Manager, ConfigBinding service, ServiceChange Handler, Policy Handler, Dashboard and Inventory, each with corresponding Helm charts under OOM (https://git.onap.org/oom/tree/kubernetes/dcaegen2/components).   Once DCAE platform components are up and running, rest of DCAE service components required for ONAP  flow are deployed via bootstrap POD, which invokes Cloudify Manager API with Blueprints for various DCAE components that are needed for the built-in collections and control loops flow support.  

-

-To keep the ONAP footprint minimal, only minimal set of MS (required for ONAP Integration usecases) are deployed via bootstrap pod. Rest of service blueprints are available for operator to deploy on-demand as required. 

+With DCAE Transformation to Helm completed in Jakarta/R10 release, all DCAE components deployment will be supported only via helm. 

+ Charts for individual MS are available under **dcaegen2-services** directory under OOM project

+ (https://git.onap.org/oom/tree/kubernetes/dcaegen2-services/components). To keep the ONAP footprint minimal, only minimal set of MS

+ (required for ONAP Integration usecases) are enabled by default on ONAP/DCAE deployment, which includes four DCAE services (HV VES

+ collector, VES collector, PNF Registration Handler, and TCA (Gen2) analytics service).

 

 More details of the DCAE deployment can be found under Installation section.

 

+Architectural Reference

+-----------------------

+

+ - `ARC DCAE Component Description <https://wiki.onap.org/display/DW/ARC+DCAE+Component+Description+-+Jakarta-R10>`_

+ - `R10 M2 ARC Proposal <https://wiki.onap.org/display/DW/DCAE+R10+M2+Architecture+Review>`_

+

 

 Usage Scenarios

 ---------------

@@ -106,13 +97,18 @@ For ONAP  DCAE participates in the following use cases.
 

 - CCVPN :  RestConf Collector, Holmes

 

-- BBS : VES Collector, PRH, BBS-Event Processor, VES-Mapper, RESTConf Collector

+- PNF Registration: VES Collector, PRH

 

-- 5G Bulk PM : DataFile Collector, PM-Mapper, HV-VES

+- 5G Bulk PM : DataFile Collector, PM-Mapper, HV-VES, PMSH

 

 - 5G OOF SON: VES collector, SON-Handler

 

 - 5G E2E Network Slicing: VES collector, Slice Analysis, DES, PM-Mapper, DFC, Datalake feeder

+

+- IBN/CCVPN : VES collector, Slice Analysis, DES,  Datalake feeder

  

 

-In addition, DCAE supports on-demand deployment and configuration of service components via CLAMP.  In such case CLAMP invokes the deployment and configuration of additional TCA instances.

+DCAE supports on-demand deployment and configuration of all its microservices via helm charts. As components can also be onboarded

+ through MOD, the flow output is distributed as helm charts which can be installed on-demand also by the operators. 

+

+Policy/CLAMP K8S participant is another ONAP client which can trigger deployment of DCAE Microservice charts.

diff --git a/docs/sections/images/R10_architecture_diagram.png b/docs/sections/images/R10_architecture_diagram.png
new file mode 100644
index 00000000..c862abd1
--- /dev/null
+++ b/docs/sections/images/R10_architecture_diagram.png
diff --git a/docs/sections/images/R9_architecture_diagram.png b/docs/sections/images/R9_architecture_diagram.png
new file mode 100644
index 00000000..0f6bdc1c
--- /dev/null
+++ b/docs/sections/images/R9_architecture_diagram.png