diff options
21 files changed, 2287 insertions, 1804 deletions
diff --git a/docs/requirements-docs.txt b/docs/requirements-docs.txt index 5a3d2f17..3b3441a8 100644 --- a/docs/requirements-docs.txt +++ b/docs/requirements-docs.txt @@ -1 +1,3 @@ -lfdocs-conf
\ No newline at end of file +lfdocs-conf +sphinx>=4.2.0 # BSD +sphinx-rtd-theme>=1.0.0 # MIT diff --git a/docs/sections/architecture.rst b/docs/sections/architecture.rst index 31164d62..04d41daa 100644 --- a/docs/sections/architecture.rst +++ b/docs/sections/architecture.rst @@ -6,46 +6,26 @@ Architecture
============
-Data Collection Analytics and Events (DCAE) is the primary data collection and analysis system of ONAP. DCAE architecture comprises of DCAE Platform and
-DCAE Service components making DCAE flexible, elastic, and expansive enough for supporting the potentially infinite number of ways of constructing intelligent
-and automated control loops on distributed and heterogeneous infrastructure.
+DCAE project provides intelligence for ONAP to support automation (via open-loop and CL) by performing network data collections, analytics & correlation and trigger actionable rootcause events.
-DCAE Platform supports the functions to deploy, host and perform LCM applications of Service components. DCAE Platform components enable model driven deployment of
-service components and middleware infrastructures that service components depend upon, such as special storage and computation platforms. When triggered by an
-invocation call (such as CLAMP or via DCAE Dashboard), DCAE Platform follows the TOSCA model of the control loop that is specified by the triggering call,
-interacts with the underlying networking and computing infrastructure such as OpenSatck installations and Kubernetes clusters to deploy and configure the virtual
-apparatus (i.e. the collectors, the analytics, and auxiliary microservices) that are needed to form the control loop, at locations that requested.
-DCAE Platform also provisions DMaaP topics and manages the distribution scopes of the topics following the prescription of the control loop model by interacting
-with controlling function of DMaaP.
+Prior to Jakara release, DCAE architecture comprised of DCAE Platform and DCAE Service components; DCAE Platform supported the functions to deploy, host and perform LCM applications of Service components.
-DCAE Service components are the functional entities that realize the collection and analytics needs of ONAP control loops. They include the collectors for various
-data collection needs, event processors for data standardization, analytics that assess collected data, and various auxiliary microservices that assist data
-collection and analytics, and support other ONAP functions. Service components and DMaaP buses form the "data plane" for DCAE, where DCAE collected data is
-transported among different DCAE service components.
+With Jakarta release, DCAE Platform component centered around Cloudify has been deprecated. All Microservice orchestration and lifecycle management will be handled through Helm/Kubernetes.
-DCAE use Consul's distributed K-V store service to manage component configurations where each key is based on the unique identity of a DCAE component (identified by ServiceComponentName), and the value is the configuration for the corresponding component. The K-V store for each service components is created during deployment. DCAE platform creates and updates the K-V pairs based on information provided as part of the control loop blueprint deployment, or through a notification/trigger received from other ONAP components such as Policy Framework and CLAMP. Either through periodically polling or proactive pushing, the DCAE components get the configuration updates in realtime and apply the configuration updates. DCAE Platform also offers dynamic template resolution for configuration parameters that are dynamic and only known by the DCAE platform, such as dynamically provisioned DMaaP topics. This approach standardizes component deployment and configuration management for DCAE service components in multi-site deployment.
+The DCAE services components includes all the microservices - collectors, analytics and event processor which supports active data-flow and processing as required by ONAP usecases. These Service components are the functional entities that realize the various
+data collection needs, event processors for data standardization, analytics that assess collected data, and various auxiliary microservices that assist automated closed loop flows.
+
+The architecture of DCAE with Helm transformation is more flexible, microsservice oriented and supports model based component design and deployment through DCAE-MOD. Also with migration to helm, DCAE miroservice deployments can be handled independently, dependencies are captured under its helm charts.
+
+Prior to Jakarta release, DCAE components relied on Consul's distributed K-V to manage and store component configuration. However with Jakarta release, Consul dependency has been removed completely across all DCAE service components.
+All Microservice configuration are resolved through files mounted via Configmap created part of
+dcae-services helm chart deployment.
DCAE Components
---------------
-The following lists the components included in ONAP DCAE . All DCAE components are offered as Docker containers. Following ONAP level deployment methods, these components can be deployed as Kubernetes Deployments and Services.
-
-- DCAE Platform
- - Core Platform
- - Cloudify Manager: TOSCA model executor. Materializes TOSCA models of control loop, or Blueprints, into properly configured and managed virtual DCAE functional components.
- - Plugins (K8S, Dmaap, Policy, Clamp, Postgres)
- - Extended Platform
- - Configuration Binding Service: Agent for service component configuration fetching; providing configuration parameter resolution.
- - Deployment Handler: API for triggering control loop deployment based on control loop's TOSCA model.
- - Policy Handler: Handler for fetching policy updates from Policy engine; and updating the configuration policies of KV entries in Consul cluster KV store for DCAE components.
- - Service Change Handler: Handler for interfacing with SDC; receiving new TOSCA models; and storing them in DCAE's own inventory.
- - DCAE Inventory-API: API for DCAE's TOSCA model store.
- - VES OpenApi Manager: Optional validator of VES_EVENT type artifacts executed during Service distributions.
- - Platform services
- - Consul: Distributed service discovery service and KV store.
- - Postgres Database: DCAE's TOSCA model store.
- - Redis Database: DCAE's transactional state store, used by TCA for supporting persistence and seamless scaling.
+The following lists the components included in ONAP DCAE. All DCAE components are offered as Docker containers. Following ONAP level deployment methods, these components can be deployed as Kubernetes Deployments and Services.
- DCAE Services
- Collectors
@@ -68,6 +48,8 @@ The following lists the components included in ONAP DCAE . All DCAE components - BBS-EventProcessor Service
- PM Subcription Handler
- DataLake Handlers (DL-Admin, DL-Feeder, DES)
+ - Misc Services
+ - VES OpenApi Manager (Optional validator of VES_EVENT type artifacts executed during Service distributions)
The figure below shows the DCAE architecture and how the components work with each other. The components on the right constitute the Platform/controller components which are statically deployed. The components on the right represent the services which can be both deployed statically or dynamically (via CLAMP)
@@ -76,20 +58,29 @@ The figure below shows the DCAE architecture and how the components work with ea The following diagram has been created on https://app.diagrams.net/. There is an editable version of the diagram
in repository under path docs/sections/images/architecture_diagram. Import this file to mentioned page to edit diagram.
-.. image:: images/R8_architecture_diagram.png
+.. image:: images/R10_architecture_diagram.png
Deployment Scenarios
--------------------
-Because DCAE service components are deployed on-demand following the control loop needs for managing ONAP deployed services, DCAE must support dynamic and on-demand deployment of service components based on ONAP control loop demands. This is why all other ONAP components are launched from the ONAP level method, DCAE only deploys a subset of its components during this ONAP deployment process and rest of DCAE components will be deployed on-demand based on usecase needs triggered by control loop request originated from CLAMP, or even by operator manually invoking DCAE's deployment API call.
+Because DCAE service components are deployed on-demand following the control loop needs for managing ONAP deployed services, DCAE must
+support dynamic and on-demand deployment of service components based on ONAP control loop demands.
-ONAP supports deployment through OOM Helm Chart currently (Heat deployment support is discontinued since R3). Hence all DCAE Platform components are deployed via Helm charts - this includes Cloudify Manager, ConfigBinding service, ServiceChange Handler, Policy Handler, Dashboard and Inventory, each with corresponding Helm charts under OOM (https://git.onap.org/oom/tree/kubernetes/dcaegen2/components). Once DCAE platform components are up and running, rest of DCAE service components required for ONAP flow are deployed via bootstrap POD, which invokes Cloudify Manager API with Blueprints for various DCAE components that are needed for the built-in collections and control loops flow support.
-
-To keep the ONAP footprint minimal, only minimal set of MS (required for ONAP Integration usecases) are deployed via bootstrap pod. Rest of service blueprints are available for operator to deploy on-demand as required.
+With DCAE Transformation to Helm completed in Jakarta/R10 release, all DCAE components deployment will be supported only via helm.
+ Charts for individual MS are available under **dcaegen2-services** directory under OOM project
+ (https://git.onap.org/oom/tree/kubernetes/dcaegen2-services/components). To keep the ONAP footprint minimal, only minimal set of MS
+ (required for ONAP Integration usecases) are enabled by default on ONAP/DCAE deployment, which includes four DCAE services (HV VES
+ collector, VES collector, PNF Registration Handler, and TCA (Gen2) analytics service).
More details of the DCAE deployment can be found under Installation section.
+Architectural Reference
+-----------------------
+
+ - `ARC DCAE Component Description <https://wiki.onap.org/display/DW/ARC+DCAE+Component+Description+-+Jakarta-R10>`_
+ - `R10 M2 ARC Proposal <https://wiki.onap.org/display/DW/DCAE+R10+M2+Architecture+Review>`_
+
Usage Scenarios
---------------
@@ -106,13 +97,18 @@ For ONAP DCAE participates in the following use cases. - CCVPN : RestConf Collector, Holmes
-- BBS : VES Collector, PRH, BBS-Event Processor, VES-Mapper, RESTConf Collector
+- PNF Registration: VES Collector, PRH
-- 5G Bulk PM : DataFile Collector, PM-Mapper, HV-VES
+- 5G Bulk PM : DataFile Collector, PM-Mapper, HV-VES, PMSH
- 5G OOF SON: VES collector, SON-Handler
- 5G E2E Network Slicing: VES collector, Slice Analysis, DES, PM-Mapper, DFC, Datalake feeder
+
+- IBN/CCVPN : VES collector, Slice Analysis, DES, Datalake feeder
-In addition, DCAE supports on-demand deployment and configuration of service components via CLAMP. In such case CLAMP invokes the deployment and configuration of additional TCA instances.
+DCAE supports on-demand deployment and configuration of all its microservices via helm charts. As components can also be onboarded
+ through MOD, the flow output is distributed as helm charts which can be installed on-demand also by the operators.
+
+Policy/CLAMP K8S participant is another ONAP client which can trigger deployment of DCAE Microservice charts.
diff --git a/docs/sections/design-components/DCAE-MOD/DCAE-MOD-Architecture.rst b/docs/sections/design-components/DCAE-MOD/DCAE-MOD-Architecture.rst index 64bd4dc2..096a448e 100644 --- a/docs/sections/design-components/DCAE-MOD/DCAE-MOD-Architecture.rst +++ b/docs/sections/design-components/DCAE-MOD/DCAE-MOD-Architecture.rst @@ -1,3 +1,6 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + ===================== DCAE MOD Architecture ===================== @@ -104,18 +107,89 @@ major functionalities: 1. It accepts changes on the flow-graph via fbp protocol -2. It generates and distributes blueprints based on the change made on +2. It generates and distributes helm charts OR blueprints based on the change made on the flow-graph + +Build Updates +~~~~~~~~~~~~~ + +New Java module - Helmgenerator-core was introduced for Helm charts +generation. MOD/Runtime has been enhanced to include this new dependency +(inaddition to Bp-generator for supporting cloudify blueprints flows). + +Below is snippet from - +https://github.com/onap/dcaegen2-platform/blob/master/mod/runtimeapi/runtime-core/pom.xml + +:: + + <dependency> + <groupId>org.onap.dcaegen2.platform</groupId> + <artifactId>helmchartgenerator-core</artifactId> + <version>1.0.3</version> + </dependency> + + +Chart Updates +~~~~~~~~~~~~~ + +MOD/Runtime Charts has been modified to include under resources, common +base templates, Charts.yaml, add-on templates and Values.yaml with +placeholder. + +|image3| + +The Helmgenerator-core modules uses these template to pull the required +dependencies and generate new chart for MS onboarded. The parameters in +component-spec provided during onboarding is used for final Values.yaml +file generation. + +Deployment +~~~~~~~~~~ + +The MOD/RuntimeAPI introduces new configuration to identify distribution +mechanism. Supported artifactType are **BLUEPRINT** or **HELM. ** + +Blueprint – Distribution to Inventory/Dashboard + +Helm – Distribution to ChartMuseum + +For Jakarta release, the charts configuration has been set to support +HELM distribution by default and configured for ONAP-internal +chart-museum registry. RuntimeAPI Chart updates +https://github.com/onap/oom/blob/master/kubernetes/dcaemod/components/dcaemod-runtime-api/values.yaml + +:: + + artifactType: "HELM" + registryBaseurl: http://chart-museum:80 + basehelmchartlocation: /helm-gen/ + + Blueprint Generator ------------------- This tool allows the user to create a blueprint from a component spec json file. -This tool is used by the runtime api. +This tool is used by the runtime api when artifactType is set to **BLUEPRINT** under +`RuntimeAPI charts + deployment <https://git.onap.org/oom/tree/kubernetes/dcaemod/components/dcaemod-runtime-api/values.yaml#n44>`__ + +Helm Generator +-------------- + +This tool allows the user to create a DCAE Services helm chart from a component spec json file. +This tool is used by the runtime api when artifactType is set to **HELM** under +`RuntimeAPI charts + deployment <https://git.onap.org/oom/tree/kubernetes/dcaemod/components/dcaemod-runtime-api/values.yaml#n44>`__ + Inventory API ------------- + +.. note:: + Used only under BLUEPRINT mode + DCAE Inventory is a web service that provides the following: @@ -146,6 +220,9 @@ dashboard. DCAE Dashboard -------------- +.. note:: + Used only under BLUEPRINT mode + The DCAE dashboard provides visibility into running DCAE services for operational purposes. It queries the DCAE Inventory for aggregate details on all the running DCAE services and for getting up-to-date @@ -165,9 +242,11 @@ The registry api offers version control and retrieval for flows. The distributor api can be used to set distribution targets. Once a flow is designed and distributed, it goes to the distributor api which is supposed to post graph changes (in accordance with fbp) to the runtime -api. The runtime api generates and distributes blueprints based on the -change made on the flow-graph. These blueprints received by the DCAE -inventory can then be viewed and deployed from the DCAE dashboard. +api. The runtime api generates and distributes deployment artifacts (either +blueprints or helm charts) based on the +change made on the flow-graph. The generated blueprints are received by the DCAE +inventory can then be viewed and deployed from the DCAE dashboard. On helm mode, +charts generated are pushed into configured Chartmuseum registry. @@ -176,3 +255,5 @@ inventory can then be viewed and deployed from the DCAE dashboard. .. |image1| image:: ../images/Onboarding-with-DCAE-MOD.png .. |image2| image:: ../images/nifi-toolbar-components.png + +.. |image3| image:: ../images/128713731_image2022.png diff --git a/docs/sections/design-components/DCAE-MOD/DCAE-MOD-User-Guide.rst b/docs/sections/design-components/DCAE-MOD/DCAE-MOD-User-Guide.rst index fb51d84a..d042028f 100644 --- a/docs/sections/design-components/DCAE-MOD/DCAE-MOD-User-Guide.rst +++ b/docs/sections/design-components/DCAE-MOD/DCAE-MOD-User-Guide.rst @@ -1,5 +1,6 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 + .. _moduserguide: =================== @@ -57,8 +58,106 @@ Types of Users and Usage Instructions: | | and validate it | | +-------+-----------------------------+-----------------------------+ +1. Pre-requisite for DCAE MOD Deployment +=========================================== + +With complete of DCAE Helm tranformation in Jakarta release, DCAE MOD has been enhanced +to support Helm chart generation for microservices onboarded. +In order to support the HELM flow through MOD, following dependency should be met + + +- An accessible ChartMuseum registry (internal or external) + +- As the provided registry is used both to pull required dependencies + and push new generated charts, all common charts used by DCAE + components must be available in this registry. + +.. note:: + By default, MOD charts are set to use local chartmuseum registry. This can be modified by + updating the `RuntimeAPI charts + deployment <https://git.onap.org/oom/tree/kubernetes/dcaemod/components/dcaemod-runtime-api/values.yaml#n44>`__ + + +ONAP deployments (gating) will include Chartmuseum installation within +ONAP cluster (charts hosted here +- https://github.com/onap/oom/tree/master/kubernetes/platform/components/chartmuseum). + +Dependent charts such as - dcaegen2-services-common, readinessCheck, +common, repositoryGenerator, postgres, mongo, serviceAccount, +certInitializer should be preloaded into this registry as MOD retrieves +them during new MS helm charts creation and linting. To support the +registry initialization, following scripts has been introduced. + +- https://github.com/onap/oom/blob/master/kubernetes/contrib/tools/registry-initialize.sh + +- https://github.com/onap/oom/blob/master/kubernetes/robot/demo-k8s.sh + +Note: Chartmuseum being a platform component, it has to be enabled +on-demand and not available with generic ONAP installation. + +Follow below steps to setup chartmuseum and pre-load required charts. + +Chartmuseum Installation +------------------------ + +Clone OOM repository and deploy optional Chartmuseum component + +**Chartmuseum Deployment** + +:: + + # git clone -b <BRANCH> http://gerrit.onap.org/r/oom --recurse-submodules + cd ~/oom/kubernetes/platform/components/chartmuseum + helm install -name dev-chartmuseum -n onap . -f ~/onap-1-override.yaml --set global.masterPassword=test1 --set global.pullPolicy=IfNotPresent -1. Deployment of DCAE MOD components via Helm charts + +.. note:: + This instance of chartmuseum registry is deployed internal to ONAP cluster and + is different from the registry setup done part `OOM + deployment <https://docs.onap.org/projects/onap-oom/en/latest/oom_quickstart_guide.html>`__ + where local helm server is setup for serving chart and to pull/push the + charts generated make process + +Chartmuseum initialization +-------------------------- + +As noted earlier, there are two scripts available for pre-load. The +`registry-initialize.sh <https://github.com/onap/oom/blob/master/kubernetes/contrib/tools/registry-initialize.sh>`__ +retrieves the Chartmuseum credential from secret and load the charts +individually based on parameter (default no parameters, will load all +DCAE service charts and its dependencies). And +`demo-k8s.sh <https://github.com/onap/oom/blob/master/kubernetes/robot/demo-k8s.sh>`__ +is wrapper script used in gating, which invokes +`registry-initialize.sh <https://github.com/onap/oom/blob/master/kubernetes/contrib/tools/registry-initialize.sh>`__ +with required parameters. + +**Chartmuseum initialization via demo-k8s.sh** + +:: + + cd ~/oom/kubernetes/robot + ./demo-k8s.sh onap registrySynch + +OR + +**Chartmuseum initialization via registry-initialize script** + +:: + + cd ~/oom/kubernetes/contrib/tools + ./registry-initialize.sh -d ../../dcaegen2-services/charts/ -n onap -r dev-chartmuseum + ./registry-initialize.sh -d ../../dcaegen2-services/charts/ -n onap -r dev-chartmuseum -p common + ./registry-initialize.sh -h repositoryGenerator -n onap -r dev-chartmuseum + ./registry-initialize.sh -h readinessCheck -n onap -r dev-chartmuseum + ./registry-initialize.sh -h dcaegen2-services-common -n onap -r dev-chartmuseum + ./registry-initialize.sh -h postgres -n onap -r dev-chartmuseum + ./registry-initialize.sh -h serviceAccount -n onap -r dev-chartmuseum + ./registry-initialize.sh -h certInitializer -n onap -r dev-chartmuseum + ./registry-initialize.sh -h mongo -n onap -r dev-chartmuseum + + + +2. Deployment of DCAE MOD components via Helm charts ======================================================= The DCAE MOD components are deployed using the standard ONAP OOM @@ -74,7 +173,8 @@ deployed. The Rancher RKE installation process sets up a suitable ingress controller. In order to enable the use of the ingress controller, it is necessary to override the OOM default global settings for ingress configuration. Specifically, the installation needs to set -the following configuration in an override file:: +the following configuration in an override file +:: ingress: enabled: true @@ -135,13 +235,21 @@ All MOD API's and UI access via ingress should use dcaemod.simpledemo.onap.org. In order to access Design UI from local, add an entry for dcaemod.simpledemo.onap.org in /etc/hosts with the correct IP (any K8S node IP can be specified). +Example below using generic override + +**Deploy MOD** + +:: + + helm install dev-dcaemod local/dcaemod --namespace onap -f ~/onap-override.yaml --set global.masterPassword=test1 --set global.pullPolicy=IfNotPresent + Using DCAE MOD without an Ingress Controller Not currently supported -2. Configuring DCAE mod +3. Configuring DCAE mod ========================== **a. Configure Nifi Registry url** @@ -179,11 +287,12 @@ IPAddress is the host address or the DNS FQDN, if there is one, for one of the K **c. Get the artifacts to test and onboard.** -Let's fetch the artifacts/ spec files +MOD components has been upgraded to use v3 specification for Helm flow support -**Component Spec for DCAE-VES-Collector :** https://git.onap.org/dcaegen2/collectors/ves/tree/dpo/spec/vescollector-componentspec.json +VESCollector +~~~~~~~~~~~~ -**Component Spec for DCAE-TCAgen2 :** https://git.onap.org/dcaegen2/collectors/ves/tree/dpo/spec/vescollector-componentspec.json +**Component Spec for DCAE-VES-Collector :** https://git.onap.org/dcaegen2/collectors/ves/tree/dpo/spec/vescollector-componentspec-v3.json **VES 5.28.4 Data Format :** https://git.onap.org/dcaegen2/collectors/ves/tree/dpo/data-formats/VES-5.28.4-dataformat.json @@ -191,8 +300,19 @@ Let's fetch the artifacts/ spec files **VES Collector Response Data Format :** https://git.onap.org/dcaegen2/collectors/ves/tree/dpo/data-formats/ves-response.json + +TCAGen2 +~~~~~~~ + +**Component Spec for DCAE-TCAgen2 :** https://git.onap.org/dcaegen2/collectors/ves/tree/dpo/spec/vescollector-componentspec.json + **TCA CL Data Format :** https://git.onap.org/dcaegen2/analytics/tca-gen2/tree/dcae-analytics/dpo/dcaeCLOutput.json +**TCA DMAAP Format :** https://git.onap.org/dcaegen2/analytics/tca-gen2/tree/dcae-analytics/dpo/dmaap.json + +**TCA AAI Data Format :** https://git.onap.org/dcaegen2/analytics/tca-gen2/tree/dcae-analytics/dpo/aai.json + + For the purpose of onboarding, a Sample Request body should be of the type -:: @@ -205,15 +325,16 @@ Request bodies of this type will be used in the onboarding requests you make usi **The prepared Sample Request body for a component dcae-ves-collector looks like so –** -See :download:`VES Collector Spec <./Sample-Input-Files/Request-body-of-Sample-Component.json>` +See :download:`VES Collector Spec <./Sample-Input-Files/Request-body-of-Sample-Component_v3.json>` **The prepared Sample request body for a sample data format looks like so -** See :download:`VES data Format <./Sample-Input-Files/Request-body-of-Sample-Data-Format.json>` +Similar updates should be done for other specification and data-formats files -**d. To onboard a data format and a component** +**d. Onboard data format and component-spec** Each component has a description that tells what it does. @@ -231,13 +352,30 @@ curl -X POST http://dcaemod.simpledemo.onap.org/onboarding/dataformats - curl -X POST http://dcaemod.simpledemo.onap.org/onboarding/components -H "Content-Type: application/json" -d @<filepath to request> +**Onboard Specs and DF** + +:: + + HOST=dcaemod.simpledemo.onap.org + curl -X POST http://$HOST/onboarding/dataformats -H "Content-Type: application/json" -d @ves-4.27.2-df.json + curl -X POST http://$HOST/onboarding/dataformats -H "Content-Type: application/json" -d @ves-5.28.4-df.json + curl -X POST http://$HOST/onboarding/dataformats -H "Content-Type: application/json" -d @ves-response-df.json + curl -X POST http://$HOST/onboarding/dataformats -H "Content-Type: application/json" -d @VES-7.30.2_ONAP-dataformat_onboard.json + curl -X POST http://$HOST/onboarding/components -H "Content-Type: application/json" -d @vescollector-componentspec-v3-mod.json + + curl -X POST http://$HOST/onboarding/dataformats -H "Content-Type: application/json" -d @dcaeCLOutput-resp.json + curl -X POST http://$HOST/onboarding/dataformats -H "Content-Type: application/json" -d @aai-resp.json + curl -X POST http://$HOST/onboarding/components -H "Content-Type: application/json" -d @tcagen2-componentspec-v3-mod.json + +You can download the Component Specification and Data Formats used for +the demo from here - `demo.zip <https://wiki.onap.org/download/attachments/128713665/demo.zip?version=1&modificationDate=1646673042000&api=v2>`__ **e. Verify the resources were created using** -curl -X GET http://<IPAddress>/onboarding/dataformats +curl -X GET http://dcaemod.simpledemo.onap.org/onboarding/dataformats -curl -X GET http://<IPAddress>/onboarding/components +curl -X GET http://dcaemod.simpledemo.onap.org/onboarding/components **f. Verify the genprocessor (which polls onboarding periodically to convert component specs to nifi processor), converted the component** @@ -248,7 +386,7 @@ processors |image1| -3. Design & Distribution Flow +4. Design & Distribution Flow ================================ @@ -339,43 +477,100 @@ get a pop up a success message like so - At this step, the design was packaged and sent to Runtime api. -The runtime is supposed to generate the blueprint out of the packaged -design/flow and push it to the DCAE inventory and the DCAE Dasboard. - -**c. Checking the components in the DCAE Dashboard** - -You should see the generated artifact/ blueprint in the DCAE Dashboard -dashboard at https://<IPAddress>:30418/ccsdk-app/login_external.htm in -our deployment. The name for each component will be appended by the flow -name followed by underscore followed by the component’s name. - -The credentials to access the DCAE Dashboard - -Login: su1234 -Password: fusion - +The runtime is supposed to generate the Helmchart for components +involved in the flow and push them to registry configured. The +RuntimeAPI logs should looks like below for successful distribution (can +be viewed through kubectl log -f command) + +**MOD/RuntimeAPI Console logs** + +:: + + 2022-03-07 18:13:25.865 INFO 1 --- [nio-9090-exec-8] o.o.d.r.web.controllers.GraphController : org.onap.dcae.runtime.web.models.GraphRequest@65efc9d3 + 2022-03-07 18:13:26.119 INFO 1 --- [nio-9090-exec-1] o.o.d.r.web.controllers.GraphController : [org.onap.dcae.runtime.web.models.Action@335a6cff, org.onap.dcae.runtime.web.models.Action@291687dd, org.onap.dcae.runtime.web.models.Action@36d57691] + 2022-03-07 18:13:26.142 INFO 1 --- [nio-9090-exec-1] o.o.d.platform.helmchartgenerator.Utils : cloning dir/file at : /tmp/chart17927059362260733428 + 2022-03-07 18:13:26.158 INFO 1 --- [nio-9090-exec-1] o.o.d.p.h.chartbuilder.HelmClientImpl : running: helm dep up /tmp/chart17927059362260733428 + Hang tight while we grab the latest from your chart repositories... + ...Successfully got an update from the "local" chart repository + Update Complete. ⎈Happy Helming!⎈ + Saving 7 charts + Downloading common from repo http://chart-museum:80 + Downloading repositoryGenerator from repo http://chart-museum:80 + Downloading readinessCheck from repo http://chart-museum:80 + Downloading dcaegen2-services-common from repo http://chart-museum:80 + Downloading postgres from repo http://chart-museum:80 + Downloading serviceAccount from repo http://chart-museum:80 + Downloading mongo from repo http://chart-museum:80 + Deleting outdated charts + 2022-03-07 18:13:26.273 INFO 1 --- [nio-9090-exec-1] o.o.d.p.h.chartbuilder.HelmClientImpl : running: helm lint /tmp/chart17927059362260733428 + 2022-03-07 18:13:30.641 INFO 1 --- [nio-9090-exec-1] o.o.d.p.h.chartbuilder.HelmClientImpl : ==> Linting /tmp/chart17927059362260733428 + 2022-03-07 18:13:30.642 INFO 1 --- [nio-9090-exec-1] o.o.d.p.h.chartbuilder.HelmClientImpl : [INFO] Chart.yaml: icon is recommended + 2022-03-07 18:13:30.642 INFO 1 --- [nio-9090-exec-1] o.o.d.p.h.chartbuilder.HelmClientImpl : + 2022-03-07 18:13:30.642 INFO 1 --- [nio-9090-exec-1] o.o.d.p.h.chartbuilder.HelmClientImpl : 1 chart(s) linted, 0 chart(s) failed + 2022-03-07 18:13:30.646 INFO 1 --- [nio-9090-exec-1] o.o.d.p.h.chartbuilder.HelmClientImpl : running: helm package -d /tmp/chart13832736430918913290 /tmp/chart17927059362260733428 + 2022-03-07 18:13:30.737 INFO 1 --- [nio-9090-exec-1] o.o.d.p.h.chartbuilder.HelmClientImpl : Successfully packaged chart and saved it to: /tmp/chart13832736430918913290/dcae-ves-collector-1.10.1.tgz + 2022-03-07 18:13:30.836 INFO 1 --- [nio-9090-exec-1] o.o.d.p.h.d.ChartMuseumDistributor : {"saved":true} + 2022-03-07 18:13:30.857 INFO 1 --- [nio-9090-exec-1] o.o.d.platform.helmchartgenerator.Utils : cloning dir/file at : /tmp/chart7638328545634423550 + 2022-03-07 18:13:30.870 INFO 1 --- [nio-9090-exec-1] o.o.d.p.h.chartbuilder.HelmClientImpl : running: helm dep up /tmp/chart7638328545634423550 + Hang tight while we grab the latest from your chart repositories... + ...Successfully got an update from the "local" chart repository + Update Complete. ⎈Happy Helming!⎈ + Saving 7 charts + Downloading common from repo http://chart-museum:80 + Downloading repositoryGenerator from repo http://chart-museum:80 + Downloading readinessCheck from repo http://chart-museum:80 + Downloading dcaegen2-services-common from repo http://chart-museum:80 + Downloading postgres from repo http://chart-museum:80 + Downloading serviceAccount from repo http://chart-museum:80 + Downloading mongo from repo http://chart-museum:80 + Deleting outdated charts + 2022-03-07 18:13:31.022 INFO 1 --- [nio-9090-exec-1] o.o.d.p.h.chartbuilder.HelmClientImpl : running: helm lint /tmp/chart7638328545634423550 + 2022-03-07 18:13:35.142 INFO 1 --- [nio-9090-exec-1] o.o.d.p.h.chartbuilder.HelmClientImpl : ==> Linting /tmp/chart7638328545634423550 + 2022-03-07 18:13:35.143 INFO 1 --- [nio-9090-exec-1] o.o.d.p.h.chartbuilder.HelmClientImpl : [INFO] Chart.yaml: icon is recommended + 2022-03-07 18:13:35.143 INFO 1 --- [nio-9090-exec-1] o.o.d.p.h.chartbuilder.HelmClientImpl : + 2022-03-07 18:13:35.143 INFO 1 --- [nio-9090-exec-1] o.o.d.p.h.chartbuilder.HelmClientImpl : 1 chart(s) linted, 0 chart(s) failed + 2022-03-07 18:13:35.148 INFO 1 --- [nio-9090-exec-1] o.o.d.p.h.chartbuilder.HelmClientImpl : running: helm package -d /tmp/chart14389934160290252569 /tmp/chart7638328545634423550 + 2022-03-07 18:13:35.238 INFO 1 --- [nio-9090-exec-1] o.o.d.p.h.chartbuilder.HelmClientImpl : Successfully packaged chart and saved it to: /tmp/chart14389934160290252569/dcae-tcagen2-1.3.1.tgz + 2022-03-07 18:13:35.303 INFO 1 --- [nio-9090-exec-1] o.o.d.p.h.d.ChartMuseumDistributor : {"saved":true} + + +5. Validation & Deployment +============================= + +** Verify if the charts are pushed into registry** + + +Charts distributed by MOD/Runtime can be verified on Chartmuseum +registry http://chart-museum:80/api/charts + +Refer to supported api under `Chartmuseum Docs <https://chartmuseum.com/docs/>`__ + +Once the charts are retrieved, they can be installed using helm install command. + +:: + + curl -X GET http://<registry:port>/charts/dcae-tcagen2-1.3.1.tgz -u onapinitializer:demo123456! -o dcae-tcagen2-1.3.1.tgz + helm install -name dev-dcaegen2-services -n onap dcae-tcagen2-1.3.1.tgz --set global.masterPassword=test1 --set global.pullPolicy=Always --set mongo.enabled=true + + + +6. Environment Cleanup +----------------------- + +**Demo Env Cleanup** + +:: + + helm delete -n onap dev-chartmuseum # To remove Chartmuseum setup completely + helm delete -n onap dev-dcaegen2-services # To remove TCAGen2 services + helm delete -n onap dev-dcaemod # To undeploy DCAEMOD + + # USE DELETE METHOD ON CHARTMUSEUM TO REMOVE ANY SPECIFIC CHART PACKAGE - example below + curl -X DELETE http://<registry:port>/api/charts/dcae-ves-collector/1.10.1 -u onapinitializer:demo123456! + curl -X DELETE http://<registry:port>/api/charts/dcae-tcagen2/1.3.1 -u onapinitializer:demo123456! + +**Remove also any persistence directory from /dockerdata-nfs/onap/ associated to chartmuseum and dcaemod** -|image20| - -|image21| - -|image22| - -The generated Blueprint can be viewed. - -|image23| - -Finally, the generated Blueprint can be deployed. - -|image24| - -You can use/import the attached input configurations files to deploy. Drag and Drop these sample JSON files to fill in the configuration values. -See :download:`VES Collector Input Configuration <./Sample-Input-Files/ves-deploy.input.json>` -See :download:`Tcagen2 Input Configuration <./Sample-Input-Files/tca-deploy.input.json>` - -|image25| - -|image26| .. |image0| image:: ../images/1.png :width: 6.5in @@ -437,22 +632,5 @@ See :download:`Tcagen2 Input Configuration <./Sample-Input-Files/tca-deploy.inpu .. |image19| image:: ../images/20.png :width: 4.91667in :height: 2.41667in -.. |image20| image:: ../images/21.png - :width: 6.5in - :height: 2.41667in -.. |image21| image:: ../images/22.png - :width: 6.5in - :height: 3in -.. |image22| image:: ../images/23.png - :width: 6.5in - :height: 2.16667in -.. |image23| image:: ../images/24.png - :width: 6.5in - :height: 2.83333in -.. |image24| image:: ../images/25.png - :width: 6.5in - :height: 3in -.. |image25| image:: ../images/26.png -.. |image26| image:: ../images/27.png diff --git a/docs/sections/design-components/DCAE-MOD/DCAE-MOD-goals.rst b/docs/sections/design-components/DCAE-MOD/DCAE-MOD-goals.rst index 23d393b1..a0521cca 100644 --- a/docs/sections/design-components/DCAE-MOD/DCAE-MOD-goals.rst +++ b/docs/sections/design-components/DCAE-MOD/DCAE-MOD-goals.rst @@ -1,3 +1,5 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 ============== MOD Objectives ============== @@ -37,7 +39,6 @@ an effort to reboot the onboarding and design experience in DCAE. - Support automated adaptation of ML model from Acumos to DCAE design & runtime environment through the Acumos Adapter. -- DCAE-MOD is developed by the DCAE team to ensure consistency across all DCAE implementation, with the long term objective to integrate with SDC as part of the Design Platform. +- DCAE-MOD is developed by the DCAE team to ensure consistency across all DCAE implementation, with the long term objective to integrate with SDC & Policy/CLAMP as part of the Design Platform. -- Integrate with ONAP User Experience portals (initially ONAP portal, later SDC portal). diff --git a/docs/sections/design-components/DCAE-MOD/Sample-Input-Files/Request-body-of-Sample-Component_v3.json b/docs/sections/design-components/DCAE-MOD/Sample-Input-Files/Request-body-of-Sample-Component_v3.json new file mode 100644 index 00000000..ebbf3f64 --- /dev/null +++ b/docs/sections/design-components/DCAE-MOD/Sample-Input-Files/Request-body-of-Sample-Component_v3.json @@ -0,0 +1,412 @@ + { + "spec": + +{ + "self": { + "version": "1.10.1", + "name": "dcae-ves-collector", + "description": "Collector for receiving VES events through restful interface", + "component_type": "docker" + }, + "streams": { + "subscribes": [], + "publishes": [{ + "format": "VES_specification", + "version": "5.28.4", + "type": "message router", + "config_key": "ves-fault" + }, + { + "format": "VES_specification", + "version": "5.28.4", + "type": "message router", + "config_key": "ves-measurement" + }, + { + "format": "VES_specification", + "version": "5.28.4", + "type": "message router", + "config_key": "ves-syslog" + }, + { + "format": "VES_specification", + "version": "5.28.4", + "type": "message router", + "config_key": "ves-heartbeat" + }, + { + "format": "VES_specification", + "version": "7.30.2", + "type": "message router", + "config_key": "ves-other" + }, + { + "format": "VES_specification", + "version": "5.28.4", + "type": "message router", + "config_key": "ves-mobileflow" + }, + { + "format": "VES_specification", + "version": "5.28.4", + "type": "message router", + "config_key": "ves-statechange" + }, + { + "format": "VES_specification", + "version": "5.28.4", + "type": "message router", + "config_key": "ves-thresholdCrossingAlert" + }, + { + "format": "VES_specification", + "version": "5.28.4", + "type": "message router", + "config_key": "ves-voicequality" + }, + { + "format": "VES_specification", + "version": "5.28.4", + "type": "message router", + "config_key": "ves-sipsignaling" + }, + { + "format": "VES_specification", + "version": "7.30.2", + "type": "message router", + "config_key": "ves-pnfRegistration" + }, + { + "format": "VES_specification", + "version": "7.30.2", + "type": "message router", + "config_key": "ves-notification" + }, + { + "format": "VES_specification", + "version": "7.30.2", + "type": "message router", + "config_key": "ves-perf3gpp" + }, + { + "format": "VES_specification", + "version": "7.30.2", + "type": "message router", + "config_key": "ves-3gpp-fault-supervision" + }, + { + "format": "VES_specification", + "version": "7.30.2", + "type": "message router", + "config_key": "ves-3gpp-provisioning" + }, + { + "format": "VES_specification", + "version": "7.30.2", + "type": "message router", + "config_key": "ves-3gpp-heartbeat" + }, + { + "format": "VES_specification", + "version": "7.30.2", + "type": "message router", + "config_key": "ves-3gpp-performance-assurance" + } + ] + }, + "services": { + "calls": [], + "provides": [{ + "route": "/eventListener/v1", + "verb": "POST", + "request": { + "format": "VES_specification", + "version": "4.27.2" + }, + "response": { + "format": "ves.coll.response", + "version": "1.0.0" + } + }, + { + "route": "/eventListener/v2", + "verb": "POST", + "request": { + "format": "VES_specification", + "version": "4.27.2" + }, + "response": { + "format": "ves.coll.response", + "version": "1.0.0" + } + }, + { + "route": "/eventListener/v3", + "verb": "POST", + "request": { + "format": "VES_specification", + "version": "4.27.2" + }, + "response": { + "format": "ves.coll.response", + "version": "1.0.0" + } + }, + { + "route": "/eventListener/v4", + "verb": "POST", + "request": { + "format": "VES_specification", + "version": "4.27.2" + }, + "response": { + "format": "ves.coll.response", + "version": "1.0.0" + } + }, + { + "route": "/eventListener/v5", + "verb": "POST", + "request": { + "format": "VES_specification", + "version": "5.28.4" + }, + "response": { + "format": "ves.coll.response", + "version": "1.0.0" + } + }, + { + "route": "/eventListener/v7", + "verb": "POST", + "request": { + "format": "VES_specification", + "version": "7.30.2" + }, + "response": { + "format": "ves.coll.response", + "version": "1.0.0" + } + } + ] + }, + "parameters": [{ + "name": "streams_publishes", + "value": "{\"ves-3gpp-fault-supervision\":{\"dmaap_info\":{\"topic_url\":\"http:\/\/message-router:3904\/events\/unauthenticated.SEC_3GPP_FAULTSUPERVISION_OUTPUT\"},\"type\":\"message_router\"},\"ves-3gpp-heartbeat\":{\"dmaap_info\":{\"topic_url\":\"http:\/\/message-router:3904\/events\/unauthenticated.SEC_3GPP_HEARTBEAT_OUTPUT\"},\"type\":\"message_router\"},\"ves-3gpp-performance-assurance\":{\"dmaap_info\":{\"topic_url\":\"http:\/\/message-router:3904\/events\/unauthenticated.SEC_3GPP_PERFORMANCEASSURANCE_OUTPUT\"},\"type\":\"message_router\"},\"ves-3gpp-provisioning\":{\"dmaap_info\":{\"topic_url\":\"http:\/\/message-router:3904\/events\/unauthenticated.SEC_3GPP_PROVISIONING_OUTPUT\"},\"type\":\"message_router\"},\"ves-fault\":{\"dmaap_info\":{\"topic_url\":\"http:\/\/message-router:3904\/events\/unauthenticated.SEC_FAULT_OUTPUT\"},\"type\":\"message_router\"},\"ves-heartbeat\":{\"dmaap_info\":{\"topic_url\":\"http:\/\/message-router:3904\/events\/unauthenticated.SEC_HEARTBEAT_OUTPUT\"},\"type\":\"message_router\"},\"ves-measurement\":{\"dmaap_info\":{\"topic_url\":\"http:\/\/message-router:3904\/events\/unauthenticated.VES_MEASUREMENT_OUTPUT\"},\"type\":\"message_router\"},\"ves-notification\":{\"dmaap_info\":{\"topic_url\":\"http:\/\/message-router:3904\/events\/unauthenticated.VES_NOTIFICATION_OUTPUT\"},\"type\":\"message_router\"},\"ves-other\":{\"dmaap_info\":{\"topic_url\":\"http:\/\/message-router:3904\/events\/unauthenticated.SEC_OTHER_OUTPUT\"},\"type\":\"message_router\"},\"ves-pnfRegistration\":{\"dmaap_info\":{\"topic_url\":\"http:\/\/message-router:3904\/events\/unauthenticated.VES_PNFREG_OUTPUT\"},\"type\":\"message_router\"}}", + "description": "standard http port collector will open for listening;", + "sourced_at_deployment": false, + "policy_editable": false, + "designer_editable": false + }, + { + "name": "collector.service.port", + "value": 8080, + "description": "standard http port collector will open for listening;", + "sourced_at_deployment": false, + "policy_editable": false, + "designer_editable": false + }, + { + "name": "collector.service.secure.port", + "value": 8443, + "description": "secure http port collector will open for listening ", + "sourced_at_deployment": false, + "policy_editable": false, + "designer_editable": true + }, + { + "name": "collector.keystore.file.location", + "value": "/opt/app/dcae-certificate/cert.jks", + "description": "fs location of keystore file in vm", + "sourced_at_deployment": false, + "policy_editable": false, + "designer_editable": false + }, + { + "name": "collector.keystore.passwordfile", + "value": "/opt/app/dcae-certificate/jks.pass", + "description": "location of keystore password file in vm", + "sourced_at_deployment": false, + "policy_editable": false, + "designer_editable": false + }, + { + "name": "collector.truststore.file.location", + "value": "/opt/app/dcae-certificate/trust.jks", + "description": "fs location of truststore file in vm", + "sourced_at_deployment": false, + "policy_editable": false, + "designer_editable": false + }, + { + "name": "collector.truststore.passwordfile", + "value": "/opt/app/dcae-certificate/trust.pass", + "description": "location of truststore password file in vm", + "sourced_at_deployment": false, + "policy_editable": false, + "designer_editable": false + }, + { + "name": "collector.dmaap.streamid", + "value": "fault=ves-fault|syslog=ves-syslog|heartbeat=ves-heartbeat|measurementsForVfScaling=ves-measurement|mobileFlow=ves-mobileflow|other=ves-other|stateChange=ves-statechange|thresholdCrossingAlert=ves-thresholdCrossingAlert|voiceQuality=ves-voicequality|sipSignaling=ves-sipsignaling|notification=ves-notification|pnfRegistration=ves-pnfRegistration|3GPP-FaultSupervision=ves-3gpp-fault-supervision|3GPP-Heartbeat=ves-3gpp-heartbeat|3GPP-Provisioning=ves-3gpp-provisioning|3GPP-PerformanceAssurance=ves-3gpp-performance-assurance", + "description": "domain-to-streamid mapping used by VESCollector to distributes events based on domain. Both primary and secondary config_key are included for resilency (multiple streamid can be included commma separated). The streamids MUST match to topic config_keys. For single site without resiliency deployment - configkeys with -secondary suffix can be removed", + "sourced_at_deployment": true, + "policy_editable": false, + "designer_editable": false + }, + { + "name": "auth.method", + "value": "noAuth", + "description": "Property to manage application mode, possible configurations: noAuth - default option - no security (http) , certOnly - auth by certificate (https), basicAuth - auth by basic auth username and password (https),certBasicAuth - auth by certificate and basic auth username / password (https),", + "sourced_at_deployment": false, + "policy_editable": false, + "designer_editable": false + }, + { + "name": "header.authlist", + "value": "sample1,$2a$10$pgjaxDzSuc6XVFEeqvxQ5u90DKJnM/u7TJTcinAlFJVaavXMWf/Zi|userid1,$2a$10$61gNubgJJl9lh3nvQvY9X.x4e5ETWJJ7ao7ZhJEvmfJigov26Z6uq|userid2,$2a$10$G52y/3uhuhWAMy.bx9Se8uzWinmbJa.dlm1LW6bYPdPkkywLDPLiy", + "description": "List of id and base 64 encoded password.For each onboarding VNF - unique userid and password should be assigned and communicated to VNF owner. Password value should be base64 encoded in config here", + "policy_editable": false, + "sourced_at_deployment": true, + "designer_editable": true + }, + { + "name": "collector.schema.checkflag", + "value": 1, + "description": "Schema check validation flag. When enabled, collector will validate input VES events against VES Schema defined on collector.schema.file ", + "sourced_at_deployment": false, + "policy_editable": false, + "designer_editable": false + }, + { + "name": "collector.schema.file", + "value": "{\"v1\":\"./etc/CommonEventFormat_27.2.json\",\"v2\":\"./etc/CommonEventFormat_27.2.json\",\"v3\":\"./etc/CommonEventFormat_27.2.json\",\"v4\":\"./etc/CommonEventFormat_27.2.json\",\"v5\":\"./etc/CommonEventFormat_28.4.1.json\",\"v7\":\"./etc/CommonEventFormat_30.2.1_ONAP.json\"}", + "description": "VES schema file name per version used for validation", + "designer_editable": true, + "sourced_at_deployment": false, + "policy_editable": false + }, + { + "name": "event.transform.flag", + "value": 1, + "description": "flag to enable tranformation rules defined under eventTransform.json; this is applicable when event tranformation rules preset should be activated for transforming <VES5.4 events to 5.4", + "sourced_at_deployment": false, + "policy_editable": false, + "designer_editable": false + }, + { + "name": "tomcat.maxthreads", + "value": "200", + "description": "Tomcat control for concurrent request", + "sourced_at_deployment": false, + "policy_editable": false, + "designer_editable": false + }, + { + "name": "collector.externalSchema.checkflag", + "value": 1, + "description": "Schema stndDefined validation flag. When enabled, collector will validate stndDefined fields in stndDefined domain events against mapped local schemas listed in file from property collector.externalSchema.mappingFileLocation.", + "sourced_at_deployment": false, + "policy_editable": false, + "designer_editable": true + }, + { + "name": "collector.externalSchema.schemasLocation", + "value": "./etc/externalRepo/", + "description": "External schemas repository. Path to schemas storage directory.", + "sourced_at_deployment": false, + "policy_editable": false, + "designer_editable": false + }, + { + "name": "collector.externalSchema.mappingFileLocation", + "value": "./etc/externalRepo/schema-map.json", + "description": "Path to JSON file containing mapping of externally located stndDefined schemas to local schema files.", + "sourced_at_deployment": false, + "policy_editable": false, + "designer_editable": false + }, + { + "name": "event.externalSchema.schemaRefPath", + "value": "$.event.stndDefinedFields.schemaReference", + "description": "An internal path from validated JSON. Defines which field is taken as public schema reference, which is later mapped.", + "sourced_at_deployment": false, + "policy_editable": false, + "designer_editable": false + }, + { + "name": "event.externalSchema.stndDefinedDataPath", + "value": "$.event.stndDefinedFields.data", + "description": "An internal path from validated JSON. Defines which field of event will be validated during stndDefined validation.", + "sourced_at_deployment": false, + "policy_editable": false, + "designer_editable": false + }, + { + "name": "collector.description.api.version.location", + "value": "etc/api_version_description.json", + "description": "Path to the file containing description of api versions", + "sourced_at_deployment": false, + "policy_editable": false, + "designer_editable": false + } + ], + "auxilary": { + "helm": { + "service": { + "type": "NodePort", + "name": "dcae-ves-collector", + "has_internal_only_ports": true, + "ports": [{ + "name": "http", + "port": 8443, + "plain_port": 8080, + "port_protocol": "http", + "nodePort": 17, + "useNodePortExt": true + }] + } + }, + "healthcheck": { + "type": "HTTP", + "interval": "15s", + "timeout": "1s", + "port": 8080, + "endpoint": "/healthcheck" + }, + "volumes": [{ + "config_volume": { + "name": "dcae-external-repo-configmap-schema-map" + }, + "container": { + "bind": "/opt/app/VESCollector/etc/externalRepo/" + } + }, { + "config_volume": { + "name": "dcae-external-repo-configmap-sa88-rel16" + }, + "container": { + "bind": "/opt/app/VESCollector/etc/externalRepo/3gpp/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/" + } + }], + "ports": [ + "8080:0", + "8443:0" + ], + "log_info": { + "log_directory": "/opt/app/VESCollector/logs/" + }, + "tls_info": { + "cert_directory": "/opt/app/dcae-certificate/", + "use_tls": true, + "use_external_tls": false + } + }, + "artifacts": [{ + "type": "docker image", + "uri": "onap/org.onap.dcaegen2.collectors.ves.vescollector:1.10.1" + }] +} +, + "owner": "onboard_dev" + } diff --git a/docs/sections/design-components/component-specification/component-json-schema.rst b/docs/sections/design-components/component-specification/component-json-schema.rst index 1d2936b0..a6afcd41 100644 --- a/docs/sections/design-components/component-specification/component-json-schema.rst +++ b/docs/sections/design-components/component-specification/component-json-schema.rst @@ -6,1001 +6,1171 @@ Component JSON Schema Definition ================================ -The schema file used for DCAE onboarding is maintained in `gerrit <https://git.onap.org/dcaegen2/platform/plain/mod/component-json-schemas/component-specification/dcae-cli-v2/component-spec-schema.json>`__ +The schema file used for DCAE onboarding is maintained in `gerrit <https://git.onap.org/dcaegen2/platform/plain/mod/component-json-schemas/component-specification/dcae-cli-v3/component-spec-schema.json>`__ The same is provided below for documentation reference. :: { - "$schema": "http://json-schema.org/draft-04/schema#", - "title": "Component specification schema", - "type": "object", - "properties": { - "self": { - "type": "object", - "properties": { - "version": { - "$ref": "#/definitions/version" + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "Component specification schema", + "type": "object", + "properties": { + "self": { + "type": "object", + "properties": { + "version": { + "$ref": "#/definitions/version" + }, + "description": { + "type": "string" + }, + "component_type": { + "type": "string", + "enum": [ + "docker", + "cdap" + ] + }, + "name": { + "$ref": "#/definitions/name" + } + }, + "required": [ + "version", + "name", + "description", + "component_type" + ] }, - "description": { - "type": "string" + "streams": { + "type": "object", + "properties": { + "publishes": { + "type": "array", + "uniqueItems": true, + "items": { + "oneOf": [ + { + "$ref": "#/definitions/publisher_http" + }, + { + "$ref": "#/definitions/publisher_message_router" + }, + { + "$ref": "#/definitions/publisher_data_router" + }, + { + "$ref": "#/definitions/publisher_kafka" + } + ] + } + }, + "subscribes": { + "type": "array", + "uniqueItems": true, + "items": { + "oneOf": [ + { + "$ref": "#/definitions/subscriber_http" + }, + { + "$ref": "#/definitions/subscriber_message_router" + }, + { + "$ref": "#/definitions/subscriber_data_router" + }, + { + "$ref": "#/definitions/subscriber_kafka" + } + ] + } + } + }, + "required": [ + "publishes", + "subscribes" + ] }, - "component_type": { - "type": "string", - "enum": [ - "docker", - "cdap" - ] + "services": { + "type": "object", + "properties": { + "calls": { + "type": "array", + "uniqueItems": true, + "items": { + "$ref": "#/definitions/caller" + } + }, + "provides": { + "type": "array", + "uniqueItems": true, + "items": { + "$ref": "#/definitions/provider" + } + } + }, + "required": [ + "calls", + "provides" + ] }, - "name": { - "$ref": "#/definitions/name" - } - }, - "required": [ - "version", - "name", - "description", - "component_type" - ] - }, - "streams": { - "type": "object", - "properties": { - "publishes": { - "type": "array", - "uniqueItems": true, - "items": { - "oneOf": [ - { "$ref": "#/definitions/publisher_http" }, - { "$ref": "#/definitions/publisher_message_router" }, - { "$ref": "#/definitions/publisher_data_router" }, - { "$ref": "#/definitions/publisher_kafka" } + "parameters": { + "anyOf": [ + { + "$ref": "#/definitions/docker-parameters" + }, + { + "$ref": "#/definitions/cdap-parameters" + } ] - } }, - "subscribes": { - "type": "array", - "uniqueItems": true, - "items": { + "auxilary": { "oneOf": [ - { "$ref": "#/definitions/subscriber_http" }, - { "$ref": "#/definitions/subscriber_message_router" }, - { "$ref": "#/definitions/subscriber_data_router" }, - { "$ref": "#/definitions/subscriber_kafka" } + { + "$ref": "#/definitions/auxilary_cdap" + }, + { + "$ref": "#/definitions/auxilary_docker" + } ] - } - } - }, - "required": [ - "publishes", - "subscribes" - ] - }, - "services": { - "type": "object", - "properties": { - "calls": { - "type": "array", - "uniqueItems": true, - "items": { - "$ref": "#/definitions/caller" - } - }, - "provides": { - "type": "array", - "uniqueItems": true, - "items": { - "$ref": "#/definitions/provider" - } - } - }, - "required": [ - "calls", - "provides" - ] - }, - "parameters" : { - "anyOf" : [ - {"$ref": "#/definitions/docker-parameters"}, - {"$ref": "#/definitions/cdap-parameters"} - ] - }, - "auxilary": { - "oneOf" : [ - {"$ref": "#/definitions/auxilary_cdap"}, - {"$ref": "#/definitions/auxilary_docker"} - ] - }, - "artifacts": { - "type": "array", - "description": "List of component artifacts", - "items": { - "$ref": "#/definitions/artifact" - } - }, - "policy_info": { - "type": "object", - "properties": { - "policy": - { - "type": "array", - "items": - { + }, + "artifacts": { + "type": "array", + "description": "List of component artifacts", + "items": { + "$ref": "#/definitions/artifact" + } + }, + "policy_info": { "type": "object", - "properties": - { - "node_label": - { - "type": "string" - }, - "policy_id": - { - "type": "string" - }, - "policy_model_id": - { - "type": "string" - } + "properties": { + "policy": { + "type": "array", + "items": { + "type": "object", + "properties": { + "node_label": { + "type": "string" + }, + "policy_id": { + "type": "string" + }, + "policy_model_id": { + "type": "string" + } + }, + "required": [ + "policy_id", + "policy_model_id" + ] + } + } }, - "required": ["node_label", "policy_model_id"] - } + "additionalProperties": false } - }, - "additionalProperties": false - } - }, - "required": [ - "self", - "streams", - "services", - "parameters", - "auxilary", - "artifacts" - ], - "additionalProperties": false, - "definitions": { - "cdap-parameters": { - "description" : "There are three seperate ways to pass parameters to CDAP: app config, app preferences, program preferences. These are all treated as optional.", - "type": "object", - "properties" : { - "program_preferences": { - "description" : "A list of {program_id, program_type, program_preference} objects where program_preference is an object passed into program_id of type program_type", - "type": "array", - "uniqueItems": true, - "items": { - "$ref": "#/definitions/program_preference" - } - }, - "app_preferences" : { - "description" : "Parameters Passed down to the CDAP preference API", - "type": "array", - "uniqueItems": true, - "items": { - "$ref": "#/definitions/parameter" - } - }, - "app_config" : { - "description" : "Parameters Passed down to the CDAP App Config", - "type": "array", - "uniqueItems": true, - "items": { - "$ref": "#/definitions/parameter" - } - } - } - }, - "program_preference": { - "type": "object", - "properties": { - "program_type": { - "$ref": "#/definitions/program_type" - }, - "program_id": { - "type": "string" - }, - "program_pref":{ - "description" : "Parameters that the CDAP developer wants pushed to this program's preferences API. Optional", - "type": "array", - "uniqueItems": true, - "items": { - "$ref": "#/definitions/parameter" - } - } - }, - "required": ["program_type", "program_id", "program_pref"] - }, - "program_type": { - "type": "string", - "enum": ["flows","mapreduce","schedules","spark","workflows","workers","services"] }, - "docker-parameters": { - "type": "array", - "uniqueItems": true, - "items": { - "$ref": "#/definitions/parameter" - } - }, - "parameter": { - "oneOf": [ - {"$ref": "#/definitions/parameter-list"}, - {"$ref": "#/definitions/parameter-other"} - ] - }, - "parameter-list": { - "properties": { - "name": { - "type": "string" - }, - "value": { - "description": "Default value for the parameter" - }, - "description": { - "description": "Description for the parameter.", - "type": "string" - }, - "type": { - "description": "Only valid type is list, the entry_schema is required - which contains the type of the list element. All properties set for the parameter apply to all elements in the list at this time", - "type": "string", - "enum": ["list"] - }, - "required": { - "description": "An optional key that declares a parameter as required (true) or not (false). Default is true.", - "type": "boolean", - "default": true - }, - "constraints": { - "description": "The optional list of sequenced constraint clauses for the parameter.", - "type": "array", - "items": { - "$ref": "#/definitions/parameter-constraints" - } - }, - "entry_schema": { - "description": "The optional property used to declare the name of the Datatype definition for entries of certain types. entry_schema must be defined when the type is list. This is the only type it is currently supported for.", - "type": "object", - "uniqueItems": true, - "items": {"$ref": "#/definitions/list-parameter"} - }, - "designer_editable": { - "description": "A required property that declares a parameter as editable by designer in SDC Tool (true) or not (false).", - "type": "boolean" - }, - "sourced_at_deployment": { - "description": "A required property that declares that a parameter is assigned at deployment time (true) or not (false).", - "type": "boolean" - }, - "policy_editable": { - "description": "A required property that declares a parameter as editable by DevOps in Policy UI (true) or not (false).", - "type": "boolean" - }, - "policy_group": { - "description": "An optional property used to group policy_editable parameters into groups. Each group will become it's own policy model. Any parameters without this property will be grouped together to form their own policy model", - "type": "string" - }, - "policy_schema" :{ - "type": "array", - "uniqueItems": true, - "items": {"$ref": "#/definitions/policy_schema_parameter"} - } - }, - "required": [ - "name", - "value", - "description", - "designer_editable", - "policy_editable", - "sourced_at_deployment", - "entry_schema" - ], - "additionalProperties": false, - "dependencies": { - "policy_schema": ["policy_editable"] - } - }, - "parameter-other": { - "properties": { - "name": { - "type": "string" - }, - "value": { - "description": "Default value for the parameter" - }, - "description": { - "description": "Description for the parameter.", - "type": "string" - }, - "type": { - "description": "The required data type for the parameter.", - "type": "string", - "enum": [ "string", "number", "boolean", "datetime" ] - }, - "required": { - "description": "An optional key that declares a parameter as required (true) or not (false). Default is true.", - "type": "boolean", - "default": true - }, - "constraints": { - "description": "The optional list of sequenced constraint clauses for the parameter.", - "type": "array", - "items": { - "$ref": "#/definitions/parameter-constraints" - } - }, - "designer_editable": { - "description": "A required property that declares a parameter as editable by designer in SDC Tool (true) or not (false).", - "type": "boolean" - }, - "sourced_at_deployment": { - "description": "A required property that declares that a parameter is assigned at deployment time (true) or not (false).", - "type": "boolean" - }, - "policy_editable": { - "description": "A required property that declares a parameter as editable in Policy UI (true) or not (false).", - "type": "boolean" - }, - "policy_group": { - "description": "An optional property used to group policy_editable parameters into groups. Each group will become it's own policy model. Any parameters without this property will be grouped together to form their own policy model", - "type": "string" - }, - "policy_schema" :{ - "description": "An optional property used to define policy_editable parameters as lists or maps", - "type": "array", - "uniqueItems": true, - "items": {"$ref": "#/definitions/policy_schema_parameter"} - } - }, - "required": [ - "name", - "value", - "description", - "designer_editable", - "sourced_at_deployment", - "policy_editable" - ], - "additionalProperties": false, - "dependencies": { - "policy_schema": ["policy_editable"] - } - }, - "list-parameter": { - "type": "object", - "properties": { - "type": { - "description": "The required data type for each parameter in the list.", - "type": "string", - "enum": ["string", "number"] - } - }, - "required": [ - "type" - ], - "additionalProperties": false - }, - "policy_schema_parameter": { - "type": "object", - "properties": { - "name": { - "type": "string" - }, - "value": { - "description": "Default value for the parameter" - }, - "description": { - "description": "Description for the parameter.", - "type": "string" - }, - "type": { - "description": "The required data type for the parameter.", - "type": "string", - "enum": [ "string", "number", "boolean", "datetime", "list", "map" ] - }, - "required": { - "description": "An optional key that declares a parameter as required (true) or not (false). Default is true.", - "type": "boolean", - "default": true - }, - "constraints": { - "description": "The optional list of sequenced constraint clauses for the parameter.", - "type": "array", - "items": { - "$ref": "#/definitions/parameter-constraints" - } - }, - "entry_schema": { - "description": "The optional key that is used to declare the name of the Datatype definition for entries of certain types. entry_schema must be defined when the type is either list or map. If the type is list and the entry type is a simple type (string, number, boolean, datetime), follow with a simple string to describe the entry type. If the type is list and the entry type is a map, follow with an array to describe the keys for the entry map. If the type is list and the entry type is also list, this is not currently supported here. If the type is map, then follow with an array to describe the keys for this map. ", - "type": "array", "uniqueItems": true, "items": {"$ref": "#/definitions/policy_schema_parameter"} - } - }, - "required": [ - "name", - "type" - ], - "additionalProperties": false - }, - "parameter-constraints": { - "type": "object", - "additionalProperties": false, - "properties": { - "equal": { - "description": "Constrains a property or parameter to a value equal to (‘=’) the value declared." - }, - "greater_than": { - "description": "Constrains a property or parameter to a value greater than (‘>’) the value declared.", - "type": "number" - }, - "greater_or_equal": { - "description": "Constrains a property or parameter to a value greater than or equal to (‘>=’) the value declared.", - "type": "number" - }, - "less_than": { - "description": "Constrains a property or parameter to a value less than (‘<’) the value declared.", - "type": "number" - }, - "less_or_equal": { - "description": "Constrains a property or parameter to a value less than or equal to (‘<=’) the value declared.", - "type": "number" - }, - "valid_values": { - "description": "Constrains a property or parameter to a value that is in the list of declared values.", - "type": "array" - }, - "length": { - "description": "Constrains the property or parameter to a value of a given length.", - "type": "number" - }, - "min_length": { - "description": "Constrains the property or parameter to a value to a minimum length.", - "type": "number" - }, - "max_length": { - "description": "Constrains the property or parameter to a value to a maximum length.", - "type": "number" - } - } - }, - "stream_message_router": { - "type": "object", - "properties": { - "format": { - "$ref": "#/definitions/name" + "required": [ + "self", + "streams", + "services", + "parameters", + "auxilary", + "artifacts" + ], + "additionalProperties": false, + "definitions": { + "cdap-parameters": { + "description": "There are three seperate ways to pass parameters to CDAP: app config, app preferences, program preferences. These are all treated as optional.", + "type": "object", + "properties": { + "program_preferences": { + "description": "A list of {program_id, program_type, program_preference} objects where program_preference is an object passed into program_id of type program_type", + "type": "array", + "uniqueItems": true, + "items": { + "$ref": "#/definitions/program_preference" + } + }, + "app_preferences": { + "description": "Parameters Passed down to the CDAP preference API", + "type": "array", + "uniqueItems": true, + "items": { + "$ref": "#/definitions/parameter" + } + }, + "app_config": { + "description": "Parameters Passed down to the CDAP App Config", + "type": "array", + "uniqueItems": true, + "items": { + "$ref": "#/definitions/parameter" + } + } + } }, - "version": { - "$ref": "#/definitions/version" + "program_preference": { + "type": "object", + "properties": { + "program_type": { + "$ref": "#/definitions/program_type" + }, + "program_id": { + "type": "string" + }, + "program_pref": { + "description": "Parameters that the CDAP developer wants pushed to this program's preferences API. Optional", + "type": "array", + "uniqueItems": true, + "items": { + "$ref": "#/definitions/parameter" + } + } + }, + "required": [ + "program_type", + "program_id", + "program_pref" + ] }, - "config_key": { - "type": "string" + "program_type": { + "type": "string", + "enum": [ + "flows", + "mapreduce", + "schedules", + "spark", + "workflows", + "workers", + "services" + ] }, - "type": { - "description": "Type of stream to be used", - "type": "string", - "enum": [ - "message router", "message_router" - ] - } - }, - "required": [ - "format", - "version", - "config_key", - "type" - ] - }, - "stream_kafka": { - "type": "object", - "properties": { - "format": { - "$ref": "#/definitions/name" + "docker-parameters": { + "type": "array", + "uniqueItems": true, + "items": { + "$ref": "#/definitions/parameter" + } }, - "version": { - "$ref": "#/definitions/version" + "parameter": { + "oneOf": [ + { + "$ref": "#/definitions/parameter-list" + }, + { + "$ref": "#/definitions/parameter-other" + } + ] }, - "config_key": { - "type": "string" + "parameter-list": { + "properties": { + "name": { + "type": "string" + }, + "value": { + "description": "Default value for the parameter" + }, + "description": { + "description": "Description for the parameter.", + "type": "string" + }, + "type": { + "description": "Only valid type is list, the entry_schema is required - which contains the type of the list element. All properties set for the parameter apply to all elements in the list at this time", + "type": "string", + "enum": [ + "list" + ] + }, + "required": { + "description": "An optional key that declares a parameter as required (true) or not (false). Default is true.", + "type": "boolean", + "default": true + }, + "constraints": { + "description": "The optional list of sequenced constraint clauses for the parameter.", + "type": "array", + "items": { + "$ref": "#/definitions/parameter-constraints" + } + }, + "entry_schema": { + "description": "The optional property used to declare the name of the Datatype definition for entries of certain types. entry_schema must be defined when the type is list. This is the only type it is currently supported for.", + "type": "object", + "uniqueItems": true, + "items": { + "$ref": "#/definitions/list-parameter" + } + }, + "designer_editable": { + "description": "A required property that declares a parameter as editable by designer in SDC Tool (true) or not (false).", + "type": "boolean" + }, + "sourced_at_deployment": { + "description": "A required property that declares that a parameter is assigned at deployment time (true) or not (false).", + "type": "boolean" + }, + "policy_editable": { + "description": "A required property that declares a parameter as editable by DevOps in Policy UI (true) or not (false).", + "type": "boolean" + }, + "policy_group": { + "description": "An optional property used to group policy_editable parameters into groups. Each group will become it's own policy model. Any parameters without this property will be grouped together to form their own policy model", + "type": "string" + }, + "policy_schema": { + "type": "array", + "uniqueItems": true, + "items": { + "$ref": "#/definitions/policy_schema_parameter" + } + } + }, + "required": [ + "name", + "value", + "description", + "designer_editable", + "policy_editable", + "sourced_at_deployment", + "entry_schema" + ], + "additionalProperties": false, + "dependencies": { + "policy_schema": [ + "policy_editable" + ] + } }, - "type": { - "description": "Type of stream to be used", - "type": "string", - "enum": [ - "kafka" - ] - } - }, - "required": [ - "format", - "version", - "config_key", - "type" - ] - }, - "publisher_http": { - "type": "object", - "properties": { - "format": { - "$ref": "#/definitions/name" + "parameter-other": { + "properties": { + "name": { + "type": "string" + }, + "value": { + "description": "Default value for the parameter" + }, + "description": { + "description": "Description for the parameter.", + "type": "string" + }, + "type": { + "description": "The required data type for the parameter.", + "type": "string", + "enum": [ + "string", + "number", + "boolean", + "datetime" + ] + }, + "required": { + "description": "An optional key that declares a parameter as required (true) or not (false). Default is true.", + "type": "boolean", + "default": true + }, + "constraints": { + "description": "The optional list of sequenced constraint clauses for the parameter.", + "type": "array", + "items": { + "$ref": "#/definitions/parameter-constraints" + } + }, + "designer_editable": { + "description": "A required property that declares a parameter as editable by designer in SDC Tool (true) or not (false).", + "type": "boolean" + }, + "sourced_at_deployment": { + "description": "A required property that declares that a parameter is assigned at deployment time (true) or not (false).", + "type": "boolean" + }, + "policy_editable": { + "description": "A required property that declares a parameter as editable in Policy UI (true) or not (false).", + "type": "boolean" + }, + "policy_group": { + "description": "An optional property used to group policy_editable parameters into groups. Each group will become it's own policy model. Any parameters without this property will be grouped together to form their own policy model", + "type": "string" + }, + "policy_schema": { + "description": "An optional property used to define policy_editable parameters as lists or maps", + "type": "array", + "uniqueItems": true, + "items": { + "$ref": "#/definitions/policy_schema_parameter" + } + } + }, + "required": [ + "name", + "value", + "description", + "designer_editable", + "sourced_at_deployment", + "policy_editable" + ], + "additionalProperties": false, + "dependencies": { + "policy_schema": [ + "policy_editable" + ] + } }, - "version": { - "$ref": "#/definitions/version" - }, - "config_key": { - "type": "string" - }, - "type": { - "description": "Type of stream to be used", - "type": "string", - "enum": [ - "http", - "https" - ] - } - }, - "required": [ - "format", - "version", - "config_key", - "type" - ] - }, - "publisher_message_router": { - "$ref": "#/definitions/stream_message_router" - }, - "publisher_data_router": { - "type": "object", - "properties": { - "format": { - "$ref": "#/definitions/name" + "list-parameter": { + "type": "object", + "properties": { + "type": { + "description": "The required data type for each parameter in the list.", + "type": "string", + "enum": [ + "string", + "number" + ] + } + }, + "required": [ + "type" + ], + "additionalProperties": false }, - "version": { - "$ref": "#/definitions/version" + "policy_schema_parameter": { + "type": "object", + "properties": { + "name": { + "type": "string" + }, + "value": { + "description": "Default value for the parameter" + }, + "description": { + "description": "Description for the parameter.", + "type": "string" + }, + "type": { + "description": "The required data type for the parameter.", + "type": "string", + "enum": [ + "string", + "number", + "boolean", + "datetime", + "list", + "map" + ] + }, + "required": { + "description": "An optional key that declares a parameter as required (true) or not (false). Default is true.", + "type": "boolean", + "default": true + }, + "constraints": { + "description": "The optional list of sequenced constraint clauses for the parameter.", + "type": "array", + "items": { + "$ref": "#/definitions/parameter-constraints" + } + }, + "entry_schema": { + "description": "The optional key that is used to declare the name of the Datatype definition for entries of certain types. entry_schema must be defined when the type is either list or map. If the type is list and the entry type is a simple type (string, number, boolean, datetime), follow with a simple string to describe the entry type. If the type is list and the entry type is a map, follow with an array to describe the keys for the entry map. If the type is list and the entry type is also list, this is not currently supported here. If the type is map, then follow with an array to describe the keys for this map. ", + "type": "array", + "uniqueItems": true, + "items": { + "$ref": "#/definitions/policy_schema_parameter" + } + } + }, + "required": [ + "name", + "type" + ], + "additionalProperties": false }, - "config_key": { - "type": "string" + "parameter-constraints": { + "type": "object", + "additionalProperties": false, + "properties": { + "equal": { + "description": "Constrains a property or parameter to a value equal to (‘=’) the value declared." + }, + "greater_than": { + "description": "Constrains a property or parameter to a value greater than (‘>’) the value declared.", + "type": "number" + }, + "greater_or_equal": { + "description": "Constrains a property or parameter to a value greater than or equal to (‘>=’) the value declared.", + "type": "number" + }, + "less_than": { + "description": "Constrains a property or parameter to a value less than (‘<’) the value declared.", + "type": "number" + }, + "less_or_equal": { + "description": "Constrains a property or parameter to a value less than or equal to (‘<=’) the value declared.", + "type": "number" + }, + "valid_values": { + "description": "Constrains a property or parameter to a value that is in the list of declared values.", + "type": "array" + }, + "length": { + "description": "Constrains the property or parameter to a value of a given length.", + "type": "number" + }, + "min_length": { + "description": "Constrains the property or parameter to a value to a minimum length.", + "type": "number" + }, + "max_length": { + "description": "Constrains the property or parameter to a value to a maximum length.", + "type": "number" + } + } }, - "type": { - "description": "Type of stream to be used", - "type": "string", - "enum": [ - "data router", "data_router" - ] - } - }, - "required": [ - "format", - "version", - "config_key", - "type" - ] - }, - "publisher_kafka": { - "$ref": "#/definitions/stream_kafka" - }, - "subscriber_http": { - "type": "object", - "properties": { - "format": { - "$ref": "#/definitions/name" + "stream_message_router": { + "type": "object", + "properties": { + "format": { + "$ref": "#/definitions/name" + }, + "version": { + "$ref": "#/definitions/version" + }, + "config_key": { + "type": "string" + }, + "type": { + "description": "Type of stream to be used", + "type": "string", + "enum": [ + "message router", + "message_router" + ] + } + }, + "required": [ + "format", + "version", + "config_key", + "type" + ] }, - "version": { - "$ref": "#/definitions/version" - }, - "route": { - "type": "string" - }, - "type": { - "description": "Type of stream to be used", - "type": "string", - "enum": [ - "http", - "https" - ] - } - }, - "required": [ - "format", - "version", - "route", - "type" - ] - }, - "subscriber_message_router": { - "$ref": "#/definitions/stream_message_router" - }, - "subscriber_data_router": { - "type": "object", - "properties": { - "format": { - "$ref": "#/definitions/name" + "stream_kafka": { + "type": "object", + "properties": { + "format": { + "$ref": "#/definitions/name" + }, + "version": { + "$ref": "#/definitions/version" + }, + "config_key": { + "type": "string" + }, + "type": { + "description": "Type of stream to be used", + "type": "string", + "enum": [ + "kafka" + ] + } + }, + "required": [ + "format", + "version", + "config_key", + "type" + ] }, - "version": { - "$ref": "#/definitions/version" + "publisher_http": { + "type": "object", + "properties": { + "format": { + "$ref": "#/definitions/name" + }, + "version": { + "$ref": "#/definitions/version" + }, + "config_key": { + "type": "string" + }, + "type": { + "description": "Type of stream to be used", + "type": "string", + "enum": [ + "http", + "https" + ] + } + }, + "required": [ + "format", + "version", + "config_key", + "type" + ] }, - "route": { - "type": "string" + "publisher_message_router": { + "$ref": "#/definitions/stream_message_router" }, - "type": { - "description": "Type of stream to be used", - "type": "string", - "enum": [ - "data router", "data_router" - ] + "publisher_data_router": { + "type": "object", + "properties": { + "format": { + "$ref": "#/definitions/name" + }, + "version": { + "$ref": "#/definitions/version" + }, + "config_key": { + "type": "string" + }, + "type": { + "description": "Type of stream to be used", + "type": "string", + "enum": [ + "data router", + "data_router" + ] + } + }, + "required": [ + "format", + "version", + "config_key", + "type" + ] }, - "config_key": { - "description": "Data router subscribers require config info to setup their endpoints to handle requests. For example, needs username and password", - "type": "string" - } - }, - "required": [ - "format", - "version", - "route", - "type", - "config_key" - ] - }, - "subscriber_kafka": { - "$ref": "#/definitions/stream_kafka" - }, - "provider" : { - "oneOf" : [ - {"$ref": "#/definitions/docker-provider"}, - {"$ref": "#/definitions/cdap-provider"} - ] - }, - "cdap-provider" : { - "type": "object", - "properties" : { - "request": { - "$ref": "#/definitions/formatPair" + "publisher_kafka": { + "$ref": "#/definitions/stream_kafka" }, - "response": { - "$ref": "#/definitions/formatPair" + "subscriber_http": { + "type": "object", + "properties": { + "format": { + "$ref": "#/definitions/name" + }, + "version": { + "$ref": "#/definitions/version" + }, + "route": { + "type": "string" + }, + "type": { + "description": "Type of stream to be used", + "type": "string", + "enum": [ + "http", + "https" + ] + } + }, + "required": [ + "format", + "version", + "route", + "type" + ] }, - "service_name" : { - "type" : "string" + "subscriber_message_router": { + "$ref": "#/definitions/stream_message_router" }, - "service_endpoint" : { - "type" : "string" + "subscriber_data_router": { + "type": "object", + "properties": { + "format": { + "$ref": "#/definitions/name" + }, + "version": { + "$ref": "#/definitions/version" + }, + "route": { + "type": "string" + }, + "type": { + "description": "Type of stream to be used", + "type": "string", + "enum": [ + "data router", + "data_router" + ] + }, + "config_key": { + "description": "Data router subscribers require config info to setup their endpoints to handle requests. For example, needs username and password", + "type": "string" + } + }, + "required": [ + "format", + "version", + "route", + "type", + "config_key" + ] }, - "verb" : { - "type": "string", - "enum": ["GET", "PUT", "POST", "DELETE"] - } - }, - "required" : [ - "request", - "response", - "service_name", - "service_endpoint", - "verb" - ] - }, - "docker-provider": { - "type": "object", - "properties": { - "request": { - "$ref": "#/definitions/formatPair" + "subscriber_kafka": { + "$ref": "#/definitions/stream_kafka" }, - "response": { - "$ref": "#/definitions/formatPair" + "provider": { + "oneOf": [ + { + "$ref": "#/definitions/docker-provider" + }, + { + "$ref": "#/definitions/cdap-provider" + } + ] }, - "route": { - "type": "string" + "cdap-provider": { + "type": "object", + "properties": { + "request": { + "$ref": "#/definitions/formatPair" + }, + "response": { + "$ref": "#/definitions/formatPair" + }, + "service_name": { + "type": "string" + }, + "service_endpoint": { + "type": "string" + }, + "verb": { + "type": "string", + "enum": [ + "GET", + "PUT", + "POST", + "DELETE" + ] + } + }, + "required": [ + "request", + "response", + "service_name", + "service_endpoint", + "verb" + ] }, - "verb": { - "type": "string", - "enum": ["GET", "PUT", "POST", "DELETE"] - } - }, - "required": [ - "request", - "response", - "route" - ] - }, - "caller": { - "type": "object", - "properties": { - "request": { - "$ref": "#/definitions/formatPair" + "docker-provider": { + "type": "object", + "properties": { + "request": { + "$ref": "#/definitions/formatPair" + }, + "response": { + "$ref": "#/definitions/formatPair" + }, + "route": { + "type": "string" + }, + "verb": { + "type": "string", + "enum": [ + "GET", + "PUT", + "POST", + "DELETE" + ] + } + }, + "required": [ + "request", + "response", + "route" + ] }, - "response": { - "$ref": "#/definitions/formatPair" + "caller": { + "type": "object", + "properties": { + "request": { + "$ref": "#/definitions/formatPair" + }, + "response": { + "$ref": "#/definitions/formatPair" + }, + "config_key": { + "type": "string" + } + }, + "required": [ + "request", + "response", + "config_key" + ] }, - "config_key": { - "type": "string" - } - }, - "required": [ - "request", - "response", - "config_key" - ] - }, - "formatPair": { - "type": "object", - "properties": { - "format": { - "$ref": "#/definitions/name" + "formatPair": { + "type": "object", + "properties": { + "format": { + "$ref": "#/definitions/name" + }, + "version": { + "$ref": "#/definitions/version" + } + } + }, + "name": { + "type": "string" }, "version": { - "$ref": "#/definitions/version" - } - } - }, - "name": { - "type": "string" - }, - "version": { - "type": "string", - "pattern": "^(\\d+\\.)(\\d+\\.)(\\*|\\d+)$" - }, - "artifact": { - "type": "object", - "description": "Component artifact object", - "properties": { - "uri": { - "type": "string", - "description": "Uri to artifact" - }, - "type": { - "type": "string", - "enum": ["jar", "docker image"] - } - }, - "required": ["uri", "type"] - }, - - "auxilary_cdap": { - "title": "cdap component specification schema", - "type": "object", - "properties": { - "streamname": { - "type": "string" - }, - "artifact_name" : { - "type": "string" - }, - "artifact_version" : { - "type": "string", - "pattern": "^(\\d+\\.)(\\d+\\.)(\\*|\\d+)$" - }, - "namespace":{ - "type": "string", - "description" : "optional" - }, - "programs": { - "type": "array", - "uniqueItems": true, - "items": { - "$ref": "#/definitions/cdap_program" - } - } - }, - "required": [ - "streamname", - "programs", - "artifact_name", - "artifact_version" - ] - }, - "cdap_program_type": { - "type": "string", - "enum": ["flows","mapreduce","schedules","spark","workflows","workers","services"] - }, - "cdap_program": { - "type": "object", - "properties": { - "program_type": { - "$ref": "#/definitions/cdap_program_type" + "type": "string", + "pattern": "^(\\d+\\.)(\\d+\\.)(\\*|\\d+)$" }, - "program_id": { - "type": "string" - } - }, - "required": ["program_type", "program_id"] - }, - - "auxilary_docker": { - "title": "Docker component specification schema", - "type": "object", - "properties": { - "healthcheck": { - "description": "Define the health check that Consul should perfom for this component", - "type": "object", - "oneOf": [ - { "$ref": "#/definitions/docker_healthcheck_http" }, - { "$ref": "#/definitions/docker_healthcheck_script" } - ] - }, - "ports": { - "description": "Port mapping to be used for Docker containers. Each entry is of the format <container port>:<host port>.", - "type": "array", - "items": { - "type": "string" - } - }, - "log_info": { - "description": "Component specific details for logging", - "type": "object", - "properties": { - "log_directory": { - "description": "The path in the container where the component writes its logs. If the component is following the EELF requirements, this would be the directory where the four EELF files are being written. (Other logs can be placed in the directory--if their names in '.log', they'll also be sent into ELK.)", - "type": "string" - }, - "alternate_fb_path": { - "description": "By default, the log volume is mounted at /var/log/onap/<component_type> in the sidecar container's file system. 'alternate_fb_path' allows overriding the default. Will affect how the log data can be found in the ELK system.", - "type": "string" - } - }, - "additionalProperties": false - }, - "tls_info": { - "description": "Component information to use tls certificates", - "type": "object", - "properties": { - "cert_directory": { - "description": "The path in the container where the component certificates will be placed by the init container", - "type": "string" + "artifact": { + "type": "object", + "description": "Component artifact object", + "properties": { + "uri": { + "type": "string", + "description": "Uri to artifact" + }, + "type": { + "type": "string", + "enum": [ + "jar", + "docker image" + ] + } }, - "use_tls": { - "description": "Boolean flag to determine if the application is using tls certificates", - "type": "boolean" + "required": [ + "uri", + "type" + ] + }, + "auxilary_cdap": { + "title": "cdap component specification schema", + "type": "object", + "properties": { + "streamname": { + "type": "string" + }, + "artifact_name": { + "type": "string" + }, + "artifact_version": { + "type": "string", + "pattern": "^(\\d+\\.)(\\d+\\.)(\\*|\\d+)$" + }, + "namespace": { + "type": "string", + "description": "optional" + }, + "programs": { + "type": "array", + "uniqueItems": true, + "items": { + "$ref": "#/definitions/cdap_program" + } + } }, - "use_external_tls": { - "description": "Boolean flag to determine if the application is using tls certificates for external communication", - "type": "boolean" - } - }, - "required": [ - "cert_directory","use_tls" - ], - "additionalProperties": false - }, - "databases": { - "description": "The databases the application is connecting to using the pgaas", - "type": "object", - "additionalProperties": { + "required": [ + "streamname", + "programs", + "artifact_name", + "artifact_version" + ] + }, + "cdap_program_type": { "type": "string", "enum": [ - "postgres" + "flows", + "mapreduce", + "schedules", + "spark", + "workflows", + "workers", + "services" ] - } - }, - "policy": { - "properties": { - "trigger_type": { - "description": "Only value of docker is supported at this time.", - "type": "string", - "enum": ["docker"] + }, + "cdap_program": { + "type": "object", + "properties": { + "program_type": { + "$ref": "#/definitions/cdap_program_type" + }, + "program_id": { + "type": "string" + } }, - "script_path": { - "description": "Script command that will be executed for policy reconfiguration", - "type": "string" - } - }, - "required": [ - "trigger_type","script_path" - ], - "additionalProperties": false - }, - "volumes": { - "description": "Volume mapping to be used for Docker containers. Each entry is of the format below", - "type": "array", - "items": { + "required": [ + "program_type", + "program_id" + ] + }, + "auxilary_docker": { + "title": "Docker component specification schema", "type": "object", - "oneOf": [ - { "$ref": "#/definitions/host_path_volume" }, - { "$ref": "#/definitions/config_map_volume" } + "properties": { + "helm": { + "type": "object", + "properties": { + "applicationEnv": { + "type": "object" + }, + "service": { + "description": "Mapping for kubernetes services", + "type": "object", + "properties": { + "type": { + "type": "string", + "enum": [ + "NodePort", + "ClusterIP" + ] + }, + "name": { + "type": "string" + }, + "ports": { + "type": "array", + "items": { + "type": "object" + } + } + }, + "required": [ + "type", + "name", + "ports" + ] + } + }, + "required": [ + "service" + ] + }, + "healthcheck": { + "description": "Define the health check that Consul should perfom for this component", + "type": "object", + "oneOf": [ + { + "$ref": "#/definitions/docker_healthcheck_http" + }, + { + "$ref": "#/definitions/docker_healthcheck_script" + } + ] + }, + "ports": { + "description": "Port mapping to be used for Docker containers. Each entry is of the format <container port>:<host port>.", + "type": "array", + "items": { + "type": "string" + } + }, + "log_info": { + "description": "Component specific details for logging", + "type": "object", + "properties": { + "log_directory": { + "description": "The path in the container where the component writes its logs. If the component is following the EELF requirements, this would be the directory where the four EELF files are being written. (Other logs can be placed in the directory--if their names in '.log', they'll also be sent into ELK.)", + "type": "string" + }, + "alternate_fb_path": { + "description": "By default, the log volume is mounted at /var/log/onap/<component_type> in the sidecar container's file system. 'alternate_fb_path' allows overriding the default. Will affect how the log data can be found in the ELK system.", + "type": "string" + } + }, + "additionalProperties": false + }, + "tls_info": { + "description": "Component information to use tls certificates", + "type": "object", + "properties": { + "cert_directory": { + "description": "The path in the container where the component certificates will be placed by the init container", + "type": "string" + }, + "use_tls": { + "description": "Boolean flag to determine if the application is using tls certificates", + "type": "boolean" + }, + "use_external_tls": { + "description": "Boolean flag to determine if the application is using tls certificates for external communication", + "type": "boolean" + } + }, + "required": [ + "cert_directory", + "use_tls" + ], + "additionalProperties": false + }, + "databases": { + "description": "The databases the application is connecting to using the pgaas", + "type": "object", + "additionalProperties": { + "type": "string", + "enum": [ + "postgres" + ] + } + }, + "policy": { + "properties": { + "trigger_type": { + "description": "Only value of docker is supported at this time.", + "type": "string", + "enum": [ + "docker" + ] + }, + "script_path": { + "description": "Script command that will be executed for policy reconfiguration", + "type": "string" + } + }, + "required": [ + "trigger_type", + "script_path" + ], + "additionalProperties": false + }, + "volumes": { + "description": "Volume mapping to be used for Docker containers. Each entry is of the format below", + "type": "array", + "items": { + "type": "object", + "oneOf": [ + { + "$ref": "#/definitions/host_path_volume" + }, + { + "$ref": "#/definitions/config_map_volume" + } + ] + } + } + }, + "required": [ + "healthcheck" + ], + "additionalProperties": false + }, + "host_path_volume": { + "type": "object", + "properties": { + "host": { + "type": "object", + "path": { + "type": "string" + } + }, + "container": { + "type": "object", + "bind": { + "type": "string" + }, + "mode": { + "type": "string" + } + } + }, + "required": [ + "host", + "container" ] - } - } - }, - "required": [ - "healthcheck" - ], - "additionalProperties": false - }, - "host_path_volume": { - "type": "object", - "properties": { - "host": { - "type": "object", - "path": { - "type": "string" - } }, - "container": { - "type": "object", - "bind": { - "type": "string" - }, - "mode": { - "type": "string" - } - } - }, - "required": ["host", "container"] - }, - "config_map_volume": { - "type": "object", - "properties": { - "config_volume": { - "type": "object", - "name": { - "type": "string" - } + "config_map_volume": { + "type": "object", + "properties": { + "config_volume": { + "type": "object", + "name": { + "type": "string" + } + }, + "container": { + "type": "object", + "bind": { + "type": "string" + }, + "mode": { + "type": "string" + } + } + }, + "required": [ + "config_volume", + "container" + ] }, - "container": { - "type": "object", - "bind": { - "type": "string" - }, - "mode": { - "type": "string" - } - } - }, - "required": ["config_volume", "container"] - }, - "docker_healthcheck_http": { - "properties": { - "type": { - "description": "Consul health check type", - "type": "string", - "enum": [ - "http", - "https" - ] - }, - "interval": { - "description": "Interval duration in seconds i.e. 10s", - "default": "15s", - "type": "string" - }, - "timeout": { - "description": "Timeout in seconds i.e. 10s", - "default": "1s", - "type": "string" - }, - "endpoint": { - "description": "Relative endpoint used by Consul to check health by making periodic HTTP GET calls", - "type": "string" - } - }, - "required": [ - "type", - "endpoint" - ] - }, - "docker_healthcheck_script": { - "properties": { - "type": { - "description": "Consul health check type", - "type": "string", - "enum": [ - "script", - "docker" - ] - }, - "interval": { - "description": "Interval duration in seconds i.e. 10s", - "default": "15s", - "type": "string" - }, - "timeout": { - "description": "Timeout in seconds i.e. 10s", - "default": "1s", - "type": "string" - }, - "script": { - "description": "Script command that will be executed by Consul to check health", - "type": "string" + "docker_healthcheck_http": { + "properties": { + "type": { + "description": "Consul health check type", + "type": "string", + "enum": [ + "http", + "https", + "HTTP", + "HTTPS" + ] + }, + "interval": { + "description": "Interval duration in seconds i.e. 10s", + "default": "15s", + "type": "string" + }, + "timeout": { + "description": "Timeout in seconds i.e. 10s", + "default": "1s", + "type": "string" + }, + "endpoint": { + "description": "Relative endpoint used by Consul to check health by making periodic HTTP GET calls", + "type": "string" + }, + "port": { + "description": "Port mapping for readiness section", + "type": "integer" + }, + "initialDelaySeconds": { + "description": "Initial delay in seconds for readiness section", + "type": "integer" + } + }, + "required": [ + "type", + "endpoint" + ] + }, + "docker_healthcheck_script": { + "properties": { + "type": { + "description": "Consul health check type", + "type": "string", + "enum": [ + "script", + "docker" + ] + }, + "interval": { + "description": "Interval duration in seconds i.e. 10s", + "default": "15s", + "type": "string" + }, + "timeout": { + "description": "Timeout in seconds i.e. 10s", + "default": "1s", + "type": "string" + }, + "script": { + "description": "Script command that will be executed by Consul to check health", + "type": "string" + }, + "initialDelaySeconds": { + "description": "Initial delay in seconds for readiness section", + "type": "integer" + } + }, + "required": [ + "type", + "script" + ] } - }, - "required": [ - "type", - "script" - ] } - } - } +} diff --git a/docs/sections/design-components/component-specification/component-type-docker.rst b/docs/sections/design-components/component-specification/component-type-docker.rst index a685e342..db338cd1 100755 --- a/docs/sections/design-components/component-specification/component-type-docker.rst +++ b/docs/sections/design-components/component-specification/component-type-docker.rst @@ -1470,7 +1470,28 @@ the Docker image. For CDAP, this is the full path to the CDAP jar. Auxilary -------- +New V3 version of component spec schema introduced - +https://github.com/onap/dcaegen2-platform/blob/master/mod/component-json-schemas/component-specification/dcae-cli-v3/component-spec-schema.json +- Added new “helm” object under “auxilary\_docker” properties + + - Includes “applicationEnv” + + - Includes “service” definition + +- Readiness Configuration support + + - docker\_healthcheck\_http + + - Added HTTP/HTTPS for supported protocol enum list + + - Added “port” + + - Added “initialDelaySeconds” + + - docker\_healthcheck\_script + + - Added “initialDelaySeconds” Health check ~~~~~~~~~~~~ diff --git a/docs/sections/design-components/component-specification/data-formats.rst b/docs/sections/design-components/component-specification/data-formats.rst index 42194fa3..b0863ca4 100755 --- a/docs/sections/design-components/component-specification/data-formats.rst +++ b/docs/sections/design-components/component-specification/data-formats.rst @@ -1,235 +1,235 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. http://creativecommons.org/licenses/by/4.0
-
-.. _data-formats:
-
-
-Data Formats
-============
-
-Data formats are descriptions of data; they are the data contract
-between your component and other components. When the components are
-‘composed’ into services in the Design tool, they can only be matched with
-components that have compatible data formats. Data formats will be
-onboarded to Design tool and assigned a UUID at that time. This UUID is then
-used to ensure compatibility amoung components. (If component X outputs
-data format ‘DF-Y’, and another component Z specifies ‘DF-Y’ as its
-input data format, then X is said to be ``composable`` with component
-Z).
-
-Since data formats will be shared across components, the onboarding
-catalog should be checked first to see if the desired data format is
-available before creating one. The vision is to have a repository of
-shared data formats that developers and teams can re-use and also
-provide them the means to extend and create new custom data formats. A
-data format is referenced by its data format id and version number.
-
-JSON schema
------------
-
- The data format specification is represented (and validated) against
- this `Data Format json schema <https://git.onap.org/dcaegen2/platform/plain/mod/component-json-schemas/data-format/dcae-cli-v1/data-format-schema.json>`__
- and described below:
-
-Meta Schema Definition
-~~~~~~~~~~~~~~~~~~~~~~
-
-The “Meta Schema” implementation defines how data format JSON schemas
-can be written to define user input. It is itself a JSON schema (thus it
-is a “meta schema”). It requires the name of the data format entry, the
-data format entry version and allows a description under “self” object.
-The meta schema version must be specified as the value of the
-“dataformatversion” key. Then the input schema itself is described as
-one of the four types listed below:
-
-+------------------+---------------------------------------------------+
-| Type | Description |
-+==================+===================================================+
-| jsonschema | inline standard JSON Schema definitions of JSON |
-| | inputs |
-+------------------+---------------------------------------------------+
-| delimitedschema | delimited data input using a JSON description and |
-| | defined delimiter |
-+------------------+---------------------------------------------------+
-| unstructured | unstructured text, and reference that allows a |
-| | pointer to another artifact for a schema. |
-+------------------+---------------------------------------------------+
-| reference | allows for XML and Protocol Buffers schema, |
-| | but can be used to reference other JSON, |
-| | delimitedschema and unstructured schemas as well. |
-+------------------+---------------------------------------------------+
-
-
-Example Schemas
----------------
-
-By reference example - Common Event Format
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-First the full JSON schema description of the Common Event Format would
-be loaded with a name of “Common Event Format” and the current version
-of “25.0.0”.
-
-Then the data format description is loaded by this schema:
-
-::
-
- {
- "self": {
- "name": "Common Event Format Definition",
- "version": "25.0.0",
- "description": "Common Event Format Definition"
-
- },
- "dataformatversion": "1.0.0",
- "reference": {
- "name": "Common Event Format",
- "format": "JSON",
- "version": "25.0.0"
- }
- }
-
-
-
-Simple JSON Example
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-
-::
-
- {
- "self": {
- "name": "Simple JSON Example",
- "version": "1.0.0",
- "description": "An example of unnested JSON schema for Input and output"
-
- },
- "dataformatversion": "1.0.0",
- "jsonschema": {
- "$schema": "http://json-schema.org/draft-04/schema#",
- "type": "object",
- "properties": {
- "raw-text": {
- "type": "string"
- }
- },
- "required": ["raw-text"],
- "additionalProperties": false
- }
- }
-
-Nested JSON Example
-~~~~~~~~~~~~~~~~~~~
-
-::
-
- {
- "self": {
- "name": "Nested JSON Example",
- "version": "1.0.0",
- "description": "An example of nested JSON schema for Input and output"
-
- },
- "dataformatversion": "1.0.0",
- "jsonschema": {
- "$schema": "http://json-schema.org/draft-04/schema#",
- "properties": {
- "numFound": {
- "type": "integer"
- },
- "start": {
- "type": "integer"
- },
- "engagements": {
- "type": "array",
- "items": {
- "properties": {
- "engagementID": {
- "type": "string",
- "transcript": {
- "type": "array",
- "items": {
- "type": {
- "type": "string"
- },
- "content": {
- "type": "string"
- },
- "senderName": {
- "type": "string"
- },
- "iso": {
- "type": "string"
- },
- "timestamp": {
- "type": "integer"
- },
- "senderId": {
- "type": "string"
- }
- }
- }
- }
- }
- }
- }
- },
- "additionalProperties": false
- }
- }
-
-Unstructured Example
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-::
-
- {
- "self": {
- "name": "Unstructured Text Example",
- "version": "25.0.0",
- "description": "An example of a unstructured text used for both input and output for "
-
- },
- "dataformatversion": "1.0.0",
- "unstructured": {
- "encoding": "UTF-8"
- }
- }
-
-
-An example of a delimited schema
---------------------------------
-
-::
-
- {
- "self": {
- "name": "Delimited Format Example",
- "version": "1.0.0",
- "description": "Delimited format example just for testing"
-
- },
- "dataformatversion": "1.0.0",
- "delimitedschema": {
- "delimiter": "|",
- "fields": [{
- "name": "field1",
- "description": "test field1",
- "fieldtype": "string"
- }, {
- "name": "field2",
- "description": "test field2",
- "fieldtype": "boolean"
- }]
- }
- }
-
-Note: The referenced data format (in this case, a schema named “Common
-Event Format” with version of “25.0.0”) must already exist in the
-onboarding catalog.
-
-Working with Data Formats
--------------------------
-
-Data Formats can be validated using `schema <https://git.onap.org/dcaegen2/platform/plain/mod/component-json-schemas/data-format/dcae-cli-v1/data-format-schema.json>`__
-Once validated, the dataformat can be onboarded using :doc:`DCAE-MOD <../DCAE-MOD/DCAE-MOD-User-Guide>`
\ No newline at end of file +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +.. _data-formats: + + +Data Formats +============ + +Data formats are descriptions of data; they are the data contract +between your component and other components. When the components are +‘composed’ into services in the Design tool, they can only be matched with +components that have compatible data formats. Data formats will be +onboarded to Design tool and assigned a UUID at that time. This UUID is then +used to ensure compatibility amoung components. (If component X outputs +data format ‘DF-Y’, and another component Z specifies ‘DF-Y’ as its +input data format, then X is said to be ``composable`` with component +Z). + +Since data formats will be shared across components, the onboarding +catalog should be checked first to see if the desired data format is +available before creating one. The vision is to have a repository of +shared data formats that developers and teams can re-use and also +provide them the means to extend and create new custom data formats. A +data format is referenced by its data format id and version number. + +JSON schema +----------- + + The data format specification is represented (and validated) against + this `Data Format json schema <https://git.onap.org/dcaegen2/platform/plain/mod/component-json-schemas/data-format/dcae-cli-v1/data-format-schema.json>`__ + and described below: + +Meta Schema Definition +~~~~~~~~~~~~~~~~~~~~~~ + +The “Meta Schema” implementation defines how data format JSON schemas +can be written to define user input. It is itself a JSON schema (thus it +is a “meta schema”). It requires the name of the data format entry, the +data format entry version and allows a description under “self” object. +The meta schema version must be specified as the value of the +“dataformatversion” key. Then the input schema itself is described as +one of the four types listed below: + ++------------------+---------------------------------------------------+ +| Type | Description | ++==================+===================================================+ +| jsonschema | inline standard JSON Schema definitions of JSON | +| | inputs | ++------------------+---------------------------------------------------+ +| delimitedschema | delimited data input using a JSON description and | +| | defined delimiter | ++------------------+---------------------------------------------------+ +| unstructured | unstructured text, and reference that allows a | +| | pointer to another artifact for a schema. | ++------------------+---------------------------------------------------+ +| reference | allows for XML and Protocol Buffers schema, | +| | but can be used to reference other JSON, | +| | delimitedschema and unstructured schemas as well. | ++------------------+---------------------------------------------------+ + + +Example Schemas +--------------- + +By reference example - Common Event Format +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +First the full JSON schema description of the Common Event Format would +be loaded with a name of “Common Event Format” and the current version +of “25.0.0”. + +Then the data format description is loaded by this schema: + +:: + + { + "self": { + "name": "Common Event Format Definition", + "version": "25.0.0", + "description": "Common Event Format Definition" + + }, + "dataformatversion": "1.0.0", + "reference": { + "name": "Common Event Format", + "format": "JSON", + "version": "25.0.0" + } + } + + + +Simple JSON Example +~~~~~~~~~~~~~~~~~~~ + + +:: + + { + "self": { + "name": "Simple JSON Example", + "version": "1.0.0", + "description": "An example of unnested JSON schema for Input and output" + + }, + "dataformatversion": "1.0.0", + "jsonschema": { + "$schema": "http://json-schema.org/draft-04/schema#", + "type": "object", + "properties": { + "raw-text": { + "type": "string" + } + }, + "required": ["raw-text"], + "additionalProperties": false + } + } + +Nested JSON Example +~~~~~~~~~~~~~~~~~~~ + +:: + + { + "self": { + "name": "Nested JSON Example", + "version": "1.0.0", + "description": "An example of nested JSON schema for Input and output" + + }, + "dataformatversion": "1.0.0", + "jsonschema": { + "$schema": "http://json-schema.org/draft-04/schema#", + "properties": { + "numFound": { + "type": "integer" + }, + "start": { + "type": "integer" + }, + "engagements": { + "type": "array", + "items": { + "properties": { + "engagementID": { + "type": "string", + "transcript": { + "type": "array", + "items": { + "type": { + "type": "string" + }, + "content": { + "type": "string" + }, + "senderName": { + "type": "string" + }, + "iso": { + "type": "string" + }, + "timestamp": { + "type": "integer" + }, + "senderId": { + "type": "string" + } + } + } + } + } + } + } + }, + "additionalProperties": false + } + } + +Unstructured Example +~~~~~~~~~~~~~~~~~~~~ + +:: + + { + "self": { + "name": "Unstructured Text Example", + "version": "25.0.0", + "description": "An example of a unstructured text used for both input and output for " + + }, + "dataformatversion": "1.0.0", + "unstructured": { + "encoding": "UTF-8" + } + } + + +An example of a delimited schema +-------------------------------- + +:: + + { + "self": { + "name": "Delimited Format Example", + "version": "1.0.0", + "description": "Delimited format example just for testing" + + }, + "dataformatversion": "1.0.0", + "delimitedschema": { + "delimiter": "|", + "fields": [{ + "name": "field1", + "description": "test field1", + "fieldtype": "string" + }, { + "name": "field2", + "description": "test field2", + "fieldtype": "boolean" + }] + } + } + +Note: The referenced data format (in this case, a schema named “Common +Event Format” with version of “25.0.0”) must already exist in the +onboarding catalog. + +Working with Data Formats +------------------------- + +Data Formats can be validated using `schema <https://git.onap.org/dcaegen2/platform/plain/mod/component-json-schemas/data-format/dcae-cli-v1/data-format-schema.json>`__ +Once validated, the dataformat can be onboarded using :doc:`DCAE-MOD <../DCAE-MOD/DCAE-MOD-User-Guide>` diff --git a/docs/sections/design-components/images/128713731_image2022.png b/docs/sections/design-components/images/128713731_image2022.png Binary files differnew file mode 100644 index 00000000..0c9356c3 --- /dev/null +++ b/docs/sections/design-components/images/128713731_image2022.png diff --git a/docs/sections/design-components/requirements-guidelines.rst b/docs/sections/design-components/requirements-guidelines.rst index c887d4ff..015688fe 100644 --- a/docs/sections/design-components/requirements-guidelines.rst +++ b/docs/sections/design-components/requirements-guidelines.rst @@ -4,223 +4,67 @@ Onboarding Pre-requisite ======================== -Before a component is onboarded into DCAE, the component developer must ensure it -is compliant with ONAP & DCAE goals and requirement in order to correctly be deployed and be managed. This -page will discuss the changes which are grouped into the following -categories: - -- :any:`Configuration management via ConfigBindingService <configuration_management>` -- :any:`Docker images <docker_images>` -- :any:`Policy Reconfiguration flow support <policy_reconfiguration>` -- :any:`Operational Requirement <operation_requirement>` - - -.. _configuration_management: - -Configuration Management ------------------------- - -All configuration for a component is stored in CONSUL under the -components uniquely generated name which is provided by the environment -variable ``HOSTNAME`` as well as ``SERVICE_NAME``. It is then made -available to the component via a remote HTTP service call to CONFIG -BINDING SERVICE. - -The main entry in CONSUL for the component contains its -**generated application configuration**. This is based on the submitted -component specification, and consists of the *interfaces* (streams and -services/calls) and *parameters* sections. Other entries may exist as -well, under specific keys, such as :dmaap . Each key represents a -specific type of information and is also available to the component by -calling CONFIG BINDING SERVICE. More on this below. - -Components are required to pull their -**generated application configuration** at application startup using the environment -setting exposed during deployment. - - -Envs -~~~~ - -The platform provides a set of environment variables into each Docker -container: - -+----------------------------+--------------+----------------------------------------+ -| Name | Type | Description | -+============================+==============+========================================+ -| ``HOSTNAME`` | string | Unique name of the component instance | -| | | that is generated | -+----------------------------+--------------+----------------------------------------+ -| ``CONSUL_HOST`` | string | Hostname of the platform's Consul | -| | | instance | -+----------------------------+--------------+----------------------------------------+ -| ``CONFIG_BINDING_SERVICE`` | string | Hostname of the platform's config | -| | | binding service instance | -| | | | -+----------------------------+--------------+----------------------------------------+ -| ``DOCKER_HOST`` | string | Host of the target platform Docker | -| | | host to run the container on | -+----------------------------+--------------+----------------------------------------+ -| ``CBS_CONFIG_URL`` | string | Fully resolved URL to query config | -| | | from CONSUL via CBS | -+----------------------------+--------------+----------------------------------------+ +Before a component is onboarded into DCAE, the component developer must ensure it +is compliant with ONAP & DCAE goals and requirement in order to correctly be deployed and be managed. .. _config_binding_service: -Config Binding Service -~~~~~~~~~~~~~~~~~~~~~~ +Config Binding Service SDK Integration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The config binding service is a platform HTTP service that is -responsible for providing clients with its fully resolve configuration -JSON at startup, and also other configurations objects -when requested. +With Jakarta release, Consul and ConfigBindingService interface has been deprecated from DCAE +All Microservice configuration are resolved through files mounted via Configmap created part of +dcae-services helm chart deployment. -At runtime, components should make an HTTP GET on: +CBS SDK library are available within DCAE which can be used by DCAE Microservices for configuration +retrieval. For details on the API - refer `CBS SDK Java Library +<https://docs.onap.org/projects/onap-dcaegen2/en/latest/sections/sdk/api.html>`__ -:: - - <config binding service hostname>:<port>/service_component/NAME +Its strongly recommended to use CBS SDK library for consistency across DCAE services to retrieve both static and policy driven configuration. -For Docker components, NAME should be set to ``HOSTNAME``, which is -provided as an ENV variable to the container. +Topic Configuration +~~~~~~~~~~~~~~~~~~~ -The binding service integrates with the streams and services section of -the component specification. For example, if you specify that you call a -service: +With Helm flow integration in MOD, topic generation feature is not supported. -:: - - "services": { - "calls": [{ - "config_key": "vnf-db", - "request": { - "format": "dcae.vnf.meta", - "version": "1.0.0" - }, - "response": { - "format": "dcae.vnf.kpi", - "version": "1.0.0" - } - }], - ... - } - -Then the config binding service will find all available IP addresses of -services meeting the containers needs, and provide them to the container -under your ``config_key``: +Applications are required to identify the topic and feed information as application +configuration. +For application onboarded through MOD, these should be included in the specification file under **parameters** :: - // your configuration - { - "vbf-db" : // see above - [IP:Port1, IP:Port2,…] // all of these meet your needs, choose one. - } - -Regarding ``<config binding service hostname>:<port>``, there is DNS -work going on to make this resolvable in a convenient way inside of your -container. - -For all Kubernetes deployments since El-Alto, an environment variable ``CBS_CONFIG_URL`` will be exposed -by platform (k8s plugins) providing the exact URL to be used for configuration retrieval. -Application can use this URL directly instead of constructing URL from HOSTNAME (which refers to ServiceComponentName) -and CONFIG_BINDING_SERVICE env's. By default, this URL will use HTTPS CBS interface - -If you are integrating with CBS SDK, then the DNS resolution and configuration fetch -are handled via library functions. - -Generated Application Configuration -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The DCAE platform uses the component specification to generate the -component’s application configuration provided at deployment time. The -component developer should expect to use this configuration JSON in the -component. - - -The following component spec snippet (from String Matching): + "parameters": [{ + "name": "streams_publishes", + "value": "{\"ves-3gpp-fault-supervision\":{\"dmaap_info\":{\"topic_url\":\"http:\/\/message-router:3904\/events\/unauthenticated.SEC_3GPP_FAULTSUPERVISION_OUTPUT\"},\"type\":\"message_router\"},\"ves-3gpp-heartbeat\":{\"dmaap_info\":{\"topic_url\":\"http:\/\/message-router:3904\/events\/unauthenticated.SEC_3GPP_HEARTBEAT_OUTPUT\"},\"type\":\"message_router\"},\"ves-3gpp-performance-assurance\":{\"dmaap_info\":{\"topic_url\":\"http:\/\/message-router:3904\/events\/unauthenticated.SEC_3GPP_PERFORMANCEASSURANCE_OUTPUT\"},\"type\":\"message_router\"},\"ves-3gpp-provisioning\":{\"dmaap_info\":{\"topic_url\":\"http:\/\/message-router:3904\/events\/unauthenticated.SEC_3GPP_PROVISIONING_OUTPUT\"},\"type\":\"message_router\"},\"ves-fault\":{\"dmaap_info\":{\"topic_url\":\"http:\/\/message-router:3904\/events\/unauthenticated.SEC_FAULT_OUTPUT\"},\"type\":\"message_router\"},\"ves-heartbeat\":{\"dmaap_info\":{\"topic_url\":\"http:\/\/message-router:3904\/events\/unauthenticated.SEC_HEARTBEAT_OUTPUT\"},\"type\":\"message_router\"},\"ves-measurement\":{\"dmaap_info\":{\"topic_url\":\"http:\/\/message-router:3904\/events\/unauthenticated.VES_MEASUREMENT_OUTPUT\"},\"type\":\"message_router\"},\"ves-notification\":{\"dmaap_info\":{\"topic_url\":\"http:\/\/message-router:3904\/events\/unauthenticated.VES_NOTIFICATION_OUTPUT\"},\"type\":\"message_router\"},\"ves-other\":{\"dmaap_info\":{\"topic_url\":\"http:\/\/message-router:3904\/events\/unauthenticated.SEC_OTHER_OUTPUT\"},\"type\":\"message_router\"},\"ves-pnfRegistration\":{\"dmaap_info\":{\"topic_url\":\"http:\/\/message-router:3904\/events\/unauthenticated.VES_PNFREG_OUTPUT\"},\"type\":\"message_router\"}}", + "description": "standard http port collector will open for listening;", + "sourced_at_deployment": false, + "policy_editable": false, + "designer_editable": false + }] +For components delivered as Helm directly, it should be specified under **applicationConfig** section in values.yaml :: - "streams":{ - "subscribes": [{ - "format": "VES_specification", - "version": "4.27.2", - "type": "message_router", - "config_key" : "mr_input" - }], - "publishes": [{ - "format": "VES_specification", - "version": "4.27.2", - "config_key": "mr_output", - "type": "message_router" - }] - }, - "services":{ - "calls": [{ - "config_key" : "aai_broker_handle", - "verb": "GET", - "request": { - "format": "get_with_query_params", - "version": "1.0.0" - }, - "response": { - "format": "aai_broker_response", - "version": "3.0.0" - } - }], - "provides": [] - }, - -Will result in the following top level keys in the configuration - -:: + streams_publishes: + ves-fault: + dmaap_info: + topic_url:"http://message-router:3904/events/unauthenticated.SEC_FAULT_OUTPUT" + type: message_router + ves-measurement: + dmaap_info: + topic_url: "http://message-router:3904/events/unauthenticated.VES_MEASUREMENT_OUTPUT" + type: message_router - "streams_publishes":{ - "mr_output":{ // notice the config key above - "aaf_password":"XXX", - "type":"message_router", - "dmaap_info":{ - "client_role": null, - "client_id": null, - "location": null, - "topic_url":"https://YOUR_HOST:3905/events/com.att.dcae.dmaap.FTL2.DCAE-CL-EVENT" // just an example - }, - "aaf_username":"XXX" - } - }, - "streams_subscribes":{ - "mr_input":{ // notice the config key above - "aaf_password":"XXX", - "type":"message_router", - "dmaap_info":{ - "client_role": null, - "client_id": null, - "location": null, - "topic_url":"https://YOUR_HOST:3905/events/com.att.dcae.dmaap.FTL2.TerrysStringMatchingTest" // just an example - }, - "aaf_username":"XXX" - } - }, - "services_calls":{ - "aai_broker_handle":[ // notice the config key above - "135.205.226.128:32768" // based on deployment time, just an example - ] - } - -These keys will always be populated whether they are empty or not. So -the minimum configuration you will get, (in the case of a component that -provides an HTTP service, doesn’t call any services, and has no streams, -is: -:: +You can find examples of topic and feed configuration used in DCAE components from charts under OOM repository - +https://github.com/onap/oom/tree/master/kubernetes/dcaegen2-services/components - "streams_publishes":{}, - "streams_subscribes":{}, - "services_calls":{} +Its recommended to follow similar topic construct for consistency across all DCAE Services. This will also enable using + `SDK DMAAP Java Library +<https://docs.onap.org/projects/onap-dcaegen2/en/latest/sections/sdk/api.html>`__ +for easier integration. -Thus your component should expect these well-known top level keys. DCAE SDK ~~~~~~~~ @@ -237,15 +81,52 @@ DCAE has SDK/libraries which can be used for service components for easy integra Policy Reconfiguration ---------------------- -Components must provide a way to receive policy reconfiguration, that -is, configuration parameters that have been updated via the Policy UI. -The component developer must either periodically poll the ConfigBindingService API -to retrieve/refresh the new configuration or provides a script (defined in the :any:`Docker -auxiliary specification <docker-auxiliary-details>`) -that will be triggered when policy update is detected by the platform. +Policy Framework based reconfiguration is supported via sidecar. The component owners are responsible for +loading the required model and creating policies required. + +Once the policies are created, the corresponding policy_id should be listed in the component_spec or helm charts override as below + +Component spec must include the policy_info object and list of policy_id to be retrieved +:: + "policy_info":{ + "policy": [ + { + "node_label": "tca_policy_00", + "policy_model_id": "onap.policies.monitoring.cdap.tca.hi.lo.app", + "policy_id": "onap.vfirewall.tca" + }, + { + "node_label":"tca_policy_01", + "policy_model_id":"onap.policies.monitoring.cdap.tca.hi.lo.app", + "policy_id":"onap.vdns.tca" + } + ] + } + +"node_label" is optional and can be ignored +"policy_model_id" refers to model uploaded into policy framework +"policy_id" refers to the instance of policy created for model specified. + +When the helm-charts are generated by DCAEMOD/Runtime, the charts will have following property defined in the values.yaml + +:: + dcaePolicySyncImage: onap/org.onap.dcaegen2.deployments.dcae-services-policy-sync:1.0.1 + policies: + policyID: | + '["onap.vfirewall.tca","onap.vdns.tca"]' + +When using dcaegen2-services-common templates, the presence of **policies** property will deploy policy-sidecar automatically which will +periodically pull configuration from Policy framework and make it available shared mountpoint to microservice container. -.. _docker_images: +More information on Policy side car can be found on this wiki - https://wiki.onap.org/display/DW/Policy+function+as+Sidecar + +.. note:: + When using DCAE CBS SDK, policy config retrieval is supported natively by the library + + + +.. _docker_images: Docker Images ------------- @@ -258,6 +139,13 @@ For ONAP microservices, the components images are expected to pushed into ONAP n part of `ONAP CI jobs <https://wiki.onap.org/display/DW/Using+Standard+Jenkins+Job+%28JJB%29+Templates>`__ +Helm Chart +---------- + +Components being delivered under ONAP/OOM must adopt dcaegen2-common-services template. +Information about using the common templates to deploy a microservice can be +found in :doc:`Helm to deploy DCAE Microservices <./dcaeservice_helm_template>`. + .. _operation_requirement: Operational Requirement @@ -266,14 +154,14 @@ Operational Requirement Logging ~~~~~~~ -All ONAP MS logging should follow logging specification defined by `logging project <https://wiki.onap.org/pages/viewpage.action?pageId=71831691>`__ +All ONAP MS logging should follow logging specification defined by `SECCOM <https://wiki.onap.org/display/DW/Jakarta+Best+Practice+Proposal+for+Standardized+Logging+Fields+-+v2>`__ The application log configuration must enable operation to choose if to be written into file or stdout or both during deployment. -S3P +S3P ~~~ -ONAP S3P (all scaling/resiliency/security/maintainability) goals should meet at the minimum level defined for DCAE project for the targeted release - -If the component is stateful, it should persist its state on external store (eg. pg, redis) to allow support for scaling and resiliency. This should be important design criteria for the component. If the components either publish/subscribe into DMAAP topic, then secure connection to DMAAP must be supported (platform will provide aaf_username/aaf_password for each topic as configuration). +ONAP S3P (all scaling/resiliency/security/maintainability) goals should meet at the minimum level defined for DCAE project for the targeted release +If the component is stateful, it should persist its state on external store (eg. pg, redis) to allow support for scaling and +resiliency. This should be important design criteria for the component. diff --git a/docs/sections/images/R10_architecture_diagram.png b/docs/sections/images/R10_architecture_diagram.png Binary files differnew file mode 100644 index 00000000..c862abd1 --- /dev/null +++ b/docs/sections/images/R10_architecture_diagram.png diff --git a/docs/sections/images/R9_architecture_diagram.png b/docs/sections/images/R9_architecture_diagram.png Binary files differnew file mode 100644 index 00000000..0f6bdc1c --- /dev/null +++ b/docs/sections/images/R9_architecture_diagram.png diff --git a/docs/sections/installation_oom.rst b/docs/sections/installation_oom.rst index f71675bf..41cf34ab 100644 --- a/docs/sections/installation_oom.rst +++ b/docs/sections/installation_oom.rst @@ -1,8 +1,8 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -DCAE Deployment (using Helm and Cloudify) -========================================= +DCAE Deployment (using Helm) +============================ This document describes the details of the Helm chart based deployment process for ONAP and how DCAE is deployed through this process. @@ -21,18 +21,22 @@ At deployment time, with a single **helm deploy** command, Helm resolves all the and invokes Kubernetes deployment operations for all the resources. All ONAP Helm charts are organized under the **kubernetes** directory of the **OOM** project, where roughly each ONAP component occupies a subdirectory. -DCAE platform components are deployed using Helm charts under the **dcaegen2** directory. -With DCAE Transformation to Helm in Istabul, all DCAE components are supported for both helm and Cloudify/Blueprint deployments. Charts for individual MS are available under **dcaegen2-services** directory under OOM project (https://git.onap.org/oom/tree/kubernetes/dcaegen2-services/components). With ONAP deployment, four DCAE services (HV VES collector, VES collector, PNF Registration Handler, and TCA (Gen2) analytics service) are bootstrapped via Helm charts. -Other DCAE Services are deployed on-demand, after ONAP/DCAE installation, through Cloudify Blueprints or helm-charts. For on-demand helm chart, refer to steps described in :ref:`Helm install/upgrade section <dcae-service-deployment>`. -Operators can deploy on-demand other MS required for their usecases also via Cloudify as described in :doc:`On-demand MS Installation <./installation_MS_ondemand>`. +With DCAE Transformation to Helm completed in Jakarta/R10 release, all DCAE components deployment will be supported only via helm. Charts for individual MS are available under **dcaegen2-services** directory under OOM project (https://git.onap.org/oom/tree/kubernetes/dcaegen2-services/components). With ONAP deployment, four DCAE services (HV VES collector, VES collector, PNF Registration Handler, and TCA (Gen2) analytics service) are bootstrapped via Helm charts. + +Other DCAE Services can be deployed on-demand via their independent helm-charts. For on-demand helm chart, refer to steps described in :ref:`Helm install/upgrade section <dcae-service-deployment>`. + + +.. note:: + DCAE platform components deployments is optionally available through Helm charts under the **dcaegen2** directory however this mode is not supported with Jakarta release. These charts will be removed in subsequent release. + DCAE Chart Organization ----------------------- -Following Helm conventions, the DCAE Helm chart directory (``oom/kubernetes/dcaegen2``) consists of the following files and subdirectories: +Following Helm conventions, the DCAE Helm chart directory (``oom/kubernetes/dcaegen2-services`` & ``oom/kubernetes/dcaegen2``) consists of the following files and subdirectories: * ``Chart.yaml``: metadata. * ``requirements.yaml``: dependency charts. @@ -41,19 +45,6 @@ Following Helm conventions, the DCAE Helm chart directory (``oom/kubernetes/dcae * ``Makefile``: make file to build DCAE charts * ``components``: subdirectory for DCAE sub-charts. -The dcaegen2 chart has the following sub-charts: - -* ``dcae-bootstrap``: deploys the DCAE bootstrap service that performs some DCAE initialization and deploys additional DCAE components. -* ``dcae-cloudify-manager``: deploys the DCAE Cloudify Manager instance. -* ``dcae-config-binding-service``: deploys the DCAE config binding service. -* ``dcae-deployment-handler``: deploys the DCAE deployment handler service. -* ``dcae-healthcheck``: deploys the DCAE healthcheck service that provides an API to check the health of all DCAE components. -* ``dcae-policy-handler``: deploys the DCAE policy handler service. -* ``dcae-redis``: deploys the DCAE Redis cluster. -* ``dcae-dashboard``: deploys the DCAE Dashboard for managing DCAE microservices deployments -* ``dcae-servicechange-handler``: deploys the DCAE service change handler service. -* ``dcae-inventory-api``: deploys the DCAE inventory API service. -* ``dcae-ves-openapi-manager``: deploys the DCAE service validator of VES_EVENT type artifacts from distributed services. The dcaegen2-services chart has the following sub-charts: @@ -79,6 +70,7 @@ The dcaegen2-services chart has the following sub-charts: * ``dcae-snmptrap-collector``: deploys the DCAE SNMPTRAP collector service. * ``dcae-son-handler``: deploys the DCAE SON-Handler microservice. * ``dcae-ves-mapper``: deploys the DCAE VES Mapper microservice. +* ``dcae-ves-openapi-manager``: deploys the DCAE service validator of VES_EVENT type artifacts from distributed services. The dcaegen2-services sub-charts depend on a set of common templates, found under the ``common`` subdirectory under ``dcaegen2-services``. @@ -86,64 +78,48 @@ The dcaegen2-services sub-charts depend on a set of common templates, found unde Information about using the common templates to deploy a microservice can be found in :doc:`Using Helm to deploy DCAE Microservices <./dcaeservice_helm_template>`. +The dcaegen2 chart has the following sub-charts: + +* ``dcae-bootstrap``: deploys the DCAE bootstrap service that performs some DCAE initialization and deploys additional DCAE components. +* ``dcae-cloudify-manager``: deploys the DCAE Cloudify Manager instance. +* ``dcae-config-binding-service``: deploys the DCAE config binding service. +* ``dcae-deployment-handler``: deploys the DCAE deployment handler service. +* ``dcae-healthcheck``: deploys the DCAE healthcheck service that provides an API to check the health of all DCAE components. +* ``dcae-policy-handler``: deploys the DCAE policy handler service. +* ``dcae-redis``: deploys the DCAE Redis cluster. +* ``dcae-dashboard``: deploys the DCAE Dashboard for managing DCAE microservices deployments +* ``dcae-servicechange-handler``: deploys the DCAE service change handler service. +* ``dcae-inventory-api``: deploys the DCAE inventory API service. + +These components are by default disabled under ONAP for Jakarta release and charts will be removed next release + DCAE Deployment --------------- At deployment time for ONAP, when the **helm deploy** command is executed, -DCAE resources defined within the subcharts - "dcaegen2" above are deployed -along with subset of DCAE Microservices (based on override file configuration -defined in `values.yaml <https://git.onap.org/oom/tree/kubernetes/dcaegen2-services/values.yaml>`_ - +only the DCAE resources defined within the subcharts - "dcaegen2-services" above are deployed +(based on override file configuration defined in `values.yaml <https://git.onap.org/oom/tree/kubernetes/dcaegen2-services/values.yaml>`_ + These include: -* DCAE bootstrap service -* DCAE healthcheck service -* DCAE platform components: - - * Cloudify Manager - * Config binding service - * Deployment handler - * Policy handler - * Service change handler - * Inventory API service - * Inventory postgres database service (launched as a dependency of the inventory API service) - * DCAE postgres database service (launched as a dependency of the bootstrap service) - * DCAE Mongo database service (launched as a dependency of the bootstrap service) - * VES OpenAPI Manager - * DCAE Service components: * VES Collector * HV-VES Collector * PNF-Registration Handler Service * Threshold Crossing Analysis (TCA-gen2) +* DCAE-Services healthcheck +* VES OpenAPI Manager Some of the DCAE subcharts include an initContainer that checks to see if other services that they need in order to run have become ready. The installation of these subcharts will pause until the needed services are available. -In addition, DCAE operations depends on a Consul server cluster. -For ONAP OOM deployment, the Consul cluster is provided as a shared -resource. Its charts are defined under the ``oom/kubernetes/consul`` -directory, not as part of the DCAE chart hierarchy. - -With Istanbul release, DCAE bootstrapped Microservice deployment are managed completely under Helm. The Cloudify -Bootstrap container preloads the microservice blueprints into DCAE Inventory, thereby making them available -for On-Demand deployment support (trigger from CLAMP or external projects). - -The dcae-bootstrap service has a number of prerequisites because the subsequently deployed DCAE components depends on a number of resources having entered their normal operation state. DCAE bootstrap job will not start before these resources are ready. They are: - - * dcae-cloudify-manager - * consul-server - * msb-discovery - * kube2msb - * dcae-config-binding-service - * dcae-db - * dcae-mongodb - * dcae-inventory-api +Since Istanbul release, DCAE bootstrapped Microservice deployment are managed completely under Helm. Additionaly tls-init-container invoked during component deployment relies on AAF to generate the required certificate hence AAF -must be enabled under OOM deployment configuration. +must be enabled under OOM deployment configuration. +As majority of DCAE services rely on DMAAP (MR and DR) interfaces, ONAP/DMAAP must also be enabled under OOM deployment configuration. DCAE Configuration ------------------ @@ -159,64 +135,15 @@ Deployment time configuration of DCAE components are defined in several places. * In a Helm chart hierarchy, values defined in values.yaml files in higher level supersedes values defined in values.yaml files in lower level; * Helm command line supplied values supersedes values defined in any values.yaml files. -In addition, for DCAE components deployed through Cloudify Manager blueprints, their configuration parameters are defined in the following places: - - * The blueprint files can contain static values for configuration parameters; - * The blueprint files are defined under the ``blueprints`` directory of the ``dcaegen2/platform/blueprints`` repo, named with "k8s" prefix. - * The blueprint files can specify input parameters and the values of these parameters will be used for configuring parameters in Blueprints. The values for these input parameters can be supplied in several ways as listed below in the order of precedence (low to high): - * The blueprint files can define default values for the input parameters; - * The blueprint input files can contain static values for input parameters of blueprints. These input files are provided as config resources under the dcae-bootstrap chart; - * The blueprint input files may contain Helm templates, which are resolved into actual deployment time values following the rules for Helm values. - - -Now we walk through an example, how to configure the Docker image for the DCAE VESCollector, which is deployed by Cloudify Manager. - -(*Note: Beginning with the Istanbul release, VESCollector is no longer deployed using Cloudify Manager during bootstrap. However, the example is still -useful for understanding how to deploy other components using a Cloudify blueprint.*) - -In the `k8s-ves.yaml <https://git.onap.org/dcaegen2/platform/blueprints/tree/blueprints/k8s-ves.yaml>`_ blueprint, the Docker image to use is defined as an input parameter with a default value: - -.. code-block:: yaml - - tag_version: - type: string - default: "nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector:1.5.4" - -The corresponding input file, ``https://git.onap.org/oom/tree/kubernetes/dcaegen2/components/dcae-bootstrap/resources/inputs/k8s-ves-inputs-tls.yaml``, -it is defined again as: - -.. code-block:: yaml - {{ if .Values.componentImages.ves }} - tag_version: {{ include "common.repository" . }}/{{ .Values.componentImages.ves }} - {{ end }} - - -Thus, when ``common.repository`` and ``componentImages.ves`` are defined in the ``values.yaml`` files, -their values will be plugged in here and the resulting ``tag_version`` value -will be passed to the blueprint as the Docker image tag to use instead of the default value in the blueprint. - -The ``componentImages.ves`` value is provided in the ``oom/kubernetes/dcaegen2/charts/dcae-bootstrap/values.yaml`` file: - -.. code-block:: yaml - - componentImages: - ves: onap/org.onap.dcaegen2.collectors.ves.vescollector:1.5.4 - - -The final result is that when DCAE bootstrap calls Cloudify Manager to deploy the DCAE VES collector, the 1.5.4 image will be deployed. - .. _dcae-service-deployment: On-demand deployment/upgrade through Helm ----------------------------------------- -Under DCAE Transformation to Helm, all DCAE components has been delivered as helm charts under +Under DCAE Transformation to Helm, all DCAE components has been delivered as helm charts under OOM repository (https://git.onap.org/oom/tree/kubernetes/dcaegen2-services). -Blueprint deployment is also available to support regression usecases; ``Istanbul will be final release where -Cloudify blueprint for components/microservices will be supported.`` - All DCAE component charts follows standard Helm structure. Each Microservice charts has predefined configuration defined under ``applicationConfig`` which can be modified or overridden at deployment time. @@ -306,15 +233,7 @@ Below is a table of default hostnames and ports for DCAE component service endpo HV-VES dcae-hv-ves-collector:6061 dcae-hv-ves-collector.onap:30222 TCA-Gen2 dcae-tcagen2:9091 NA PRH dcae-prh:8100 NA - Policy Handler policy-handler:25577 NA - Deployment Handler deployment-handler:8443 NA - Inventory inventory:8080 NA - Config binding config-binding-service:10000/10001 NA - DCAE Healthcheck dcae-healthcheck:80 NA DCAE MS Healthcheck dcae-ms-healthcheck:8080 NA - Cloudify Manager dcae-cloudify-manager:80 NA - DCAE Dashboard dashboard:8443 dashboard:30418 - DCAE mongo dcae-mongo-read:27017 NA =================== ================================== ======================================================= In addition, a number of ONAP service endpoints that are used by DCAE components are listed as follows @@ -323,15 +242,10 @@ for reference by DCAE developers and testers: ==================== ============================ ================================ Component Cluster Internal (host:port) Cluster external (svc_name:port) ==================== ============================ ================================ - Consul Server consul-server-ui:8500 NA Robot robot:88 robot:30209 TCP Message router message-router:3904 NA Message router message-router:3905 message-router-external:30226 Message router Kafka message-router-kafka:9092 NA - MSB Discovery msb-discovery:10081 msb-discovery:30281 - Logging log-kibana:5601 log-kibana:30253 - AAI aai:8080 aai:30232 - AAI aai:8443 aai:30233 ==================== ============================ ================================ Uninstalling DCAE @@ -341,7 +255,7 @@ All of the DCAE components deployed using the OOM Helm charts will be deleted by the ``helm undeploy`` command. This command can be used to uninstall all of ONAP by undeploying the top-level Helm release that was created by the ``helm deploy`` command. The command can also be used to -uninstall just DCAE, by having the command undeploy the `top_level_release_name`-``dcaegen2`` +uninstall just DCAE, by having the command undeploy the `top_level_release_name`-``dcaegen2-services`` Helm sub-release. Helm will undeploy only the components that were originally deployed using @@ -353,22 +267,6 @@ used for the deployment (typically ``onap``) after running the undeploy operation. Deleting the namespace will get rid of any remaining resources in the namespace, including the components deployed by Cloudify Manager. -When uninstalling DCAE alone, deleting the namespace would delete the -rest of ONAP as well. To delete DCAE alone, and to make sure all of the -DCAE components deployed by Cloudify Manager are uninstalled: - -* Find the Cloudify Manager pod identifier, using a command like: - - ``kubectl -n onap get pods | grep dcae-cloudify-manager`` -* Execute the DCAE cleanup script on the Cloudify Manager pod, using a command like: - - ``kubectl -n onap exec`` `cloudify-manager-pod-id` ``-- /scripts/dcae-cleanup.sh`` -* Finally, run ``helm undeploy`` against the DCAE Helm subrelease - -The DCAE cleanup script uses Cloudify Manager and the DCAE Kubernetes -plugin to instruct Kubernetes to delete the components deployed by Cloudify -Manager. This includes the components deployed when the DCAE bootstrap -service ran and any components deployed after bootstrap. To undeploy the DCAE services deployed via Helm (the hv-ves-collector, ves-collector, tcagen2, and prh), use the ``helm undeploy`` command against the `top_level_release_name`-``dcaegen2-services`` diff --git a/docs/sections/sdk/apis.rst b/docs/sections/sdk/apis.rst index 823027f2..80d1489d 100644 --- a/docs/sections/sdk/apis.rst +++ b/docs/sections/sdk/apis.rst @@ -8,6 +8,7 @@ Available APIs .. toctree:: :depth: 3 +.. _config_binding_service_sdk: cbs-client - a Config Binding Service client -------------------------------------------- diff --git a/docs/sections/services/bbs-event-processor/development_info.rst b/docs/sections/services/bbs-event-processor/development_info.rst deleted file mode 100644 index 4f4b4b1b..00000000 --- a/docs/sections/services/bbs-event-processor/development_info.rst +++ /dev/null @@ -1,30 +0,0 @@ -Compiling BBS-EP -================ - -BBS-ep is a sub-project of dcaegen2/services (inside components directory). -To build just the BBS-ep component, run the following maven command from within **components/bbs-event-processor** directory -`mvn clean install` - - -API Endpoints -============= - -Running with dev-mode of BBS-EP - - Heartbeat: **GET http://<container_address>:8100/heartbeat** - - Start Polling for events: **POST http://<container_address>:8100/start-tasks** - - Stop Polling for events: **POST http://<container_address>:8100/cancel-tasks** - - Execute just one polling for PNF re-registration internal events: **POST http://<container_address>:8100/poll-reregistration-events** - - Execute just one polling for CPE authentication events: **POST http://<container_address>:8100/poll-cpe-authentication-events** - - Change application logging level: **POST http://<container_address>:8100/logging/{level}** - -More detailed API specifications can be found in :doc:`../../apis/swagger-bbs-event-processor`. - -Maven GroupId: -============== - -org.onap.dcaegen2.services.components - -Maven Parent ArtifactId: -======================== - -org.onap.oparen:oparent:1.2.3
\ No newline at end of file diff --git a/docs/sections/services/bbs-event-processor/functionality.rst b/docs/sections/services/bbs-event-processor/functionality.rst deleted file mode 100644 index 8ddf18d2..00000000 --- a/docs/sections/services/bbs-event-processor/functionality.rst +++ /dev/null @@ -1,13 +0,0 @@ -Functionality -============= - -PNF re-registration processing logic - -.. image:: ../../images/bbs-ep-pnf-relocation.png - -CPE authentication processing logic - -.. image:: ../../images/bbs-ep-cpe-authentication.png - -For more details about the exact flows and where BBS-EP fits in the overall BBS use case flows, visit -* https://wiki.onap.org/display/DW/BBS+Notifications
\ No newline at end of file diff --git a/docs/sections/services/bbs-event-processor/index.rst b/docs/sections/services/bbs-event-processor/index.rst deleted file mode 100644 index f9fc2d8b..00000000 --- a/docs/sections/services/bbs-event-processor/index.rst +++ /dev/null @@ -1,31 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - -================== -BBS-EventProcessor -================== - - -Overview -======== - -BBE-ep is responsible for handling two types of events for the BBS use case. - -First are PNF re-registration internal events published by PRH. BBS-ep must process these internal events to understand if they -actually constitute ONT(CPE) relocation events. In the relocation case, it publishes an event towards unauthenticated.DCAE_CL_OUTPUT -DMaaP topic to trigger further Policy actions related to BBS use case. - -Second type of events are CPE authentication events originally published by the Edge SDN M&C component of BBS use case architecture. -Through RestConf-Collector or VES-Collector, these events are consumed by BBS-ep and they are forwared towards unauthenticated.DCAE_CL_OUTPUT -DMaaP topic to trigger further Policy actions related to BBS use case. - -BBE-ep periodically polls for the two events. Polling interval is configurable and can be changed dynamically from Consul. Its implementation -is based on Reactive Streams (Reactor library), so it is fully asynchronous and non-blocking. - - -.. toctree:: - :maxdepth: 1 - - ./installation - ./functionality - ./development_info diff --git a/docs/sections/services/bbs-event-processor/installation.rst b/docs/sections/services/bbs-event-processor/installation.rst deleted file mode 100644 index dcb8269e..00000000 --- a/docs/sections/services/bbs-event-processor/installation.rst +++ /dev/null @@ -1,90 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. _bbs-installation: - -Installation -============ - -BBS-ep is delivered as a Spring-Boot application ready to be deployed in Docker (via docker-compose). - -The following docker-compose-yaml file shows a default configuration. The file can be run using `docker compose up` command: - -.. code-block:: yaml - - version: '3' - services: - bbs-event-processor: - image: onap/org.onap.dcaegen2.services.components.bbs-event-processor:latest - container_name: bbs-event-processor - hostname: bbs-event-processor - ports: - - 32100:8100 - environment: - CONFIGS_DMAAP_CONSUMER_RE-REGISTRATION_DMAAPHOSTNAME: 10.133.115.190 - CONFIGS_DMAAP_CONSUMER_RE-REGISTRATION_DMAAPPORTNUMBER: 30227 - CONFIGS_DMAAP_CONSUMER_RE-REGISTRATION_DMAAPTOPICNAME: /events/unauthenticated.PNF_UPDATE - CONFIGS_DMAAP_CONSUMER_RE-REGISTRATION_CONSUMERGROUP: foo - CONFIGS_DMAAP_CONSUMER_RE-REGISTRATION_CONSUMERID: bar - CONFIGS_DMAAP_CONSUMER_CPE-AUTHENTICATION_DMAAPHOSTNAME: 10.133.115.190 - CONFIGS_DMAAP_CONSUMER_CPE-AUTHENTICATION_DMAAPPORTNUMBER: 30227 - CONFIGS_DMAAP_CONSUMER_CPE-AUTHENTICATION_DMAAPTOPICNAME: /events/unauthenticated.CPE_AUTHENTICATION - CONFIGS_DMAAP_CONSUMER_CPE-AUTHENTICATION_CONSUMERGROUP: foo - CONFIGS_DMAAP_CONSUMER_CPE-AUTHENTICATION_CONSUMERID: bar - CONFIGS_DMAAP_PRODUCER_DMAAPHOSTNAME: 10.133.115.190 - CONFIGS_DMAAP_PRODUCER_DMAAPPORTNUMBER: 30227 - CONFIGS_DMAAP_PRODUCER_DMAAPTOPICNAME: /events/unauthenticated.DCAE_CL_OUTPUT - CONFIGS_AAI_CLIENT_AAIHOST: 10.133.115.190 - CONFIGS_AAI_CLIENT_AAIPORT: 30233 - CONFIGS_APPLICATION_PIPELINESPOLLINGINTERVALSEC: 30 - CONFIGS_APPLICATION_PIPELINESTIMEOUTSEC: 15 - CONFIGS_APPLICATION_RE-REGISTRATION_POLICYSCOPE: policyScope - CONFIGS_APPLICATION_RE-REGISTRATION_CLCONTROLNAME: controlName - CONFIGS_APPLICATION_CPE-AUTHENTICATION_POLICYSCOPE: policyScope - CONFIGS_APPLICATION_CPE-AUTHENTICATION_CLCONTROLNAME: controlName - CONFIGS_SECURITY_TRUSTSTOREPATH: KeyStore.jks - CONFIGS_SECURITY_TRUSTSTOREPASSWORDPATH: KeyStorePass.txt - CONFIGS_SECURITY_KEYSTOREPATH: KeyStore.jks - CONFIGS_SECURITY_KEYSTOREPASSWORDPATH: KeyStorePass.txt - LOGGING_LEVEL_ORG_ONAP_BBS: TRACE - -BBS-ep can be dynamically deployed in DCAE’s Cloudify environment via its blueprint deployment artifact. - -Blueprint can be found in - - https://git.onap.org/dcaegen2/platform/blueprints/tree/blueprints/k8s-bbs-event-processor.yaml - -Steps to deploy are shown below - -- Enter the Bootstrap POD -- Validate blueprint - .. code-block:: bash - - cfy blueprints validate /blueprints/k8s-bbs-event-processor.yaml -- Upload validated blueprint - .. code-block:: bash - - - cfy blueprints upload -b bbs-ep /blueprints/k8s-bbs-event-processor.yaml -- Create deployment - .. code-block:: bash - - - cfy deployments create -b bbs-ep -i /blueprints/k8s-bbs-event-processor.yaml bbs-ep -- Deploy blueprint - .. code-block:: bash - - - cfy executions start -d bbs-ep install - -To undeploy BBS-ep, steps are shown below - -- Uninstall running BBS-ep and delete deployment - .. code-block:: bash - - - cfy uninstall bbs-ep -- Delete blueprint - .. code-block:: bash - - - cfy blueprints delete bbs-ep
\ No newline at end of file diff --git a/docs/sections/services/serviceindex.rst b/docs/sections/services/serviceindex.rst index 386215dd..1701f0fb 100644 --- a/docs/sections/services/serviceindex.rst +++ b/docs/sections/services/serviceindex.rst @@ -25,7 +25,6 @@ Event Processor .. toctree::
:maxdepth: 1
- ./bbs-event-processor/index.rst
./datalake-handler/index.rst
./mapper/index.rst
./pm-mapper/index.rst
diff --git a/docs/sections/services/snmptrap/installation.rst b/docs/sections/services/snmptrap/installation.rst index d134a895..9c549948 100644 --- a/docs/sections/services/snmptrap/installation.rst +++ b/docs/sections/services/snmptrap/installation.rst @@ -24,7 +24,7 @@ configuration assets to instantiated containers as needed. Also required is a working DMAAP/MR environment. trapd publishes traps to DMAAP/MR as JSON messages and expects the host -resources and publishing credentials to be included in the *Config Binding Service* +resources and publishing credentials to be included in the *Config Binding Service* config. Installation @@ -33,7 +33,7 @@ Installation The following command will download the latest trapd container from nexus and launch it in the container named "trapd": - ``docker run --detach -t --rm -p 162:6162/udp -P --name=trapd nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.snmptrap:2.0.3 ./bin/snmptrapd.sh start`` + ``docker run --detach -t --rm -p 162:6162/udp -P --name=trapd nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.snmptrap:2.0.6 ./bin/snmptrapd.sh start`` Running an instance of **trapd** will result in arriving traps being published to the topic specified by Config Binding Services. |
