diff options
| author |   Michael Hwang <mhwang@research.att.com> | 2017-11-06 14:52:37 -0500 |
|---|---|---|
| committer | Michael Hwang <mhwang@research.att.com> | 2017-11-06 14:53:29 -0500 |
| commit | 3d2b2aed3172d48325ed91ee6c1546c5172bac5e (patch) | |
| tree | ec0ae314c46c33998b11ae7a1e4e48d0720425ee | |
| parent | a73548f4eb6ba4bdb2b2064c7e24a4b74b467f5a (diff) | |
Merge in accumulated changes
Change-Id: I3cd9da556c31efe879f0ff1eaf48f63e4fc353aa
Issue-Id: DCAEGEN2-189
Signed-off-by: Michael Hwang <mhwang@research.att.com>
| -rw-r--r-- | platformdoc/README.md | 12 | ||||
| -rw-r--r-- | platformdoc/docs/architecture/services.md | 7 | ||||
| -rw-r--r-- | platformdoc/docs/components/component-specification/cdap-specification.md | 70 | ||||
| -rw-r--r-- | platformdoc/docs/components/component-specification/common-specification.md | 189 | ||||
| -rw-r--r-- | platformdoc/docs/components/component-specification/configuration-grid.md | 13 | ||||
| -rw-r--r-- | platformdoc/docs/components/component-specification/docker-specification.md | 207 | ||||
| -rw-r--r-- | platformdoc/docs/components/component-specification/start-here.md | 21 | ||||
| -rw-r--r-- | platformdoc/docs/components/component-specification/streams-grid.md | 60 | ||||
| -rw-r--r-- | platformdoc/docs/components/component-type-docker.md | 19 | ||||
| -rw-r--r-- | platformdoc/docs/components/data-formats.md | 12 | ||||
| -rw-r--r-- | platformdoc/docs/components/dcae-cli/quickstart.md | 2 | ||||
| -rw-r--r-- | platformdoc/docs/components/dcae-cli/walkthrough.md | 47 | ||||
| -rw-r--r-- | platformdoc/docs/components/intro.md | 60 | ||||
| -rw-r--r-- | platformdoc/docs/index.md | 2 | ||||
| -rw-r--r-- | platformdoc/mkdocs.yml | 20 |
15 files changed, 476 insertions, 265 deletions
diff --git a/platformdoc/README.md b/platformdoc/README.md index 917c087f..ea5739ab 100644 --- a/platformdoc/README.md +++ b/platformdoc/README.md @@ -20,6 +20,7 @@ This is a [Mkdocs](http://www.mkdocs.org/) project. To serve a local version of ``` 2. Clone this repo + 3. Run the following at the root of the cloned repo: ``` @@ -35,19 +36,18 @@ This is a [Mkdocs](http://www.mkdocs.org/) project. To serve a local version of mkdocs build ``` -2. Build and push the Docker image - note the repository and group `YOUR_NEXUS_DOCKER_REGISTRY/onap` can be customized and replaced: +2. Build the Docker image, tag as desired - eg `dcae-platform-documentation:YYYYMMDD` ``` - docker build -t YOUR_NEXUS_DOCKER_REGISTRY/onap/dcae-platform-documentation:latest . - docker push YOUR_NEXUS_DOCKER_REGISTRY/onap/dcae-platform-documentation:latest + docker build -t dcae-platform-documentation:20171002 . ``` 3. Run the Docker container: ``` export DOCKER_HOST=tcp://<target docker host> - # REVIEW: Does this always pull latest? - docker pull YOUR_NEXUS_DOCKER_REGISTRY/onap/dcae-platform-documentation:latest - docker run -d --name dpd -p 80:80 YOUR_NEXUS_DOCKER_REGISTRY/onap/dcae-platform-documentation:latest + docker stop dpd + docker rm -f dpd + docker run -d --name dpd -p 80:80 dcae-platform-documentation:YYYYMMDD ``` diff --git a/platformdoc/docs/architecture/services.md b/platformdoc/docs/architecture/services.md new file mode 100644 index 00000000..985953c3 --- /dev/null +++ b/platformdoc/docs/architecture/services.md @@ -0,0 +1,7 @@ +# 'Services' Overview + +DCAE Services run on the DCAE platform. A service performs either monitors the virtualized network services or does analytics. A service is composed of one or more components, and performs a business need. + +Services are created in a 'Service Design and Creation' tool called 'Service Assurance Flow Designer' by a Service Designer. This is done by +* 1. 'compose'ing a service from the available SDC catalog components (actually from the TOSCA models representing the components), then +* 2. 'submit'ing the service, which generates a Cloudify blueprint, is then automatically uploaded to INVENTORY, and then deployed by DEPLOYMENT HANDLER (once a DTI Event is triggered for the service). It should be noted that some service flows, specifally 'closed-loop' flows, are not initiated by DTI, but are done by CLAMP. diff --git a/platformdoc/docs/components/component-specification/cdap-specification.md b/platformdoc/docs/components/component-specification/cdap-specification.md index 479cc944..d4d01406 100644 --- a/platformdoc/docs/components/component-specification/cdap-specification.md +++ b/platformdoc/docs/components/component-specification/cdap-specification.md @@ -1,31 +1,21 @@ # Component specification (CDAP) -## Overview +The CDAP component specification contains the following groups of information. Many of these are common to both CDAP and Docker components and are therefore described in the common specification. -This page contains details specific to CDAP applications. +* [Metadata](common-specification.md#metadata) +* [Interfaces](common-specification.md#interfaces) including the associated [Data Formats](/components/data-formats.md) +* [Parameters](#parameters) - for specifying parameters in your AppConfig, AppPreferences, and ProgramPreferences to the Designer and Policy. This of course is CDAP-specific and is described below. +* [Auxiliary Details](#auxiliary-details) +* [List of artifacts](common-specification.md#artifacts) -The component specification contains the following top-level groups of information: - -* Component [Metadata](#metadata) -* [Parameters](#Parameters): the section for specifying parameters in your AppConfig, AppPreferences, and ProgramPreferences to the Designer and Policy. -* [Interfaces](#interfaces): the connections from this component to other components -* [Auxiliary details](#auxiliary) -* [List of artifacts](#artifacts) - -Note: this page does not talk about [DCAE specific requirements](/components/component-type-cdap.md) that your component must adhere to. Please see that page for discussions about DMaaP, Policy, Metrics, and more. ## Current Limitations and TODOs -* Currently we only support CDAP applications that have a stream. * The integration of DMD is likely to significantly change the [Interfaces](#interfaces) section in this specification, see [DMaaP abstraction](/components/component-type-cdap.md#dmaap-abstraction). -## Metadata - -See [Metadata](common-specification.md#metadata) - ## Parameters -There is a `parameters` section in your component specification. This section contains three optional keys: [app_config](#appconfig), [app_preferences](#apppreferences), and [propram_preferences](#prorgram_preferences): +There is a `parameters` section in your component specification. This section contains three optional keys: [app_config](#appconfig), [app_preferences](#apppreferences), and [propram_preferences](#program_preferences): ``` "parameters" : { "app_config" : [ ...], @@ -42,21 +32,15 @@ There is a `parameters` section in your component specification. This section co * Despite the AppConfig being optional to *specify* in the case that you have no parameters in your AppConfig, it is *required for processing* in your application. That is because the DCAE platform will place important information into your AppConfig as discussed below. ### Parameter -We use the following definition of a _single parameter_, which is used in several places below: -``` - { - "name" : "param name", - "value" : "developer default", - "description" : "tell policy/ops what this does" - } -``` + +The following CDAP specific definitions use `param1` to refer to the common parameter layout in [Parameter](common-specification.md#parameters) ### AppConfig The `app_config` key refers to the [CDAP AppConfig](http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/configuration.html). It is expected to be a JSON: ``` "app_config" : [ // list of JSON - param1, // see Parameter above + param1, // common parameter layout ... ] ``` @@ -70,15 +54,15 @@ Unfortunately, at the time of writing, the AppConfig is a Java map of `string:st >>> ``` -The final AppConfig (after the designer and policy override parameter values) is passed into CDAP's AppConfig API when starting your application. +The final AppConfig (after the designer and policy override parameter values) is passed into CDAP's AppConfig API when starting the application. ### AppPreferences -In addition to the CDAP AppConfig, we support [Application Preferences](http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/preferences.html#set-preferences). +In addition to the CDAP AppConfig, the platform supports [Application Preferences](http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/preferences.html#set-preferences). The format of the `app_preferences` value is the same as the above: ``` "app_preferences" : [ // list of JSON - param1, // see Parameter above + param1, // common parameter layout ... ] ``` @@ -94,22 +78,20 @@ Preferences can also be specified [per program](http://docs.cask.co/cdap/current "program_id" : "program name 1", // the name of this CDAP program "program_type" : "e.g., flows", // "must be one of flows, mapreduce, schedules, spark, workflows, workers, or services", "program_pref" : [ // list of JSON - param1, // see Parameter above + param1, // common parameter layout ... ] }, - // repeat for each program you want to pass a preferences JSON to + // repeat for each program that receives a program_preferences JSON ] ``` Each `program_pref` JSON is passed into the CDAP API as the preference for `program_id`. -## Interfaces -See [Interfaces](common-specification.md#interfaces) NOTE: for CDAP, this section is very likely to change when DMD is available. The _future_ vision, as per [DMaaP intentionally abstracted](/components/component-type-cdap.md#dmaap-abstraction) is that you would publish your data as a series of files on HDFS, and DMD will pick them up and send them to the appropriate DMaaP feeds or directly when needed. -## Auxiliary +## Auxiliary Details `auxiliary` contains details about CDAP specific parameters. @@ -126,7 +108,7 @@ program_id | string | name of CDAP entity (eg "WhoFlow") Example: ```json -+"auxiliary": { +"auxiliary": { "streamname" : "who", "artifact_name" : "HelloWorld", "artifact_version" : "3.4.3", @@ -143,21 +125,3 @@ The `programs` key is identical to the `program_preferences` key discussed [abov * each JSON in the list does not contain `program_pref` * this is required! You must include all of your programs in this, as it is used to start each program as well as for DCAE to perform periodic healthchecks on your application. Don't forget about your services; they are programs too. -+ -## Artifacts - -`artifacts` contains a list of artifacts associated with this component. This is where you specify your CDAP jar. - -Property Name | Type | Description -------------- | ---- | ----------- -artifacts | JSON array | Each entry is a artifact object - -Each artifact object has: - -Property Name | Type | Description -------------- | ---- | ----------- -uri | string | *Required*. Uri to the artifact -type | string | *Required*. For CDAP, use `jar` - -This file is uploading using the CLI tool at the same time as your component specification. - diff --git a/platformdoc/docs/components/component-specification/common-specification.md b/platformdoc/docs/components/component-specification/common-specification.md index 77f2a7f0..06f7f76c 100644 --- a/platformdoc/docs/components/component-specification/common-specification.md +++ b/platformdoc/docs/components/component-specification/common-specification.md @@ -1,11 +1,11 @@ -# Component specification (Common elements) +# Common Elements of the Component Specification -This page describes component specification sections that are common (or nearly) to both Docker and CDAP. +This page describes the component specification (JSON) sections that are common to both Docker and CDAP components. Differences for each are pointed out below. Elements that are very different, and described in the CDAP or Docker specific pages. ## Component Metadata -Metadata refers to the properties found under the `self` JSON. This group of properties are used to uniquely identify this component specification and identify the component that this specification is used to capture. The metadata section, represented under `self`, is used to uniquely identify your component among all components in the catalog. +Metadata refers to the properties found under the `self` JSON. This group of properties is used to uniquely identify this component specification and identify the component that this specification is used to capture. -From the specification example above: +Example: ``` "self": { @@ -16,26 +16,27 @@ From the specification example above: }, ``` -Here is a breakdown of the schema: +`self` Schema: Property Name | Type | Description ------------- | ---- | ----------- version | string | *Required*. Semantic version for this specification -name | string | *Required*. Full name of this component which is also used as this component's catalog id. The name includes a namespace that is dot-separated. -description | string | Human-readable text blurb describing the component and the components functional purpose. +name | string | *Required*. Full name of this component which is also used as this component's catalog id. +description | string | *Required*. Human-readable text describing the component and the components functional purpose. component_type | string | *Required*. Identify what containerization technology this component uses: `docker` or `cdap`. ## Interfaces Interfaces are the JSON objects found under the `streams` key and the `services` key. These are used to describe the interfaces that the component uses and the interfaces that the component provides. The description of each interface includes the associated [data format](/components/data-formats.md). ### Streams - * The `streams` JSON is for specifying that you produce data that can be consumed by other components, and the streams you expect to subscribe to produced by other components. These are "fire and forget" type interfaces where the publisher of a stream does not expect or parse a response from the subscriber. -* The term `stream` here is abstract and neither refers to "CDAP streams" or "DMaaP feeds": while a stream is very likely a DMaaP feed, it could be a direct stream of data being routed via HTTP too. It abstractly refers to a sequence of data leaving a publisher. + * The `streams` JSON is for specifying data produced for consumption by other components, and the streams expected to subscribe to that is produced by other components. These are "fire and forget" type interfaces where the publisher of a stream does not expect or parse a response from the subscriber. +* The term `stream` here is abstract and neither refers to "CDAP streams" or "DMaaP feeds". While a stream is very likely a DMaaP feed, it could be a direct stream of data being routed via HTTP too. It abstractly refers to a sequence of data leaving a publisher. * Streams have anonymous publish/subscribe semantics, which decouples the production of information from its consumption. * In general, components are not aware of who they are communicating with. -* Instead, components that are interested in data subscribe to the relevant stream; components that generate data publish to the relevant stream. +* Instead, components that are interested in data, subscribe to the relevant stream; components that generate data publish to the relevant stream. * There can be multiple publishers and subscribers to a stream. Streams are intended for unidirectional, streaming communication. + Streams interfaces that implement an HTTP endpoint must support POST. Streams are split into: @@ -47,7 +48,7 @@ publishes | JSON list | *Required*. List of all stream interfaces that this com #### Subscribes -From the example specification: +Example: ```json "streams": { @@ -63,18 +64,20 @@ From the example specification: This describes that `asimov.component.kpi_anomaly` exposes an HTTP endpoint called `/data` which accepts requests that have the data format of `dcae.vnf.kpi` version `1.0.0`. -The JSON object schema used in `subscribes`: +`subscribes` Schema: Property Name | Type | Description ------------- | ---- | ----------- format | string | *Required*. Data format id of the data format that is used by this interface version | string | *Required*. Data format version of the data format that is used by this interface -route | string | *Required*. The HTTP route that this interface listens on +route | string | *Required for HTTP and data router*. The HTTP route that this interface listens on +config_key | string | *Required for message_router and data router*. The HTTP route that this interface listens on type | string | *Required*. Type of stream: `http`, `message_router`, `data_router` + ##### Message router -Message router subscribers are http clients rather than http services and performs a http `GET` call. Thus, message router subscribers description is structured like message router publishers and requires `config_key`: +Message router subscribers are http clients rather than http services and performs a http a `GET` call. Thus, message router subscribers description is structured like message router publishers and requires `config_key`: ```json "streams": { @@ -90,7 +93,7 @@ Message router subscribers are http clients rather than http services and perfor ##### Data router -Data router subscribers are http or https services that handle `PUT` requests from data router. Developers must provide the `route` or url path/endpoint that is expected to handle data router requests. This will be used to construct the delivery url needed to register your subscriber to the provisioned feed. Developers must also provide a `config_key` because there is dynamic configuration information associated with the feed that your application will need e.g. username and password. See the page on [DMaaP connection objects](../dcae-cli/dmaap-connection-objects) for more details on the configuration information. +Data router subscribers are http or https services that handle `PUT` requests from data router. Developers must provide the `route` or url path/endpoint that is expected to handle data router requests. This will be used to construct the delivery url needed to register the subscriber to the provisioned feed. Developers must also provide a `config_key` because there is dynamic configuration information associated with the feed that the application will need e.g. username and password. See the page on [DMaaP connection objects](../dcae-cli/dmaap-connection-objects) for more details on the configuration information. Example (not tied to the larger example): @@ -109,7 +112,7 @@ Example (not tied to the larger example): #### Publishes -From the example specification: +Example: ```json "streams": { @@ -126,18 +129,36 @@ From the example specification: This describes that `asimov.component.kpi_anomaly` publishes by making POST requests to streams that support the data format `asimov.format.integerClassification` version `1.0.0`. -The JSON object schema used in `publishes`: +`publishes` Schema: Property Name | Type | Description ------------- | ---- | ----------- format | string | *Required*. Data format id of the data format that is used by this interface version | string | *Required*. Data format version of the data format that is used by this interface -config_key | string | *Required*. The JSON key in the generated application configuration that will be used to pass the downstream component connection information. -type | string | *Required*. Type of stream: `http`, `message router` +config_key | string | *Required*. The JSON key in the generated application configuration that will be used to pass the downstream component's (the subscriber's) connection information. +type | string | *Required*. Type of stream: `http`, `message router`, `data router` + +##### Message router + +Message router publishers are http clients of DMaap message_router. Developers must provide a `config_key` because there is dynamic configuration information associated with the feed that the application will need to receive e.g. topic url, username, password. See the page on [DMaaP connection objects](../dcae-cli/dmaap-connection-objects/#message_router) for more details on the configuration information. + +Example (not tied to the larger example): + +```json +"streams": { +... + "publishes": [{ + "config_key": "some-pub-mr", + "format": "sandbox.platform.any", + "type": "message_router", + "version": "0.1.0" + }] +} +``` ##### Data router -Data router publishers are http clients that make `PUT` requests to data router. Developers must also provide a `config_key` because there is dynamic configuration information associated with the feed that your application will need to receive e.g. publish url, username, password. See the page on [DMaaP connection objects](../dcae-cli/dmaap-connection-objects) for more details on the configuration information. +Data router publishers are http clients that make `PUT` requests to data router. Developers must also provide a `config_key` because there is dynamic configuration information associated with the feed that the application will need to receive e.g. publish url, username, password. See the page on [DMaaP connection objects](../dcae-cli/dmaap-connection-objects) for more details on the configuration information. Example (not tied to the larger example): @@ -153,6 +174,11 @@ Example (not tied to the larger example): } ``` +#### Quick Reference + +Refer to this [Quick Reference](/components/component-specification/streams-grid.md) for a comparison of the Streams 'Publishes' and 'Subscribes' sections. + + ### Services * The publish / subscribe model is a very flexible communication paradigm, but its many-to-many one-way transport is not appropriate for RPC @@ -167,10 +193,10 @@ calls | JSON list | *Required*. List of all service interfaces that this compon provides | JSON list | *Required*. List of all service interfaces that this component exposes and provides #### Calls -The JSON `services/calls` is for specifying that your component relies on an HTTP(S) service---your component sends that service an HTTP request, and that service responds with an HTTP reply. +The JSON `services/calls` is for specifying that the component relies on an HTTP(S) service---the component sends that service an HTTP request, and that service responds with an HTTP reply. An example of this is how string matching (SM) depends on the AAI Broker. SM performs a synchronous REST call to the AAI broker, providing it the VMNAME of the VNF, and the AAI Broker responds with additional details about the VNF. This dependency is expressed via `services/calls`. In contrast, the output of string matching (the alerts it computes) is sent directly to policy as a fire-and-forget interface, so that is an example of a `stream`. -From the example specification: +Example: ```json "services": { @@ -191,7 +217,7 @@ From the example specification: This describes that `asimov.component.kpi_anomaly` will make HTTP calls to a downstream component that accepts requests of data format `dcae.vnf.meta` version `1.0.0` and is expecting the response to be `dcae.vnf.kpi` version `1.0.0`. -The JSON object schema used in `calls`: +`calls` Schema: Property Name | Type | Description ------------- | ---- | ----------- @@ -208,7 +234,7 @@ version | string | *Required*. Data format version of the data format that is u #### Provides -From the example specification: +Example: ```json "services": { @@ -229,7 +255,7 @@ From the example specification: This describes that `asimov.component.kpi_anomaly` provides a service interface and it is exposed on the `/score-vnf` HTTP endpoint. The endpoint accepts requests that have the data format `dcae.vnf.meta` version `1.0.0` and gives back a response of `asimov.format.integerClassification` version `1.0.0`. -The JSON object schema used in `provides`: +`provides` Schema for a Docker component: Property Name | Type | Description ------------- | ---- | ----------- @@ -256,3 +282,116 @@ Note, for CDAP, there is a slight variation due to the way CDAP exposes services } ] ``` + +`provides` Schema for a CDAP component: + +Property Name | Type | Description +------------- | ---- | ----------- +request | JSON object | *Required*. Description of the expected request data format for this interface +response | JSON object | *Required*. Description of the expected response for this interface +service_name | string | *Required*. The CDAP service name (eg "Greeting") +service_endpoint | string | *Required*. The CDAP service endpoint for this service_name (eg "/greet") +verb | string | *Required*. 'GET', 'PUT' or 'POST' + + +## Parameters + +`parameters` is where to specify the component's application configuration parameters that are not connection information. + +Property Name | Type | Description +------------- | ---- | ----------- +parameters | JSON array | Each entry is a parameter object + +Parameter object has the following available properties: + +Property Name | Type | Description | Default +------------- | ---- | ----------- | ------- +name | string | *Required*. The property name that will be used as the key in the generated config | +value | any | *Required*. The default value for the given parameter | +description | string | *Required*. Human-readable text describing the parameter like what its for | +type | string | The required data type for the parameter | +required | boolean | An optional key that declares a parameter as required (true) or not (false) | true +constraints | array | The optional list of sequenced constraint clauses for the parameter. See below | +entry_schema | string | The optional key that is used to declare the name of the Datatype definition for entries of set types such as the TOSCA 'list' or 'map'. Only 1 level is supported at this time | +designer_editable | boolean | An optional key that declares a parameter to be editable by designer (true) or not (false) | true +sourced_at_deployment | boolean | An optional key that declares a parameter's value to be assigned at deployment time (true) | false +policy_editable | boolean | An optional key that declares a parameter to be editable by policy (true) or not (false) | true +policy_schema | array | The optional list of schema definitions used for policy. See below | + +Example: + +```json +"parameters": [ + { + "name": "threshold", + "value": 0.75, + "description": "Probability threshold to exceed to be anomalous" + } +] +``` + +Many of the parameter properties have been copied from TOSCA model property definitions and are to be used for service design composition and policy creation. See [section 3.5.8 *Property definition*](http://docs.oasis-open.org/tosca/TOSCA-Simple-Profile-YAML/v1.1/TOSCA-Simple-Profile-YAML-v1.1.html). + +The property `constraints` is a list of objects where each constraint object: + +Property Name | Type | Description +------------- | ---- | ----------- +equal | | Constrains a property or parameter to a value equal to (‘=’) the value declared +greater_than | number | Constrains a property or parameter to a value greater than (‘>’) the value declared +greater_or_equal | number | Constrains a property or parameter to a value greater than or equal to (‘>=’) the value declared +less_than | number | Constrains a property or parameter to a value less than (‘<’) the value declared +less_or_equal | number | Constrains a property or parameter to a value less than or equal to (‘<=’) the value declared +valid_values | array | Constrains a property or parameter to a value that is in the list of declared values +length | number | Constrains the property or parameter to a value of a given length +min_length | number | Constrains the property or parameter to a value to a minimum length +max_length | number | Constrains the property or parameter to a value to a maximum length + +`threshold` is the configuration parameter and will get set to 0.75 when the configuration gets generated. + +The property `policy_schema` is a list of objects where each policy_schema object: + +Property Name | Type | Description | Default +------------- | ---- | ----------- | ------- +name | string | *Required*. parameter name +value | string | default value for the parameter +description | string | parameter description +type | enum | *Required*. data type of the parameter, 'string', 'number', 'boolean', 'datetime', 'list', or 'map' +required | boolean | is parameter required or not? | true +constraints | array | The optional list of sequenced constraint clauses for the parameter. See above | +entry_schema | string | The optional key that is used to declare the name of the Datatype definition for certain types. entry_schema must be defined when the type is either list or map. If the type is list and the entry type is a simple type (string, number, bookean, datetime), follow with an string to describe the entry | +| | If the type is list and the entry type is a map, follow with an array to describe the keys for the entry map | +| | If the type is list and the entry type is a list, that is not currently supported +| | If the type is map, follow with an aray to describe the keys for the map | + +## Generated Application Configuration + +The above example for component `asimov.component.kpi_anomaly` will get transformed into the following application configuration JSON that is fully resolved and provided at runtime by calling the `config binding service`: + +```json +{ + "streams_publishes": { + "prediction": ["10.100.1.100:32567"] + }, + "streams_subscribes": {}, + "threshold": 0.75, + "services_calls": { + "vnf-db": ["10.100.1.101:32890"] + } +} +``` + +## Artifacts + +`artifacts` contains a list of artifacts associated with this component. For Docker, this is the full path (including the registry) to the Docker image. For CDAP, this is the full path to the CDAP jar. + +Property Name | Type | Description +------------- | ---- | ----------- +artifacts | JSON array | Each entry is a artifact object + +`artifact` Schema: + +Property Name | Type | Description +------------- | ---- | ----------- +uri | string | *Required*. Uri to the artifact, full path +type | string | *Required*. `docker image` or `jar` + diff --git a/platformdoc/docs/components/component-specification/configuration-grid.md b/platformdoc/docs/components/component-specification/configuration-grid.md new file mode 100644 index 00000000..e9054593 --- /dev/null +++ b/platformdoc/docs/components/component-specification/configuration-grid.md @@ -0,0 +1,13 @@ + +# Configuration Quick Reference + +The following types of configuration are supported by the DCAE Platform. + +|Input Sources|Default Values|Designer Input |Clamp Input|Policy Input|Runtime Input| +|---------|---------|----------|---------|----------|---| +|Notes||This applies only to components that are self-service (supported by SDC) |This applies only to components that are part of a closed-loop interface || | +|Who provides?|Component Developer|Service Designer|CLAMP|Operations|Runtime Platform| +| When/Where it is provided |During onboarding – in the component specification | At design time – in the SDC UI | At installation – in the CLAMP UI | Anytime – in the POLICY GUI | When the component is deployed | +|Component Specification Details|For CDAP:<br> ‘value’ Name and KV pairs in AppConfig or AppPreferences For Docker:<br> ‘value’ is provided for variable in ‘parameter’ section|‘designer-editable’ must be set to ‘true’ for variable in ‘parameter’ section. || ‘policy-editable’ must be set to ‘true’ and ‘policy_schema’ must be provided for variable in ‘parameter’ section |'sourced_at_<br>deployment' must be set to 'true' for variable in 'parameter' section | +| How it is used | This is passed to the component in the generated configuration if not overridden.|This overrides any values previously set, but can be overridden by CLAMP or POLICY.|This overrides any values previously set, but can be overridden by POLICY.|This overrides any values previously set, but can be overridden at any point thereafter.|This overrides any values previously set, but can be overridden at any point thereafter by Policy. | +| Additional Info for Component Developer|For CDAP:<br> ‘value’ is provided for variable in the ‘AppConfig’ or ‘AppPreferences’ sections<br><br> For Docker:<br> ‘value’ is provided for variable in ‘parameter’ section|||For Docker: In the auxiliary section:<br> {"policy": {"trigger_type": "policy","script_path": “/opt/app/reconfigure.sh”} }<br> Script interface must be "opt/app/reconfigure.sh” $trigger_type $updated_policies $updated_appl_config" <br> where $updated_policies is a json provided by the Policy Handler and <br> $update_appl_config is the post-merged appl config which may contain unresolved configuration that didn’t come from policy.<br> Suggestion is for script to call CONFIG BINDING SERVICE to resolve any configuration. | diff --git a/platformdoc/docs/components/component-specification/docker-specification.md b/platformdoc/docs/components/component-specification/docker-specification.md index 2bff6c47..79d3a88b 100644 --- a/platformdoc/docs/components/component-specification/docker-specification.md +++ b/platformdoc/docs/components/component-specification/docker-specification.md @@ -1,96 +1,39 @@ # Component specification (Docker) -This page contains details specific to Dockerized applications. +The Docker component specification contains the following groups of information. Many of these are common to both Docker and CDAP components and are therefore described in the common specification. -The component specification contains the following top-level groups of information: +* [Metadata](common-specification.md#metadata) +* [Interfaces](common-specification.md#interfaces) including the associated [Data Formats](/components/data-formats.md) +* [Parameters](common-specification.md#parameters) +* [Auxiliary Details](#auxiliary-details) +* [List of Artifacts](common-specification.md#artifacts) -* [Component metadata](#metadata) -* [Component interfaces](#interfaces) including the associated [data formats](/components/data-formats.md) -* [Configuration parameters](#configuration-parameters) -* [Auxiliary details](#auxilary) -* [List of artifacts](#artifacts) +## Auxiliary Details -## Metadata +`auxiliary` contains Docker specific details like health check, port mapping, volume mapping, and policy reconfiguration script details. -See [Metadata](common-specification.md#metadata) - -## Interfaces - -See [Interfaces](common-specification.md#interfaces) - -## Configuration parameters - -`parameters` is where to specify the component's application configuration parameters that are not connection information. - -Property Name | Type | Description -------------- | ---- | ----------- -parameters | JSON array | Each entry is a parameter object - -Parameter object has the following available properties: - -Property Name | Type | Description | Default -------------- | ---- | ----------- | ------- -name | string | *Required*. The property name that will be used as the key in the generated config | -value | any | *Required*. The default value for the given parameter | -description | string | *Required*. Human-readable text describing the parameter like what its for | -type | string | The required data type for the parameter | -required | boolean | An optional key that declares a parameter as required (true) or not (false) | true -constraints | array | The optional list of sequenced constraint clauses for the parameter | -entry_schema | string | The optional key that is used to declare the name of the Datatype definition for entries of set types such as the TOSCA list or map | -designer_editable | boolean | An optional key that declares a parameter to be editable by designer (true) or not (false) | true -policy_editable | boolean | An optional key that declares a parameter to be editable by policy (true) or not (false) | true -policy_schema | array | The optional list of schema definitions used for policy | - -Many of the parameter properties have been copied from TOSCA model property definitions and are to be used for service design composition and policy creation. See [section 3.5.8 *Property definition*](http://docs.oasis-open.org/tosca/TOSCA-Simple-Profile-YAML/v1.1/TOSCA-Simple-Profile-YAML-v1.1.html). - -The property `constraints` is a list of objects where each constraint object: - -Property Name | Type | Description -------------- | ---- | ----------- -equal | | Constrains a property or parameter to a value equal to (‘=’) the value declared -greater_than | number | Constrains a property or parameter to a value greater than (‘>’) the value declared -greater_or_equal | number | Constrains a property or parameter to a value greater than or equal to (‘>=’) the value declared -less_than | number | Constrains a property or parameter to a value less than (‘<’) the value declared -less_or_equal | number | Constrains a property or parameter to a value less than or equal to (‘<=’) the value declared -valid_values | array | Constrains a property or parameter to a value that is in the list of declared values -length | number | Constrains the property or parameter to a value of a given length -min_length | number | Constrains the property or parameter to a value to a minimum length -max_length | number | Constrains the property or parameter to a value to a maximum length - -From the example specification: - -```json -"parameters": [ - { - "name": "threshold", - "value": 0.75, - "description": "Probability threshold to exceed to be anomalous" - } -] -``` - -`threshold` is the configuration parameter and will get set to 0.75 when the configuration gets generated. - -## Auxiliary - -`auxilary` contains Docker specific details like health check and port mapping information. - -Property Name | Type | Description +Name | Type | Description ------------- | ---- | ----------- healthcheck | JSON object | *Required*. Health check definition details ports | JSON array | each array item maps a container port to the host port. See example below. +volume | JSON array | each array item contains a host and container object. See example below. | | +policy | JSON array | *Required*. Policy script details + +### Health Check Definition -### Health check definition +The platform uses Consul to perform periodic health check calls. Consul provides different types of [check definitions](https://www.consul.io/docs/agent/checks.html). The platform currently supports http and docker health checks. -The platform uses Consul to perform periodic health check calls. Consul provides different types of [check definitions](https://www.consul.io/docs/agent/checks.html). The platform currently supports http and docker health checks. +When choosing a value for interval, consider that too frequent healthchecks will put unnecessary load on Consul and DCAE. If there is a problematic resource, then more frequent healthchecks are warranted (eg 15s or 60s), but as stablility increases, so can these values, (eg 300s). -#### http +When choosing a value for timeout, consider that too small a number will result in increasing timeout failures, and too large a number will result in a delay in the notification of resource problem. A suggestion is to start with 5s and workd from there. + +#### http Property Name | Type | Description ------------- | ---- | ----------- type | string | *Required*. `http` -interval | string | Interval duration in seconds i.e. `15s` -timeout | string | Timeout in seconds i.e. `1s` +interval | string | Interval duration in seconds i.e. `60s` +timeout | string | Timeout in seconds i.e. `5s` endpoint | string | *Required*. GET endpoint provided by the component for Consul to call to check health Example: @@ -106,7 +49,7 @@ Example: } ``` -#### docker +#### docker script example Property Name | Type | Description ------------- | ---- | ----------- @@ -130,9 +73,7 @@ Example: } ``` -### Ports example - -Example: +### Ports ```json "auxilary": { @@ -142,23 +83,93 @@ Example: In the example above, container port 8080 maps to host port 8000. -## Artifacts +### Volume Mapping -`artifacts` contains a list of artifacts associated with this component. For Docker, this would be where you specify your Docker image full path including registry. +```json +"auxilary": { + "volumes": [ + { + "container": { + "bind": "/tmp/docker.sock", + "mode": "ro" + }, + "host": { + "path": "/var/run/docker.sock" + } + } + ] +} +``` + +At the top-level: Property Name | Type | Description ------------- | ---- | ----------- -artifacts | JSON array | Each entry is a artifact object +volumes | array | Contains container and host objects -Each artifact object has: +The `container` object contains: Property Name | Type | Description ------------- | ---- | ----------- -uri | string | *Required*. Uri to the artifact -type | string | *Required*. `docker image` or `jar` +bind | string | path to the container volume +mode | string | "ro" - indicates read-only volume + | | "" - indicates that the contain can write into the bind mount -## Example -Here is a full example of a component spec: +The `host` object contains: + +Property Name | Type | Description +------------- | ---- | ----------- +path | string | path to the host volume + +Here's an example of the minimal JSON that must be provided as an input: + +```json +"auxilary": { + "volumes": [ + { + "container": { + "bind": "/tmp/docker.sock" + }, + "host": { + "path": "/var/run/docker.sock" + } + } + ] +} +``` + +In the example above, the container volume "/tmp/docker.sock" maps to host volume "/var/run/docker.sock". + +### Policy + +Policy changes made in the Policy UI will be provided to the Docker component by triggering a script that is defined here. + +Property Name | Type | Description +------------- | ---- | ----------- +reconfigure_type | string | *Required*. Current value supported is `policy` +script_path | string | *Required*. Current value for 'policy' reconfigure_type must be "/opt/app/reconfigure.sh" + +Example: + +```json +"auxilary": { + "policy": { + "reconfigure_type": "policy", + "script_path": "/opt/app/reconfigure.sh" + } +} +``` + +The docker script interface is as follows: <br><br> `/opt/app/reconfigure.sh $reconfigure_type {"updated policies": <updated policies object>, "application config": <applcation config object>} + +Name | Type | Description +---- | ---- | ----------- +reconfigure_type | string | "policy" +updated_policies | json | TBD +updated_appl_config | json | complete generated app_config, not fully-resolved, but `policy-enabled` parameters have been updated. In order to get the complete updated app_config, the component would have to call `config-binding-service`. + + +## Docker Component Spec - Complete Example ```json { @@ -222,25 +233,9 @@ Here is a full example of a component spec: } }, "artifacts": [{ - "uri": "YOUR_NEXUS_DOCKER_REGISTRY/kpi_anomaly:1.0.0", + "uri": "fake.nexus.com/dcae/kpi_anomaly:1.0.0", "type": "docker image" }] } ``` -## Generate application configuration - -The above example `asimov.component.kpi_anomaly` will get transformed into the following application configuration JSON that is fully resolved and provided at runtime by calling the config binding service: - -```json -{ - "streams_publishes": { - "prediction": ["10.100.1.100:32567"] - }, - "streams_subscribes": {}, - "threshold": 0.75, - "services_calls": { - "vnf-db": ["10.100.1.101:32890"] - } -} -``` diff --git a/platformdoc/docs/components/component-specification/start-here.md b/platformdoc/docs/components/component-specification/start-here.md index 3816dd06..e69de29b 100644 --- a/platformdoc/docs/components/component-specification/start-here.md +++ b/platformdoc/docs/components/component-specification/start-here.md @@ -1,21 +0,0 @@ -# Component specification - -Every component that onboards onto the DCAE platform requires a component specification. The component specification is a JSON artifact that fully describes your components. The format of the component specification is standardized for CDAP applications and Dockerized applications and is validated using [a JSON schema](ONAP URL TBD). - -The component specification is used by: - -* Design - TOSCA models are generated from the component specification so that your component can be used by designers to compose new DCAE services in SDC. -* Policy - TOSCA models are generated from the component specification so that operations can create policy models used to dynamically configure your component. -* Runtime platform - Your component's application configuration (JSON) is generated from the component specification and will be provided to your component at runtime. - -## dcae-cli - -Use the [`dcae-cli`](../dcae-cli/quickstart) tool to manage your component specification and to test your components with it. - -The dcae-cli can also be used to view component specifications that have already been added and published. Please check out the [shared catalog](../dcae-cli/walkthrough/#shared-catalog) for examples for both Docker and CDAP. - -## Next - -If you are building a CDAP application, review the [component specification details for CDAP](cdap-specification.md). - -If you are building a Dockerized application, review the [component specification details for Docker](docker-specification.md). diff --git a/platformdoc/docs/components/component-specification/streams-grid.md b/platformdoc/docs/components/component-specification/streams-grid.md new file mode 100644 index 00000000..b6d64745 --- /dev/null +++ b/platformdoc/docs/components/component-specification/streams-grid.md @@ -0,0 +1,60 @@ + +# Streams Formatting Quick Reference + +Each of the following tables represents an example of a publisher and its subscriber, which are of course, different components. This focuses on the fields that are ‘different’ for each of these TYPEs, to illustrate the relationship between `config_key`, dmaap connection object, and the generated configuration. Some notes on specific properties: + +* `config_key` is an arbitrary string, chosen by the component developer. It is returned in the generated configuration where it contains specific values for the target connection +* `format`, `version`, and `type` properties in the subscriber would match these properties in the publisher +* `aaf_username` and `aaf_password` may be different between the publisher and the subscriber + + + +### Using http + +#### *Publishing Component* + +| component spec | runtime platform generated config | +|----------------|-----------------------------------| +|"streams":{<br> "publishes":[{<br> "config_key":"prediction",<br> "format":"some-format",<br> "type":"http",<br> "version":"0.1.0"<br> }]<br>}<br>|"streams_publishes":{<br> "prediction":["10.100.1.100:32567/data"] | + +#### *Subscribing Component* + +| component spec | runtime platform generated config | +|----------------|-----------------------------------| +|"streams":{<br> "subscribes":[{<br> "route":"/data",<br> "format":"some-format",<br> "type":"http",<br> "version":"0.1.0"<br> }]<br>}<br>|"N/A"| + + + +### Using Message Router + +#### *Publishing Component* + +Note: When deploying, this component should be deployed first so satisfy downstream dependencies. Refer to the –force option in component ‘run’ command for more information. + +| component spec | Dmaap Connection Object | runtime platform generated config | +|----------------|-------------------------| --------------------------------- | +|"streams":{<br> "publishes":[{<br> "config_key":"mr_output",<br> "format":"some-format",<br> "type":"message_router",<br> "version":"0.1.0"<br> }]<br>} | {<br> "type":"message_router",<br> "dmaap_info": {<br> "topic_url": "https://we-are-message-router.us:3905/events/some-topic" }<br>} <br><br>*Note: For message router, this object is identical for the publisher and the subscriber* | "streams_publishes":{<br> "mr_output":{<br> "aaf_username":"pub-user",<br> "aaf_password":"pub-pwd",<br> "type":"message_router",<br> "dmaap_info":{<br> "topic_url":"https://we-are-message-router.us:3905/events/some-topic"}<br> }<br>},<br> "streams_subscribes":{<br>…<br>} + +#### *Subscribing Component* + +| component spec | Dmaap Connection Object | runtime platform generated config | +|----------------|-------------------------| --------------------------------- | +|"streams":{<br> "subscribes":[{<br> "config_key":"mr_input",<br> "format":"some-format",<br> "type":"message_router",<br> "version":"0.1.0"<br> }]<br>} | {<br> "type":"message_router",<br> "dmaap_info": {<br> "topic_url": "https://we-are-message-router.us:3905/events/some-topic" }<br>} <br><br>*Note: For message router, this object is identical for the publisher and the subscriber* | "streams_publishes":{<br>…<br>},<br> "streams_subscribes":{<br> "mr_input":{<br> "aaf_username":"sub-user",<br> "aaf_password":"sub-pwd",<br> "type":"message_router",<br> "dmaap_info":{<br> "topic_url":"https://we-are-message-router.us:3905/events/some-topic"}<br> }<br>} + + + + +### Using Data Router + +#### *Publishing Component* + +| component spec | Dmaap Connection Object | runtime platform generated config | +|----------------|-------------------------| --------------------------------- | +|"streams":{<br> "publishes":[{<br> "config_key":"dr_output",<br> "format":"some-format",<br> "type":"data_router",<br> "version":"0.1.0"<br> }]<br>} | {<br> "type":"data_router",<br> "dmaap_info": {<br> "location": "mtc00",<br> "publish_url": "https://we-are-data-router.us/feed/xyz", <br> "log_url": "https://we-are-data-router.us/feed/xyz/logs",<br> "username": "pub-user",<br> "password": "pub-password",<br> "publisher_id": "123456"}<br>} | streams_publishes":{<br> "dr_output":{<br> "type":"data_router",<br> "dmaap_info":{<br> "location":"mtc00",<br> "publish_url":"https://we-are-data-router.us/feed/xyz",<br> "log_url":"https://we-are-data-router.us/feed/xyz/logs",<br> "username":"pub-user",<br> "password":"pub-password",<br> "publisher_id":"123456"}<br> }<br>},<br> "streams_subscribes":{<br> …<br> } + +#### *Subscribing Component* + +| component spec | Dmaap Connection Object | runtime platform generated config | +|----------------|-------------------------| --------------------------------- | +|"streams":{<br> "subscribes":[{<br> "config_key":"dr_input",<br> "format":"some-format",<br> "type":"data_router",<br> "version":"0.1.0",<br> "route":"/target-path"<br> }]<br>} | {<br> "type":"data_router",<br> "dmaap_info": {<br> "location": "mtc00",<br> "delivery_url": "https://my-subscriber-app.dcae:8080/target-path", <br> "username": "sub-user",<br> "password": "sub-password",<br> "subscriber_id": "789012"}<br>} | "streams_publishes":{<br> …<br> },<br> "streams_subscribes":{<br> "dr_input":{<br> "type":"data_router",<br> "dmaap_info":{<br> "location":"mtc00",<br> "delivery_url":"https://my-subscriber-app.dcae:8080/target-path",<br> "username":"sub-user",<br> "password":"sub-password",<br> "subscriber_id":"789012"}<br> }<br>} + diff --git a/platformdoc/docs/components/component-type-docker.md b/platformdoc/docs/components/component-type-docker.md index e9f014d8..fd3358dc 100644 --- a/platformdoc/docs/components/component-type-docker.md +++ b/platformdoc/docs/components/component-type-docker.md @@ -76,9 +76,10 @@ Regarding `<config binding service hostname>:<port>`, there is DNS work going on However, currently you will be given a name as an ENV variable, `CONFIG_BINDING_SERVICE`, and you will need to query Consul's service discovery to get `<config binding service hostname>:<port>`. -### Policy Reconfiguration +### Policy Reconfiguration +*(Draft and subject to change)* -*Details coming soon* +Components must provide a way to receive policy reconfiguration, that is, configuration parameters that have been updated via the Policy UI. The component developer provides a docker script (defined in the [Docker auxiliary specification](component-specification/docker-specification#policy-example)) that will be triggered when this occurs. ### DMaaP @@ -94,6 +95,18 @@ Docker images must be pushed to the environment specific Nexus repository. This Use the Docker command-line to [tag](https://docs.docker.com/engine/reference/commandline/tag/) your Docker image where the *target image* must contain the registry host name and port. +For example, an application called laika has been tagged for an example Nexus registry: + +``` +$ docker images +REPOSITORY TAG IMAGE ID CREATED SIZE +YOUR_NEXUS_DOCKER_REGISTRY/laika 0.4.0 154cc382df61 7 weeks ago 710.5 MB +laika 0.4.0 154cc382df61 7 weeks ago 710.5 MB +``` + +Note, the Docker registry that is used may require a login to authenticate. + + ``` docker login YOUR_NEXUS_DOCKER_REGISTRY ``` @@ -116,7 +129,7 @@ After tagging, upload your image to the remote registry using the Docker [push c docker push YOUR_NEXUS_DOCKER_REGISTRY/laika:0.4.0 ``` -*NOTE* Replace `laika` with your application's name. Replace the `0.4.0` version with your application's version. +*NOTE* Replace `laika` with your application's name. Replace the `0.4.0` version with your application's version. ### Ports diff --git a/platformdoc/docs/components/data-formats.md b/platformdoc/docs/components/data-formats.md index 62681154..7d17e984 100644 --- a/platformdoc/docs/components/data-formats.md +++ b/platformdoc/docs/components/data-formats.md @@ -11,23 +11,23 @@ The vision is to have a repository of shared data formats that developers and te # Meta Schema Definition -The current "Meta Schema" implementation defines how data formats can be written. It requires the name of the data format entry, the data format entry version and allows a description under "self". The meta schema version as "dataformatversion" must be specified. Then the schema is described. There are four types of schema descriptions - jsonschema for inline JSON Schema definitions, delimitedschema for JSON schema descriptions of delimited data, unstructured for unstructured text, and reference that allows a pointer to another artifact for a schema. The reference allows for XML schema, but can be used for JSON, Delimited Format, and Unstructured formats as well. +The "Meta Schema" implementation defines how data format JSON schemas can be written to define user input. It is itself a JSON schema (thus it is a "meta schema"). It requires the name of the data format entry, the data format entry version and allows a description under "self" object. The meta schema version must be specified as the value of the "dataformatversion" key. Then the input schema itself is described. There are four types of schema descriptions objects - jsonschema for inline standard JSON Schema definitions of JSON inputs, delimitedschema for delimited data input using a defined JSON description, unstructured for unstructured text, and reference that allows a pointer to another artifact for a schema. The reference allows for XML schema, but can be used as a pointer to JSON, Delimited Format, and Unstructured schemas as well. -The current implementation is defined by the "format_schema" at the link below. There are descriptions of each entity: +The current Meta Schema implementation is defined at the link below. [schema](ONAP URL TBD) #TCA Example -TCA Input - Common Event Format by referemce +TCA Input - Common Event Format by reference -First the full json schema description of the Commen Event Format would be loaded with a name of "Common Event Format" and the current version of "25.0.0". +First the full JSON schema description of the Common Event Format would be loaded with a name of "Common Event Format" and the current version of "25.0.0". Then the data format description is loaded by the example at this link: [tcainput](ONAP URL TBD) -TCA Output JSON inline example: (TBD Types ok?) +TCA Output JSON inline example: [tcaoutput](ONAP URL TBD) @@ -45,7 +45,7 @@ CUDA Unstructured Example: [unstructuredtext](ONAP URL TBD) -#A possible example of a delimited schema +# An example of a delimited schema ``` { diff --git a/platformdoc/docs/components/dcae-cli/quickstart.md b/platformdoc/docs/components/dcae-cli/quickstart.md index 5644b4fe..905b1372 100644 --- a/platformdoc/docs/components/dcae-cli/quickstart.md +++ b/platformdoc/docs/components/dcae-cli/quickstart.md @@ -38,7 +38,7 @@ To do an upgrade, use the `--upgrade` flag. When you run the tool for the first time, the tool will create a [configuration directory](http://click.pocoo.org/5/api/#click.get_app_dir) and generate a configuration file. -Configuration is first sourced from an remote server that the platform team manages. This is overlaid with configuration details that you will be prompted to input particularly your AT&T UID. +Configuration is first sourced from an remote server that the platform team manages. This is overlaid with configuration details that you will be prompted to input particularly your user id. #### `--reinit` diff --git a/platformdoc/docs/components/dcae-cli/walkthrough.md b/platformdoc/docs/components/dcae-cli/walkthrough.md index e646c435..da54e12e 100644 --- a/platformdoc/docs/components/dcae-cli/walkthrough.md +++ b/platformdoc/docs/components/dcae-cli/walkthrough.md @@ -27,6 +27,7 @@ Options: Commands: add Tracks a data format file SPECIFICATION... + generate Generates a data format file from JSON input examples list Lists all your data formats publish Publishes data format to make publicly... show Provides more information about FORMAT @@ -70,6 +71,14 @@ Data formats for mh677g Go ahead and add other referenced data formats. +If you have JSON input you can generate a data format like this: + +``` +$ dcae_cli data_format --keywords generate myname:1.0.0 myjsoninputfile +``` + +where `myname` is the name of your data format, `1.0.0` is an example version, and `myjsoninputfile` is an example JSON input file (a directory of input JSON files can also be provided). The `--keywords` option adds additional data attributes that can be completed to provide a more detailed data characterization. In any event the output should be reviewed for accuracy. The data format is written to stdout. + ## Adding component `component` is the sub-command that is used to work with operations for components: @@ -220,6 +229,7 @@ Use the "--deployed" option to see more details on deployments For Docker containers, you can run either attached or unattached. Attached means that the dcae-cli tool will launch the container and not terminate. The dcae-cli while attached will stream in the logs of the Docker container. Doing a Ctrl-C will terminate the run session which means undeploy your container and force a clean up automatically. +Running unattached means simply deploy your container. You will need to execute `undeploy` when you are done testing. #### CDAP **NOTE** Make sure your CDAP jar has been uploaded to Nexus. @@ -356,6 +366,43 @@ DCAE.Run | WARNING | Your component is a data router subscriber. Here are the de ``` +### *Sourced at deployment* testing + +Components may have configuration parameters whose values are to be sourced at deployment time. For example, there are components whose configuration parameters are to come from DTI events which are only available when the component is getting deployed. These configuration parameters must be setup correctly in the [component specification](http://localhost:8000/components/component-specification/docker-specification/#configuration-parameters) by setting the property `sourced_at_deployment` to `true` for each and every parameter that is expected to come in at deployment time. + +Once your component specification has been updated correctly, you must use the `--inputs-file` command-line argument when running the commands `dev` or `run` with your component. This is to simulate providing the dynamic, deployment time values for those parameters marked as `sourced_at_deployment`. + +For example, if your component specification has the following configuration parameters: + +``` +"parameters": [{ + "name": "vnf-ip", + "value": "", + "sourced_at_deployment": true +}, +{ + "name": "static-param", + "value": 5 +}] +``` + +You would have to pass in an inputs file that looks like: + +``` +{ + "vnf-ip": "10.100.1.100" +} +``` + +Your application configuration would look like: + +``` +{ + "vnf-ip": "10.100.1.100", + "static-param": 5 +} +``` + ## Publishing component Once components have their component specifications crafted and validated and have been tested, components should be published in the shared onboarding catalog using the `publish` sub-command for both data formats and components. You must publish all data formats of a component before publishing a component. diff --git a/platformdoc/docs/components/intro.md b/platformdoc/docs/components/intro.md index 2e0087c1..6bd3b2d1 100644 --- a/platformdoc/docs/components/intro.md +++ b/platformdoc/docs/components/intro.md @@ -2,55 +2,47 @@ DCAE components are services that provide a specific functionality and are written to be composable with other DCAE service components. The DCAE platform is responsible for running and managing DCAE service components reliably. -Currently, the DCAE platform supports two types of components, CDAP applications and Docker containers. For each, there are requirements that must be met for the component to integrate into the DCAE platform (see [CDAP](component-type-docker.md) and [Docker](component-type-docker.md)). +Currently, the DCAE platform supports two types of components, CDAP applications and Docker containers. For each, there are requirements that must be met for the component to integrate into the DCAE platform (see [CDAP](component-type-cdap.md) and [Docker](component-type-docker.md)). -## Onboarding - -There is an onboarding process that all component developers will have to take their components through to ensure that their DCAE service components are compliant with the DCAE platform and are authorized to push their component into the ASDC catalog. A command-line tool called [`dcae-cli`](ONAP URL TBD) is provided to help you through this process. The high level summary of the onboarding process is: - -1. Defining your [Data formats](data-formats.md) -2. Defining your component specification], which is a JSON that is used to describe and configure the component is to be provided by component developers upon the completion of micro-service implementation and certification testing steps. See [docker](component-specification/docker-specification.md) and [CDAP](component-specification/cdap-specification.md). -3. Testing your component locally -4. Pairwise-testing your component with any other components you connect to -5. Pushing your component and data formats into the ASDC catalog - -## The whys - -### Components require data formats... +### Components requires one or more data formats. Components are software applications that do some function. Components don't run independently, they depend upon other components. A component's function could require connecting to other components to fulfill that function. A component could also be providing its function as a service through an interface for other components to use. -The challenge is that a component cannot connect to or be connected with any other component. The upstream and downstream components must *speak* the same vocabulary or *data format*. The output of an upstream component must match your component's input or your component's output must match the downstream component's input. This is necessary for components to function without errors and correctly. +Components cannot connect to or be connected with any other component. The upstream and downstream components must *speak* the same vocabulary or *data format*. The output of an one component must match another component's input. This is necessary for components to function correctly and without errors. + +The platform requires data formats to ensure that a component will be run with other *compatible* components. -All components must have data formats. The platform requires this to validate and to ensure that your component will be run with *compatible* components. +Data formats can and should be shared by multiple components. -Components *should* have one to many data formats. Data formats should be shared. -### Components require a component specification... +### Each Component requires a component specification. -One design goal of the DCAE was to have a single place for component developers to describe their component. They should *not* have to worry about Tosca, blueprints, yaml files, etc. The component specification (and it's linked data formats) is the only piece of information about a component that each component developer has to provide. Here are some benefits: +The component specification is a JSON artifact that fully specifies the component, it's interfaces, and configuration. It's standardized for CDAP and Docker applications and is validated using a [JSON schema](ONAP URL TBD). -1. The component specification fully specifies your inputs and outputs, so DCAE knows how it can compose your component. The component specification is where you define what types of other components your component connects with and what types of other components can connect to you. You are defining your inputs and outputs. +The component specification fully specifies all the configuration parameters of the component. This is used by the designer and by policy (future) to configure the runtime behavior of the component. -2. The component specification fully specifies all the configuration parameters of your component. This is used by the designer and by policy to configure the runtime behavior of your component. +The component specification is used to *generate* application configuration in a standardized JSON that the platform will make available to the component. This application configuration JSON will include: -3. The component specification is used to *generate* your application configuration in a standardized JSON that the platform will make available. This application configuration JSON will include: +* Parameters that have been assigned values from the component specification, policy, and/or the designer +* Connection details of downstream components -* Parameters that have been assigned values from you, policy, and/or design -* Connection details of downstream components you are dependent upon +The component specification is transformed by DCAE tooling (explained later) into TOSCA models (one for the component, and in the future, one for Policy). The TOSCA models then get transformed into Cloudify blueprints. -4. The component specification is transformed by DCAE tooling into Policy models and Cloudify blueprints, so you need not worry about these. +The component specification is used by: -Every component should have one component specification. +* dcae_cli tool - to validate it +* Design Tools - TOSCA models are generated from the component specification so that the component can be used by designers to compose new DCAE services in SDC. +* Policy (future) - TOSCA models are generated from the component specification so that operations can create policy models used to dynamically configure the component. +* the runtime platform - The component's application configuration (JSON) is generated from the component specification and will be provided to the component at runtime. -### Component developers should use the dcae-cli... +## Onboarding -The dcae-cli was developed to empower component developers with developing their components for the DCAE platform. It is intended to help with: +Onboarding is a process that ensures that the component is compliant with the DCAE platform rules. A command-line tool called [`dcae-cli`](/components/dcae-cli/quickstart.md) is provided to help with onboarding. The high level summary of the onboarding process is: -* Crafting your component specification and your data formats -* Finding and sharing existing data formats -* Finding and running existing components -* Testing and launching your component as it would be run on the platform -* Push your component into the catalog to be shared and to be used by designers +1. Defining the [data formats](data-formats.md) if they don't already exist. +2. Defining the [component specification](/components/component-specification/common-specification.md). See [docker](component-specification/docker-specification.md) and [CDAP](component-specification/cdap-specification.md). +3. Use the dcae_cli tool to [add the data formats](/components/dcae-cli/walkthrough/#adding-data-formats) and [add the component](/components/dcae-cli/walkthrough/#adding-component) to the onboarding catalog. This process will validate them as well. +4. Use the dcae_cli tool to [deploy](/components/dcae-cli/walkthrough/#development-and-testing) the component. (The component is deployed to the environment indicated in the [profile](/components/dcae-cli/walkthrough/#setting-profile) section. +5. Test the component. Also do pairwise-test the component with any other components it connects with. +7. Publish the component and data formats into the Service Design and Creation (SDC) 'catalog'. (Currently, this is a manual step). -All this from a single command-line tool. diff --git a/platformdoc/docs/index.md b/platformdoc/docs/index.md index 6b7475fd..93689beb 100644 --- a/platformdoc/docs/index.md +++ b/platformdoc/docs/index.md @@ -10,5 +10,5 @@ This is the home for the documetation for the DCAE platform where you will find: ## Background -DCAE is composed of a platform and services. The DCAE platform are the technical pieces responsible for the deployment and the management of DCAE services. The DCAE services are the technical pieces responsible for monitoring network services. +DCAE is composed of a platform and services. The DCAE platform is all of the technical pieces responsible for the deployment and the management of DCAE services. The DCAE services are the components responsible for monitoring network services, and data collection and analytics. diff --git a/platformdoc/mkdocs.yml b/platformdoc/mkdocs.yml index f724f902..3cbe14c1 100644 --- a/platformdoc/mkdocs.yml +++ b/platformdoc/mkdocs.yml @@ -7,20 +7,22 @@ pages: - Home: 'index.md' - Architecture: - Technologies: 'architecture/pieces.md' + - Services: 'architecture/services.md' - Service Discovery: 'architecture/service-discovery.md' - Component Developers: - Overview: 'components/intro.md' - - Component Specification: - - Start here: 'components/component-specification/start-here.md' - - For CDAP: 'components/component-specification/cdap-specification.md' - - For Docker: 'components/component-specification/docker-specification.md' - - Common elements: 'components/component-specification/common-specification.md' - - Generated configuration: 'components/component-specification/generated-configuration.md' - - Data Formats: 'components/data-formats.md' - - Component Requirements: + - Requirements/Guidelines: - CDAP: 'components/component-type-cdap.md' - Docker: 'components/component-type-docker.md' - - dcae-cli: + - Component Specification: + - Common Elements: 'components/component-specification/common-specification.md' + - CDAP Specific: 'components/component-specification/cdap-specification.md' + - Docker Specific: 'components/component-specification/docker-specification.md' + - Generated Configuration: 'components/component-specification/generated-configuration.md' + - Streams Quick Reference: 'components/component-specification/streams-grid.md' + - Configuration Quick Reference: 'components/component-specification/configuration-grid.md' + - Data Formats: 'components/data-formats.md' + - dcae-cli Tool: - Quickstart: 'components/dcae-cli/quickstart.md' - Walk-through: 'components/dcae-cli/walkthrough.md' - DMaaP connection objects: 'components/dcae-cli/dmaap-connection-objects.md' |