diff options
Diffstat (limited to 'platformdoc/docs')
20 files changed, 1806 insertions, 0 deletions
diff --git a/platformdoc/docs/architecture/pieces.md b/platformdoc/docs/architecture/pieces.md new file mode 100644 index 00000000..7787359f --- /dev/null +++ b/platformdoc/docs/architecture/pieces.md @@ -0,0 +1,7 @@ +# Platform technologies + +* [Cloudify](http://getcloudify.org/) +* [Consul](https://www.consul.io/) +* [Docker](https://www.docker.com/) +* [CDAP](https://cdap.io/) +* [Registrator](https://github.com/gliderlabs/registrator) diff --git a/platformdoc/docs/architecture/service-discovery.md b/platformdoc/docs/architecture/service-discovery.md new file mode 100644 index 00000000..25d69fcf --- /dev/null +++ b/platformdoc/docs/architecture/service-discovery.md @@ -0,0 +1,15 @@ +# Service Discovery + +Service discovery is an architecture pattern used for components (micro-services) to locate each other. The DCAE platform uses [server-side discovery](http://microservices.io/patterns/server-side-discovery.html) and is using [Consul](https://www.consul.io/) as the service registry solution. + +## Service registration + +All components are required to register with Consul in order to be discovered. There are two methods of registration: self and 3rd party. The DCAE platform uses 3rd party registration which means components don't actually make the registration calls but defers that responsibility to a platform service. + +### Implementation for Docker + +[Registrator](http://gliderlabs.com/registrator/latest/) is an open source application that is responsible for registering all components that run as Docker containers. Registrator watches the local Docker engine's activity log and will register and unregister a Docker container when the container is started and stopped. + +### Implementation for CDAP + +The CDAP broker is a REST web service that is responsible for registering all components that run as CDAP applications. diff --git a/platformdoc/docs/components/component-specification/cdap-specification.md b/platformdoc/docs/components/component-specification/cdap-specification.md new file mode 100644 index 00000000..479cc944 --- /dev/null +++ b/platformdoc/docs/components/component-specification/cdap-specification.md @@ -0,0 +1,163 @@ +# Component specification (CDAP) + +## Overview + +This page contains details specific to CDAP applications. + +The component specification contains the following top-level groups of information: + +* Component [Metadata](#metadata) +* [Parameters](#Parameters): the section for specifying parameters in your AppConfig, AppPreferences, and ProgramPreferences to the Designer and Policy. +* [Interfaces](#interfaces): the connections from this component to other components +* [Auxiliary details](#auxiliary) +* [List of artifacts](#artifacts) + +Note: this page does not talk about [DCAE specific requirements](/components/component-type-cdap.md) that your component must adhere to. Please see that page for discussions about DMaaP, Policy, Metrics, and more. + +## Current Limitations and TODOs + +* Currently we only support CDAP applications that have a stream. +* The integration of DMD is likely to significantly change the [Interfaces](#interfaces) section in this specification, see [DMaaP abstraction](/components/component-type-cdap.md#dmaap-abstraction). + +## Metadata + +See [Metadata](common-specification.md#metadata) + +## Parameters + +There is a `parameters` section in your component specification. This section contains three optional keys: [app_config](#appconfig), [app_preferences](#apppreferences), and [propram_preferences](#prorgram_preferences): +``` +"parameters" : { + "app_config" : [ ...], + "app_preferences" : [ ...], + "program_preferences" : [...] + // any additional keys are ignored +} +``` + +* Each section details the parameters that are a part of each of these CDAP constructs (see below). +* All such parameters will be exposed to the designer and to policy for override. +* These parameters should have default values specified by the component developer where necessary, i.e., parameters that _must_ come from the designer/policy should not have defaults. +* All of these keys are optional because not every CDAP application uses preferences and not every application uses the AppConfig. However, you should specify at least one, or else your application will have no parameters exposed to policy or to the DCAE designer, which means it would be non-configurable. +* Despite the AppConfig being optional to *specify* in the case that you have no parameters in your AppConfig, it is *required for processing* in your application. That is because the DCAE platform will place important information into your AppConfig as discussed below. + +### Parameter +We use the following definition of a _single parameter_, which is used in several places below: +``` + { + "name" : "param name", + "value" : "developer default", + "description" : "tell policy/ops what this does" + } +``` + +### AppConfig + +The `app_config` key refers to the [CDAP AppConfig](http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/configuration.html). It is expected to be a JSON: +``` +"app_config" : [ // list of JSON + param1, // see Parameter above + ... +] +``` +Unfortunately, at the time of writing, the AppConfig is a Java map of `string:string`, which means you cannot have more complex structures (than string) as any value in your AppConfig. However, there is a way to bypass this constraint: you can pass a JSON by encoding the JSON as a string. E.g., the `json.dumps()` and it's converse `loads` methods in Python: +``` +>>> import json +>>> json.dumps({"foo" : "bar"}) # This is a real JSON +'{"foo": "bar"}' # It is now a string: pass this in as your parameter value +>>> json.loads('{"foo": "bar"}') # Do the equivelent of this in your application +{u'foo': u'bar'} # ...and you'll get back a real JSON +>>> +``` + +The final AppConfig (after the designer and policy override parameter values) is passed into CDAP's AppConfig API when starting your application. + +### AppPreferences + +In addition to the CDAP AppConfig, we support [Application Preferences](http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/preferences.html#set-preferences). +The format of the `app_preferences` value is the same as the above: +``` +"app_preferences" : [ // list of JSON + param1, // see Parameter above + ... +] +``` + +The final Application Preferences JSON (after the designer and policy override parameter values) is passed into CDAP's Preferences API when starting your application. + +### ProgramPreferences + +Preferences can also be specified [per program](http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/lifecycle.html#program-lifecycle) in CDAP. This key's value is a list of JSON with the following format: +``` +"program_preferences" : [ // note: this is a list of JSON + { + "program_id" : "program name 1", // the name of this CDAP program + "program_type" : "e.g., flows", // "must be one of flows, mapreduce, schedules, spark, workflows, workers, or services", + "program_pref" : [ // list of JSON + param1, // see Parameter above + ... + ] + }, + // repeat for each program you want to pass a preferences JSON to +] +``` +Each `program_pref` JSON is passed into the CDAP API as the preference for `program_id`. + +## Interfaces +See [Interfaces](common-specification.md#interfaces) + +NOTE: for CDAP, this section is very likely to change when DMD is available. +The _future_ vision, as per [DMaaP intentionally abstracted](/components/component-type-cdap.md#dmaap-abstraction) is that you would publish your data as a series of files on HDFS, and DMD will pick them up and send them to the appropriate DMaaP feeds or directly when needed. + +## Auxiliary + +`auxiliary` contains details about CDAP specific parameters. + +Property Name | Type | Description +------------- | ---- | ----------- +streamname | string | *Required*. +artifact_name | string | +artifact_version | string | the version of your CDAP JAR artifact +namespace | string | the CDAP namespace to deploy into, default is 'default' +programs | array | contains each CDAP entity represented in the artifact +program_type | string | CDAP entity (eg "flows") +program_id | string | name of CDAP entity (eg "WhoFlow") + +Example: + +```json ++"auxiliary": { + "streamname" : "who", + "artifact_name" : "HelloWorld", + "artifact_version" : "3.4.3", + "namespace" : "hw", + "programs" : [ + {"program_type" : "flows", "program_id" : "WhoFlow"}, + {"program_type" : "services", "program_id" : "Greeting"}, + ... + ], +} +``` +The `programs` key is identical to the `program_preferences` key discussed [above](#programpreferences) except: + +* each JSON in the list does not contain `program_pref` +* this is required! You must include all of your programs in this, as it is used to start each program as well as for DCAE to perform periodic healthchecks on your application. Don't forget about your services; they are programs too. + ++ +## Artifacts + +`artifacts` contains a list of artifacts associated with this component. This is where you specify your CDAP jar. + +Property Name | Type | Description +------------- | ---- | ----------- +artifacts | JSON array | Each entry is a artifact object + +Each artifact object has: + +Property Name | Type | Description +------------- | ---- | ----------- +uri | string | *Required*. Uri to the artifact +type | string | *Required*. For CDAP, use `jar` + +This file is uploading using the CLI tool at the same time as your component specification. + diff --git a/platformdoc/docs/components/component-specification/common-specification.md b/platformdoc/docs/components/component-specification/common-specification.md new file mode 100644 index 00000000..77f2a7f0 --- /dev/null +++ b/platformdoc/docs/components/component-specification/common-specification.md @@ -0,0 +1,258 @@ +# Component specification (Common elements) + +This page describes component specification sections that are common (or nearly) to both Docker and CDAP. + +## Component Metadata +Metadata refers to the properties found under the `self` JSON. This group of properties are used to uniquely identify this component specification and identify the component that this specification is used to capture. The metadata section, represented under `self`, is used to uniquely identify your component among all components in the catalog. + +From the specification example above: + +``` +"self": { + "version": "1.0.0", + "name": "asimov.component.kpi_anomaly", + "description": "Classifies VNF KPI data as anomalous", + "component_type": "docker" +}, +``` + +Here is a breakdown of the schema: + +Property Name | Type | Description +------------- | ---- | ----------- +version | string | *Required*. Semantic version for this specification +name | string | *Required*. Full name of this component which is also used as this component's catalog id. The name includes a namespace that is dot-separated. +description | string | Human-readable text blurb describing the component and the components functional purpose. +component_type | string | *Required*. Identify what containerization technology this component uses: `docker` or `cdap`. + +## Interfaces +Interfaces are the JSON objects found under the `streams` key and the `services` key. These are used to describe the interfaces that the component uses and the interfaces that the component provides. The description of each interface includes the associated [data format](/components/data-formats.md). + +### Streams + * The `streams` JSON is for specifying that you produce data that can be consumed by other components, and the streams you expect to subscribe to produced by other components. These are "fire and forget" type interfaces where the publisher of a stream does not expect or parse a response from the subscriber. +* The term `stream` here is abstract and neither refers to "CDAP streams" or "DMaaP feeds": while a stream is very likely a DMaaP feed, it could be a direct stream of data being routed via HTTP too. It abstractly refers to a sequence of data leaving a publisher. +* Streams have anonymous publish/subscribe semantics, which decouples the production of information from its consumption. +* In general, components are not aware of who they are communicating with. +* Instead, components that are interested in data subscribe to the relevant stream; components that generate data publish to the relevant stream. +* There can be multiple publishers and subscribers to a stream. Streams are intended for unidirectional, streaming communication. + +Streams interfaces that implement an HTTP endpoint must support POST. + +Streams are split into: + +Property Name | Type | Description +------------- | ---- | ----------- +subscribes | JSON list | *Required*. List of all available stream interfaces that this component has that can be used for subscribing +publishes | JSON list | *Required*. List of all stream interfaces that this component will publish onto + +#### Subscribes + +From the example specification: + +```json +"streams": { + "subscribes": [{ + "format": "dcae.vnf.kpi", + "version": "1.0.0", + "route": "/data", // for CDAP this value is not used + "type": "http" + }], +... +} +``` + +This describes that `asimov.component.kpi_anomaly` exposes an HTTP endpoint called `/data` which accepts requests that have the data format of `dcae.vnf.kpi` version `1.0.0`. + +The JSON object schema used in `subscribes`: + +Property Name | Type | Description +------------- | ---- | ----------- +format | string | *Required*. Data format id of the data format that is used by this interface +version | string | *Required*. Data format version of the data format that is used by this interface +route | string | *Required*. The HTTP route that this interface listens on +type | string | *Required*. Type of stream: `http`, `message_router`, `data_router` + +##### Message router + +Message router subscribers are http clients rather than http services and performs a http `GET` call. Thus, message router subscribers description is structured like message router publishers and requires `config_key`: + +```json +"streams": { + "subscribes": [{ + "format": "dcae.some-format", + "version": "1.0.0", + "config_key": "some_format_handle", + "type": "message router" + }], +... +} +``` + +##### Data router + +Data router subscribers are http or https services that handle `PUT` requests from data router. Developers must provide the `route` or url path/endpoint that is expected to handle data router requests. This will be used to construct the delivery url needed to register your subscriber to the provisioned feed. Developers must also provide a `config_key` because there is dynamic configuration information associated with the feed that your application will need e.g. username and password. See the page on [DMaaP connection objects](../dcae-cli/dmaap-connection-objects) for more details on the configuration information. + +Example (not tied to the larger example): + +```json +"streams": { + "subscribes": [{ + "config_key": "some-sub-dr", + "format": "sandbox.platform.any", + "route": "/identity", + "type": "data_router", + "version": "0.1.0" + }], +... +} +``` + +#### Publishes + +From the example specification: + +```json +"streams": { +... + "publishes": [{ + "format": "asimov.format.integerClassification", + "version": "1.0.0", + "config_key": "prediction", + "type": "http" + }] +}, + +``` + +This describes that `asimov.component.kpi_anomaly` publishes by making POST requests to streams that support the data format `asimov.format.integerClassification` version `1.0.0`. + +The JSON object schema used in `publishes`: + +Property Name | Type | Description +------------- | ---- | ----------- +format | string | *Required*. Data format id of the data format that is used by this interface +version | string | *Required*. Data format version of the data format that is used by this interface +config_key | string | *Required*. The JSON key in the generated application configuration that will be used to pass the downstream component connection information. +type | string | *Required*. Type of stream: `http`, `message router` + +##### Data router + +Data router publishers are http clients that make `PUT` requests to data router. Developers must also provide a `config_key` because there is dynamic configuration information associated with the feed that your application will need to receive e.g. publish url, username, password. See the page on [DMaaP connection objects](../dcae-cli/dmaap-connection-objects) for more details on the configuration information. + +Example (not tied to the larger example): + +```json +"streams": { +... + "publishes": [{ + "config_key": "some-pub-dr", + "format": "sandbox.platform.any", + "type": "data_router", + "version": "0.1.0" + }] +} +``` + +### Services + +* The publish / subscribe model is a very flexible communication paradigm, but its many-to-many one-way transport is not appropriate for RPC +request / reply interactions, which are often required in a distributed system. +* Request / reply is done via a Service, which is defined by a pair of messages: one for the request and one for the reply. + +Services are split into: + +Property Name | Type | Description +------------- | ---- | ----------- +calls | JSON list | *Required*. List of all service interfaces that this component will call +provides | JSON list | *Required*. List of all service interfaces that this component exposes and provides + +#### Calls +The JSON `services/calls` is for specifying that your component relies on an HTTP(S) service---your component sends that service an HTTP request, and that service responds with an HTTP reply. +An example of this is how string matching (SM) depends on the AAI Broker. SM performs a synchronous REST call to the AAI broker, providing it the VMNAME of the VNF, and the AAI Broker responds with additional details about the VNF. This dependency is expressed via `services/calls`. In contrast, the output of string matching (the alerts it computes) is sent directly to policy as a fire-and-forget interface, so that is an example of a `stream`. + +From the example specification: + +```json +"services": { + "calls": [{ + "config_key": "vnf-db", + "request": { + "format": "dcae.vnf.meta", + "version": "1.0.0" + }, + "response": { + "format": "dcae.vnf.kpi", + "version": "1.0.0" + } + }], +... +} +``` + +This describes that `asimov.component.kpi_anomaly` will make HTTP calls to a downstream component that accepts requests of data format `dcae.vnf.meta` version `1.0.0` and is expecting the response to be `dcae.vnf.kpi` version `1.0.0`. + +The JSON object schema used in `calls`: + +Property Name | Type | Description +------------- | ---- | ----------- +request | JSON object | *Required*. Description of the expected request for this downstream interface +response | JSON object | *Required*. Description of the expected response for this downstream interface +config_key | string | *Required*. The JSON key in the generated application configuration that will be used to pass the downstream component connection information. + +The JSON object schema for both `request` and `response`: + +Property Name | Type | Description +------------- | ---- | ----------- +format | string | *Required*. Data format id of the data format that is used by this interface +version | string | *Required*. Data format version of the data format that is used by this interface + +#### Provides + +From the example specification: + +```json +"services": { +... + "provides": [{ + "route": "/score-vnf", + "request": { + "format": "dcae.vnf.meta", + "version": "1.0.0" + }, + "response": { + "format": "asimov.format.integerClassification", + "version": "1.0.0" + } + }] +}, +``` + +This describes that `asimov.component.kpi_anomaly` provides a service interface and it is exposed on the `/score-vnf` HTTP endpoint. The endpoint accepts requests that have the data format `dcae.vnf.meta` version `1.0.0` and gives back a response of `asimov.format.integerClassification` version `1.0.0`. + +The JSON object schema used in `provides`: + +Property Name | Type | Description +------------- | ---- | ----------- +request | JSON object | *Required*. Description of the expected request for this interface +response | JSON object | *Required*. Description of the expected response for this interface +route | string | *Required*. The HTTP route that this interface listens on + +The JSON object schema for both `request` and `response`: + +Property Name | Type | Description +------------- | ---- | ----------- +format | string | *Required*. Data format id of the data format that is used by this interface +version | string | *Required*. Data format version of the data format that is used by this interface + +Note, for CDAP, there is a slight variation due to the way CDAP exposes services: +``` + "provides":[ // note this is a list of JSON + { + "request":{ ...}, + "response":{ ...}, + "service_name":"name CDAP service", + "service_endpoint":"greet", // E.g the URL is /services/service_name/methods/service_endpoint + "verb":"GET" // GET, PUT, or POST + } + ] +``` diff --git a/platformdoc/docs/components/component-specification/docker-specification.md b/platformdoc/docs/components/component-specification/docker-specification.md new file mode 100644 index 00000000..2bff6c47 --- /dev/null +++ b/platformdoc/docs/components/component-specification/docker-specification.md @@ -0,0 +1,246 @@ +# Component specification (Docker) + +This page contains details specific to Dockerized applications. + +The component specification contains the following top-level groups of information: + +* [Component metadata](#metadata) +* [Component interfaces](#interfaces) including the associated [data formats](/components/data-formats.md) +* [Configuration parameters](#configuration-parameters) +* [Auxiliary details](#auxilary) +* [List of artifacts](#artifacts) + +## Metadata + +See [Metadata](common-specification.md#metadata) + +## Interfaces + +See [Interfaces](common-specification.md#interfaces) + +## Configuration parameters + +`parameters` is where to specify the component's application configuration parameters that are not connection information. + +Property Name | Type | Description +------------- | ---- | ----------- +parameters | JSON array | Each entry is a parameter object + +Parameter object has the following available properties: + +Property Name | Type | Description | Default +------------- | ---- | ----------- | ------- +name | string | *Required*. The property name that will be used as the key in the generated config | +value | any | *Required*. The default value for the given parameter | +description | string | *Required*. Human-readable text describing the parameter like what its for | +type | string | The required data type for the parameter | +required | boolean | An optional key that declares a parameter as required (true) or not (false) | true +constraints | array | The optional list of sequenced constraint clauses for the parameter | +entry_schema | string | The optional key that is used to declare the name of the Datatype definition for entries of set types such as the TOSCA list or map | +designer_editable | boolean | An optional key that declares a parameter to be editable by designer (true) or not (false) | true +policy_editable | boolean | An optional key that declares a parameter to be editable by policy (true) or not (false) | true +policy_schema | array | The optional list of schema definitions used for policy | + +Many of the parameter properties have been copied from TOSCA model property definitions and are to be used for service design composition and policy creation. See [section 3.5.8 *Property definition*](http://docs.oasis-open.org/tosca/TOSCA-Simple-Profile-YAML/v1.1/TOSCA-Simple-Profile-YAML-v1.1.html). + +The property `constraints` is a list of objects where each constraint object: + +Property Name | Type | Description +------------- | ---- | ----------- +equal | | Constrains a property or parameter to a value equal to (‘=’) the value declared +greater_than | number | Constrains a property or parameter to a value greater than (‘>’) the value declared +greater_or_equal | number | Constrains a property or parameter to a value greater than or equal to (‘>=’) the value declared +less_than | number | Constrains a property or parameter to a value less than (‘<’) the value declared +less_or_equal | number | Constrains a property or parameter to a value less than or equal to (‘<=’) the value declared +valid_values | array | Constrains a property or parameter to a value that is in the list of declared values +length | number | Constrains the property or parameter to a value of a given length +min_length | number | Constrains the property or parameter to a value to a minimum length +max_length | number | Constrains the property or parameter to a value to a maximum length + +From the example specification: + +```json +"parameters": [ + { + "name": "threshold", + "value": 0.75, + "description": "Probability threshold to exceed to be anomalous" + } +] +``` + +`threshold` is the configuration parameter and will get set to 0.75 when the configuration gets generated. + +## Auxiliary + +`auxilary` contains Docker specific details like health check and port mapping information. + +Property Name | Type | Description +------------- | ---- | ----------- +healthcheck | JSON object | *Required*. Health check definition details +ports | JSON array | each array item maps a container port to the host port. See example below. + +### Health check definition + +The platform uses Consul to perform periodic health check calls. Consul provides different types of [check definitions](https://www.consul.io/docs/agent/checks.html). The platform currently supports http and docker health checks. + +#### http + +Property Name | Type | Description +------------- | ---- | ----------- +type | string | *Required*. `http` +interval | string | Interval duration in seconds i.e. `15s` +timeout | string | Timeout in seconds i.e. `1s` +endpoint | string | *Required*. GET endpoint provided by the component for Consul to call to check health + +Example: + +```json +"auxilary": { + "healthcheck": { + "type": "http", + "interval": "15s", + "timeout": "1s", + "endpoint": "/my-health" + } +} +``` + +#### docker + +Property Name | Type | Description +------------- | ---- | ----------- +type | string | *Required*. `docker` +interval | string | Interval duration in seconds i.e. `15s` +timeout | string | Timeout in seconds i.e. `1s` +script | string | *Required*. Full path of script that exists in the Docker container to be executed + +Consul will use the [Docker exec API](https://docs.docker.com/engine/api/v1.29/#tag/Exec) to periodically call your script in your container. It will examine the script result to identify whether your component is healthy. Your component is considered healthy when the script returns `0` otherwise your component is considered not healthy. + +Example: + +```json +"auxilary": { + "healthcheck": { + "type": "docker", + "script": "/app/resources/check_health.py", + "timeout": "30s", + "interval": "180s" + } +} +``` + +### Ports example + +Example: + +```json +"auxilary": { + "ports": ["8080:8000"] +} +``` + +In the example above, container port 8080 maps to host port 8000. + +## Artifacts + +`artifacts` contains a list of artifacts associated with this component. For Docker, this would be where you specify your Docker image full path including registry. + +Property Name | Type | Description +------------- | ---- | ----------- +artifacts | JSON array | Each entry is a artifact object + +Each artifact object has: + +Property Name | Type | Description +------------- | ---- | ----------- +uri | string | *Required*. Uri to the artifact +type | string | *Required*. `docker image` or `jar` + +## Example +Here is a full example of a component spec: + +```json +{ + "self": { + "version": "1.0.0", + "name": "asimov.component.kpi_anomaly", + "description": "Classifies VNF KPI data as anomalous", + "component_type": "docker" + }, + "streams": { + "subscribes": [{ + "format": "dcae.vnf.kpi", + "version": "1.0.0", + "route": "/data", + "type": "http" + }], + "publishes": [{ + "format": "asimov.format.integerClassification", + "version": "1.0.0", + "config_key": "prediction", + "type": "http" + }] + }, + "services": { + "calls": [{ + "config_key": "vnf-db", + "request": { + "format": "dcae.vnf.meta", + "version": "1.0.0" + }, + "response": { + "format": "dcae.vnf.kpi", + "version": "1.0.0" + } + }], + "provides": [{ + "route": "/score-vnf", + "request": { + "format": "dcae.vnf.meta", + "version": "1.0.0" + }, + "response": { + "format": "asimov.format.integerClassification", + "version": "1.0.0" + } + }] + }, + "parameters": [ + { + "name": "threshold", + "value": 0.75, + "description": "Probability threshold to exceed to be anomalous" + } + ], + "auxilary": { + "healthcheck": { + "type": "http", + "interval": "15s", + "timeout": "1s", + "endpoint": "/my-health" + } + }, + "artifacts": [{ + "uri": "YOUR_NEXUS_DOCKER_REGISTRY/kpi_anomaly:1.0.0", + "type": "docker image" + }] +} +``` + +## Generate application configuration + +The above example `asimov.component.kpi_anomaly` will get transformed into the following application configuration JSON that is fully resolved and provided at runtime by calling the config binding service: + +```json +{ + "streams_publishes": { + "prediction": ["10.100.1.100:32567"] + }, + "streams_subscribes": {}, + "threshold": 0.75, + "services_calls": { + "vnf-db": ["10.100.1.101:32890"] + } +} +``` diff --git a/platformdoc/docs/components/component-specification/generated-configuration.md b/platformdoc/docs/components/component-specification/generated-configuration.md new file mode 100644 index 00000000..258358de --- /dev/null +++ b/platformdoc/docs/components/component-specification/generated-configuration.md @@ -0,0 +1,91 @@ +# Generated configuration + +The DCAE platform relies on the component specification to generate the component's application configuration JSON at deployment time. The component developer should expect to use this configuration JSON in their application to provision themselves. + +Pro-tip: As you build your component specification, you can use the [dcae-cli `dev` command](../dcae-cli/walkthrough/#dev) to view what the resulting application configuration will look like. + +## Streams and services + +For both Docker and CDAP, when your component is deployed, any `streams` and `services/calls` you specified will be injected into your configuration under the following well known structure. +Your component is required to parse this information if you have any connectivity to DMaaP or are calling another DCAE component. + +More details about the DMaaP connection objects are found [here](../dcae-cli/dmaap-connection-objects/). + +This is best served with an example. + +The following component spec snippet (from String Matching): +``` +"streams":{ + "subscribes": [{ + "format": "VES_specification", + "version": "4.27.2", + "type": "message_router", + "config_key" : "mr_input" + }], + "publishes": [{ + "format": "VES_specification", + "version": "4.27.2", + "config_key": "mr_output", + "type": "message_router" + }] + }, + "services":{ + "calls": [{ + "config_key" : "aai_broker_handle", + "verb": "GET", + "request": { + "format": "get_with_query_params", + "version": "1.0.0" + }, + "response": { + "format": "aai_broker_response", + "version": "3.0.0" + } + }], + "provides": [] + }, +``` + +Will turn into the following top level keys in your configuration (for CDAP, this will be under AppConfig) + +``` + "streams_publishes":{ + "mr_output":{ // notice the config key above + "aaf_password":"XXX", + "type":"message_router", + "dmaap_info":{ + "client_role": null, + "client_id": null, + "location": null, + "topic_url":"XXX" + }, + "aaf_username":"XXX" + } + }, + "streams_subscribes":{ + "mr_input":{ // notice the config key above + "aaf_password":"XXX", + "type":"message_router", + "dmaap_info":{ + "client_role": null, + "client_id": null, + "location": null, + "topic_url":"XXX" + }, + "aaf_username":"XXX" + } + }, + "services_calls":{ + "aai_broker_handle":[ // notice the config key above + "SOME_IP:32768" // based on deployment time, just an example + ] + } +``` +These keys will always be populated regardless of whether they are empty. So the minimal you will get, in the case of a component that provides an HTTP service and does not call any services and has no streams, is: +``` + "streams_publishes":{}, + "streams_subscribes":{}, + "services_calls":{} +``` + +Thus your component should expect these well-known top level keys. diff --git a/platformdoc/docs/components/component-specification/start-here.md b/platformdoc/docs/components/component-specification/start-here.md new file mode 100644 index 00000000..3816dd06 --- /dev/null +++ b/platformdoc/docs/components/component-specification/start-here.md @@ -0,0 +1,21 @@ +# Component specification + +Every component that onboards onto the DCAE platform requires a component specification. The component specification is a JSON artifact that fully describes your components. The format of the component specification is standardized for CDAP applications and Dockerized applications and is validated using [a JSON schema](ONAP URL TBD). + +The component specification is used by: + +* Design - TOSCA models are generated from the component specification so that your component can be used by designers to compose new DCAE services in SDC. +* Policy - TOSCA models are generated from the component specification so that operations can create policy models used to dynamically configure your component. +* Runtime platform - Your component's application configuration (JSON) is generated from the component specification and will be provided to your component at runtime. + +## dcae-cli + +Use the [`dcae-cli`](../dcae-cli/quickstart) tool to manage your component specification and to test your components with it. + +The dcae-cli can also be used to view component specifications that have already been added and published. Please check out the [shared catalog](../dcae-cli/walkthrough/#shared-catalog) for examples for both Docker and CDAP. + +## Next + +If you are building a CDAP application, review the [component specification details for CDAP](cdap-specification.md). + +If you are building a Dockerized application, review the [component specification details for Docker](docker-specification.md). diff --git a/platformdoc/docs/components/component-type-cdap.md b/platformdoc/docs/components/component-type-cdap.md new file mode 100644 index 00000000..253dd2f8 --- /dev/null +++ b/platformdoc/docs/components/component-type-cdap.md @@ -0,0 +1,52 @@ +# CDAP Component Requirements/Information + +This page contains information about CDAP app development in DCAE. + +## Uploading your Jar File +The DCAE component specification has you input your `jar_url`, the URL on Nexus to your Jar file. This DCAE controller deploys out of Nexus. +You can upload your jar(s) using the following command, replacing NAME: +``` +curl -v --user 'user:password' http://YOUR_NEXUS_RAW_REPO/NAME.jar --upload-file NAME.jar +``` +During the CLI Tool Usage, in your spec, supply `http://YOUR_NEXUS_RAW_REPO/NAME.jar` as the JAR artifact URL. + +## Policy Reconfiguration +We support reconfiguration of both AppConfig and AppPreferences. + +For AppConfig, we support CDAPs "update" API to [reconfigure an application](http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/lifecycle.html\#update-an-application}}). + +For AppPreferences, we: + +1. Stop your programs + +2. Set the new preferences + +3. Start your programs + +At the time of writing, there is no way to update a CDAP application's AppConfig or AppPreferences, without a restart, *and notify* the application. The latter is a future promised feature by CASK---the ability to update preferences and inform the application that something is changed (so it repulls). +As CDAP currently stands however, given the above, if you are building a stateful application, you must persist your state often (e.g., to a CDAP dataset), as you may be restarted at any time with an updated configuration, or stopped&started at any time with updated preferences. + +## Metrics +Metrics are pulled from your CDAP application on a periodic basis and (in the future: pushed to a central DCAE metric store, currently: just dropped). +For this to be useful, your application should provide [metrics](http://docs.cask.co/cdap/current/en/admin-manual/operations/metrics.html). +While nothing in the DCAE runtime enforces that your CDAP application tracks metrics, your metrics (or lack thereof) will be visible in the DCAE dashboard and to operations. + +## Future DMaaP abstraction +Shown below is our _vision_ for how DMaaP is abstracted from component developers: + + + +Today, this is a vision; it is not in place. +Today, each CDAP app is built with built in assumptions about where they are getting their data from. +Some CDAP apps have the built in assumption of a UEB feed. Some MR. Some DR. +This becomes very difficult to orchestrate when each app in the catalog has built in data assumptions. + +The goal of this vision is to _decouple_ the data plane from the analytics plane. +Analytics should be agnostic to _how_ they are receiving their data beyond "filesystem" or "HTTP". +Analytics developers shouldn't have to worry about the data plane, that should be taken care of by the platform. +They should be spending their time on the problem at hand---the analytic. + +This also allows each CDAP application to have a standard set of interfaces: HTTP and HDFS: + + + diff --git a/platformdoc/docs/components/component-type-docker.md b/platformdoc/docs/components/component-type-docker.md new file mode 100644 index 00000000..e9f014d8 --- /dev/null +++ b/platformdoc/docs/components/component-type-docker.md @@ -0,0 +1,166 @@ +# Component Requirements: Docker + +## Overview + +Component developers are required to provide artifacts for the platform to be able to deploy your component including: + +* [Component specification](component-specification/docker-specification) +* [Data formats](data-formats) +* [Auxiliary component specification](component-specification/docker-specification#auxiliary) +* [Docker image](#docker-on-the-platform) + +In addition, components will have to be enhanced to be compliant with the DCAE platform in order to correctly be deployed and be managed. This page will discuss the changes which are grouped into the following categories: + +* [Service registration](#service-registration) +* [Configuration management on the new platform](#configuration) +* [Operational concerns](#operational-concerns) + +To help component developers to make and to test the changes needed to have components run on the platform, a command-line tool called [`dcae-cli`](dcae-cli/quickstart) is provided by the platform team. + +## Service registration + +Every [Docker component is registered](../architecture/service-discovery) with the platform's service discovery layer. Docker components are not expected to make the explicit make registration calls because that is done by through a platform 3rd party registration service. A couple things are needed from component developers in order for this registration to occur successfully: + +1. Docker images must be created from a Dockerfile that has an [`EXPOSE`](https://docs.docker.com/engine/reference/builder/#/expose) instruction. This applies to components that listen on a port. +2. Component healthcheck details must be provided in the Docker auxiliary component specification + +### Expose port + +Components that listen on a specific port must explicitly declare in their Dockerfile that port using the `EXPOSE` instruction before building the image. +Warning! At the current time, you can not expose multiple ports in your Dockerfile or registration *will not work* correctly. + +### Health check + +Component developers are required to provide a way for the platform to periodically check the health of their running components. The platform uses Consul to perform these periodic calls. Consul provides different types of [check definitions](https://www.consul.io/docs/agent/checks.html). The details of the definition used by your component is to be provided through the [Docker auxiliary specification](component-specification/docker-specification#auxiliary). + +## Configuration + +The component's application configuration is generated from the submitted component specification into a JSON representation. The DCAE platform will provide the configuration JSON by making it available on a remote HTTP service. Components are required to pull their configuration JSON at application startup. The configuration JSON is stored under the components uniquely generated name which is provided by the environment variable `HOSTNAME`. + +You can see more details on the generated application configuration [here](component-specification/docker-specification#generate-application-configuration). + +### Config binding service +The config binding service is a platform HTTP service that is responsible for providing clients with a fully resolved configuration JSON at runtime. Components should make an HTTP GET on: + +``` +<config binding service hostname>:<port>/service_component/NAME +``` +For Docker components, NAME should be set to `HOSTNAME`, which was provided as an ENV variable inside of your container. + +The binding service integrates with the streams and services section of your component specification. For example, if you specify that you call a service: +``` +"services": { + "calls": [{ + "config_key": "vnf-db", + "request": { + "format": "dcae.vnf.meta", + "version": "1.0.0" + }, + "response": { + "format": "dcae.vnf.kpi", + "version": "1.0.0" + } + }], +... +} +``` +Then the config binding service will find all available IP addresses of services meeting your needs, and provide them to you under your `config_key`: +``` +// your configuration +{ + "vbf-db" : // see above + [IP:Port1, IP:Port2,…] // all of these meet your needs, choose one. +} +``` +Regarding `<config binding service hostname>:<port>`, there is DNS work going on to make this resolvable in a convenient way inside of your container. +However, currently you will be given a name as an ENV variable, `CONFIG_BINDING_SERVICE`, and you will need to query Consul's service discovery to get +`<config binding service hostname>:<port>`. + +### Policy Reconfiguration + +*Details coming soon* + +### DMaaP + +Components can be publishers or subscribers to either message router topics or data router feeds. This is defined in the component specification under the `streams` section where you can specify whether your component is expected to subscribe or to publish to a [message router topic](component-specification/common-specification/#message-router) or to a [data router feed](component-specification/common-specification/#data-router). Given a composition with components that use DMaaP, the platform will provision the topic or feed and provide the necessary [connection details](dcae-cli/dmaap-connection-objects) at runtime for each DMaaP dependent component. These connection details will be provided through your application's [generated configuration](component-specification/generated-configuration). + +In order to test DMaaP connections in onboarding, the developer (currently) must provision all test topics and feeds manually and provide the [dcae-cli with the connection details](dcae-cli/walkthrough/#dmaap-testing) when deploying your application. + +## Docker on the platform + +### Images + +Docker images must be pushed to the environment specific Nexus repository. This requires tagging your build with the full name of you image which includes the Nexus repository name. + +Use the Docker command-line to [tag](https://docs.docker.com/engine/reference/commandline/tag/) your Docker image where the *target image* must contain the registry host name and port. + +``` +docker login YOUR_NEXUS_DOCKER_REGISTRY +``` + +Tag your image: + +``` +docker tag laika:0.4.0 YOUR_NEXUS_DOCKER_REGISTRY/laika:0.4.0 +``` + +Or build and tag: + +``` +docker build -t YOUR_NEXUS_DOCKER_REGISTRY/laika:0.4.0 . +``` + +After tagging, upload your image to the remote registry using the Docker [push command](https://docs.docker.com/engine/reference/commandline/push/). Note that the registry may require a login. Use the Docker [login command](https://docs.docker.com/engine/reference/commandline/login/) before pushing in that case. + +``` +docker push YOUR_NEXUS_DOCKER_REGISTRY/laika:0.4.0 +``` + +*NOTE* Replace `laika` with your application's name. Replace the `0.4.0` version with your application's version. + +### Ports + +On the DCAE platform, Docker components are run with the `--publish-all` or `-P` argument. This means the Docker container for your component will be listening on a random port and that random port will be mapped to the port [you exposed](#service-registration). + +### Envs + +The platform provides a set of environment variables into each Docker container: + +Name | Type | Description +---- | ---- | ----------- +`HOSTNAME` | string | Unique name of the component instance that is generated +`CONSUL_HOST` | string | Hostname of the platform's Consul instance +`CONFIG_BINDING_SERVICE` | string | Hostname of the platform's config binding service instance +`DOCKER_HOST` | string | Host of the target platform Docker host to run the container on + +## Operational concerns + +### Logging + +Currently the platform uses the default `json-file` logging driver for Docker. For onboarding testing, component developers can access their logs from their Docker containers either by running their component using the `--attached` flag or by using the `docker logs` command. The requirement is that applications must write to stdout and/or stderr. + +To use the `docker logs` command for your deployed running Docker container, + +1. You must have Docker installed on your local machine +2. Have the generated name of your component. This is generated for you when you execute `dcae_cli component dev` or `dcae_cli component run`. +3. Find the target Docker host using the `dcae_cli profiles show` command: + +``` +$ dcae_cli profiles show YOUR_PROFILE_NAME +{ + ... + "docker_host": "YOUR_DOCKER_HOST:2376" +} +``` + +4. Set your Docker client to point to the target Docker host: + +``` +$ export DOCKER_HOST="tcp://YOUR_DOCKER_HOST:2376" +``` + +5. Use the `docker logs` command: + +``` +$ docker logs <generated component name> +``` diff --git a/platformdoc/docs/components/data-formats.md b/platformdoc/docs/components/data-formats.md new file mode 100644 index 00000000..62681154 --- /dev/null +++ b/platformdoc/docs/components/data-formats.md @@ -0,0 +1,72 @@ +# Data Formats + +Because the DCAE designer composes your component with others at service design time, in most cases you do not know what specific component(s) your component will send data to during runtime. +Thus, it is vital that DCAE has a language of describing the data passed between components, so that it is known which components are composable with others. +Data formats are descriptions of data---they are the data contract between your component and other components. +You need to describe the available outputs and assumed inputs of your components as data formats. +These data descriptions are onboarded into ASDC, and each receives a UUID. +If component X outputs data format DF-Y, and another component Z specifies DF-Y as their input data format, then X is said to be _composable_ with that component. +The data formats are referenced in the component specifications by the data format's id and version. +The vision is to have a repository of shared data formats that developers and teams can re-use and also provide them the means to extend and create new custom data formats. + +# Meta Schema Definition + +The current "Meta Schema" implementation defines how data formats can be written. It requires the name of the data format entry, the data format entry version and allows a description under "self". The meta schema version as "dataformatversion" must be specified. Then the schema is described. There are four types of schema descriptions - jsonschema for inline JSON Schema definitions, delimitedschema for JSON schema descriptions of delimited data, unstructured for unstructured text, and reference that allows a pointer to another artifact for a schema. The reference allows for XML schema, but can be used for JSON, Delimited Format, and Unstructured formats as well. + +The current implementation is defined by the "format_schema" at the link below. There are descriptions of each entity: + +[schema](ONAP URL TBD) + +#TCA Example + +TCA Input - Common Event Format by referemce + +First the full json schema description of the Commen Event Format would be loaded with a name of "Common Event Format" and the current version of "25.0.0". + +Then the data format description is loaded by the example at this link: + +[tcainput](ONAP URL TBD) + +TCA Output JSON inline example: (TBD Types ok?) + +[tcaoutput](ONAP URL TBD) + +#CUDA Example + +CUDA Simple JSON Example: + +[simplejson](ONAP URL TBD) + +CUDA Nested JSON Example: + +[nestedjson](ONAP URL TBD) + +CUDA Unstructured Example: + +[unstructuredtext](ONAP URL TBD) + +#A possible example of a delimited schema + +``` +{ + "self": { + "name": "Delimited Format Example", + "version": "1.0.0", + "description": "Delimited format example just for testing" + + }, + "dataformatversion": "1.0.0", + "delimitedschema": { + "delimiter": "|", + "fields": [{ + "name": "field1", + "description": "test field1", + "fieldtype": "string" + }, { + "name": "field2", + "description": "test field2", + "fieldtype": "boolean" + }] + } +} +``` diff --git a/platformdoc/docs/components/dcae-cli/dmaap-connection-objects.md b/platformdoc/docs/components/dcae-cli/dmaap-connection-objects.md new file mode 100644 index 00000000..cf03ccec --- /dev/null +++ b/platformdoc/docs/components/dcae-cli/dmaap-connection-objects.md @@ -0,0 +1,156 @@ +# DMaaP connection objects + +DMaaP connection objects are JSON objects that: + +1. Components should expect at runtime in their application configuration and is to be used to connect to the appropriate DMaaP feed or topic. +2. Developers must provide through the command-line argument `--dmaap-file` to test their component with manually provisioned feeds and topics. + +This page is a reference to the specific structure that each type of DMaaP stream requires. + +Note for #1 that components should expect the entire object with all properties at runtime where the default will be `null` unless specified otherwise. + +Note for #2 that developers are not required to provide the entire object. The required properties will be labeled with "*required as input*". + +## Message router + +Publishers and subscribers both have the same JSON object structure. Here's an example: + +```json +{ + "type": "message_router", + "aaf_username": "some-user", + "aaf_password": "some-password", + "dmaap_info": { + "client_role": "com.dcae.member", + "client_id": "1500462518108", + "location": "mtc00", + "topic_url": "https://we-are-message-router.us:3905/events/some-topic" + } +} +``` + +At the top-level: + +Property Name | Type | Description +------------- | ---- | ----------- +type | string | *Required as input*. Must be `message_router` for message router topics +aaf_username | string | AAF username message router clients use to authenticate with secure topics +aaf_password | string | AAF password message router clients use to authenticate with secure topics +dmaap_info | JSON object | *Required as input*. Contains the topic connection details + +The `dmaap_info` object contains: + +Property Name | Type | Description +------------- | ---- | ----------- +client_role | string | AAF client role that's requesting publish or subscribe access to the topic +client_id | string | Client id for given AAF client +location | string | DCAE location for the publisher or subscriber, used to set up routing +topic_url | string | *Required as input*. URL for accessing the topic to publish or receive events + +Here's an example of the minimal JSON that must be provided as an input: + +```json +{ + "type": "message_router", + "dmaap_info": { + "topic_url": "https://we-are-message-router.us:3905/events/some-topic" + } +} +``` + +## Data router + +### Publisher + +Here's an example of what the JSON object connection for data router publisher looks like: + +```json +{ + "type": "data_router", + "dmaap_info": { + "location": "mtc00", + "publish_url": "https://we-are-data-router.us/feed/xyz", + "log_url": "https://we-are-data-router.us/feed/xyz/logs", + "username": "some-user", + "password": "some-password", + "publisher_id": "123456" + } +} +``` + +At the top-level: + +Property Name | Type | Description +------------- | ---- | ----------- +type | string | *Required as input*. Must be `data_router` for data router feeds +dmaap_info | JSON object | *Required as input*. Contains the topic connection details + +The `dmaap_info` object contains: + +Property Name | Type | Description +------------- | ---- | ----------- +location | string | DCAE location for the publisher, used to set up routing +publish_url | string | *Required as input*. URL to which the publisher makes Data Router publish requests +log_url | string | URL from which log data for the feed can be obtained +username | string | Username the publisher uses to authenticate to Data Router +password | string | Password the publisher uses to authenticate to Data Router +publisher_id | string | Publisher id in Data Router + +Here's an example of the minimal JSON that must be provided as an input: + +```json +{ + "type": "data_router", + "dmaap_info": { + "publish_url": "https://we-are-data-router.us/feed/xyz" + } +} +``` + +### Subscriber + +Here's an example of what the JSON object connection for data router subscriber looks like: + +```json +{ + "type": "data_router", + "dmaap_info": { + "location": "mtc00", + "delivery_url": "https://my-subscriber-app.dcae:8080/target-path", + "username": "some-user", + "password": "some-password", + "subscriber_id": "789012" + } +} +``` + +At the top-level: + +Property Name | Type | Description +------------- | ---- | ----------- +type | string | *Required as input*. Must be `data_router` for data router feeds +dmaap_info | JSON object | *Required as input*. Contains the topic connection details + +The `dmaap_info` object contains: + +Property Name | Type | Description +------------- | ---- | ----------- +location | string | DCAE location for the publisher, used to set up routing +delivery_url | string | URL to which the Data Router should deliver files +username | string | Username Data Router uses to authenticate to the subscriber when delivering files +password | string | Password Data Router uses to authenticate to the subscriber when delivering files +subscriber_id | string | Subscriber id in Data Router + +Here's an example of the minimal JSON that must be provided as an input: + +```json +{ + "type": "data_router", + "dmaap_info": { + } +} +``` + +Developers are recommended to use `username` and `password` since this is the recommended security practice. + +Note that the dcae-cli will construct the `delivery_url` when deploying the component since this can only be known at deployment time. diff --git a/platformdoc/docs/components/dcae-cli/quickstart.md b/platformdoc/docs/components/dcae-cli/quickstart.md new file mode 100644 index 00000000..5644b4fe --- /dev/null +++ b/platformdoc/docs/components/dcae-cli/quickstart.md @@ -0,0 +1,79 @@ +# Quickstart + +The `dcae-cli` is a Python command-line tool built to aide component developers with the development and testing of their micro-service component for the DCAE platform. It will help developers do functional and integration testing of their components locally and on remote environments as simple as possible. + +The tool requires the component developers to create a valid component specification for their component which is used by the tool. This same component specification will be published in the [onboarding catalog](../../glossary#onboarding-catalog) at the end of development and testing. + +The git repository can be found [here](ONAP LINK TBD) + +## Pre-requisite + +### For Docker + +There are two options for development with Docker: developing locally on your machine which requires Docker to be installed and developing remotely by deploying onto remote infrastructure. + +#### For local development + +* You must install [Docker engine](https://docs.docker.com/engine/installation/) locally on your machine. +* You must know the *external ip* of where the Docker engine is running. The external ip is needed so that service discovery will wire up correctly. + - For OSX users, this means making sure the VirtualBox VM that is running your Docker engine has a bridged adapter and getting the ip of that adapter. + +#### For remote development + +You need access to a remote host with Docker engine installed with remote API access. You must have the associated connection information: domain name or IP and port (should be either 2375 or 2376). This information should be set in [an active profile](walkthrough.md#setting-profile). + +### For CDAP + +TBD + +## Install + +``` +pip install --extra-index-url https://YOUR_NEXUS_PYPI_SERVER/simple dcae-cli +``` + +To do an upgrade, use the `--upgrade` flag. + +### Configuration + +When you run the tool for the first time, the tool will create a [configuration directory](http://click.pocoo.org/5/api/#click.get_app_dir) and generate a configuration file. + +Configuration is first sourced from an remote server that the platform team manages. This is overlaid with configuration details that you will be prompted to input particularly your AT&T UID. + +#### `--reinit` + +Configuration can be re-initialized or reset. There is a `--reinit` flag that is to be used to re-initialize your configuration and your environment profiles. + +To re-initialize: + +``` +$ dcae_cli --reinit +``` + +### Verify + +Verify that its installed: + +``` +$ dcae_cli --help +Usage: dcae_cli [OPTIONS] COMMAND [ARGS]... + +Options: + -v, --verbose Prints INFO-level logs to screen. + --reinit Re-initialize dcae-cli configuration + --version Show the version and exit. + --help Show this message and exit. + +Commands: + catalog + component + data_format + profiles +``` +## Version + +You can verify the version of the dcae-cli with the following command: + +``` +$ dcae_cli --version +``` diff --git a/platformdoc/docs/components/dcae-cli/walkthrough.md b/platformdoc/docs/components/dcae-cli/walkthrough.md new file mode 100644 index 00000000..e646c435 --- /dev/null +++ b/platformdoc/docs/components/dcae-cli/walkthrough.md @@ -0,0 +1,391 @@ +# Walk-through + +The goal of this quickstart is to provide an overview of the functionalities of the `dcae-cli` and walk you through the capabilities: + +* [Adding data formats](#adding-data-formats) +* [Adding component](#adding-component) +* [Setting profile](#setting-profile) +* [Development and testing](#development-and-testing) +* [Publishing component](#publishing-component) +* [Shared catalog](#shared-catalog) + +This walk-through uses example projects: + +* [laika](ONAP URL TBD) +* [CDAP examples](ONAP URL TBD) + +## Adding data formats + +`data_format` is the sub-command that is used to execute operations that manage [data formats](../data-formats.md). + +``` +$ dcae_cli data_format --help +Usage: dcae_cli data_format [OPTIONS] COMMAND [ARGS]... + +Options: + --help Show this message and exit. + +Commands: + add Tracks a data format file SPECIFICATION... + list Lists all your data formats + publish Publishes data format to make publicly... + show Provides more information about FORMAT +``` + +Your data format must be in the catalog in order to use in the component specification. Check the catalog using the `data_format list` sub-command: + +``` +$ dcae_cli data_format list + +Data formats for mh677g ++------+---------+-------------+--------+----------+ +| Name | Version | Description | Status | Modified | ++------+---------+-------------+--------+----------+ +| | | | | | ++------+---------+-------------+--------+----------+ +``` + +The fields `name`, `version`, `description` are referenced from the data format JSON from the `self` JSON. + +There are no data formats so you must add the data formats that your component specification references. Use the `data_format add` sub-command: + +Here's an example command: + +``` +dcae_cli data_format add health.json +``` + +Verify that it was added: + +``` +$ dcae_cli data_format list + +Data formats for mh677g ++-------------------------------+---------+-------------------------------------------+--------+----------------------------+ +| Name | Version | Description | Status | Modified | ++-------------------------------+---------+-------------------------------------------+--------+----------------------------+ +| sandbox.platform.laika.health | 0.1.0 | Data format used for the /health endpoint | staged | 2017-05-23 04:02:38.952799 | ++-------------------------------+---------+-------------------------------------------+--------+----------------------------+ +``` + +Go ahead and add other referenced data formats. + +## Adding component + +`component` is the sub-command that is used to work with operations for components: + +``` +$ dcae_cli component --help +Usage: dcae_cli component [OPTIONS] COMMAND [ARGS]... + +Options: + --help Show this message and exit. + +Commands: + add + dev Set up component in development for... + list Lists components in the public catalog. + publish Pushes COMPONENT to the public catalog + run Runs the latest version of COMPONENT. + show Provides more information about COMPONENT + undeploy Undeploys the latest version of COMPONENT. +``` + +Your component must be accessible from the catalog in order for it to be used. Check the catalog using the `component list` sub-command: + +``` +$ dcae_cli component list +Active profile: solutioning + ++------+---------+------+-------------+--------+----------+-----------+ +| Name | Version | Type | Description | Status | Modified | #Deployed | ++------+---------+------+-------------+--------+----------+-----------+ +| | | | | | | | ++------+---------+------+-------------+--------+----------+-----------+ + +Use the "--deployed" option to see more details on deployments +``` + +The fields `name`, `version`, `type`, `description` are referenced from the component specification's `self` JSON. + +There are no components so you must add your component. Use the `component add` sub-command. The command is the same for docker and cdap components: + +``` +$ dcae_cli component add --help +Usage: dcae_cli component add [OPTIONS] SPECIFICATION + +Options: + --update Updates a locally added component if it has not been already + pushed + --help Show this message and exit. +``` + +*Note* use the `--update` flag to replace existing staged instances. + +The `component dev` sub-command can be useful in validating and experimenting when crafting your component specification. See details about `dev` under [Development and testing](#development-and-testing). + +Once we add the components laika and helloworld, let's verify that they got added ok: + +``` +$ dcae_cli component list +Active profile: solutioning + ++-------------------------+---------+--------+---------------------------------------------------------------+--------+----------------------------+-----------+ +| Name | Version | Type | Description | Status | Modified | #Deployed | ++-------------------------+---------+--------+---------------------------------------------------------------+--------+----------------------------+-----------+ +| cdap.helloworld.endnode | 0.8.0 | cdap | cdap test component | staged | 2017-05-23 04:14:35.588075 | 0 | +| sandbox.platform.laika | 0.5.0 | docker | Web service used as a stand-alone test DCAE service compone.. | staged | 2017-05-23 04:07:44.065610 | 0 | ++-------------------------+---------+--------+---------------------------------------------------------------+--------+----------------------------+-----------+ + +Use the "--deployed" option to see more details on deployments +``` + +## Setting profile + +`profile` is the sub-command that is used to manage profiles. These profiles contain environment variables used to connect to different environments. This is used in the running and deployment of your component using the `dcae_cli component run` command. The `dcae-cli` ships with profiles for `solutioning` and `rework`. + +``` +$ dcae_cli profiles --help +Usage: dcae_cli profiles [OPTIONS] COMMAND [ARGS]... + +Options: + --help Show this message and exit. + +Commands: + activate Sets profile NAME as the active profile + create Creates a new profile NAME initialized with... + delete Deletes profile NAME + list Lists available profiles + set Updates profile NAME such that KEY=VALUE + show Prints the profile dictionary +``` + +To see what variables a profile contains, you can use the `show` command, as in `dcae_cli profiles show PROFILE_NAME` + +Use the `create` sub-command to create your own profile and assign new values using the `set` command. Afterwards you will need to `activate` the profile you wish to use. First take a look at which profile is active: + +``` +$ dcae_cli profiles list + rework +* solutioning +``` + +The active profile is `solutioning` so to activate *rework* to use `rework`: + +``` +$ dcae_cli profiles activate rework +``` + +Check + +``` +$ dcae_cli profiles list +* rework + solutioning +``` + +## Development and testing + +The following operations under the sub-command `component` are aimed to help developers with testing: + +* `run` +* `undeploy` +* `dev` + +### `run` + +The `run` operation is to be used for running your application in its container remotely on the activated environment. Docker containers have the additional option to run locally on your development machine. + +In order to run your application, you must have added your data formats and your component to your catalog. + +Let's verify that your component is in the catalog: + +``` +$ dcae_cli component list +Active profile: solutioning + ++-------------------------+---------+--------+---------------------------------------------------------------+--------+----------------------------+-----------+ +| Name | Version | Type | Description | Status | Modified | #Deployed | ++-------------------------+---------+--------+---------------------------------------------------------------+--------+----------------------------+-----------+ +| cdap.helloworld.endnode | 0.8.0 | cdap | cdap test component | staged | 2017-05-23 04:14:35.588075 | 0 | +| sandbox.platform.laika | 0.5.0 | docker | Web service used as a stand-alone test DCAE service compone.. | staged | 2017-05-23 04:07:44.065610 | 0 | ++-------------------------+---------+--------+---------------------------------------------------------------+--------+----------------------------+-----------+ + +Use the "--deployed" option to see more details on deployments +``` + +#### Docker + +**NOTE** Make sure your Docker image has been uploaded to the shared registry. + +For Docker containers, you can run either attached or unattached. Attached means that the dcae-cli tool will launch the container and not terminate. The dcae-cli while attached will stream in the logs of the Docker container. Doing a Ctrl-C will terminate the run session which means undeploy your container and force a clean up automatically. + +#### CDAP + +**NOTE** Make sure your CDAP jar has been uploaded to Nexus. + +TODO + +### `undeploy` + +The `undeploy` operation is to be used to undeploy any instances of a specified component, version that you have deployed. This includes cleaning up of configuration. + +Let's undeploy `sandbox.platform.laika` that was deployed from the previous section: + +``` +$ dcae_cli component undeploy sandbox.platform.laika:0.5.0 +DCAE.Undeploy | WARNING | Undeploying components: 1 +DCAE.Undeploy | WARNING | Undeployed components: 1 +``` + +### `dev` + +The `dev` operation is a convenient operation that can be useful for the development and testing of your component. It can be used to: + +* Help validate your experimental component specification before uploading to the catalog +* Generate the application configuration from the component specification and make it available in a test environment. This allows you to view your resulting configuration for local development and to help debug potential related issues. + +Let's say you have a component specification called `component-spec.json`: + +``` +$ dcae_cli component dev component-spec.json +Ready for component development + +Setup these environment varibles. Run "source env_solutioning": + +export DOCKER_HOST=SOME_DOCKER_HOST:2376 +export SERVICE_CHECK_INTERVAL=15s +export CONFIG_BINDING_SERVICE=config_binding_service +export HOSTNAME=mh677g.95740959-63d2-492a-b964-62a6dce2591d.0-6-0.sandbox-platform-laika +export CONSUL_HOST=SOME_CONSUL_HOST +export CDAP_BROKER=cdap_broker +export SERVICE_NAME=mh677g.95740959-63d2-492a-b964-62a6dce2591d.0-6-0.sandbox-platform-laika +export SERVICE_CHECK_TIMEOUT=1s +export SERVICE_CHECK_HTTP=/health + +Press any key to stop and to clean up +``` + +Your application configuration is now available under the name `mh677g.95740959-63d2-492a-b964-62a6dce2591d.0-6-0.sandbox-platform-laika`. + +To view the resulting configuration, you can `curl` a request to the config binding service or programmatically fetch your configuration within your application. + +You need to first query Consul to get the ip and port of config binding service: + +``` +curl http://$CONSUL_HOST:8500/v1/catalog/service/$CONFIG_BINDING_SERVICE +[ + { + "ID": "983d5c94-c508-4a8a-9be3-5912bd09786b", + "Node": "realsolcnsl00", + "Address": "10.226.1.22", + "TaggedAddresses": { + "lan": "10.226.1.22", + "wan": "10.226.1.22" + }, + "NodeMeta": {}, + "ServiceID": "5f371f295c90:config_binding_service:10000", + "ServiceName": "config_binding_service", + "ServiceTags": [], + "ServiceAddress": "XXXX", + "ServicePort": 32770, + "ServiceEnableTagOverride": false, + "CreateIndex": 487, + "ModifyIndex": 487 + } +] +``` + +### DMaaP testing + +Currently, the dcae-cli does not have the capability of provisioning topics. In order to do testing with message router topics or with data router feeds, the developer must provision the topic or the feed manually and provide the connection details in the form of a JSON in a file to the dcae-cli. This file is to be passed in when using the `run` and `dev` commands with the option `--dmaap-file`. + +The structure of the DMaaP JSON is an object of config keys to matching topic or feed connection details. Config keys are the `config_key` values specified in your component specification streams section where the streams must be type message router or data router. Information about the associated connection details can be found on [this page](dmaap-connection-objects.md). Please check it out. + +For example, if you have a component specification that has the following streams entry: + +```json +"streams": { + "publishes": [{ + "format": "ves", + "version": "1.0.0", + "type": "message router", + "config_key": "ves_connection" + }] +} +``` + +Then to deploy and to run your component, you must use the `--dmaap-file` command and pass in a JSON that looks like: + +```json +{ + "ves_connection": { + "type": "message_router", + "dmaap_info": { + "topic_url": "https://we-are-message-router.us:3905/events/some-topic" + } + } +} +``` + +The provided DMaaP JSON is used to simulate the output of provisioning and will be used to merge with the generated application configuration at runtime. + +Your final application config will look like: + +```json +{ + "streams_publishes": { + "ves_connection": { + "type": "message_router", + "dmaap_info": { + "topic_url": "https://we-are-message-router.us:3905/events/some-topic" + } + } + } +} +``` + +#### Data router subscribers + +Note for data router subscriber testing, you will need the delivery url in order to provision the subscriber to the feed. This is constructed at deployment time and will be provided by the dcae-cli after you deploy your component. The delivery urls will be displayed to the screen: + +``` +DCAE.Run | WARNING | Your component is a data router subscriber. Here are the delivery urls: + + some-sub-dr: http://SOME_IP:32838/identity + +``` + +## Publishing component + +Once components have their component specifications crafted and validated and have been tested, components should be published in the shared onboarding catalog using the `publish` sub-command for both data formats and components. You must publish all data formats of a component before publishing a component. + +Publishing will change the status of a component, be made accessible for other developers to use, and will generate the associated TOSCA models for use in designing of compositions. + +``` +dcae_cli component publish sandbox.platform.laika:0.5.0 +``` + +## Shared catalog + +`catalog` is the sub-command used to access and to browse the shared onboarding catalog to view components and data formats that have been published and that are being worked on. Components and data formats have two statuses `staged` and `published`. + +Staged means that the resource has been simply added and is under development. It is to be used only by the owner. Published means that the resource has been fully developed and tested and is ready to be shared. + +Published components can be deployed by non-owners and published data formats can be used in component specifications of non-owners. + +There are two available operations: + +``` +$ dcae_cli catalog --help +Usage: dcae_cli catalog [OPTIONS] COMMAND [ARGS]... + +Options: + --help Show this message and exit. + +Commands: + list + show +``` + +Staged components can be viewed under the `list` operation using the `--expanded` flag. diff --git a/platformdoc/docs/components/intro.md b/platformdoc/docs/components/intro.md new file mode 100644 index 00000000..2e0087c1 --- /dev/null +++ b/platformdoc/docs/components/intro.md @@ -0,0 +1,56 @@ +# Overview + +DCAE components are services that provide a specific functionality and are written to be composable with other DCAE service components. The DCAE platform is responsible for running and managing DCAE service components reliably. + +Currently, the DCAE platform supports two types of components, CDAP applications and Docker containers. For each, there are requirements that must be met for the component to integrate into the DCAE platform (see [CDAP](component-type-docker.md) and [Docker](component-type-docker.md)). + +## Onboarding + +There is an onboarding process that all component developers will have to take their components through to ensure that their DCAE service components are compliant with the DCAE platform and are authorized to push their component into the ASDC catalog. A command-line tool called [`dcae-cli`](ONAP URL TBD) is provided to help you through this process. The high level summary of the onboarding process is: + +1. Defining your [Data formats](data-formats.md) +2. Defining your component specification], which is a JSON that is used to describe and configure the component is to be provided by component developers upon the completion of micro-service implementation and certification testing steps. See [docker](component-specification/docker-specification.md) and [CDAP](component-specification/cdap-specification.md). +3. Testing your component locally +4. Pairwise-testing your component with any other components you connect to +5. Pushing your component and data formats into the ASDC catalog + +## The whys + +### Components require data formats... + +Components are software applications that do some function. Components don't run independently, they depend upon other components. A component's function could require connecting to other components to fulfill that function. A component could also be providing its function as a service through an interface for other components to use. + +The challenge is that a component cannot connect to or be connected with any other component. The upstream and downstream components must *speak* the same vocabulary or *data format*. The output of an upstream component must match your component's input or your component's output must match the downstream component's input. This is necessary for components to function without errors and correctly. + +All components must have data formats. The platform requires this to validate and to ensure that your component will be run with *compatible* components. + +Components *should* have one to many data formats. Data formats should be shared. + +### Components require a component specification... + +One design goal of the DCAE was to have a single place for component developers to describe their component. They should *not* have to worry about Tosca, blueprints, yaml files, etc. The component specification (and it's linked data formats) is the only piece of information about a component that each component developer has to provide. Here are some benefits: + +1. The component specification fully specifies your inputs and outputs, so DCAE knows how it can compose your component. The component specification is where you define what types of other components your component connects with and what types of other components can connect to you. You are defining your inputs and outputs. + +2. The component specification fully specifies all the configuration parameters of your component. This is used by the designer and by policy to configure the runtime behavior of your component. + +3. The component specification is used to *generate* your application configuration in a standardized JSON that the platform will make available. This application configuration JSON will include: + +* Parameters that have been assigned values from you, policy, and/or design +* Connection details of downstream components you are dependent upon + +4. The component specification is transformed by DCAE tooling into Policy models and Cloudify blueprints, so you need not worry about these. + +Every component should have one component specification. + +### Component developers should use the dcae-cli... + +The dcae-cli was developed to empower component developers with developing their components for the DCAE platform. It is intended to help with: + +* Crafting your component specification and your data formats +* Finding and sharing existing data formats +* Finding and running existing components +* Testing and launching your component as it would be run on the platform +* Push your component into the catalog to be shared and to be used by designers + +All this from a single command-line tool. diff --git a/platformdoc/docs/glossary.md b/platformdoc/docs/glossary.md new file mode 100644 index 00000000..a9f23214 --- /dev/null +++ b/platformdoc/docs/glossary.md @@ -0,0 +1,19 @@ +# Glossary + +## ASDC + +The ECOMP resource catalog. We assume the existince and usage of ASDC, though for near term testing purposes, we use a mock version of the catalog. +We assume all DCAE artifacts: components like collectors, CDAP applications, data formats (see below), etc, are onboarded and searchable in the catalog. +Further, we assume that every catalog artifact has a ``UUID", a globally unique identifier that identifies that artifact. + +## Component + +Refers to a DCAE service component which is a single micro-service that is written to be run by the DCAE platform and to be composeable to form a DCAE service. + +## Data format + +Artifact that describes a data structure and schema. + +## Onboarding catalog + +The onboarding catalog is the intermediary store for project details, component specifications and data formats. This is to be used by component developers through the `dcae-cli` to browse and to launch existing components and push their own component artifacts in preparation to move their component to the ASDC resource catalog. diff --git a/platformdoc/docs/images/IO.graffle b/platformdoc/docs/images/IO.graffle Binary files differnew file mode 100644 index 00000000..47c18b2a --- /dev/null +++ b/platformdoc/docs/images/IO.graffle diff --git a/platformdoc/docs/images/dmd vision.graffle b/platformdoc/docs/images/dmd vision.graffle Binary files differnew file mode 100644 index 00000000..c6ad0d51 --- /dev/null +++ b/platformdoc/docs/images/dmd vision.graffle diff --git a/platformdoc/docs/images/dmdvision.png b/platformdoc/docs/images/dmdvision.png Binary files differnew file mode 100644 index 00000000..cc6f195f --- /dev/null +++ b/platformdoc/docs/images/dmdvision.png diff --git a/platformdoc/docs/images/io.png b/platformdoc/docs/images/io.png Binary files differnew file mode 100644 index 00000000..26c5eba9 --- /dev/null +++ b/platformdoc/docs/images/io.png diff --git a/platformdoc/docs/index.md b/platformdoc/docs/index.md new file mode 100644 index 00000000..6b7475fd --- /dev/null +++ b/platformdoc/docs/index.md @@ -0,0 +1,14 @@ +# DCAE Platform + +## Welcome + +This is the home for the documetation for the DCAE platform where you will find: + +* Details of the platform's architecture +* User guides for the different roles + - [Component developers](components/intro.md) + +## Background + +DCAE is composed of a platform and services. The DCAE platform are the technical pieces responsible for the deployment and the management of DCAE services. The DCAE services are the technical pieces responsible for monitoring network services. + |
