diff options
Diffstat (limited to 'platformdoc/docs/components/component-specification')
7 files changed, 0 insertions, 1068 deletions
diff --git a/platformdoc/docs/components/component-specification/cdap-specification.md b/platformdoc/docs/components/component-specification/cdap-specification.md deleted file mode 100644 index ce17d811..00000000 --- a/platformdoc/docs/components/component-specification/cdap-specification.md +++ /dev/null @@ -1,127 +0,0 @@ -# Component specification (CDAP) - -The CDAP component specification contains the following groups of information. Many of these are common to both CDAP and Docker components and are therefore described in the common specification. - -* [Metadata](common-specification.md#metadata) -* [Interfaces](common-specification.md#interfaces) including the associated [Data Formats](/components/data-formats.md) -* [Parameters](#parameters) - for specifying parameters in your AppConfig, AppPreferences, and ProgramPreferences to the Designer and Policy. This of course is CDAP-specific and is described below. -* [Auxiliary Details](#auxiliary-details) -* [List of artifacts](common-specification.md#artifacts) - - -## Current Limitations and TODOs - -* The integration of DMD is likely to significantly change the [Interfaces](common-specification.md#interfaces) section in this specification.. - -## Parameters - -There is a `parameters` section in your component specification. This section contains three optional keys: [app_config](#appconfig), [app_preferences](#apppreferences), and [program_preferences](#programpreferences): -``` -"parameters" : { - "app_config" : [ ...], - "app_preferences" : [ ...], - "program_preferences" : [...] - // any additional keys are ignored -} -``` - -* Each section details the parameters that are a part of each of these CDAP constructs (see below). -* All such parameters will be exposed to the designer and to policy for override. -* These parameters should have default values specified by the component developer where necessary, i.e., parameters that _must_ come from the designer/policy should not have defaults. -* All of these keys are optional because not every CDAP application uses preferences and not every application uses the AppConfig. However, you should specify at least one, or else your application will have no parameters exposed to policy or to the DCAE designer, which means it would be non-configurable. -* Despite the AppConfig being optional to *specify* in the case that you have no parameters in your AppConfig, it is *required for processing* in your application. That is because the DCAE platform will place important information into your AppConfig as discussed below. - -### Parameter - -The following CDAP specific definitions use `param1` to refer to the common parameter layout in [Parameter](common-specification.md#parameters) - -### AppConfig - -The `app_config` key refers to the [CDAP AppConfig](http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/configuration.html). It is expected to be a JSON: -``` -"app_config" : [ // list of JSON - param1, // common parameter layout - ... -] -``` -Unfortunately, at the time of writing, the AppConfig is a Java map of `string:string`, which means you cannot have more complex structures (than string) as any value in your AppConfig. However, there is a way to bypass this constraint: you can pass a JSON by encoding the JSON as a string. E.g., the `json.dumps()` and it's converse `loads` methods in Python: -``` ->>> import json ->>> json.dumps({"foo" : "bar"}) # This is a real JSON -'{"foo": "bar"}' # It is now a string: pass this in as your parameter value ->>> json.loads('{"foo": "bar"}') # Do the equivelent of this in your application -{u'foo': u'bar'} # ...and you'll get back a real JSON ->>> -``` - -The final AppConfig (after the designer and policy override parameter values) is passed into CDAP's AppConfig API when starting the application. - -### AppPreferences - -In addition to the CDAP AppConfig, the platform supports [Application Preferences](http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/preferences.html#set-preferences). -The format of the `app_preferences` value is the same as the above: -``` -"app_preferences" : [ // list of JSON - param1, // common parameter layout - ... -] -``` - -The final Application Preferences JSON (after the designer and policy override parameter values) is passed into CDAP's Preferences API when starting your application. - -### ProgramPreferences - -Preferences can also be specified [per program](http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/lifecycle.html#program-lifecycle) in CDAP. This key's value is a list of JSON with the following format: -``` -"program_preferences" : [ // note: this is a list of JSON - { - "program_id" : "program name 1", // the name of this CDAP program - "program_type" : "e.g., flows", // "must be one of flows, mapreduce, schedules, spark, workflows, workers, or services", - "program_pref" : [ // list of JSON - param1, // common parameter layout - ... - ] - }, - // repeat for each program that receives a program_preferences JSON -] -``` -Each `program_pref` JSON is passed into the CDAP API as the preference for `program_id`. - - -NOTE: for CDAP, this section is very likely to change when DMD is available. -The _future_ vision is that you would publish your data as a series of files on HDFS, and DMD will pick them up and send them to the appropriate DMaaP feeds or directly when needed. - -## Auxiliary Details - -`auxiliary` contains details about CDAP specific parameters. - -Property Name | Type | Description -------------- | ---- | ----------- -streamname | string | *Required*. -artifact_name | string | -artifact_version | string | the version of your CDAP JAR artifact -namespace | string | the CDAP namespace to deploy into, default is 'default' -programs | array | contains each CDAP entity represented in the artifact -program_type | string | CDAP entity (eg "flows") -program_id | string | name of CDAP entity (eg "WhoFlow") - -Example: - -```json -"auxiliary": { - "streamname" : "who", - "artifact_name" : "HelloWorld", - "artifact_version" : "3.4.3", - "namespace" : "hw", - "programs" : [ - {"program_type" : "flows", "program_id" : "WhoFlow"}, - {"program_type" : "services", "program_id" : "Greeting"}, - ... - ], -} -``` -The `programs` key is identical to the `program_preferences` key discussed [above](#programpreferences) except: - -* each JSON in the list does not contain `program_pref` -* this is required! You must include all of your programs in this, as it is used to start each program as well as for DCAE to perform periodic healthchecks on your application. Don't forget about your services; they are programs too. - diff --git a/platformdoc/docs/components/component-specification/common-specification.md b/platformdoc/docs/components/component-specification/common-specification.md deleted file mode 100644 index 14c1297c..00000000 --- a/platformdoc/docs/components/component-specification/common-specification.md +++ /dev/null @@ -1,402 +0,0 @@ -# Common Elements of the Component Specification - -This page describes the component specification (JSON) sections that are common to both Docker and CDAP components. Differences for each are pointed out below. Elements that are very different, and described in the CDAP or Docker specific pages. - -## Meta Schema Definition - -The component specification is represented (and validated) against this -[Component Spec json schema](https://codecloud.web.att.com/projects/ST_DCAECNTR/repos/component-json-schemas/browse/component-spec-schema.json) and described below: - -The "Meta Schema" implementation defines how component specification JSON schemas can be written to define user input. It is itself a JSON schema (thus it is a "meta schema"). It requires the name of the component entry, component type (either 'cdap' or 'docker') and a description under "self" object. The meta schema version must be specified as the value of the "version" key. Then the input schema itself is described. - -The table below shows the four types of schema descriptions supported -There are four types of schema descriptions objects - jsonschema for inline standard JSON Schema definitions of JSON inputs, delimitedschema for delimited data input using a JSON description defined by AT&T, unstructured for unstructured text, and reference that allows a pointer to another artifact for a schema. The reference allows for XML schema, but can be used as a pointer to JSON, Delimited Format, and Unstructured schemas as well. - -## Component Metadata -Metadata refers to the properties found under the `self` JSON. This group of properties is used to uniquely identify this component specification and identify the component that this specification is used to capture. - -Example: - -``` -"self": { - "version": "1.0.0", - "name": "asimov.component.kpi_anomaly", - "description": "Classifies VNF KPI data as anomalous", - "component_type": "docker" -}, -``` - -`self` Schema: - -Property Name | Type | Description -------------- | ---- | ----------- -version | string | *Required*. Semantic version for this specification -name | string | *Required*. Full name of this component which is also used as this component's catalog id. -description | string | *Required*. Human-readable text describing the component and the components functional purpose. -component_type | string | *Required*. Identify what containerization technology this component uses: `docker` or `cdap`. - -## Interfaces -Interfaces are the JSON objects found under the `streams` key and the `services` key. These are used to describe the interfaces that the component uses and the interfaces that the component provides. The description of each interface includes the associated [data format](/components/data-formats.md). - -### Streams - * The `streams` JSON is for specifying data produced for consumption by other components, and the streams expected to subscribe to that is produced by other components. These are "fire and forget" type interfaces where the publisher of a stream does not expect or parse a response from the subscriber. -* The term `stream` here is abstract and neither refers to "CDAP streams" or "DMaaP feeds". While a stream is very likely a DMaaP feed, it could be a direct stream of data being routed via HTTP too. It abstractly refers to a sequence of data leaving a publisher. -* Streams have anonymous publish/subscribe semantics, which decouples the production of information from its consumption. -Like the component specification, the data format specification is represented/validated against this -[Data Format json schema](https://codecloud.web.att.com/projects/ST_DCAECNTR/repos/component-json-schemas/browse/data-format-schema.json) - -* In general, components are not aware of who they are communicating with. -* Instead, components that are interested in data, subscribe to the relevant stream; components that generate data publish to the relevant stream. -* There can be multiple publishers and subscribers to a stream. Streams are intended for unidirectional, streaming communication. - - -Streams interfaces that implement an HTTP endpoint must support POST. - -Streams are split into: - -Property Name | Type | Description -------------- | ---- | ----------- -subscribes | JSON list | *Required*. List of all available stream interfaces that this component has that can be used for subscribing -publishes | JSON list | *Required*. List of all stream interfaces that this component will publish onto - -#### Subscribes - -Example: - -```json -"streams": { - "subscribes": [{ - "format": "dcae.vnf.kpi", - "version": "1.0.0", - "route": "/data", // for CDAP this value is not used - "type": "http" - }], -... -} -``` - -This describes that `asimov.component.kpi_anomaly` exposes an HTTP endpoint called `/data` which accepts requests that have the data format of `dcae.vnf.kpi` version `1.0.0`. - -`subscribes` Schema: - -Property Name | Type | Description -------------- | ---- | ----------- -format | string | *Required*. Data format id of the data format that is used by this interface -version | string | *Required*. Data format version of the data format that is used by this interface -route | string | *Required for HTTP and data router*. The HTTP route that this interface listens on -config_key | string | *Required for message_router and data router*. The HTTP route that this interface listens on -type | string | *Required*. Type of stream: `http`, `message_router`, `data_router` - - -##### Message router - -Message router subscribers are http clients rather than http services and performs a http a `GET` call. Thus, message router subscribers description is structured like message router publishers and requires `config_key`: - -```json -"streams": { - "subscribes": [{ - "format": "dcae.some-format", - "version": "1.0.0", - "config_key": "some_format_handle", - "type": "message router" - }], -... -} -``` - -##### Data router - -Data router subscribers are http or https services that handle `PUT` requests from data router. Developers must provide the `route` or url path/endpoint that is expected to handle data router requests. This will be used to construct the delivery url needed to register the subscriber to the provisioned feed. Developers must also provide a `config_key` because there is dynamic configuration information associated with the feed that the application will need e.g. username and password. See the page on [DMaaP connection objects](/components/component-specification/dmaap-connection-objects/#data-router) for more details on the configuration information. - -Example (not tied to the larger example): - -```json -"streams": { - "subscribes": [{ - "config_key": "some-sub-dr", - "format": "sandbox.platform.any", - "route": "/identity", - "type": "data_router", - "version": "0.1.0" - }], -... -} -``` - -#### Publishes - -Example: - -```json -"streams": { -... - "publishes": [{ - "format": "asimov.format.integerClassification", - "version": "1.0.0", - "config_key": "prediction", - "type": "http" - }] -}, - -``` - -This describes that `asimov.component.kpi_anomaly` publishes by making POST requests to streams that support the data format `asimov.format.integerClassification` version `1.0.0`. - -`publishes` Schema: - -Property Name | Type | Description -------------- | ---- | ----------- -format | string | *Required*. Data format id of the data format that is used by this interface -version | string | *Required*. Data format version of the data format that is used by this interface -config_key | string | *Required*. The JSON key in the generated application configuration that will be used to pass the downstream component's (the subscriber's) connection information. -type | string | *Required*. Type of stream: `http`, `message router`, `data router` - -##### Message router - -Message router publishers are http clients of DMaap message_router. Developers must provide a `config_key` because there is dynamic configuration information associated with the feed that the application needs to receive e.g. topic url, username, password. See the page on [DMaaP connection objects](/components/component-specification/dmaap-connection-objects/#message-router) for more details on the configuration information. - -Example (not tied to the larger example): - -```json -"streams": { -... - "publishes": [{ - "config_key": "some-pub-mr", - "format": "sandbox.platform.any", - "type": "message_router", - "version": "0.1.0" - }] -} -``` - -##### Data router - -Data router publishers are http clients that make `PUT` requests to data router. Developers must also provide a `config_key` because there is dynamic configuration information associated with the feed that the application will need to receive e.g. publish url, username, password. See the page on [DMaaP connection objects](/components/component-specification/dmaap-connection-objects/#data-router) for more details on the configuration information. - -Example (not tied to the larger example): - -```json -"streams": { -... - "publishes": [{ - "config_key": "some-pub-dr", - "format": "sandbox.platform.any", - "type": "data_router", - "version": "0.1.0" - }] -} -``` - -#### Quick Reference - -Refer to this [Quick Reference](/components/component-specification/streams-grid.md) for a comparison of the Streams 'Publishes' and 'Subscribes' sections. - - -### Services - -* The publish / subscribe model is a very flexible communication paradigm, but its many-to-many one-way transport is not appropriate for RPC -request / reply interactions, which are often required in a distributed system. -* Request / reply is done via a Service, which is defined by a pair of messages: one for the request and one for the reply. - -Services are split into: - -Property Name | Type | Description -------------- | ---- | ----------- -calls | JSON list | *Required*. List of all service interfaces that this component will call -provides | JSON list | *Required*. List of all service interfaces that this component exposes and provides - -#### Calls -The JSON `services/calls` is for specifying that the component relies on an HTTP(S) service---the component sends that service an HTTP request, and that service responds with an HTTP reply. -An example of this is how string matching (SM) depends on the AAI Broker. SM performs a synchronous REST call to the AAI broker, providing it the VMNAME of the VNF, and the AAI Broker responds with additional details about the VNF. This dependency is expressed via `services/calls`. In contrast, the output of string matching (the alerts it computes) is sent directly to policy as a fire-and-forget interface, so that is an example of a `stream`. - -Example: - -```json -"services": { - "calls": [{ - "config_key": "vnf-db", - "request": { - "format": "dcae.vnf.meta", - "version": "1.0.0" - }, - "response": { - "format": "dcae.vnf.kpi", - "version": "1.0.0" - } - }], -... -} -``` - -This describes that `asimov.component.kpi_anomaly` will make HTTP calls to a downstream component that accepts requests of data format `dcae.vnf.meta` version `1.0.0` and is expecting the response to be `dcae.vnf.kpi` version `1.0.0`. - -`calls` Schema: - -Property Name | Type | Description -------------- | ---- | ----------- -request | JSON object | *Required*. Description of the expected request for this downstream interface -response | JSON object | *Required*. Description of the expected response for this downstream interface -config_key | string | *Required*. The JSON key in the generated application configuration that will be used to pass the downstream component connection information. - -The JSON object schema for both `request` and `response`: - -Property Name | Type | Description -------------- | ---- | ----------- -format | string | *Required*. Data format id of the data format that is used by this interface -version | string | *Required*. Data format version of the data format that is used by this interface - -#### Provides - -Example: - -```json -"services": { -... - "provides": [{ - "route": "/score-vnf", - "request": { - "format": "dcae.vnf.meta", - "version": "1.0.0" - }, - "response": { - "format": "asimov.format.integerClassification", - "version": "1.0.0" - } - }] -}, -``` - -This describes that `asimov.component.kpi_anomaly` provides a service interface and it is exposed on the `/score-vnf` HTTP endpoint. The endpoint accepts requests that have the data format `dcae.vnf.meta` version `1.0.0` and gives back a response of `asimov.format.integerClassification` version `1.0.0`. - -`provides` Schema for a Docker component: - -Property Name | Type | Description -------------- | ---- | ----------- -request | JSON object | *Required*. Description of the expected request for this interface -response | JSON object | *Required*. Description of the expected response for this interface -route | string | *Required*. The HTTP route that this interface listens on - -The JSON object schema for both `request` and `response`: - -Property Name | Type | Description -------------- | ---- | ----------- -format | string | *Required*. Data format id of the data format that is used by this interface -version | string | *Required*. Data format version of the data format that is used by this interface - -Note, for CDAP, there is a slight variation due to the way CDAP exposes services: -``` - "provides":[ // note this is a list of JSON - { - "request":{ ...}, - "response":{ ...}, - "service_name":"name CDAP service", - "service_endpoint":"greet", // E.g the URL is /services/service_name/methods/service_endpoint - "verb":"GET" // GET, PUT, or POST - } - ] -``` - -`provides` Schema for a CDAP component: - -Property Name | Type | Description -------------- | ---- | ----------- -request | JSON object | *Required*. Description of the expected request data format for this interface -response | JSON object | *Required*. Description of the expected response for this interface -service_name | string | *Required*. The CDAP service name (eg "Greeting") -service_endpoint | string | *Required*. The CDAP service endpoint for this service_name (eg "/greet") -verb | string | *Required*. 'GET', 'PUT' or 'POST' - - -## Parameters - -`parameters` is where to specify the component's application configuration parameters that are not connection information. - -Property Name | Type | Description -------------- | ---- | ----------- -parameters | JSON array | Each entry is a parameter object - -Parameter object has the following available properties: - -Property Name | Type | Description | Default -------------- | ---- | ----------- | ------- -name | string | *Required*. The property name that will be used as the key in the generated config | -value | any | *Required*. The default value for the given parameter | -description | string | *Required*. Human-readable text describing the parameter like what its for | -type | string | The required data type for the parameter | -required | boolean | An optional key that declares a parameter as required (true) or not (false) | true -constraints | array | The optional list of sequenced constraint clauses for the parameter. See below | -entry_schema | string | The optional key that is used to declare the name of the Datatype definition for entries of set types such as the TOSCA 'list' or 'map'. Only 1 level is supported at this time | -designer_editable | boolean | An optional key that declares a parameter to be editable by designer (true) or not (false) | -sourced_at_deployment | boolean | An optional key that declares a parameter's value to be assigned at deployment time (true) | -policy_editable | boolean | An optional key that declares a parameter to be editable by policy (true) or not (false) | -policy_schema | array | The optional list of schema definitions used for policy. See below | - -Example: - -```json -"parameters": [ - { - "name": "threshold", - "value": 0.75, - "description": "Probability threshold to exceed to be anomalous" - } -] -``` - -Many of the parameter properties have been copied from TOSCA model property definitions and are to be used for service design composition and policy creation. See [section 3.5.8 *Property definition*](http://docs.oasis-open.org/tosca/TOSCA-Simple-Profile-YAML/v1.1/TOSCA-Simple-Profile-YAML-v1.1.html). - -The property `constraints` is a list of objects where each constraint object: - -Property Name | Type | Description -------------- | ---- | ----------- -equal | | Constrains a property or parameter to a value equal to (‘=’) the value declared -greater_than | number | Constrains a property or parameter to a value greater than (‘>’) the value declared -greater_or_equal | number | Constrains a property or parameter to a value greater than or equal to (‘>=’) the value declared -less_than | number | Constrains a property or parameter to a value less than (‘<’) the value declared -less_or_equal | number | Constrains a property or parameter to a value less than or equal to (‘<=’) the value declared -valid_values | array | Constrains a property or parameter to a value that is in the list of declared values -length | number | Constrains the property or parameter to a value of a given length -min_length | number | Constrains the property or parameter to a value to a minimum length -max_length | number | Constrains the property or parameter to a value to a maximum length - -`threshold` is the configuration parameter and will get set to 0.75 when the configuration gets generated. - -The property `policy_schema` is a list of objects where each policy_schema object: - -Property Name | Type | Description | Default -------------- | ---- | ----------- | ------- -name | string | *Required*. parameter name -value | string | default value for the parameter -description | string | parameter description -type | enum | *Required*. data type of the parameter, 'string', 'number', 'boolean', 'datetime', 'list', or 'map' -required | boolean | is parameter required or not? | true -constraints | array | The optional list of sequenced constraint clauses for the parameter. See above | -entry_schema | string | The optional key that is used to declare the name of the Datatype definition for certain types. entry_schema must be defined when the type is either list or map. If the type is list and the entry type is a simple type (string, number, bookean, datetime), follow with an string to describe the entry | -| | If the type is list and the entry type is a map, follow with an array to describe the keys for the entry map | -| | If the type is list and the entry type is a list, that is not currently supported -| | If the type is map, follow with an aray to describe the keys for the map | - -## Generated Application Configuration - -The Platform generates configuration for the component (based on the component spec) at deployment time. The generated application configuration will be made up of the Parameters, Streams, and Services sections, after any provisioning for streams and services. The component developer can see what this configuration will look like by reviewing the [component dev command](/components/dcae_cli/commands/#run-the-dev-command). - -## Artifacts - -`artifacts` contains a list of artifacts associated with this component. For Docker, this is the full path (including the registry) to the Docker image. For CDAP, this is the full path to the CDAP jar. - -Property Name | Type | Description -------------- | ---- | ----------- -artifacts | JSON array | Each entry is a artifact object - -`artifact` Schema: - -Property Name | Type | Description -------------- | ---- | ----------- -uri | string | *Required*. Uri to the artifact, full path -type | string | *Required*. `docker image` or `jar` - -# Working with Component Specs - -Components can be added to the onboarding catalog (which first validates the component spec) by using the [dcae_cli Tool](http://dcae-platform.research.att.com/components/dcae-cli/quickstart/). Here you can also list the components, show the contents of a component, publish the component, validate the generated configuration for the component, deploy (run) and undeploy the component. For a list of these capabilities, see [Component Commands](/components/dcae-cli/commands/#component). - - diff --git a/platformdoc/docs/components/component-specification/configuration-grid.md b/platformdoc/docs/components/component-specification/configuration-grid.md deleted file mode 100644 index e1154218..00000000 --- a/platformdoc/docs/components/component-specification/configuration-grid.md +++ /dev/null @@ -1,19 +0,0 @@ - -# Configuration Quick Reference - -#### Default Values - -The component developer can provide default values for any `parameter` in the component specification. These defaults will be passed to the component in its generated configuration. - -#### Overridden/Entered Values - -Depending on the other properties set for the parameter, the default value can be overridden at 'design-time', 'deploy-time' or once the microservice is running ('run-time'). -(*In the future, when Policy is supported, configuration will also be able to be provided/changed in the Policy UI at any time).* - -||Design-Time Input |CLAMP Input|Policy Input (future) |Deploy-Time Input|Run-Time Input (DTI)| -|-----|-----|-----|-----|----------|----------| -|Description|Applies to SDC self-service components|Applies to components deployed by CLAMP |(not yet supported)|Applies to manually deployed services| Applies to components supporting DTI reconfiguration| -|Input provided by|Service Designer|CLAMP|Operations|DevOps|Runtime Platform(DTI)| -|How it is provided |In the SDC UI |In the CLAMP UI |In the POLICY GUI |In the DCAE Dashboard (or Jenkins job)|In the DTI Event -|Component Specification Details|‘designer-editable’ set to ‘true’| None. Developer provides CLAMP an email with parameters to be supported|‘policy_editable’ must be set to ‘true’ and ‘policy_schema’ must be provided|'sourced_at_<br>deployment' must be set to 'true'|parameter 'dcae_target_type' defined with default value set to supported vnfType-vnfFuncId, with properties 'designer_editable' and 'sourced_at_deployment' set appropriately| -|Additional Info for Component Developer|||For Docker only: In the auxiliary section:<br> {"policy": {"trigger_type": "policy","script_path": “/opt/app/reconfigure.sh”} }<br> Script interface would then be "/opt/app/reconfigure.sh” $trigger_type $updated_policy" <br> where $updated_policy is json provided by the Policy Handler.||For Docker only: In the auxiliary section:<br> {"dti": “/opt/app/reconfigure.sh”} <br> Script interface would then be "/opt/app/reconfigure.sh” $trigger_type $updated_dti" <br> where $updated_dti is json provided by the DTI Plugin.| diff --git a/platformdoc/docs/components/component-specification/dmaap-connection-objects.md b/platformdoc/docs/components/component-specification/dmaap-connection-objects.md deleted file mode 100644 index b1b650fd..00000000 --- a/platformdoc/docs/components/component-specification/dmaap-connection-objects.md +++ /dev/null @@ -1,169 +0,0 @@ -# DMaaP connection objects - -DMaaP Connection Objects are JSON objects that: - -1. At Runtime - this is generated by the DCAE Platform and passed to the component in its application_configuration to be used to connect to the DMaaP feed or topic. Components will receive the entire object with all properties populated (default will be `null) unless specified otherwise. -2. During dcae_cli testing - this is provided through the command-line argument `--dmaap-file` to test the component with manually provisioned feeds and topics. Developers are not required to provide the entire object. The required properties are labeled below with "*Required as input*". - -## Message Router - -Publishers and subscribers have the same generated `Dmaap Connection Object` structure. Here's an example for any given config-key: -(This is what will be in application_configuration) - -```json -{ - "type": "message_router", - "aaf_username": "some-user", - "aaf_password": "some-password", - "dmaap_info": { - "client_role": "com.dcae.member", - "client_id": "1500462518108", - "location": "mtc00", - "topic_url": "https://we-are-message-router.us:3905/events/some-topic" - } -} -``` - -At the top-level: - -Property Name | Type | Description -------------- | ---- | ----------- -type | string | *Required as input*. Must be `message_router` for message router topics -aaf_username | string | AAF username message router clients use to authenticate with secure topics -aaf_password | string | AAF password message router clients use to authenticate with secure topics -dmaap_info | JSON object | *Required as input*. Contains the topic connection details - -The `dmaap_info` object contains: - -Property Name | Type | Description -------------- | ---- | ----------- -client_role | string | AAF client role that's requesting publish or subscribe access to the topic -client_id | string | Client id for given AAF client -location | string | DCAE location for the publisher or subscriber, used to set up routing -topic_url | string | *Required as input*. URL for accessing the topic to publish or receive events - -The --dmaap-file argument (to the component `run` or `dev` command), must minimally contain: - -```json -{ - "type": "message_router", - "dmaap_info": { - "topic_url": "https://we-are-message-router.us:3905/events/some-topic" - } -} -``` - -## Data Router - -### Publisher - -Here's an example of what the generated `Dmaap Connection Object` for Data Router Publisher looks like: -(This is what will be in application_configuration) - -```json -{ - "type": "data_router", - "dmaap_info": { - "location": "mtc00", - "publish_url": "https://we-are-data-router.us/feed/xyz", - "log_url": "https://we-are-data-router.us/feed/xyz/logs", - "username": "some-user", - "password": "some-password", - "publisher_id": "123456" - } -} -``` - -At the top-level: - -Property Name | Type | Description -------------- | ---- | ----------- -type | string | *Required as input*. Must be `data_router` for data router feeds -dmaap_info | JSON object | *Required as input*. Contains the feed connection details - -The `dmaap_info` object contains: - -Property Name | Type | Description -------------- | ---- | ----------- -location | string | DCAE location for the publisher, used to set up routing -publish_url | string | *Required as input*. URL to which the publisher makes Data Router publish requests -log_url | string | URL from which log data for the feed can be obtained -username | string | Username the publisher uses to authenticate to Data Router -password | string | Password the publisher uses to authenticate to Data Router -publisher_id | string | Publisher id in Data Router - -The --dmaap-file argument (to the component `run` or `dev` command), must minimally contain: - -```json -{ - "type": "data_router", - "dmaap_info": { - "publish_url": "https://we-are-data-router.us/feed/xyz" - } -} -``` - -### Subscriber - -Here's an example of what the generated `Dmaap Connection Object` for a Data Router Subscriber looks like: -(This is what will be passed in application_configuration) - -```json -{ - "type": "data_router", - "dmaap_info": { - "location": "mtc00", - "delivery_url": "https://my-subscriber-app.dcae:8080/target-path", - "username": "some-user", - "password": "some-password", - "subscriber_id": "789012" - } -} -``` - -At the top-level: - -Property Name | Type | Description -------------- | ---- | ----------- -type | string | *Required as input*. Must be `data_router` for data router feeds -dmaap_info | JSON object | *Required as input*. Contains the feed connection details - -The `dmaap_info` object contains: - -Property Name | Type | Description -------------- | ---- | ----------- -location | string | DCAE location for the subscriber, used to set up routing -delivery_url | string | URL to which the Data Router should deliver files -username | string | Username Data Router uses to authenticate to the subscriber when delivering files -password | string | Password Data Router uses to authenticate to the subscriber when delivering files -subscriber_id | string | Subscriber id in Data Router - -The --dmaap-file argument (to the component `run` or `dev` command), must minimally contain: - -```json -{ - "type": "data_router", - "dmaap_info": { - } -} -``` - -It is the recommended security practice that `username` and `password` are specified. -You cannot provide the `delivery_url` in your dmaap-file, because it's not constructed until deployment time. Therefore, after the test deployment, go back to the Data Router Feed and provide the delivery_url (in order to start receiving the feeds). - -### Data Router Example - -(After the Data Router feed has been manually provisioned) - -``` -$ dcae_cli component run --dmaap-file $dmaap_file $component-name -DCAE.Run | WARNING | Your component is a data router subscriber. Here are the delivery urls: - - some-sub-dr: http://135.205.226.128:32838/identity -``` - -(Update the Data Router Feed to provide the delivery_url). - - - - diff --git a/platformdoc/docs/components/component-specification/docker-specification.md b/platformdoc/docs/components/component-specification/docker-specification.md deleted file mode 100644 index 60905046..00000000 --- a/platformdoc/docs/components/component-specification/docker-specification.md +++ /dev/null @@ -1,273 +0,0 @@ -# Component specification (Docker) - -The Docker component specification contains the following groups of information. Many of these are common to both Docker and CDAP components and are therefore described in the common specification. - -* [Metadata](common-specification.md#metadata) -* [Interfaces](common-specification.md#interfaces) including the associated [Data Formats](/components/data-formats.md) -* [Parameters](common-specification.md#parameters) -* [Auxiliary Details](#auxiliary-details) -* [List of Artifacts](common-specification.md#artifacts) - -## Auxiliary Details - -`auxiliary` contains Docker specific details like health check, port mapping, volume mapping, dti and policy reconfiguration script details. (Policy reconfiguration is not yet supported). - -Name | Type | Description -------------- | ---- | ----------- -healthcheck | JSON object | *Required*. Health check definition details -ports | JSON array | each array item maps a container port to the host port. See example below. -volume | JSON array | each array item contains a host and container object. See example below. -reconfigs | string | DTI reconfiguration script details -*Planned for 1806* | | -policy | JSON array | *Required*. Policy reconfiguration script details - -### Health Check Definition - -The platform uses Consul to perform periodic health check calls. Consul provides different types of [check definitions](https://www.consul.io/docs/agent/checks.html). The platform currently supports http and docker health checks. - -When choosing a value for interval, consider that too frequent healthchecks will put unnecessary load on Consul and DCAE. If there is a problematic resource, then more frequent healthchecks are warranted (eg 15s or 60s), but as stablility increases, so can these values, (eg 300s). - -When choosing a value for timeout, consider that too small a number will result in increasing timeout failures, and too large a number will result in a delay in the notification of the resource problem. A suggestion is to start with 5s and work from there. - -#### http - -Property Name | Type | Description -------------- | ---- | ----------- -type | string | *Required*. `http` -interval | string | Interval duration in seconds i.e. `60s` -timeout | string | Timeout in seconds i.e. `5s` -endpoint | string | *Required*. GET endpoint provided by the component for Consul to call to check health - -Example: - -```json -"auxilary": { - "healthcheck": { - "type": "http", - "interval": "15s", - "timeout": "1s", - "endpoint": "/my-health" - } -} -``` - -#### docker script example - -Property Name | Type | Description -------------- | ---- | ----------- -type | string | *Required*. `docker` -interval | string | Interval duration in seconds i.e. `15s` -timeout | string | Timeout in seconds i.e. `1s` -script | string | *Required*. Full path of script that exists in the Docker container to be executed - -Consul will use the [Docker exec API](https://docs.docker.com/engine/api/v1.29/#tag/Exec) to periodically call your script in your container. It will examine the script result to identify whether your component is healthy. Your component is considered healthy when the script returns `0` otherwise your component is considered not healthy. - -Example: - -```json -"auxilary": { - "healthcheck": { - "type": "docker", - "script": "/app/resources/check_health.py", - "timeout": "30s", - "interval": "180s" - } -} -``` - -### Ports - -This method of exposing/mapping a local port to a host port is NOT RECOMMENDED because of the possibility of port conflicts. If multiple instances of a docker container will be running, there definitely will be port conflicts. Use at your own risk. (The preferred way to expose a port is to do so in the Dockerfile as described [here](/components/component-type-docker/#ports)). - -```json -"auxilary": { - "ports": ["8080:8000"] -} -``` - -In the example above, container port 8080 maps to host port 8000. - -### Volume Mapping - -```json -"auxilary": { - "volumes": [ - { - "container": { - "bind": "/tmp/docker.sock", - "mode": "ro" - }, - "host": { - "path": "/var/run/docker.sock" - } - } - ] -} -``` - -At the top-level: - -Property Name | Type | Description -------------- | ---- | ----------- -volumes | array | Contains container and host objects - -The `container` object contains: - -Property Name | Type | Description -------------- | ---- | ----------- -bind | string | path to the container volume -mode | string | "ro" - indicates read-only volume - | | "" - indicates that the contain can write into the bind mount - -The `host` object contains: - -Property Name | Type | Description -------------- | ---- | ----------- -path | string | path to the host volume - -Here's an example of the minimal JSON that must be provided as an input: - -```json -"auxilary": { - "volumes": [ - { - "container": { - "bind": "/tmp/docker.sock" - }, - "host": { - "path": "/var/run/docker.sock" - } - } - ] -} -``` - -In the example above, the container volume "/tmp/docker.sock" maps to host volume "/var/run/docker.sock". - -### DTI Reconfiguration - -DTI changes will be provided to the Docker component by triggering a script that is defined here. - -Property Name | Type | Description -------------- | ---- | ----------- -dti | string | *Required*. Suggested value is "/opt/app/reconfigure.sh" - -Example: - -```json -"auxilary": { - "dti": "/opt/app/reconfigure.sh" -} -``` - -The docker script interface is as follows: <br><br> `/opt/app/reconfigure.sh $reconfigure_type {<updated_dti object>} - -Name | Type | Description ----- | ---- | ----------- -reconfigure_type | string | "dti" -updated_dti | json | dti_event object - -The dti_event object can be seen [here](https://codecloud.web.att.com/projects/ST_DCAE/repos/com.att.dcae.orch.dti-handler/browse/dti_inputs.yaml). - -An example of a DTI reconfiguration script can be found [here](https://codecloud.web.att.com/projects/ST_DCAECNTR/repos/docker-cloudify/browse/examples/reconfigure.py?at=refs%2Fheads%2Frelease%2F1802). - - -### Policy (not yet supported) - -Policy changes made in the Policy UI will be provided to the Docker component by triggering a script that is defined here. - -Property Name | Type | Description -------------- | ---- | ----------- -reconfigure_type | string | *Required*. Current value supported is `policy` -script_path | string | *Required*. Current value for 'policy' reconfigure_type must be "/opt/app/reconfigure.sh" - -Example: - -```json -"auxilary": { - "policy": { - "reconfigure_type": "policy", - "script_path": "/opt/app/reconfigure.sh" - } -} -``` - -The docker script interface is as follows: <br><br> `/opt/app/reconfigure.sh $reconfigure_type {"updated policies": <updated policies object>, "application config": <applcation config object>} - -Name | Type | Description ----- | ---- | ----------- -reconfigure_type | string | "policy" -updated_policies | json | TBD -updated_appl_config | json | complete generated app_config, not fully-resolved, but `policy-enabled` parameters have been updated. In order to get the complete updated app_config, the component would have to call `config-binding-service`. - - -## Docker Component Spec - Complete Example - -```json -{ - "self": { - "version": "1.0.0", - "name": "asimov.component.kpi_anomaly", - "description": "Classifies VNF KPI data as anomalous", - "component_type": "docker" - }, - "streams": { - "subscribes": [{ - "format": "dcae.vnf.kpi", - "version": "1.0.0", - "route": "/data", - "type": "http" - }], - "publishes": [{ - "format": "asimov.format.integerClassification", - "version": "1.0.0", - "config_key": "prediction", - "type": "http" - }] - }, - "services": { - "calls": [{ - "config_key": "vnf-db", - "request": { - "format": "dcae.vnf.meta", - "version": "1.0.0" - }, - "response": { - "format": "dcae.vnf.kpi", - "version": "1.0.0" - } - }], - "provides": [{ - "route": "/score-vnf", - "request": { - "format": "dcae.vnf.meta", - "version": "1.0.0" - }, - "response": { - "format": "asimov.format.integerClassification", - "version": "1.0.0" - } - }] - }, - "parameters": [ - { - "name": "threshold", - "value": 0.75, - "description": "Probability threshold to exceed to be anomalous" - } - ], - "auxilary": { - "healthcheck": { - "type": "http", - "interval": "15s", - "timeout": "1s", - "endpoint": "/my-health" - } - }, - "artifacts": [{ - "uri": "fake.nexus.att.com/dcae/kpi_anomaly:1.0.0", - "type": "docker image" - }] -} -``` - diff --git a/platformdoc/docs/components/component-specification/examples-grid.md b/platformdoc/docs/components/component-specification/examples-grid.md deleted file mode 100644 index 9597e537..00000000 --- a/platformdoc/docs/components/component-specification/examples-grid.md +++ /dev/null @@ -1,18 +0,0 @@ -# Component Spec Examples - -The following table shows CDAP and Docker component spec examples for some of the commonly used platform features. Please note that these examples were chosen for the specific features that they contain, and should not necessarily be used for examples of other features, as they may have been captured before they were complete or modified to best utilize those features. - -| Feature | CDAP Example | Docker Example | -|---------|--------------|----------------| -| Dmaap MR subscriber | [String Matching](https://codecloud.web.att.com/projects/ST_DCAECNTR/repos/dcae_cli_examples/browse/sm/sm_spec.json) | TBD | -| Dmaap MR publisher | [String Matching](https://codecloud.web.att.com/projects/ST_DCAECNTR/repos/dcae_cli_examples/browse/sm/sm_spec.json) | [Syslog spec](https://codecloud.web.att.com/projects/ST_DCAECNTR/repos/dcae_cli_examples/browse/syslog/syslog.json) | -| Dmaap DR subscriber/DMD | TBD | [Sessionization](https://codecloud.web.att.com/projects/ST_DCAECNTR/repos/dcae_cli_examples/browse/sessionization/sessionization-componentspec.json) | -| Dmaap DR publisher/DMD | TBD | [Sessionization](https://codecloud.web.att.com/projects/ST_DCAECNTR/repos/dcae_cli_examples/browse/sessionization/sessionization-componentspec.json) | -| Policy (simple) | [tca_policy2](https://codecloud.web.att.com/projects/ST_DCAECNTR/repos/dcae_cli_examples/browse/tca_with_policy2/tca_with_policy2.json) | TBD | -| Policy (complex) | [tca_policy](https://codecloud.web.att.com/projects/ST_DCAECNTR/repos/dcae_cli_examples/browse/tca_with_policy/tca_with_policy.json) | TBD | -| Healthcheck | N/A (the platform does this) | [FOI spec](https://codecloud.web.att.com/projects/ST_DCAECNTR/repos/dcae_cli_examples/browse/foi/foisftpcollector_componentspec.json) -| Service Calls | [String Matching](https://codecloud.web.att.com/projects/ST_DCAECNTR/repos/dcae_cli_examples/browse/sm/sm_spec.json) | TBD | -| Service Provides | [cuda](https://codecloud.web.att.com/projects/ST_DCAECNTR/repos/dcae_cli_examples/browse/cuda/spec.json) | [VES](https://codecloud.web.att.com/projects/ST_DCAECNTR/repos/dcae_cli_examples/browse/ves/vescollector-componentspec.json) | -| DTI Reconfig | TBD | [FOI-sftp](https://codecloud.web.att.com/projects/ST_DCAECNTR/repos/dcae_cli_examples/browse/foi-sftp/component_spec.json) | - - diff --git a/platformdoc/docs/components/component-specification/streams-grid.md b/platformdoc/docs/components/component-specification/streams-grid.md deleted file mode 100644 index b6d64745..00000000 --- a/platformdoc/docs/components/component-specification/streams-grid.md +++ /dev/null @@ -1,60 +0,0 @@ - -# Streams Formatting Quick Reference - -Each of the following tables represents an example of a publisher and its subscriber, which are of course, different components. This focuses on the fields that are ‘different’ for each of these TYPEs, to illustrate the relationship between `config_key`, dmaap connection object, and the generated configuration. Some notes on specific properties: - -* `config_key` is an arbitrary string, chosen by the component developer. It is returned in the generated configuration where it contains specific values for the target connection -* `format`, `version`, and `type` properties in the subscriber would match these properties in the publisher -* `aaf_username` and `aaf_password` may be different between the publisher and the subscriber - - - -### Using http - -#### *Publishing Component* - -| component spec | runtime platform generated config | -|----------------|-----------------------------------| -|"streams":{<br> "publishes":[{<br> "config_key":"prediction",<br> "format":"some-format",<br> "type":"http",<br> "version":"0.1.0"<br> }]<br>}<br>|"streams_publishes":{<br> "prediction":["10.100.1.100:32567/data"] | - -#### *Subscribing Component* - -| component spec | runtime platform generated config | -|----------------|-----------------------------------| -|"streams":{<br> "subscribes":[{<br> "route":"/data",<br> "format":"some-format",<br> "type":"http",<br> "version":"0.1.0"<br> }]<br>}<br>|"N/A"| - - - -### Using Message Router - -#### *Publishing Component* - -Note: When deploying, this component should be deployed first so satisfy downstream dependencies. Refer to the –force option in component ‘run’ command for more information. - -| component spec | Dmaap Connection Object | runtime platform generated config | -|----------------|-------------------------| --------------------------------- | -|"streams":{<br> "publishes":[{<br> "config_key":"mr_output",<br> "format":"some-format",<br> "type":"message_router",<br> "version":"0.1.0"<br> }]<br>} | {<br> "type":"message_router",<br> "dmaap_info": {<br> "topic_url": "https://we-are-message-router.us:3905/events/some-topic" }<br>} <br><br>*Note: For message router, this object is identical for the publisher and the subscriber* | "streams_publishes":{<br> "mr_output":{<br> "aaf_username":"pub-user",<br> "aaf_password":"pub-pwd",<br> "type":"message_router",<br> "dmaap_info":{<br> "topic_url":"https://we-are-message-router.us:3905/events/some-topic"}<br> }<br>},<br> "streams_subscribes":{<br>…<br>} - -#### *Subscribing Component* - -| component spec | Dmaap Connection Object | runtime platform generated config | -|----------------|-------------------------| --------------------------------- | -|"streams":{<br> "subscribes":[{<br> "config_key":"mr_input",<br> "format":"some-format",<br> "type":"message_router",<br> "version":"0.1.0"<br> }]<br>} | {<br> "type":"message_router",<br> "dmaap_info": {<br> "topic_url": "https://we-are-message-router.us:3905/events/some-topic" }<br>} <br><br>*Note: For message router, this object is identical for the publisher and the subscriber* | "streams_publishes":{<br>…<br>},<br> "streams_subscribes":{<br> "mr_input":{<br> "aaf_username":"sub-user",<br> "aaf_password":"sub-pwd",<br> "type":"message_router",<br> "dmaap_info":{<br> "topic_url":"https://we-are-message-router.us:3905/events/some-topic"}<br> }<br>} - - - - -### Using Data Router - -#### *Publishing Component* - -| component spec | Dmaap Connection Object | runtime platform generated config | -|----------------|-------------------------| --------------------------------- | -|"streams":{<br> "publishes":[{<br> "config_key":"dr_output",<br> "format":"some-format",<br> "type":"data_router",<br> "version":"0.1.0"<br> }]<br>} | {<br> "type":"data_router",<br> "dmaap_info": {<br> "location": "mtc00",<br> "publish_url": "https://we-are-data-router.us/feed/xyz", <br> "log_url": "https://we-are-data-router.us/feed/xyz/logs",<br> "username": "pub-user",<br> "password": "pub-password",<br> "publisher_id": "123456"}<br>} | streams_publishes":{<br> "dr_output":{<br> "type":"data_router",<br> "dmaap_info":{<br> "location":"mtc00",<br> "publish_url":"https://we-are-data-router.us/feed/xyz",<br> "log_url":"https://we-are-data-router.us/feed/xyz/logs",<br> "username":"pub-user",<br> "password":"pub-password",<br> "publisher_id":"123456"}<br> }<br>},<br> "streams_subscribes":{<br> …<br> } - -#### *Subscribing Component* - -| component spec | Dmaap Connection Object | runtime platform generated config | -|----------------|-------------------------| --------------------------------- | -|"streams":{<br> "subscribes":[{<br> "config_key":"dr_input",<br> "format":"some-format",<br> "type":"data_router",<br> "version":"0.1.0",<br> "route":"/target-path"<br> }]<br>} | {<br> "type":"data_router",<br> "dmaap_info": {<br> "location": "mtc00",<br> "delivery_url": "https://my-subscriber-app.dcae:8080/target-path", <br> "username": "sub-user",<br> "password": "sub-password",<br> "subscriber_id": "789012"}<br>} | "streams_publishes":{<br> …<br> },<br> "streams_subscribes":{<br> "dr_input":{<br> "type":"data_router",<br> "dmaap_info":{<br> "location":"mtc00",<br> "delivery_url":"https://my-subscriber-app.dcae:8080/target-path",<br> "username":"sub-user",<br> "password":"sub-password",<br> "subscriber_id":"789012"}<br> }<br>} - |