diff options
| author |   Ralph Knag <rhknag@research.att.com> | 2017-12-05 12:05:57 -0500 |
|---|---|---|
| committer | Ralph Knag <rhknag@research.att.com> | 2017-12-05 22:45:35 -0500 |
| commit | 1fca6acb2918beef86a5a78dc659683830908cd2 (patch) | |
| tree | 5854d05874e1e52b58a4836d653b0d1728dec304 | |
| parent | 3ffda69097e20d0fcff44f837c6f68870e0a4528 (diff) | |
DCAE Controller documentation DCAEGEN2-213
Issue-ID: DCAEGEN2-213
Change-Id: I7f2023b7f88b73eef852eca0bbf9086e14903cd6
Signed-off-by: Ralph Knag <rhknag@research.att.com>
24 files changed, 5592 insertions, 0 deletions
diff --git a/docs/index.rst b/docs/index.rst index e2ea600e..253d7d67 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -19,5 +19,6 @@ Data Collection, Analytics, and Events (DCAE) ./sections/configuration.rst ./sections/administration.rst ./sections/humaninterfaces.rst + ./sections/components/component-development.rst ./sections/release-notes.rst diff --git a/docs/sections/components/architecture/pieces.rst b/docs/sections/components/architecture/pieces.rst new file mode 100755 index 00000000..9e57c02b --- /dev/null +++ b/docs/sections/components/architecture/pieces.rst @@ -0,0 +1,11 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+Platform technologies
+=====================
+
+- `Cloudify <http://getcloudify.org/>`__
+- `Consul <https://www.consul.io/>`__
+- `Docker <https://www.docker.com/>`__
+- `CDAP <https://cdap.io/>`__
+- `Registrator <https://github.com/gliderlabs/registrator>`__
diff --git a/docs/sections/components/architecture/service-discovery.rst b/docs/sections/components/architecture/service-discovery.rst new file mode 100755 index 00000000..1eeaef88 --- /dev/null +++ b/docs/sections/components/architecture/service-discovery.rst @@ -0,0 +1,36 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+Service Discovery
+=================
+
+Service discovery is an architecture pattern used for components
+(micro-services) to locate each other. The DCAE platform uses
+`server-side
+discovery <http://microservices.io/patterns/server-side-discovery.html>`__
+and is using `Consul <https://www.consul.io/>`__ as the service registry
+solution.
+
+Service registration
+--------------------
+
+All components are required to register with Consul in order to be
+discovered. There are two methods of registration: self and 3rd party.
+The DCAE platform uses 3rd party registration which means components
+don’t actually make the registration calls but defers that
+responsibility to a platform service.
+
+Implementation for Docker
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+`Registrator <http://gliderlabs.com/registrator/latest/>`__ is an open
+source application that is responsible for registering all components
+that run as Docker containers. Registrator watches the local Docker
+engine’s activity log and will register and unregister a Docker
+container when the container is started and stopped.
+
+Implementation for CDAP
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The CDAP broker is a REST web service that is responsible for
+registering all components that run as CDAP applications.
diff --git a/docs/sections/components/architecture/services.rst b/docs/sections/components/architecture/services.rst new file mode 100755 index 00000000..9cfd084c --- /dev/null +++ b/docs/sections/components/architecture/services.rst @@ -0,0 +1,19 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+‘Services’ Overview
+===================
+
+DCAE Services run on the DCAE platform. A service performs either
+monitors the virtualized network services or does analytics. A service
+is composed of one or more components, and performs a business need.
+
+Services are created in a ‘Service Design and Creation’ tool called
+‘Service Assurance Flow Designer’ by a Service Designer. This is done by
+\* 1. ‘compose’ing a service from the available SDC catalog components
+(actually from the TOSCA models representing the components), then \* 2.
+’submit’ing the service, which generates a Cloudify blueprint, is then
+automatically uploaded to INVENTORY, and then deployed by DEPLOYMENT
+HANDLER (once a DTI Event is triggered for the service). It should be
+noted that some service flows, specifally ’closed-loop’ flows, are not
+initiated by DTI, but are done by CLAMP.
diff --git a/docs/sections/components/component-development.rst b/docs/sections/components/component-development.rst new file mode 100644 index 00000000..2877822a --- /dev/null +++ b/docs/sections/components/component-development.rst @@ -0,0 +1,16 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +DCAE Component Development +========================== + +.. toctree:: + :maxdepth: 1 + + ./intro.rst + ./requirements-guidelines.rst + ./component-specification/component-specification.rst + ./data-formats.rst + ./dcae-cli/dcae-cli-tool.rst + ./glossary.rst + diff --git a/docs/sections/components/component-json-schema.rst b/docs/sections/components/component-json-schema.rst new file mode 100644 index 00000000..9cd5e4f9 --- /dev/null +++ b/docs/sections/components/component-json-schema.rst @@ -0,0 +1,745 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +.. _dcae-component-schema: + +DCAE Component JSON Schema +========================== + +:: + + { + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "Component specification schema", + "type": "object", + "properties": { + "self": { + "type": "object", + "properties": { + "version": { + "$ref": "#/definitions/version" + }, + "description": { + "type": "string" + }, + "component_type": { + "type": "string", + "enum": [ + "docker", + "cdap" + ] + }, + "name": { + "$ref": "#/definitions/name" + } + }, + "required": [ + "version", + "name", + "description", + "component_type" + ] + }, + "streams": { + "type": "object", + "properties": { + "publishes": { + "type": "array", + "uniqueItems": true, + "items": { + "oneOf": [ + { "$ref": "#/definitions/publisher_http" }, + { "$ref": "#/definitions/publisher_message_router" }, + { "$ref": "#/definitions/publisher_data_router" } + ] + } + }, + "subscribes": { + "type": "array", + "uniqueItems": true, + "items": { + "oneOf": [ + { "$ref": "#/definitions/subscriber_http" }, + { "$ref": "#/definitions/subscriber_message_router" }, + { "$ref": "#/definitions/subscriber_data_router" } + ] + } + } + }, + "required": [ + "publishes", + "subscribes" + ] + }, + "services": { + "type": "object", + "properties": { + "calls": { + "type": "array", + "uniqueItems": true, + "items": { + "$ref": "#/definitions/caller" + } + }, + "provides": { + "type": "array", + "uniqueItems": true, + "items": { + "$ref": "#/definitions/provider" + } + } + }, + "required": [ + "calls", + "provides" + ] + }, + "parameters" : { + "anyOf" : [ + {"$ref": "#/definitions/docker-parameters"}, + {"$ref": "#/definitions/cdap-parameters"} + ] + }, + "auxilary": { + "oneOf" : [ + {"$ref": "#/definitions/auxilary_cdap"}, + {"$ref": "#/definitions/auxilary_docker"} + ] + }, + "artifacts": { + "type": "array", + "description": "List of component artifacts", + "items": { + "$ref": "#/definitions/artifact" + } + } + }, + "required": [ + "self", + "streams", + "services", + "parameters", + "auxilary", + "artifacts" + ], + "additionalProperties": false, + "definitions": { + "cdap-parameters": { + "description" : "There are three seperate ways to pass parameters to CDAP: app config, app preferences, program preferences. These are all treated as optional.", + "type": "object", + "properties" : { + "program_preferences": { + "description" : "A list of {program_id, program_type, program_preference} objects where program_preference is an object passed into program_id of type program_type", + "type": "array", + "uniqueItems": true, + "items": { + "$ref": "#/definitions/program_preference" + } + }, + "app_preferences" : { + "description" : "Parameters Passed down to the CDAP preference API", + "type": "array", + "uniqueItems": true, + "items": { + "$ref": "#/definitions/parameter" + } + }, + "app_config" : { + "description" : "Parameters Passed down to the CDAP App Config", + "type": "array", + "uniqueItems": true, + "items": { + "$ref": "#/definitions/parameter" + } + } + } + }, + "program_preference": { + "type": "object", + "properties": { + "program_type": { + "$ref": "#/definitions/program_type" + }, + "program_id": { + "type": "string" + }, + "program_pref":{ + "description" : "Parameters that the CDAP developer wants pushed to this program's preferences API. Optional", + "type": "array", + "uniqueItems": true, + "items": { + "$ref": "#/definitions/parameter" + } + } + }, + "required": ["program_type", "program_id", "program_pref"] + }, + "program_type": { + "type": "string", + "enum": ["flows","mapreduce","schedules","spark","workflows","workers","services"] + }, + "docker-parameters": { + "type": "array", + "uniqueItems": true, + "items": { + "$ref": "#/definitions/parameter" + } + }, + "parameter": { + "type": "object", + "properties": { + "name": { + "type": "string" + }, + "value": { + "description": "Default value for the parameter" + }, + "description": { + "description": "Description for the parameter.", + "type": "string" + }, + "type": { + "description": "The required data type for the parameter.", + "type": "string", + "enum": [ "string", "number", "boolean", "datetime" ] + }, + "required": { + "description": "An optional key that declares a parameter as required (true) or not (false). Default is true.", + "type": "boolean", + "default": true + }, + "constraints": { + "description": "The optional list of sequenced constraint clauses for the parameter.", + "type": "array", + "items": { + "$ref": "#/definitions/parameter-constraints" + } + }, + "entry_schema": { + "description": "used for complex data type in the future. 'type' must be map or array for entry_schema to kick_in. ", + "type": "string" + }, + "designer_editable": { + "description": "An optional key that declares a parameter to be editable by designer (true) or not (false). Default is true.", + "type": "boolean", + "default": true + }, + "policy_editable": { + "description": "An optional key that declares a parameter to be editable by policy (true) or not (false). Default is true.", + "type": "boolean", + "default": false + }, + "sourced_at_deployment": { + "description": "An optional key that declares a parameter's value to be assigned at deployment time (true). Default is false.", + "type": "boolean", + "default": false + }, + "policy_schema" :{ + "type": "array", + "uniqueItems": true, + "items": {"$ref": "#/definitions/policy_schema_parameter"} + } + }, + "required": [ + "name", + "value", + "description" + ], + "additionalProperties": false, + "dependencies": { "policy_schema": ["policy_editable"]} + }, + "policy_schema_parameter": { + "type": "object", + "properties": { + "name": { + "type": "string" + }, + "value": { + "description": "Default value for the parameter" + }, + "description": { + "description": "Description for the parameter.", + "type": "string" + }, + "type": { + "description": "The required data type for the parameter.", + "type": "string", + "enum": [ "string", "number", "boolean", "datetime", "list", "map" ] + }, + "required": { + "description": "An optional key that declares a parameter as required (true) or not (false). Default is true.", + "type": "boolean", + "default": true + }, + "constraints": { + "description": "The optional list of sequenced constraint clauses for the parameter.", + "type": "array", + "items": { + "$ref": "#/definitions/parameter-constraints" + } + }, + "entry_schema": { + "description": "The optional key that is used to declare the name of the Datatype definition for entries of certain types. entry_schema must be defined when the type is either list or map. If the type is list and the entry type is a simple type (string, number, boolean, datetime), follow with a simple string to describe the entry type. If the type is list and the entry type is a map, follow with an array to describe the keys for the entry map. If the type is list and the entry type is also list, this is not currently supported here. If the type is map, then follow with an array to describe the keys for this map. ", + "type": "array", "uniqueItems": true, "items": {"$ref": "#/definitions/policy_schema_parameter"} + } + }, + "required": [ + "name", + "type" + ], + "additionalProperties": false + }, + "parameter-constraints": { + "type": "object", + "additionalProperties": false, + "properties": { + "equal": { + "description": "Constrains a property or parameter to a value equal to (‘=’) the value declared." + }, + "greater_than": { + "description": "Constrains a property or parameter to a value greater than (‘>’) the value declared.", + "type": "number" + }, + "greater_or_equal": { + "description": "Constrains a property or parameter to a value greater than or equal to (‘>=’) the value declared.", + "type": "number" + }, + "less_than": { + "description": "Constrains a property or parameter to a value less than (‘<’) the value declared.", + "type": "number" + }, + "less_or_equal": { + "description": "Constrains a property or parameter to a value less than or equal to (‘<=’) the value declared.", + "type": "number" + }, + "valid_values": { + "description": "Constrains a property or parameter to a value that is in the list of declared values.", + "type": "array" + }, + "length": { + "description": "Constrains the property or parameter to a value of a given length.", + "type": "number" + }, + "min_length": { + "description": "Constrains the property or parameter to a value to a minimum length.", + "type": "number" + }, + "max_length": { + "description": "Constrains the property or parameter to a value to a maximum length.", + "type": "number" + } + } + }, + "stream_message_router": { + "type": "object", + "properties": { + "format": { + "$ref": "#/definitions/name" + }, + "version": { + "$ref": "#/definitions/version" + }, + "config_key": { + "type": "string" + }, + "type": { + "description": "Type of stream to be used", + "type": "string", + "enum": [ + "message router", "message_router" + ] + } + }, + "required": [ + "format", + "version", + "config_key", + "type" + ] + }, + "publisher_http": { + "type": "object", + "properties": { + "format": { + "$ref": "#/definitions/name" + }, + "version": { + "$ref": "#/definitions/version" + }, + "config_key": { + "type": "string" + }, + "type": { + "description": "Type of stream to be used", + "type": "string", + "enum": [ + "http", + "https" + ] + } + }, + "required": [ + "format", + "version", + "config_key", + "type" + ] + }, + "publisher_message_router": { + "$ref": "#/definitions/stream_message_router" + }, + "publisher_data_router": { + "type": "object", + "properties": { + "format": { + "$ref": "#/definitions/name" + }, + "version": { + "$ref": "#/definitions/version" + }, + "config_key": { + "type": "string" + }, + "type": { + "description": "Type of stream to be used", + "type": "string", + "enum": [ + "data router", "data_router" + ] + } + }, + "required": [ + "format", + "version", + "config_key", + "type" + ] + }, + "subscriber_http": { + "type": "object", + "properties": { + "format": { + "$ref": "#/definitions/name" + }, + "version": { + "$ref": "#/definitions/version" + }, + "route": { + "type": "string" + }, + "type": { + "description": "Type of stream to be used", + "type": "string", + "enum": [ + "http", + "https" + ] + } + }, + "required": [ + "format", + "version", + "route", + "type" + ] + }, + "subscriber_message_router": { + "$ref": "#/definitions/stream_message_router" + }, + "subscriber_data_router": { + "type": "object", + "properties": { + "format": { + "$ref": "#/definitions/name" + }, + "version": { + "$ref": "#/definitions/version" + }, + "route": { + "type": "string" + }, + "type": { + "description": "Type of stream to be used", + "type": "string", + "enum": [ + "data router", "data_router" + ] + }, + "config_key": { + "description": "Data router subscribers require config info to setup their endpoints to handle requests. For example, needs username and password", + "type": "string" + } + }, + "required": [ + "format", + "version", + "route", + "type", + "config_key" + ] + }, + "provider" : { + "oneOf" : [ + {"$ref": "#/definitions/docker-provider"}, + {"$ref": "#/definitions/cdap-provider"} + ] + }, + "cdap-provider" : { + "type": "object", + "properties" : { + "request": { + "$ref": "#/definitions/formatPair" + }, + "response": { + "$ref": "#/definitions/formatPair" + }, + "service_name" : { + "type" : "string" + }, + "service_endpoint" : { + "type" : "string" + }, + "verb" : { + "type": "string", + "enum": ["GET", "PUT", "POST", "DELETE"] + } + }, + "required" : [ + "request", + "response", + "service_name", + "service_endpoint", + "verb" + ] + }, + "docker-provider": { + "type": "object", + "properties": { + "request": { + "$ref": "#/definitions/formatPair" + }, + "response": { + "$ref": "#/definitions/formatPair" + }, + "route": { + "type": "string" + }, + "verb": { + "type": "string", + "enum": ["GET", "PUT", "POST", "DELETE"] + } + }, + "required": [ + "request", + "response", + "route" + ] + }, + "caller": { + "type": "object", + "properties": { + "request": { + "$ref": "#/definitions/formatPair" + }, + "response": { + "$ref": "#/definitions/formatPair" + }, + "config_key": { + "type": "string" + } + }, + "required": [ + "request", + "response", + "config_key" + ] + }, + "formatPair": { + "type": "object", + "properties": { + "format": { + "$ref": "#/definitions/name" + }, + "version": { + "$ref": "#/definitions/version" + } + } + }, + "name": { + "type": "string" + }, + "version": { + "type": "string", + "pattern": "^(\\d+\\.)(\\d+\\.)(\\*|\\d+)$" + }, + "artifact": { + "type": "object", + "description": "Component artifact object", + "properties": { + "uri": { + "type": "string", + "description": "Uri to artifact" + }, + "type": { + "type": "string", + "enum": ["jar", "docker image"] + } + }, + "required": ["uri", "type"] + }, + + "auxilary_cdap": { + "title": "cdap component specification schema", + "type": "object", + "properties": { + "streamname": { + "type": "string" + }, + "artifact_name" : { + "type": "string" + }, + "artifact_version" : { + "type": "string", + "pattern": "^(\\d+\\.)(\\d+\\.)(\\*|\\d+)$" + }, + "namespace":{ + "type": "string", + "description" : "optional" + }, + "programs": { + "type": "array", + "uniqueItems": true, + "items": { + "$ref": "#/definitions/cdap_program" + } + } + }, + "required": [ + "streamname", + "programs", + "artifact_name", + "artifact_version" + ] + }, + "cdap_program_type": { + "type": "string", + "enum": ["flows","mapreduce","schedules","spark","workflows","workers","services"] + }, + "cdap_program": { + "type": "object", + "properties": { + "program_type": { + "$ref": "#/definitions/cdap_program_type" + }, + "program_id": { + "type": "string" + } + }, + "required": ["program_type", "program_id"] + }, + + "auxilary_docker": { + "title": "Docker component specification schema", + "type": "object", + "properties": { + "healthcheck": { + "description": "Define the health check that Consul should perfom for this component", + "type": "object", + "oneOf": [ + { "$ref": "#/definitions/docker_healthcheck_http" }, + { "$ref": "#/definitions/docker_healthcheck_script" } + ] + }, + "ports": { + "description": "Port mapping to be used for Docker containers. Each entry is of the format <container port>:<host port>.", + "type": "array", + "items": { + "type": "string" + } + }, + "volumes": { + "description": "Volume mapping to be used for Docker containers. Each entry is of the format below", + "type": "array", + "items": { + "type": "object", + "properties": { + "host":{ + "type":"object", + "path": {"type": "string"} + }, + "container":{ + "type":"object", + "bind": { "type": "string"}, + "mode": { "type": "string"} + } + } + } + } + }, + "required": [ + "healthcheck" + ], + "additionalProperties": false + }, + "docker_healthcheck_http": { + "properties": { + "type": { + "description": "Consul health check type", + "type": "string", + "enum": [ + "http", + "https" + ] + }, + "interval": { + "description": "Interval duration in seconds i.e. 10s", + "default": "15s", + "type": "string" + }, + "timeout": { + "description": "Timeout in seconds i.e. 10s", + "default": "1s", + "type": "string" + }, + "endpoint": { + "description": "Relative endpoint used by Consul to check health by making periodic HTTP GET calls", + "type": "string" + } + }, + "required": [ + "type", + "endpoint" + ] + }, + "docker_healthcheck_script": { + "properties": { + "type": { + "description": "Consul health check type", + "type": "string", + "enum": [ + "script", + "docker" + ] + }, + "interval": { + "description": "Interval duration in seconds i.e. 10s", + "default": "15s", + "type": "string" + }, + "timeout": { + "description": "Timeout in seconds i.e. 10s", + "default": "1s", + "type": "string" + }, + "script": { + "description": "Script command that will be executed by Consul to check health", + "type": "string" + } + }, + "required": [ + "type", + "script" + ] + } + } + } diff --git a/docs/sections/components/component-specification/cdap-specification.rst b/docs/sections/components/component-specification/cdap-specification.rst new file mode 100755 index 00000000..022b46d7 --- /dev/null +++ b/docs/sections/components/component-specification/cdap-specification.rst @@ -0,0 +1,209 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+.. _cdap-specification:
+
+Component specification (CDAP)
+==============================
+
+The CDAP component specification contains the following groups of
+information. Many of these are common to both CDAP and Docker components
+and are therefore described in the common specification.
+
+- :any:`Metadata <metadata>`
+- :any:`Interfaces <interfaces>` including the
+ associated :any:`Data Formats <data-formats>`
+- :any:`Parameters <parameters>` - for specifying parameters in your
+ AppConfig, AppPreferences, and ProgramPreferences to the Designer and
+ Policy. This of course is CDAP-specific and is described below.
+- :any:`Auxiliary Details <auxiliary-details>`
+- :any:`List of artifacts <artifacts>`
+
+Current Limitations and TODOs
+-----------------------------
+
+- The integration of DMD is likely to significantly change the
+ :any:`Interfaces <interfaces>` section in this specification, see
+ :any:`DMaaP abstraction <dmaap-abstraction>`.
+
+.. _parameters:
+
+Parameters
+----------
+
+There is a ``parameters`` section in your component specification. This
+section contains three optional keys: `app_config <#appconfig>`__,
+`app_preferences <#apppreferences>`__, and
+`propram_preferences <#programpreferences>`__:
+
+::
+
+ "parameters" : {
+ "app_config" : [ ...],
+ "app_preferences" : [ ...],
+ "program_preferences" : [...]
+ // any additional keys are ignored
+ }
+
+- Each section details the parameters that are a part of each of these
+ CDAP constructs (see below).
+- All such parameters will be exposed to the designer and to policy for
+ override.
+- These parameters should have default values specified by the
+ component developer where necessary, i.e., parameters that *must*
+ come from the designer/policy should not have defaults.
+- All of these keys are optional because not every CDAP application
+ uses preferences and not every application uses the AppConfig.
+ However, you should specify at least one, or else your application
+ will have no parameters exposed to policy or to the DCAE designer,
+ which means it would be non-configurable.
+- Despite the AppConfig being optional to *specify* in the case that
+ you have no parameters in your AppConfig, it is *required for
+ processing* in your application. That is because the DCAE platform
+ will place important information into your AppConfig as discussed
+ below.
+
+Parameter
+~~~~~~~~~
+
+The following CDAP specific definitions use ``param1`` to refer to the
+common parameter layout in
+:any:`Parameter <parameters>`
+
+AppConfig
+~~~~~~~~~
+
+The ``app_config`` key refers to the `CDAP AppConfig <http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/configuration.html>`_.
+It is expected to be a JSON:
+
+::
+
+ "app_config" : [ // list of JSON
+ param1, // common parameter layout
+ ...
+ ]
+
+Unfortunately, at the time of writing, the AppConfig is a Java map of
+``string:string``, which means you cannot have more complex structures
+(than string) as any value in your AppConfig. However, there is a way to
+bypass this constraint: you can pass a JSON by encoding the JSON as a
+string. E.g., the ``json.dumps()`` and it’s converse ``loads`` methods
+in Python:
+
+::
+
+ >>> import json
+ >>> json.dumps({"foo" : "bar"}) # This is a real JSON
+ '{"foo": "bar"}' # It is now a string: pass this in as your parameter value
+ >>> json.loads('{"foo": "bar"}') # Do the equivelent of this in your application
+ {u'foo': u'bar'} # ...and you'll get back a real JSON
+ >>>
+
+The final AppConfig (after the designer and policy override parameter
+values) is passed into CDAP’s AppConfig API when starting the
+application.
+
+
+AppPreferences
+~~~~~~~~~~~~~~
+
+In addition to the CDAP AppConfig, the platform supports `Application Preferences <http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/preferences.html#set-preferences>`_.
+The format of the ``app_preferences`` value is the same as the above:
+
+::
+
+ "app_preferences" : [ // list of JSON
+ param1, // common parameter layout
+ ...
+ ]
+
+The final Application Preferences JSON (after the designer and policy
+override parameter values) is passed into CDAP’s Preferences API when
+starting your application.
+
+
+ProgramPreferences
+~~~~~~~~~~~~~~~~~~
+
+Preferences can also be specified `per program <http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/lifecycle.html#program-lifecycle>`_
+in CDAP. This key’s value is a list of JSON with the following format:
+
+::
+
+ "program_preferences" : [ // note: this is a list of JSON
+ {
+ "program_id" : "program name 1", // the name of this CDAP program
+ "program_type" : "e.g., flows", // "must be one of flows, mapreduce, schedules, spark, workflows, workers, or services",
+ "program_pref" : [ // list of JSON
+ param1, // common parameter layout
+ ...
+ ]
+ },
+ // repeat for each program that receives a program_preferences JSON
+ ]
+
+Each ``program_pref`` JSON is passed into the CDAP API as the preference
+for ``program_id``.
+
+NOTE: for CDAP, this section is very likely to change when DMD is
+available. The *future* vision, as per :any:`DMaaP intentionally abstracted <dmaap-abstraction>` is
+that you would publish your data as a series of files on HDFS, and DMD
+will pick them up and send them to the appropriate DMaaP feeds or
+directly when needed.
+
+.. _auxiliary-details:
+
+Auxiliary Details
+-----------------
+
+*auxiliary* contains details about CDAP specific parameters.
+
++----------------------+----------------------+----------------------+
+| Property Name | Type | Description |
++======================+======================+======================+
+| streamname | string | *Required*. |
++----------------------+----------------------+----------------------+
+| artifact_name | string | |
++----------------------+----------------------+----------------------+
+| artifact_version | string | the version of your |
+| | | CDAP JAR artifact |
++----------------------+----------------------+----------------------+
+| namespace | string | the CDAP namespace |
+| | | to deploy into, |
+| | | default is ‘default’ |
++----------------------+----------------------+----------------------+
+| programs | array | contains each CDAP |
+| | | entity represented |
+| | | in the artifact |
++----------------------+----------------------+----------------------+
+| program_type | string | CDAP entity (eg |
+| | | “flows”) |
++----------------------+----------------------+----------------------+
+| program_id | string | name of CDAP entity |
+| | | (eg “WhoFlow”) |
++----------------------+----------------------+----------------------+
+
+Example:
+
+.. code:: json
+
+ "auxiliary": {
+ "streamname" : "who",
+ "artifact_name" : "HelloWorld",
+ "artifact_version" : "3.4.3",
+ "namespace" : "hw",
+ "programs" : [
+ {"program_type" : "flows", "program_id" : "WhoFlow"},
+ {"program_type" : "services", "program_id" : "Greeting"},
+ ...
+ ],
+ }
+
+The ``programs`` key is identical to the ``program_preferences`` key
+discussed `above <#programpreferences>`__ except:
+
+- each JSON in the list does not contain ``program_pref``
+- this is required! You must include all of your programs in this, as
+ it is used to start each program as well as for DCAE to perform
+ periodic healthchecks on your application. Don’t forget about your
+ services; they are programs too.
diff --git a/docs/sections/components/component-specification/common-specification.rst b/docs/sections/components/component-specification/common-specification.rst new file mode 100755 index 00000000..a29610f5 --- /dev/null +++ b/docs/sections/components/component-specification/common-specification.rst @@ -0,0 +1,1418 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+.. _common-specification:
+
+Common Elements of the Component Specification
+==============================================
+
+This page describes the component specification (JSON) sections that are
+common to both Docker and CDAP components. Differences for each are
+pointed out below. Elements that are very different, and described in
+the CDAP or Docker specific pages.
+
+.. _metadata:
+
+Component Metadata
+------------------
+
+Metadata refers to the properties found under the ``self`` JSON. This
+group of properties is used to uniquely identify this component
+specification and identify the component that this specification is used
+to capture.
+
+Example:
+
+::
+
+ "self": {
+ "version": "1.0.0",
+ "name": "asimov.component.kpi_anomaly",
+ "description": "Classifies VNF KPI data as anomalous",
+ "component_type": "docker"
+ },
+
+``self`` Schema:
+
++-------------+--------+----------------+
+| Property | Type | Description |
+| Name | | |
++=============+========+================+
+| version | string | *Required*. |
+| | | Semantic |
+| | | version |
+| | | for this |
+| | | specification |
++-------------+--------+----------------+
+| name | string | *Required*. |
+| | | Full |
+| | | name of |
+| | | this |
+| | | component |
+| | | which is |
+| | | also |
+| | | used as |
+| | | this |
+| | | component's |
+| | | catalog |
+| | | id. |
++-------------+--------+----------------+
+| description | string | *Required* |
+| | | Human-readable |
+| | | text |
+| | | describing |
+| | | the |
+| | | component |
+| | | and the |
+| | | components |
+| | | functional |
+| | | purpose. |
++-------------+--------+----------------+
+| component_t\| string | *Required* |
+| ype | | Identify |
+| | | what |
+| | | containe\ |
+| | | rization |
+| | | technolo\ |
+| | | gy |
+| | | this |
+| | | componen\ |
+| | | t |
+| | | uses: |
+| | | *docker* |
+| | | or |
+| | | *cdap*. |
+| | | |
++-------------+--------+----------------+
+
+.. _interfaces:
+
+Interfaces
+----------
+
+Interfaces are the JSON objects found under the ``streams`` key and the
+``services`` key. These are used to describe the interfaces that the
+component uses and the interfaces that the component provides. The
+description of each interface includes the associated :any:`data
+format <data-formats>`.
+
+Streams
+~~~~~~~
+
+- The ``streams`` JSON is for specifying data produced for consumption
+ by other components, and the streams expected to subscribe to that is
+ produced by other components. These are “fire and forget” type
+ interfaces where the publisher of a stream does not expect or parse a
+ response from the subscriber.
+- The term ``stream`` here is abstract and neither refers to “CDAP
+ streams” or “DMaaP feeds”. While a stream is very likely a DMaaP
+ feed, it could be a direct stream of data being routed via HTTP too.
+ It abstractly refers to a sequence of data leaving a publisher.
+- Streams have anonymous publish/subscribe semantics, which decouples
+ the production of information from its consumption.
+- In general, components are not aware of who they are communicating
+ with.
+- Instead, components that are interested in data, subscribe to the
+ relevant stream; components that generate data publish to the
+ relevant stream.
+- There can be multiple publishers and subscribers to a stream. Streams
+ are intended for unidirectional, streaming communication.
+
+Streams interfaces that implement an HTTP endpoint must support POST.
+
+Streams are split into:
+
++-------------+----+----------+
+| Property | Ty\| Descript\|
+| Name | pe | ion |
++=============+====+==========+
+| subscribes | JS\| *Require\|
+| | ON | d*. |
+| | li\| List of |
+| | st | all |
+| | | availabl\|
+| | | e |
+| | | stream |
+| | | interfac\|
+| | | es |
+| | | that |
+| | | this |
+| | | componen\|
+| | | t |
+| | | has that |
+| | | can be |
+| | | used for |
+| | | subscrib\|
+| | | ing |
++-------------+----+----------+
+| publishes | JS\| *Require\|
+| | ON | d*. |
+| | li\| List of |
+| | st | all |
+| | | stream |
+| | | interfac\|
+| | | es |
+| | | that |
+| | | this |
+| | | componen\|
+| | | t |
+| | | will |
+| | | publish |
+| | | onto |
++-------------+----+----------+
+
+Subscribes
+^^^^^^^^^^
+
+Example:
+
+.. code:: json
+
+ "streams": {
+ "subscribes": [{
+ "format": "dcae.vnf.kpi",
+ "version": "1.0.0",
+ "route": "/data", // for CDAP this value is not used
+ "type": "http"
+ }],
+ ...
+ }
+
+This describes that ``asimov.component.kpi_anomaly`` exposes an HTTP
+endpoint called ``/data`` which accepts requests that have the data
+format of ``dcae.vnf.kpi`` version ``1.0.0``.
+
+``subscribes`` Schema:
+
++-------------+----+--------------------+
+| Property | Ty\| Descript\ |
+| Name | pe | ion |
++=============+====+====================+
+| format | st\| *Require\ |
+| | ri\| d*. |
+| | ng | Data |
+| | | format |
+| | | id of |
+| | | the data |
+| | | format |
+| | | that is |
+| | | used by |
+| | | this |
+| | | interfac\ |
+| | | e |
++-------------+----+--------------------+
+| version | st\| *Require\ |
+| | ri\| d*. |
+| | ng | Data |
+| | | format |
+| | | version |
+| | | of the |
+| | | data |
+| | | format |
+| | | that is |
+| | | used by |
+| | | this |
+| | | interfac\ |
+| | | e |
++-------------+----+--------------------+
+| route | st\| *Require\ |
+| | ri\| d |
+| | ng | for HTTP |
+| | | and data |
+| | | router*. |
+| | | The HTTP |
+| | | route |
+| | | that |
+| | | this |
+| | | interfac\ |
+| | | e |
+| | | listens |
+| | | on |
++-------------+----+--------------------+
+| config_key | st\| *Require\ |
+| | ri\| d \ |
+| | ng | for \ |
+| | | message_router\ |
+| | | and data \ |
+| | | router*. |
+| | | The HTTP |
+| | | route |
+| | | that |
+| | | this |
+| | | interfac\ |
+| | | e |
+| | | listens |
+| | | on |
++-------------+----+--------------------+
+| type | st\| *Require\ |
+| | ri\| d*. |
+| | ng | Type of |
+| | | stream: |
+| | | ``http`` |
+| | | , |
+| | | ``message_router`` |
+| | | , |
+| | | ``data_router`` |
++-------------+----+--------------------+
+
+.. _message-router:
+
+Message router
+''''''''''''''
+
+Message router subscribers are http clients rather than http services
+and performs a http a ``GET`` call. Thus, message router subscribers
+description is structured like message router publishers and requires
+``config_key``:
+
+.. code:: json
+
+ "streams": {
+ "subscribes": [{
+ "format": "dcae.some-format",
+ "version": "1.0.0",
+ "config_key": "some_format_handle",
+ "type": "message router"
+ }],
+ ...
+ }
+
+
+.. _data-router:
+
+Data router
+'''''''''''
+
+Data router subscribers are http or https services that handle ``PUT``
+requests from data router. Developers must provide the ``route`` or url
+path/endpoint that is expected to handle data router requests. This will
+be used to construct the delivery url needed to register the subscriber
+to the provisioned feed. Developers must also provide a ``config_key``
+because there is dynamic configuration information associated with the
+feed that the application will need e.g. username and password. See the
+page on :doc:`DMaaP connection objects <../dcae-cli/dmaap-connection-objects>` for more details on
+the configuration information.
+
+Example (not tied to the larger example):
+
+.. code:: json
+
+ "streams": {
+ "subscribes": [{
+ "config_key": "some-sub-dr",
+ "format": "sandbox.platform.any",
+ "route": "/identity",
+ "type": "data_router",
+ "version": "0.1.0"
+ }],
+ ...
+ }
+
+Publishes
+^^^^^^^^^
+
+Example:
+
+.. code:: json
+
+ "streams": {
+ ...
+ "publishes": [{
+ "format": "asimov.format.integerClassification",
+ "version": "1.0.0",
+ "config_key": "prediction",
+ "type": "http"
+ }]
+ },
+
+This describes that ``asimov.component.kpi_anomaly`` publishes by making
+POST requests to streams that support the data format
+``asimov.format.integerClassification`` version ``1.0.0``.
+
+``publishes`` Schema:
+
++-------------+----+--------------------+
+| Property | Ty\| Descript\ |
+| Name | pe | ion |
++=============+====+====================+
+| format | st\| *Require\ |
+| | ri\| d*. |
+| | ng | Data |
+| | | format |
+| | | id of |
+| | | the data |
+| | | format |
+| | | that is |
+| | | used by |
+| | | this |
+| | | interfac\ |
+| | | e |
++-------------+----+--------------------+
+| version | st\| *Require\ |
+| | ri\| d*. |
+| | ng | Data |
+| | | format |
+| | | version |
+| | | of the |
+| | | data |
+| | | format |
+| | | that is |
+| | | used by |
+| | | this |
+| | | interfac\ |
+| | | e |
++-------------+----+--------------------+
+| config_key | st\| *Require\ |
+| | ri\| d*. |
+| | ng | The JSON |
+| | | key in |
+| | | the |
+| | | generate\ |
+| | | d |
+| | | applicat |
+| | | ion |
+| | | configur\ |
+| | | ation |
+| | | that |
+| | | will be |
+| | | used to |
+| | | pass the |
+| | | downstre\ |
+| | | am |
+| | | componen\ |
+| | | t’s |
+| | | (the |
+| | | subscrib\ |
+| | | er’s) |
+| | | connecti\ |
+| | | on |
+| | | informat\ |
+| | | ion. |
++-------------+----+--------------------+
+| type | st\| *Require\ |
+| | ri\| d*. |
+| | ng | Type of |
+| | | stream: |
+| | | ``http`` |
+| | | , |
+| | | ``message_router`` |
+| | | , |
+| | | ``data_router`` |
++-------------+----+--------------------+
+
+.. message-router-1:
+
+Message router
+''''''''''''''
+
+Message router publishers are http clients of DMaap message_router.
+Developers must provide a ``config_key`` because there is dynamic
+configuration information associated with the feed that the application
+will need to receive e.g. topic url, username, password. See the page on
+:doc:`DMaaP connection objects <../dcae-cli/dmaap-connection-objects>` for more details on
+the configuration information.
+
+Example (not tied to the larger example):
+
+.. code:: json
+
+ "streams": {
+ ...
+ "publishes": [{
+ "config_key": "some-pub-mr",
+ "format": "sandbox.platform.any",
+ "type": "message_router",
+ "version": "0.1.0"
+ }]
+ }
+
+.. data-router-1:
+
+Data router
+'''''''''''
+
+Data router publishers are http clients that make ``PUT`` requests to
+data router. Developers must also provide a ``config_key`` because there
+is dynamic configuration information associated with the feed that the
+application will need to receive e.g. publish url, username, password.
+See the page on :doc:`DMaaP connection objects <../dcae-cli/dmaap-connection-objects>` for more details on
+the configuration information.
+
+Example (not tied to the larger example):
+
+.. code:: json
+
+ "streams": {
+ ...
+ "publishes": [{
+ "config_key": "some-pub-dr",
+ "format": "sandbox.platform.any",
+ "type": "data_router",
+ "version": "0.1.0"
+ }]
+ }
+
+Quick Reference
+^^^^^^^^^^^^^^^
+
+Refer to this :doc:`Quick Reference <streams-grid>` for a
+comparison of the Streams ‘Publishes’ and ‘Subscribes’ sections.
+
+Services
+~~~~~~~~
+
+- The publish / subscribe model is a very flexible communication
+ paradigm, but its many-to-many one-way transport is not appropriate
+ for RPC request / reply interactions, which are often required in a
+ distributed system.
+- Request / reply is done via a Service, which is defined by a pair of
+ messages: one for the request and one for the reply.
+
+Services are split into:
+
++-------------+----+----------+
+| Property | Ty\| Descript\|
+| Name | pe | ion |
++=============+====+==========+
+| calls | JS\| *Require\|
+| | ON | d*. |
+| | li\| List of |
+| | st | all |
+| | | service |
+| | | interfac\|
+| | | es |
+| | | that |
+| | | this |
+| | | componen\|
+| | | t |
+| | | will |
+| | | call |
++-------------+----+----------+
+| provides | JS\| *Require\|
+| | ON | d*. |
+| | li\| List of |
+| | st | all |
+| | | service |
+| | | interfac\|
+| | | es |
+| | | that |
+| | | this |
+| | | componen\|
+| | | t |
+| | | exposes |
+| | | and |
+| | | provides |
++-------------+----+----------+
+
+Calls
+^^^^^
+
+The JSON ``services/calls`` is for specifying that the component relies
+on an HTTP(S) service—the component sends that service an HTTP request,
+and that service responds with an HTTP reply. An example of this is how
+string matching (SM) depends on the AAI Broker. SM performs a
+synchronous REST call to the AAI broker, providing it the VMNAME of the
+VNF, and the AAI Broker responds with additional details about the VNF.
+This dependency is expressed via ``services/calls``. In contrast, the
+output of string matching (the alerts it computes) is sent directly to
+policy as a fire-and-forget interface, so that is an example of a
+``stream``.
+
+Example:
+
+.. code:: json
+
+ "services": {
+ "calls": [{
+ "config_key": "vnf-db",
+ "request": {
+ "format": "dcae.vnf.meta",
+ "version": "1.0.0"
+ },
+ "response": {
+ "format": "dcae.vnf.kpi",
+ "version": "1.0.0"
+ }
+ }],
+ ...
+ }
+
+This describes that ``asimov.component.kpi_anomaly`` will make HTTP
+calls to a downstream component that accepts requests of data format
+``dcae.vnf.meta`` version ``1.0.0`` and is expecting the response to be
+``dcae.vnf.kpi`` version ``1.0.0``.
+
+``calls`` Schema:
+
++-------------+----+----------+
+| Property | Ty\| Descript\|
+| Name | pe | ion |
++=============+====+==========+
+| request | JS\| *Require\|
+| | ON | d*. |
+| | ob\| Descript\|
+| | je\| ion |
+| | ct | of the |
+| | | expected |
+| | | request |
+| | | for this |
+| | | downstre\|
+| | | am |
+| | | interfac\|
+| | | e |
++-------------+----+----------+
+| response | JS\| *Require\|
+| | ON | d*. |
+| | ob\| Descript\|
+| | je\| ion |
+| | ct | of the |
+| | | expected |
+| | | response |
+| | | for this |
+| | | downstre\|
+| | | am |
+| | | interfac\|
+| | | e |
++-------------+----+----------+
+| config_key | st\| *Require\|
+| | ri\| d*. |
+| | ng | The JSON |
+| | | key in |
+| | | the |
+| | | generate\|
+| | | d |
+| | | applicat |
+| | | ion |
+| | | configur\|
+| | | ation |
+| | | that |
+| | | will be |
+| | | used to |
+| | | pass the |
+| | | downstre\|
+| | | am |
+| | | componen |
+| | | t |
+| | | connecti\|
+| | | on |
+| | | informat\|
+| | | ion. |
++-------------+----+----------+
+
+The JSON object schema for both ``request`` and ``response``:
+
++-------------+----+----------+
+| Property | Ty\| Descript\|
+| Name | pe | ion |
++=============+====+==========+
+| format | st\| *Require\|
+| | ri\| d*. |
+| | ng | Data |
+| | | format |
+| | | id of |
+| | | the data |
+| | | format |
+| | | that is |
+| | | used by |
+| | | this |
+| | | interfac\|
+| | | e |
++-------------+----+----------+
+| version | st\| *Require\|
+| | ri\| d*. |
+| | ng | Data |
+| | | format |
+| | | version |
+| | | of the |
+| | | data |
+| | | format |
+| | | that is |
+| | | used by |
+| | | this |
+| | | interfac\|
+| | | e |
++-------------+----+----------+
+
+Provides
+^^^^^^^^
+
+Example:
+
+.. code:: json
+
+ "services": {
+ ...
+ "provides": [{
+ "route": "/score-vnf",
+ "request": {
+ "format": "dcae.vnf.meta",
+ "version": "1.0.0"
+ },
+ "response": {
+ "format": "asimov.format.integerClassification",
+ "version": "1.0.0"
+ }
+ }]
+ },
+
+This describes that ``asimov.component.kpi_anomaly`` provides a service
+interface and it is exposed on the ``/score-vnf`` HTTP endpoint. The
+endpoint accepts requests that have the data format ``dcae.vnf.meta``
+version ``1.0.0`` and gives back a response of
+``asimov.format.integerClassification`` version ``1.0.0``.
+
+``provides`` Schema for a Docker component:
+
++-------------+----+----------+
+| Property | Ty\| Descript\|
+| Name | pe | ion |
++=============+====+==========+
+| request | JS\| *Require\|
+| | ON | d*. |
+| | ob\| Descript\|
+| | je\| ion |
+| | ct | of the |
+| | | expected |
+| | | request |
+| | | for this |
+| | | interfac\|
+| | | e |
++-------------+----+----------+
+| response | JS\| *Require\|
+| | ON | d*. |
+| | ob\| Descript\|
+| | je\| ion |
+| | ct | of the |
+| | | expected |
+| | | response |
+| | | for this |
+| | | interfac\|
+| | | e |
++-------------+----+----------+
+| route | st\| *Require\|
+| | ri\| d*. |
+| | ng | The HTTP |
+| | | route |
+| | | that |
+| | | this |
+| | | interfac\|
+| | | e |
+| | | listens |
+| | | on |
++-------------+----+----------+
+
+The JSON object schema for both ``request`` and ``response``:
+
++-------------+----+----------+
+| Property | Ty\| Descript\|
+| Name | pe | ion |
++=============+====+==========+
+| format | st\| *Require\|
+| | ri\| d*. |
+| | ng | Data |
+| | | format |
+| | | id of |
+| | | the data |
+| | | format |
+| | | that is |
+| | | used by |
+| | | this |
+| | | interfac\|
+| | | e |
++-------------+----+----------+
+| version | st\| *Require\|
+| | ri\| d*. |
+| | ng | Data |
+| | | format |
+| | | version |
+| | | of the |
+| | | data |
+| | | format |
+| | | that is |
+| | | used by |
+| | | this |
+| | | interfac\|
+| | | e |
++-------------+----+----------+
+
+Note, for CDAP, there is a slight variation due to the way CDAP exposes
+services:
+
+::
+
+ "provides":[ // note this is a list of JSON
+ {
+ "request":{ ...},
+ "response":{ ...},
+ "service_name":"name CDAP service",
+ "service_endpoint":"greet", // E.g the URL is /services/service_name/methods/service_endpoint
+ "verb":"GET" // GET, PUT, or POST
+ }
+ ]
+
+``provides`` Schema for a CDAP component:
+
++-------------+----+-----------+
+| Property | Ty\| Descript\ |
+| Name | pe | ion |
++=============+====+===========+
+| request | JS\| *Require\ |
+| | ON | d*. |
+| | ob\| Descript\ |
+| | je\| ion |
+| | ct | of the |
+| | | expected |
+| | | request |
+| | | data |
+| | | format |
+| | | for this |
+| | | interfac\ |
+| | | e |
++-------------+----+-----------+
+| response | JS\| *Require\ |
+| | ON | d*. |
+| | ob\| Descript\ |
+| | je\| ion |
+| | ct | of the |
+| | | expected |
+| | | response |
+| | | for this |
+| | | interfac\ |
+| | | e |
++-------------+----+-----------+
+| service_nam\| st\| *Require\ |
+| e | ri\| d*. |
+| | ng | The CDAP |
+| | | service |
+| | | name (eg |
+| | | “Greetin\ |
+| | | g”) |
++-------------+----+-----------+
+| service_end | st\| *Require\ |
+| point | ri\| d*. |
+| | ng | The CDAP |
+| | | service |
+| | | endpoint |
+| | | for this |
+| | | service_n\|
+| | | ame |
+| | | (eg |
+| | | “/greet” |
+| | | ) |
++-------------+----+-----------+
+| verb | st\| *Require\ |
+| | ri\| d*. |
+| | ng | ‘GET’, |
+| | | ‘PUT’ or |
+| | | ‘POST’ |
++-------------+----+-----------+
+
+Parameters
+----------
+
+``parameters`` is where to specify the component’s application
+configuration parameters that are not connection information.
+
++---------------+------------+----------------------------------+
+| Property Name | Type | Description |
++===============+============+==================================+
+| parameters | JSON array | Each entry is a parameter object |
++---------------+------------+----------------------------------+
+
+Parameter object has the following available properties:
+
++--------------+----+----------+------+
+| Property | Ty\| Descript\| Defa\|
+| Name | pe | ion | ult |
++==============+====+==========+======+
+| name | st\| *Require\| |
+| | ri\| d*. | |
+| | ng | The | |
+| | | property | |
+| | | name | |
+| | | that | |
+| | | will be | |
+| | | used as | |
+| | | the key | |
+| | | in the | |
+| | | generate\| |
+| | | d | |
+| | | config | |
++--------------+----+----------+------+
+| value | an\| *Require\| |
+| | y | d*. | |
+| | | The | |
+| | | default | |
+| | | value | |
+| | | for the | |
+| | | given | |
+| | | paramete\| |
+| | | r | |
++--------------+----+----------+------+
+| description | st\| *Require\| |
+| | ri\| d*. | |
+| | ng | Human-re\| |
+| | | adable | |
+| | | text | |
+| | | describi\| |
+| | | ng | |
+| | | the | |
+| | | paramete\| |
+| | | r | |
+| | | like | |
+| | | what its | |
+| | | for | |
++--------------+----+----------+------+
+| type | st\| The | |
+| | ri\| required | |
+| | ng | data | |
+| | | type for | |
+| | | the | |
+| | | paramete\| |
+| | | r | |
++--------------+----+----------+------+
+| required | bo\| An | true |
+| | ol\| optional | |
+| | ea\| key that | |
+| | n | declares | |
+| | | a | |
+| | | paramete\| |
+| | | r | |
+| | | as | |
+| | | required | |
+| | | (true) | |
+| | | or not | |
+| | | (false) | |
++--------------+----+----------+------+
+| constraints | ar\| The | |
+| | ra\| optional | |
+| | y | list of | |
+| | | sequence | |
+| | | d | |
+| | | constrai\| |
+| | | nt | |
+| | | clauses | |
+| | | for the | |
+| | | paramete\| |
+| | | r. | |
+| | | See | |
+| | | below | |
++--------------+----+----------+------+
+| entry_schem\ | st\| The | |
+| a | ri\| optional | |
+| | ng | key that | |
+| | | is used | |
+| | | to | |
+| | | declare | |
+| | | the name | |
+| | | of the | |
+| | | Datatype | |
+| | | definiti\| |
+| | | on | |
+| | | for | |
+| | | entries | |
+| | | of set | |
+| | | types | |
+| | | such as | |
+| | | the | |
+| | | TOSCA | |
+| | | ‘list’ | |
+| | | or | |
+| | | ‘map’. | |
+| | | Only 1 | |
+| | | level is | |
+| | | supporte\| |
+| | | d | |
+| | | at this | |
+| | | time | |
++--------------+----+----------+------+
+| designer_ed\ | bo\| An | true |
+| itable | ol\| optional | |
+| | ea\| key that | |
+| | n | declares | |
+| | | a | |
+| | | paramete\| |
+| | | r | |
+| | | to be | |
+| | | editable | |
+| | | by | |
+| | | designer | |
+| | | (true) | |
+| | | or not | |
+| | | (false) | |
++--------------+----+----------+------+
+| sourced_at_d\| bo\| An | fals\|
+| eployment | ol\| optional | e |
+| | ea\| key that | |
+| | n | declares | |
+| | | a | |
+| | | paramete\| |
+| | | r’s | |
+| | | value to | |
+| | | be | |
+| | | assigned | |
+| | | at | |
+| | | deployme\| |
+| | | nt | |
+| | | time | |
+| | | (true) | |
++--------------+----+----------+------+
+| policy_edit\ | bo\| An | true |
+| able | ol\| optional | |
+| | ea\| key that | |
+| | n | declares | |
+| | | a | |
+| | | paramete\| |
+| | | r | |
+| | | to be | |
+| | | editable | |
+| | | by | |
+| | | policy | |
+| | | (true) | |
+| | | or not | |
+| | | (false) | |
++--------------+----+----------+------+
+| policy_sche\ | ar\| The | |
+| ma | ra\| optional | |
+| | y | list of | |
+| | | schema | |
+| | | definiti\| |
+| | | ons | |
+| | | used for | |
+| | | policy. | |
+| | | See | |
+| | | below | |
++--------------+----+----------+------+
+
+Example:
+
+.. code:: json
+
+ "parameters": [
+ {
+ "name": "threshold",
+ "value": 0.75,
+ "description": "Probability threshold to exceed to be anomalous"
+ }
+ ]
+
+Many of the parameter properties have been copied from TOSCA model
+property definitions and are to be used for service design composition
+and policy creation. See `section 3.5.8 *Property
+definition* <http://docs.oasis-open.org/tosca/TOSCA-Simple-Profile-YAML/v1.1/TOSCA-Simple-Profile-YAML-v1.1.html>`__.
+
+The property ``constraints`` is a list of objects where each constraint
+object:
+
++--------------+----+----------+
+| Property | Ty\| Descript\|
+| Name | pe | ion |
++==============+====+==========+
+| equal | | Constrai\|
+| | | ns |
+| | | a |
+| | | property |
+| | | or |
+| | | paramete\|
+| | | r |
+| | | to a |
+| | | value |
+| | | equal to |
+| | | (‘=’) |
+| | | the |
+| | | value |
+| | | declared |
++--------------+----+----------+
+| greater_tha\ | nu\| Constrai\|
+| n | mb\| ns |
+| | er | a |
+| | | property |
+| | | or |
+| | | paramete |
+| | | r |
+| | | to a |
+| | | value |
+| | | greater |
+| | | than |
+| | | (‘>’) |
+| | | the |
+| | | value |
+| | | declared |
++--------------+----+----------+
+| greater_or_e\| nu\| Constrai\|
+| qual | mb\| ns |
+| | er | a |
+| | | property |
+| | | or |
+| | | paramete\|
+| | | r |
+| | | to a |
+| | | value |
+| | | greater |
+| | | than or |
+| | | equal to |
+| | | (‘>=’) |
+| | | the |
+| | | value |
+| | | declared |
++--------------+----+----------+
+| less_than | nu\| Constrai\|
+| | mb\| ns |
+| | er | a |
+| | | property |
+| | | or |
+| | | paramete\|
+| | | r |
+| | | to a |
+| | | value |
+| | | less |
+| | | than |
+| | | (‘<’) |
+| | | the |
+| | | value |
+| | | declared |
++--------------+----+----------+
+| less_or_equ\ | nu\| Constrai\|
+| al | mb\| ns |
+| | er | a |
+| | | property |
+| | | or |
+| | | paramete\|
+| | | r |
+| | | to a |
+| | | value |
+| | | less |
+| | | than or |
+| | | equal to |
+| | | (‘<=’) |
+| | | the |
+| | | value |
+| | | declared |
++--------------+----+----------+
+| valid_value\ | ar\| Constrai\|
+| s | ra\| ns |
+| | y | a |
+| | | property |
+| | | or |
+| | | paramete\|
+| | | r |
+| | | to a |
+| | | value |
+| | | that is |
+| | | in the |
+| | | list of |
+| | | declared |
+| | | values |
++--------------+----+----------+
+| length | nu\| Constrai\|
+| | mb\| ns |
+| | er | the |
+| | | property |
+| | | or |
+| | | paramete\|
+| | | r |
+| | | to a |
+| | | value of |
+| | | a given |
+| | | length |
++--------------+----+----------+
+| min_length | nu\| Constrai\|
+| | mb\| ns |
+| | er | the |
+| | | property |
+| | | or |
+| | | paramete\|
+| | | r |
+| | | to a |
+| | | value to |
+| | | a |
+| | | minimum |
+| | | length |
++--------------+----+----------+
+| max_length | nu\| Constrai\|
+| | mb\| ns |
+| | er | the |
+| | | property |
+| | | or |
+| | | paramete\|
+| | | r |
+| | | to a |
+| | | value to |
+| | | a |
+| | | maximum |
+| | | length |
++--------------+----+----------+
+
+``threshold`` is the configuration parameter and will get set to 0.75
+when the configuration gets generated.
+
+The property ``policy_schema`` is a list of objects where each
+policy_schema object:
+
++-------------+----+----------+------+
+| Property | Ty\| Descript\| Defa\|
+| Name | pe | ion | ult |
++=============+====+==========+======+
+| name | st\| *Require\| |
+| | ri\| d*. | |
+| | ng | paramete\| |
+| | | r | |
+| | | name | |
++-------------+----+----------+------+
+| value | st\| default | |
+| | ri\| value | |
+| | ng | for the | |
+| | | paramete\| |
+| | | r | |
++-------------+----+----------+------+
+| description | st\| paramete\| |
+| | ri\| r | |
+| | ng | descript\| |
+| | | ion | |
++-------------+----+----------+------+
+| type | en\| *Require\| |
+| | um | d*. | |
+| | | data | |
+| | | type of | |
+| | | the | |
+| | | paramete\| |
+| | | r, | |
+| | | ‘string’ | |
+| | | , | |
+| | | ‘number’ | |
+| | | , | |
+| | | ‘boolean | |
+| | | ’, | |
+| | | ‘datetim\| |
+| | | e’, | |
+| | | ‘list’, | |
+| | | or ‘map’ | |
++-------------+----+----------+------+
+| required | bo\| is | true |
+| | ol\| paramete\| |
+| | ea\| r | |
+| | n | required | |
+| | | or not? | |
++-------------+----+----------+------+
+| constraints | ar\| The | |
+| | ra\| optional | |
+| | y | list of | |
+| | | sequence\| |
+| | | d | |
+| | | constrai\| |
+| | | nt | |
+| | | clauses | |
+| | | for the | |
+| | | paramete\| |
+| | | r. | |
+| | | See | |
+| | | above | |
++-------------+----+----------+------+
+| entry_schem\| st\| The | |
+| a | ri\| optional | |
+| | ng | key that | |
+| | | is used | |
+| | | to | |
+| | | declare | |
+| | | the name | |
+| | | of the | |
+| | | Datatype | |
+| | | definiti\| |
+| | | on | |
+| | | for | |
+| | | certain | |
+| | | types. | |
+| | | entry_sc\| |
+| | | hema | |
+| | | must be | |
+| | | defined | |
+| | | when the | |
+| | | type is | |
+| | | either | |
+| | | list or | |
+| | | map. If | |
+| | | the type | |
+| | | is list | |
+| | | and the | |
+| | | entry | |
+| | | type is | |
+| | | a simple | |
+| | | type | |
+| | | (string, | |
+| | | number, | |
+| | | bookean, | |
+| | | datetime | |
+| | | ), | |
+| | | follow | |
+| | | with an | |
+| | | string | |
+| | | to | |
+| | | describe | |
+| | | the | |
+| | | entry | |
++-------------+----+----------+------+
+| | If | | |
+| | th\| | |
+| | e | | |
+| | ty\| | |
+| | pe | | |
+| | is | | |
+| | li\| | |
+| | st | | |
+| | an\| | |
+| | d | | |
+| | th\| | |
+| | e | | |
+| | en\| | |
+| | tr\| | |
+| | y | | |
+| | ty\| | |
+| | pe | | |
+| | is | | |
+| | a | | |
+| | ma\| | |
+| | p, | | |
+| | fo\| | |
+| | ll\| | |
+| | ow | | |
+| | wi\| | |
+| | th | | |
+| | an | | |
+| | ar\| | |
+| | ra\| | |
+| | y | | |
+| | to | | |
+| | de\| | |
+| | sc\| | |
+| | ri\| | |
+| | be | | |
+| | th\| | |
+| | e | | |
+| | ke\| | |
+| | ys | | |
+| | fo\| | |
+| | r | | |
+| | th\| | |
+| | e | | |
+| | en\| | |
+| | tr\| | |
+| | y | | |
+| | ma\| | |
+| | p | | |
++-------------+----+----------+------+
+| | If | | |
+| | th\| | |
+| | e | | |
+| | ty\| | |
+| | pe | | |
+| | is | | |
+| | li\| | |
+| | st | | |
+| | an\| | |
+| | d | | |
+| | th\| | |
+| | e | | |
+| | en\| | |
+| | tr\| | |
+| | y | | |
+| | ty\| | |
+| | pe | | |
+| | is | | |
+| | a | | |
+| | li\| | |
+| | st | | |
+| | , | | |
+| | th\| | |
+| | at | | |
+| | is | | |
+| | no\| | |
+| | t | | |
+| | cu\| | |
+| | rr\| | |
+| | en\| | |
+| | tl\| | |
+| | y | | |
+| | su\| | |
+| | pp\| | |
+| | or\| | |
+| | te\| | |
+| | d \| | |
++-------------+----+----------+------+
+| | If | | |
+| | th\| | |
+| | e | | |
+| | ty\| | |
+| | pe | | |
+| | is | | |
+| | ma\| | |
+| | p, | | |
+| | fo\| | |
+| | ll\| | |
+| | ow | | |
+| | wi\| | |
+| | th | | |
+| | an | | |
+| | ar\| | |
+| | ay | | |
+| | to | | |
+| | de\| | |
+| | sc\| | |
+| | ri\| | |
+| | be | | |
+| | th\| | |
+| | e | | |
+| | ke\| | |
+| | ys | | |
+| | fo\| | |
+| | r | | |
+| | th\| | |
+| | e | | |
+| | ma\| | |
+| | p | | |
++-------------+----+----------+------+
+
+Generated Application Configuration
+-----------------------------------
+
+The above example for component ``asimov.component.kpi_anomaly`` will
+get transformed into the following application configuration JSON that
+is fully resolved and provided at runtime by calling the
+``config binding service``:
+
+.. code:: json
+
+ {
+ "streams_publishes": {
+ "prediction": ["10.100.1.100:32567"]
+ },
+ "streams_subscribes": {},
+ "threshold": 0.75,
+ "services_calls": {
+ "vnf-db": ["10.100.1.101:32890"]
+ }
+ }
+
+.. _artifacts:
+
+Artifacts
+---------
+
+``artifacts`` contains a list of artifacts associated with this
+component. For Docker, this is the full path (including the registry) to
+the Docker image. For CDAP, this is the full path to the CDAP jar.
+
++---------------+------------+---------------------------------+
+| Property Name | Type | Description |
++===============+============+=================================+
+| artifacts | JSON array | Each entry is a artifact object |
++---------------+------------+---------------------------------+
+
+``artifact`` Schema:
+
++---------------+--------+--------------------------------------------+
+| Property Name | Type | Description |
++===============+========+============================================+
+| uri | string | *Required*. Uri to the artifact, full path |
++---------------+--------+--------------------------------------------+
+| type | string | *Required*. ``docker image`` or ``jar`` |
++---------------+--------+--------------------------------------------+
diff --git a/docs/sections/components/component-specification/component-specification.rst b/docs/sections/components/component-specification/component-specification.rst new file mode 100644 index 00000000..759f3dc9 --- /dev/null +++ b/docs/sections/components/component-specification/component-specification.rst @@ -0,0 +1,16 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +Component Specification +======================= + +.. toctree:: + :maxdepth: 1 + + ./common-specification.rst + ./cdap-specification.rst + ./docker-specification.rst + ./generated-configuration.rst + ./streams-grid.rst + ./configuration-grid.rst + diff --git a/docs/sections/components/component-specification/configuration-grid.rst b/docs/sections/components/component-specification/configuration-grid.rst new file mode 100755 index 00000000..956e1ff2 --- /dev/null +++ b/docs/sections/components/component-specification/configuration-grid.rst @@ -0,0 +1,315 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+Configuration Quick Reference
+=============================
+
+The following types of configuration are supported by the DCAE Platform.
+
++---------+---------+----------+---------+-----------+---+
+| Input | Default | Designer | Clamp | Policy | R\|
+| Sources | Values | Input | Input | Input | u\|
+| | | | | | n\|
+| | | | | | t\|
+| | | | | | i\|
+| | | | | | m\|
+| | | | | | e |
+| | | | | | I\|
+| | | | | | n\|
+| | | | | | p\|
+| | | | | | u\|
+| | | | | | t |
++=========+=========+==========+=========+===========+===+
+| Notes | | This | This | | |
+| | | applies | applies | | |
+| | | only to | only to | | |
+| | | componen\| compone\| | |
+| | | ts | nts | | |
+| | | that are | that | | |
+| | | self-ser\| are | | |
+| | | vice | part of | | |
+| | | (support\| a | | |
+| | | ed | closed-\| | |
+| | | by SDC) | loop | | |
+| | | | interfa\| | |
+| | | | ce | | |
++---------+---------+----------+---------+-----------+---+
+| Who | Compone\| Service | CLAMP | Operatio\ | R\|
+| provide\| nt | Designer | | ns | u\|
+| s? | Develop | | | | n\|
+| | er | | | | t\|
+| | | | | | i\|
+| | | | | | m\|
+| | | | | | e |
+| | | | | | P\|
+| | | | | | l\|
+| | | | | | a\|
+| | | | | | t\|
+| | | | | | f\|
+| | | | | | o\|
+| | | | | | r\|
+| | | | | | m |
++---------+---------+----------+---------+-----------+---+
+| When/Wh\| During | At | At | Anytime | W\|
+| ere | onboard\| design | install\| – in the | h\|
+| it is | ing | time – | ation | POLICY | e\|
+| provide | – in | in the | – in | GUI | n |
+| d | the | SDC UI | the | | t\|
+| | compone\| | CLAMP | | h\|
+| | nt | | UI | | e |
+| | specifi\| | | | c\|
+| | cation | | | | o\|
+| | | | | | m\|
+| | | | | | p\|
+| | | | | | o\|
+| | | | | | n\|
+| | | | | | e\|
+| | | | | | n\|
+| | | | | | t |
+| | | | | | i\|
+| | | | | | s |
+| | | | | | d\|
+| | | | | | e\|
+| | | | | | p\|
+| | | | | | l\|
+| | | | | | o\|
+| | | | | | y\|
+| | | | | | e\|
+| | | | | | d |
++---------+---------+----------+---------+-----------+---+
+| Compone\| For | ‘designe\| | ‘policy-\ | ‘\|
+| nt | CDAP: | r-editab\| | editable\ | s\|
+| Specifi\| ‘value’ | le’ | | ’ | o\|
+| cation | Name | must be | | must be | u\|
+| Details | and KV | set to | | set to | r\|
+| | pairs | ‘true’ | | ‘true’ | c\|
+| | in | for | | and | e\|
+| | AppConf\| variable | | ‘policy_s\| d\|
+| | ig | in | | chema’ | _\|
+| | or | ‘paramet\| | must be | a\|
+| | AppPref\| er’ | | provided | t\|
+| | erences | section. | | for | _\|
+| | For | | | variable | \ |
+| | Docker: | | | in | d\|
+| | ‘value’ | | | ‘paramet\ | e\|
+| | is | | | er’ | p\|
+| | provide | | | section | l\|
+| | d | | | | o\|
+| | for | | | | y\|
+| | variabl\| | | | m\|
+| | e | | | | e\|
+| | in | | | | n\|
+| | ‘parame\| | | | t\|
+| | ter’ | | | | ’ |
+| | section | | | | m\|
+| | | | | | u\|
+| | | | | | s\|
+| | | | | | t |
+| | | | | | b\|
+| | | | | | e |
+| | | | | | s\|
+| | | | | | e\|
+| | | | | | t |
+| | | | | | t\|
+| | | | | | o |
+| | | | | | ‘\|
+| | | | | | t\|
+| | | | | | r\|
+| | | | | | u\|
+| | | | | | e |
+| | | | | | ’\|
+| | | | | | f\|
+| | | | | | o\|
+| | | | | | r |
+| | | | | | v\|
+| | | | | | a\|
+| | | | | | r\|
+| | | | | | i\|
+| | | | | | a\|
+| | | | | | b\|
+| | | | | | l\|
+| | | | | | e\|
+| | | | | | i\|
+| | | | | | n\|
+| | | | | | ‘ |
+| | | | | | p\|
+| | | | | | a\|
+| | | | | | r\|
+| | | | | | a\|
+| | | | | | m\|
+| | | | | | e\|
+| | | | | | t\|
+| | | | | | e\|
+| | | | | | r\|
+| | | | | | ’ |
+| | | | | | s\|
+| | | | | | e\|
+| | | | | | c\|
+| | | | | | t\|
+| | | | | | i\|
+| | | | | | o\|
+| | | | | | n |
++---------+---------+----------+---------+-----------+---+
+
++---------+---------+----------+---------+-----------+---+
+| How it | This is | This | This | This | T\|
+| is used | passed | override\| overrid\| override\ | h\|
+| | to the | s | es | s | i\|
+| | compone\| any | any | any | s |
+| | nt | values | values | values | o\|
+| | in the | previous\| previou\| previous\ | v\|
+| | generat\| ly | sly | ly | e\|
+| | ed | set, but | set, | set, but | r\|
+| | configu\| can be | but can | can be | r\|
+| | ration | overridd\| be | overridd\ | i\|
+| | if not | en | overrid\| en | d\|
+| | overrid\| by CLAMP | den | at any | e\|
+| | den. | or | by | point | s |
+| | | POLICY. | POLICY. | thereaft\ | a\|
+| | | | | er. | n\|
+| | | | | | y |
+| | | | | | v\|
+| | | | | | a\|
+| | | | | | l\|
+| | | | | | u\|
+| | | | | | e\|
+| | | | | | s |
+| | | | | | p\|
+| | | | | | r\|
+| | | | | | e\|
+| | | | | | v\|
+| | | | | | i\|
+| | | | | | o\|
+| | | | | | u\|
+| | | | | | s\|
+| | | | | | l\|
+| | | | | | y |
+| | | | | | s\|
+| | | | | | e\|
+| | | | | | t\|
+| | | | | | , |
+| | | | | | b\|
+| | | | | | u\|
+| | | | | | t |
+| | | | | | c\|
+| | | | | | a\|
+| | | | | | n |
+| | | | | | b\|
+| | | | | | e |
+| | | | | | o\|
+| | | | | | v\|
+| | | | | | e\|
+| | | | | | r\|
+| | | | | | r\|
+| | | | | | i\|
+| | | | | | d\|
+| | | | | | d\|
+| | | | | | e\|
+| | | | | | n |
+| | | | | | a\|
+| | | | | | t |
+| | | | | | a\|
+| | | | | | n\|
+| | | | | | y |
+| | | | | | p\|
+| | | | | | o\|
+| | | | | | i\|
+| | | | | | n\|
+| | | | | | t |
+| | | | | | t\|
+| | | | | | h\|
+| | | | | | e\|
+| | | | | | r\|
+| | | | | | e\|
+| | | | | | a\|
+| | | | | | f\|
+| | | | | | t\|
+| | | | | | e\|
+| | | | | | r |
+| | | | | | b\|
+| | | | | | y |
+| | | | | | P\|
+| | | | | | o\|
+| | | | | | l\|
+| | | | | | i\|
+| | | | | | c\|
+| | | | | | y\|
+| | | | | | . |
++---------+---------+----------+---------+-----------+---+
+| Additio\| For | | | For | |
+| nal | CDAP: | | | Docker: | |
+| Info | ‘value’ | | | In the | |
+| for | is | | | auxiliar\ | |
+| Compone\| provide\| | | y | |
+| nt | d | | | section: | |
+| Develop\| for | | | {“policy | |
+| er | variabl\| | | ”: | |
+| | e | | | {“trigge\ | |
+| | in the | | | r_type”: | |
+| | ‘AppCon\| | | “policy” | |
+| | fig’ | | | ,“script\ | |
+| | or | | | _path”: | |
+| | ‘AppPre\| | | “/opt/ap\ | |
+| | ference\| | | p/reconf\ | |
+| | s’ | | | igure.sh\ | |
+| | section\| | | ”} | |
+| | s | | | } Script | |
+| | For | | | interfac\ | |
+| | Docker: | | | e | |
+| | ‘value’ | | | must be | |
+| | is | | | “opt/app\ | |
+| | provide\| | | /reconfi\ | |
+| | d | | | gure.sh” | |
+| | for | | | $trigger\ | |
+| | variabl\| | | _type | |
+| | e | | | $updated\ | |
+| | in | | | _policie\ | |
+| | ‘parame\| | | s | |
+| | ter’ | | | $updated\ | |
+| | section | | | _appl_co\ | |
+| | | | | nfig" | |
+| | | | | where | |
+| | | | | $updated\ | |
+| | | | | _policie\ | |
+| | | | | s | |
+| | | | | is a | |
+| | | | | json | |
+| | | | | provided | |
+| | | | | by the | |
+| | | | | Policy | |
+| | | | | Handler | |
+| | | | | and | |
+| | | | | $update_a\| |
+| | | | | ppl_con\ | |
+| | | | | fig | |
+| | | | | is the | |
+| | | | | post-mer\ | |
+| | | | | ged | |
+| | | | | appl\ | |
+| | | | | config | |
+| | | | | which | |
+| | | | | may | |
+| | | | | contain | |
+| | | | | unresolv\ | |
+| | | | | ed | |
+| | | | | configur\ | |
+| | | | | ation | |
+| | | | | that | |
+| | | | | didn’t | |
+| | | | | come | |
+| | | | | from | |
+| | | | | policy. | |
+| | | | | Suggesti\ | |
+| | | | | on | |
+| | | | | is for | |
+| | | | | script | |
+| | | | | to call | |
+| | | | | CONFIG | |
+| | | | | BINDING | |
+| | | | | SERVICE | |
+| | | | | to | |
+| | | | | resolve | |
+| | | | | any | |
+| | | | | configur\ | |
+| | | | | ation. | |
++---------+---------+----------+---------+-----------+---+
diff --git a/docs/sections/components/component-specification/docker-specification.rst b/docs/sections/components/component-specification/docker-specification.rst new file mode 100755 index 00000000..38612e17 --- /dev/null +++ b/docs/sections/components/component-specification/docker-specification.rst @@ -0,0 +1,454 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+.. _docker-specification:
+
+Component specification (Docker)
+================================
+
+The Docker component specification contains the following groups of
+information. Many of these are common to both Docker and CDAP components
+and are therefore described in the common specification.
+
+- :any:`Metadata <metadata>`
+- :any:`Interfaces <interfaces>` including the
+ associated :any:`Data Formats <data-formats>`
+- :any:`Parameters <parameters>`
+- :any:`Auxiliary Details <docker-auxiliary-details>`
+- :any:`List of Artifacts <artifacts>`
+
+.. _docker-auxiliary-details:
+
+Auxiliary Details
+-----------------
+
+``auxiliary`` contains Docker specific details like health check, port
+mapping, volume mapping, and policy reconfiguration script details.
+
++-------------+----+----------+
+| Name | Ty\| Descript\|
+| | pe | ion |
++=============+====+==========+
+| healthcheck | JS\| *Require\|
+| | ON | d*. |
+| | ob\| Health |
+| | je\| check |
+| | ct | definiti\|
+| | | on |
+| | | details |
++-------------+----+----------+
+| ports | JS\| each |
+| | ON | array |
+| | ar\| item |
+| | ra\| maps a |
+| | y | containe\|
+| | | r |
+| | | port to |
+| | | the host |
+| | | port. |
+| | | See |
+| | | example |
+| | | below. |
++-------------+----+----------+
+| volume | JS\| each |
+| | ON | array |
+| | ar\| item |
+| | ra\| contains |
+| | y | a host |
+| | | and |
+| | | containe\|
+| | | r |
+| | | object. |
+| | | See |
+| | | example |
+| | | below. |
++-------------+----+----------+
+| policy | JS\| *Require\|
+| | ON | d*. |
+| | ar\| Policy |
+| | ra\| script |
+| | y | details |
++-------------+----+----------+
+
+Health Check Definition
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The platform uses Consul to perform periodic health check calls. Consul
+provides different types of `check definitions <https://www.consul.io/docs/agent/checks.html>`_. The
+platform currently supports http and docker health checks.
+
+When choosing a value for interval, consider that too frequent
+healthchecks will put unnecessary load on Consul and DCAE. If there is a
+problematic resource, then more frequent healthchecks are warranted (eg
+15s or 60s), but as stablility increases, so can these values, (eg
+300s).
+
+When choosing a value for timeout, consider that too small a number will
+result in increasing timeout failures, and too large a number will
+result in a delay in the notification of resource problem. A suggestion
+is to start with 5s and workd from there.
+
+http
+^^^^
+
++-------------+----+----------+
+| Property | Ty\| Descript\|
+| Name | pe | ion |
++=============+====+==========+
+| type | st\| *Require\|
+| | ri\| d*. |
+| | ng | ``http`` |
++-------------+----+----------+
+| interval | st\| Interval |
+| | ri\| duration |
+| | ng | in |
+| | | seconds |
+| | | i.e. |
+| | | ``60s`` |
++-------------+----+----------+
+| timeout | st\| Timeout |
+| | ri\| in |
+| | ng | seconds |
+| | | i.e. |
+| | | ``5s`` |
++-------------+----+----------+
+| endpoint | st\| *Require\|
+| | ri\| d*. |
+| | ng | GET |
+| | | endpoint |
+| | | provided |
+| | | by the |
+| | | componen\|
+| | | t |
+| | | for |
+| | | Consul |
+| | | to call |
+| | | to check |
+| | | health |
++-------------+----+----------+
+
+Example:
+
+.. code:: json
+
+ "auxilary": {
+ "healthcheck": {
+ "type": "http",
+ "interval": "15s",
+ "timeout": "1s",
+ "endpoint": "/my-health"
+ }
+ }
+
+docker script example
+^^^^^^^^^^^^^^^^^^^^^
+
++-------------+----+------------+
+| Property | Ty\| Descript\ |
+| Name | pe | ion |
++=============+====+============+
+| type | st\| *Require\ |
+| | ri\| d*. |
+| | ng | ``docker`` |
++-------------+----+------------+
+| interval | st\| Interval |
+| | ri\| duration |
+| | ng | in |
+| | | seconds |
+| | | i.e. |
+| | | ``15s`` |
++-------------+----+------------+
+| timeout | st\| Timeout |
+| | ri\| in |
+| | ng | seconds |
+| | | i.e. |
+| | | ``1s`` |
++-------------+----+------------+
+| script | st\| *Require\ |
+| | ri\| d*. |
+| | ng | Full |
+| | | path of |
+| | | script |
+| | | that |
+| | | exists |
+| | | in the |
+| | | Docker |
+| | | containe\ |
+| | | r |
+| | | to be |
+| | | executed |
++-------------+----+------------+
+
+Consul will use the `Docker exec API <https://docs.docker.com/engine/api/v1.29/#tag/Exec>`_ to
+periodically call your script in your container. It will examine the
+script result to identify whether your component is healthy. Your
+component is considered healthy when the script returns ``0`` otherwise
+your component is considered not healthy.
+
+Example:
+
+.. code:: json
+
+ "auxilary": {
+ "healthcheck": {
+ "type": "docker",
+ "script": "/app/resources/check_health.py",
+ "timeout": "30s",
+ "interval": "180s"
+ }
+ }
+
+Ports
+~~~~~
+
+.. code:: json
+
+ "auxilary": {
+ "ports": ["8080:8000"]
+ }
+
+In the example above, container port 8080 maps to host port 8000.
+
+Volume Mapping
+~~~~~~~~~~~~~~
+
+.. code:: json
+
+ "auxilary": {
+ "volumes": [
+ {
+ "container": {
+ "bind": "/tmp/docker.sock",
+ "mode": "ro"
+ },
+ "host": {
+ "path": "/var/run/docker.sock"
+ }
+ }
+ ]
+ }
+
+At the top-level:
+
++---------------+-------+-------------------------------------+
+| Property Name | Type | Description |
++===============+=======+=====================================+
+| volumes | array | Contains container and host objects |
++---------------+-------+-------------------------------------+
+
+The ``container`` object contains:
+
++----------------------+----------------------+----------------------+
+| Property Name | Type | Description |
++======================+======================+======================+
+| bind | string | path to the |
+| | | container volume |
++----------------------+----------------------+----------------------+
+| mode | string | “ro” - indicates |
+| | | read-only volume |
+| | | “” - indicates that |
+| | | the container can |
+| | | write into the bind |
+| | | mount |
++----------------------+----------------------+----------------------+
+
+The ``host`` object contains:
+
++---------------+--------+-------------------------+
+| Property Name | Type | Description |
++===============+========+=========================+
+| path | string | path to the host volume |
++---------------+--------+-------------------------+
+
+Here’s an example of the minimal JSON that must be provided as an input:
+
+.. code:: json
+
+ "auxilary": {
+ "volumes": [
+ {
+ "container": {
+ "bind": "/tmp/docker.sock"
+ },
+ "host": {
+ "path": "/var/run/docker.sock"
+ }
+ }
+ ]
+ }
+
+In the example above, the container volume “/tmp/docker.sock” maps to
+host volume “/var/run/docker.sock”.
+
+Policy
+~~~~~~
+
+Policy changes made in the Policy UI will be provided to the Docker
+component by triggering a script that is defined here.
+
++-------------+----+------------+
+| Property | Ty\| Descript\ |
+| Name | pe | ion |
++=============+====+============+
+| reconfigure | st\| *Require\ |
+| _type | ri\| d*. |
+| | ng | Current |
+| | | value |
+| | | supporte |
+| | | d |
+| | | is |
+| | | ``policy`` |
++-------------+----+------------+
+| script_path | st\| *Require\ |
+| | ri\| d*. |
+| | ng | Current |
+| | | value |
+| | | for |
+| | | ‘policy’ |
+| | | reconfig\ |
+| | | ure_type |
+| | | must be |
+| | | “/opt/ap\ |
+| | | p/reconf\ |
+| | | igure.sh |
+| | | ” |
++-------------+----+------------+
+
+Example:
+
+.. code:: json
+
+ "auxilary": {
+ "policy": {
+ "reconfigure_type": "policy",
+ "script_path": "/opt/app/reconfigure.sh"
+ }
+ }
+
+The docker script interface is as follows: \`/opt/app/reconfigure.sh
+$reconfigure_type {“updated policies”: , “application config”: }
+
++-----+----+---------------------------+
+| Na\ | Ty\| Descript\ |
+| me | pe | ion |
++=====+====+===========================+
+| re\ | st\| “policy” |
+| co\ | ri\| |
+| nf\ | ng | |
+| ig\ | | |
+| ur\ | | |
+| e_t\| | |
+| y\ | | |
+| pe\ | | |
++-----+----+---------------------------+
+| up\ | js\| TBD |
+| da\ | on | |
+| te\ | | |
+| d_p\| | |
+| o\ | | |
+| li\ | | |
+| ci\ | | |
+| es | | |
++-----+----+---------------------------+
+| up\ | js\| complete |
+| da\ | on | generate\ |
+| te\ | | d |
+| d_a\| | app_conf\ |
+| p\ | | ig, |
+| pl\ | | not |
+| _c\ | | fully-re\ |
+| on\ | | solved, |
+| fi\ | | but |
+| g | | ``policy-enabled`` |
+| | | paramete\ |
+| | | rs |
+| | | have |
+| | | been |
+| | | updated. |
+| | | In order |
+| | | to get |
+| | | the |
+| | | complete |
+| | | updated |
+| | | app_conf\ |
+| | | ig, |
+| | | the |
+| | | componen\ |
+| | | t |
+| | | would |
+| | | have to |
+| | | call |
+| | | ``config-binding-service``|
+| | | . |
++-----+----+---------------------------+
+
+Docker Component Spec - Complete Example
+----------------------------------------
+
+.. code:: json
+
+ {
+ "self": {
+ "version": "1.0.0",
+ "name": "asimov.component.kpi_anomaly",
+ "description": "Classifies VNF KPI data as anomalous",
+ "component_type": "docker"
+ },
+ "streams": {
+ "subscribes": [{
+ "format": "dcae.vnf.kpi",
+ "version": "1.0.0",
+ "route": "/data",
+ "type": "http"
+ }],
+ "publishes": [{
+ "format": "asimov.format.integerClassification",
+ "version": "1.0.0",
+ "config_key": "prediction",
+ "type": "http"
+ }]
+ },
+ "services": {
+ "calls": [{
+ "config_key": "vnf-db",
+ "request": {
+ "format": "dcae.vnf.meta",
+ "version": "1.0.0"
+ },
+ "response": {
+ "format": "dcae.vnf.kpi",
+ "version": "1.0.0"
+ }
+ }],
+ "provides": [{
+ "route": "/score-vnf",
+ "request": {
+ "format": "dcae.vnf.meta",
+ "version": "1.0.0"
+ },
+ "response": {
+ "format": "asimov.format.integerClassification",
+ "version": "1.0.0"
+ }
+ }]
+ },
+ "parameters": [
+ {
+ "name": "threshold",
+ "value": 0.75,
+ "description": "Probability threshold to exceed to be anomalous"
+ }
+ ],
+ "auxilary": {
+ "healthcheck": {
+ "type": "http",
+ "interval": "15s",
+ "timeout": "1s",
+ "endpoint": "/my-health"
+ }
+ },
+ "artifacts": [{
+ "uri": "fake.nexus.com/dcae/kpi_anomaly:1.0.0",
+ "type": "docker image"
+ }]
+ }
diff --git a/docs/sections/components/component-specification/generated-configuration.rst b/docs/sections/components/component-specification/generated-configuration.rst new file mode 100755 index 00000000..ba5ae4a0 --- /dev/null +++ b/docs/sections/components/component-specification/generated-configuration.rst @@ -0,0 +1,114 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+.. _generated-configuration:
+
+Generated configuration
+=======================
+
+The DCAE platform relies on the component specification to generate the
+component’s application configuration JSON at deployment time. The
+component developer should expect to use this configuration JSON in
+their application to provision themselves.
+
+Pro-tip: As you build your component specification, you can use the
+:any:`dcae-cli dev command <walkthrough-dev>` to view what
+the resulting application configuration will look like.
+
+Streams and services
+--------------------
+
+For both Docker and CDAP, when your component is deployed, any
+``streams`` and ``services/calls`` you specified will be injected into
+your configuration under the following well known structure. Your
+component is required to parse this information if you have any
+connectivity to DMaaP or are calling another DCAE component.
+
+More details about the DMaaP connection objects are found
+:doc:`here <../dcae-cli/dmaap-connection-objects>`.
+
+This is best served with an example.
+
+The following component spec snippet (from String Matching):
+
+::
+
+ "streams":{
+ "subscribes": [{
+ "format": "VES_specification",
+ "version": "4.27.2",
+ "type": "message_router",
+ "config_key" : "mr_input"
+ }],
+ "publishes": [{
+ "format": "VES_specification",
+ "version": "4.27.2",
+ "config_key": "mr_output",
+ "type": "message_router"
+ }]
+ },
+ "services":{
+ "calls": [{
+ "config_key" : "aai_broker_handle",
+ "verb": "GET",
+ "request": {
+ "format": "get_with_query_params",
+ "version": "1.0.0"
+ },
+ "response": {
+ "format": "aai_broker_response",
+ "version": "3.0.0"
+ }
+ }],
+ "provides": []
+ },
+
+Will turn into the following top level keys in your configuration (for
+CDAP, this will be under AppConfig)
+
+::
+
+ "streams_publishes":{
+ "mr_output":{ // notice the config key above
+ "aaf_password":"XXX",
+ "type":"message_router",
+ "dmaap_info":{
+ "client_role": null,
+ "client_id": null,
+ "location": null,
+ "topic_url":"XXX"
+ },
+ "aaf_username":"XXX"
+ }
+ },
+ "streams_subscribes":{
+ "mr_input":{ // notice the config key above
+ "aaf_password":"XXX",
+ "type":"message_router",
+ "dmaap_info":{
+ "client_role": null,
+ "client_id": null,
+ "location": null,
+ "topic_url":"XXX"
+ },
+ "aaf_username":"XXX"
+ }
+ },
+ "services_calls":{
+ "aai_broker_handle":[ // notice the config key above
+ "SOME_IP:32768" // based on deployment time, just an example
+ ]
+ }
+
+These keys will always be populated regardless of whether they are
+empty. So the minimal you will get, in the case of a component that
+provides an HTTP service and does not call any services and has no
+streams, is:
+
+::
+
+ "streams_publishes":{},
+ "streams_subscribes":{},
+ "services_calls":{}
+
+Thus your component should expect these well-known top level keys.
diff --git a/docs/sections/components/component-specification/start-here.rst b/docs/sections/components/component-specification/start-here.rst new file mode 100755 index 00000000..d3f5a12f --- /dev/null +++ b/docs/sections/components/component-specification/start-here.rst @@ -0,0 +1 @@ +
diff --git a/docs/sections/components/component-specification/streams-grid.rst b/docs/sections/components/component-specification/streams-grid.rst new file mode 100755 index 00000000..6105e9e1 --- /dev/null +++ b/docs/sections/components/component-specification/streams-grid.rst @@ -0,0 +1,149 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+.. _streams-grid:
+
+Streams Formatting Quick Reference
+==================================
+
+Each of the following tables represents an example of a publisher and
+its subscriber, which are of course, different components. This focuses
+on the fields that are ‘different’ for each of these TYPEs, to
+illustrate the relationship between ``config_key``, dmaap connection
+object, and the generated configuration. Some notes on specific
+properties:
+
+- ``config_key`` is an arbitrary string, chosen by the component
+ developer. It is returned in the generated configuration where it
+ contains specific values for the target connection
+- ``format``, ``version``, and ``type`` properties in the subscriber
+ would match these properties in the publisher
+- ``aaf_username`` and ``aaf_password`` may be different between the
+ publisher and the subscriber
+
+Using http
+~~~~~~~~~~
+
+*Publishing Component*
+^^^^^^^^^^^^^^^^^^^^^^
+
++-----------------------------+----------------------------------------+
+| component \ | runtime platform generated config |
+| spec | |
++=============================+========================================+
+| "streams":{ | "streams_publishes":{ |
+| "publishes":[{ | "prediction":"10.100.1.100:32567/data" |
+| "config_key":"prediction", | |
+| "format":"some-format", | |
+| "type":"http", | |
+| "version":"0.1.0" } | |
+| ]} | |
++-----------------------------+----------------------------------------+
+
+*Subscribing Component*
+^^^^^^^^^^^^^^^^^^^^^^^
+
++-----------------------------+----------------------------------------+
+| component | runtime platform generated config |
+| spec | |
++=============================+========================================+
+| “streams”:{ | "N/A" |
+| "subscribes":[{ | |
+| "route":"/data", | |
+| "format":"some-format", | |
+| "type":"http" | |
+| "version":"0.1.0" } | |
+| ]} | |
++-----------------------------+----------------------------------------+
+
+Using Message Router
+~~~~~~~~~~~~~~~~~~~~
+
+.. publishing-component-1:
+
+*Publishing Component*
+^^^^^^^^^^^^^^^^^^^^^^
+
+Note: When deploying, this component should be deployed first so satisfy
+downstream dependencies. Refer to the –force option in component ‘run’
+command for more information.
+
++---------------+------------------------+-----------------------------------------------------------------------------+
+| component \ | Dmaap Connection \ | runtime platform generated \ |
+| spec | Object | config |
++===============+========================+=============================================================================+
+| “streams”:{ | { “dmaap_info”: | “streams_publishes”:{ |
+| “config_k\| {} \ *Note: For \ | “aaf_username”:“pub-user”, |
+| ey”:“mr_out\ | message router, this \ | “type”:“message_router”, |
+| put”, “t\ | object is identical \ | “topic_url”:"https://we-are-message-router.us:3905/events/some-topic"\ |
+| ype”:“messag\ | for the publisher and \| "streams_subscribes":{...} |
+| e_router”, | the subscriber* | |
+| }]} | | |
++---------------+------------------------+-----------------------------------------------------------------------------+
+
+*Subscribing Component*
+^^^^^^^^^^^^^^^^^^^^^^^
+
++---------------+------------------------+-----------------------------------------------------------------------------+
+| component \ | Dmaap Connection \ | runtime platform generated \ |
+| spec | Object | config |
++===============+========================+=============================================================================+
+| “streams”:{ | { “dmaap_info”: | “streams_publishes”:{…}, |
+| “config_k\| {} \ *Note: For \ | “streams_subscribes”:{ |
+| ey”:“mr_inp\ | message router, this \ | “aaf_username”:“sub-user”, |
+| ut”, “ty\ | object is identical \ | “type”:“message_router”, |
+| pe”:“message\ | for the publisher and \| “topic_url”:“https://we-are-message-router.us:3905/events/some-topic" |
+| _router”, | the subscriber* | |
+| }]} | | |
++---------------+------------------------+-----------------------------------------------------------------------------+
+
+Using Data Router
+~~~~~~~~~~~~~~~~~
+
+.. publishing-component-2:
+
+*Publishing Component*
+^^^^^^^^^^^^^^^^^^^^^^
+
++---------------+-----------------------------------------------+-----------------------------------------------+
+| component spec| Dmaap Connection Object | runtime platform generated config |
++===============+===============================================+===============================================+
+| “streams”:{ | { “dmaap_info”: { | streams_publishes“:{ ”typ\ |
+| “config_key: | “location”: | e“:”data_router“, "location":"mtc00" |
+| “dr_output" | “mtc00”, | , |
+| , "type": | “publish_url”: | "publish_url“: |
+| “data_r\ | "https://we-are-data-router.us/feed/xyz"\ | "http://we-are-data-router.us/feed/xyz" |
+| outer”, }] | , | , |
+| } | “log_url”:\ | "log_url“:\ |
+| | \ | ”https://we-are-data-router.us/feed/xyz/logs" |
+| | "https://we-are-data-router.us/feed/xyz/logs"\| , |
+| | , | ”username“:”pub-user“, |
+| | “username”: | ”publisher_id“:”123456\ |
+| | “pub-user”, | “}}, |
+| | “password”: | ”streams_subscribes“:{ |
+| | “pub-password”, | … } |
+| | “publisher_id”: | |
+| | “123456”}} | |
++---------------+-----------------------------------------------+-----------------------------------------------+
+
+.. subscribing-component-1:
+
+*Subscribing Component*
+^^^^^^^^^^^^^^^^^^^^^^^
+
++---------------+---------------------------------------------------+---------------------------------------------------------------------------+
+| component \ | Dmaap Connection \ | runtime platform generated \ |
+| spec | Object | config |
++===============+===================================================+===========================================================================+
+| “streams”:{ | { “dmaap_info”: | “streams_publishes”:{ … }, |
+| “config_k\| { “location”: | “streams_subscribes”:{ |
+| ey”:“dr_inp\ | “mtc00”, | “type”:“data_router”, |
+| ut”, “ty\ | “delivery_url”: | “location”:“mtc00”, |
+| pe”:“data_ro\ | "https://my-subscriber-app.dcae:8080/target-path"\| “delivery_url”:"https://my-subscriber-app.dcae:8080/target-path"\|
+| uter”, | \ | \ |
+| “route”: | , | , |
+| “/target-pat\ | “password”: | \ |
+| h”} | “sub-password”, | “username”:“sub-user”, |
+| | “subscriber_id”: | |
+| | “789012”}} | “subscriber_id”:“789012”}} |
++---------------+---------------------------------------------------+---------------------------------------------------------------------------+
diff --git a/docs/sections/components/component-type-cdap.rst b/docs/sections/components/component-type-cdap.rst new file mode 100755 index 00000000..0f5150fd --- /dev/null +++ b/docs/sections/components/component-type-cdap.rst @@ -0,0 +1,90 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+.. _cdap-requirements:
+
+CDAP Component Requirements
+===========================
+
+This page contains information about CDAP app development in DCAE.
+
+Uploading your Jar File
+-----------------------
+
+The DCAE component specification has you input your ``jar_url``, the URL
+on Nexus to your Jar file. This DCAE controller deploys out of Nexus.
+You can upload your jar(s) using the following command, replacing NAME:
+
+::
+
+ curl -v --user 'user:password' http://YOUR_NEXUS_RAW_REPO/NAME.jar --upload-file NAME.jar
+
+During the CLI Tool Usage, in your spec, supply
+``http://YOUR_NEXUS_RAW_REPO/NAME.jar`` as the JAR artifact URL.
+
+Policy Reconfiguration
+----------------------
+
+We support reconfiguration of both AppConfig and AppPreferences.
+
+For AppConfig, we support CDAPs “update” API to `reconfigure an application <http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/lifecycle.html#update-an-application%7D%7D>`_.
+
+For AppPreferences, we:
+
+1. Stop your programs
+
+2. Set the new preferences
+
+3. Start your programs
+
+At the time of writing, there is no way to update a CDAP application’s
+AppConfig or AppPreferences, without a restart, *and notify* the
+application. The latter is a future promised feature by CASK—the ability
+to update preferences and inform the application that something is
+changed (so it repulls). As CDAP currently stands however, given the
+above, if you are building a stateful application, you must persist your
+state often (e.g., to a CDAP dataset), as you may be restarted at any
+time with an updated configuration, or stopped&started at any time with
+updated preferences.
+
+Metrics
+-------
+
+Metrics are pulled from your CDAP application on a periodic basis and
+(in the future: pushed to a central DCAE metric store, currently: just
+dropped). For this to be useful, your application should provide `metrics <http://docs.cask.co/cdap/current/en/admin-manual/operations/metrics.html>`_.
+While nothing in the DCAE runtime enforces that your CDAP application
+tracks metrics, your metrics (or lack thereof) will be visible in the
+DCAE dashboard and to operations.
+
+.. _dmaap-abstraction:
+
+Future DMaaP abstraction
+------------------------
+
+Shown below is our *vision* for how DMaaP is abstracted from component
+developers:
+
+.. figure:: ../images/dmdvision.png
+ :alt: Screenshot
+
+ Screenshot
+
+Today, this is a vision; it is not in place. Today, each CDAP app is
+built with built in assumptions about where they are getting their data
+from. Some CDAP apps have the built in assumption of a UEB feed. Some
+MR. Some DR. This becomes very difficult to orchestrate when each app in
+the catalog has built in data assumptions.
+
+The goal of this vision is to *decouple* the data plane from the
+analytics plane. Analytics should be agnostic to *how* they are
+receiving their data beyond “filesystem” or “HTTP”. Analytics developers
+shouldn’t have to worry about the data plane, that should be taken care
+of by the platform. They should be spending their time on the problem at
+hand—the analytic.
+
+This also allows each CDAP application to have a standard set of
+interfaces: HTTP and HDFS: |Screenshot|
+
+.. |Screenshot| image:: ../images/io.png
+
diff --git a/docs/sections/components/component-type-docker.rst b/docs/sections/components/component-type-docker.rst new file mode 100755 index 00000000..449e58a8 --- /dev/null +++ b/docs/sections/components/component-type-docker.rst @@ -0,0 +1,319 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+.. _docker-requirements:
+
+Docker Component Requirements
+=============================
+
+Overview
+--------
+
+Component developers are required to provide artifacts for the platform
+to be able to deploy your component including:
+
+- :any:`Component specification <docker-specification>`
+- :doc:`Data formats <data-formats>`
+- :any:`Auxiliary component specification <docker-auxiliary-details>`
+- :any:`Docker image <docker-on-the-platform>`
+
+In addition, components will have to be enhanced to be compliant with
+the DCAE platform in order to correctly be deployed and be managed. This
+page will discuss the changes which are grouped into the following
+categories:
+
+- :any:`Service registration <service-registration>`
+- :any:`Configuration management on the new platform <docker-configuration>`
+- :any:`Operational concerns <operational-concerns>`
+
+To help component developers to make and to test the changes needed to
+have components run on the platform, a command-line tool called
+:doc:`dcae-cli <dcae-cli/quickstart>` is provided by the platform team.
+
+.. _service-registration:
+
+Service registration
+--------------------
+
+Every :doc:`Docker component is registered <architecture/service-discovery>` with the platform’s
+service discovery layer. Docker components are not expected to make the
+explicit make registration calls because that is done by through a
+platform 3rd party registration service. A couple things are needed from
+component developers in order for this registration to occur
+successfully:
+
+1. Docker images must be created from a Dockerfile that has an `EXPOSE <https://docs.docker.com/engine/reference/builder/#/expose>`_ instruction. This applies to components that listen on a port.
+2. Component healthcheck details must be provided in the Docker
+ auxiliary component specification
+
+Expose port
+~~~~~~~~~~~
+
+Components that listen on a specific port must explicitly declare in
+their Dockerfile that port using the ``EXPOSE`` instruction before
+building the image. Warning! At the current time, you can not expose
+multiple ports in your Dockerfile or registration *will not work*
+correctly.
+
+Health check
+~~~~~~~~~~~~
+
+Component developers are required to provide a way for the platform to periodically check the health of their running components. The platform uses Consul to perform these periodic calls. Consul provides different types of `check definitions <https://www.consul.io/docs/agent/checks.html>`_. The details of the definition used by your component is to be provided through the :any:`Docker auxiliary specification <docker-auxiliary-details>`.
+
+.. _docker-configuration:
+
+Configuration
+-------------
+
+The component’s application configuration is generated from the
+submitted component specification into a JSON representation. The DCAE
+platform will provide the configuration JSON by making it available on a
+remote HTTP service. Components are required to pull their configuration
+JSON at application startup. The configuration JSON is stored under the
+components uniquely generated name which is provided by the environment
+variable ``HOSTNAME``.
+
+You can see more details on the generated application configuration
+:any:`here <docker-specification>`.
+
+Config binding service
+~~~~~~~~~~~~~~~~~~~~~~
+
+The config binding service is a platform HTTP service that is
+responsible for providing clients with a fully resolved configuration
+JSON at runtime. Components should make an HTTP GET on:
+
+::
+
+ <config binding service hostname>:<port>/service_component/NAME
+
+For Docker components, NAME should be set to ``HOSTNAME``, which was
+provided as an ENV variable inside of your container.
+
+The binding service integrates with the streams and services section of
+your component specification. For example, if you specify that you call
+a service:
+
+::
+
+ "services": {
+ "calls": [{
+ "config_key": "vnf-db",
+ "request": {
+ "format": "dcae.vnf.meta",
+ "version": "1.0.0"
+ },
+ "response": {
+ "format": "dcae.vnf.kpi",
+ "version": "1.0.0"
+ }
+ }],
+ ...
+ }
+
+Then the config binding service will find all available IP addresses of
+services meeting your needs, and provide them to you under your
+``config_key``:
+
+::
+
+ // your configuration
+ {
+ "vbf-db" : // see above
+ [IP:Port1, IP:Port2,…] // all of these meet your needs, choose one.
+ }
+
+Regarding ``<config binding service hostname>:<port>``, there is DNS
+work going on to make this resolvable in a convenient way inside of your
+container. However, currently you will be given a name as an ENV
+variable, ``CONFIG_BINDING_SERVICE``, and you will need to query
+Consul’s service discovery to get
+``<config binding service hostname>:<port>``.
+
+Policy Reconfiguration
+~~~~~~~~~~~~~~~~~~~~~~
+
+*(Draft and subject to change)*
+
+Components must provide a way to receive policy reconfiguration, that
+is, configuration parameters that have been updated via the Policy UI.
+The component developer provides a docker script (defined in the :any:`Docker auxiliary specification <docker-specification>`)
+that will be triggered when this occurs.
+
+DMaaP
+~~~~~
+
+Components can be publishers or subscribers to either message router
+topics or data router feeds. This is defined in the component
+specification under the ``streams`` section where you can specify
+whether your component is expected to subscribe or to publish to a
+:any:`message router topic <message-router>` or to a :any:`data router feed <data-router>`.
+Given a composition with components that use DMaaP, the platform will
+provision the topic or feed and provide the necessary :any:`connection details <dcae-cli/dmaap-connection-objects>` at runtime for each DMaaP
+dependent component. These connection details will be provided through
+your application’s :any:`generated configuration <component-specification/generated-configuration>`.
+
+In order to test DMaaP connections in onboarding, the developer
+(currently) must provision all test topics and feeds manually and
+provide the :any:`dcae-cli with the connection details <dmaap-testing>` when deploying your
+application.
+
+.. _docker-on-the-platform:
+
+Docker on the platform
+----------------------
+
+Images
+~~~~~~
+
+Docker images must be pushed to the environment specific Nexus
+repository. This requires tagging your build with the full name of you
+image which includes the Nexus repository name.
+
+Use the Docker command-line to `tag <https://docs.docker.com/engine/reference/commandline/tag/>`_ your Docker image where the *target image* must contain the registry host name and port.
+
+For example, an application called laika has been tagged for an example
+Nexus registry:
+
+::
+
+ $ docker images
+ REPOSITORY TAG IMAGE ID CREATED SIZE
+ YOUR_NEXUS_DOCKER_REGISTRY/laika 0.4.0 154cc382df61 7 weeks ago 710.5 MB
+ laika 0.4.0 154cc382df61 7 weeks ago 710.5 MB
+
+Note, the Docker registry that is used may require a login to
+authenticate.
+
+::
+
+ docker login YOUR_NEXUS_DOCKER_REGISTRY
+
+Tag your image:
+
+::
+
+ docker tag laika:0.4.0 YOUR_NEXUS_DOCKER_REGISTRY/laika:0.4.0
+
+Or build and tag:
+
+::
+
+ docker build -t YOUR_NEXUS_DOCKER_REGISTRY/laika:0.4.0 .
+
+After tagging, upload your image to the remote registry using the Docker `push command <https://docs.docker.com/engine/reference/commandline/push/>`_ .
+Note that the registry may require a login. Use the Docker `login command <https://docs.docker.com/engine/reference/commandline/login/>`_
+before pushing in that case.
+
+::
+
+ docker push YOUR_NEXUS_DOCKER_REGISTRY/laika:0.4.0
+
+*NOTE* Replace ``laika`` with your application’s name. Replace the
+``0.4.0`` version with your application’s version.
+
+Ports
+~~~~~
+
+On the DCAE platform, Docker components are run with the ``--publish-all`` or ``-P`` argument. This means the Docker container
+for your component will be listening on a random port and that random
+port will be mapped to the port :any:`you exposed <service-registration>`.
+
+Envs
+~~~~
+
+The platform provides a set of environment variables into each Docker
+container:
+
++----------------------------+----+----------+
+| Na\ | Ty\| Descript\|
+| me | pe | ion |
++============================+====+==========+
+| ``HOSTNAME`` | st\| Unique |
+| | ri\| name of |
+| | ng | the |
+| | | componen\|
+| | | t |
+| | | instance |
+| | | that is |
+| | | generate\|
+| | | d |
++----------------------------+----+----------+
+| ``CONSOL_HOST`` | st\| Hostname |
+| | ri\| of the |
+| | ng | platform\|
+| | | ’s |
+| | | Consul |
+| | | instance |
+| | | |
+| | | |
++----------------------------+----+----------+
+| ``CONFIG_BINDING_SERVICE`` | st\| Hostname |
+| | ri\| of the |
+| | ng | platform\|
+| | | ’s |
+| | | config |
+| | | binding |
+| | | service |
+| | | instance |
+| | | |
+| | | |
+| | | |
+| | | |
+| | | |
++----------------------------+----+----------+
+| ``DOCKER_HOSTS`` | st\| Host of |
+| | ri\| the |
+| | ng | target |
+| | | platform |
+| | | Docker |
+| | | host to |
+| | | run the |
+| | | containe\|
+| | | r |
+| | | on |
++----------------------------+----+----------+
+
+.. _operational-concerns:
+
+Operational concerns
+--------------------
+
+Logging
+~~~~~~~
+
+Currently the platform uses the default ``json-file`` logging driver for
+Docker. For onboarding testing, component developers can access their
+logs from their Docker containers either by running their component
+using the ``--attached`` flag or by using the ``docker logs`` command.
+The requirement is that applications must write to stdout and/or stderr.
+
+To use the ``docker logs`` command for your deployed running Docker
+container,
+
+1. You must have Docker installed on your local machine
+2. Have the generated name of your component. This is generated for you
+ when you execute ``dcae_cli component dev`` or
+ ``dcae_cli component run``.
+3. Find the target Docker host using the ``dcae_cli profiles show``
+ command:
+
+::
+
+ $ dcae_cli profiles show YOUR_PROFILE_NAME
+ {
+ ...
+ "docker_host": "YOUR_DOCKER_HOST:2376"
+ }
+
+4. Set your Docker client to point to the target Docker host:
+
+::
+
+ $ export DOCKER_HOST="tcp://YOUR_DOCKER_HOST:2376"
+
+5. Use the ``docker logs`` command:
+
+::
+
+ $ docker logs <generated component name>
diff --git a/docs/sections/components/data-formats.rst b/docs/sections/components/data-formats.rst new file mode 100755 index 00000000..ecb019c2 --- /dev/null +++ b/docs/sections/components/data-formats.rst @@ -0,0 +1,418 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+.. _data-formats:
+
+Data Formats
+============
+
+| Because the DCAE designer composes your component with others at
+ service design time, in most cases you do not know what specific
+ component(s) your component will send data to during runtime. Thus, it
+ is vital that DCAE has a language of describing the data passed
+ between components, so that it is known which components are
+ composable with others. Data formats are descriptions of data—they are
+ the data contract between your component and other components. You
+ need to describe the available outputs and assumed inputs of your
+ components as data formats. These data descriptions are onboarded into
+ ASDC, and each receives a UUID. If component X outputs data format
+ DF-Y, and another component Z specifies DF-Y as their input data
+ format, then X is said to be *composable* with that component. The
+ data formats are referenced in the component specifications by the
+ data format’s id and version.
+| The vision is to have a repository of shared data formats that
+ developers and teams can re-use and also provide them the means to
+ extend and create new custom data formats.
+
+.. _dataformat_metadata:
+
+Meta Schema Definition
+----------------------
+
+The “Meta Schema” implementation defines how data format JSON schemas
+can be written to define user input. It is itself a JSON schema (thus it
+is a “meta schema”). It requires the name of the data format entry, the
+data format entry version and allows a description under “self” object.
+The meta schema version must be specified as the value of the
+“dataformatversion” key. Then the input schema itself is described.
+There are four types of schema descriptions objects - jsonschema for
+inline standard JSON Schema definitions of JSON inputs, delimitedschema
+for delimited data input using a defined JSON description, unstructured
+for unstructured text, and reference that allows a pointer to another
+artifact for a schema. The reference allows for XML schema, but can be
+used as a pointer to JSON, Delimited Format, and Unstructured schemas as
+well.
+
+The current Meta Schema implementation is defined below:
+
+::
+
+ {
+ "$schema": "http://json-schema.org/draft-04/schema#",
+ "title": "Data format specification schema Version 1.0",
+ "type": "object",
+ "oneOf": [{
+ "properties": {
+ "self": {
+ "$ref": "#/definitions/self"
+ },
+ "dataformatversion": {
+ "$ref": "#/definitions/dataformatversion"
+ },
+ "reference": {
+
+ "type": "object",
+ "description": "A reference to an external schema - name/version is used to access the artifact",
+ "properties": {
+ "name": {
+ "$ref": "#/definitions/name"
+ },
+ "version": {
+ "$ref": "#/definitions/version"
+ },
+ "format": {
+ "$ref": "#/definitions/format"
+ }
+ },
+ "required": [
+ "name",
+ "version",
+ "format"
+ ],
+ "additionalProperties": false
+ }
+ },
+ "required": ["self", "dataformatversion", "reference"],
+ "additionalProperties": false
+ }, {
+ "properties": {
+ "self": {
+ "$ref": "#/definitions/self"
+ },
+ "dataformatversion": {
+ "$ref": "#/definitions/dataformatversion"
+ },
+ "jsonschema": {
+ "$schema": "http://json-schema.org/draft-04/schema#",
+ "description": "The actual JSON schema for this data format"
+ }
+
+ },
+ "required": ["self", "dataformatversion", "jsonschema"],
+ "additionalProperties": false
+ }, {
+ "properties": {
+ "self": {
+ "$ref": "#/definitions/self"
+ },
+ "dataformatversion": {
+ "$ref": "#/definitions/dataformatversion"
+ },
+ "delimitedschema": {
+ "type": "object",
+ "description": "A JSON schema for delimited files",
+ "properties": {
+ "delimiter": {
+ "enum": [",", "|", "\t", ";"]
+ },
+ "fields": {
+ "type": "array",
+ "description": "Array of field descriptions",
+ "items": {
+ "$ref": "#/definitions/field"
+ }
+ }
+ },
+ "additionalProperties": false
+ }
+ },
+ "required": ["self", "dataformatversion", "delimitedschema"],
+ "additionalProperties": false
+ }, {
+ "properties": {
+ "self": {
+ "$ref": "#/definitions/self"
+ },
+ "dataformatversion": {
+ "$ref": "#/definitions/dataformatversion"
+ },
+ "unstructured": {
+ "type": "object",
+ "description": "A JSON schema for unstructured text",
+ "properties": {
+ "encoding": {
+ "type": "string",
+ "enum": ["ASCII", "UTF-8", "UTF-16", "UTF-32"]
+ }
+ },
+ "additionalProperties": false
+
+ }
+ },
+ "required": ["self", "dataformatversion", "unstructured"],
+ "additionalProperties": false
+ }],
+ "definitions": {
+ "name": {
+ "type": "string"
+ },
+ "version": {
+ "type": "string",
+ "pattern": "^(\\d+\\.)(\\d+\\.)(\\*|\\d+)$"
+ },
+ "self": {
+ "description": "Identifying Information for the Data Format - name/version can be used to access the artifact",
+ "type": "object",
+ "properties": {
+ "name": {
+ "$ref": "#/definitions/name"
+ },
+ "version": {
+ "$ref": "#/definitions/version"
+ },
+ "description": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "name",
+ "version"
+ ],
+ "additionalProperties": false
+ },
+ "format": {
+ "description": "Reference schema type",
+ "type": "string",
+ "enum": [
+ "JSON",
+ "Delimited Format",
+ "XML",
+ "Unstructured"
+ ]
+ },
+ "field": {
+ "description": "A field definition for the delimited schema",
+ "type": "object",
+ "properties": {
+ "name": {
+ "type": "string"
+ },
+ "description": {
+ "type": "string"
+ },
+ "fieldtype": {
+ "description": "the field type - from the XML schema types",
+ "type": "string",
+ "enum": ["string", "boolean",
+ "decimal", "float", "double",
+ "duration", "dateTime", "time",
+ "date", "gYearMonth", "gYear",
+ "gMonthDay", "gDay", "gMonth",
+ "hexBinary", "base64Binary",
+ "anyURI", "QName", "NOTATION",
+ "normalizedString", "token",
+ "language", "IDREFS", "ENTITIES",
+ "NMTOKEN", "NMTOKENS", "Name",
+ "NCName", "ID", "IDREF", "ENTITY",
+ "integer", "nonPositiveInteger",
+ "negativeInteger", "long", "int",
+ "short", "byte",
+ "nonNegativeInteger", "unsignedLong",
+ "unsignedInt", "unsignedShort",
+ "unsignedByte", "positiveInteger"
+
+ ]
+ },
+ "fieldPattern": {
+ "description": "Regular expression that defines the field format",
+ "type": "integer"
+ },
+ "fieldMaxLength": {
+ "description": "The maximum length of the field",
+ "type": "integer"
+ },
+ "fieldMinLength": {
+ "description": "The minimum length of the field",
+ "type": "integer"
+ },
+ "fieldMinimum": {
+ "description": "The minimum numeric value of the field",
+ "type": "integer"
+ },
+ "fieldMaximum": {
+ "description": "The maximum numeric value of the field",
+ "type": "integer"
+ }
+ },
+ "additionalProperties": false
+ },
+ "dataformatversion": {
+ "type": "string",
+ "enum": ["1.0.0"]
+ }
+ }
+ }
+
+Examples
+-----------
+
+By reference example - Common Event Format
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+First the full JSON schema description of the Common Event Format would
+be loaded with a name of “Common Event Format” and the current version
+of “25.0.0”.
+
+Then the data format description is loaded by this schema:
+
+::
+
+ {
+ "self": {
+ "name": "Common Event Format Definition",
+ "version": "25.0.0",
+ "description": "Common Event Format Definition"
+
+ },
+ "dataformatversion": "1.0.0",
+ "reference": {
+ "name": "Common Event Format",
+ "format": "JSON",
+ "version": "25.0.0"
+ }
+ }
+
+
+
+Simple JSON Example
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+::
+
+ {
+ "self": {
+ "name": "Simple JSON Example",
+ "version": "1.0.0",
+ "description": "An example of unnested JSON schema for Input and output"
+
+ },
+ "dataformatversion": "1.0.0",
+ "jsonschema": {
+ "$schema": "http://json-schema.org/draft-04/schema#",
+ "type": "object",
+ "properties": {
+ "raw-text": {
+ "type": "string"
+ }
+ },
+ "required": ["raw-text"],
+ "additionalProperties": false
+ }
+ }
+
+Nested JSON Example
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ {
+ "self": {
+ "name": "Nested JSON Example",
+ "version": "1.0.0",
+ "description": "An example of nested JSON schema for Input and output"
+
+ },
+ "dataformatversion": "1.0.0",
+ "jsonschema": {
+ "$schema": "http://json-schema.org/draft-04/schema#",
+ "properties": {
+ "numFound": {
+ "type": "integer"
+ },
+ "start": {
+ "type": "integer"
+ },
+ "engagements": {
+ "type": "array",
+ "items": {
+ "properties": {
+ "engagementID": {
+ "type": "string",
+ "transcript": {
+ "type": "array",
+ "items": {
+ "type": {
+ "type": "string"
+ },
+ "content": {
+ "type": "string"
+ },
+ "senderName": {
+ "type": "string"
+ },
+ "iso": {
+ "type": "string"
+ },
+ "timestamp": {
+ "type": "integer"
+ },
+ "senderId": {
+ "type": "string"
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "additionalProperties": false
+ }
+ }
+
+Unstructured Example
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ {
+ "self": {
+ "name": "Unstructured Text Example",
+ "version": "25.0.0",
+ "description": "An example of a unstructured text used for both input and output for "
+
+ },
+ "dataformatversion": "1.0.0",
+ "unstructured": {
+ "encoding": "UTF-8"
+ }
+ }
+
+
+An example of a delimited schema
+--------------------------------
+
+::
+
+ {
+ "self": {
+ "name": "Delimited Format Example",
+ "version": "1.0.0",
+ "description": "Delimited format example just for testing"
+
+ },
+ "dataformatversion": "1.0.0",
+ "delimitedschema": {
+ "delimiter": "|",
+ "fields": [{
+ "name": "field1",
+ "description": "test field1",
+ "fieldtype": "string"
+ }, {
+ "name": "field2",
+ "description": "test field2",
+ "fieldtype": "boolean"
+ }]
+ }
+ }
diff --git a/docs/sections/components/dcae-cli/dcae-cli-tool.rst b/docs/sections/components/dcae-cli/dcae-cli-tool.rst new file mode 100644 index 00000000..6f564cee --- /dev/null +++ b/docs/sections/components/dcae-cli/dcae-cli-tool.rst @@ -0,0 +1,13 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +dcae-cli Tool +============= + +.. toctree:: + :maxdepth: 1 + + ./quickstart.rst + ./walkthrough.rst + ./dmaap-connection-objects.rst + diff --git a/docs/sections/components/dcae-cli/dmaap-connection-objects.rst b/docs/sections/components/dcae-cli/dmaap-connection-objects.rst new file mode 100755 index 00000000..cad3b0c9 --- /dev/null +++ b/docs/sections/components/dcae-cli/dmaap-connection-objects.rst @@ -0,0 +1,415 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+DMaaP connection objects
+========================
+
+DMaaP connection objects are JSON objects that:
+
+1. Components should expect at runtime in their application
+ configuration and is to be used to connect to the appropriate DMaaP
+ feed or topic.
+2. Developers must provide through the command-line argument
+ ``--dmaap-file`` to test their component with manually provisioned
+ feeds and topics.
+
+This page is a reference to the specific structure that each type of
+DMaaP stream requires.
+
+Note for #1 that components should expect the entire object with all
+properties at runtime where the default will be ``null`` unless
+specified otherwise.
+
+Note for #2 that developers are not required to provide the entire
+object. The required properties will be labeled with “*required as
+input*”.
+
+.. _dmaap-message-router:
+
+Message router
+--------------
+
+Publishers and subscribers both have the same JSON object structure.
+Here’s an example:
+
+.. code:: json
+
+ {
+ "type": "message_router",
+ "aaf_username": "some-user",
+ "aaf_password": "some-password",
+ "dmaap_info": {
+ "client_role": "com.dcae.member",
+ "client_id": "1500462518108",
+ "location": "mtc00",
+ "topic_url": "https://we-are-message-router.us:3905/events/some-topic"
+ }
+ }
+
+
+At the top-level:
+
++-------------+----+--------------------+
+| Property | Ty\| Descript\ |
+| Name | pe | ion |
++=============+====+====================+
+| type | st\| *Require\ |
+| | ri\| d \ |
+| | ng | as \ |
+| | | input*. |
+| | | Must be |
+| | | ``message_router`` |
+| | | for |
+| | | message |
+| | | router |
+| | | topics |
++-------------+----+--------------------+
+| aaf_usernam\| st\| AAF |
+| e | ri\| username |
+| | ng | message |
+| | | router |
+| | | clients |
+| | | use to |
+| | | authenti\ |
+| | | cate |
+| | | with |
+| | | secure |
+| | | topics |
++-------------+----+--------------------+
+| aaf_passwor\| st\| AAF |
+| d | ri\| password |
+| | ng | message |
+| | | router |
+| | | clients |
+| | | use to |
+| | | authenti\ |
+| | | cate |
+| | | with |
+| | | secure |
+| | | topics |
++-------------+----+--------------------+
+| dmaap_info | JS\| *Require\ |
+| | ON | d \ |
+| | ob\| as \ |
+| | je\| input*. |
+| | ct | Contains |
+| | | the |
+| | | topic |
+| | | connecti\ |
+| | | on |
+| | | details |
++-------------+----+--------------------+
+
+The ``dmaap_info`` object contains:
+
++-------------+----+----------+
+| Property | Ty\| Descript\|
+| Name | pe | ion |
++=============+====+==========+
+| client_role | st\| AAF |
+| | ri\| client |
+| | ng | role |
+| | | that’s |
+| | | requesti\|
+| | | ng |
+| | | publish |
+| | | or |
+| | | subscrib\|
+| | | e |
+| | | access |
+| | | to the |
+| | | topic |
++-------------+----+----------+
+| client_id | st\| Client |
+| | ri\| id for |
+| | ng | given |
+| | | AAF |
+| | | client |
++-------------+----+----------+
+| location | st\| DCAE |
+| | ri\| location |
+| | ng | for the |
+| | | publishe\|
+| | | r |
+| | | or |
+| | | subscrib\|
+| | | er, |
+| | | used to |
+| | | set up |
+| | | routing |
++-------------+----+----------+
+| topic_url | st\| *Require\|
+| | ri\| d \ |
+| | ng | as \ |
+| | | input*. |
+| | | URL for |
+| | | accessin\|
+| | | g |
+| | | the |
+| | | topic to |
+| | | publish |
+| | | or |
+| | | receive |
+| | | events |
++-------------+----+----------+
+
+Here’s an example of the minimal JSON that must be provided as an input:
+
+.. code:: json
+
+ {
+ "type": "message_router",
+ "dmaap_info": {
+ "topic_url": "https://we-are-message-router.us:3905/events/some-topic"
+ }
+ }
+
+.. _dmaap-data-router:
+
+Data router
+-----------
+
+Publisher
+~~~~~~~~~
+
+Here’s an example of what the JSON object connection for data router
+publisher looks like:
+
+.. code:: json
+
+ {
+ "type": "data_router",
+ "dmaap_info": {
+ "location": "mtc00",
+ "publish_url": "https://we-are-data-router.us/feed/xyz",
+ "log_url": "https://we-are-data-router.us/feed/xyz/logs",
+ "username": "some-user",
+ "password": "some-password",
+ "publisher_id": "123456"
+ }
+ }
+
+At the top-level:
+
++-------------+----+----------------+
+| Property | Ty\| Descript\ |
+| Name | pe | ion |
++=============+====+================+
+| type | st\| *Require\ |
+| | ri\| d \ |
+| | ng | as \ |
+| | | input*. |
+| | | Must be |
+| | | ``data_router``|
+| | | for data |
+| | | router |
+| | | feeds |
++-------------+----+----------------+
+| dmaap_info | JS\| *Require\ |
+| | ON | d \ |
+| | ob\| as \ |
+| | je\| input*. |
+| | ct | Contains |
+| | | the |
+| | | topic |
+| | | connecti\ |
+| | | on |
+| | | details |
++-------------+----+----------------+
+
+The ``dmaap_info`` object contains:
+
++-------------+----+----------+
+| Property | Ty\| Descript\|
+| Name | pe | ion |
++=============+====+==========+
+| location | st\| DCAE |
+| | ri\| location |
+| | ng | for the |
+| | | publishe\|
+| | | r, |
+| | | used to |
+| | | set up |
+| | | routing |
++-------------+----+----------+
+| publish_url | st\| *Require\|
+| | ri\| d \ |
+| | ng | as \ |
+| | | input*. |
+| | | URL to |
+| | | which |
+| | | the |
+| | | publishe\|
+| | | r |
+| | | makes |
+| | | Data |
+| | | Router |
+| | | publish |
+| | | requests |
++-------------+----+----------+
+| log_url | st\| URL from |
+| | ri\| which |
+| | ng | log data |
+| | | for the |
+| | | feed can |
+| | | be |
+| | | obtained |
++-------------+----+----------+
+| username | st\| Username |
+| | ri\| the |
+| | ng | publishe\|
+| | | r |
+| | | uses to |
+| | | authenti\|
+| | | cate |
+| | | to Data |
+| | | Router |
++-------------+----+----------+
+| password | st\| Password |
+| | ri\| the |
+| | ng | publishe\|
+| | | r |
+| | | uses to |
+| | | authenti\|
+| | | cate |
+| | | to Data |
+| | | Router |
++-------------+----+----------+
+| publisher_i | st\| Publishe\|
+| d | ri\| r |
+| | ng | id in |
+| | | Data |
+| | | Router |
++-------------+----+----------+
+
+Here’s an example of the minimal JSON that must be provided as an input:
+
+.. code:: json
+
+ {
+ "type": "data_router",
+ "dmaap_info": {
+ "publish_url": "https://we-are-data-router.us/feed/xyz"
+ }
+ }
+
+Subscriber
+~~~~~~~~~~
+
+Here’s an example of what the JSON object connection for data router
+subscriber looks like:
+
+.. code:: json
+
+ {
+ "type": "data_router",
+ "dmaap_info": {
+ "location": "mtc00",
+ "delivery_url": "https://my-subscriber-app.dcae:8080/target-path",
+ "username": "some-user",
+ "password": "some-password",
+ "subscriber_id": "789012"
+ }
+ }
+
+At the top-level:
+
++-------------+----+----------------+
+| Property | Ty\| Descript\ |
+| Name | pe | ion |
++=============+====+================+
+| type | st\| *Require\ |
+| | ri\| d |
+| | ng | as \ |
+| | | input*. |
+| | | Must be |
+| | | ``data_router``|
+| | | for data |
+| | | router |
+| | | feeds |
++-------------+----+----------------+
+| dmaap_info | JS\| *Require\ |
+| | ON | d \ |
+| | ob\| as \ |
+| | je\| input*. |
+| | ct | Contains |
+| | | the |
+| | | topic |
+| | | connecti\ |
+| | | on |
+| | | details |
++-------------+----+----------------+
+
+The ``dmaap_info`` object contains:
+
++--------------+----+----------+
+| Property | Ty\| Descript\|
+| Name | pe | ion |
++==============+====+==========+
+| location | st\| DCAE |
+| | ri\| location |
+| | ng | for the |
+| | | publishe\|
+| | | r, |
+| | | used to |
+| | | set up |
+| | | routing |
++--------------+----+----------+
+| delivery_ur\ | st\| URL to |
+| l | ri\| which |
+| | ng | the Data |
+| | | Router |
+| | | should |
+| | | deliver |
+| | | files |
++--------------+----+----------+
+| username | st\| Username |
+| | ri\| Data |
+| | ng | Router |
+| | | uses to |
+| | | authenti\|
+| | | cate |
+| | | to the |
+| | | subscrib\|
+| | | er |
+| | | when |
+| | | deliveri\|
+| | | ng |
+| | | files |
++--------------+----+----------+
+| password | st\| Password |
+| | ri\| Data |
+| | ng | Router |
+| | | uses to |
+| | | authenti\|
+| | | cate |
+| | | to the |
+| | | subscrib\|
+| | | er |
+| | | when |
+| | | deliveri\|
+| | | ng |
+| | | files |
++--------------+----+----------+
+| subscriber_i\| st | Subscrib\|
+| d | ri | er |
+| | ng | id in |
+| | | Data |
+| | | Router |
++--------------+----+----------+
+
+Here’s an example of the minimal JSON that must be provided as an input:
+
+.. code:: json
+
+ {
+ "type": "data_router",
+ "dmaap_info": {
+ }
+ }
+
+Developers are recommended to use ``username`` and ``password`` since
+this is the recommended security practice.
+
+Note that the dcae-cli will construct the ``delivery_url`` when
+deploying the component since this can only be known at deployment time.
diff --git a/docs/sections/components/dcae-cli/quickstart.rst b/docs/sections/components/dcae-cli/quickstart.rst new file mode 100755 index 00000000..60370330 --- /dev/null +++ b/docs/sections/components/dcae-cli/quickstart.rst @@ -0,0 +1,122 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+.. _quickstart:
+
+Quickstart
+==========
+
+The ``dcae-cli`` is a Python command-line tool built to aide component
+developers with the development and testing of their micro-service
+component for the DCAE platform. It will help developers do functional
+and integration testing of their components locally and on remote
+environments as simple as possible.
+
+The tool requires the component developers to create a valid component
+specification for their component which is used by the tool. This same
+component specification will be published in the :any:`onboarding catalog <glossary-onboarding-catalog>` at the end of development
+and testing.
+
+.. The git repository can be found `here <ONAP%20LINK%20TBD>`__
+
+Pre-requisite
+-------------
+
+For Docker
+~~~~~~~~~~
+
+There are two options for development with Docker: developing locally on
+your machine which requires Docker to be installed and developing
+remotely by deploying onto remote infrastructure.
+
+For local development
+^^^^^^^^^^^^^^^^^^^^^
+
+- You must install `Docker
+ engine <https://docs.docker.com/engine/installation/>`__ locally on
+ your machine.
+- You must know the *external ip* of where the Docker engine is
+ running. The external ip is needed so that service discovery will
+ wire up correctly.
+
+ - For OSX users, this means making sure the VirtualBox VM that is
+ running your Docker engine has a bridged adapter and getting the
+ ip of that adapter.
+
+For remote development
+^^^^^^^^^^^^^^^^^^^^^^
+
+You need access to a remote host with Docker engine installed with
+remote API access. You must have the associated connection information:
+domain name or IP and port (should be either 2375 or 2376). This
+information should be set in :any:`an active profile <setting-profile>`.
+
+For CDAP
+~~~~~~~~
+
+TBD
+
+Install
+-------
+
+::
+
+ pip install --extra-index-url https://YOUR_NEXUS_PYPI_SERVER/simple dcae-cli
+
+To do an upgrade, use the ``--upgrade`` flag.
+
+Configuration
+~~~~~~~~~~~~~
+
+When you run the tool for the first time, the tool will create a
+`configuration
+directory <http://click.pocoo.org/5/api/#click.get_app_dir>`__ and
+generate a configuration file.
+
+Configuration is first sourced from an remote server that the platform
+team manages. This is overlaid with configuration details that you will
+be prompted to input particularly your user id.
+
+``--reinit``
+^^^^^^^^^^^^
+
+Configuration can be re-initialized or reset. There is a ``--reinit``
+flag that is to be used to re-initialize your configuration and your
+environment profiles.
+
+To re-initialize:
+
+::
+
+ $ dcae_cli --reinit
+
+Verify
+~~~~~~
+
+Verify that its installed:
+
+::
+
+ $ dcae_cli --help
+ Usage: dcae_cli [OPTIONS] COMMAND [ARGS]...
+
+ Options:
+ -v, --verbose Prints INFO-level logs to screen.
+ --reinit Re-initialize dcae-cli configuration
+ --version Show the version and exit.
+ --help Show this message and exit.
+
+ Commands:
+ catalog
+ component
+ data_format
+ profiles
+
+Version
+-------
+
+You can verify the version of the dcae-cli with the following command:
+
+::
+
+ $ dcae_cli --version
diff --git a/docs/sections/components/dcae-cli/walkthrough.rst b/docs/sections/components/dcae-cli/walkthrough.rst new file mode 100755 index 00000000..d33c35fb --- /dev/null +++ b/docs/sections/components/dcae-cli/walkthrough.rst @@ -0,0 +1,565 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+.. _walkthrough:
+
+Walk-through
+============
+
+The goal of this quickstart is to provide an overview of the
+functionalities of the ``dcae-cli`` and walk you through the
+capabilities:
+
+- `Adding data formats <#adding-data-formats>`__
+- `Adding component <#adding-component>`__
+- `Setting profile <#setting-profile>`__
+- `Development and testing <#development-and-testing>`__
+- `Publishing component <#publishing-component>`__
+- `Shared catalog <#shared-catalog>`__
+
+.. This walk-through uses example projects: COMMENTED OUT FOR NOW TBD
+..
+.. - `laika <ONAP%20URL%20TBD>`__
+.. - `CDAP examples <ONAP%20URL%20TBD>`__
+
+.. _adding-data-formats:
+
+Adding data formats
+-------------------
+
+``data_format`` is the sub-command that is used to execute operations
+that manage `data formats <../data-formats.md>`__.
+
+::
+
+ $ dcae_cli data_format --help
+ Usage: dcae_cli data_format [OPTIONS] COMMAND [ARGS]...
+
+ Options:
+ --help Show this message and exit.
+
+ Commands:
+ add Tracks a data format file SPECIFICATION...
+ generate Generates a data format file from JSON input examples
+ list Lists all your data formats
+ publish Publishes data format to make publicly...
+ show Provides more information about FORMAT
+
+Your data format must be in the catalog in order to use in the component
+specification. Check the catalog using the ``data_format list``
+sub-command:
+
+::
+
+ $ dcae_cli data_format list
+
+ Data formats for mh677g
+ +------+---------+-------------+--------+----------+
+ | Name | Version | Description | Status | Modified |
+ +------+---------+-------------+--------+----------+
+ | | | | | |
+ +------+---------+-------------+--------+----------+
+
+The fields ``name``, ``version``, ``description`` are referenced from
+the data format JSON from the ``self`` JSON.
+
+There are no data formats so you must add the data formats that your
+component specification references. Use the ``data_format add``
+sub-command:
+
+Here’s an example command:
+
+::
+
+ dcae_cli data_format add health.json
+
+Verify that it was added:
+
+::
+
+ $ dcae_cli data_format list
+
+ Data formats for mh677g
+ +-------------------------------+---------+-------------------------------------------+--------+----------------------------+
+ | Name | Version | Description | Status | Modified |
+ +-------------------------------+---------+-------------------------------------------+--------+----------------------------+
+ | sandbox.platform.laika.health | 0.1.0 | Data format used for the /health endpoint | staged | 2017-05-23 04:02:38.952799 |
+ +-------------------------------+---------+-------------------------------------------+--------+----------------------------+
+
+Go ahead and add other referenced data formats.
+
+If you have JSON input you can generate a data format like this:
+
+::
+
+ $ dcae_cli data_format --keywords generate myname:1.0.0 myjsoninputfile
+
+where ``myname`` is the name of your data format, ``1.0.0`` is an
+example version, and ``myjsoninputfile`` is an example JSON input file
+(a directory of input JSON files can also be provided). The
+``--keywords`` option adds additional data attributes that can be
+completed to provide a more detailed data characterization. In any event
+the output should be reviewed for accuracy. The data format is written
+to stdout.
+
+.. _adding-component:
+
+Adding component
+----------------
+
+``component`` is the sub-command that is used to work with operations
+for components:
+
+::
+
+ $ dcae_cli component --help
+ Usage: dcae_cli component [OPTIONS] COMMAND [ARGS]...
+
+ Options:
+ --help Show this message and exit.
+
+ Commands:
+ add
+ dev Set up component in development for...
+ list Lists components in the public catalog.
+ publish Pushes COMPONENT to the public catalog
+ run Runs the latest version of COMPONENT.
+ show Provides more information about COMPONENT
+ undeploy Undeploys the latest version of COMPONENT.
+
+Your component must be accessible from the catalog in order for it to be
+used. Check the catalog using the ``component list`` sub-command:
+
+::
+
+ $ dcae_cli component list
+ Active profile: solutioning
+
+ +------+---------+------+-------------+--------+----------+-----------+
+ | Name | Version | Type | Description | Status | Modified | #Deployed |
+ +------+---------+------+-------------+--------+----------+-----------+
+ | | | | | | | |
+ +------+---------+------+-------------+--------+----------+-----------+
+
+ Use the "--deployed" option to see more details on deployments
+
+The fields ``name``, ``version``, ``type``, ``description`` are
+referenced from the component specification’s ``self`` JSON.
+
+There are no components so you must add your component. Use the
+``component add`` sub-command. The command is the same for docker and
+cdap components:
+
+::
+
+ $ dcae_cli component add --help
+ Usage: dcae_cli component add [OPTIONS] SPECIFICATION
+
+ Options:
+ --update Updates a locally added component if it has not been already
+ pushed
+ --help Show this message and exit.
+
+*Note* use the ``--update`` flag to replace existing staged instances.
+
+The ``component dev`` sub-command can be useful in validating and
+experimenting when crafting your component specification. See details
+about ``dev`` under `Development and
+testing <#development-and-testing>`__.
+
+Once we add the components laika and helloworld, let’s verify that they
+got added ok:
+
+::
+
+ $ dcae_cli component list
+ Active profile: solutioning
+
+ +-------------------------+---------+--------+---------------------------------------------------------------+--------+----------------------------+-----------+
+ | Name | Version | Type | Description | Status | Modified | #Deployed |
+ +-------------------------+---------+--------+---------------------------------------------------------------+--------+----------------------------+-----------+
+ | cdap.helloworld.endnode | 0.8.0 | cdap | cdap test component | staged | 2017-05-23 04:14:35.588075 | 0 |
+ | sandbox.platform.laika | 0.5.0 | docker | Web service used as a stand-alone test DCAE service compone.. | staged | 2017-05-23 04:07:44.065610 | 0 |
+ +-------------------------+---------+--------+---------------------------------------------------------------+--------+----------------------------+-----------+
+
+ Use the "--deployed" option to see more details on deployments
+
+.. _setting-profile:
+
+Setting profile
+---------------
+
+``profile`` is the sub-command that is used to manage profiles. These
+profiles contain environment variables used to connect to different
+environments. This is used in the running and deployment of your
+component using the ``dcae_cli component run`` command. The ``dcae-cli``
+ships with profiles for ``solutioning`` and ``rework``.
+
+::
+
+ $ dcae_cli profiles --help
+ Usage: dcae_cli profiles [OPTIONS] COMMAND [ARGS]...
+
+ Options:
+ --help Show this message and exit.
+
+ Commands:
+ activate Sets profile NAME as the active profile
+ create Creates a new profile NAME initialized with...
+ delete Deletes profile NAME
+ list Lists available profiles
+ set Updates profile NAME such that KEY=VALUE
+ show Prints the profile dictionary
+
+To see what variables a profile contains, you can use the ``show``
+command, as in ``dcae_cli profiles show PROFILE_NAME``
+
+Use the ``create`` sub-command to create your own profile and assign new
+values using the ``set`` command. Afterwards you will need to
+``activate`` the profile you wish to use. First take a look at which
+profile is active:
+
+::
+
+ $ dcae_cli profiles list
+ rework
+ * solutioning
+
+The active profile is ``solutioning`` so to activate *rework* to use
+``rework``:
+
+::
+
+ $ dcae_cli profiles activate rework
+
+Check
+
+::
+
+ $ dcae_cli profiles list
+ * rework
+ solutioning
+
+.. _development-and-testing:
+
+Development and testing
+-----------------------
+
+The following operations under the sub-command ``component`` are aimed
+to help developers with testing:
+
+- ``run``
+- ``undeploy``
+- ``dev``
+
+``run``
+~~~~~~~
+
+The ``run`` operation is to be used for running your application in its
+container remotely on the activated environment. Docker containers have
+the additional option to run locally on your development machine.
+
+In order to run your application, you must have added your data formats
+and your component to your catalog.
+
+Let’s verify that your component is in the catalog:
+
+::
+
+ $ dcae_cli component list
+ Active profile: solutioning
+
+ +-------------------------+---------+--------+---------------------------------------------------------------+--------+----------------------------+-----------+
+ | Name | Version | Type | Description | Status | Modified | #Deployed |
+ +-------------------------+---------+--------+---------------------------------------------------------------+--------+----------------------------+-----------+
+ | cdap.helloworld.endnode | 0.8.0 | cdap | cdap test component | staged | 2017-05-23 04:14:35.588075 | 0 |
+ | sandbox.platform.laika | 0.5.0 | docker | Web service used as a stand-alone test DCAE service compone.. | staged | 2017-05-23 04:07:44.065610 | 0 |
+ +-------------------------+---------+--------+---------------------------------------------------------------+--------+----------------------------+-----------+
+
+ Use the "--deployed" option to see more details on deployments
+
+Docker
+^^^^^^
+
+**NOTE** Make sure your Docker image has been uploaded to the shared
+registry.
+
+For Docker containers, you can run either attached or unattached.
+Attached means that the dcae-cli tool will launch the container and not
+terminate. The dcae-cli while attached will stream in the logs of the
+Docker container. Doing a Ctrl-C will terminate the run session which
+means undeploy your container and force a clean up automatically.
+
+Running unattached means simply deploy your container. You will need to
+execute ``undeploy`` when you are done testing. #### CDAP
+
+**NOTE** Make sure your CDAP jar has been uploaded to Nexus.
+
+TODO
+
+``undeploy``
+~~~~~~~~~~~~
+
+The ``undeploy`` operation is to be used to undeploy any instances of a
+specified component, version that you have deployed. This includes
+cleaning up of configuration.
+
+Let’s undeploy ``sandbox.platform.laika`` that was deployed from the
+previous section:
+
+::
+
+ $ dcae_cli component undeploy sandbox.platform.laika:0.5.0
+ DCAE.Undeploy | WARNING | Undeploying components: 1
+ DCAE.Undeploy | WARNING | Undeployed components: 1
+
+.. _walkthrough-dev:
+
+``dev``
+~~~~~~~
+
+The ``dev`` operation is a convenient operation that can be useful for
+the development and testing of your component. It can be used to:
+
+- Help validate your experimental component specification before
+ uploading to the catalog
+- Generate the application configuration from the component
+ specification and make it available in a test environment. This
+ allows you to view your resulting configuration for local development
+ and to help debug potential related issues.
+
+Let’s say you have a component specification called
+``component-spec.json``:
+
+::
+
+ $ dcae_cli component dev component-spec.json
+ Ready for component development
+
+ Setup these environment varibles. Run "source env_solutioning":
+
+ export DOCKER_HOST=SOME_DOCKER_HOST:2376
+ export SERVICE_CHECK_INTERVAL=15s
+ export CONFIG_BINDING_SERVICE=config_binding_service
+ export HOSTNAME=mh677g.95740959-63d2-492a-b964-62a6dce2591d.0-6-0.sandbox-platform-laika
+ export CONSUL_HOST=SOME_CONSUL_HOST
+ export CDAP_BROKER=cdap_broker
+ export SERVICE_NAME=mh677g.95740959-63d2-492a-b964-62a6dce2591d.0-6-0.sandbox-platform-laika
+ export SERVICE_CHECK_TIMEOUT=1s
+ export SERVICE_CHECK_HTTP=/health
+
+ Press any key to stop and to clean up
+
+Your application configuration is now available under the name
+``mh677g.95740959-63d2-492a-b964-62a6dce2591d.0-6-0.sandbox-platform-laika``.
+
+To view the resulting configuration, you can ``curl`` a request to the
+config binding service or programmatically fetch your configuration
+within your application.
+
+You need to first query Consul to get the ip and port of config binding
+service:
+
+::
+
+ curl http://$CONSUL_HOST:8500/v1/catalog/service/$CONFIG_BINDING_SERVICE
+ [
+ {
+ "ID": "983d5c94-c508-4a8a-9be3-5912bd09786b",
+ "Node": "realsolcnsl00",
+ "Address": "10.226.1.22",
+ "TaggedAddresses": {
+ "lan": "10.226.1.22",
+ "wan": "10.226.1.22"
+ },
+ "NodeMeta": {},
+ "ServiceID": "5f371f295c90:config_binding_service:10000",
+ "ServiceName": "config_binding_service",
+ "ServiceTags": [],
+ "ServiceAddress": "XXXX",
+ "ServicePort": 32770,
+ "ServiceEnableTagOverride": false,
+ "CreateIndex": 487,
+ "ModifyIndex": 487
+ }
+ ]
+
+.. _dmaap-testing:
+
+DMaaP testing
+~~~~~~~~~~~~~
+
+Currently, the dcae-cli does not have the capability of provisioning
+topics. In order to do testing with message router topics or with data
+router feeds, the developer must provision the topic or the feed
+manually and provide the connection details in the form of a JSON in a
+file to the dcae-cli. This file is to be passed in when using the
+``run`` and ``dev`` commands with the option ``--dmaap-file``.
+
+The structure of the DMaaP JSON is an object of config keys to matching
+topic or feed connection details. Config keys are the ``config_key``
+values specified in your component specification streams section where
+the streams must be type message router or data router. Information
+about the associated connection details can be found on `this
+page <dmaap-connection-objects.md>`__. Please check it out.
+
+For example, if you have a component specification that has the
+following streams entry:
+
+.. code:: json
+
+ "streams": {
+ "publishes": [{
+ "format": "ves",
+ "version": "1.0.0",
+ "type": "message router",
+ "config_key": "ves_connection"
+ }]
+ }
+
+Then to deploy and to run your component, you must use the
+``--dmaap-file`` command and pass in a JSON that looks like:
+
+.. code:: json
+
+ {
+ "ves_connection": {
+ "type": "message_router",
+ "dmaap_info": {
+ "topic_url": "https://we-are-message-router.us:3905/events/some-topic"
+ }
+ }
+ }
+
+The provided DMaaP JSON is used to simulate the output of provisioning
+and will be used to merge with the generated application configuration
+at runtime.
+
+Your final application config will look like:
+
+.. code:: json
+
+ {
+ "streams_publishes": {
+ "ves_connection": {
+ "type": "message_router",
+ "dmaap_info": {
+ "topic_url": "https://we-are-message-router.us:3905/events/some-topic"
+ }
+ }
+ }
+ }
+
+Data router subscribers
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Note for data router subscriber testing, you will need the delivery url
+in order to provision the subscriber to the feed. This is constructed at
+deployment time and will be provided by the dcae-cli after you deploy
+your component. The delivery urls will be displayed to the screen:
+
+::
+
+ DCAE.Run | WARNING | Your component is a data router subscriber. Here are the delivery urls:
+
+ some-sub-dr: http://SOME_IP:32838/identity
+
+*Sourced at deployment* testing
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Components may have configuration parameters whose values are to be
+sourced at deployment time. For example, there are components whose
+configuration parameters are to come from DTI events which are only
+available when the component is getting deployed. These configuration
+parameters must be setup correctly in the `component
+specification <http://localhost:8000/components/component-specification/docker-specification/#configuration-parameters>`__
+by setting the property ``sourced_at_deployment`` to ``true`` for each
+and every parameter that is expected to come in at deployment time.
+
+Once your component specification has been updated correctly, you must
+use the ``--inputs-file`` command-line argument when running the
+commands ``dev`` or ``run`` with your component. This is to simulate
+providing the dynamic, deployment time values for those parameters
+marked as ``sourced_at_deployment``.
+
+For example, if your component specification has the following
+configuration parameters:
+
+::
+
+ "parameters": [{
+ "name": "vnf-ip",
+ "value": "",
+ "sourced_at_deployment": true
+ },
+ {
+ "name": "static-param",
+ "value": 5
+ }]
+
+You would have to pass in an inputs file that looks like:
+
+::
+
+ {
+ "vnf-ip": "10.100.1.100"
+ }
+
+Your application configuration would look like:
+
+::
+
+ {
+ "vnf-ip": "10.100.1.100",
+ "static-param": 5
+ }
+
+Publishing component
+--------------------
+
+Once components have their component specifications crafted and
+validated and have been tested, components should be published in the
+shared onboarding catalog using the ``publish`` sub-command for both
+data formats and components. You must publish all data formats of a
+component before publishing a component.
+
+Publishing will change the status of a component, be made accessible for
+other developers to use, and will generate the associated TOSCA models
+for use in designing of compositions.
+
+::
+
+ dcae_cli component publish sandbox.platform.laika:0.5.0
+
+Shared catalog
+--------------
+
+``catalog`` is the sub-command used to access and to browse the shared
+onboarding catalog to view components and data formats that have been
+published and that are being worked on. Components and data formats have
+two statuses ``staged`` and ``published``.
+
+Staged means that the resource has been simply added and is under
+development. It is to be used only by the owner. Published means that
+the resource has been fully developed and tested and is ready to be
+shared.
+
+Published components can be deployed by non-owners and published data
+formats can be used in component specifications of non-owners.
+
+There are two available operations:
+
+::
+
+ $ dcae_cli catalog --help
+ Usage: dcae_cli catalog [OPTIONS] COMMAND [ARGS]...
+
+ Options:
+ --help Show this message and exit.
+
+ Commands:
+ list
+ show
+
+Staged components can be viewed under the ``list`` operation using the
+``--expanded`` flag.
diff --git a/docs/sections/components/glossary.rst b/docs/sections/components/glossary.rst new file mode 100644 index 00000000..6c048cf3 --- /dev/null +++ b/docs/sections/components/glossary.rst @@ -0,0 +1,37 @@ + +.. http://creativecommons.org/licenses/by/4.0 + +.. _glossary: + +Glossary +======== + +.. _glossary-asdc: + +ASDC +---- + +The ECOMP resource catalog. We assume the existince and usage of ASDC, though for near term testing purposes, we use a mock version of the catalog. +We assume all DCAE artifacts: components like collectors, CDAP applications, data formats (see below), etc, are onboarded and searchable in the catalog. +Further, we assume that every catalog artifact has a *UUID*, a globally unique identifier that identifies that artifact. + +.. _glossary-component: + +Component +--------- + +Refers to a DCAE service component which is a single micro-service that is written to be run by the DCAE platform and to be composeable to form a DCAE service. + +.. _glossary-data-format: + +Data format +----------- + +Artifact that describes a data structure and schema. + +.. _glossary-onboarding-catalog: + +Onboarding catalog +------------------ + +The onboarding catalog is the intermediary store for project details, component specifications and data formats. This is to be used by component developers through the `dcae-cli` to browse and to launch existing components and push their own component artifacts in preparation to move their component to the ASDC resource catalog. diff --git a/docs/sections/components/intro.rst b/docs/sections/components/intro.rst new file mode 100755 index 00000000..8a31e844 --- /dev/null +++ b/docs/sections/components/intro.rst @@ -0,0 +1,97 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+.. _intro:
+
+
+Component Developer Overview
+============================
+
+DCAE components are services that provide a specific functionality and
+are written to be composable with other DCAE service components. The
+DCAE platform is responsible for running and managing DCAE service
+components reliably.
+
+Currently, the DCAE platform supports two types of components, CDAP
+applications and Docker containers. For each, there are requirements
+that must be met for the component to integrate into the DCAE platform
+(see :doc:`CDAP <component-type-cdap>` and :doc:`Docker <component-type-docker>`.
+
+Components requires one or more data formats.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Components are software applications that do some function. Components
+don’t run independently, they depend upon other components. A
+component’s function could require connecting to other components to
+fulfill that function. A component could also be providing its function
+as a service through an interface for other components to use.
+
+Components cannot connect to or be connected with any other component.
+The upstream and downstream components must *speak* the same vocabulary
+or *data format*. The output of an one component must match another
+component’s input. This is necessary for components to function
+correctly and without errors.
+
+The platform requires data formats to ensure that a component will be
+run with other *compatible* components.
+
+Data formats can and should be shared by multiple components.
+
+Each Component requires a component specification.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The component specification is a JSON artifact that fully specifies the
+component, it’s interfaces, and configuration. It’s standardized for
+CDAP and Docker applications and is validated using a :any:`JSON
+schema <dcae-component-schema>`.
+
+The component specification fully specifies all the configuration
+parameters of the component. This is used by the designer and by policy
+(future) to configure the runtime behavior of the component.
+
+The component specification is used to *generate* application
+configuration in a standardized JSON that the platform will make
+available to the component. This application configuration JSON will
+include:
+
+- Parameters that have been assigned values from the component
+ specification, policy, and/or the designer
+- Connection details of downstream components
+
+The component specification is transformed by DCAE tooling (explained
+later) into TOSCA models (one for the component, and in the future, one
+for Policy). The TOSCA models then get transformed into Cloudify
+blueprints.
+
+The component specification is used by:
+
+- dcae_cli tool - to validate it
+- Design Tools - TOSCA models are generated from the component
+ specification so that the component can be used by designers to
+ compose new DCAE services in SDC.
+- Policy (future) - TOSCA models are generated from the component
+ specification so that operations can create policy models used to
+ dynamically configure the component.
+- the runtime platform - The component’s application configuration
+ (JSON) is generated from the component specification and will be
+ provided to the component at runtime.
+
+Onboarding
+----------
+
+Onboarding is a process that ensures that the component is compliant
+with the DCAE platform rules. A command-line tool called :doc:`dcae-cli <dcae-cli/quickstart>` is provided to help with onboarding. The high level summary of the onboarding process is:
+
+1. Defining the :doc:`data formats <data-formats>` if they don’t already
+ exist.
+2. Define the :doc:`component specification <component-specification/common-specification>`. See :doc:`Docker <component-specification/docker-specification>` and :doc:`CDAP <component-specification/cdap-specification>`.
+3. Use the dcae_cli tool to :any:`add the data formats <adding-data-formats>`
+ and :any:`add the component <adding-component>` to
+ the onboarding catalog. This process will validate them as well.
+4. Use the dcae_cli tool to :any:`deploy <development-and-testing>`
+ the component. (The component is deployed to the environment
+ indicated in :any:`profile <setting-profile>`).
+5. Test the component. Also do pairwise-test the component with any
+ other components it connects with.
+6. Publish the component and data formats into the Service Design and
+ Creation (SDC) ‘catalog’. (Currently, this is a manual step).
diff --git a/docs/sections/components/requirements-guidelines.rst b/docs/sections/components/requirements-guidelines.rst new file mode 100644 index 00000000..1f1870eb --- /dev/null +++ b/docs/sections/components/requirements-guidelines.rst @@ -0,0 +1,12 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +Requirements/Guidelines +======================= + +.. toctree:: + :maxdepth: 1 + + ./component-type-cdap.rst + ./component-type-docker.rst + |