From 1fca6acb2918beef86a5a78dc659683830908cd2 Mon Sep 17 00:00:00 2001 From: Ralph Knag Date: Tue, 5 Dec 2017 12:05:57 -0500 Subject: DCAE Controller documentation DCAEGEN2-213 Issue-ID: DCAEGEN2-213 Change-Id: I7f2023b7f88b73eef852eca0bbf9086e14903cd6 Signed-off-by: Ralph Knag --- .../component-specification/cdap-specification.rst | 209 +++ .../common-specification.rst | 1418 ++++++++++++++++++++ .../component-specification.rst | 16 + .../component-specification/configuration-grid.rst | 315 +++++ .../docker-specification.rst | 454 +++++++ .../generated-configuration.rst | 114 ++ .../component-specification/start-here.rst | 1 + .../component-specification/streams-grid.rst | 149 ++ 8 files changed, 2676 insertions(+) create mode 100755 docs/sections/components/component-specification/cdap-specification.rst create mode 100755 docs/sections/components/component-specification/common-specification.rst create mode 100644 docs/sections/components/component-specification/component-specification.rst create mode 100755 docs/sections/components/component-specification/configuration-grid.rst create mode 100755 docs/sections/components/component-specification/docker-specification.rst create mode 100755 docs/sections/components/component-specification/generated-configuration.rst create mode 100755 docs/sections/components/component-specification/start-here.rst create mode 100755 docs/sections/components/component-specification/streams-grid.rst (limited to 'docs/sections/components/component-specification') diff --git a/docs/sections/components/component-specification/cdap-specification.rst b/docs/sections/components/component-specification/cdap-specification.rst new file mode 100755 index 00000000..022b46d7 --- /dev/null +++ b/docs/sections/components/component-specification/cdap-specification.rst @@ -0,0 +1,209 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +.. _cdap-specification: + +Component specification (CDAP) +============================== + +The CDAP component specification contains the following groups of +information. Many of these are common to both CDAP and Docker components +and are therefore described in the common specification. + +- :any:`Metadata ` +- :any:`Interfaces ` including the + associated :any:`Data Formats ` +- :any:`Parameters ` - for specifying parameters in your + AppConfig, AppPreferences, and ProgramPreferences to the Designer and + Policy. This of course is CDAP-specific and is described below. +- :any:`Auxiliary Details ` +- :any:`List of artifacts ` + +Current Limitations and TODOs +----------------------------- + +- The integration of DMD is likely to significantly change the + :any:`Interfaces ` section in this specification, see + :any:`DMaaP abstraction `. + +.. _parameters: + +Parameters +---------- + +There is a ``parameters`` section in your component specification. This +section contains three optional keys: `app_config <#appconfig>`__, +`app_preferences <#apppreferences>`__, and +`propram_preferences <#programpreferences>`__: + +:: + + "parameters" : { + "app_config" : [ ...], + "app_preferences" : [ ...], + "program_preferences" : [...] + // any additional keys are ignored + } + +- Each section details the parameters that are a part of each of these + CDAP constructs (see below). +- All such parameters will be exposed to the designer and to policy for + override. +- These parameters should have default values specified by the + component developer where necessary, i.e., parameters that *must* + come from the designer/policy should not have defaults. +- All of these keys are optional because not every CDAP application + uses preferences and not every application uses the AppConfig. + However, you should specify at least one, or else your application + will have no parameters exposed to policy or to the DCAE designer, + which means it would be non-configurable. +- Despite the AppConfig being optional to *specify* in the case that + you have no parameters in your AppConfig, it is *required for + processing* in your application. That is because the DCAE platform + will place important information into your AppConfig as discussed + below. + +Parameter +~~~~~~~~~ + +The following CDAP specific definitions use ``param1`` to refer to the +common parameter layout in +:any:`Parameter ` + +AppConfig +~~~~~~~~~ + +The ``app_config`` key refers to the `CDAP AppConfig `_. +It is expected to be a JSON: + +:: + + "app_config" : [ // list of JSON + param1, // common parameter layout + ... + ] + +Unfortunately, at the time of writing, the AppConfig is a Java map of +``string:string``, which means you cannot have more complex structures +(than string) as any value in your AppConfig. However, there is a way to +bypass this constraint: you can pass a JSON by encoding the JSON as a +string. E.g., the ``json.dumps()`` and it’s converse ``loads`` methods +in Python: + +:: + + >>> import json + >>> json.dumps({"foo" : "bar"}) # This is a real JSON + '{"foo": "bar"}' # It is now a string: pass this in as your parameter value + >>> json.loads('{"foo": "bar"}') # Do the equivelent of this in your application + {u'foo': u'bar'} # ...and you'll get back a real JSON + >>> + +The final AppConfig (after the designer and policy override parameter +values) is passed into CDAP’s AppConfig API when starting the +application. + + +AppPreferences +~~~~~~~~~~~~~~ + +In addition to the CDAP AppConfig, the platform supports `Application Preferences `_. +The format of the ``app_preferences`` value is the same as the above: + +:: + + "app_preferences" : [ // list of JSON + param1, // common parameter layout + ... + ] + +The final Application Preferences JSON (after the designer and policy +override parameter values) is passed into CDAP’s Preferences API when +starting your application. + + +ProgramPreferences +~~~~~~~~~~~~~~~~~~ + +Preferences can also be specified `per program `_ +in CDAP. This key’s value is a list of JSON with the following format: + +:: + + "program_preferences" : [ // note: this is a list of JSON + { + "program_id" : "program name 1", // the name of this CDAP program + "program_type" : "e.g., flows", // "must be one of flows, mapreduce, schedules, spark, workflows, workers, or services", + "program_pref" : [ // list of JSON + param1, // common parameter layout + ... + ] + }, + // repeat for each program that receives a program_preferences JSON + ] + +Each ``program_pref`` JSON is passed into the CDAP API as the preference +for ``program_id``. + +NOTE: for CDAP, this section is very likely to change when DMD is +available. The *future* vision, as per :any:`DMaaP intentionally abstracted ` is +that you would publish your data as a series of files on HDFS, and DMD +will pick them up and send them to the appropriate DMaaP feeds or +directly when needed. + +.. _auxiliary-details: + +Auxiliary Details +----------------- + +*auxiliary* contains details about CDAP specific parameters. + ++----------------------+----------------------+----------------------+ +| Property Name | Type | Description | ++======================+======================+======================+ +| streamname | string | *Required*. | ++----------------------+----------------------+----------------------+ +| artifact_name | string | | ++----------------------+----------------------+----------------------+ +| artifact_version | string | the version of your | +| | | CDAP JAR artifact | ++----------------------+----------------------+----------------------+ +| namespace | string | the CDAP namespace | +| | | to deploy into, | +| | | default is ‘default’ | ++----------------------+----------------------+----------------------+ +| programs | array | contains each CDAP | +| | | entity represented | +| | | in the artifact | ++----------------------+----------------------+----------------------+ +| program_type | string | CDAP entity (eg | +| | | “flows”) | ++----------------------+----------------------+----------------------+ +| program_id | string | name of CDAP entity | +| | | (eg “WhoFlow”) | ++----------------------+----------------------+----------------------+ + +Example: + +.. code:: json + + "auxiliary": { + "streamname" : "who", + "artifact_name" : "HelloWorld", + "artifact_version" : "3.4.3", + "namespace" : "hw", + "programs" : [ + {"program_type" : "flows", "program_id" : "WhoFlow"}, + {"program_type" : "services", "program_id" : "Greeting"}, + ... + ], + } + +The ``programs`` key is identical to the ``program_preferences`` key +discussed `above <#programpreferences>`__ except: + +- each JSON in the list does not contain ``program_pref`` +- this is required! You must include all of your programs in this, as + it is used to start each program as well as for DCAE to perform + periodic healthchecks on your application. Don’t forget about your + services; they are programs too. diff --git a/docs/sections/components/component-specification/common-specification.rst b/docs/sections/components/component-specification/common-specification.rst new file mode 100755 index 00000000..a29610f5 --- /dev/null +++ b/docs/sections/components/component-specification/common-specification.rst @@ -0,0 +1,1418 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +.. _common-specification: + +Common Elements of the Component Specification +============================================== + +This page describes the component specification (JSON) sections that are +common to both Docker and CDAP components. Differences for each are +pointed out below. Elements that are very different, and described in +the CDAP or Docker specific pages. + +.. _metadata: + +Component Metadata +------------------ + +Metadata refers to the properties found under the ``self`` JSON. This +group of properties is used to uniquely identify this component +specification and identify the component that this specification is used +to capture. + +Example: + +:: + + "self": { + "version": "1.0.0", + "name": "asimov.component.kpi_anomaly", + "description": "Classifies VNF KPI data as anomalous", + "component_type": "docker" + }, + +``self`` Schema: + ++-------------+--------+----------------+ +| Property | Type | Description | +| Name | | | ++=============+========+================+ +| version | string | *Required*. | +| | | Semantic | +| | | version | +| | | for this | +| | | specification | ++-------------+--------+----------------+ +| name | string | *Required*. | +| | | Full | +| | | name of | +| | | this | +| | | component | +| | | which is | +| | | also | +| | | used as | +| | | this | +| | | component's | +| | | catalog | +| | | id. | ++-------------+--------+----------------+ +| description | string | *Required* | +| | | Human-readable | +| | | text | +| | | describing | +| | | the | +| | | component | +| | | and the | +| | | components | +| | | functional | +| | | purpose. | ++-------------+--------+----------------+ +| component_t\| string | *Required* | +| ype | | Identify | +| | | what | +| | | containe\ | +| | | rization | +| | | technolo\ | +| | | gy | +| | | this | +| | | componen\ | +| | | t | +| | | uses: | +| | | *docker* | +| | | or | +| | | *cdap*. | +| | | | ++-------------+--------+----------------+ + +.. _interfaces: + +Interfaces +---------- + +Interfaces are the JSON objects found under the ``streams`` key and the +``services`` key. These are used to describe the interfaces that the +component uses and the interfaces that the component provides. The +description of each interface includes the associated :any:`data +format `. + +Streams +~~~~~~~ + +- The ``streams`` JSON is for specifying data produced for consumption + by other components, and the streams expected to subscribe to that is + produced by other components. These are “fire and forget” type + interfaces where the publisher of a stream does not expect or parse a + response from the subscriber. +- The term ``stream`` here is abstract and neither refers to “CDAP + streams” or “DMaaP feeds”. While a stream is very likely a DMaaP + feed, it could be a direct stream of data being routed via HTTP too. + It abstractly refers to a sequence of data leaving a publisher. +- Streams have anonymous publish/subscribe semantics, which decouples + the production of information from its consumption. +- In general, components are not aware of who they are communicating + with. +- Instead, components that are interested in data, subscribe to the + relevant stream; components that generate data publish to the + relevant stream. +- There can be multiple publishers and subscribers to a stream. Streams + are intended for unidirectional, streaming communication. + +Streams interfaces that implement an HTTP endpoint must support POST. + +Streams are split into: + ++-------------+----+----------+ +| Property | Ty\| Descript\| +| Name | pe | ion | ++=============+====+==========+ +| subscribes | JS\| *Require\| +| | ON | d*. | +| | li\| List of | +| | st | all | +| | | availabl\| +| | | e | +| | | stream | +| | | interfac\| +| | | es | +| | | that | +| | | this | +| | | componen\| +| | | t | +| | | has that | +| | | can be | +| | | used for | +| | | subscrib\| +| | | ing | ++-------------+----+----------+ +| publishes | JS\| *Require\| +| | ON | d*. | +| | li\| List of | +| | st | all | +| | | stream | +| | | interfac\| +| | | es | +| | | that | +| | | this | +| | | componen\| +| | | t | +| | | will | +| | | publish | +| | | onto | ++-------------+----+----------+ + +Subscribes +^^^^^^^^^^ + +Example: + +.. code:: json + + "streams": { + "subscribes": [{ + "format": "dcae.vnf.kpi", + "version": "1.0.0", + "route": "/data", // for CDAP this value is not used + "type": "http" + }], + ... + } + +This describes that ``asimov.component.kpi_anomaly`` exposes an HTTP +endpoint called ``/data`` which accepts requests that have the data +format of ``dcae.vnf.kpi`` version ``1.0.0``. + +``subscribes`` Schema: + ++-------------+----+--------------------+ +| Property | Ty\| Descript\ | +| Name | pe | ion | ++=============+====+====================+ +| format | st\| *Require\ | +| | ri\| d*. | +| | ng | Data | +| | | format | +| | | id of | +| | | the data | +| | | format | +| | | that is | +| | | used by | +| | | this | +| | | interfac\ | +| | | e | ++-------------+----+--------------------+ +| version | st\| *Require\ | +| | ri\| d*. | +| | ng | Data | +| | | format | +| | | version | +| | | of the | +| | | data | +| | | format | +| | | that is | +| | | used by | +| | | this | +| | | interfac\ | +| | | e | ++-------------+----+--------------------+ +| route | st\| *Require\ | +| | ri\| d | +| | ng | for HTTP | +| | | and data | +| | | router*. | +| | | The HTTP | +| | | route | +| | | that | +| | | this | +| | | interfac\ | +| | | e | +| | | listens | +| | | on | ++-------------+----+--------------------+ +| config_key | st\| *Require\ | +| | ri\| d \ | +| | ng | for \ | +| | | message_router\ | +| | | and data \ | +| | | router*. | +| | | The HTTP | +| | | route | +| | | that | +| | | this | +| | | interfac\ | +| | | e | +| | | listens | +| | | on | ++-------------+----+--------------------+ +| type | st\| *Require\ | +| | ri\| d*. | +| | ng | Type of | +| | | stream: | +| | | ``http`` | +| | | , | +| | | ``message_router`` | +| | | , | +| | | ``data_router`` | ++-------------+----+--------------------+ + +.. _message-router: + +Message router +'''''''''''''' + +Message router subscribers are http clients rather than http services +and performs a http a ``GET`` call. Thus, message router subscribers +description is structured like message router publishers and requires +``config_key``: + +.. code:: json + + "streams": { + "subscribes": [{ + "format": "dcae.some-format", + "version": "1.0.0", + "config_key": "some_format_handle", + "type": "message router" + }], + ... + } + + +.. _data-router: + +Data router +''''''''''' + +Data router subscribers are http or https services that handle ``PUT`` +requests from data router. Developers must provide the ``route`` or url +path/endpoint that is expected to handle data router requests. This will +be used to construct the delivery url needed to register the subscriber +to the provisioned feed. Developers must also provide a ``config_key`` +because there is dynamic configuration information associated with the +feed that the application will need e.g. username and password. See the +page on :doc:`DMaaP connection objects <../dcae-cli/dmaap-connection-objects>` for more details on +the configuration information. + +Example (not tied to the larger example): + +.. code:: json + + "streams": { + "subscribes": [{ + "config_key": "some-sub-dr", + "format": "sandbox.platform.any", + "route": "/identity", + "type": "data_router", + "version": "0.1.0" + }], + ... + } + +Publishes +^^^^^^^^^ + +Example: + +.. code:: json + + "streams": { + ... + "publishes": [{ + "format": "asimov.format.integerClassification", + "version": "1.0.0", + "config_key": "prediction", + "type": "http" + }] + }, + +This describes that ``asimov.component.kpi_anomaly`` publishes by making +POST requests to streams that support the data format +``asimov.format.integerClassification`` version ``1.0.0``. + +``publishes`` Schema: + ++-------------+----+--------------------+ +| Property | Ty\| Descript\ | +| Name | pe | ion | ++=============+====+====================+ +| format | st\| *Require\ | +| | ri\| d*. | +| | ng | Data | +| | | format | +| | | id of | +| | | the data | +| | | format | +| | | that is | +| | | used by | +| | | this | +| | | interfac\ | +| | | e | ++-------------+----+--------------------+ +| version | st\| *Require\ | +| | ri\| d*. | +| | ng | Data | +| | | format | +| | | version | +| | | of the | +| | | data | +| | | format | +| | | that is | +| | | used by | +| | | this | +| | | interfac\ | +| | | e | ++-------------+----+--------------------+ +| config_key | st\| *Require\ | +| | ri\| d*. | +| | ng | The JSON | +| | | key in | +| | | the | +| | | generate\ | +| | | d | +| | | applicat | +| | | ion | +| | | configur\ | +| | | ation | +| | | that | +| | | will be | +| | | used to | +| | | pass the | +| | | downstre\ | +| | | am | +| | | componen\ | +| | | t’s | +| | | (the | +| | | subscrib\ | +| | | er’s) | +| | | connecti\ | +| | | on | +| | | informat\ | +| | | ion. | ++-------------+----+--------------------+ +| type | st\| *Require\ | +| | ri\| d*. | +| | ng | Type of | +| | | stream: | +| | | ``http`` | +| | | , | +| | | ``message_router`` | +| | | , | +| | | ``data_router`` | ++-------------+----+--------------------+ + +.. message-router-1: + +Message router +'''''''''''''' + +Message router publishers are http clients of DMaap message_router. +Developers must provide a ``config_key`` because there is dynamic +configuration information associated with the feed that the application +will need to receive e.g. topic url, username, password. See the page on +:doc:`DMaaP connection objects <../dcae-cli/dmaap-connection-objects>` for more details on +the configuration information. + +Example (not tied to the larger example): + +.. code:: json + + "streams": { + ... + "publishes": [{ + "config_key": "some-pub-mr", + "format": "sandbox.platform.any", + "type": "message_router", + "version": "0.1.0" + }] + } + +.. data-router-1: + +Data router +''''''''''' + +Data router publishers are http clients that make ``PUT`` requests to +data router. Developers must also provide a ``config_key`` because there +is dynamic configuration information associated with the feed that the +application will need to receive e.g. publish url, username, password. +See the page on :doc:`DMaaP connection objects <../dcae-cli/dmaap-connection-objects>` for more details on +the configuration information. + +Example (not tied to the larger example): + +.. code:: json + + "streams": { + ... + "publishes": [{ + "config_key": "some-pub-dr", + "format": "sandbox.platform.any", + "type": "data_router", + "version": "0.1.0" + }] + } + +Quick Reference +^^^^^^^^^^^^^^^ + +Refer to this :doc:`Quick Reference ` for a +comparison of the Streams ‘Publishes’ and ‘Subscribes’ sections. + +Services +~~~~~~~~ + +- The publish / subscribe model is a very flexible communication + paradigm, but its many-to-many one-way transport is not appropriate + for RPC request / reply interactions, which are often required in a + distributed system. +- Request / reply is done via a Service, which is defined by a pair of + messages: one for the request and one for the reply. + +Services are split into: + ++-------------+----+----------+ +| Property | Ty\| Descript\| +| Name | pe | ion | ++=============+====+==========+ +| calls | JS\| *Require\| +| | ON | d*. | +| | li\| List of | +| | st | all | +| | | service | +| | | interfac\| +| | | es | +| | | that | +| | | this | +| | | componen\| +| | | t | +| | | will | +| | | call | ++-------------+----+----------+ +| provides | JS\| *Require\| +| | ON | d*. | +| | li\| List of | +| | st | all | +| | | service | +| | | interfac\| +| | | es | +| | | that | +| | | this | +| | | componen\| +| | | t | +| | | exposes | +| | | and | +| | | provides | ++-------------+----+----------+ + +Calls +^^^^^ + +The JSON ``services/calls`` is for specifying that the component relies +on an HTTP(S) service—the component sends that service an HTTP request, +and that service responds with an HTTP reply. An example of this is how +string matching (SM) depends on the AAI Broker. SM performs a +synchronous REST call to the AAI broker, providing it the VMNAME of the +VNF, and the AAI Broker responds with additional details about the VNF. +This dependency is expressed via ``services/calls``. In contrast, the +output of string matching (the alerts it computes) is sent directly to +policy as a fire-and-forget interface, so that is an example of a +``stream``. + +Example: + +.. code:: json + + "services": { + "calls": [{ + "config_key": "vnf-db", + "request": { + "format": "dcae.vnf.meta", + "version": "1.0.0" + }, + "response": { + "format": "dcae.vnf.kpi", + "version": "1.0.0" + } + }], + ... + } + +This describes that ``asimov.component.kpi_anomaly`` will make HTTP +calls to a downstream component that accepts requests of data format +``dcae.vnf.meta`` version ``1.0.0`` and is expecting the response to be +``dcae.vnf.kpi`` version ``1.0.0``. + +``calls`` Schema: + ++-------------+----+----------+ +| Property | Ty\| Descript\| +| Name | pe | ion | ++=============+====+==========+ +| request | JS\| *Require\| +| | ON | d*. | +| | ob\| Descript\| +| | je\| ion | +| | ct | of the | +| | | expected | +| | | request | +| | | for this | +| | | downstre\| +| | | am | +| | | interfac\| +| | | e | ++-------------+----+----------+ +| response | JS\| *Require\| +| | ON | d*. | +| | ob\| Descript\| +| | je\| ion | +| | ct | of the | +| | | expected | +| | | response | +| | | for this | +| | | downstre\| +| | | am | +| | | interfac\| +| | | e | ++-------------+----+----------+ +| config_key | st\| *Require\| +| | ri\| d*. | +| | ng | The JSON | +| | | key in | +| | | the | +| | | generate\| +| | | d | +| | | applicat | +| | | ion | +| | | configur\| +| | | ation | +| | | that | +| | | will be | +| | | used to | +| | | pass the | +| | | downstre\| +| | | am | +| | | componen | +| | | t | +| | | connecti\| +| | | on | +| | | informat\| +| | | ion. | ++-------------+----+----------+ + +The JSON object schema for both ``request`` and ``response``: + ++-------------+----+----------+ +| Property | Ty\| Descript\| +| Name | pe | ion | ++=============+====+==========+ +| format | st\| *Require\| +| | ri\| d*. | +| | ng | Data | +| | | format | +| | | id of | +| | | the data | +| | | format | +| | | that is | +| | | used by | +| | | this | +| | | interfac\| +| | | e | ++-------------+----+----------+ +| version | st\| *Require\| +| | ri\| d*. | +| | ng | Data | +| | | format | +| | | version | +| | | of the | +| | | data | +| | | format | +| | | that is | +| | | used by | +| | | this | +| | | interfac\| +| | | e | ++-------------+----+----------+ + +Provides +^^^^^^^^ + +Example: + +.. code:: json + + "services": { + ... + "provides": [{ + "route": "/score-vnf", + "request": { + "format": "dcae.vnf.meta", + "version": "1.0.0" + }, + "response": { + "format": "asimov.format.integerClassification", + "version": "1.0.0" + } + }] + }, + +This describes that ``asimov.component.kpi_anomaly`` provides a service +interface and it is exposed on the ``/score-vnf`` HTTP endpoint. The +endpoint accepts requests that have the data format ``dcae.vnf.meta`` +version ``1.0.0`` and gives back a response of +``asimov.format.integerClassification`` version ``1.0.0``. + +``provides`` Schema for a Docker component: + ++-------------+----+----------+ +| Property | Ty\| Descript\| +| Name | pe | ion | ++=============+====+==========+ +| request | JS\| *Require\| +| | ON | d*. | +| | ob\| Descript\| +| | je\| ion | +| | ct | of the | +| | | expected | +| | | request | +| | | for this | +| | | interfac\| +| | | e | ++-------------+----+----------+ +| response | JS\| *Require\| +| | ON | d*. | +| | ob\| Descript\| +| | je\| ion | +| | ct | of the | +| | | expected | +| | | response | +| | | for this | +| | | interfac\| +| | | e | ++-------------+----+----------+ +| route | st\| *Require\| +| | ri\| d*. | +| | ng | The HTTP | +| | | route | +| | | that | +| | | this | +| | | interfac\| +| | | e | +| | | listens | +| | | on | ++-------------+----+----------+ + +The JSON object schema for both ``request`` and ``response``: + ++-------------+----+----------+ +| Property | Ty\| Descript\| +| Name | pe | ion | ++=============+====+==========+ +| format | st\| *Require\| +| | ri\| d*. | +| | ng | Data | +| | | format | +| | | id of | +| | | the data | +| | | format | +| | | that is | +| | | used by | +| | | this | +| | | interfac\| +| | | e | ++-------------+----+----------+ +| version | st\| *Require\| +| | ri\| d*. | +| | ng | Data | +| | | format | +| | | version | +| | | of the | +| | | data | +| | | format | +| | | that is | +| | | used by | +| | | this | +| | | interfac\| +| | | e | ++-------------+----+----------+ + +Note, for CDAP, there is a slight variation due to the way CDAP exposes +services: + +:: + + "provides":[ // note this is a list of JSON + { + "request":{ ...}, + "response":{ ...}, + "service_name":"name CDAP service", + "service_endpoint":"greet", // E.g the URL is /services/service_name/methods/service_endpoint + "verb":"GET" // GET, PUT, or POST + } + ] + +``provides`` Schema for a CDAP component: + ++-------------+----+-----------+ +| Property | Ty\| Descript\ | +| Name | pe | ion | ++=============+====+===========+ +| request | JS\| *Require\ | +| | ON | d*. | +| | ob\| Descript\ | +| | je\| ion | +| | ct | of the | +| | | expected | +| | | request | +| | | data | +| | | format | +| | | for this | +| | | interfac\ | +| | | e | ++-------------+----+-----------+ +| response | JS\| *Require\ | +| | ON | d*. | +| | ob\| Descript\ | +| | je\| ion | +| | ct | of the | +| | | expected | +| | | response | +| | | for this | +| | | interfac\ | +| | | e | ++-------------+----+-----------+ +| service_nam\| st\| *Require\ | +| e | ri\| d*. | +| | ng | The CDAP | +| | | service | +| | | name (eg | +| | | “Greetin\ | +| | | g”) | ++-------------+----+-----------+ +| service_end | st\| *Require\ | +| point | ri\| d*. | +| | ng | The CDAP | +| | | service | +| | | endpoint | +| | | for this | +| | | service_n\| +| | | ame | +| | | (eg | +| | | “/greet” | +| | | ) | ++-------------+----+-----------+ +| verb | st\| *Require\ | +| | ri\| d*. | +| | ng | ‘GET’, | +| | | ‘PUT’ or | +| | | ‘POST’ | ++-------------+----+-----------+ + +Parameters +---------- + +``parameters`` is where to specify the component’s application +configuration parameters that are not connection information. + ++---------------+------------+----------------------------------+ +| Property Name | Type | Description | ++===============+============+==================================+ +| parameters | JSON array | Each entry is a parameter object | ++---------------+------------+----------------------------------+ + +Parameter object has the following available properties: + ++--------------+----+----------+------+ +| Property | Ty\| Descript\| Defa\| +| Name | pe | ion | ult | ++==============+====+==========+======+ +| name | st\| *Require\| | +| | ri\| d*. | | +| | ng | The | | +| | | property | | +| | | name | | +| | | that | | +| | | will be | | +| | | used as | | +| | | the key | | +| | | in the | | +| | | generate\| | +| | | d | | +| | | config | | ++--------------+----+----------+------+ +| value | an\| *Require\| | +| | y | d*. | | +| | | The | | +| | | default | | +| | | value | | +| | | for the | | +| | | given | | +| | | paramete\| | +| | | r | | ++--------------+----+----------+------+ +| description | st\| *Require\| | +| | ri\| d*. | | +| | ng | Human-re\| | +| | | adable | | +| | | text | | +| | | describi\| | +| | | ng | | +| | | the | | +| | | paramete\| | +| | | r | | +| | | like | | +| | | what its | | +| | | for | | ++--------------+----+----------+------+ +| type | st\| The | | +| | ri\| required | | +| | ng | data | | +| | | type for | | +| | | the | | +| | | paramete\| | +| | | r | | ++--------------+----+----------+------+ +| required | bo\| An | true | +| | ol\| optional | | +| | ea\| key that | | +| | n | declares | | +| | | a | | +| | | paramete\| | +| | | r | | +| | | as | | +| | | required | | +| | | (true) | | +| | | or not | | +| | | (false) | | ++--------------+----+----------+------+ +| constraints | ar\| The | | +| | ra\| optional | | +| | y | list of | | +| | | sequence | | +| | | d | | +| | | constrai\| | +| | | nt | | +| | | clauses | | +| | | for the | | +| | | paramete\| | +| | | r. | | +| | | See | | +| | | below | | ++--------------+----+----------+------+ +| entry_schem\ | st\| The | | +| a | ri\| optional | | +| | ng | key that | | +| | | is used | | +| | | to | | +| | | declare | | +| | | the name | | +| | | of the | | +| | | Datatype | | +| | | definiti\| | +| | | on | | +| | | for | | +| | | entries | | +| | | of set | | +| | | types | | +| | | such as | | +| | | the | | +| | | TOSCA | | +| | | ‘list’ | | +| | | or | | +| | | ‘map’. | | +| | | Only 1 | | +| | | level is | | +| | | supporte\| | +| | | d | | +| | | at this | | +| | | time | | ++--------------+----+----------+------+ +| designer_ed\ | bo\| An | true | +| itable | ol\| optional | | +| | ea\| key that | | +| | n | declares | | +| | | a | | +| | | paramete\| | +| | | r | | +| | | to be | | +| | | editable | | +| | | by | | +| | | designer | | +| | | (true) | | +| | | or not | | +| | | (false) | | ++--------------+----+----------+------+ +| sourced_at_d\| bo\| An | fals\| +| eployment | ol\| optional | e | +| | ea\| key that | | +| | n | declares | | +| | | a | | +| | | paramete\| | +| | | r’s | | +| | | value to | | +| | | be | | +| | | assigned | | +| | | at | | +| | | deployme\| | +| | | nt | | +| | | time | | +| | | (true) | | ++--------------+----+----------+------+ +| policy_edit\ | bo\| An | true | +| able | ol\| optional | | +| | ea\| key that | | +| | n | declares | | +| | | a | | +| | | paramete\| | +| | | r | | +| | | to be | | +| | | editable | | +| | | by | | +| | | policy | | +| | | (true) | | +| | | or not | | +| | | (false) | | ++--------------+----+----------+------+ +| policy_sche\ | ar\| The | | +| ma | ra\| optional | | +| | y | list of | | +| | | schema | | +| | | definiti\| | +| | | ons | | +| | | used for | | +| | | policy. | | +| | | See | | +| | | below | | ++--------------+----+----------+------+ + +Example: + +.. code:: json + + "parameters": [ + { + "name": "threshold", + "value": 0.75, + "description": "Probability threshold to exceed to be anomalous" + } + ] + +Many of the parameter properties have been copied from TOSCA model +property definitions and are to be used for service design composition +and policy creation. See `section 3.5.8 *Property +definition* `__. + +The property ``constraints`` is a list of objects where each constraint +object: + ++--------------+----+----------+ +| Property | Ty\| Descript\| +| Name | pe | ion | ++==============+====+==========+ +| equal | | Constrai\| +| | | ns | +| | | a | +| | | property | +| | | or | +| | | paramete\| +| | | r | +| | | to a | +| | | value | +| | | equal to | +| | | (‘=’) | +| | | the | +| | | value | +| | | declared | ++--------------+----+----------+ +| greater_tha\ | nu\| Constrai\| +| n | mb\| ns | +| | er | a | +| | | property | +| | | or | +| | | paramete | +| | | r | +| | | to a | +| | | value | +| | | greater | +| | | than | +| | | (‘>’) | +| | | the | +| | | value | +| | | declared | ++--------------+----+----------+ +| greater_or_e\| nu\| Constrai\| +| qual | mb\| ns | +| | er | a | +| | | property | +| | | or | +| | | paramete\| +| | | r | +| | | to a | +| | | value | +| | | greater | +| | | than or | +| | | equal to | +| | | (‘>=’) | +| | | the | +| | | value | +| | | declared | ++--------------+----+----------+ +| less_than | nu\| Constrai\| +| | mb\| ns | +| | er | a | +| | | property | +| | | or | +| | | paramete\| +| | | r | +| | | to a | +| | | value | +| | | less | +| | | than | +| | | (‘<’) | +| | | the | +| | | value | +| | | declared | ++--------------+----+----------+ +| less_or_equ\ | nu\| Constrai\| +| al | mb\| ns | +| | er | a | +| | | property | +| | | or | +| | | paramete\| +| | | r | +| | | to a | +| | | value | +| | | less | +| | | than or | +| | | equal to | +| | | (‘<=’) | +| | | the | +| | | value | +| | | declared | ++--------------+----+----------+ +| valid_value\ | ar\| Constrai\| +| s | ra\| ns | +| | y | a | +| | | property | +| | | or | +| | | paramete\| +| | | r | +| | | to a | +| | | value | +| | | that is | +| | | in the | +| | | list of | +| | | declared | +| | | values | ++--------------+----+----------+ +| length | nu\| Constrai\| +| | mb\| ns | +| | er | the | +| | | property | +| | | or | +| | | paramete\| +| | | r | +| | | to a | +| | | value of | +| | | a given | +| | | length | ++--------------+----+----------+ +| min_length | nu\| Constrai\| +| | mb\| ns | +| | er | the | +| | | property | +| | | or | +| | | paramete\| +| | | r | +| | | to a | +| | | value to | +| | | a | +| | | minimum | +| | | length | ++--------------+----+----------+ +| max_length | nu\| Constrai\| +| | mb\| ns | +| | er | the | +| | | property | +| | | or | +| | | paramete\| +| | | r | +| | | to a | +| | | value to | +| | | a | +| | | maximum | +| | | length | ++--------------+----+----------+ + +``threshold`` is the configuration parameter and will get set to 0.75 +when the configuration gets generated. + +The property ``policy_schema`` is a list of objects where each +policy_schema object: + ++-------------+----+----------+------+ +| Property | Ty\| Descript\| Defa\| +| Name | pe | ion | ult | ++=============+====+==========+======+ +| name | st\| *Require\| | +| | ri\| d*. | | +| | ng | paramete\| | +| | | r | | +| | | name | | ++-------------+----+----------+------+ +| value | st\| default | | +| | ri\| value | | +| | ng | for the | | +| | | paramete\| | +| | | r | | ++-------------+----+----------+------+ +| description | st\| paramete\| | +| | ri\| r | | +| | ng | descript\| | +| | | ion | | ++-------------+----+----------+------+ +| type | en\| *Require\| | +| | um | d*. | | +| | | data | | +| | | type of | | +| | | the | | +| | | paramete\| | +| | | r, | | +| | | ‘string’ | | +| | | , | | +| | | ‘number’ | | +| | | , | | +| | | ‘boolean | | +| | | ’, | | +| | | ‘datetim\| | +| | | e’, | | +| | | ‘list’, | | +| | | or ‘map’ | | ++-------------+----+----------+------+ +| required | bo\| is | true | +| | ol\| paramete\| | +| | ea\| r | | +| | n | required | | +| | | or not? | | ++-------------+----+----------+------+ +| constraints | ar\| The | | +| | ra\| optional | | +| | y | list of | | +| | | sequence\| | +| | | d | | +| | | constrai\| | +| | | nt | | +| | | clauses | | +| | | for the | | +| | | paramete\| | +| | | r. | | +| | | See | | +| | | above | | ++-------------+----+----------+------+ +| entry_schem\| st\| The | | +| a | ri\| optional | | +| | ng | key that | | +| | | is used | | +| | | to | | +| | | declare | | +| | | the name | | +| | | of the | | +| | | Datatype | | +| | | definiti\| | +| | | on | | +| | | for | | +| | | certain | | +| | | types. | | +| | | entry_sc\| | +| | | hema | | +| | | must be | | +| | | defined | | +| | | when the | | +| | | type is | | +| | | either | | +| | | list or | | +| | | map. If | | +| | | the type | | +| | | is list | | +| | | and the | | +| | | entry | | +| | | type is | | +| | | a simple | | +| | | type | | +| | | (string, | | +| | | number, | | +| | | bookean, | | +| | | datetime | | +| | | ), | | +| | | follow | | +| | | with an | | +| | | string | | +| | | to | | +| | | describe | | +| | | the | | +| | | entry | | ++-------------+----+----------+------+ +| | If | | | +| | th\| | | +| | e | | | +| | ty\| | | +| | pe | | | +| | is | | | +| | li\| | | +| | st | | | +| | an\| | | +| | d | | | +| | th\| | | +| | e | | | +| | en\| | | +| | tr\| | | +| | y | | | +| | ty\| | | +| | pe | | | +| | is | | | +| | a | | | +| | ma\| | | +| | p, | | | +| | fo\| | | +| | ll\| | | +| | ow | | | +| | wi\| | | +| | th | | | +| | an | | | +| | ar\| | | +| | ra\| | | +| | y | | | +| | to | | | +| | de\| | | +| | sc\| | | +| | ri\| | | +| | be | | | +| | th\| | | +| | e | | | +| | ke\| | | +| | ys | | | +| | fo\| | | +| | r | | | +| | th\| | | +| | e | | | +| | en\| | | +| | tr\| | | +| | y | | | +| | ma\| | | +| | p | | | ++-------------+----+----------+------+ +| | If | | | +| | th\| | | +| | e | | | +| | ty\| | | +| | pe | | | +| | is | | | +| | li\| | | +| | st | | | +| | an\| | | +| | d | | | +| | th\| | | +| | e | | | +| | en\| | | +| | tr\| | | +| | y | | | +| | ty\| | | +| | pe | | | +| | is | | | +| | a | | | +| | li\| | | +| | st | | | +| | , | | | +| | th\| | | +| | at | | | +| | is | | | +| | no\| | | +| | t | | | +| | cu\| | | +| | rr\| | | +| | en\| | | +| | tl\| | | +| | y | | | +| | su\| | | +| | pp\| | | +| | or\| | | +| | te\| | | +| | d \| | | ++-------------+----+----------+------+ +| | If | | | +| | th\| | | +| | e | | | +| | ty\| | | +| | pe | | | +| | is | | | +| | ma\| | | +| | p, | | | +| | fo\| | | +| | ll\| | | +| | ow | | | +| | wi\| | | +| | th | | | +| | an | | | +| | ar\| | | +| | ay | | | +| | to | | | +| | de\| | | +| | sc\| | | +| | ri\| | | +| | be | | | +| | th\| | | +| | e | | | +| | ke\| | | +| | ys | | | +| | fo\| | | +| | r | | | +| | th\| | | +| | e | | | +| | ma\| | | +| | p | | | ++-------------+----+----------+------+ + +Generated Application Configuration +----------------------------------- + +The above example for component ``asimov.component.kpi_anomaly`` will +get transformed into the following application configuration JSON that +is fully resolved and provided at runtime by calling the +``config binding service``: + +.. code:: json + + { + "streams_publishes": { + "prediction": ["10.100.1.100:32567"] + }, + "streams_subscribes": {}, + "threshold": 0.75, + "services_calls": { + "vnf-db": ["10.100.1.101:32890"] + } + } + +.. _artifacts: + +Artifacts +--------- + +``artifacts`` contains a list of artifacts associated with this +component. For Docker, this is the full path (including the registry) to +the Docker image. For CDAP, this is the full path to the CDAP jar. + ++---------------+------------+---------------------------------+ +| Property Name | Type | Description | ++===============+============+=================================+ +| artifacts | JSON array | Each entry is a artifact object | ++---------------+------------+---------------------------------+ + +``artifact`` Schema: + ++---------------+--------+--------------------------------------------+ +| Property Name | Type | Description | ++===============+========+============================================+ +| uri | string | *Required*. Uri to the artifact, full path | ++---------------+--------+--------------------------------------------+ +| type | string | *Required*. ``docker image`` or ``jar`` | ++---------------+--------+--------------------------------------------+ diff --git a/docs/sections/components/component-specification/component-specification.rst b/docs/sections/components/component-specification/component-specification.rst new file mode 100644 index 00000000..759f3dc9 --- /dev/null +++ b/docs/sections/components/component-specification/component-specification.rst @@ -0,0 +1,16 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +Component Specification +======================= + +.. toctree:: + :maxdepth: 1 + + ./common-specification.rst + ./cdap-specification.rst + ./docker-specification.rst + ./generated-configuration.rst + ./streams-grid.rst + ./configuration-grid.rst + diff --git a/docs/sections/components/component-specification/configuration-grid.rst b/docs/sections/components/component-specification/configuration-grid.rst new file mode 100755 index 00000000..956e1ff2 --- /dev/null +++ b/docs/sections/components/component-specification/configuration-grid.rst @@ -0,0 +1,315 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +Configuration Quick Reference +============================= + +The following types of configuration are supported by the DCAE Platform. + ++---------+---------+----------+---------+-----------+---+ +| Input | Default | Designer | Clamp | Policy | R\| +| Sources | Values | Input | Input | Input | u\| +| | | | | | n\| +| | | | | | t\| +| | | | | | i\| +| | | | | | m\| +| | | | | | e | +| | | | | | I\| +| | | | | | n\| +| | | | | | p\| +| | | | | | u\| +| | | | | | t | ++=========+=========+==========+=========+===========+===+ +| Notes | | This | This | | | +| | | applies | applies | | | +| | | only to | only to | | | +| | | componen\| compone\| | | +| | | ts | nts | | | +| | | that are | that | | | +| | | self-ser\| are | | | +| | | vice | part of | | | +| | | (support\| a | | | +| | | ed | closed-\| | | +| | | by SDC) | loop | | | +| | | | interfa\| | | +| | | | ce | | | ++---------+---------+----------+---------+-----------+---+ +| Who | Compone\| Service | CLAMP | Operatio\ | R\| +| provide\| nt | Designer | | ns | u\| +| s? | Develop | | | | n\| +| | er | | | | t\| +| | | | | | i\| +| | | | | | m\| +| | | | | | e | +| | | | | | P\| +| | | | | | l\| +| | | | | | a\| +| | | | | | t\| +| | | | | | f\| +| | | | | | o\| +| | | | | | r\| +| | | | | | m | ++---------+---------+----------+---------+-----------+---+ +| When/Wh\| During | At | At | Anytime | W\| +| ere | onboard\| design | install\| – in the | h\| +| it is | ing | time – | ation | POLICY | e\| +| provide | – in | in the | – in | GUI | n | +| d | the | SDC UI | the | | t\| +| | compone\| | CLAMP | | h\| +| | nt | | UI | | e | +| | specifi\| | | | c\| +| | cation | | | | o\| +| | | | | | m\| +| | | | | | p\| +| | | | | | o\| +| | | | | | n\| +| | | | | | e\| +| | | | | | n\| +| | | | | | t | +| | | | | | i\| +| | | | | | s | +| | | | | | d\| +| | | | | | e\| +| | | | | | p\| +| | | | | | l\| +| | | | | | o\| +| | | | | | y\| +| | | | | | e\| +| | | | | | d | ++---------+---------+----------+---------+-----------+---+ +| Compone\| For | ‘designe\| | ‘policy-\ | ‘\| +| nt | CDAP: | r-editab\| | editable\ | s\| +| Specifi\| ‘value’ | le’ | | ’ | o\| +| cation | Name | must be | | must be | u\| +| Details | and KV | set to | | set to | r\| +| | pairs | ‘true’ | | ‘true’ | c\| +| | in | for | | and | e\| +| | AppConf\| variable | | ‘policy_s\| d\| +| | ig | in | | chema’ | _\| +| | or | ‘paramet\| | must be | a\| +| | AppPref\| er’ | | provided | t\| +| | erences | section. | | for | _\| +| | For | | | variable | \ | +| | Docker: | | | in | d\| +| | ‘value’ | | | ‘paramet\ | e\| +| | is | | | er’ | p\| +| | provide | | | section | l\| +| | d | | | | o\| +| | for | | | | y\| +| | variabl\| | | | m\| +| | e | | | | e\| +| | in | | | | n\| +| | ‘parame\| | | | t\| +| | ter’ | | | | ’ | +| | section | | | | m\| +| | | | | | u\| +| | | | | | s\| +| | | | | | t | +| | | | | | b\| +| | | | | | e | +| | | | | | s\| +| | | | | | e\| +| | | | | | t | +| | | | | | t\| +| | | | | | o | +| | | | | | ‘\| +| | | | | | t\| +| | | | | | r\| +| | | | | | u\| +| | | | | | e | +| | | | | | ’\| +| | | | | | f\| +| | | | | | o\| +| | | | | | r | +| | | | | | v\| +| | | | | | a\| +| | | | | | r\| +| | | | | | i\| +| | | | | | a\| +| | | | | | b\| +| | | | | | l\| +| | | | | | e\| +| | | | | | i\| +| | | | | | n\| +| | | | | | ‘ | +| | | | | | p\| +| | | | | | a\| +| | | | | | r\| +| | | | | | a\| +| | | | | | m\| +| | | | | | e\| +| | | | | | t\| +| | | | | | e\| +| | | | | | r\| +| | | | | | ’ | +| | | | | | s\| +| | | | | | e\| +| | | | | | c\| +| | | | | | t\| +| | | | | | i\| +| | | | | | o\| +| | | | | | n | ++---------+---------+----------+---------+-----------+---+ + ++---------+---------+----------+---------+-----------+---+ +| How it | This is | This | This | This | T\| +| is used | passed | override\| overrid\| override\ | h\| +| | to the | s | es | s | i\| +| | compone\| any | any | any | s | +| | nt | values | values | values | o\| +| | in the | previous\| previou\| previous\ | v\| +| | generat\| ly | sly | ly | e\| +| | ed | set, but | set, | set, but | r\| +| | configu\| can be | but can | can be | r\| +| | ration | overridd\| be | overridd\ | i\| +| | if not | en | overrid\| en | d\| +| | overrid\| by CLAMP | den | at any | e\| +| | den. | or | by | point | s | +| | | POLICY. | POLICY. | thereaft\ | a\| +| | | | | er. | n\| +| | | | | | y | +| | | | | | v\| +| | | | | | a\| +| | | | | | l\| +| | | | | | u\| +| | | | | | e\| +| | | | | | s | +| | | | | | p\| +| | | | | | r\| +| | | | | | e\| +| | | | | | v\| +| | | | | | i\| +| | | | | | o\| +| | | | | | u\| +| | | | | | s\| +| | | | | | l\| +| | | | | | y | +| | | | | | s\| +| | | | | | e\| +| | | | | | t\| +| | | | | | , | +| | | | | | b\| +| | | | | | u\| +| | | | | | t | +| | | | | | c\| +| | | | | | a\| +| | | | | | n | +| | | | | | b\| +| | | | | | e | +| | | | | | o\| +| | | | | | v\| +| | | | | | e\| +| | | | | | r\| +| | | | | | r\| +| | | | | | i\| +| | | | | | d\| +| | | | | | d\| +| | | | | | e\| +| | | | | | n | +| | | | | | a\| +| | | | | | t | +| | | | | | a\| +| | | | | | n\| +| | | | | | y | +| | | | | | p\| +| | | | | | o\| +| | | | | | i\| +| | | | | | n\| +| | | | | | t | +| | | | | | t\| +| | | | | | h\| +| | | | | | e\| +| | | | | | r\| +| | | | | | e\| +| | | | | | a\| +| | | | | | f\| +| | | | | | t\| +| | | | | | e\| +| | | | | | r | +| | | | | | b\| +| | | | | | y | +| | | | | | P\| +| | | | | | o\| +| | | | | | l\| +| | | | | | i\| +| | | | | | c\| +| | | | | | y\| +| | | | | | . | ++---------+---------+----------+---------+-----------+---+ +| Additio\| For | | | For | | +| nal | CDAP: | | | Docker: | | +| Info | ‘value’ | | | In the | | +| for | is | | | auxiliar\ | | +| Compone\| provide\| | | y | | +| nt | d | | | section: | | +| Develop\| for | | | {“policy | | +| er | variabl\| | | ”: | | +| | e | | | {“trigge\ | | +| | in the | | | r_type”: | | +| | ‘AppCon\| | | “policy” | | +| | fig’ | | | ,“script\ | | +| | or | | | _path”: | | +| | ‘AppPre\| | | “/opt/ap\ | | +| | ference\| | | p/reconf\ | | +| | s’ | | | igure.sh\ | | +| | section\| | | ”} | | +| | s | | | } Script | | +| | For | | | interfac\ | | +| | Docker: | | | e | | +| | ‘value’ | | | must be | | +| | is | | | “opt/app\ | | +| | provide\| | | /reconfi\ | | +| | d | | | gure.sh” | | +| | for | | | $trigger\ | | +| | variabl\| | | _type | | +| | e | | | $updated\ | | +| | in | | | _policie\ | | +| | ‘parame\| | | s | | +| | ter’ | | | $updated\ | | +| | section | | | _appl_co\ | | +| | | | | nfig" | | +| | | | | where | | +| | | | | $updated\ | | +| | | | | _policie\ | | +| | | | | s | | +| | | | | is a | | +| | | | | json | | +| | | | | provided | | +| | | | | by the | | +| | | | | Policy | | +| | | | | Handler | | +| | | | | and | | +| | | | | $update_a\| | +| | | | | ppl_con\ | | +| | | | | fig | | +| | | | | is the | | +| | | | | post-mer\ | | +| | | | | ged | | +| | | | | appl\ | | +| | | | | config | | +| | | | | which | | +| | | | | may | | +| | | | | contain | | +| | | | | unresolv\ | | +| | | | | ed | | +| | | | | configur\ | | +| | | | | ation | | +| | | | | that | | +| | | | | didn’t | | +| | | | | come | | +| | | | | from | | +| | | | | policy. | | +| | | | | Suggesti\ | | +| | | | | on | | +| | | | | is for | | +| | | | | script | | +| | | | | to call | | +| | | | | CONFIG | | +| | | | | BINDING | | +| | | | | SERVICE | | +| | | | | to | | +| | | | | resolve | | +| | | | | any | | +| | | | | configur\ | | +| | | | | ation. | | ++---------+---------+----------+---------+-----------+---+ diff --git a/docs/sections/components/component-specification/docker-specification.rst b/docs/sections/components/component-specification/docker-specification.rst new file mode 100755 index 00000000..38612e17 --- /dev/null +++ b/docs/sections/components/component-specification/docker-specification.rst @@ -0,0 +1,454 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +.. _docker-specification: + +Component specification (Docker) +================================ + +The Docker component specification contains the following groups of +information. Many of these are common to both Docker and CDAP components +and are therefore described in the common specification. + +- :any:`Metadata ` +- :any:`Interfaces ` including the + associated :any:`Data Formats ` +- :any:`Parameters ` +- :any:`Auxiliary Details ` +- :any:`List of Artifacts ` + +.. _docker-auxiliary-details: + +Auxiliary Details +----------------- + +``auxiliary`` contains Docker specific details like health check, port +mapping, volume mapping, and policy reconfiguration script details. + ++-------------+----+----------+ +| Name | Ty\| Descript\| +| | pe | ion | ++=============+====+==========+ +| healthcheck | JS\| *Require\| +| | ON | d*. | +| | ob\| Health | +| | je\| check | +| | ct | definiti\| +| | | on | +| | | details | ++-------------+----+----------+ +| ports | JS\| each | +| | ON | array | +| | ar\| item | +| | ra\| maps a | +| | y | containe\| +| | | r | +| | | port to | +| | | the host | +| | | port. | +| | | See | +| | | example | +| | | below. | ++-------------+----+----------+ +| volume | JS\| each | +| | ON | array | +| | ar\| item | +| | ra\| contains | +| | y | a host | +| | | and | +| | | containe\| +| | | r | +| | | object. | +| | | See | +| | | example | +| | | below. | ++-------------+----+----------+ +| policy | JS\| *Require\| +| | ON | d*. | +| | ar\| Policy | +| | ra\| script | +| | y | details | ++-------------+----+----------+ + +Health Check Definition +~~~~~~~~~~~~~~~~~~~~~~~ + +The platform uses Consul to perform periodic health check calls. Consul +provides different types of `check definitions `_. The +platform currently supports http and docker health checks. + +When choosing a value for interval, consider that too frequent +healthchecks will put unnecessary load on Consul and DCAE. If there is a +problematic resource, then more frequent healthchecks are warranted (eg +15s or 60s), but as stablility increases, so can these values, (eg +300s). + +When choosing a value for timeout, consider that too small a number will +result in increasing timeout failures, and too large a number will +result in a delay in the notification of resource problem. A suggestion +is to start with 5s and workd from there. + +http +^^^^ + ++-------------+----+----------+ +| Property | Ty\| Descript\| +| Name | pe | ion | ++=============+====+==========+ +| type | st\| *Require\| +| | ri\| d*. | +| | ng | ``http`` | ++-------------+----+----------+ +| interval | st\| Interval | +| | ri\| duration | +| | ng | in | +| | | seconds | +| | | i.e. | +| | | ``60s`` | ++-------------+----+----------+ +| timeout | st\| Timeout | +| | ri\| in | +| | ng | seconds | +| | | i.e. | +| | | ``5s`` | ++-------------+----+----------+ +| endpoint | st\| *Require\| +| | ri\| d*. | +| | ng | GET | +| | | endpoint | +| | | provided | +| | | by the | +| | | componen\| +| | | t | +| | | for | +| | | Consul | +| | | to call | +| | | to check | +| | | health | ++-------------+----+----------+ + +Example: + +.. code:: json + + "auxilary": { + "healthcheck": { + "type": "http", + "interval": "15s", + "timeout": "1s", + "endpoint": "/my-health" + } + } + +docker script example +^^^^^^^^^^^^^^^^^^^^^ + ++-------------+----+------------+ +| Property | Ty\| Descript\ | +| Name | pe | ion | ++=============+====+============+ +| type | st\| *Require\ | +| | ri\| d*. | +| | ng | ``docker`` | ++-------------+----+------------+ +| interval | st\| Interval | +| | ri\| duration | +| | ng | in | +| | | seconds | +| | | i.e. | +| | | ``15s`` | ++-------------+----+------------+ +| timeout | st\| Timeout | +| | ri\| in | +| | ng | seconds | +| | | i.e. | +| | | ``1s`` | ++-------------+----+------------+ +| script | st\| *Require\ | +| | ri\| d*. | +| | ng | Full | +| | | path of | +| | | script | +| | | that | +| | | exists | +| | | in the | +| | | Docker | +| | | containe\ | +| | | r | +| | | to be | +| | | executed | ++-------------+----+------------+ + +Consul will use the `Docker exec API `_ to +periodically call your script in your container. It will examine the +script result to identify whether your component is healthy. Your +component is considered healthy when the script returns ``0`` otherwise +your component is considered not healthy. + +Example: + +.. code:: json + + "auxilary": { + "healthcheck": { + "type": "docker", + "script": "/app/resources/check_health.py", + "timeout": "30s", + "interval": "180s" + } + } + +Ports +~~~~~ + +.. code:: json + + "auxilary": { + "ports": ["8080:8000"] + } + +In the example above, container port 8080 maps to host port 8000. + +Volume Mapping +~~~~~~~~~~~~~~ + +.. code:: json + + "auxilary": { + "volumes": [ + { + "container": { + "bind": "/tmp/docker.sock", + "mode": "ro" + }, + "host": { + "path": "/var/run/docker.sock" + } + } + ] + } + +At the top-level: + ++---------------+-------+-------------------------------------+ +| Property Name | Type | Description | ++===============+=======+=====================================+ +| volumes | array | Contains container and host objects | ++---------------+-------+-------------------------------------+ + +The ``container`` object contains: + ++----------------------+----------------------+----------------------+ +| Property Name | Type | Description | ++======================+======================+======================+ +| bind | string | path to the | +| | | container volume | ++----------------------+----------------------+----------------------+ +| mode | string | “ro” - indicates | +| | | read-only volume | +| | | “” - indicates that | +| | | the container can | +| | | write into the bind | +| | | mount | ++----------------------+----------------------+----------------------+ + +The ``host`` object contains: + ++---------------+--------+-------------------------+ +| Property Name | Type | Description | ++===============+========+=========================+ +| path | string | path to the host volume | ++---------------+--------+-------------------------+ + +Here’s an example of the minimal JSON that must be provided as an input: + +.. code:: json + + "auxilary": { + "volumes": [ + { + "container": { + "bind": "/tmp/docker.sock" + }, + "host": { + "path": "/var/run/docker.sock" + } + } + ] + } + +In the example above, the container volume “/tmp/docker.sock” maps to +host volume “/var/run/docker.sock”. + +Policy +~~~~~~ + +Policy changes made in the Policy UI will be provided to the Docker +component by triggering a script that is defined here. + ++-------------+----+------------+ +| Property | Ty\| Descript\ | +| Name | pe | ion | ++=============+====+============+ +| reconfigure | st\| *Require\ | +| _type | ri\| d*. | +| | ng | Current | +| | | value | +| | | supporte | +| | | d | +| | | is | +| | | ``policy`` | ++-------------+----+------------+ +| script_path | st\| *Require\ | +| | ri\| d*. | +| | ng | Current | +| | | value | +| | | for | +| | | ‘policy’ | +| | | reconfig\ | +| | | ure_type | +| | | must be | +| | | “/opt/ap\ | +| | | p/reconf\ | +| | | igure.sh | +| | | ” | ++-------------+----+------------+ + +Example: + +.. code:: json + + "auxilary": { + "policy": { + "reconfigure_type": "policy", + "script_path": "/opt/app/reconfigure.sh" + } + } + +The docker script interface is as follows: \`/opt/app/reconfigure.sh +$reconfigure_type {“updated policies”: , “application config”: } + ++-----+----+---------------------------+ +| Na\ | Ty\| Descript\ | +| me | pe | ion | ++=====+====+===========================+ +| re\ | st\| “policy” | +| co\ | ri\| | +| nf\ | ng | | +| ig\ | | | +| ur\ | | | +| e_t\| | | +| y\ | | | +| pe\ | | | ++-----+----+---------------------------+ +| up\ | js\| TBD | +| da\ | on | | +| te\ | | | +| d_p\| | | +| o\ | | | +| li\ | | | +| ci\ | | | +| es | | | ++-----+----+---------------------------+ +| up\ | js\| complete | +| da\ | on | generate\ | +| te\ | | d | +| d_a\| | app_conf\ | +| p\ | | ig, | +| pl\ | | not | +| _c\ | | fully-re\ | +| on\ | | solved, | +| fi\ | | but | +| g | | ``policy-enabled`` | +| | | paramete\ | +| | | rs | +| | | have | +| | | been | +| | | updated. | +| | | In order | +| | | to get | +| | | the | +| | | complete | +| | | updated | +| | | app_conf\ | +| | | ig, | +| | | the | +| | | componen\ | +| | | t | +| | | would | +| | | have to | +| | | call | +| | | ``config-binding-service``| +| | | . | ++-----+----+---------------------------+ + +Docker Component Spec - Complete Example +---------------------------------------- + +.. code:: json + + { + "self": { + "version": "1.0.0", + "name": "asimov.component.kpi_anomaly", + "description": "Classifies VNF KPI data as anomalous", + "component_type": "docker" + }, + "streams": { + "subscribes": [{ + "format": "dcae.vnf.kpi", + "version": "1.0.0", + "route": "/data", + "type": "http" + }], + "publishes": [{ + "format": "asimov.format.integerClassification", + "version": "1.0.0", + "config_key": "prediction", + "type": "http" + }] + }, + "services": { + "calls": [{ + "config_key": "vnf-db", + "request": { + "format": "dcae.vnf.meta", + "version": "1.0.0" + }, + "response": { + "format": "dcae.vnf.kpi", + "version": "1.0.0" + } + }], + "provides": [{ + "route": "/score-vnf", + "request": { + "format": "dcae.vnf.meta", + "version": "1.0.0" + }, + "response": { + "format": "asimov.format.integerClassification", + "version": "1.0.0" + } + }] + }, + "parameters": [ + { + "name": "threshold", + "value": 0.75, + "description": "Probability threshold to exceed to be anomalous" + } + ], + "auxilary": { + "healthcheck": { + "type": "http", + "interval": "15s", + "timeout": "1s", + "endpoint": "/my-health" + } + }, + "artifacts": [{ + "uri": "fake.nexus.com/dcae/kpi_anomaly:1.0.0", + "type": "docker image" + }] + } diff --git a/docs/sections/components/component-specification/generated-configuration.rst b/docs/sections/components/component-specification/generated-configuration.rst new file mode 100755 index 00000000..ba5ae4a0 --- /dev/null +++ b/docs/sections/components/component-specification/generated-configuration.rst @@ -0,0 +1,114 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +.. _generated-configuration: + +Generated configuration +======================= + +The DCAE platform relies on the component specification to generate the +component’s application configuration JSON at deployment time. The +component developer should expect to use this configuration JSON in +their application to provision themselves. + +Pro-tip: As you build your component specification, you can use the +:any:`dcae-cli dev command ` to view what +the resulting application configuration will look like. + +Streams and services +-------------------- + +For both Docker and CDAP, when your component is deployed, any +``streams`` and ``services/calls`` you specified will be injected into +your configuration under the following well known structure. Your +component is required to parse this information if you have any +connectivity to DMaaP or are calling another DCAE component. + +More details about the DMaaP connection objects are found +:doc:`here <../dcae-cli/dmaap-connection-objects>`. + +This is best served with an example. + +The following component spec snippet (from String Matching): + +:: + + "streams":{ + "subscribes": [{ + "format": "VES_specification", + "version": "4.27.2", + "type": "message_router", + "config_key" : "mr_input" + }], + "publishes": [{ + "format": "VES_specification", + "version": "4.27.2", + "config_key": "mr_output", + "type": "message_router" + }] + }, + "services":{ + "calls": [{ + "config_key" : "aai_broker_handle", + "verb": "GET", + "request": { + "format": "get_with_query_params", + "version": "1.0.0" + }, + "response": { + "format": "aai_broker_response", + "version": "3.0.0" + } + }], + "provides": [] + }, + +Will turn into the following top level keys in your configuration (for +CDAP, this will be under AppConfig) + +:: + + "streams_publishes":{ + "mr_output":{ // notice the config key above + "aaf_password":"XXX", + "type":"message_router", + "dmaap_info":{ + "client_role": null, + "client_id": null, + "location": null, + "topic_url":"XXX" + }, + "aaf_username":"XXX" + } + }, + "streams_subscribes":{ + "mr_input":{ // notice the config key above + "aaf_password":"XXX", + "type":"message_router", + "dmaap_info":{ + "client_role": null, + "client_id": null, + "location": null, + "topic_url":"XXX" + }, + "aaf_username":"XXX" + } + }, + "services_calls":{ + "aai_broker_handle":[ // notice the config key above + "SOME_IP:32768" // based on deployment time, just an example + ] + } + +These keys will always be populated regardless of whether they are +empty. So the minimal you will get, in the case of a component that +provides an HTTP service and does not call any services and has no +streams, is: + +:: + + "streams_publishes":{}, + "streams_subscribes":{}, + "services_calls":{} + +Thus your component should expect these well-known top level keys. diff --git a/docs/sections/components/component-specification/start-here.rst b/docs/sections/components/component-specification/start-here.rst new file mode 100755 index 00000000..d3f5a12f --- /dev/null +++ b/docs/sections/components/component-specification/start-here.rst @@ -0,0 +1 @@ + diff --git a/docs/sections/components/component-specification/streams-grid.rst b/docs/sections/components/component-specification/streams-grid.rst new file mode 100755 index 00000000..6105e9e1 --- /dev/null +++ b/docs/sections/components/component-specification/streams-grid.rst @@ -0,0 +1,149 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +.. _streams-grid: + +Streams Formatting Quick Reference +================================== + +Each of the following tables represents an example of a publisher and +its subscriber, which are of course, different components. This focuses +on the fields that are ‘different’ for each of these TYPEs, to +illustrate the relationship between ``config_key``, dmaap connection +object, and the generated configuration. Some notes on specific +properties: + +- ``config_key`` is an arbitrary string, chosen by the component + developer. It is returned in the generated configuration where it + contains specific values for the target connection +- ``format``, ``version``, and ``type`` properties in the subscriber + would match these properties in the publisher +- ``aaf_username`` and ``aaf_password`` may be different between the + publisher and the subscriber + +Using http +~~~~~~~~~~ + +*Publishing Component* +^^^^^^^^^^^^^^^^^^^^^^ + ++-----------------------------+----------------------------------------+ +| component \ | runtime platform generated config | +| spec | | ++=============================+========================================+ +| "streams":{   | "streams_publishes":{  | +| "publishes":[{ | "prediction":"10.100.1.100:32567/data" | +| "config_key":"prediction", |  | +| "format":"some-format", | | +| "type":"http", | | +| "version":"0.1.0"   } | | +| ]} | | ++-----------------------------+----------------------------------------+ + +*Subscribing Component* +^^^^^^^^^^^^^^^^^^^^^^^ + ++-----------------------------+----------------------------------------+ +| component | runtime platform generated config | +| spec | | ++=============================+========================================+ +| “streams”:{    | "N/A" | +| "subscribes":[{ | | +| "route":"/data", | | +| "format":"some-format", | | +| "type":"http" | | +| "version":"0.1.0"   } | | +| ]} | | ++-----------------------------+----------------------------------------+ + +Using Message Router +~~~~~~~~~~~~~~~~~~~~ + +.. publishing-component-1: + +*Publishing Component* +^^^^^^^^^^^^^^^^^^^^^^ + +Note: When deploying, this component should be deployed first so satisfy +downstream dependencies. Refer to the –force option in component ‘run’ +command for more information. + ++---------------+------------------------+-----------------------------------------------------------------------------+ +| component \ | Dmaap Connection \ | runtime platform generated \ | +| spec | Object | config | ++===============+========================+=============================================================================+ +| “streams”:{  | {     “dmaap_info”: | “streams_publishes”:{    | +|     “config_k\| {} \ *Note: For \ | “aaf_username”:“pub-user”,   | +| ey”:“mr_out\ | message router, this \ |   “type”:“message_router”,   | +| put”,     “t\ | object is identical \ |      “topic_url”:"https://we-are-message-router.us:3905/events/some-topic"\ | +| ype”:“messag\ | for the publisher and \| "streams_subscribes":{...} | +| e_router”,   | the subscriber* | | +|  }]} | | | ++---------------+------------------------+-----------------------------------------------------------------------------+ + +*Subscribing Component* +^^^^^^^^^^^^^^^^^^^^^^^ + ++---------------+------------------------+-----------------------------------------------------------------------------+ +| component \ | Dmaap Connection \ | runtime platform generated \ | +| spec | Object | config | ++===============+========================+=============================================================================+ +| “streams”:{  | {     “dmaap_info”: | “streams_publishes”:{…}, | +|     “config_k\| {} \ *Note: For \ | “streams_subscribes”:{    | +| ey”:“mr_inp\ | message router, this \ | “aaf_username”:“sub-user”,   | +| ut”,     “ty\ | object is identical \ |   “type”:“message_router”,   | +| pe”:“message\ | for the publisher and \|      “topic_url”:“https://we-are-message-router.us:3905/events/some-topic" | +| _router”,    | the subscriber* | | +| }]} | | | ++---------------+------------------------+-----------------------------------------------------------------------------+ + +Using Data Router +~~~~~~~~~~~~~~~~~ + +.. publishing-component-2: + +*Publishing Component* +^^^^^^^^^^^^^^^^^^^^^^ + ++---------------+-----------------------------------------------+-----------------------------------------------+ +| component spec| Dmaap Connection Object | runtime platform generated config | ++===============+===============================================+===============================================+ +| “streams”:{  | {    “dmaap_info”: { | streams_publishes“:{    ”typ\ | +| “config_key: |      “location”: | e“:”data_router“,       "location":"mtc00" | +| “dr_output" | “mtc00”,      | , | +| , "type": | “publish_url”: | "publish_url“: | +| “data_r\ | "https://we-are-data-router.us/feed/xyz"\ | "http://we-are-data-router.us/feed/xyz" | +| outer”,   }] | , | , | +| } | “log_url”:\ | "log_url“:\ | +| | \ | ”https://we-are-data-router.us/feed/xyz/logs" | +| | "https://we-are-data-router.us/feed/xyz/logs"\| , | +| | , | ”username“:”pub-user“, | +| | “username”: | ”publisher_id“:”123456\ | +| | “pub-user”,      | “}}, | +| | “password”: |  ”streams_subscribes“:{ | +| | “pub-password”,      | … } | +| | “publisher_id”: | | +| | “123456”}} | | ++---------------+-----------------------------------------------+-----------------------------------------------+ + +.. subscribing-component-1: + +*Subscribing Component* +^^^^^^^^^^^^^^^^^^^^^^^ + ++---------------+---------------------------------------------------+---------------------------------------------------------------------------+ +| component \ | Dmaap Connection \ | runtime platform generated \ | +| spec | Object | config | ++===============+===================================================+===========================================================================+ +| “streams”:{  | {      “dmaap_info”: | “streams_publishes”:{ … }, | +|     “config_k\| {      “location”: | “streams_subscribes”:{       | +| ey”:“dr_inp\ | “mtc00”,      | “type”:“data_router”,        | +| ut”,     “ty\ | “delivery_url”: |   “location”:“mtc00”, | +| pe”:“data_ro\ | "https://my-subscriber-app.dcae:8080/target-path"\|          “delivery_url”:"https://my-subscriber-app.dcae:8080/target-path"\| +| uter”, | \ | \ | +|     “route”: | , | , | +| “/target-pat\ |      “password”: | \        | +| h”} | “sub-password”,      | “username”:“sub-user”, | +| | “subscriber_id”: |          | +| | “789012”}} | “subscriber_id”:“789012”}} | ++---------------+---------------------------------------------------+---------------------------------------------------------------------------+ -- cgit