diff options
Diffstat (limited to 'docs/sections/components/component-specification/cdap-specification.rst')
| -rwxr-xr-x | docs/sections/components/component-specification/cdap-specification.rst | 209 |
1 files changed, 209 insertions, 0 deletions
diff --git a/docs/sections/components/component-specification/cdap-specification.rst b/docs/sections/components/component-specification/cdap-specification.rst new file mode 100755 index 00000000..022b46d7 --- /dev/null +++ b/docs/sections/components/component-specification/cdap-specification.rst @@ -0,0 +1,209 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+.. _cdap-specification:
+
+Component specification (CDAP)
+==============================
+
+The CDAP component specification contains the following groups of
+information. Many of these are common to both CDAP and Docker components
+and are therefore described in the common specification.
+
+- :any:`Metadata <metadata>`
+- :any:`Interfaces <interfaces>` including the
+ associated :any:`Data Formats <data-formats>`
+- :any:`Parameters <parameters>` - for specifying parameters in your
+ AppConfig, AppPreferences, and ProgramPreferences to the Designer and
+ Policy. This of course is CDAP-specific and is described below.
+- :any:`Auxiliary Details <auxiliary-details>`
+- :any:`List of artifacts <artifacts>`
+
+Current Limitations and TODOs
+-----------------------------
+
+- The integration of DMD is likely to significantly change the
+ :any:`Interfaces <interfaces>` section in this specification, see
+ :any:`DMaaP abstraction <dmaap-abstraction>`.
+
+.. _parameters:
+
+Parameters
+----------
+
+There is a ``parameters`` section in your component specification. This
+section contains three optional keys: `app_config <#appconfig>`__,
+`app_preferences <#apppreferences>`__, and
+`propram_preferences <#programpreferences>`__:
+
+::
+
+ "parameters" : {
+ "app_config" : [ ...],
+ "app_preferences" : [ ...],
+ "program_preferences" : [...]
+ // any additional keys are ignored
+ }
+
+- Each section details the parameters that are a part of each of these
+ CDAP constructs (see below).
+- All such parameters will be exposed to the designer and to policy for
+ override.
+- These parameters should have default values specified by the
+ component developer where necessary, i.e., parameters that *must*
+ come from the designer/policy should not have defaults.
+- All of these keys are optional because not every CDAP application
+ uses preferences and not every application uses the AppConfig.
+ However, you should specify at least one, or else your application
+ will have no parameters exposed to policy or to the DCAE designer,
+ which means it would be non-configurable.
+- Despite the AppConfig being optional to *specify* in the case that
+ you have no parameters in your AppConfig, it is *required for
+ processing* in your application. That is because the DCAE platform
+ will place important information into your AppConfig as discussed
+ below.
+
+Parameter
+~~~~~~~~~
+
+The following CDAP specific definitions use ``param1`` to refer to the
+common parameter layout in
+:any:`Parameter <parameters>`
+
+AppConfig
+~~~~~~~~~
+
+The ``app_config`` key refers to the `CDAP AppConfig <http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/configuration.html>`_.
+It is expected to be a JSON:
+
+::
+
+ "app_config" : [ // list of JSON
+ param1, // common parameter layout
+ ...
+ ]
+
+Unfortunately, at the time of writing, the AppConfig is a Java map of
+``string:string``, which means you cannot have more complex structures
+(than string) as any value in your AppConfig. However, there is a way to
+bypass this constraint: you can pass a JSON by encoding the JSON as a
+string. E.g., the ``json.dumps()`` and it’s converse ``loads`` methods
+in Python:
+
+::
+
+ >>> import json
+ >>> json.dumps({"foo" : "bar"}) # This is a real JSON
+ '{"foo": "bar"}' # It is now a string: pass this in as your parameter value
+ >>> json.loads('{"foo": "bar"}') # Do the equivelent of this in your application
+ {u'foo': u'bar'} # ...and you'll get back a real JSON
+ >>>
+
+The final AppConfig (after the designer and policy override parameter
+values) is passed into CDAP’s AppConfig API when starting the
+application.
+
+
+AppPreferences
+~~~~~~~~~~~~~~
+
+In addition to the CDAP AppConfig, the platform supports `Application Preferences <http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/preferences.html#set-preferences>`_.
+The format of the ``app_preferences`` value is the same as the above:
+
+::
+
+ "app_preferences" : [ // list of JSON
+ param1, // common parameter layout
+ ...
+ ]
+
+The final Application Preferences JSON (after the designer and policy
+override parameter values) is passed into CDAP’s Preferences API when
+starting your application.
+
+
+ProgramPreferences
+~~~~~~~~~~~~~~~~~~
+
+Preferences can also be specified `per program <http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/lifecycle.html#program-lifecycle>`_
+in CDAP. This key’s value is a list of JSON with the following format:
+
+::
+
+ "program_preferences" : [ // note: this is a list of JSON
+ {
+ "program_id" : "program name 1", // the name of this CDAP program
+ "program_type" : "e.g., flows", // "must be one of flows, mapreduce, schedules, spark, workflows, workers, or services",
+ "program_pref" : [ // list of JSON
+ param1, // common parameter layout
+ ...
+ ]
+ },
+ // repeat for each program that receives a program_preferences JSON
+ ]
+
+Each ``program_pref`` JSON is passed into the CDAP API as the preference
+for ``program_id``.
+
+NOTE: for CDAP, this section is very likely to change when DMD is
+available. The *future* vision, as per :any:`DMaaP intentionally abstracted <dmaap-abstraction>` is
+that you would publish your data as a series of files on HDFS, and DMD
+will pick them up and send them to the appropriate DMaaP feeds or
+directly when needed.
+
+.. _auxiliary-details:
+
+Auxiliary Details
+-----------------
+
+*auxiliary* contains details about CDAP specific parameters.
+
++----------------------+----------------------+----------------------+
+| Property Name | Type | Description |
++======================+======================+======================+
+| streamname | string | *Required*. |
++----------------------+----------------------+----------------------+
+| artifact_name | string | |
++----------------------+----------------------+----------------------+
+| artifact_version | string | the version of your |
+| | | CDAP JAR artifact |
++----------------------+----------------------+----------------------+
+| namespace | string | the CDAP namespace |
+| | | to deploy into, |
+| | | default is ‘default’ |
++----------------------+----------------------+----------------------+
+| programs | array | contains each CDAP |
+| | | entity represented |
+| | | in the artifact |
++----------------------+----------------------+----------------------+
+| program_type | string | CDAP entity (eg |
+| | | “flows”) |
++----------------------+----------------------+----------------------+
+| program_id | string | name of CDAP entity |
+| | | (eg “WhoFlow”) |
++----------------------+----------------------+----------------------+
+
+Example:
+
+.. code:: json
+
+ "auxiliary": {
+ "streamname" : "who",
+ "artifact_name" : "HelloWorld",
+ "artifact_version" : "3.4.3",
+ "namespace" : "hw",
+ "programs" : [
+ {"program_type" : "flows", "program_id" : "WhoFlow"},
+ {"program_type" : "services", "program_id" : "Greeting"},
+ ...
+ ],
+ }
+
+The ``programs`` key is identical to the ``program_preferences`` key
+discussed `above <#programpreferences>`__ except:
+
+- each JSON in the list does not contain ``program_pref``
+- this is required! You must include all of your programs in this, as
+ it is used to start each program as well as for DCAE to perform
+ periodic healthchecks on your application. Don’t forget about your
+ services; they are programs too.
|