From 64559741f071b37556b7745ffa2cdcc25f40af94 Mon Sep 17 00:00:00 2001 From: VENKATESH KUMAR Date: Wed, 29 Apr 2020 18:53:53 -0400 Subject: dcae design doc updates Change-Id: I7145f840d9a7de34bb6a615fe992ba22e1ff0380 Signed-off-by: VENKATESH KUMAR Issue-ID: DCAEGEN2-2024 Issue-ID: DCAEGEN2-1865 Signed-off-by: VENKATESH KUMAR --- .../component-specification/cdap-specification.rst | 208 --------------------- 1 file changed, 208 deletions(-) delete mode 100755 docs/sections/components/component-specification/cdap-specification.rst (limited to 'docs/sections/components/component-specification/cdap-specification.rst') diff --git a/docs/sections/components/component-specification/cdap-specification.rst b/docs/sections/components/component-specification/cdap-specification.rst deleted file mode 100755 index 2b26442a..00000000 --- a/docs/sections/components/component-specification/cdap-specification.rst +++ /dev/null @@ -1,208 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - -.. _cdap-specification: - -Component specification (CDAP) -============================== - -The CDAP component specification contains the following groups of -information. Many of these are common to both CDAP and Docker components -and are therefore described in the common specification. - -- :any:`Metadata ` -- :any:`Interfaces ` including the - associated :any:`Data Formats ` -- :any:`Parameters ` - for specifying parameters in your - AppConfig, AppPreferences, and ProgramPreferences to the Designer and - Policy. This of course is CDAP-specific and is described below. -- :any:`Auxiliary Details ` -- :any:`List of artifacts ` - -Current Limitations and TODOs ------------------------------ - -- The integration of DMD is likely to significantly change the - :any:`Interfaces ` section in this specification. - -.. _parameters: - -Parameters ----------- - -There is a ``parameters`` section in your component specification. This -section contains three optional keys: `app_config <#appconfig>`__, -`app_preferences <#apppreferences>`__, and -`program_preferences <#programpreferences>`__: - -:: - - "parameters" : { - "app_config" : [ ...], - "app_preferences" : [ ...], - "program_preferences" : [...] - // any additional keys are ignored - } - -- Each section details the parameters that are a part of each of these - CDAP constructs (see below). -- All such parameters will be exposed to the designer and to policy for - override. -- These parameters should have default values specified by the - component developer where necessary, i.e., parameters that *must* - come from the designer/policy should not have defaults. -- All of these keys are optional because not every CDAP application - uses preferences and not every application uses the AppConfig. - However, you should specify at least one, or else your application - will have no parameters exposed to policy or to the DCAE designer, - which means it would be non-configurable. -- Despite the AppConfig being optional to *specify* in the case that - you have no parameters in your AppConfig, it is *required for - processing* in your application. That is because the DCAE platform - will place important information into your AppConfig as discussed - below. - -Parameter -~~~~~~~~~ - -The following CDAP specific definitions use ``param1`` to refer to the -common parameter layout in -:any:`Parameter ` - -AppConfig -~~~~~~~~~ - -The ``app_config`` key refers to the `CDAP AppConfig `_. -It is expected to be a JSON: - -:: - - "app_config" : [ // list of JSON - param1, // common parameter layout - ... - ] - -Unfortunately, at the time of writing, the AppConfig is a Java map of -``string:string``, which means you cannot have more complex structures -(than string) as any value in your AppConfig. However, there is a way to -bypass this constraint: you can pass a JSON by encoding the JSON as a -string. E.g., the ``json.dumps()`` and it’s converse ``loads`` methods -in Python: - -:: - - >>> import json - >>> json.dumps({"foo" : "bar"}) # This is a real JSON - '{"foo": "bar"}' # It is now a string: pass this in as your parameter value - >>> json.loads('{"foo": "bar"}') # Do the equivelent of this in your application - {u'foo': u'bar'} # ...and you'll get back a real JSON - >>> - -The final AppConfig (after the designer and policy override parameter -values) is passed into CDAP’s AppConfig API when starting the -application. - - -AppPreferences -~~~~~~~~~~~~~~ - -In addition to the CDAP AppConfig, the platform supports `Application Preferences `_. -The format of the ``app_preferences`` value is the same as the above: - -:: - - "app_preferences" : [ // list of JSON - param1, // common parameter layout - ... - ] - -The final Application Preferences JSON (after the designer and policy -override parameter values) is passed into CDAP’s Preferences API when -starting your application. - - -ProgramPreferences -~~~~~~~~~~~~~~~~~~ - -Preferences can also be specified `per program `_ -in CDAP. This key’s value is a list of JSON with the following format: - -:: - - "program_preferences" : [ // note: this is a list of JSON - { - "program_id" : "program name 1", // the name of this CDAP program - "program_type" : "e.g., flows", // "must be one of flows, mapreduce, schedules, spark, workflows, workers, or services", - "program_pref" : [ // list of JSON - param1, // common parameter layout - ... - ] - }, - // repeat for each program that receives a program_preferences JSON - ] - -Each ``program_pref`` JSON is passed into the CDAP API as the preference -for ``program_id``. - -NOTE: for CDAP, this section is very likely to change when DMD is -available. The *future* vision is -that you would publish your data as a series of files on HDFS, and DMD -will pick them up and send them to the appropriate DMaaP feeds or -directly when needed. - -.. _auxiliary-details: - -Auxiliary Details ------------------ - -*auxiliary* contains details about CDAP specific parameters. - -+----------------------+----------------------+----------------------+ -| Property Name | Type | Description | -+======================+======================+======================+ -| streamname | string | *Required*. | -+----------------------+----------------------+----------------------+ -| artifact_name | string | | -+----------------------+----------------------+----------------------+ -| artifact_version | string | the version of your | -| | | CDAP JAR artifact | -+----------------------+----------------------+----------------------+ -| namespace | string | the CDAP namespace | -| | | to deploy into, | -| | | default is ‘default’ | -+----------------------+----------------------+----------------------+ -| programs | array | contains each CDAP | -| | | entity represented | -| | | in the artifact | -+----------------------+----------------------+----------------------+ -| program_type | string | CDAP entity (eg | -| | | “flows”) | -+----------------------+----------------------+----------------------+ -| program_id | string | name of CDAP entity | -| | | (eg “WhoFlow”) | -+----------------------+----------------------+----------------------+ - -Example: - -.. code:: json - - "auxiliary": { - "streamname" : "who", - "artifact_name" : "HelloWorld", - "artifact_version" : "3.4.3", - "namespace" : "hw", - "programs" : [ - {"program_type" : "flows", "program_id" : "WhoFlow"}, - {"program_type" : "services", "program_id" : "Greeting"}, - ... - ], - } - -The ``programs`` key is identical to the ``program_preferences`` key -discussed `above <#programpreferences>`__ except: - -- each JSON in the list does not contain ``program_pref`` -- this is required! You must include all of your programs in this, as - it is used to start each program as well as for DCAE to perform - periodic healthchecks on your application. Don’t forget about your - services; they are programs too. -- cgit 1.2.3-korg