Centralize readthedocs docs

Move readthedocs documentation into ccsdk/distribution so that there
is a single CCSDK project parent as opposed to separate repo-specific
parents.

Change-Id: I8f7fc649fd534dfc7e4a00f04728e0233ff4d23c
Issue-ID: CCSDK-140
Signed-off-by: Dan Timoney <dtimoney@att.com>

1 files changed, 439 insertions, 0 deletions

diff --git a/docs/platform/plugins/dmaap.rst b/docs/platform/plugins/dmaap.rst
new file mode 100644
index 00000000..b49eb4f8
--- /dev/null
+++ b/docs/platform/plugins/dmaap.rst
@@ -0,0 +1,439 @@
+Cloudify DMaaP Plugin
+---------------------
+
+Cloudify plugin for creating and managing DMaaP Data Router feeds and
+subscriptions and DMaaP Message Router topics. The plugin uses the DMaaP
+Bus Controller API.
+
+Plugin Support for DMaaP Data Router
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Plugin Types for DMaaP Data Router
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The Cloudify type definitions for DMaaP Data Router nodes and
+relationships are defined in ```dmaap.yaml`` <./dmaap.yaml>`__.
+
+There are four node types for DMaaP Data Router:
+
+-  ``ccsdk.nodes.Feed``: This type represents a feed that does not yet
+   exist and that should be created when the install workflow is run
+   against a blueprint that contains a node of this type.
+
++------------------------+----------+-------------+---------------------------------------------------------------------------------------------+
+| Property               | Type     | Required?   | Description                                                                                 |
++========================+==========+=============+=============================================================================================+
+| feed\_name             | string   | no          | a name that identifies the feed (plugin will generate if absent)                            |
++------------------------+----------+-------------+---------------------------------------------------------------------------------------------+
+| feed\_version          | string   | no          | version number for the feed (feed\_name + feed\_version uniquely identify the feed in DR)   |
++------------------------+----------+-------------+---------------------------------------------------------------------------------------------+
+| feed\_description      | string   | no          | human-readable description of the feed                                                      |
++------------------------+----------+-------------+---------------------------------------------------------------------------------------------+
+| aspr\_classification   | string   | no          | AT&T ASPR classification of the feed                                                        |
++------------------------+----------+-------------+---------------------------------------------------------------------------------------------+
+
+-  ``ccsdk.nodes.ExistingFeed``: This type represents a feed that
+   already exists. Nodes of this type are placed in a blueprint so that
+   other nodes in the blueprint can be set up as publishers or
+   subscribers to the feed. The table below shows the properties that a
+   node of this type may have.
+
++------------+----------+-------------+---------------------------------------------------------------+
+| Property   | Type     | Required?   | Description                                                   |
++============+==========+=============+===============================================================+
+| feed\_id   | string   | yes         | Feed identifier assigned by DMaaP when the feed was created   |
++------------+----------+-------------+---------------------------------------------------------------+
+
+-  ``ccsdk.nodes.ExternalTargetFeed``: This type represents a feed
+   created in an external DMaaP environment (i.e., an environment that
+   the plugin cannot access to make provisioning requests, such as a
+   shared corporate system). Nodes of this type are placed in a
+   blueprint so that other feed nodes of type ``ccsdk.nodes.Feed`` or
+   ``ccsdk.nodes.ExistingFeed`` can be set up to "bridge" to external
+   feeds by publishing data to the external feeds. The table below shows
+   the properties that a node of this type may have.
+
++------------+----------+-------------+----------------------------------------------------------------+
+| Property   | Type     | Required?   | Description                                                    |
++============+==========+=============+================================================================+
+| url        | string   | yes         | The publish URL of the external feed.                          |
++------------+----------+-------------+----------------------------------------------------------------+
+| username   | string   | yes         | The username to be used when delivering to the external feed   |
++------------+----------+-------------+----------------------------------------------------------------+
+| userpw     | string   | yes         | The password to be used when delivering to the external feed   |
++------------+----------+-------------+----------------------------------------------------------------+
+
+*Note: These properties are usually obtained by manually creating a feed
+in the external DMaaP DR system and then creating a publisher for that
+feed.*
+
+-  ``ccsdk.nodes.ExternalSourceFeed``: This type represents a feed
+   created in an external DMaaP environment (i.e., an environment that
+   the plugin cannot access to makes provisioning requests, such as a
+   shared corporate system). Nodes of this type are place in a blueprint
+   so that they can be set up to "bridge" to other feed nodes of type
+   ``ccsdk.nodes.Feed`` or ``ccsdk.nodes.ExistingFeed``. This type has
+   no node properties, but when a bridge is set up, the url, username,
+   and password are attached to the node as runtime\_properties, using
+   the name of the target feed node as the top-level key.
+
+There are five relationship types for DMaaP Data Router:
+
+-  ``ccsdk.relationships.publish_files``, used to indicate that the
+   relationship's source node sends is a publisher to the Data Router
+   feed represented by the relationship's target node.
+-  ``ccsdk.relationships.subscribe_to_files``, used to indicate that the
+   relationship's source node is a subscriber to the Data Router feed
+   represented by the relationship's target node.
+-  ``ccsdk.relationships.bridges_to``, used to indicate that the
+   relationship's source node (a ``ccsdk.nodes.Feed`` or
+   ``ccsdk.nodes.ExistingFeed``) should be set up to forward data
+   ("bridge") to the relationship's target feed (another
+   ``ccsdk.nodes.Feed`` or ``ccsdk.nodes.ExistingFeed``).
+-  ``ccsdk.relationships.bridges_to_external``, used to indicate that
+   the relationship's source node (a ``ccsdk.nodes.Feed`` or
+   ``ccsdk.nodes.ExistingFeed``) should be set up to forward data
+   ("bridge") to the relationship's target node (a feed in an external
+   DMaaP system, represented by a ``ccsdk.nodes.ExternalTargetFeed``
+   node).
+-  ``ccsdk.relationships.bridges_from_external_to_internal``, used to
+   indicate the the relationship's source node (a feed in an external
+   DMaaP system, represented by a ``ccsdk.nodes.ExternalSourceFeed``
+   node) should be set up to forward date ("bridge") to the
+   relationship's target node (an internal ONAP feed, represented by a
+   ``ccsdk.nodes.Feed`` or ``ccsdk.nodes.ExistingFeed`` node).
+
+The plugin code implements the lifecycle operations needed to create and
+delete feeds and to add and remove publishers and subscribers. It also
+implements the operations needed to set up bridging between feeds.
+
+Interaction with Other Plugins
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When creating a new feed or processing a reference to an existing feed,
+the plugin operates independently of other plugins.
+
+When processing a ``ccsdk.relationships.publish_files`` relationship or
+a ``ccsdk.relationships.subscribe_to_files`` relationship, this plugin
+needs to obtain data from the source node and, in the case of
+``publish_files``, provide data to the source node. Certain conventions
+are therefore needed for passing data between this plugin and the
+plugins responsible for the source nodes in these relationships. In
+Cloudify, the mechanism for sharing data among plugins is the
+``ctx.instance.runtime_properties`` dictionary associated with each
+node.
+
+A given source node may have relationships with several feeds. For
+example, an ONAP DCAE data collector might publish two different types
+of data to two different feeds. An ONAP DCAE analytics module might
+subscribe to one feed to get input for its processing and publish its
+results to a different feed. When this DMaaP plugin and the plugin for
+the source node exchange information, they need to do in a way that lets
+them distinguish among different feeds. We do this through a simple
+convention: for each source node to feed relationship, the source node
+plugin will create a property in the source node's
+``runtime_properties`` dictionary. The name of the property will be the
+same as the name of the target node of the relationship. For instance,
+if a node has a ``publishes_files`` relationship with a target node
+named ``feed00``, then the plugin that's responsible for managing the
+source node with create an entry in the source node's
+``runtime_properties`` dictionary named ``feed00``. This entry itself
+will be a dictionary.
+
+The content of this data exchange dictionary depends on whether the
+source node is a publisher (i.e., the relationship is ``publish_files``)
+or a subscriber (i.e., the relationship is ``subscribe_to_files``).
+
+For the ``publish_files`` relationship, the data exchange dictionary has
+the following properties:
+
++----------------+----------------------+--------------------------------------------------------------------------------------------------+
+| Property       | Set by               | Description                                                                                      |
++================+======================+==================================================================================================+
+| location       | source node plugin   | the DMaaP location for the publisher, used to set up routing                                     |
++----------------+----------------------+--------------------------------------------------------------------------------------------------+
+| publish\_url   | DMaaP plugin         | the URL to which the publisher makes Data Router publish requests                                |
++----------------+----------------------+--------------------------------------------------------------------------------------------------+
+| log\_url       | DMaaP plugin         | the URL from which log data for the feed can be obtained                                         |
++----------------+----------------------+--------------------------------------------------------------------------------------------------+
+| username       | DMaaP plugin         | the username (generated by the DMaaP plugin) the publisher uses to authenticate to Data Router   |
++----------------+----------------------+--------------------------------------------------------------------------------------------------+
+| password       | DMaaP plugin         | the password (generated by the DMaaP plugin) the publisher uses to authenticate to Data Router   |
++----------------+----------------------+--------------------------------------------------------------------------------------------------+
+
+For the ``subscribe_to_files`` relationship, the data exchange
+dictionary has the following properties:
+
++-----------------+----------------------+-----------------------------------------------------------------------------------------+
+| Property        | Set by               | Description                                                                             |
++=================+======================+=========================================================================================+
+| location        | source node plugin   | the DMaaP location for the subscriber, used to set up routing                           |
++-----------------+----------------------+-----------------------------------------------------------------------------------------+
+| delivery\_url   | source node plugin   | the URL to which the Data Router should deliver files                                   |
++-----------------+----------------------+-----------------------------------------------------------------------------------------+
+| username        | source node plugin   | the username Data Router uses to authenticate to the subscriber when delivering files   |
++-----------------+----------------------+-----------------------------------------------------------------------------------------+
+| password        | source node plugin   | the username Data Router uses to authenticate to the subscriber when delivering file    |
++-----------------+----------------------+-----------------------------------------------------------------------------------------+
+
+Plugin Support for DMaaP Message Router
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Plugin Types for DMaaP Message Router
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The Cloudify type definitions for DMaaP Message Router nodes and
+relationships are defined in ```dmaap.yaml`` <./dmaap.yaml>`__.
+
+There are two node types for DMaaP Message Router:
+
+-  ``ccsdk.nodes.Topic``: This type represents a topic that does not yet
+   exist and that should be created when the install workflow is run
+   against a blueprint that contains a node of this type.
+
++----------------------+-----------+-------------+-----------------------------------------------------------------------------+
+| Property             | Type      | Required?   | Description                                                                 |
++======================+===========+=============+=============================================================================+
+| topic\_name          | string    | no          | a name that uniquely identifies the feed (plugin will generate if absent)   |
++----------------------+-----------+-------------+-----------------------------------------------------------------------------+
+| topic\_description   | string    | no          | human-readable description of the feed                                      |
++----------------------+-----------+-------------+-----------------------------------------------------------------------------+
+| txenable             | boolean   | no          | flag indicating whether transactions are enabled for this topic             |
++----------------------+-----------+-------------+-----------------------------------------------------------------------------+
+| replication\_case    | string    | no          | type of replication required for the topic (defaults to no replication)     |
++----------------------+-----------+-------------+-----------------------------------------------------------------------------+
+| global\_mr\_url      | string    | no          | Global MR host name for replication to a global MR instance                 |
++----------------------+-----------+-------------+-----------------------------------------------------------------------------+
+
+Note: In order to set up topics, a user should be familiar with message
+router and how it is configured, and this README is not the place to
+explain the details of message router. Here are a couple of pieces of
+information that might be helpful. Currently, the allowed values for
+``replication_case`` are:
+
+-  ``REPLICATION_NONE``
+-  ``REPLICATION_EDGE_TO_CENTRAL``
+-  ``REPLICATION_EDGE_TO_CENTRAL_TO_GLOBAL``
+-  ``REPLICATION_CENTRAL_TO_EDGE``
+-  ``REPLICATION_CENTRAL_TO_GLOBAL``
+-  ``REPLICATION_GLOBAL_TO_CENTRAL``
+-  ``REPLICATION_GLOBAL_TO_CENTRAL_TO_EDGE``
+
+The ``global_mr_url`` is actually a host name, not a full URL. It points
+to a host in a global message router cluster. (A 'global' message router
+cluster is one that's not part of ONAP.)
+
+-  ``ccsdk.nodes.ExistingTopic``: This type represents a topic that
+   already exists. Nodes of this type are placed in a blueprint so that
+   other nodes in the blueprint can be set up as publishers or
+   subscribers to the topic. The table below shows the properties that a
+   node of this type may have.
+
++------------+----------+-------------+--------------------------------------------+
+| Property   | Type     | Required?   | Description                                |
++============+==========+=============+============================================+
+| fqtn       | string   | yes         | fully-qualified topic name for the topic   |
++------------+----------+-------------+--------------------------------------------+
+
+Interaction with Other Plugins
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When creating a new topic or processing a reference to an existing
+topic, the plugin operates independently of other plugins.
+
+When processing a ``ccsdk.relationships.publish_events`` relationship or
+a ``ccsdk.relationships.subscribe_to_events`` relationship, this plugin
+needs to obtain data from and provide data to the source node. Certain
+conventions are therefore needed for passing data between this plugin
+and the plugins responsible for the source nodes in these relationships.
+In Cloudify, the mechanism for sharing data among plugins is the
+``ctx.instance.runtime_properties`` dictionary associated with each
+node.
+
+A given source node may have relationships with several topics. For
+example, an ONAP DCAE analytics module might subscribe to one topic to
+get input for its processing and publish its results to a different
+topic. When this DMaaP plugin and the plugin for the source node
+exchange information, they need to do in a way that lets them
+distinguish among different feeds. We do this through a simple
+convention: for each source node to topic relationship, the source node
+plugin will create a property in the source node's
+``runtime_properties`` dictionary. The name of the property will be the
+same as the name of the target node of the relationship. For instance,
+if a node has a ``publishes_events`` relationship with a target node
+named ``topic00``, then the plugin that's responsible for managing the
+source node with create an entry in the source node's
+``runtime_properties`` dictionary named ``topic00``. This entry itself
+will be a dictionary.
+
+For both types of relationship, the data exchange dictionary has the
+following properties:
+
++----------------+----------------------+----------------------------------------------------------------------------------+
+| Property       | Set by               | Description                                                                      |
++================+======================+==================================================================================+
+| location       | source node plugin   | the DMaaP location for the publisher or subscriber, used to set up routing       |
++----------------+----------------------+----------------------------------------------------------------------------------+
+| client\_role   | source node plugin   | the AAF client role that's requesting publish or subscribe access to the topic   |
++----------------+----------------------+----------------------------------------------------------------------------------+
+| topic\_url     | DMaaP plugin         | the URL for accessing the topic to publish or receive events                     |
++----------------+----------------------+----------------------------------------------------------------------------------+
+
+Interaction with Consul configuration store
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In addition to storing the results of DMaaP Data Router and DMaaP
+Message Router provisioning operations in ``runtime_properties``, the
+DMaaP plugin also stores these results into the ONAP configuration
+store, which resides in a `Consul key-value
+store <https://www.consul.io/>`__. This allows DMaaP clients (components
+that act as publishers, subscribers, or both) to retrieve their DMaaP
+configuration information from Consul, rather than having the plugin
+that deploys the client directly configure the client using data in
+``runtime_properties``.
+
+The ``runtime_properties`` for a client must contain a property called
+``service_component_name``. If this property is not present, the plugin
+will raise a NonRecoverableError and cause the installation to fail.
+
+If ``service_component_name`` is present, then the plugin will use a
+Consul key consisting of the value of ``service_component_name``
+prepended to the fixed string ``:dmaap``. For example, if the
+``service_component_name`` is ``client123``, the plugin will use
+``client123:dmaap`` as the key for storing DMaaP information into
+Consul. Information for all of the feeds and topics for a client are
+stored under the same key.
+
+The value stored is a nested JSON object. At the top level of the object
+are properties representing each topic or feed for which the component
+is a publisher or subscriber. The name of the property is the node name
+of the target feed or topic. The value of the property is another JSON
+object that corresponds to the dictionary that the plugin created in
+``runtime_properties`` corresponding to the target feed or topic. Note
+that the information in Consul includes all of the properties for the
+feed or topic, those set by the source node plugin as well as those set
+by the DMaaP plugin.
+
+Examples:
+
+Data Router publisher, target feed ``feed00``:
+
+::
+
+    {
+      "feed00": {
+        "username": "rC9QR51I",
+        "log_url": "https://dmaap.example.com/feedlog/972",
+        "publish_url": "https://dmaap.example.com/publish/972",
+        "location": "loc00",
+        "password": "QOQeUh5KLR",
+        "publisher_id": "972.360gm"
+      }
+    }
+
+Data Router subscriber, target feed ``feed01``:
+
+::
+
+    {
+      "feed01": {
+        "username": "drdeliver",
+        "password": "1loveDataR0uter",
+        "location": "loc00",
+        "delivery_url": "https://example.com/whatever",
+        "subscriber_id": "1550"
+      }
+    }
+
+Message Router publisher to ``topic00``, subscriber to ``topic01``. Note
+how each topic appears as a top-level property in the object.
+
+::
+
+    {
+      "topic00": {
+        "topic_url": "https://dmaap.example.com:3905/events/org.onap.ccsdk.dmaap.FTL2.outboundx",
+        "client_role": "org.onap.ccsdk.member",
+        "location": "loc00",
+        "client_id": "1494621774522"
+      },
+      "topic01": {
+        "topic_url": "https://dmaap.example.com:3905/events/org.onap.ccsdk.dmaap.FTL2.inboundx",
+        "client_role": "org.onap.ccsdk.member",
+        "location": "loc00",
+        "client_id": "1494621778627"
+      }
+    }
+
+Packaging and installing
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The DMaaP plugin is meant to be used as a `Cloudify managed
+plugin <http://docs.getcloudify.org/3.4.0/plugins/using-plugins/>`__.
+Managed plugins are packaged using
+```wagon`` <https://github.com/cloudify-cosmo/wagon>`__.
+
+To package this plugin, executing the following command in the top-level
+directory of this plugin, from a Python environment in which ``wagon``
+has been installed:
+
+::
+
+    wagon create -s . -r -o /path/to/directory/for/wagon/output
+
+Once the wagon file is built, it can be uploaded to a Cloudify Manager
+host using the ``cfy plugins upload`` command described in the
+documentation above.
+
+Managed plugins can also be loaded at the time a Cloudify Manager host
+is installed, via the installation blueprint and inputs file. We expect
+that this plugin will be loaded at Cloudify Manager installation time,
+and that ``cfy plugins upload`` will be used only for delivering patches
+between releases.
+
+Configuration
+~~~~~~~~~~~~~
+
+The plugin needs to be configured with certain parameters needed to
+access the DMaaP Bus Controller. In keeping with the ONAP architecture,
+this information is stored in Consul.
+
+The plugin finds the address and port of the DMaaP Bus Controller using
+the Consul service discovery facility. The plugin expects the Bus
+Controller to be registered under the name ``dmaap_bus_controller``.
+
+Additional parameters come from the ``dmaap`` key in the Cloudify
+Manager's Consul configuration, which is stored in the Consul KV store
+under the key name 'cloudify\_manager'. The table below lists the
+properties in the configuration:
+
++----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+
+| Property       | Type     | Required?   | Default      | Description                                                                                 |
++================+==========+=============+==============+=============================================================================================+
+| ``username``   | string   | Yes         | (none)       | The username for logging into DMaaP Bus Controller                                          |
++----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+
+| ``password``   | string   | Yes         | (none)       | The password for logging into DMaaP Bus Controller                                          |
++----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+
+| ``owner``      | string   | Yes         | (none)       | The name to be used as the owner for entities created by the plugin                         |
++----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+
+| ``protocol``   | string   | No          | ``https``    | The protocol (URL scheme) used to access the DMaaP bus controller (``http`` or ``https``)   |
++----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+
+| ``path``       | string   | No          | ``webapi``   | The path to the root of the DMaaP Bus Controller API endpoint                               |
++----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+
+
+Here is an example of a Cloudify Manager configuration object showing
+only the ``dmaap`` key:
+
+::
+
+    {
+      "dmaap": {
+        "username": "dmaap.client@ccsdkorch.onap.org",
+        "password": "guessmeifyoucan"
+        "owner": "ccsdkorc"
+      },
+
+      (other configuration here)
+
+    }
+