path: root/docs/dmaap.rst

Cloudify DMaaP Plugin
---------------------

Cloudify plugin for creating and managing DMaaP Data Router feeds and
subscriptions and DMaaP Message Router topics. The plugin uses the DMaaP
Bus Controller API.

Plugin Support for DMaaP Data Router
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Plugin Types for DMaaP Data Router
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The Cloudify type definitions for DMaaP Data Router nodes and
relationships are defined in ```dmaap.yaml`` <./dmaap.yaml>`__.

There are four node types for DMaaP Data Router:

-  ``ccsdk.nodes.Feed``: This type represents a feed that does not yet
   exist and that should be created when the install workflow is run
   against a blueprint that contains a node of this type.

+------------------------+----------+-------------+---------------------------------------------------------------------------------------------+
| Property               | Type     | Required?   | Description                                                                                 |
+========================+==========+=============+=============================================================================================+
| feed\_name             | string   | no          | a name that identifies the feed (plugin will generate if absent)                            |
+------------------------+----------+-------------+---------------------------------------------------------------------------------------------+
| feed\_version          | string   | no          | version number for the feed (feed\_name + feed\_version uniquely identify the feed in DR)   |
+------------------------+----------+-------------+---------------------------------------------------------------------------------------------+
| feed\_description      | string   | no          | human-readable description of the feed                                                      |
+------------------------+----------+-------------+---------------------------------------------------------------------------------------------+
| aspr\_classification   | string   | no          | AT&T ASPR classification of the feed                                                        |
+------------------------+----------+-------------+---------------------------------------------------------------------------------------------+

-  ``ccsdk.nodes.ExistingFeed``: This type represents a feed that
   already exists. Nodes of this type are placed in a blueprint so that
   other nodes in the blueprint can be set up as publishers or
   subscribers to the feed. The table below shows the properties that a
   node of this type may have.

+------------+----------+-------------+---------------------------------------------------------------+
| Property   | Type     | Required?   | Description                                                   |
+============+==========+=============+===============================================================+
| feed\_id   | string   | yes         | Feed identifier assigned by DMaaP when the feed was created   |
+------------+----------+-------------+---------------------------------------------------------------+

-  ``ccsdk.nodes.ExternalTargetFeed``: This type represents a feed
   created in an external DMaaP environment (i.e., an environment that
   the plugin cannot access to make provisioning requests, such as a
   shared corporate system). Nodes of this type are placed in a
   blueprint so that other feed nodes of type ``ccsdk.nodes.Feed`` or
   ``ccsdk.nodes.ExistingFeed`` can be set up to "bridge" to external
   feeds by publishing data to the external feeds. The table below shows
   the properties that a node of this type may have.

+------------+----------+-------------+----------------------------------------------------------------+
| Property   | Type     | Required?   | Description                                                    |
+============+==========+=============+================================================================+
| url        | string   | yes         | The publish URL of the external feed.                          |
+------------+----------+-------------+----------------------------------------------------------------+
| username   | string   | yes         | The username to be used when delivering to the external feed   |
+------------+----------+-------------+----------------------------------------------------------------+
| userpw     | string   | yes         | The password to be used when delivering to the external feed   |
+------------+----------+-------------+----------------------------------------------------------------+

*Note: These properties are usually obtained by manually creating a feed
in the external DMaaP DR system and then creating a publisher for that
feed.*

-  ``ccsdk.nodes.ExternalSourceFeed``: This type represents a feed
   created in an external DMaaP environment (i.e., an environment that
   the plugin cannot access to makes provisioning requests, such as a
   shared corporate system). Nodes of this type are place in a blueprint
   so that they can be set up to "bridge" to other feed nodes of type
   ``ccsdk.nodes.Feed`` or ``ccsdk.nodes.ExistingFeed``. This type has
   no node properties, but when a bridge is set up, the url, username,
   and password are attached to the node as runtime\_properties, using
   the name of the target feed node as the top-level key.

There are five relationship types for DMaaP Data Router:

-  ``ccsdk.relationships.publish_files``, used to indicate that the
   relationship's source node sends is a publisher to the Data Router
   feed represented by the relationship's target node.
-  ``ccsdk.relationships.subscribe_to_files``, used to indicate that the
   relationship's source node is a subscriber to the Data Router feed
   represented by the relationship's target node.
-  ``ccsdk.relationships.bridges_to``, used to indicate that the
   relationship's source node (a ``ccsdk.nodes.Feed`` or
   ``ccsdk.nodes.ExistingFeed``) should be set up to forward data
   ("bridge") to the relationship's target feed (another
   ``ccsdk.nodes.Feed`` or ``ccsdk.nodes.ExistingFeed``).
-  ``ccsdk.relationships.bridges_to_external``, used to indicate that
   the relationship's source node (a ``ccsdk.nodes.Feed`` or
   ``ccsdk.nodes.ExistingFeed``) should be set up to forward data
   ("bridge") to the relationship's target node (a feed in an external
   DMaaP system, represented by a ``ccsdk.nodes.ExternalTargetFeed``
   node).
-  ``ccsdk.relationships.bridges_from_external_to_internal``, used to
   indicate the the relationship's source node (a feed in an external
   DMaaP system, represented by a ``ccsdk.nodes.ExternalSourceFeed``
   node) should be set up to forward date ("bridge") to the
   relationship's target node (an internal ONAP feed, represented by a
   ``ccsdk.nodes.Feed`` or ``ccsdk.nodes.ExistingFeed`` node).

The plugin code implements the lifecycle operations needed to create and
delete feeds and to add and remove publishers and subscribers. It also
implements the operations needed to set up bridging between feeds.

Interaction with Other Plugins
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

When creating a new feed or processing a reference to an existing feed,
the plugin operates independently of other plugins.

When processing a ``ccsdk.relationships.publish_files`` relationship or
a ``ccsdk.relationships.subscribe_to_files`` relationship, this plugin
needs to obtain data from the source node and, in the case of
``publish_files``, provide data to the source node. Certain conventions
are therefore needed for passing data between this plugin and the
plugins responsible for the source nodes in these relationships. In
Cloudify, the mechanism for sharing data among plugins is the
``ctx.instance.runtime_properties`` dictionary associated with each
node.

A given source node may have relationships with several feeds. For
example, an ONAP DCAE data collector might publish two different types
of data to two different feeds. An ONAP DCAE analytics module might
subscribe to one feed to get input for its processing and publish its
results to a different feed. When this DMaaP plugin and the plugin for
the source node exchange information, they need to do in a way that lets
them distinguish among different feeds. We do this through a simple
convention: for each source node to feed relationship, the source node
plugin will create a property in the source node's
``runtime_properties`` dictionary. The name of the property will be the
same as the name of the target node of the relationship. For instance,
if a node has a ``publishes_files`` relationship with a target node
named ``feed00``, then the plugin that's responsible for managing the
source node with create an entry in the source node's
``runtime_properties`` dictionary named ``feed00``. This entry itself
will be a dictionary.

The content of this data exchange dictionary depends on whether the
source node is a publisher (i.e., the relationship is ``publish_files``)
or a subscriber (i.e., the relationship is ``subscribe_to_files``).

For the ``publish_files`` relationship, the data exchange dictionary has
the following properties:

+----------------+----------------------+--------------------------------------------------------------------------------------------------+
| Property       | Set by               | Description                                                                                      |
+================+======================+==================================================================================================+
| location       | source node plugin   | the DMaaP location for the publisher, used to set up routing                                     |
+----------------+----------------------+--------------------------------------------------------------------------------------------------+
| publish\_url   | DMaaP plugin         | the URL to which the publisher makes Data Router publish requests                                |
+----------------+----------------------+--------------------------------------------------------------------------------------------------+
| log\_url       | DMaaP plugin         | the URL from which log data for the feed can be obtained                                         |
+----------------+----------------------+--------------------------------------------------------------------------------------------------+
| username       | DMaaP plugin         | the username (generated by the DMaaP plugin) the publisher uses to authenticate to Data Router   |
+----------------+----------------------+--------------------------------------------------------------------------------------------------+
| password       | DMaaP plugin         | the password (generated by the DMaaP plugin) the publisher uses to authenticate to Data Router   |
+----------------+----------------------+--------------------------------------------------------------------------------------------------+

For the ``subscribe_to_files`` relationship, the data exchange
dictionary has the following properties:

+-----------------+----------------------+-----------------------------------------------------------------------------------------+
| Property        | Set by               | Description                                                                             |
+=================+======================+=========================================================================================+
| location        | source node plugin   | the DMaaP location for the subscriber, used to set up routing                           |
+-----------------+----------------------+-----------------------------------------------------------------------------------------+
| delivery\_url   | source node plugin   | the URL to which the Data Router should deliver files                                   |
+-----------------+----------------------+-----------------------------------------------------------------------------------------+
| username        | source node plugin   | the username Data Router uses to authenticate to the subscriber when delivering files   |
+-----------------+----------------------+-----------------------------------------------------------------------------------------+
| password        | source node plugin   | the username Data Router uses to authenticate to the subscriber when delivering file    |
+-----------------+----------------------+-----------------------------------------------------------------------------------------+

Plugin Support for DMaaP Message Router
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Plugin Types for DMaaP Message Router
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The Cloudify type definitions for DMaaP Message Router nodes and
relationships are defined in ```dmaap.yaml`` <./dmaap.yaml>`__.

There are two node types for DMaaP Message Router:

-  ``ccsdk.nodes.Topic``: This type represents a topic that does not yet
   exist and that should be created when the install workflow is run
   against a blueprint that contains a node of this type.

+----------------------+-----------+-------------+-----------------------------------------------------------------------------+
| Property             | Type      | Required?   | Description                                                                 |
+======================+===========+=============+=============================================================================+
| topic\_name          | string    | no          | a name that uniquely identifies the feed (plugin will generate if absent)   |
+----------------------+-----------+-------------+-----------------------------------------------------------------------------+
| topic\_description   | string    | no          | human-readable description of the feed                                      |
+----------------------+-----------+-------------+-----------------------------------------------------------------------------+
| txenable             | boolean   | no          | flag indicating whether transactions are enabled for this topic             |
+----------------------+-----------+-------------+-----------------------------------------------------------------------------+
| replication\_case    | string    | no          | type of replication required for the topic (defaults to no replication)     |
+----------------------+-----------+-------------+-----------------------------------------------------------------------------+
| global\_mr\_url      | string    | no          | Global MR host name for replication to a global MR instance                 |
+----------------------+-----------+-------------+-----------------------------------------------------------------------------+

Note: In order to set up topics, a user should be familiar with message
router and how it is configured, and this README is not the place to
explain the details of message router. Here are a couple of pieces of
information that might be helpful. Currently, the allowed values for
``replication_case`` are:

-  ``REPLICATION_NONE``
-  ``REPLICATION_EDGE_TO_CENTRAL``
-  ``REPLICATION_EDGE_TO_CENTRAL_TO_GLOBAL``
-  ``REPLICATION_CENTRAL_TO_EDGE``
-  ``REPLICATION_CENTRAL_TO_GLOBAL``
-  ``REPLICATION_GLOBAL_TO_CENTRAL``
-  ``REPLICATION_GLOBAL_TO_CENTRAL_TO_EDGE``

The ``global_mr_url`` is actually a host name, not a full URL. It points
to a host in a global message router cluster. (A 'global' message router
cluster is one that's not part of ONAP.)

-  ``ccsdk.nodes.ExistingTopic``: This type represents a topic that
   already exists. Nodes of this type are placed in a blueprint so that
   other nodes in the blueprint can be set up as publishers or
   subscribers to the topic. The table below shows the properties that a
   node of this type may have.

+------------+----------+-------------+--------------------------------------------+
| Property   | Type     | Required?   | Description                                |
+============+==========+=============+============================================+
| fqtn       | string   | yes         | fully-qualified topic name for the topic   |
+------------+----------+-------------+--------------------------------------------+

Interaction with Other Plugins
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

When creating a new topic or processing a reference to an existing
topic, the plugin operates independently of other plugins.

When processing a ``ccsdk.relationships.publish_events`` relationship or
a ``ccsdk.relationships.subscribe_to_events`` relationship, this plugin
needs to obtain data from and provide data to the source node. Certain
conventions are therefore needed for passing data between this plugin
and the plugins responsible for the source nodes in these relationships.
In Cloudify, the mechanism for sharing data among plugins is the
``ctx.instance.runtime_properties`` dictionary associated with each
node.

A given source node may have relationships with several topics. For
example, an ONAP DCAE analytics module might subscribe to one topic to
get input for its processing and publish its results to a different
topic. When this DMaaP plugin and the plugin for the source node
exchange information, they need to do in a way that lets them
distinguish among different feeds. We do this through a simple
convention: for each source node to topic relationship, the source node
plugin will create a property in the source node's
``runtime_properties`` dictionary. The name of the property will be the
same as the name of the target node of the relationship. For instance,
if a node has a ``publishes_events`` relationship with a target node
named ``topic00``, then the plugin that's responsible for managing the
source node with create an entry in the source node's
``runtime_properties`` dictionary named ``topic00``. This entry itself
will be a dictionary.

For both types of relationship, the data exchange dictionary has the
following properties:

+----------------+----------------------+----------------------------------------------------------------------------------+
| Property       | Set by               | Description                                                                      |
+================+======================+==================================================================================+
| location       | source node plugin   | the DMaaP location for the publisher or subscriber, used to set up routing       |
+----------------+----------------------+----------------------------------------------------------------------------------+
| client\_role   | source node plugin   | the AAF client role that's requesting publish or subscribe access to the topic   |
+----------------+----------------------+----------------------------------------------------------------------------------+
| topic\_url     | DMaaP plugin         | the URL for accessing the topic to publish or receive events                     |
+----------------+----------------------+----------------------------------------------------------------------------------+

Interaction with Consul configuration store
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In addition to storing the results of DMaaP Data Router and DMaaP
Message Router provisioning operations in ``runtime_properties``, the
DMaaP plugin also stores these results into the ONAP configuration
store, which resides in a `Consul key-value
store <https://www.consul.io/>`__. This allows DMaaP clients (components
that act as publishers, subscribers, or both) to retrieve their DMaaP
configuration information from Consul, rather than having the plugin
that deploys the client directly configure the client using data in
``runtime_properties``.

The ``runtime_properties`` for a client must contain a property called
``service_component_name``. If this property is not present, the plugin
will raise a NonRecoverableError and cause the installation to fail.

If ``service_component_name`` is present, then the plugin will use a
Consul key consisting of the value of ``service_component_name``
prepended to the fixed string ``:dmaap``. For example, if the
``service_component_name`` is ``client123``, the plugin will use
``client123:dmaap`` as the key for storing DMaaP information into
Consul. Information for all of the feeds and topics for a client are
stored under the same key.

The value stored is a nested JSON object. At the top level of the object
are properties representing each topic or feed for which the component
is a publisher or subscriber. The name of the property is the node name
of the target feed or topic. The value of the property is another JSON
object that corresponds to the dictionary that the plugin created in
``runtime_properties`` corresponding to the target feed or topic. Note
that the information in Consul includes all of the properties for the
feed or topic, those set by the source node plugin as well as those set
by the DMaaP plugin.

Examples:

Data Router publisher, target feed ``feed00``:

::

    {
      "feed00": {
        "username": "rC9QR51I",
        "log_url": "https://dmaap.example.com/feedlog/972",
        "publish_url": "https://dmaap.example.com/publish/972",
        "location": "loc00",
        "password": "QOQeUh5KLR",
        "publisher_id": "972.360gm"
      }
    }

Data Router subscriber, target feed ``feed01``:

::

    {
      "feed01": {
        "username": "drdeliver",
        "password": "1loveDataR0uter",
        "location": "loc00",
        "delivery_url": "https://example.com/whatever",
        "subscriber_id": "1550"
      }
    }

Message Router publisher to ``topic00``, subscriber to ``topic01``. Note
how each topic appears as a top-level property in the object.

::

    {
      "topic00": {
        "topic_url": "https://dmaap.example.com:3905/events/org.onap.ccsdk.dmaap.FTL2.outboundx",
        "client_role": "org.onap.ccsdk.member",
        "location": "loc00",
        "client_id": "1494621774522"
      },
      "topic01": {
        "topic_url": "https://dmaap.example.com:3905/events/org.onap.ccsdk.dmaap.FTL2.inboundx",
        "client_role": "org.onap.ccsdk.member",
        "location": "loc00",
        "client_id": "1494621778627"
      }
    }

Packaging and installing
~~~~~~~~~~~~~~~~~~~~~~~~

The DMaaP plugin is meant to be used as a `Cloudify managed
plugin <http://docs.getcloudify.org/3.4.0/plugins/using-plugins/>`__.
Managed plugins are packaged using
```wagon`` <https://github.com/cloudify-cosmo/wagon>`__.

To package this plugin, executing the following command in the top-level
directory of this plugin, from a Python environment in which ``wagon``
has been installed:

::

    wagon create -s . -r -o /path/to/directory/for/wagon/output

Once the wagon file is built, it can be uploaded to a Cloudify Manager
host using the ``cfy plugins upload`` command described in the
documentation above.

Managed plugins can also be loaded at the time a Cloudify Manager host
is installed, via the installation blueprint and inputs file. We expect
that this plugin will be loaded at Cloudify Manager installation time,
and that ``cfy plugins upload`` will be used only for delivering patches
between releases.

Configuration
~~~~~~~~~~~~~

The plugin needs to be configured with certain parameters needed to
access the DMaaP Bus Controller. In keeping with the ONAP architecture,
this information is stored in Consul.

The plugin finds the address and port of the DMaaP Bus Controller using
the Consul service discovery facility. The plugin expects the Bus
Controller to be registered under the name ``dmaap_bus_controller``.

Additional parameters come from the ``dmaap`` key in the Cloudify
Manager's Consul configuration, which is stored in the Consul KV store
under the key name 'cloudify\_manager'. The table below lists the
properties in the configuration:

+----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+
| Property       | Type     | Required?   | Default      | Description                                                                                 |
+================+==========+=============+==============+=============================================================================================+
| ``username``   | string   | Yes         | (none)       | The username for logging into DMaaP Bus Controller                                          |
+----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+
| ``password``   | string   | Yes         | (none)       | The password for logging into DMaaP Bus Controller                                          |
+----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+
| ``owner``      | string   | Yes         | (none)       | The name to be used as the owner for entities created by the plugin                         |
+----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+
| ``protocol``   | string   | No          | ``https``    | The protocol (URL scheme) used to access the DMaaP bus controller (``http`` or ``https``)   |
+----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+
| ``path``       | string   | No          | ``webapi``   | The path to the root of the DMaaP Bus Controller API endpoint                               |
+----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+

Here is an example of a Cloudify Manager configuration object showing
only the ``dmaap`` key:

::

    {
      "dmaap": {
        "username": "dmaap.client@ccsdkorch.onap.org",
        "password": "guessmeifyoucan"
        "owner": "ccsdkorc"
      },

      (other configuration here)

    }