diff options
Diffstat (limited to 'docs/dmaap.rst')
| -rw-r--r-- | docs/dmaap.rst | 439 |
1 files changed, 439 insertions, 0 deletions
diff --git a/docs/dmaap.rst b/docs/dmaap.rst new file mode 100644 index 0000000..b49eb4f --- /dev/null +++ b/docs/dmaap.rst @@ -0,0 +1,439 @@ +Cloudify DMaaP Plugin +--------------------- + +Cloudify plugin for creating and managing DMaaP Data Router feeds and +subscriptions and DMaaP Message Router topics. The plugin uses the DMaaP +Bus Controller API. + +Plugin Support for DMaaP Data Router +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Plugin Types for DMaaP Data Router +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The Cloudify type definitions for DMaaP Data Router nodes and +relationships are defined in ```dmaap.yaml`` <./dmaap.yaml>`__. + +There are four node types for DMaaP Data Router: + +- ``ccsdk.nodes.Feed``: This type represents a feed that does not yet + exist and that should be created when the install workflow is run + against a blueprint that contains a node of this type. + ++------------------------+----------+-------------+---------------------------------------------------------------------------------------------+ +| Property | Type | Required? | Description | ++========================+==========+=============+=============================================================================================+ +| feed\_name | string | no | a name that identifies the feed (plugin will generate if absent) | ++------------------------+----------+-------------+---------------------------------------------------------------------------------------------+ +| feed\_version | string | no | version number for the feed (feed\_name + feed\_version uniquely identify the feed in DR) | ++------------------------+----------+-------------+---------------------------------------------------------------------------------------------+ +| feed\_description | string | no | human-readable description of the feed | ++------------------------+----------+-------------+---------------------------------------------------------------------------------------------+ +| aspr\_classification | string | no | AT&T ASPR classification of the feed | ++------------------------+----------+-------------+---------------------------------------------------------------------------------------------+ + +- ``ccsdk.nodes.ExistingFeed``: This type represents a feed that + already exists. Nodes of this type are placed in a blueprint so that + other nodes in the blueprint can be set up as publishers or + subscribers to the feed. The table below shows the properties that a + node of this type may have. + ++------------+----------+-------------+---------------------------------------------------------------+ +| Property | Type | Required? | Description | ++============+==========+=============+===============================================================+ +| feed\_id | string | yes | Feed identifier assigned by DMaaP when the feed was created | ++------------+----------+-------------+---------------------------------------------------------------+ + +- ``ccsdk.nodes.ExternalTargetFeed``: This type represents a feed + created in an external DMaaP environment (i.e., an environment that + the plugin cannot access to make provisioning requests, such as a + shared corporate system). Nodes of this type are placed in a + blueprint so that other feed nodes of type ``ccsdk.nodes.Feed`` or + ``ccsdk.nodes.ExistingFeed`` can be set up to "bridge" to external + feeds by publishing data to the external feeds. The table below shows + the properties that a node of this type may have. + ++------------+----------+-------------+----------------------------------------------------------------+ +| Property | Type | Required? | Description | ++============+==========+=============+================================================================+ +| url | string | yes | The publish URL of the external feed. | ++------------+----------+-------------+----------------------------------------------------------------+ +| username | string | yes | The username to be used when delivering to the external feed | ++------------+----------+-------------+----------------------------------------------------------------+ +| userpw | string | yes | The password to be used when delivering to the external feed | ++------------+----------+-------------+----------------------------------------------------------------+ + +*Note: These properties are usually obtained by manually creating a feed +in the external DMaaP DR system and then creating a publisher for that +feed.* + +- ``ccsdk.nodes.ExternalSourceFeed``: This type represents a feed + created in an external DMaaP environment (i.e., an environment that + the plugin cannot access to makes provisioning requests, such as a + shared corporate system). Nodes of this type are place in a blueprint + so that they can be set up to "bridge" to other feed nodes of type + ``ccsdk.nodes.Feed`` or ``ccsdk.nodes.ExistingFeed``. This type has + no node properties, but when a bridge is set up, the url, username, + and password are attached to the node as runtime\_properties, using + the name of the target feed node as the top-level key. + +There are five relationship types for DMaaP Data Router: + +- ``ccsdk.relationships.publish_files``, used to indicate that the + relationship's source node sends is a publisher to the Data Router + feed represented by the relationship's target node. +- ``ccsdk.relationships.subscribe_to_files``, used to indicate that the + relationship's source node is a subscriber to the Data Router feed + represented by the relationship's target node. +- ``ccsdk.relationships.bridges_to``, used to indicate that the + relationship's source node (a ``ccsdk.nodes.Feed`` or + ``ccsdk.nodes.ExistingFeed``) should be set up to forward data + ("bridge") to the relationship's target feed (another + ``ccsdk.nodes.Feed`` or ``ccsdk.nodes.ExistingFeed``). +- ``ccsdk.relationships.bridges_to_external``, used to indicate that + the relationship's source node (a ``ccsdk.nodes.Feed`` or + ``ccsdk.nodes.ExistingFeed``) should be set up to forward data + ("bridge") to the relationship's target node (a feed in an external + DMaaP system, represented by a ``ccsdk.nodes.ExternalTargetFeed`` + node). +- ``ccsdk.relationships.bridges_from_external_to_internal``, used to + indicate the the relationship's source node (a feed in an external + DMaaP system, represented by a ``ccsdk.nodes.ExternalSourceFeed`` + node) should be set up to forward date ("bridge") to the + relationship's target node (an internal ONAP feed, represented by a + ``ccsdk.nodes.Feed`` or ``ccsdk.nodes.ExistingFeed`` node). + +The plugin code implements the lifecycle operations needed to create and +delete feeds and to add and remove publishers and subscribers. It also +implements the operations needed to set up bridging between feeds. + +Interaction with Other Plugins +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When creating a new feed or processing a reference to an existing feed, +the plugin operates independently of other plugins. + +When processing a ``ccsdk.relationships.publish_files`` relationship or +a ``ccsdk.relationships.subscribe_to_files`` relationship, this plugin +needs to obtain data from the source node and, in the case of +``publish_files``, provide data to the source node. Certain conventions +are therefore needed for passing data between this plugin and the +plugins responsible for the source nodes in these relationships. In +Cloudify, the mechanism for sharing data among plugins is the +``ctx.instance.runtime_properties`` dictionary associated with each +node. + +A given source node may have relationships with several feeds. For +example, an ONAP DCAE data collector might publish two different types +of data to two different feeds. An ONAP DCAE analytics module might +subscribe to one feed to get input for its processing and publish its +results to a different feed. When this DMaaP plugin and the plugin for +the source node exchange information, they need to do in a way that lets +them distinguish among different feeds. We do this through a simple +convention: for each source node to feed relationship, the source node +plugin will create a property in the source node's +``runtime_properties`` dictionary. The name of the property will be the +same as the name of the target node of the relationship. For instance, +if a node has a ``publishes_files`` relationship with a target node +named ``feed00``, then the plugin that's responsible for managing the +source node with create an entry in the source node's +``runtime_properties`` dictionary named ``feed00``. This entry itself +will be a dictionary. + +The content of this data exchange dictionary depends on whether the +source node is a publisher (i.e., the relationship is ``publish_files``) +or a subscriber (i.e., the relationship is ``subscribe_to_files``). + +For the ``publish_files`` relationship, the data exchange dictionary has +the following properties: + ++----------------+----------------------+--------------------------------------------------------------------------------------------------+ +| Property | Set by | Description | ++================+======================+==================================================================================================+ +| location | source node plugin | the DMaaP location for the publisher, used to set up routing | ++----------------+----------------------+--------------------------------------------------------------------------------------------------+ +| publish\_url | DMaaP plugin | the URL to which the publisher makes Data Router publish requests | ++----------------+----------------------+--------------------------------------------------------------------------------------------------+ +| log\_url | DMaaP plugin | the URL from which log data for the feed can be obtained | ++----------------+----------------------+--------------------------------------------------------------------------------------------------+ +| username | DMaaP plugin | the username (generated by the DMaaP plugin) the publisher uses to authenticate to Data Router | ++----------------+----------------------+--------------------------------------------------------------------------------------------------+ +| password | DMaaP plugin | the password (generated by the DMaaP plugin) the publisher uses to authenticate to Data Router | ++----------------+----------------------+--------------------------------------------------------------------------------------------------+ + +For the ``subscribe_to_files`` relationship, the data exchange +dictionary has the following properties: + ++-----------------+----------------------+-----------------------------------------------------------------------------------------+ +| Property | Set by | Description | ++=================+======================+=========================================================================================+ +| location | source node plugin | the DMaaP location for the subscriber, used to set up routing | ++-----------------+----------------------+-----------------------------------------------------------------------------------------+ +| delivery\_url | source node plugin | the URL to which the Data Router should deliver files | ++-----------------+----------------------+-----------------------------------------------------------------------------------------+ +| username | source node plugin | the username Data Router uses to authenticate to the subscriber when delivering files | ++-----------------+----------------------+-----------------------------------------------------------------------------------------+ +| password | source node plugin | the username Data Router uses to authenticate to the subscriber when delivering file | ++-----------------+----------------------+-----------------------------------------------------------------------------------------+ + +Plugin Support for DMaaP Message Router +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Plugin Types for DMaaP Message Router +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The Cloudify type definitions for DMaaP Message Router nodes and +relationships are defined in ```dmaap.yaml`` <./dmaap.yaml>`__. + +There are two node types for DMaaP Message Router: + +- ``ccsdk.nodes.Topic``: This type represents a topic that does not yet + exist and that should be created when the install workflow is run + against a blueprint that contains a node of this type. + ++----------------------+-----------+-------------+-----------------------------------------------------------------------------+ +| Property | Type | Required? | Description | ++======================+===========+=============+=============================================================================+ +| topic\_name | string | no | a name that uniquely identifies the feed (plugin will generate if absent) | ++----------------------+-----------+-------------+-----------------------------------------------------------------------------+ +| topic\_description | string | no | human-readable description of the feed | ++----------------------+-----------+-------------+-----------------------------------------------------------------------------+ +| txenable | boolean | no | flag indicating whether transactions are enabled for this topic | ++----------------------+-----------+-------------+-----------------------------------------------------------------------------+ +| replication\_case | string | no | type of replication required for the topic (defaults to no replication) | ++----------------------+-----------+-------------+-----------------------------------------------------------------------------+ +| global\_mr\_url | string | no | Global MR host name for replication to a global MR instance | ++----------------------+-----------+-------------+-----------------------------------------------------------------------------+ + +Note: In order to set up topics, a user should be familiar with message +router and how it is configured, and this README is not the place to +explain the details of message router. Here are a couple of pieces of +information that might be helpful. Currently, the allowed values for +``replication_case`` are: + +- ``REPLICATION_NONE`` +- ``REPLICATION_EDGE_TO_CENTRAL`` +- ``REPLICATION_EDGE_TO_CENTRAL_TO_GLOBAL`` +- ``REPLICATION_CENTRAL_TO_EDGE`` +- ``REPLICATION_CENTRAL_TO_GLOBAL`` +- ``REPLICATION_GLOBAL_TO_CENTRAL`` +- ``REPLICATION_GLOBAL_TO_CENTRAL_TO_EDGE`` + +The ``global_mr_url`` is actually a host name, not a full URL. It points +to a host in a global message router cluster. (A 'global' message router +cluster is one that's not part of ONAP.) + +- ``ccsdk.nodes.ExistingTopic``: This type represents a topic that + already exists. Nodes of this type are placed in a blueprint so that + other nodes in the blueprint can be set up as publishers or + subscribers to the topic. The table below shows the properties that a + node of this type may have. + ++------------+----------+-------------+--------------------------------------------+ +| Property | Type | Required? | Description | ++============+==========+=============+============================================+ +| fqtn | string | yes | fully-qualified topic name for the topic | ++------------+----------+-------------+--------------------------------------------+ + +Interaction with Other Plugins +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When creating a new topic or processing a reference to an existing +topic, the plugin operates independently of other plugins. + +When processing a ``ccsdk.relationships.publish_events`` relationship or +a ``ccsdk.relationships.subscribe_to_events`` relationship, this plugin +needs to obtain data from and provide data to the source node. Certain +conventions are therefore needed for passing data between this plugin +and the plugins responsible for the source nodes in these relationships. +In Cloudify, the mechanism for sharing data among plugins is the +``ctx.instance.runtime_properties`` dictionary associated with each +node. + +A given source node may have relationships with several topics. For +example, an ONAP DCAE analytics module might subscribe to one topic to +get input for its processing and publish its results to a different +topic. When this DMaaP plugin and the plugin for the source node +exchange information, they need to do in a way that lets them +distinguish among different feeds. We do this through a simple +convention: for each source node to topic relationship, the source node +plugin will create a property in the source node's +``runtime_properties`` dictionary. The name of the property will be the +same as the name of the target node of the relationship. For instance, +if a node has a ``publishes_events`` relationship with a target node +named ``topic00``, then the plugin that's responsible for managing the +source node with create an entry in the source node's +``runtime_properties`` dictionary named ``topic00``. This entry itself +will be a dictionary. + +For both types of relationship, the data exchange dictionary has the +following properties: + ++----------------+----------------------+----------------------------------------------------------------------------------+ +| Property | Set by | Description | ++================+======================+==================================================================================+ +| location | source node plugin | the DMaaP location for the publisher or subscriber, used to set up routing | ++----------------+----------------------+----------------------------------------------------------------------------------+ +| client\_role | source node plugin | the AAF client role that's requesting publish or subscribe access to the topic | ++----------------+----------------------+----------------------------------------------------------------------------------+ +| topic\_url | DMaaP plugin | the URL for accessing the topic to publish or receive events | ++----------------+----------------------+----------------------------------------------------------------------------------+ + +Interaction with Consul configuration store +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In addition to storing the results of DMaaP Data Router and DMaaP +Message Router provisioning operations in ``runtime_properties``, the +DMaaP plugin also stores these results into the ONAP configuration +store, which resides in a `Consul key-value +store <https://www.consul.io/>`__. This allows DMaaP clients (components +that act as publishers, subscribers, or both) to retrieve their DMaaP +configuration information from Consul, rather than having the plugin +that deploys the client directly configure the client using data in +``runtime_properties``. + +The ``runtime_properties`` for a client must contain a property called +``service_component_name``. If this property is not present, the plugin +will raise a NonRecoverableError and cause the installation to fail. + +If ``service_component_name`` is present, then the plugin will use a +Consul key consisting of the value of ``service_component_name`` +prepended to the fixed string ``:dmaap``. For example, if the +``service_component_name`` is ``client123``, the plugin will use +``client123:dmaap`` as the key for storing DMaaP information into +Consul. Information for all of the feeds and topics for a client are +stored under the same key. + +The value stored is a nested JSON object. At the top level of the object +are properties representing each topic or feed for which the component +is a publisher or subscriber. The name of the property is the node name +of the target feed or topic. The value of the property is another JSON +object that corresponds to the dictionary that the plugin created in +``runtime_properties`` corresponding to the target feed or topic. Note +that the information in Consul includes all of the properties for the +feed or topic, those set by the source node plugin as well as those set +by the DMaaP plugin. + +Examples: + +Data Router publisher, target feed ``feed00``: + +:: + + { + "feed00": { + "username": "rC9QR51I", + "log_url": "https://dmaap.example.com/feedlog/972", + "publish_url": "https://dmaap.example.com/publish/972", + "location": "loc00", + "password": "QOQeUh5KLR", + "publisher_id": "972.360gm" + } + } + +Data Router subscriber, target feed ``feed01``: + +:: + + { + "feed01": { + "username": "drdeliver", + "password": "1loveDataR0uter", + "location": "loc00", + "delivery_url": "https://example.com/whatever", + "subscriber_id": "1550" + } + } + +Message Router publisher to ``topic00``, subscriber to ``topic01``. Note +how each topic appears as a top-level property in the object. + +:: + + { + "topic00": { + "topic_url": "https://dmaap.example.com:3905/events/org.onap.ccsdk.dmaap.FTL2.outboundx", + "client_role": "org.onap.ccsdk.member", + "location": "loc00", + "client_id": "1494621774522" + }, + "topic01": { + "topic_url": "https://dmaap.example.com:3905/events/org.onap.ccsdk.dmaap.FTL2.inboundx", + "client_role": "org.onap.ccsdk.member", + "location": "loc00", + "client_id": "1494621778627" + } + } + +Packaging and installing +~~~~~~~~~~~~~~~~~~~~~~~~ + +The DMaaP plugin is meant to be used as a `Cloudify managed +plugin <http://docs.getcloudify.org/3.4.0/plugins/using-plugins/>`__. +Managed plugins are packaged using +```wagon`` <https://github.com/cloudify-cosmo/wagon>`__. + +To package this plugin, executing the following command in the top-level +directory of this plugin, from a Python environment in which ``wagon`` +has been installed: + +:: + + wagon create -s . -r -o /path/to/directory/for/wagon/output + +Once the wagon file is built, it can be uploaded to a Cloudify Manager +host using the ``cfy plugins upload`` command described in the +documentation above. + +Managed plugins can also be loaded at the time a Cloudify Manager host +is installed, via the installation blueprint and inputs file. We expect +that this plugin will be loaded at Cloudify Manager installation time, +and that ``cfy plugins upload`` will be used only for delivering patches +between releases. + +Configuration +~~~~~~~~~~~~~ + +The plugin needs to be configured with certain parameters needed to +access the DMaaP Bus Controller. In keeping with the ONAP architecture, +this information is stored in Consul. + +The plugin finds the address and port of the DMaaP Bus Controller using +the Consul service discovery facility. The plugin expects the Bus +Controller to be registered under the name ``dmaap_bus_controller``. + +Additional parameters come from the ``dmaap`` key in the Cloudify +Manager's Consul configuration, which is stored in the Consul KV store +under the key name 'cloudify\_manager'. The table below lists the +properties in the configuration: + ++----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+ +| Property | Type | Required? | Default | Description | ++================+==========+=============+==============+=============================================================================================+ +| ``username`` | string | Yes | (none) | The username for logging into DMaaP Bus Controller | ++----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+ +| ``password`` | string | Yes | (none) | The password for logging into DMaaP Bus Controller | ++----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+ +| ``owner`` | string | Yes | (none) | The name to be used as the owner for entities created by the plugin | ++----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+ +| ``protocol`` | string | No | ``https`` | The protocol (URL scheme) used to access the DMaaP bus controller (``http`` or ``https``) | ++----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+ +| ``path`` | string | No | ``webapi`` | The path to the root of the DMaaP Bus Controller API endpoint | ++----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+ + +Here is an example of a Cloudify Manager configuration object showing +only the ``dmaap`` key: + +:: + + { + "dmaap": { + "username": "dmaap.client@ccsdkorch.onap.org", + "password": "guessmeifyoucan" + "owner": "ccsdkorc" + }, + + (other configuration here) + + } + |