Make platformdoc docker container

Issue-Id: DCAEGEN2-60
Change-Id: Id57ab539707a43883ff957a23d2459a9b12df22e
Signed-off-by: Lusheng Ji <lji@research.att.com>

3 files changed, 626 insertions, 0 deletions

diff --git a/platformdoc/docs/components/dcae-cli/dmaap-connection-objects.md b/platformdoc/docs/components/dcae-cli/dmaap-connection-objects.md
new file mode 100644
index 00000000..cf03ccec
--- /dev/null
+++ b/platformdoc/docs/components/dcae-cli/dmaap-connection-objects.md
@@ -0,0 +1,156 @@
+# DMaaP connection objects
+
+DMaaP connection objects are JSON objects that:
+
+1. Components should expect at runtime in their application configuration and is to be used to connect to the appropriate DMaaP feed or topic.
+2. Developers must provide through the command-line argument `--dmaap-file` to test their component with manually provisioned feeds and topics.
+
+This page is a reference to the specific structure that each type of DMaaP stream requires.
+
+Note for #1 that components should expect the entire object with all properties at runtime where the default will be `null` unless specified otherwise.
+
+Note for #2 that developers are not required to provide the entire object.  The required properties will be labeled with "*required as input*".
+
+## Message router
+
+Publishers and subscribers both have the same JSON object structure.  Here's an example:
+
+```json
+{
+    "type": "message_router",
+    "aaf_username": "some-user",
+    "aaf_password": "some-password",
+    "dmaap_info": {
+        "client_role": "com.dcae.member",
+        "client_id": "1500462518108",
+        "location": "mtc00",
+        "topic_url": "https://we-are-message-router.us:3905/events/some-topic"
+    }
+}
+```
+
+At the top-level:
+
+Property Name | Type | Description
+------------- | ---- | -----------
+type | string | *Required as input*.  Must be `message_router` for message router topics
+aaf_username | string | AAF username message router clients use to authenticate with secure topics
+aaf_password | string | AAF password message router clients use to authenticate with secure topics
+dmaap_info | JSON object | *Required as input*. Contains the topic connection details
+
+The `dmaap_info` object contains:
+
+Property Name | Type | Description
+------------- | ---- | -----------
+client_role | string | AAF client role that's requesting publish or subscribe access to the topic
+client_id | string | Client id for given AAF client
+location | string | DCAE location for the publisher or subscriber, used to set up routing
+topic_url | string | *Required as input*. URL for accessing the topic to publish or receive events
+
+Here's an example of the minimal JSON that must be provided as an input:
+
+```json
+{
+    "type": "message_router",
+    "dmaap_info": {
+        "topic_url": "https://we-are-message-router.us:3905/events/some-topic"
+    }
+}
+```
+
+## Data router
+
+### Publisher
+
+Here's an example of what the JSON object connection for data router publisher looks like:
+
+```json
+{
+    "type": "data_router",
+    "dmaap_info": {
+        "location": "mtc00",
+        "publish_url": "https://we-are-data-router.us/feed/xyz",
+        "log_url": "https://we-are-data-router.us/feed/xyz/logs",
+        "username": "some-user",
+        "password": "some-password",
+        "publisher_id": "123456"
+    } 
+}
+```
+
+At the top-level:
+
+Property Name | Type | Description
+------------- | ---- | -----------
+type | string | *Required as input*.  Must be `data_router` for data router feeds
+dmaap_info | JSON object | *Required as input*. Contains the topic connection details
+
+The `dmaap_info` object contains:
+
+Property Name | Type | Description
+------------- | ---- | -----------
+location | string | DCAE location for the publisher, used to set up routing
+publish_url | string | *Required as input*. URL to which the publisher makes Data Router publish requests
+log_url | string | URL from which log data for the feed can be obtained
+username | string | Username the publisher uses to authenticate to Data Router 
+password | string | Password the publisher uses to authenticate to Data Router
+publisher_id | string | Publisher id in Data Router
+
+Here's an example of the minimal JSON that must be provided as an input:
+
+```json
+{
+    "type": "data_router",
+    "dmaap_info": {
+        "publish_url": "https://we-are-data-router.us/feed/xyz"
+    }
+}
+```
+
+### Subscriber
+
+Here's an example of what the JSON object connection for data router subscriber looks like:
+
+```json
+{
+    "type": "data_router",
+    "dmaap_info": {
+        "location": "mtc00",
+        "delivery_url": "https://my-subscriber-app.dcae:8080/target-path",
+        "username": "some-user",
+        "password": "some-password",
+        "subscriber_id": "789012"
+    } 
+}
+```
+
+At the top-level:
+
+Property Name | Type | Description
+------------- | ---- | -----------
+type | string | *Required as input*.  Must be `data_router` for data router feeds
+dmaap_info | JSON object | *Required as input*. Contains the topic connection details
+
+The `dmaap_info` object contains:
+
+Property Name | Type | Description
+------------- | ---- | -----------
+location | string | DCAE location for the publisher, used to set up routing
+delivery_url | string | URL to which the Data Router should deliver files
+username | string | Username Data Router uses to authenticate to the subscriber when delivering files
+password | string | Password Data Router uses to authenticate to the subscriber when delivering files
+subscriber_id | string | Subscriber id in Data Router
+
+Here's an example of the minimal JSON that must be provided as an input:
+
+```json
+{
+    "type": "data_router",
+    "dmaap_info": {
+    }
+}
+```
+
+Developers are recommended to use `username` and `password` since this is the recommended security practice.
+
+Note that the dcae-cli will construct the `delivery_url` when deploying the component since this can only be known at deployment time.
diff --git a/platformdoc/docs/components/dcae-cli/quickstart.md b/platformdoc/docs/components/dcae-cli/quickstart.md
new file mode 100644
index 00000000..5644b4fe
--- /dev/null
+++ b/platformdoc/docs/components/dcae-cli/quickstart.md
@@ -0,0 +1,79 @@
+# Quickstart
+
+The `dcae-cli` is a Python command-line tool built to aide component developers with the development and testing of their micro-service component for the DCAE platform.  It will help developers do functional and integration testing of their components locally and on remote environments as simple as possible.
+
+The tool requires the component developers to create a valid component specification for their component which is used by the tool.  This same component specification will be published in the [onboarding catalog](../../glossary#onboarding-catalog) at the end of development and testing.
+
+The git repository can be found [here](ONAP LINK TBD)
+
+## Pre-requisite
+
+### For Docker
+
+There are two options for development with Docker: developing locally on your machine which requires Docker to be installed and developing remotely by deploying onto remote infrastructure.
+
+#### For local development
+
+* You must install [Docker engine](https://docs.docker.com/engine/installation/) locally on your machine.
+* You must know the *external ip* of where the Docker engine is running.  The external ip is needed so that service discovery will wire up correctly.
+    - For OSX users, this means making sure the VirtualBox VM that is running your Docker engine has a bridged adapter and getting the ip of that adapter.
+
+#### For remote development
+
+You need access to a remote host with Docker engine installed with remote API access.  You must have the associated connection information: domain name or IP and port (should be either 2375 or 2376).  This information should be set in [an active profile](walkthrough.md#setting-profile).
+
+### For CDAP
+
+TBD
+
+## Install
+
+```
+pip install --extra-index-url https://YOUR_NEXUS_PYPI_SERVER/simple dcae-cli
+```
+
+To do an upgrade, use the `--upgrade` flag.
+
+### Configuration
+
+When you run the tool for the first time, the tool will create a [configuration directory](http://click.pocoo.org/5/api/#click.get_app_dir) and generate a configuration file.
+
+Configuration is first sourced from an remote server that the platform team manages.  This is overlaid with configuration details that you will be prompted to input particularly your AT&T UID.
+
+#### `--reinit`
+
+Configuration can be re-initialized or reset.  There is a `--reinit` flag that is to be used to re-initialize your configuration and your environment profiles.
+
+To re-initialize:
+
+```
+$ dcae_cli --reinit
+```
+
+### Verify
+
+Verify that its installed:
+
+```
+$ dcae_cli --help
+Usage: dcae_cli [OPTIONS] COMMAND [ARGS]...
+
+Options:
+  -v, --verbose  Prints INFO-level logs to screen.
+  --reinit       Re-initialize dcae-cli configuration
+  --version      Show the version and exit.
+  --help         Show this message and exit.
+
+Commands:
+  catalog
+  component
+  data_format
+  profiles
+```
+## Version
+
+You can verify the version of the dcae-cli with the following command:
+
+```
+$ dcae_cli --version
+```
diff --git a/platformdoc/docs/components/dcae-cli/walkthrough.md b/platformdoc/docs/components/dcae-cli/walkthrough.md
new file mode 100644
index 00000000..e646c435
--- /dev/null
+++ b/platformdoc/docs/components/dcae-cli/walkthrough.md
@@ -0,0 +1,391 @@
+# Walk-through
+
+The goal of this quickstart is to provide an overview of the functionalities of the `dcae-cli` and walk you through the capabilities:
+
+* [Adding data formats](#adding-data-formats)
+* [Adding component](#adding-component)
+* [Setting profile](#setting-profile)
+* [Development and testing](#development-and-testing)
+* [Publishing component](#publishing-component)
+* [Shared catalog](#shared-catalog)
+
+This walk-through uses example projects:
+
+* [laika](ONAP URL TBD)
+* [CDAP examples](ONAP URL TBD)
+
+## Adding data formats
+
+`data_format` is the sub-command that is used to execute operations that manage [data formats](../data-formats.md).
+
+```
+$ dcae_cli data_format --help
+Usage: dcae_cli data_format [OPTIONS] COMMAND [ARGS]...
+
+Options:
+  --help  Show this message and exit.
+
+Commands:
+  add      Tracks a data format file SPECIFICATION...
+  list     Lists all your data formats
+  publish  Publishes data format to make publicly...
+  show     Provides more information about FORMAT
+```
+
+Your data format must be in the catalog in order to use in the component specification.  Check the catalog using the `data_format list` sub-command:
+
+```
+$ dcae_cli data_format list
+
+Data formats for mh677g
++------+---------+-------------+--------+----------+
+| Name | Version | Description | Status | Modified |
++------+---------+-------------+--------+----------+
+|      |         |             |        |          |
++------+---------+-------------+--------+----------+
+```
+
+The fields `name`, `version`, `description` are referenced from the data format JSON from the `self` JSON.
+
+There are no data formats so you must add the data formats that your component specification references.  Use the `data_format add` sub-command:
+
+Here's an example command:
+
+```
+dcae_cli data_format add health.json
+```
+
+Verify that it was added:
+
+```
+$ dcae_cli data_format list
+
+Data formats for mh677g
++-------------------------------+---------+-------------------------------------------+--------+----------------------------+
+| Name                          | Version | Description                               | Status | Modified                   |
++-------------------------------+---------+-------------------------------------------+--------+----------------------------+
+| sandbox.platform.laika.health | 0.1.0   | Data format used for the /health endpoint | staged | 2017-05-23 04:02:38.952799 |
++-------------------------------+---------+-------------------------------------------+--------+----------------------------+
+```
+
+Go ahead and add other referenced data formats.
+
+## Adding component
+
+`component` is the sub-command that is used to work with operations for components:
+
+```
+$ dcae_cli component --help
+Usage: dcae_cli component [OPTIONS] COMMAND [ARGS]...
+
+Options:
+  --help  Show this message and exit.
+
+Commands:
+  add
+  dev       Set up component in development for...
+  list      Lists components in the public catalog.
+  publish   Pushes COMPONENT to the public catalog
+  run       Runs the latest version of COMPONENT.
+  show      Provides more information about COMPONENT
+  undeploy  Undeploys the latest version of COMPONENT.
+```
+
+Your component must be accessible from the catalog in order for it to be used.  Check the catalog using the `component list` sub-command:
+
+```
+$ dcae_cli component list
+Active profile: solutioning
+
++------+---------+------+-------------+--------+----------+-----------+
+| Name | Version | Type | Description | Status | Modified | #Deployed |
++------+---------+------+-------------+--------+----------+-----------+
+|      |         |      |             |        |          |           |
++------+---------+------+-------------+--------+----------+-----------+
+
+Use the "--deployed" option to see more details on deployments
+```
+
+The fields `name`, `version`, `type`, `description` are referenced from the component specification's `self` JSON.
+
+There are no components so you must add your component.  Use the `component add` sub-command.  The command is the same for docker and cdap components:
+
+```
+$ dcae_cli component add --help
+Usage: dcae_cli component add [OPTIONS] SPECIFICATION
+
+Options:
+  --update  Updates a locally added component if it has not been already
+            pushed
+  --help    Show this message and exit.
+```
+
+*Note* use the `--update` flag to replace existing staged instances.
+
+The `component dev` sub-command can be useful in validating and experimenting when crafting your component specification.  See details about `dev` under [Development and testing](#development-and-testing).
+
+Once we add the components laika and helloworld, let's verify that they got added ok:
+
+```
+$ dcae_cli component list
+Active profile: solutioning
+
++-------------------------+---------+--------+---------------------------------------------------------------+--------+----------------------------+-----------+
+| Name                    | Version | Type   | Description                                                   | Status | Modified                   | #Deployed |
++-------------------------+---------+--------+---------------------------------------------------------------+--------+----------------------------+-----------+
+| cdap.helloworld.endnode | 0.8.0   | cdap   | cdap test component                                           | staged | 2017-05-23 04:14:35.588075 | 0         |
+| sandbox.platform.laika  | 0.5.0   | docker | Web service used as a stand-alone test DCAE service compone.. | staged | 2017-05-23 04:07:44.065610 | 0         |
++-------------------------+---------+--------+---------------------------------------------------------------+--------+----------------------------+-----------+
+
+Use the "--deployed" option to see more details on deployments
+```
+
+## Setting profile
+
+`profile` is the sub-command that is used to manage profiles.  These profiles contain environment variables used to connect to different environments.  This is used in the running and deployment of your component using the `dcae_cli component run` command.  The `dcae-cli` ships with profiles for `solutioning` and `rework`.
+
+```
+$ dcae_cli profiles --help
+Usage: dcae_cli profiles [OPTIONS] COMMAND [ARGS]...
+
+Options:
+  --help  Show this message and exit.
+
+Commands:
+  activate  Sets profile NAME as the active profile
+  create    Creates a new profile NAME initialized with...
+  delete    Deletes profile NAME
+  list      Lists available profiles
+  set       Updates profile NAME such that KEY=VALUE
+  show      Prints the profile dictionary
+```
+
+To see what variables a profile contains, you can use the `show` command, as in `dcae_cli profiles show PROFILE_NAME`
+
+Use the `create` sub-command to create your own profile and assign new values using the `set` command.  Afterwards you will need to `activate` the profile you wish to use.  First take a look at which profile is active:
+
+```
+$ dcae_cli profiles list
+   rework
+*  solutioning
+```
+
+The active profile is `solutioning` so to activate *rework* to use `rework`:
+
+```
+$ dcae_cli profiles activate rework
+```
+
+Check
+
+```
+$ dcae_cli profiles list
+*  rework
+   solutioning
+```
+
+## Development and testing
+
+The following operations under the sub-command `component` are aimed to help developers with testing:
+
+* `run`
+* `undeploy`
+* `dev`
+
+### `run`
+
+The `run` operation is to be used for running your application in its container remotely on the activated environment.  Docker containers have the additional option to run locally on your development machine.
+
+In order to run your application, you must have added your data formats and your component to your catalog.
+
+Let's verify that your component is in the catalog:
+
+```
+$ dcae_cli component list                                                                                       
+Active profile: solutioning
+
++-------------------------+---------+--------+---------------------------------------------------------------+--------+----------------------------+-----------+
+| Name                    | Version | Type   | Description                                                   | Status | Modified                   | #Deployed |
++-------------------------+---------+--------+---------------------------------------------------------------+--------+----------------------------+-----------+
+| cdap.helloworld.endnode | 0.8.0   | cdap   | cdap test component                                           | staged | 2017-05-23 04:14:35.588075 | 0         |
+| sandbox.platform.laika  | 0.5.0   | docker | Web service used as a stand-alone test DCAE service compone.. | staged | 2017-05-23 04:07:44.065610 | 0         |
++-------------------------+---------+--------+---------------------------------------------------------------+--------+----------------------------+-----------+
+
+Use the "--deployed" option to see more details on deployments
+```
+
+#### Docker
+
+**NOTE** Make sure your Docker image has been uploaded to the shared registry.
+
+For Docker containers, you can run either attached or unattached.  Attached means that the dcae-cli tool will launch the container and not terminate.  The dcae-cli while attached will stream in the logs of the Docker container.  Doing a Ctrl-C will terminate the run session which means undeploy your container and force a clean up automatically.
+
+#### CDAP
+
+**NOTE** Make sure your CDAP jar has been uploaded to Nexus.
+
+TODO
+
+### `undeploy`
+
+The `undeploy` operation is to be used to undeploy any instances of a specified component, version that you have deployed.  This includes cleaning up of configuration.
+
+Let's undeploy `sandbox.platform.laika` that was deployed from the previous section:
+
+```
+$ dcae_cli component undeploy sandbox.platform.laika:0.5.0
+DCAE.Undeploy | WARNING | Undeploying components: 1
+DCAE.Undeploy | WARNING | Undeployed components: 1
+```
+
+### `dev`
+
+The `dev` operation is a convenient operation that can be useful for the development and testing of your component.  It can be used to:
+
+* Help validate your experimental component specification before uploading to the catalog
+* Generate the application configuration from the component specification and make it available in a test environment.  This allows you to view your resulting configuration for local development and to help debug potential related issues.
+
+Let's say you have a component specification called `component-spec.json`:
+
+```
+$ dcae_cli component dev component-spec.json 
+Ready for component development
+
+Setup these environment varibles. Run "source env_solutioning":
+
+export DOCKER_HOST=SOME_DOCKER_HOST:2376
+export SERVICE_CHECK_INTERVAL=15s
+export CONFIG_BINDING_SERVICE=config_binding_service
+export HOSTNAME=mh677g.95740959-63d2-492a-b964-62a6dce2591d.0-6-0.sandbox-platform-laika
+export CONSUL_HOST=SOME_CONSUL_HOST
+export CDAP_BROKER=cdap_broker
+export SERVICE_NAME=mh677g.95740959-63d2-492a-b964-62a6dce2591d.0-6-0.sandbox-platform-laika
+export SERVICE_CHECK_TIMEOUT=1s
+export SERVICE_CHECK_HTTP=/health
+
+Press any key to stop and to clean up
+```
+
+Your application configuration is now available under the name `mh677g.95740959-63d2-492a-b964-62a6dce2591d.0-6-0.sandbox-platform-laika`.
+
+To view the resulting configuration, you can `curl` a request to the config binding service or programmatically fetch your configuration within your application.
+
+You need to first query Consul to get the ip and port of config binding service:
+
+```
+curl http://$CONSUL_HOST:8500/v1/catalog/service/$CONFIG_BINDING_SERVICE
+[
+  {
+    "ID": "983d5c94-c508-4a8a-9be3-5912bd09786b",
+    "Node": "realsolcnsl00",
+    "Address": "10.226.1.22",
+    "TaggedAddresses": {
+      "lan": "10.226.1.22",
+      "wan": "10.226.1.22"
+    },
+    "NodeMeta": {},
+    "ServiceID": "5f371f295c90:config_binding_service:10000",
+    "ServiceName": "config_binding_service",
+    "ServiceTags": [],
+    "ServiceAddress": "XXXX",
+    "ServicePort": 32770,
+    "ServiceEnableTagOverride": false,
+    "CreateIndex": 487,
+    "ModifyIndex": 487
+  }
+]
+```
+
+### DMaaP testing
+
+Currently, the dcae-cli does not have the capability of provisioning topics.  In order to do testing with message router topics or with data router feeds, the developer must provision the topic or the feed manually and provide the connection details in the form of a JSON in a file to the dcae-cli.  This file is to be passed in when using the `run` and `dev` commands with the option `--dmaap-file`.
+
+The structure of the DMaaP JSON is an object of config keys to matching topic or feed connection details.  Config keys are the `config_key` values specified in your component specification streams section where the streams must be type message router or data router.  Information about the associated connection details can be found on [this page](dmaap-connection-objects.md).  Please check it out.
+
+For example, if you have a component specification that has the following streams entry:
+
+```json
+"streams": {
+    "publishes": [{
+        "format": "ves",
+        "version": "1.0.0",
+        "type": "message router",
+        "config_key": "ves_connection"
+    }]
+}
+```
+
+Then to deploy and to run your component, you must use the `--dmaap-file` command and pass in a JSON that looks like:
+
+```json
+{
+    "ves_connection": {
+        "type": "message_router",
+        "dmaap_info": {
+            "topic_url": "https://we-are-message-router.us:3905/events/some-topic"
+        }
+    }
+}
+```
+
+The provided DMaaP JSON is used to simulate the output of provisioning and will be used to merge with the generated application configuration at runtime.
+
+Your final application config will look like:
+
+```json
+{
+    "streams_publishes": {
+        "ves_connection": {
+            "type": "message_router",
+            "dmaap_info": {
+                "topic_url": "https://we-are-message-router.us:3905/events/some-topic"
+            }
+        }
+    }
+}
+```
+
+#### Data router subscribers
+
+Note for data router subscriber testing, you will need the delivery url in order to provision the subscriber to the feed.  This is constructed at deployment time and will be provided by the dcae-cli after you deploy your component.  The delivery urls will be displayed to the screen:
+
+```
+DCAE.Run | WARNING | Your component is a data router subscriber. Here are the delivery urls: 
+
+        some-sub-dr: http://SOME_IP:32838/identity
+
+```
+
+## Publishing component
+
+Once components have their component specifications crafted and validated and have been tested, components should be published in the shared onboarding catalog using the `publish` sub-command for both data formats and components.  You must publish all data formats of a component before publishing a component.
+
+Publishing will change the status of a component, be made accessible for other developers to use, and will generate the associated TOSCA models for use in designing of compositions.
+
+```
+dcae_cli component publish sandbox.platform.laika:0.5.0
+```
+
+## Shared catalog
+
+`catalog` is the sub-command used to access and to browse the shared onboarding catalog to view components and data formats that have been published and that are being worked on.  Components and data formats have two statuses `staged` and `published`.
+
+Staged means that the resource has been simply added and is under development.  It is to be used only by the owner.  Published means that the resource has been fully developed and tested and is ready to be shared.
+
+Published components can be deployed by non-owners and published data formats can be used in component specifications of non-owners.
+
+There are two available operations:
+
+```
+$ dcae_cli catalog --help
+Usage: dcae_cli catalog [OPTIONS] COMMAND [ARGS]...
+
+Options:
+  --help  Show this message and exit.
+
+Commands:
+  list
+  show
+```
+
+Staged components can be viewed under the `list` operation using the `--expanded` flag.