Add docker and relationships plugin

Change-Id: I323599ae2965f081f2061b6791635bbeddb09811
Issue-Id: DCAEGEN2-79
Signed-off-by: Michael Hwang <mhwang@research.att.com>

11 files changed, 597 insertions, 0 deletions

diff --git a/relationships/.gitignore b/relationships/.gitignore
new file mode 100644
index 0000000..5c75135
--- /dev/null
+++ b/relationships/.gitignore
@@ -0,0 +1,93 @@
+cfyhelper.sh
+.cloudify/
+*.wgn
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*,cover
+.hypothesis/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# dotenv
+.env
+
+# virtualenv
+.venv/
+venv/
+ENV/
+
+# Spyder project settings
+.spyderproject
+
+# Rope project settings
+.ropeproject
diff --git a/relationships/LICENSE.txt b/relationships/LICENSE.txt
new file mode 100644
index 0000000..cb8008a
--- /dev/null
+++ b/relationships/LICENSE.txt
@@ -0,0 +1,32 @@
+============LICENSE_START=======================================================
+org.onap.dcae
+================================================================================
+Copyright (c) 2017 AT&T Intellectual Property. All rights reserved.
+================================================================================
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+============LICENSE_END=========================================================
+
+ECOMP is a trademark and service mark of AT&T Intellectual Property.
+
+
+Copyright (c) 2017 AT&T Intellectual Property. All rights reserved.
+===================================================================
+Licensed under the Creative Commons License, Attribution 4.0 Intl.  (the "License");
+you may not use this documentation except in compliance with the License.
+You may obtain a copy of the License at
+       https://creativecommons.org/licenses/by/4.0/
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
diff --git a/relationships/README.md b/relationships/README.md
new file mode 100644
index 0000000..82d60f3
--- /dev/null
+++ b/relationships/README.md
@@ -0,0 +1,54 @@
+# relationships-cloudify
+
+This repo contains Cloudify artifacts to support custom functionality in processing relationships between nodes.
+
+* `component_connected_to` - Used to connect service component nodes together
+* `component_contained_in` - Used to place service component nodes on their proper container host or cluster
+
+## `component_connected_to`
+
+This Cloudify relationship is used to collect the relationship information of all the target nodes of a source node and to provide this information to the source node application.  Currently, this information is provided by being stored in the platform Consul instance under the key `<source node's name>:rel`.
+
+Each target node is expected to provide its own name (name used for service registration) to the source node.  These target names are passed as a runtime property of the target node context under the key `service_component_name`.
+
+### Special features
+
+#### `target_name_override`
+
+The preconfigure operation uses the task function `add_relationship` which has an *optional* input parameter `target_name_override`.  The target name is passed into the source node's relationship information that is used by the source node underlying application to connect to the target node underlying application.  If not used, the default behavior is to expect the target name to come from the target node as a runtime property under the key `service_component_name`.
+
+##### When should you use this?
+
+When you know the target node does not populate `service_component_name` into its runtime properties.
+
+##### Usage example
+
+Here is an example of how you would use the `target_name_override` input parameter in a blueprint:
+
+```yaml
+node_templates:
+
+  some-source:
+    type: dcae.nodes.rework.DockerContainer
+    properties:
+        service_component_type:
+            'laika'
+        service_id:
+            { get_input: service_id }
+        location_id:
+            'rework-central'
+        application_config:
+            { get_input: some-source-config }
+        image:
+            'dcae-rework/laika:0.4.0'
+    relationships:
+      - type: dcae.relationships.component_connected_to
+        target: some-target
+        target_interfaces:
+            cloudify.interfaces.relationship_lifecycle:
+                preconfigure:
+                    inputs:
+                        target_name_override:
+                            'some-target'
+```
+
diff --git a/relationships/example_register_to_blueprint.yaml b/relationships/example_register_to_blueprint.yaml
new file mode 100644
index 0000000..52ee40b
--- /dev/null
+++ b/relationships/example_register_to_blueprint.yaml
@@ -0,0 +1,24 @@
+tosca_definitions_version: cloudify_dsl_1_3
+
+imports:
+  - http://www.getcloudify.org/spec/cloudify/3.4/types.yaml
+  - {{ ONAPTEMPLATE_RAWREPOURL_org_onap_dcaegen2 }}/type_files/relationship/1.0.0/node-type.yaml
+
+node_templates:
+
+    src:
+        type: cloudify.nodes.Root
+        relationships:
+          - type: dcae.relationships.component_registered_to
+            target: tgt #agree this is kind of weird to be a relationship type now with a dummy target
+            target_interfaces:
+                cloudify.interfaces.relationship_lifecycle:
+                    preconfigure:
+                        inputs:
+                            address_to_register: "666.666.666.666"
+                            port_to_register: "666"
+                            name_to_register: "TEST_REGISTERED_TO_SERVICE"
+                            location_id: "rework-central"
+
+    tgt: #do relationships always need targets? 
+        type: cloudify.nodes.Root
diff --git a/relationships/relationship-types.yaml b/relationships/relationship-types.yaml
new file mode 100644
index 0000000..d0ab59f
--- /dev/null
+++ b/relationships/relationship-types.yaml
@@ -0,0 +1,65 @@
+tosca_definitions_version: cloudify_dsl_1_3
+
+imports:
+    - http://www.getcloudify.org/spec/cloudify/3.4/types.yaml
+
+plugins:
+  relationships:
+    executor: 'central_deployment_agent'
+    package_name: relationshipplugin
+    package_version: 1.0.0
+
+relationships:
+    # The relationship type here is to be used between service component nodes. What is achieved here is
+    # functionality in providing this relationship information to the service components so that they can
+    # do service discovery.
+    #
+    # This function will create/add to the rels list for components. So going from a "collector node -> analytics node"
+    # for example, this is kind of the edge and will add:
+    # 
+    # ```
+    # "collector_name:rel": ["analytics_name"]
+    # ```
+    #
+    # To the key value store.
+    #
+    dcae.relationships.component_connected_to:
+        derived_from: cloudify.relationships.connected_to
+        # These operations are for adding and removing relationships from Consul
+        # http://getcloudify.org/guide/3.1/reference-builtin-workflows.html
+        target_interfaces:
+            cloudify.interfaces.relationship_lifecycle:
+                preconfigure:
+                    # Adds target to the source relationship list
+                    implementation: relationships.relationshipplugin.add_relationship
+                unlink:
+                    # Removes target from the source relationship list
+                    implementation: relationships.relationshipplugin.remove_relationship
+
+    dcae.relationships.component_contained_in:
+        derived_from: cloudify.relationships.contained_in
+        target_interfaces:
+            cloudify.interfaces.relationship_lifecycle:
+                preconfigure:
+                    implementation: relationships.relationshipplugin.forward_destination_info
+                # TODO: Is there a need for "unlink"?
+
+    dcae.relationships.component_registered_to:
+        #Operates on a relationship A -> B and makes the following assumptions:
+        derived_from: cloudify.relationships.connected_to
+        target_interfaces:
+            cloudify.interfaces.relationship_lifecycle:
+                preconfigure:
+                    implementation: relationships.relationshipplugin.tasks.registered_to
+                    inputs:
+                       address_to_register:
+                           type: string
+                       port_to_register:
+                           type: string
+                       name_to_register:
+                           type: string
+                unlink:
+                    implementation: relationships.relationshipplugin.tasks.registered_to_delete
+
+
+
diff --git a/relationships/relationshipplugin/__init__.py b/relationships/relationshipplugin/__init__.py
new file mode 100644
index 0000000..259e52c
--- /dev/null
+++ b/relationships/relationshipplugin/__init__.py
@@ -0,0 +1,23 @@
+# ============LICENSE_START=======================================================
+# org.onap.dcae
+# ================================================================================
+# Copyright (c) 2017 AT&T Intellectual Property. All rights reserved.
+# ================================================================================
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============LICENSE_END=========================================================
+#
+# ECOMP is a trademark and service mark of AT&T Intellectual Property.
+
+
+from .tasks import add_relationship, remove_relationship, \
+    forward_destination_info
diff --git a/relationships/relationshipplugin/discovery.py b/relationships/relationshipplugin/discovery.py
new file mode 100644
index 0000000..bd0e369
--- /dev/null
+++ b/relationships/relationshipplugin/discovery.py
@@ -0,0 +1,84 @@
+# ============LICENSE_START=======================================================
+# org.onap.dcae
+# ================================================================================
+# Copyright (c) 2017 AT&T Intellectual Property. All rights reserved.
+# ================================================================================
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============LICENSE_END=========================================================
+#
+# ECOMP is a trademark and service mark of AT&T Intellectual Property.
+
+from urlparse import urlparse
+import json
+import consul
+
+
+class DiscoveryError(RuntimeError):
+    pass
+
+def _create_rel_key(service_component_name):
+    return "{0}:rel".format(service_component_name)
+
+def _parse_host(host):
+    """Parse host string
+
+    Returns:
+        Tuple of the hostname and port to use to connect to Consul
+    """
+    def parse_urlparse_result(pr):
+        if not pr.hostname:
+            raise DiscoveryError("Invalid Consul host provided: {0}".format(host))
+
+        try:
+            # Port 8500 is the Consul default
+            return (pr.hostname, pr.port if pr.port else 8500)
+        except ValueError as e:
+            # Something bad happened with port
+            raise DiscoveryError("Invalid Consul host provided: {0}".format(host))
+
+    pr = urlparse(host)
+
+    # urlparse requires scheme to be set in order to be useful
+    if pr.scheme and pr.netloc:
+        return parse_urlparse_result(pr)
+    else:
+        return parse_urlparse_result(urlparse("http://{0}".format(host)))
+
+
+def create_kv_conn(host):
+    """Create connection to key-value store
+
+    Returns a Consul client to the specified Consul host
+    """
+    (hostname, port) = _parse_host(host)
+    return consul.Consul(host=hostname, port=port)
+
+def store_relationship(kv_conn, source_name, target_name):
+    # TODO: Rel entry may already exist in a one-to-many situation. Need to
+    # support that.
+    rel_key = _create_rel_key(source_name)
+    rel_value = [target_name] if target_name else []
+
+    kv_conn.kv.put(rel_key, json.dumps(rel_value))
+    print("Added relationship for {0}".format(rel_key))
+
+def delete_relationship(kv_conn, service_component_name):
+    rel_key = _create_rel_key(service_component_name)
+    index, rels = kv_conn.kv.get(rel_key)
+
+    if rels:
+        rels = json.loads(rels["Value"].decode("utf-8"))
+        kv_conn.kv.delete(rel_key)
+        return rels
+    else:
+        return []
diff --git a/relationships/relationshipplugin/tasks.py b/relationships/relationshipplugin/tasks.py
new file mode 100644
index 0000000..b17403c
--- /dev/null
+++ b/relationships/relationshipplugin/tasks.py
@@ -0,0 +1,131 @@
+# ============LICENSE_START=======================================================
+# org.onap.dcae
+# ================================================================================
+# Copyright (c) 2017 AT&T Intellectual Property. All rights reserved.
+# ================================================================================
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============LICENSE_END=========================================================
+#
+# ECOMP is a trademark and service mark of AT&T Intellectual Property.
+
+import json
+from cloudify import ctx
+from cloudify.decorators import operation
+from cloudify.exceptions import NonRecoverableError
+from relationshipplugin import discovery as dis
+import requests
+
+
+SERVICE_COMPONENT_NAME = "service_component_name"
+SELECTED_CONTAINER_DESTINATION = "selected_container_destination"
+CONSUL_HOST = "consul_host"
+
+CONSUL_HOSTNAME = "localhost"
+
+
+# Lifecycle interface calls for component_connect_to
+
+# NOTE: ctx.source and ctx.target are RelationshipSubjectContext
+# Order of operation of relationships is bit confusing. These operations are
+# implemented for `target_interfaces`. By observation, the target node processed,
+# then the source is created, the relationship is run then the source is started.
+# http://getcloudify.org/guide/3.1/dsl-spec-relationships.html#relationship-interfaces
+
+@operation
+def add_relationship(**kwargs):
+    """Adds target to the source relationship list"""
+    try:
+        conn = dis.create_kv_conn(CONSUL_HOSTNAME)
+
+        source_name = ctx.source.instance.runtime_properties[SERVICE_COMPONENT_NAME]
+        # The use case for using the target name override is for the platform
+        # blueprint where the cdap broker needs to connect to a cdap cluster but
+        # the cdap cluster does not not use the component plugins so the name is
+        # not generated.
+        # REVIEW: Re-review this
+        target_name = kwargs["target_name_override"] \
+            if "target_name_override" in kwargs \
+            else ctx.target.instance.runtime_properties[SERVICE_COMPONENT_NAME]
+
+        dis.store_relationship(conn, source_name, target_name)
+        ctx.logger.info("Created relationship: {0} to {1}".format(source_name,
+            target_name))
+    except Exception as e:
+        ctx.logger.error("Unexpected error while adding relationship: {0}"
+                .format(str(e)))
+        raise NonRecoverableError(e)
+
+@operation
+def remove_relationship(**kwargs):
+    """Removes target from the source relationship list"""
+    try:
+        conn = dis.create_kv_conn(CONSUL_HOSTNAME)
+
+        source_name = ctx.source.instance.runtime_properties[SERVICE_COMPONENT_NAME]
+        dis.delete_relationship(conn, source_name)
+        ctx.logger.info("Removed relationship: {0}".format(source_name))
+    except Exception as e:
+        ctx.logger.error("Unexpected error while removing relationship: {0}"
+                .format(str(e)))
+        raise NonRecoverableError(e)
+
+
+# Lifecycle interface calls for component_contained_in
+
+@operation
+def forward_destination_info(**kwargs):
+    try:
+        selected_target = ctx.target.instance.runtime_properties[SERVICE_COMPONENT_NAME]
+        ctx.source.instance.runtime_properties[SELECTED_CONTAINER_DESTINATION] = selected_target
+        ctx.logger.info("Forwarding selected target: {0}".format(ctx.source.instance.id))
+    except Exception as e:
+        ctx.logger.error("Unexpected error while forwarding selected target: {0}"
+                .format(str(e)))
+        raise NonRecoverableError(e)
+
+@operation
+def registered_to(**kwargs):
+    """
+    Intended to be used in platform blueprints, but possible to be reused elsewhere
+    """
+    ctx.logger.info(str(kwargs))
+    address = kwargs["address_to_register"]
+    name = kwargs["name_to_register"]
+    port = kwargs["port_to_register"]
+
+    (consul_host, consul_port) = (CONSUL_HOSTNAME, 8500)
+    #Storing in source because that's who is getting registered
+    ctx.source.instance.runtime_properties[CONSUL_HOST] = "http://{0}:{1}".format(consul_host, consul_port)
+    ctx.source.instance.runtime_properties["name_to_register"] = name #careful! delete does not have access to inputs
+    
+    try:
+        response = requests.put(url = "{0}/v1/agent/service/register".format(ctx.source.instance.runtime_properties[CONSUL_HOST]),
+                     json = {
+                              "name" : name,
+                              "Address" : address,
+                              "Port" : int(port)
+                            },
+                     headers={'Content-Type': 'application/json'})
+        response.raise_for_status() #bomb if not 2xx
+    except Exception as e:
+        ctx.logger.error("Error while registering: {0}".format(str(e)))
+        raise NonRecoverableError(e)
+
+@operation
+def registered_to_delete(**kwargs):
+    """
+    The deletion/opposite of registered_to
+    """
+    requests.put(url = "{0}/v1/agent/service/deregister/{1}".format(ctx.source.instance.runtime_properties[CONSUL_HOST], ctx.source.instance.runtime_properties["name_to_register"]),
+                 headers={'Content-Type': 'Content-Type: application/json'})
+    #this is on delete so do not do any checking
diff --git a/relationships/requirements.txt b/relationships/requirements.txt
new file mode 100644
index 0000000..59c8c70
--- /dev/null
+++ b/relationships/requirements.txt
@@ -0,0 +1,12 @@
+bottle==0.12.7
+cloudify-plugins-common==3.4
+cloudify-rest-client==3.4
+Jinja2==2.7.2
+MarkupSafe==0.23
+networkx==1.8.1
+pika==0.9.14
+proxy-tools==0.1.0
+python-consul==0.6.1
+requests==2.7.0
+requests-toolbelt==0.7.0
+six==1.10.0
diff --git a/relationships/setup.py b/relationships/setup.py
new file mode 100644
index 0000000..fbbf077
--- /dev/null
+++ b/relationships/setup.py
@@ -0,0 +1,35 @@
+# ============LICENSE_START=======================================================
+# org.onap.dcae
+# ================================================================================
+# Copyright (c) 2017 AT&T Intellectual Property. All rights reserved.
+# ================================================================================
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============LICENSE_END=========================================================
+#
+# ECOMP is a trademark and service mark of AT&T Intellectual Property.
+
+import os
+from setuptools import setup
+
+setup(
+    name='relationshipplugin',
+    description='',
+    version="1.0.0",
+    author='Michael Hwang, Tommy Carpenter',
+    packages=['relationshipplugin'],
+    zip_safe=False,
+    install_requires=[
+        "python-consul>=0.6.0",
+        "cloudify-plugins-common==3.4.0"
+    ]
+)
diff --git a/relationships/tests/test_discovery.py b/relationships/tests/test_discovery.py
new file mode 100644
index 0000000..ae2d34b
--- /dev/null
+++ b/relationships/tests/test_discovery.py
@@ -0,0 +1,44 @@
+# ============LICENSE_START=======================================================
+# org.onap.dcae
+# ================================================================================
+# Copyright (c) 2017 AT&T Intellectual Property. All rights reserved.
+# ================================================================================
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============LICENSE_END=========================================================
+#
+# ECOMP is a trademark and service mark of AT&T Intellectual Property.
+
+import pytest
+from relationshipplugin import discovery as dis
+
+
+def test_create_kv_conn_parse_host():
+    # Just hostname
+    hostname = "some-consul.far.away"
+    assert (hostname, 8500) == dis._parse_host(hostname)
+
+    # Hostname:port
+    port = 8080
+    host = "{0}:{1}".format(hostname, port)
+    assert (hostname, port) == dis._parse_host(host)
+
+    # Invalid port
+    port = "abc"
+    host = "{0}:{1}".format(hostname, port)
+    with pytest.raises(dis.DiscoveryError):
+        dis._parse_host(host)
+
+    # Hanging colon
+    port = ""
+    host = "{0}:{1}".format(hostname, port)
+    assert (hostname, 8500) == dis._parse_host(host)