diff options
Diffstat (limited to 'docs/sections/distribution.rst')
| -rw-r--r-- | docs/sections/distribution.rst | 749 |
1 files changed, 0 insertions, 749 deletions
diff --git a/docs/sections/distribution.rst b/docs/sections/distribution.rst deleted file mode 100644 index df7a4c2..0000000 --- a/docs/sections/distribution.rst +++ /dev/null @@ -1,749 +0,0 @@ -Python/Linux Distribution Notes -=============================== - -*Updated: 10 Nov 2017 23:30 GMT* - -This document exists to help bridge the gap between the Conductor python -package and any downstream distribution. The steps outlined herein may -be taken into consideration when creating an AT&T SWM package, -Ubuntu/Debian package, Chef cookbook, or Ansible playbook. - -Components ----------- - -Conductor consists of five services that work together: - -- **``conductor-api``**: An HTTP REST API -- **``conductor-controller``**: Validation, translation, and - status/results -- **``conductor-data``**: Inventory provider and service controller - gateway -- **``conductor-solver``**: Processing and solution calculation -- **``conductor-reservation``**: Reserves the suggested solution solved - by Solver component. - -Workflow --------- - -- Deployment **plans** are created, viewed, and deleted via - ``conductor-api`` and its `REST API <doc/api/README.md>`__. -- Included within each ``conductor-api`` plan request is a `Homing - Template <doc/template/README.md>`__. -- Homing Templates describe a set of inventory demands and constraints - to be solved against. -- ``conductor-api`` hands off all API requests to - ``conductor-controller`` for handling. -- All deployment plans are assigned a unique identifier (UUID-4), which - can be used to check for solution status asynchronously. (Conductor - does not support callbacks at this time.) -- ``conductor-controller`` ensures templates are well-formed and valid. - Errors and remediation are made visible through ``conductor-api``. - When running in debug mode, the API will also include a python - traceback in the response body, if available. -- ``conductor-controller`` uses ``conductor-data`` to resolve demands - against a particular **inventory provider** (e.g., A&AI). -- ``conductor-controller`` translates the template into a format - suitable for solving. -- As each template is translated, ``conductor-solver`` begins working - on it. -- ``conductor-solver`` uses ``conductor-data`` to resolve constraints - against a particular **service controller** (e.g., SDN-C). -- ``conductor-solver`` determines the most suitable inventory to - recommend. -- ``conductor-reservation`` attempts to reserve the solved solution in - SDN-GC - -**NOTE**: There is no Command Line Interface or Python API Library at -this time. - -Pre-Flight and Pre-Installation Considerations ----------------------------------------------- - -AT&T Application Identifiers and Roles -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -- App/Tool Name: ECOMP Conductor -- MOTS Application ID: 26213 -- MechID: m04308 -- ECOMP Feature ID: F13704 -- PMT: 461306 -- UAM Role Name: Conductor Production Support -- UAM Role id: 0000025248 - -Root -~~~~ - -Be aware that some commands may require ``sudo``, depending on the -account being used to perform the installation. - -Proxy -~~~~~ - -If line-of-sight to internet-facing repositories is permitted and -available, set the following shell environment variables if AT&T proxy -services are required: - -.. code:: bash - - $ export http_proxy="http://one.proxy.att.com:8080/" - $ export https_proxy="http://one.proxy.att.com:8080/" - -Requirements -~~~~~~~~~~~~ - -Conductor is officially supported on `Ubuntu 14.04 LTS (Trusty -Tahr) <http://releases.ubuntu.com/14.04/>`__, though it should also work -on newer releases. - -Ensure the following Ubuntu packages are present, as they may not be -included by default: - -- `libffi-dev <http://packages.ubuntu.com/trusty/libffi-dev>`__ -- `postgresql-server-dev-9.3 <http://packages.ubuntu.com/trusty/postgresql-server-dev-9.3>`__ -- `python2.7 <http://packages.ubuntu.com/trusty/python2.7>`__ - -``conductor-api`` may be run as-is for development and test purposes. -When used in a production environment, it is recommended that -``conductor-api`` run under a multithreaded httpd service supporting -`WSGI <https://www.wikipedia.org/wiki/Web_Server_Gateway_Interface>`__, -tuned as appropriate. - -Configuration instructions for **apache2 httpd** and **nginx** are -included herein. Respective package requirements are: - -- `apache2 <http://packages.ubuntu.com/trusty/apache2>`__ and - `libapache2-mod-wsgi <http://packages.ubuntu.com/trusty/libapache2-mod-wsgi>`__ -- `nginx <http://packages.ubuntu.com/trusty/nginx>`__ and - `uwsgi <http://packages.ubuntu.com/trusty/uwsgi>`__ - -All Conductor services use AT&T `Music <https://github.com/att/music>`__ -for data storage/persistence and/or as a RPC transport mechanism. -Consult the `Music Local Installation -Guide <https://github.com/att/music/blob/master/README.md>`__ for -installation/configuration steps. - -Networking -~~~~~~~~~~ - -All conductor services require line-of-sight access to all Music -servers/ports. - -The ``conductor-api`` service uses TCP port 8091. - -Security -~~~~~~~~ - -``conductor-api`` is accessed via HTTP. SSL/TLS certificates and -AuthN/AuthZ (e.g., AAF) are not supported at this time. - -Conductor makes use of plugins that act as gateways to *inventory -providers* and *service controllers*. At present, two plugins are -supported out-of-the-box: **A&AI** and **SDN-C**, respectively. - -A&AI requires two-way SSL/TLS. Certificates must be registered and -whitelisted with A&AI. SDN-C uses HTTP Basic Authentication. Consult -with each respective service for official information on how to obtain -access. - -Storage -~~~~~~~ - -For a cloud environment in particular, it may be desirable to use a -separate block storage device (e.g., an OpenStack Cinder volume) for -logs, configuration, and other data persistence. In this way, it becomes -a trivial matter to replace the entire VM if necessary, followed by -reinstallation of the app and any supplemental configuration. Take this -into consideration when setting various Conductor config options. - -Python Virtual Environments -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -At present, Conductor installation is only supported at the (upstream) -python package level and not the (downstream) Ubuntu distribution or SWM -package levels. - -To mitigate/eliminate the risk of introducing conflicts with other -python applications or Ubuntu/SWM package dependencies, consider -installing Conductor in a `python virtual -environment <http://docs.python-guide.org/en/latest/dev/virtualenvs/>`__ -(or *venv* for short). - -Example venv-aware WSGI app configurations, sysvinit scripts, and -upstart scripts can be found in the Conductor repository under -`examples </examples/>`__. - -Python Package Dependencies -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Conductor is installed using the python ``pip`` command. ``pip`` uses a -python project's `requirements manifest </requirements.txt>`__ to -install all python module dependencies. - -**NOTE**: When line-of-sight access to a PyPI-compatible package index -is not available, advance installation of Conductor's python package -dependencies is required *before* installation. - -Other Production Environment Considerations -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -TBD. ``:)`` - -Over time, considerations may include services such as: - -- AAF -- AppMetrics -- Introscope -- Nagios -- Splunk -- UAM - -Installation and Configuration ------------------------------- - -**IMPORTANT**: Perform the steps in this section after *optionally* -configuring and activating a python virtual environment. - -Installing From a PyPI Repository -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -In ONAP, the ``conductor`` package can be found on \`\`\`\`. - -Installation is via the ``pip`` command. Here is an example ``pip.conf`` -file that uses both the internet and intranet-facing PyPI repositories: - -.. code:: ini - - [global] - index = https://pypi.python.org/pypi - index-url = https://pypi.python.org/simple - extra-index-url = - trusted-host = - -Once the configuration is in place, installation is simple: - -.. code:: bash - - $ pip install of-has - -To upgrade or downgrade, simply re-run ``pip install`` using the -appropriate ``pip`` command line options. - -**NOTE**: Be sure proxy settings are in place if they're required to -access ``pypi.python.org``. - -Installing From Source -~~~~~~~~~~~~~~~~~~~~~~ - -Conductor source in ONAP is maintained in -https://gerrit.onap.org/r/optf/has. - -Clone the git repository, and then install from within the ``conductor`` -directory: - -.. code:: bash - - $ git clone https://gerrit.onap.org/r/optf/has - Cloning into 'conductor'... - remote: Counting objects: 2291, done. - remote: Compressing objects: 88% (1918/2179) - remote: Compressing objects: 100% (2179/2179), done. - remote: Total 2291 (delta 1422), reused 0 (delta 0) - Receiving objects: 100% (2291/2291), 477.59 KiB | 0 bytes/s, done. - Resolving deltas: 100% (1422/1422), done. - $ cd conductor - $ pip install . - -The latest source can be pulled from ONAP at any time and reinstalled: - -.. code:: bash - - $ git pull - $ pip install . - -Verifying Installation -~~~~~~~~~~~~~~~~~~~~~~ - -Each of the five Conductor services may be invoked with the ``--help`` -option: - -.. code:: bash - - $ conductor-api -- --help - $ conductor-controller --help - $ conductor-data --help - $ conductor-solver --help - $ conductor-reservation --help - -**NOTE**: The ``conductor-api`` command is deliberate. ``--`` is used as -as separator between the arguments used to start the WSGI server and the -arguments passed to the WSGI application. - -Post-Flight and Post-Installation Considerations ------------------------------------------------- - -User and Group -~~~~~~~~~~~~~~ - -It's good practice to create an unprivileged account (e.g., a user/group -named ``conductor``) and run all Conductor services as that user: - -.. code:: bash - - $ sudo addgroup --system conductor - $ sudo adduser --system --home /var/lib/conductor --ingroup conductor --no-create-home --shell /bin/false conductor - -SSL/TLS Certificates -~~~~~~~~~~~~~~~~~~~~ - -The A&AI Inventory Provider Plugin requiries two-way SSL/TLS. After -provisioning a certificate per A&AI guidelines, it will be necessary to -securely install the certificate, key, and certificate authority bundle. - -When running conductor services as ``conductor:conductor`` -(recommended), consider co-locating all of these files under the -configuration directory. For example, when using ``/etc/conductor``: - -.. code:: bash - - $ # Certificate files (crt extension, 644 permissions) - $ sudo mkdir /etc/conductor/ssl/certs - $ # Private Certificate Key files (key extension, 640 permissions) - $ sudo mkdir /etc/conductor/ssl/private - $ # Certificate Authority (CA) Bundles (crt extension, 644 permissions) - $ sudo mkdir /etc/conductor/ssl/ca-certificates - $ # Add files to newly created directories, then set ownership - $ sudo chmod -R conductor:conductor /etc/conductor/ssl - -For a hypothetical domain name ``imacculate.client.research.att.com``, -example filenames could be as follows: - -.. code:: bash - - $ find ssl -type f -printf '%M %u:%g %f\n' - -rw-r----- conductor:conductor imacculate.client.research.att.com.key - -rw-r--r-- conductor:conductor Symantec_Class_3_Secure_Server_CA.crt - -rw-r--r-- conductor:conductor imacculate.client.research.att.com.crt - -When running conductor services as ``root``, consider these existing -Ubuntu filesystem locations for SSL/TLS files: - -**Certificate** files (``crt`` extension) are typically stored in -``/etc/ssl/certs`` with ``root:root`` ownership and 644 permissions. - -**Private Certificate Key** files (``key`` extension) are typically -stored in ``/etc/ssl/private`` with ``root:root`` ownership and 640 -permissions. - -**Certificate Authority (CA) Bundles** (``crt`` extension) are typically -stored in ``/usr/share/ca-certificates/conductor`` with ``root:root`` -ownership, and 644 permissions. These Bundle files are then symlinked -within ``/etc/ssl/certs`` using equivalent filenames, a ``pem`` -extension, and ``root:root`` ownership. - -**NOTE**: LUKS (Linux Unified Key Setup) is not supported by Conductor -at this time. - -Configuration -~~~~~~~~~~~~~ - -Configuration files are located in ``etc/conductor`` relative to the -python environment Conductor is installed in. - -To generate a sample configuration file, change to the directory just -above where ``etc/conductor`` is located (e.g., ``/`` for the default -environment, or the virtual environment root directory). Then: - -.. code:: bash - - $ oslo-config-generator --config-file=etc/conductor/conductor-config-generator.conf - -This will generate ``etc/conductor/conductor.conf.sample``. - -Because the configuration directory and files will include credentials, -consider removing world permissions: - -.. code:: bash - - $ find etc/conductor -type f -exec chmod 640 {} + - $ find etc/conductor -type d -exec chmod 750 {} + - -The sample config may then be copied and edited. Be sure to backup any -previous ``conductor.conf`` if necessary. - -.. code:: bash - - $ cd etc/conductor - $ cp -p conductor.conf.sample conductor.conf - -``conductor.conf`` is fully annotated with descriptions of all options. -Defaults are included, with all options commented out. Conductor will -use defaults even if an option is not present in the file. To change an -option, simply uncomment it and edit its value. - -With the exception of the ``DEFAULT`` section, it's best to restart the -Conductor services after making any config changes. In some cases, only -one particular service actually needs to be restarted. When in doubt, -however, it's best to restart all of them. - -A few options in particular warrant special attention: - -:: - - [DEFAULT] - - # If set to true, the logging level will be set to DEBUG instead of the default - # INFO level. (boolean value) - # Note: This option can be changed without restarting. - #debug = false - -For more verbose logging across all Conductor services, set ``debug`` to -true. - -:: - - [aai] - - # Base URL for A&AI, up to and not including the version, and without a - # trailing slash. (string value) - #server_url = https://controller:8443/aai - - # SSL/TLS certificate file in pem format. This certificate must be registered - # with the A&AI endpoint. (string value) - #certificate_file = certificate.pem - - # Private Certificate Key file in pem format. (string value) - #certificate_key_file = certificate_key.pem - - # Certificate Authority Bundle file in pem format. Must contain the appropriate - # trust chain for the Certificate file. (string value) - #certificate_authority_bundle_file = certificate_authority_bundle.pem - -Set ``server_url`` to the A&AI server URL, to but not including the -version, omitting any trailing slash. Conductor supports A&AI API v9 at -a minimum. - -Set the ``certificate`` prefixed keys to the appropriate SSL/TLS-related -files. - -**IMPORTANT**: The A&AI server may have a mismatched host/domain name -and SSL/TLS certificate. In such cases, certificate verification will -fail. To mitigate this, ``certificate_authority_bundle_file`` may be set -to an empty value. While Conductor normally requires a CA Bundle -(otherwise why bother using SSL/TLS), this requirement has been -temporarily relaxed so that development and testing may continue. - -:: - - [messaging_server] - - # Log debug messages. Default value is False. (boolean value) - #debug = false - -When the ``DEFAULT`` section's ``debug`` option is ``true``, set this -section's ``debug`` option to ``true`` to enable detailed Conductor-side -RPC-over-Music debug messages. - -Be aware, it is voluminous. "You have been warned." ``:)`` - -:: - - [music_api] - - # List of hostnames (round-robin access) (list value) - #hostnames = localhost - - # Log debug messages. Default value is False. (boolean value) - #debug = false - -Set ``hostnames`` to match wherever the Music REST API is being hosted -(wherever Apache Tomcat and ``MUSIC.war`` are located). - -When the ``DEFAULT`` section's ``debug`` option is ``true``, set this -section's ``debug`` option to ``true`` to enable detailed Conductor-side -MUSIC API debug messages. - -The previous comment around the volume of log lines applies even more so -here. (Srsly. We're not kidding.) - -**IMPORTANT**: Conductor does not presently use Music's atomic -consistency features due to concern around lock creation/acquisition. -Instead, Conductor uses eventual consistency. For this reason, -consistency issues may occur when using Music in a multi-server, High -Availability configuration. - -:: - - [sdnc] - - # Base URL for SDN-C. (string value) - #server_url = https://controller:8443/restconf - - # Basic Authentication Username (string value) - #username = <None> - - # Basic Authentication Password (string value) - #password = <None> - -Set ``server_url`` to the SDN-C server URL, omitting any trailing slash. - -Set ``username`` and ``password`` to the appropriate values as directed -by SDN-C. - -Running for the First Time -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Each Conductor component may be run interactively. In this case, the -user does not necessarily matter. - -When running interactively, it is suggested to run each command in a -separate terminal session and in the following order: - -.. code:: bash - - conductor-data --config-file=/etc/conductor/conductor.conf - conductor-controller --config-file=/etc/conductor/conductor.conf - conductor-solver --config-file=/etc/conductor/conductor.conf - conductor-reservation --config-file=/etc/conductor/conductor.conf - conductor-api --port=8091 -- --config-file=/etc/conductor/conductor.conf - -Optionally, use an application like -`screen <http://packages.ubuntu.com/trusty/screen>`__ to nest all five -terminal sessions within one detachable session. (This is also the same -package used by -`DevStack <https://docs.openstack.org/developer/devstack/>`__.) - -To verify that ``conductor-api`` can be reached, browse to -``http://HOST:8091/``, where HOST is the hostname ``conductor-api`` is -running on. No AuthN/AuthZ is required at this time. Depending on -network considerations, it may be necessary to use a command like -``wget`` instead of a desktop browser. - -The response should look similar to: - -.. code:: json - - { - "versions": { - "values": [ - { - "status": "development", - "updated": "2016-11-01T00:00:00Z", - "media-types": [ - { - "base": "application/json", - "type": "application/vnd.ecomp.homing-v1+json" - } - ], - "id": "v1", - "links": [ - { - "href": "http://127.0.0.1:8091/v1", - "rel": "self" - }, - { - "href": "http://conductor.research.att.com/", - "type": "text/html", - "rel": "describedby" - } - ] - } - ] - } - } - -Sample API Calls and Homing Templates -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -A `Postman <http://getpostman.com/>`__ collection illustrating sample -requests is available upon request. The collection will also be added in -a future revision. - -`Sample homing templates </doc/examples/README.md>`__ are also -available. - -Ubuntu Service Scripts -~~~~~~~~~~~~~~~~~~~~~~ - -Ubuntu sysvinit (init.d) and upstart (init) scripts are typically -installed at the Ubuntu package level. Since there is no such packaging -at this time, example scripts have been provided in the repository. - -To install, place all Conductor `sysvinit -scripts </examples/distribution/ubuntu/init.d>`__ in ``/etc/init.d``, -and all `upstart scripts </examples/distribution/ubuntu/init>`__ in -``/etc/init``. - -Set file permissions: - -.. code:: bash - - $ sudo chmod 644 /etc/init/conductor* - $ sudo chmod 755 /etc/init.d/conductor* - -If a python virtual environment is being used, edit each -``/etc/init/conductor*`` and ``/etc/init.d/conductor*`` prefixed file so -that ``PYTHON_HOME`` is set to the python virtual environment root -directory. - -Next, enable the scripts: - -.. code:: bash - - $ sudo update-rc.d conductor-api defaults - $ sudo update-rc.d conductor-controller defaults - $ sudo update-rc.d conductor-data defaults - $ sudo update-rc.d conductor-solver defaults - $ sudo update-rc.d conductor-reservation defaults - $ sudo initctl reload-configuration - -Conductor components may now be started/stopped like any other Ubuntu -service, for example: - -.. code:: bash - - $ sudo service conductor-api start - $ sudo service conductor-api status - $ sudo service conductor-api restart - $ sudo service conductor-api stop - -Conductor service scripts automatically create directories for ``log``, -``lock``, ``run``, ``lib``, and ``log`` files, e.g., -``/var/log/conductor`` and so on. - -Log File Rotation -~~~~~~~~~~~~~~~~~ - -Sample ``logrotate.d`` configuration files have been provided in the -repository. - -To install, place all Conductor `logrotate -files </examples/distribution/ubuntu/logrotate.d>`__ in -``/etc/logrotate.d``. - -Set file ownership and permissions: - -.. code:: bash - - $ sudo chown root:root /etc/logrotate.d/conductor* - $ sudo chmod 644 /etc/logrotate.d/conductor* - -``logrotate.d`` automatically recognizes new files at the next log -rotation opportunity and does not require restarting. - -Running conductor-api Under apache2 httpd and mod\_wsgi -------------------------------------------------------- - -Sample configuration files have been provided in the repository. - -These instructions presume a ``conductor`` user exists. See the -**Service Scripts** section for details. - -First, set up a few directories: - -.. code:: bash - - $ sudo mkdir -p /var/www/conductor - $ sudo mkdir /var/log/apache2/conductor - -To install, place the Conductor `WSGI application -file </conductor/api/app.wsgi>`__ in ``/var/www/conductor``. - -Set the owner/group of both directories/files to ``conductor``: - -.. code:: bash - - $ sudo chown -R conductor:conductor /var/log/apache2/conductor /var/www/conductor - -Next, place the Conductor `apache2 httpd site config -file </examples/apache2/conductor.conf>`__ in -``/etc/apache2/sites-available``. - -Set the owner/group to ``root``: - -.. code:: bash - - $ sudo chown -R root:root /etc/apache2/sites-available/conductor.conf - -If Conductor was installed in a python virtual environment, append -``python-home=VENV`` to ``WSGIDaemonProcess``, where ``VENV`` is the -python virtual environment root directory. - -**IMPORTANT**: Before proceeding, disable the ``conductor-api`` sysvinit -and upstart services, as the REST API will now be handled by apache2 -httpd. Otherwise there will be a port conflict, and you will be sad. - -Enable the Conductor site, ensure the configuration syntax is valid, and -gracefully restart apache2 httpd. - -.. code:: bash - - $ sudo a2ensite conductor - $ sudo apachectl -t - Syntax OK - $ sudo apachectl graceful - -To disable the Conductor site, run ``sudo a2dissite conductor``, then -gracefully restart once again. Optionally, re-enable the -``conductor-api`` sysvinit and upstart services. - -Running conductor-api Under nginx and uWSGI -------------------------------------------- - -Sample configuration files have been provided in the repository. - -These instructions presume a ``conductor`` user exists. See the -**Service Scripts** section for details. - -To install, place the Conductor `nginx config -files </examples/nginx/>`__ and `WSGI application -file </conductor/api/app.wsgi>`__ in ``/etc/nginx`` (taking care to -backup any prior configuration files). It may be desirable to -incorporate Conductor's ``nginx.conf`` into the existing config. - -Rename ``app.wsgi`` to ``conductor.wsgi``: - -.. code:: bash - - $ cd /etc/nginx - $ sudo mv app.wsgi conductor.wsgi - -In ``nginx.conf``, set ``CONDUCTOR_API_FQDN`` to the server name. - -**IMPORTANT**: Before proceeding, disable the ``conductor-api`` sysvinit -and upstart services, as the REST API will now be handled by nginx. -Otherwise there will be a port conflict, and you will be sad. - -Restart nginx: - -.. code:: bash - - $ sudo service nginx restart - -Then, run ``conductor-api`` under nginx using uWSGI: - -.. code:: bash - - $ sudo uwsgi -s /tmp/uwsgi.sock --chmod-socket=777 --wsgi-file /etc/nginx/conductor.wsgi --callable application --set port=8091 - -To use a python virtual environment, add ``--venv VENV`` to the -``uwsgi`` command, where ``VENV`` is the python virtual environment root -directory. - -Uninstallation --------------- - -Activate a virtual environment (venv) first, if necessary, then -uninstall with: - -.. code:: bash - - $ pip uninstall ecomp-conductor - -Remove any previously made configuration file changes, user accounts, -Ubuntu/SWM packages, and other settings as needed. - -Bug Reporting and Feedback --------------------------- - -... is encouraged. Please raise an issue at: -https://jira.onap.org/projects/OPTFRA/summary |