diff options
author | krishnaa96 <krishna.moorthy6@wipro.com> | 2021-06-01 09:57:40 +0530 |
---|---|---|
committer | krishnaa96 <krishna.moorthy6@wipro.com> | 2021-06-02 12:35:06 +0530 |
commit | 211b609bd426b22fe7a0fc60a970a787acc006ca (patch) | |
tree | 2ed0a472d5d552fbf2c39a9b85449f7850b440ae | |
parent | 5034a07fa6174039e8fb88e2d508932ed05a120c (diff) |
Update docs for music migration
- Update the Architecture diagram
- Split Installation page into
configuration and advanced options
Issue-ID: OPTFRA-964
Signed-off-by: krishnaa96 <krishna.moorthy6@wipro.com>
Change-Id: I214b05ef0f88cb11f50aa6156450506074ed06c0
-rw-r--r-- | docs/index.rst | 2 | ||||
-rw-r--r-- | docs/sections/architecture.rst | 57 | ||||
-rw-r--r-- | docs/sections/configuration.rst | 159 | ||||
-rw-r--r-- | docs/sections/diagrams/HAS_SeedCode_Architecture_R2.jpg | bin | 321115 -> 0 bytes | |||
-rw-r--r-- | docs/sections/diagrams/HAS_SeedCode_Architecture_R9.jpg | bin | 0 -> 346149 bytes | |||
-rw-r--r-- | docs/sections/installation-advanced.rst | 150 | ||||
-rw-r--r-- | docs/sections/installation.rst | 744 | ||||
-rw-r--r-- | docs/sections/logging.rst | 22 |
8 files changed, 468 insertions, 666 deletions
diff --git a/docs/index.rst b/docs/index.rst index 59759d4..86c78f1 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -42,6 +42,8 @@ is an orchestration service, or Nova is a compute service. ./sections/offeredapis.rst ./sections/consumedapis.rst ./sections/installation.rst + ./sections/configuration.rst + ./sections/installation-advanced.rst ./sections/logging.rst ./sections/homingspecification.rst Example Homing Templates <./sections/example.rst> diff --git a/docs/sections/architecture.rst b/docs/sections/architecture.rst index cfbd535..5588c75 100644 --- a/docs/sections/architecture.rst +++ b/docs/sections/architecture.rst @@ -39,7 +39,7 @@ sample homing template has been drawn up for residential vCPE Homing Use Case. HAS Architecture (R2) ---------------------- -.. image:: ./diagrams/HAS_SeedCode_Architecture_R2.jpg +.. image:: ./diagrams/HAS_SeedCode_Architecture_R9.jpg Lifecycle of a Homing request in HAS -------------------------------------------- @@ -59,3 +59,58 @@ A sample heuristic greedy algorithm of HAS (using a vCPE as example) ------------------------------------------------------------------------ .. image:: ./diagrams/HASHeuristicGreedyAlgorithm.jpg + +Components +---------- + +Conductor consists of five services that work together: + +- **``conductor-api``**: An HTTP REST API +- **``conductor-controller``**: Validation, translation, and + status/results +- **``conductor-data``**: Inventory provider and service controller + gateway +- **``conductor-solver``**: Processing and solution calculation +- **``conductor-reservation``**: Reserves the suggested solution solved + by Solver component. + +Workflow +-------- + +- Deployment **plans** are created, viewed, and deleted via + ``conductor-api`` and its `REST API <doc/api/README.md>`__. +- Included within each ``conductor-api`` plan request is a `Homing + Template <doc/template/README.md>`__. +- Homing Templates describe a set of inventory demands and constraints + to be solved against. +- ``conductor-api`` hands off all API requests to + ``conductor-controller`` for handling. +- All deployment plans are assigned a unique identifier (UUID-4), which + can be used to check for solution status asynchronously. (Conductor + does not support callbacks at this time.) +- ``conductor-controller`` ensures templates are well-formed and valid. + Errors and remediation are made visible through ``conductor-api``. + When running in debug mode, the API will also include a python + traceback in the response body, if available. +- ``conductor-controller`` uses ``conductor-data`` to resolve demands + against a particular **inventory provider** (e.g., A&AI). +- ``conductor-controller`` translates the template into a format + suitable for solving. +- As each template is translated, ``conductor-solver`` begins working + on it. +- ``conductor-solver`` uses ``conductor-data`` to resolve constraints + against a particular **service controller** (e.g., SDN-C). +- ``conductor-solver`` determines the most suitable inventory to + recommend. +- ``conductor-reservation`` attempts to reserve the solved solution in + SDN-GC + +**NOTE**: There is no Command Line Interface or Python API Library at +this time. + +DB Backend +---------- + +All Conductor services use a DB backend for data storage/persistence and/or +as a RPC transport mechanism. The current implementation supports two services which can be used as backend - `Music <https://github.com/att/music>`__ and +`ETCD <https://github.com/etcd-io/etcd>`__. diff --git a/docs/sections/configuration.rst b/docs/sections/configuration.rst new file mode 100644 index 0000000..968cece --- /dev/null +++ b/docs/sections/configuration.rst @@ -0,0 +1,159 @@ +Configuration +============= + +Configuration files are located in ``etc/conductor`` relative to the +python environment Conductor is installed in. + +To generate a sample configuration file, change to the directory just +above where ``etc/conductor`` is located (e.g., ``/`` for the default +environment, or the virtual environment root directory). Then: + +.. code:: bash + + $ oslo-config-generator --config-file=etc/conductor/conductor-config-generator.conf + +This will generate ``etc/conductor/conductor.conf.sample``. + +Because the configuration directory and files will include credentials, +consider removing world permissions: + +.. code:: bash + + $ find etc/conductor -type f -exec chmod 640 {} + + $ find etc/conductor -type d -exec chmod 750 {} + + +The sample config may then be copied and edited. Be sure to backup any +previous ``conductor.conf`` if necessary. + +.. code:: bash + + $ cd etc/conductor + $ cp -p conductor.conf.sample conductor.conf + +``conductor.conf`` is fully annotated with descriptions of all options. +Defaults are included, with all options commented out. Conductor will +use defaults even if an option is not present in the file. To change an +option, simply uncomment it and edit its value. + +With the exception of the ``DEFAULT`` section, it's best to restart the +Conductor services after making any config changes. In some cases, only +one particular service actually needs to be restarted. When in doubt, +however, it's best to restart all of them. + +A few options in particular warrant special attention: + +:: + + [DEFAULT] + + # If set to true, the logging level will be set to DEBUG instead of the default + # INFO level. (boolean value) + # Note: This option can be changed without restarting. + #debug = false + +For more verbose logging across all Conductor services, set ``debug`` to +true. + +:: + + [db_options] + + # db_backend to use + db_backend = etcd + + # Use music mock api + music_mock = False + +Set ``db_backend`` to the db(music/etcd) which is being deployed. Based on this +options, conductor will decide on using the corresponding client to access the +backend. + +:: + + [aai] + + # Base URL for A&AI, up to and not including the version, and without a + # trailing slash. (string value) + #server_url = https://controller:8443/aai + + # SSL/TLS certificate file in pem format. This certificate must be registered + # with the A&AI endpoint. (string value) + #certificate_file = certificate.pem + + # Private Certificate Key file in pem format. (string value) + #certificate_key_file = certificate_key.pem + + # Certificate Authority Bundle file in pem format. Must contain the appropriate + # trust chain for the Certificate file. (string value) + #certificate_authority_bundle_file = certificate_authority_bundle.pem + +Set ``server_url`` to the A&AI server URL, to but not including the +version, omitting any trailing slash. Conductor supports A&AI API v9 at +a minimum. + +Set the ``certificate`` prefixed keys to the appropriate SSL/TLS-related +files. + +**IMPORTANT**: The A&AI server may have a mismatched host/domain name +and SSL/TLS certificate. In such cases, certificate verification will +fail. To mitigate this, ``certificate_authority_bundle_file`` may be set +to an empty value. While Conductor normally requires a CA Bundle +(otherwise why bother using SSL/TLS), this requirement has been +temporarily relaxed so that development and testing may continue. + +:: + + [messaging_server] + + # Log debug messages. Default value is False. (boolean value) + #debug = false + +When the ``DEFAULT`` section's ``debug`` option is ``true``, set this +section's ``debug`` option to ``true`` to enable detailed Conductor-side +RPC-over-Music debug messages. + +Be aware, it is voluminous. "You have been warned." ``:)`` + +:: + + [music_api] + + # List of hostnames (round-robin access) (list value) + #hostnames = localhost + + # Log debug messages. Default value is False. (boolean value) + #debug = false + +Set ``hostnames`` to match wherever the Music REST API is being hosted +(wherever Apache Tomcat and ``MUSIC.war`` are located). + +When the ``DEFAULT`` section's ``debug`` option is ``true``, set this +section's ``debug`` option to ``true`` to enable detailed Conductor-side +MUSIC API debug messages. + +The previous comment around the volume of log lines applies even more so +here. (Srsly. We're not kidding.) + +**IMPORTANT**: Conductor does not presently use Music's atomic +consistency features due to concern around lock creation/acquisition. +Instead, Conductor uses eventual consistency. For this reason, +consistency issues may occur when using Music in a multi-server, High +Availability configuration. + +:: + + [sdnc] + + # Base URL for SDN-C. (string value) + #server_url = https://controller:8443/restconf + + # Basic Authentication Username (string value) + #username = <None> + + # Basic Authentication Password (string value) + #password = <None> + +Set ``server_url`` to the SDN-C server URL, omitting any trailing slash. + +Set ``username`` and ``password`` to the appropriate values as directed +by SDN-C. diff --git a/docs/sections/diagrams/HAS_SeedCode_Architecture_R2.jpg b/docs/sections/diagrams/HAS_SeedCode_Architecture_R2.jpg Binary files differdeleted file mode 100644 index 1d247ca..0000000 --- a/docs/sections/diagrams/HAS_SeedCode_Architecture_R2.jpg +++ /dev/null diff --git a/docs/sections/diagrams/HAS_SeedCode_Architecture_R9.jpg b/docs/sections/diagrams/HAS_SeedCode_Architecture_R9.jpg Binary files differnew file mode 100644 index 0000000..4141cd9 --- /dev/null +++ b/docs/sections/diagrams/HAS_SeedCode_Architecture_R9.jpg diff --git a/docs/sections/installation-advanced.rst b/docs/sections/installation-advanced.rst new file mode 100644 index 0000000..c0cf62e --- /dev/null +++ b/docs/sections/installation-advanced.rst @@ -0,0 +1,150 @@ +Installation - Advanced Options +=============================== + +Running conductor-api Under apache2 httpd and mod\_wsgi +------------------------------------------------------- + +``conductor-api`` may be run as-is for development and test purposes. +When used in a production environment, it is recommended that +``conductor-api`` run under a multithreaded httpd service supporting +`WSGI <https://www.wikipedia.org/wiki/Web_Server_Gateway_Interface>`__, +tuned as appropriate. + +Configuration instructions for **apache2 httpd** and **nginx** are +included herein. Respective package requirements are: + +- `apache2 <http://packages.ubuntu.com/focal/apache2>`__ and + `libapache2-mod-wsgi <http://packages.ubuntu.com/focal/libapache2-mod-wsgi>`__ +- `nginx <http://packages.ubuntu.com/focal/nginx>`__ and + `uwsgi <http://packages.ubuntu.com/focal/uwsgi>`__ + + +Sample configuration files have been provided in the repository. + +These instructions presume a ``conductor`` user exists. See the +**Service Scripts** section for details. + +First, set up a few directories: + +.. code:: bash + + $ sudo mkdir -p /var/www/conductor + $ sudo mkdir /var/log/apache2/conductor + +To install, place the Conductor `WSGI application +file </conductor/api/app.wsgi>`__ in ``/var/www/conductor``. + +Set the owner/group of both directories/files to ``conductor``: + +.. code:: bash + + $ sudo chown -R conductor:conductor /var/log/apache2/conductor /var/www/conductor + +Next, place the Conductor `apache2 httpd site config +file </examples/apache2/conductor.conf>`__ in +``/etc/apache2/sites-available``. + +Set the owner/group to ``root``: + +.. code:: bash + + $ sudo chown -R root:root /etc/apache2/sites-available/conductor.conf + +If Conductor was installed in a python virtual environment, append +``python-home=VENV`` to ``WSGIDaemonProcess``, where ``VENV`` is the +python virtual environment root directory. + +**IMPORTANT**: Before proceeding, disable the ``conductor-api`` sysvinit +and upstart services, as the REST API will now be handled by apache2 +httpd. Otherwise there will be a port conflict, and you will be sad. + +Enable the Conductor site, ensure the configuration syntax is valid, and +gracefully restart apache2 httpd. + +.. code:: bash + + $ sudo a2ensite conductor + $ sudo apachectl -t + Syntax OK + $ sudo apachectl graceful + +To disable the Conductor site, run ``sudo a2dissite conductor``, then +gracefully restart once again. Optionally, re-enable the +``conductor-api`` sysvinit and upstart services. + +Running conductor-api Under nginx and uWSGI +------------------------------------------- + +Sample configuration files have been provided in the repository. + +These instructions presume a ``conductor`` user exists. See the +**Service Scripts** section for details. + +To install, place the Conductor `nginx config +files </examples/nginx/>`__ and `WSGI application +file </conductor/api/app.wsgi>`__ in ``/etc/nginx`` (taking care to +backup any prior configuration files). It may be desirable to +incorporate Conductor's ``nginx.conf`` into the existing config. + +Rename ``app.wsgi`` to ``conductor.wsgi``: + +.. code:: bash + + $ cd /etc/nginx + $ sudo mv app.wsgi conductor.wsgi + +In ``nginx.conf``, set ``CONDUCTOR_API_FQDN`` to the server name. + +**IMPORTANT**: Before proceeding, disable the ``conductor-api`` sysvinit +and upstart services, as the REST API will now be handled by nginx. +Otherwise there will be a port conflict, and you will be sad. + +Restart nginx: + +.. code:: bash + + $ sudo service nginx restart + +Then, run ``conductor-api`` under nginx using uWSGI: + +.. code:: bash + + $ sudo uwsgi -s /tmp/uwsgi.sock --chmod-socket=777 --wsgi-file /etc/nginx/conductor.wsgi --callable application --set port=8091 + +To use a python virtual environment, add ``--venv VENV`` to the +``uwsgi`` command, where ``VENV`` is the python virtual environment root +directory. + +Networking +---------- + +All conductor services require line-of-sight access to all Music/ETCD +servers/ports. + +The ``conductor-api`` service uses TCP port 8091. + +Security +-------- + +``conductor-api`` is accessed via HTTP. SSL/TLS certificates and +AuthN/AuthZ (e.g., AAF) are supported at this time in kubernetes +environment. + +Conductor makes use of plugins that act as gateways to *inventory +providers* and *service controllers*. At present, two plugins are +supported out-of-the-box: **A&AI** and **SDN-C**, respectively. + +A&AI requires two-way SSL/TLS. Certificates must be registered and +whitelisted with A&AI. SDN-C uses HTTP Basic Authentication. Consult +with each respective service for official information on how to obtain +access. + +Storage +------- + +For a cloud environment in particular, it may be desirable to use a +separate block storage device (e.g., an OpenStack Cinder volume) for +logs, configuration, and other data persistence. In this way, it becomes +a trivial matter to replace the entire VM if necessary, followed by +reinstallation of the app and any supplemental configuration. Take this +into consideration when setting various Conductor config options. diff --git a/docs/sections/installation.rst b/docs/sections/installation.rst index d4b93d6..217a6dc 100644 --- a/docs/sections/installation.rst +++ b/docs/sections/installation.rst @@ -1,239 +1,112 @@ -Python/Linux Distribution Notes -=============================== - -*Updated: 10 Nov 2017 23:30 GMT* - -This document exists to help bridge the gap between the Conductor python -package and any downstream distribution. The steps outlined herein may -be taken into consideration when creating an AT&T SWM package, -Ubuntu/Debian package, Chef cookbook, or Ansible playbook. - -Components ----------- - -Conductor consists of five services that work together: - -- **``conductor-api``**: An HTTP REST API -- **``conductor-controller``**: Validation, translation, and - status/results -- **``conductor-data``**: Inventory provider and service controller - gateway -- **``conductor-solver``**: Processing and solution calculation -- **``conductor-reservation``**: Reserves the suggested solution solved - by Solver component. - -Workflow --------- - -- Deployment **plans** are created, viewed, and deleted via - ``conductor-api`` and its `REST API <doc/api/README.md>`__. -- Included within each ``conductor-api`` plan request is a `Homing - Template <doc/template/README.md>`__. -- Homing Templates describe a set of inventory demands and constraints - to be solved against. -- ``conductor-api`` hands off all API requests to - ``conductor-controller`` for handling. -- All deployment plans are assigned a unique identifier (UUID-4), which - can be used to check for solution status asynchronously. (Conductor - does not support callbacks at this time.) -- ``conductor-controller`` ensures templates are well-formed and valid. - Errors and remediation are made visible through ``conductor-api``. - When running in debug mode, the API will also include a python - traceback in the response body, if available. -- ``conductor-controller`` uses ``conductor-data`` to resolve demands - against a particular **inventory provider** (e.g., A&AI). -- ``conductor-controller`` translates the template into a format - suitable for solving. -- As each template is translated, ``conductor-solver`` begins working - on it. -- ``conductor-solver`` uses ``conductor-data`` to resolve constraints - against a particular **service controller** (e.g., SDN-C). -- ``conductor-solver`` determines the most suitable inventory to - recommend. -- ``conductor-reservation`` attempts to reserve the solved solution in - SDN-GC - -**NOTE**: There is no Command Line Interface or Python API Library at -this time. - -Pre-Flight and Pre-Installation Considerations ----------------------------------------------- - -AT&T Application Identifiers and Roles -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -- App/Tool Name: ECOMP Conductor -- MOTS Application ID: 26213 -- MechID: m04308 -- ECOMP Feature ID: F13704 -- PMT: 461306 -- UAM Role Name: Conductor Production Support -- UAM Role id: 0000025248 - -Root -~~~~ - -Be aware that some commands may require ``sudo``, depending on the -account being used to perform the installation. - -Proxy -~~~~~ - -If line-of-sight to internet-facing repositories is permitted and -available, set the following shell environment variables if AT&T proxy -services are required: +Installation +============ -.. code:: bash - - $ export http_proxy="http://one.proxy.att.com:8080/" - $ export https_proxy="http://one.proxy.att.com:8080/" - -Requirements -~~~~~~~~~~~~ - -Conductor is officially supported on `Ubuntu 14.04 LTS (Trusty -Tahr) <http://releases.ubuntu.com/14.04/>`__, though it should also work -on newer releases. +OOF-HAS OOM Charts +------------------ -Ensure the following Ubuntu packages are present, as they may not be -included by default: - -- `libffi-dev <http://packages.ubuntu.com/trusty/libffi-dev>`__ -- `postgresql-server-dev-9.3 <http://packages.ubuntu.com/trusty/postgresql-server-dev-9.3>`__ -- `python2.7 <http://packages.ubuntu.com/trusty/python2.7>`__ - -``conductor-api`` may be run as-is for development and test purposes. -When used in a production environment, it is recommended that -``conductor-api`` run under a multithreaded httpd service supporting -`WSGI <https://www.wikipedia.org/wiki/Web_Server_Gateway_Interface>`__, -tuned as appropriate. +HAS charts are located in the `OOM repository <https://git.onap.org/oom/>`__ -Configuration instructions for **apache2 httpd** and **nginx** are -included herein. Respective package requirements are: +Please refer OOM documentation for deploying/undeploying the OOF compoenents +via helm charts in the k8s environment. -- `apache2 <http://packages.ubuntu.com/trusty/apache2>`__ and - `libapache2-mod-wsgi <http://packages.ubuntu.com/trusty/libapache2-mod-wsgi>`__ -- `nginx <http://packages.ubuntu.com/trusty/nginx>`__ and - `uwsgi <http://packages.ubuntu.com/trusty/uwsgi>`__ +Local Installation +------------------ -All Conductor services use AT&T `Music <https://github.com/att/music>`__ -for data storage/persistence and/or as a RPC transport mechanism. -Consult the `Music Local Installation -Guide <https://github.com/att/music/blob/master/README.md>`__ for -installation/configuration steps. +HAS components can be deployed in two ways in a local environment for development +and testing. -Networking -~~~~~~~~~~ +Docker Installation +------------------- -All conductor services require line-of-sight access to all Music -servers/ports. - -The ``conductor-api`` service uses TCP port 8091. - -Security -~~~~~~~~ +Building Docker Images +~~~~~~~~~~~~~~~~~~~~~~ -``conductor-api`` is accessed via HTTP. SSL/TLS certificates and -AuthN/AuthZ (e.g., AAF) are not supported at this time. +Build the HAS docker images using the maven build from the root of the project -Conductor makes use of plugins that act as gateways to *inventory -providers* and *service controllers*. At present, two plugins are -supported out-of-the-box: **A&AI** and **SDN-C**, respectively. +.. code:: bash -A&AI requires two-way SSL/TLS. Certificates must be registered and -whitelisted with A&AI. SDN-C uses HTTP Basic Authentication. Consult -with each respective service for official information on how to obtain -access. + git clone --depth 1 https://gerrit.onap.org/r/optf/has + cd has + mvn clean install -Storage -~~~~~~~ +Installing the components and simulators +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -For a cloud environment in particular, it may be desirable to use a -separate block storage device (e.g., an OpenStack Cinder volume) for -logs, configuration, and other data persistence. In this way, it becomes -a trivial matter to replace the entire VM if necessary, followed by -reinstallation of the app and any supplemental configuration. Take this -into consideration when setting various Conductor config options. +HAS docker containers can be installed using the shell scripts in the CSIT directory +which includes the script to deploy the startup dependencies(SMS, ETCD) and a +few simulators. -Python Virtual Environments -~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. code:: bash -At present, Conductor installation is only supported at the (upstream) -python package level and not the (downstream) Ubuntu distribution or SWM -package levels. + export WORKSPACE=$(pwd)/csit + ./csit/plans/default/setup.sh -To mitigate/eliminate the risk of introducing conflicts with other -python applications or Ubuntu/SWM package dependencies, consider -installing Conductor in a `python virtual -environment <http://docs.python-guide.org/en/latest/dev/virtualenvs/>`__ -(or *venv* for short). +Similarly the installed components can be deleted using the teardown script. -Example venv-aware WSGI app configurations, sysvinit scripts, and -upstart scripts can be found in the Conductor repository under -`examples </examples/>`__. +.. code:: bash -Python Package Dependencies -~~~~~~~~~~~~~~~~~~~~~~~~~~~ + export WORKSPACE=$(pwd)/csit + ./csit/plans/default/teardown.sh -Conductor is installed using the python ``pip`` command. ``pip`` uses a -python project's `requirements manifest </requirements.txt>`__ to -install all python module dependencies. +Note: The simulator setup can be disabled by the commenting out the commands from +the setup scripts. -**NOTE**: When line-of-sight access to a PyPI-compatible package index -is not available, advance installation of Conductor's python package -dependencies is required *before* installation. +Installation from the source +---------------------------- -Other Production Environment Considerations -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +HAS components can be installed locally by directly in a linux based environment. +This will be significant for testing and debugging during developme -TBD. ``:)`` +Requirements +~~~~~~~~~~~~ -Over time, considerations may include services such as: +Conductor is officially supported on most of the linux based environment, but of +the development and testing were done on Ubuntu based machines. -- AAF -- AppMetrics -- Introscope -- Nagios -- Splunk -- UAM +Ensure the following packages are present, as they may not be +included by default: -Installation and Configuration ------------------------------- +- libffi-dev +- python3.8 -**IMPORTANT**: Perform the steps in this section after *optionally* -configuring and activating a python virtual environment. +Installing Dependent Components(AAF-SMS, ETCD/MUSIC) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Installing From a PyPI Repository -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The scripts to install and uninstall the components are present in the +CSIT directory. -In ONAP, the ``conductor`` package can be found on \`\`\`\`. +**Note**: For setting up SMS, ETCD and MUSIC, docker must be present in +the machine. -Installation is via the ``pip`` command. Here is an example ``pip.conf`` -file that uses both the internet and intranet-facing PyPI repositories: +For installing/uninstalling AAF-SMS, -.. code:: ini +.. code:: bash - [global] - index = https://pypi.python.org/pypi - index-url = https://pypi.python.org/simple - extra-index-url = - trusted-host = + cd csit/scripts + # install SMS + source setup-sms.sh + # uninstall SMS + docker stop sms + docker stop vault + docker rm sms + docker rm vault -Once the configuration is in place, installation is simple: +For installing/uninstalling ETCD .. code:: bash - $ pip install --no-cache-dir of-has - -To upgrade or downgrade, simply re-run ``pip install --no-cache-dir`` using the -appropriate ``pip`` command line options. - -**NOTE**: Be sure proxy settings are in place if they're required to -access ``pypi.python.org``. + cd csit/scripts + # install etcd + source etcd_Script.sh + # uninstall etcd + source etcd_teardown_script.sh Installing From Source ~~~~~~~~~~~~~~~~~~~~~~ +**IMPORTANT**: Perform the steps in this section after *optionally* +configuring and activating a python virtual environment. + Conductor source in ONAP is maintained in https://gerrit.onap.org/r/optf/has. @@ -242,23 +115,9 @@ directory: .. code:: bash - $ git clone --depth 1 https://gerrit.onap.org/r/optf/has - Cloning into 'conductor'... - remote: Counting objects: 2291, done. - remote: Compressing objects: 88% (1918/2179) - remote: Compressing objects: 100% (2179/2179), done. - remote: Total 2291 (delta 1422), reused 0 (delta 0) - Receiving objects: 100% (2291/2291), 477.59 KiB | 0 bytes/s, done. - Resolving deltas: 100% (1422/1422), done. - $ cd conductor - $ pip install --no-cache-dir . - -The latest source can be pulled from ONAP at any time and reinstalled: - -.. code:: bash - - $ git pull - $ pip install --no-cache-dir . + git clone --depth 1 https://gerrit.onap.org/r/optf/has + cd conductor + pip install --no-cache-dir -e . Verifying Installation ~~~~~~~~~~~~~~~~~~~~~~ @@ -268,227 +127,16 @@ option: .. code:: bash - $ conductor-api -- --help - $ conductor-controller --help - $ conductor-data --help - $ conductor-solver --help - $ conductor-reservation --help + conductor-api -- --help + conductor-controller --help + conductor-data --help + conductor-solver --help + conductor-reservation --help **NOTE**: The ``conductor-api`` command is deliberate. ``--`` is used as as separator between the arguments used to start the WSGI server and the arguments passed to the WSGI application. -Post-Flight and Post-Installation Considerations ------------------------------------------------- - -User and Group -~~~~~~~~~~~~~~ - -It's good practice to create an unprivileged account (e.g., a user/group -named ``conductor``) and run all Conductor services as that user: - -.. code:: bash - - $ sudo addgroup --system conductor - $ sudo adduser --system --home /var/lib/conductor --ingroup conductor --no-create-home --shell /bin/false conductor - -SSL/TLS Certificates -~~~~~~~~~~~~~~~~~~~~ - -The A&AI Inventory Provider Plugin requiries two-way SSL/TLS. After -provisioning a certificate per A&AI guidelines, it will be necessary to -securely install the certificate, key, and certificate authority bundle. - -When running conductor services as ``conductor:conductor`` -(recommended), consider co-locating all of these files under the -configuration directory. For example, when using ``/etc/conductor``: - -.. code:: bash - - $ # Certificate files (crt extension, 644 permissions) - $ sudo mkdir /etc/conductor/ssl/certs - $ # Private Certificate Key files (key extension, 640 permissions) - $ sudo mkdir /etc/conductor/ssl/private - $ # Certificate Authority (CA) Bundles (crt extension, 644 permissions) - $ sudo mkdir /etc/conductor/ssl/ca-certificates - $ # Add files to newly created directories, then set ownership - $ sudo chmod -R conductor:conductor /etc/conductor/ssl - -For a hypothetical domain name ``imacculate.client.research.att.com``, -example filenames could be as follows: - -.. code:: bash - - $ find ssl -type f -printf '%M %u:%g %f\n' - -rw-r----- conductor:conductor imacculate.client.research.att.com.key - -rw-r--r-- conductor:conductor Symantec_Class_3_Secure_Server_CA.crt - -rw-r--r-- conductor:conductor imacculate.client.research.att.com.crt - -When running conductor services as ``root``, consider these existing -Ubuntu filesystem locations for SSL/TLS files: - -**Certificate** files (``crt`` extension) are typically stored in -``/etc/ssl/certs`` with ``root:root`` ownership and 644 permissions. - -**Private Certificate Key** files (``key`` extension) are typically -stored in ``/etc/ssl/private`` with ``root:root`` ownership and 640 -permissions. - -**Certificate Authority (CA) Bundles** (``crt`` extension) are typically -stored in ``/usr/share/ca-certificates/conductor`` with ``root:root`` -ownership, and 644 permissions. These Bundle files are then symlinked -within ``/etc/ssl/certs`` using equivalent filenames, a ``pem`` -extension, and ``root:root`` ownership. - -**NOTE**: LUKS (Linux Unified Key Setup) is not supported by Conductor -at this time. - -Configuration -~~~~~~~~~~~~~ - -Configuration files are located in ``etc/conductor`` relative to the -python environment Conductor is installed in. - -To generate a sample configuration file, change to the directory just -above where ``etc/conductor`` is located (e.g., ``/`` for the default -environment, or the virtual environment root directory). Then: - -.. code:: bash - - $ oslo-config-generator --config-file=etc/conductor/conductor-config-generator.conf - -This will generate ``etc/conductor/conductor.conf.sample``. - -Because the configuration directory and files will include credentials, -consider removing world permissions: - -.. code:: bash - - $ find etc/conductor -type f -exec chmod 640 {} + - $ find etc/conductor -type d -exec chmod 750 {} + - -The sample config may then be copied and edited. Be sure to backup any -previous ``conductor.conf`` if necessary. - -.. code:: bash - - $ cd etc/conductor - $ cp -p conductor.conf.sample conductor.conf - -``conductor.conf`` is fully annotated with descriptions of all options. -Defaults are included, with all options commented out. Conductor will -use defaults even if an option is not present in the file. To change an -option, simply uncomment it and edit its value. - -With the exception of the ``DEFAULT`` section, it's best to restart the -Conductor services after making any config changes. In some cases, only -one particular service actually needs to be restarted. When in doubt, -however, it's best to restart all of them. - -A few options in particular warrant special attention: - -:: - - [DEFAULT] - - # If set to true, the logging level will be set to DEBUG instead of the default - # INFO level. (boolean value) - # Note: This option can be changed without restarting. - #debug = false - -For more verbose logging across all Conductor services, set ``debug`` to -true. - -:: - - [aai] - - # Base URL for A&AI, up to and not including the version, and without a - # trailing slash. (string value) - #server_url = https://controller:8443/aai - - # SSL/TLS certificate file in pem format. This certificate must be registered - # with the A&AI endpoint. (string value) - #certificate_file = certificate.pem - - # Private Certificate Key file in pem format. (string value) - #certificate_key_file = certificate_key.pem - - # Certificate Authority Bundle file in pem format. Must contain the appropriate - # trust chain for the Certificate file. (string value) - #certificate_authority_bundle_file = certificate_authority_bundle.pem - -Set ``server_url`` to the A&AI server URL, to but not including the -version, omitting any trailing slash. Conductor supports A&AI API v9 at -a minimum. - -Set the ``certificate`` prefixed keys to the appropriate SSL/TLS-related -files. - -**IMPORTANT**: The A&AI server may have a mismatched host/domain name -and SSL/TLS certificate. In such cases, certificate verification will -fail. To mitigate this, ``certificate_authority_bundle_file`` may be set -to an empty value. While Conductor normally requires a CA Bundle -(otherwise why bother using SSL/TLS), this requirement has been -temporarily relaxed so that development and testing may continue. - -:: - - [messaging_server] - - # Log debug messages. Default value is False. (boolean value) - #debug = false - -When the ``DEFAULT`` section's ``debug`` option is ``true``, set this -section's ``debug`` option to ``true`` to enable detailed Conductor-side -RPC-over-Music debug messages. - -Be aware, it is voluminous. "You have been warned." ``:)`` - -:: - - [music_api] - - # List of hostnames (round-robin access) (list value) - #hostnames = localhost - - # Log debug messages. Default value is False. (boolean value) - #debug = false - -Set ``hostnames`` to match wherever the Music REST API is being hosted -(wherever Apache Tomcat and ``MUSIC.war`` are located). - -When the ``DEFAULT`` section's ``debug`` option is ``true``, set this -section's ``debug`` option to ``true`` to enable detailed Conductor-side -MUSIC API debug messages. - -The previous comment around the volume of log lines applies even more so -here. (Srsly. We're not kidding.) - -**IMPORTANT**: Conductor does not presently use Music's atomic -consistency features due to concern around lock creation/acquisition. -Instead, Conductor uses eventual consistency. For this reason, -consistency issues may occur when using Music in a multi-server, High -Availability configuration. - -:: - - [sdnc] - - # Base URL for SDN-C. (string value) - #server_url = https://controller:8443/restconf - - # Basic Authentication Username (string value) - #username = <None> - - # Basic Authentication Password (string value) - #password = <None> - -Set ``server_url`` to the SDN-C server URL, omitting any trailing slash. - -Set ``username`` and ``password`` to the appropriate values as directed -by SDN-C. - Running for the First Time ~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -506,51 +154,6 @@ separate terminal session and in the following order: conductor-reservation --config-file=/etc/conductor/conductor.conf conductor-api --port=8091 -- --config-file=/etc/conductor/conductor.conf -Optionally, use an application like -`screen <http://packages.ubuntu.com/trusty/screen>`__ to nest all five -terminal sessions within one detachable session. (This is also the same -package used by -`DevStack <https://docs.openstack.org/developer/devstack/>`__.) - -To verify that ``conductor-api`` can be reached, browse to -``http://HOST:8091/``, where HOST is the hostname ``conductor-api`` is -running on. No AuthN/AuthZ is required at this time. Depending on -network considerations, it may be necessary to use a command like -``wget`` instead of a desktop browser. - -The response should look similar to: - -.. code:: json - - { - "versions": { - "values": [ - { - "status": "development", - "updated": "2016-11-01T00:00:00Z", - "media-types": [ - { - "base": "application/json", - "type": "application/vnd.ecomp.homing-v1+json" - } - ], - "id": "v1", - "links": [ - { - "href": "http://127.0.0.1:8091/v1", - "rel": "self" - }, - { - "href": "http://conductor.research.att.com/", - "type": "text/html", - "rel": "describedby" - } - ] - } - ] - } - } - Sample API Calls and Homing Templates ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -558,192 +161,5 @@ A `Postman <http://getpostman.com/>`__ collection illustrating sample requests is available upon request. The collection will also be added in a future revision. -`Sample homing templates </doc/examples/README.md>`__ are also +`Sample homing templates <example.html>`__ are also available. - -Ubuntu Service Scripts -~~~~~~~~~~~~~~~~~~~~~~ - -Ubuntu sysvinit (init.d) and upstart (init) scripts are typically -installed at the Ubuntu package level. Since there is no such packaging -at this time, example scripts have been provided in the repository. - -To install, place all Conductor `sysvinit -scripts </examples/distribution/ubuntu/init.d>`__ in ``/etc/init.d``, -and all `upstart scripts </examples/distribution/ubuntu/init>`__ in -``/etc/init``. - -Set file permissions: - -.. code:: bash - - $ sudo chmod 644 /etc/init/conductor* - $ sudo chmod 755 /etc/init.d/conductor* - -If a python virtual environment is being used, edit each -``/etc/init/conductor*`` and ``/etc/init.d/conductor*`` prefixed file so -that ``PYTHON_HOME`` is set to the python virtual environment root -directory. - -Next, enable the scripts: - -.. code:: bash - - $ sudo update-rc.d conductor-api defaults - $ sudo update-rc.d conductor-controller defaults - $ sudo update-rc.d conductor-data defaults - $ sudo update-rc.d conductor-solver defaults - $ sudo update-rc.d conductor-reservation defaults - $ sudo initctl reload-configuration - -Conductor components may now be started/stopped like any other Ubuntu -service, for example: - -.. code:: bash - - $ sudo service conductor-api start - $ sudo service conductor-api status - $ sudo service conductor-api restart - $ sudo service conductor-api stop - -Conductor service scripts automatically create directories for ``log``, -``lock``, ``run``, ``lib``, and ``log`` files, e.g., -``/var/log/conductor`` and so on. - -Log File Rotation -~~~~~~~~~~~~~~~~~ - -Sample ``logrotate.d`` configuration files have been provided in the -repository. - -To install, place all Conductor `logrotate -files </examples/distribution/ubuntu/logrotate.d>`__ in -``/etc/logrotate.d``. - -Set file ownership and permissions: - -.. code:: bash - - $ sudo chown root:root /etc/logrotate.d/conductor* - $ sudo chmod 644 /etc/logrotate.d/conductor* - -``logrotate.d`` automatically recognizes new files at the next log -rotation opportunity and does not require restarting. - -Running conductor-api Under apache2 httpd and mod\_wsgi -------------------------------------------------------- - -Sample configuration files have been provided in the repository. - -These instructions presume a ``conductor`` user exists. See the -**Service Scripts** section for details. - -First, set up a few directories: - -.. code:: bash - - $ sudo mkdir -p /var/www/conductor - $ sudo mkdir /var/log/apache2/conductor - -To install, place the Conductor `WSGI application -file </conductor/api/app.wsgi>`__ in ``/var/www/conductor``. - -Set the owner/group of both directories/files to ``conductor``: - -.. code:: bash - - $ sudo chown -R conductor:conductor /var/log/apache2/conductor /var/www/conductor - -Next, place the Conductor `apache2 httpd site config -file </examples/apache2/conductor.conf>`__ in -``/etc/apache2/sites-available``. - -Set the owner/group to ``root``: - -.. code:: bash - - $ sudo chown -R root:root /etc/apache2/sites-available/conductor.conf - -If Conductor was installed in a python virtual environment, append -``python-home=VENV`` to ``WSGIDaemonProcess``, where ``VENV`` is the -python virtual environment root directory. - -**IMPORTANT**: Before proceeding, disable the ``conductor-api`` sysvinit -and upstart services, as the REST API will now be handled by apache2 -httpd. Otherwise there will be a port conflict, and you will be sad. - -Enable the Conductor site, ensure the configuration syntax is valid, and -gracefully restart apache2 httpd. - -.. code:: bash - - $ sudo a2ensite conductor - $ sudo apachectl -t - Syntax OK - $ sudo apachectl graceful - -To disable the Conductor site, run ``sudo a2dissite conductor``, then -gracefully restart once again. Optionally, re-enable the -``conductor-api`` sysvinit and upstart services. - -Running conductor-api Under nginx and uWSGI -------------------------------------------- - -Sample configuration files have been provided in the repository. - -These instructions presume a ``conductor`` user exists. See the -**Service Scripts** section for details. - -To install, place the Conductor `nginx config -files </examples/nginx/>`__ and `WSGI application -file </conductor/api/app.wsgi>`__ in ``/etc/nginx`` (taking care to -backup any prior configuration files). It may be desirable to -incorporate Conductor's ``nginx.conf`` into the existing config. - -Rename ``app.wsgi`` to ``conductor.wsgi``: - -.. code:: bash - - $ cd /etc/nginx - $ sudo mv app.wsgi conductor.wsgi - -In ``nginx.conf``, set ``CONDUCTOR_API_FQDN`` to the server name. - -**IMPORTANT**: Before proceeding, disable the ``conductor-api`` sysvinit -and upstart services, as the REST API will now be handled by nginx. -Otherwise there will be a port conflict, and you will be sad. - -Restart nginx: - -.. code:: bash - - $ sudo service nginx restart - -Then, run ``conductor-api`` under nginx using uWSGI: - -.. code:: bash - - $ sudo uwsgi -s /tmp/uwsgi.sock --chmod-socket=777 --wsgi-file /etc/nginx/conductor.wsgi --callable application --set port=8091 - -To use a python virtual environment, add ``--venv VENV`` to the -``uwsgi`` command, where ``VENV`` is the python virtual environment root -directory. - -Uninstallation --------------- - -Activate a virtual environment (venv) first, if necessary, then -uninstall with: - -.. code:: bash - - $ pip uninstall ecomp-conductor - -Remove any previously made configuration file changes, user accounts, -Ubuntu/SWM packages, and other settings as needed. - -Bug Reporting and Feedback --------------------------- - -... is encouraged. Please raise an issue at: -https://jira.onap.org/projects/OPTFRA/summary diff --git a/docs/sections/logging.rst b/docs/sections/logging.rst index 86b63f9..2078ade 100644 --- a/docs/sections/logging.rst +++ b/docs/sections/logging.rst @@ -13,4 +13,24 @@ Timestamp|RequestId|ServiceInstanceId|ThreadId|Virtual Server Name|ServiceName|I The logger util module can be found at: -<>/has/conductor/conductor/common/utils/conductor_logging_util.py
\ No newline at end of file +<>/has/conductor/conductor/common/utils/conductor_logging_util.py + +Log File Rotation +----------------- + +Sample ``logrotate.d`` configuration files have been provided in the +repository. + +To install, place all Conductor `logrotate +files </examples/distribution/ubuntu/logrotate.d>`__ in +``/etc/logrotate.d``. + +Set file ownership and permissions: + +.. code:: bash + + $ sudo chown root:root /etc/logrotate.d/conductor* + $ sudo chmod 644 /etc/logrotate.d/conductor* + +``logrotate.d`` automatically recognizes new files at the next log +rotation opportunity and does not require restarting. |