[DOCS] Add helm3 install guide

Document helm 3 deployment as our experimental feature.

Issue-ID: OOM-2562
Change-Id: I188f53d7b90657d710109e6966220e0cfb9db8be
Signed-off-by: Jakub Latusek <j.latusek@samsung.com>
[Fix issues reported in review]
Signed-off-by: Krzysztof Opasiak <k.opasiak@samsung.com>
(cherry picked from commit 222b48bc355af6bf2a6dc10622b71c9be00e590d)

3 files changed, 982 insertions, 0 deletions

diff --git a/docs/index.rst b/docs/index.rst
index c3902ecae0..68b38de9aa 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -12,7 +12,9 @@ OOM Documentation Repository
 
   oom_project_description.rst
   oom_quickstart_guide.rst
+  oom_quickstart_guide_helm3.rst
   oom_user_guide.rst
+  oom_user_guide_helm3.rst
   oom_developer_guide.rst
   oom_cloud_setup_guide.rst
   release-notes.rst
diff --git a/docs/oom_quickstart_guide_helm3.rst b/docs/oom_quickstart_guide_helm3.rst
new file mode 100644
index 0000000000..5a3076426e
--- /dev/null
+++ b/docs/oom_quickstart_guide_helm3.rst
@@ -0,0 +1,252 @@
+.. This work is licensed under a
+.. Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. Copyright 2019-2020 Amdocs, Bell Canada, Orange, Samsung
+.. _oom_quickstart_guide_helm3:
+.. _quick-start-label-helm3:
+
+OOM Quick Start Guide Helm3 (experimental)
+###########################################
+
+.. figure:: oomLogoV2-medium.png
+   :align: right
+
+Once a Kubernetes environment is available (follow the instructions in
+:ref:`cloud-setup-guide-label` if you don't have a cloud environment
+available), follow the following instructions to deploy ONAP.
+
+**Step 1.** Clone the OOM repository from ONAP gerrit::
+
+  > git clone -b <BRANCH> http://gerrit.onap.org/r/oom --recurse-submodules
+  > cd oom/kubernetes
+
+where <BRANCH> can be an official release tag, such as
+
+* 4.0.0-ONAP for Dublin
+* 5.0.1-ONAP for El Alto
+* 6.0.0 for Frankfurt
+* 7.0.0 for Guilin
+
+**Step 2.** Install Helm Plugins required to deploy ONAP::
+
+  > cp -R ~/oom/kubernetes/helm/plugins/ ~/.local/share/helm/plugins
+  > helm plugin install https://github.com/chartmuseum/helm-push.git
+
+**Step 3** Install Chartmuseum::
+
+  > curl -LO https://s3.amazonaws.com/chartmuseum/release/latest/bin/linux/amd64/chartmuseum
+  > chmod +x ./chartmuseum
+  > mv ./chartmuseum /usr/local/bin
+
+**Step 4.** Customize the Helm charts like `oom/kubernetes/onap/values.yaml` or
+an override file like `onap-all.yaml`, `onap-vfw.yaml` or `openstack.yaml` file
+to suit your deployment with items like the OpenStack tenant information.
+
+.. note::
+  Standard and example override files (e.g. `onap-all.yaml`, `openstack.yaml`) can be found in
+  the `oom/kubernetes/onap/resources/overrides/` directory.
+
+
+ a. You may want to selectively enable or disable ONAP components by changing
+    the ``enabled: true/false`` flags.
+
+
+ b. Encrypt the OpenStack password using the shell tool for Robot and put it in
+    the Robot Helm charts or Robot section of `openstack.yaml`
+
+
+ c. Encrypt the OpenStack password using the java based script for SO Helm charts
+    or SO section of `openstack.yaml`.
+
+
+ d. Update the OpenStack parameters that will be used by Robot, SO and APPC Helm
+    charts or use an override file to replace them.
+
+ e. Add in the command line a value for the global master password (global.masterPassword).
+
+
+
+a. Enabling/Disabling Components:
+Here is an example of the nominal entries that need to be provided.
+We have different values file available for different contexts.
+
+.. literalinclude:: ../kubernetes/onap/values.yaml
+   :language: yaml
+
+
+b. Generating ROBOT Encrypted Password:
+The Robot encrypted Password uses the same encryption.key as SO but an
+openssl algorithm that works with the python based Robot Framework.
+
+.. note::
+  To generate Robot ``openStackEncryptedPasswordHere``::
+
+    cd so/resources/config/mso/
+    /oom/kubernetes/so/resources/config/mso# echo -n "<openstack tenant password>" | openssl aes-128-ecb -e -K `cat encryption.key` -nosalt | xxd -c 256 -p``
+
+c. Generating SO Encrypted Password:
+The SO Encrypted Password uses a java based encryption utility since the
+Java encryption library is not easy to integrate with openssl/python that
+Robot uses in Dublin and upper versions.
+
+.. note::
+  To generate SO ``openStackEncryptedPasswordHere`` and ``openStackSoEncryptedPassword``
+  ensure `default-jdk` is installed::
+
+    apt-get update; apt-get install default-jdk
+
+  Then execute::
+
+    SO_ENCRYPTION_KEY=`cat ~/oom/kubernetes/so/resources/config/mso/encryption.key`
+    OS_PASSWORD=XXXX_OS_CLEARTESTPASSWORD_XXXX
+
+    git clone http://gerrit.onap.org/r/integration
+    cd integration/deployment/heat/onap-rke/scripts
+
+    javac Crypto.java
+    java Crypto "$OS_PASSWORD" "$SO_ENCRYPTION_KEY"
+
+d. Update the OpenStack parameters:
+
+There are assumptions in the demonstration VNF Heat templates about the
+networking available in the environment. To get the most value out of these
+templates and the automation that can help confirm the setup is correct, please
+observe the following constraints.
+
+
+``openStackPublicNetId:``
+  This network should allow Heat templates to add interfaces.
+  This need not be an external network, floating IPs can be assigned to the
+  ports on the VMs that are created by the heat template but its important that
+  neutron allow ports to be created on them.
+
+``openStackPrivateNetCidr: "10.0.0.0/16"``
+  This ip address block is used to assign OA&M addresses on VNFs to allow ONAP
+  connectivity. The demonstration Heat templates assume that 10.0 prefix can be
+  used by the VNFs and the demonstration ip addressing plan embodied in the
+  preload template prevent conflicts when instantiating the various VNFs. If
+  you need to change this, you will need to modify the preload data in the
+  Robot Helm chart like integration_preload_parameters.py and the
+  demo/heat/preload_data in the Robot container. The size of the CIDR should
+  be sufficient for ONAP and the VMs you expect to create.
+
+``openStackOamNetworkCidrPrefix: "10.0"``
+  This ip prefix mush match the openStackPrivateNetCidr and is a helper
+  variable to some of the Robot scripts for demonstration. A production
+  deployment need not worry about this setting but for the demonstration VNFs
+  the ip asssignment strategy assumes 10.0 ip prefix.
+
+Example Keystone v2.0
+
+.. literalinclude:: example-integration-override.yaml
+   :language: yaml
+
+Example Keystone v3  (required for Rocky and later releases)
+
+.. literalinclude:: example-integration-override-v3.yaml
+   :language: yaml
+
+
+**Step 5.** To setup a local Helm server to server up the ONAP charts::
+
+  > chartmuseum --storage local --storage-local-rootdir ~/helm3-storage -port 8879 &
+
+Note the port number that is listed and use it in the Helm repo add as
+follows::
+
+  > helm repo add local http://127.0.0.1:8879
+
+**Step 6.** Verify your Helm repository setup with::
+
+  > helm repo list
+  NAME   URL
+  local  http://127.0.0.1:8879
+
+**Step 7.** Build a local Helm repository (from the kubernetes directory)::
+
+  > make SKIP_LINT=TRUE [HELM_BIN=<HELM_PATH>] all ; make SKIP_LINT=TRUE [HELM_BIN=<HELM_PATH>] onap
+
+`HELM_BIN`
+  Sets the helm binary to be used. The default value use helm from PATH
+
+
+**Step 8.** Display the onap charts that available to be deployed::
+
+  > helm repo update
+  > helm search repo onap
+
+.. literalinclude:: helm-search.txt
+
+.. note::
+  The setup of the Helm repository is a one time activity. If you make changes
+  to your deployment charts or values be sure to use ``make`` to update your
+  local Helm repository.
+
+**Step 9.** Once the repo is setup, installation of ONAP can be done with a
+single command
+
+.. note::
+  The ``--timeout 900s`` is currently required in Dublin and later
+  versions up to address long running initialization tasks for DMaaP
+  and SO. Without this timeout value both applications may fail to
+  deploy.
+
+.. danger::
+  We've added the master password on the command line.
+  You shouldn't put it in a file for safety reason
+  please don't forget to change the value to something random
+
+  A space is also added in front of the command so "history" doesn't catch it.
+  This masterPassword is very sensitive, please be careful!
+
+
+To deploy all ONAP applications use this command::
+
+    > cd oom/kubernetes
+    >  helm deploy dev local/onap --namespace onap --set global.masterPassword=myAwesomePasswordThatINeedToChange -f onap/resources/overrides/onap-all.yaml -f onap/resources/overrides/environment.yaml -f onap/resources/overrides/openstack.yaml --timeout 900s
+
+All override files may be customized (or replaced by other overrides) as per
+needs.
+
+`onap-all.yaml`
+  Enables the modules in the ONAP deployment. As ONAP is very modular, it is
+  possible to customize ONAP and disable some components through this
+  configuration file.
+
+`onap-all-ingress-nginx-vhost.yaml`
+  Alternative version of the `onap-all.yaml` but with global ingress controller
+  enabled. It requires the cluster configured with the nginx ingress controller
+  and load balancer. Please use this file instead `onap-all.yaml` if you want
+  to use experimental ingress controller feature.
+
+`environment.yaml`
+  Includes configuration values specific to the deployment environment.
+
+  Example: adapt readiness and liveness timers to the level of performance of
+  your infrastructure
+
+`openstack.yaml`
+  Includes all the OpenStack related information for the default target tenant
+  you want to use to deploy VNFs from ONAP and/or additional parameters for the
+  embedded tests.
+
+**Step 10.** Verify ONAP installation
+
+Use the following to monitor your deployment and determine when ONAP is ready
+for use::
+
+  > kubectl get pods -n onap -o=wide
+
+.. note::
+  While all pods may be in a Running state, it is not a guarantee that all components are running fine.
+
+  Launch the healthcheck tests using Robot to verify that the components are healthy::
+
+    > ~/oom/kubernetes/robot/ete-k8s.sh onap health
+
+**Step 11.** Undeploy ONAP
+::
+
+  > helm undeploy dev
+
+More examples of using the deploy and undeploy plugins can be found here: https://wiki.onap.org/display/DW/OOM+Helm+%28un%29Deploy+plugins
diff --git a/docs/oom_user_guide_helm3.rst b/docs/oom_user_guide_helm3.rst
new file mode 100644
index 0000000000..b687fe8bd3
--- /dev/null
+++ b/docs/oom_user_guide_helm3.rst
@@ -0,0 +1,728 @@
+.. This work is licensed under a Creative Commons Attribution 4.0
+.. International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. Copyright 2018-2020 Amdocs, Bell Canada, Orange, Samsung
+.. _oom_user_guide:
+
+.. Links
+.. _Curated applications for Kubernetes: https://github.com/kubernetes/charts
+.. _Services: https://kubernetes.io/docs/concepts/services-networking/service/
+.. _ReplicaSet: https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/
+.. _StatefulSet: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/
+.. _Helm Documentation: https://docs.helm.sh/helm/
+.. _Helm: https://docs.helm.sh/
+.. _Kubernetes: https://Kubernetes.io/
+.. _Kubernetes LoadBalancer: https://kubernetes.io/docs/concepts/services-networking/service/#type-loadbalancer
+.. _user-guide-label:
+
+OOM User Guide helm3 (experimental)
+###################################
+
+The ONAP Operations Manager (OOM) provide the ability to manage the entire
+life-cycle of an ONAP installation, from the initial deployment to final
+decommissioning. This guide provides instructions for users of ONAP to
+use the Kubernetes_/Helm_ system as a complete ONAP management system.
+
+This guide provides many examples of Helm command line operations.  For a
+complete description of these commands please refer to the `Helm
+Documentation`_.
+
+.. figure:: oomLogoV2-medium.png
+   :align: right
+
+The following sections describe the life-cycle operations:
+
+- Deploy_ - with built-in component dependency management
+- Configure_ - unified configuration across all ONAP components
+- Monitor_ - real-time health monitoring feeding to a Consul UI and Kubernetes
+- Heal_- failed ONAP containers are recreated automatically
+- Scale_ - cluster ONAP services to enable seamless scaling
+- Upgrade_ - change-out containers or configuration with little or no service
+  impact
+- Delete_ - cleanup individual containers or entire deployments
+
+.. figure:: oomLogoV2-Deploy.png
+   :align: right
+
+Deploy
+======
+
+The OOM team with assistance from the ONAP project teams, have built a
+comprehensive set of Helm charts, yaml files very similar to TOSCA files, that
+describe the composition of each of the ONAP components and the relationship
+within and between components. Using this model Helm is able to deploy all of
+ONAP with a few simple commands.
+
+Pre-requisites
+--------------
+Your environment must have both the Kubernetes `kubectl` and Helm setup as a
+one time activity.
+
+Install Kubectl
+~~~~~~~~~~~~~~~
+Enter the following to install kubectl (on Ubuntu, there are slight differences
+on other O/Ss), the Kubernetes command line interface used to manage a
+Kubernetes cluster::
+
+  > curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.8.10/bin/linux/amd64/kubectl
+  > chmod +x ./kubectl
+  > sudo mv ./kubectl /usr/local/bin/kubectl
+  > mkdir ~/.kube
+
+Paste kubectl config from Rancher (see the :ref:`cloud-setup-guide-label` for
+alternative Kubernetes environment setups) into the `~/.kube/config` file.
+
+Verify that the Kubernetes config is correct::
+
+  > kubectl get pods --all-namespaces
+
+At this point you should see six Kubernetes pods running.
+
+Install Helm
+~~~~~~~~~~~~
+Helm is used by OOM for package and configuration management. To install Helm,
+enter the following::
+
+  > wget https://get.helm.sh/helm-v3.3.4-linux-amd64.tar.gz
+  > tar -zxvf helm-v3.3.4-linux-amd64.tar.gz
+  > sudo mv linux-amd64/helm /usr/local/bin/helm
+
+Verify the Helm version with::
+
+  > helm version
+
+Install the Helm Repo
+---------------------
+Once kubectl and Helm are setup, one needs to setup a local Helm server to
+server up the ONAP charts::
+
+  > helm install osn/onap
+
+.. note::
+  The osn repo is not currently available so creation of a local repository is
+  required.
+
+Helm is able to use charts served up from a repository and comes setup with a
+default CNCF provided `Curated applications for Kubernetes`_ repository called
+stable which should be removed to avoid confusion::
+
+  > helm repo remove stable
+
+.. To setup the Open Source Networking Nexus repository for helm enter::
+..  > helm repo add osn 'https://nexus3.onap.org:10001/helm/helm-repo-in-nexus/master/'
+
+To prepare your system for an installation of ONAP, you'll need to::
+
+  > git clone -b guilin --recurse-submodules -j2 http://gerrit.onap.org/r/oom
+  > cd oom/kubernetes
+
+
+To install a local Helm server::
+
+  > curl -LO https://s3.amazonaws.com/chartmuseum/release/latest/bin/linux/amd64/chartmuseum
+  > chmod +x ./chartmuseum
+  > mv ./chartmuseum /usr/local/bin
+
+To setup a local Helm server to server up the ONAP charts::
+
+  > mkdir -p ~/helm3-storage
+  > chartmuseum --storage local --storage-local-rootdir ~/helm3-storage -port 8879 &
+
+Note the port number that is listed and use it in the Helm repo add as
+follows::
+
+  > helm repo add local http://127.0.0.1:8879
+
+To get a list of all of the available Helm chart repositories::
+
+  > helm repo list
+  NAME   URL
+  local  http://127.0.0.1:8879
+
+Then build your local Helm repository::
+
+  > make SKIP_LINT=TRUE [HELM_BIN=<HELM_PATH>] all
+
+`HELM_BIN`
+  Sets the helm binary to be used. The default value use helm from PATH
+
+The Helm search command reads through all of the repositories configured on the
+system, and looks for matches::
+
+  > helm search -l
+  NAME                    VERSION    DESCRIPTION
+  local/appc              2.0.0      Application Controller
+  local/clamp             2.0.0      ONAP Clamp
+  local/common            2.0.0      Common templates for inclusion in other charts
+  local/onap              2.0.0      Open Network Automation Platform (ONAP)
+  local/robot             2.0.0      A helm Chart for kubernetes-ONAP Robot
+  local/so                2.0.0      ONAP Service Orchestrator
+
+In any case, setup of the Helm repository is a one time activity.
+
+Next, install Helm Plugins required to deploy the ONAP Casablanca release::
+
+  > cp -R ~/oom/kubernetes/helm/plugins/ ~/.local/share/helm/plugins
+
+Once the repo is setup, installation of ONAP can be done with a single
+command::
+
+  > helm deploy development local/onap --namespace onap
+
+This will install ONAP from a local repository in a 'development' Helm release.
+As described below, to override the default configuration values provided by
+OOM, an environment file can be provided on the command line as follows::
+
+  > helm deploy development local/onap --namespace onap -f overrides.yaml
+
+To get a summary of the status of all of the pods (containers) running in your
+deployment::
+
+  > kubectl get pods --all-namespaces -o=wide
+
+.. note::
+  The Kubernetes namespace concept allows for multiple instances of a component
+  (such as all of ONAP) to co-exist with other components in the same
+  Kubernetes cluster by isolating them entirely.  Namespaces share only the
+  hosts that form the cluster thus providing isolation between production and
+  development systems as an example.  The OOM deployment of ONAP in Beijing is
+  now done within a single Kubernetes namespace where in Amsterdam a namespace
+  was created for each of the ONAP components.
+
+.. note::
+  The Helm `--name` option refers to a release name and not a Kubernetes namespace.
+
+
+To install a specific version of a single ONAP component (`so` in this example)
+with the given release name enter::
+
+  > helm deploy so onap/so --version 3.0.1
+
+To display details of a specific resource or group of resources type::
+
+  > kubectl describe pod so-1071802958-6twbl
+
+where the pod identifier refers to the auto-generated pod identifier.
+
+.. figure:: oomLogoV2-Configure.png
+   :align: right
+
+Configure
+=========
+
+Each project within ONAP has its own configuration data generally consisting
+of: environment variables, configuration files, and database initial values.
+Many technologies are used across the projects resulting in significant
+operational complexity and an inability to apply global parameters across the
+entire ONAP deployment. OOM solves this problem by introducing a common
+configuration technology, Helm charts, that provide a hierarchical
+configuration with the ability to override values with higher
+level charts or command line options.
+
+The structure of the configuration of ONAP is shown in the following diagram.
+Note that key/value pairs of a parent will always take precedence over those
+of a child. Also note that values set on the command line have the highest
+precedence of all.
+
+.. graphviz::
+
+   digraph config {
+      {
+         node     [shape=folder]
+         oValues  [label="values.yaml"]
+         demo     [label="onap-demo.yaml"]
+         prod     [label="onap-production.yaml"]
+         oReq     [label="requirements.yaml"]
+         soValues [label="values.yaml"]
+         soReq    [label="requirements.yaml"]
+         mdValues [label="values.yaml"]
+      }
+      {
+         oResources  [label="resources"]
+      }
+      onap -> oResources
+      onap -> oValues
+      oResources -> environments
+      oResources -> oReq
+      oReq -> so
+      environments -> demo
+      environments -> prod
+      so -> soValues
+      so -> soReq
+      so -> charts
+      charts -> mariadb
+      mariadb -> mdValues
+
+   }
+
+The top level onap/values.yaml file contains the values required to be set
+before deploying ONAP.  Here is the contents of this file:
+
+.. include:: ../kubernetes/onap/values.yaml
+   :code: yaml
+
+One may wish to create a value file that is specific to a given deployment such
+that it can be differentiated from other deployments.  For example, a
+onap-development.yaml file may create a minimal environment for development
+while onap-production.yaml might describe a production deployment that operates
+independently of the developer version.
+
+For example, if the production OpenStack instance was different from a
+developer's instance, the onap-production.yaml file may contain a different
+value for the vnfDeployment/openstack/oam_network_cidr key as shown below.
+
+.. code-block:: yaml
+
+  nsPrefix: onap
+  nodePortPrefix: 302
+  apps: consul msb mso message-router sdnc vid robot portal policy appc aai
+  sdc dcaegen2 log cli multicloud clamp vnfsdk aaf kube2msb
+  dataRootDir: /dockerdata-nfs
+
+  # docker repositories
+  repository:
+    onap: nexus3.onap.org:10001
+    oom: oomk8s
+    aai: aaionap
+    filebeat: docker.elastic.co
+
+  image:
+    pullPolicy: Never
+
+  # vnf deployment environment
+  vnfDeployment:
+    openstack:
+      ubuntu_14_image: "Ubuntu_14.04.5_LTS"
+      public_net_id: "e8f51956-00dd-4425-af36-045716781ffc"
+      oam_network_id: "d4769dfb-c9e4-4f72-b3d6-1d18f4ac4ee6"
+      oam_subnet_id: "191f7580-acf6-4c2b-8ec0-ba7d99b3bc4e"
+      oam_network_cidr: "192.168.30.0/24"
+  <...>
+
+
+To deploy ONAP with this environment file, enter::
+
+  > helm deploy local/onap -n onap -f environments/onap-production.yaml
+
+.. include:: environments_onap_demo.yaml
+   :code: yaml
+
+When deploying all of ONAP a requirements.yaml file control which and what
+version of the ONAP components are included.  Here is an excerpt of this
+file:
+
+.. code-block:: yaml
+
+  # Referencing a named repo called 'local'.
+  # Can add this repo by running commands like:
+  # > helm serve
+  # > helm repo add local http://127.0.0.1:8879
+  dependencies:
+  <...>
+    - name: so
+      version: ~2.0.0
+      repository: '@local'
+      condition: so.enabled
+  <...>
+
+The ~ operator in the `so` version value indicates that the latest "2.X.X"
+version of `so` shall be used thus allowing the chart to allow for minor
+upgrades that don't impact the so API; hence, version 2.0.1 will be installed
+in this case.
+
+The onap/resources/environment/onap-dev.yaml (see the excerpt below) enables
+for fine grained control on what components are included as part of this
+deployment. By changing this `so` line to `enabled: false` the `so` component
+will not be deployed.  If this change is part of an upgrade the existing `so`
+component will be shut down. Other `so` parameters and even `so` child values
+can be modified, for example the `so`'s `liveness` probe could be disabled
+(which is not recommended as this change would disable auto-healing of `so`).
+
+.. code-block:: yaml
+
+  #################################################################
+  # Global configuration overrides.
+  #
+  # These overrides will affect all helm charts (ie. applications)
+  # that are listed below and are 'enabled'.
+  #################################################################
+  global:
+  <...>
+
+  #################################################################
+  # Enable/disable and configure helm charts (ie. applications)
+  # to customize the ONAP deployment.
+  #################################################################
+  aaf:
+    enabled: false
+  <...>
+  so: # Service Orchestrator
+    enabled: true
+
+    replicaCount: 1
+
+    liveness:
+      # necessary to disable liveness probe when setting breakpoints
+      # in debugger so K8s doesn't restart unresponsive container
+      enabled: true
+
+  <...>
+
+Accessing the ONAP Portal using OOM and a Kubernetes Cluster
+------------------------------------------------------------
+
+The ONAP deployment created by OOM operates in a private IP network that isn't
+publicly accessible (i.e. OpenStack VMs with private internal network) which
+blocks access to the ONAP Portal. To enable direct access to this Portal from a
+user's own environment (a laptop etc.) the portal application's port 8989 is
+exposed through a `Kubernetes LoadBalancer`_ object.
+
+Typically, to be able to access the Kubernetes nodes publicly a public address
+is assigned. In OpenStack this is a floating IP address.
+
+When the `portal-app` chart is deployed a Kubernetes service is created that
+instantiates a load balancer.  The LB chooses the private interface of one of
+the nodes as in the example below (10.0.0.4 is private to the K8s cluster only).
+Then to be able to access the portal on port 8989 from outside the K8s &
+OpenStack environment, the user needs to assign/get the floating IP address that
+corresponds to the private IP as follows::
+
+  > kubectl -n onap get services|grep "portal-app"
+  portal-app  LoadBalancer   10.43.142.201   10.0.0.4   8989:30215/TCP,8006:30213/TCP,8010:30214/TCP   1d   app=portal-app,release=dev
+
+
+In this example, use the 10.0.0.4 private address as a key find the
+corresponding public address which in this example is 10.12.6.155. If you're
+using OpenStack you'll do the lookup with the horizon GUI or the OpenStack CLI
+for your tenant (openstack server list).  That IP is then used in your
+`/etc/hosts` to map the fixed DNS aliases required by the ONAP Portal as shown
+below::
+
+  10.12.6.155 portal.api.simpledemo.onap.org
+  10.12.6.155 vid.api.simpledemo.onap.org
+  10.12.6.155 sdc.api.fe.simpledemo.onap.org
+  10.12.6.155 sdc.workflow.plugin.simpledemo.onap.org
+  10.12.6.155 sdc.dcae.plugin.simpledemo.onap.org
+  10.12.6.155 portal-sdk.simpledemo.onap.org
+  10.12.6.155 policy.api.simpledemo.onap.org
+  10.12.6.155 aai.api.sparky.simpledemo.onap.org
+  10.12.6.155 cli.api.simpledemo.onap.org
+  10.12.6.155 msb.api.discovery.simpledemo.onap.org
+  10.12.6.155 msb.api.simpledemo.onap.org
+  10.12.6.155 clamp.api.simpledemo.onap.org
+  10.12.6.155 so.api.simpledemo.onap.org
+  10.12.6.155 sdc.workflow.plugin.simpledemo.onap.org
+
+Ensure you've disabled any proxy settings the browser you are using to access
+the portal and then simply access now the new ssl-encrypted URL:
+https://portal.api.simpledemo.onap.org:30225/ONAPPORTAL/login.htm
+
+.. note::
+  Using the HTTPS based Portal URL the Browser needs to be configured to accept
+  unsecure credentials.
+  Additionally when opening an Application inside the Portal, the Browser
+  might block the content, which requires to disable the blocking and reloading
+  of the page
+
+.. note::
+  Besides the ONAP Portal the Components can deliver additional user interfaces,
+  please check the Component specific documentation.
+
+.. note::
+
+   | Alternatives Considered:
+
+   -  Kubernetes port forwarding was considered but discarded as it would require
+      the end user to run a script that opens up port forwarding tunnels to each of
+      the pods that provides a portal application widget.
+
+   -  Reverting to a VNC server similar to what was deployed in the Amsterdam
+      release was also considered but there were many issues with resolution, lack
+      of volume mount, /etc/hosts dynamic update, file upload that were a tall order
+      to solve in time for the Beijing release.
+
+   Observations:
+
+   -  If you are not using floating IPs in your Kubernetes deployment and directly attaching
+      a public IP address (i.e. by using your public provider network) to your K8S Node
+      VMs' network interface, then the output of 'kubectl -n onap get services | grep "portal-app"'
+      will show your public IP instead of the private network's IP. Therefore,
+      you can grab this public IP directly (as compared to trying to find the floating
+      IP first) and map this IP in /etc/hosts.
+
+.. figure:: oomLogoV2-Monitor.png
+   :align: right
+
+Monitor
+=======
+
+All highly available systems include at least one facility to monitor the
+health of components within the system.  Such health monitors are often used as
+inputs to distributed coordination systems (such as etcd, Zookeeper, or Consul)
+and monitoring systems (such as Nagios or Zabbix). OOM provides two mechanisms
+to monitor the real-time health of an ONAP deployment:
+
+- a Consul GUI for a human operator or downstream monitoring systems and
+  Kubernetes liveness probes that enable automatic healing of failed
+  containers, and
+- a set of liveness probes which feed into the Kubernetes manager which
+  are described in the Heal section.
+
+Within ONAP, Consul is the monitoring system of choice and deployed by OOM in
+two parts:
+
+- a three-way, centralized Consul server cluster is deployed as a highly
+  available monitor of all of the ONAP components, and
+- a number of Consul agents.
+
+The Consul server provides a user interface that allows a user to graphically
+view the current health status of all of the ONAP components for which agents
+have been created - a sample from the ONAP Integration labs follows:
+
+.. figure:: consulHealth.png
+   :align: center
+
+To see the real-time health of a deployment go to: http://<kubernetes IP>:30270/ui/
+where a GUI much like the following will be found:
+
+
+.. figure:: oomLogoV2-Heal.png
+   :align: right
+
+Heal
+====
+
+The ONAP deployment is defined by Helm charts as mentioned earlier.  These Helm
+charts are also used to implement automatic recoverability of ONAP components
+when individual components fail. Once ONAP is deployed, a "liveness" probe
+starts checking the health of the components after a specified startup time.
+
+Should a liveness probe indicate a failed container it will be terminated and a
+replacement will be started in its place - containers are ephemeral. Should the
+deployment specification indicate that there are one or more dependencies to
+this container or component (for example a dependency on a database) the
+dependency will be satisfied before the replacement container/component is
+started. This mechanism ensures that, after a failure, all of the ONAP
+components restart successfully.
+
+To test healing, the following command can be used to delete a pod::
+
+  > kubectl delete pod [pod name] -n [pod namespace]
+
+One could then use the following command to monitor the pods and observe the
+pod being terminated and the service being automatically healed with the
+creation of a replacement pod::
+
+  > kubectl get pods --all-namespaces -o=wide
+
+.. figure:: oomLogoV2-Scale.png
+   :align: right
+
+Scale
+=====
+
+Many of the ONAP components are horizontally scalable which allows them to
+adapt to expected offered load.  During the Beijing release scaling is static,
+that is during deployment or upgrade a cluster size is defined and this cluster
+will be maintained even in the presence of faults. The parameter that controls
+the cluster size of a given component is found in the values.yaml file for that
+component.  Here is an excerpt that shows this parameter:
+
+.. code-block:: yaml
+
+  # default number of instances
+  replicaCount: 1
+
+In order to change the size of a cluster, an operator could use a helm upgrade
+(described in detail in the next section) as follows::
+
+  > helm upgrade --set replicaCount=3 onap/so/mariadb
+
+The ONAP components use Kubernetes provided facilities to build clustered,
+highly available systems including: Services_ with load-balancers, ReplicaSet_,
+and StatefulSet_.  Some of the open-source projects used by the ONAP components
+directly support clustered configurations, for example ODL and MariaDB Galera.
+
+The Kubernetes Services_ abstraction to provide a consistent access point for
+each of the ONAP components, independent of the pod or container architecture
+of that component.  For example, SDN-C uses OpenDaylight clustering with a
+default cluster size of three but uses a Kubernetes service to and change the
+number of pods in this abstract this cluster from the other ONAP components
+such that the cluster could change size and this change is isolated from the
+other ONAP components by the load-balancer implemented in the ODL service
+abstraction.
+
+A ReplicaSet_ is a construct that is used to describe the desired state of the
+cluster.  For example 'replicas: 3' indicates to Kubernetes that a cluster of 3
+instances is the desired state.  Should one of the members of the cluster fail,
+a new member will be automatically started to replace it.
+
+Some of the ONAP components many need a more deterministic deployment; for
+example to enable intra-cluster communication. For these applications the
+component can be deployed as a Kubernetes StatefulSet_ which will maintain a
+persistent identifier for the pods and thus a stable network id for the pods.
+For example: the pod names might be web-0, web-1, web-{N-1} for N 'web' pods
+with corresponding DNS entries such that intra service communication is simple
+even if the pods are physically distributed across multiple nodes. An example
+of how these capabilities can be used is described in the Running Consul on
+Kubernetes tutorial.
+
+.. figure:: oomLogoV2-Upgrade.png
+   :align: right
+
+Upgrade
+=======
+
+Helm has built-in capabilities to enable the upgrade of pods without causing a
+loss of the service being provided by that pod or pods (if configured as a
+cluster).  As described in the OOM Developer's Guide, ONAP components provide
+an abstracted 'service' end point with the pods or containers providing this
+service hidden from other ONAP components by a load balancer. This capability
+is used during upgrades to allow a pod with a new image to be added to the
+service before removing the pod with the old image. This 'make before break'
+capability ensures minimal downtime.
+
+Prior to doing an upgrade, determine of the status of the deployed charts::
+
+  > helm list
+  NAME REVISION UPDATED                  STATUS    CHART     NAMESPACE
+  so   1        Mon Feb 5 10:05:22 2018  DEPLOYED  so-2.0.1  default
+
+When upgrading a cluster a parameter controls the minimum size of the cluster
+during the upgrade while another parameter controls the maximum number of nodes
+in the cluster.  For example, SNDC configured as a 3-way ODL cluster might
+require that during the upgrade no fewer than 2 pods are available at all times
+to provide service while no more than 5 pods are ever deployed across the two
+versions at any one time to avoid depleting the cluster of resources. In this
+scenario, the SDNC cluster would start with 3 old pods then Kubernetes may add
+a new pod (3 old, 1 new), delete one old (2 old, 1 new), add two new pods (2
+old, 3 new) and finally delete the 2 old pods (3 new).  During this sequence
+the constraints of the minimum of two pods and maximum of five would be
+maintained while providing service the whole time.
+
+Initiation of an upgrade is triggered by changes in the Helm charts.  For
+example, if the image specified for one of the pods in the SDNC deployment
+specification were to change (i.e. point to a new Docker image in the nexus3
+repository - commonly through the change of a deployment variable), the
+sequence of events described in the previous paragraph would be initiated.
+
+For example, to upgrade a container by changing configuration, specifically an
+environment value::
+
+  > helm deploy onap onap/so --version 2.0.1 --set enableDebug=true
+
+Issuing this command will result in the appropriate container being stopped by
+Kubernetes and replaced with a new container with the new environment value.
+
+To upgrade a component to a new version with a new configuration file enter::
+
+  > helm deploy onap onap/so --version 2.0.2 -f environments/demo.yaml
+
+To fetch release history enter::
+
+  > helm history so
+  REVISION UPDATED                  STATUS     CHART     DESCRIPTION
+  1        Mon Feb 5 10:05:22 2018  SUPERSEDED so-2.0.1  Install complete
+  2        Mon Feb 5 10:10:55 2018  DEPLOYED   so-2.0.2  Upgrade complete
+
+Unfortunately, not all upgrades are successful.  In recognition of this the
+lineup of pods within an ONAP deployment is tagged such that an administrator
+may force the ONAP deployment back to the previously tagged configuration or to
+a specific configuration, say to jump back two steps if an incompatibility
+between two ONAP components is discovered after the two individual upgrades
+succeeded.
+
+This rollback functionality gives the administrator confidence that in the
+unfortunate circumstance of a failed upgrade the system can be rapidly brought
+back to a known good state.  This process of rolling upgrades while under
+service is illustrated in this short YouTube video showing a Zero Downtime
+Upgrade of a web application while under a 10 million transaction per second
+load.
+
+For example, to roll-back back to previous system revision enter::
+
+  > helm rollback so 1
+
+  > helm history so
+  REVISION UPDATED                  STATUS     CHART     DESCRIPTION
+  1        Mon Feb 5 10:05:22 2018  SUPERSEDED so-2.0.1  Install complete
+  2        Mon Feb 5 10:10:55 2018  SUPERSEDED so-2.0.2  Upgrade complete
+  3        Mon Feb 5 10:14:32 2018  DEPLOYED   so-2.0.1  Rollback to 1
+
+.. note::
+
+  The description field can be overridden to document actions taken or include
+  tracking numbers.
+
+Many of the ONAP components contain their own databases which are used to
+record configuration or state information.  The schemas of these databases may
+change from version to version in such a way that data stored within the
+database needs to be migrated between versions. If such a migration script is
+available it can be invoked during the upgrade (or rollback) by Container
+Lifecycle Hooks. Two such hooks are available, PostStart and PreStop, which
+containers can access by registering a handler against one or both. Note that
+it is the responsibility of the ONAP component owners to implement the hook
+handlers - which could be a shell script or a call to a specific container HTTP
+endpoint - following the guidelines listed on the Kubernetes site. Lifecycle
+hooks are not restricted to database migration or even upgrades but can be used
+anywhere specific operations need to be taken during lifecycle operations.
+
+OOM uses Helm K8S package manager to deploy ONAP components. Each component is
+arranged in a packaging format called a chart - a collection of files that
+describe a set of k8s resources. Helm allows for rolling upgrades of the ONAP
+component deployed. To upgrade a component Helm release you will need an
+updated Helm chart. The chart might have modified, deleted or added values,
+deployment yamls, and more.  To get the release name use::
+
+  > helm ls
+
+To easily upgrade the release use::
+
+  > helm upgrade [RELEASE] [CHART]
+
+To roll back to a previous release version use::
+
+  > helm rollback [flags] [RELEASE] [REVISION]
+
+For example, to upgrade the onap-so helm release to the latest SO container
+release v1.1.2:
+
+- Edit so values.yaml which is part of the chart
+- Change "so: nexus3.onap.org:10001/openecomp/so:v1.1.1" to
+  "so: nexus3.onap.org:10001/openecomp/so:v1.1.2"
+- From the chart location run::
+
+  > helm upgrade onap-so
+
+The previous so pod will be terminated and a new so pod with an updated so
+container will be created.
+
+.. figure:: oomLogoV2-Delete.png
+   :align: right
+
+Delete
+======
+
+Existing deployments can be partially or fully removed once they are no longer
+needed.  To minimize errors it is recommended that before deleting components
+from a running deployment the operator perform a 'dry-run' to display exactly
+what will happen with a given command prior to actually deleting anything.  For
+example::
+
+  > helm undeploy onap --dry-run
+
+will display the outcome of deleting the 'onap' release from the
+deployment.
+To completely delete a release and remove it from the internal store enter::
+
+  > helm undeploy onap
+
+One can also remove individual components from a deployment by changing the
+ONAP configuration values.  For example, to remove `so` from a running
+deployment enter::
+
+  > helm undeploy onap-so
+
+will remove `so` as the configuration indicates it's no longer part of the
+deployment. This might be useful if a one wanted to replace just `so` by
+installing a custom version.