diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/index.rst | 2 | ||||
| -rw-r--r-- | docs/oom_quickstart_guide_helm3.rst | 252 | ||||
| -rw-r--r-- | docs/oom_user_guide_helm3.rst | 728 |
3 files changed, 982 insertions, 0 deletions
diff --git a/docs/index.rst b/docs/index.rst index c3902ecae0..68b38de9aa 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -12,7 +12,9 @@ OOM Documentation Repository oom_project_description.rst oom_quickstart_guide.rst + oom_quickstart_guide_helm3.rst oom_user_guide.rst + oom_user_guide_helm3.rst oom_developer_guide.rst oom_cloud_setup_guide.rst release-notes.rst diff --git a/docs/oom_quickstart_guide_helm3.rst b/docs/oom_quickstart_guide_helm3.rst new file mode 100644 index 0000000000..5a3076426e --- /dev/null +++ b/docs/oom_quickstart_guide_helm3.rst @@ -0,0 +1,252 @@ +.. This work is licensed under a +.. Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. Copyright 2019-2020 Amdocs, Bell Canada, Orange, Samsung +.. _oom_quickstart_guide_helm3: +.. _quick-start-label-helm3: + +OOM Quick Start Guide Helm3 (experimental) +########################################### + +.. figure:: oomLogoV2-medium.png + :align: right + +Once a Kubernetes environment is available (follow the instructions in +:ref:`cloud-setup-guide-label` if you don't have a cloud environment +available), follow the following instructions to deploy ONAP. + +**Step 1.** Clone the OOM repository from ONAP gerrit:: + + > git clone -b <BRANCH> http://gerrit.onap.org/r/oom --recurse-submodules + > cd oom/kubernetes + +where <BRANCH> can be an official release tag, such as + +* 4.0.0-ONAP for Dublin +* 5.0.1-ONAP for El Alto +* 6.0.0 for Frankfurt +* 7.0.0 for Guilin + +**Step 2.** Install Helm Plugins required to deploy ONAP:: + + > cp -R ~/oom/kubernetes/helm/plugins/ ~/.local/share/helm/plugins + > helm plugin install https://github.com/chartmuseum/helm-push.git + +**Step 3** Install Chartmuseum:: + + > curl -LO https://s3.amazonaws.com/chartmuseum/release/latest/bin/linux/amd64/chartmuseum + > chmod +x ./chartmuseum + > mv ./chartmuseum /usr/local/bin + +**Step 4.** Customize the Helm charts like `oom/kubernetes/onap/values.yaml` or +an override file like `onap-all.yaml`, `onap-vfw.yaml` or `openstack.yaml` file +to suit your deployment with items like the OpenStack tenant information. + +.. note:: + Standard and example override files (e.g. `onap-all.yaml`, `openstack.yaml`) can be found in + the `oom/kubernetes/onap/resources/overrides/` directory. + + + a. You may want to selectively enable or disable ONAP components by changing + the ``enabled: true/false`` flags. + + + b. Encrypt the OpenStack password using the shell tool for Robot and put it in + the Robot Helm charts or Robot section of `openstack.yaml` + + + c. Encrypt the OpenStack password using the java based script for SO Helm charts + or SO section of `openstack.yaml`. + + + d. Update the OpenStack parameters that will be used by Robot, SO and APPC Helm + charts or use an override file to replace them. + + e. Add in the command line a value for the global master password (global.masterPassword). + + + +a. Enabling/Disabling Components: +Here is an example of the nominal entries that need to be provided. +We have different values file available for different contexts. + +.. literalinclude:: ../kubernetes/onap/values.yaml + :language: yaml + + +b. Generating ROBOT Encrypted Password: +The Robot encrypted Password uses the same encryption.key as SO but an +openssl algorithm that works with the python based Robot Framework. + +.. note:: + To generate Robot ``openStackEncryptedPasswordHere``:: + + cd so/resources/config/mso/ + /oom/kubernetes/so/resources/config/mso# echo -n "<openstack tenant password>" | openssl aes-128-ecb -e -K `cat encryption.key` -nosalt | xxd -c 256 -p`` + +c. Generating SO Encrypted Password: +The SO Encrypted Password uses a java based encryption utility since the +Java encryption library is not easy to integrate with openssl/python that +Robot uses in Dublin and upper versions. + +.. note:: + To generate SO ``openStackEncryptedPasswordHere`` and ``openStackSoEncryptedPassword`` + ensure `default-jdk` is installed:: + + apt-get update; apt-get install default-jdk + + Then execute:: + + SO_ENCRYPTION_KEY=`cat ~/oom/kubernetes/so/resources/config/mso/encryption.key` + OS_PASSWORD=XXXX_OS_CLEARTESTPASSWORD_XXXX + + git clone http://gerrit.onap.org/r/integration + cd integration/deployment/heat/onap-rke/scripts + + javac Crypto.java + java Crypto "$OS_PASSWORD" "$SO_ENCRYPTION_KEY" + +d. Update the OpenStack parameters: + +There are assumptions in the demonstration VNF Heat templates about the +networking available in the environment. To get the most value out of these +templates and the automation that can help confirm the setup is correct, please +observe the following constraints. + + +``openStackPublicNetId:`` + This network should allow Heat templates to add interfaces. + This need not be an external network, floating IPs can be assigned to the + ports on the VMs that are created by the heat template but its important that + neutron allow ports to be created on them. + +``openStackPrivateNetCidr: "10.0.0.0/16"`` + This ip address block is used to assign OA&M addresses on VNFs to allow ONAP + connectivity. The demonstration Heat templates assume that 10.0 prefix can be + used by the VNFs and the demonstration ip addressing plan embodied in the + preload template prevent conflicts when instantiating the various VNFs. If + you need to change this, you will need to modify the preload data in the + Robot Helm chart like integration_preload_parameters.py and the + demo/heat/preload_data in the Robot container. The size of the CIDR should + be sufficient for ONAP and the VMs you expect to create. + +``openStackOamNetworkCidrPrefix: "10.0"`` + This ip prefix mush match the openStackPrivateNetCidr and is a helper + variable to some of the Robot scripts for demonstration. A production + deployment need not worry about this setting but for the demonstration VNFs + the ip asssignment strategy assumes 10.0 ip prefix. + +Example Keystone v2.0 + +.. literalinclude:: example-integration-override.yaml + :language: yaml + +Example Keystone v3 (required for Rocky and later releases) + +.. literalinclude:: example-integration-override-v3.yaml + :language: yaml + + +**Step 5.** To setup a local Helm server to server up the ONAP charts:: + + > chartmuseum --storage local --storage-local-rootdir ~/helm3-storage -port 8879 & + +Note the port number that is listed and use it in the Helm repo add as +follows:: + + > helm repo add local http://127.0.0.1:8879 + +**Step 6.** Verify your Helm repository setup with:: + + > helm repo list + NAME URL + local http://127.0.0.1:8879 + +**Step 7.** Build a local Helm repository (from the kubernetes directory):: + + > make SKIP_LINT=TRUE [HELM_BIN=<HELM_PATH>] all ; make SKIP_LINT=TRUE [HELM_BIN=<HELM_PATH>] onap + +`HELM_BIN` + Sets the helm binary to be used. The default value use helm from PATH + + +**Step 8.** Display the onap charts that available to be deployed:: + + > helm repo update + > helm search repo onap + +.. literalinclude:: helm-search.txt + +.. note:: + The setup of the Helm repository is a one time activity. If you make changes + to your deployment charts or values be sure to use ``make`` to update your + local Helm repository. + +**Step 9.** Once the repo is setup, installation of ONAP can be done with a +single command + +.. note:: + The ``--timeout 900s`` is currently required in Dublin and later + versions up to address long running initialization tasks for DMaaP + and SO. Without this timeout value both applications may fail to + deploy. + +.. danger:: + We've added the master password on the command line. + You shouldn't put it in a file for safety reason + please don't forget to change the value to something random + + A space is also added in front of the command so "history" doesn't catch it. + This masterPassword is very sensitive, please be careful! + + +To deploy all ONAP applications use this command:: + + > cd oom/kubernetes + > helm deploy dev local/onap --namespace onap --set global.masterPassword=myAwesomePasswordThatINeedToChange -f onap/resources/overrides/onap-all.yaml -f onap/resources/overrides/environment.yaml -f onap/resources/overrides/openstack.yaml --timeout 900s + +All override files may be customized (or replaced by other overrides) as per +needs. + +`onap-all.yaml` + Enables the modules in the ONAP deployment. As ONAP is very modular, it is + possible to customize ONAP and disable some components through this + configuration file. + +`onap-all-ingress-nginx-vhost.yaml` + Alternative version of the `onap-all.yaml` but with global ingress controller + enabled. It requires the cluster configured with the nginx ingress controller + and load balancer. Please use this file instead `onap-all.yaml` if you want + to use experimental ingress controller feature. + +`environment.yaml` + Includes configuration values specific to the deployment environment. + + Example: adapt readiness and liveness timers to the level of performance of + your infrastructure + +`openstack.yaml` + Includes all the OpenStack related information for the default target tenant + you want to use to deploy VNFs from ONAP and/or additional parameters for the + embedded tests. + +**Step 10.** Verify ONAP installation + +Use the following to monitor your deployment and determine when ONAP is ready +for use:: + + > kubectl get pods -n onap -o=wide + +.. note:: + While all pods may be in a Running state, it is not a guarantee that all components are running fine. + + Launch the healthcheck tests using Robot to verify that the components are healthy:: + + > ~/oom/kubernetes/robot/ete-k8s.sh onap health + +**Step 11.** Undeploy ONAP +:: + + > helm undeploy dev + +More examples of using the deploy and undeploy plugins can be found here: https://wiki.onap.org/display/DW/OOM+Helm+%28un%29Deploy+plugins diff --git a/docs/oom_user_guide_helm3.rst b/docs/oom_user_guide_helm3.rst new file mode 100644 index 0000000000..b687fe8bd3 --- /dev/null +++ b/docs/oom_user_guide_helm3.rst @@ -0,0 +1,728 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 +.. International License. +.. http://creativecommons.org/licenses/by/4.0 +.. Copyright 2018-2020 Amdocs, Bell Canada, Orange, Samsung +.. _oom_user_guide: + +.. Links +.. _Curated applications for Kubernetes: https://github.com/kubernetes/charts +.. _Services: https://kubernetes.io/docs/concepts/services-networking/service/ +.. _ReplicaSet: https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/ +.. _StatefulSet: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/ +.. _Helm Documentation: https://docs.helm.sh/helm/ +.. _Helm: https://docs.helm.sh/ +.. _Kubernetes: https://Kubernetes.io/ +.. _Kubernetes LoadBalancer: https://kubernetes.io/docs/concepts/services-networking/service/#type-loadbalancer +.. _user-guide-label: + +OOM User Guide helm3 (experimental) +################################### + +The ONAP Operations Manager (OOM) provide the ability to manage the entire +life-cycle of an ONAP installation, from the initial deployment to final +decommissioning. This guide provides instructions for users of ONAP to +use the Kubernetes_/Helm_ system as a complete ONAP management system. + +This guide provides many examples of Helm command line operations. For a +complete description of these commands please refer to the `Helm +Documentation`_. + +.. figure:: oomLogoV2-medium.png + :align: right + +The following sections describe the life-cycle operations: + +- Deploy_ - with built-in component dependency management +- Configure_ - unified configuration across all ONAP components +- Monitor_ - real-time health monitoring feeding to a Consul UI and Kubernetes +- Heal_- failed ONAP containers are recreated automatically +- Scale_ - cluster ONAP services to enable seamless scaling +- Upgrade_ - change-out containers or configuration with little or no service + impact +- Delete_ - cleanup individual containers or entire deployments + +.. figure:: oomLogoV2-Deploy.png + :align: right + +Deploy +====== + +The OOM team with assistance from the ONAP project teams, have built a +comprehensive set of Helm charts, yaml files very similar to TOSCA files, that +describe the composition of each of the ONAP components and the relationship +within and between components. Using this model Helm is able to deploy all of +ONAP with a few simple commands. + +Pre-requisites +-------------- +Your environment must have both the Kubernetes `kubectl` and Helm setup as a +one time activity. + +Install Kubectl +~~~~~~~~~~~~~~~ +Enter the following to install kubectl (on Ubuntu, there are slight differences +on other O/Ss), the Kubernetes command line interface used to manage a +Kubernetes cluster:: + + > curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.8.10/bin/linux/amd64/kubectl + > chmod +x ./kubectl + > sudo mv ./kubectl /usr/local/bin/kubectl + > mkdir ~/.kube + +Paste kubectl config from Rancher (see the :ref:`cloud-setup-guide-label` for +alternative Kubernetes environment setups) into the `~/.kube/config` file. + +Verify that the Kubernetes config is correct:: + + > kubectl get pods --all-namespaces + +At this point you should see six Kubernetes pods running. + +Install Helm +~~~~~~~~~~~~ +Helm is used by OOM for package and configuration management. To install Helm, +enter the following:: + + > wget https://get.helm.sh/helm-v3.3.4-linux-amd64.tar.gz + > tar -zxvf helm-v3.3.4-linux-amd64.tar.gz + > sudo mv linux-amd64/helm /usr/local/bin/helm + +Verify the Helm version with:: + + > helm version + +Install the Helm Repo +--------------------- +Once kubectl and Helm are setup, one needs to setup a local Helm server to +server up the ONAP charts:: + + > helm install osn/onap + +.. note:: + The osn repo is not currently available so creation of a local repository is + required. + +Helm is able to use charts served up from a repository and comes setup with a +default CNCF provided `Curated applications for Kubernetes`_ repository called +stable which should be removed to avoid confusion:: + + > helm repo remove stable + +.. To setup the Open Source Networking Nexus repository for helm enter:: +.. > helm repo add osn 'https://nexus3.onap.org:10001/helm/helm-repo-in-nexus/master/' + +To prepare your system for an installation of ONAP, you'll need to:: + + > git clone -b guilin --recurse-submodules -j2 http://gerrit.onap.org/r/oom + > cd oom/kubernetes + + +To install a local Helm server:: + + > curl -LO https://s3.amazonaws.com/chartmuseum/release/latest/bin/linux/amd64/chartmuseum + > chmod +x ./chartmuseum + > mv ./chartmuseum /usr/local/bin + +To setup a local Helm server to server up the ONAP charts:: + + > mkdir -p ~/helm3-storage + > chartmuseum --storage local --storage-local-rootdir ~/helm3-storage -port 8879 & + +Note the port number that is listed and use it in the Helm repo add as +follows:: + + > helm repo add local http://127.0.0.1:8879 + +To get a list of all of the available Helm chart repositories:: + + > helm repo list + NAME URL + local http://127.0.0.1:8879 + +Then build your local Helm repository:: + + > make SKIP_LINT=TRUE [HELM_BIN=<HELM_PATH>] all + +`HELM_BIN` + Sets the helm binary to be used. The default value use helm from PATH + +The Helm search command reads through all of the repositories configured on the +system, and looks for matches:: + + > helm search -l + NAME VERSION DESCRIPTION + local/appc 2.0.0 Application Controller + local/clamp 2.0.0 ONAP Clamp + local/common 2.0.0 Common templates for inclusion in other charts + local/onap 2.0.0 Open Network Automation Platform (ONAP) + local/robot 2.0.0 A helm Chart for kubernetes-ONAP Robot + local/so 2.0.0 ONAP Service Orchestrator + +In any case, setup of the Helm repository is a one time activity. + +Next, install Helm Plugins required to deploy the ONAP Casablanca release:: + + > cp -R ~/oom/kubernetes/helm/plugins/ ~/.local/share/helm/plugins + +Once the repo is setup, installation of ONAP can be done with a single +command:: + + > helm deploy development local/onap --namespace onap + +This will install ONAP from a local repository in a 'development' Helm release. +As described below, to override the default configuration values provided by +OOM, an environment file can be provided on the command line as follows:: + + > helm deploy development local/onap --namespace onap -f overrides.yaml + +To get a summary of the status of all of the pods (containers) running in your +deployment:: + + > kubectl get pods --all-namespaces -o=wide + +.. note:: + The Kubernetes namespace concept allows for multiple instances of a component + (such as all of ONAP) to co-exist with other components in the same + Kubernetes cluster by isolating them entirely. Namespaces share only the + hosts that form the cluster thus providing isolation between production and + development systems as an example. The OOM deployment of ONAP in Beijing is + now done within a single Kubernetes namespace where in Amsterdam a namespace + was created for each of the ONAP components. + +.. note:: + The Helm `--name` option refers to a release name and not a Kubernetes namespace. + + +To install a specific version of a single ONAP component (`so` in this example) +with the given release name enter:: + + > helm deploy so onap/so --version 3.0.1 + +To display details of a specific resource or group of resources type:: + + > kubectl describe pod so-1071802958-6twbl + +where the pod identifier refers to the auto-generated pod identifier. + +.. figure:: oomLogoV2-Configure.png + :align: right + +Configure +========= + +Each project within ONAP has its own configuration data generally consisting +of: environment variables, configuration files, and database initial values. +Many technologies are used across the projects resulting in significant +operational complexity and an inability to apply global parameters across the +entire ONAP deployment. OOM solves this problem by introducing a common +configuration technology, Helm charts, that provide a hierarchical +configuration with the ability to override values with higher +level charts or command line options. + +The structure of the configuration of ONAP is shown in the following diagram. +Note that key/value pairs of a parent will always take precedence over those +of a child. Also note that values set on the command line have the highest +precedence of all. + +.. graphviz:: + + digraph config { + { + node [shape=folder] + oValues [label="values.yaml"] + demo [label="onap-demo.yaml"] + prod [label="onap-production.yaml"] + oReq [label="requirements.yaml"] + soValues [label="values.yaml"] + soReq [label="requirements.yaml"] + mdValues [label="values.yaml"] + } + { + oResources [label="resources"] + } + onap -> oResources + onap -> oValues + oResources -> environments + oResources -> oReq + oReq -> so + environments -> demo + environments -> prod + so -> soValues + so -> soReq + so -> charts + charts -> mariadb + mariadb -> mdValues + + } + +The top level onap/values.yaml file contains the values required to be set +before deploying ONAP. Here is the contents of this file: + +.. include:: ../kubernetes/onap/values.yaml + :code: yaml + +One may wish to create a value file that is specific to a given deployment such +that it can be differentiated from other deployments. For example, a +onap-development.yaml file may create a minimal environment for development +while onap-production.yaml might describe a production deployment that operates +independently of the developer version. + +For example, if the production OpenStack instance was different from a +developer's instance, the onap-production.yaml file may contain a different +value for the vnfDeployment/openstack/oam_network_cidr key as shown below. + +.. code-block:: yaml + + nsPrefix: onap + nodePortPrefix: 302 + apps: consul msb mso message-router sdnc vid robot portal policy appc aai + sdc dcaegen2 log cli multicloud clamp vnfsdk aaf kube2msb + dataRootDir: /dockerdata-nfs + + # docker repositories + repository: + onap: nexus3.onap.org:10001 + oom: oomk8s + aai: aaionap + filebeat: docker.elastic.co + + image: + pullPolicy: Never + + # vnf deployment environment + vnfDeployment: + openstack: + ubuntu_14_image: "Ubuntu_14.04.5_LTS" + public_net_id: "e8f51956-00dd-4425-af36-045716781ffc" + oam_network_id: "d4769dfb-c9e4-4f72-b3d6-1d18f4ac4ee6" + oam_subnet_id: "191f7580-acf6-4c2b-8ec0-ba7d99b3bc4e" + oam_network_cidr: "192.168.30.0/24" + <...> + + +To deploy ONAP with this environment file, enter:: + + > helm deploy local/onap -n onap -f environments/onap-production.yaml + +.. include:: environments_onap_demo.yaml + :code: yaml + +When deploying all of ONAP a requirements.yaml file control which and what +version of the ONAP components are included. Here is an excerpt of this +file: + +.. code-block:: yaml + + # Referencing a named repo called 'local'. + # Can add this repo by running commands like: + # > helm serve + # > helm repo add local http://127.0.0.1:8879 + dependencies: + <...> + - name: so + version: ~2.0.0 + repository: '@local' + condition: so.enabled + <...> + +The ~ operator in the `so` version value indicates that the latest "2.X.X" +version of `so` shall be used thus allowing the chart to allow for minor +upgrades that don't impact the so API; hence, version 2.0.1 will be installed +in this case. + +The onap/resources/environment/onap-dev.yaml (see the excerpt below) enables +for fine grained control on what components are included as part of this +deployment. By changing this `so` line to `enabled: false` the `so` component +will not be deployed. If this change is part of an upgrade the existing `so` +component will be shut down. Other `so` parameters and even `so` child values +can be modified, for example the `so`'s `liveness` probe could be disabled +(which is not recommended as this change would disable auto-healing of `so`). + +.. code-block:: yaml + + ################################################################# + # Global configuration overrides. + # + # These overrides will affect all helm charts (ie. applications) + # that are listed below and are 'enabled'. + ################################################################# + global: + <...> + + ################################################################# + # Enable/disable and configure helm charts (ie. applications) + # to customize the ONAP deployment. + ################################################################# + aaf: + enabled: false + <...> + so: # Service Orchestrator + enabled: true + + replicaCount: 1 + + liveness: + # necessary to disable liveness probe when setting breakpoints + # in debugger so K8s doesn't restart unresponsive container + enabled: true + + <...> + +Accessing the ONAP Portal using OOM and a Kubernetes Cluster +------------------------------------------------------------ + +The ONAP deployment created by OOM operates in a private IP network that isn't +publicly accessible (i.e. OpenStack VMs with private internal network) which +blocks access to the ONAP Portal. To enable direct access to this Portal from a +user's own environment (a laptop etc.) the portal application's port 8989 is +exposed through a `Kubernetes LoadBalancer`_ object. + +Typically, to be able to access the Kubernetes nodes publicly a public address +is assigned. In OpenStack this is a floating IP address. + +When the `portal-app` chart is deployed a Kubernetes service is created that +instantiates a load balancer. The LB chooses the private interface of one of +the nodes as in the example below (10.0.0.4 is private to the K8s cluster only). +Then to be able to access the portal on port 8989 from outside the K8s & +OpenStack environment, the user needs to assign/get the floating IP address that +corresponds to the private IP as follows:: + + > kubectl -n onap get services|grep "portal-app" + portal-app LoadBalancer 10.43.142.201 10.0.0.4 8989:30215/TCP,8006:30213/TCP,8010:30214/TCP 1d app=portal-app,release=dev + + +In this example, use the 10.0.0.4 private address as a key find the +corresponding public address which in this example is 10.12.6.155. If you're +using OpenStack you'll do the lookup with the horizon GUI or the OpenStack CLI +for your tenant (openstack server list). That IP is then used in your +`/etc/hosts` to map the fixed DNS aliases required by the ONAP Portal as shown +below:: + + 10.12.6.155 portal.api.simpledemo.onap.org + 10.12.6.155 vid.api.simpledemo.onap.org + 10.12.6.155 sdc.api.fe.simpledemo.onap.org + 10.12.6.155 sdc.workflow.plugin.simpledemo.onap.org + 10.12.6.155 sdc.dcae.plugin.simpledemo.onap.org + 10.12.6.155 portal-sdk.simpledemo.onap.org + 10.12.6.155 policy.api.simpledemo.onap.org + 10.12.6.155 aai.api.sparky.simpledemo.onap.org + 10.12.6.155 cli.api.simpledemo.onap.org + 10.12.6.155 msb.api.discovery.simpledemo.onap.org + 10.12.6.155 msb.api.simpledemo.onap.org + 10.12.6.155 clamp.api.simpledemo.onap.org + 10.12.6.155 so.api.simpledemo.onap.org + 10.12.6.155 sdc.workflow.plugin.simpledemo.onap.org + +Ensure you've disabled any proxy settings the browser you are using to access +the portal and then simply access now the new ssl-encrypted URL: +https://portal.api.simpledemo.onap.org:30225/ONAPPORTAL/login.htm + +.. note:: + Using the HTTPS based Portal URL the Browser needs to be configured to accept + unsecure credentials. + Additionally when opening an Application inside the Portal, the Browser + might block the content, which requires to disable the blocking and reloading + of the page + +.. note:: + Besides the ONAP Portal the Components can deliver additional user interfaces, + please check the Component specific documentation. + +.. note:: + + | Alternatives Considered: + + - Kubernetes port forwarding was considered but discarded as it would require + the end user to run a script that opens up port forwarding tunnels to each of + the pods that provides a portal application widget. + + - Reverting to a VNC server similar to what was deployed in the Amsterdam + release was also considered but there were many issues with resolution, lack + of volume mount, /etc/hosts dynamic update, file upload that were a tall order + to solve in time for the Beijing release. + + Observations: + + - If you are not using floating IPs in your Kubernetes deployment and directly attaching + a public IP address (i.e. by using your public provider network) to your K8S Node + VMs' network interface, then the output of 'kubectl -n onap get services | grep "portal-app"' + will show your public IP instead of the private network's IP. Therefore, + you can grab this public IP directly (as compared to trying to find the floating + IP first) and map this IP in /etc/hosts. + +.. figure:: oomLogoV2-Monitor.png + :align: right + +Monitor +======= + +All highly available systems include at least one facility to monitor the +health of components within the system. Such health monitors are often used as +inputs to distributed coordination systems (such as etcd, Zookeeper, or Consul) +and monitoring systems (such as Nagios or Zabbix). OOM provides two mechanisms +to monitor the real-time health of an ONAP deployment: + +- a Consul GUI for a human operator or downstream monitoring systems and + Kubernetes liveness probes that enable automatic healing of failed + containers, and +- a set of liveness probes which feed into the Kubernetes manager which + are described in the Heal section. + +Within ONAP, Consul is the monitoring system of choice and deployed by OOM in +two parts: + +- a three-way, centralized Consul server cluster is deployed as a highly + available monitor of all of the ONAP components, and +- a number of Consul agents. + +The Consul server provides a user interface that allows a user to graphically +view the current health status of all of the ONAP components for which agents +have been created - a sample from the ONAP Integration labs follows: + +.. figure:: consulHealth.png + :align: center + +To see the real-time health of a deployment go to: http://<kubernetes IP>:30270/ui/ +where a GUI much like the following will be found: + + +.. figure:: oomLogoV2-Heal.png + :align: right + +Heal +==== + +The ONAP deployment is defined by Helm charts as mentioned earlier. These Helm +charts are also used to implement automatic recoverability of ONAP components +when individual components fail. Once ONAP is deployed, a "liveness" probe +starts checking the health of the components after a specified startup time. + +Should a liveness probe indicate a failed container it will be terminated and a +replacement will be started in its place - containers are ephemeral. Should the +deployment specification indicate that there are one or more dependencies to +this container or component (for example a dependency on a database) the +dependency will be satisfied before the replacement container/component is +started. This mechanism ensures that, after a failure, all of the ONAP +components restart successfully. + +To test healing, the following command can be used to delete a pod:: + + > kubectl delete pod [pod name] -n [pod namespace] + +One could then use the following command to monitor the pods and observe the +pod being terminated and the service being automatically healed with the +creation of a replacement pod:: + + > kubectl get pods --all-namespaces -o=wide + +.. figure:: oomLogoV2-Scale.png + :align: right + +Scale +===== + +Many of the ONAP components are horizontally scalable which allows them to +adapt to expected offered load. During the Beijing release scaling is static, +that is during deployment or upgrade a cluster size is defined and this cluster +will be maintained even in the presence of faults. The parameter that controls +the cluster size of a given component is found in the values.yaml file for that +component. Here is an excerpt that shows this parameter: + +.. code-block:: yaml + + # default number of instances + replicaCount: 1 + +In order to change the size of a cluster, an operator could use a helm upgrade +(described in detail in the next section) as follows:: + + > helm upgrade --set replicaCount=3 onap/so/mariadb + +The ONAP components use Kubernetes provided facilities to build clustered, +highly available systems including: Services_ with load-balancers, ReplicaSet_, +and StatefulSet_. Some of the open-source projects used by the ONAP components +directly support clustered configurations, for example ODL and MariaDB Galera. + +The Kubernetes Services_ abstraction to provide a consistent access point for +each of the ONAP components, independent of the pod or container architecture +of that component. For example, SDN-C uses OpenDaylight clustering with a +default cluster size of three but uses a Kubernetes service to and change the +number of pods in this abstract this cluster from the other ONAP components +such that the cluster could change size and this change is isolated from the +other ONAP components by the load-balancer implemented in the ODL service +abstraction. + +A ReplicaSet_ is a construct that is used to describe the desired state of the +cluster. For example 'replicas: 3' indicates to Kubernetes that a cluster of 3 +instances is the desired state. Should one of the members of the cluster fail, +a new member will be automatically started to replace it. + +Some of the ONAP components many need a more deterministic deployment; for +example to enable intra-cluster communication. For these applications the +component can be deployed as a Kubernetes StatefulSet_ which will maintain a +persistent identifier for the pods and thus a stable network id for the pods. +For example: the pod names might be web-0, web-1, web-{N-1} for N 'web' pods +with corresponding DNS entries such that intra service communication is simple +even if the pods are physically distributed across multiple nodes. An example +of how these capabilities can be used is described in the Running Consul on +Kubernetes tutorial. + +.. figure:: oomLogoV2-Upgrade.png + :align: right + +Upgrade +======= + +Helm has built-in capabilities to enable the upgrade of pods without causing a +loss of the service being provided by that pod or pods (if configured as a +cluster). As described in the OOM Developer's Guide, ONAP components provide +an abstracted 'service' end point with the pods or containers providing this +service hidden from other ONAP components by a load balancer. This capability +is used during upgrades to allow a pod with a new image to be added to the +service before removing the pod with the old image. This 'make before break' +capability ensures minimal downtime. + +Prior to doing an upgrade, determine of the status of the deployed charts:: + + > helm list + NAME REVISION UPDATED STATUS CHART NAMESPACE + so 1 Mon Feb 5 10:05:22 2018 DEPLOYED so-2.0.1 default + +When upgrading a cluster a parameter controls the minimum size of the cluster +during the upgrade while another parameter controls the maximum number of nodes +in the cluster. For example, SNDC configured as a 3-way ODL cluster might +require that during the upgrade no fewer than 2 pods are available at all times +to provide service while no more than 5 pods are ever deployed across the two +versions at any one time to avoid depleting the cluster of resources. In this +scenario, the SDNC cluster would start with 3 old pods then Kubernetes may add +a new pod (3 old, 1 new), delete one old (2 old, 1 new), add two new pods (2 +old, 3 new) and finally delete the 2 old pods (3 new). During this sequence +the constraints of the minimum of two pods and maximum of five would be +maintained while providing service the whole time. + +Initiation of an upgrade is triggered by changes in the Helm charts. For +example, if the image specified for one of the pods in the SDNC deployment +specification were to change (i.e. point to a new Docker image in the nexus3 +repository - commonly through the change of a deployment variable), the +sequence of events described in the previous paragraph would be initiated. + +For example, to upgrade a container by changing configuration, specifically an +environment value:: + + > helm deploy onap onap/so --version 2.0.1 --set enableDebug=true + +Issuing this command will result in the appropriate container being stopped by +Kubernetes and replaced with a new container with the new environment value. + +To upgrade a component to a new version with a new configuration file enter:: + + > helm deploy onap onap/so --version 2.0.2 -f environments/demo.yaml + +To fetch release history enter:: + + > helm history so + REVISION UPDATED STATUS CHART DESCRIPTION + 1 Mon Feb 5 10:05:22 2018 SUPERSEDED so-2.0.1 Install complete + 2 Mon Feb 5 10:10:55 2018 DEPLOYED so-2.0.2 Upgrade complete + +Unfortunately, not all upgrades are successful. In recognition of this the +lineup of pods within an ONAP deployment is tagged such that an administrator +may force the ONAP deployment back to the previously tagged configuration or to +a specific configuration, say to jump back two steps if an incompatibility +between two ONAP components is discovered after the two individual upgrades +succeeded. + +This rollback functionality gives the administrator confidence that in the +unfortunate circumstance of a failed upgrade the system can be rapidly brought +back to a known good state. This process of rolling upgrades while under +service is illustrated in this short YouTube video showing a Zero Downtime +Upgrade of a web application while under a 10 million transaction per second +load. + +For example, to roll-back back to previous system revision enter:: + + > helm rollback so 1 + + > helm history so + REVISION UPDATED STATUS CHART DESCRIPTION + 1 Mon Feb 5 10:05:22 2018 SUPERSEDED so-2.0.1 Install complete + 2 Mon Feb 5 10:10:55 2018 SUPERSEDED so-2.0.2 Upgrade complete + 3 Mon Feb 5 10:14:32 2018 DEPLOYED so-2.0.1 Rollback to 1 + +.. note:: + + The description field can be overridden to document actions taken or include + tracking numbers. + +Many of the ONAP components contain their own databases which are used to +record configuration or state information. The schemas of these databases may +change from version to version in such a way that data stored within the +database needs to be migrated between versions. If such a migration script is +available it can be invoked during the upgrade (or rollback) by Container +Lifecycle Hooks. Two such hooks are available, PostStart and PreStop, which +containers can access by registering a handler against one or both. Note that +it is the responsibility of the ONAP component owners to implement the hook +handlers - which could be a shell script or a call to a specific container HTTP +endpoint - following the guidelines listed on the Kubernetes site. Lifecycle +hooks are not restricted to database migration or even upgrades but can be used +anywhere specific operations need to be taken during lifecycle operations. + +OOM uses Helm K8S package manager to deploy ONAP components. Each component is +arranged in a packaging format called a chart - a collection of files that +describe a set of k8s resources. Helm allows for rolling upgrades of the ONAP +component deployed. To upgrade a component Helm release you will need an +updated Helm chart. The chart might have modified, deleted or added values, +deployment yamls, and more. To get the release name use:: + + > helm ls + +To easily upgrade the release use:: + + > helm upgrade [RELEASE] [CHART] + +To roll back to a previous release version use:: + + > helm rollback [flags] [RELEASE] [REVISION] + +For example, to upgrade the onap-so helm release to the latest SO container +release v1.1.2: + +- Edit so values.yaml which is part of the chart +- Change "so: nexus3.onap.org:10001/openecomp/so:v1.1.1" to + "so: nexus3.onap.org:10001/openecomp/so:v1.1.2" +- From the chart location run:: + + > helm upgrade onap-so + +The previous so pod will be terminated and a new so pod with an updated so +container will be created. + +.. figure:: oomLogoV2-Delete.png + :align: right + +Delete +====== + +Existing deployments can be partially or fully removed once they are no longer +needed. To minimize errors it is recommended that before deleting components +from a running deployment the operator perform a 'dry-run' to display exactly +what will happen with a given command prior to actually deleting anything. For +example:: + + > helm undeploy onap --dry-run + +will display the outcome of deleting the 'onap' release from the +deployment. +To completely delete a release and remove it from the internal store enter:: + + > helm undeploy onap + +One can also remove individual components from a deployment by changing the +ONAP configuration values. For example, to remove `so` from a running +deployment enter:: + + > helm undeploy onap-so + +will remove `so` as the configuration indicates it's no longer part of the +deployment. This might be useful if a one wanted to replace just `so` by +installing a custom version. |