diff options
Diffstat (limited to 'docs/sections/guides')
16 files changed, 2479 insertions, 0 deletions
diff --git a/docs/sections/guides/access_guides/oom_access_info.rst b/docs/sections/guides/access_guides/oom_access_info.rst new file mode 100644 index 0000000000..2e779105f2 --- /dev/null +++ b/docs/sections/guides/access_guides/oom_access_info.rst @@ -0,0 +1,21 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 +.. International License. +.. http://creativecommons.org/licenses/by/4.0 +.. Copyright (C) 2022 Nordix Foundation + +.. _oom_access_info_guide: + +OOM Access Info +--------------- + +.. figure:: ../../resources/images/oom_logo/oomLogoV2-medium.png + :align: right + +Some relevant information regarding accessing OOM from outside the cluster etc + + +.. toctree:: + :maxdepth: 1 + + oom_ingress_access.rst + diff --git a/docs/sections/guides/access_guides/oom_ingress_access.rst b/docs/sections/guides/access_guides/oom_ingress_access.rst new file mode 100644 index 0000000000..0c64375098 --- /dev/null +++ b/docs/sections/guides/access_guides/oom_ingress_access.rst @@ -0,0 +1,18 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 +.. International License. +.. http://creativecommons.org/licenses/by/4.0 +.. Copyright (C) 2022 Nordix Foundation + +.. Links + + +.. figure:: ../../resources/images/oom_logo/oomLogoV2-medium.png + :align: right + +.. _oom_ingress_access: + + +Ingress access to OOM +##################### + +TBD diff --git a/docs/sections/guides/deployment_guides/oom_customize_overrides.rst b/docs/sections/guides/deployment_guides/oom_customize_overrides.rst new file mode 100644 index 0000000000..3acb8b6ee6 --- /dev/null +++ b/docs/sections/guides/deployment_guides/oom_customize_overrides.rst @@ -0,0 +1,48 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 +.. International License. +.. http://creativecommons.org/licenses/by/4.0 +.. Copyright (C) 2022 Nordix Foundation + +.. Links +.. _helm deploy: https://github.com/onap/oom/blob/master/kubernetes/helm/plugins/deploy/deploy.sh + +.. _oom_customize_overrides: + +OOM Custom Overrides +#################### + +The OOM `helm deploy`_ plugin requires deployment configuration as input, usually in the form of override yaml files. +These input files determine what ONAP components get deployed, and the configuration of the OOM deployment. + +Other helm config options like `--set log.enabled=true|false` are available. + +See the `helm deploy`_ plugin usage section for more detail, or it the plugin has already been installed, execute the following:: + + > helm deploy --help + +Users can customize the override files to suit their required deployment. + +.. note:: + Standard and example override files (e.g. `onap-all.yaml`, `onap-all-ingress-istio.yaml`) + can be found in the `oom/kubernetes/onap/resources/overrides/` directory. + + * Users can selectively enable or disable ONAP components by changing the ``enabled: true/false`` flags. + + * Add to the command line a value for the global master password (ie. --set global.masterPassword=My_superPassw0rd). + + +Enabling/Disabling Components +----------------------------- +Here is an example of the nominal entries that need to be provided. +Different values files are available for different contexts. + +.. collapse:: Default ONAP values.yaml + + .. include:: ../../../../kubernetes/onap/values.yaml + :code: yaml + +| + +Some other heading +------------------ +adva
\ No newline at end of file diff --git a/docs/sections/guides/deployment_guides/oom_deployment.rst b/docs/sections/guides/deployment_guides/oom_deployment.rst new file mode 100644 index 0000000000..110736939e --- /dev/null +++ b/docs/sections/guides/deployment_guides/oom_deployment.rst @@ -0,0 +1,42 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 +.. International License. +.. http://creativecommons.org/licenses/by/4.0 +.. Copyright (C) 2022 Nordix Foundation + +.. Links +.. _ONAP Release Long Term Roadmap: https://wiki.onap.org/display/DW/Long+Term+Roadmap + +.. _oom_deploy_guide: + +OOM Deployment Guide +-------------------- + +.. figure:: ../../resources/images/oom_logo/oomLogoV2-medium.png + :align: right + +ONAP OOM supports several options for the deployment of ONAP using it's helm charts. + + * :ref:`oom_helm_release_repo_deploy` + * :ref:`oom_helm_testing_repo_deploy` + * :ref:`oom_dev_testing_local_deploy` + +.. warning:: + | **Pre-requisites** + | The following sections must be completed before continuing with deployment: + + | :ref:`Set up your base platform<oom_base_setup_guide>` + + +Each deployment method can be customized to deploy a subset of ONAP component applications. +See the :ref:`oom_customize_overrides` section for more details. + + +.. toctree:: + :hidden: + + oom_customize_overrides.rst + oom_helm_release_repo_deploy.rst + oom_helm_testing_repo_deploy.rst + oom_dev_testing_local_deploy.rst + + diff --git a/docs/sections/guides/deployment_guides/oom_dev_testing_local_deploy.rst b/docs/sections/guides/deployment_guides/oom_dev_testing_local_deploy.rst new file mode 100644 index 0000000000..50701dd597 --- /dev/null +++ b/docs/sections/guides/deployment_guides/oom_dev_testing_local_deploy.rst @@ -0,0 +1,87 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 +.. International License. +.. http://creativecommons.org/licenses/by/4.0 +.. Copyright (C) 2022 Nordix Foundation + +.. Links + +.. _oom_dev_testing_local_deploy: + +OOM Developer Testing Deployment +================================ + +Developing and testing changes to the existing OOM project can be done locally by setting up some additional +tools to host the updated helm charts. + +**Step 1.** Clone the OOM repository from ONAP gerrit:: + + > git clone http://gerrit.onap.org/r/oom + + > cd oom/kubernetes + + +**Step 2.** Install Helm Plugin required to push helm charts to local repo:: + + > helm plugin install https://github.com/chartmuseum/helm-push.git --version 0.9.0 + +.. note:: + The ``--version 0.9.0`` is required as new version of helm (3.7.0 and up) is + now using ``push`` directly and helm-push is using ``cm-push`` starting + version ``0.10.0`` and up. + +**Step 3.** Install Chartmuseum + +Chart museum is required to host the helm charts locally when deploying in a development environment:: + + > curl -LO https://s3.amazonaws.com/chartmuseum/release/latest/bin/linux/amd64/chartmuseum + + > chmod +x ./chartmuseum + + > mv ./chartmuseum /usr/local/bin + +**Step 4.** To setup a local Helm server to store the ONAP charts:: + + > mkdir -p ~/helm3-storage + + > chartmuseum --storage local --storage-local-rootdir ~/helm3-storage -port 8879 & + +Note the port number that is listed and use it in the Helm repo add as follows:: + + > helm repo add local http://127.0.0.1:8879 + +**Step 5.** Verify your Helm repository setup with:: + + > helm repo list + NAME URL + local http://127.0.0.1:8879 + +**Step 6.** Build a local Helm repository (from the kubernetes directory):: + + > make SKIP_LINT=TRUE [HELM_BIN=<HELM_PATH>] all + +`HELM_BIN` + Sets the helm binary to be used. The default value use helm from PATH + + +**Step 7.** Display the onap charts that are available to be deployed:: + + > helm repo update + + > helm search repo local + + +.. collapse:: Helm search repo output + + .. include:: ../../resources/helm/helm-search.txt + :code: yaml + +| + +.. note:: + The setup of the Helm repository is a one time activity. If you make changes + to your deployment charts or values be sure to use ``make`` to update your + local Helm repository. + + + + diff --git a/docs/sections/guides/deployment_guides/oom_helm_release_repo_deploy.rst b/docs/sections/guides/deployment_guides/oom_helm_release_repo_deploy.rst new file mode 100644 index 0000000000..f932360e44 --- /dev/null +++ b/docs/sections/guides/deployment_guides/oom_helm_release_repo_deploy.rst @@ -0,0 +1,44 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 +.. International License. +.. http://creativecommons.org/licenses/by/4.0 +.. Copyright (C) 2022 Nordix Foundation + +.. Links +.. _ONAP helm release repository: https://nexus3.onap.org/service/rest/repository/browse/onap-helm-release/ +.. _ONAP Release Long Term Roadmap: https://wiki.onap.org/display/DW/Long+Term+Roadmap + +.. _oom_helm_release_repo_deploy: + +OOM Helm Release Deployment +=========================== + +ONAP hosts the OOM release helm charts in it's `ONAP helm release repository`_. + +This is the officially supported repository for the deployment of OOM. + +.. note:: + ONAP supports up to N-1 releases. See `ONAP Release Long Term Roadmap`_ for more details. + +Add the OOM release repo & Deploy +--------------------------------- +Add the repository: + +- To add the onap release helm repo, execute the following:: + + > helm repo add onap-release https://nexus3.onap.org/repository/onap-helm-release/ + +.. note:: + The following helm command will deploy ONAP charts, with `all` OOM components enabled as per the onap-all.yml overrides file provided to the `-f` flag. + + To customize what applications are deployed, see the :ref:`oom_customize_overrides` section for more details, to provide your own custom overrides yaml file. + +- To deploy a release, execute the following, substituting the <version> tag with your preferred release (ie. 11.0.0):: + + > helm deploy dev onap-release/onap --namespace onap --create-namespace --set global.masterPassword=myAwesomePasswordThatINeedToChange --version <version> -f oom/kubernetes/onap/resources/overrides/onap-all.yaml + + + + + + + diff --git a/docs/sections/guides/deployment_guides/oom_helm_testing_repo_deploy.rst b/docs/sections/guides/deployment_guides/oom_helm_testing_repo_deploy.rst new file mode 100644 index 0000000000..a0dafcef9b --- /dev/null +++ b/docs/sections/guides/deployment_guides/oom_helm_testing_repo_deploy.rst @@ -0,0 +1,46 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 +.. International License. +.. http://creativecommons.org/licenses/by/4.0 +.. Copyright (C) 2022 Nordix Foundation + +.. Links +.. _ONAP helm testing repository: https://nexus3.onap.org/service/rest/repository/browse/onap-helm-testing/ +.. _OOM: https://github.com/onap/oom + +.. _oom_helm_testing_repo_deploy: + +OOM Helm Testing Deployment +=========================== + +ONAP hosts the OOM `testing` helm charts in it's `ONAP helm testing repository`_. + +This is helm repo contains: + + * The `latest` charts built from the head of the `OOM`_ project's master + branch, tagged with the version number of the current development cycle (ie. 11.0.0). + + +Add the OOM testing repo & Deploy +--------------------------------- +.. note:: + The testing helm charts for earlier releases are not fully supported. Test at your own risk. + +Add the repository: + +- To add the onap testing helm repo, execute the following:: + + > helm repo add onap-testing https://nexus3.onap.org/repository/onap-helm-testing/ + +.. note:: + The following helm command will deploy ONAP charts, with `all` OOM components enabled as per the onap-all.yml overrides file provided to the `-f` flag. + + To customize what applications are deployed, see the :ref:`oom_customize_overrides` section for more details, to provide your own custom overrides yaml file. + +- To deploy the latest charts, we need to target the repo added previously:: + + > helm deploy dev onap-testing/onap --namespace onap --create-namespace --set global.masterPassword=myAwesomePasswordThatINeedToChange -f oom/kubernetes/onap/resources/overrides/onap-all.yaml + +This will deploy the latest testing version of the OOM helm charts. + + + diff --git a/docs/sections/guides/development_guides/oom_dev_config_management.rst b/docs/sections/guides/development_guides/oom_dev_config_management.rst new file mode 100644 index 0000000000..36a02dc85d --- /dev/null +++ b/docs/sections/guides/development_guides/oom_dev_config_management.rst @@ -0,0 +1,444 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 +.. International License. +.. http://creativecommons.org/licenses/by/4.0 +.. Copyright 2018-2020 Amdocs, Bell Canada, Orange, Samsung +.. Modification copyright (C) 2022 Nordix Foundation + +.. Links + +.. _oom_dev_config_management: + + +Configuration Management +######################## + +ONAP is a large system composed of many components - each of which are complex +systems in themselves - that needs to be deployed in a number of different +ways. For example, within a single operator's network there may be R&D +deployments under active development, pre-production versions undergoing system +testing and production systems that are operating live networks. Each of these +deployments will differ in significant ways, such as the version of the +software images deployed. In addition, there may be a number of application +specific configuration differences, such as operating system environment +variables. The following describes how the Helm configuration management +system is used within the OOM project to manage both ONAP infrastructure +configuration as well as ONAP components configuration. + +One of the artifacts that OOM/Kubernetes uses to deploy ONAP components is the +deployment specification, yet another yaml file. Within these deployment specs +are a number of parameters as shown in the following example: + +.. code-block:: yaml + + apiVersion: apps/v1 + kind: StatefulSet + metadata: + labels: + app.kubernetes.io/name: zookeeper + helm.sh/chart: zookeeper + app.kubernetes.io/component: server + app.kubernetes.io/managed-by: Tiller + app.kubernetes.io/instance: onap-oof + name: onap-oof-zookeeper + namespace: onap + spec: + <...> + replicas: 3 + selector: + matchLabels: + app.kubernetes.io/name: zookeeper + app.kubernetes.io/component: server + app.kubernetes.io/instance: onap-oof + serviceName: onap-oof-zookeeper-headless + template: + metadata: + labels: + app.kubernetes.io/name: zookeeper + helm.sh/chart: zookeeper + app.kubernetes.io/component: server + app.kubernetes.io/managed-by: Tiller + app.kubernetes.io/instance: onap-oof + spec: + <...> + affinity: + containers: + - name: zookeeper + <...> + image: gcr.io/google_samples/k8szk:v3 + imagePullPolicy: Always + <...> + ports: + - containerPort: 2181 + name: client + protocol: TCP + - containerPort: 3888 + name: election + protocol: TCP + - containerPort: 2888 + name: server + protocol: TCP + <...> + +Note that within the statefulset specification, one of the container arguments +is the key/value pair image: gcr.io/google_samples/k8szk:v3 which +specifies the version of the zookeeper software to deploy. Although the +statefulset specifications greatly simplify statefulset, maintenance of the +statefulset specifications themselves become problematic as software versions +change over time or as different versions are required for different +statefulsets. For example, if the R&D team needs to deploy a newer version of +mariadb than what is currently used in the production environment, they would +need to clone the statefulset specification and change this value. Fortunately, +this problem has been solved with the templating capabilities of Helm. + +The following example shows how the statefulset specifications are modified to +incorporate Helm templates such that key/value pairs can be defined outside of +the statefulset specifications and passed during instantiation of the component. + +.. code-block:: yaml + + apiVersion: apps/v1 + kind: StatefulSet + metadata: + name: {{ include "common.fullname" . }} + namespace: {{ include "common.namespace" . }} + labels: {{- include "common.labels" . | nindent 4 }} + spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: {{- include "common.matchLabels" . | nindent 6 }} + # serviceName is only needed for StatefulSet + # put the postfix part only if you have add a postfix on the service name + serviceName: {{ include "common.servicename" . }}-{{ .Values.service.postfix }} + <...> + template: + metadata: + labels: {{- include "common.labels" . | nindent 8 }} + annotations: {{- include "common.tplValue" (dict "value" .Values.podAnnotations "context" $) | nindent 8 }} + name: {{ include "common.name" . }} + spec: + <...> + containers: + - name: {{ include "common.name" . }} + image: {{ .Values.image }} + imagePullPolicy: {{ .Values.global.pullPolicy | default .Values.pullPolicy }} + ports: + {{- range $index, $port := .Values.service.ports }} + - containerPort: {{ $port.port }} + name: {{ $port.name }} + {{- end }} + {{- range $index, $port := .Values.service.headlessPorts }} + - containerPort: {{ $port.port }} + name: {{ $port.name }} + {{- end }} + <...> + +This version of the statefulset specification has gone through the process of +templating values that are likely to change between statefulsets. Note that the +image is now specified as: image: {{ .Values.image }} instead of a +string used previously. During the statefulset phase, Helm (actually the Helm +sub-component Tiller) substitutes the {{ .. }} entries with a variable defined +in a values.yaml file. The content of this file is as follows: + +.. code-block:: yaml + + <...> + image: gcr.io/google_samples/k8szk:v3 + replicaCount: 3 + <...> + + +Within the values.yaml file there is an image key with the value +`gcr.io/google_samples/k8szk:v3` which is the same value used in +the non-templated version. Once all of the substitutions are complete, the +resulting statefulset specification ready to be used by Kubernetes. + +When creating a template consider the use of default values if appropriate. +Helm templating has built in support for DEFAULT values, here is +an example: + +.. code-block:: yaml + + imagePullSecrets: + - name: "{{ .Values.nsPrefix | default "onap" }}-docker-registry-key" + +The pipeline operator ("|") used here hints at that power of Helm templates in +that much like an operating system command line the pipeline operator allow +over 60 Helm functions to be embedded directly into the template (note that the +Helm template language is a superset of the Go template language). These +functions include simple string operations like upper and more complex flow +control operations like if/else. + +OOM is mainly helm templating. In order to have consistent deployment of the +different components of ONAP, some rules must be followed. + +Templates are provided in order to create Kubernetes resources (Secrets, +Ingress, Services, ...) or part of Kubernetes resources (names, labels, +resources requests and limits, ...). + +a full list and simple description is done in +`kubernetes/common/common/documentation.rst`. + +Service template +---------------- + +In order to create a Service for a component, you have to create a file (with +`service` in the name. +For normal service, just put the following line: + +.. code-block:: yaml + + {{ include "common.service" . }} + +For headless service, the line to put is the following: + +.. code-block:: yaml + + {{ include "common.headlessService" . }} + +The configuration of the service is done in component `values.yaml`: + +.. code-block:: yaml + + service: + name: NAME-OF-THE-SERVICE + postfix: MY-POSTFIX + type: NodePort + annotations: + someAnnotationsKey: value + ports: + - name: tcp-MyPort + port: 5432 + nodePort: 88 + - name: http-api + port: 8080 + nodePort: 89 + - name: https-api + port: 9443 + nodePort: 90 + +`annotations` and `postfix` keys are optional. +if `service.type` is `NodePort`, then you have to give `nodePort` value for your +service ports (which is the end of the computed nodePort, see example). + +It would render the following Service Resource (for a component named +`name-of-my-component`, with version `x.y.z`, helm deployment name +`my-deployment` and `global.nodePortPrefix` `302`): + +.. code-block:: yaml + + apiVersion: v1 + kind: Service + metadata: + annotations: + someAnnotationsKey: value + name: NAME-OF-THE-SERVICE-MY-POSTFIX + labels: + app.kubernetes.io/name: name-of-my-component + helm.sh/chart: name-of-my-component-x.y.z + app.kubernetes.io/instance: my-deployment-name-of-my-component + app.kubernetes.io/managed-by: Tiller + spec: + ports: + - port: 5432 + targetPort: tcp-MyPort + nodePort: 30288 + - port: 8080 + targetPort: http-api + nodePort: 30289 + - port: 9443 + targetPort: https-api + nodePort: 30290 + selector: + app.kubernetes.io/name: name-of-my-component + app.kubernetes.io/instance: my-deployment-name-of-my-component + type: NodePort + +In the deployment or statefulSet file, you needs to set the good labels in +order for the service to match the pods. + +here's an example to be sure it matches (for a statefulSet): + +.. code-block:: yaml + + apiVersion: apps/v1 + kind: StatefulSet + metadata: + name: {{ include "common.fullname" . }} + namespace: {{ include "common.namespace" . }} + labels: {{- include "common.labels" . | nindent 4 }} + spec: + selector: + matchLabels: {{- include "common.matchLabels" . | nindent 6 }} + # serviceName is only needed for StatefulSet + # put the postfix part only if you have add a postfix on the service name + serviceName: {{ include "common.servicename" . }}-{{ .Values.service.postfix }} + <...> + template: + metadata: + labels: {{- include "common.labels" . | nindent 8 }} + annotations: {{- include "common.tplValue" (dict "value" .Values.podAnnotations "context" $) | nindent 8 }} + name: {{ include "common.name" . }} + spec: + <...> + containers: + - name: {{ include "common.name" . }} + ports: + {{- range $index, $port := .Values.service.ports }} + - containerPort: {{ $port.port }} + name: {{ $port.name }} + {{- end }} + {{- range $index, $port := .Values.service.headlessPorts }} + - containerPort: {{ $port.port }} + name: {{ $port.name }} + {{- end }} + <...> + +The configuration of the service is done in component `values.yaml`: + +.. code-block:: yaml + + service: + name: NAME-OF-THE-SERVICE + headless: + postfix: NONE + annotations: + anotherAnnotationsKey : value + publishNotReadyAddresses: true + headlessPorts: + - name: tcp-MyPort + port: 5432 + - name: http-api + port: 8080 + - name: https-api + port: 9443 + +`headless.annotations`, `headless.postfix` and +`headless.publishNotReadyAddresses` keys are optional. + +If `headless.postfix` is not set, then we'll add `-headless` at the end of the +service name. + +If it set to `NONE`, there will be not postfix. + +And if set to something, it will add `-something` at the end of the service +name. + +It would render the following Service Resource (for a component named +`name-of-my-component`, with version `x.y.z`, helm deployment name +`my-deployment` and `global.nodePortPrefix` `302`): + +.. code-block:: yaml + + apiVersion: v1 + kind: Service + metadata: + annotations: + anotherAnnotationsKey: value + name: NAME-OF-THE-SERVICE + labels: + app.kubernetes.io/name: name-of-my-component + helm.sh/chart: name-of-my-component-x.y.z + app.kubernetes.io/instance: my-deployment-name-of-my-component + app.kubernetes.io/managed-by: Tiller + spec: + clusterIP: None + ports: + - port: 5432 + targetPort: tcp-MyPort + nodePort: 30288 + - port: 8080 + targetPort: http-api + nodePort: 30289 + - port: 9443 + targetPort: https-api + nodePort: 30290 + publishNotReadyAddresses: true + selector: + app.kubernetes.io/name: name-of-my-component + app.kubernetes.io/instance: my-deployment-name-of-my-component + type: ClusterIP + +Previous example of StatefulSet would also match (except for the `postfix` part +obviously). + +Creating Deployment or StatefulSet +---------------------------------- + +Deployment and StatefulSet should use the `apps/v1` (which has appeared in +v1.9). +As seen on the service part, the following parts are mandatory: + +.. code-block:: yaml + + apiVersion: apps/v1 + kind: StatefulSet + metadata: + name: {{ include "common.fullname" . }} + namespace: {{ include "common.namespace" . }} + labels: {{- include "common.labels" . | nindent 4 }} + spec: + selector: + matchLabels: {{- include "common.matchLabels" . | nindent 6 }} + # serviceName is only needed for StatefulSet + # put the postfix part only if you have add a postfix on the service name + serviceName: {{ include "common.servicename" . }}-{{ .Values.service.postfix }} + <...> + template: + metadata: + labels: {{- include "common.labels" . | nindent 8 }} + annotations: {{- include "common.tplValue" (dict "value" .Values.podAnnotations "context" $) | nindent 8 }} + name: {{ include "common.name" . }} + spec: + <...> + containers: + - name: {{ include "common.name" . }} + +Dependency Management +--------------------- +These Helm charts describe the desired state +of an ONAP deployment and instruct the Kubernetes container manager as to how +to maintain the deployment in this state. These dependencies dictate the order +in-which the containers are started for the first time such that such +dependencies are always met without arbitrary sleep times between container +startups. For example, the SDC back-end container requires the Elastic-Search, +Cassandra and Kibana containers within SDC to be ready and is also dependent on +DMaaP (or the message-router) to be ready - where ready implies the built-in +"readiness" probes succeeded - before becoming fully operational. When an +initial deployment of ONAP is requested the current state of the system is NULL +so ONAP is deployed by the Kubernetes manager as a set of Docker containers on +one or more predetermined hosts. The hosts could be physical machines or +virtual machines. When deploying on virtual machines the resulting system will +be very similar to "Heat" based deployments, i.e. Docker containers running +within a set of VMs, the primary difference being that the allocation of +containers to VMs is done dynamically with OOM and statically with "Heat". +Example SO deployment descriptor file shows SO's dependency on its mariadb +data-base component: + +SO deployment specification excerpt: + +.. code-block:: yaml + + apiVersion: apps/v1 + kind: Deployment + metadata: + name: {{ include "common.fullname" . }} + namespace: {{ include "common.namespace" . }} + labels: {{- include "common.labels" . | nindent 4 }} + spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: {{- include "common.matchLabels" . | nindent 6 }} + template: + metadata: + labels: + app: {{ include "common.name" . }} + release: {{ .Release.Name }} + spec: + initContainers: + - command: + - /app/ready.py + args: + - --container-name + - so-mariadb + env: + ...
\ No newline at end of file diff --git a/docs/sections/guides/development_guides/oom_dev_container_orchestration.rst b/docs/sections/guides/development_guides/oom_dev_container_orchestration.rst new file mode 100644 index 0000000000..b137bff8b6 --- /dev/null +++ b/docs/sections/guides/development_guides/oom_dev_container_orchestration.rst @@ -0,0 +1,366 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 +.. International License. +.. http://creativecommons.org/licenses/by/4.0 +.. Copyright 2018-2020 Amdocs, Bell Canada, Orange, Samsung +.. Modification copyright (C) 2022 Nordix Foundation + +.. Links +.. _Kubernetes: https://Kubernetes.io/ +.. _AWS Elastic Block Store: https://aws.amazon.com/ebs/ +.. _Azure File: https://docs.microsoft.com/en-us/azure/storage/files/storage-files-introduction +.. _GCE Persistent Disk: https://cloud.google.com/compute/docs/disks/ +.. _Gluster FS: https://www.gluster.org/ +.. _Kubernetes Storage Class: https://Kubernetes.io/docs/concepts/storage/storage-classes/ +.. _Assigning Pods to Nodes: https://Kubernetes.io/docs/concepts/configuration/assign-pod-node/ + + +.. _oom_dev_container_orch: + +Kubernetes Container Orchestration +################################## + +The ONAP components are managed by the Kubernetes_ container management system +which maintains the desired state of the container system as described by one +or more deployment descriptors - similar in concept to OpenStack HEAT +Orchestration Templates. The following sections describe the fundamental +objects managed by Kubernetes, the network these components use to communicate +with each other and other entities outside of ONAP and the templates that +describe the configuration and desired state of the ONAP components. + +**Name Spaces** + +Within the namespaces are Kubernetes services that provide external +connectivity to pods that host Docker containers. + +ONAP Components to Kubernetes Object Relationships +-------------------------------------------------- +Kubernetes deployments consist of multiple objects: + +- **nodes** - a worker machine - either physical or virtual - that hosts + multiple containers managed by Kubernetes. +- **services** - an abstraction of a logical set of pods that provide a + micro-service. +- **pods** - one or more (but typically one) container(s) that provide specific + application functionality. +- **persistent volumes** - One or more permanent volumes need to be established + to hold non-ephemeral configuration and state data. + +The relationship between these objects is shown in the following figure: + +.. .. uml:: +.. +.. @startuml +.. node PH { +.. component Service { +.. component Pod0 +.. component Pod1 +.. } +.. } +.. +.. database PV +.. @enduml + +.. figure:: ../../resources/images/k8s/kubernetes_objects.png + +OOM uses these Kubernetes objects as described in the following sections. + +Nodes +~~~~~ +OOM works with both physical and virtual worker machines. + +* Virtual Machine Deployments - If ONAP is to be deployed onto a set of virtual + machines, the creation of the VMs is outside of the scope of OOM and could be + done in many ways, such as + + * manually, for example by a user using the OpenStack Horizon dashboard or + AWS EC2, or + * automatically, for example with the use of a OpenStack Heat Orchestration + Template which builds an ONAP stack, Azure ARM template, AWS CloudFormation + Template, or + * orchestrated, for example with Cloudify creating the VMs from a TOSCA + template and controlling their life cycle for the life of the ONAP + deployment. + +* Physical Machine Deployments - If ONAP is to be deployed onto physical + machines there are several options but the recommendation is to use Rancher + along with Helm to associate hosts with a Kubernetes cluster. + +Pods +~~~~ +A group of containers with shared storage and networking can be grouped +together into a Kubernetes pod. All of the containers within a pod are +co-located and co-scheduled so they operate as a single unit. Within ONAP +Amsterdam release, pods are mapped one-to-one to docker containers although +this may change in the future. As explained in the Services section below the +use of Pods within each ONAP component is abstracted from other ONAP +components. + +Services +~~~~~~~~ +OOM uses the Kubernetes service abstraction to provide a consistent access +point for each of the ONAP components independent of the pod or container +architecture of that component. For example, the SDNC component may introduce +OpenDaylight clustering as some point and change the number of pods in this +component to three or more but this change will be isolated from the other ONAP +components by the service abstraction. A service can include a load balancer +on its ingress to distribute traffic between the pods and even react to dynamic +changes in the number of pods if they are part of a replica set. + +Persistent Volumes +~~~~~~~~~~~~~~~~~~ +To enable ONAP to be deployed into a wide variety of cloud infrastructures a +flexible persistent storage architecture, built on Kubernetes persistent +volumes, provides the ability to define the physical storage in a central +location and have all ONAP components securely store their data. + +When deploying ONAP into a public cloud, available storage services such as +`AWS Elastic Block Store`_, `Azure File`_, or `GCE Persistent Disk`_ are +options. Alternatively, when deploying into a private cloud the storage +architecture might consist of Fiber Channel, `Gluster FS`_, or iSCSI. Many +other storage options existing, refer to the `Kubernetes Storage Class`_ +documentation for a full list of the options. The storage architecture may vary +from deployment to deployment but in all cases a reliable, redundant storage +system must be provided to ONAP with which the state information of all ONAP +components will be securely stored. The Storage Class for a given deployment is +a single parameter listed in the ONAP values.yaml file and therefore is easily +customized. Operation of this storage system is outside the scope of the OOM. + +.. code-block:: yaml + + Insert values.yaml code block with storage block here + +Once the storage class is selected and the physical storage is provided, the +ONAP deployment step creates a pool of persistent volumes within the given +physical storage that is used by all of the ONAP components. ONAP components +simply make a claim on these persistent volumes (PV), with a persistent volume +claim (PVC), to gain access to their storage. + +The following figure illustrates the relationships between the persistent +volume claims, the persistent volumes, the storage class, and the physical +storage. + +.. graphviz:: + + digraph PV { + label = "Persistance Volume Claim to Physical Storage Mapping" + { + node [shape=cylinder] + D0 [label="Drive0"] + D1 [label="Drive1"] + Dx [label="Drivex"] + } + { + node [shape=Mrecord label="StorageClass:ceph"] + sc + } + { + node [shape=point] + p0 p1 p2 + p3 p4 p5 + } + subgraph clusterSDC { + label="SDC" + PVC0 + PVC1 + } + subgraph clusterSDNC { + label="SDNC" + PVC2 + } + subgraph clusterSO { + label="SO" + PVCn + } + PV0 -> sc + PV1 -> sc + PV2 -> sc + PVn -> sc + + sc -> {D0 D1 Dx} + PVC0 -> PV0 + PVC1 -> PV1 + PVC2 -> PV2 + PVCn -> PVn + + # force all of these nodes to the same line in the given order + subgraph { + rank = same; PV0;PV1;PV2;PVn;p0;p1;p2 + PV0->PV1->PV2->p0->p1->p2->PVn [style=invis] + } + + subgraph { + rank = same; D0;D1;Dx;p3;p4;p5 + D0->D1->p3->p4->p5->Dx [style=invis] + } + + } + +In-order for an ONAP component to use a persistent volume it must make a claim +against a specific persistent volume defined in the ONAP common charts. Note +that there is a one-to-one relationship between a PVC and PV. The following is +an excerpt from a component chart that defines a PVC: + +.. code-block:: yaml + + Insert PVC example here + +OOM Networking with Kubernetes +------------------------------ + +- DNS +- Ports - Flattening the containers also expose port conflicts between the + containers which need to be resolved. + + +Pod Placement Rules +------------------- +OOM will use the rich set of Kubernetes node and pod affinity / +anti-affinity rules to minimize the chance of a single failure resulting in a +loss of ONAP service. Node affinity / anti-affinity is used to guide the +Kubernetes orchestrator in the placement of pods on nodes (physical or virtual +machines). For example: + +- if a container used Intel DPDK technology the pod may state that it as + affinity to an Intel processor based node, or +- geographical based node labels (such as the Kubernetes standard zone or + region labels) may be used to ensure placement of a DCAE complex close to the + VNFs generating high volumes of traffic thus minimizing networking cost. + Specifically, if nodes were pre-assigned labels East and West, the pod + deployment spec to distribute pods to these nodes would be: + +.. code-block:: yaml + + nodeSelector: + failure-domain.beta.Kubernetes.io/region: {{ .Values.location }} + +- "location: West" is specified in the `values.yaml` file used to deploy + one DCAE cluster and "location: East" is specified in a second `values.yaml` + file (see OOM Configuration Management for more information about + configuration files like the `values.yaml` file). + +Node affinity can also be used to achieve geographic redundancy if pods are +assigned to multiple failure domains. For more information refer to `Assigning +Pods to Nodes`_. + +.. note:: + One could use Pod to Node assignment to totally constrain Kubernetes when + doing initial container assignment to replicate the Amsterdam release + OpenStack Heat based deployment. Should one wish to do this, each VM would + need a unique node name which would be used to specify a node constaint + for every component. These assignment could be specified in an environment + specific values.yaml file. Constraining Kubernetes in this way is not + recommended. + +Kubernetes has a comprehensive system called Taints and Tolerations that can be +used to force the container orchestrator to repel pods from nodes based on +static events (an administrator assigning a taint to a node) or dynamic events +(such as a node becoming unreachable or running out of disk space). There are +no plans to use taints or tolerations in the ONAP Beijing release. Pod +affinity / anti-affinity is the concept of creating a spacial relationship +between pods when the Kubernetes orchestrator does assignment (both initially +an in operation) to nodes as explained in Inter-pod affinity and anti-affinity. +For example, one might choose to co-located all of the ONAP SDC containers on a +single node as they are not critical runtime components and co-location +minimizes overhead. On the other hand, one might choose to ensure that all of +the containers in an ODL cluster (SDNC and APPC) are placed on separate nodes +such that a node failure has minimal impact to the operation of the cluster. +An example of how pod affinity / anti-affinity is shown below: + +Pod Affinity / Anti-Affinity + +.. code-block:: yaml + + apiVersion: v1 + kind: Pod + metadata: + name: with-pod-affinity + spec: + affinity: + podAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: security + operator: In + values: + - S1 + topologyKey: failure-domain.beta.Kubernetes.io/zone + podAntiAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + labelSelector: + matchExpressions: + - key: security + operator: In + values: + - S2 + topologyKey: Kubernetes.io/hostname + containers: + - name: with-pod-affinity + image: gcr.io/google_containers/pause:2.0 + +This example contains both podAffinity and podAntiAffinity rules, the first +rule is is a must (requiredDuringSchedulingIgnoredDuringExecution) while the +second will be met pending other considerations +(preferredDuringSchedulingIgnoredDuringExecution). Preemption Another feature +that may assist in achieving a repeatable deployment in the presence of faults +that may have reduced the capacity of the cloud is assigning priority to the +containers such that mission critical components have the ability to evict less +critical components. Kubernetes provides this capability with Pod Priority and +Preemption. Prior to having more advanced production grade features available, +the ability to at least be able to re-deploy ONAP (or a subset of) reliably +provides a level of confidence that should an outage occur the system can be +brought back on-line predictably. + +Health Checks +------------- + +Monitoring of ONAP components is configured in the agents within JSON files and +stored in gerrit under the consul-agent-config, here is an example from the AAI +model loader (aai-model-loader-health.json): + +.. code-block:: json + + { + "service": { + "name": "A&AI Model Loader", + "checks": [ + { + "id": "model-loader-process", + "name": "Model Loader Presence", + "script": "/consul/config/scripts/model-loader-script.sh", + "interval": "15s", + "timeout": "1s" + } + ] + } + } + +Liveness Probes +--------------- + +These liveness probes can simply check that a port is available, that a +built-in health check is reporting good health, or that the Consul health check +is positive. For example, to monitor the SDNC component has following liveness +probe can be found in the SDNC DB deployment specification: + +.. code-block:: yaml + + sdnc db liveness probe + + livenessProbe: + exec: + command: ["mysqladmin", "ping"] + initialDelaySeconds: 30 periodSeconds: 10 + timeoutSeconds: 5 + +The 'initialDelaySeconds' control the period of time between the readiness +probe succeeding and the liveness probe starting. 'periodSeconds' and +'timeoutSeconds' control the actual operation of the probe. Note that +containers are inherently ephemeral so the healing action destroys failed +containers and any state information within it. To avoid a loss of state, a +persistent volume should be used to store all data that needs to be persisted +over the re-creation of a container. Persistent volumes have been created for +the database components of each of the projects and the same technique can be +used for all persistent state information.
\ No newline at end of file diff --git a/docs/sections/guides/development_guides/oom_dev_helm_chart_info.rst b/docs/sections/guides/development_guides/oom_dev_helm_chart_info.rst new file mode 100644 index 0000000000..533f60e29b --- /dev/null +++ b/docs/sections/guides/development_guides/oom_dev_helm_chart_info.rst @@ -0,0 +1,172 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 +.. International License. +.. http://creativecommons.org/licenses/by/4.0 +.. Copyright 2018-2020 Amdocs, Bell Canada, Orange, Samsung +.. Modification copyright (C) 2022 Nordix Foundation + +.. Links +.. _Helm Charts: https://artifacthub.io/packages/search +.. _aai: https://github.com/onap/oom/tree/master/kubernetes/aai +.. _name.tpl: https://github.com/onap/oom/blob/master/kubernetes/common/common/templates/_name.tpl +.. _namespace.tpl: https://github.com/onap/oom/blob/master/kubernetes/common/common/templates/_namespace.tpl + +.. _oom_helm_chart_info: + +Helm Charts +########### + +A Helm chart is a collection of files that describe a related set of Kubernetes +resources. A simple chart might be used to deploy something simple, like a +memcached pod, while a complex chart might contain many micro-service arranged +in a hierarchy as found in the `aai`_ ONAP component. + +Charts are created as files laid out in a particular directory tree, then they +can be packaged into versioned archives to be deployed. There is a public +archive of `Helm Charts`_ on ArtifactHUB that includes many technologies applicable +to ONAP. Some of these charts have been used in ONAP and all of the ONAP charts +have been created following the guidelines provided. + +An example structure of the OOM common helm charts is shown below: + +.. code-block:: bash + + common + ├── cassandra + │  ├── Chart.yaml + │  ├── resources + │  │  ├── config + │  │  │  └── docker-entrypoint.sh + │  │  ├── exec.py + │  │  └── restore.sh + │  ├── templates + │  │  ├── backup + │  │  │  ├── configmap.yaml + │  │  │  ├── cronjob.yaml + │  │  │  ├── pv.yaml + │  │  │  └── pvc.yaml + │  │  ├── configmap.yaml + │  │  ├── pv.yaml + │  │  ├── service.yaml + │  │  └── statefulset.yaml + │  └── values.yaml + ├── common + │  ├── Chart.yaml + │  ├── templates + │  │  ├── _createPassword.tpl + │  │  ├── _ingress.tpl + │  │  ├── _labels.tpl + │  │  ├── _mariadb.tpl + │  │  ├── _name.tpl + │  │  ├── _namespace.tpl + │  │  ├── _repository.tpl + │  │  ├── _resources.tpl + │  │  ├── _secret.yaml + │  │  ├── _service.tpl + │  │  ├── _storage.tpl + │  │  └── _tplValue.tpl + │  └── values.yaml + ├── ... + └── postgres-legacy +   ├── Chart.yaml + ├── charts + └── configs + +The common section of charts consists of a set of templates that assist with +parameter substitution (`name.tpl`_, `namespace.tpl`_, etc) and a set of +charts for components used throughout ONAP. When the common components are used +by other charts they are instantiated each time or we can deploy a shared +instances for several components. + +All of the ONAP components have charts that follow the pattern shown below: + +.. code-block:: bash + + name-of-my-component + ├── Chart.yaml + ├── component + │  └── subcomponent-folder + ├── charts + │  └── subchart-folder + ├── resources + │  ├── folder1 + │  │  ├── file1 + │  │  └── file2 + │  └── folder1 + │  ├── file3 + │  └── folder3 + │    └── file4 + ├── templates + │  ├── NOTES.txt + │  ├── configmap.yaml + │  ├── deployment.yaml + │  ├── ingress.yaml + │  ├── job.yaml + │  ├── secrets.yaml + │  └── service.yaml + └── values.yaml + +Note that the /components sub dir may include a hierarchy of sub +components and in themselves can be quite complex. + +You can use either `charts` or `components` folder for your subcomponents. +`charts` folder means that the subcomponent will always been deployed. + +`components` folders means we can choose if we want to deploy the subcomponent. + +This choice is done in root `values.yaml`: + +.. code-block:: yaml + + --- + global: + key: value + + component1: + enabled: true + component2: + enabled: true + +Then in `Chart.yaml` dependencies section, you'll use these values: + +.. code-block:: yaml + + --- + dependencies: + - name: common + version: ~x.y-0 + repository: '@local' + - name: component1 + version: ~x.y-0 + repository: 'file://components/component1' + condition: component1.enabled + - name: component2 + version: ~x.y-0 + repository: 'file://components/component2' + condition: component2.enabled + +Configuration of the components varies somewhat from component to component but +generally follows the pattern of one or more `configmap.yaml` files which can +directly provide configuration to the containers in addition to processing +configuration files stored in the `config` directory. It is the responsibility +of each ONAP component team to update these configuration files when changes +are made to the project containers that impact configuration. + +The following section describes how the hierarchical ONAP configuration system +is key to management of such a large system. + + +.. MISC +.. ==== +.. Note that although OOM uses Kubernetes facilities to minimize the effort +.. required of the ONAP component owners to implement a successful rolling +.. upgrade strategy there are other considerations that must be taken into +.. consideration. +.. For example, external APIs - both internal and external to ONAP - should be +.. designed to gracefully accept transactions from a peer at a different +.. software version to avoid deadlock situations. Embedded version codes in +.. messages may facilitate such capabilities. +.. +.. Within each of the projects a new configuration repository contains all of +.. the project specific configuration artifacts. As changes are made within +.. the project, it's the responsibility of the project team to make appropriate +.. changes to the configuration data. diff --git a/docs/sections/guides/development_guides/oom_development.rst b/docs/sections/guides/development_guides/oom_development.rst new file mode 100644 index 0000000000..169e211450 --- /dev/null +++ b/docs/sections/guides/development_guides/oom_development.rst @@ -0,0 +1,64 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 +.. International License. +.. http://creativecommons.org/licenses/by/4.0 +.. Copyright (C) 2022 Nordix Foundation + +.. Links +.. _Helm: https://docs.helm.sh/ +.. _Helm Charts: https://github.com/kubernetes/charts +.. _Kubernetes: https://Kubernetes.io/ +.. _Docker: https://www.docker.com/ +.. _Nexus: https://nexus.onap.org/ + +.. _oom_dev_guide: + +OOM Developer Guide +################### + +.. figure:: ../../resources/images/oom_logo/oomLogoV2-medium.png + :align: right + +ONAP consists of a large number of components, each of which are substantial +projects within themselves, which results in a high degree of complexity in +deployment and management. To cope with this complexity the ONAP Operations +Manager (OOM) uses a Helm_ model of ONAP - Helm being the primary management +system for Kubernetes_ container systems - to drive all user driven life-cycle +management operations. The Helm model of ONAP is composed of a set of +hierarchical Helm charts that define the structure of the ONAP components and +the configuration of these components. These charts are fully parameterized +such that a single environment file defines all of the parameters needed to +deploy ONAP. A user of ONAP may maintain several such environment files to +control the deployment of ONAP in multiple environments such as development, +pre-production, and production. + + + +.. rubric:: Container Background + +Linux containers allow for an application and all of its operating system +dependencies to be packaged and deployed as a single unit without including a +guest operating system as done with virtual machines. The most popular +container solution is Docker_ which provides tools for container management +like the Docker Host (dockerd) which can create, run, stop, move, or delete a +container. Docker has a very popular registry of containers images that can be +used by any Docker system; however, in the ONAP context, Docker images are +built by the standard CI/CD flow and stored in Nexus_ repositories. OOM uses +the "standard" ONAP docker containers and three new ones specifically created +for OOM. + +Containers are isolated from each other primarily via name spaces within the +Linux kernel without the need for multiple guest operating systems. As such, +multiple containers can be deployed with little overhead such as all of ONAP +can be deployed on a single host. With some optimization of the ONAP components +(e.g. elimination of redundant database instances) it may be possible to deploy +ONAP on a single laptop computer. + +The following sections describe how the ONAP Helm charts are constructed. + +.. toctree:: + :maxdepth: 1 + + oom_dev_helm_chart_info.rst + oom_dev_config_management.rst + oom_dev_container_orchestration.rst + diff --git a/docs/sections/guides/infra_guides/oom_base_config_setup.rst b/docs/sections/guides/infra_guides/oom_base_config_setup.rst new file mode 100644 index 0000000000..d228f5df56 --- /dev/null +++ b/docs/sections/guides/infra_guides/oom_base_config_setup.rst @@ -0,0 +1,187 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 +.. International License. +.. http://creativecommons.org/licenses/by/4.0 +.. Copyright (C) 2022 Nordix Foundation + +.. Links +.. _HELM Best Practices Guide: https://docs.helm.sh/chart_best_practices/#requirements +.. _helm installation guide: https://helm.sh/docs/intro/install/ +.. _kubectl installation guide: https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/ +.. _Curated applications for Kubernetes: https://github.com/kubernetes/charts +.. _Cert-Manager Installation documentation: https://cert-manager.io/docs/installation/kubernetes/ +.. _Cert-Manager kubectl plugin documentation: https://cert-manager.io/docs/usage/kubectl-plugin/ +.. _Strimzi Apache Kafka Operator helm Installation documentation: https://strimzi.io/docs/operators/in-development/deploying.html#deploying-cluster-operator-helm-chart-str + +.. _oom_base_setup_guide: + +OOM Base Platform +################# + +As part of the initial base setup of the host Kubernetes cluster, +the following mandatory installation and configuration steps must be completed. + +.. contents:: + :backlinks: top + :depth: 1 + :local: +.. + +For additional platform add-ons, see the :ref:`oom_base_optional_addons` section. + +Install & configure kubectl +*************************** +The Kubernetes command line interface used to manage a Kubernetes cluster needs to be installed +and configured to run as non root. + +For additional information regarding kubectl installation and configuration see the `kubectl installation guide`_ + +To install kubectl, execute the following, replacing the <recommended-kubectl-version> with the version defined +in the :ref:`versions_table` table:: + + > curl -LO https://dl.k8s.io/release/v<recommended-kubectl-version>/bin/linux/amd64/kubectl + + > chmod +x ./kubectl + + > sudo mv ./kubectl /usr/local/bin/kubectl + + > mkdir ~/.kube + + > cp kube_config_cluster.yml ~/.kube/config.onap + + > export KUBECONFIG=~/.kube/config.onap + + > kubectl config use-context onap + +Validate the installation:: + + > kubectl get nodes + +:: + + NAME STATUS ROLES AGE VERSION + onap-control-1 Ready controlplane,etcd 3h53m v1.23.8 + onap-control-2 Ready controlplane,etcd 3h53m v1.23.8 + onap-k8s-1 Ready worker 3h53m v1.23.8 + onap-k8s-2 Ready worker 3h53m v1.23.8 + onap-k8s-3 Ready worker 3h53m v1.23.8 + onap-k8s-4 Ready worker 3h53m v1.23.8 + onap-k8s-5 Ready worker 3h53m v1.23.8 + onap-k8s-6 Ready worker 3h53m v1.23.8 + + +Install & configure helm +************************ +Helm is used for package and configuration management of the relevant helm charts. +For additional information, see the `helm installation guide`_ + +To install helm, execute the following, replacing the <recommended-helm-version> with the version defined +in the :ref:`versions_table` table:: + + > wget https://get.helm.sh/helm-v<recommended-helm-version>-linux-amd64.tar.gz + + > tar -zxvf helm-v<recommended-helm-version>-linux-amd64.tar.gz + + > sudo mv linux-amd64/helm /usr/local/bin/helm + +Verify the helm version with:: + + > helm version + +Helm's default CNCF provided `Curated applications for Kubernetes`_ repository called +*stable* can be removed to avoid confusion:: + + > helm repo remove stable + +Install the additional OOM plugins required to un/deploy the OOM helm charts:: + + > git clone http://gerrit.onap.org/r/oom + + > cp -R ~/oom/kubernetes/helm/plugins/ /usr/local/bin/helm/plugins + +Verify the plugins are installed:: + + > helm plugin ls + +:: + + NAME VERSION DESCRIPTION + deploy 1.0.0 install (upgrade if release exists) parent charty and all subcharts as separate but related releases + undeploy 1.0.0 delete parent chart and subcharts that were deployed as separate releases + + +Install the strimzi kafka operator +********************************** +Strimzi Apache Kafka provides a way to run an Apache Kafka cluster on Kubernetes +in various deployment configurations by using kubernetes operators. +Operators are a method of packaging, deploying, and managing Kubernetes applications. + +Strimzi Operators extend the Kubernetes functionality, automating common +and complex tasks related to a Kafka deployment. By implementing +knowledge of Kafka operations in code, the Kafka administration +tasks are simplified and require less manual intervention. + +The Strimzi cluster operator is deployed using helm to install the parent chart +containing all of the required custom resource definitions. This should be done +by a kubernetes administrator to allow for deployment of custom resources in to +any kubernetes namespace within the cluster. + +Full installation instructions can be found in the +`Strimzi Apache Kafka Operator helm Installation documentation`_. + +To add the required helm repository, execute the following:: + + > helm repo add strimzi https://strimzi.io/charts/ + +To install the strimzi kafka operator, execute the following, replacing the <recommended-strimzi-version> with the version defined +in the :ref:`versions_table` table:: + + > helm install strimzi-kafka-operator strimzi/strimzi-kafka-operator --namespace strimzi-system --version <recommended-strimzi-version> --set watchAnyNamespace=true --create-namespace + +Verify the installation:: + + > kubectl get po -n strimzi-system + +:: + + NAME READY STATUS RESTARTS AGE + strimzi-cluster-operator-7f7d6b46cf-mnpjr 1/1 Running 0 2m + + +Install Cert-Manager +******************** + +Cert-Manager is a native Kubernetes certificate management controller. +It can help with issuing certificates from a variety of sources, such as +Let’s Encrypt, HashiCorp Vault, Venafi, a simple signing key pair, self +signed or external issuers. It ensures certificates are valid and up to +date, and attempt to renew certificates at a configured time before expiry. + +Cert-Manager is deployed using regular YAML manifests which include all +the needed resources (the CustomResourceDefinitions, cert-manager, +namespace, and the webhook component). + +Full installation instructions, including details on how to configure extra +functionality in Cert-Manager can be found in the +`Cert-Manager Installation documentation`_. + +There is also a kubectl plugin (kubectl cert-manager) that can help you +to manage cert-manager resources inside your cluster. For installation +steps, please refer to `Cert-Manager kubectl plugin documentation`_. + + +To install cert-manager, execute the following, replacing the <recommended-cm-version> with the version defined +in the :ref:`versions_table` table:: + + > kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v<recommended-cm-version>/cert-manager.yaml + +Verify the installation:: + + > kubectl get po -n cert-manager + +:: + + NAME READY STATUS RESTARTS AGE + cert-manager-776c4cfcb6-vgnpw 1/1 Running 0 2m + cert-manager-cainjector-7d9668978d-hdxf7 1/1 Running 0 2m + cert-manager-webhook-66c8f6c75-dxmtz 1/1 Running 0 2m + diff --git a/docs/sections/guides/infra_guides/oom_base_optional_addons.rst b/docs/sections/guides/infra_guides/oom_base_optional_addons.rst new file mode 100644 index 0000000000..4b4fbf7883 --- /dev/null +++ b/docs/sections/guides/infra_guides/oom_base_optional_addons.rst @@ -0,0 +1,41 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 +.. International License. +.. http://creativecommons.org/licenses/by/4.0 +.. Copyright (C) 2022 Nordix Foundation + +.. Links +.. _Prometheus stack README: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#readme + +.. _oom_base_optional_addons: + +OOM Optional Addons +################### + +The following optional applications can be added to your kubernetes environment. + +Install Prometheus Stack +************************ + +Prometheus is an open-source systems monitoring and alerting toolkit with +an active ecosystem. + +Kube Prometheus Stack is a collection of Kubernetes manifests, Grafana +dashboards, and Prometheus rules combined with documentation and scripts to +provide easy to operate end-to-end Kubernetes cluster monitoring with +Prometheus using the Prometheus Operator. As it includes both Prometheus +Operator and Grafana dashboards, there is no need to set up them separately. +See the `Prometheus stack README`_ for more information. + +To install the prometheus stack, execute the following: + +- Add the prometheus-community Helm repository:: + + > helm repo add prometheus-community https://prometheus-community.github.io/helm-charts + +- Update your local Helm chart repository cache:: + + > helm repo update + +- To install prometheus, execute the following, replacing the <recommended-pm-version> with the version defined in the :ref:`versions_table` table:: + + > helm install prometheus prometheus-community/kube-prometheus-stack --namespace=prometheus --create-namespace --version=<recommended-pm-version> diff --git a/docs/sections/guides/infra_guides/oom_infra_setup.rst b/docs/sections/guides/infra_guides/oom_infra_setup.rst new file mode 100644 index 0000000000..d8fb743f42 --- /dev/null +++ b/docs/sections/guides/infra_guides/oom_infra_setup.rst @@ -0,0 +1,72 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 +.. International License. +.. http://creativecommons.org/licenses/by/4.0 +.. Copyright (C) 2022 Nordix Foundation + +.. Links +.. _Kubernetes: https://kubernetes.io/ +.. _Kubernetes best practices: https://kubernetes.io/docs/setup/best-practices/cluster-large/ +.. _kubelet confg guide: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ + +.. _oom_infra_setup_guide: + +OOM Infrastructure Guide +######################## + +.. figure:: ../../resources/images/oom_logo/oomLogoV2-medium.png + :align: right + +OOM deploys and manages ONAP on a pre-established Kubernetes_ cluster - the +creation of this cluster is outside of the scope of the OOM project as there +are many options including public clouds with pre-established environments. +If creation of a Kubernetes cluster is required, the life-cycle of this +cluster is independent of the life-cycle of the ONAP components themselves. + +.. rubric:: Minimum Hardware Configuration + +Some recommended hardware requirements are provided below. Note that this is for a +full ONAP deployment (all components). + +.. table:: OOM Hardware Requirements + + ===== ===== ====== ==================== + RAM HD vCores Ports + ===== ===== ====== ==================== + 224GB 160GB 112 0.0.0.0/0 (all open) + ===== ===== ====== ==================== + +Customizing ONAP to deploy only components that are needed will drastically reduce these requirements. +See the :ref:`OOM customized deployment<oom_customize_overrides>` section for more details. + +.. note:: + | Kubernetes supports a maximum of 110 pods per node - this can be overcome by modifying your kubelet config. + | See the `kubelet confg guide`_ for more information. + + | The use of many small nodes is preferred over a few larger nodes (for example 14 x 16GB - 8 vCores each). + + | OOM can be deployed on a private set of physical hosts or VMs (or even a combination of the two). + +.. rubric:: Software Requirements + +The versions of software that are supported by OOM are as follows: + +.. _versions_table: + +.. table:: OOM Software Requirements + + ============== =========== ======= ======== ======== ============ ================= ======= + Release Kubernetes Helm kubectl Docker Cert-Manager Prometheus Stack Strimzi + ============== =========== ======= ======== ======== ============ ================= ======= + Jakarta 1.22.4 3.6.3 1.22.4 20.10.x 1.8.0 35.x 0.28.0 + Kohn 1.23.8 3.8.2 1.23.8 20.10.x 1.8.0 35.x 0.31.1 + ============== =========== ======= ======== ======== ============ ================= ======= + + +.. toctree:: + :hidden: + + oom_base_config_setup.rst + oom_base_optional_addons.rst + oom_setup_ingress_controller.rst + + diff --git a/docs/sections/guides/infra_guides/oom_setup_ingress_controller.rst b/docs/sections/guides/infra_guides/oom_setup_ingress_controller.rst new file mode 100644 index 0000000000..8c261fdfd7 --- /dev/null +++ b/docs/sections/guides/infra_guides/oom_setup_ingress_controller.rst @@ -0,0 +1,176 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 +.. International License. +.. http://creativecommons.org/licenses/by/4.0 +.. Copyright 2020, Samsung Electronics +.. Modification copyright (C) 2022 Nordix Foundation + +.. Links +.. _metallb Metal Load Balancer installation: https://metallb.universe.tf/installation/ + +.. _oom_setup_ingress_controller: + +OOM Ingress controller setup +############################ + +.. warning:: + This guide should prob go in the Optional addons section + +This optional guide provides instruction how to setup experimental ingress controller +feature. For this, we are hosting our cluster on OpenStack VMs and using the +Rancher Kubernetes Engine (RKE) to deploy and manage our Kubernetes Cluster and +ingress controller + +.. contents:: + :backlinks: top + :depth: 1 + :local: +.. + +The result at the end of this tutorial will be: + +#. Customization of the cluster.yaml file for ingress controller support + +#. Installation and configuration test DNS server for ingress host resolution + on testing machines + +#. Installation and configuration MLB (Metal Load Balancer) required for + exposing ingress service + +#. Installation and configuration NGINX ingress controller + +#. Additional info how to deploy ONAP with services exposed via Ingress + controller + +Customize cluster.yml file +************************** +Before setup cluster for ingress purposes DNS cluster IP and ingress provider +should be configured and following: + +.. code-block:: yaml + + --- + <...> + restore: + restore: false + snapshot_name: "" + ingress: + provider: none + dns: + provider: coredns + upstreamnameservers: + - <custer_dns_ip>:31555 + +Where the <cluster_dns_ip> should be set to the same IP as the CONTROLPANE +node. + +For external load balancer purposes, minimum one of the worker node should be +configured with external IP address accessible outside the cluster. It can be +done using the following example node configuration: + +.. code-block:: yaml + + --- + <...> + - address: <external_ip> + internal_address: <internal_ip> + port: "22" + role: + - worker + hostname_override: "onap-worker-0" + user: ubuntu + ssh_key_path: "~/.ssh/id_rsa" + <...> + +Where the <external_ip> is external worker node IP address, and <internal_ip> +is internal node IP address if it is required. + + +DNS server configuration and installation +***************************************** +DNS server deployed on the Kubernetes cluster makes it easy to use services +exposed through ingress controller because it resolves all subdomain related to +the ONAP cluster to the load balancer IP. Testing ONAP cluster requires a lot +of entries on the target machines in the /etc/hosts. Adding many entries into +the configuration files on testing machines is quite problematic and error +prone. The better wait is to create central DNS server with entries for all +virtual host pointed to simpledemo.onap.org and add custom DNS server as a +target DNS server for testing machines and/or as external DNS for Kubernetes +cluster. + +DNS server has automatic installation and configuration script, so installation +is quite easy:: + + > cd kubernetes/contrib/dns-server-for-vhost-ingress-testing + + > ./deploy\_dns.sh + +After DNS deploy you need to setup DNS entry on the target testing machine. +Because DNS listen on non standard port configuration require iptables rules +on the target machine. Please follow the configuration proposed by the deploy +scripts. +Example output depends on the IP address and example output looks like bellow:: + + DNS server already deployed: + 1. You can add the DNS server to the target machine using following commands: + sudo iptables -t nat -A OUTPUT -p tcp -d 192.168.211.211 --dport 53 -j DNAT --to-destination 10.10.13.14:31555 + sudo iptables -t nat -A OUTPUT -p udp -d 192.168.211.211 --dport 53 -j DNAT --to-destination 10.10.13.14:31555 + sudo sysctl -w net.ipv4.conf.all.route_localnet=1 + sudo sysctl -w net.ipv4.ip_forward=1 + 2. Update /etc/resolv.conf file with nameserver 192.168.211.211 entry on your target machine + + +MetalLB Load Balancer installation and configuration +**************************************************** + +By default pure Kubernetes cluster requires external load balancer if we want +to expose external port using LoadBalancer settings. For this purpose MetalLB +can be used. Before installing the MetalLB you need to ensure that at least one +worker has assigned IP accessible outside the cluster. + +MetalLB Load balancer can be easily installed using automatic install script:: + + > cd kubernetes/contrib/metallb-loadbalancer-inst + + > ./install-metallb-on-cluster.sh + + +Configuration of the Nginx ingress controller +********************************************* + +After installation of the DNS server and ingress controller, we can install and +configure ingress controller. +It can be done using the following commands:: + + > cd kubernetes/contrib/ingress-nginx-post-inst + + > kubectl apply -f nginx_ingress_cluster_config.yaml + + > kubectl apply -f nginx_ingress_enable_optional_load_balacer_service.yaml + +After deploying the NGINX ingress controller, you can ensure that the ingress port is +exposed as load balancer service with an external IP address:: + + > kubectl get svc -n ingress-nginx + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + default-http-backend ClusterIP 10.10.10.10 <none> 80/TCP 25h + ingress-nginx LoadBalancer 10.10.10.11 10.12.13.14 80:31308/TCP,443:30314/TCP 24h + + +ONAP with ingress exposed services +********************************** +If you want to deploy onap with services exposed through ingress controller you +can use full onap deploy yaml:: + + > onap/resources/overrides/onap-all-ingress-nginx-vhost.yaml + +Ingress also can be enabled on any onap setup override using following code: + +.. code-block:: yaml + + --- + <...> + global: + <...> + ingress: + enabled: true + diff --git a/docs/sections/guides/user_guides/oom_user_guide.rst b/docs/sections/guides/user_guides/oom_user_guide.rst new file mode 100644 index 0000000000..c0f4f6ef73 --- /dev/null +++ b/docs/sections/guides/user_guides/oom_user_guide.rst @@ -0,0 +1,651 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 +.. International License. +.. http://creativecommons.org/licenses/by/4.0 +.. Copyright (C) 2022 Nordix Foundation + +.. Links +.. _Curated applications for Kubernetes: https://github.com/kubernetes/charts +.. _Services: https://kubernetes.io/docs/concepts/services-networking/service/ +.. _ReplicaSet: https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/ +.. _StatefulSet: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/ +.. _Helm Documentation: https://docs.helm.sh/helm/ +.. _Helm: https://docs.helm.sh/ +.. _Kubernetes: https://Kubernetes.io/ +.. _Kubernetes LoadBalancer: https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer +.. _user-guide-label: + + +.. _oom_user_guide: + + +OOM User Guide +############## + +.. warning:: + + **THIS PAGE NEEDS TO BE EITHER REWRITTEN OR SOMETING AS SOME INFO IS NO LONGER RELEVANT** + +The ONAP Operations Manager (OOM) provide the ability to manage the entire +life-cycle of an ONAP installation, from the initial deployment to final +decommissioning. This guide provides instructions for users of ONAP to +use the Kubernetes_/Helm_ system as a complete ONAP management system. + +This guide provides many examples of Helm command line operations. For a +complete description of these commands please refer to the `Helm +Documentation`_. + +.. figure:: ../../resources/images/oom_logo/oomLogoV2-medium.png + :align: right + +The following sections describe the life-cycle operations: + +- Deploy_ - with built-in component dependency management +- Configure_ - unified configuration across all ONAP components +- Monitor_ - real-time health monitoring feeding to a Consul UI and Kubernetes +- Heal_- failed ONAP containers are recreated automatically +- Scale_ - cluster ONAP services to enable seamless scaling +- Upgrade_ - change-out containers or configuration with little or no service impact +- Delete_ - cleanup individual containers or entire deployments + +.. figure:: ../../resources/images/oom_logo/oomLogoV2-Deploy.png + :align: right + +Deploy +====== + +The OOM team with assistance from the ONAP project teams, have built a +comprehensive set of Helm charts, yaml files very similar to TOSCA files, that +describe the composition of each of the ONAP components and the relationship +within and between components. Using this model Helm is able to deploy all of +ONAP with a few simple commands. + +Please refer to the :ref:`oom_deploy_guide` for deployment pre-requisites and options + +.. note:: + Refer to the :ref:`oom_customize_overrides` section on how to update overrides.yaml and values.yaml + +.. figure:: ../../resources/images/oom_logo/oomLogoV2-Configure.png + :align: right + +Configure +========= + +Each project within ONAP has its own configuration data generally consisting +of: environment variables, configuration files, and database initial values. +Many technologies are used across the projects resulting in significant +operational complexity and an inability to apply global parameters across the +entire ONAP deployment. OOM solves this problem by introducing a common +configuration technology, Helm charts, that provide a hierarchical +configuration with the ability to override values with higher +level charts or command line options. + +The structure of the configuration of ONAP is shown in the following diagram. +Note that key/value pairs of a parent will always take precedence over those +of a child. Also note that values set on the command line have the highest +precedence of all. + +.. graphviz:: + + digraph config { + { + node [shape=folder] + oValues [label="values.yaml"] + demo [label="onap-demo.yaml"] + prod [label="onap-production.yaml"] + oReq [label="Chart.yaml"] + soValues [label="values.yaml"] + soReq [label="Chart.yaml"] + mdValues [label="values.yaml"] + } + { + oResources [label="resources"] + } + onap -> oResources + onap -> oValues + oResources -> environments + oResources -> oReq + oReq -> so + environments -> demo + environments -> prod + so -> soValues + so -> soReq + so -> charts + charts -> mariadb + mariadb -> mdValues + + } + +The top level onap/values.yaml file contains the values required to be set +before deploying ONAP. Here is the contents of this file: + +.. collapse:: Default ONAP values.yaml + + .. include:: ../../../../kubernetes/onap/values.yaml + :code: yaml + +| + + +One may wish to create a value file that is specific to a given deployment such +that it can be differentiated from other deployments. For example, a +onap-development.yaml file may create a minimal environment for development +while onap-production.yaml might describe a production deployment that operates +independently of the developer version. + +For example, if the production OpenStack instance was different from a +developer's instance, the onap-production.yaml file may contain a different +value for the vnfDeployment/openstack/oam_network_cidr key as shown below. + +.. code-block:: yaml + + nsPrefix: onap + nodePortPrefix: 302 + apps: consul msb mso message-router sdnc vid robot portal policy appc aai + sdc dcaegen2 log cli multicloud clamp vnfsdk aaf kube2msb + dataRootDir: /dockerdata-nfs + + # docker repositories + repository: + onap: nexus3.onap.org:10001 + oom: oomk8s + aai: aaionap + filebeat: docker.elastic.co + + image: + pullPolicy: Never + + # vnf deployment environment + vnfDeployment: + openstack: + ubuntu_14_image: "Ubuntu_14.04.5_LTS" + public_net_id: "e8f51956-00dd-4425-af36-045716781ffc" + oam_network_id: "d4769dfb-c9e4-4f72-b3d6-1d18f4ac4ee6" + oam_subnet_id: "191f7580-acf6-4c2b-8ec0-ba7d99b3bc4e" + oam_network_cidr: "192.168.30.0/24" + <...> + + +To deploy ONAP with this environment file, enter:: + + > helm deploy local/onap -n onap -f onap/resources/environments/onap-production.yaml --set global.masterPassword=password + + +.. collapse:: Default ONAP values.yaml + + .. include:: ../../resources/yaml/environments_onap_demo.yaml + :code: yaml + +| + +When deploying all of ONAP, the dependencies section of the Chart.yaml file +controls which and what version of the ONAP components are included. +Here is an excerpt of this file: + +.. code-block:: yaml + + dependencies: + <...> + - name: so + version: ~11.0.0 + repository: '@local' + condition: so.enabled + <...> + +The ~ operator in the `so` version value indicates that the latest "10.X.X" +version of `so` shall be used thus allowing the chart to allow for minor +upgrades that don't impact the so API; hence, version 10.0.1 will be installed +in this case. + +The onap/resources/environment/dev.yaml (see the excerpt below) enables +for fine grained control on what components are included as part of this +deployment. By changing this `so` line to `enabled: false` the `so` component +will not be deployed. If this change is part of an upgrade the existing `so` +component will be shut down. Other `so` parameters and even `so` child values +can be modified, for example the `so`'s `liveness` probe could be disabled +(which is not recommended as this change would disable auto-healing of `so`). + +.. code-block:: yaml + + ################################################################# + # Global configuration overrides. + # + # These overrides will affect all helm charts (ie. applications) + # that are listed below and are 'enabled'. + ################################################################# + global: + <...> + + ################################################################# + # Enable/disable and configure helm charts (ie. applications) + # to customize the ONAP deployment. + ################################################################# + aaf: + enabled: false + <...> + so: # Service Orchestrator + enabled: true + + replicaCount: 1 + + liveness: + # necessary to disable liveness probe when setting breakpoints + # in debugger so K8s doesn't restart unresponsive container + enabled: true + + <...> + +Accessing the ONAP Portal using OOM and a Kubernetes Cluster +------------------------------------------------------------ + +The ONAP deployment created by OOM operates in a private IP network that isn't +publicly accessible (i.e. OpenStack VMs with private internal network) which +blocks access to the ONAP Portal. To enable direct access to this Portal from a +user's own environment (a laptop etc.) the portal application's port 8989 is +exposed through a `Kubernetes LoadBalancer`_ object. + +Typically, to be able to access the Kubernetes nodes publicly a public address +is assigned. In OpenStack this is a floating IP address. + +When the `portal-app` chart is deployed a Kubernetes service is created that +instantiates a load balancer. The LB chooses the private interface of one of +the nodes as in the example below (10.0.0.4 is private to the K8s cluster only). +Then to be able to access the portal on port 8989 from outside the K8s & +OpenStack environment, the user needs to assign/get the floating IP address that +corresponds to the private IP as follows:: + + > kubectl -n onap get services|grep "portal-app" + portal-app LoadBalancer 10.43.142.201 10.0.0.4 8989:30215/TCP,8006:30213/TCP,8010:30214/TCP 1d app=portal-app,release=dev + + +In this example, use the 11.0.0.4 private address as a key find the +corresponding public address which in this example is 10.12.6.155. If you're +using OpenStack you'll do the lookup with the horizon GUI or the OpenStack CLI +for your tenant (openstack server list). That IP is then used in your +`/etc/hosts` to map the fixed DNS aliases required by the ONAP Portal as shown +below:: + + 10.12.6.155 portal.api.simpledemo.onap.org + 10.12.6.155 vid.api.simpledemo.onap.org + 10.12.6.155 sdc.api.fe.simpledemo.onap.org + 10.12.6.155 sdc.workflow.plugin.simpledemo.onap.org + 10.12.6.155 sdc.dcae.plugin.simpledemo.onap.org + 10.12.6.155 portal-sdk.simpledemo.onap.org + 10.12.6.155 policy.api.simpledemo.onap.org + 10.12.6.155 aai.api.sparky.simpledemo.onap.org + 10.12.6.155 cli.api.simpledemo.onap.org + 10.12.6.155 msb.api.discovery.simpledemo.onap.org + 10.12.6.155 msb.api.simpledemo.onap.org + 10.12.6.155 clamp.api.simpledemo.onap.org + 10.12.6.155 so.api.simpledemo.onap.org + 10.12.6.155 sdc.workflow.plugin.simpledemo.onap.org + +Ensure you've disabled any proxy settings the browser you are using to access +the portal and then simply access now the new ssl-encrypted URL: +``https://portal.api.simpledemo.onap.org:30225/ONAPPORTAL/login.htm`` + +.. note:: + Using the HTTPS based Portal URL the Browser needs to be configured to accept + unsecure credentials. + Additionally when opening an Application inside the Portal, the Browser + might block the content, which requires to disable the blocking and reloading + of the page + +.. note:: + Besides the ONAP Portal the Components can deliver additional user interfaces, + please check the Component specific documentation. + +.. note:: + + | Alternatives Considered: + + - Kubernetes port forwarding was considered but discarded as it would + require the end user to run a script that opens up port forwarding tunnels + to each of the pods that provides a portal application widget. + + - Reverting to a VNC server similar to what was deployed in the Amsterdam + release was also considered but there were many issues with resolution, + lack of volume mount, /etc/hosts dynamic update, file upload that were + a tall order to solve in time for the Beijing release. + + Observations: + + - If you are not using floating IPs in your Kubernetes deployment and + directly attaching a public IP address (i.e. by using your public provider + network) to your K8S Node VMs' network interface, then the output of + 'kubectl -n onap get services | grep "portal-app"' + will show your public IP instead of the private network's IP. Therefore, + you can grab this public IP directly (as compared to trying to find the + floating IP first) and map this IP in /etc/hosts. + +.. figure:: ../../resources/images/oom_logo/oomLogoV2-Monitor.png + :align: right + +Monitor +======= + +All highly available systems include at least one facility to monitor the +health of components within the system. Such health monitors are often used as +inputs to distributed coordination systems (such as etcd, Zookeeper, or Consul) +and monitoring systems (such as Nagios or Zabbix). OOM provides two mechanisms +to monitor the real-time health of an ONAP deployment: + +- a Consul GUI for a human operator or downstream monitoring systems and + Kubernetes liveness probes that enable automatic healing of failed + containers, and +- a set of liveness probes which feed into the Kubernetes manager which + are described in the Heal section. + +Within ONAP, Consul is the monitoring system of choice and deployed by OOM in +two parts: + +- a three-way, centralized Consul server cluster is deployed as a highly + available monitor of all of the ONAP components, and +- a number of Consul agents. + +The Consul server provides a user interface that allows a user to graphically +view the current health status of all of the ONAP components for which agents +have been created - a sample from the ONAP Integration labs follows: + +.. figure:: ../../resources/images/consul/consulHealth.png + :align: center + +To see the real-time health of a deployment go to: ``http://<kubernetes IP>:30270/ui/`` +where a GUI much like the following will be found: + +.. note:: + If Consul GUI is not accessible, you can refer this + `kubectl port-forward <https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/>`_ method to access an application + +.. figure:: ../../resources/images/oom_logo/oomLogoV2-Heal.png + :align: right + +Heal +==== + +The ONAP deployment is defined by Helm charts as mentioned earlier. These Helm +charts are also used to implement automatic recoverability of ONAP components +when individual components fail. Once ONAP is deployed, a "liveness" probe +starts checking the health of the components after a specified startup time. + +Should a liveness probe indicate a failed container it will be terminated and a +replacement will be started in its place - containers are ephemeral. Should the +deployment specification indicate that there are one or more dependencies to +this container or component (for example a dependency on a database) the +dependency will be satisfied before the replacement container/component is +started. This mechanism ensures that, after a failure, all of the ONAP +components restart successfully. + +To test healing, the following command can be used to delete a pod:: + + > kubectl delete pod [pod name] -n [pod namespace] + +One could then use the following command to monitor the pods and observe the +pod being terminated and the service being automatically healed with the +creation of a replacement pod:: + + > kubectl get pods --all-namespaces -o=wide + +.. figure:: ../../resources/images/oom_logo/oomLogoV2-Scale.png + :align: right + +Scale +===== + +Many of the ONAP components are horizontally scalable which allows them to +adapt to expected offered load. During the Beijing release scaling is static, +that is during deployment or upgrade a cluster size is defined and this cluster +will be maintained even in the presence of faults. The parameter that controls +the cluster size of a given component is found in the values.yaml file for that +component. Here is an excerpt that shows this parameter: + +.. code-block:: yaml + + # default number of instances + replicaCount: 1 + +In order to change the size of a cluster, an operator could use a helm upgrade +(described in detail in the next section) as follows:: + + > helm upgrade [RELEASE] [CHART] [flags] + +The RELEASE argument can be obtained from the following command:: + + > helm list + +Below is the example for the same:: + + > helm list + NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE + dev 1 Wed Oct 14 13:49:52 2020 DEPLOYED onap-11.0.0 Kohn onap + dev-cassandra 5 Thu Oct 15 14:45:34 2020 DEPLOYED cassandra-11.0.0 onap + dev-contrib 1 Wed Oct 14 13:52:53 2020 DEPLOYED contrib-11.0.0 onap + dev-mariadb-galera 1 Wed Oct 14 13:55:56 2020 DEPLOYED mariadb-galera-11.0.0 onap + +Here the Name column shows the RELEASE NAME, In our case we want to try the +scale operation on cassandra, thus the RELEASE NAME would be dev-cassandra. + +Now we need to obtain the chart name for cassandra. Use the below +command to get the chart name:: + + > helm search cassandra + +Below is the example for the same:: + + > helm search cassandra + NAME CHART VERSION APP VERSION DESCRIPTION + local/cassandra 11.0.0 ONAP cassandra + local/portal-cassandra 11.0.0 Portal cassandra + local/aaf-cass 11.0.0 ONAP AAF cassandra + local/sdc-cs 11.0.0 ONAP Service Design and Creation Cassandra + +Here the Name column shows the chart name. As we want to try the scale +operation for cassandra, thus the corresponding chart name is local/cassandra + + +Now we have both the command's arguments, thus we can perform the +scale operation for cassandra as follows:: + + > helm upgrade dev-cassandra local/cassandra --set replicaCount=3 + +Using this command we can scale up or scale down the cassandra db instances. + + +The ONAP components use Kubernetes provided facilities to build clustered, +highly available systems including: Services_ with load-balancers, ReplicaSet_, +and StatefulSet_. Some of the open-source projects used by the ONAP components +directly support clustered configurations, for example ODL and MariaDB Galera. + +The Kubernetes Services_ abstraction to provide a consistent access point for +each of the ONAP components, independent of the pod or container architecture +of that component. For example, SDN-C uses OpenDaylight clustering with a +default cluster size of three but uses a Kubernetes service to and change the +number of pods in this abstract this cluster from the other ONAP components +such that the cluster could change size and this change is isolated from the +other ONAP components by the load-balancer implemented in the ODL service +abstraction. + +A ReplicaSet_ is a construct that is used to describe the desired state of the +cluster. For example 'replicas: 3' indicates to Kubernetes that a cluster of 3 +instances is the desired state. Should one of the members of the cluster fail, +a new member will be automatically started to replace it. + +Some of the ONAP components many need a more deterministic deployment; for +example to enable intra-cluster communication. For these applications the +component can be deployed as a Kubernetes StatefulSet_ which will maintain a +persistent identifier for the pods and thus a stable network id for the pods. +For example: the pod names might be web-0, web-1, web-{N-1} for N 'web' pods +with corresponding DNS entries such that intra service communication is simple +even if the pods are physically distributed across multiple nodes. An example +of how these capabilities can be used is described in the Running Consul on +Kubernetes tutorial. + +.. figure:: ../../resources/images/oom_logo/oomLogoV2-Upgrade.png + :align: right + +Upgrade +======= + +Helm has built-in capabilities to enable the upgrade of pods without causing a +loss of the service being provided by that pod or pods (if configured as a +cluster). As described in the OOM Developer's Guide, ONAP components provide +an abstracted 'service' end point with the pods or containers providing this +service hidden from other ONAP components by a load balancer. This capability +is used during upgrades to allow a pod with a new image to be added to the +service before removing the pod with the old image. This 'make before break' +capability ensures minimal downtime. + +Prior to doing an upgrade, determine of the status of the deployed charts:: + + > helm list + NAME REVISION UPDATED STATUS CHART NAMESPACE + so 1 Mon Feb 5 10:05:22 2020 DEPLOYED so-11.0.0 onap + +When upgrading a cluster a parameter controls the minimum size of the cluster +during the upgrade while another parameter controls the maximum number of nodes +in the cluster. For example, SNDC configured as a 3-way ODL cluster might +require that during the upgrade no fewer than 2 pods are available at all times +to provide service while no more than 5 pods are ever deployed across the two +versions at any one time to avoid depleting the cluster of resources. In this +scenario, the SDNC cluster would start with 3 old pods then Kubernetes may add +a new pod (3 old, 1 new), delete one old (2 old, 1 new), add two new pods (2 +old, 3 new) and finally delete the 2 old pods (3 new). During this sequence +the constraints of the minimum of two pods and maximum of five would be +maintained while providing service the whole time. + +Initiation of an upgrade is triggered by changes in the Helm charts. For +example, if the image specified for one of the pods in the SDNC deployment +specification were to change (i.e. point to a new Docker image in the nexus3 +repository - commonly through the change of a deployment variable), the +sequence of events described in the previous paragraph would be initiated. + +For example, to upgrade a container by changing configuration, specifically an +environment value:: + + > helm upgrade so onap/so --version 11.0.1 --set enableDebug=true + +Issuing this command will result in the appropriate container being stopped by +Kubernetes and replaced with a new container with the new environment value. + +To upgrade a component to a new version with a new configuration file enter:: + + > helm upgrade so onap/so --version 11.0.1 -f environments/demo.yaml + +To fetch release history enter:: + + > helm history so + REVISION UPDATED STATUS CHART DESCRIPTION + 1 Mon Jul 5 10:05:22 2022 SUPERSEDED so-11.0.0 Install complete + 2 Mon Jul 5 10:10:55 2022 DEPLOYED so-11.0.1 Upgrade complete + +Unfortunately, not all upgrades are successful. In recognition of this the +lineup of pods within an ONAP deployment is tagged such that an administrator +may force the ONAP deployment back to the previously tagged configuration or to +a specific configuration, say to jump back two steps if an incompatibility +between two ONAP components is discovered after the two individual upgrades +succeeded. + +This rollback functionality gives the administrator confidence that in the +unfortunate circumstance of a failed upgrade the system can be rapidly brought +back to a known good state. This process of rolling upgrades while under +service is illustrated in this short YouTube video showing a Zero Downtime +Upgrade of a web application while under a 10 million transaction per second +load. + +For example, to roll-back back to previous system revision enter:: + + > helm rollback so 1 + + > helm history so + REVISION UPDATED STATUS CHART DESCRIPTION + 1 Mon Jul 5 10:05:22 2022 SUPERSEDED so-11.0.0 Install complete + 2 Mon Jul 5 10:10:55 2022 SUPERSEDED so-11.0.1 Upgrade complete + 3 Mon Jul 5 10:14:32 2022 DEPLOYED so-11.0.0 Rollback to 1 + +.. note:: + + The description field can be overridden to document actions taken or include + tracking numbers. + +Many of the ONAP components contain their own databases which are used to +record configuration or state information. The schemas of these databases may +change from version to version in such a way that data stored within the +database needs to be migrated between versions. If such a migration script is +available it can be invoked during the upgrade (or rollback) by Container +Lifecycle Hooks. Two such hooks are available, PostStart and PreStop, which +containers can access by registering a handler against one or both. Note that +it is the responsibility of the ONAP component owners to implement the hook +handlers - which could be a shell script or a call to a specific container HTTP +endpoint - following the guidelines listed on the Kubernetes site. Lifecycle +hooks are not restricted to database migration or even upgrades but can be used +anywhere specific operations need to be taken during lifecycle operations. + +OOM uses Helm K8S package manager to deploy ONAP components. Each component is +arranged in a packaging format called a chart - a collection of files that +describe a set of k8s resources. Helm allows for rolling upgrades of the ONAP +component deployed. To upgrade a component Helm release you will need an +updated Helm chart. The chart might have modified, deleted or added values, +deployment yamls, and more. To get the release name use:: + + > helm ls + +To easily upgrade the release use:: + + > helm upgrade [RELEASE] [CHART] + +To roll back to a previous release version use:: + + > helm rollback [flags] [RELEASE] [REVISION] + +For example, to upgrade the onap-so helm release to the latest SO container +release v1.1.2: + +- Edit so values.yaml which is part of the chart +- Change "so: nexus3.onap.org:10001/openecomp/so:v1.1.1" to + "so: nexus3.onap.org:10001/openecomp/so:v1.1.2" +- From the chart location run:: + + > helm upgrade onap-so + +The previous so pod will be terminated and a new so pod with an updated so +container will be created. + +.. figure:: ../../resources/images/oom_logo/oomLogoV2-Delete.png + :align: right + +Delete +====== + +Existing deployments can be partially or fully removed once they are no longer +needed. To minimize errors it is recommended that before deleting components +from a running deployment the operator perform a 'dry-run' to display exactly +what will happen with a given command prior to actually deleting anything. +For example:: + + > helm undeploy onap --dry-run + +will display the outcome of deleting the 'onap' release from the +deployment. +To completely delete a release and remove it from the internal store enter:: + + > helm undeploy onap + +Once complete undeploy is done then delete the namespace as well +using following command:: + + > kubectl delete namespace <name of namespace> + +.. note:: + You need to provide the namespace name which you used during deployment, + below is the example:: + + > kubectl delete namespace onap + +One can also remove individual components from a deployment by changing the +ONAP configuration values. For example, to remove `so` from a running +deployment enter:: + + > helm undeploy onap-so + +will remove `so` as the configuration indicates it's no longer part of the +deployment. This might be useful if a one wanted to replace just `so` by +installing a custom version. |