diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/oom_developer_guide.rst | 667 |
1 files changed, 429 insertions, 238 deletions
diff --git a/docs/oom_developer_guide.rst b/docs/oom_developer_guide.rst index a2ccc1ba60..c3fb603d04 100644 --- a/docs/oom_developer_guide.rst +++ b/docs/oom_developer_guide.rst @@ -79,159 +79,123 @@ have been created following the guidelines provided. The top level of the ONAP charts is shown below: -.. graphviz:: +.. code-block:: bash + + common + ├── cassandra + │ ├── Chart.yaml + │ ├── requirements.yaml + │ ├── resources + │ │ ├── config + │ │ │ └── docker-entrypoint.sh + │ │ ├── exec.py + │ │ └── restore.sh + │ ├── templates + │ │ ├── backup + │ │ │ ├── configmap.yaml + │ │ │ ├── cronjob.yaml + │ │ │ ├── pv.yaml + │ │ │ └── pvc.yaml + │ │ ├── configmap.yaml + │ │ ├── pv.yaml + │ │ ├── service.yaml + │ │ └── statefulset.yaml + │ └── values.yaml + ├── common + │ ├── Chart.yaml + │ ├── templates + │ │ ├── _createPassword.tpl + │ │ ├── _ingress.tpl + │ │ ├── _labels.tpl + │ │ ├── _mariadb.tpl + │ │ ├── _name.tpl + │ │ ├── _namespace.tpl + │ │ ├── _repository.tpl + │ │ ├── _resources.tpl + │ │ ├── _secret.yaml + │ │ ├── _service.tpl + │ │ ├── _storage.tpl + │ │ └── _tplValue.tpl + │ └── values.yaml + ├── ... + └── postgres-legacy + ├── Chart.yaml + ├── requirements.yaml + ├── charts + └── configs - digraph onap_top_chart { - rankdir="LR"; - { - node [shape=folder] - oValues [label="values.yaml"] - oChart [label="Chart.yaml"] - dev [label="dev.yaml"] - prod [label="prod.yaml"] - crb [label="clusterrolebindings.yaml"] - secrets [label="secrets.yaml"] - } - { - node [style=dashed] - vCom [label="component"] - } +The common section of charts consists of a set of templates that assist with +parameter substitution (`_name.tpl`, `_namespace.tpl` and others) and a set of charts +for components used throughout ONAP. When the common components are used by other charts they +are instantiated each time or we can deploy a shared instances for several components. - onap -> oValues - onap -> oChart - onap -> templates - onap -> resources - oValues -> vCom - resources -> environments - environments -> dev - environments -> prod - templates -> crb - templates -> secrets - } +All of the ONAP components have charts that follow the pattern shown below: -Within the `values.yaml` file at the `onap` level, one will find a set of -boolean values that control which of the ONAP components get deployed as shown -below: +.. code-block:: bash + + name-of-my-component + ├── Chart.yaml + ├── requirements.yaml + ├── component + │ └── subcomponent-folder + ├── charts + │ └── subchart-folder + ├── resources + │ ├── folder1 + │ │ ├── file1 + │ │ └── file2 + │ └── folder1 + │ ├── file3 + │ └── folder3 + │ └── file4 + ├── templates + │ ├── NOTES.txt + │ ├── configmap.yaml + │ ├── deployment.yaml + │ ├── ingress.yaml + │ ├── job.yaml + │ ├── secrets.yaml + │ └── service.yaml + └── values.yaml + +Note that the component charts / components may include a hierarchy of sub +components and in themselves can be quite complex. + +You can use either `charts` or `components` folder for your subcomponents. +`charts` folder means that the subcomponent will always been deployed. + +`components` folders means we can choose if we want to deploy the sub component. + +This choice is done in root `values.yaml`: .. code-block:: yaml - aaf: # Application Authorization Framework - enabled: false - <...> - so: # Service Orchestrator - enabled: true - -By setting these flags a custom deployment can be created and used during -deployment by using the `-f` Helm option as follows:: - - > helm install local/onap -name development -f dev.yaml - -Note that there are one or more example deployment files in the -`onap/resources/environments/` directory. It is best practice to create a -unique deployment file for each environment used to ensure consistent -behaviour. - -To aid in the long term supportability of ONAP, a set of common charts have -been created (and will be expanded in subsequent releases of ONAP) that can be -used by any of the ONAP components by including the common component in its -`requirements.yaml` file. The common components are arranged as follows: - -.. graphviz:: - - digraph onap_common_chart { - rankdir="LR"; - { - node [shape=folder] - mValues [label="values.yaml"] - ccValues [label="values.yaml"] - comValues [label="values.yaml"] - comChart [label="Chart.yaml"] - ccChart [label="Chart.yaml"] - mChart [label="Chart.yaml"] - - mReq [label="requirements.yaml"] - mService [label="service.yaml"] - mMap [label="configmap.yaml"] - ccName [label="_name.tpl"] - ccNS [label="_namespace.tpl"] - } - { - cCom [label="common"] - mTemp [label="templates"] - ccTemp [label="templates"] - } - { - more [label="...",style=dashed] - } - - common -> comValues - common -> comChart - common -> cCom - common -> mysql - common -> more - - cCom -> ccChart - cCom -> ccValues - cCom -> ccTemp - ccTemp -> ccName - ccTemp -> ccNS - - mysql -> mValues - mysql -> mChart - mysql -> mReq - mysql -> mTemp - mTemp -> mService - mTemp -> mMap - } + --- + global: + key: value -The common section of charts consists of a set of templates that assist with -parameter substitution (`_name.tpl` and `_namespace.tpl`) and a set of charts -for components used throughout ONAP. Initially `mysql` is in the common area -but this will expand to include other databases like `mariadb-galera`, -`postgres`, and `cassandra`. Other candidates for common components include -`redis` and`kafka`. When the common components are used by other charts they -are instantiated each time. In subsequent ONAP releases some of the common -components could be a setup as services that are used by multiple ONAP -components thus minimizing the deployment and operational costs. - -All of the ONAP components have charts that follow the pattern shown below: - -.. graphviz:: - - digraph onap_component_chart { - rankdir="LR"; - { - node [shape=folder] - cValues [label="values.yaml"] - cChart [label="Chart.yaml"] - cService [label="service.yaml"] - cMap [label="configmap.yaml"] - cFiles [label="config file(s)"] - } - { - cCharts [label="charts"] - cTemp [label="templates"] - cRes [label="resources"] + component1: + enabled: true + component2: + enabled: true - } - { - sCom [label="component",style=dashed] - } +Then in `requirements.yaml`, you'll use these values: - component -> cValues - component -> cChart - component -> cCharts - component -> cTemp - component -> cRes - cTemp -> cService - cTemp -> cMap - cRes -> config - config -> cFiles - cCharts -> sCom - } +.. code-block:: yaml -Note that the component charts may include a hierarchy of components and in -themselves can be quite complex. + --- + dependencies: + - name: common + version: ~x.y-0 + repository: '@local' + - name: component1 + version: ~x.y-0 + repository: 'file://components/component1' + condition: component1.enabled + - name: component2 + version: ~x.y-0 + repository: 'file://components/component2' + condition: component2.enabled Configuration of the components varies somewhat from component to component but generally follows the pattern of one or more `configmap.yaml` files which can @@ -260,126 +224,134 @@ configuration as well as ONAP components configuration. One of the artifacts that OOM/Kubernetes uses to deploy ONAP components is the deployment specification, yet another yaml file. Within these deployment specs -are a number of parameters as shown in the following mariadb example: +are a number of parameters as shown in the following example: .. code-block:: yaml - apiVersion: extensions/v1beta1 - kind: Deployment + apiVersion: apps/v1 + kind: StatefulSet metadata: - name: mariadb + labels: + app.kubernetes.io/name: zookeeper + helm.sh/chart: zookeeper + app.kubernetes.io/component: server + app.kubernetes.io/managed-by: Tiller + app.kubernetes.io/instance: onap-oof + name: onap-oof-zookeeper + namespace: onap spec: - <...> + <...> + replicas: 3 + selector: + matchLabels: + app.kubernetes.io/name: zookeeper + app.kubernetes.io/component: server + app.kubernetes.io/instance: onap-oof + serviceName: onap-oof-zookeeper-headless template: - <...> + metadata: + labels: + app.kubernetes.io/name: zookeeper + helm.sh/chart: zookeeper + app.kubernetes.io/component: server + app.kubernetes.io/managed-by: Tiller + app.kubernetes.io/instance: onap-oof spec: - hostname: mariadb + <...> + affinity: containers: - - args: - image: nexus3.onap.org:10001/mariadb:10.1.11 - name: "mariadb" - env: - - name: MYSQL_ROOT_PASSWORD - value: password - - name: MARIADB_MAJOR - value: "10.1" + - name: zookeeper + <...> + image: gcr.io/google_samples/k8szk:v3 + imagePullPolicy: Always <...> - imagePullSecrets: - - name: onap-docker-registry-key - -Note that within the deployment specification, one of the container arguments -is the key/value pair image: nexus3.onap.org:10001/mariadb:10.1.11 which -specifies the version of the mariadb software to deploy. Although the -deployment specifications greatly simplify deployment, maintenance of the -deployment specifications themselves become problematic as software versions + ports: + - containerPort: 2181 + name: client + protocol: TCP + - containerPort: 3888 + name: election + protocol: TCP + - containerPort: 2888 + name: server + protocol: TCP + <...> + +Note that within the statefulset specification, one of the container arguments +is the key/value pair image: gcr.io/google_samples/k8szk:v3 which +specifies the version of the zookeeper software to deploy. Although the +statefulset specifications greatly simplify statefulset, maintenance of the +statefulset specifications themselves become problematic as software versions change over time or as different versions are required for different -deployments. For example, if the R&D team needs to deploy a newer version of +statefulsets. For example, if the R&D team needs to deploy a newer version of mariadb than what is currently used in the production environment, they would -need to clone the deployment specification and change this value. Fortunately, +need to clone the statefulset specification and change this value. Fortunately, this problem has been solved with the templating capabilities of Helm. -The following example shows how the deployment specifications are modified to +The following example shows how the statefulset specifications are modified to incorporate Helm templates such that key/value pairs can be defined outside of -the deployment specifications and passed during instantiation of the component. +the statefulset specifications and passed during instantiation of the component. .. code-block:: yaml - apiVersion: extensions/v1beta1 - kind: Deployment + apiVersion: apps/v1 + kind: StatefulSet metadata: - name: mariadb - namespace: "{{ .Values.nsPrefix }}-mso" + name: {{ include "common.fullname" . }} + namespace: {{ include "common.namespace" . }} + labels: {{- include "common.labels" . | nindent 4 }} spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: {{- include "common.matchLabels" . | nindent 6 }} + # serviceName is only needed for StatefulSet + # put the postfix part only if you have add a postfix on the service name + serviceName: {{ include "common.servicename" . }}-{{ .Values.service.postfix }} <...> template: - <...> + metadata: + labels: {{- include "common.labels" . | nindent 8 }} + annotations: {{- include "common.tplValue" (dict "value" .Values.podAnnotations "context" $) | nindent 8 }} + name: {{ include "common.name" . }} spec: - hostname: mariadb - containers: - - args: - image: {{ .Values.image.mariadb }} - imagePullPolicy: {{ .Values.pullPolicy }} - name: "mariadb" - env: - - name: MYSQL_ROOT_PASSWORD - value: password - - name: MARIADB_MAJOR - value: "10.1" <...> - imagePullSecrets: - - name: "{{ .Values.nsPrefix }}-docker-registry-key"apiVersion: extensions/v1beta1 - kind: Deployment - metadata: - name: mariadb - namespace: "{{ .Values.nsPrefix }}-mso" - spec: - <...> - template: - <...> - spec: - hostname: mariadb containers: - - args: - image: {{ .Values.image.mariadb }} - imagePullPolicy: {{ .Values.pullPolicy }} - name: "mariadb" - env: - - name: MYSQL_ROOT_PASSWORD - value: password - - name: MARIADB_MAJOR - value: "10.1" - <...> - imagePullSecrets: - - name: "{{ .Values.nsPrefix }}-docker-registry-key" - -This version of the deployment specification has gone through the process of -templating values that are likely to change between deployments. Note that the -image is now specified as: image: {{ .Values.image.mariadb }} instead of a -string used previously. During the deployment phase, Helm (actually the Helm + - name: {{ include "common.name" . }} + image: {{ .Values.image }} + imagePullPolicy: {{ .Values.global.pullPolicy | default .Values.pullPolicy }} + ports: + {{- range $index, $port := .Values.service.ports }} + - containerPort: {{ $port.port }} + name: {{ $port.name }} + {{- end }} + {{- range $index, $port := .Values.service.headlessPorts }} + - containerPort: {{ $port.port }} + name: {{ $port.name }} + {{- end }} + <...> + +This version of the statefulset specification has gone through the process of +templating values that are likely to change between statefulsets. Note that the +image is now specified as: image: {{ .Values.image }} instead of a +string used previously. During the statefulset phase, Helm (actually the Helm sub-component Tiller) substitutes the {{ .. }} entries with a variable defined in a values.yaml file. The content of this file is as follows: .. code-block:: yaml - nsPrefix: onap - pullPolicy: IfNotPresent - image: - readiness: oomk8s/readiness-check:2.0.0 - mso: nexus3.onap.org:10001/openecomp/mso:1.0-STAGING-latest - mariadb: nexus3.onap.org:10001/mariadb:10.1.11 + <...> + image: gcr.io/google_samples/k8szk:v3 + replicaCount: 3 + <...> + -Within the values.yaml file there is an image section with the key/value pair -mariadb: nexus3.onap.org:10001/mariadb:10.1.11 which is the same value used in +Within the values.yaml file there is an image key with the value +`gcr.io/google_samples/k8szk:v3` which is the same value used in the non-templated version. Once all of the substitutions are complete, the -resulting deployment specification ready to be used by Kubernetes. - -Also note that in this example, the namespace key/value pair is specified in -the values.yaml file. This key/value pair will be global across the entire -ONAP deployment and is therefore a prime example of where configuration -hierarchy can be very useful. +resulting statefulset specification ready to be used by Kubernetes. -When creating a deployment template consider the use of default values if -appropriate. Helm templating has built in support for DEFAULT values, here is +When creating a template consider the use of default values if appropriate. +Helm templating has built in support for DEFAULT values, here is an example: .. code-block:: yaml @@ -394,6 +366,227 @@ Helm template language is a superset of the Go template language). These functions include simple string operations like upper and more complex flow control operations like if/else. +OOM is mainly helm templating. In order to have consistent deployment of the +different components of ONAP, some rules must be followed. + +Templates are provided in order to create Kubernetes resources (Secrets, +Ingress, Services, ...) or part of Kubernetes resources (names, labels, +resources requests and limits, ...). + +Service template +---------------- + +In order to create a Service for a component, you have to create a file (with +`service` in the name. +For normal service, just put the following line: + +.. code-block:: yaml + + {{ include "common.service" . }} + +For headless service, the line to put is the following: + +.. code-block:: yaml + + {{ include "common.headlessService" . }} + +The configuration of the service is done in component `values.yaml`: + +.. code-block:: yaml + + service: + name: NAME-OF-THE-SERVICE + postfix: MY-POSTFIX + type: NodePort + annotations: + someAnnotationsKey: value + ports: + - name: tcp-MyPort + port: 5432 + nodePort: 88 + - name: http-api + port: 8080 + nodePort: 89 + - name: https-api + port: 9443 + nodePort: 90 + +`annotations` and `postfix` keys are optional. +if `service.type` is `NodePort`, then you have to give `nodePort` value for your +service ports (which is the end of the computed nodePort, see example). + +It would render the following Service Resource (for a component named +`name-of-my-component`, with version `x.y.z`, helm deployment name +`my-deployment` and `global.nodePortPrefix` `302`): + +.. code-block:: yaml + + apiVersion: v1 + kind: Service + metadata: + annotations: + someAnnotationsKey: value + name: NAME-OF-THE-SERVICE-MY-POSTFIX + labels: + app.kubernetes.io/name: name-of-my-component + helm.sh/chart: name-of-my-component-x.y.z + app.kubernetes.io/instance: my-deployment-name-of-my-component + app.kubernetes.io/managed-by: Tiller + spec: + ports: + - port: 5432 + targetPort: tcp-MyPort + nodePort: 30288 + - port: 8080 + targetPort: http-api + nodePort: 30289 + - port: 9443 + targetPort: https-api + nodePort: 30290 + selector: + app.kubernetes.io/name: name-of-my-component + app.kubernetes.io/instance: my-deployment-name-of-my-component + type: NodePort + +In the deployment or statefulSet file, you needs to set the good labels in order +for the service to match the pods. + +here's an example to be sure it matchs (for a statefulSet): + +.. code-block:: yaml + + apiVersion: apps/v1 + kind: StatefulSet + metadata: + name: {{ include "common.fullname" . }} + namespace: {{ include "common.namespace" . }} + labels: {{- include "common.labels" . | nindent 4 }} + spec: + selector: + matchLabels: {{- include "common.matchLabels" . | nindent 6 }} + # serviceName is only needed for StatefulSet + # put the postfix part only if you have add a postfix on the service name + serviceName: {{ include "common.servicename" . }}-{{ .Values.service.postfix }} + <...> + template: + metadata: + labels: {{- include "common.labels" . | nindent 8 }} + annotations: {{- include "common.tplValue" (dict "value" .Values.podAnnotations "context" $) | nindent 8 }} + name: {{ include "common.name" . }} + spec: + <...> + containers: + - name: {{ include "common.name" . }} + ports: + {{- range $index, $port := .Values.service.ports }} + - containerPort: {{ $port.port }} + name: {{ $port.name }} + {{- end }} + {{- range $index, $port := .Values.service.headlessPorts }} + - containerPort: {{ $port.port }} + name: {{ $port.name }} + {{- end }} + <...> + +The configuration of the service is done in component `values.yaml`: + +.. code-block:: yaml + + service: + name: NAME-OF-THE-SERVICE + headless: + postfix: NONE + annotations: + anotherAnnotationsKey : value + publishNotReadyAddresses: true + headlessPorts: + - name: tcp-MyPort + port: 5432 + - name: http-api + port: 8080 + - name: https-api + port: 9443 + +`headless.annotations`, `headless.postfix` and +`headless.publishNotReadyAddresses` keys are optional. + +If `headless.postfix` is not set, then we'll add `-headless` at the end of the +service name. + +If it set to `NONE`, there will be not postfix. + +And if set to something, it will add `-something` at the end of the service +name. + +It would render the following Service Resource (for a component named +`name-of-my-component`, with version `x.y.z`, helm deployment name +`my-deployment` and `global.nodePortPrefix` `302`): + +.. code-block:: yaml + + apiVersion: v1 + kind: Service + metadata: + annotations: + anotherAnnotationsKey: value + name: NAME-OF-THE-SERVICE + labels: + app.kubernetes.io/name: name-of-my-component + helm.sh/chart: name-of-my-component-x.y.z + app.kubernetes.io/instance: my-deployment-name-of-my-component + app.kubernetes.io/managed-by: Tiller + spec: + clusterIP: None + ports: + - port: 5432 + targetPort: tcp-MyPort + nodePort: 30288 + - port: 8080 + targetPort: http-api + nodePort: 30289 + - port: 9443 + targetPort: https-api + nodePort: 30290 + publishNotReadyAddresses: true + selector: + app.kubernetes.io/name: name-of-my-component + app.kubernetes.io/instance: my-deployment-name-of-my-component + type: ClusterIP + +Previous example of StatefulSet would also match (except for the `postfix` part +obviously). + +Creating Deployment or StatefulSet +---------------------------------- + +Deployment and StatefulSet should use the `apps/v1` (which has appeared in +v1.9). +As seen on the service part, the following parts are mandatory: + +.. code-block:: yaml + + apiVersion: apps/v1 + kind: StatefulSet + metadata: + name: {{ include "common.fullname" . }} + namespace: {{ include "common.namespace" . }} + labels: {{- include "common.labels" . | nindent 4 }} + spec: + selector: + matchLabels: {{- include "common.matchLabels" . | nindent 6 }} + # serviceName is only needed for StatefulSet + # put the postfix part only if you have add a postfix on the service name + serviceName: {{ include "common.servicename" . }}-{{ .Values.service.postfix }} + <...> + template: + metadata: + labels: {{- include "common.labels" . | nindent 8 }} + annotations: {{- include "common.tplValue" (dict "value" .Values.podAnnotations "context" $) | nindent 8 }} + name: {{ include "common.name" . }} + spec: + <...> + containers: + - name: {{ include "common.name" . }} ONAP Application Configuration ------------------------------ @@ -423,18 +616,16 @@ SO deployment specification excerpt: .. code-block:: yaml - apiVersion: extensions/v1beta1 + apiVersion: apps/v1 kind: Deployment metadata: - name: {{ include "common.name" . }} + name: {{ include "common.fullname" . }} namespace: {{ include "common.namespace" . }} - labels: - app: {{ include "common.name" . }} - chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }} - release: {{ .Release.Name }} - heritage: {{ .Release.Service }} + labels: {{- include "common.labels" . | nindent 4 }} spec: replicas: {{ .Values.replicaCount }} + selector: + matchLabels: {{- include "common.matchLabels" . | nindent 6 }} template: metadata: labels: |