.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 .. Copyright 2019 Samsung Electronics Co., Ltd. .. _oooi_installguide: OOM ONAP Offline Installer - Installation Guide =============================================== This document describes the correct offline installation procedure for `OOM ONAP`_, which is done by the ansible based `offline-installer `_. Before you dive into the installation you should prepare the offline installer itself - the installer consists of at least two packages/resources. You can read about it in the `Build Guide`_, which provides the instructions for creating them. This current version of the *Installation Guide* supports `El Alto release`_. ----- .. _oooi_installguide_preparations: Part 1. Prerequisites --------------------- OOM ONAP deployment has certain hardware resource requirements - `El Alto requirements`_: Community recommended footprint from `El Alto requirements`_ page is 16 VMs ``224 GB RAM`` and ``112 vCPUs``. We will not follow strictly this setup due to such demanding resource consumption and so we will deploy our installation across four nodes (VMs) instead of sixteen. Our simplified setup is definitively not supported or recommended - you are free to diverge - you can follow the official guidelines or make completely different layout, but the minimal count of nodes should not drop below three - otherwise you may have to do some tweaking to make it work, which is not covered here (there is a pod count limit for a single kubernetes node - you can read more about it in this `discussion `_). .. _oooi_installguide_preparations_k8s_cluster: Kubernetes cluster ~~~~~~~~~~~~~~~~~~ The four nodes/VMs will be running these services: - **infra-node**:: - nexus - nginx proxy - dns - kubernetes-etcd - kubernetes-control-plane **NOTE:** kubernetes-* roles can be collocated directly with kubernetes nodes and not necessarily on infra node. - **kubernetes node 1-3**:: - kubernetes worker You don't need to care about these services now - that is the responsibility of the installer (described below). Just start four VMs as seen in this table (or according to your needs as we hinted above): .. _Overview table of the kubernetes cluster: Kubernetes cluster overview ^^^^^^^^^^^^^^^^^^^^^^^^^^^ In El Alto we are using RKE as k8s orchestrator method, however everyone is free to diverge from this example and can set it up in own way omitting our rke playbook execution. =================== ================== ==================== ============== ============ =============== KUBERNETES NODE OS NETWORK CPU RAM STORAGE =================== ================== ==================== ============== ============ =============== **infra-node** RHEL/CentOS 7.6 ``10.8.8.100/24`` ``8 vCPUs`` ``8 GB`` ``100 GB`` **kube-node1** RHEL/CentOS 7.6 ``10.8.8.101/24`` ``16 vCPUs`` ``56+ GB`` ``100 GB`` **kube-node2** RHEL/CentOS 7.6 ``10.8.8.102/24`` ``16 vCPUs`` ``56+ GB`` ``100 GB`` **kube-node3** RHEL/CentOS 7.6 ``10.8.8.103/24`` ``16 vCPUs`` ``56+ GB`` ``100 GB`` SUM ``56 vCPUs`` ``176+ GB`` ``400 GB`` =========================================================== ============== ============ =============== Unfortunately, the offline installer supports only **RHEL 7.x** or **CentOS 7.6** distribution as of now. So, your VMs should be preinstalled with this operating system - the hypervisor and platform can be of your choosing. We will expect from now on that you installed four VMs and they are connected to the shared network. All VMs must be reachable from our *install-server* (below), which can be the hypervisor, *infra-node* or completely different machine. But in either of these cases the *install-server* must be able to connect over ssh to all of these nodes. .. _oooi_installguide_preparations_installserver: Install-server ~~~~~~~~~~~~~~ We will use distinct *install-server* and keep it separate from the four-node cluster. But if you wish so, you can use *infra-node* for this goal (if you use the default ``'chroot'`` option of the installer), but in that case double the size of the storage requirement! Prerequisites for the *install-server*: - packages described in `Build Guide`_ - extra ``100 GB`` storage (to have space where to store these packages) - installed ``'chroot'`` and/or ``'docker'`` system commands - network connection to the nodes - especially functioning ssh client Our *install-server* will have ip: ``10.8.8.4``. **NOTE:** All the subsequent commands below, are executed from within this *install-server*. ----- .. _oooi_installguide_config: Part 2. Preparation and configuration ------------------------------------- We *MUST* do all the following instructions from the *install-server* and also we will be running them as a user ``root``. But that is not necessary - you can without any problem p
{
  "sdc-connections": {
    "sdc-controller1": {
      "user": "User1",
      "consumerGroup": "consumerGroup1",
      "consumerId": "consumerId1",
      "environmentName": "environmentName1",
      "sdcAddress": "localhost:${docker.http-cache.port.host}",
      "password": "enc:QpF2TcrdRSFADqDxH1HwDYdYUIbMxOj-TrGd6Vqvwzd",
      "pollingInterval": 10,
      "pollingTimeout": 30,
      "messageBusAddresses": [
        "localhost"
      ]
    },
    "sdc-controller2": {
      "user": "User2",
      "consumerGroup": "consumerGroup2",
      "consumerId": "consumerId2",
      "environmentName": "environmentName2",
      "sdcAddress": "localhost:${docker.http-cache.port.host}",
      "password": "enc:QpF2TcrdRSFADqDxH1HwDYdYUIbMxOj-TrGd6Vqvwzd",
      "pollingInterval": 10,
      "pollingTimeout": 30,
      "messageBusAddresses": [
        "localhost"
      ]
    }
  }
}
city we will showcase generating of a private key without any such protection:: $ ssh-keygen -N "" -f ~/.ssh/identity The next step will be to distribute the public key to these nodes and from that point no password is needed:: $ for ip in 100 101 102 103 ; do ssh-copy-id -i ~/.ssh/identity.pub root@10.8.8.${ip} ; done This command behaves almost identically to the ``'setup.yml'`` playbook. If you generated the ssh key manually then you can now run the ``'setup.yml'`` playbook like this and achieve the same result as in the first execution:: $ ./run_playbook.sh -i inventory/hosts.yml setup.yml This time it should not ask you for any password - of course this is very redundant, because you just distributed two ssh keys for no good reason. We can finally edit and finish the configuration of the ``'hosts.yml'``: - if you used the ``'setup.yml'`` playbook then you can just leave this line as it is:: ansible_ssh_private_key_file: /root/.ssh/offline_ssh_key - if you created a ssh key manually then change it like this:: ansible_ssh_private_key_file: /root/.ssh/identity ----- .. _oooi_installguide_install: Part 3. Installation -------------------- We should have the configuration complete and be ready to start the installation. The installation is done via ansible playbooks, which are run either inside a **chroot** environment (default) or from the **docker** container. If for some reason you want to run playbooks from the docker instead of chroot then you cannot use *infra-node* or any other *kube-node* as the *install-server* - otherwise you risk that installation will fail due to restarting of the docker service. If you built your ``'sw'`` package well then there should be the file ``'ansible_chroot.tgz'`` inside the ``'docker'`` directory. If not then you must create it - to learn how to do that and to get more info about the scripts dealing with docker and chroot, go to `Appendix 1. Ansible execution/bootstrap`_ We will use the default chroot option so we don't need any docker service to be running. Installation is actually very straightforward now:: $ ./run_playbook.sh -i inventory/hosts.yml -e @application/application_configuration.yml site.yml This will take a while so be patient. ``'site.yml'`` playbook actually runs in the order the following playbooks: - ``upload_resources.yml`` - ``infrastructure.yml`` - ``rke.yml`` - ``application.yml`` ---- .. _oooi_installguide_postinstall: Part 4. Post-installation and troubleshooting --------------------------------------------- After all of the playbooks are run successfully, it will still take a lot of time until all pods are up and running. You can monitor your newly created kubernetes cluster for example like this:: $ ssh -i ~/.ssh/offline_ssh_key root@10.8.8.100 # tailor this command to connect to your infra-node $ watch -d -n 5 'kubectl get pods --all-namespaces' Alternatively you can monitor progress with ``helm_deployment_status.py`` script located in offline-installer directory. Transfer it to infra-node and run:: $ python helm_deployment_status.py -n # namespace defaults to onap To automatically verify functionality with healthchecks after deployment becomes ready or after timeout period expires, append ``-hp`` switch followed by the full path to the healthcheck script and ``--health-mode`` optional switch with appropriate mode supported by that script (``health`` by default, ``--help`` displays available modes):: $ python helm_deployment_status.py -hp //helm_charts/robot/ete-k8s.sh --health-mode It is strongly recommended to tailor ``helm_deployment_status.py`` to your needs since default values might not be what you'd expect. The defaults can be displayed with ``--help`` switch. Final result of installation varies based on number of k8s nodes used and distribution of pods. In some dev envs we quite frequently hit problems with not all pods properly deployed. In successful deployments all jobs should be in successful state. This can be verified using :: $ kubectl get jobs -n If some of the job is hanging in some wrong end-state like ``'BackoffLimitExceeded'`` manual intervention is required to heal this and make also dependent jobs passing. More details about particular job state can be obtained using :: $ kubectl describe job -n If manual intervention is required, one can remove failing job and retry helm install command directly, which will not launch full deployment but rather check current state of the system and rebuild parts which are not up & running. Exact commands are as follows :: $ kubectl delete job -n $ helm deploy --namespace E.g. helm deploy dev local/onap --namespace onap Once all pods are properly deployed and in running state, one can verify functionality e.g. by running onap healthchecks :: $ cd //helm_charts/robot $ ./ete-k8s.sh onap health For better work with terminal screen and jq packages were added . It can be installed from resources directory. Screen is a terminal multiplexer. With screen it is possible to have more terminal instances active. Screen as well keeps active SSH connections even terminal is closed. Jq can be used for editing json data format as output of kubectl. For example jq was used to troubleshoot `SDNC-739 (UEB - Listener in Crashloopback) `_ :: $ kubectl -n onap get job onap-sdc-sdc-be-config-backend -o json | jq "del(.spec.selector)" | jq "del(.spec.template.metadata.labels)" | kubectl -n onap replace --force -f - ----- .. _oooi_installguide_appendix1: Appendix 1. Ansible execution/bootstrap --------------------------------------- There are two ways how to easily run the installer's ansible playbooks: - If you already have or can install a docker then you can build the provided ``'Dockerfile'`` for the ansible and run playbooks in the docker container. - Another way to deploy ansible is via chroot environment which is bundled together within this directory. (Re)build docker image and/or chroot archive ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Inside the ``'docker'`` directory is the ``'Dockerfile'`` and ``'build_ansible_image.sh'`` script. You can run ``'build_ansible_image.sh'`` script on some machine with the internet connectivity and it will download all required packages needed for building the ansible docker image and for exporting it into a flat chroot environment. Built image is exported into ``'ansible_chroot.tgz'`` archive in the same (``'docker'``) directory. This script has two optional arguments: #. ansible version #. docker image name **Note:** if optional arguments are not used, docker image name will be set to ``'ansible'`` by default. Launching ansible playbook using chroot environment ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is the default and preferred way of running ansible playbooks in an offline environment as there is no dependency on docker to be installed on the system. Chroot environment is already provided by included archive ``'ansible_chroot.tgz'``. It should be available in the ``'docker'`` directory as the end-result of the packaging script or after manual run of the ``'build_ansible_image.sh'`` script referenced above. All playbooks can be executed via ``'./run_playbook.sh'`` wrapper script. To get more info about the way how the ``'./run_playbook.sh'`` wrapper script should be used, run:: $ ./run_playbook.sh The main purpose of this wrapper script is to provide the ansible framework to a machine where it was bootstrapped without need of installing additional packages. The user can run this to display ``'ansible-playbook'`` command help:: $ ./run_playbook.sh --help Developers notes ~~~~~~~~~~~~~~~~ * There are two scripts which work in tandem for creating and running chroot * First one can convert docker image into chroot directory * Second script will automate chrooting (necessary steps for chroot to work and cleanup) * Both of them have help - just run:: $ cd docker $ ./create_docker_chroot.sh help $ ./run_chroot.sh help Example usage:: $ sudo su $ docker/create_docker_chroot.sh convert some_docker_image ./new_name_for_chroot $ cat ./new_name_for_chroot/README.md $ docker/run_chroot.sh execute ./new_name_for_chroot cat /etc/os-release 2>/dev/null Launching ansible playbook using docker container (ALTERNATIVE APPROACH) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This option is here just to keep support for the older method which relies on a running docker service. For the offline deployment use the chroot option as indicated above. You will not need ``'ansible_chroot.tgz'`` archive anymore, but the new requirement is a prebuilt docker image of ansible (based on the provided ``'Dockerfile'``). It should be available in your local docker repository (otherwise the default name ``'ansible'`` may fetch unwanted image from default registry!). To trigger this functionality and to run ``'ansible-playbook'`` inside a docker container instead of the chroot environment, you must first set the ``ANSIBLE_DOCKER_IMAGE`` variable. The value must be a name of the built ansible docker image. Usage is basically the same as with the default chroot way - the only difference is the existence of the environment variable:: $ ANSIBLE_DOCKER_IMAGE=ansible ./run_playbook.sh --help ----- .. _Build Guide: ./BuildGuide.rst .. _El Alto requirements: https://onap.readthedocs.io/en/elalto/guides/onap-developer/settingup/index.html#installing-onap .. _El Alto release: https://docs.onap.org/en/elalto/release/ .. _OOM ONAP: https://wiki.onap.org/display/DW/ONAP+Operations+Manager+Project