summaryrefslogtreecommitdiffstats
path: root/docs/TestingGuide.rst
diff options
context:
space:
mode:
authorSamuli Silvius <s.silvius@partner.samsung.com>2019-04-04 10:55:12 +0000
committerGerrit Code Review <gerrit@onap.org>2019-04-04 10:55:12 +0000
commitd425e37b6bf8c69511d4f726660ce2d4e6d5e77c (patch)
tree92029a29714cbb9fd1183d49860b40aa122ab70d /docs/TestingGuide.rst
parent05b9001fa01b3f1076a5a21f063ca40421a66333 (diff)
parent542bdfd8ae6e5290cfb1feabbfbf5412041941ee (diff)
Merge "Document molecule testing"
Diffstat (limited to 'docs/TestingGuide.rst')
-rw-r--r--docs/TestingGuide.rst365
1 files changed, 365 insertions, 0 deletions
diff --git a/docs/TestingGuide.rst b/docs/TestingGuide.rst
new file mode 100644
index 00000000..caf3d927
--- /dev/null
+++ b/docs/TestingGuide.rst
@@ -0,0 +1,365 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. Copyright 2019 Samsung Electronics Co., Ltd.
+
+OOM ONAP Offline Installer Testing Guide
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This testing guide describes how offline installer can be tested in local
+development environment (laptop) without the need for actual servers.
+
+Documentation refers to files/directories in ``ansible`` directory of this repository.
+
+Introduction
+============
+
+Offline installer uses Molecule_ for testing all roles.
+
+Molecule is tool for ansible roles development and testing. In this project
+Molecule is used for integration type of testing for both roles and playbooks.
+Role code is tested against simulated host.
+
+Molecule is designed to test single Ansible_ role in isolation. Offline installer however
+has many small roles that are dependent on each other and also execution order for roles
+is meaningful. In that respect Molecule's design does not offer sufficient level
+of testing as it's lacking playbook level of scenario testing by default.
+Luckily Molecule is highly configurable and it is possible to achieve a higher level of
+testing scenarios for the offline installer.
+
+Testing with Molecule is divided to two levels of testing:
+ 1) role level testing (as per Molecule design)
+ 2) playbook level testing (offline installer own setup)
+
+Purpose
+=======
+
+The purpose of using testing framework like Molecule is to make possible for developer to
+verify ansible code changes locally in own laptop without the need for big resources.
+
+Developer is also expected to do development of the Ansible code and the Molecule test
+code at the same time.
+Offline installer does not have unittest level of testing for the ansible code.
+
+Any commit made to ansible code base needs to first pass Molecule tests before
+it's merged.
+
+Test levels
+===========
+
+To cover both testing levels (role and playbook) with maximum benefit and minimum
+copy-pasting, the testing code should be written in reusable way.
+
+Reusable test code can be achieved by writing all prepare/cleanup and other
+helping code as a roles into main test directory.
+Also testinfra_ test code can be shared between different roles and between different scenarios
+of one role.
+
+Testing of role and one scenario (one execution run of molecule) is fully
+defined by **molecule.yml** file.
+
+molecule.yml file is always located in directory:
+
+ <tested-role>/molecule/<scenario>/molecule.yml
+
+i.e. one role can have multiple scenarios (different configuration, OS etc. whatever user wants)
+to execute tests for same role. Each scenario has own molecule.yml file and own testinfra
+tests.
+
+Molecule.yml file is the only file that cannot be re-used (except with symbolic links) but
+all other resources can be reused by referencing those in molecule.yml file or/and indirectly
+from resources molecule.yml is pointing to.
+
+**tested-role** is clear in case of normal role level testing, but in playbook level testing the
+tested-role is just an invented role name and directory with molecule directory inside but no
+actual ansible role code.
+
+Role level testing
+------------------
+
+The target is to test single role in isolation just like Molecule is designed.
+Role level testing is supposed to cover:
+
+- Syntax checking (Yamllint_, `Ansible lint`_, flake8_)
+- Ansible code testing
+- Idempotence testing
+- Verifying role results from target hosts (testinfra tests)
+
+Ansible code testing can/should also cover all different options how this role
+can be run (`scenario <https://molecule.readthedocs.io/en/latest/configuration.html#root-scenario>`_).
+Different molecule runs can be implemented as own scenarios (in addition to default scenario)
+or default scenario playbook can be extended to run role tests multiple times just adjusting
+configuration between.
+
+Example with nexus role
+::
+
+ ├── infrastructure.yml
+ ├── roles
+ │   ├── nexus
+ │   │   ├── defaults
+ │   │   ├── files
+ │   │   ├── molecule
+ │   │   │   └── default
+ │   │   │   ├── molecule.yml
+ │   │   │   ├── playbook.yml
+ │   │   │   ├── prepare.yml
+ │   │   │   └── tests
+ │   │   ├── tasks
+ │   │   └── vars
+
+Playbook level testing
+----------------------
+
+Playbook level testing is this project's (offline installer) own
+setup and way of using Molecule. The target is to raise testing level
+from single role testing up to single playbook testing.
+
+Playbook level testing can be used also to run multiple playbooks and/or
+playbooks multiple times with different configuration.
+
+The aim is to verify multiple roles working together i.e. higher level of
+integration testing.
+
+Practically the **tested-role** is just a wrapper directory to conform
+molecule required directory structure and provide a name for the test.
+Directory itself does not contain any ansible role code, but just
+molecule files configured to run multiple other roles.
+
+Playbook level test directories should be named consistently according to
+tested playbook and prefix string ``play`` and with optional description
+if there are multiple scenarios for single playbook:
+
+ play-<playbookname>[-<description>]
+
+E.g.
+
+- ``play-infrastructure``
+- ``play-resources``
+
+As role's are tested with own molecule tests in isolation, playbook level tests
+should focus to integration of the roles and should avoid of repeating same tests
+as done already for individual roles.
+
+Playbook level testing is supposed to cover:
+ - Ansible code testing
+
+Basically it's easier to highlight what is supposed to be **avoided** in playbook level
+testing for the reason not to repeat the same that is done already in role level testing.
+
+- Syntax checking is left out already by default as molecule does linting only for the
+ role code where molecule is run, and in this case tested-role is empty.
+
+- Idempotence can be tested, but should be disabled (by default) in molecule.yml because
+ it takes too much time and was tested already for individual roles.
+
+- Verifying target hosts with testinfra tests can be done but then something else
+ should be tested as in role based tests. And if those 2 would overlap it's better
+ to leave them out.
+
+Example with infrastructure playbook level test files
+::
+
+ ├── infrastructure.yml
+ └── test
+ ├── play-infrastructure
+ │   └── molecule
+ │   └── default
+ │   ├── molecule.yml
+ │   ├── playbook.yml
+ │   ├── prepare.yml
+ │   └── tests
+
+Test code reuse and naming
+===========================
+
+As both testing levels test the same Ansible roles, there are a need
+to share common code for both of them.
+
+Testinfra_ Python code should be shared when also playbook level
+tests verify target hosts. However sharing is not limited only for the 2 test levels
+but also between different roles.
+
+Individual role have testinfra tests on directory:
+
+ roles/<role>/molecule/<scenario>/tests
+
+and any commonly usable testinfra Python code should be placed to directory:
+
+ test/testinfra
+
+Ansible role testing uses several resources defined by provisioner section of
+molecule.yml
+https://molecule.readthedocs.io/en/latest/configuration.html#provisioner
+
+Most common resources that are written for role testing are:
+
+- playbook.yml (mandatory but can include specific code)
+- prepare.yml
+- cleanup.yml
+- create.yml
+- destroy.yml
+
+all of which can be just placed to scenario directory together with playbook.yml
+(without editing molecule.yml when in default directory) and all of which can
+include ansible code to do something e.g. prepare role for testing.
+
+Example molecule files:
+
+Role level tests for nexus role:
+ - roles/nexus/molecule/default/molecule.yml
+ - roles/nexus/molecule/default/playbook.yml
+ - roles/nexus/molecule/default/prepare.yml
+playbook level tests for infrastructure playbook:
+ - test/play-infrastructure/molecule/default/molecule.yml
+ - test/play-infrastructure/molecule/default/playbook.yml
+ - test/play-infrastructure/molecule/default/prepare.yml
+
+Sharing all test code should be done by writing them in the form of ansible
+roles and placing commonly usable roles into:
+
+ test/roles/<testrole>
+
+Test roles should be named consistently according to action it's needed and
+role for it's for together with optional description:
+
+ <action>-<role>[-<description>]
+
+Examples of commonly used test roles
+::
+
+ ├── infrastructure.yml
+ └── test
+ ├── play-infrastructure
+ └── roles
+ ├── post-certificates
+ ├── prepare-common
+ ├── prepare-dns
+ ├── prepare-docker
+ ├── prepare-nexus
+ └── prepare-nginx
+
+Molecule platform images
+========================
+
+Molecule can build images of the tested hosts on the fly with default
+Dockerfile template (docker driver) or from a Dockerfile provided by user.
+In case of Vagrant driver used box image can be also fully customized by user.
+
+To speed up testing it's preferred to pre-build needed images to be usable in
+local docker repository in case of docker driver or Vagrant image cache in case
+of Vagrant driver.
+
+Used Dockerfiles/Box definitions are kept in following directory structure
+::
+
+ └── test
+ └── images
+ ├── docker
+ │   ├── build-all.sh
+ │   ├── centos7
+ │   │   ├── build.sh
+ │   │   ├── dbus.service
+ │   │   └── Dockerfile
+ │   └── ubuntu
+ │   ├── build.sh
+ │   └── Dockerfile
+ └── vagrant
+
+Build images
+------------
+
+Build all platforms images before running Molecule tests. Building can be done
+with the following single command:
+
+ test/images/docker/build-all.sh
+
+Install
+=======
+
+Molecule can be installed in multiple ways and in this guide 2 different ways is
+covered.
+
+- Install Molecule with pip in virtual environment
+- Use Molecule provided docker container to run Molecule
+
+Install with pip
+----------------
+
+This is a OS dependent and some prerequisites needs to be installed, but after
+prerequisites are installed installing Molecule can be done by calling following
+script:
+
+ source test/bin/install-molecule.sh
+
+As for the required OS packages, see example for Ubuntu in the install-molecule.sh
+script's comments or from Molecule_ pages.
+
+Note that sourcing the script is not needed to get Molecule installed but it leaves
+you already into virtual environment and ready to run Molecule.
+
+To get out from virtual environment issue:
+
+ deactivate
+
+And next time to activate virtual environment again before running Molecule, issue:
+
+ source ~/molecule_venv/bin/activate
+
+And here the directory ``~/molecule_venv`` is just the default virtual environment
+path that install-molecule.sh script is using and can be overridden with
+``VENV_PATH`` environment variable.
+
+Use Molecule docker container
+-----------------------------
+
+Molecule provides docker containers images via quay.io_ where Molecule, Ansible
+and all needed dependencies are build to the image.
+
+In this way of using Molecule, no installation is needed and only docker is the
+prerequisite for running Molecule.
+
+For using provided image to test offline-installer roles, following scripts are
+provided:
+
+Build container image:
+ ``test/molecule-docker/build.sh``
+
+This will build image named ``molecule-dev`` with strict version tag.
+
+Set molecule into the PATH:
+ ``source test/bin/set_molecule_paths.sh``
+
+That will add the actual Molecule run wrapper script test/bin/molecule.sh to path
+usable from everywhere similarly than molecule with pip and virtual environment.
+
+Run Molecule wrapper script:
+ ``test/bin/molecule.sh``
+
+For running Molecule. Using ``molecule-dev`` image and the exact version defined by
+test/docker/build.sh script.
+
+Usage
+=====
+
+Basic usage of molecule tests. See more detailed instructions from Molecule_
+
+Run complete testing for a role or a playbook:
+
+1. cd roles/<role> or cd test/play-<playbook-name>
+2. molecule test
+
+Develop a role code and run testing during the coding:
+
+1. cd roles/<role>
+2. Edit ansible code and molecule test code when needed
+3. molecule converge
+4. Repeat steps 2 and 3 until code is ready and molecule tests are passing
+5. molecule test
+
+.. _Molecule: https://molecule.readthedocs.io
+.. _quay.io: https://quay.io/repository/ansible/molecule
+.. _Testinfra: https://testinfra.readthedocs.io
+.. _Flake8: http://flake8.pycqa.org
+.. _Yamllint: https://github.com/adrienverge/yamllint
+.. _Ansible Lint: https://github.com/ansible/ansible-lint
+.. _Ansible: https://www.ansible.com/