From 9643b0c11bdafd26ea0ac5127325aa8cb09f0c03 Mon Sep 17 00:00:00 2001 From: mrichomme Date: Sat, 14 Nov 2020 22:36:57 +0100 Subject: Refactor Integration official documentation Issue-ID: INT-1736 Signed-off-by: mrichomme Change-Id: Ia7b6425358eb9b07e293881dabd5345697af1c39 --- docs/integration-tests.rst | 149 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 149 insertions(+) create mode 100644 docs/integration-tests.rst (limited to 'docs/integration-tests.rst') diff --git a/docs/integration-tests.rst b/docs/integration-tests.rst new file mode 100644 index 000000000..be3a04c7b --- /dev/null +++ b/docs/integration-tests.rst @@ -0,0 +1,149 @@ +.. This work is licensed under a + Creative Commons Attribution 4.0 International License. +.. integration-tests: + +Tests +===== + +.. important:: + Integration is in charged of several types of tests: + + - Use Cases: developped by use case teams, usually complex, demonstrating high value capabilities ofr ONAP. They may be partially automated and even + integrated in CD. + - CSIT Tests: functional tests created by the projects, partially hosted in CSIT repository + - Automatic Test Cases: these use cases are usually more simple and aims to validate that ONAP is working properly. + These tests have been developed to validate ONAP as a software solution. + In theory all the main function shall be covered by such tests in order to have more robust CI/CD and then avoid regressions. + These tests are usually developed and maintained by the integration team. + +We may also indicate that when the development of the test framework python-onapsk +follows standard developement quality rules and imposes the creation of +unit/functional/integration tests. +As an example python-onapsdk requires a unit test coverage of 98% before merging +a new feature, which is far above the project criteria in SonarCLoud today. + +Use Cases +--------- + +The use cases are described in :ref:`Verified Use cases `. + +CSIT Tests +---------- + +The CSIT tests are functional tests executed by the projects on mocked +environement to validate their components. +Historically it was hosted in a CSIT repository. + +Integration team invited the projects to bring back such tests bacdk to home +repository for 2 main reasons: + +- integration cannot be a bottleneck: +2/merge from integration needed for each + project +- most of the tests are abandonned and not maintained when hosted in a third party + repository leading to CI/CD time waste and misleading test reporting + +In Guilin a PoC to help the project to re-insource their functional tests have +been initiated. +See `CSIT wiki page `__ +for details. + +Automatic Tests +--------------- + +These tests are run daily/weekly on each new gate (new patchset in OOM, clamp +or SO). They can be in any language (bash, go, python,...), leveraging any test +framework (robotframework, MTS, python-onapsdk). +They are all embedded in `xtesting `__ dockers. + +.. hint:: + Automatic tests are currently divided in 4 different categories: + + - infrastructure-healthcheck: tests from OOM checking the ONAP namespace, certificates,.. + - healthcheck: basic tests on components + - smoke tests: end to end tests + - security tests + +A dashboard summarizing the status and providing the links to the test result +page or the logs is automatically created at the end of the execution of the +tests. + +.. figure:: files/tests/test-dashboard.png + +All the pages and artifacts are pushed to LF backend: + +- Daily chaines: https://logs.onap.org/onap-integration/daily +- Weekly chains: https://logs.onap.org/onap-integration/weekly +- Gating chains: https://logs.onap.org/onap-integration/gating + + +Infrastructure Healthcheck Tests +................................ + +.. csv-table:: Infrastructure Healthcheck Tests + :file: ./files/csv/tests-infrastructure-healthcheck.csv + :widths: 20,40,20,20 + :delim: ; + :header-rows: 1 + +See `Infrastructure Healthcheck README `__ +to adapt then run infrastructure healthcheck tests on your own system. + +Please note that the onap-k8s is run 2 times in CD chains. It is run just after +the installation (onap-k8s) and at the end of the test execution (onap-k8s-teardown) +in order to collect the logs of the different components during the test execution. + +.. figure:: files/tests/test-onap-k8s.png + +Healhcheck Tests +................ + +.. csv-table:: Healthcheck Tests + :file: ./files/csv/tests-healthcheck.csv + :widths: 20,40,20,20 + :delim: ; + :header-rows: 1 + +See `Healthcheck README `__ +to adapt then run healthcheck tests on your own system. + +Smoke Tests +........... + +.. csv-table:: Smoke Tests + :file: ./files/csv/tests-smoke.csv + :widths: 20,40,20,20 + :delim: ; + :header-rows: 1 + +See `Python smoke test README `__ +to adapt and run robot based smoke tests. +An html page is generated by the pythonsdk-test tests. + +.. figure:: files/tests/test-basic-cnf.png + + +See `Robot smoke test README `__ +to adapt and run pythonsdk based smoke tests. +Standard Robot html pages are generated. See :ref:`Robot page `. + +Security Tests +............... + +.. csv-table:: Security Tests + :file: ./files/csv/tests-security.csv + :widths: 20,40,20,20 + :delim: ; + :header-rows: 1 + +See `Security test README `__ +to adapt then run the security tests on your own system. + +Note for security tests, integration team follows `SECCOM recommendations and +apply waivers granted by SECCOM if needed through xfail lists `__. + +Stability Testing +----------------- + +Ensuring the stability of ONAP is one of the missions of the Integration team. +CI chains and stability tests are performed to help stabilising the release. +See :ref:`Integration stability tests ` for details. -- cgit 1.2.3-korg