diff options
| -rw-r--r-- | docs/contributing.rst | 215 | ||||
| -rw-r--r-- | docs/index.rst | 2 | ||||
| -rw-r--r-- | docs/release-notes.rst | 74 |
3 files changed, 260 insertions, 31 deletions
diff --git a/docs/contributing.rst b/docs/contributing.rst index 2643f7c..8b561af 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -2,43 +2,107 @@ .. http://creativecommons.org/licenses/by/4.0 .. Copyright 2019 AT&T Intellectual Property. All rights reserved. -Contributing -############ +How to Contribute +################# + +Overview +~~~~~~~~ + +This section will provide details on how to contribute changes to the project +covering both the mechanics of how to contribute new code as well as +how to adhere to the code quality and coding practices of the project. + + +Prerequisites +~~~~~~~~~~~~~ + +As this project is part of the overall ONAP project, there are some common +guidelines and activities you will need to adhere to: + +- Acquire a `Linux Foundation ID <https://identity.linuxfoundation.org/>`__ +- Configure your `ONAP Gerrit Account <https://wiki.onap.org/display/DW/Configuring+Gerrit>`__ + as it is used for all code submissions and reviews +- Install VVP per the :ref:`installation instructions <vvp-installation>` +- Most features will require some knowledge of both `Openstack Heat <https://wiki.openstack.org/wiki/Heat>`__ + and the `ONAP Heat requirements <https://onap.readthedocs.io/en/latest/submodules/vnfrqts/requirements.git/docs/Chapter5/Heat/index.html>`__ +- Knowledge of writing tests in `pytest <https://pytest.readthedocs.io/>`__ + +Other useful links: + +- All work is documented and tracked via the `VVP Project <https://jira.onap.org/projects/VVP/issues/>`__ + in the `ONAP JIRA instance <https://jira.onap.org/>`__. Login is via your + Linux Foundation ID +- Proposals for new features, general information about the projects, + meeting minutes, and ONAP process information is located on the + `ONAP Wiki <https://wiki.onap.org/>`__ +- The VVP project hosts a weekly meeting to plan upcoming work, discuss open + issues, and align on priorities. Please consider attending if possible if you + intend to contribute to the project. Refer to the ONAP Calendar `<https://wiki.onap.org/pages/viewpage.action?pageId=6587439>`__ + for scheduling details Objective ~~~~~~~~~ -**The objective for the VVP test suite is for each -test to directly correlate with at least one requirement in the** -`VNF Requirements <https://onap.readthedocs.io/en/latest/submodules/vnfrqts/requirements.git/docs/index.html>`__ -**project in ONAP. If the test you intend to write doesn't -have a corresponding requirement in the VNF Requirements project, consider -making a contribution to that project first.** +The primary focus of VVP is ensuring that a VNF that is described using Openstack +Heat complies with the ONAP Heat requirements specified in the `VNF Requirements (VNFRQTS) <https://onap.readthedocs.io/en/latest/submodules/vnfrqts/requirements.git/docs/index.html>`__ +project. If a VNF does not comply with these rules, then it may not successfully +be modeled in SDC, fail to instantiate, be improperly inventoried in A&AI, or +fail orchestration. -Convenience vs Convention -~~~~~~~~~~~~~~~~~~~~~~~~~ +The project aims to validate every mandatory requirement in the VNF Requirements +related to Heat (i.e. all requirements with a **MUST** or **MUST NOT** keyword) -There are a lot of ways to write tests. Priorities for the VVP test suite are +Heat templates are validated using tests written in ``pytest``. Each test will +validate one or more requirements. Typically we strive to have 1 test per +requirement, but there are situations where it is easiest and clearest to +validate multiple, tightly related requirements with a single test. - - Accuracy - - User Comprehension +Every test **MUST** have a corresponding requirement in the ONAP VNF Requirements +project. If your contribution is a test and there is not an appropriate +requirement, then please consider making a contribution to that project first. -The test suite is often used by people who don't write code, or people -who aren't devoted to writing python validation tests. - -The output of failed validation tests can be difficult to read, so -keep that in mind when you are deciding whether to create another -level of abstraction vs having some code duplication or verbose tests. Writing Tests ############# +Coding Conventions +~~~~~~~~~~~~~~~~~~ + +* Follow `PEP-8 conventions <https://www.python.org/dev/peps/pep-0008/>`__ + * NOTE: The only variation is that the line-length can be 88 + characters vs. 80 +* Familiarize yourself with the utilities that exist in the following utility + modules and leverage them to avoid duplication. + + - ``ice_validator/tests/helpers.py`` + - ``ice_validator/tests/structures.py`` + - ``ice_validator/tests/utils/**`` + +* Ensure all source files include the standard Apache License 2.0 text and + appropriate copyright statement (see other source files for example) +* All code must pass standard quality checks prior to submission which can be + executed via ``tox`` or by running ``checks.py`` +* When parsing YAML, always use the ``tests/cached_yaml`` module versus the + default ``yaml`` module. This will greatly improve performance due to the + large number of yaml files that must be parsed. +* In an effort to keep the number of dependencies down, please favor using + the Python standard library unless using an external library significantly + improves or reduces the needed code to implement the functionality. +* For security purposes the following hardening activities are used: + + - Avoid usage of ``yaml.load`` and always use ``yaml.safe_load`` (Note: if you + use ``cached_yaml`` as instructed above, then this is covered automatically) + - Docker containers must not be run as root (see current Dockerfile for + an example) + - Inspect and resolve all findings by the bandit scans + File Name ~~~~~~~~~ -Test files are written in python, and should go into the +Test files are written in Python, and should go into the ``/validation-scripts/ice_validator/tests/`` directory. They should be prefixed -with ``test_``. If not, ``pytest`` will not discover your test. +with ``test_``. If not, ``pytest`` will not discover your test. The file name +should reflect what is being tested. Test Name ~~~~~~~~~ @@ -56,11 +120,11 @@ For example: Requirement Decorator ~~~~~~~~~~~~~~~~~~~~~ -Each test function should be decorated with a requirement ID from the +Each test function must be decorated with a requirement ID from the VNF Requirements project. The following is required to be imported at the top of the test file: -``from .helpers import validates`` +``from tests.helpers import validates`` Then, your test function should be decorated like this: @@ -75,10 +139,19 @@ report that includes the requirements that were violated. If a test is not decorated it is unclear what the reason for a failure is, and the implication is that the test is not needed. +The validation reports will show the requirement text that was violated and +it will be pulled from the ``heat_requirements.json`` file. This file +is published by the VNFRQTS project, and VVP maintains a copy of the file. +Your requirement should be present in this file. The ``update_reqs.py`` +command can be used to re-synchronize the VVP copy with VNFRQTS master. + Test Parameters ~~~~~~~~~~~~~~~ -Each test should be parameterized based on what artifact is being validated. +There are several dynamic fixtures that can be injected into a test based on +what the test is attempting to validate. Each test should be parameterized based +on what artifact is being validated. + Available parameters are enumerated in ``/validation-scripts/ice_validator/tests/parameterizers.py``. Below is a description of the most commonly used: @@ -143,8 +216,9 @@ Optional: Pytest Markers and Validation Categories The VVP test suite has the concept of a ``base`` test. These are used as sanity tests and are executed before the other tests, and if they fail the -test suite execution is halted. If you are writing a ``base`` test, mark your -test like this: +test suite execution is halted. A test should be annotated with ``base`` if the +failure is likely to generate many subsequent failures (ex: improperly formatted +YAML). If you are writing a ``base`` test, mark your test like this: .. code-block:: python @@ -155,16 +229,17 @@ test like this: def test_my_new_requirement(): The VVP test suite also has the concept of a ``category`` to -define what additional set of optional tests to execute. The way it works -is by using ``categories`` decorator. +define what additional set of optional tests to execute when requested by the +end user. The way it works is by applying the ``categories`` decorator to the +test. By default, all ``base`` tests and tests with no category are executed. If you want an additional category to run, pass the command line argument: ``--category=<category>`` -This will execute all ``base`` tests, non-marked tests, -and tests marked like the following: +This will extend the default set of tests to also include tests marked with +the requested category like the following: .. code-block:: python @@ -176,3 +251,83 @@ and tests marked like the following: This should be used sparingly, and in practice consider reviewing a requirement with the VNF Requirements team before adding a test to a category. + +Testing your Test +~~~~~~~~~~~~~~~~~ + +Every Heat validation test must have a unit test that validates the test is +working as expected. This is handled by creating a one or more "fixtures" that +will exercise the test and validate the expected result. + +The fixtures are stored in the ``ice_validator/tests/fixtures`` directory under +a directory that matches the test file name **exactly**. + +For example, if your test is named ``test_neutron_ports.py``, then the test +fixtures must be in the ``ice_validator/tests/fixtures/test_neutron_ports/`` +directory. + +At minimum, each test must have one example of heat templates/files that +pass (stored in the ``pass`` subdirectory), and one example that fails ( +stored in the ``fail`` subdirectory). These templates do not need to be complete, +valid Heat template - they only need to include the minimum content to +validate the test. + +If you need to test multiple conditions or branches of your test, then you +can nest other directories under your test's fixture directory. Each nested +directory, must in turn have a ``pass`` and ``fail`` subdirectory. + +.. code-block:: + + ice_validator/ + |--- tests/ + |--- fixtures/ + |--- test_neutron_ports/ + |--- scenario_one/ + | |--- pass/ + | |--- fail/ + |--- scenario_two/ + |--- pass/ + |--- fail/ + +To execute all tests for the entire suite, issue the following commmand +from the ``ice_validator`` directory: + +``pytest --self-test`` + +If you wish to selectively execute your test against one of the fixtures, +then issue the following command from the ``ice_validator`` directory: + +``pytest tests/<test_file>.py --template-directory=tests/fixtures/<test_file>/<scenario>`` + +If you have contributed code outside of a ``tests_*.py`` file, then you should +create suitable tests for that functionality in the ``app_tests`` directory. +The tests should be compatible with ``pytest``, but these tests +do not use the fixtures mechanism. + +Submitting Your Changes For Review +################################## + +Once you have completed your changes and tested they work as expected, then the +next step is to validate they are ready for submission. The ``checks.py`` +module in the root directory contains are variety of code quality checks +that the build server will execute. These can be executed locally using ``tox`` +or simply running ``checks.py``. + +At the time of this writing, the following checks will be performed: + +- Executing the full test suite (``app_tests`` and ``--self-test``) +- flake8 code style validation +- Ensuring the ``heat_requirements.json`` file is up-to-date with VNFRQTS + (run ``update_reqs.py`` if this check fails) +- Ensures all mandatory tests from VNFRQTS have tests in VVP +- Security checks via bandit + +Once all tests are passed, then refer to `Pushing Changes Using Git <https://wiki.onap.org/display/DW/Pushing+Changes+Using+Git>`__ +for details on how to submit your change. + +Once your change has been submitted, please add the following individuals as +reviewers at minimum: + +- Steven Stark +- Trevor Lovett +- Steven Wright diff --git a/docs/index.rst b/docs/index.rst index abfcfa9..b381d9f 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -28,6 +28,6 @@ setup, and execute the validations. installation.rst validation-scripts.rst - contributing.rst humaninterfaces.rst + contributing.rst release-notes.rst diff --git a/docs/release-notes.rst b/docs/release-notes.rst index 1a6c24b..2fdb5fa 100644 --- a/docs/release-notes.rst +++ b/docs/release-notes.rst @@ -7,6 +7,80 @@ VVP Project Release Notes =============================== +Version: 5.1.0 +-------------- + +:Release Date: TBD + +**Removed Features** +- None + +**New Features** + +- OpenStack Heat Validation - VVP now includes the latest version of OpenStack + Heat, and can be used to validate that the Heat not only complies with ONAP + rules, but is also valid Heat (similar to stack-validate) (``VVP-218``) +- Preload Template Generation - VVP will not create preload templates based + on the Heat template being validated. The user can optionally populate the + template by specifying environment files (``VVP-227``, ``VVP-277``) +- Added ``checks.py`` to consolidate various quality checks that can now + be performed cross-platform and consistently between the local and build + environment. + + - Added quality check to ensure VVP includes the latest version of Heat + requirements from the VVP project + - All code is now scanned for security issues using the Bandit library + (``VVP-244``) + +- Updated Availability Zone tests to align with latest VNF Requirements + (``VVP-226``) +- Performance enhancements - improved performance of validation of large + templates by 30-70% (``VVP-225``) +- VVP GUI can be customized to display configurable disclaimer text, and + also allow the acceptance of terms-and-condition, or other legal agreements + before allowing the user to use the tool (``VVP-195``) +- Enhanced report readability by removing unnecessary columns and other + enhancements (``VVP-184``) +- Removed dependency on ``yamllint`` library to remove dependency on L/GPL code + (``VVP-201``) +- Allow error messages with line breaks (``VVP-225``) +- Various enhancements to remove redundant tests or improve error messages + + + +**Bug Fixes** +- Fixed errors in ``test_environment_file_parameters`` where wrong variables + were being checked (``VVP-267``) +- VVP GUI fails to open reports when tools is launched from a network share + (``VVP-266``) +- Escape error messages before display in HTML report (``VVP-159``) +- Improved error message when Heat archives included nested directories which + are not allowed (``VVP-217``) +- Relaxed validation of ``get_param`` usage to better comply with SDC + implementation (``VVP-220``) + + +**Known Issues** + +- None + +**Security Notes** + +VVP code has been formally scanned during build time using NexusIQ and no +Critical vulnerability was found. + +Additionally, all VVP code is now scanned using the `Bandit <https://bandit.readthedocs.io/en/latest/>`__ +library. All potential issues reported this scanning process have been +addressed or marked as non-issues using the ``# nosec`` marker in the source +code. + +Quick Links: + +- `VVP project page <https://wiki.onap.org/display/DW/VNF+Validation+Program+Project>`_ +- `Passing Badge information for VVP <https://bestpractices.coreinfrastructure.org/en/projects/1738>`_ + + + Version: 4.0.0 -------------- |