[VVP] Update the Contributing.rst and Release Notes

Addresses:
- How to add tests for the validations
- Hardening practices followed by the project
- How to perform quality checks before submission

Issue-ID: VVP-250
Issue-ID: VVP-251
Issue-ID: VVP-253

Change-Id: I8be8626372bd25ef2089491a2f1e4938b21cdcfa
Signed-off-by: Lovett, Trevor <trevor.lovett@att.com>

3 files changed, 260 insertions, 31 deletions

diff --git a/docs/contributing.rst b/docs/contributing.rst
index 2643f7c..8b561af 100644
--- a/docs/contributing.rst
+++ b/docs/contributing.rst
@@ -2,43 +2,107 @@
 .. http://creativecommons.org/licenses/by/4.0
 .. Copyright 2019 AT&T Intellectual Property.  All rights reserved.
 
-Contributing
-############
+How to Contribute
+#################
+
+Overview
+~~~~~~~~
+
+This section will provide details on how to contribute changes to the project
+covering both the mechanics of how to contribute new code as well as
+how to adhere to the code quality and coding practices of the project.
+
+
+Prerequisites
+~~~~~~~~~~~~~
+
+As this project is part of the overall ONAP project, there are some common
+guidelines and activities you will need to adhere to:
+
+- Acquire a `Linux Foundation ID <https://identity.linuxfoundation.org/>`__
+- Configure your `ONAP Gerrit Account <https://wiki.onap.org/display/DW/Configuring+Gerrit>`__
+  as it is used for all code submissions and reviews
+- Install VVP per the :ref:`installation instructions <vvp-installation>`
+- Most features will require some knowledge of both `Openstack Heat <https://wiki.openstack.org/wiki/Heat>`__
+  and the `ONAP Heat requirements <https://onap.readthedocs.io/en/latest/submodules/vnfrqts/requirements.git/docs/Chapter5/Heat/index.html>`__
+- Knowledge of writing tests in `pytest <https://pytest.readthedocs.io/>`__
+
+Other useful links:
+
+- All work is documented and tracked via the `VVP Project <https://jira.onap.org/projects/VVP/issues/>`__
+  in the `ONAP JIRA instance <https://jira.onap.org/>`__. Login is via your
+  Linux Foundation ID
+- Proposals for new features, general information about the projects,
+  meeting minutes, and ONAP process information is located on the
+  `ONAP Wiki <https://wiki.onap.org/>`__
+- The VVP project hosts a weekly meeting to plan upcoming work, discuss open
+  issues, and align on priorities.  Please consider attending if possible if you
+  intend to contribute to the project.  Refer to the ONAP Calendar `<https://wiki.onap.org/pages/viewpage.action?pageId=6587439>`__
+  for scheduling details
 
 Objective
 ~~~~~~~~~
 
-**The objective for the VVP test suite is for each
-test to directly correlate with at least one requirement in the**
-`VNF Requirements <https://onap.readthedocs.io/en/latest/submodules/vnfrqts/requirements.git/docs/index.html>`__
-**project in ONAP. If the test you intend to write doesn't
-have a corresponding requirement in the VNF Requirements project, consider
-making a contribution to that project first.**
+The primary focus of VVP is ensuring that a VNF that is described using Openstack
+Heat complies with the ONAP Heat requirements specified in the `VNF Requirements (VNFRQTS) <https://onap.readthedocs.io/en/latest/submodules/vnfrqts/requirements.git/docs/index.html>`__
+project.  If a VNF does not comply with these rules, then it may not successfully
+be modeled in SDC, fail to instantiate, be improperly inventoried in A&AI, or
+fail orchestration.
 
-Convenience vs Convention
-~~~~~~~~~~~~~~~~~~~~~~~~~
+The project aims to validate every mandatory requirement in the VNF Requirements
+related to Heat (i.e. all requirements with a **MUST** or **MUST NOT** keyword)
 
-There are a lot of ways to write tests. Priorities for the VVP test suite are
+Heat templates are validated using tests written in ``pytest``.  Each test will
+validate one or more requirements. Typically we strive to have 1 test per
+requirement, but there are situations where it is easiest and clearest to
+validate multiple, tightly related requirements with a single test.
 
- - Accuracy
- - User Comprehension
+Every test **MUST** have a corresponding requirement in the ONAP VNF Requirements
+project.  If your contribution is a test and there is not an appropriate
+requirement, then please consider making a contribution to that project first.
 
-The test suite is often used by people who don't write code, or people
-who aren't devoted to writing python validation tests.
-
-The output of failed validation tests can be difficult to read, so
-keep that in mind when you are deciding whether to create another
-level of abstraction vs having some code duplication or verbose tests.
 
 Writing Tests
 #############
 
+Coding Conventions
+~~~~~~~~~~~~~~~~~~
+
+* Follow `PEP-8 conventions <https://www.python.org/dev/peps/pep-0008/>`__
+    * NOTE: The only variation is that the line-length can be 88
+      characters vs. 80
+* Familiarize yourself with the utilities that exist in the following utility
+  modules and leverage them to avoid duplication.
+
+    - ``ice_validator/tests/helpers.py``
+    - ``ice_validator/tests/structures.py``
+    - ``ice_validator/tests/utils/**``
+
+* Ensure all source files include the standard Apache License 2.0 text and
+  appropriate copyright statement (see other source files for example)
+* All code must pass standard quality checks prior to submission which can be
+  executed via ``tox`` or by running ``checks.py``
+* When parsing YAML, always use the ``tests/cached_yaml`` module versus the
+  default ``yaml`` module.  This will greatly improve performance due to the
+  large number of yaml files that must be parsed.
+* In an effort to keep the number of dependencies down, please favor using
+  the Python standard library unless using an external library significantly
+  improves or reduces the needed code to implement the functionality.
+* For security purposes the following hardening activities are used:
+
+    - Avoid usage of ``yaml.load`` and always use ``yaml.safe_load`` (Note: if you
+      use ``cached_yaml`` as instructed above, then this is covered automatically)
+    - Docker containers must not be run as root (see current Dockerfile for
+      an example)
+    - Inspect and resolve all findings by the bandit scans
+
 File Name
 ~~~~~~~~~
 
-Test files are written in python, and should go into the
+Test files are written in Python, and should go into the
 ``/validation-scripts/ice_validator/tests/`` directory. They should be prefixed
-with ``test_``. If not, ``pytest`` will not discover your test.
+with ``test_``. If not, ``pytest`` will not discover your test.  The file name
+should reflect what is being tested.
 
 Test Name
 ~~~~~~~~~
@@ -56,11 +120,11 @@ For example:
 Requirement Decorator
 ~~~~~~~~~~~~~~~~~~~~~
 
-Each test function should be decorated with a requirement ID from the
+Each test function must be decorated with a requirement ID from the
 VNF Requirements project. The following is required to be imported at
 the top of the test file:
 
-``from .helpers import validates``
+``from tests.helpers import validates``
 
 Then, your test function should be decorated like this:
 
@@ -75,10 +139,19 @@ report that includes the requirements that were violated. If a test is not
 decorated it is unclear what the reason for a failure is, and the
 implication is that the test is not needed.
 
+The validation reports will show the requirement text that was violated and
+it will be pulled from the ``heat_requirements.json`` file.  This file
+is published by the VNFRQTS project, and VVP maintains a copy of the file.
+Your requirement should be present in this file.  The ``update_reqs.py``
+command can be used to re-synchronize the VVP copy with VNFRQTS master.
+
 Test Parameters
 ~~~~~~~~~~~~~~~
 
-Each test should be parameterized based on what artifact is being validated.
+There are several dynamic fixtures that can be injected into a test based on
+what the test is attempting to validate. Each test should be parameterized based
+on what artifact is being validated.
+
 Available parameters are enumerated in
 ``/validation-scripts/ice_validator/tests/parameterizers.py``. Below is a description
 of the most commonly used:
@@ -143,8 +216,9 @@ Optional: Pytest Markers and Validation Categories
 
 The VVP test suite has the concept of a ``base`` test. These are used as
 sanity tests and are executed before the other tests, and if they fail the
-test suite execution is halted. If you are writing a ``base`` test, mark your
-test like this:
+test suite execution is halted. A test should be annotated with ``base`` if the
+failure is likely to generate many subsequent failures (ex: improperly formatted
+YAML).  If you are writing a ``base`` test, mark your test like this:
 
 .. code-block:: python
 
@@ -155,16 +229,17 @@ test like this:
   def test_my_new_requirement():
 
 The VVP test suite also has the concept of a ``category`` to
-define what additional set of optional tests to execute. The way it works
-is by using ``categories`` decorator.
+define what additional set of optional tests to execute when requested by the
+end user. The way it works is by applying the ``categories`` decorator to the
+test.
 
 By default, all ``base`` tests and tests with no category are executed.
 If you want an additional category to run, pass the command line argument:
 
 ``--category=<category>``
 
-This will execute all ``base`` tests, non-marked tests,
-and tests marked like the following:
+This will extend the default set of tests to also include tests marked with
+the requested category like the following:
 
 .. code-block:: python
 
@@ -176,3 +251,83 @@ and tests marked like the following:
 
 This should be used sparingly, and in practice consider reviewing a requirement
 with the VNF Requirements team before adding a test to a category.
+
+Testing your Test
+~~~~~~~~~~~~~~~~~
+
+Every Heat validation test must have a unit test that validates the test is
+working as expected.  This is handled by creating a one or more "fixtures" that
+will exercise the test and validate the expected result.
+
+The fixtures are stored in the ``ice_validator/tests/fixtures`` directory under
+a directory that matches the test file name **exactly**.
+
+For example, if your test is named ``test_neutron_ports.py``, then the test
+fixtures must be in the ``ice_validator/tests/fixtures/test_neutron_ports/``
+directory.
+
+At minimum, each test must have one example of heat templates/files that
+pass (stored in the ``pass`` subdirectory), and one example that fails (
+stored in the ``fail`` subdirectory).  These templates do not need to be complete,
+valid Heat template - they only need to include the minimum content to
+validate the test.
+
+If you need to test multiple conditions or branches of your test, then you
+can nest other directories under your test's fixture directory.  Each nested
+directory, must in turn have a ``pass`` and ``fail`` subdirectory.
+
+.. code-block::
+
+    ice_validator/
+    |--- tests/
+         |--- fixtures/
+              |--- test_neutron_ports/
+                   |--- scenario_one/
+                   |    |--- pass/
+                   |    |--- fail/
+                   |--- scenario_two/
+                        |--- pass/
+                        |--- fail/
+
+To execute all tests for the entire suite, issue the following commmand
+from the ``ice_validator`` directory:
+
+``pytest --self-test``
+
+If you wish to selectively execute your test against one of the fixtures,
+then issue the following command from the ``ice_validator`` directory:
+
+``pytest tests/<test_file>.py --template-directory=tests/fixtures/<test_file>/<scenario>``
+
+If you have contributed code outside of a ``tests_*.py`` file, then you should
+create suitable tests for that functionality in the ``app_tests`` directory.
+The tests should be compatible with ``pytest``, but these tests
+do not use the fixtures mechanism.
+
+Submitting Your Changes For Review
+##################################
+
+Once you have completed your changes and tested they work as expected, then the
+next step is to validate they are ready for submission.  The ``checks.py``
+module in the root directory contains are variety of code quality checks
+that the build server will execute.  These can be executed locally using ``tox``
+or simply running ``checks.py``.
+
+At the time of this writing, the following checks will be performed:
+
+- Executing the full test suite (``app_tests`` and ``--self-test``)
+- flake8 code style validation
+- Ensuring the ``heat_requirements.json`` file is up-to-date with VNFRQTS
+  (run ``update_reqs.py`` if this check fails)
+- Ensures all mandatory tests from VNFRQTS have tests in VVP
+- Security checks via bandit
+
+Once all tests are passed, then refer to `Pushing Changes Using Git <https://wiki.onap.org/display/DW/Pushing+Changes+Using+Git>`__
+for details on how to submit your change.
+
+Once your change has been submitted, please add the following individuals as
+reviewers at minimum:
+
+- Steven Stark
+- Trevor Lovett
+- Steven Wright
diff --git a/docs/index.rst b/docs/index.rst
index abfcfa9..b381d9f 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -28,6 +28,6 @@ setup, and execute the validations.
 
    installation.rst
    validation-scripts.rst
-   contributing.rst
    humaninterfaces.rst
+   contributing.rst
    release-notes.rst
diff --git a/docs/release-notes.rst b/docs/release-notes.rst
index 1a6c24b..2fdb5fa 100644
--- a/docs/release-notes.rst
+++ b/docs/release-notes.rst
@@ -7,6 +7,80 @@ VVP Project Release Notes
 ===============================
 
 
+Version: 5.1.0
+--------------
+
+:Release Date: TBD
+
+**Removed Features**
+- None
+
+**New Features**
+
+- OpenStack Heat Validation - VVP now includes the latest version of OpenStack
+  Heat, and can be used to validate that the Heat not only complies with ONAP
+  rules, but is also valid Heat (similar to stack-validate) (``VVP-218``)
+- Preload Template Generation - VVP will not create preload templates based
+  on the Heat template being validated. The user can optionally populate the
+  template by specifying environment files (``VVP-227``, ``VVP-277``)
+- Added ``checks.py`` to consolidate various quality checks that can now
+  be performed cross-platform and consistently between the local and build
+  environment.
+
+    - Added quality check to ensure VVP includes the latest version of Heat
+      requirements from the VVP project
+    - All code is now scanned for security issues using the Bandit library
+      (``VVP-244``)
+
+- Updated Availability Zone tests to align with latest VNF Requirements
+  (``VVP-226``)
+- Performance enhancements - improved performance of validation of large
+  templates by 30-70% (``VVP-225``)
+- VVP GUI can be customized to display configurable disclaimer text, and
+  also allow the acceptance of terms-and-condition, or other legal agreements
+  before allowing the user to use the tool (``VVP-195``)
+- Enhanced report readability by removing unnecessary columns and other
+  enhancements (``VVP-184``)
+- Removed dependency on ``yamllint`` library to remove dependency on L/GPL code
+  (``VVP-201``)
+- Allow error messages with line breaks (``VVP-225``)
+- Various enhancements to remove redundant tests or improve error messages
+
+
+
+**Bug Fixes**
+- Fixed errors in ``test_environment_file_parameters`` where wrong variables
+  were being checked (``VVP-267``)
+- VVP GUI fails to open reports when tools is launched from a network share
+  (``VVP-266``)
+- Escape error messages before display in HTML report (``VVP-159``)
+- Improved error message when Heat archives included nested directories which
+  are not allowed (``VVP-217``)
+- Relaxed validation of ``get_param`` usage to better comply with SDC
+  implementation (``VVP-220``)
+
+
+**Known Issues**
+
+- None
+
+**Security Notes**
+
+VVP code has been formally scanned during build time using NexusIQ and no
+Critical vulnerability was found.
+
+Additionally, all VVP code is now scanned using the `Bandit <https://bandit.readthedocs.io/en/latest/>`__
+library.  All potential issues reported this scanning process have been
+addressed or marked as non-issues using the ``# nosec`` marker in the source
+code.
+
+Quick Links:
+
+- `VVP project page <https://wiki.onap.org/display/DW/VNF+Validation+Program+Project>`_
+- `Passing Badge information for VVP <https://bestpractices.coreinfrastructure.org/en/projects/1738>`_
+
+
+
 Version: 4.0.0
 --------------