[VVP] ADD how to contribute information

Change-Id: I7a2dbd7f110d3c129521eb59d53b19c30cc3b8a2
Issue-ID: VVP-125
Signed-off-by: stark, steven <steven.stark@att.com>

1 files changed, 208 insertions, 14 deletions

diff --git a/README.rst b/README.rst
index 8c3e782..87348ea 100644
--- a/README.rst
+++ b/README.rst
@@ -5,6 +5,8 @@
 Manual Heat Template Validation
 ===============================
 
+.. contents::
+
 validation-scripts
 ------------------
 
@@ -49,7 +51,7 @@ To validate Heat templates just run this the command under the folder ``ice_vali
 where ``<Directory>`` is the full path to a folder containing heat templates.
 
 Output
-______
+######
 
 After performing a validation, an output folder will be created.
 
@@ -60,18 +62,18 @@ of the ONAP VNF Heat Template Guideline violations. If there are no violations,
 the report will say ``No validation errors found.``
 
 Interpreting the Output
-_______________________
+#######################
 
 The report file will have 4 columns for details about a violation, and one
 row for each violation. Below contains details about each column.
 
 File
-####
+~~~~
 
 This is the file(s) that contained the violation
 
 Error Message
-#############
+~~~~~~~~~~~~~
 
 This shows the test and brief error message from the validation script that
 contained the violation. There is a ``Full Details`` button to show the
@@ -80,7 +82,7 @@ about what element is involved with the violation (such as the parameter
 name, resource id, etc...).
 
 Requirement(s)
-##############
+~~~~~~~~~~~~~~
 
 This column contains the requirement(s) that each test/violation is
 mapped to. These requirements are taken directly from the VNF Requirements
@@ -88,7 +90,7 @@ project Heat Orchestration Template Guidelines section.
 
 
 Resolution Steps
-################
+~~~~~~~~~~~~~~~~
 
 For some violations, there are pre-defined resolution steps that
 indicate what action the user should take to resolve the violation.
@@ -97,18 +99,210 @@ indicate what action the user should take to resolve the violation.
 can be resolved simply by reviewing the requirements that have been violated
 in the previous column.
 
+How to Contribute
+_________________
+
+Before getting started
+######################
+
+Objective
+~~~~~~~~~
+
+**The objective for the VVP test suite is for each
+test to directly correlate with at least one requirement in the**
+`VNF Requirements <https://onap.readthedocs.io/en/latest/submodules/vnfrqts/requirements.git/docs/index.html>`__
+**project in ONAP. If the test you intend to write doesn't
+have a corresponding requirement in the VNF Requirements project, consider
+making a contribution to that project first.**
+
+Convenience vs Convention
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are a lot of ways to write tests. Priorities for the VVP test suite are
+
+ - Accuracy
+ - User Comprehension
+
+The test suite is often used by people who don't write code, or people
+who aren't devoted to writing python validation tests.
+
+The output of failed validation tests can be difficult to read, so
+keep that in mind when you are deciding whether to create another
+level of abstraction vs having some code duplication or verbose tests.
+
+Developer Guide
+###############
+
+File Name
+~~~~~~~~~
+
+Test files are written in python, and should go into the
+``/validation-scripts/ice_validator/tests/`` directory. They should be prefixed
+with ``test_``. If not, ``pytest`` will not discover your
+test if you don't follow this convention.
+
+Test Name
+~~~~~~~~~
+
+Tests are functions defined in the test file, and also must be prefixed with
+``test_``. If not, ``pytest`` will not collect them during execution.
+For example:
+
+**test_my_new_requirement_file.py**
+
+.. code-block:: python
+
+  def test_my_new_requirement():
+
+Requirement Decorator
+~~~~~~~~~~~~~~~~~~~~~
+
+Each test function should be decorated with a requirement ID from the
+VNF Requirements project. The following is required to be imported at
+the top of the test file:
+
+``from .helpers import validates``
+
+Then, your test function should be decorated like this:
+
+.. code-block:: python
+
+  @validates("R-123456",
+             "R-123457") # these requirement IDs should come from the VNFRQTS project
+  def test_my_new_requirement():
+
+This decorator is used at the end of the test suite execution to generate a
+report that includes the requirements that were violated. If a test is not
+decorated it is unclear what the reason for a failure is, and the
+implication is that the test is not needed.
+
+Test Parameters
+~~~~~~~~~~~~~~~
+
+Each test should be parameterized based on what artifact is being validated.
+Available parameters are enumerated in
+``/validation-scripts/ice_validator/tests/parameterizers.py``. Below is a description
+of the most commonly used:
+
+  - ``heat_template``: parameter is the full path name for a file with the
+    extenstion ``.yaml`` or ``.yml``,
+    if the file also has a corresponding file with the same name but
+    extension ``.env``.
+  - ``yaml_file``: parameter is the full path name for a file with the
+    extenstion ``.yaml`` or ``.yml``
+  - ``yaml_files``: parameter is a list of all files with the extenstion
+    ``.yaml`` or ``.yml``.
+  - ``volume_template``: parameter is the full path name for a file name
+    that ends with ``_volume`` and the extension ``.yaml`` or ``.yml``.
+
+There are many others that can also be used, check ``parameterizers.py`` for
+the full list.
+
+The parameter that you decide to use determines how many times a test is
+executed, and what data is available to validate. For example, if the
+test suite is executed against a directory with 10 ``.yaml`` files, and
+a test is using the parameter ``yaml_file``, the test will be executed
+once for each file, for a total of 10 executions. If the parameter
+``yaml_files`` (note the plural) is used instead, the test will
+only execute once.
+
+Here's an example for how to parameterize a test:
+
+.. code-block:: python
+
+  @validates("R-123456",
+             "R-123457")
+  def test_my_new_requirement(yaml_file): # this test will execute for each .yaml or .yml
+
+Collecting Failures
+~~~~~~~~~~~~~~~~~~~
+
+To raise a violation to ``pytest`` to be collected and included on the final
+violation report, use the ``assert`` statement. Example:
+
+.. code-block:: python
+
+  @validates("R-123456",
+             "R-123457")
+  def test_my_new_requirement(yaml_file):
+    my test logic
+    ...
+    ...
+    ...
+
+    assert not failure_condition, error_message
+
+As one of the VVP priorities is User Comprehension, the ``error_message``
+should be readable and include helpful information for triaging the failure,
+such as the ``yaml_file``, the parameter the test was checking, etc...
+
+If the assert statement fails, the failure is collected by ``pytest``, and the
+associated requirements and error_message are included in the final report.
+
+Optional: Pytest Markers and Validation Profiles
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The VVP test suite has the concept of a ``base`` test. These are used as
+sanity tests and are executed before the other tests, and if they fail the
+test suite execution is halted. If you are writing a ``base`` test, mark your
+test like this:
+
+.. code-block:: python
+
+  import pytest
+
+  @pytest.mark.base # this is the base test marker
+  @validates("R-123456")
+  def test_my_new_requirement():
+
+The VVP test suite also has the concept of a ``validation profile`` to
+define what set of tests to execute. The way it works is by using ``pytest``
+markers.
+
+By default, all ``base`` tests and non-marked tests are executed. If you want
+an additional profile to run, pass the command line argument:
+
+``--validation-profile=<my_validation_profile>``
+
+This will execute all ``base`` tests, non-marked tests,
+and tests marked like the following:
+
+.. code-block:: python
+
+  import pytest
+
+  @pytest.mark.<my_validation_profile> # this is an additional pytest marker
+  @validates("R-123456")
+  def test_my_new_requirement():
+
+This should be used sparingly, and in practice consider reviewing a requirement
+with the VNF Requirements team before adding a test to a validation profile.
 
 Self-Test Suite
-_______________
+~~~~~~~~~~~~~~~
+
+The VVP test suite includes an extensive self-test suite. This can be
+executed by running
+
+``</path/to/validation-scripts/ice_validator>$ pytest --self-test tests/``
+
+This self test suite is used as a check for any new or modified tests.
 
-The ``ice_validator`` includes an extensive self-test suite. It is a
-**requirement** for any additions or changes to the test suite to
-successfully and cleanly complete a tox run. Simply run ``tox`` from
-the project root as:
+If you are adding a new test, a new self-test ``fixture`` **MUST** be created
+in the directory ``/validation-scripts/ice_validator/tests/fixtures``. The
+directory should be named identical to the new python file (without the ``.py``
+extension), and it should contain 2 subdirectories: ``pass`` and ``fail``.
 
-``$ tox``
+These directories should include heat templates that pass and fail the new test.
+For Example, if I have created a new test called ``test_my_new_requirement.py``
+, I should create:
 
-You can also run it under the folder ``ice_validator``:
+``/validation-scripts/ice_validator/tests/fixtures/test_my_new_requirement/pass``
+``/validation-scripts/ice_validator/tests/fixtures/test_my_new_requirement/pass/pass.yaml``
+``/validation-scripts/ice_validator/tests/fixtures/test_my_new_requirement/fail``
+``/validation-scripts/ice_validator/tests/fixtures/test_my_new_requirement/fail/fail.yaml``
 
-``$ pytest --self-test``
+When executing the self-test suite, the templates in these folders are
+expected to pass and fail, respectively, **ONLY** for the corresponding test.
+They don't need to pass the whole test suite.