diff options
Diffstat (limited to 'docs/validation-scripts.rst')
| -rw-r--r-- | docs/validation-scripts.rst | 379 |
1 files changed, 314 insertions, 65 deletions
diff --git a/docs/validation-scripts.rst b/docs/validation-scripts.rst index 0ff6e32..b327ad4 100644 --- a/docs/validation-scripts.rst +++ b/docs/validation-scripts.rst @@ -1,105 +1,349 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. Copyright 2018 AT&T Intellectual Property. All rights reserved. +.. Copyright 2019 AT&T Intellectual Property. All rights reserved. -Manual Heat Template Validation -=============================== +Heat Validation via Command Line +================================ -validation-scripts ------------------- +.. note:: + This section assumes you have already acquired and setup the validation + scripts as described in the :ref:`VVP Installation <vvp-installation>` + section. -This project contains validation scripts to test -that a set of Heat Templates adheres to -the ONAP VNF Heat Orchestration Template guidelines. +Basic Script Execution +---------------------- -For more information on the ONAP Heat Orchestration -Template Guidelines, vist the `Heat Guidelines <https://onap.readthedocs.io/en/latest/submodules/vnfrqts/requirements.git/docs/Chapter5/Heat/index.html>`__ +1. The scripts must be executed from the ``ice_validator`` directory. + Navigate to the directory first:: -About -_____ + > cd <vvp-directory>/ice_validator -The validation scripts project allows performing heat template -validation without installing the full VVP platform. The following -instructions apply to running these validation scripts in that manner. +2. To run the script with default options, issue the following command where + ``<Directory>`` is the directory containing the Heat templates:: + + /path/to/validation-scripts/ice_validator>$ pytest tests --template-directory=<Directory>`` + +3. Test results will be written to the consoles describing any failures that + are encountered. If no failures are found, then a success message will + be displayed. Additional reports can be found in the + ``<VVP Directory>/ice_validator/output`` directory. See + :ref:`Reports <vvp-reports>` for more details. + + +.. _vvp-cmd-options: + +Command Line Options +-------------------- + +Additional options can be specified at the command line to customize output +and execution. VVP uses the `pytest <https://docs.pytest.org>`__ +framework for test execution, and therefore all pytest options are available. + +Please refer to the pytest documentation for information on the command-line +options that framework provides. We will only document the specific options +that are used by VVP:: + + --template-directory=TEMPLATE_DIR + Directory which holds the templates for validation + + --category=CATEGORY_NAME + Additional validations that can be selected. + The only supported value at this time is + environment_file which will enforce that specific + parameter values are excluded from the environment + file per ONAP Heat requirements. + + --self-test Test the unit tests against their fixtured data + + --report-format=REPORT_FORMAT + Format of output report (html, csv, excel, json) + + --continue-on-failure + Continue validation even when structural errors exist + in input files + + --output-directory=OUTPUT_DIR + Alternate directory for report output. + +.. _vvp-reports: + +VVP Reports +----------- + +After completion of the validations, several reports will be written to the +ouput directory (specified via ``--output-directory`` or +``<vvp-directory>/ice_validator/output`` by default). + + ++-----------------------+------------------------------------------------------+ +| Report | Description | ++=======================+======================================================+ +| ``report.{html,csv,`` | Full report of validation results showing the pass | +| ``xlsx,json}``| or failure of the validation and details on all | +| | errors. See :ref:`Validation Report <vvp-report>` | +| | for information on the HTML, CSV, and Excel versions | +| | of the report, or | +| | :ref:`JSON Report <vvp-json-report>` for details on | +| | the machine-readable results. | ++-----------------------+------------------------------------------------------+ +| ``traceability.csv`` | This shows a mapping from Heat requirement to | +| | test case. | ++-----------------------+------------------------------------------------------+ +| ``mapping_errors.csv``| File should be empty, but if present shows tests | +| | that are mapped to requirements that no longer exist | ++-----------------------+------------------------------------------------------+ +| ``traceability.rst`` | Similar information to ``traceability.csv``, but | +| | in reStructuredText for use in the VNF Requirements | +| | documentation. | ++-----------------------+------------------------------------------------------+ +| ``failures`` | **Deprecated** JSON version of test failures. Use | +| | ``report.json`` instead. | ++-----------------------+------------------------------------------------------+ + +.. _vvp-report: + +Validation Report +~~~~~~~~~~~~~~~~~ + +If the report format of ``html`` (*default*), ``excel``, or ``csv`` are +requested via the ``--report-format`` option, then the a report file will +be written to the output directory. Regardless of format, the file will contain +a header section that summarizes the results and files scanned, and an error +section that has a row for each failure with four columns. + +Header Information +^^^^^^^^^^^^^^^^^^ + ++-----------------------+------------------------------------------------------+ +| Header Element | Description | ++=======================+======================================================+ +| Categories Selected | Any additional categories selected via ``--category``| ++-----------------------+------------------------------------------------------+ +| Tool Version | Version of the tool that produced the report | ++-----------------------+------------------------------------------------------+ +| Report Generated At | The timestamp of when the report was generated | ++-----------------------+------------------------------------------------------+ +| Directory Validated | Absolute path to directory validated | ++-----------------------+------------------------------------------------------+ +| Checksum | Unique MD5 has of the contents of template directory | ++-----------------------+------------------------------------------------------+ +| Total Errors | Number of errors/violations found | ++-----------------------+------------------------------------------------------+ + + +Error Information +^^^^^^^^^^^^^^^^^ + +If any violations are found, then there will be one row for each violation +with the following columns: + ++-----------------------+------------------------------------------------------+ +| Column Name | Description | ++=======================+======================================================+ +| Files | The file or files that were scanned as part of the | +| | test. | ++-----------------------+------------------------------------------------------+ +| Tests | Name of the test case (not shown in HTML version) | ++-----------------------+------------------------------------------------------+ +| Error Message | This shows the test and brief error message from the | +| | test that failed. This will contain details about | +| | the element that triggered the violation such as the | +| | parameter name, resource ID, etc. | +| | | +| | In the HTML version of the report this column will | +| | also show the test case name, and provide a link to | +| | ``Full Details`` the raw output of the test | ++-----------------------+------------------------------------------------------+ +| Requirements | The requirement ID and text that was violated | ++-----------------------+------------------------------------------------------+ +| Resolution Steps | For some violations, there are pre-defined resolution| +| | steps that indicate what action the user should take | +| | to resolve the violation. | +| | | +| | **Note**: Not all violations will have resolution | +| | steps, rather the error message and requirement is | +| | sufficient. | ++-----------------------+------------------------------------------------------+ +| Raw Test Output | Full output from the pytest test case. This not a | +| | dedicated column in the HTML version of the report. | ++-----------------------+------------------------------------------------------+ + + +.. _vvp-json-report: + +JSON Report +~~~~~~~~~~~ + +This report is intended to provide a machine-readable version of the test +execution, and provides the most comprehensive summary of the test execution +and results. + +File Header/Top Level +^^^^^^^^^^^^^^^^^^^^^ + +JSON Report <vvp-json-report> +The top level will include a summary of available execution metadata. + +NOTE: The ``tests`` and ``requirements`` entries are elided in the +example below. + +**Example Header**: + +.. code-block:: javascript + + { + "version": "dublin", + "template_directory": "/path/to/template", + "timestamp": "2019-01-21T02:11:07.305000", + "checksum": "6296aa211870634f9b4a23477c5eab28", + "profile": "", + "outcome": "FAIL", + "tests": [], + "requirements": [], + } + +**Header Definition**: + +.. csv-table:: + :header-rows: 1 + :file: vvp_json_header.csv + +.. _vvp-test-result: + +Test Result +^^^^^^^^^^^ + +For each test result a JSON object will be provided that informs the consumer +what tests was run, its result, and the requirements it validated. + +**Example Test Result**: + +.. code-block:: javascript + + { + "files": [ + "/Users/username/Desktop/stark_template2/STARKDB-nested-1.yaml", + "/Users/username/Desktop/stark_template2/base_starkdb.yaml", + ], + "test_module": "test_resource_indices", + "test_case": "test_indices_start_at_0_increment", + "result": "FAIL", + "error": " Index values associated with resource ID prefix STARKDB_server_ do not start at 0\n", + "requirements": [ + { + "id": "R-11690", + "text": "When a VNF's Heat Orchestration Template's Resource ID contains an\n``{index}``, the ``{index}`` is a numeric value that **MUST** start at\nzero and **MUST** increment by one.\n\nAs stated in R-16447,\n*a VNF's <resource ID> MUST be unique across all Heat\nOrchestration Templates and all HEAT Orchestration Template\nNested YAML files that are used to create the VNF*. While the ``{index}``\nwill start at zero in the VNF, the ``{index}`` may not start at zero\nin a given Heat Orchestration Template or HEAT Orchestration Template\nNested YAML file.", + "keyword": "MUST" + } + ] + } + + +**Test Result Definition**: + +.. csv-table:: + :header-rows: 1 + :file: vvp_json_test_result.csv -Installation -____________ +.. _vvp-req-metadata: -This software is not platform dependent and can be run in a Windows, Unix or -OS X environment. +Requirement Metadata +^^^^^^^^^^^^^^^^^^^^ -Satisfy Dependencies -#################### +For each test case, the following requirement metadata will be reported. +**Example Requirement Metadata**: - These can be installed using pip (assuming pip is installed) with the command: +.. code-block:: javascript -``$ pip install -r requirements.txt`` + { + "id": "R-11690", + "text": "When a VNF's Heat Orchestration Template's Resource ID contains an\n``{index}``, the ``{index}`` is a numeric value that **MUST** start at\nzero and **MUST** increment by one.\n\nAs stated in R-16447,\n*a VNF's <resource ID> MUST be unique across all Heat\nOrchestration Templates and all HEAT Orchestration Template\nNested YAML files that are used to create the VNF*. While the ``{index}``\nwill start at zero in the VNF, the ``{index}`` may not start at zero\nin a given Heat Orchestration Template or HEAT Orchestration Template\nNested YAML file.", + "keyword": "MUST" + } -Use -___ -Clone this project. +**Requirement Metadata Definition**: -To validate Heat templates just run this the command under the folder ``ice_validator``: ++-----------------+-------------+--------------+-----------+-------------------+ +| Field Name | Required | Data Type | Valid | Description | +| | Optional | | Values | | +| | Conditional | | | | ++=================+=============+==============+===========+===================+ +| ``id`` | Required | ``string`` | | Requirement ID | +| | | | | from the VNFRQTS | +| | | | | project | ++-----------------+-------------+--------------+-----------+-------------------+ +| ``text`` | Required | ``string`` | | Full text of | +| | | | | requirement | ++-----------------+-------------+--------------+-----------+-------------------+ +| ``keyword`` | Required | ``string`` | MUST, | RFC 2119 keyword | +| | | | MUST NOT, | of the requirement| +| | | | MAY, | | +| | | | SHOULD, | | +| | | | SHOULD NOT| | ++-----------------+-------------+--------------+-----------+-------------------+ -``</path/to/validation-scripts/ice_validator>$ pytest --tap-stream --template-directory=<Directory>`` -where ``<Directory>`` is the full path to a folder containing heat templates. +.. _vvp-req-result: -Output -______ +Requirement Result +^^^^^^^^^^^^^^^^^^ -After performing a validation, an output folder will be created. +The file also includes an aggregated view of adherence to the VNF Requirements +validated by the validation scripts. Since some requirements have multiple +test cases, these results roll-up the result to an aggregated result for each +requirement. This section does not include detailed test results. If you +require detailed error information, then refer to the tests section of the +results. -``/path/to/validation-scripts/ice_validator/output/`` +**Example Requirement Result**: -This folder will contain a file ``report.html`` which contains a list of all -of the ONAP VNF Heat Template Guideline violations. If there are no violations, -the report will say ``No validation errors found.`` +.. code-block:: javascript -Interpreting the Output -_______________________ + { + "id": "R-16447", + "text": "A VNF's <resource ID> **MUST** be unique across all Heat\nOrchestration Templates and all HEAT Orchestration Template\nNested YAML files that are used to create the VNF.", + "keyword": "MUST", + "result": "FAIL" + "errors": [ + "The error message" + ] + } -The report file will have 4 columns for details about a violation, and one -row for each violation. Below contains details about each column. -File -#### +**Requirement Result Definition**: -This is the file(s) that contained the violation +.. csv-table:: + :header-rows: 1 + :file: vvp_json_req_result.csv -Error Message -############# -This shows the test and brief error message from the validation script that -contained the violation. There is a ``Full Details`` button to show the -complete raw test output. The error message will also contain details -about what element is involved with the violation (such as the parameter -name, resource id, etc...). +.. _vvp-docker-execution: -Requirement(s) -############## +Docker Execution +---------------- -This column contains the requirement(s) that each test/violation is -mapped to. These requirements are taken directly from the VNF Requirements -project Heat Orchestration Template Guidelines section. +A version of VVP is also provided as a Docker image. If your environment +supports Docker, then this eliminates the need to setup and install +the application from source code. +To execute from Docker, issue the following command where +``<Local Template Directory>`` is where the Heat templates are located and +``<Local Report Directory>`` is where you would like the reports to be +written on your local machine:: -Resolution Steps -################ - -For some violations, there are pre-defined resolution steps that -indicate what action the user should take to resolve the violation. - -**Note**: Not all violations will have resolution steps. Most violations -can be resolved simply by reviewing the requirements that have been violated -in the previous column. + docker run --rm -i -v ~/<Local Template Directory>/:/template \ + -v ~/<Local Report Directory>:/reports \ + onap/vvp/validation-scripts --template-directory=/template \ + --output-directory=/reports +The same :ref:`command line options <vvp-cmd-options>` can be used with the +Docker image that are used with the version from source. Self-Test Suite -_______________ +--------------- The ``ice_validator`` includes an extensive self-test suite. It is a **requirement** for any additions or changes to the test suite to @@ -111,3 +355,8 @@ the project root as: You can also run it under the folder ``ice_validator``: ``$ pytest --self-test`` + + + + + |