summaryrefslogtreecommitdiffstats
path: root/docs
diff options
context:
space:
mode:
authorLovett, Trevor <trevor.lovett@att.com>2019-02-15 18:17:14 -0600
committerLovett, Trevor <trevor.lovett@att.com>2019-02-16 08:47:41 -0600
commitbcbac1a1498b4f064fba575f7cbe8cb9b3663cd8 (patch)
tree8a185fa51017e0788162dec8ffa5d9d9d7d697cc /docs
parent023d74cc2d436659398743c412a602c6c1777fcb (diff)
[VVP] Doc for GUI, Docker, and Deprecation
Revising documentation to reflect the deprecation of the VVP web application as well as adding documentation for the new additions such as: * Alternate report options * Native GUI tool * Docker execution Change-Id: Ia1b1e99e7d5fb657286efb281049e4550a80d1b6 Issue-ID: VVP-137 Signed-off-by: Lovett, Trevor (tl2972) <tl2972@att.com>
Diffstat (limited to 'docs')
-rw-r--r--docs/administration.rst27
-rwxr-xr-xdocs/architecture.pngbin30004 -> 0 bytes
-rw-r--r--docs/architecture.rst77
-rw-r--r--docs/conf.py1
-rw-r--r--docs/configuration.rst9
-rw-r--r--docs/delivery.rst11
-rw-r--r--docs/humaninterfaces.rst113
-rw-r--r--docs/index.rst32
-rw-r--r--docs/installation.rst52
-rw-r--r--docs/logging.rst14
-rw-r--r--docs/offeredapis.rst10
-rw-r--r--docs/release-notes.rst44
-rw-r--r--docs/validation-scripts.rst379
-rw-r--r--docs/vvp_app.pngbin0 -> 38936 bytes
-rw-r--r--docs/vvp_json_header.csv17
-rw-r--r--docs/vvp_json_req_result.csv17
-rw-r--r--docs/vvp_json_test_result.csv17
17 files changed, 574 insertions, 246 deletions
diff --git a/docs/administration.rst b/docs/administration.rst
deleted file mode 100644
index 0fb3fe5..0000000
--- a/docs/administration.rst
+++ /dev/null
@@ -1,27 +0,0 @@
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. http://creativecommons.org/licenses/by/4.0
-.. Copyright 2017 AT&T Intellectual Property. All rights reserved.
-
-Administration
-================
-
-There is a functionality which part of Django framework to manage all
-models and tables located on our application's db.
-The django based containers are: engagement-manager, cms, ci tests
-
-Usually, the admin dashboard url is http(s)://{{domain}}/admin/
-Throght there you will see all models exists in the related application
-(in the shape of tables) and will be able to create/update/delete there rows.
-
-The VVP application allow you to login into portal as admin (usually created
-under admin@{{your-domain}}.com). The admin allow you to manage VVP entities,
-update templates and display all availiable engagements and VFs.
-You can reach the screen by clicking the user bubble on top-right
-of the screen, and from the drop-down menu - click the 'Admin' button.
-
-Processes
------------
-
-
-Actions
----------
diff --git a/docs/architecture.png b/docs/architecture.png
deleted file mode 100755
index f65e1ae..0000000
--- a/docs/architecture.png
+++ /dev/null
Binary files differ
diff --git a/docs/architecture.rst b/docs/architecture.rst
deleted file mode 100644
index 671abc8..0000000
--- a/docs/architecture.rst
+++ /dev/null
@@ -1,77 +0,0 @@
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. http://creativecommons.org/licenses/by/4.0
-.. Copyright 2017 AT&T Intellectual Property. All rights reserved.
-
-
-Architecture
-===============
-
-The architecture of VVP based on micro-services. Each element has
-dedicated container.
-VVP uses rocket and kubernetes to manage the VVP system.
-There is a devkit component which allows you to render and deploy VVP.
-It's render and deploy the rockets with kuberenetes management (via quay.io).
-
-We store files on Ceph distributed storage system that includes 'RADOS Gateway'
-which allows us to store files in AWS S3 based API (buckets etc..).
-
-|image0|
-
-Architecture Alignment
-----------------------------
-
-How does this project fit into the rest of the ONAP Architecture?
-
- * The VNF Validation Program will utilize the architecture to
- validate VNFs against it.
-
-What other ONAP projects does this project depend on?
-
- * This project depends on SDC, VNF Validation Program, and VNF Requirements
-
-How does this align with external standards/specifications?
-
- * Are there dependencies with other open source projects?
- * OpenStack
-
-Interactions
--------------------
-
-This is the list of containers part of VVP application:
-
- * ci-uwsgi - end-to-end flow tests based on Seleniunm
- * cms-nginx - webserver of CMS
- * cms-uwsgi - backend uwsgi server which hosts django application
- * em-nginx - webserver of engagement manager
- * em-uwsgi - backend uwsgi server which hosts django application
- * ext-haproxy - load balancer for external transport
- * int-haproxy - load balancer for internal (container to container)
- transport
- * gitlab - holds all customers files in repos
- * imagescanner - scan for validity and viruses on users files
- * jenkins - run validation tasks
- * portal - run the UI/UX application AKA VVP-Portal
- * postgresql - store all data of engagement manager
- * redis - in memory key-value store for all project
- * celery - task queue manager which manage all validation tasks
-
-Repo name:
-
- * org.onap.vvp/devkit
- * org.onap.vvp/ansible-ice-bootstrap
- * org.onap.vvp/portal
- * org.onap.vvp/engagementmgr
- * org.onap.vvp/cms
- * org.onap.vvp/jenkins
- * org.onap.vvp/haproxy
- * org.onap.vvp/postgresql
- * org.onap.vvp/gitlab
- * org.onap.vvp/jeeves
- * org.onap.vvp/test-engine
- * org.onap.vvp/validation-scripts
- * org.onap.vvp/documentation
- * org.onap.vvp/image-scanner
-
-.. |image0| image:: architecture.png
- :width: 11in
- :height: 5in
diff --git a/docs/conf.py b/docs/conf.py
index 38b57a5..7e484bc 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -51,7 +51,6 @@ extensions = [
'sphinx.ext.doctest',
'sphinx.ext.graphviz',
'sphinx.ext.todo',
- 'sphinx.ext.pngmath',
'sphinx.ext.viewcode',
]
diff --git a/docs/configuration.rst b/docs/configuration.rst
deleted file mode 100644
index 03a4ae4..0000000
--- a/docs/configuration.rst
+++ /dev/null
@@ -1,9 +0,0 @@
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. http://creativecommons.org/licenses/by/4.0
-.. Copyright 2017 AT&T Intellectual Property. All rights reserved.
-
-Configuration
-================
-
-All the configuration is happening on vvp-devkit component.
-There you will define all related fields for each component in a dedicated .yaml file.
diff --git a/docs/delivery.rst b/docs/delivery.rst
deleted file mode 100644
index e722662..0000000
--- a/docs/delivery.rst
+++ /dev/null
@@ -1,11 +0,0 @@
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. http://creativecommons.org/licenses/by/4.0
-.. Copyright 2017 AT&T Intellectual Property. All rights reserved.
-
-Delivery
-==========
-
-Describe how functions are packaged into run-time components.
-For some components a block diagram may be useful.
-
-All the project deploy is under ansible files which rendered and deployed in devkit component.
diff --git a/docs/humaninterfaces.rst b/docs/humaninterfaces.rst
index 8f609f1..522c713 100644
--- a/docs/humaninterfaces.rst
+++ b/docs/humaninterfaces.rst
@@ -1,11 +1,110 @@
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
-.. Copyright 2017 AT&T Intellectual Property. All rights reserved.
+.. Copyright 2019 AT&T Intellectual Property. All rights reserved.
-Human Interfaces
-======================
+Graphical User Interface
+========================
-The project is to develop a validation program to provide assurance of VNF
-interoperability with ONAP. Obtaining a validation shall be a self-service
-activity and should be against a reference release of ONAP for use by the
-VNF provider & any other validation authority.
+If desired, a graphical user interface is also provided when the application
+has been :doc:`installed from source <installation>`. This can provide a
+convenient wrapper that enables users to perform validations without using
+extensive command-line usage.
+
+At this time the application can only be run from source, but in the future
+a packaged version may be provided.
+
+
+How to Start the Application
+----------------------------
+
+1. Ensure you have installed the application and its dependencies from
+ source as described in the :doc:`Installation chapter <installation>`
+
+2. Navigate to the ``ice_validator`` directory::
+
+ > cd <vvp-directory>/ice_validator
+
+3. Launch the gui using the ``vvp.py`` command::
+
+ > python vvp.py
+
+
+How to Use the Tool
+-------------------
+
+.. note::
+ The look-and-feel of the application will vary slightly depending
+ on the Operating System of the host machine. The screenshot below is how
+ the application looks on a Windows machine.
+
+**Sample Screenshot of VVP GUI Application**
+
+.. image:: vvp_app.png
+
+
+All configuration options available to the command-line version of the
+application are exposed as options on the left-hand side of the GUI.
+
+Additional Validation Categories
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This allows the base set of tests to be extended by selecting additional
+categories of tests. At this time, only one additional category is supported.
+
+This maps to the ``--category`` command-line option.
+
++----------------------+-------------------------------------------------------+
+| Category | Description |
++======================+=======================================================+
+| Environment | When selected, VVP will flag parameters in environment|
+| File Compliance | files that should be excluded per the ONAP Heat |
+| | requirements. De-selecting this can be useful when |
+| | when testing instantiation directly in OpenStack |
+| | without ONAP orchestration. This is equivalent to |
+| | specifying ``--category=environment_file`` from the |
+| | command-line. |
++----------------------+-------------------------------------------------------+
+
+Settings
+~~~~~~~~
+
+**Verbosity** - Controls the level of comparison output displayed in the
+underlying pytest validation output. Default maps to pytest ``-v`` level of
+verbosity.
+
+**Report Format** - Controls the format of the output report generated after
+validation. The options are: HTML (the default), Excel, and CSV. This is
+equivalent the ``--report-format`` command-line option with the exception,
+that the JSON format is not supported via the GUI.
+
+**Input Format** - Controls the expected format of the template files. This
+can either be a ZIP file containing the Heat templates or a directory
+containing the Heat templates. There is no ZIP file option for the command-line
+script at this time.
+
+**Halt on Basic Failures** - VVP deems certain tests as "base tests" which if
+failed have the potential to generate a large number of other errors. This
+would include tests such as validating that the Heat templates are valid
+YAML. If checked, the tool will immediately stop all other tests and show
+a report of this single failure. This can be useful in reducing the number
+of errors to sift through in these situations. De-selecting this option
+is the equivalent of specifying ``--continue-on-failure`` as a command-line option.
+
+Running Validations
+~~~~~~~~~~~~~~~~~~~
+
+1. Select the desired input format in the settings (ZIP or Directory)
+2. Select the [...] button next to the Template Location input box
+3. Select the directory or ZIP file containing the Heat templates, and then
+ click Open
+4. Once the input is selected, select the "Validate Templates" button to
+ start the validation process. The white box to the right will
+ display output as the validations are executed.
+5. Once validation is complete a summary of pass and fail will be written
+ to the output window, and a "View Report" option will appear on the
+ left-hand control panel.
+6. Select the "View Report" option, and the report will be opened in the
+ the appropriate application based on report format.
+7. If you have questions about report output, please refer to the
+ :ref:`Validation Report reference material <vvp-report>` for more
+ information.
diff --git a/docs/index.rst b/docs/index.rst
index e139e9d..906a352 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -1,26 +1,32 @@
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
-.. Copyright 2017 AT&T Intellectual Property. All rights reserved.
+.. Copyright 2019 AT&T Intellectual Property. All rights reserved.
VVP Documentation
-====================
+=================
-The project is to develop a validation program to provide assurance of
-VNF interoperability with ONAP. Obtaining a validation shall be a
-self-service activity and should be against a reference release of
-ONAP for use by the VNF provider & any other validation authority.
+The VNF Validation Platform (VVP) is an application to validate that `OpenStack
+Heat Templates <https://wiki.openstack.org/wiki/Heat>`__ comply with the
+ONAP requirements and guidelines documented in the :doc:`Heat section <../../../../../submodules/vnfrqts/requirements.git/docs/Chapter5/Heat/index>` of
+the ONAP's :doc:`VNF Requirements and Guidelines documentation <../../../../../guides/onap-provider/index>`.
+Adherence to these guidelines ensures that a VNF can be successfully
+onboarded, modeled, instantiated, and orchestrated by ONAP to the fullest
+extent possible.
+
+VVP is a utility written in Python that can be executed via a
+:doc:`command-line script <validation-scripts>`,
+:ref:`Docker container <vvp-docker-execution>`, or a
+native :doc:`Desktop GUI application <humaninterfaces>` to analyze and report
+on the compliance of a given set of Heat templates to the ONAP requirements.
+
+This guide will provide the user with instructions on how to acquire,
+setup, and execute the validations.
.. toctree::
:maxdepth: 1
+ installation.rst
validation-scripts.rst
- administration.rst
- architecture.rst
- configuration.rst
- delivery.rst
humaninterfaces.rst
- installation.rst
- logging.rst
- offeredapis.rst
release-notes.rst
diff --git a/docs/installation.rst b/docs/installation.rst
index 0353ec4..a20bbed 100644
--- a/docs/installation.rst
+++ b/docs/installation.rst
@@ -1,14 +1,54 @@
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
-.. Copyright 2017 AT&T Intellectual Property. All rights reserved.
+.. Copyright 2019 AT&T Intellectual Property. All rights reserved.
+
+.. _vvp-installation:
Installation
-==================
+============
+
+The VNF Validation Platform (VVP) can be run from source as a normal
+Python executable or a Docker image is provided. This section will
+describe how to setup and install the ``vvp/validation-scripts`` and
+run from source.
-Installation happens through the vvp-devkit project.
+Installation and configuration of Docker is beyond the scope of this document,
+but you can refer to the :ref:`Docker Execution <vvp-docker-execution>`
+instructions for more details on running the validations as from
+the Docker image.
-Environment
+Pre-requisites
--------------
-Steps
-----------
+This document assumes you have the following system-level utilities
+installed.
+
+Please refer to the respective sites for these tools for the appropriate
+installation instructions for your given operating system.
+
+* `Python 3.6+ <https://www.python.org/downloads/>`__
+* `Git <https://git-scm.com/>`__
+
+Setup
+------
+
+The source code for VVP can be obtained from the `ONAP Gerrit site <https://gerrit.onap.org/r/#/admin/projects/vvp/validation-scripts>`__
+or its `GitHub mirror <https://github.com/onap/vvp-validation-scripts>`__.
+
+1. Clone the source from your desired repository host:
+
+ Choose **one** of the following ``git clone`` commands::
+
+ > git clone https://github.com/onap/vvp-validation-scripts.git
+ > git clone https://gerrit.onap.org/r/vvp/validation-scripts
+
+2. (*Optional*) If desired, you can create a virtual Python environment to
+ avoid installing VVP's dependencies in your system level installation
+ of Python::
+
+ > python -m venv vvp
+ > source vvp/activate
+
+3. Install the required dependencies with the following command::
+
+ > python pip install -r requirements.txt
diff --git a/docs/logging.rst b/docs/logging.rst
deleted file mode 100644
index 4c795b2..0000000
--- a/docs/logging.rst
+++ /dev/null
@@ -1,14 +0,0 @@
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. http://creativecommons.org/licenses/by/4.0
-.. Copyright 2017 AT&T Intellectual Property. All rights reserved.
-
-Logging
-==========
-Login into portal/em admin/cms admin
-
-Where to Access Information
---------------------------------
-
-
-Error / Warning Messages
-----------------------------
diff --git a/docs/offeredapis.rst b/docs/offeredapis.rst
deleted file mode 100644
index 20a696b..0000000
--- a/docs/offeredapis.rst
+++ /dev/null
@@ -1,10 +0,0 @@
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. http://creativecommons.org/licenses/by/4.0
-.. Copyright 2017 AT&T Intellectual Property. All rights reserved.
-
-
-Offered APIs
-==============
-
-
-None Available at this time
diff --git a/docs/release-notes.rst b/docs/release-notes.rst
index 2e9c394..e5876a9 100644
--- a/docs/release-notes.rst
+++ b/docs/release-notes.rst
@@ -1,16 +1,47 @@
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
-.. Copyright 2017 AT&T Intellectual Property. All rights reserved.
+.. Copyright 2019 AT&T Intellectual Property. All rights reserved.
Release Notes
===============================
-.. note::
- VNF Onboarding is a challenge across the industry because of a lack of a
- standard form for VNFs.
- The VVP project was created to provide validation scripts for
- VNFs to test validity against the ONAP Requirements.
+
+Version: 4.0.0 (Dublin)
+-----------------------
+
+:Release Date: TBD
+
+**Removed Features**
+
+- The VVP web application has been deprecated and is no longer supported
+ as of the Dublin release. The validation scripts continue to be supported
+ and enhanced, but contributions to the web-related repositories are now locked
+ and VVP will no longer be supported for deployment via ONAP Operations
+ Manager (OOM).
+
+ The following repositories are now locked as of this release:
+
+ - ``vvp/ansibile-ice-bootstrap``
+ - ``vvp/cms``
+ - ``vvp/devkit``
+ - ``vvp/engagementmgr``
+ - ``vvp/gitlab``
+ - ``vvp/image-scanner``
+ - ``vvp/jenkins``
+ - ``vvp/portal``
+ - ``vvp/postgresql``
+ - ``vvp/test-engine``
+
+**New Features**
+
+- A new :doc:`GUI application <humaninterfaces>` has been contributed and can
+ be used to execute validations in a user-friendly way without using complex
+ command line options.
+- VVP is now packaged as a Docker container eliminating the need to run the
+ application from source code. See the :ref:`Docker Execution <vvp-docker-execution>`
+ instructions for more details.
+
Version: 3.0.0
--------------
@@ -70,6 +101,7 @@ Quick Links:
- Initial release - none
**Other**
+
NA
===========
diff --git a/docs/validation-scripts.rst b/docs/validation-scripts.rst
index 0ff6e32..b327ad4 100644
--- a/docs/validation-scripts.rst
+++ b/docs/validation-scripts.rst
@@ -1,105 +1,349 @@
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
-.. Copyright 2018 AT&T Intellectual Property. All rights reserved.
+.. Copyright 2019 AT&T Intellectual Property. All rights reserved.
-Manual Heat Template Validation
-===============================
+Heat Validation via Command Line
+================================
-validation-scripts
-------------------
+.. note::
+ This section assumes you have already acquired and setup the validation
+ scripts as described in the :ref:`VVP Installation <vvp-installation>`
+ section.
-This project contains validation scripts to test
-that a set of Heat Templates adheres to
-the ONAP VNF Heat Orchestration Template guidelines.
+Basic Script Execution
+----------------------
-For more information on the ONAP Heat Orchestration
-Template Guidelines, vist the `Heat Guidelines <https://onap.readthedocs.io/en/latest/submodules/vnfrqts/requirements.git/docs/Chapter5/Heat/index.html>`__
+1. The scripts must be executed from the ``ice_validator`` directory.
+ Navigate to the directory first::
-About
-_____
+ > cd <vvp-directory>/ice_validator
-The validation scripts project allows performing heat template
-validation without installing the full VVP platform. The following
-instructions apply to running these validation scripts in that manner.
+2. To run the script with default options, issue the following command where
+ ``<Directory>`` is the directory containing the Heat templates::
+
+ /path/to/validation-scripts/ice_validator>$ pytest tests --template-directory=<Directory>``
+
+3. Test results will be written to the consoles describing any failures that
+ are encountered. If no failures are found, then a success message will
+ be displayed. Additional reports can be found in the
+ ``<VVP Directory>/ice_validator/output`` directory. See
+ :ref:`Reports <vvp-reports>` for more details.
+
+
+.. _vvp-cmd-options:
+
+Command Line Options
+--------------------
+
+Additional options can be specified at the command line to customize output
+and execution. VVP uses the `pytest <https://docs.pytest.org>`__
+framework for test execution, and therefore all pytest options are available.
+
+Please refer to the pytest documentation for information on the command-line
+options that framework provides. We will only document the specific options
+that are used by VVP::
+
+ --template-directory=TEMPLATE_DIR
+ Directory which holds the templates for validation
+
+ --category=CATEGORY_NAME
+ Additional validations that can be selected.
+ The only supported value at this time is
+ environment_file which will enforce that specific
+ parameter values are excluded from the environment
+ file per ONAP Heat requirements.
+
+ --self-test Test the unit tests against their fixtured data
+
+ --report-format=REPORT_FORMAT
+ Format of output report (html, csv, excel, json)
+
+ --continue-on-failure
+ Continue validation even when structural errors exist
+ in input files
+
+ --output-directory=OUTPUT_DIR
+ Alternate directory for report output.
+
+.. _vvp-reports:
+
+VVP Reports
+-----------
+
+After completion of the validations, several reports will be written to the
+ouput directory (specified via ``--output-directory`` or
+``<vvp-directory>/ice_validator/output`` by default).
+
+
++-----------------------+------------------------------------------------------+
+| Report | Description |
++=======================+======================================================+
+| ``report.{html,csv,`` | Full report of validation results showing the pass |
+| ``xlsx,json}``| or failure of the validation and details on all |
+| | errors. See :ref:`Validation Report <vvp-report>` |
+| | for information on the HTML, CSV, and Excel versions |
+| | of the report, or |
+| | :ref:`JSON Report <vvp-json-report>` for details on |
+| | the machine-readable results. |
++-----------------------+------------------------------------------------------+
+| ``traceability.csv`` | This shows a mapping from Heat requirement to |
+| | test case. |
++-----------------------+------------------------------------------------------+
+| ``mapping_errors.csv``| File should be empty, but if present shows tests |
+| | that are mapped to requirements that no longer exist |
++-----------------------+------------------------------------------------------+
+| ``traceability.rst`` | Similar information to ``traceability.csv``, but |
+| | in reStructuredText for use in the VNF Requirements |
+| | documentation. |
++-----------------------+------------------------------------------------------+
+| ``failures`` | **Deprecated** JSON version of test failures. Use |
+| | ``report.json`` instead. |
++-----------------------+------------------------------------------------------+
+
+.. _vvp-report:
+
+Validation Report
+~~~~~~~~~~~~~~~~~
+
+If the report format of ``html`` (*default*), ``excel``, or ``csv`` are
+requested via the ``--report-format`` option, then the a report file will
+be written to the output directory. Regardless of format, the file will contain
+a header section that summarizes the results and files scanned, and an error
+section that has a row for each failure with four columns.
+
+Header Information
+^^^^^^^^^^^^^^^^^^
+
++-----------------------+------------------------------------------------------+
+| Header Element | Description |
++=======================+======================================================+
+| Categories Selected | Any additional categories selected via ``--category``|
++-----------------------+------------------------------------------------------+
+| Tool Version | Version of the tool that produced the report |
++-----------------------+------------------------------------------------------+
+| Report Generated At | The timestamp of when the report was generated |
++-----------------------+------------------------------------------------------+
+| Directory Validated | Absolute path to directory validated |
++-----------------------+------------------------------------------------------+
+| Checksum | Unique MD5 has of the contents of template directory |
++-----------------------+------------------------------------------------------+
+| Total Errors | Number of errors/violations found |
++-----------------------+------------------------------------------------------+
+
+
+Error Information
+^^^^^^^^^^^^^^^^^
+
+If any violations are found, then there will be one row for each violation
+with the following columns:
+
++-----------------------+------------------------------------------------------+
+| Column Name | Description |
++=======================+======================================================+
+| Files | The file or files that were scanned as part of the |
+| | test. |
++-----------------------+------------------------------------------------------+
+| Tests | Name of the test case (not shown in HTML version) |
++-----------------------+------------------------------------------------------+
+| Error Message | This shows the test and brief error message from the |
+| | test that failed. This will contain details about |
+| | the element that triggered the violation such as the |
+| | parameter name, resource ID, etc. |
+| | |
+| | In the HTML version of the report this column will |
+| | also show the test case name, and provide a link to |
+| | ``Full Details`` the raw output of the test |
++-----------------------+------------------------------------------------------+
+| Requirements | The requirement ID and text that was violated |
++-----------------------+------------------------------------------------------+
+| Resolution Steps | For some violations, there are pre-defined resolution|
+| | steps that indicate what action the user should take |
+| | to resolve the violation. |
+| | |
+| | **Note**: Not all violations will have resolution |
+| | steps, rather the error message and requirement is |
+| | sufficient. |
++-----------------------+------------------------------------------------------+
+| Raw Test Output | Full output from the pytest test case. This not a |
+| | dedicated column in the HTML version of the report. |
++-----------------------+------------------------------------------------------+
+
+
+.. _vvp-json-report:
+
+JSON Report
+~~~~~~~~~~~
+
+This report is intended to provide a machine-readable version of the test
+execution, and provides the most comprehensive summary of the test execution
+and results.
+
+File Header/Top Level
+^^^^^^^^^^^^^^^^^^^^^
+
+JSON Report <vvp-json-report>
+The top level will include a summary of available execution metadata.
+
+NOTE: The ``tests`` and ``requirements`` entries are elided in the
+example below.
+
+**Example Header**:
+
+.. code-block:: javascript
+
+ {
+ "version": "dublin",
+ "template_directory": "/path/to/template",
+ "timestamp": "2019-01-21T02:11:07.305000",
+ "checksum": "6296aa211870634f9b4a23477c5eab28",
+ "profile": "",
+ "outcome": "FAIL",
+ "tests": [],
+ "requirements": [],
+ }
+
+**Header Definition**:
+
+.. csv-table::
+ :header-rows: 1
+ :file: vvp_json_header.csv
+
+.. _vvp-test-result:
+
+Test Result
+^^^^^^^^^^^
+
+For each test result a JSON object will be provided that informs the consumer
+what tests was run, its result, and the requirements it validated.
+
+**Example Test Result**:
+
+.. code-block:: javascript
+
+ {
+ "files": [
+ "/Users/username/Desktop/stark_template2/STARKDB-nested-1.yaml",
+ "/Users/username/Desktop/stark_template2/base_starkdb.yaml",
+ ],
+ "test_module": "test_resource_indices",
+ "test_case": "test_indices_start_at_0_increment",
+ "result": "FAIL",
+ "error": " Index values associated with resource ID prefix STARKDB_server_ do not start at 0\n",
+ "requirements": [
+ {
+ "id": "R-11690",
+ "text": "When a VNF's Heat Orchestration Template's Resource ID contains an\n``{index}``, the ``{index}`` is a numeric value that **MUST** start at\nzero and **MUST** increment by one.\n\nAs stated in R-16447,\n*a VNF's <resource ID> MUST be unique across all Heat\nOrchestration Templates and all HEAT Orchestration Template\nNested YAML files that are used to create the VNF*. While the ``{index}``\nwill start at zero in the VNF, the ``{index}`` may not start at zero\nin a given Heat Orchestration Template or HEAT Orchestration Template\nNested YAML file.",
+ "keyword": "MUST"
+ }
+ ]
+ }
+
+
+**Test Result Definition**:
+
+.. csv-table::
+ :header-rows: 1
+ :file: vvp_json_test_result.csv
-Installation
-____________
+.. _vvp-req-metadata:
-This software is not platform dependent and can be run in a Windows, Unix or
-OS X environment.
+Requirement Metadata
+^^^^^^^^^^^^^^^^^^^^
-Satisfy Dependencies
-####################
+For each test case, the following requirement metadata will be reported.
+**Example Requirement Metadata**:
- These can be installed using pip (assuming pip is installed) with the command:
+.. code-block:: javascript
-``$ pip install -r requirements.txt``
+ {
+ "id": "R-11690",
+ "text": "When a VNF's Heat Orchestration Template's Resource ID contains an\n``{index}``, the ``{index}`` is a numeric value that **MUST** start at\nzero and **MUST** increment by one.\n\nAs stated in R-16447,\n*a VNF's <resource ID> MUST be unique across all Heat\nOrchestration Templates and all HEAT Orchestration Template\nNested YAML files that are used to create the VNF*. While the ``{index}``\nwill start at zero in the VNF, the ``{index}`` may not start at zero\nin a given Heat Orchestration Template or HEAT Orchestration Template\nNested YAML file.",
+ "keyword": "MUST"
+ }
-Use
-___
-Clone this project.
+**Requirement Metadata Definition**:
-To validate Heat templates just run this the command under the folder ``ice_validator``:
++-----------------+-------------+--------------+-----------+-------------------+
+| Field Name | Required | Data Type | Valid | Description |
+| | Optional | | Values | |
+| | Conditional | | | |
++=================+=============+==============+===========+===================+
+| ``id`` | Required | ``string`` | | Requirement ID |
+| | | | | from the VNFRQTS |
+| | | | | project |
++-----------------+-------------+--------------+-----------+-------------------+
+| ``text`` | Required | ``string`` | | Full text of |
+| | | | | requirement |
++-----------------+-------------+--------------+-----------+-------------------+
+| ``keyword`` | Required | ``string`` | MUST, | RFC 2119 keyword |
+| | | | MUST NOT, | of the requirement|
+| | | | MAY, | |
+| | | | SHOULD, | |
+| | | | SHOULD NOT| |
++-----------------+-------------+--------------+-----------+-------------------+
-``</path/to/validation-scripts/ice_validator>$ pytest --tap-stream --template-directory=<Directory>``
-where ``<Directory>`` is the full path to a folder containing heat templates.
+.. _vvp-req-result:
-Output
-______
+Requirement Result
+^^^^^^^^^^^^^^^^^^
-After performing a validation, an output folder will be created.
+The file also includes an aggregated view of adherence to the VNF Requirements
+validated by the validation scripts. Since some requirements have multiple
+test cases, these results roll-up the result to an aggregated result for each
+requirement. This section does not include detailed test results. If you
+require detailed error information, then refer to the tests section of the
+results.
-``/path/to/validation-scripts/ice_validator/output/``
+**Example Requirement Result**:
-This folder will contain a file ``report.html`` which contains a list of all
-of the ONAP VNF Heat Template Guideline violations. If there are no violations,
-the report will say ``No validation errors found.``
+.. code-block:: javascript
-Interpreting the Output
-_______________________
+ {
+ "id": "R-16447",
+ "text": "A VNF's <resource ID> **MUST** be unique across all Heat\nOrchestration Templates and all HEAT Orchestration Template\nNested YAML files that are used to create the VNF.",
+ "keyword": "MUST",
+ "result": "FAIL"
+ "errors": [
+ "The error message"
+ ]
+ }
-The report file will have 4 columns for details about a violation, and one
-row for each violation. Below contains details about each column.
-File
-####
+**Requirement Result Definition**:
-This is the file(s) that contained the violation
+.. csv-table::
+ :header-rows: 1
+ :file: vvp_json_req_result.csv
-Error Message
-#############
-This shows the test and brief error message from the validation script that
-contained the violation. There is a ``Full Details`` button to show the
-complete raw test output. The error message will also contain details
-about what element is involved with the violation (such as the parameter
-name, resource id, etc...).
+.. _vvp-docker-execution:
-Requirement(s)
-##############
+Docker Execution
+----------------
-This column contains the requirement(s) that each test/violation is
-mapped to. These requirements are taken directly from the VNF Requirements
-project Heat Orchestration Template Guidelines section.
+A version of VVP is also provided as a Docker image. If your environment
+supports Docker, then this eliminates the need to setup and install
+the application from source code.
+To execute from Docker, issue the following command where
+``<Local Template Directory>`` is where the Heat templates are located and
+``<Local Report Directory>`` is where you would like the reports to be
+written on your local machine::
-Resolution Steps
-################
-
-For some violations, there are pre-defined resolution steps that
-indicate what action the user should take to resolve the violation.
-
-**Note**: Not all violations will have resolution steps. Most violations
-can be resolved simply by reviewing the requirements that have been violated
-in the previous column.
+ docker run --rm -i -v ~/<Local Template Directory>/:/template \
+ -v ~/<Local Report Directory>:/reports \
+ onap/vvp/validation-scripts --template-directory=/template \
+ --output-directory=/reports
+The same :ref:`command line options <vvp-cmd-options>` can be used with the
+Docker image that are used with the version from source.
Self-Test Suite
-_______________
+---------------
The ``ice_validator`` includes an extensive self-test suite. It is a
**requirement** for any additions or changes to the test suite to
@@ -111,3 +355,8 @@ the project root as:
You can also run it under the folder ``ice_validator``:
``$ pytest --self-test``
+
+
+
+
+
diff --git a/docs/vvp_app.png b/docs/vvp_app.png
new file mode 100644
index 0000000..8a27249
--- /dev/null
+++ b/docs/vvp_app.png
Binary files differ
diff --git a/docs/vvp_json_header.csv b/docs/vvp_json_header.csv
new file mode 100644
index 0000000..5827f8b
--- /dev/null
+++ b/docs/vvp_json_header.csv
@@ -0,0 +1,17 @@
+Field Name,"Required/
+Optional/
+Conditional",Data Type,Valid Values,Description
+``checksum``,Required,``string``,,MD5 hash of all file contents in the ``template_directory``
+``outcome``,Required,``string``,"``PASS``
+``FAIL``
+``ERROR``","One of the valid values:
+
+* ``PASS`` - All tests passed successfully (some may have been skipped as not applicable based on the contents of the template)
+* ``FAIL`` - At least one test failed. In this scenario the templates will need to be corrected to comply with the requirements
+* ``ERROR`` - An unexpected error occurred during test setup. Some or all tests may have not executed. Issue should be referred to the VVP team for investigation."
+``categories``,Optional,array of ``string``,,Categories selected via the ``--category`` option.
+``template_directory``,Required,``string``,,Absolute path of the directory containing the Heat templates that were validated
+``version``,Required,``string``,,Version of the validation scripts that produced the report
+``timestamp``,Required,``string``,,ISO 8601 Timestamp in UTC of when the report was generated
+``tests``,Required,List of :ref:`Test Result <vvp-test-result>`,,See :ref:`Test Result <vvp-test-result>`
+``requirements``,Required,List of :ref:`Requirement Result <vvp-req-result>`,,See :ref:`Requirement Result <vvp-req-result>`
diff --git a/docs/vvp_json_req_result.csv b/docs/vvp_json_req_result.csv
new file mode 100644
index 0000000..42b96b0
--- /dev/null
+++ b/docs/vvp_json_req_result.csv
@@ -0,0 +1,17 @@
+Field Name,Required/Optional/Conditional,Data Type,Valid Values,Description
+``id``,Required,``string``,,"Requirement ID from the VNFRQTS project.
+
+**NOTE**:a requirement ID of ""Unmapped"" may be included if one or more tests are not mapped to a requirement."
+``text``,Required,``string``,,Full text of the requirement.
+``keyword``,Required,``string``,"``MUST``,
+``MUST NOT``,
+``MAY``,
+``""SHOULD""``,
+``""SHOULD NOT""``",RFC 2119 keyword associated with the requirement
+``result``,Required,``string``,"``PASS``, ``SKIP``, ``FAIL``, ``ERROR``","One of the valid values:
+
+* ``PASS`` - The test case passed with no violations
+* ``SKIP`` - The test case was skipped because it was deemed not applicable
+* ``FAIL`` - The test case completed, but it found a violation
+* ``ERROR`` - An unexpected error was found while setting up the test case"
+``errors``,Required,List of ``string``,,Error messages associated with this requirement. This will be an empty string if the result is ``PASS`` or ``SKIP``
diff --git a/docs/vvp_json_test_result.csv b/docs/vvp_json_test_result.csv
new file mode 100644
index 0000000..14982c0
--- /dev/null
+++ b/docs/vvp_json_test_result.csv
@@ -0,0 +1,17 @@
+Field Name,Required/Optional/Conditional,Data Type,Valid Values,Description
+``files``,Required,list of ``string``,,"List of files that were passed to the test case.
+
+**NOTE**: If ``result`` is ``ERROR`` this may be an empty list"
+``test_module``,Required,``string``,,Name of module/file anme that contains the test case
+``test_case``,Required,``string``,,Name of the test case
+``result``,Required,``string``,"* ``PASS``
+* ``SKIP``
+* ``FAIL``
+* ``ERROR``","One of the valid values:
+
+* ``PASS`` - The test case passed with no violations
+* ``SKIP`` - The test case was deemed not applicable
+* ``FAIL`` - The test case completed, but a violation was found
+* ``ERROR`` - An unexpected error was found while setting up the test case"
+``error``,Required,``string``,,"If the test failed or encountered an error, then ths will be a message summarizing the error. If the test passed or was skipped, then this will be an empty string"
+``requirements``,Required,List of :ref:`Requirement Metadata <vvp-req-metadata>`,,See :ref:`Requirement Metadata <vvp-req-metadata>`