From bcbac1a1498b4f064fba575f7cbe8cb9b3663cd8 Mon Sep 17 00:00:00 2001 From: "Lovett, Trevor" Date: Fri, 15 Feb 2019 18:17:14 -0600 Subject: [VVP] Doc for GUI, Docker, and Deprecation Revising documentation to reflect the deprecation of the VVP web application as well as adding documentation for the new additions such as: * Alternate report options * Native GUI tool * Docker execution Change-Id: Ia1b1e99e7d5fb657286efb281049e4550a80d1b6 Issue-ID: VVP-137 Signed-off-by: Lovett, Trevor (tl2972) --- docs/administration.rst | 27 --- docs/architecture.png | Bin 30004 -> 0 bytes docs/architecture.rst | 77 --------- docs/conf.py | 1 - docs/configuration.rst | 9 - docs/delivery.rst | 11 -- docs/humaninterfaces.rst | 113 ++++++++++++- docs/index.rst | 32 ++-- docs/installation.rst | 52 +++++- docs/logging.rst | 14 -- docs/offeredapis.rst | 10 -- docs/release-notes.rst | 44 ++++- docs/validation-scripts.rst | 379 ++++++++++++++++++++++++++++++++++-------- docs/vvp_app.png | Bin 0 -> 38936 bytes docs/vvp_json_header.csv | 17 ++ docs/vvp_json_req_result.csv | 17 ++ docs/vvp_json_test_result.csv | 17 ++ 17 files changed, 574 insertions(+), 246 deletions(-) delete mode 100644 docs/administration.rst delete mode 100755 docs/architecture.png delete mode 100644 docs/architecture.rst delete mode 100644 docs/configuration.rst delete mode 100644 docs/delivery.rst delete mode 100644 docs/logging.rst delete mode 100644 docs/offeredapis.rst create mode 100644 docs/vvp_app.png create mode 100644 docs/vvp_json_header.csv create mode 100644 docs/vvp_json_req_result.csv create mode 100644 docs/vvp_json_test_result.csv (limited to 'docs') diff --git a/docs/administration.rst b/docs/administration.rst deleted file mode 100644 index 0fb3fe5..0000000 --- a/docs/administration.rst +++ /dev/null @@ -1,27 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. Copyright 2017 AT&T Intellectual Property. All rights reserved. - -Administration -================ - -There is a functionality which part of Django framework to manage all -models and tables located on our application's db. -The django based containers are: engagement-manager, cms, ci tests - -Usually, the admin dashboard url is http(s)://{{domain}}/admin/ -Throght there you will see all models exists in the related application -(in the shape of tables) and will be able to create/update/delete there rows. - -The VVP application allow you to login into portal as admin (usually created -under admin@{{your-domain}}.com). The admin allow you to manage VVP entities, -update templates and display all availiable engagements and VFs. -You can reach the screen by clicking the user bubble on top-right -of the screen, and from the drop-down menu - click the 'Admin' button. - -Processes ------------ - - -Actions ---------- diff --git a/docs/architecture.png b/docs/architecture.png deleted file mode 100755 index f65e1ae..0000000 Binary files a/docs/architecture.png and /dev/null differ diff --git a/docs/architecture.rst b/docs/architecture.rst deleted file mode 100644 index 671abc8..0000000 --- a/docs/architecture.rst +++ /dev/null @@ -1,77 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. Copyright 2017 AT&T Intellectual Property. All rights reserved. - - -Architecture -=============== - -The architecture of VVP based on micro-services. Each element has -dedicated container. -VVP uses rocket and kubernetes to manage the VVP system. -There is a devkit component which allows you to render and deploy VVP. -It's render and deploy the rockets with kuberenetes management (via quay.io). - -We store files on Ceph distributed storage system that includes 'RADOS Gateway' -which allows us to store files in AWS S3 based API (buckets etc..). - -|image0| - -Architecture Alignment ----------------------------- - -How does this project fit into the rest of the ONAP Architecture? - - * The VNF Validation Program will utilize the architecture to - validate VNFs against it. - -What other ONAP projects does this project depend on? - - * This project depends on SDC, VNF Validation Program, and VNF Requirements - -How does this align with external standards/specifications? - - * Are there dependencies with other open source projects? - * OpenStack - -Interactions -------------------- - -This is the list of containers part of VVP application: - - * ci-uwsgi - end-to-end flow tests based on Seleniunm - * cms-nginx - webserver of CMS - * cms-uwsgi - backend uwsgi server which hosts django application - * em-nginx - webserver of engagement manager - * em-uwsgi - backend uwsgi server which hosts django application - * ext-haproxy - load balancer for external transport - * int-haproxy - load balancer for internal (container to container) - transport - * gitlab - holds all customers files in repos - * imagescanner - scan for validity and viruses on users files - * jenkins - run validation tasks - * portal - run the UI/UX application AKA VVP-Portal - * postgresql - store all data of engagement manager - * redis - in memory key-value store for all project - * celery - task queue manager which manage all validation tasks - -Repo name: - - * org.onap.vvp/devkit - * org.onap.vvp/ansible-ice-bootstrap - * org.onap.vvp/portal - * org.onap.vvp/engagementmgr - * org.onap.vvp/cms - * org.onap.vvp/jenkins - * org.onap.vvp/haproxy - * org.onap.vvp/postgresql - * org.onap.vvp/gitlab - * org.onap.vvp/jeeves - * org.onap.vvp/test-engine - * org.onap.vvp/validation-scripts - * org.onap.vvp/documentation - * org.onap.vvp/image-scanner - -.. |image0| image:: architecture.png - :width: 11in - :height: 5in diff --git a/docs/conf.py b/docs/conf.py index 38b57a5..7e484bc 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -51,7 +51,6 @@ extensions = [ 'sphinx.ext.doctest', 'sphinx.ext.graphviz', 'sphinx.ext.todo', - 'sphinx.ext.pngmath', 'sphinx.ext.viewcode', ] diff --git a/docs/configuration.rst b/docs/configuration.rst deleted file mode 100644 index 03a4ae4..0000000 --- a/docs/configuration.rst +++ /dev/null @@ -1,9 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. Copyright 2017 AT&T Intellectual Property. All rights reserved. - -Configuration -================ - -All the configuration is happening on vvp-devkit component. -There you will define all related fields for each component in a dedicated .yaml file. diff --git a/docs/delivery.rst b/docs/delivery.rst deleted file mode 100644 index e722662..0000000 --- a/docs/delivery.rst +++ /dev/null @@ -1,11 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. Copyright 2017 AT&T Intellectual Property. All rights reserved. - -Delivery -========== - -Describe how functions are packaged into run-time components. -For some components a block diagram may be useful. - -All the project deploy is under ansible files which rendered and deployed in devkit component. diff --git a/docs/humaninterfaces.rst b/docs/humaninterfaces.rst index 8f609f1..522c713 100644 --- a/docs/humaninterfaces.rst +++ b/docs/humaninterfaces.rst @@ -1,11 +1,110 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. Copyright 2017 AT&T Intellectual Property. All rights reserved. +.. Copyright 2019 AT&T Intellectual Property. All rights reserved. -Human Interfaces -====================== +Graphical User Interface +======================== -The project is to develop a validation program to provide assurance of VNF -interoperability with ONAP. Obtaining a validation shall be a self-service -activity and should be against a reference release of ONAP for use by the -VNF provider & any other validation authority. +If desired, a graphical user interface is also provided when the application +has been :doc:`installed from source `. This can provide a +convenient wrapper that enables users to perform validations without using +extensive command-line usage. + +At this time the application can only be run from source, but in the future +a packaged version may be provided. + + +How to Start the Application +---------------------------- + +1. Ensure you have installed the application and its dependencies from + source as described in the :doc:`Installation chapter ` + +2. Navigate to the ``ice_validator`` directory:: + + > cd /ice_validator + +3. Launch the gui using the ``vvp.py`` command:: + + > python vvp.py + + +How to Use the Tool +------------------- + +.. note:: + The look-and-feel of the application will vary slightly depending + on the Operating System of the host machine. The screenshot below is how + the application looks on a Windows machine. + +**Sample Screenshot of VVP GUI Application** + +.. image:: vvp_app.png + + +All configuration options available to the command-line version of the +application are exposed as options on the left-hand side of the GUI. + +Additional Validation Categories +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This allows the base set of tests to be extended by selecting additional +categories of tests. At this time, only one additional category is supported. + +This maps to the ``--category`` command-line option. + ++----------------------+-------------------------------------------------------+ +| Category | Description | ++======================+=======================================================+ +| Environment | When selected, VVP will flag parameters in environment| +| File Compliance | files that should be excluded per the ONAP Heat | +| | requirements. De-selecting this can be useful when | +| | when testing instantiation directly in OpenStack | +| | without ONAP orchestration. This is equivalent to | +| | specifying ``--category=environment_file`` from the | +| | command-line. | ++----------------------+-------------------------------------------------------+ + +Settings +~~~~~~~~ + +**Verbosity** - Controls the level of comparison output displayed in the +underlying pytest validation output. Default maps to pytest ``-v`` level of +verbosity. + +**Report Format** - Controls the format of the output report generated after +validation. The options are: HTML (the default), Excel, and CSV. This is +equivalent the ``--report-format`` command-line option with the exception, +that the JSON format is not supported via the GUI. + +**Input Format** - Controls the expected format of the template files. This +can either be a ZIP file containing the Heat templates or a directory +containing the Heat templates. There is no ZIP file option for the command-line +script at this time. + +**Halt on Basic Failures** - VVP deems certain tests as "base tests" which if +failed have the potential to generate a large number of other errors. This +would include tests such as validating that the Heat templates are valid +YAML. If checked, the tool will immediately stop all other tests and show +a report of this single failure. This can be useful in reducing the number +of errors to sift through in these situations. De-selecting this option +is the equivalent of specifying ``--continue-on-failure`` as a command-line option. + +Running Validations +~~~~~~~~~~~~~~~~~~~ + +1. Select the desired input format in the settings (ZIP or Directory) +2. Select the [...] button next to the Template Location input box +3. Select the directory or ZIP file containing the Heat templates, and then + click Open +4. Once the input is selected, select the "Validate Templates" button to + start the validation process. The white box to the right will + display output as the validations are executed. +5. Once validation is complete a summary of pass and fail will be written + to the output window, and a "View Report" option will appear on the + left-hand control panel. +6. Select the "View Report" option, and the report will be opened in the + the appropriate application based on report format. +7. If you have questions about report output, please refer to the + :ref:`Validation Report reference material ` for more + information. diff --git a/docs/index.rst b/docs/index.rst index e139e9d..906a352 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,26 +1,32 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. Copyright 2017 AT&T Intellectual Property. All rights reserved. +.. Copyright 2019 AT&T Intellectual Property. All rights reserved. VVP Documentation -==================== +================= -The project is to develop a validation program to provide assurance of -VNF interoperability with ONAP. Obtaining a validation shall be a -self-service activity and should be against a reference release of -ONAP for use by the VNF provider & any other validation authority. +The VNF Validation Platform (VVP) is an application to validate that `OpenStack +Heat Templates `__ comply with the +ONAP requirements and guidelines documented in the :doc:`Heat section <../../../../../submodules/vnfrqts/requirements.git/docs/Chapter5/Heat/index>` of +the ONAP's :doc:`VNF Requirements and Guidelines documentation <../../../../../guides/onap-provider/index>`. +Adherence to these guidelines ensures that a VNF can be successfully +onboarded, modeled, instantiated, and orchestrated by ONAP to the fullest +extent possible. + +VVP is a utility written in Python that can be executed via a +:doc:`command-line script `, +:ref:`Docker container `, or a +native :doc:`Desktop GUI application ` to analyze and report +on the compliance of a given set of Heat templates to the ONAP requirements. + +This guide will provide the user with instructions on how to acquire, +setup, and execute the validations. .. toctree:: :maxdepth: 1 + installation.rst validation-scripts.rst - administration.rst - architecture.rst - configuration.rst - delivery.rst humaninterfaces.rst - installation.rst - logging.rst - offeredapis.rst release-notes.rst diff --git a/docs/installation.rst b/docs/installation.rst index 0353ec4..a20bbed 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -1,14 +1,54 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. Copyright 2017 AT&T Intellectual Property. All rights reserved. +.. Copyright 2019 AT&T Intellectual Property. All rights reserved. + +.. _vvp-installation: Installation -================== +============ + +The VNF Validation Platform (VVP) can be run from source as a normal +Python executable or a Docker image is provided. This section will +describe how to setup and install the ``vvp/validation-scripts`` and +run from source. -Installation happens through the vvp-devkit project. +Installation and configuration of Docker is beyond the scope of this document, +but you can refer to the :ref:`Docker Execution ` +instructions for more details on running the validations as from +the Docker image. -Environment +Pre-requisites -------------- -Steps ----------- +This document assumes you have the following system-level utilities +installed. + +Please refer to the respective sites for these tools for the appropriate +installation instructions for your given operating system. + +* `Python 3.6+ `__ +* `Git `__ + +Setup +------ + +The source code for VVP can be obtained from the `ONAP Gerrit site `__ +or its `GitHub mirror `__. + +1. Clone the source from your desired repository host: + + Choose **one** of the following ``git clone`` commands:: + + > git clone https://github.com/onap/vvp-validation-scripts.git + > git clone https://gerrit.onap.org/r/vvp/validation-scripts + +2. (*Optional*) If desired, you can create a virtual Python environment to + avoid installing VVP's dependencies in your system level installation + of Python:: + + > python -m venv vvp + > source vvp/activate + +3. Install the required dependencies with the following command:: + + > python pip install -r requirements.txt diff --git a/docs/logging.rst b/docs/logging.rst deleted file mode 100644 index 4c795b2..0000000 --- a/docs/logging.rst +++ /dev/null @@ -1,14 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. Copyright 2017 AT&T Intellectual Property. All rights reserved. - -Logging -========== -Login into portal/em admin/cms admin - -Where to Access Information --------------------------------- - - -Error / Warning Messages ----------------------------- diff --git a/docs/offeredapis.rst b/docs/offeredapis.rst deleted file mode 100644 index 20a696b..0000000 --- a/docs/offeredapis.rst +++ /dev/null @@ -1,10 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. Copyright 2017 AT&T Intellectual Property. All rights reserved. - - -Offered APIs -============== - - -None Available at this time diff --git a/docs/release-notes.rst b/docs/release-notes.rst index 2e9c394..e5876a9 100644 --- a/docs/release-notes.rst +++ b/docs/release-notes.rst @@ -1,16 +1,47 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. Copyright 2017 AT&T Intellectual Property. All rights reserved. +.. Copyright 2019 AT&T Intellectual Property. All rights reserved. Release Notes =============================== -.. note:: - VNF Onboarding is a challenge across the industry because of a lack of a - standard form for VNFs. - The VVP project was created to provide validation scripts for - VNFs to test validity against the ONAP Requirements. + +Version: 4.0.0 (Dublin) +----------------------- + +:Release Date: TBD + +**Removed Features** + +- The VVP web application has been deprecated and is no longer supported + as of the Dublin release. The validation scripts continue to be supported + and enhanced, but contributions to the web-related repositories are now locked + and VVP will no longer be supported for deployment via ONAP Operations + Manager (OOM). + + The following repositories are now locked as of this release: + + - ``vvp/ansibile-ice-bootstrap`` + - ``vvp/cms`` + - ``vvp/devkit`` + - ``vvp/engagementmgr`` + - ``vvp/gitlab`` + - ``vvp/image-scanner`` + - ``vvp/jenkins`` + - ``vvp/portal`` + - ``vvp/postgresql`` + - ``vvp/test-engine`` + +**New Features** + +- A new :doc:`GUI application ` has been contributed and can + be used to execute validations in a user-friendly way without using complex + command line options. +- VVP is now packaged as a Docker container eliminating the need to run the + application from source code. See the :ref:`Docker Execution ` + instructions for more details. + Version: 3.0.0 -------------- @@ -70,6 +101,7 @@ Quick Links: - Initial release - none **Other** + NA =========== diff --git a/docs/validation-scripts.rst b/docs/validation-scripts.rst index 0ff6e32..b327ad4 100644 --- a/docs/validation-scripts.rst +++ b/docs/validation-scripts.rst @@ -1,105 +1,349 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. Copyright 2018 AT&T Intellectual Property. All rights reserved. +.. Copyright 2019 AT&T Intellectual Property. All rights reserved. -Manual Heat Template Validation -=============================== +Heat Validation via Command Line +================================ -validation-scripts ------------------- +.. note:: + This section assumes you have already acquired and setup the validation + scripts as described in the :ref:`VVP Installation ` + section. -This project contains validation scripts to test -that a set of Heat Templates adheres to -the ONAP VNF Heat Orchestration Template guidelines. +Basic Script Execution +---------------------- -For more information on the ONAP Heat Orchestration -Template Guidelines, vist the `Heat Guidelines `__ +1. The scripts must be executed from the ``ice_validator`` directory. + Navigate to the directory first:: -About -_____ + > cd /ice_validator -The validation scripts project allows performing heat template -validation without installing the full VVP platform. The following -instructions apply to running these validation scripts in that manner. +2. To run the script with default options, issue the following command where + ```` is the directory containing the Heat templates:: + + /path/to/validation-scripts/ice_validator>$ pytest tests --template-directory=`` + +3. Test results will be written to the consoles describing any failures that + are encountered. If no failures are found, then a success message will + be displayed. Additional reports can be found in the + ``/ice_validator/output`` directory. See + :ref:`Reports ` for more details. + + +.. _vvp-cmd-options: + +Command Line Options +-------------------- + +Additional options can be specified at the command line to customize output +and execution. VVP uses the `pytest `__ +framework for test execution, and therefore all pytest options are available. + +Please refer to the pytest documentation for information on the command-line +options that framework provides. We will only document the specific options +that are used by VVP:: + + --template-directory=TEMPLATE_DIR + Directory which holds the templates for validation + + --category=CATEGORY_NAME + Additional validations that can be selected. + The only supported value at this time is + environment_file which will enforce that specific + parameter values are excluded from the environment + file per ONAP Heat requirements. + + --self-test Test the unit tests against their fixtured data + + --report-format=REPORT_FORMAT + Format of output report (html, csv, excel, json) + + --continue-on-failure + Continue validation even when structural errors exist + in input files + + --output-directory=OUTPUT_DIR + Alternate directory for report output. + +.. _vvp-reports: + +VVP Reports +----------- + +After completion of the validations, several reports will be written to the +ouput directory (specified via ``--output-directory`` or +``/ice_validator/output`` by default). + + ++-----------------------+------------------------------------------------------+ +| Report | Description | ++=======================+======================================================+ +| ``report.{html,csv,`` | Full report of validation results showing the pass | +| ``xlsx,json}``| or failure of the validation and details on all | +| | errors. See :ref:`Validation Report ` | +| | for information on the HTML, CSV, and Excel versions | +| | of the report, or | +| | :ref:`JSON Report ` for details on | +| | the machine-readable results. | ++-----------------------+------------------------------------------------------+ +| ``traceability.csv`` | This shows a mapping from Heat requirement to | +| | test case. | ++-----------------------+------------------------------------------------------+ +| ``mapping_errors.csv``| File should be empty, but if present shows tests | +| | that are mapped to requirements that no longer exist | ++-----------------------+------------------------------------------------------+ +| ``traceability.rst`` | Similar information to ``traceability.csv``, but | +| | in reStructuredText for use in the VNF Requirements | +| | documentation. | ++-----------------------+------------------------------------------------------+ +| ``failures`` | **Deprecated** JSON version of test failures. Use | +| | ``report.json`` instead. | ++-----------------------+------------------------------------------------------+ + +.. _vvp-report: + +Validation Report +~~~~~~~~~~~~~~~~~ + +If the report format of ``html`` (*default*), ``excel``, or ``csv`` are +requested via the ``--report-format`` option, then the a report file will +be written to the output directory. Regardless of format, the file will contain +a header section that summarizes the results and files scanned, and an error +section that has a row for each failure with four columns. + +Header Information +^^^^^^^^^^^^^^^^^^ + ++-----------------------+------------------------------------------------------+ +| Header Element | Description | ++=======================+======================================================+ +| Categories Selected | Any additional categories selected via ``--category``| ++-----------------------+------------------------------------------------------+ +| Tool Version | Version of the tool that produced the report | ++-----------------------+------------------------------------------------------+ +| Report Generated At | The timestamp of when the report was generated | ++-----------------------+------------------------------------------------------+ +| Directory Validated | Absolute path to directory validated | ++-----------------------+------------------------------------------------------+ +| Checksum | Unique MD5 has of the contents of template directory | ++-----------------------+------------------------------------------------------+ +| Total Errors | Number of errors/violations found | ++-----------------------+------------------------------------------------------+ + + +Error Information +^^^^^^^^^^^^^^^^^ + +If any violations are found, then there will be one row for each violation +with the following columns: + ++-----------------------+------------------------------------------------------+ +| Column Name | Description | ++=======================+======================================================+ +| Files | The file or files that were scanned as part of the | +| | test. | ++-----------------------+------------------------------------------------------+ +| Tests | Name of the test case (not shown in HTML version) | ++-----------------------+------------------------------------------------------+ +| Error Message | This shows the test and brief error message from the | +| | test that failed. This will contain details about | +| | the element that triggered the violation such as the | +| | parameter name, resource ID, etc. | +| | | +| | In the HTML version of the report this column will | +| | also show the test case name, and provide a link to | +| | ``Full Details`` the raw output of the test | ++-----------------------+------------------------------------------------------+ +| Requirements | The requirement ID and text that was violated | ++-----------------------+------------------------------------------------------+ +| Resolution Steps | For some violations, there are pre-defined resolution| +| | steps that indicate what action the user should take | +| | to resolve the violation. | +| | | +| | **Note**: Not all violations will have resolution | +| | steps, rather the error message and requirement is | +| | sufficient. | ++-----------------------+------------------------------------------------------+ +| Raw Test Output | Full output from the pytest test case. This not a | +| | dedicated column in the HTML version of the report. | ++-----------------------+------------------------------------------------------+ + + +.. _vvp-json-report: + +JSON Report +~~~~~~~~~~~ + +This report is intended to provide a machine-readable version of the test +execution, and provides the most comprehensive summary of the test execution +and results. + +File Header/Top Level +^^^^^^^^^^^^^^^^^^^^^ + +JSON Report +The top level will include a summary of available execution metadata. + +NOTE: The ``tests`` and ``requirements`` entries are elided in the +example below. + +**Example Header**: + +.. code-block:: javascript + + { + "version": "dublin", + "template_directory": "/path/to/template", + "timestamp": "2019-01-21T02:11:07.305000", + "checksum": "6296aa211870634f9b4a23477c5eab28", + "profile": "", + "outcome": "FAIL", + "tests": [], + "requirements": [], + } + +**Header Definition**: + +.. csv-table:: + :header-rows: 1 + :file: vvp_json_header.csv + +.. _vvp-test-result: + +Test Result +^^^^^^^^^^^ + +For each test result a JSON object will be provided that informs the consumer +what tests was run, its result, and the requirements it validated. + +**Example Test Result**: + +.. code-block:: javascript + + { + "files": [ + "/Users/username/Desktop/stark_template2/STARKDB-nested-1.yaml", + "/Users/username/Desktop/stark_template2/base_starkdb.yaml", + ], + "test_module": "test_resource_indices", + "test_case": "test_indices_start_at_0_increment", + "result": "FAIL", + "error": " Index values associated with resource ID prefix STARKDB_server_ do not start at 0\n", + "requirements": [ + { + "id": "R-11690", + "text": "When a VNF's Heat Orchestration Template's Resource ID contains an\n``{index}``, the ``{index}`` is a numeric value that **MUST** start at\nzero and **MUST** increment by one.\n\nAs stated in R-16447,\n*a VNF's MUST be unique across all Heat\nOrchestration Templates and all HEAT Orchestration Template\nNested YAML files that are used to create the VNF*. While the ``{index}``\nwill start at zero in the VNF, the ``{index}`` may not start at zero\nin a given Heat Orchestration Template or HEAT Orchestration Template\nNested YAML file.", + "keyword": "MUST" + } + ] + } + + +**Test Result Definition**: + +.. csv-table:: + :header-rows: 1 + :file: vvp_json_test_result.csv -Installation -____________ +.. _vvp-req-metadata: -This software is not platform dependent and can be run in a Windows, Unix or -OS X environment. +Requirement Metadata +^^^^^^^^^^^^^^^^^^^^ -Satisfy Dependencies -#################### +For each test case, the following requirement metadata will be reported. +**Example Requirement Metadata**: - These can be installed using pip (assuming pip is installed) with the command: +.. code-block:: javascript -``$ pip install -r requirements.txt`` + { + "id": "R-11690", + "text": "When a VNF's Heat Orchestration Template's Resource ID contains an\n``{index}``, the ``{index}`` is a numeric value that **MUST** start at\nzero and **MUST** increment by one.\n\nAs stated in R-16447,\n*a VNF's MUST be unique across all Heat\nOrchestration Templates and all HEAT Orchestration Template\nNested YAML files that are used to create the VNF*. While the ``{index}``\nwill start at zero in the VNF, the ``{index}`` may not start at zero\nin a given Heat Orchestration Template or HEAT Orchestration Template\nNested YAML file.", + "keyword": "MUST" + } -Use -___ -Clone this project. +**Requirement Metadata Definition**: -To validate Heat templates just run this the command under the folder ``ice_validator``: ++-----------------+-------------+--------------+-----------+-------------------+ +| Field Name | Required | Data Type | Valid | Description | +| | Optional | | Values | | +| | Conditional | | | | ++=================+=============+==============+===========+===================+ +| ``id`` | Required | ``string`` | | Requirement ID | +| | | | | from the VNFRQTS | +| | | | | project | ++-----------------+-------------+--------------+-----------+-------------------+ +| ``text`` | Required | ``string`` | | Full text of | +| | | | | requirement | ++-----------------+-------------+--------------+-----------+-------------------+ +| ``keyword`` | Required | ``string`` | MUST, | RFC 2119 keyword | +| | | | MUST NOT, | of the requirement| +| | | | MAY, | | +| | | | SHOULD, | | +| | | | SHOULD NOT| | ++-----------------+-------------+--------------+-----------+-------------------+ -``$ pytest --tap-stream --template-directory=`` -where ```` is the full path to a folder containing heat templates. +.. _vvp-req-result: -Output -______ +Requirement Result +^^^^^^^^^^^^^^^^^^ -After performing a validation, an output folder will be created. +The file also includes an aggregated view of adherence to the VNF Requirements +validated by the validation scripts. Since some requirements have multiple +test cases, these results roll-up the result to an aggregated result for each +requirement. This section does not include detailed test results. If you +require detailed error information, then refer to the tests section of the +results. -``/path/to/validation-scripts/ice_validator/output/`` +**Example Requirement Result**: -This folder will contain a file ``report.html`` which contains a list of all -of the ONAP VNF Heat Template Guideline violations. If there are no violations, -the report will say ``No validation errors found.`` +.. code-block:: javascript -Interpreting the Output -_______________________ + { + "id": "R-16447", + "text": "A VNF's **MUST** be unique across all Heat\nOrchestration Templates and all HEAT Orchestration Template\nNested YAML files that are used to create the VNF.", + "keyword": "MUST", + "result": "FAIL" + "errors": [ + "The error message" + ] + } -The report file will have 4 columns for details about a violation, and one -row for each violation. Below contains details about each column. -File -#### +**Requirement Result Definition**: -This is the file(s) that contained the violation +.. csv-table:: + :header-rows: 1 + :file: vvp_json_req_result.csv -Error Message -############# -This shows the test and brief error message from the validation script that -contained the violation. There is a ``Full Details`` button to show the -complete raw test output. The error message will also contain details -about what element is involved with the violation (such as the parameter -name, resource id, etc...). +.. _vvp-docker-execution: -Requirement(s) -############## +Docker Execution +---------------- -This column contains the requirement(s) that each test/violation is -mapped to. These requirements are taken directly from the VNF Requirements -project Heat Orchestration Template Guidelines section. +A version of VVP is also provided as a Docker image. If your environment +supports Docker, then this eliminates the need to setup and install +the application from source code. +To execute from Docker, issue the following command where +```` is where the Heat templates are located and +```` is where you would like the reports to be +written on your local machine:: -Resolution Steps -################ - -For some violations, there are pre-defined resolution steps that -indicate what action the user should take to resolve the violation. - -**Note**: Not all violations will have resolution steps. Most violations -can be resolved simply by reviewing the requirements that have been violated -in the previous column. + docker run --rm -i -v ~//:/template \ + -v ~/:/reports \ + onap/vvp/validation-scripts --template-directory=/template \ + --output-directory=/reports +The same :ref:`command line options ` can be used with the +Docker image that are used with the version from source. Self-Test Suite -_______________ +--------------- The ``ice_validator`` includes an extensive self-test suite. It is a **requirement** for any additions or changes to the test suite to @@ -111,3 +355,8 @@ the project root as: You can also run it under the folder ``ice_validator``: ``$ pytest --self-test`` + + + + + diff --git a/docs/vvp_app.png b/docs/vvp_app.png new file mode 100644 index 0000000..8a27249 Binary files /dev/null and b/docs/vvp_app.png differ diff --git a/docs/vvp_json_header.csv b/docs/vvp_json_header.csv new file mode 100644 index 0000000..5827f8b --- /dev/null +++ b/docs/vvp_json_header.csv @@ -0,0 +1,17 @@ +Field Name,"Required/ +Optional/ +Conditional",Data Type,Valid Values,Description +``checksum``,Required,``string``,,MD5 hash of all file contents in the ``template_directory`` +``outcome``,Required,``string``,"``PASS`` +``FAIL`` +``ERROR``","One of the valid values: + +* ``PASS`` - All tests passed successfully (some may have been skipped as not applicable based on the contents of the template) +* ``FAIL`` - At least one test failed. In this scenario the templates will need to be corrected to comply with the requirements +* ``ERROR`` - An unexpected error occurred during test setup. Some or all tests may have not executed. Issue should be referred to the VVP team for investigation." +``categories``,Optional,array of ``string``,,Categories selected via the ``--category`` option. +``template_directory``,Required,``string``,,Absolute path of the directory containing the Heat templates that were validated +``version``,Required,``string``,,Version of the validation scripts that produced the report +``timestamp``,Required,``string``,,ISO 8601 Timestamp in UTC of when the report was generated +``tests``,Required,List of :ref:`Test Result `,,See :ref:`Test Result ` +``requirements``,Required,List of :ref:`Requirement Result `,,See :ref:`Requirement Result ` diff --git a/docs/vvp_json_req_result.csv b/docs/vvp_json_req_result.csv new file mode 100644 index 0000000..42b96b0 --- /dev/null +++ b/docs/vvp_json_req_result.csv @@ -0,0 +1,17 @@ +Field Name,Required/Optional/Conditional,Data Type,Valid Values,Description +``id``,Required,``string``,,"Requirement ID from the VNFRQTS project. + +**NOTE**:a requirement ID of ""Unmapped"" may be included if one or more tests are not mapped to a requirement." +``text``,Required,``string``,,Full text of the requirement. +``keyword``,Required,``string``,"``MUST``, +``MUST NOT``, +``MAY``, +``""SHOULD""``, +``""SHOULD NOT""``",RFC 2119 keyword associated with the requirement +``result``,Required,``string``,"``PASS``, ``SKIP``, ``FAIL``, ``ERROR``","One of the valid values: + +* ``PASS`` - The test case passed with no violations +* ``SKIP`` - The test case was skipped because it was deemed not applicable +* ``FAIL`` - The test case completed, but it found a violation +* ``ERROR`` - An unexpected error was found while setting up the test case" +``errors``,Required,List of ``string``,,Error messages associated with this requirement. This will be an empty string if the result is ``PASS`` or ``SKIP`` diff --git a/docs/vvp_json_test_result.csv b/docs/vvp_json_test_result.csv new file mode 100644 index 0000000..14982c0 --- /dev/null +++ b/docs/vvp_json_test_result.csv @@ -0,0 +1,17 @@ +Field Name,Required/Optional/Conditional,Data Type,Valid Values,Description +``files``,Required,list of ``string``,,"List of files that were passed to the test case. + +**NOTE**: If ``result`` is ``ERROR`` this may be an empty list" +``test_module``,Required,``string``,,Name of module/file anme that contains the test case +``test_case``,Required,``string``,,Name of the test case +``result``,Required,``string``,"* ``PASS`` +* ``SKIP`` +* ``FAIL`` +* ``ERROR``","One of the valid values: + +* ``PASS`` - The test case passed with no violations +* ``SKIP`` - The test case was deemed not applicable +* ``FAIL`` - The test case completed, but a violation was found +* ``ERROR`` - An unexpected error was found while setting up the test case" +``error``,Required,``string``,,"If the test failed or encountered an error, then ths will be a message summarizing the error. If the test passed or was skipped, then this will be an empty string" +``requirements``,Required,List of :ref:`Requirement Metadata `,,See :ref:`Requirement Metadata ` -- cgit 1.2.3-korg