Add StndDefined validation subsection to VES RTD

Change-Id: I9e9a03ce6786787cf47b15a888d0cfd1bac30d01
Signed-off-by: Michal Banka <michal.banka@nokia.com>
Signed-off-by: Edyta Krukowska <edyta.krukowska@nokia.com>
Issue-ID: DCAEGEN2-2265

1 files changed, 271 insertions, 0 deletions

diff --git a/docs/sections/services/ves-http/stnd-defined-validation.rst b/docs/sections/services/ves-http/stnd-defined-validation.rst
new file mode 100644
index 00000000..c4184e46
--- /dev/null
+++ b/docs/sections/services/ves-http/stnd-defined-validation.rst
@@ -0,0 +1,271 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+StndDefined Events Collection Mechanism
+=======================================
+
+Description
+-----------
+
+This mechanism can be used to validate any JSON content incoming as JsonNode using OpenAPI standardized schemas.
+During validation externally located schemas are mapped to local schema files.
+
+Validated JSON must have one field that will refer to an external schema, which will be mapped to local file and then
+validation of any chosen part of JSON is executed using local schema.
+
+StndDefined validation is integrated with the event collecting functionality available under the endpoint
+*/eventListener/v7*. Process of event collecting includes steps in the following order:
+
+1. General event validation (1st stage validation)
+2. Event transformation
+3. **StndDefined event validation** (2nd stage validation)
+4. Event routing to DMaaP
+
+Mapping file is cached on stndDefined validator creation, so it's not read every time validation is performed.
+Schemas' content couldn't be cached due to an external library restrictions (OpenAPI4j).
+
+The value of the 'stndDefinedNamespace' field in any incoming stndDefined JSON event is used to match the topic from
+property *collector.dmaap.streamid*.
+
+StndDefined properties
+----------------------
+
+There are 5 additional properties related to stndDefined validation in collector.properties file.
+
++----------------------------------------------+--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+-------------------------------------------------------------------------------+
+| Name                                         | Description                                                                    | Example                                                                        | Note                                                                          |
++==============================================+================================================================================+================================================================================+===============================================================================+
+| collector.externalSchema.checkflag           | Flag is responsible for turning on/off stndDefined data validation.            | -1 or 1                                                                        |                                                                               |
+|                                              | By default this flag is set to 1, which means that the validation is enabled.  |                                                                                |                                                                               |
+|                                              | In case flag is set to -1, validation is disabled.                             |                                                                                |                                                                               |
++----------------------------------------------+--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+-------------------------------------------------------------------------------+
+| collector.externalSchema.mappingFileLocation | This should be a local filesystem path to file with mappings of public URLs    | etc/externalRepo/schema-map.json                                               |                                                                               |
+|                                              | to local URLs.                                                                 |                                                                                |                                                                               |
++----------------------------------------------+--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+-------------------------------------------------------------------------------+
+| collector.externalSchema.schemasLocation     | Schemas location is a directory under which stndDefined validator will search  | ./etc/externalRepo/ and first mapping from example mappingFile below is taken, |                                                                               |
+|                                              | for local schemas.                                                             | validator will look for schema under the path:                                 |                                                                               |
+|                                              |                                                                                | ./etc/externalRepo/3gpp/rep/sa5/data-models/blob/REL-16/OpenAPI/faultMnS.yaml  |                                                                               |
++----------------------------------------------+--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+-------------------------------------------------------------------------------+
+| event.externalSchema.schemaRefPath           | This is an internal path from validated JSON. It should define which field     | /event/stndDefinedFields/schemaReference                                       | In SDK version 1.4.2 this path doesn’t use JSON path notation (with . signs). |
+|                                              | will be taken as public schema reference, which is later mapped.               |                                                                                | It might change in further versions                                           |
++----------------------------------------------+--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+-------------------------------------------------------------------------------+
+| event.externalSchema.stndDefinedDataPath     | This is internal path from validated JSON.                                     | /event/stndDefinedFields/data                                                  | In SDK version 1.4.2 this path doesn’t use JSON path notation (with . signs). |
+|                                              | It should define which field will be validated.                                |                                                                                | It might change in further versions                                           |
++----------------------------------------------+--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+-------------------------------------------------------------------------------+
+
+Format of the schema mapping file is a JSON file with list of mappings, as shown in the example below.
+
+.. code-block:: json
+
+    [
+        {
+            "publicURL": "https://forge.3gpp.org/rep/sa5/data-models/blob/REL-16/OpenAPI/faultMnS.yaml",
+            "localURL": "3gpp/rep/sa5/data-models/blob/REL-16/OpenAPI/faultMnS.yaml"
+        },
+        {
+            "publicURL": "https://forge.3gpp.org/rep/sa5/data-models/blob/REL-16/OpenAPI/heartbeatNtf.yaml",
+            "localURL": "3gpp/rep/sa5/data-models/blob/REL-16/OpenAPI/heartbeatNtf.yaml"
+        },
+        {
+            "publicURL": "https://forge.3gpp.org/rep/sa5/data-models/blob/REL-16/OpenAPI/PerDataFileReportMnS.yaml",
+            "localURL": "3gpp/rep/sa5/data-models/blob/REL-16/OpenAPI/PerDataFileReportMnS.yaml"
+        },
+        {
+            "publicURL": "https://forge.3gpp.org/rep/sa5/data-models/blob/master/OpenAPI/provMnS.yaml",
+            "localURL": "3gpp/rep/sa5/data-models/blob/REL-16/OpenAPI/provMnS.yaml"
+        }
+    ]
+
+
+Requirements for stndDefined validation
+---------------------------------------
+
+To run stndDefined validation, both *collector.schema.checkflag* and *collector.externalSchema.checkflag* must be set to 1.
+
+Despite the flag set, the validation will not start when:
+
+- Domain of the incoming event is not 'stndDefined'.
+- General event validation (1st stage) failed.
+- Field of event referenced under the property *event.externalSchema.schemaRefPath* (by default */event/stndDefinedFields/schemaReference*):
+    - Has an empty value.
+    - Does not exist in the incoming event.
+
+Validation scenarios
+--------------------
+
+Positive scenario, which returns 202 Accepted HTTP code after successful stndDefined validation:
+
+- *collector.schema.checkflag* and *collector.externalSchema.checkflag* is set to 1
+- Mapping file has valid format
+- Schema file mapped from referenced in the event is valid
+- The incoming event is valid against schema
+
+Below are scenarios when, the stndDefined validation will end with failure and return 400 Bad Request HTTP code:
+
+- One of stndDefined data fields has wrong type or value
+- StndDefined data has an empty body or is missing required field
+- Field of event referenced under the property event.externalSchema.schemaRefPath has publicURL which is not mapped in the schemas mappings
+- Field defining public schema in event (by default */event/stndDefinedFields/schemaReference*) after "#" has non existing reference in schema file
+
+Schemas repository description
+------------------------------
+
+Schemas and mapping file location might be configured to any local directory through properties in collector.properties
+as described in 'StndDefined properties' section.
+
+By default schemas repository is located under *etc/externalSchema* directory, as well as schemas mapping file called
+*schema-map.json*. There are files stored in the project repository which are schemas from 3GPP organisation. Every
+organisation which adds or mounts external schemas should store them in folder named by organisation
+name. Further folders structure may be whatever as long as schemas are correctly referenced in the mapping file.
+
+Sample directory tree of *etc* directory:
+
+.. code-block:: text
+
+    etc
+    ├── ...
+    └── externalRepo
+        ├── schema-map.json
+        └── 3gpp
+            └── rep
+                └── sa5
+                    └── data-models
+                        └── blob
+                            └── REL-16
+                                └── OpenAPI
+                                    ├── faultMnS.yaml
+                                    ├── heartbeatNtf.yaml
+                                    ├── PerDataFileReportMnS.yaml
+                                    └── provMnS.yaml
+
+Routing of stndDefined domain events
+------------------------------------
+
+All events, expect those with 'stndDefined' domain, are routed to DMaaP topics basing on domain value. Events with
+'stndDefined' domain are sent to proper topic basing on field 'stndDefinedNamespace'.
+
+This is the only difference from standard event routing, specific for 'stndDefined' domain. As in every other event
+routing value is being mapped for specific DMaaP stream. Stream ID to DMaaP channels mappings are located in
+*etc/collector.properties* file under property *collector.dmaap.streamid*. Channels descriptions are in
+*etc/DmaapConfig.json*, where destination DMaaP topics are selected.
+
+With stndDefined domain managment 4 new mappings were added. Their routing has been described in the table below:
+
+    +---------------------------+--------------------------------+------------------------------------------------------+
+    | Stream ID                 | Channel                        | DMaaP Stream                                         |
+    +===========================+================================+======================================================+
+    | 3GPP-FaultSupervision     | ves-3gpp-fault-supervision     | unauthenticated.SEC_3GPP_FAULTSUPERVISION_OUTPUT     |
+    +---------------------------+--------------------------------+------------------------------------------------------+
+    | 3GPP-Heartbeat            | ves-3gpp-heartbeat             | unauthenticated.SEC_3GPP_HEARTBEAT_OUTPUT            |
+    +---------------------------+--------------------------------+------------------------------------------------------+
+    | 3GPP-Provisioning         | ves-3gpp-provisioning          | unauthenticated.SEC_3GPP_PROVISIONING_OUTPUT         |
+    +---------------------------+--------------------------------+------------------------------------------------------+
+    | 3GPP-PerformanceAssurance | ves-3gpp-performance-assurance | unauthenticated.SEC_3GPP_PERFORMANCEASSURANCE_OUTPUT |
+    +---------------------------+--------------------------------+------------------------------------------------------+
+
+
+Error scenarios behaviour
+-------------------------
+
+There are few error scenarios described in 'Validation scenarios' section. This section will describe user point of view
+of VES Collector behaviour when they happen. Messages returned as HTTP response contain data described below for each
+scenario.
+
+1. StndDefined fields validation related errors
+
+1.1. Schema file referred under the path from property *event.externalSchema.schemaRefPath* (by default */event/stndDefinedFields/schemaReference*) not present in the schema repository.
+
+    +---------------------+------------------------------------------------------------------+
+    | Property Name       | Property Description                                             |
+    +=====================+==================================================================+
+    | MessageId           | SVC2004                                                          |
+    +---------------------+------------------------------------------------------------------+
+    | Text                | "Invalid input value for %1 %2: %3"                              |
+    +---------------------+------------------------------------------------------------------+
+    | Variables           | %1 – “attribute”                                                 |
+    |                     | %2 – "event.stndDefinedFields.schemaReference"                   |
+    |                     | %3 – "Referred external schema not present in schema repository" |
+    +---------------------+------------------------------------------------------------------+
+    | HTTP status code(s) | 400 Bad request                                                  |
+    +---------------------+------------------------------------------------------------------+
+
+1.2. File referred under the path from property *event.externalSchema.schemaRefPath* (by default */event/stndDefinedFields/schemaReference*) exists, but internal reference (part of URL after #) is incorrect.
+
+    +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
+    | Property Name       | Property Description                                                                                                              |
+    +=====================+===================================================================================================================================+
+    | MessageId           | SVC2000                                                                                                                           |
+    +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
+    | Text                | The following service error occurred: %1. Error code is %2                                                                        |
+    +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
+    | Variables           | %1 - "event.stndDefinedFields.schemaReference value does not correspond to any external event schema file in externalSchema repo" |
+    |                     | %2 - "400"                                                                                                                        |
+    +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
+    | HTTP status code(s) | 400 Bad request                                                                                                                   |
+    +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
+
+1.3. StndDefined validation executed, but event contents do not validate with referenced schema.
+
+    +---------------------+---------------------------------------------------------------------------------------------+
+    | Property Name       | Property Description                                                                        |
+    +=====================+=============================================================================================+
+    | MessageId           | SVC2000                                                                                     |
+    +---------------------+---------------------------------------------------------------------------------------------+
+    | Text                | The following service error occurred: %1. Error code is %2                                  |
+    +---------------------+---------------------------------------------------------------------------------------------+
+    | Variables           | %1 - "event.stndDefinedFields.data invalid against event.stndDefinedFields.schemaReference" |
+    |                     | %2 - "400"                                                                                  |
+    +---------------------+---------------------------------------------------------------------------------------------+
+    | HTTP status code(s) | 400 Bad request                                                                             |
+    +---------------------+---------------------------------------------------------------------------------------------+
+
+2. Problems with routing of stndDefined domain.
+
+2.1. StndDefinedNamespace field not received in the incoming event.
+
+    +---------------------+-----------------------------------------------------+
+    | Property Name       | Property Description                                |
+    +=====================+=====================================================+
+    | MessageId           | SVC2006                                             |
+    +---------------------+-----------------------------------------------------+
+    | Text                | Mandatory input %1 %2 is missing from request       |
+    +---------------------+-----------------------------------------------------+
+    | Variables           | %1 – “attribute”                                    |
+    |                     | %2 – "event.commonEventHeader.stndDefinedNamespace" |
+    +---------------------+-----------------------------------------------------+
+    | HTTP status code(s) | 400 Bad Request                                     |
+    +---------------------+-----------------------------------------------------+
+
+2.2. StndDefinedNamespace field present, but value is empty.
+
+    +---------------------+-----------------------------------------------------+
+    | Property Name       | Property Description                                |
+    +=====================+=====================================================+
+    | MessageId           | SVC2006                                             |
+    +---------------------+-----------------------------------------------------+
+    | Text                | Mandatory input %1 %2 is empty in request           |
+    +---------------------+-----------------------------------------------------+
+    | Variables           | %1 – “attribute”                                    |
+    |                     | %2 – "event.commonEventHeader.stndDefinedNamespace" |
+    +---------------------+-----------------------------------------------------+
+    | HTTP status code(s) | 400 Bad Request                                     |
+    +---------------------+-----------------------------------------------------+
+
+2.3. StndDefinedNamespace field present, but value doesn't match any stream ID mapping.
+
+    +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
+    | Property Name       | Property Description                                                                                                                      |
+    +=====================+===========================================================================================================================================+
+    | MessageId           | SVC2004                                                                                                                                   |
+    +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
+    | Text                | "Invalid input value for %1 %2: %3"                                                                                                       |
+    +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
+    | Variables           | %1 – “attribute”                                                                                                                          |
+    |                     | %2 – "event.commonEventHeader.stndDefinedNamespace"                                                                                       |
+    |                     | %3 – "stndDefinedNamespace received not present in VES Collector routing configuration. Unable to route event to appropriate DMaaP topic" |
+    +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
+    | HTTP status code(s) | 400 Bad request                                                                                                                           |
+    +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
+
+
+