From 4bea9eea18354bc3d93cfe0aab372c839e40e1f6 Mon Sep 17 00:00:00 2001 From: Aleem Raja Date: Tue, 30 Apr 2019 13:13:52 +0530 Subject: Update DCAEGEN2, mapper services doc Changing documentation as per changes made for Dublin release Change-Id: I2ef0e85c235b78ea7eeab6f160d47b16039f3950 Issue-ID: DCAEGEN2-1461 Signed-off-by: Aleem Raja Update DCAEGEN2, mapper services doc Changed File index.rst installation.rst Added File: troubleshooting.rst Issue-ID: DCAEGEN2-1461 Change-Id: I2ef0e85c235b78ea7eeab6f160d47b16039f3950 Signed-off-by: Aleem Raja Update DCAEGEN2, mapper services doc Changed Files: index.rst installation.rst Added File: troubleshooting.rst Change-Id: I2ef0e85c235b78ea7eeab6f160d47b16039f3950 Issue-ID: DCAEGEN2-1461 Signed-off-by: Aleem Raja Update DCAEGEN2, mapper services doc Changed File index.rst Issue-ID: DCAEGEN2-1461 Change-Id: I2ef0e85c235b78ea7eeab6f160d47b16039f3950 Signed-off-by: Aleem Raja --- docs/sections/services/mapper/delivery.rst | 10 +- docs/sections/services/mapper/flow-rest-conf.png | Bin 0 -> 36111 bytes docs/sections/services/mapper/flow.png | Bin 35180 -> 42354 bytes docs/sections/services/mapper/flow.rst | 40 ++++-- docs/sections/services/mapper/index.rst | 14 +- docs/sections/services/mapper/installation.rst | 73 ++++++---- docs/sections/services/mapper/mappingfile.rst | 162 +++++++++++++--------- docs/sections/services/mapper/troubleshooting.rst | 82 +++++++++++ 8 files changed, 263 insertions(+), 118 deletions(-) create mode 100644 docs/sections/services/mapper/flow-rest-conf.png create mode 100644 docs/sections/services/mapper/troubleshooting.rst diff --git a/docs/sections/services/mapper/delivery.rst b/docs/sections/services/mapper/delivery.rst index 10968b7e..6cb2cf2f 100644 --- a/docs/sections/services/mapper/delivery.rst +++ b/docs/sections/services/mapper/delivery.rst @@ -1,17 +1,15 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. Copyright 2018 Tech Mahindra Ltd. +.. Copyright 2018-2019 Tech Mahindra Ltd. Delivery ======== -Mapper is delivered with **2 Docker container** having spring boot microservices, **UniversalVesAdapter** and **SnmMapper**. UniversalVesAdapter converts SNMP trap JSON to VES and snmpmapper is just helper application for uploading the mapping file to DB. +Mapper is delivered with **1 Docker container** having spring boot microservice, **UniversalVesAdapter**. UniversalVesAdapter converts telementary data to VES. -| In current release, the installation of mapper service is not integrated with DCAE's installation process and needs to be done manually. +| In current release, the UniversalVesAdapter is integrated with DCAE's config binding service. On start, it fetches the initial configuration from CBS and uses the same. Currently it is not having functionality to refresh the configuration changes made into Consul KV store. Docker Containers --------------- Docker images can be pulled from ONAP Nexus repository with below commands: -| docker pull nexus3.onap.org:10003/snapshots/onap/org.onap.dcaegen2.services.mapper.vesadapter.universalvesadaptor:latest - -| docker pull nexus3.onap.org:10003/snapshots/onap/org.onap.dcaegen2.services.mapper.vesadapter.snmpmapper:latest + ``docker pull nexus3.onap.org:10001/onap/org.onap.dcaegen2.services.mapper.vesadapter.universalvesadaptor:1.0.0-SNAPSHOT`` diff --git a/docs/sections/services/mapper/flow-rest-conf.png b/docs/sections/services/mapper/flow-rest-conf.png new file mode 100644 index 00000000..723977dc Binary files /dev/null and b/docs/sections/services/mapper/flow-rest-conf.png differ diff --git a/docs/sections/services/mapper/flow.png b/docs/sections/services/mapper/flow.png index 614d5d5f..86af4c4e 100644 Binary files a/docs/sections/services/mapper/flow.png and b/docs/sections/services/mapper/flow.png differ diff --git a/docs/sections/services/mapper/flow.rst b/docs/sections/services/mapper/flow.rst index cff2485e..9fe3bfbb 100644 --- a/docs/sections/services/mapper/flow.rst +++ b/docs/sections/services/mapper/flow.rst @@ -1,17 +1,39 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. Copyright 2018-2019 Tech Mahindra Ltd. + +============ +Flow for converting Rest Conf Collector notification ============ -Flow +.. [1] RestConf Collector generates rcc-notication in JSON format and publishes it on DMaaP topic **unathenticated.DCAE_RCC_OUTPUT** +.. [2] The Universal VES Adapter(UVA) microservice has subscribed to this DMaaP topic. +.. [3] On receiving an event from DMaaP, the adapter uses the corresponding mapping file and converts received notification into the VES event. It uses the notification-id from the received notification to find the required mapping file. +.. [4] Those notifications for which no mapping file is identified, a default mapping file is used with generic mappings to create the VES event. +.. [5] The VES formatted Event will be then published on DMaaP topic **unauthenticated.VES_PNFREG_OUTPUT**. + + +.. image:: ./flow-rest-conf.png + :height: 200px + :width: 300 px + :scale: 50 % + :alt: alternate text + :align: left + + + ============ +Flow for converting SNMP Collector notification ============ -.. [#] VNF submits SNMP traps to the SNMP trap collecto. -.. [#] Collector converts the trap into JSON format and publishes it on DMaaP topic **ONAP-COLLECTOR-SNMPTRAP** -.. [#] The Universal VES Adapter(UVA) microservice has subscribed to this DMaaP topic. -.. [#] On receiving an event from DMaaP, the adapter uses the corresponding mapping file from repository and converts received event into the VES event. It uses the enterprise ID from the received event to find the required mapping file. -.. [#] Those SNMP Traps for which no mapping file is identified, a default mapping file is used with generic mappings to create the VES object. -.. [#] The VES formatted Event will be then published on DMaaP topic **unauthenticated.SEC_FAULT_OUTPUT**. +.. [1] VNF submits SNMP traps to the SNMP collector. +.. [2] Collector converts the trap into JSON format and publishes it on DMaaP topic **unauthenticated.ONAP-COLLECTOR-SNMPTRAP** +.. [3] The Universal VES Adapter(UVA) microservice has subscribed to this DMaaP topic. +.. [4] On receiving an event from DMaaP, the adapter uses the corresponding mapping file and converts received event into the VES event. It uses the enterprise ID from the received event to find the required mapping file. +.. [5] Those SNMP Traps for which no mapping file is identified, a default mapping file is used with generic mappings to create the VES event. +.. [6] The VES formatted Event will be then published on DMaaP topic **unauthenticated.SEC_FAULT_OUTPUT**. .. image:: ./flow.png - :height: 100px - :width: 200 px + :height: 200px + :width: 300 px :scale: 50 % :alt: alternate text :align: left \ No newline at end of file diff --git a/docs/sections/services/mapper/index.rst b/docs/sections/services/mapper/index.rst index b142ded0..f88f4214 100644 --- a/docs/sections/services/mapper/index.rst +++ b/docs/sections/services/mapper/index.rst @@ -1,6 +1,6 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. Copyright 2018 Tech Mahindra Ltd. +.. Copyright 2018-2019 Tech Mahindra Ltd. Mapper @@ -8,16 +8,9 @@ Mapper | **Problem:** Different VNF vendors generate event and telemetry data in different formats. Out of the box, all VNF vendors may not support VES format. | **Solution**: A generic adapter which can convert different formats of event and telemetry data to VES format can be of use here. -| *Note*: Currently support snmp trap telemetry data - -There are 2 micro services in the mapper functionality. - -**Universal VES Adapter** - It converts the telemetry data into the required ves format and publishes to the dmaap for further action to be taken. - *Note*: Currently it supports conversion of snmp traps generated by SNMP TRAP Collector. With some work, it can support other collectors as well. - - -**SnmpMapper** - It provides web UI to upload mapping files which are then stored into the postgres database. VES adapter refers to these mapping files at the time of converting the events into VES events. +| *Note*: Currently mapping files are available for SNMP collector and RESTConf collector. +**VES-Mapper** converts the telemetry data into the required VES format and publishes to the DMaaP for further action to be taken by the DCAE analytics applications. .. toctree:: @@ -28,3 +21,4 @@ There are 2 micro services in the mapper functionality. ./installation.rst ./mappingfile.rst ./SampleSnmpTrapConversion + ./troubleshooting.rst \ No newline at end of file diff --git a/docs/sections/services/mapper/installation.rst b/docs/sections/services/mapper/installation.rst index 1d05ebb8..7559f4a5 100644 --- a/docs/sections/services/mapper/installation.rst +++ b/docs/sections/services/mapper/installation.rst @@ -1,48 +1,67 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. Copyright 2018 Tech Mahindra Ltd. +.. Copyright 2018-2019 Tech Mahindra Ltd. Installation ============ -Currently both Universal VES Adapter and SnmpMapper are not integrated with CONFIG_BINDING_SERVICE to get their configuration. All the required configuration needs to be passed as docker environment parameters. -| **Universal Ves Adapter** +**Deployment Prerequisite/dependencies** - To run Universal Ves Adapter container on standalone mode, following docker environment parameters are required. +VES-Mapper can be deployed individually though it will throw errors if it can't reach to DMaaP instance's APIs. To test it functionally, DMaaP is the only required prerequisite outside DCAE. As VES-Mapper is integrated with Consul / CBS, it fetches the initial configuration from Consul. +**Note:** Currently VES-Mapper fetches configuration from Consul only during initialization. It does not periodically refresh the local configuration by getting updates from Consul. This is planned for E release. -- DMAAPHOST - Should contain an address to DMaaP, so that event publishing can work -- MR_PORT - DMaaP Exposed Port -- URL_JDBC - JDBC URL where postgres db is installed -- JDBC_USERNAME - Username for postgres DB user -- JDBC_PASSWORD - Password for postgres DB user -- CONSUL_HOST - a host address where Consul service lies -- CONFIG_BINDING_SERVICE - name of CBS as it is registered in Consul -- HOSTNAME - name of Universal Ves Adapter application as it is registered in CBS catalog +**Blueprint/model/image** -Sample docker run command could be - +VES-Mapper blueprint is available @ +https://gerrit.onap.org/r/gitweb?p=dcaegen2/services/mapper.git;a=blob;f=UniversalVesAdapter/dpo/blueprints/k8s-vesmapper.yaml-template.yaml;h=refs/head/master - ``docker run -d -p 8085:8085/tcp -e URL_JDBC=jdbc:postgresql://10.53.172.138:5432/postgres -e JDBC_USERNAME=postgres -e JDBC_PASSWORD=root -e DMAAPHOST=10.53.172.156 -e CONSUL_HOST=10.53.172.109 -e HOSTNAME=static-dcaegen2-services-mua -e MR_DEFAULT_PORT_NUMBER=3904 -e CONFIG_BINDING_SERVICE=config_binding_service nexus3.onap.org:10003/onap/org.onap.dcaegen2.services.mapper.vesadapter.universalvesadaptor:latest`` +VES-Mapper docker image is available in Nexus repo @ `nexus3.onap.org:10001/onap/org.onap.dcaegen2.services.mapper.vesadapter.universalvesadaptor:1.0.0-SNAPSHOT `_ -Note: you should have a postgresql and DMaap instance running in your system and provide their credentials and hostname to run in standalone mode. +**1.To Run via blueprint** -| **SnmpMapper** +*a. Verify DMaaP configurations in the blueprint as per setup* -- To run SnmpMapper container on standalone mode, following docker environment parameters are required. + Dmaap Configuration consists of subscribe url to fetch notifications from the respective collector and publish url to publish ves event. -- DMAAPHOST-should contain an address to DMaaP, this will be used in future reference -- MR_PORT - DMaaP Exposed Port -- URL_JDBC- JDBC URL where postgres db is installed -- JDBC_USERNAME- Username for postgres DB user -- JDBC_PASSWORD-Password for postgres DB user -- CONSUL_HOST - a host address where Consul service lies -- CONFIG_BINDING_SERVICE - name of CBS as it is registered in Consul -- HOSTNAME - name of Universal Ves Adapter application as it is registered in CBS catalog -Sample docker run command could be - +``streams_publishes`` and ``streams_subscribes`` points to the publishing topic and subscribe topic respectively. + +update these ``urls`` as per your DMaaP configurations in the blueprint. + +*b. Verify the Smooks mapping files in the blueprint as per the usecase. Blueprint contains default mapping file for each supported collector ( SNMP Collector and RESTConf collector currently) which may serve the purpose for the usecase. + +``mapping-files`` in ``collectors`` contains the contents of the mapping file. + +*c. Upload the blueprint in the DCAE's Cloudify instance* + +For this step, DCAE's Cloudify instance should be in running state. Transfer blueprint file in DCAE bootstrap POD under /blueprints directory. Log-in to the DCAE bootstrap POD's main container. + +Validate blueprint + + ``cfy blueprints validate /blueprints/k8s-vesmapper.yaml-template.yaml`` + +Use following command for validated blueprint to upload: + + ``cfy blueprints upload -b ves-mapper /blueprints/k8s-vesmapper.yaml-template.yaml`` + +*d. Create the Deployment* +After VES-Mapper's validated blueprint is uploaded, create Cloudify Deployment by following command + + ``cfy deployments create -b ves-mapper ves-mapper`` + +*e. Deploy the component by using following command* + + ``cfy executions start -d ves-mapper install`` + + +**2.To run on standalone mode** + +Though this is not a preferred way, to run VES-Mapper container on standalone mode using local configuration file carried in the docker image, following docker run command can be used. + + ``docker run -d nexus3.onap.org:10003/onap/org.onap.dcaegen2.services.mapper.vesadapter.universalvesadaptor:1.0.0-SNAPSHOT`` - ``docker run -d -p 8084:8080/tcp -e URL_JDBC=jdbc:postgresql://10.53.172.138:5432/postgres -e JDBC_USERNAME=postgres -e JDBC_PASSWORD=root -e DMAAPHOST=10.53.172.156 -e CONSUL_HOST=10.53.172.109 -e HOSTNAME=static-dcaegen2-services-msnmp -e MR_DEFAULT_PORT_NUMBER=3904 -e CONFIG_BINDING_SERVICE=config_binding_service nexus3.onap.org:10003/onap/org.onap.dcaegen2.services.mapper.snmpmapper:latest`` diff --git a/docs/sections/services/mapper/mappingfile.rst b/docs/sections/services/mapper/mappingfile.rst index e99ba5a9..daf515d3 100644 --- a/docs/sections/services/mapper/mappingfile.rst +++ b/docs/sections/services/mapper/mappingfile.rst @@ -1,84 +1,114 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. Copyright 2018 Tech Mahindra Ltd. +.. Copyright 2018-2019 Tech Mahindra Ltd. Mapping File ============ Mapping file is needed by Universal VES Adapter to convert the telemetry data into the VES format. The Adapter uses Smooks Framework to do the data format conversion by using the mapping files. -Currently it requires to write the mapping file manually and upload it using the SnmpMapper application. The file gets stored into the postgres database. The pgInventory instance is used for this. | To know more about smooks framework check the following link: | http://www.smooks.org/guide +SNMP Collector Default Mapping File +============ Following is the default snmp mapping file which is used when no mapping file is found while processing event from SNMP Trap Collector. .. code-block:: xml - - - - - - - - - - - - - - - - - - - - - "3.0" - "XXXX" - "VesAdapter" - "fault" - commonEventHeader.domain+"_"+"_"+ faultFields.alarmCondition; - - - - "Medium" - "VesAdapter" - - - - - "MINOR" - "SNMP Agent" - "SNMP Fault" - - - "Active" - - - - - - - - - - - - - + + + + + + + + + + + + + + + '3.0' + 'FaultField' + 'XXXX' + 'VESMapper' + org.onap.dcaegen2.ves.domain.ves54.CommonEventHeader.Domain.FAULT + commonEventHeader.domain + + + + org.onap.dcaegen2.ves.domain.ves54.CommonEventHeader.Priority.NORMAL + 'VesAdapter' + + + + + 'SNMP Fault' + org.onap.dcaegen2.ves.domain.ves54.FaultFields.VfStatus.ACTIVE + org.onap.dcaegen2.ves.domain.ves54.FaultFields.EventSeverity.MINOR + + + + + + + + + + +RestConf Collector Default Mapping File +============ +Following is the default RestConf collector mapping file which is used when no mapping file is found while processing notification from RestConf Collector. +.. code-block:: xml + + + + + + + + + + + + + + + org.onap.dcaegen2.ves.domain.ves70.CommonEventHeader.Version._4_0_1 + 'pnfRegistration' + org.onap.dcaegen2.ves.domain.ves70.CommonEventHeader.VesEventListenerVersion._7_0_1 + 'registration_'+commonEventHeader.ts1 + 'VESMapper' + org.onap.dcaegen2.ves.domain.ves70.CommonEventHeader.Domain.PNF_REGISTRATION + commonEventHeader.domain + + commonEventHeader.ts1 + commonEventHeader.ts1 + org.onap.dcaegen2.ves.domain.ves70.CommonEventHeader.Priority.NORMAL + pnfRegistrationFields.vendorName+'-'+pnfRegistrationFields.serialNumber + + + org.onap.dcaegen2.ves.domain.ves70.PnfRegistrationFields.PnfRegistrationFieldsVersion._2_0 + + + + + + + + + + + + + + + + + + diff --git a/docs/sections/services/mapper/troubleshooting.rst b/docs/sections/services/mapper/troubleshooting.rst new file mode 100644 index 00000000..196faa0d --- /dev/null +++ b/docs/sections/services/mapper/troubleshooting.rst @@ -0,0 +1,82 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +.. _troubleshooting: + +Troubleshooting +=============== + +**NOTE** + +According to **ONAP** logging policy, **Mapper** logs contain all required markers as well as service and client specific Mapped Diagnostic Context (later referred as MDC) + +Default console log pattern: + +:: + + |%date{"HH:mm:ss.SSSXXX", UTC}\t[ %thread\t] %highlight(%-5level)\t - %msg\t + +A sample, fully qualified message implementing this pattern: + +:: + + |11:10:13.230 [rcc-notification] INFO metricsLogger - fetch and publish from and to Dmaap started:rcc-notification + + +For simplicity, all log messages in this section are shortened to contain only: + * logger name + * log level + * message + +Error and warning logs contain also: + * exception message + * stack trace + +**Do not rely on exact log messages or their presence, as they are often subject to change.** + +Deployment/Installation errors +-------------------- + +**Missing Default Config File in case of using local config instead of Consul** + +:: + + + |13:04:37.535 [main] ERROR errorLogger - Default Config file kv.json is missing + |13:04:37.537 [main] ERROR errorLogger - Application stoped due to missing default Config file + |13:04:37.538 [main] INFO o.s.s.c.ThreadPoolTaskExecutor - Shutting down ExecutorService 'applicationTaskExecutor' + |15:40:43.982 [main] WARN debugLogger - All Smooks objects closed +**These log messages are printed when the default configuration file "kv.json", was not present.** + + + +**Invalid Default Config File in case of using local config instead of Consul** + +If Default Config File is an invalid json file, we will get below exception + +:: + + |15:19:52.489 [main] ERROR o.s.boot.SpringApplication - Application run failed + |java.lang.IllegalStateException: Failed to execute CommandLineRunner + at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:816) + at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:797) + at org.springframework.boot.SpringApplication.run(SpringApplication.java:324) + at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260) + at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248) + at org.onap.universalvesadapter.Application.main(Application.java:29) + |Caused by: org.json.JSONException: Expected a ',' or '}' at 8100 [character 2 line 54] + at org.json.JSONTokener.syntaxError(JSONTokener.java:433) + at org.json.JSONObject.(JSONObject.java:229) + at org.json.JSONObject.(JSONObject.java:321) + at org.onap.universalvesadapter.utils.FetchDynamicConfig.verifyConfigChange(FetchDynamicConfig.java:97) + at org.onap.universalvesadapter.utils.FetchDynamicConfig.cbsCall(FetchDynamicConfig.java:66) + at org.onap.universalvesadapter.service.VESAdapterInitializer.run(VESAdapterInitializer.java:83) + at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:813) + ... 5 common frames omitted + |15:19:52.492 [main] INFO o.s.s.c.ThreadPoolTaskExecutor - Shutting down ExecutorService 'applicationTaskExecutor' + |15:19:52.493 [main] WARN debugLogger - All Smooks objects closed + + +**Invalid Smooks mapping file** + +If VES-Mapper blueprint or local config file contains invalid Smooks mapping file, then we will get below SAXException / JsonProcessingException / JsonSyntaxException / JsonParseException while processing the incoming notifications and the notification will be dropped without converting into required VES event. All such dropped notifications will be logged in error log file. \ No newline at end of file -- cgit 1.2.3-korg