How-to for vFW monitoring PNDA application

Change-Id: I37169f934ea988a94b6c96adc0c224111992880e
Issue-ID: DCAEGEN2-1023
Signed-off-by: Cristina Precup <cprecup@cisco.com>

1 files changed, 145 insertions, 0 deletions

diff --git a/pnda-ztt-app/README.md b/pnda-ztt-app/README.md
index 9ab1d53..50a4add 100644
--- a/pnda-ztt-app/README.md
+++ b/pnda-ztt-app/README.md
@@ -11,3 +11,148 @@ timeseries datapoints and writes them to OpenTSDB.
 This application demonstrates ingest of VES events from a virtual firewall into PNDA. The
 extracted metrics get stored in HDFS and the timeseries database. This enables direct
 visualization via Grafana as well as downstream Spark based analytics.
+
+The __Virtual firewall (vFW)__ use case that is part of ONAP will have the following flow for its data:
+
+    ONAP : VES -> ONAP : DMaaP -> Logstash -> PNDA : Kafka -> PNDA : Spark application -> PNDA : OpenTSDB
+
+## Prerequisites
+
+### ONAP : DMaaP data exposure
+
+_Instructions_
+
+If the port of the ONAP : DMaaP component is not exposed, it is required to adjust this accordingly:
+
+```bash
+# Find the message-router-kafka pod
+kubectl -n onap get pods | grep message-router-kafka
+
+# Create a loadbalancer for the ONAP : DMaaP component
+kubectl -n onap expose deployment <k8s-cluster-name>-message-router-kafka --type=LoadBalancer --name=<k8s-cluster-name>-message-router-kafka-lb
+
+# Check the outcome
+kubectl -n onap get services | grep message-router-kafka-lb
+```
+
+This adds a public endpoint to the _message-router-kafka_ pod. It gives a public IP address and exposes the port `9092` through the new service of type LoadBalancer. In addition to this, we need to add the hostname `message-router-kafka` and the public IP address to the instances that run Logstash and the Spark application.
+
+### Setup the Logstash component
+
+Logstash is used for three reasons:
+
+- to read the data from the ONAP : DMaaP component
+- to serialize the data using [AVRO encoding](https://avro.apache.org/docs/current/)
+- to pass the data to the PNDA : Kafka broker
+
+This application has been tested with _Logstash 5.6.8_.
+
+_Instructions_
+
+Launch a Logstash instance part of a network that can reach the public endpoint of the _message-router-kafka_ pod.
+
+```bash
+# Download Logstash
+sudo yum install wget
+wget https://artifacts.elastic.co/downloads/logstash/logstash-5.6.8.tar.gz
+tar zxvf logstash-5.6.8.tar.gz
+
+# Download PNDA Logstash AVRO codec
+git clone https://github.com/pndaproject/logstash-codec-pnda-avro
+cd logstash-codec-pnda-avro
+git checkout release/3.6
+
+# Download Ruby
+curl -sSL https://get.rvm.io | bash -s stable —ruby         
+ruby -v
+
+# Build the codec
+gem build logstash-codec-pnda-avro.gemspec
+
+# Install the codec as a Logstash plugin
+../logstash-5.6.8/bin/logstash-plugin install logstash-codec-pnda-avro-3.1.1-java.gem  
+```
+
+Create a Logstash configuration file:
+```bash
+cat logstash.conf
+input {
+  kafka {
+    topics => ["unauthenticated.VES_MEASUREMENT_OUTPUT"]
+    bootstrap_servers => "<IP-ONAP-MESSAGE-ROUTER-KAFKA>:9092"
+  }
+}
+
+filter {
+  mutate {
+    add_field => {
+      "src" => "ves"
+      "host_ip" => "onap.demon"
+    }
+    rename => { "message" => "rawdata"}
+  }
+  ruby {
+    code => "event.set('timestamp', (event.get('@timestamp').to_f * 1000).to_i)"
+  }
+}
+
+output {
+  kafka {
+    topic_id => "ves.avro"
+    bootstrap_servers => "<IP-PNDA-KAFKA>:9094"
+    value_serializer => "org.apache.kafka.common.serialization.ByteArraySerializer"
+    codec => pnda-avro{
+      schema_uri => "dataplatform-raw.avsc"
+    }
+  }
+  stdout {
+    codec => "json"
+  }
+}
+
+```
+
+Launch Logstash:
+```bash
+../logstash-5.6.8/bin/logstash -f logstash.conf &
+```
+
+### Create the output topic in PNDA Kafka
+```bash
+/opt/pnda/kafka/bin/kafka-topics.sh --create --topic ves.avro --partitions 2 --replication-factor 1 --zookeeper=<IP-PNDA-KAFKA-ZOOKEEPER>:2181
+```
+
+### Specify the PNDA gateway
+
+Before building the application, configure the server that will be used for the package deployment by overwriting the value of the `SERVER` variable in the Makefile. Replace the `knox.example.com` with the address of the PNDA gateway.
+
+```bash
+cd pnda-ztt-app
+vi Makefile
+
+SERVER=https://knox.example.com:8443/gateway/pnda/repository
+```
+
+### Build the application
+
+```bash
+make app
+``` 
+
+### Package the application
+```bash
+make package
+```
+
+### Upload the application to the package manager
+```bash
+make deploy
+```
+
+### Deploy the application
+
+Access the [list of packages in the PNDA console](https://knox.example.com:8443/gateway/pnda/console#/packages/pnda-ztt-app). Click _+ Deploy_ in order to deploy the application.
+
+### Launch the application
+
+Access the [list of applications in the PNDA console](https://knox.example.com:8443/gateway/pnda/console#/applications). Click _Create New Application_.