summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
-rw-r--r--pnda-ztt-app/README.md145
1 files changed, 145 insertions, 0 deletions
diff --git a/pnda-ztt-app/README.md b/pnda-ztt-app/README.md
index 9ab1d53..50a4add 100644
--- a/pnda-ztt-app/README.md
+++ b/pnda-ztt-app/README.md
@@ -11,3 +11,148 @@ timeseries datapoints and writes them to OpenTSDB.
This application demonstrates ingest of VES events from a virtual firewall into PNDA. The
extracted metrics get stored in HDFS and the timeseries database. This enables direct
visualization via Grafana as well as downstream Spark based analytics.
+
+The __Virtual firewall (vFW)__ use case that is part of ONAP will have the following flow for its data:
+
+ ONAP : VES -> ONAP : DMaaP -> Logstash -> PNDA : Kafka -> PNDA : Spark application -> PNDA : OpenTSDB
+
+## Prerequisites
+
+### ONAP : DMaaP data exposure
+
+_Instructions_
+
+If the port of the ONAP : DMaaP component is not exposed, it is required to adjust this accordingly:
+
+```bash
+# Find the message-router-kafka pod
+kubectl -n onap get pods | grep message-router-kafka
+
+# Create a loadbalancer for the ONAP : DMaaP component
+kubectl -n onap expose deployment <k8s-cluster-name>-message-router-kafka --type=LoadBalancer --name=<k8s-cluster-name>-message-router-kafka-lb
+
+# Check the outcome
+kubectl -n onap get services | grep message-router-kafka-lb
+```
+
+This adds a public endpoint to the _message-router-kafka_ pod. It gives a public IP address and exposes the port `9092` through the new service of type LoadBalancer. In addition to this, we need to add the hostname `message-router-kafka` and the public IP address to the instances that run Logstash and the Spark application.
+
+### Setup the Logstash component
+
+Logstash is used for three reasons:
+
+- to read the data from the ONAP : DMaaP component
+- to serialize the data using [AVRO encoding](https://avro.apache.org/docs/current/)
+- to pass the data to the PNDA : Kafka broker
+
+This application has been tested with _Logstash 5.6.8_.
+
+_Instructions_
+
+Launch a Logstash instance part of a network that can reach the public endpoint of the _message-router-kafka_ pod.
+
+```bash
+# Download Logstash
+sudo yum install wget
+wget https://artifacts.elastic.co/downloads/logstash/logstash-5.6.8.tar.gz
+tar zxvf logstash-5.6.8.tar.gz
+
+# Download PNDA Logstash AVRO codec
+git clone https://github.com/pndaproject/logstash-codec-pnda-avro
+cd logstash-codec-pnda-avro
+git checkout release/3.6
+
+# Download Ruby
+curl -sSL https://get.rvm.io | bash -s stable —ruby
+ruby -v
+
+# Build the codec
+gem build logstash-codec-pnda-avro.gemspec
+
+# Install the codec as a Logstash plugin
+../logstash-5.6.8/bin/logstash-plugin install logstash-codec-pnda-avro-3.1.1-java.gem
+```
+
+Create a Logstash configuration file:
+```bash
+cat logstash.conf
+input {
+ kafka {
+ topics => ["unauthenticated.VES_MEASUREMENT_OUTPUT"]
+ bootstrap_servers => "<IP-ONAP-MESSAGE-ROUTER-KAFKA>:9092"
+ }
+}
+
+filter {
+ mutate {
+ add_field => {
+ "src" => "ves"
+ "host_ip" => "onap.demon"
+ }
+ rename => { "message" => "rawdata"}
+ }
+ ruby {
+ code => "event.set('timestamp', (event.get('@timestamp').to_f * 1000).to_i)"
+ }
+}
+
+output {
+ kafka {
+ topic_id => "ves.avro"
+ bootstrap_servers => "<IP-PNDA-KAFKA>:9094"
+ value_serializer => "org.apache.kafka.common.serialization.ByteArraySerializer"
+ codec => pnda-avro{
+ schema_uri => "dataplatform-raw.avsc"
+ }
+ }
+ stdout {
+ codec => "json"
+ }
+}
+
+```
+
+Launch Logstash:
+```bash
+../logstash-5.6.8/bin/logstash -f logstash.conf &
+```
+
+### Create the output topic in PNDA Kafka
+```bash
+/opt/pnda/kafka/bin/kafka-topics.sh --create --topic ves.avro --partitions 2 --replication-factor 1 --zookeeper=<IP-PNDA-KAFKA-ZOOKEEPER>:2181
+```
+
+### Specify the PNDA gateway
+
+Before building the application, configure the server that will be used for the package deployment by overwriting the value of the `SERVER` variable in the Makefile. Replace the `knox.example.com` with the address of the PNDA gateway.
+
+```bash
+cd pnda-ztt-app
+vi Makefile
+
+SERVER=https://knox.example.com:8443/gateway/pnda/repository
+```
+
+### Build the application
+
+```bash
+make app
+```
+
+### Package the application
+```bash
+make package
+```
+
+### Upload the application to the package manager
+```bash
+make deploy
+```
+
+### Deploy the application
+
+Access the [list of packages in the PNDA console](https://knox.example.com:8443/gateway/pnda/console#/packages/pnda-ztt-app). Click _+ Deploy_ in order to deploy the application.
+
+### Launch the application
+
+Access the [list of applications in the PNDA console](https://knox.example.com:8443/gateway/pnda/console#/applications). Click _Create New Application_.