diff options
| -rw-r--r-- | pnda-ztt-app/README.md | 145 |
1 files changed, 145 insertions, 0 deletions
diff --git a/pnda-ztt-app/README.md b/pnda-ztt-app/README.md index 9ab1d53..50a4add 100644 --- a/pnda-ztt-app/README.md +++ b/pnda-ztt-app/README.md @@ -11,3 +11,148 @@ timeseries datapoints and writes them to OpenTSDB. This application demonstrates ingest of VES events from a virtual firewall into PNDA. The extracted metrics get stored in HDFS and the timeseries database. This enables direct visualization via Grafana as well as downstream Spark based analytics. + +The __Virtual firewall (vFW)__ use case that is part of ONAP will have the following flow for its data: + + ONAP : VES -> ONAP : DMaaP -> Logstash -> PNDA : Kafka -> PNDA : Spark application -> PNDA : OpenTSDB + +## Prerequisites + +### ONAP : DMaaP data exposure + +_Instructions_ + +If the port of the ONAP : DMaaP component is not exposed, it is required to adjust this accordingly: + +```bash +# Find the message-router-kafka pod +kubectl -n onap get pods | grep message-router-kafka + +# Create a loadbalancer for the ONAP : DMaaP component +kubectl -n onap expose deployment <k8s-cluster-name>-message-router-kafka --type=LoadBalancer --name=<k8s-cluster-name>-message-router-kafka-lb + +# Check the outcome +kubectl -n onap get services | grep message-router-kafka-lb +``` + +This adds a public endpoint to the _message-router-kafka_ pod. It gives a public IP address and exposes the port `9092` through the new service of type LoadBalancer. In addition to this, we need to add the hostname `message-router-kafka` and the public IP address to the instances that run Logstash and the Spark application. + +### Setup the Logstash component + +Logstash is used for three reasons: + +- to read the data from the ONAP : DMaaP component +- to serialize the data using [AVRO encoding](https://avro.apache.org/docs/current/) +- to pass the data to the PNDA : Kafka broker + +This application has been tested with _Logstash 5.6.8_. + +_Instructions_ + +Launch a Logstash instance part of a network that can reach the public endpoint of the _message-router-kafka_ pod. + +```bash +# Download Logstash +sudo yum install wget +wget https://artifacts.elastic.co/downloads/logstash/logstash-5.6.8.tar.gz +tar zxvf logstash-5.6.8.tar.gz + +# Download PNDA Logstash AVRO codec +git clone https://github.com/pndaproject/logstash-codec-pnda-avro +cd logstash-codec-pnda-avro +git checkout release/3.6 + +# Download Ruby +curl -sSL https://get.rvm.io | bash -s stable —ruby +ruby -v + +# Build the codec +gem build logstash-codec-pnda-avro.gemspec + +# Install the codec as a Logstash plugin +../logstash-5.6.8/bin/logstash-plugin install logstash-codec-pnda-avro-3.1.1-java.gem +``` + +Create a Logstash configuration file: +```bash +cat logstash.conf +input { + kafka { + topics => ["unauthenticated.VES_MEASUREMENT_OUTPUT"] + bootstrap_servers => "<IP-ONAP-MESSAGE-ROUTER-KAFKA>:9092" + } +} + +filter { + mutate { + add_field => { + "src" => "ves" + "host_ip" => "onap.demon" + } + rename => { "message" => "rawdata"} + } + ruby { + code => "event.set('timestamp', (event.get('@timestamp').to_f * 1000).to_i)" + } +} + +output { + kafka { + topic_id => "ves.avro" + bootstrap_servers => "<IP-PNDA-KAFKA>:9094" + value_serializer => "org.apache.kafka.common.serialization.ByteArraySerializer" + codec => pnda-avro{ + schema_uri => "dataplatform-raw.avsc" + } + } + stdout { + codec => "json" + } +} + +``` + +Launch Logstash: +```bash +../logstash-5.6.8/bin/logstash -f logstash.conf & +``` + +### Create the output topic in PNDA Kafka +```bash +/opt/pnda/kafka/bin/kafka-topics.sh --create --topic ves.avro --partitions 2 --replication-factor 1 --zookeeper=<IP-PNDA-KAFKA-ZOOKEEPER>:2181 +``` + +### Specify the PNDA gateway + +Before building the application, configure the server that will be used for the package deployment by overwriting the value of the `SERVER` variable in the Makefile. Replace the `knox.example.com` with the address of the PNDA gateway. + +```bash +cd pnda-ztt-app +vi Makefile + +SERVER=https://knox.example.com:8443/gateway/pnda/repository +``` + +### Build the application + +```bash +make app +``` + +### Package the application +```bash +make package +``` + +### Upload the application to the package manager +```bash +make deploy +``` + +### Deploy the application + +Access the [list of packages in the PNDA console](https://knox.example.com:8443/gateway/pnda/console#/packages/pnda-ztt-app). Click _+ Deploy_ in order to deploy the application. + +### Launch the application + +Access the [list of applications in the PNDA console](https://knox.example.com:8443/gateway/pnda/console#/applications). Click _Create New Application_. |