aboutsummaryrefslogtreecommitdiffstats
path: root/test/mocks/datafilecollector-testharness/simulator-group
diff options
context:
space:
mode:
authorBjornMagnussonXA <bjorn.magnusson@est.tech>2019-04-10 13:04:08 +0000
committerBjornMagnussonXA <bjorn.magnusson@est.tech>2019-04-10 13:04:08 +0000
commitf4e1836eab2cff83bc2542383ed7f97d98a7b68c (patch)
tree5c3efb2db7bddb5213fa60ba15dc1643a835c653 /test/mocks/datafilecollector-testharness/simulator-group
parentefbb5adf8bd45c606565dd1fea1f71f930d3e5dc (diff)
Enhancements of simulators for DFC
New test cases for MR and DR simulators Added simulator monitor server for simulator supervision Update - Adapted simulators to changed naming of ftp files Update2 - Added some more test behaviour to simulators. Issue-ID: DCAEGEN2-1313 Change-Id: I90a346d44ab4e7c9f5a65f599b64f1907525dd51 Signed-off-by: BjornMagnussonXA <bjorn.magnusson@est.tech>
Diffstat (limited to 'test/mocks/datafilecollector-testharness/simulator-group')
-rw-r--r--test/mocks/datafilecollector-testharness/simulator-group/README.md46
-rwxr-xr-xtest/mocks/datafilecollector-testharness/simulator-group/docker-compose-setup.sh11
-rw-r--r--test/mocks/datafilecollector-testharness/simulator-group/docker-compose-template.yml22
-rwxr-xr-xtest/mocks/datafilecollector-testharness/simulator-group/setup-ftp-files-for-image.sh43
-rwxr-xr-xtest/mocks/datafilecollector-testharness/simulator-group/sim-monitor-start.sh7
-rw-r--r--test/mocks/datafilecollector-testharness/simulator-group/sim-monitor.js165
-rwxr-xr-xtest/mocks/datafilecollector-testharness/simulator-group/simulators-start.sh30
7 files changed, 291 insertions, 33 deletions
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/README.md b/test/mocks/datafilecollector-testharness/simulator-group/README.md
index 4d448f8d2..5981c7978 100644
--- a/test/mocks/datafilecollector-testharness/simulator-group/README.md
+++ b/test/mocks/datafilecollector-testharness/simulator-group/README.md
@@ -4,15 +4,17 @@ Mainly this is needed for CSIT tests but can be used also for local testing.
###Preparation
+Do the manual steps to prepare the simulator images
+
Build the mr-sim image.
cd ../mr-sim
-Run the docker build command to build the image for the MR simulator: 'docker build -t mrsim:latest .
+Run the docker build command to build the image for the MR simulator: 'docker build -t mrsim:latest .'
cd ../dr-sim
-Run the docker build command to build the image for the DR simulators: `docker build -t drsim_common:latest .
+Run the docker build command to build the image for the DR simulators: `docker build -t drsim_common:latest .'
cd ../simulator-group
@@ -21,21 +23,53 @@ Copy the 'configuration' and 'tls' catalogues from the ftps-sftp-server dir.
Check the README.md in ftps-sftp-server dir in case the cert need to be updated.
-cp -r ./ftps-sftp-server/configuration .
+cp -r ../ftps-sftp-server/configuration .
cp -r ../ftps-sftp-server/tls .
###Execution
-Edit the `docker-compose-setup.sh` to setup the env variables to the desired test behavior for each simulators.
+Edit the `docker-compose-setup.sh` (or create a copy) to setup the env variables to the desired test behavior for each simulators.
See each simulator to find a description of the available settings.
+The following env variables shall be set (example values).
+Note that NUM_FTPFILES and NUM_PNFS controls the number of ftp files created in the ftp servers.
+A total of NUM_FTPFILES * NUM_PNFS ftp files will be created in each dtp server (4 files in the below example).
+Large settings will be time consuming at start of the servers.
+
+DR_TC="--tc normal" #Normal behavior of the DR sim
+
+DR_REDIR_TC="--tc normal" #Normal behavior of the DR redirect sim
+
+MR_TC="--tc100" #One 1 MB file in one event, once.
+
+BC_TC="" #Not in use yet
+
+NUM_FTPFILES="2" #Two file for each PNF
-Run the script `docker-compose-setup.sh`to create a docker-compose with the desired settings. All simulators
-will be started with the generated docker-compose.yml file
+NUM_PNFS="2" #Two PNFs
+
+Run the script `docker-compose-setup.sh`to create a docker-compose with the desired settings. The desired setting
+in the script need to be manually adapted to for each specific simulator behavior according to the above. Check each simulator for available
+parameters.
+All simulators will be started with the generated docker-compose.yml file
+
+To generate ftp url with IP different from localhost, set SFTP_SIM_IP and/or FTPS_SIM_IP env variables to the addreses of the ftp servers before starting.
+So farm, this only works when the simulator python script is started from the command line.
Kill all the containers with `simulators-kill.se`
`simulators_start.sh` is for CSIT test and requires the env variables for test setting to be present in the shell.
`setup-ftp-files.for-image.sh` is for CSIT and executed when the ftp servers are started from the docker-compose-setup.sh`.
+To make DFC to be able to connect to the simulator containers, DFC need to run in host mode.
+Start DFC by the following cmd: `docker run -d --network="host" --name dfc_app <dfc-image> `
+
+`<dfc-image>` could be either the locally built image `onap/org.onap.dcaegen2.collectors.datafile.datafile-app-server`
+or the one in nexus `nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.datafile.datafile-app-server`.
+
+
+
+###Simulator monitor
+Start the simulator monitor server with `sim-monitor-start.sh` and the open a browser with the url `localhost:9999/mon`
+to see the statisics page with data from MR sim, DR sim and DR redir sim.
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-setup.sh b/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-setup.sh
index d0171a5e3..af36d0562 100755
--- a/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-setup.sh
+++ b/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-setup.sh
@@ -2,10 +2,13 @@
#Script for manually starting all simulators with test setting below
-export DR_TC=""
-export DR_REDIR_TC=""
+export DR_TC="--tc normal"
+export DR_REDIR_TC="--tc normal"
export MR_TC="--tc100"
-export BC_TC=""
-export NUM_FTPFILES="250"
+export BC_TC="" #Not in use yet
+export NUM_FTPFILES="10"
+export NUM_PNFS="700"
+export FILE_SIZE="1MB"
+export FTP_TYPE="SFTP"
source ./simulators-start.sh \ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-template.yml b/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-template.yml
index e7c700759..89a45a476 100644
--- a/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-template.yml
+++ b/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-template.yml
@@ -1,8 +1,17 @@
version: '2'
+
+networks:
+ dfcnet:
+ ipam:
+ config:
+ - subnet: 192.168.100.0/16
+
services:
drsim:
- network_mode: "host"
+ networks:
+ dfcnet:
+ ipv4_address: 192.168.100.2
image: drsim_common:latest
ports:
- "3906:3906"
@@ -11,6 +20,9 @@ services:
command: node dmaapDR.js ${DR_TC}
drsim_redir:
+ networks:
+ dfcnet:
+ ipv4_address: 192.168.100.3
image: drsim_common:latest
ports:
- "3908:3908"
@@ -19,6 +31,9 @@ services:
command: node dmaapDR_redir.js ${DR_REDIR_TC}
mrsim:
+ networks:
+ dfcnet:
+ ipv4_address: 192.168.100.1
image: mrsim:latest
ports:
- "2222:2222"
@@ -26,17 +41,17 @@ services:
command: python mr-sim.py ${MR_TC}
sftp-server:
+ network_mode: bridge
container_name: dfc_sftp-server
image: atmoz/sftp:alpine
ports:
- "1022:22"
-# volumes:
-# - ./files/onap/sftp/:/home/onap/
restart: on-failure
command: onap:pano:1001
ftpes-server-vsftpd:
+ network_mode: bridge
container_name: dfc_ftpes-server-vsftpd
image: docker.io/panubo/vsftpd
ports:
@@ -54,6 +69,5 @@ services:
- ./tls/dfc.crt:/etc/ssl/private/dfc.crt:ro
- ./configuration/vsftpd_ssl.conf:/etc/vsftpd_ssl.conf:ro
-# - ./files/onap/ftps/:/srv/
restart: on-failure
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/setup-ftp-files-for-image.sh b/test/mocks/datafilecollector-testharness/simulator-group/setup-ftp-files-for-image.sh
index d8e57a3d8..6c2b4f29f 100755
--- a/test/mocks/datafilecollector-testharness/simulator-group/setup-ftp-files-for-image.sh
+++ b/test/mocks/datafilecollector-testharness/simulator-group/setup-ftp-files-for-image.sh
@@ -1,28 +1,47 @@
-#!/bin/bash
+#!/usr/bin/env bash
# Script to create files for the FTP server to return upon request.
# The file names matches the files names in the events polled from the MR simulator.
# Intended for execution in the running ftp containers in the ftp-root dir.
-echo "Running ftp file creations"
-
NUM=200 #Default number of files
+PNFS=1 #Default number of PNFs
+FSIZE="ALL"
if [ $# -eq 1 ]; then
NUM=$1
+elif [ $# -eq 2 ]; then
+ NUM=$1
+ PNFS=$2
+elif [ $# -eq 3 ]; then
+ NUM=$1
+ PNFS=$2
+ FSIZE=$3
+ if [ $3 != "1KB" ] && [ $3 != "1MB" ] && [ $3 != "5MB" ] && [ $3 != "50MB" ] && [ $3 != "ALL" ]; then
+ echo "File size shall be 1KB|1MB|5MB|50MB|ALL"
+ exit
+ fi
+else
+ echo "Wrong args, usage: setup-ftp-files-for-image.sh [ <num-files> [ <num-PNFs> [ 1KB|1MB|5MB|50MB ] ] ]"
+ exit
fi
+echo "Running ftp file creations. " $PNFS " PNFs and " $NUM " files for each PNF with file size(s) "$FSIZE
+
+truncate -s 1KB 1KB.tar.gz
truncate -s 1MB 1MB.tar.gz
truncate -s 5MB 5MB.tar.gz
truncate -s 50MB 50MB.tar.gz
-
-i=0
-while [ $i -lt $NUM ]; do #Problem with for loop and var substituion in curly bracket....so used good old style loop
- ln -s 1MB.tar.gz 1MB_$i.tar.gz
- ln -s 5MB.tar.gz 5MB_$i.tar.gz
- let i=i+1
+p=0
+while [ $p -lt $PNFS ]; do
+ i=0
+ while [ $i -lt $NUM ]; do #Problem with for loop and var substituion in curly bracket....so used good old style loop
+ if [ $FSIZE = "ALL" ] || [ $FSIZE = "1KB" ]; then ln -s 1KB.tar.gz 'A20000626.2315+0200-2330+0200_PNF'$p'-'$i'-1KB.tar.gz'; fi
+ if [ $FSIZE = "ALL" ] || [ $FSIZE = "1MB" ]; then ln -s 1MB.tar.gz 'A20000626.2315+0200-2330+0200_PNF'$p'-'$i'-1MB.tar.gz'; fi
+ if [ $FSIZE = "ALL" ] || [ $FSIZE = "5MB" ]; then ln -s 5MB.tar.gz 'A20000626.2315+0200-2330+0200_PNF'$p'-'$i'-5MB.tar.gz'; fi
+ if [ $FSIZE = "ALL" ] || [ $FSIZE = "50MB" ]; then ln -s 50MB.tar.gz 'A20000626.2315+0200-2330+0200_PNF'$p'-'$i'-50MB.tar.gz'; fi
+ let i=i+1
+ done
+ let p=p+1
done
-
-
-ln -s 50MB.tar.gz 50MB_0.tar.gz #Large file, only for single file test
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/sim-monitor-start.sh b/test/mocks/datafilecollector-testharness/simulator-group/sim-monitor-start.sh
new file mode 100755
index 000000000..52c8c1cac
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/sim-monitor-start.sh
@@ -0,0 +1,7 @@
+#/bin/bash
+
+#Script to start the sim-monitor
+
+#Re-using modules for dr-sim
+cp -r ../dr-sim/node_modules .
+node sim-monitor.js \ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/sim-monitor.js b/test/mocks/datafilecollector-testharness/simulator-group/sim-monitor.js
new file mode 100644
index 000000000..e4a19c21e
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/sim-monitor.js
@@ -0,0 +1,165 @@
+var http = require('http');
+
+var express = require('express');
+var app = express();
+
+//I am alive
+app.get("/",function(req, res){
+ res.send("ok");
+})
+
+//Get parameter valuye from other server
+function getSimCtr(url, cb) {
+ var data = '';
+ http.get(url, (resp) => {
+ // A chunk of data has been recieved.
+ resp.on('data', (chunk) => {
+ data += chunk;
+ });
+
+ // The whole response has been received.
+ resp.on('end', () => {
+ //Pad data to fixed length
+ var i = 20-data.length;
+ while(i>0) {
+ data = data+"&nbsp;";
+ i--;
+ }
+ cb(data);
+ });
+
+ }).on("error", (err) => {
+ console.log("Error: " + err.message);
+ cb("no response from simulator");
+ });
+};
+
+//Status variables, for parameters values fetched from other simulators
+var mr1, mr2, mr3, mr4, mr5, mr6, mr7;
+
+var dr1, dr2, dr3, dr4, dr5, dr6, dr7, dr8, dr9;
+
+var drr1, drr2, drr3, drr4, drr5, drr6;
+
+app.get("/mon",function(req, res){
+
+ //MR
+ getSimCtr("http://127.0.0.1:2222/ctr_requests", function(data) {
+ mr1 = data;
+ });
+ getSimCtr("http://127.0.0.1:2222/ctr_responses", function(data) {
+ mr2 = data;
+ });
+ getSimCtr("http://127.0.0.1:2222/ctr_unique_files", function(data) {
+ mr3 = data;
+ });
+ getSimCtr("http://127.0.0.1:2222/tc_info", function(data) {
+ mr4 = data;
+ });
+ getSimCtr("http://127.0.0.1:2222/ctr_events", function(data) {
+ mr5 = data;
+ });
+ getSimCtr("http://127.0.0.1:2222/execution_time", function(data) {
+ mr6 = data;
+ });
+ getSimCtr("http://127.0.0.1:2222/ctr_unique_PNFs", function(data) {
+ mr7 = data;
+ });
+
+ //DR
+ getSimCtr("http://127.0.0.1:3906/ctr_publish_query", function(data) {
+ dr1 = data;
+ });
+ getSimCtr("http://127.0.0.1:3906/ctr_publish_query_published", function(data) {
+ dr2 = data;
+ });
+ getSimCtr("http://127.0.0.1:3906/ctr_publish_query_not_published", function(data) {
+ dr3 = data;
+ });
+ getSimCtr("http://127.0.0.1:3906/ctr_publish_req", function(data) {
+ dr4 = data;
+ });
+ getSimCtr("http://127.0.0.1:3906/ctr_publish_req_redirect", function(data) {
+ dr5 = data;
+ });
+ getSimCtr("http://127.0.0.1:3906/ctr_publish_req_published", function(data) {
+ dr6 = data;
+ });
+ getSimCtr("http://127.0.0.1:3906/ctr_published_files", function(data) {
+ dr7 = data;
+ });
+ getSimCtr("http://127.0.0.1:3906/tc_info", function(data) {
+ dr8 = data;
+ });
+ getSimCtr("http://127.0.0.1:3906/execution_time", function(data) {
+ dr9 = data;
+ });
+
+ //DR REDIR
+ getSimCtr("http://127.0.0.1:3908/ctr_publish_requests", function(data) {
+ drr1 = data;
+ });
+ getSimCtr("http://127.0.0.1:3908/ctr_publish_responses", function(data) {
+ drr2 = data;
+ });
+ getSimCtr("http://127.0.0.1:3908/tc_info", function(data) {
+ drr3 = data;
+ });
+ getSimCtr("http://127.0.0.1:3908/execution_time", function(data) {
+ drr4 = data;
+ });
+ getSimCtr("http://127.0.0.1:3908/time_lastpublish", function(data) {
+ drr5 = data;
+ });
+ getSimCtr("http://127.0.0.1:3908/dwl_volume", function(data) {
+ drr6 = data;
+ });
+
+ //Build web page
+ var str = "<!DOCTYPE html>" +
+ "<html>" +
+ "<head>" +
+ "<meta http-equiv=\"refresh\" content=\"5\">"+ //5 sec auto reefresh
+ "<title>Simulator monitor</title>"+
+ "</head>" +
+ "<body>" +
+ "<h3>MR Simulator</h3>" +
+ "<font face=\"courier\">"+
+ "MR TC:........................................." + mr4 + "<br>" +
+ "Execution time (mm.ss):........................" + mr6 + "<br>" +
+ "Number of requests (polls):...................." + mr1 + "<br>" +
+ "Number of responses (polls):..................." + mr2 + "<br>" +
+ "Number of unique files in all responses:......." + mr3 + "<br>" +
+ "Number of events..............................." + mr5 + "<br>" +
+ "Number of unique PNFs.........................." + mr7 + "<br>" +
+ "</font>"+
+ "<h3>DR Simulator</h3>" +
+ "<font face=\"courier\">"+
+ "DR TC:........................................." + dr8 + "<br>" +
+ "Execution time (mm.ss):........................" + dr9 + "<br>" +
+ "Number of queries:............................." + dr1 + "<br>" +
+ "Number of query responses, file published:....." + dr2 + "<br>" +
+ "Number of query responses, file not published:." + dr3 + "<br>" +
+ "Number of requests:............................" + dr4 + "<br>" +
+ "Number of responses with redirect:............." + dr5 + "<br>" +
+ "Number of responses without redirect:.........." + dr6 + "<br>" +
+ "Number of published files:....................." + dr7 + "<br>" +
+ "</font>"+
+ "<h3>DR Redirect Simulator</h3>" +
+ "<font face=\"courier\">"+
+ "DR REDIR TC:..................................." + drr3 + "<br>" +
+ "Execution time (mm.ss):........................" + drr4 + "<br>" +
+ "Number of requests:............................" + drr1 + "<br>" +
+ "Number of responses:..........................." + drr2 + "<br>" +
+ "Downloaded volume (bytes):....................." + drr6 + "<br>" +
+ "Last publish (mm:ss):.........................." + drr5 + "<br>" +
+ "</font>"+
+ "</body>" +
+ "</html>";
+ res.send(str);
+})
+
+var httpServer = http.createServer(app);
+var httpPort=9999;
+httpServer.listen(httpPort);
+console.log("Simulator monitor listening (http) at "+httpPort); \ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/simulators-start.sh b/test/mocks/datafilecollector-testharness/simulator-group/simulators-start.sh
index ef1b90a38..cc5ce32ec 100755
--- a/test/mocks/datafilecollector-testharness/simulator-group/simulators-start.sh
+++ b/test/mocks/datafilecollector-testharness/simulator-group/simulators-start.sh
@@ -35,11 +35,27 @@ if [ -z "$NUM_FTPFILES" ]
then
NUM_FTPFILES=200
fi
+if [ -z "$NUM_PNFS" ]
+ then
+ NUM_PNFS=1
+fi
+if [ -z "$FILE_SIZE" ]
+ then
+ FILE_SIZE="ALL"
+fi
+if [ -z "$FTP_TYPE" ]
+ then
+ FTP_TYPE="ALL"
+fi
-
-docker cp setup-ftp-files-for-image.sh $SFTP_SIM:/tmp/
-docker exec -w /home/onap/ $SFTP_SIM /tmp/setup-ftp-files-for-image.sh $NUM_FTPFILES >/dev/null 2>&1
-
-docker cp setup-ftp-files-for-image.sh $FTPS_SIM:/tmp/setup-ftp-files-for-image.sh
-docker exec -w /srv $FTPS_SIM /tmp/setup-ftp-files-for-image.sh $NUM_FTPFILES >/dev/null 2>&1
-
+if [ $FTP_TYPE = "ALL" ] || [ $FTP_TYPE = "SFTP" ]; then
+ echo "Creating files for SFTP server, may take time...."
+ docker cp setup-ftp-files-for-image.sh $SFTP_SIM:/tmp/
+ docker exec -w /home/onap/ $SFTP_SIM /tmp/setup-ftp-files-for-image.sh $NUM_FTPFILES $NUM_PNFS $FILE_SIZE #>/dev/null 2>&1
+fi
+if [ $FTP_TYPE = "ALL" ] || [ $FTP_TYPE = "FTPS" ]; then
+ echo "Creating files for FTPS server, may take time...."
+ docker cp setup-ftp-files-for-image.sh $FTPS_SIM:/tmp/setup-ftp-files-for-image.sh
+ docker exec -w /srv $FTPS_SIM /tmp/setup-ftp-files-for-image.sh $NUM_FTPFILES $NUM_PNFS $FILE_SIZE #>/dev/null 2>&1
+fi
+echo "Done: All simulators started and configured"