Add HTTP to DFC tests

- testing environment preparation for DFC with HTTP client
- add suitable TCs for CSIT tests

Issue-ID: DCAEGEN2-2527
Signed-off-by: Krzysztof Gajewski <krzysztof.gajewski@nokia.com>
Change-Id: Ic419c38125201ed6578a308ba258709f20ae65bd

7 files changed, 218 insertions, 13 deletions

diff --git a/test/mocks/datafilecollector-testharness/simulator-group/README.md b/test/mocks/datafilecollector-testharness/simulator-group/README.md
index a37c2fb8d..75abec9f1 100644
--- a/test/mocks/datafilecollector-testharness/simulator-group/README.md
+++ b/test/mocks/datafilecollector-testharness/simulator-group/README.md
@@ -16,7 +16,7 @@ In general these steps are needed to run the simulator group and dfc
 
 # Overview of the simulators.
 
-There are 5 different types of simulators. For futher details, see the README.md in each simulator dir.
+There are 6 different types of simulators. For futher details, see the README.md in each simulator dir.
 
 1. The MR simulator emits fileready events, upon poll requests, with new and historice file references
    It is possible to configire the change identifier and file prefixes for these identifiers and for which consumer groups
@@ -34,6 +34,7 @@ There are 5 different types of simulators. For futher details, see the README.md
    possible to configure the distrubution of files over these 5 servers (from 1 up to 5 severs). At start of the server, the server is
    populated with files to download.
 5. The FTPES simulator(s) is the same as the SFTP except that it using the FTPES protocol.
+6. The HTTP simulator(s) is the same as SFTP except that it using the HTTP protocol.
 
 # Build the simulator images
 
@@ -72,8 +73,8 @@ Do the manual steps to prepare the simulator images:
 Edit the `docker-compose-setup.sh` (or create a copy) to setup the env variables to the desired test behavior for each simulators.
 See each simulator to find a description of the available settings (DR_TC, DR_REDIR_TC and MR_TC).
 The following env variables shall be set (example values).
-Note that NUM_FTPFILES and NUM_PNFS controls the number of ftp files created in the ftp servers.
-A total of NUM_FTPFILES \* NUM_PNFS ftp files will be created in each ftp server (4 files in the below example).
+Note that NUM_FTPFILES, NUM_HTTPFILES and NUM_PNFS controls the number of ftp/http files created in the ftp/http servers.
+A total of NUM_FTPFILES \* NUM_PNFS (or NUM_HTTPFILES \* NUM_PNFS) ftp/http files will be created in each ftp/http server (4 files in the below example for ftp server).
 Large settings will be time consuming at start of the servers.
 Note that the number of files must match the number of file references emitted from the MR sim.
 
@@ -94,23 +95,24 @@ FILE_SIZE="1MB"               #File size for FTP file (1KB, 1MB, 5MB, 50MB or AL
 FTP_TYPE="SFTP"               #Type of FTP files to generate (SFTP, FTPES or ALL)
 
 If `FTP_TYPE` is set to `ALL`, both ftp servers will be populated with the same files. If set to `SFTP` or `FTPES` then only the server serving that protocol will be populated with files.
+`HTTP_TYPE` for now is only prepared for `HTTP` protocol. `HTTPS` protocol will be served in the future
 
 Run the script `docker-compose-setup.sh`to create a docker-compose with the desired settings. The desired setting
 in the script need to be manually adapted to for each specific simulator behavior according to the above. Check each simulator for available
 parameters.
 All simulators will be started with the generated docker-compose.yml file
 
-To generate ftp url with IP different from localhost, set SFTP_SIM_IP and/or FTPES_SIM_IP env variables to the addreses of the ftp servers before starting. 
+To generate ftp/http url with IP different from localhost, set SFTP_SIM_IP and/or FTPES_SIM_IP and/or HTTP_SIM_IP env variables to the addreses of the ftp servers before starting.
 So farm, this only works when the simulator python script is started from the command line.
 
 Kill all the containers with `simulators-kill.se`
 
 `simulators_start.sh` is for CSIT test and requires the env variables for test setting to be present in the shell.
 
-`setup-ftp-files.for-image.sh` is for CSIT and executed when the ftp servers are started from the docker-compose-setup.sh\`.
+`setup-ftp-files.for-image.sh` and `setup-http-files-for-image.sh` is for CSIT and executed when the ftp/http servers are started from the docker-compose-setup.sh\`.
 
 To make DFC to be able to connect to the simulator containers, DFC need to run in host mode.
-Start DFC by the following cmd: `docker run -d --network="host" --name dfc_app <dfc-image> `
+Start DFC by the following cmd: ` docker run -d --network="host" --name dfc_app <dfc-image>  `
 
 `<dfc-image>` could be either the locally built image `onap/org.onap.dcaegen2.collectors.datafile.datafile-app-server`
 or the one in nexus `nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.datafile.datafile-app-server`.
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-setup.sh b/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-setup.sh
index e6df022ce..ad45559eb 100755
--- a/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-setup.sh
+++ b/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-setup.sh
@@ -13,15 +13,24 @@ export DR_FEEDS="1:A,2:B,3:C,4:D"                      # Comma-separated of <fee
 export DR_REDIR_TC="--tc normal"                       # Test behaviour for DR redir sim
 export DR_REDIR_FEEDS="1:A,2:B,3:C,4:D"                # Comma-separated of <feed-id>:<file-name-prefixes> for DR redir sim
 
+export NUM_PNFS="700"                                  # Number of unuqie PNFs to generate file for
+export FILE_SIZE="1MB"                                 # File size for file (1KB, 1MB, 5MB, 50MB or ALL)
+
 export NUM_FTPFILES="105"                              # Number of FTP files to generate per PNF
-export NUM_PNFS="700"                                  # Number of unuqie PNFs to generate FTP file for
-export FILE_SIZE="1MB"                                 # File size for FTP file (1KB, 1MB, 5MB, 50MB or ALL)
 export FTP_TYPE="SFTP"                                 # Type of FTP files to generate (SFTP, FTPES or ALL)
 export FTP_FILE_PREFIXES="A,B,C,D"                     # Comma separated list of file name prefixes for ftp files
 export NUM_FTP_SERVERS=1                               # Number of FTP server to distribute the PNFs (Max 5)
 
+export NUM_HTTPFILES="105"                              # Number of HTTP files to generate per PNF
+export HTTP_TYPE="HTTP"                                 # Type of HTTP files to generate (HTTP, HTTPS or ALL)
+export HTTP_FILE_PREFIXES="A,B,C,D"                     # Comma separated list of file name prefixes for http files
+export NUM_HTTP_SERVERS=1                               # Number of HTTP server to distribute the PNFs (Max 5)
+export BASIC_AUTH_LOGIN=demo
+export BASIC_AUTH_PASSWORD=demo123456!
+
 export SFTP_SIMS="localhost:21,localhost:22,localhost:23,localhost:24,localhost:25"  # Comma separated list for SFTP servers host:port
 export FTPES_SIMS="localhost:1022,localhost:1023,localhost:1024,localhost:1026,localhost:1026" # Comma separated list for FTPES servers host:port
+export HTTP_SIMS="localhost:81,localhost:82,localhost:83,localhost:84,localhost:85"  # Comma separated list for HTTP servers host:port
 
 export DR_REDIR_SIM="localhost"                               # Hostname of DR redirect server
 
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-template.yml b/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-template.yml
index 41cafc386..c4ed2fec3 100644
--- a/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-template.yml
+++ b/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-template.yml
@@ -77,7 +77,9 @@ services:
     environment:
       SFTP_SIMS: ${SFTP_SIMS}
       FTPES_SIMS: ${FTPES_SIMS}
+      HTTP_SIMS: ${HTTP_SIMS}
       NUM_FTP_SERVERS: ${NUM_FTP_SERVERS}
+      NUM_HTTP_SERVERS: ${NUM_HTTP_SERVERS}
       MR_GROUPS: ${MR_GROUPS}
       MR_FILE_PREFIX_MAPPING: ${MR_FILE_PREFIX_MAPPING}
     image: mrsim:latest
@@ -225,3 +227,48 @@ services:
       - ./tls:/etc/ssl/private/:rw
     depends_on:
       - tls-init-container
+
+  http-server0:
+    networks:
+      - dfcnet
+    container_name: dfc_http-server0
+    image: http_httpd:latest
+    ports:
+      - "81:80"
+    restart: on-failure
+
+  http-server1:
+    networks:
+      - dfcnet
+    container_name: dfc_http-server1
+    image: http_httpd:latest
+    ports:
+      - "82:80"
+    restart: on-failure
+
+  http-server2:
+    networks:
+      - dfcnet
+    container_name: dfc_http-server2
+    image: http_httpd:latest
+    ports:
+      - "83:80"
+    restart: on-failure
+
+  http-server3:
+    networks:
+      - dfcnet
+    container_name: dfc_http-server3
+    image: http_httpd:latest
+    ports:
+      - "84:80"
+    restart: on-failure
+
+  http-server4:
+    networks:
+      - dfcnet
+    container_name: dfc_http-server4
+    image: http_httpd:latest
+    ports:
+      - "85:80"
+    restart: on-failure
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/prepare-images.sh b/test/mocks/datafilecollector-testharness/simulator-group/prepare-images.sh
index 11e4c0e11..fa56724bb 100755
--- a/test/mocks/datafilecollector-testharness/simulator-group/prepare-images.sh
+++ b/test/mocks/datafilecollector-testharness/simulator-group/prepare-images.sh
@@ -17,3 +17,7 @@ cd ../ftpes-sftp-server
 
 docker build -t ftpes_vsftpd:latest -f Dockerfile-ftpes .
 
+#Build image for http server
+cd ../http-https-server
+
+docker build -t http_httpd:latest -f Dockerfile-http .
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/setup-http-files-for-image.sh b/test/mocks/datafilecollector-testharness/simulator-group/setup-http-files-for-image.sh
new file mode 100755
index 000000000..9d54bc3ef
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/setup-http-files-for-image.sh
@@ -0,0 +1,64 @@
+#!/usr/bin/env sh
+
+# Script to create files for the HTTP server to return upon request.
+# The file names matches the files names in the events polled from the MR simulator.
+# Intended for execution in the running http containers in the http-root dir.
+
+NUM=200 #Default number of files
+PNFS=1 #Default number of PNFs
+FSIZE="ALL"
+PREFIXES="A"
+HTTP_SERV_INDEX=0
+NUM_HTTP_SERVERS=1
+
+if [ $# -ge 1 ]; then
+    NUM=$1
+fi
+if [ $# -ge 2 ]; then
+    PNFS=$2
+fi
+if [ $# -ge 3 ]; then
+    FSIZE=$3
+    if [ $3 != "1KB" ] && [ $3 != "1MB" ] && [ $3 != "5MB" ]  && [ $3 != "50MB" ]  && [ $3 != "ALL" ]; then
+    	echo "File size shall be 1KB|1MB|5MB|50MB|ALL"
+    	exit
+    fi
+fi
+if [ $# -ge 4 ]; then
+	PREFIXES=$4
+fi
+if [ $# -ge 5 ]; then
+	NUM_HTTP_SERVERS=$5
+fi
+if [ $# -ge 6 ]; then
+	HTTP_SERV_INDEX=$6
+fi
+if [ $# -lt 1 ] || [ $# -gt 6 ]; then
+    echo "Wrong args, usage: setup-http-files-for-image.sh [ <num-files> [ <num-PNFs> [ 1KB|1MB|5MB|50MB [ <comma-separated-file-name-prefixs> [ <number-of-http-servers> <http-server-index> ] ] ] ] ] ]"
+    exit
+fi
+
+echo "Running http file creations. " $PNFS " PNFs and " $NUM " files for each PNF with file size(s) " $FSIZE "and file prefixe(s) " $PREFIXES " in http servers with index " $HTTP_SERV_INDEX
+
+truncate -s 1KB 1KB.tar.gz
+truncate -s 1MB 1MB.tar.gz
+truncate -s 5MB 5MB.tar.gz
+truncate -s 50MB 50MB.tar.gz
+
+for fnp in ${PREFIXES//,/ }
+do
+	p=0
+	while [ $p -lt $PNFS ]; do
+		if [[ $(($p%$NUM_HTTP_SERVERS)) == $HTTP_SERV_INDEX ]]; then
+    		i=0
+    		while [ $i -lt $NUM ]; do  #Problem with for loop and var substituion in curly bracket....so used good old style loop
+    			if [ $FSIZE = "ALL" ] || [ $FSIZE = "1KB" ]; then ln -s 1KB.tar.gz $fnp'20000626.2315+0200-2330+0200_PNF'$p'-'$i'-1KB.tar.gz' >& /dev/null; fi
+        		if [ $FSIZE = "ALL" ] || [ $FSIZE = "1MB" ]; then ln -s 1MB.tar.gz $fnp'20000626.2315+0200-2330+0200_PNF'$p'-'$i'-1MB.tar.gz' >& /dev/null; fi
+        		if [ $FSIZE = "ALL" ] || [ $FSIZE = "5MB" ]; then ln -s 5MB.tar.gz $fnp'20000626.2315+0200-2330+0200_PNF'$p'-'$i'-5MB.tar.gz' >& /dev/null; fi
+        		if [ $FSIZE = "ALL" ] || [ $FSIZE = "50MB" ]; then ln -s 50MB.tar.gz $fnp'20000626.2315+0200-2330+0200_PNF'$p'-'$i'-50MB.tar.gz' >& /dev/null; fi
+    			let i=i+1
+    		done
+    	fi
+    	let p=p+1
+	done
+done
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/simulators-kill.sh b/test/mocks/datafilecollector-testharness/simulator-group/simulators-kill.sh
index 32045ea56..697850339 100755
--- a/test/mocks/datafilecollector-testharness/simulator-group/simulators-kill.sh
+++ b/test/mocks/datafilecollector-testharness/simulator-group/simulators-kill.sh
@@ -16,6 +16,11 @@ docker kill dfc_ftpes-server-vsftpd1
 docker kill dfc_ftpes-server-vsftpd2
 docker kill dfc_ftpes-server-vsftpd3
 docker kill dfc_ftpes-server-vsftpd4
+docker kill dfc_http-server0
+docker kill dfc_http-server1
+docker kill dfc_http-server2
+docker kill dfc_http-server3
+docker kill dfc_http-server4
 docker kill dfc_cbs
 docker kill dfc_consul
 
@@ -33,7 +38,12 @@ docker rm dfc_ftpes-server-vsftpd1
 docker rm dfc_ftpes-server-vsftpd2
 docker rm dfc_ftpes-server-vsftpd3
 docker rm dfc_ftpes-server-vsftpd4
+docker rm dfc_http-server0
+docker rm dfc_http-server1
+docker rm dfc_http-server2
+docker rm dfc_http-server3
+docker rm dfc_http-server4
 docker rm dfc_cbs
 docker rm dfc_consul
 
-echo "done"
\ No newline at end of file
+echo "done"
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/simulators-start.sh b/test/mocks/datafilecollector-testharness/simulator-group/simulators-start.sh
index 25750d38f..895949caa 100755
--- a/test/mocks/datafilecollector-testharness/simulator-group/simulators-start.sh
+++ b/test/mocks/datafilecollector-testharness/simulator-group/simulators-start.sh
@@ -12,6 +12,18 @@ server_check() {
 	echo "Simulator " $1 " on localhost:$2$3 - no response"
 }
 
+basic_auth_server_check() {
+	for i in {1..10}; do
+		res=$(curl  -s -o /dev/null -w "%{http_code}" http://$BASIC_AUTH_LOGIN:$BASIC_AUTH_PASSWORD@localhost:$2$3)
+		if [ $res -gt 199 ] && [ $res -lt 300 ]; then
+			echo "Simulator " $1 " on localhost:$2$3 responded ok"
+			return
+		fi
+		sleep 1
+	done
+	echo "Simulator " $1 " on localhost:$2$3 - no response"
+}
+
 server_check_https() {
 	for i in {1..10}; do
 		res=$(curl  -k -s -o /dev/null -w "%{http_code}" https://localhost:$2$3)
@@ -55,6 +67,16 @@ DOCKER_SIM_NWNAME="dfcnet"
 echo "Creating docker network $DOCKER_SIM_NWNAME, if needed"
 docker network ls| grep $DOCKER_SIM_NWNAME > /dev/null || docker network create $DOCKER_SIM_NWNAME
 
+if [ -z "$NUM_FTP_SERVERS" ]
+ then
+ export NUM_FTP_SERVERS=1
+fi
+
+if [ -z "$NUM_HTTP_SERVERS" ]
+ then
+ export NUM_HTTP_SERVERS=1
+fi
+
 docker-compose -f docker-compose-template.yml config > docker-compose.yml
 
 docker-compose up -d
@@ -64,6 +86,7 @@ sudo chown $(id -u):$(id -g) consul/consul/
 
 declare -a SFTP_SIM
 declare -a FTPES_SIM
+declare -a HTTP_SIM
 
 DR_SIM="$(docker ps -q --filter='name=dfc_dr-sim')"
 DR_RD_SIM="$(docker ps -q --filter='name=dfc_dr-redir-sim')"
@@ -78,6 +101,11 @@ FTPES_SIM[1]="$(docker ps -q --filter='name=dfc_ftpes-server-vsftpd1')"
 FTPES_SIM[2]="$(docker ps -q --filter='name=dfc_ftpes-server-vsftpd2')"
 FTPES_SIM[3]="$(docker ps -q --filter='name=dfc_ftpes-server-vsftpd3')"
 FTPES_SIM[4]="$(docker ps -q --filter='name=dfc_ftpes-server-vsftpd4')"
+HTTP_SIM[0]="$(docker ps -q --filter='name=dfc_http-server0')"
+HTTP_SIM[1]="$(docker ps -q --filter='name=dfc_http-server1')"
+HTTP_SIM[2]="$(docker ps -q --filter='name=dfc_http-server2')"
+HTTP_SIM[3]="$(docker ps -q --filter='name=dfc_http-server3')"
+HTTP_SIM[4]="$(docker ps -q --filter='name=dfc_http-server4')"
 CBS_SIM="$(docker ps -q --filter='name=dfc_cbs')"
 CONSUL_SIM="$(docker ps -q --filter='name=dfc_consul')"
 
@@ -96,6 +124,11 @@ if [ $(docker inspect --format '{{ .State.Running }}' $DR_SIM) ] && \
 [ $(docker inspect --format '{{ .State.Running }}' ${FTPES_SIM[2]}) ] && \
 [ $(docker inspect --format '{{ .State.Running }}' ${FTPES_SIM[3]}) ] && \
 [ $(docker inspect --format '{{ .State.Running }}' ${FTPES_SIM[4]}) ] && \
+[ $(docker inspect --format '{{ .State.Running }}' ${HTTP_SIM[0]}) ] && \
+[ $(docker inspect --format '{{ .State.Running }}' ${HTTP_SIM[1]}) ] && \
+[ $(docker inspect --format '{{ .State.Running }}' ${HTTP_SIM[2]}) ] && \
+[ $(docker inspect --format '{{ .State.Running }}' ${HTTP_SIM[3]}) ] && \
+[ $(docker inspect --format '{{ .State.Running }}' ${HTTP_SIM[4]}) ] && \
 [ $(docker inspect --format '{{ .State.Running }}' $CBS_SIM) ] && \
 [ $(docker inspect --format '{{ .State.Running }}' $CONSUL_SIM) ]
  then
@@ -107,6 +140,16 @@ if [ $(docker inspect --format '{{ .State.Running }}' $DR_SIM) ] && \
  fi
 done
 
+if [ -z "$BASIC_AUTH_LOGIN" ]
+ then
+ BASIC_AUTH_LOGIN=demo
+fi
+
+if [ -z "$BASIC_AUTH_PASSWORD" ]
+ then
+ BASIC_AUTH_PASSWORD=demo123456!
+fi
+
 server_check      "cbs          " 10000 "/healthcheck"
 server_check      "consul       " 8500 "/v1/catalog/service/agent"
 server_check      "DR sim       " 3906 "/"
@@ -125,6 +168,11 @@ sftp_server_check "SFTP server 1" 1023
 sftp_server_check "SFTP server 2" 1024
 sftp_server_check "SFTP server 3" 1025
 sftp_server_check "SFTP server 4" 1026
+basic_auth_server_check "HTTP server 0" 81
+basic_auth_server_check "HTTP server 1" 82
+basic_auth_server_check "HTTP server 2" 83
+basic_auth_server_check "HTTP server 3" 84
+basic_auth_server_check "HTTP server 4" 85
 
 echo ""
 
@@ -150,10 +198,6 @@ if [ -z "$FTP_FILE_PREFIXES" ]
  FTP_FILE_PREFIXES="A"
 fi
 
-if [ -z "$NUM_FTP_SERVERS" ]
- then
- NUM_FTP_SERVERS=1
-fi
 
 
 if [ $FTP_TYPE = "ALL" ] || [ $FTP_TYPE = "SFTP" ]; then
@@ -176,4 +220,29 @@ if [ $FTP_TYPE = "ALL" ] || [ $FTP_TYPE = "FTPES" ]; then
 		let p=p+1
 	done
 fi
+
+#Populate the http server with files. Note some common variables with ftp files!
+if [ -z "$NUM_HTTPFILES" ]
+ then
+ NUM_HTTPFILES=200
+fi
+if [ -z "$HTTP_TYPE" ]
+ then
+ HTTP_TYPE="ALL"
+fi
+if [ -z "$HTTP_FILE_PREFIXES" ]
+ then
+ HTTP_FILE_PREFIXES="A"
+fi
+
+if [ $HTTP_TYPE = "ALL" ] || [ $HTTP_TYPE = "HTTP" ]; then
+	echo "Creating files for HTTP server, may take time...."
+	p=0
+	while [ $p -lt $NUM_HTTP_SERVERS ]; do
+		docker cp setup-http-files-for-image.sh ${HTTP_SIM[$p]}:/tmp/setup-http-files-for-image.sh
+		#Double slash needed for docker on win...
+		docker exec -w //usr//local//apache2//htdocs ${HTTP_SIM[$p]} //tmp/setup-http-files-for-image.sh $NUM_HTTPFILES $NUM_PNFS $FILE_SIZE $HTTP_FILE_PREFIXES $NUM_HTTP_SERVERS $p #>/dev/null 2>&1
+		let p=p+1
+	done
+fi
 echo "Done: All simulators started and configured"