Add HTTPS to DFC tests

- testing environment preparation for DFC with HTTPS client
  (expand existing http server)
- add suitable TCs for CSIT tests

Issue-ID: DCAEGEN2-2528
Signed-off-by: Krzysztof Gajewski <krzysztof.gajewski@nokia.com>
Change-Id: Ifa8abb827e269d76144fc74c65031dd387fe5fe7

2 files changed, 123 insertions, 11 deletions

diff --git a/test/mocks/datafilecollector-testharness/mr-sim/README.md b/test/mocks/datafilecollector-testharness/mr-sim/README.md
index 055e7d914..a8d2b7aaa 100644
--- a/test/mocks/datafilecollector-testharness/mr-sim/README.md
+++ b/test/mocks/datafilecollector-testharness/mr-sim/README.md
@@ -10,7 +10,7 @@ This readme contains:
 
 The MR-sim is a python script delivering batches of events including one or more fileReady for one or more PNFs.
 It is possible to configure number of events, PNFs, consumer groups, exising or missing files, file prefixes and change identifier.
-In addition, MR sim can be configured to deliver file url for up to 5 FTP and 5 HTTP servers (simulating the PNFs).
+In addition, MR sim can be configured to deliver file url for up to 5 FTP and 5 HTTP/HTTPS/HTTPS with no auth servers (simulating the PNFs).
 
 ## Building and running
 
@@ -33,8 +33,10 @@ The following envrionment vaiables are used:
 - **FTPES_SIMS** - A comma-separated list of hostname:port for the FTP servers to generate ftpes file urls for. If not set MR sim will assume 'localhost:21'. Minimum 1 and maximum 5 host-port pairs can be given.
 - **SFTP_SIMS** - A comma-separated list of hostname:port for the FTP servers to generate sftp file urls for. If not set MR sim will assume 'localhost:1022'. Minimum 1 and maximum 5 host-port pairs can be given.
 - **HTTP_SIMS** - A comma-separated list of hostname:port for the HTTP servers to generate http file urls for. If not set MR sim will assume 'localhost:81'. Minimum 1 and maximum 5 host-port pairs can be given.
+- **HTTPS_SIMS** - A comma-separated list of hostname:port for the HTTPS servers (configured for client certificate authentication and basic authentication; certificates were obtained using CMPv2 server) to generate http file urls for. If not set MR sim will assume 'localhost:444'. Minimum 1 and maximum 5 host-port pairs can be given.
+- **HTTPS_SIMS_NO_AUTH** - A comma-separated list of hostname:port for the HTTPS servers with no autorization to generate http file urls for. If not set MR sim will assume 'localhost:8081'. Minimum 1 and maximum 5 host-port pairs can be given.
 - **NUM_FTP_SERVERS** - Number of FTP servers to use out of those specified in the envrioment variables above. The number shall be in the range 1-5.
-- **NUM_HTTP_SERVERS** - Number of HTTP servers to use out of those specified in the envrioment variables above. The number shall be in the range 1-5.
+- **NUM_HTTP_SERVERS** - Number of HTTP/HTTPS/HTTPS with no authorization servers to use out of those specified in the envrioment variables above. The number shall be in the range 1-5.
 - **MR_GROUPS** - A comma-separated list of consummer-group:changeId\[:changeId]\*. Defines which change identifier that should be used for each consumer group. If not set the MR-sim will assume 'OpenDcae-c12:PM_MEAS_FILES'.not set the MR-sim will assume 'OpenDcae-c12:PM_MEAS_FILES'.
 - **MR_FILE_PREFIX_MAPPING** - A comma-separated list of changeId:filePrefix. Defines which file prefix to use for each change identifier, needed to distinguish files for each change identifiers. If not set the MR-sim will assume 'PM_MEAS_FILES:A
 
@@ -179,7 +181,9 @@ TC1500 - 700 ME, SFTP, 1MB files, 100 files per event, 35 events per poll, simul
 
 Changing the first digit in tc number will change the test case to run FTPES or HTTP instead. Eg. TC201 is FTPES version of TC101.
 
-TC2XX is same as TC1XX but with FTPES, TC3XX is same as TC1XX but with HTTP
+TC2XX is same as TC1XX but with FTPES, TC3XX is same as TC1XX but with HTTP, TC4XX is same as TC1XX but with HTTPS
+(with basic authorization). Note, in the case of HTTPS, some tests may not have direct correspondence in FTP tests
+(TC403, TC404 described in the end of this section).
 
 TC6XX is same as TC5XX but with FTPES
 
@@ -187,6 +191,10 @@ TC8XX is same as TC7XX but with FTPES
 
 TC2XXX is same as TC1XXX but with FTPES
 
+TC403 - One ME, HTTPS with client certificate authentication, 1 1MB file, 1 event
+
+TC404 - One ME, HTTPS with no client authentication, 1 1MB file, 1 event
+
 ## Developer workflow
 
 1. `sudo apt install python3-venv`
diff --git a/test/mocks/datafilecollector-testharness/mr-sim/mr-sim.py b/test/mocks/datafilecollector-testharness/mr-sim/mr-sim.py
index 809f18661..323dde618 100644
--- a/test/mocks/datafilecollector-testharness/mr-sim/mr-sim.py
+++ b/test/mocks/datafilecollector-testharness/mr-sim/mr-sim.py
@@ -1,3 +1,14 @@
+# COPYRIGHT NOTICE STARTS HERE
+#
+# Modifications copyright (C) 2021 Nokia. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# COPYRIGHT NOTICE ENDS HERE
 import argparse
 import os
 import sys
@@ -19,10 +30,13 @@ ftpes_hosts = []
 ftpes_ports = []
 http_hosts = []
 http_ports = []
+https_hosts = []
+https_ports = []
+https_hosts_no_auth = []
+https_ports_no_auth = []
 num_ftp_servers = 1
 num_http_servers = 1
 
-
 def sumList(ctrArray):
     tmp = 0
     for i in range(len(ctrArray)):
@@ -525,6 +539,18 @@ def MR_reply(consumerGroup, consumerId):
     elif args.tc302:
         return tc100(groupIndex, changeId, filePrefix, "http", "50MB")
 
+    elif args.tc400:
+        return tc100(groupIndex, changeId, filePrefix, "https", "1MB")
+    elif args.tc401:
+        return tc100(groupIndex, changeId, filePrefix, "https", "5MB")
+    elif args.tc402:
+        return tc100(groupIndex, changeId, filePrefix, "https", "50MB")
+    elif args.tc403:
+        return tc100(groupIndex, changeId, filePrefix, "httpsCAuth", "1MB")
+    elif args.tc404:
+        return tc100(groupIndex, changeId, filePrefix, "httpsNoAuth", "1MB")
+
+
 #### Test case functions
 
 
@@ -543,7 +569,8 @@ def tc100(groupIndex, changeId, filePrefix, schemeType, fileSize):
     fileName = createFileName(groupIndex, filePrefix, nodeName, seqNr, fileSize)
     msg = getEventHead(groupIndex, changeId, nodeName) + getEventName(fileName, schemeType, "onap", "pano",
                                                                       nodeIndex) + getEventEnd()
-    if schemeType == "http":
+    if (schemeType == "http") or (schemeType == "https") \
+            or (schemeType == "httpsCAuth") or (schemeType == "httpsNoAuth"):
         msg = getEventHead(groupIndex, changeId, nodeName) + getEventName(fileName, schemeType, "demo", "demo123456!",
                                                                           nodeIndex) + getEventEnd()
     fileMap[groupIndex][seqNr * hash(filePrefix)] = seqNr
@@ -1192,20 +1219,37 @@ def getEventName(fn, type, user, passwd, nodeIndex):
     nodeIndex = nodeIndex % num_ftp_servers
     port = sftp_ports[nodeIndex]
     ip = sftp_hosts[nodeIndex]
-    if (type == "ftpes"):
+    location_variant = type + """://""" + user + """:""" + passwd + """@""" + ip + """:""" + str(port)
+    if type == "ftpes":
         port = ftpes_ports[nodeIndex]
         ip = ftpes_hosts[nodeIndex]
-    elif (type == "http"):
+        location_variant = type + """://""" + user + """:""" + passwd + """@""" + ip + """:""" + str(port)
+    elif type == "http":
         nodeIndex = nodeIndex % num_http_servers
         port = http_ports[nodeIndex]
         ip = http_hosts[nodeIndex]
+        location_variant = type + """://""" + user + """:""" + passwd + """@""" + ip + """:""" + str(port)
+    elif type == "https":
+        nodeIndex = nodeIndex % num_http_servers
+        port = https_ports[nodeIndex]
+        ip = https_hosts[nodeIndex]
+        location_variant = type + """://""" + user + """:""" + passwd + """@""" + ip + """:""" + str(port)
+    elif type == "httpsCAuth":
+        alt_type = "https"
+        port = https_ports[nodeIndex]
+        ip = https_hosts[nodeIndex]
+        location_variant = alt_type + """://""" + ip + """:""" + str(port)
+    elif type == "httpsNoAuth":
+        alt_type = "https"
+        port = https_ports_no_auth[nodeIndex]
+        ip = https_hosts_no_auth[nodeIndex]
+        location_variant = alt_type + """://""" + ip + """:""" + str(port)
 
     nameStr = """{
                   "name": \"""" + fn + """",
                   "hashMap": {
                     "fileFormatType": "org.3GPP.32.435#measCollec",
-                    "location": \"""" + type + """://""" + user + """:""" + passwd + """@""" + ip + """:""" + str(
-        port) + """/""" + fn + """",
+                    "location": \"""" + location_variant + """/""" + fn + """",
                     "fileFormatVersion": "V10",
                     "compression": "gzip"
                   }
@@ -1248,14 +1292,18 @@ if __name__ == "__main__":
     sftp_sims = os.environ.get('SFTP_SIMS', 'localhost:1022')
     ftpes_sims = os.environ.get('FTPES_SIMS', 'localhost:21')
     http_sims = os.environ.get('HTTP_SIMS', 'localhost:81')
+    https_sims = os.environ.get('HTTPS_SIMS', 'localhost:444')
+    https_sims_no_auth = os.environ.get('HTTPS_SIMS_NO_AUTH', 'localhost:8081')
     num_ftp_servers = int(os.environ.get('NUM_FTP_SERVERS', 1))
     num_http_servers = int(os.environ.get('NUM_HTTP_SERVERS', 1))
 
     print("Configured sftp sims: " + sftp_sims)
     print("Configured ftpes sims: " + ftpes_sims)
     print("Configured http sims: " + http_sims)
+    print("Configured https sims: " + https_sims)
+    print("Configured https with no authorization sims: " + https_sims_no_auth)
     print("Configured number of ftp servers: " + str(num_ftp_servers))
-    print("Configured number of http servers: " + str(num_http_servers))
+    print("Configured number of http/https/https with no auth servers: " + str(num_http_servers) + " each")
 
     tmp = sftp_sims.split(',')
     for i in range(len(tmp)):
@@ -1275,6 +1323,19 @@ if __name__ == "__main__":
         http_hosts.append(hp[0])
         http_ports.append(hp[1])
 
+    tmp = https_sims.split(',')
+    for i in range(len(tmp)):
+        hp = tmp[i].split(':')
+        https_hosts.append(hp[0])
+        https_ports.append(hp[1])
+
+    tmp = https_sims_no_auth.split(',')
+    for i in range(len(tmp)):
+        hp = tmp[i].split(':')
+        https_hosts_no_auth.append(hp[0])
+        https_ports_no_auth.append(hp[1])
+
+
     groups = os.environ.get('MR_GROUPS', 'OpenDcae-c12:PM_MEAS_FILES')
     print("Groups detected: " + groups)
     configuredPrefixes = os.environ.get('MR_FILE_PREFIX_MAPPING', 'PM_MEAS_FILES:A')
@@ -1637,6 +1698,28 @@ if __name__ == "__main__":
         action='store_true',
         help='TC302 - One ME, HTTP, 1 50MB file, 1 event')
 
+    # HTTPS TCs with single ME
+    parser.add_argument(
+        '--tc400',
+        action='store_true',
+        help='TC400 - One ME, HTTPS, 1 1MB file, 1 event')
+    parser.add_argument(
+        '--tc401',
+        action='store_true',
+        help='TC401 - One ME, HTTPS, 1 5MB file, 1 event')
+    parser.add_argument(
+        '--tc402',
+        action='store_true',
+        help='TC402 - One ME, HTTPS, 1 50MB file, 1 event')
+    parser.add_argument(
+        '--tc403',
+        action='store_true',
+        help='TC403 - One ME, HTTPS client certificare authentication, 1 1MB file, 1 event')
+    parser.add_argument(
+        '--tc404',
+        action='store_true',
+        help='TC404 - One ME, HTTPS no client authentication, 1 1MB file, 1 event')
+
     args = parser.parse_args()
 
     if args.tc100:
@@ -1778,6 +1861,17 @@ if __name__ == "__main__":
     elif args.tc302:
         tc_num = "TC# 302"
 
+    elif args.tc400:
+        tc_num = "TC# 400"
+    elif args.tc401:
+        tc_num = "TC# 401"
+    elif args.tc402:
+        tc_num = "TC# 402"
+    elif args.tc403:
+        tc_num = "TC# 403"
+    elif args.tc404:
+        tc_num = "TC# 404"
+
     else:
         print("No TC was defined")
         print("use --help for usage info")
@@ -1797,8 +1891,18 @@ if __name__ == "__main__":
         print("Using " + str(http_hosts[i]) + ":" + str(http_ports[i]) + " for http server with index " + str(
             i) + " for http server address and port in file urls.")
 
+    for i in range(len(https_hosts)):
+        print("Using " + str(https_hosts[i]) + ":" + str(https_ports[i]) + " for https server with index " + str(
+            i) + " for https server address and port in file urls.")
+
+    for i in range(len(https_hosts_no_auth)):
+        print("Using " + str(https_hosts_no_auth[i]) + ":" + str(https_ports_no_auth[i])
+              + " for https server with no authentication with index " + str(i)
+              + " for https server address and port in file urls.")
+
     print("Using up to " + str(num_ftp_servers) + " ftp servers, for each protocol for PNFs.")
-    print("Using up to " + str(num_http_servers) + " http servers, for each protocol for PNFs.")
+    print("Using up to " + str(num_http_servers)
+          + " http/https/https with no auth servers, for each protocol for PNFs.")
 
 
     def https_app(**kwargs):