1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
|
==============================
DFC (DataFile Collector)
==============================
:Date: 2018-09-21
.. contents::
:depth: 3
..
Overview
========
DFC will orchestrate the collection of bulk PM data flow:
1. Subscribes to fileReady DMaaP topic
2. Collects the file from the xNF
3. Sends new event to DataRouter with file.
Introduction
============
DFC is delivered as one **Docker container** which hosts application server and can be started by `docker-compose`.
Functionality
=============
.. image:: ../images/DFC.png
Paths
=====
GET /events/unauthenticated.VES_NOTIFICATION_OUTPUT
-----------------------------------------------
Description
~~~~~~~~~~~
Reads fileReady events from DMaaP (Data Movement as a Platform)
Responses
~~~~~~~~~
+-----------+-------------------------------------------+
| HTTP Code | Description |
+===========+===========================================+
| **200** | successful response |
+-----------+-------------------------------------------+
POST /publish
--------------------------------------
Description
~~~~~~~~~~~
Publish the collected file/s as a stream to DataRouter
- file as stream
- compression
- fileFormatType
- fileFormatVersion
Responses
~~~~~~~~~
+-----------+-------------------------------------------+
| HTTP Code | Description |
+===========+===========================================+
| **200** | successful response |
+-----------+-------------------------------------------+
Compiling DFC
=============
Whole project (top level of DFC directory) and each module (sub module directory) can be compiled using
`mvn clean install` command.
Configuration file: Config/datafile_endpoints.json
Main API Endpoints
==================
Running with dev-mode of DFC
- Heartbeat: **http://<container_address>:8100/heartbeat** or **https://<container_address>:8433/heartbeat**
- Start DFC: **http://<container_address>:8100/start** or **https://<container_address>:8433/start**
- Stop DFC: **http://<container_address>:8100/stopDatafile** or **https://<container_address>:8433/stopDatafile**
The external port allocated for 8100 (http) is 30245.
Maven GroupId:
==============
org.onap.dcaegen2.collectors
Maven Parent ArtifactId:
========================
dcae-collectors
Maven Children Artifacts:
=========================
1. datafile-app-server: DFC server
2. datafile-dmaap-client: Contains implementation of DmaaP client
3. datafile-commons: Common code for whole DFC modules
4. docker-compose: Contains the docker-compose
Configuration of Certificates in test environment(For FTP over TLS):
====================================================================
DFC supports two protocols: FTPES and SFTP.
For FTPES, it is mutual authentication with certificates.
In our test environment, we use vsftpd to simulate xNF, and we generate self-signed
keys & certificates on both vsftpd server and DFC.
1. generate key & certificate with openssl for DFC：
openssl genrsa -out dfc.key 2048
openssl req -new -out dfc.csr -key dfc.key
openssl x509 -req -days 365 -in dfc.csr -signkey dfc.key -out dfc.crt
2. generate key & certificate with openssl for vsftpd:
openssl genrsa -out ftp.key 2048
openssl req -new -out ftp.csr -key ftp.key
openssl x509 -req -days 365 -in ftp.csr -signkey ftp.key -out ftp.crt
3. configure java keystore in DFC:
We have two keystore files, one for TrustManager, one for KeyManager.
First, create a jks keystore for TrustManager:
keytool -keystore ftp.jks -genkey -alias ftp
Second, convert your certificate in a DER format :
openssl x509 -outform der -in ftp.crt -out ftp.der
And after, import it in the keystore :
keytool -import -alias ftp -keystore ftp.jks -file ftp.der
For KeyManager:
First, create a jks keystore:
keytool -keystore dfc.jks -genkey -alias dfc
Second, import dfc.crt and dfc.key to dfc.jks. This is a bit troublesome.
Step one: Convert x509 Cert and Key to a pkcs12 file
openssl pkcs12 -export -in dfc.crt -inkey dfc.key \
-out dfc.p12 -name [some-alias] \
Note: Make sure you put a password on the p12 file - otherwise you'll get
a null reference exception when you try to import it. (In case anyone else had this headache).
Note 2: You might want to add the -chainoption to preserve the full certificate chain.
Step two: Convert the pkcs12 file to a java keystore
keytool -importkeystore \
-deststorepass [changeit] -destkeypass [changeit] -destkeystore dfc.jks \
-srckeystore dfc.p12 -srcstoretype PKCS12 -srcstorepass some-password \
-alias [some-alias]
Finished
4. configure vsftpd:
update /etc/vsftpd/vsftpd.conf:
rsa_cert_file=/etc/ssl/private/ftp.crt
rsa_private_key_file=/etc/ssl/private/ftp.key
ssl_enable=YES
allow_anon_ssl=NO
force_local_data_ssl=YES
force_local_logins_ssl=YES
ssl_tlsv1=YES
ssl_sslv2=YES
ssl_sslv3=YES
require_ssl_reuse=NO
ssl_ciphers=HIGH
require_cert=YES
ssl_request_cert=YES
ca_certs_file=/home/vsftpd/myuser/dfc.crt
5. configure config/datafile_endpoints.json:
Update the file accordingly:
"ftpesConfiguration": {
"keyCert": "/config/dfc.jks",
"keyPassword": "[yourpassword]",
"trustedCA": "/config/ftp.jks",
"trustedCAPassword": "[yourpassword]"
}
6. This has been tested with vsftpd and dfc, with self-signed certificates.
In real deployment, we should use ONAP-CA signed certificate for DFC, and vendor-CA signed
certificate for xNF.
|
