summaryrefslogtreecommitdiffstats
path: root/docs/sections/services/dfc
diff options
context:
space:
mode:
Diffstat (limited to 'docs/sections/services/dfc')
-rw-r--r--docs/sections/services/dfc/architecture.rst4
-rw-r--r--docs/sections/services/dfc/certificates.rst44
-rw-r--r--docs/sections/services/dfc/configuration.rst26
-rw-r--r--docs/sections/services/dfc/consumedapis.rst2
-rw-r--r--docs/sections/services/dfc/delivery.rst1
-rw-r--r--docs/sections/services/dfc/http-notes.rst9
-rw-r--r--docs/sections/services/dfc/installation-helm.rst2
-rw-r--r--docs/sections/services/dfc/logging.rst4
-rw-r--r--docs/sections/services/dfc/troubleshooting.rst20
9 files changed, 58 insertions, 54 deletions
diff --git a/docs/sections/services/dfc/architecture.rst b/docs/sections/services/dfc/architecture.rst
index 75913dbb..230a3419 100644
--- a/docs/sections/services/dfc/architecture.rst
+++ b/docs/sections/services/dfc/architecture.rst
@@ -20,7 +20,7 @@ DFC will handle the collection of bulk PM data flow:
DFC is delivered as one **Docker container** which hosts application server.
See `Delivery`_ for more information about the docker container.
-.. _Delivery: ./delivery.html
+.. _Delivery: ./delivery.rst
Functionality
"""""""""""""
@@ -49,5 +49,5 @@ affect the handling of others.
Generalized DFC
""""""""""""""""
From version 1.2.1 and onwards, the DFC has more general use. Instead of only handling PM files, any kind of files
-are handled. The 'changeIdentifier' field in the FileReady VES event (which is reported from the PNFs) identifies the
+are handled. The 'changeIdentifier' field in the FileReady VES event (which is reported from the PNFs) identifies the
file type. This is mapped to a publishing stream in the DR.
diff --git a/docs/sections/services/dfc/certificates.rst b/docs/sections/services/dfc/certificates.rst
index 9c4d46b2..d272dd8e 100644
--- a/docs/sections/services/dfc/certificates.rst
+++ b/docs/sections/services/dfc/certificates.rst
@@ -21,7 +21,7 @@ keys & certificates on both vsftpd server and DFC.
1. Generate key/certificate with openssl for DFC:
-------------------------------------------------
-.. code:: bash
+.. code-block:: bash
openssl genrsa -out dfc.key 2048
openssl req -new -out dfc.csr -key dfc.key
@@ -29,7 +29,7 @@ keys & certificates on both vsftpd server and DFC.
2. Generate key & certificate with openssl for vsftpd:
------------------------------------------------------
-.. code:: bash
+.. code-block:: bash
openssl genrsa -out ftp.key 2048
openssl req -new -out ftp.csr -key ftp.key
@@ -43,20 +43,20 @@ We have two keystore files, one for TrustManager, one for KeyManager.
1. First, convert your certificate in a DER format :
- .. code:: bash
+ .. code-block:: bash
openssl x509 -outform der -in ftp.crt -out ftp.der
2. And after copy existing keystore and password from container:
- .. code:: bash
+ .. code-block:: bash
kubectl cp <DFC pod>:/opt/app/datafile/etc/cert/trust.jks trust.jks
kubectl cp <DFC pod>:/opt/app/datafile/etc/cert/trust.pass trust.pass
3. Import DER certificate in the keystore :
- .. code:: bash
+ .. code-block:: bash
keytool -import -alias ftp -keystore trust.jks -file ftp.der
@@ -66,42 +66,48 @@ We have two keystore files, one for TrustManager, one for KeyManager.
Convert x509 Cert and Key to a pkcs12 file
- .. code:: bash
+ .. code-block:: bash
openssl pkcs12 -export -in dfc.crt -inkey dfc.key -out cert.p12 -name dfc
Note: Make sure you put a password on the p12 file - otherwise you'll get a null reference exception when you try to import it.
2. Create password files for cert.p12
- .. code:: bash
- printf "[your password]" > p12.pass
+ .. code-block:: bash
+
+ printf "[your password]" > p12.pass
4. Update existing KeyStore files
---------------------------------
Copy the new trust.jks and cert.p12 and password files from local environment to the DFC container.
- .. code:: bash
- mkdir mycert
- cp cert.p12 mycert/
- cp p12.pass mycert/
- cp trust.jks mycert/
- cp trust.pass mycert/
- kubectl cp mycert/ <DFC pod>:/opt/app/datafile/etc/cert/
+ .. code-block:: bash
+
+ mkdir mycert
+ cp cert.p12 mycert/
+ cp p12.pass mycert/
+ cp trust.jks mycert/
+ cp trust.pass mycert/
+ kubectl cp mycert/ <DFC pod>:/opt/app/datafile/etc/cert/
5. Update configuration in consul
-----------------------------------
Change path in consul:
- .. code:: bash
+
+.. code-block:: bash
+
dmaap.ftpesConfig.keyCert": "/opt/app/datafile/etc/cert/mycert/cert.p12
dmaap.ftpesConfig.keyPasswordPath": "/opt/app/datafile/etc/cert/mycert/p12.pass
dmaap.ftpesConfig.trustedCa": "/opt/app/datafile/etc/cert/mycert/trust.jks
dmaap.ftpesConfig.trustedCaPasswordPath": "/opt/app/datafile/etc/cert/mycert/trust.pass
Consul's address: http://<worker external IP>:<Consul External Port>
- .. code:: bash
- kubectl -n onap get svc | grep consul
+
+ .. code-block:: bash
+
+ kubectl -n onap get svc | grep consul
.. image:: ./consule-certificate-update.png
@@ -132,4 +138,4 @@ Consul's address: http://<worker external IP>:<Consul External Port>
7. Other conditions
---------------------------------------------------------------------------
This has been tested with vsftpd and dfc, with self-signed certificates.
- In real deployment, we should use ONAP-CA signed certificate for DFC, and vendor-CA signed certificate for xNF
+ In real deployment, we should use ONAP-CA signed certificate for DFC, and vendor-CA signed certificate for xNF.
diff --git a/docs/sections/services/dfc/configuration.rst b/docs/sections/services/dfc/configuration.rst
index 69375c76..22f8d691 100644
--- a/docs/sections/services/dfc/configuration.rst
+++ b/docs/sections/services/dfc/configuration.rst
@@ -38,10 +38,10 @@ The user can also enable secure communication with the DMaaP Message Router.
DFC can handle multiple stream identifiers. For each stream identifier/feed combination the user must provide the
** stream identifier**, **feed name**, and **feed location**.
-**Note!** The **feed name** provided should be used by the consumer/s to set up the subscription to the feed.
+**Note!** The **feed name** provided should be used by the consumer/s to set up the subscription to the feed.
The **stream identifier** shall be defined as an item under the **streams_publishes** tag in the "**applicationConfig**"
-section.
+section.
.. code-block:: yaml
@@ -68,10 +68,10 @@ section.
Under this tag the internal "**feed identifier**" for the feed shall also be added to get the
info about the feed substituted in by CBS (that's what the <<>> tags are for).
-The **feed name** and **feed location** are defined as inputs for the user to provide in helm chart values.yaml. An example snapshot on default configuration is provided below.
+The **feed name** and **feed location** are defined as inputs for the user to provide in helm chart values.yaml. An example snapshot on default configuration is provided below.
.. code-block:: yaml
-
+
# DataRouter Feed Configuration
drFeedConfig:
- feedName: bulk_pm_feed
@@ -79,7 +79,7 @@ The **feed name** and **feed location** are defined as inputs for the user to pr
feedVersion: "0.0"
asprClassification: unclassified
feedDescription: DFC Feed Creation
-
+
# DataRouter Publisher Configuration
drPubConfig:
- feedName: bulk_pm_feed
@@ -95,7 +95,7 @@ Turn On/Off StrictHostChecking
**StrictHostChecking** is a SSH connection option which prevents Man in the Middle (MitM) attacks. If it is enabled, client checks HostName and public key provided by server and compares it with keys stored locally. Only if matching entry is found, SSH connection can be established.
By default in DataFile Collector this option is enabled (true) and requires to provide known_hosts list to DFC container.
-**Important: DFC requires public keys in sha-rsa KeyAlgorithm**
+**Important: DFC requires public keys in sha-rsa KeyAlgorithm**
**Known_hosts file** is a list in following format:
@@ -103,7 +103,7 @@ By default in DataFile Collector this option is enabled (true) and requires to p
<HostName/HostIP> <KeyAlgorithms> <Public Key>
-e.g:
+e.g:
.. code-block:: bash
@@ -137,7 +137,7 @@ e.g:
3. Mount newly created Config Map as Volume to DFC by editing DFC deployment. **DFC deployment contains 3 containers, pay attention to mount the file to the appropriate container.**
.. code-block:: yaml
-
+
...
kind: Deployment
metadata:
@@ -165,7 +165,7 @@ e.g:
Known_hosts file path can be controlled by Environment Variable *KNOWN_HOSTS_FILE_PATH*. Full (absolute) path has to be provided. Sample deployment with changed known_hosts file path can be seen below.
.. code-block:: yaml
-
+
...
kind: Deployment
metadata:
@@ -177,7 +177,7 @@ Known_hosts file path can be controlled by Environment Variable *KNOWN_HOSTS_FIL
spec:
containers:
- image: <DFC image>
- envs:
+ envs:
- name: KNOWN_HOSTS_FILE_PATH
value: /home/datafile/.ssh/new/path/<known_hosts file name, e.g. my_custom_keys>
...
@@ -207,7 +207,7 @@ e.g:
kubectl -n onap edit cm onap-dcae-dfc-known-hosts
-To delete and create again Config Map execute:
+To delete and create again Config Map execute:
.. code-block:: bash
@@ -226,7 +226,7 @@ To turn off StrictHostChecking, set below option to false. It could be changed i
**WARNING: such operation is not recommended as it decreases DFC security and exposes DFC to MitM attacks.**
-.. code-block:: bash
+.. code-block:: yaml
"sftp.security.strictHostKeyChecking": false
@@ -236,7 +236,7 @@ Disable TLS connection
----------------------
The TLS connection in the external interface is enabled by default. To disable TLS, use the following application property:
-.. code-block:: bash
+.. code-block:: yaml
"dmaap.certificateConfig.enableCertAuth": false
diff --git a/docs/sections/services/dfc/consumedapis.rst b/docs/sections/services/dfc/consumedapis.rst
index 258164a7..b1cf714b 100644
--- a/docs/sections/services/dfc/consumedapis.rst
+++ b/docs/sections/services/dfc/consumedapis.rst
@@ -69,4 +69,4 @@ Responses
| HTTP Code | Description |
+===========+=====================+
| **200** | successful response |
-+-----------+---------------------+ \ No newline at end of file
++-----------+---------------------+
diff --git a/docs/sections/services/dfc/delivery.rst b/docs/sections/services/dfc/delivery.rst
index b193bf30..508cc954 100644
--- a/docs/sections/services/dfc/delivery.rst
+++ b/docs/sections/services/dfc/delivery.rst
@@ -27,4 +27,3 @@ Choose your preferred settings (ssh, http or https, with or without hook) and ru
DFC deployment is handled through Helm charts under OOM repository `here`_.
.. _here: https://gerrit.onap.org/r/gitweb?p=oom.git;a=tree;f=kubernetes/dcaegen2-services/components/dcae-datafile-collector
-
diff --git a/docs/sections/services/dfc/http-notes.rst b/docs/sections/services/dfc/http-notes.rst
index c45c7bd8..0fe3a758 100644
--- a/docs/sections/services/dfc/http-notes.rst
+++ b/docs/sections/services/dfc/http-notes.rst
@@ -67,15 +67,14 @@ HTTPS connection with DFC
The file ready message for https server is the same as used in other protocols and http. The only difference is that the scheme is set to
"https":
-.. code-block:: bash
+.. code-block:: json
- ...
- "arrayOfNamedHashMap": [
+ {"arrayOfNamedHashMap": [
{
"name": "C_28532_measData_file.xml",
"hashMap": {
- "location": "https://login:password@server.com:443/file.xml.gz",
- ...
+ "location": "https://login:password@server.com:443/file.xml.gz"
+ }}]}
The processed uri depends on the https connection type that has to be established (client certificate authentication, basic
authentication, and no authentication).
diff --git a/docs/sections/services/dfc/installation-helm.rst b/docs/sections/services/dfc/installation-helm.rst
index cfef688b..62c1709f 100644
--- a/docs/sections/services/dfc/installation-helm.rst
+++ b/docs/sections/services/dfc/installation-helm.rst
@@ -71,4 +71,4 @@ Example yaml file with DataFile Collector configuration:
type: message_router
-More information about properties could be found in configuration section, see :ref:`dfc_configuration`
+More information about properties could be found in configuration section, see :ref:`dfc_configuration`.
diff --git a/docs/sections/services/dfc/logging.rst b/docs/sections/services/dfc/logging.rst
index 51395eb0..ed218515 100644
--- a/docs/sections/services/dfc/logging.rst
+++ b/docs/sections/services/dfc/logging.rst
@@ -9,9 +9,9 @@ file located in datafile-app-server/config folder.
To activate logging, please follow the instructions on this `page`_.
-.. _page: ../troubleshooting.rst
+.. _page: ./troubleshooting.rst
**Where is the log file?**
-The log file is located under /var/log/ONAP/ and called application.log. \ No newline at end of file
+The log file is located under /var/log/ONAP/ and called application.log.
diff --git a/docs/sections/services/dfc/troubleshooting.rst b/docs/sections/services/dfc/troubleshooting.rst
index 680bf1ff..3bb132de 100644
--- a/docs/sections/services/dfc/troubleshooting.rst
+++ b/docs/sections/services/dfc/troubleshooting.rst
@@ -82,7 +82,7 @@ DFC uses a number of configuration parameters. You can find below the kind of re
-Wrong trustedCaPassword:
-.. code-block:: json
+.. code-block:: none
org.onap.dcaegen2.collectors.datafile.tasks.FileCollector |2019-04-24T14:05:54.494Z |WARN |Failed to download file: PNF0 A20000626.2315+0200-2330+0200_PNF0-0-1MB.tar.gz, reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.IOException: Keystore was tampered with, or password was incorrect |RequestID=A20000626.2315+0200-2330+0200_PNF0-0-1MB.tar.gz | | |FileCollectorWorker-2 |
\... |WARN |Failed to download file: ..., reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.IOException: Keystore was tampered with, or password was incorrect ...
@@ -93,7 +93,7 @@ DFC uses a number of configuration parameters. You can find below the kind of re
-Wrong trustedCa:
-.. code-block:: json
+.. code-block:: none
org.onap.dcaegen2.collectors.datafile.tasks.FileCollector |2019-04-24T14:11:22.584Z |WARN |Failed to download file: PNF0 A20000626.2315+0200-2330+0200_PNF0-0-1MB.tar.gz, reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.FileNotFoundException: **WRONGconfig/ftp.jks** |RequestID=A20000626.2315+0200-2330+0200_PNF0-0-1MB.tar.gz | | |FileCollectorWorker-2 |
\... |WARN |Failed to download file: ..., reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.FileNotFoundException: WRONGconfig/ftp.jks ...
@@ -103,7 +103,7 @@ DFC uses a number of configuration parameters. You can find below the kind of re
-Wrong keyPassword:
-.. code-block:: json
+.. code-block:: none
org.onap.dcaegen2.collectors.datafile.tasks.FileCollector |2019-04-24T14:15:40.694Z |WARN |Failed to download file: PNF0 A20000626.2315+0200-2330+0200_PNF0-0-1MB.tar.gz, reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.IOException: Keystore was tampered with, or password was incorrect |RequestID=A20000626.2315+0200-2330+0200_PNF0-0-1MB.tar.gz | | |FileCollectorWorker-2 |
\... |WARN |Failed to download file: ..., reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.IOException: Keystore was tampered with, or password was incorrect ...
@@ -113,7 +113,7 @@ DFC uses a number of configuration parameters. You can find below the kind of re
-Wrong keyCert:
-.. code-block:: json
+.. code-block:: none
org.onap.dcaegen2.collectors.datafile.tasks.FileCollector |2019-04-24T14:20:46.308Z |WARN |Failed to download file: PNF0 A20000626.2315+0200-2330+0200_PNF0-0-1MB.tar.gz, reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.FileNotFoundException: **WRONGconfig/dfc.jks (No such file or directory)** |RequestID=A20000626.2315+0200-2330+0200_PNF0-0-1MB.tar.gz | | |FileCollectorWorker-2 |
\... |WARN |Failed to download file: ..., reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.FileNotFoundException: WRONGconfig/dfc.jks (No such file or directory) ...
@@ -123,7 +123,7 @@ DFC uses a number of configuration parameters. You can find below the kind of re
-Wrong consumer dmaapHostName:
-.. code-block:: json
+.. code-block:: none
org.onap.dcaegen2.collectors.datafile.tasks.ScheduledTasks |2019-04-24T14:27:06.578Z |ERROR |Polling for file ready message failed, exception: java.net.UnknownHostException: **WRONGlocalhost**: Try again, config: DmaapConsumerConfiguration{consumerId=C12, consumerGroup=OpenDcae-c12, timeoutMs=-1, messageLimit=1, **dmaapHostName=WRONGlocalhost**, dmaapPortNumber=2222, dmaapTopicName=/events/unauthenticated.VES_NOTIFICATION_OUTPUT, dmaapProtocol=http, dmaapUserName=, dmaapUserPassword=, dmaapContentType=application/json, trustStorePath=change it, trustStorePasswordPath=change it, keyStorePath=change it, keyStorePasswordPath=change it, enableDmaapCertAuth=false} |RequestID=90fe7450-0bc2-4bf6-a2f0-2aeef6f196ae | | |reactor-http-epoll-3 |
\... |ERROR |Polling for file ready message failed, exception: java.net.UnknownHostException: *WRONGlocalhost*, config: DmaapConsumerConfiguration{..., dmaapHostName=*WRONGlocalhost*, ...} ...
@@ -134,7 +134,7 @@ DFC uses a number of configuration parameters. You can find below the kind of re
-Wrong consumer dmaapPortNumber:
-.. code-block:: json
+.. code-block:: none
org.onap.dcaegen2.collectors.datafile.tasks.ScheduledTasks |2019-04-24T14:33:35.286Z |ERROR |Polling for file ready message failed, exception: io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: localhost/127.0.0.1:**WRONGport**, config: DmaapConsumerConfiguration{consumerId=C12, consumerGroup=OpenDcae-c12, timeoutMs=-1, messageLimit=1, dmaapHostName=localhost, **dmaapPortNumber=WRONGport**, dmaapTopicName=/events/unauthenticated.VES_NOTIFICATION_OUTPUT, dmaapProtocol=http, dmaapUserName=, dmaapUserPassword=, dmaapContentType=application/json, trustStorePath=change it, trustStorePasswordPath=change it, keyStorePath=change it, keyStorePasswordPath=change it, enableDmaapCertAuth=false} |RequestID=b57c68fe-84bf-442f-accd-ea821a5a321f | | |reactor-http-epoll-3 |
\... |ERROR |Polling for file ready message failed, exception: io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: localhost/127.0.0.1:*WRONGport*, config: DmaapConsumerConfiguration{..., dmaapPortNumber=*WRONGport*, ...} ...
@@ -145,7 +145,7 @@ DFC uses a number of configuration parameters. You can find below the kind of re
-Wrong consumer dmaapTopicName:
-.. code-block:: json
+.. code-block:: none
org.onap.dcaegen2.collectors.datafile.tasks.ScheduledTasks |2019-04-24T14:38:07.097Z |ERROR |Polling for file ready message failed, exception: java.lang.RuntimeException: DmaaPConsumer HTTP 404 NOT_FOUND, config: DmaapConsumerConfiguration{consumerId=C12, consumerGroup=OpenDcae-c12, timeoutMs=-1, messageLimit=1, dmaapHostName=localhost, dmaapPortNumber=2222, **dmaapTopicName=/events/unauthenticated.VES_NOTIFICATION_OUTPUTWRONG**, dmaapProtocol=http, dmaapUserName=, dmaapUserPassword=, dmaapContentType=application/json, trustStorePath=change it, trustStorePasswordPath=change it, keyStorePath=change it, keyStorePasswordPath=change it, enableDmaapCertAuth=false} |RequestID=8bd71bac-68af-494b-9518-3ab4478371cf | | |reactor-http-epoll-4 |
\... |ERROR |Polling for file ready message failed, exception: java.lang.RuntimeException: DmaaPConsumer HTTP 404 NOT_FOUND, config: DmaapConsumerConfiguration{..., dmaapTopicName=*/events/unauthenticated.VES_NOTIFICATION_OUTPUTWRONG*, ...} ...
@@ -161,10 +161,10 @@ Missing known_hosts file
""""""""""""""""""""""""
When StrictHostKeyChecking is enabled and DFC cannot find a known_hosts file, the warning information shown below is visible in the logfile. In this case, DFC acts like StrictHostKeyChecking is disabled.
-.. code-block:: bash
+.. code-block:: none
- org.onap.dcaegen2.collectors.datafile.ftp.SftpClient |2020-07-24T06:32:56.010Z
- |WARN |StrictHostKeyChecking is enabled but environment variable KNOWN_HOSTS_FILE_PATH is not set or points to not existing file [/home/datafile/.ssh/known_hosts] --> falling back to StrictHostKeyChecking='no'.
+ org.onap.dcaegen2.collectors.datafile.ftp.SftpClient |2020-07-24T06:32:56.010Z
+ |WARN |StrictHostKeyChecking is enabled but environment variable KNOWN_HOSTS_FILE_PATH is not set or points to not existing file [/home/datafile/.ssh/known_hosts] --> falling back to StrictHostKeyChecking='no'.
To resolve this warning, provide a known_hosts file or disable StrictHostKeyChecking, see DFC config page - :ref:`strict_host_checking_config`.