summaryrefslogtreecommitdiffstats
path: root/docs/sections/services/datalake-handler
diff options
context:
space:
mode:
authorVijay Venkatesh Kumar <vv770d@att.com>2020-02-21 19:22:38 +0000
committerGerrit Code Review <gerrit@onap.org>2020-02-21 19:22:38 +0000
commit729478d078e7570824472d02ff0d4e115cb33144 (patch)
tree8abb1c40fb158e27cf1175ff3ed19bc8a0a4b437 /docs/sections/services/datalake-handler
parentcc673dc6fef15973fa48cc55a69c64291ed5e92b (diff)
parentbb94cb7d7a20e62da9eed8b07c3a82fd28bd38eb (diff)
Merge "DL-Handler Doc"
Diffstat (limited to 'docs/sections/services/datalake-handler')
-rw-r--r--docs/sections/services/datalake-handler/DL-DES.PNGbin0 -> 38114 bytes
-rw-r--r--docs/sections/services/datalake-handler/arch.PNGbin0 -> 79479 bytes
-rw-r--r--docs/sections/services/datalake-handler/dbschema.PNGbin0 -> 74430 bytes
-rw-r--r--docs/sections/services/datalake-handler/feeder-arch.PNGbin0 -> 64492 bytes
-rw-r--r--docs/sections/services/datalake-handler/index.rst37
-rw-r--r--docs/sections/services/datalake-handler/installation.rst4
-rw-r--r--docs/sections/services/datalake-handler/overview.rst76
-rw-r--r--docs/sections/services/datalake-handler/userguide.rst4
8 files changed, 121 insertions, 0 deletions
diff --git a/docs/sections/services/datalake-handler/DL-DES.PNG b/docs/sections/services/datalake-handler/DL-DES.PNG
new file mode 100644
index 00000000..aa3c494c
--- /dev/null
+++ b/docs/sections/services/datalake-handler/DL-DES.PNG
Binary files differ
diff --git a/docs/sections/services/datalake-handler/arch.PNG b/docs/sections/services/datalake-handler/arch.PNG
new file mode 100644
index 00000000..eb809c6c
--- /dev/null
+++ b/docs/sections/services/datalake-handler/arch.PNG
Binary files differ
diff --git a/docs/sections/services/datalake-handler/dbschema.PNG b/docs/sections/services/datalake-handler/dbschema.PNG
new file mode 100644
index 00000000..14c0d931
--- /dev/null
+++ b/docs/sections/services/datalake-handler/dbschema.PNG
Binary files differ
diff --git a/docs/sections/services/datalake-handler/feeder-arch.PNG b/docs/sections/services/datalake-handler/feeder-arch.PNG
new file mode 100644
index 00000000..cb995fac
--- /dev/null
+++ b/docs/sections/services/datalake-handler/feeder-arch.PNG
Binary files differ
diff --git a/docs/sections/services/datalake-handler/index.rst b/docs/sections/services/datalake-handler/index.rst
new file mode 100644
index 00000000..56ada5f2
--- /dev/null
+++ b/docs/sections/services/datalake-handler/index.rst
@@ -0,0 +1,37 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+
+DataLake-Handler MS
+==============
+
+**DataLake-Handler MS** is a software component of ONAP that can systematically persist the events from DMaaP into supported Big Data storage systems.
+It has a Admin UI, where a system administrator configures which Topics to be monitored, and to which data storage to store the data.
+It is also used to manage the settings of the storage and associated data analytics tool.
+The second part is the Feeder, which does the data transfer work and is horizontal scalable.
+
+.. image:: DL-DES.PNG
+
+DataLake-Handler MS overview and functions
+------------------------------------------
+
+.. toctree::
+ :maxdepth: 1
+
+ ./overview.rst
+
+DataLake-Handler MS Installation Steps and Configurations
+---------------------------------------------------------
+
+.. toctree::
+ :maxdepth: 1
+
+ ./installation.rst
+
+DataLake-Handler MS Admin UI User Guide
+---------------------------------------
+
+.. toctree::
+ :maxdepth: 1
+
+ ./userguide.rst
diff --git a/docs/sections/services/datalake-handler/installation.rst b/docs/sections/services/datalake-handler/installation.rst
new file mode 100644
index 00000000..2235198a
--- /dev/null
+++ b/docs/sections/services/datalake-handler/installation.rst
@@ -0,0 +1,4 @@
+Installation
+============
+
+DataLake handler microservice can be deployed using ... (TODO) \ No newline at end of file
diff --git a/docs/sections/services/datalake-handler/overview.rst b/docs/sections/services/datalake-handler/overview.rst
new file mode 100644
index 00000000..51dab104
--- /dev/null
+++ b/docs/sections/services/datalake-handler/overview.rst
@@ -0,0 +1,76 @@
+.. This work is licensed under a Creative Commons Attribution 4.0
+ International License. http://creativecommons.org/licenses/by/4.0
+
+.. _docs_Datalake_Handler_MS:
+
+Architecture
+------------
+
+
+Background
+~~~~~~~~~~
+There are large amount of data flowing among ONAP components, mostly via DMaaP and Web Services.
+For example, all events/feed collected by DCAE collectors go through DMaaP.
+DMaaP is backed by Kafka, which is a system for Publish-Subscribe,
+where data is not meant to be permanent and gets deleted after certain retention period.
+Kafka is not a database, means that data there is not for query.
+Though some components may store processed result into their local databases, most of the raw data will eventually lost.
+We should provide a systematic way to store these raw data, and even the processed result,
+which will serve as the source for data analytics and machine learning, providing insight to the network operation.
+
+
+Relations with Other ONAP Components
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The architecture below depicts the DataLake MS as a part of ONAP. Only the relevant interactions and components are shown.
+
+.. image:: ./arch.PNG
+
+Note that not all data storage systems in the picture are supported. In R6, the following storage are supported:
+ - MongoDB
+ - Couchbase
+ - Elasticsearch and Kibana
+ - HDFS
+Depending on demands, new systems may be added to the supported list. In the following we use the term database for the storage,
+even though HDFS is a file system (but with simple settings, it can be treats as a database, e.g. Hive.)
+
+Note that once the data is stored in databases, other ONAP components and systems will directly query data from the databases,
+without interacting with DataLake Handler.
+
+Description
+~~~~~~~~~~~
+DataLate Handler's main function is to monitor and persist data flow through DMaaP. The databases are outside of ONAP scope,
+since the data is expected to be huge, and a database may be a complicated cluster consisting of thousand of nodes.
+
+Admin UI
+~~~~~~~~
+A system administrator uses DataLake Admin UI to:
+ - Configure external database connections, such as host, port, login.
+ - Configure which Topics to monitor, which databases to store the data for each Topic.
+ - Pre-configured 3rd Party Tools dashboards and templates.
+
+This UI tool is used to manage all the Dayalake settings stored in MariaDB. Here is the database schema:
+
+.. image:: ./dbschema.PNG
+
+Feeder
+~~~~~~
+Architecture
+.. image:: ./feeder-arch.PNG
+
+Features
+
+ - Read data directly from Kafka for performance.
+ - Support for pluggable databases. To add a new database, we only need to implement its corrosponding service.
+ - Support REST API for inter-component communications. Besides managing DatAlake settings in MariaDB,
+ Admin UI also use this API to start/stop Feeder, query Feeder status and statistics.
+ - Use MariaDB to store settings.
+ - Support data processing features. Before persisting data, data can be massaged in Feeder.
+ Currently two features are implemented: Correlate Cleared Message (in org.onap.datalake.feeder.service.db.ElasticsearchService)
+ and Flatten JSON Array (org.onap.datalake.feeder.service.StoreService).
+ - Connection to Kafka and DBs are secured
+
+
+Links
+~~~~~
+ - DataLake Development Environment Setup https://wiki.onap.org/display/DW/DataLake+Development+Environment+Setup
+ - Source Code https://gerrit.onap.org/r/gitweb?p=dcaegen2/services.git;a=tree;f=components/datalake-handler;hb=HEAD
diff --git a/docs/sections/services/datalake-handler/userguide.rst b/docs/sections/services/datalake-handler/userguide.rst
new file mode 100644
index 00000000..4a0957f5
--- /dev/null
+++ b/docs/sections/services/datalake-handler/userguide.rst
@@ -0,0 +1,4 @@
+Admin UI User Guide
+---------------------
+
+To be filled.