From bb94cb7d7a20e62da9eed8b07c3a82fd28bd38eb Mon Sep 17 00:00:00 2001 From: Guobiao Mo Date: Tue, 11 Feb 2020 17:14:33 -0800 Subject: DL-Handler Doc Issue-ID: DCAEGEN2-2028 Change-Id: Ifa26667a9ef1d6d8be711b1454d3e4ff6ea9a74b Signed-off-by: Guobiao Mo --- docs/sections/services/datalake-handler/DL-DES.PNG | Bin 0 -> 38114 bytes docs/sections/services/datalake-handler/arch.PNG | Bin 0 -> 79479 bytes .../services/datalake-handler/dbschema.PNG | Bin 0 -> 74430 bytes .../services/datalake-handler/feeder-arch.PNG | Bin 0 -> 64492 bytes docs/sections/services/datalake-handler/index.rst | 37 ++++++++++ .../services/datalake-handler/installation.rst | 4 ++ .../services/datalake-handler/overview.rst | 76 +++++++++++++++++++++ .../services/datalake-handler/userguide.rst | 4 ++ 8 files changed, 121 insertions(+) create mode 100644 docs/sections/services/datalake-handler/DL-DES.PNG create mode 100644 docs/sections/services/datalake-handler/arch.PNG create mode 100644 docs/sections/services/datalake-handler/dbschema.PNG create mode 100644 docs/sections/services/datalake-handler/feeder-arch.PNG create mode 100644 docs/sections/services/datalake-handler/index.rst create mode 100644 docs/sections/services/datalake-handler/installation.rst create mode 100644 docs/sections/services/datalake-handler/overview.rst create mode 100644 docs/sections/services/datalake-handler/userguide.rst (limited to 'docs/sections/services/datalake-handler') diff --git a/docs/sections/services/datalake-handler/DL-DES.PNG b/docs/sections/services/datalake-handler/DL-DES.PNG new file mode 100644 index 00000000..aa3c494c Binary files /dev/null and b/docs/sections/services/datalake-handler/DL-DES.PNG differ diff --git a/docs/sections/services/datalake-handler/arch.PNG b/docs/sections/services/datalake-handler/arch.PNG new file mode 100644 index 00000000..eb809c6c Binary files /dev/null and b/docs/sections/services/datalake-handler/arch.PNG differ diff --git a/docs/sections/services/datalake-handler/dbschema.PNG b/docs/sections/services/datalake-handler/dbschema.PNG new file mode 100644 index 00000000..14c0d931 Binary files /dev/null and b/docs/sections/services/datalake-handler/dbschema.PNG differ diff --git a/docs/sections/services/datalake-handler/feeder-arch.PNG b/docs/sections/services/datalake-handler/feeder-arch.PNG new file mode 100644 index 00000000..cb995fac Binary files /dev/null and b/docs/sections/services/datalake-handler/feeder-arch.PNG differ diff --git a/docs/sections/services/datalake-handler/index.rst b/docs/sections/services/datalake-handler/index.rst new file mode 100644 index 00000000..56ada5f2 --- /dev/null +++ b/docs/sections/services/datalake-handler/index.rst @@ -0,0 +1,37 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + + +DataLake-Handler MS +============== + +**DataLake-Handler MS** is a software component of ONAP that can systematically persist the events from DMaaP into supported Big Data storage systems. +It has a Admin UI, where a system administrator configures which Topics to be monitored, and to which data storage to store the data. +It is also used to manage the settings of the storage and associated data analytics tool. +The second part is the Feeder, which does the data transfer work and is horizontal scalable. + +.. image:: DL-DES.PNG + +DataLake-Handler MS overview and functions +------------------------------------------ + +.. toctree:: + :maxdepth: 1 + + ./overview.rst + +DataLake-Handler MS Installation Steps and Configurations +--------------------------------------------------------- + +.. toctree:: + :maxdepth: 1 + + ./installation.rst + +DataLake-Handler MS Admin UI User Guide +--------------------------------------- + +.. toctree:: + :maxdepth: 1 + + ./userguide.rst diff --git a/docs/sections/services/datalake-handler/installation.rst b/docs/sections/services/datalake-handler/installation.rst new file mode 100644 index 00000000..2235198a --- /dev/null +++ b/docs/sections/services/datalake-handler/installation.rst @@ -0,0 +1,4 @@ +Installation +============ + +DataLake handler microservice can be deployed using ... (TODO) \ No newline at end of file diff --git a/docs/sections/services/datalake-handler/overview.rst b/docs/sections/services/datalake-handler/overview.rst new file mode 100644 index 00000000..51dab104 --- /dev/null +++ b/docs/sections/services/datalake-handler/overview.rst @@ -0,0 +1,76 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 + International License. http://creativecommons.org/licenses/by/4.0 + +.. _docs_Datalake_Handler_MS: + +Architecture +------------ + + +Background +~~~~~~~~~~ +There are large amount of data flowing among ONAP components, mostly via DMaaP and Web Services. +For example, all events/feed collected by DCAE collectors go through DMaaP. +DMaaP is backed by Kafka, which is a system for Publish-Subscribe, +where data is not meant to be permanent and gets deleted after certain retention period. +Kafka is not a database, means that data there is not for query. +Though some components may store processed result into their local databases, most of the raw data will eventually lost. +We should provide a systematic way to store these raw data, and even the processed result, +which will serve as the source for data analytics and machine learning, providing insight to the network operation. + + +Relations with Other ONAP Components +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The architecture below depicts the DataLake MS as a part of ONAP. Only the relevant interactions and components are shown. + +.. image:: ./arch.PNG + +Note that not all data storage systems in the picture are supported. In R6, the following storage are supported: + - MongoDB + - Couchbase + - Elasticsearch and Kibana + - HDFS +Depending on demands, new systems may be added to the supported list. In the following we use the term database for the storage, +even though HDFS is a file system (but with simple settings, it can be treats as a database, e.g. Hive.) + +Note that once the data is stored in databases, other ONAP components and systems will directly query data from the databases, +without interacting with DataLake Handler. + +Description +~~~~~~~~~~~ +DataLate Handler's main function is to monitor and persist data flow through DMaaP. The databases are outside of ONAP scope, +since the data is expected to be huge, and a database may be a complicated cluster consisting of thousand of nodes. + +Admin UI +~~~~~~~~ +A system administrator uses DataLake Admin UI to: + - Configure external database connections, such as host, port, login. + - Configure which Topics to monitor, which databases to store the data for each Topic. + - Pre-configured 3rd Party Tools dashboards and templates. + +This UI tool is used to manage all the Dayalake settings stored in MariaDB. Here is the database schema: + +.. image:: ./dbschema.PNG + +Feeder +~~~~~~ +Architecture +.. image:: ./feeder-arch.PNG + +Features + + - Read data directly from Kafka for performance. + - Support for pluggable databases. To add a new database, we only need to implement its corrosponding service. + - Support REST API for inter-component communications. Besides managing DatAlake settings in MariaDB, + Admin UI also use this API to start/stop Feeder, query Feeder status and statistics. + - Use MariaDB to store settings. + - Support data processing features. Before persisting data, data can be massaged in Feeder. + Currently two features are implemented: Correlate Cleared Message (in org.onap.datalake.feeder.service.db.ElasticsearchService) + and Flatten JSON Array (org.onap.datalake.feeder.service.StoreService). + - Connection to Kafka and DBs are secured + + +Links +~~~~~ + - DataLake Development Environment Setup https://wiki.onap.org/display/DW/DataLake+Development+Environment+Setup + - Source Code https://gerrit.onap.org/r/gitweb?p=dcaegen2/services.git;a=tree;f=components/datalake-handler;hb=HEAD diff --git a/docs/sections/services/datalake-handler/userguide.rst b/docs/sections/services/datalake-handler/userguide.rst new file mode 100644 index 00000000..4a0957f5 --- /dev/null +++ b/docs/sections/services/datalake-handler/userguide.rst @@ -0,0 +1,4 @@ +Admin UI User Guide +--------------------- + +To be filled. -- cgit 1.2.3-korg