aboutsummaryrefslogtreecommitdiffstats
path: root/datafile-app-server
AgeCommit message (Collapse)AuthorFilesLines
2019-08-12bump dfc version to 1.2.2el-altoVijay Venkatesh Kumar1-1/+1
Change-Id: I080c26db2639eff2174d4aac334990af67019f46 Signed-off-by: Vijay Venkatesh Kumar <vv770d@att.com> Issue-ID: DCAEGEN2-1669 (cherry picked from commit 50cd922ec5647a13601c2c5100718790244c2fe3)
2019-07-20remove docker bind from mvnVijay Venkatesh Kumar1-2/+7
Change-Id: I5787075e559bcfb56373de5cfa7e09365f16f8b8 Signed-off-by: Vijay Venkatesh Kumar <vv770d@att.com> Issue-ID: DCAEGEN2-1690 Issue-ID: DCAEGEN2-1578
2019-07-12Cbs Client integrationYongchaoWu27-368/+445
Cbs Client is integrated to read configurations from consul Issue-ID: DCAEGEN2-1595 Change-Id: Idb0ebd34eba077f9c1cb584abab4d8722b56f6c5 Signed-off-by: YongchaoWu <yongchao.wu@est.tech>
2019-07-11JJB alignment across DCAE components - DCAE Datafile Collectorelinuxhenrik1-1/+1
Change-Id: I27dd76a8f1b84a8ab26080381321c509c1117a9b Issue-ID: DCAEGEN2-1578 Signed-off-by: elinuxhenrik <henrik.b.andersson@est.tech>
2019-07-02DFC - housekeepingelinuxhenrik22-150/+655
Removed Sonar warnings. Improved code coverage. Improved code formatting Change-Id: I0e8287cecb18a8976635427697dfaeb823da204b Issue-ID: DCAEGEN2-1645 Signed-off-by: elinuxhenrik <henrik.b.andersson@est.tech>
2019-06-14Code formatting with mavenPatrikBuhr50-1081/+1104
This commit only contains code formastting changes done by command: mvn formatter:format spotless:apply process-sources Change-Id: I0fb9d166ad5d9171cdeee9f26b6d353bca74069c Issue-ID: DCAEGEN2-1538 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-06-14Bugfix, improved behaviour for large filesPatrikBuhr14-97/+300
Previously files was read into a buffer for publishing. This does not work when files are bigger than the available memory. After the fix , files are streamed instead. Implemented a new REST primitive for exposing status and statistics. To be used for test and trouble shooting. Change-Id: Iab5a1ee9ffcbf6836fcf709d115bf25ab0391732 Issue-ID: DCAEGEN2-1532 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-06-05Datafile - Improve JUnit testselinuxhenrik23-257/+501
Improve JUnit tests for the Datafile Collector. Add tests of logging. Remove dependency to old JUnit version. Improve some log messages. Improve code coverage. Change-Id: Ie8204619ab7fe4294c4ac0dd8f030973653ef4a3 Issue-ID: DCAEGEN2-1490 Signed-off-by: elinuxhenrik <henrik.b.andersson@est.tech>
2019-06-04Blueprint alignmentYongchaoWu7-146/+94
Blueprint is aligned with the dmaap topic configuration Change-Id: I4303724eed21047b87a8b7438970625b08811d22 Issue-ID: DCAEGEN2-1145 Signed-off-by: YongchaoWu <yongchao.wu@est.tech>
2019-06-04Merge "Fix a small bug for swagger-ui"Henrik Andersson1-0/+8
2019-06-04Merge "Bugfixes, Generalizing Data File Collection to handle any type of file"Henrik Andersson1-9/+7
2019-06-03Bugfixes, Generalizing Data File Collection to handle any type of filePatrikBuhr1-9/+7
- When a change ID was not configured, the task counter was not decreased. The result was that the DFC stopped polling, - When the check if a file is a already published fails (most likely to a problem is the DR), the DFC will try to publish it (instead of just ingnoring it). Change-Id: If9f5b962210f809d5d2ae0aa60d3a7f99099c058 Issue-ID: DCAEGEN2-1532 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-06-03DFC, Purging the cache of published files occasionally failsPatrikBuhr1-5/+5
Solved a threading issue. Change-Id: I1272cadd5d8bc8bb8ffc4c3d33c9b5c706470864 Issue-ID: DCAEGEN2-1588 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-05-29Generalizing Data File Collection to handle any type of filePatrikBuhr43-955/+1097
Extension of the DFC to be able to handle any file types types, which are published on different DR feeds. This association between file type and DR feed is defined by configuration. The file type is defined by the changeIdentifier in the fileReady VES message reported from the PNF. The creation of DR feeds and configuration will be done by the DMAAP plugin, but that is not tested yet. Change-Id: I13b36acd926a6941ee733e6b37922049fb54a5d9 Issue-ID: DCAEGEN2-1532 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-05-28Fix a small bug for swagger-uiyanhuanwang1-0/+8
swagger-ui and actuator-ui are missing. This commit is to fix this bug. Change-Id: I237ad1739dd23b7316a3efecb40a9bd78247ff33 Issue-ID: DCAEGEN2-1535 Signed-off-by: yanhuanwang <martin.yan.seu@gmail.com>
2019-05-22Merge "Fix for robustness"Henrik Andersson3-9/+16
2019-05-17Fix, skip FTP retry in certain casesPatrikBuhr7-13/+77
In certain conditions there is no reason to retry fetching files. For instance when the file is removed in the PNF or when the password/certificate is wrong. When the DFC is started there are always queued VES events that referes to removed files which in turn results in that the DFC will retry fetching these files in vain. The DFC house keeps its number of concurrents tasks to not exeed quotas for memory,open file descriptors etc. As more threads are occupied with retrying, the fewer threads can do their intended work, which decreases the throughput. Testing has showed that already when the number of PNFs are 10, the throughput is radically decreased (and the problem is then escalating). Change-Id: I4833724a3ef3509025f4a0a438c6c8025932b0f4 Issue-ID: DCAEGEN2-1508 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-05-17Fix for robustnessPatrikBuhr3-9/+16
In the long time stability test it has showed that when there are an exessive amount of FileReady events, the DFC will fetch these from the MR and build up an internal queue. This has the effect that the DFC will consume too much memory. Itmight also affect load balancing (in case of several DFC instances) so that one might consume all events and the others get nothing. The DFC should not fetch new FileReady events when it fully loaded with work. Change-Id: I58665edd678d2f1c8a32d0e56455228b522aab40 Issue-ID: DCAEGEN2-1509 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-05-13Bugfix: add MDC context to http redirect logsRehanRaza4-9/+32
Change-Id: Icdf00480eed43536a8b4693cbb208560f0881c95 Issue-ID: DCAEGEN2-1462 Signed-off-by: RehanRaza <muhammad.rehan.raza@est.tech>
2019-05-10Merge "Clean up pom files"Vijay Venkatesh Kumar1-119/+64
2019-05-08Code improvement, FilePublishInformation serializingPatrikBuhr5-94/+36
Updated serializer of FilePublishInformation so that only elements that should be part of publishing is included. Change-Id: If5632af8bf9fd26f8bc96b25d8bfbccf63dfec2f Issue-ID: DCAEGEN2-1488 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-05-08Clean up pom fileselinuxhenrik1-119/+64
Change-Id: Ifede82e15ebc38af92ade8259a6e1f641946b815 Issue-ID: DCAEGEN2-1487 Signed-off-by: elinuxhenrik <henrik.b.andersson@est.tech>
2019-05-07Merge "Bugfix, trace context"Henrik Andersson1-4/+2
2019-05-05Ownership fix in config directoryTamasBakai3-3/+3
Change-Id: Iec3ca75c42642da27c52bccd6d60811c434f778a Issue-ID: DCAEGEN2-1466 Signed-off-by: TamasBakai <tamas.bakai@est.tech>
2019-05-02Bugfix, trace contextPatrikBuhr1-4/+2
When retrying of file publish, the RequestID in the trace context is incorect. The context must be expicitly updated since retrying is executed in a separate thread. Change-Id: I1c5dc0ba0e53c1219e23f93d8c5b420996a16c93 Issue-ID: DCAEGEN2-1462 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-04-25Extend Swagger for actuator end-points also1.1.21.1.1TamasBakai1-10/+12
Change-Id: Ia036f7d79d25ceb2f1e465d79f17db1acbabbfbf Issue-ID: DCAEGEN2-1443 Signed-off-by: TamasBakai <tamas.bakai@est.tech>
2019-04-16Bugfix, serializing in DataRouterPublisherPatrikBuhr2-1/+8
This fixes a bug introduced in c7b0c7014688c447d411f791a094440403e65228. The new property must be removed before publishing to data router. The consequense of the bug is that the DR rejects the publishing. Change-Id: Ib42cbed4decdba4c13678cea47dff35ca77679a5 Issue-ID: DCAEGEN2-1305 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-04-12Improved loggingPatrikBuhr12-102/+169
File name is used in Requext_ID to make it more easy to trouble shoot. Change-Id: Ied98766f1a177a9bda35ec5892a60d06619ff3a7 Issue-ID: DCAEGEN2-1305 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-04-12Fixed build problemsPatrikBuhr2-29/+8
Fixed a build problem cause by a merge error from previous commits. Fixes warnings from maven compile Change-Id: I1ce88b2686b3be5ecd17113cbfde3e65297c27c6 Issue-ID: DCAEGEN2-1118 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-04-11Merge "Robustness issue"Vijay Venkatesh Kumar2-19/+40
2019-04-11Merge "Enabling Spring boot actuator"Vijay Venkatesh Kumar2-4/+65
2019-04-11Robustness issuePatrikBuhr3-20/+41
Limiting the number of paralell threads when retying of ftp or publish. Previously there was no upper limit on how many paralel threads that could be started by retry attempts. For instance, a worker thread with 100 new files would start 100 new threads when the ftt server was unavailable. Change-Id: Ia8792f03ea55c0c467ef248ddc9d59187c06a946 Issue-ID: DCAEGEN2-1118 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-04-11Enabling Spring boot actuatorPatrikBuhr2-4/+65
This makes it possible by a REST API: - change logging levels - get the application log - get build info for the DFC - get various metrixes for the execution such as heap consumption, number of threads etc. Change-Id: I570bc0db6a9b9977ecfd83389b463fe652e4ba16 Issue-ID: DCAEGEN2-1428 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-04-11Extend heartbeat to also include som health statuselinuxhenrik6-11/+146
Add info about the internal status of DFC to the healtCheck message. Change-Id: Ic2ad3e5b97c42d6affe1ac4e1d7cf2274027f881 Issue-ID: DCAEGEN2-1403 Signed-off-by: elinuxhenrik <henrik.b.andersson@est.tech>
2019-04-11Housekeepingelinuxhenrik69-646/+2808
No functional changes made in this commit. Removed CheckStyle and Sonar warnings. Formatted code. Renamed methods and classes for better understanding. Removed unnecessary classes. Moved all code to single project. Change-Id: Ie3feb6c6a985e94a382812aa083dcf57bc46c7b3 Issue-ID: DCAEGEN2-1367 Signed-off-by: elinuxhenrik <henrik.b.andersson@est.tech>
2019-04-10Updating certificatesTamasBakai11-20/+70
Issue-ID: DCAEGEN2-1300 Change-Id: I9ce457b69e5866e77a465fb040d196b8905ca274 Signed-off-by: TamasBakai <tamas.bakai@est.tech>
2019-04-04Improved loggingPatrikBuhr15-222/+162
Fixed problem with startup which lead to that the REST API was not working running locally. Fixed problem with DmaapProducerHttpClient which would use no timeout, which can lead to infinitly haninging threads. A long timeout is used instead. Change-Id: I28469b1b3aaad0dab4cf247bb8af968e71a60133 Issue-ID: DCAEGEN2-1305 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-03-29Change file name sent to DataRouterelinuxhenrik6-30/+16
In the previous commit for this issue, the file name sent to DataRouter was changed. This turned out to be wrong, so in this commit it is changed back again. Change-Id: I9b0d726f406758846fd256626e73edf59d710c8b Issue-ID: DCAEGEN2-1256 Signed-off-by: elinuxhenrik <henrik.b.andersson@est.tech>
2019-03-28Merge "Fix DataFile - Align Dmaap Topic configuration"Vijay Venkatesh Kumar6-471/+8
2019-03-22Local filename updated, stability issuesPatrikBuhr18-254/+205
The local filename is changed so it contains PNF name instead of the PNF IP address. The paralellism is restricted to 100 worker threads in order to solve problems with too many open file descriptors and out of memory. Logging is improved. Change-Id: I24ce2e23020cc253a3c7bebac1ab5cf703b5b144 Issue-ID: DCAEGEN2-1118 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-03-21Add check to DataRouter if file has been publishedmaximesson16-579/+1124
For each file in the FileReady message that DFC does not know if it has been published yet, it should ask DataRouter if it has been published already to avoid downloading and publishing a file more than once. Change-Id: I18117a6e968ec929aa255052a4c44f890a8ed39d Issue-ID: DCAEGEN2-1256 Signed-off-by: maximesson <maxime.bonneau@est.tech>
2019-03-18Fix DataFile - Align Dmaap Topic configurationYongchaoWu6-471/+8
Remove additional configuration in Blueprints Remove additional configruation in component specs Change-Id: I0d546227af0be485f2ffed27a93c701a7654ff8e Issue-ID: DCAEGEN2-1145 Signed-off-by: YongchaoWu <yongchao.wu@est.tech>
2019-03-18Merge "Thread safety issues"Tony Hansen7-158/+145
2019-03-14DataFile - Align Dmaap Topic configurationYongchaoWu6-57/+594
Blueprints alignment Configbinding service support Modify the policy model complaint to R4 model expected by Policy Change-Id: Ic53f4ab2a5867e5cdf0fc735be180715077154ee Issue-ID: DCAEGEN2-1145 Signed-off-by: YongchaoWu <yongchao.wu@est.tech>
2019-03-14Thread safety issuesPatrikBuhr7-158/+145
The TrustManager is now loaded and initialized once in a thread safe way (instead of each time it is used). Removed some unneeded wrappers. Using AutoCloseable for FTP clients to make sure they are closed in case of exceptions. Made AppConfig thread safe. Change-Id: Ia6a2c8a76bf960013180fdd7c53ae0ff17b26505 Issue-ID: DCAEGEN2-1118 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-03-08DFC logging according to ONAP specificationRehanRaza13-119/+214
Change-Id: I6fe18ce3bdbc6d0b1cf5c5e65534cab694cfb898 Issue-ID: DCAEGEN2-1305 Signed-off-by: RehanRaza <muhammad.rehan.raza@est.tech>
2019-03-08Merge "Improvement of the parallelism"Tony Hansen3-67/+379
2019-03-07Change to docker-maven-pluginRehanRaza2-20/+26
Change-Id: I5dc3c5b69b2ef764992e2db76eb9cc154ff5adf7 Issue-ID: DCAEGEN2-1269 Signed-off-by: RehanRaza <muhammad.rehan.raza@est.tech>
2019-03-06Improvement of the parallelismPatrikBuhr3-67/+379
The reactive framework Scedulers uses to few threads. (the same number as the number of processors). That is too few for an io-intense application like this where CPU is not the limiting factor. Change-Id: Ia5f41e75716d309f47dce5f5273b739f7e6d136a Issue-ID: DCAEGEN2-1118 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-03-06Fix dockerfile-maven-plugin in pom.xmlRehanRaza1-3/+9
Change-Id: Id0389e1011fdce9dee4fc0d330973ef34bc2ed6a Issue-ID: DCAEGEN2-1269 Signed-off-by: RehanRaza <muhammad.rehan.raza@est.tech>