aboutsummaryrefslogtreecommitdiffstats
path: root/datafile-app-server/src/test
AgeCommit message (Collapse)AuthorFilesLines
2021-02-11Add HTTPS to collect files from xNFs1.5.3Krzysztof Gajewski13-46/+644
- plus small refactoring related to above - update to version 1.5.3 Issue-ID: DCAEGEN2-2528 Signed-off-by: Krzysztof Gajewski <krzysztof.gajewski@nokia.com> Change-Id: I2531c85967964f1359bafd5b694afbf662edf54e
2020-12-30Add HTTP as new protocol to collect files from xNFsKrzysztof Gajewski10-12/+332
- HTTP basic auth included - small code refactoring related to the task Issue-ID: DCAEGEN2-2527 Signed-off-by: Krzysztof Gajewski <krzysztof.gajewski@nokia.com> Change-Id: I13ec80e996861e14d2c561087c4af3b34d861030
2020-12-02Upgrade Reactor and SpringBoot1.5.0tkogut11-90/+85
Issue-ID: DCAEGEN2-2537 Signed-off-by: tkogut <tomasz.kogut@nokia.com> Change-Id: Ide560554f11d01737b351d6138430efe6005791d
2020-08-25Update dcae SDK from 1.1.6 to 1.4.2Remigiusz Janeczek13-429/+162
Bump project version from 1.4.2 to 1.4.3 Update deprecated calls to JsonParser Make logs always go to file and console Issue-ID: DCAEGEN2-2267 Signed-off-by: Remigiusz Janeczek <remigiusz.janeczek@nokia.com> Change-Id: Ib8d7f82b3daf03ca327581c9a5dc4f6f27a20141
2020-07-30Removed support for FTPS protocol (from now only FTPeS and sFTP will be ↵Jan Malkiewicz7-48/+52
supported). Issue-ID: DCAEGEN2-2367 Signed-off-by: Jan Malkiewicz <jan.malkiewicz@nokia.com> Change-Id: Ifada2899431c56bdbfcf3aad73fdc9b376ae9c4f
2020-07-28Add sftp strict host key checking to DFC.Jan Malkiewicz6-39/+115
Issue-ID: DCAEGEN2-2219 Signed-off-by: Jan Malkiewicz <jan.malkiewicz@nokia.com> Change-Id: Iadf6c6bd743c42ebb3bf9ad8ac443fc0f3f58063
2019-10-16DataFileCollector use wrong KeyManagerFactoryburdziak1-8/+8
Issue-ID: DCAEGEN2-1854 Signed-off-by: burdziak <olaf.burdziakowski@nokia.com> Change-Id: I71c7526097014e10d0ef091e38a929b81ba1f627
2019-09-18Personalized client certificate authentication via FTPes in 5G Bulk PM ↵burdziak1-0/+27
usecase doesn't work Issue-ID: DCAEGEN2-1754 Signed-off-by: burdziak <olaf.burdziakowski@nokia.com> Change-Id: Idec3efce8412dd175ecf75f55790a24cec651104
2019-09-06TLS init container usageecaiyanlinux8-17/+19
Change-Id: Iec51c722f6a3a519fd7eb0f37f8b801eebbc039b Issue-ID: DCAEGEN2-1702 Signed-off-by: ecaiyanlinux <martin.c.yan@est.tech>
2019-08-12DFC HousekeepingPatrikBuhr9-68/+71
Removed warnings. Improved code formatting Removed unused files Removed datafile_endpoints.json from the container Change-Id: I7334775be793cd9cb709ca1e031c620c2c1c2b3f Issue-ID: DCAEGEN2-1645 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-07-12Cbs Client integrationYongchaoWu13-203/+244
Cbs Client is integrated to read configurations from consul Issue-ID: DCAEGEN2-1595 Change-Id: Idb0ebd34eba077f9c1cb584abab4d8722b56f6c5 Signed-off-by: YongchaoWu <yongchao.wu@est.tech>
2019-07-02DFC - housekeepingelinuxhenrik10-67/+504
Removed Sonar warnings. Improved code coverage. Improved code formatting Change-Id: I0e8287cecb18a8976635427697dfaeb823da204b Issue-ID: DCAEGEN2-1645 Signed-off-by: elinuxhenrik <henrik.b.andersson@est.tech>
2019-06-14Code formatting with mavenPatrikBuhr17-784/+791
This commit only contains code formastting changes done by command: mvn formatter:format spotless:apply process-sources Change-Id: I0fb9d166ad5d9171cdeee9f26b6d353bca74069c Issue-ID: DCAEGEN2-1538 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-06-14Bugfix, improved behaviour for large filesPatrikBuhr5-28/+50
Previously files was read into a buffer for publishing. This does not work when files are bigger than the available memory. After the fix , files are streamed instead. Implemented a new REST primitive for exposing status and statistics. To be used for test and trouble shooting. Change-Id: Iab5a1ee9ffcbf6836fcf709d115bf25ab0391732 Issue-ID: DCAEGEN2-1532 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-06-05Datafile - Improve JUnit testselinuxhenrik9-159/+397
Improve JUnit tests for the Datafile Collector. Add tests of logging. Remove dependency to old JUnit version. Improve some log messages. Improve code coverage. Change-Id: Ie8204619ab7fe4294c4ac0dd8f030973653ef4a3 Issue-ID: DCAEGEN2-1490 Signed-off-by: elinuxhenrik <henrik.b.andersson@est.tech>
2019-05-29Generalizing Data File Collection to handle any type of filePatrikBuhr15-520/+444
Extension of the DFC to be able to handle any file types types, which are published on different DR feeds. This association between file type and DR feed is defined by configuration. The file type is defined by the changeIdentifier in the fileReady VES message reported from the PNF. The creation of DR feeds and configuration will be done by the DMAAP plugin, but that is not tested yet. Change-Id: I13b36acd926a6941ee733e6b37922049fb54a5d9 Issue-ID: DCAEGEN2-1532 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-05-22Merge "Fix for robustness"Henrik Andersson1-0/+1
2019-05-17Fix, skip FTP retry in certain casesPatrikBuhr1-0/+17
In certain conditions there is no reason to retry fetching files. For instance when the file is removed in the PNF or when the password/certificate is wrong. When the DFC is started there are always queued VES events that referes to removed files which in turn results in that the DFC will retry fetching these files in vain. The DFC house keeps its number of concurrents tasks to not exeed quotas for memory,open file descriptors etc. As more threads are occupied with retrying, the fewer threads can do their intended work, which decreases the throughput. Testing has showed that already when the number of PNFs are 10, the throughput is radically decreased (and the problem is then escalating). Change-Id: I4833724a3ef3509025f4a0a438c6c8025932b0f4 Issue-ID: DCAEGEN2-1508 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-05-17Fix for robustnessPatrikBuhr1-0/+1
In the long time stability test it has showed that when there are an exessive amount of FileReady events, the DFC will fetch these from the MR and build up an internal queue. This has the effect that the DFC will consume too much memory. Itmight also affect load balancing (in case of several DFC instances) so that one might consume all events and the others get nothing. The DFC should not fetch new FileReady events when it fully loaded with work. Change-Id: I58665edd678d2f1c8a32d0e56455228b522aab40 Issue-ID: DCAEGEN2-1509 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-05-13Bugfix: add MDC context to http redirect logsRehanRaza2-4/+14
Change-Id: Icdf00480eed43536a8b4693cbb208560f0881c95 Issue-ID: DCAEGEN2-1462 Signed-off-by: RehanRaza <muhammad.rehan.raza@est.tech>
2019-05-08Code improvement, FilePublishInformation serializingPatrikBuhr2-57/+4
Updated serializer of FilePublishInformation so that only elements that should be part of publishing is included. Change-Id: If5632af8bf9fd26f8bc96b25d8bfbccf63dfec2f Issue-ID: DCAEGEN2-1488 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-04-16Bugfix, serializing in DataRouterPublisherPatrikBuhr1-1/+5
This fixes a bug introduced in c7b0c7014688c447d411f791a094440403e65228. The new property must be removed before publishing to data router. The consequense of the bug is that the DR rejects the publishing. Change-Id: Ib42cbed4decdba4c13678cea47dff35ca77679a5 Issue-ID: DCAEGEN2-1305 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-04-12Improved loggingPatrikBuhr6-28/+49
File name is used in Requext_ID to make it more easy to trouble shoot. Change-Id: Ied98766f1a177a9bda35ec5892a60d06619ff3a7 Issue-ID: DCAEGEN2-1305 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-04-11Merge "Robustness issue"Vijay Venkatesh Kumar1-0/+2
2019-04-11Robustness issuePatrikBuhr1-0/+2
Limiting the number of paralell threads when retying of ftp or publish. Previously there was no upper limit on how many paralel threads that could be started by retry attempts. For instance, a worker thread with 100 new files would start 100 new threads when the ftt server was unavailable. Change-Id: Ia8792f03ea55c0c467ef248ddc9d59187c06a946 Issue-ID: DCAEGEN2-1118 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-04-11Extend heartbeat to also include som health statuselinuxhenrik2-0/+118
Add info about the internal status of DFC to the healtCheck message. Change-Id: Ic2ad3e5b97c42d6affe1ac4e1d7cf2274027f881 Issue-ID: DCAEGEN2-1403 Signed-off-by: elinuxhenrik <henrik.b.andersson@est.tech>
2019-04-11Housekeepingelinuxhenrik25-442/+1322
No functional changes made in this commit. Removed CheckStyle and Sonar warnings. Formatted code. Renamed methods and classes for better understanding. Removed unnecessary classes. Moved all code to single project. Change-Id: Ie3feb6c6a985e94a382812aa083dcf57bc46c7b3 Issue-ID: DCAEGEN2-1367 Signed-off-by: elinuxhenrik <henrik.b.andersson@est.tech>
2019-04-10Updating certificatesTamasBakai3-13/+8
Issue-ID: DCAEGEN2-1300 Change-Id: I9ce457b69e5866e77a465fb040d196b8905ca274 Signed-off-by: TamasBakai <tamas.bakai@est.tech>
2019-04-04Improved loggingPatrikBuhr5-37/+12
Fixed problem with startup which lead to that the REST API was not working running locally. Fixed problem with DmaapProducerHttpClient which would use no timeout, which can lead to infinitly haninging threads. A long timeout is used instead. Change-Id: I28469b1b3aaad0dab4cf247bb8af968e71a60133 Issue-ID: DCAEGEN2-1305 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-03-29Change file name sent to DataRouterelinuxhenrik2-9/+5
In the previous commit for this issue, the file name sent to DataRouter was changed. This turned out to be wrong, so in this commit it is changed back again. Change-Id: I9b0d726f406758846fd256626e73edf59d710c8b Issue-ID: DCAEGEN2-1256 Signed-off-by: elinuxhenrik <henrik.b.andersson@est.tech>
2019-03-22Local filename updated, stability issuesPatrikBuhr6-53/+60
The local filename is changed so it contains PNF name instead of the PNF IP address. The paralellism is restricted to 100 worker threads in order to solve problems with too many open file descriptors and out of memory. Logging is improved. Change-Id: I24ce2e23020cc253a3c7bebac1ab5cf703b5b144 Issue-ID: DCAEGEN2-1118 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-03-21Add check to DataRouter if file has been publishedmaximesson8-480/+777
For each file in the FileReady message that DFC does not know if it has been published yet, it should ask DataRouter if it has been published already to avoid downloading and publishing a file more than once. Change-Id: I18117a6e968ec929aa255052a4c44f890a8ed39d Issue-ID: DCAEGEN2-1256 Signed-off-by: maximesson <maxime.bonneau@est.tech>
2019-03-14Thread safety issuesPatrikBuhr3-71/+56
The TrustManager is now loaded and initialized once in a thread safe way (instead of each time it is used). Removed some unneeded wrappers. Using AutoCloseable for FTP clients to make sure they are closed in case of exceptions. Made AppConfig thread safe. Change-Id: Ia6a2c8a76bf960013180fdd7c53ae0ff17b26505 Issue-ID: DCAEGEN2-1118 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-03-08DFC logging according to ONAP specificationRehanRaza4-45/+47
Change-Id: I6fe18ce3bdbc6d0b1cf5c5e65534cab694cfb898 Issue-ID: DCAEGEN2-1305 Signed-off-by: RehanRaza <muhammad.rehan.raza@est.tech>
2019-03-06Improvement of the parallelismPatrikBuhr2-29/+315
The reactive framework Scedulers uses to few threads. (the same number as the number of processors). That is too few for an io-intense application like this where CPU is not the limiting factor. Change-Id: Ia5f41e75716d309f47dce5f5273b739f7e6d136a Issue-ID: DCAEGEN2-1118 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-02-27Purging of cached informationPatrikBuhr1-0/+64
The datafile collector has a cache will all previously published files. The cache is on regular intevals purged so that non used entries are removed so that it does not grow infinitely. Added a unit test. Change-Id: I8897fee4522c97031f735b1d6774803dcb73926b Issue-ID: DCAEGEN2-1118 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-02-22Fixed some sonar issuesPatrikBuhr2-16/+21
Changed some Flux to Mono. Removed some obfuscating wrappers. Issue-ID: DCAEGEN2-1118 Change-Id: I76dcaea7c69608cf404389fad93f7539f735aad2 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-02-19Refactor: fix over engineering in ConfigurationYongchaoWu2-57/+56
Remove the duplicated function, move the over engineering code. Make the code simple and understandable. Issue-ID: DCAEGEN2-1222 Change-Id: Iab1737c084421d038b724d2fb41c9d17ebe4dbeb Signed-off-by: YongchaoWu <yongchao.wu@est.tech>
2019-02-15Merge "Add Readme.md"Tony Hansen1-14/+41
2019-02-15Running of file collection in paralellPatrikBuhr8-348/+462
Each FileReady message is run in a separate thread to increase the thoughput. Fetching of files from PNFs is retryed by using the reactive framework. Robustness to temporary failures is increased by retrying to publish fetched files. Fixed so that well known ports (FTPS/SFTP) are used if omitted in the FileReady message URL. Change-Id: I5dfc75a08da0e870fafa3ee1bc83574aca16aabd Issue-ID: DCAEGEN2-1118 Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
2019-02-13Add Readme.mdyanhuanwang1-14/+41
Format code Change-Id: Iba56718868043d3b19c71892f363fbb838f4988a Signed-off-by: Chengkai Yan <martin.c.yan@est.tech> Issue-ID: DCAEGEN2-1039
2019-01-30Fix for integration with SDKpwielebs2-2/+2
Change-Id: I9947531e2cad813ac8936769147bee710db5a335 Issue-ID: DCAEGEN2-1096 Signed-off-by: pwielebs <piotr.wielebski@nokia.com>
2019-01-25Integration with SDKpwielebs5-21/+62
* New fields added to DmaapPublisherConfiguraion * New fields added to DmaapConsumerConfiguration * Dmaap Consumer web client replaced by SDK's consumer web client * UTs aligned * Disable enforcer plugin in DFC * Update oparent to 1.2.1 Change-Id: I3569180c15227bc2c8df74fd070571b0aa56fa04 Issue-ID: DCAEGEN2-1096 Signed-off-by: pwielebs <piotr.wielebski@nokia.com>
2019-01-08use sdk-cbsmicdzied2-165/+0
Change-Id: I615d57927c163d022683ad3cae2703adda48cf88 Issue-ID: DCAEGEN2-1064 Signed-off-by: micdzied <michal.1.dziedzic@nokia.com>
2018-12-03Make DFC handle multiple messages from MRelinuxhenrik1-17/+125
DMaaP MessageRouter might send multiple messages in the same respons when DFC polls. This was not handled by DFC. Change-Id: I78c2fc7f4512a07fadf61c2cf1f6399d466fc873 Issue-ID: DCAEGEN2-1001 Signed-off-by: elinuxhenrik <henrik.b.andersson@est.tech>
2018-11-28Fix sonar issueselinuxhenrik4-55/+89
Change-Id: I4aff14b6afc5faaf95f28286dc6f2f741191e403 Issue-ID: DCAEGEN2-991 Signed-off-by: elinuxhenrik <henrik.b.andersson@est.tech>
2018-11-16Update meta-dataChengkai Yan5-177/+519
Add commonEventHeader to X-ATT-DR-META Format code with formatter:off/on Change-Id: I490620af127a03face48bf7b2ad3abf0bb80d4fa Signed-off-by: Chengkai Yan <martin.c.yan@est.tech> Issue-ID: DCAEGEN2-935
2018-10-30Add code to validate changeType & changeIdentifierChengkai Yan1-5/+45
Change-Id: Ib6731a7529ebb06588e17f91a986764f042b26b7 Signed-off-by: Chengkai Yan <martin.c.yan@est.tech> Issue-ID: DCAEGEN2-923
2018-10-09Fix retry when file download fails1.0.2elinuxhenrik1-14/+69
When the Datafile collector is unable to download the file from an xNF it now retries to collect the file. Change-Id: I61f68f9cf7af1a7fab160b0e936daafd1a23aaf8 Issue-ID: DCAEGEN2-864 Signed-off-by: elinuxhenrik <henrik.b.andersson@est.tech>
2018-10-03Fix mutual authenticationVijay Venkatesh Kumar6-262/+184
Last version from demo. Change-Id: Ib41581bf6f9eb92a03edf8434261d3674b6e3e39 Issue-ID: DCAEGEN2-860 Signed-off-by: elinuxhenrik <henrik.b.andersson@est.tech> Signed-off-by: Vijay Venkatesh Kumar <vv770d@att.com>