Age | Commit message (Collapse) | Author | Files | Lines |
|
In the long time stability test it has showed that when there are an exessive amount of FileReady events, the DFC will fetch these from the MR and build up an internal queue.
This has the effect that the DFC will consume too much memory. Itmight also affect load balancing (in case of several DFC instances) so that one might consume all events and the others get nothing.
The DFC should not fetch new FileReady events when it fully loaded with work.
Change-Id: I58665edd678d2f1c8a32d0e56455228b522aab40
Issue-ID: DCAEGEN2-1509
Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
|
|
Change-Id: Icdf00480eed43536a8b4693cbb208560f0881c95
Issue-ID: DCAEGEN2-1462
Signed-off-by: RehanRaza <muhammad.rehan.raza@est.tech>
|
|
|
|
Updated serializer of FilePublishInformation so that only elements that
should be part of publishing is included.
Change-Id: If5632af8bf9fd26f8bc96b25d8bfbccf63dfec2f
Issue-ID: DCAEGEN2-1488
Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
|
|
Change-Id: Ifede82e15ebc38af92ade8259a6e1f641946b815
Issue-ID: DCAEGEN2-1487
Signed-off-by: elinuxhenrik <henrik.b.andersson@est.tech>
|
|
|
|
Change-Id: Iec3ca75c42642da27c52bccd6d60811c434f778a
Issue-ID: DCAEGEN2-1466
Signed-off-by: TamasBakai <tamas.bakai@est.tech>
|
|
When retrying of file publish, the RequestID in the trace
context is incorect. The context must be expicitly updated
since retrying is executed in a separate thread.
Change-Id: I1c5dc0ba0e53c1219e23f93d8c5b420996a16c93
Issue-ID: DCAEGEN2-1462
Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
|
|
Change-Id: Ia036f7d79d25ceb2f1e465d79f17db1acbabbfbf
Issue-ID: DCAEGEN2-1443
Signed-off-by: TamasBakai <tamas.bakai@est.tech>
|
|
This fixes a bug introduced in c7b0c7014688c447d411f791a094440403e65228.
The new property must be removed before publishing to data router.
The consequense of the bug is that the DR rejects the publishing.
Change-Id: Ib42cbed4decdba4c13678cea47dff35ca77679a5
Issue-ID: DCAEGEN2-1305
Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
|
|
File name is used in Requext_ID to make it
more easy to trouble shoot.
Change-Id: Ied98766f1a177a9bda35ec5892a60d06619ff3a7
Issue-ID: DCAEGEN2-1305
Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
|
|
Fixed a build problem cause by a merge
error from previous commits.
Fixes warnings from maven compile
Change-Id: I1ce88b2686b3be5ecd17113cbfde3e65297c27c6
Issue-ID: DCAEGEN2-1118
Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
|
|
|
|
|
|
Limiting the number of paralell threads when
retying of ftp or publish.
Previously there was no upper limit on how many paralel
threads that could be started by retry attempts.
For instance, a worker thread with 100 new files
would start 100 new threads when the ftt server
was unavailable.
Change-Id: Ia8792f03ea55c0c467ef248ddc9d59187c06a946
Issue-ID: DCAEGEN2-1118
Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
|
|
This makes it possible by a REST API:
- change logging levels
- get the application log
- get build info for the DFC
- get various metrixes for the execution
such as heap consumption, number of threads etc.
Change-Id: I570bc0db6a9b9977ecfd83389b463fe652e4ba16
Issue-ID: DCAEGEN2-1428
Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
|
|
Add info about the internal status of DFC to the healtCheck message.
Change-Id: Ic2ad3e5b97c42d6affe1ac4e1d7cf2274027f881
Issue-ID: DCAEGEN2-1403
Signed-off-by: elinuxhenrik <henrik.b.andersson@est.tech>
|
|
No functional changes made in this commit.
Removed CheckStyle and Sonar warnings.
Formatted code.
Renamed methods and classes for better understanding.
Removed unnecessary classes.
Moved all code to single project.
Change-Id: Ie3feb6c6a985e94a382812aa083dcf57bc46c7b3
Issue-ID: DCAEGEN2-1367
Signed-off-by: elinuxhenrik <henrik.b.andersson@est.tech>
|
|
Issue-ID: DCAEGEN2-1300
Change-Id: I9ce457b69e5866e77a465fb040d196b8905ca274
Signed-off-by: TamasBakai <tamas.bakai@est.tech>
|
|
Fixed problem with startup which lead
to that the REST API was not working running
locally.
Fixed problem with DmaapProducerHttpClient which would use
no timeout, which can lead to infinitly haninging threads.
A long timeout is used instead.
Change-Id: I28469b1b3aaad0dab4cf247bb8af968e71a60133
Issue-ID: DCAEGEN2-1305
Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
|
|
In the previous commit for this issue, the file name sent to DataRouter
was changed. This turned out to be wrong, so in this commit it is
changed back again.
Change-Id: I9b0d726f406758846fd256626e73edf59d710c8b
Issue-ID: DCAEGEN2-1256
Signed-off-by: elinuxhenrik <henrik.b.andersson@est.tech>
|
|
|
|
Change-Id: I147b698c40bb80a09c885c16d08137dcb4a0cf69
Signed-off-by: Vijay Venkatesh Kumar <vv770d@att.com>
Issue-ID: DCAEGEN2-1270
Signed-off-by: Vijay Venkatesh Kumar <vv770d@att.com>
|
|
The local filename is changed so it contains PNF name
instead of the PNF IP address.
The paralellism is restricted to 100 worker threads in order to
solve problems with too many open file descriptors and out of memory.
Logging is improved.
Change-Id: I24ce2e23020cc253a3c7bebac1ab5cf703b5b144
Issue-ID: DCAEGEN2-1118
Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
|
|
For each file in the FileReady message that DFC does not know if it has been
published yet, it should ask DataRouter if it has been published already to
avoid downloading and publishing a file more than once.
Change-Id: I18117a6e968ec929aa255052a4c44f890a8ed39d
Issue-ID: DCAEGEN2-1256
Signed-off-by: maximesson <maxime.bonneau@est.tech>
|
|
Remove additional configuration in Blueprints
Remove additional configruation in component
specs
Change-Id: I0d546227af0be485f2ffed27a93c701a7654ff8e
Issue-ID: DCAEGEN2-1145
Signed-off-by: YongchaoWu <yongchao.wu@est.tech>
|
|
|
|
Blueprints alignment
Configbinding service support
Modify the policy model complaint to R4 model expected by Policy
Change-Id: Ic53f4ab2a5867e5cdf0fc735be180715077154ee
Issue-ID: DCAEGEN2-1145
Signed-off-by: YongchaoWu <yongchao.wu@est.tech>
|
|
The TrustManager is now loaded and initialized once
in a thread safe way (instead of each time it is used).
Removed some unneeded wrappers.
Using AutoCloseable for FTP clients to make
sure they are closed in case of exceptions.
Made AppConfig thread safe.
Change-Id: Ia6a2c8a76bf960013180fdd7c53ae0ff17b26505
Issue-ID: DCAEGEN2-1118
Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
|
|
|
|
|
|
Change-Id: I6fe18ce3bdbc6d0b1cf5c5e65534cab694cfb898
Issue-ID: DCAEGEN2-1305
Signed-off-by: RehanRaza <muhammad.rehan.raza@est.tech>
|
|
|
|
Change-Id: I5dc3c5b69b2ef764992e2db76eb9cc154ff5adf7
Issue-ID: DCAEGEN2-1269
Signed-off-by: RehanRaza <muhammad.rehan.raza@est.tech>
|
|
The reactive framework Scedulers uses to few threads.
(the same number as the number of processors).
That is too few for an io-intense application like this
where CPU is not the limiting factor.
Change-Id: Ia5f41e75716d309f47dce5f5273b739f7e6d136a
Issue-ID: DCAEGEN2-1118
Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
|
|
Change-Id: Id0389e1011fdce9dee4fc0d330973ef34bc2ed6a
Issue-ID: DCAEGEN2-1269
Signed-off-by: RehanRaza <muhammad.rehan.raza@est.tech>
|
|
|
|
Change-Id: Icf8a37ea63e78d70256571d1bce424e629a04b55
Issue-ID: DCAEGEN2-1269
Signed-off-by: RehanRaza <muhammad.rehan.raza@est.tech>
|
|
Add blueprints, dataformat, component specs,
policy and tosca models.
Issue-ID: DCAEGEN2-1156
Change-Id: Id1b6573c3cd72a90ab2feca2180ce1d04ecf98c4
Signed-off-by: YongchaoWu <yongchao.wu@est.tech>
|
|
The datafile collector has a cache will all previously published files.
The cache is on regular intevals purged so that non used entries
are removed so that it does not grow infinitely.
Added a unit test.
Change-Id: I8897fee4522c97031f735b1d6774803dcb73926b
Issue-ID: DCAEGEN2-1118
Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
|
|
|
|
Changed some Flux to Mono.
Removed some obfuscating wrappers.
Issue-ID: DCAEGEN2-1118
Change-Id: I76dcaea7c69608cf404389fad93f7539f735aad2
Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
|
|
Change-Id: I82504ebebc8ddf48f6ee6b1c1dfbe433c60ab1aa
Issue-ID: DCAEGEN2-1269
Signed-off-by: RehanRaza <muhammad.rehan.raza@est.tech>
|
|
Remove the duplicated function, move the over
engineering code. Make the code simple and
understandable.
Issue-ID: DCAEGEN2-1222
Change-Id: Iab1737c084421d038b724d2fb41c9d17ebe4dbeb
Signed-off-by: YongchaoWu <yongchao.wu@est.tech>
|
|
Update DFC description
Change-Id: I778014f3e0ec215c5ce761d087e28bcf06b7296c
Signed-off-by: yanhuanwang <martin.yan.seu@gmail.com>
Issue-ID: DCAEGEN2-1039
|
|
|
|
|
|
Each FileReady message is run in a separate thread to increase the
thoughput.
Fetching of files from PNFs is retryed by using the reactive
framework.
Robustness to temporary failures is increased by retrying to publish
fetched files.
Fixed so that well known ports (FTPS/SFTP) are used if omitted in the
FileReady message URL.
Change-Id: I5dfc75a08da0e870fafa3ee1bc83574aca16aabd
Issue-ID: DCAEGEN2-1118
Signed-off-by: PatrikBuhr <patrik.buhr@est.tech>
|
|
Change-Id: I1ffb840a18b0c75cffe21229db743f6570073148
Issue-ID: DCAEGEN2-1210
Signed-off-by: elinuxhenrik <henrik.b.andersson@est.tech>
|
|
Format code
Change-Id: Iba56718868043d3b19c71892f363fbb838f4988a
Signed-off-by: Chengkai Yan <martin.c.yan@est.tech>
Issue-ID: DCAEGEN2-1039
|