aboutsummaryrefslogtreecommitdiffstats
path: root/sources
AgeCommit message (Collapse)AuthorFilesLines
2019-06-26Add environment configuration to kafka consumerFilip Krzywka6-125/+174
- HV-VES-specific environment prefix moved inside HvVes modules to allow simpler no-prefix API for other modules - created OptionDSL for brevity Change-Id: I2fabbda1280cc0f913f8a0a04b4a055f39ed1fae Issue-ID: DCAEGEN2-1626 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-06-25Implement message counting in consumerkjaniak19-54/+468
Issue-ID: DCAEGEN2-1635 Change-Id: I2666de7bad27052d9cefa0f687ad0772d4c9a95d Signed-off-by: kjaniak <kornel.janiak@nokia.com>
2019-06-25Merge "Expose Prometheus metrics in Kafka Consumer"Piotr Jaszczyk8-4/+210
2019-06-19Expose Prometheus metrics in Kafka ConsumerJakub Dudycz8-4/+210
Change-Id: I9d0568a5fc296604d7ac7a45b8bbd0289b845938 Signed-off-by: Jakub Dudycz <jakub.dudycz@nokia.com> Issue-ID: DCAEGEN2-1626
2019-06-18Fix request intervalFilip Krzywka2-11/+8
In previous implementation DistinctUntilChangedSubscriber always requested from upstream 256 events, which resulted in immediate 256 requests to CBS. Request amount is not configurable in other way than hard-limiting using `limitRequest`, which limits request amount for single subscriber. (At least in our pipeline) To avoid multiple manual subscribes, this commit changed CbsClientAdapter to use Mono instead of Flux for CbsRequests and repeat this Mono conditionally. Flux inside of repeatWhen is emitting event after each onComplete received from upstream Mono and resubscribes to it if condition is met. This seemed like good place to put our interval mechanism, which is always-pass condition, but condition resolving blocks for variable duration. Change-Id: I04d1e657ec4d82185f6f07422c25c2d2ff23e60d Issue-ID: DCAEGEN2-1557 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-06-14Create Kafka Consumer project stubJakub Dudycz5-0/+166
Change-Id: Ib45ca6a9ab282dab509e34f9a3d3682373d794d3 Signed-off-by: Jakub Dudycz <jakub.dudycz@nokia.com> Issue-ID: DCAEGEN2-1619
2019-06-12Retry infinitelyFilip Krzywka9-104/+429
- changed specification to retry infinitely - moved MDC to cbsAdapter constructor as in whole module it contains only local context (instanceID etc.). Also permuted constructor params to match ConfigurationProviders order - refactored module tests as ground for future enhancements Change-Id: Ic074b9c421b60662e5512c55c7b1dfb90ab0d2ea Issue-ID: DCAEGEN2-1557 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-06-07Run container commands as non-rootFilip Krzywka1-0/+7
- 1410 uig/gid is arbitrary, note however that any volume mounted into container should limit permissions to this number - also reduced firstRequestDelay in configuration provided through local docker-compose Change-Id: I77eeebeec5931db5c04f8f6f24d7c6fe7a121015 Issue-ID: DCAEGEN2-1557 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-06-05Support CBS request interval reconfigurationkjaniak8-91/+297
Change-Id: Ie8892e33b2f6a58d6076f66e6cc6a2df830dfa48 Issue-ID: DCAEGEN2-1525 Signed-off-by: kjaniak <kornel.janiak@nokia.com>
2019-05-10Enable Kafka consumer offset committingFilip Krzywka1-5/+6
It appears that reactor-kafka is setting auto.commit property to false, which makes our CSITs fail nondeterministically due to automatic reset of consumer offset. By acknowledging manually, we will mark every message for committing, which will be performed according to ConsumerConfiguration. This way, Kafka broker should persist consumer offset. Change-Id: I0c5156ff8df9bb3341e733e50a3c6866fdd94976 Issue-ID: DCAEGEN2-1495 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-05-06Update versions for El Alto releasePiotr Jaszczyk15-16/+16
Change-Id: If86dcbf8695cc50a540fe892a4f70b3ad24ac5d1 Issue-ID: DCAEGEN2-1474 Signed-off-by: Piotr Jaszczyk <piotr.jaszczyk@nokia.com>
2019-04-24Use AAF credentials from stream definitionPiotr Jaszczyk7-32/+225
Change-Id: I4fc20c116c60f6e7d46215a32c33884cd957e93b Issue-ID: DCAEGEN2-1448 Signed-off-by: Piotr Jaszczyk <piotr.jaszczyk@nokia.com>
2019-04-24Use SASL auth in kafka connectionsFilip Krzywka2-1/+29
Change-Id: I55a9289901a6a44f3d07a3cf4e5a028399a5d0dc Issue-ID: DCAEGEN2-1448 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-04-16Add internal modifiers to comply with new HvVesCustomRuleIzabela Zawadzka21-35/+35
Change-Id: I33ad24179d05ded58f6784544be9bbdd56210720 Signed-off-by: Izabela Zawadzka <izabela.zawadzka@nokia.com> Issue-ID: DCAEGEN2-1438
2019-04-12Move cmd ssl logic to xnf-simulatorJakub Dudycz14-35/+14
Change-Id: I562acbb0b7522e6cefea2a2dc1a9e292c1227b41 Signed-off-by: Jakub Dudycz <jakub.dudycz@nokia.com> Issue-ID: DCAEGEN2-565
2019-04-12Creation of server modulekjaniak35-195/+335
Issue-ID: DCAEGEN2-1390 Change-Id: I07410b16ed6566b933d5f1efa35bddb965225794 Signed-off-by: kjaniak <kornel.janiak@nokia.com> Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-04-12Merge "Extract transforming logic from validator"Piotr Jaszczyk17-350/+700
2019-04-12Merge "Fix xnf-simulator message sending bug"Piotr Jaszczyk2-8/+10
2019-04-12Extract transforming logic from validatorFilip Krzywka17-350/+700
Change-Id: Ic019b1796e17d24f14f41a817af6e5ecd8c7244b Issue-ID: DCAEGEN2-1416 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-04-11Fix xnf-simulator message sending bugJakub Dudycz2-8/+10
Fix bug where xnf-simulator was failing to send messages with invalid wire frame. Generated events couldn't pass HV-VES client validatation, because they had empty payload. Change-Id: I20fb6254e432de096acfcfb8f0a0a5f912ca08c0 Signed-off-by: Jakub Dudycz <jakub.dudycz@nokia.com> Issue-ID: DCAEGEN2-1413
2019-04-10Read passwords from filesPiotr Jaszczyk13-62/+81
Key- and trust-store passwords should be read from files in order to work with DCAE tls-init-container. Change-Id: Ibe454663328268f33f8be25ef9ec129f1ce1d396 Issue-ID: DCAEGEN2-1412 Signed-off-by: Piotr Jaszczyk <piotr.jaszczyk@nokia.com>
2019-04-10Merge "Move maxPayloadSizeBytes to CollectorConfiguration"Piotr Jaszczyk20-44/+116
2019-04-10Move maxPayloadSizeBytes to CollectorConfigurationJakub Dudycz20-44/+116
Signed-off-by: Jakub Dudycz <jakub.dudycz@nokia.com> Issue-ID: DCAEGEN2-1387 Change-Id: I37151a58f5244841243dc531912af2ef50ea5d3c
2019-04-10Allow retrieving multiple kafka topics statusFilip Krzywka8-127/+223
Change-Id: I5e8433873e5d594e6df9da8c4893b0f54614efae Issue-ID: DCAEGEN2-1399 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-04-10Merge "Parse whole dynamic configuration"Piotr Jaszczyk14-232/+200
2019-04-09Parse whole dynamic configurationJakub Dudycz14-232/+200
Change-Id: I96e4cf3ac75920ed909da9063ba0b788b55474e4 Signed-off-by: Jakub Dudycz <jakub.dudycz@nokia.com> Issue-ID: DCAEGEN2-1386
2019-04-09Provide enhanced component descriptionPiotr Jaszczyk1-2/+6
Change-Id: Iaed53f9acc9cdc830b2069de6d96d5f2965d9466 Issue-ID: DCAEGEN2-1394 Signed-off-by: Piotr Jaszczyk <piotr.jaszczyk@nokia.com>
2019-04-08Flatten configuration structureFilip Krzywka13-457/+230
Change-Id: I9a5a53f3c4cf0973d071f2bc6962016b5613972d Issue-ID: DCAEGEN2-710 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-04-04Get rid of arrow-effects usagePiotr Jaszczyk33-289/+166
Also clean-up dependencies + use Kotlin BOM to force single kotlin-stdlib on classpath. Issue-ID: DCAEGEN2-1392 Change-Id: I447c4686707de81f35f7734255ce0b13c997c4a4 Signed-off-by: Piotr Jaszczyk <piotr.jaszczyk@nokia.com>
2019-04-03Simplify factory/provider logicPiotr Jaszczyk10-55/+40
Change-Id: I59467c41e1de63ead7c190a7c8fd688e3216237a Issue-ID: DCAEGEN2-1385 Signed-off-by: Piotr Jaszczyk <piotr.jaszczyk@nokia.com>
2019-04-03Fix shutting down when new config received bugPiotr Jaszczyk20-113/+131
When new configuration has been received and at least one client connection has been active the collector used to shut down. Also got rid of some more IO monad usage. Change-Id: I7981ff388ff1264a79d722727ef3005cf39e9f0d Issue-ID: DCAEGEN2-1382 Signed-off-by: Piotr Jaszczyk <piotr.jaszczyk@nokia.com>
2019-04-02Allow reuse of security passwordsFilip Krzywka6-28/+87
- moved logic of creating passwords into Validator Change-Id: Ieabab24734250bfacea96eff2b7102465ff8ed57 Issue-ID: DCAEGEN2-1380 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-04-02Merge configurationsFilip Krzywka19-145/+122
- changed temporarily HV-VES default log level to DEBUG as in current implementation we are applying LogLevel defined in configuration file only if we successfully retrieve one from configuration-module, which means that inside of this module we are logging on default level (from logback file). This should be fixed in future work - reduced log level on SDK's CbsClientImpl as it's logging frequency was too high Change-Id: If50df18df099c34bfc36d39b045140f9b9ad87f6 Issue-ID: DCAEGEN2-1347 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-04-01Merge "Move ConfigurationProvider to config module"Piotr Wielebski23-395/+159
2019-04-01Move ConfigurationProvider to config modulePiotr Jaszczyk23-395/+159
Change-Id: Ic6f955f4e777e06e7c7eed6e08c0cac470e9a51d Issue-ID: DCAEGEN2-1347 Signed-off-by: Piotr Jaszczyk <piotr.jaszczyk@nokia.com>
2019-03-29Fix subscription error in DcaeAppSimulatorIzabela Zawadzka1-1/+1
Change-Id: I6548d7e6c676498988ddcdde4331385e87ee746d Signed-off-by: Izabela Zawadzka <izabela.zawadzka@nokia.com> Issue-ID: DCAEGEN2-1372
2019-03-29Read HV-VES healthcheck api port from cmd linekjaniak7-56/+136
Change-Id: I6b4680a62512ef6ba15a0454e109b19619a997a6 Issue-ID: DCAEGEN2-1364 Signed-off-by: kjaniak <kornel.janiak@nokia.com>
2019-03-29Merge "Remove IO monad usage from simulators"Piotr Jaszczyk10-179/+148
2019-03-28Redefine RoutingFilip Krzywka27-445/+504
As all needed information to route messege is contained inside of KafkaSink message, we can simply put this object as part of single Route. Change-Id: I2e7df2e0193eb2af5283980d4d5c8df03ac94df9 Issue-ID: DCAEGEN2-1347 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-03-28Remove IO monad usage from simulatorsIzabela Zawadzka10-179/+148
Change-Id: I1c470777b91230f4a44a4960ca534e4b20c1ac43 Issue-ID: DCAEGEN2-1372 Signed-off-by: Izabela Zawadzka <izabela.zawadzka@nokia.com>
2019-03-28Added partial config merger and updated config file formatpbiniek11-60/+298
Change-Id: I470d3a8b2d43f5b45d958320b620205457d1cf23 Issue-ID: DCAEGEN2-1370 Signed-off-by: Pawel Biniek <pawel.biniek@nokia.com>
2019-03-27Move routing functionality inside RouterFilip Krzywka7-95/+37
- also removed Routing-DSL as it won't be needed anymore Change-Id: Ifc7bc7641a60936b5257c0bff7a8c51dddc30687 Issue-ID: DCAEGEN2-1347 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-03-27Remove routing from configuration fileFilip Krzywka10-324/+57
Routing should only be available from CBS. Change-Id: Idfada36dcda4192d84a5b601907d27fe0db942ce Issue-ID: DCAEGEN2-1347 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-03-27Remove dummyMode configuration optionFilip Krzywka12-93/+14
As it's not really used by anyone and it's functionality does not bring much value Change-Id: I99b07e484a8494a036f1f1b07e21666e044edbdb Issue-ID: DCAEGEN2-1347 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-03-27Update SDK usage to comply with new APIFilip Krzywka7-20/+19
Change-Id: Ifa5847617b7cadaadd80a82a3b279ac2beb4b258 Issue-ID: DCAEGEN2-1347 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-03-26Use DataStream API from CBS clientkjaniak11-140/+176
Change-Id: Ief92f793282288938c6663616e9613c6df2d8ddb Issue-ID: DCAEGEN2-1346 Signed-off-by: kjaniak <kornel.janiak@nokia.com>
2019-03-22Use updated CBS Client APIPiotr Jaszczyk1-2/+2
Change-Id: I476900dd81d6b23e3b35a3f93c71742efdca7a6e Issue-ID: DCAEGEN2-1363 Signed-off-by: Piotr Jaszczyk <piotr.jaszczyk@nokia.com>
2019-03-21Make security keys optional with SSL disabledFilip Krzywka7-74/+300
Change-Id: I0b0dd63cb3609cddf6aa5e63cf0a0f147a0aca91 Issue-ID: DCAEGEN2-1340 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-03-21Include basic configuration in Docker imagePiotr Jaszczyk8-4/+103
Makes the collector start out of the box, even without valid CBS configuration. Note: the basic configuration will need to be changed later. Issue-ID: DCAEGEN2-1340 Change-Id: I3c2b502f253677e78a4f4990002e683613b8869e Signed-off-by: Piotr Jaszczyk <piotr.jaszczyk@nokia.com>
2019-03-20Remove environment variables and program argumentsJakub Dudycz46-868/+914
- Move all command line program arguments to json file. - Reorganize configuration classes and the way they are passed through application - Implement HV VES configuration stream - Create concrete configuration from partial one - Modify main HV-VES server starting pipeline Change-Id: I6cf874b6904ed768e4820b8132f5f760299c929e Signed-off-by: Jakub Dudycz <jakub.dudycz@nokia.com> Issue-ID: DCAEGEN2-1340