aboutsummaryrefslogtreecommitdiffstats
path: root/sources
AgeCommit message (Collapse)AuthorFilesLines
2021-04-29Fix hv-ves configuration loading from envsBartosz Gardziejewski1-1/+2
Signed-off-by: Bartosz Gardziejewski <bartosz.gardziejewski@nokia.com> Change-Id: I53c4cec9803cc2f17a8be92e159b5d64130b83af Issue-ID: DCAEGEN2-2749
2021-04-22Change openjdk baseOS img to integration-java11Alexander Mazuruk22-67/+63
Benefits from switching over: * minimal {java11,python} images maintained by integration team * using currently "blessed by seccom" versions (:latest tag used) * should limit spread of legal issues across layers * integration images will be the first to have automated compliance documentation * should limit spread of base layers (contributing to deployment footprint - more base layers = more to download, more to store etc...) Issue-ID: INT-1864 Issue-ID: DCAEGEN2-2420 Signed-off-by: Alexander Mazuruk <a.mazuruk@samsung.com> Change-Id: Ibd6f379b869306c5f14f06f6efe60c5a3c32f779
2021-04-01Add stndDefined domain to HV-VES1.8.0Michal Banka22-77/+145
- Updated SDK dependency to 1.8.2 so HV-VES recognizes stndDefinedNamespace field during events deserialization - Added new domain stndDefined - Added routings specific for stndDefined events which is controled by stndDefinedNamespace field, not domain - Bumped HV-VES to 1.8.0 Signed-off-by: Michal Banka <michal.banka@nokia.com> Change-Id: Ia27f2de7a4b64df5e8a182888006f7e8a941ca63 Issue-ID: DCAEGEN2-2702
2021-03-30Fix unstable test1.7.0Tomasz Pietruszkiewicz1-2/+2
Change-Id: Ie77e5f64a6c4ca9c21300da1eef19ad5d9f5c393 Issue-ID: DCAEGEN2-2696 Signed-off-by: Tomasz Pietruszkiewicz <tomasz.pietruszkiewicz@nokia.com>
2021-03-29Update all HV-VES dependencies to newest versionsTomasz Pietruszkiewicz20-23/+27
Change-Id: Iba0d2dce0374a3fe44f0b6e3b302b02e782d8d56 Issue-ID: DCAEGEN2-2696 Signed-off-by: Tomasz Pietruszkiewicz <tomasz.pietruszkiewicz@nokia.com>
2020-12-28Upgrade hv-ves, reactor, protobuf and sdk versionsPawel25-33/+37
Reactor from Dysprosium-SR11 to 2020.0.2 Protobuf from 3.6.1 to 3.14.0 SDK from 1.4.4 to 1.5.0 Issue-ID: DCAEGEN2-2537 Signed-off-by: Pawel <pawel.kasperkiewicz@nokia.com> Change-Id: Ie3fc82c67ba096efcd0c584cf3f7f35128962efe
2020-11-30Update DCAE-SDK version to 1.4.4Remigiusz Janeczek17-18/+18
Allow HV-VES to recover after exception from CbsClient Issue-ID: DCAEGEN2-2516 Signed-off-by: Remigiusz Janeczek <remigiusz.janeczek@nokia.com> Change-Id: I9dd3c5b5479c05e8e366c7bfbbb6062122986fa1
2020-05-29Add micrometer metricsPiotr Marcinkiewicz1-0/+2
Add file descriptor metrics Issue-ID: DCAEGEN2-1576 Change-Id: I7221b8d671e9ee8e022b6b7b6fd9b908f18b78be Signed-off-by: Piotr Marcinkiewicz <piotr.marcinkiewicz@nokia.com>
2020-04-30Make offset consumer more genericJoanna Jeremicz4-22/+51
Offsets are now correctly read despite of partitions number on topic. Issue-ID: DCAEGEN2-1576 Change-Id: Id89bebbcf1b6181a2f9fd5a4a4600329d046c5e5 Signed-off-by: Joanna Jeremicz <joanna.jeremicz@nokia.com>
2020-04-23Add metric for total latency without routingRemigiusz Janeczek3-34/+74
Add metric for time between Producer and HV-VES output without sending to Kafka Refactor metrics test Add new latencies and individual cores usage to Grafana Issue-ID: DCAEGEN2-1576 Signed-off-by: Remigiusz Janeczek <remigiusz.janeczek@nokia.com> Change-Id: I6112db76be1c7108c18336b50f9f12d5ce62c24a
2020-04-22Add of message travel time metrickjaniak5-15/+57
Message travel time: Producer -> HV-VES input introduced. Tests for new metric added. Change-Id: I36347ff53abb3f274e4358af26db49fe8bac95ed Issue-ID: DCAEGEN2-1576 Signed-off-by: kjaniak <kornel.janiak@nokia.com>
2020-04-22Top up version to 1.5.0kjaniak17-18/+18
Because of Frankfurt branching done on 1.4.1 (1.4.0 was released) minor version need to be top up. Issue-ID: DCAEGEN2-1576 Change-Id: I92f6ab3fc565ec22b5c8cdd1050e2b9e12446153 Signed-off-by: kjaniak <kornel.janiak@nokia.com>
2020-02-13Top up oparent versionTomasz Wrobel1-2/+2
Issue-ID: DCAEGEN2-2014 Signed-off-by: Tomasz Wrobel <tomasz.wrobel@nokia.com> Change-Id: I4cd32f48b7b9f1f572591ac1dc4bed091a99adde
2020-01-29Patch version increased (1.4.0 -> 1.4.1)Joanna Jeremicz17-18/+18
Issue-ID: DCAEGEN2-1982 Signed-off-by: Joanna Jeremicz <joanna.jeremicz@nokia.com> Change-Id: I1c26cd3384ac443984cfbfed9b0bed722e1b31eb
2020-01-15Change hv-ves version after last releasekjaniak17-18/+18
Since this version will include java 11 minor version top up. Change-Id: Ie45f1a6afc33c2199d0a3f4d048420769158a6e6 Issue-ID: DCAEGEN2-1982 Signed-off-by: kjaniak <kornel.janiak@nokia.com>
2019-12-10Prepare release 1.3.1Joanna Jeremicz17-18/+18
Issue-ID: DCAEGEN2-1823 Signed-off-by: Joanna Jeremicz <joanna.jeremicz@nokia.com> Change-Id: Ifb02692ac982f28a916fa0e70548216e891216fe
2019-11-22Fix HV-VES static code vulnerabilitiesJoanna Jeremicz1-2/+2
- Update kafka-clients version - Update SDK version - Update reactor-bom version Issue-ID: DCAEGEN2-1823 Signed-off-by: Joanna Jeremicz <joanna.jeremicz@nokia.com> Change-Id: Ib2d49c46e2d1b58d5131b9af6fc27804d443da71
2019-11-07Fix for 30s bucket problem in metricskjaniak1-2/+3
Change-Id: I9a6b362f59a47454cb36d9a54861d1bcc1b934be Issue-ID: DCAEGEN2-1890 Signed-off-by: kjaniak <kornel.janiak@nokia.com>
2019-10-30Add metric for processing without routingkjaniak6-9/+58
Performance tests need better check of processing time in HV-VES. Change-Id: I0792c4ac014a7b8907ef314a3fd9981776dc0b35 Issue-ID: DCAEGEN2-1890 Signed-off-by: kjaniak <kornel.janiak@nokia.com>
2019-10-23Fix performance test issuesRemigiusz Janeczek2-0/+5
Issue-ID: DCAEGEN2-1783 Signed-off-by: Remigiusz Janeczek <remigiusz.janeczek@nokia.com> Change-Id: I367d109fc5903cd3fd6dbfeeda77d30d780a8a09
2019-10-16Move certificates directory creation to dockerfileRemigiusz Janeczek1-0/+2
Issue-ID: DCAEGEN2-1783 Signed-off-by: Remigiusz Janeczek <remigiusz.janeczek@nokia.com> Change-Id: If4d9c0dab255ce4f2024f06da40c724dd98d7e5f
2019-10-08Add graphs for performance monitoringkjaniak3-11/+16
Performance tests need graphs representing: total, message queue on HV-VES, offset per kafka topic partition, incoming data rate. Added deletion of producers at test end. Gauges for offsets turned to Range vector (distinguishable based on tag). Since Producer is to fast it was reconfigured to send messages with 1 ms interval. Change-Id: I09fc1edbcd824294d0bb89acc2ea2306a6df00ca Issue-ID: DCAEGEN2-1807 Signed-off-by: kjaniak <kornel.janiak@nokia.com>
2019-09-26Add partition offset metric to each topic partitionkjaniak5-26/+46
Before this commit offset consumer was able to fetch offset just from one partition. This commit solve this. Change-Id: I2c2c300219e43ab422b237094ad775ca8795169e Issue-ID: DCAEGEN2-1783 Signed-off-by: kjaniak <kornel.janiak@nokia.com>
2019-07-12Use consumers in mainIzabela Zawadzka4-9/+65
It includes --disable-processing flag. Also fixed some issues with script for local performance test. Also added KafkaConsumer::poll in OffsetKafka Consumer - without it KafkaConsumer::assignment returns empty set Signed-off-by: Izabela Zawadzka <izabela.zawadzka@nokia.com> Issue-ID: DCAEGEN2-1657 Change-Id: I95fadb45f321398346094dfa0c4a6e9da954c186
2019-07-08Merge "Measure message travel time"Piotr Jaszczyk8-156/+293
2019-07-05Increase coverage in new consumerFilip Krzywka1-0/+140
Change-Id: I58e88182e84a7d807d481e8cbd7196d8b82309ac Issue-ID: DCAEGEN2-1619 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-07-05Merge "Add performance test producer deployment yaml"Piotr Jaszczyk1-111/+0
2019-07-05Add performance test producer deployment yamlJakub Dudycz1-111/+0
- create cloud performance test directory structure - move consumer deployment file to "tools/performence/cloud" directory Change-Id: Ia5c6d8ed15a2188ff9306cf788052f432dae1e33 Signed-off-by: Jakub Dudycz <jakub.dudycz@nokia.com> Issue-ID: DCAEGEN2-1652
2019-07-04Measure message travel timeIzabela Zawadzka8-156/+293
Issue-ID: DCAEGEN2-1654 Signed-off-by: Izabela Zawadzka <izabela.zawadzka@nokia.com> Change-Id: Ifd6597209c5be51d5b4ff5faf7b3b1b1a2871403
2019-07-04Merge "Change handling of offset metric for consumer app"Piotr Jaszczyk5-15/+26
2019-07-03Merge "Add K8S deployment for kafka-consumer"Piotr Jaszczyk1-0/+111
2019-07-03Change handling of offset metric for consumer appkjaniak5-15/+26
Change-Id: Ia83469783d64d0174624489df671006e72bc8948 Issue-ID: DCAEGEN2-1635 Signed-off-by: kjaniak <kornel.janiak@nokia.com>
2019-07-02Reorganize kafka modulekjaniak18-196/+486
Change-Id: I2eb7a8a6e92c9d89586b877f4cae438497b62ae2 Issue-ID: DCAEGEN2-1635 Signed-off-by: kjaniak <kornel.janiak@nokia.com>
2019-06-28Add K8S deployment for kafka-consumerFilip Krzywka1-0/+111
Change-Id: Idba5c738dc2062b30c32bb928d701614743f0828 Issue-ID: DCAEGEN2-1619 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-06-27JJB alignment across DCAE components - DCAE HV-VESrajendrajaiswal17-18/+18
Change-Id: I815c32a91a3e710d9076495dbafcba420bde4da5 Issue-ID: DCAEGEN2-1578 Signed-off-by: rajendrajaiswal <rajendra.jaiswal@ericsson.com>
2019-06-26Implement kafka consumer metricsFilip Krzywka8-104/+199
- bump Micrometer version 1.0.8 -> 1.1.5 Change-Id: I7a00fbf252df0f31f12f8743e8719701bd282ce2 Issue-ID: DCAEGEN2-1626 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-06-26Add environment configuration to kafka consumerFilip Krzywka6-125/+174
- HV-VES-specific environment prefix moved inside HvVes modules to allow simpler no-prefix API for other modules - created OptionDSL for brevity Change-Id: I2fabbda1280cc0f913f8a0a04b4a055f39ed1fae Issue-ID: DCAEGEN2-1626 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-06-25Implement message counting in consumerkjaniak19-54/+468
Issue-ID: DCAEGEN2-1635 Change-Id: I2666de7bad27052d9cefa0f687ad0772d4c9a95d Signed-off-by: kjaniak <kornel.janiak@nokia.com>
2019-06-25Merge "Expose Prometheus metrics in Kafka Consumer"Piotr Jaszczyk8-4/+210
2019-06-19Expose Prometheus metrics in Kafka ConsumerJakub Dudycz8-4/+210
Change-Id: I9d0568a5fc296604d7ac7a45b8bbd0289b845938 Signed-off-by: Jakub Dudycz <jakub.dudycz@nokia.com> Issue-ID: DCAEGEN2-1626
2019-06-18Fix request intervalFilip Krzywka2-11/+8
In previous implementation DistinctUntilChangedSubscriber always requested from upstream 256 events, which resulted in immediate 256 requests to CBS. Request amount is not configurable in other way than hard-limiting using `limitRequest`, which limits request amount for single subscriber. (At least in our pipeline) To avoid multiple manual subscribes, this commit changed CbsClientAdapter to use Mono instead of Flux for CbsRequests and repeat this Mono conditionally. Flux inside of repeatWhen is emitting event after each onComplete received from upstream Mono and resubscribes to it if condition is met. This seemed like good place to put our interval mechanism, which is always-pass condition, but condition resolving blocks for variable duration. Change-Id: I04d1e657ec4d82185f6f07422c25c2d2ff23e60d Issue-ID: DCAEGEN2-1557 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-06-14Create Kafka Consumer project stubJakub Dudycz5-0/+166
Change-Id: Ib45ca6a9ab282dab509e34f9a3d3682373d794d3 Signed-off-by: Jakub Dudycz <jakub.dudycz@nokia.com> Issue-ID: DCAEGEN2-1619
2019-06-12Retry infinitelyFilip Krzywka9-104/+429
- changed specification to retry infinitely - moved MDC to cbsAdapter constructor as in whole module it contains only local context (instanceID etc.). Also permuted constructor params to match ConfigurationProviders order - refactored module tests as ground for future enhancements Change-Id: Ic074b9c421b60662e5512c55c7b1dfb90ab0d2ea Issue-ID: DCAEGEN2-1557 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-06-07Run container commands as non-rootFilip Krzywka1-0/+7
- 1410 uig/gid is arbitrary, note however that any volume mounted into container should limit permissions to this number - also reduced firstRequestDelay in configuration provided through local docker-compose Change-Id: I77eeebeec5931db5c04f8f6f24d7c6fe7a121015 Issue-ID: DCAEGEN2-1557 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-06-05Support CBS request interval reconfigurationkjaniak8-91/+297
Change-Id: Ie8892e33b2f6a58d6076f66e6cc6a2df830dfa48 Issue-ID: DCAEGEN2-1525 Signed-off-by: kjaniak <kornel.janiak@nokia.com>
2019-05-10Enable Kafka consumer offset committingFilip Krzywka1-5/+6
It appears that reactor-kafka is setting auto.commit property to false, which makes our CSITs fail nondeterministically due to automatic reset of consumer offset. By acknowledging manually, we will mark every message for committing, which will be performed according to ConsumerConfiguration. This way, Kafka broker should persist consumer offset. Change-Id: I0c5156ff8df9bb3341e733e50a3c6866fdd94976 Issue-ID: DCAEGEN2-1495 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-05-06Update versions for El Alto releasePiotr Jaszczyk15-16/+16
Change-Id: If86dcbf8695cc50a540fe892a4f70b3ad24ac5d1 Issue-ID: DCAEGEN2-1474 Signed-off-by: Piotr Jaszczyk <piotr.jaszczyk@nokia.com>
2019-04-24Use AAF credentials from stream definitionPiotr Jaszczyk7-32/+225
Change-Id: I4fc20c116c60f6e7d46215a32c33884cd957e93b Issue-ID: DCAEGEN2-1448 Signed-off-by: Piotr Jaszczyk <piotr.jaszczyk@nokia.com>
2019-04-24Use SASL auth in kafka connectionsFilip Krzywka2-1/+29
Change-Id: I55a9289901a6a44f3d07a3cf4e5a028399a5d0dc Issue-ID: DCAEGEN2-1448 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
2019-04-16Add internal modifiers to comply with new HvVesCustomRuleIzabela Zawadzka21-35/+35
Change-Id: I33ad24179d05ded58f6784544be9bbdd56210720 Signed-off-by: Izabela Zawadzka <izabela.zawadzka@nokia.com> Issue-ID: DCAEGEN2-1438