aboutsummaryrefslogtreecommitdiffstats
path: root/k6-tests
AgeCommit message (Collapse)AuthorFilesLines
2024-11-22Define new Suite (js): Endurancehalil.cakal6-139/+218
- have one ncmp-test-runner.js with different configs: kpi.json and endurance.json - move scenarios and thresholds settings into the json configs Issue-ID: CPS-2444 Change-Id: I72f65036dc3f7762284b32903e1d05474dbed6c2 Signed-off-by: halil.cakal <halil.cakal@est.tech>
2024-11-18Add new k6 test profile for running edurance testshalil.cakal9-38/+86
- assign a profile name to the existing k6 tests (kpi) - add control which profile is run using k6 config file - both test profiles can run in parallel Issue-ID: CPS-2464 Change-Id: I9fea13f12e2da46bd55b4315c68209843c1abe06 Signed-off-by: halil.cakal <halil.cakal@est.tech>
2024-11-15[k6] Lower load for legacy async batch operation testdanielhanrahan2-3/+2
Legacy async batch operation test is failing and reporting zero as result due to timeouts in the test. - Lower total batch requests to 100 instead of 1000 - Limit the number of kafka messages per batch to 200 instead of 1000 to avoid timeout issues. Issue-ID: CPS-2274 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I06c04173ca1ed0f3580df6a73b5d647ab7ed0c71
2024-11-13[k6] Output git build info in k6 logdanielhanrahan1-1/+6
- output build info in k6 log using actuator endpoint - use quiet output while pulling images for less log noise Issue-ID: CPS-2445 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I4464b70650828c842c94677dddcc4d112f9758e0
2024-11-04Cm Avc Event to have same keympriyank1-1/+7
- incoming Cm Avc Event from DMI Plugin is consumed and forwarded to target topic - the key from source topic to be used in the target topic while forwarding - with same key the ordering of the message will be preserved - NOTE: the RTD related changes will be a separate patchset Issue-ID: CPS-2436 Change-Id: Ie692663706b378022ec0d621d92ca5054bad8d1b Signed-off-by: mpriyank <priyank.maheshwari@est.tech>
2024-10-29[k6] Increase timeouts after DMI stub changesdanielhanrahan1-2/+2
This commit changes k6 setup and teardown timeouts to 20 minutes. This fixes current test failures due to time-outs. Issue-ID: CPS-2410 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: Ifdbf26ed1669cdb3a2fc9afb551308757511281b
2024-10-25[k6] Update tests for DMI stub changes for modulesdanielhanrahan3-4/+4
It is needed to update k6 tests due to changes made in the DMI stub for dynamic module generation: - Increase setup timeout to 30 minutes - Use 5 module set tags: tagA to tagE - Update module search to look for module100 Issue-ID: CPS-2410 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I17830e4d858883743476d297af90d1ceaba1eb44
2024-10-23Resize cps expectationhalil.cakal1-1/+1
- set expectation to 70 (cmhanldes/second) for discovery case Issue-ID: CPS-2350 Change-Id: I3d0f9f5e20de87f79ebbe569c7905f4eefed35e8 Signed-off-by: halil.cakal <halil.cakal@est.tech>
2024-10-17Refactor and Optimize ModuleSyncWatchdog for Improved Lock Handling and ↵sourabh_sourabh1-1/+1
Queue Management - Enhanced logging for populateWorkQueueIfNeeded and resetPreviouslyLockedCmHandles methods, improving clarity and error handling. - Improved readability and maintainability of the locking mechanism with Hazelcast's FencedLock. - Optimized error handling in catch blocks, logging detailed exception messages and stack traces for better troubleshooting. - Refined lock acquisition and release flow, with clear log messages for both successful and failed lock operations, ensuring safe handling of Hazelcast distributed locks. Issue-ID: CPS-2403 Change-Id: Ie089f36a817d4965782235b51ee987ef054516b1 Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>
2024-10-09Merge "Added K6 script test to check a throughput of CPS kafka consumer"Daniel Hanrahan2-0/+137
2024-10-03Added K6 script test to check a throughput of CPS kafka consumersourabh_sourabh2-0/+137
- Added new K6 script to produce cm avc enents. (It is not a part regular automated tests) - Added a json resource (avc event) Issue-ID: CPS-2329 Change-Id: I6446bc120382257c5039df17fd34b84b7c6f6550 Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>
2024-10-03Re-adjust cps expectationshalil.cakal1-13/+13
- aligning cps expectation for new plots Issue-ID: CPS-2350 Change-Id: I5b300c4f5432bf2bdfe8a8b0d52b2ce60bd338d6 Signed-off-by: halil.cakal <halil.cakal@est.tech>
2024-09-24[k6] CM handles searches using different filtersdanielhanrahan4-30/+181
Each 5 VUs for searches uses a different filter: no filter, module, property, cps path, and trust level. This is needed since each search may have different performance. This also helps reduce load versus doing a combined module and property search all the time. Issue-ID: CPS-2349 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I33a75260f97697d2061d10b80ba7eeb2f6c346f5
2024-09-18Change execution order of the legacy async batch read operation in k6 suitehalil.cakal1-2/+3
- run the test after the parallel cases - add 30 seconds safe-period of time waiting for completion of active threads (remained from previous test cases) Issue-ID: CPS-2414 Change-Id: I1260b37cd4f1974eebcccf867b36172b407a9646 Signed-off-by: halil.cakal <halil.cakal@est.tech>
2024-09-13Add CPS limit to the plotshalil.cakal1-16/+16
- as of now the cps limits given manually according to the latest result - there will be another commit will change the plot job in cicd repo to reflect the values on the plots Issue-ID: CPS-2350 Change-Id: Ic371f46894f1825dca1d26ec65ae0f62aa06c327 Signed-off-by: halil.cakal <halil.cakal@est.tech>
2024-09-12Calculate throughput for legacy batch read data operationshalil.cakal3-53/+66
- stress the cps-and-ncmp module for batch read data operation - for this aim use shared-ireations during async http request - as parallel to this, consume expected number of messages as fast as possible Issue-ID: CPS-2268 Change-Id: I1b6724479dac7391cbb6407fda52d15176aa8985 Signed-off-by: halil.cakal <halil.cakal@est.tech>
2024-09-10[k6] Fix incorrect results when errors occur in k6 tests (case 2)egernug1-1/+5
De-registration is reported as 200,000 handles per second if all operations fail - Added a running total for successful de-registrations of CM Handles Issue-ID: CPS-2346 Change-Id: Ib4302d3b605ff1eaf4ecadac11b4e851a69210df Signed-off-by: egernug <gerard.nugent@est.tech>
2024-09-10Fix high-cardinality metrics in k6, causing high memory usesourabh_sourabh4-58/+101
- Added unique IDs as metric tags for all the endpoints. - Re-arranged order of public and prive js methods. Issue-ID: CPS-2331 Change-Id: Ib876a647fb35110c50670c7222986e8a8a6f5ca0 Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>
2024-09-05[k6] Minor correction to plot titlesdanielhanrahan1-2/+2
Plot title say Module filter instead of Module and Property. Issue-ID: CPS-2349 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: Ie23080a6085960b89132ad38d07d0c2b6f12471b
2024-09-04[k6] Align tests with updated requirementsdanielhanrahan5-76/+75
- Test load increased to 5 CM handle searches and 5 Id searches to reflect current requirements. - CM handles searches now use a combined search filter including 1 module and 1 public property, so we are testing with a heavier load than required, for early warning of trouble. - Test load reduced for passthrough operations to 4 VUs per operation to reflect addition of batch operation. - Added test of passthrough write operation with alternate ID. - Additional refactoring. Issue-ID: CPS-2349 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I88d2fe431f74821e8e0e976441efcf82aa320849
2024-09-03Merge "Fix huge logs produced when errors occur in k6"Daniel Hanrahan1-1/+3
2024-08-30[k6] Report failure rate for all tests in a single plotdanielhanrahan2-16/+2
Instead of separate plots of test failures for each test case, we have one single plot showing failures. Issue-ID: CPS-2347 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: Ie79b75c34903a7df2037474b3536d4867c1a61e0
2024-08-30[k6] Fix k6 tests after DMI stub changesdanielhanrahan2-6/+6
A change to DMI stub has broken k6 module search due to module rename and increased registration time. - increase setupTimeout by 2 minutes to factor in DMI startup delay - module search uses module common to all module set tags - only record times for searches returning number of CM-handles Issue-ID: CPS-2353 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I9067dc73015cfb475cdce1e7a3cc2d5401449caf
2024-08-30Fix huge logs produced when errors occur in k6sourabh_sourabh1-1/+3
- Changed k6 config to ignore all the error logs. Issue-ID: CPS-2344 Change-Id: Iad7be43ae5bb909518f9f7d8e386a97c96c349e3 Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>
2024-08-27[k6] Exclude timings when error occurdanielhanrahan2-28/+37
If an error occurs during k6 test, e.g. tcp connection refused, the timing of failed response is included in the average response time. This can lead to incorrect plots when errors occur, e.g. operations appear to be faster when really the operations are failing. - use custom Trend metrics for all measurements - only log results when operations return expected response codes Issue-ID: CPS-2345 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: Iab1db5e5ca9530f33da239187348cabcd0605648
2024-08-27Merge "Plot request failures for each k6 test"Daniel Hanrahan3-37/+52
2024-08-27Merge "[k6] Always pull latest cps and dmi stub images"Priyank Maheshwari1-1/+7
2024-08-27Install dependencies for k6 and csitdanielhanrahan2-0/+51
- install docker-compose in csit and k6 tests - install k6 with kafka extension in k6 tests Issue-ID: CPS-2376 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I8a67813a96dc7f321d4abe0aa91e236fd1060f3e
2024-08-26Plot request failures for each k6 testhalil.cakal3-37/+52
- add request failures to plot for each tests - simplify setup and teardown function Issue-ID: CPS-2347 Change-Id: I81226518af7bda12b0489ec46de08e27c96cc16d Signed-off-by: halil.cakal <halil.cakal@est.tech>
2024-08-26[k6] Always pull latest cps and dmi stub imagesdanielhanrahan1-1/+7
Presently, k6 tests are using old cached DMI stub version in the Jenkins job. This commit ensures latest images are pulled by cleaning old images if a variable is set. Issue-ID: CPS-2368 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I66a749c7cdeb7daee06a00b06dff50ead6854f28
2024-08-16[k6] Fix teardown script shutting down all containersdanielhanrahan1-8/+2
The teardown script in k6 shuts down ALL docker containers when done. This can cause failures in the groovy tests when running in parallel, as the groovy test container for DB also shuts down. This fix is to shut down using docker-compose. Issue-ID: CPS-2367 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: Ic55ca6396dfc3170e4a326f34214d3a819977f50
2024-08-16[k6] Remove incorrect passthrough ratesdanielhanrahan2-14/+8
Reported rates for passthrough operations is incorrectly reported, due to it being averaged over whole test time including setup and teardown. It was agreed to simply not report rates, and only report NCMP overhead. - remove http request rates for passthrough operations - change VUs from passthrough read so that there are 10 total Issue-ID: CPS-2264 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: Ida498955e69dffb332d6efeb8a8d73343565f9a1
2024-08-15Fix syntax error on k6 testdanielhanrahan1-1/+1
Issue-ID: CPS-2357 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: Ibbd32282608ff707c8030d43df3cf85043242218
2024-08-14Add moduleSetTag on k6 testemaclee2-2/+4
Issue-ID: CPS-2357 Change-Id: I21b980b1e36988d3b5daf07e4f6a14fe4309cd20 Signed-off-by: emaclee <lee.anjella.macabuhay@est.tech>
2024-08-12K6 performance test for Alternate Id Readseanbeirne4-1/+28
Issue-Id: CPS-2279 Change-Id: Iade53a85afb183c0df259c5e7fc12fa0f6986218 Signed-off-by: seanbeirne <sean.beirne@est.tech>
2024-07-31Merge "Add k6 for legacy async batch passthrough read operation"Daniel Hanrahan4-6/+79
2024-07-31Add k6 for legacy async batch passthrough read operationhalil.cakal4-6/+79
- add kafka support to k6 codebase - add two new tests: data operation batch passthrough read and data operation async batch read (from kafka) - calculate the events/second via Trend - add kafka ui support to docker-compose Note: Before this commit being merged, the k6 executable should have been compiled with the kafka extension of xk6-kafka Issue-ID: CPS-2268 Change-Id: Ib7777b7bc9f15b210ea36d3541cba0e0c943f883 Signed-off-by: halil.cakal <halil.cakal@est.tech>
2024-07-25[k6] Fix response code check for passthrough write testdanielhanrahan1-1/+1
The POST operation returns 201, not 200. This causes checks to fail when running k6 manually. Issue-ID: CPS-2267 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I3c0e771b89aa09e41c9dbdf1c532599e1ab5440d
2024-07-18[k6] Measure NCMP overhead for passthrough operationsdanielhanrahan2-13/+30
Measure NCMP overhead by subtracting DMI delay from request duration. Issue-ID: CPS-2269 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: Ieb4758e9f7cab119da4c28e4a3febfa1361b2613
2024-07-16Report throughput for passthrough read operationhalil.cakal2-3/+2
- change threshold from avg duration to req per second - the delay for dmi has been hanlded in seprate commit in dmi repository - add delays through docker-compose env variables Issue-ID:CPS-2285 Change-Id: I35978b21c208fdb4209b1610160dc5545ee22965 Signed-off-by: halil.cakal <halil.cakal@est.tech>
2024-07-12Add passthrough write (CUD) operations to k6halil.cakal4-3/+26
- the delay for dmi has been hanlded in seprate commit - for simplicity only the POST request is considered Issue-ID: CPS-2267 Change-Id: I62ede04dd8e2744fc723566742396df31c42ea5d Signed-off-by: halil.cakal <halil.cakal@est.tech>
2024-07-09Merge "Setup automated tests with two replicas of cps-and-ncmp"Daniel Hanrahan1-2/+10
2024-07-05[k6] Measure CM-handle (de)registration in CM-handles/secdanielhanrahan10-231/+130
As per characteristics requirements document: - measure registration in CM-handles/second - measure deregistration in CM-handles/second - summary table includes test case number, description, units of measurement, actual value and limit. Issue-ID: CPS-2269 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I838004da1c230ab722f49c2adacf34e730d7ac79
2024-07-01Setup automated tests with two replicas of cps-and-ncmphalil.cakal1-2/+10
- this change for two test suits: CSIT and K6 - increase cps-and-ncmp replicas to 2 - introduce nginx as load-balancer - set proxies inside nginx for the cps-and-ncmp URI paths - upgrade hazelcast-spring to 5.3.7 Issue-ID: CPS-2038 Change-Id: If1b6c1e1218af16b24cc061507ea9b25d277d143 Signed-off-by: halil.cakal <halil.cakal@est.tech>
2024-06-24Clean up the k6 test suitehalil.cakal10-324/+9
- Remove the tests that are not in the Characteristics - Set .625 seconds/operation for cm handle id search with module filter - Set 13 seconds/operation for cm handle search with module filter Issue-ID: CPS-2286 Change-Id: I88d5773abadb29ce46914029332384ae8e55e032 Signed-off-by: halil.cakal <halil.cakal@est.tech>
2024-06-14[k6] Add JVM warmup phase (#2)danielhanrahan4-9/+14
To account for JVM warmup effect, the k6 test suite is run twice: initially, the whole test suite is run without recording results. - Added environment variables to allow setting total CM-handles and batch size for registration. By reducing total CM-handles during warm-up, many more operations will be run. Issue-ID: CPS-2264 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: Ib1d1e76d31719d788a3e026797faa0a425b320a2
2024-06-14[k6] Refactoring k6 tests (#1)danielhanrahan15-117/+174
This commit moves all common request logic into a common folder. It is needed to avoid duplication before adding JVM warmup phase. - move registration-related code into common folder - move passthrough operations into common folder Issue-ID: CPS-2208 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: Ia9ebf61d21044b43063bde153f9c526e67d607c8
2024-06-06[k6] Refactor k6 tests for CM handle searchesdanielhanrahan8-66/+46
A cleanup of the k6 tests to avoid code duplication Issue-ID: CPS-2208 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: If1e58ee87300cd28ed72b365d9a86e2709dc6c13
2024-05-28Adjust K6 thresholdsdanielhanrahan11-28/+28
- Adjust timings based on previous runs. - Reduce from 5 to 4 parallel clients for CM handle searches to avoid OutOfMemoryErrors. Issue-ID: CPS-2208 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I339cc3df84b2295bb11371ab73b4948f43615c15
2024-05-27Print summary report for K6 testshalil.cakal15-80/+183
- add a generic template to print summary of k6 test results - remove container logs since its around 50 mb Issue-ID: CPS-2215 Change-Id: I8971fc30b9dc8be2ed16eda6755d17a91f608b48 Signed-off-by: halil.cakal <halil.cakal@est.tech> Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Signed-off-by: halil.cakal <halil.cakal@est.tech>