summaryrefslogtreecommitdiffstats
path: root/cps-ri/src/main
AgeCommit message (Collapse)AuthorFilesLines
2023-06-14Lower memory usage in FragmentRepositorydanielhanrahan5-205/+151
Avoid using Spring Data "interface projection" in FragmentRepository. The use of FragmentExtract in FragmentRepository is causing an overhead of around 5 kilobytes per fragment, which is leading to abnormally high memory usage when queries return a large number of nodes. For example, around 250MB of additional memory is needlessly used when fetching 50,000 datanodes. - Remove FragmentExtract interface and FragmentEntityArranger class. - Add FragmentPrefetchRepository, using JdbcTemplate and RowMapper to fetch FragmentEntity descendants in a single SQL query. - Many CpsDataService operations have memory reductions: - queryDataNodes - getDataNodesForMultipleXpaths - updateDataNodesAndDescendants - updateNodeLeaves - and any NCMP methods using the above. Issue-ID: CPS-1716 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: Ic47a2c9eb34150ed76bd5ce452fe1c9aaf9b4c5c
2023-06-13Merge "Move integration test (DataService)"Toine Siebelink2-5/+4
2023-06-12Fix for recursive SQL returning extra level of descendantsdanielhanrahan1-2/+2
Recursive SQL queries are incorrectly returning maxDepth + 1 levels of fragments due to incorrect comparison in the WHERE clause. This code is used by getDataNodes and queryDataNodes to prefetch the descendants before converting to datanodes. Prefetching extra descendants causes high memory usage. Issue-ID: CPS-1716 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: Ie22215c0c7018cd293f73cf75adca2cd55f218a6
2023-06-12Move integration test (DataService)ToineSiebelink2-5/+4
- Moved all Data Intergation Test - Added/improved some module Moudle Servcie test too - Removed some unused (impossible) code-paths in prod code - Replaced null's with meaningful strings (in edge case scenarios) - Replaced @Shared with static (shorter) in some test Issue-ID: CPS-1687 Signed-off-by: ToineSiebelink <toine.siebelink@est.tech> Change-Id: Ib4b1ec1706ace7e3f5ac03d85177a6f7d4f6b8d0
2023-05-30Add <,> operators support to cps-pathRudrangi Anupriya1-17/+32
Issue-ID: CPS-1273 Change-Id: I5d562463b9a49abfe0436047a637857d10596fff Signed-off-by: Rudrangi Anupriya <ra00745022@techmahindra.com>
2023-05-17Entity ID types do not match types in database definitionToineSiebelink17-36/+86
As Liquibase is used, JPA does not manage database definition. Within Liquibase changelog, only Anchor and Fragment have BIGINT (Long) IDs. Other entities have INTEGER IDs. - Change AnchorEntity/AnchorRepository ID type to Long - Change YangResourceEntity/YangResourceRepository and ModuleReferenceRepository ID to Integer - Modify datatypes of foreign key references in liquibase changelog to use correct types for dataspace_id, schema_set_id, yang_resource_id - Update test liquibase changelog Issue-ID: CPS-1676 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I26de6712b2e74d166fc5a48b1f8149de6a1e5199 Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
2023-05-17Move integration test for adminServiceToineSiebelink1-4/+2
- finalized moving of admin service interation test (just some scenarios were missing) - deleted old test class (code coverage in RI module dropped) - line/branch coverage now 100/100 (was 97/91) - small refactoring; renaming and removed unnecessary check Signed-off-by: ToineSiebelink <toine.siebelink@est.tech> Change-Id: Ic683548ea5e1e4e252f257c0f1034c5cf76e498d
2023-05-12Remove dataspace_id column from Fragment tabledanielhanrahan8-26/+150
- Remove dataspace_id column from fragment table and drop associated index, foreign key constraint and uniqueness constaint. - Add uniqueness constraint using only fragment xpath and anchor_id. - Add not-null constraint to anchor_id. - Update code and test data to remove references to dataspace_id. - Repopulate the dataspace_id in Liquibase rollback (verified with manual testing of update/rollback). Issue-ID: CPS-1677 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I467cccba25ac5d884ec790064ca58150281b7e1d
2023-05-11Move integration test for moduleServiceToineSiebelink1-41/+23
- all modules service interation test moved - added some aditional scenarios to increase coverage - removed unused method(s) - remove javdoc on private methods - some minor refactoring like replace 'var' with correct type Issue-ID: CPS-1687 Signed-off-by: ToineSiebelink <toine.siebelink@est.tech> Change-Id: I03ff1f3562cfe38318e8b9af81be47a1fe667da2
2023-05-10Move integration test (DataService)ToineSiebelink1-7/+0
- dropped ri module coverage to alow for move - removed performance tests (already better tests in intergration module) - added 'save' iue cases in new integration-test - removed no longer used version for save single data node (todo: remove old tests for similar functionality, will wait until new test cover same) Issue-ID: CPS-1687 Signed-off-by: ToineSiebelink <toine.siebelink@est.tech> Change-Id: I4c2e788e0a313938a923cd1e7e52584d739ec6f4
2023-05-09Support for Patch across multiple data nodes--global2-13/+31
- Added new method updateMultipleDataLeaves to perform Update on multiple data nodes - Deprecated singular method of update data node(updateDataLeaves) - Refactored code where singular version was used - Updated release notes Issue-ID: CPS-1006 Signed-off-by: <as00745003@techmahindra.com> Change-Id: If67280e2dd3ad566de9a8217489f168415e624bc
2023-05-08Merge "Use recursive SQL to fetch descendants in CpsPath queries (CPS-1664 #4)"Sourabh Sourabh3-99/+37
2023-05-08Merge "Fix and refactor query across anchors (CPS-1664 #3)"Sourabh Sourabh7-68/+154
2023-05-08Merge "[CPS] RI: Code Refactoring # Replace '[\s\S]' to '.' as it is same as ↵Toine Siebelink1-8/+3
'.' while grouping in regex # Updated the groovy files to add 'assert' and to replace deprecated method # Removed deprecated 'storeDataNode' method."
2023-05-08Merge "Don't handle ancestor-axis using regex quickfind (CPS-1664 #2)"Sourabh Sourabh2-10/+6
2023-05-05[CPS] RI: Code Refactoringraviteja.karumuri1-8/+3
# Replace '[\s\S]' to '.' as it is same as '.' while grouping in regex # Updated the groovy files to add 'assert' and to replace deprecated method # Removed deprecated 'storeDataNode' method. Issue-ID: CPS-1594 Signed-off-by: raviteja.karumuri <raviteja.karumuri@est.tech> Change-Id: Iaa5f8b0cc8d66473c993060a8fc5a54e37e373b1
2023-05-04Fragment Entity does not represent the correct relationship with Anchordanielhanrahan1-2/+1
FragmentEntity incorrectly has OneToOne relationship to AnchorEntity, instead of ManyToOne. The reason this has not caused errors so far is that Liquibase is being used to generate database definition, not JPA. Issue-ID: CPS-863 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I8d3c5dc19193a84fa59526de7f2cbab7ab20a8fc
2023-05-04Use recursive SQL to fetch descendants in CpsPath queries (CPS-1664 #4)danielhanrahan3-99/+37
- Add recursive SQL method to fetch descendants from queries. This changes worst-case complexity from quadratic to linear, resulting in extremely large performance increase for large number of datanodes. - Remove RegexQuickFind algorithm as it is no longer faster. - Updated query performance test timings Issue-ID: CPS-1664 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: If5f0b54a88af1cb681006bbeca7043345dcdc8da
2023-05-04Fix and refactor query across anchors (CPS-1664 #3)danielhanrahan7-68/+154
- Fix CPS-1580: Query Across All Anchors Does NOT Filter on Dataspace - Fix CPS-1582: NullPointerException in queryDataNodesAcrossAnchors - Improve performance of queryDataNodesAcrossAnchors - Refactor queryDataNodes - Refactor FragmentQueryBuilder Issue-ID: CPS-1580 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I8b83d4c580280d8719ab2ac33f3914f53e798774
2023-05-04Don't handle ancestor-axis using regex quickfind (CPS-1664 #2)danielhanrahan2-10/+6
This is a preparatory patch to make refactoring easier in CPS-1664 #3 Issue-ID: CPS-1664 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I1f94fef137ae41f7dedc585d8314f13493885d1b
2023-05-04Merge "Sensible equals and hashCode for FragmentEntity (CPS-1664 #1)"Toine Siebelink3-1/+10
2023-05-04Fix for quickfind with descendants incl. list entriesToineSiebelink2-24/+38
- add specialized regex - fixed legacy issues with absolute paths (added ^ in regex's) - split into 2 methods for normal queries and quickfind - fixed order of private methods Issue-ID: CPS-1671 Signed-off-by: ToineSiebelink <toine.siebelink@est.tech> Change-Id: I785a230cd0e3e0359e3cbb0fa754a21b6e2be7ff
2023-05-04Sensible equals and hashCode for FragmentEntity (CPS-1664 #1)danielhanrahan3-1/+10
Include Anchor and Xpath in equals and hashCode methods for FragmentEntity. (This also requires adding equals and hashCode for AnchorEntity and DataspaceEntity.) The combination of dataspace, anchor, and xpath uniquely identify a fragment/datanode. This allows FragmentEntity objects returned from query across anchors to be stored in Set collections. Performance was observed to be unaffected by the change. Issue-ID: CPS-1664 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I2c7e3957e392af36f5230d08c9bbd550f44e7444
2023-05-03canUseRegexQuickfind does not test for contains-conditiondanielhanrahan1-1/+2
This bug causes queries using contains-condition to return wrong results, when INCLUDE_ALL_DESCENDANTS option is set. Issue-ID: CPS-1663 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: If323e0e3b54d8c9a213dffda46897b2efb01fe2a
2023-04-26[CPS] RI, Service: Code Refactoringraviteja.karumuri2-22/+0
# Remove unused method "updateDataNodeAndDescendants" # Remove unused production code Issue-ID: CPS-1594 Signed-off-by: raviteja.karumuri <raviteja.karumuri@est.tech> Change-Id: Id909e9da3e9181c181dfdfe62f051451d8e488a2
2023-04-24Remove 32K limit from queries with collection parametersdanielhanrahan7-160/+113
SQL queries taking collection parameters currently create a seperate query parameter for each collection element. There is a limit of around 2^15 (32,768) query parameters. Postgres DB natively supports array types, in which the whole array is transmitted in binary as a single parameter. Changing queries to use arrays removes the 32K limit on queries. - Add support for Postgres arrays to queries - Change repository methods to use arrays, and provide overloaded versions taking collection parameters - Update tests to reflect 32K limit being lifted Issue-ID: CPS-1573 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I64f2aeaedbe54bfe12e3079cba0f2216759142c3
2023-04-21Merge "Add contains condition support to cps-path"Luke Gleeson1-0/+11
2023-04-21Add contains condition support to cps-pathRudrangi Anupriya1-0/+11
Issue-ID: CPS-1272 Change-Id: Ic81d1322cacc64a8752916324b801d02be47d34f Signed-off-by: Rudrangi Anupriya <ra00745022@techmahindra.com>
2023-04-20Migrate hibernate-types to hypersistence-utilsdanielhanrahan1-1/+1
There is newer release of the dependency that provides JSONB support. The newer version provides additional features, such as Postgres arrays. It was renamed from hibernate-types to hypersistence-utils. Issue-ID: CPS-1573 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: Iab556850041b592a8c4d515270dfe4bd9547957a
2023-04-17Remove CollectionUtils dependencylukegleeson1-2/+1
Remove dependency creating pipeline issue Issue-ID: CPS-1626 Signed-off-by: lukegleeson <luke.gleeson@est.tech> Change-Id: I64a6019a8196aa98dd3874d08f373c4fb9e3b8c9
2023-04-14Use cascade delete in fragments tabledanielhanrahan4-30/+40
- Add Liquibase change to cascade delete fragments - Remove cascade delete logic from FragmentNativeRepository - Remove unused method deleteFragmentEntity Issue-ID: CPS-1617 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: Iecf4f3022b402e0328609ddc11a879c83e5d7f30
2023-04-14Remove @NotNull from repository methodsdanielhanrahan4-29/+23
Issue-ID: CPS-1573 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I55312bdc5454ed9ca60441b3968d19a61028eb66
2023-04-14Add OR operator to cps-pathRudrangi Anupriya1-16/+22
Issue-ID: CPS-1215 Change-Id: I91fdf5bddcc4fc12a8cf9dbce75f77c832c55871 Signed-off-by: Rudrangi Anupriya <ra00745022@techmahindra.com>
2023-04-11Populate Dataspace field of DataNodedanielhanrahan1-0/+1
Issue-ID: CPS-1603 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I85aca8bccc28d36aa4cefa0a32cb9bdf2156618a
2023-03-21Query data nodes across all anchors under one dataspacerajesh.kumar6-8/+88
Issue-ID: CPS-1396 Change-ID: I73f97f986a817d423f93a8d922dcd9647b1412ab Signed-off-by: rajesh.kumar <rk00747546@techmahindra.com>
2023-03-14Reduce dataspace/anchor lookups in CpsDataServicedanielhanrahan1-1/+1
- Use Anchor object instead of names in processDataUpdatedEventAsync - Use Anchor object instead of names in buildDataNodes - Avoid unnecessary ArrayList copy in updateDataNodesAndDescendants Issue-ID: CPS-1536 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I46936a655a3d151357e55b68e1c4161e07100e84
2023-03-13Fetch fragment entities using recursive SQL querydanielhanrahan5-131/+39
- Add SQL query that can fetch needed fragments to any given depth - Update getFragmentEntities to use new query - Remove now unused FragmentRepositoryMultiPathQuery - Remove unused TempTableCreator method - Result: getDataNodesForMultipleXpaths is up to 10 times faster Issue-ID: CPS-1525 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I07cbc9da5ab994ce7e0c2b02d7ca05089f05dab0
2023-03-13Merge "Reduce dataspace/anchor lookups in CpsDataPersistenceService"Toine Siebelink2-143/+83
2023-03-13Merge "Add new metrics to the get data node method"Toine Siebelink1-1/+8
2023-03-13Reduce dataspace/anchor lookups in CpsDataPersistenceServicedanielhanrahan2-144/+84
- Remove unneeded calls to DataspaceRepository::getByName - Remove unneeded calls to AnchorRepository::getByDataspaceAndName - Refactor FragmentRepository Issue-ID: CPS-1536 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I2121f6247070ee4a7c000e06ec66a6278b758540
2023-03-13Make single deleteDataNode use plural deleteDataNodesdanielhanrahan1-71/+31
- Make deleteDataNode and deleteListDataNode call deleteDataNodes - Add onlySupportListDeletion option to deleteDataNodes to support original deleteListDataNode behaviour - Allow delete root xpath in deleteDataNodes - Fix incorrect use of PathParsingException in deleteDataNode - Update performance tests timings Issue-ID: CPS-1523 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: I92c3c3ce606a5ab2cb8e6779d1ee0f9853529982
2023-03-09Add new metrics to the get data node methodkissand1-1/+8
Issue-ID: CPS-1509 Change-Id: I2eded9a23b56f5263591c710c7a707cf281f93d7 Signed-off-by: kissand <andras.zoltan.kiss@est.tech>
2023-03-08Add index to schema_set_yang_resources tabledanielhanrahan2-2/+19
Currently, the database is sequentially scanning when deleting from schema_set_yang_resources table as there are no indexes on the table. Adding an index doubles performance of NCMP CM-handle deregistration. Issue-ID: CPS-1535 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: Ia6ac7ef95d1662f49a77fc72112f1e29a67079ff
2023-03-07Drop temp table when transaction finishesdanielhanrahan1-1/+2
By default, temp tables are persisted until the end of the session. This causes "Out of shared memory" errors in the database, which then auto-vacuums the orphaned temp tables to free memory. To fix this, we drop temp tables when the transaction finishes, using ON COMMIT DROP. Issue-ID: CPS-1533 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: Iac47d85e9621531a4a95b99c4becc8cc95242721
2023-03-01Add DataNodeNotFoundException to deleteDataNodesdanielhanrahan2-10/+24
Current implementation of NCMP handle de-registration relies on DataNodeNotFoundException being thrown to report errors. - Make deleteDataNodes throw DataNodeFoundExceptionBatch - Update performance test timings Issue-ID: CPS-1481 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: Ib833bdb4a23d24e1784bdaf4e5e5e8a9acb41c54
2023-02-28Skip deleting list xpaths that are list elementsdanielhanrahan1-4/+10
List elements cannot be lists. Filtering list elements before trying to delete lists by xpaths doubles performance when deleting list elements (which doubles performance of CM handle de-registration). Issue-ID: CPS-1511 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: Ieb6002212d006396d468f27f853708b5aa1e31f2
2023-02-24Improve performance for update data nodesdanielhanrahan1-39/+34
- Extract method getFragmentEntities from getDataNodes - Remove unused code from getFragmentEntity - Use bulk getFragmentEntities in updateDataNodesAndDescendants - Update performance test timings Issue-ID: CPS-1504 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: Icd2c6a0e6009e152de43090cbc23a21349703da2
2023-02-21Improve performance of deleteDataNodes SQLdanielhanrahan1-35/+38
- Use SQL IN operator instead of temp table when deleting nodes - Use Postgresql LIKE ANY array operator when deleting lists - Update delete perf test timings - Refactor adding cascade delete constraint Issue-ID: CPS-1502 Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech> Change-Id: Ic90b867e7c71ec1981f05a9122322ece84dd8bde
2023-02-20Merge "FragmentEntity stuck in memory on "Already exists" error"Toine Siebelink1-0/+2
2023-02-20Merge "CPS-1401 Implement V2 of GET Data Node API"Luke Gleeson1-10/+15