diff options
| author |   morganrol <morgan.richomme@orange.com> | 2021-04-06 19:57:44 +0200 |
|---|---|---|
| committer |   Bartek Grzybowski <b.grzybowski@partner.samsung.com> | 2021-05-06 09:06:47 +0000 |
| commit | 8cae510b7e65b85f33a5e3b4c8fdedc1229515da (patch) | |
| tree | 46c037832b5bcec47e5769a7d18fdc155e616b4b | |
| parent | 7df9a28f41a11afc2572c0a1f2558503f07a907e (diff) | |
[DOC] Honolulu documentationhonolulu
Update documentation for Honolulu
Issue-ID: INT-1888
Signed-off-by: morganrol <morgan.richomme@orange.com>
Change-Id: I66e6f397bcda898445e256dd8e22dfcb20847408
(cherry picked from commit 81dc70f5e575ae4a972fc50e038733a0dc8481aa)
48 files changed, 763 insertions, 675 deletions
diff --git a/docs/docs_usecases_release.rst b/docs/docs_usecases_release.rst index ffd954617..9165959cd 100644 --- a/docs/docs_usecases_release.rst +++ b/docs/docs_usecases_release.rst @@ -5,14 +5,14 @@ :orphan: -Guilin Use Cases and Requirements -================================= +Honolulu Use Cases and Requirements +=================================== Description ----------- This session includes use cases and functional requirements which have been -officially verified in Guilin release by the ONAP community. +officially verified in Honolulu release by the ONAP community. For each use case or functional requirement, you can find contact names and a link to the associated documentation. @@ -31,3 +31,4 @@ This documentation deals with usecases/release_automated_usecases.rst usecases/release_requirements.rst usecases/release_non_functional_requirements.rst + usecases/deprecated_usecases.rst diff --git a/docs/files/csv/issues-integration.csv b/docs/files/csv/issues-integration.csv index 70ead70c8..ef939c297 100644 --- a/docs/files/csv/issues-integration.csv +++ b/docs/files/csv/issues-integration.csv @@ -1,11 +1,6 @@ Issue key;Issue id;Summary;Priority;Status;Created -INT-1770;56498;PNF registarte test case very offen fails at general step Submit SDC Software Product;Medium;Open;05/Nov/20 7:32 PM -INT-1755;56342;[PMSH] Wrong AAI api version in CSITs;Medium;Open;28/Oct/20 11:20 AM -INT-1738;55962;SO CSIT failing due to invalid syntax;Medium;Open;09/Oct/20 2:54 PM -INT-1726;55683;SDC intitlement pool creation is failing;Medium;Open;24/Sep/20 2:44 PM -INT-1720;55314;Robot certificates is too long and has bad owner;Medium;Open;14/Sep/20 6:57 AM -INT-1717;55218;Robot has python 2.7 pods;High;Open;10/Sep/20 7:31 AM -INT-1716;55175;robot pod is launched as root;High;Open;09/Sep/20 4:32 PM -INT-1660;53648;Fix 5g bulkpm CSITs for pm-mapper java 11;Medium;Open;16/Jul/20 5:28 AM -INT-1659;53599;MDONS Extnesion: Integration Test;Medium;In Progress;15/Jul/20 2:58 AM -INT-1655;53455;SDNC CSIT failing for test plan sdnc_netconf_tls_post_deploy.;Medium;Delivered;08/Jul/20 5:24 PM +INT-1883;59237;Reccurent various errors in smoke tests;High;In progress;12/Mar/21 +INT-1880;59163;Xtesting - basic_vm - values from basic_vm-service.yaml not applied;Low;In progress;11/Mar/21 +INT-1873;58953;Race condition between cmpv2 and Basic DCAE health check;Medium;In progress;24/Feb/21 +INT-1843;57940;Upload 5G-core-nf-simulator code;Medium;In progress;25/Jan/21 +INT-1828;57751;Update Transport simu documentation;Medium;In progress;11/Jan/21 diff --git a/docs/files/csv/issues-testsuite.csv b/docs/files/csv/issues-testsuite.csv index 2cbe8f9b6..907b87bcf 100644 --- a/docs/files/csv/issues-testsuite.csv +++ b/docs/files/csv/issues-testsuite.csv @@ -1,2 +1,2 @@ Issue key;Issue id;Summary;Priority;Status;Created -TEST-277;56680;Issues with basic_clamp test;Medium;Open;12/Nov/20 9:24 AM +TEST-306;58636;Resiliency testing;Medium;In progress;17/Feb/21 diff --git a/docs/files/csv/release-demo-features.csv b/docs/files/csv/release-demo-features.csv index 69899d044..92a8d9af3 100644 --- a/docs/files/csv/release-demo-features.csv +++ b/docs/files/csv/release-demo-features.csv @@ -1,10 +1,8 @@ Issue-ID;Description -INT-1788;Add Lasse as Integration committer -INT-1785;Create demo 1.7.0 for guilin -INT-1658;vFW CNF CDS scripts update for native Helm support -INT-1240;ROBOT e2e test cases improvements for csar/zip tosca package validation in VNFSDK and SDC Onbarding -INT-1658;vFW CDS CNF CBA with native k8s profile upload -INT-1724;Update INFO.yaml -INT-1600;VNF Software Version parameter added to CDS Heat templates -INT-1625;Update pom files in master to 1.7.0-SNAPSHOT -SO-2983;Fix for missing VNFC rel information +INT-1868;vFW CNF fix for issue with config-deploy operation +INT-1868;vFW CNF updates after integration tests +INT-1868;Real vFW CNF config-assign and config-deploy +INT-1868;Implementation of config-assign and config-deploy actions +INT-1868;Add cofig-assign and config-deploy to cds workflow +INT-1658;Check orchestration status during instantiation and deletion vFW_CNF_CDS usecase +INT-1658;Automation scripts for vFW_CNF_CDS usecase diff --git a/docs/files/csv/release-integration-features.csv b/docs/files/csv/release-integration-features.csv index 484cdb07d..39714891e 100644 --- a/docs/files/csv/release-integration-features.csv +++ b/docs/files/csv/release-integration-features.csv @@ -1,117 +1,30 @@ Issue-ID;Description -INT-1736;Update branch references -INT-1788;Add Lasse as Integration committer -INT-1690;Release Docs Update for CCVPN - Transport Slicing in Guilin -INT-1736;Correct typos and grammar -INT-1707;Fix Data Type of External RAN NSSMF simulator -INT-1736;Refactor Integration official documentation -INT-1777;Updated docs_E2E_network_slicing.rst -INT-1776;[DOC] Adding few notes and sample outputs to the robot framework documentation -INT-1772;Update OOF PCI document -INT-1740;PNF Software Upgrade documentation -INT-1759;Minor corrections to doc index -INT-1764;Display waiver content for the security tests -INT-1764;Display waiver content for the security tests -INT-1763;Update rke config for windriver deployment -INT-1690;Update for MDONS Extersion in Guilin -INT-1616;optimize size and time using "--no-cache-dir" -INT-1723;Setup basic functional test of NSSMF RAN simulator -INT-1601;Set up and tear down test environment properly -INT-1724;Update INFO.yaml -INT-1723;Add missing dependency for the werkzeug module in setooptools as well -INT-1723;Add missing dependency for the werkzeug module -INT-1707;Add an External RAN NSSMF simulator -INT-1570;Add timeout on SSL internal cert retrieval -INT-1710;Remove old SECCOM waiver lists source -INT-1570;check_certificates: create result dir if needed -TEST-239;update vcpe case -TEST-259;Fixed CBA sdnc_model_version into vFW_CDS_CNF documentation -POLICY-2794;update policy staging images -INT-1601;Add support for detecting Java 8 -INT-1704;AAI-Simulator: Pulling from PNF-SW-UP CSIT to Integration repo -DCAEGEN2-2267;Update tests for Datafile Collector (DCAE SDK update) -INT-1570;Add MANIFEST.in to include j2 file in the python package -POLICY-2794;update policy staging images -INT-1695;Reset security xfail list for Guilin -INT-1696;Update release note for Frankfurt maintenance release -INT-1672;Drop using symlinks for documentation markup rendering -INT-1601;Replace deprecated "with_items" with loops -INT-1601;Add versions of tools used during verification -INT-1677;Manual revert of kubernetes version in rke scripts -INT-1677;Update Kubernetes rke version -INT-1601;Use external volumes for OpenStack virtual machines -INT-1601;Add hosts configuration for Service Mesh ONAP -INT-1601;Make Vagrant-based environment safe defaults -INT-1601;Add separate NICs to access OpenStack virtual machines -INT-1601;Add Ansible roles for OpenStack security groups -INT-1601;Add Ansible roles for OpenStack hosts -INT-1601;Add Ansible roles for OpenStack network -INT-1601;Add Vagrant-based development environment -INT-1601;Introduce ONAP CI Lab infrastructure setup guide -INT-1601;Introduce ONAP CI Lab setup guide -DCAEGEN2-2367;Adjust DFC test files to use FTPES instead of FTPS -INT-1571;Import upstream component version inspection tool -INT-1662;Fix vsftpd config setup -INT-1663;Add README file -INT-1662;Add HOT parameters for the simulator -INT-1661;Setup build env on instance -INT-1661;Setup runner script for the simulator in HOT -DCAEGEN2-2367;Update tls-init-container -INT-1657;[PMSHSIM] Refactoring pmsh-pnf-sim -DCAEGEN2-2219;Add sftp strict host key checking to DFC consul configs -INT-1652;Update release notes for Frankfurt Maintenance release -INT-1570;Fix check certification python pakaging -INT-1649;Add basic cloud-init setup for the simulator instance -INT-1641;Create basic Heat template -INT-1570;Initiate check certificate validity test -INT-1571;Update Operation box in Vagrant environment -INT-1639;netconf-pnp-simulator: fix supervisor patching failure -INT-1642;Create runtime setup for the script -INT-1643;Fix --conf option validation -OOM-2437;Add compare NodePort and Ingress scan -OOM-2434;Fix ingress default http and https ports -INT-1630;Start/stop the 3GPP measurements file generator at simulator start/stop -INT-1629;Pass options to each lifecycle method separately -INT-1629;Validate the subcommands on each instance method invocation -INT-1629;Remove the existing_sim_instances instance attribute -INT-1627;Support setting custom username/password for the file server service -INT-1623;Drop shell wrapper for docker compose operations -INT-1623;Run the 3GPP measurements file generator in bootstrap phase -INT-1621;Generate vsftpd config file in bootstrap phase -INT-1621;Make the config generation private method more generic -INT-1625;Update release note to include demo artifacts -INT-1626;OUpdate OOF_SON use case documentation -ONAPARC-551;Allow using separate storage pool for Vagrant-based minimal ONAP deployment -ONAPARC-551;Adjust default storage device argument on Vagrant-based minimal ONAP deployment -ONAPARC-551;Increase Helm timeout on Vagrant-based minimal ONAP deployment -ONAPARC-551;Set masterPassword on Vagrant-based minimal ONAP deployment -ONAPARC-551;Disable Helm charts linting on Vagrant-based minimal ONAP deployment -ONAPARC-551;Make Helm timeout adjustable on Vagrant-based minimal ONAP deployment -INT-1551;Update stability test documentation -INT-1619;Add a helper method to generate docker compose file -INT-1619;Simplify network name in Compose template -INT-1618;Add helper method to generate simulator config -INT-1617;Update Mass PNF simulator documentation -INT-1617;Drop action methods decorator -INT-1617;Setup and start docker-compose simulator app directly in Python module -INT-1617;Chown vsftpd config file to root while bootstrapping -INT-1617;Add internal routine for logs archiving -INT-1617;Remove useless RUNNING_COMPOSE_CONFIG assignment -INT-1617;Drop shell debug mode in command wrapper method -OOM-2331;Update helm and kubectl versions -INT-1615;Correct ReadTheDoc theme -INT-1457;[vFW_CNF_CDS] Fix typo in Postman collection request -INT-1613;Support stopping sending PNF registration messages in Python module -INT-1613;Add test case for invalid '--count' option value -INT-1611;Drop shell wrapper for 'trigger_custom' action as well -INT-1611;Support triggering simulator app directly from MassPnfSim module -INT-1610;Drop shell wrapper for stopping simulator instances -INT-1531;[vFW_CNF_CDS] UAT testing doc update -INT-1551;Update doc to reference 1.6.3 RC2 frankfurt image -INT-1468;Update E2E_network_slicing doc -INT-1610;Add helper routine for getting iter range in lifecycle methods -INT-1606;Drop shell wrapper for getting simulators status -INT-1606;Add helper method to get running docker containers names list -INT-1606;Add helper method to get simulator instance data -INT-1466;Update OOF_SON use case documentation -INT-1608;Remove postman subdirectory +INT-1874;E2E_network_slicing-H-release known issues update +INT-1601;Flatten Ansible role structure +INT-1906;Add instructions for the simulator wrapper +INT-1869;Update 5G PNF PnP test documentation about information on how to use NF Simulator +INT-1869;[INTEGRATION] NF simulator documentation - fix indents and bullet points +INT-1908;[INT] Fix netconf-pnp-simulator Python dependencies version +INT-1867;vFW CNF use case changes for CNFO +INT-1869;[INTEGRATION] Add NF simulator documentation in read the docs +INT-1888;[DOC] Honolulu documentation +INT-1900;Update the simulators list and description +INT-1900;Update the simulators repository list +INT-1900;Fix repositories links +INT-1875;Updating release notes for CCVPN - Transport Slicing (REQ-456). +INT-1874;E2E_network_slicing - Honolulu updates +INT-1888;[DOCUMENTATION] Add a simulator section in documentation +INT-1877;Fix Integration-netconf-pnp-simulator-docker-merge-master jenkins build +INT-1875;Documentation update for CCVPN - Transport Slicing (REQ-456) +INT-1872;G-release docs_E2E_network_slicing.rst- Reverting the overwrites +TEST-289;[DOC] Improve automated tests doc section +DCAEGEN2-2528;Add HTTPS to DFC tests +INT-1601;Make adding Ubuntu Bionic image to the DevStack idempotent +INT-1601;Create cluster operator access information +INT-1601;Inject operator key when launching cluster instances +INT-1601;Divide host group into cluster and operation instances +INT-1601;Accept host group as an argument to allow reusing roles +INT-1834;Fix version test in CI +ONAPARC-551;Force new location of default Helm chart repository +ONAPARC-551;Update Vagrant-based minimal ONAP deployment to Guilin +DCAEGEN2-2527;Add HTTP to DFC tests diff --git a/docs/files/csv/release-integration-ref.csv b/docs/files/csv/release-integration-ref.csv index 94d07c9fc..0b70ad1b8 100644 --- a/docs/files/csv/release-integration-ref.csv +++ b/docs/files/csv/release-integration-ref.csv @@ -1,22 +1,21 @@ Repository;Revision -demo;28107b9b3116d05eead52f9b88bfa385f4762f0a -integration;b5e15a27d466406483c02e3d5ce07666b286fa3b -integration/csit;e50b1f4517d353e31626d06fa3a5bea079a07c44 -integration/docker/onap-java11;1508c9d6e2afa05808e50bb835c48669b1cf9b03 -integration/docker/onap-python;b2f19c061037bbf899330f5394972fb6a2ab1ed1 -integration/seccom;a8a3e4197944e10d162173cea355c567823fe723 -integration/simulators/5G-core-nf-simulator;13a7b572fcc29ba2440c7eeaca1561b4fecf85d1 +demo;660f9f9e9f4450f12e948b9601fa185af53a9006 +integration;fd30314fcd836a0f0ead81cd6f4c15a678a58228 +integration/csit;e5b006b384d3c514fa64f83c9724f6ce04c2522f +integration/docker/onap-java11;5f1789c7c781cffebb5bbb4b81766a1f91416e1c +integration/docker/onap-python;c8f336a7137e0f34d7cb7a55ee4fcb9cf104b7b4 +integration/seccom;383788aac9e8debe55c19677dda06401595a1dd5 +integration/simulators/5G-core-nf-simulator;2f7418ace1ff799e0e77dd69bbfa099fc100c307 integration/simulators/dc-simulator;cca3a1ede66374b42b3e126a6c3a75c96dd26490 -integration/simulators/ran-simulator;d543c807598ecc4b23cf5dc5b792cad0393f43d1 -integration/terraform;69e2530c06f7bc569f1ad50f877f4129f0b7dd56 -integration/terragrunt;78bb7f2409bb8f4fb826423c39641a3c9f7c07aa +integration/simulators/ran-simulator;9764b6bb545a8edd4979c481627c38c1892f207e +integration/terraform;017fc08036e8c5a7aac70970f0db400aeaed192c +integration/terragrunt;563e0bf506444b97b172998f7304a5ef6297b266 integration/usecases/BBS;7804110df11239058a0bb5b81705521d2d1457d3 integration/usecases/mdons;d8daa086b9cfa97e223ecfc3260a3065c135bb43 -integration/xtesting;9e626d69b71d27daee985ba140730b367b0f71c7 -oparent;a03c1fa9ec368a4352c1e2c3853747133be3aa29 -testsuite; e46e8492687746118ccf444b342a70b95897e2b9 -testsuite/cds;8e75c091c8f2079b8ae59ae544fbd19bbef3f963 +integration/xtesting;cedc9a1de18778e18b994ea1495a44dd8a1d33c9 +oparent;5ef7981d5e998180524c8c936c31015e71c5921f +testsuite;ffe24cbee57c7529ab26fe597d14608ee5617a91 +testsuite/cds;2a26e7f038190ef736a73443ee56169fea6815a9 testsuite/heatbridge;d0d4a8b7af52078c7f4848b2519f17ca380d6915 -testsuite/oom;f3afcb0e60f54840d386084ee3ee2e364b2650f1 -testsuite/python-testing-utilestestsuite/pythonsdk-tests;3c2183191fc1a2d01b24c7e91a144dad8467cd20 -testsuite/robot-utils;03cbae00fc115f35791e8175cdd1a777cc5deb77 +testsuite/python-testing-utils;0b72a331ff8c8fa17995cd399d66c9bc500af747 +testsuite/pythonsdk-tests;ea75b7596605a2bb7e5f51aa39ca6e5a962c59db diff --git a/docs/files/csv/release-oparent-features.csv b/docs/files/csv/release-oparent-features.csv index 8864c0bd6..7593de5f7 100644 --- a/docs/files/csv/release-oparent-features.csv +++ b/docs/files/csv/release-oparent-features.csv @@ -1,15 +1,5 @@ Issue-ID;Description +INT-1711;Bump oparent 3.2.1-SNAPSHOT +INT-1711;Release 3.2.0 oparent INT-1766;Upgrade latest dependencies -INT-1756;Set default to guilin -INT-1486;Bump oparent to version 3.1.1-SNAPSHOT -INT-1486;Release oparent version 3.1.0 -INT-1489;Add CVE Profile -POLICY-2387;Updating jetty and including jetty-security -INT-1488;Add formatter plugin -INT-1271;Upgrade checkstyle to version 8.32 Google style -INT-1486;Upgrade latest dependencies - part 2 -INT-1575;Upgrade to latest dependencies -INT-1464;Add jetty-servlets as a dependency -INT-1574;Do not use alpha version of logback -INT-1540;Bump oparent master to 3.1.0 -INT-1539;Update INFO.yaml +INT-1756;Bump master to 3.2.0-SNAPSHOT diff --git a/docs/files/csv/release-pythonsdk-features.csv b/docs/files/csv/release-pythonsdk-features.csv new file mode 100644 index 000000000..934e3f71e --- /dev/null +++ b/docs/files/csv/release-pythonsdk-features.csv @@ -0,0 +1,27 @@ +Issue-ID;Description +INT-1894;[VMMACRO] Do not use multicloud when running basic_vm_macro +INT-1819;[CLAMP] raise exception in case of Policy or DCAE errors +INT-1842;Remove masspnfsimulator as a git submodule. +TEST-316;Basic VM macro +INT-1822;[CLAMP] Integrate basic_clamp +TEST-315;[SDC] Add a timer before certifying +INT-1822;Nodeport cleanup in basic_cds test +INT-1819;Wait 15 minutes for orchestration requests +INT-1822;Timeout on orchestration requests +INT-1829;[PNF] Extend timer before sending request to simulator +TEST-305;[SDC] Certify only when needed +TEST-303;Remove pvc from pnf simulator helm chart +TEST-302;Remove cert authent in basic_clamp +TEST-299;[SDC] Add manifest to basic packages +INT-1822;Add missing logger for Pnf Scenario +TEST-288;Wrapper for simulators +TEST-288;[TEST] Collect steps cleanup reports +TEST-292;[INT] Catch all onaptests and onapsdk exceptions for scenarios +TEST-280;Consider settings custom exceptions for reporting +TEST-278;Python SDK 7.4.0 compatibility +TEST-277;[TESTS] cleanup tests and remove reference to ubuntu16 +TEST-245;PNF simulator CNF instantiation and registation steps +TEST-245;Add basic_pnf entry point +INT-1796;[TESTCASE] fix basic_onboard scenario +TEST-245;[PythonSDK-tests] Add basic_onboard testcase +TEST-279;Fix steps time measurement diff --git a/docs/files/csv/release-testsuite-features.csv b/docs/files/csv/release-testsuite-features.csv index b9842892b..ec47327f4 100644 --- a/docs/files/csv/release-testsuite-features.csv +++ b/docs/files/csv/release-testsuite-features.csv @@ -1,50 +1,13 @@ Issue-ID;Description -INT-1788;Add Lasse as Integration committer -INT-1770;PNF registarte - prevent test case fail at general step Submit SDC Software Product -INT-1768;Create testsuite:1.7.2 for RC1 -INT-1240;ROBOT e2e test cases improvements for csar/zip tosca package validation in VNFSDK and SDC Onbarding -INT-1258;Adapt suite setup to wait until topic is created -INT-1761;Bump Guilin 1.7.1 testcase docker -INT-1258;Adapt events to have the newest 3GPP path in schemaReference -INT-1754;5gbulkpm k8splugin version update for sftp component -INT-1731;Add man in the middle CMPv2 E2E test case in robot test suite certificate SANs verification -INT-1258;Fix resource paths for xtesting purposes -INT-1258;Add Suite Setup to Ves Test Suite -INT-1258;Add E2E integration tests to cover new stndDefined VES events -INT-1716;Fix Docker file -SO-3295;Revert "Adding NSSMF adapter healthcheck in SO" -INT-1731;Add CMPv2 E2E test case in robot test suite -CCSDK-2768;Improved CDS Health Check -SO-3295;Adding NSSMF adapter healthcheck in SO -SDC-3287;Enhanced SDC Healthcheck -TEST-266;Self release 1.7.0 -INT-1727;Fix docker staging build -INT-1637;Synch csar download path for vCPE -INT-1726;Add entitlement pool type to entitlement_pool.jinja -INT-1636;Add customer to Load vCPE Models -INT-1724;Update INFO.yaml -SDNC-1349;SDNC new Healthcheck TC -OPTFRA-851;Modify healthcheck for OOF-OSDF -INT-1668;fix Get Service Id issue in pnf registartion test case in xtesting gating and master -AAI-3123;A&AI health check improvement: -INT-1668;Remove from CI PNF registration test that uses deprecated vnf api -POLICY-2817;fix delete policy test in new policy healthcheck -POLICY-2817;Missing Variable Added -POLICY-2817;Improved Policy Health Check -INT-1668;Remove from CI PNF registration test that uses deprecated vnf api -POLICY-2003;Remove deprecated Policy Component Tests -INT-1702;Add tcagen2 blueprint -INT-1702;Update dcaegen2 tca policy type -INT-1675;hv-ves bad certificate -INT-1678;5gBulkPm sftp bp fix -INT-1668;Fix randomly failing test case for pnf refgstarte -INT-1668;Fix randomly failing test case for pnf refgstarte in xtetsting daily -VFC-1712;Remove health-check related to components which are not in Guilin Release -TEST-248;run cds bootstrap before vlb_cds instantiation -INT-1652;Self release testsuite 1.6.4 -VFC-1689;[VFC]Remove ems driver health check -TEST-247;Some corrections for VLB CDS -TEST-234;change owning-entity-id from hard coded to variable -INT-1553;Release testsuite 1.6.3 for Frankfurt RC2 -INT-1521;Fix 5g_bulk_PM and pnf_registrate test case errors -INT-1604;Fix problem with SSL in HV-Ves automated test case +INT-1907;Close all browsers at healthcheck suite teardown +TEST-329;[HEALTCHECK] Exclude appc healthcheck test from CI +INT-1815;[ROBOT] Add new E2E test for 5G BulkPM usecase from HTTP server to ONAP gating +HOLMES-416;Re-introduced Holmes Health Check Tasks +DCAEGEN2-2660;Add DCAE Microservice Health Check +INT-1855 DCAEGEN2-2548;[ROBOT] Add DCAEMOD usecase test to xtestnig image +POLICY-3118;invalid controllerName in policies robot tests +INT-1855;Update DCAE MOD smoke/verification test using simple app +POLICY-2526;Updates to check for 202 response during Policy-PAP Deployment +INT-1855;Add DCAE MOD smoke/verification test using simple app +INT-1804;Fix CMPv2 test cases after pnf-simulator changes +INT-1801;Extend 5g bulk pm test suite in order to have possibility to test custom PM FILES diff --git a/docs/files/csv/repo-integration.csv b/docs/files/csv/repo-integration.csv index 688687034..a801ef27a 100644 --- a/docs/files/csv/repo-integration.csv +++ b/docs/files/csv/repo-integration.csv @@ -3,9 +3,12 @@ integration;Historical main repository including documentation, simulators (e.g. integration/csit;Repository hosting some tooling to start component functional tests in Jenkins (To be deprecated in Guilin as such tests must be reinsourced by the projects);`link <https://git.onap.org/integration/csit>`__ integration/docker/onap-java11;Java11 baseline image conformed to SECCOM recommendations;`link <https://git.onap.org/integration/docker/onap-java11>`__ integration/docker/onap-python;Python baseline image conformed to SECCOM recommendations;`link <https://git.onap.org/integration/docker/onap-python>`__ +**integration/ietf-actn-tools**;IETF ACTN tools introduced in Honolulu);`link <https://git.onap.org/integration/ietf-actn-tools>`__ integration/seccom;Repory hosting seccom recommended versions and security test waivers;`link <https://git.onap.org/integration/seccom>`__ -**integration/terraform**;**Terraform based alternative infrastructure installation**;`link <https://git.onap.org/integration/terraform>`__ -**integration/terragrunt**;**Compagnon repository of terraform**;`link <https://git.onap.org/integration/terragrunt>`__ +integration/terraform;Terraform based alternative infrastructure installation;`link <https://git.onap.org/integration/terraform>`__ +integration/terragrunt;Compagnon repository of terraform;`link <https://git.onap.org/integration/terragrunt>`__ integration/usecases/bbs;BBS use case introduced in Dublin and extracted from global repository in frankfurt;`link <https://git.onap.org/integration/usecases/bbs>`__ integration/usecases/mdons;MDONS use case introduced in Frankfurt;`link <https://git.onap.org/integration/usecases/mdons>`__ +**integration/usecases/A1-policy-enforcement**;A1 policy enforcement introduced in Honolulu;`link <https://git.onap.org/integration/usecases/A1-policy-enforcement>`__ +**integration/usecases/A1-policy-enforcement-r-apps**;A1 policy enforcement (analyticis part) introduced in Honolulu;`link <https://git.onap.org/integration/usecases/A1-policy-enforcement-r-apps>`__ integration/xtesting;Repository in charge to build th xtesting dockers used in CI/CD chains;`link <https://git.onap.org/integration/xtesting>`__ diff --git a/docs/files/csv/repo-simulators.csv b/docs/files/csv/repo-simulators.csv index 3dea7bf58..7320bc535 100644 --- a/docs/files/csv/repo-simulators.csv +++ b/docs/files/csv/repo-simulators.csv @@ -4,10 +4,10 @@ integration/simulators/dc-simulator;Data Center simulator;`link <https://git.ona integration/simulators/pnf-simulator;PNF Simulator;`link <https://git.onap.org/integration/simulators/pnf-simulator>`__ integration/simulators/ran-simulator;RAN simulator;`link <https://git.onap.org/integration/simulators/ran-simulator>`__ integration/simulators/masspnf-simulator;Mass PNF Simulator;`link <https://git.onap.org/integration/simulators/masspnf-simulator>`__ -integration/simulators/A1-policy-enforcement-simulator;A1 Policy Enforcement Simulator;`link <https://git.onap.org/integration/simulators/A1-policy-enforcement-simulator>`__ -integration/simulators/core-nssmf-simulator;Core NSSMF Simulator;`link <https://git.onap.org/integration/simulators/core-nssmf-simulator>`__ -integration/simulators/nf-simulator;NF simulator;`link <https://git.onap.org/integration/simulators/nf-simulator>`__ -integration/simulators/nf-simulator/avcn-manager;NF simulator avcn manager;`link <https://git.onap.org/integration/simulators/nf-simulator/avcn-manager>`__ -integration/simulators/nf-simulator/netconf-server;NF simulator netconf server;`link <https://git.onap.org/integration/simulators/nf-simulator/netconf-server>`__ -integration/simulators/nf-simulator/pm-https-server;NF simulator pm https server;`link <https://git.onap.org/integration/simulators/nf-simulator/pm-https-server>`__ -integration/simulators/nf-simulator/ves-client;NF simulator ves client;`link <https://git.onap.org/integration/simulators/nf-simulator/ves-client>`__ +**integration/simulators/A1-policy-enforcement-simulator**;A1 Policy Enforcement Simulator;`link <https://git.onap.org/integration/simulators/A1-policy-enforcement-simulator>`__ +**integration/simulators/core-nssmf-simulator**;Core NSSMF Simulator;`link <https://git.onap.org/integration/simulators/core-nssmf-simulator>`__ +**integration/simulators/nf-simulator**;NF simulator;`link <https://git.onap.org/integration/simulators/nf-simulator>`__ +**integration/simulators/nf-simulator/avcn-manager**;NF simulator avcn manager;`link <https://git.onap.org/integration/simulators/nf-simulator/avcn-manager>`__ +**integration/simulators/nf-simulator/netconf-server**;NF simulator netconf server;`link <https://git.onap.org/integration/simulators/nf-simulator/netconf-server>`__ +**integration/simulators/nf-simulator/pm-https-server**;NF simulator pm https server;`link <https://git.onap.org/integration/simulators/nf-simulator/pm-https-server>`__ +**integration/simulators/nf-simulator/ves-client**;NF simulator ves client;`link <https://git.onap.org/integration/simulators/nf-simulator/ves-client>`__ diff --git a/docs/files/csv/repo-testsuite.csv b/docs/files/csv/repo-testsuite.csv index a00d8af03..67fc345c7 100644 --- a/docs/files/csv/repo-testsuite.csv +++ b/docs/files/csv/repo-testsuite.csv @@ -1,8 +1,11 @@ Repository;Description;Link testsuite;repository hosting the robot test suites;`link <https://git.onap.org/testsuite>`__ -**testsuite/cds**;**Repository hosting (standalone) CDS test suites shared by Bell Canada team, not yet integrated in CI/CD**;`link <https://git.onap.org/testsuite/cds>`__ +testsuite/cds;Repository hosting (standalone) CDS test suites shared by Bell Canada team, not yet integrated in CI/CD;`link <https://git.onap.org/testsuite/cds>`__ +**testsuite/cds-mock-odl**;needed for cds regression tests;`link <https://git.onap.org/testsuite/cds-mock-odl>`__ +**testsuite/cds-mock-server**;needed for cds regression tests;`link <https://git.onap.org/testsuite/cds-mock-server>`__ +**testsuite/cds-mock-ssh**;needed for cds regression tests;`link <https://git.onap.org/testsuite/cds-mock-ssh>`__ testsuite/heatbridge;python utils to manage the heatbridge function to enrich cloud information to AAI (deprecated);`link <https://git.onap.org/testsuite/heatbridge>`__ testsuite/oom;Helm chart for robot pod (to be deprecated in Honolulu and moved back to OOM);`link <https://git.onap.org/testsuite/oom>`__ testsuite/python-testing-utils;Python and robot util libraries used for robot tests;`link <https://git.onap.org/testsuite/python-testing-utils>`__ -**testsuite/pythonsdk-tests**;**Repository hosting the test scenarios leveraging python-onapsdk for end to end smoke tests**;`link <https://git.onap.org/testsuite/pythonsdk-tests>`__ -**testsuite/robot-utils**;**Repository aiming to provide a robot wrapper for python-onapsdk**;`link <https://git.onap.org/testsuite/robot-utils>`__ +testsuite/pythonsdk-tests;Repository hosting the test scenarios leveraging python-onapsdk for end to end smoke tests;`link <https://git.onap.org/testsuite/pythonsdk-tests>`__ +testsuite/robot-utils;Repository aiming to provide a robot wrapper for python-onapsdk;`link <https://git.onap.org/testsuite/robot-utils>`__ diff --git a/docs/files/csv/stability_cluster_metric_cpu.csv b/docs/files/csv/stability_cluster_metric_cpu.csv index 9259086ef..e77d61691 100644 --- a/docs/files/csv/stability_cluster_metric_cpu.csv +++ b/docs/files/csv/stability_cluster_metric_cpu.csv @@ -1,2 +1,2 @@ -Namespace;Pods;Workloads;Memory Usage;CPU Requests;CPU Requests %;CPU Limits;CPU Limits % -onap;242;181;10.31;79.93;13%;247.2;4% +Namespace;CPU Utilisation (from requests);CPU utilisation (from limits);Memory Utilisation (from requests);Memory Utilisation (from limits) +onap;2.22%;0.816%;19%;7.4% diff --git a/docs/files/csv/stability_top10_cpu.csv b/docs/files/csv/stability_top10_cpu.csv new file mode 100644 index 000000000..cdd93c1a9 --- /dev/null +++ b/docs/files/csv/stability_top10_cpu.csv @@ -0,0 +1,11 @@ +Pod;CPU Usage;CPU Requests;CPU Request %;CPU Limits;CPU Limits % +onap-robot;0.92;1;92%;4;23% +onap-cassandra-0;0.4;0.4;101%;1.6;25% +onap-cassandra-2;0.36;0.4;83%;1.6;22% +onap-ejbca;0.35;0.02;1771%;3;11% +onap-aws;0.35;6;6%;; +onap-cassandra-1;0.33;0.4;83%;1.6;21% +onap-oof-has-api;0.12;2;6%;8;1% +onap-music-cassandra-2;0.12;0.32;32%;1;12% +onap-dcae-cloudify-manager;0.11;2;6%;4;3% +onap-music-cassandra-1;0.09;0.32;29%;1;9% diff --git a/docs/files/csv/stability_top10_memory.csv b/docs/files/csv/stability_top10_memory.csv index 127d717ae..504afc2ac 100644 --- a/docs/files/csv/stability_top10_memory.csv +++ b/docs/files/csv/stability_top10_memory.csv @@ -1,11 +1,11 @@ Pod;Memory Usage;Memory Requests;Memory Requests %;Memory Limits;Memory Limits % -onap-sdnc-0;5.56 GiB;2 Gi;278%;4 GiB;139% -onap-portal-cassandra;5.5 GiB;2.8 GiB;160%;3.75 GiB;146% -onap-appc;5.28 GiB;2 GiB;264%;4 GiB; 132% -onap-cassandra-1;4.7 GiB;2.5 GiB;188%;4 GiB;117% -onap-cassandra-2;4.7 GiB;2.5 GiB;188%;4 GiB;117% -onap-cassandra-3;4.7 GiB;2.5 GiB;188%;4 GiB;117% +onap-portal-cassandra;3.34 GiB;6.2 GiB;53.8%;7.5 GiB;45% +onap-cassandra-2;2.6 GiB;5 GiB;52%;8 GiB;32% +onap-cassandra-0;2.6 GiB;5 GiB;52%;8 GiB;32% +onap-cassandra-1;2.54 GiB;5 GiB;51%;8 GiB;32% +onap-appc;2.46 GiB;4 GiB;62%;8 GiB; 32% +onap-sdnc;2.43 GiB;4 GiB;61%;8 GiB; 30% +onap-policy-mariadb-0;2.4 GiB;1.96 GiB;122%;5.5 GiB;44% onap-dcae-cloudify-manager;4.7 GiB;2 GiB;233%;4 GiB;115% -onap-clamp-dash-es;3.57 GiB; 2.5 GiB;143%;4 GiB;89% -onap-so-bpmn-infra;3.51 GiB;1 GiB; 351%;4 GiB;88% -onap-awx;3.21 GiB;6 GiB;53%;; +onap-awx;1.72 GiB;12 GiB;14%;; +onap-aaf-cass;1.45 GiB;2.5 GiB;58%;3.5 GiB;41.4% diff --git a/docs/files/csv/stability_top10_net.csv b/docs/files/csv/stability_top10_net.csv new file mode 100644 index 000000000..b86ba909f --- /dev/null +++ b/docs/files/csv/stability_top10_net.csv @@ -0,0 +1,11 @@ +Pod;Current Receive Bandwidth;Current Transmit Bandwidth;Rate of Received Packets;Rate of Transmitted Packets, Rate of Received Packets Dropped;Rate of Transmitted Packets Dropped +onap-oof-has-api;372 kB/s;670 B/s;9.21 p/s;5.4 p/s;0 p/s;0 p/s +onap-cassandra-2;231 kB/s;155 kB/s;90.35 p/s;69.84 p/s;0 p/s;0 p/s +onap-cassandra-1;228 kB/s;156 kB/s;87 p/s;82 p/s;0 p/s;0 p/s +onap-cassandra-0;144 kB/s;245 kB/s;63 p/s;75 p/s;0 p/s;0 p/s +onap-message-router-0;17 kB/s;18 kB/s;187 p/s;188 p/s;0 p/s;0 p/s +onap-portal-app;15 kB/s;4.7 kB/s;187 p/s;188 p/s;0 p/s;0 p/s +onap-consul;14 kB/s;3.9 kB/s;38 p/s;40 p/s;0 p/s;0 p/s +onap-message-router-kafka-010kB/s;10 kB/s;112 p/s;115 p/s;0 p/s;0 p/s +onap-dcaemodul-onboarding-apt_install;6.7 kB/s;6.1 kB/s;45 p/s;47 p/s;0 p/s;0 p/s +onap-message-router-kafka-2;6.3 kB/s;6.1 kB/s;70 p/s;72 p/s;0 p/s;0 p/s diff --git a/docs/files/csv/tests-healthcheck.csv b/docs/files/csv/tests-healthcheck.csv index 51eddde63..8071c4345 100644 --- a/docs/files/csv/tests-healthcheck.csv +++ b/docs/files/csv/tests-healthcheck.csv @@ -1,8 +1,10 @@ Tests;Description;Code;Comments core;Robot healthcheck tests of the core components (AA&I, DMAAP, Portal, SDC, SDNC, SO);`robot tests <https://git.onap.org/testsuite/tree/robot/testsuites/health-check.robot>`__;`robotframework <https://robotframework.org/>`__ -full;Robot healthcheck tests for all the components;`robot tests <https://git.onap.org/testsuite/tree/robot/testsuites/health-check.robot>`__;`robotframework <https://robotframework.org/>`__ +full;Robot healthcheck tests for all the components, **holmes healthcheck** have been reintroduced;`robot tests <https://git.onap.org/testsuite/tree/robot/testsuites/health-check.robot>`__;`robotframework <https://robotframework.org/>`__ healthdist;Check the onboarding and distribution of the vFW;`robot tests <https://git.onap.org/testsuite/tree/robot/testsuites/health-check.robot>`__;`robotframework <https://robotframework.org/>`__ postinstall;Check dmaap and AA&I Design model DB tests;`robot tests <https://git.onap.org/testsuite/tree/robot/testsuites/post-install-tests.robot>`__;`robotframework <https://robotframework.org/>`__ -cmpv2 (new);CMPv2 Usecase functionality;`code <https://git.onap.org/testsuite/tree/robot/testsuites/cmpv2.robot>`__;`robotframework <https://robotframework.org/>`__ ves-collector (new);Suite for checking handling events by VES Collector;`code <https://git.onap.org/testsuite/tree/robot/testsuites/ves.robot>`__;`robotframework <https://robotframework.org/>`__ hv-ves;HV-VES 'Sunny Scenario' Robot Framework test - message is sent to the collector and Kafka topic is checked if the message has been published. Content is decoded and checked.;`code <https://git.onap.org/testsuite/tree/robot/testsuites/hvves-ci.robot>`__;`robotframework <https://robotframework.org/>`__ +**basic_cds**;CDS blueprint enrichment, open a nodeport on CDS then enrich CDS CBA;`code <https://git.onap.org/testsuite/pythonsdk-tests/tree/src/onaptests/scenario/cds_blueprint_enrichment.py>`__;`onap_pythonsdk <https://gitlab.com/Orange-OpenSource/lfn/onap/python-onapsdk>`__, `pythonsdk-tests <https://git.onap.org/testsuite/pythonsdk-tests/tree/>`__ +**basic_onboard**;onboard a model, subset of most of the other basic_* tests, created to perform stability testing;`code <https://git.onap.org/testsuite/pythonsdk-tests/tree/src/onaptests/scenario/basic_onboard.py>`__;`onap_pythonsdk <https://gitlab.com/Orange-OpenSource/lfn/onap/python-onapsdk>`__, `pythonsdk-tests <https://git.onap.org/testsuite/pythonsdk-tests/tree/>`__ +**dcaemod**;Check new DCAEmod;`robot tests <https://git.onap.org/testsuite/tree/robot/testsuites/dcaemod.robot>`__;`robotframework <https://robotframework.org/>`__ diff --git a/docs/files/csv/tests-infrastructure-healthcheck.csv b/docs/files/csv/tests-infrastructure-healthcheck.csv index 43e4b139e..a61e02991 100644 --- a/docs/files/csv/tests-infrastructure-healthcheck.csv +++ b/docs/files/csv/tests-infrastructure-healthcheck.csv @@ -1,4 +1,5 @@ Tests;Description;Code;Comments -onap-helm;Verify Helm chart status;`python module <https://gitlab.com/Orange-OpenSource/lfn/tools/helm-onap-status>`__; +onap-helm;Verify Helm chart status, the test has been updated to take into account Helm3;`python module <https://gitlab.com/Orange-OpenSource/lfn/tools/helm-onap-status>`__; onap-k8s;Check common resources of the ONAP Kubernetes namespace ;`python module <https://gitlab.com/Orange-OpenSource/lfn/tools/kubernetes-status>`__;kubernetes python library nodeport_check_certs;This test list the nodeports and tries to get SSL information to evaluate the validity of the certificates (expiration and issuer) used on the nodeports;`python module <https://git.onap.org/integration/tree/test/security/check_certificates>`__;pyopenssl, kubernetes python libraries +**internal_check_certs**;This test list the internal ports and tries to get SSL information to evaluate the validity of the certificates (expiration and issuer) used ;`python module <https://git.onap.org/integration/tree/test/security/check_certificates>`__;pyopenssl, kubernetes python libraries diff --git a/docs/files/csv/tests-security.csv b/docs/files/csv/tests-security.csv index 07e05d0ba..4e88b0ada 100644 --- a/docs/files/csv/tests-security.csv +++ b/docs/files/csv/tests-security.csv @@ -7,3 +7,4 @@ http_public_endpoints;check that there is no public http endpoints exposed in ON jdpw_ports;check that there are no internal java ports;`bash script <https://git.onap.org/integration/plain/test/security/check_for_jdwp.sh>`__;kubectl, procfs kube_hunter;security suite to search k8s vulnerabilities (upstream src aquasecurity);`kube-Hunter <https://github.com/aquasecurity/kube-hunter>`__; `kube-Hunter <https://github.com/aquasecurity/kube-hunter>`__ versions;check that Java and Python are available only in versions recommended by SECCOM. This test is long and run only in Weekly CI chains;`python module <https://git.onap.org/integration/tree/test/security/check_versions>`__;cerberus, kubernetes python lib, +**tern**;Check the component licenses within the ONAP dockers;`bash script <https://gitlab.com/Orange-OpenSource/lfn/onap/xtesting-onap/-/blob/master/scripts/run_tern.sh>`__;kubectl diff --git a/docs/files/csv/tests-smoke.csv b/docs/files/csv/tests-smoke.csv index a83b5375c..b990624c1 100644 --- a/docs/files/csv/tests-smoke.csv +++ b/docs/files/csv/tests-smoke.csv @@ -1,6 +1,9 @@ Tests;Description;Code;Comments basic_vm;Onboard, distribute and instantiate an Openstack VM using à la carte BPMN, replaced the former basic_vm test;`code <https://git.onap.org/testsuite/pythonsdk-tests/tree/src/onaptests/scenario/basic_vm.py>`__;`onap_pythonsdk <https://gitlab.com/Orange-OpenSource/lfn/onap/python-onapsdk>`__, `pythonsdk-tests <https://git.onap.org/testsuite/pythonsdk-tests/tree/>`__ -basic_network (new);Onboard, distribute and instantiate a Neutron network;`code <https://git.onap.org/testsuite/pythonsdk-tests/tree/src/onaptests/scenario/basic_network.py>`__;`onap_pythonsdk <https://gitlab.com/Orange-OpenSource/lfn/onap/python-onapsdk>`__, `pythonsdk-tests <https://git.onap.org/testsuite/pythonsdk-tests/tree/>`__ +basic_network;Onboard, distribute and instantiate a Neutron network;`code <https://git.onap.org/testsuite/pythonsdk-tests/tree/src/onaptests/scenario/basic_network.py>`__;`onap_pythonsdk <https://gitlab.com/Orange-OpenSource/lfn/onap/python-onapsdk>`__, `pythonsdk-tests <https://git.onap.org/testsuite/pythonsdk-tests/tree/>`__ basic_cnf;Onboard (new), distribute and instantiate a Kubernetes pods;`code <https://git.onap.org/testsuite/pythonsdk-tests/tree/src/onaptests/scenario/basic_cnf.py>`__;`onap_pythonsdk <https://gitlab.com/Orange-OpenSource/lfn/onap/python-onapsdk>`__, `pythonsdk-tests <https://git.onap.org/testsuite/pythonsdk-tests/tree/>`__ -5gbulkpm;5G Bulk PM Usecase functionality;`code <https://git.onap.org/testsuite/tree/robot/testsuites/usecases/5gbulkpm.robot>`__;`robotframework <https://robotframework.org/>`__ +**5gbulkpm**;5G Bulk PM Usecase functionality. The test has been significantly enhanced in Honolulu;`code <https://git.onap.org/testsuite/tree/robot/testsuites/usecases/5gbulkpm.robot>`__;`robotframework <https://robotframework.org/>`__ pnf-registrate;Executes the PNF registration test cases including setup and teardown;`code <https://git.onap.org/testsuite/tree/robot/testsuites/pnf-registration.robot>`__;`robotframework <https://robotframework.org/>`__ +cmpv2;CMPv2 Usecase functionality;`code <https://git.onap.org/testsuite/tree/robot/testsuites/cmpv2.robot>`__;`robotframework <https://robotframework.org/>`__ +**basic_clamp**;distribute a model, enrich it with a tca blueprint, call Clamp to create a loop then instantiate it in Policy and DCAE;`code <https://git.onap.org/testsuite/pythonsdk-tests/tree/src/onaptests/scenario/basic_clamp.py>`__;`onap_pythonsdk <https://gitlab.com/Orange-OpenSource/lfn/onap/python-onapsdk>`__, `pythonsdk-tests <https://git.onap.org/testsuite/pythonsdk-tests/tree/>`__ +**basic_vm_macro**;Instantiate a VM using macro bpmn;`code <https://git.onap.org/testsuite/pythonsdk-tests/tree/src/onaptests/scenario/basic_vm_macro.py>`__;`onap_pythonsdk <https://gitlab.com/Orange-OpenSource/lfn/onap/python-onapsdk>`__, `pythonsdk-tests <https://git.onap.org/testsuite/pythonsdk-tests/tree/>`__ diff --git a/docs/files/csv/usecases-deprecated.csv b/docs/files/csv/usecases-deprecated.csv index 7b84dd4ab..d5066cfea 100644 --- a/docs/files/csv/usecases-deprecated.csv +++ b/docs/files/csv/usecases-deprecated.csv @@ -1,4 +1,6 @@ Use Case;Link;Last Valid Version;Comments +vFirewall with closed loop;:ref:`official doc <docs_vfw>`;Guilin;Shall still be OK in Honolulu but not tested yet +Scale Out;:ref:`official doc <docs_scaleout>`;Guilin;Shall still be OK in Honolulu but not tested yet vCPE Use Case;:ref:`official doc <docs_vcpe>`;El Alto;No resources to test on Frankfurt vIPsec with HPA Use Case;:ref:`official doc<docs_vipsec_hpa>`;El Alto;No resources to test on Frankfurt Change Management Schedule Optimization;:ref:`official doc<docs_CM_schedule_optimizer>`;El Alto;No resources to test on Frankfurt diff --git a/docs/files/csv/usecases-functional-requirements.csv b/docs/files/csv/usecases-functional-requirements.csv index e7fb7f4ba..908db9578 100644 --- a/docs/files/csv/usecases-functional-requirements.csv +++ b/docs/files/csv/usecases-functional-requirements.csv @@ -1,16 +1,9 @@ Issue key;Summary;Contact;Comment -REQ-381;Bulk PM / PM Data Control Extension;Mark Scott; -REQ-352;Extend ORAN A1 Adapter and add A1 Policy Management;John Keeney; -REQ-341;ONAP CNF orchestration - Enhancements;Lukasz Rajewski;Changes of this feature are described in :ref:`vFW CNF use case <docs_vFW_CNF_CDS>` -REQ-338;Support for Test Result Auto Analysis & Certification （NFV Testing Automatic Platform）;Lei Huang; -REQ-337;Support for Test Task Auto Execution （NFV Testing Automatic Platform）;Lei Huang; -REQ-336;Support for Test Environment Auto Deploy（NFV Testing Automatic Platform）;Lei Huang; -REQ-335;Support for Test Topology Auto Design （NFV Testing Automatic Platform）;Lei Huang; -REQ-334;ETSI-Alignment for Guilin and Honolulu;Byung-Woo Jun; -REQ-331;Policy Based Filtering;Pam Dragosh; -REQ-327;ONAP/3GPP & O-RAN Alignment-Standards Defined Notifications over VES (Guilin);Marge Hillis; -REQ-324;Support xNF Software Upgrade in association to schema updates;Zu Qiang; -REQ-322;Configuration & Persistency Service in R7;Tony Finnerty; -REQ-321;CMPv2 Enhancements for R7;Pawel Baniewski;cmpv2 automated test integrated in CI/CD, see :ref:`automated test page <release_automated_usecases>` -REQ-319;PNF PreOnboarding in R7;Ben Cheung; -REQ-318;PNF Plug & Play in R7;Ben Cheung; +REQ-463;ONAP to support Multi Tenancy (part 2);Olivier Phénix ; +REQ-457;Extend ORAN A1 Adapter and add A1 Policy Management;John Keeney; +REQ-458;ONAP CNF orchestration - Enhancements;Lukasz Rajewski;Changes of this feature are described in :ref:`vFW CNF use case <docs_vFW_CNF_CDS>` +REQ-446;NF Software Upgrade enhancement;Zu Qiang; +REQ-433;ONAP/3GPP & O-RAN Alignment-Standards Defined Notifications over VES (Honolulu);Damian Nowak; +REQ-432;IPv4/IPv6 dual stack support in ONAP (Honolulu);Damian Nowak, +REQ-431;CMPv2 Enhancements for R7;Pawel Baniewski;cmpv2 automated test integrated in CI/CD, see :ref:`automated test page <release_automated_usecases>` +REQ-400;ETSI-Alignment for Guilin and Honolulu;Byung-Woo Jun; diff --git a/docs/files/csv/usecases-non-functional-requirements.csv b/docs/files/csv/usecases-non-functional-requirements.csv index 82f4c1614..871efa8bd 100644 --- a/docs/files/csv/usecases-non-functional-requirements.csv +++ b/docs/files/csv/usecases-non-functional-requirements.csv @@ -1,18 +1,5 @@ Issue key;Summary;Contact;Comment -REQ-386;Apply common Swagger style and documentation generation tools to create robust ONAP API documentation;Andy Mayer; -REQ-385;IPv4/IPv6 dual stack support in ONAP (Guilin);Damian Nowak; -REQ-376;Flow management must be activated for ONAP.;Fabian Rouzaut;Marginal progress, it was not possible to collect the flow to setup a matrix flow -REQ-374;ONAP shall use STDOUT for logs collection;Krzysztof Opasiak; Most components (that are not in maintenance mode) are using now stdout for logs collection. We hope to finish this in next release. -REQ-373;ONAP must complete update of the Python language (from 2.7 -> 3.8);Pawel Pawlak;Significant progress done in Guilin tracked by weekly version test integrated in CI/CD -REQ-366;Containers must crash properly when a failure occurs;Sylvain Desbureaux;Good progress to be completed in Honolulu -REQ-365;Containers must have no more than one main process;Sylvain Desbureaux; -REQ-362;All containers must run as non-root user;Krzysztof Opasiak;The list of waivers has been significantly reduced in Guilin, progress tracker with root test integrated in all CI/CD chains -REQ-361;Continue hardcoded passwords removal;Krzysztof Opasiak; Majority of projects use certInitializer for certificate generation. Next subset of passwords removed. -REQ-360;Application config should be fully prepared before starting the application container;Sylvain Desbureaux; -REQ-358;No root (superuser) access to database from application container;Krzysztof Opasiak; Nice progress but still some of projects access DB as root. -REQ-357;ONAP shall increase the number of Docker Benchmark tests;Sylvain Desbureaux;More tests integrated in CI/CD but enhancements expected in Honolulu -REQ-351;ONAP must complete update of the java language (from v8 -> v11) ;Pawel Pawlak;Significant progress done in Guilin tracked by weekly version test integrated in CI/CD -REQ-350;Each ONAP project shall improve its CII Badging score by improving input validation and documenting it in their CII Badging site ;Tony L Hansen; -REQ-349;Each ONAP project shall define code coverage improvements and achieve at least 55% code coverage;Amy Zwarico; -REQ-340;ONAP to support Multi - tenancy ;Olivier Phenix; -REQ-323;Each project will update the vulnerable direct dependencies in their code base;Amy Zwarico; +REQ-453;Smart Operator Intent Translation in UUI based on IBN - R8 5G Slicing Support;Dong Wang; +REQ-430;PNF Plug & Play in R8;Damian Nowak; +REQ-428;5G Service Modeling in R8 - Modeling Work;Benjamin Cheung; +REQ-427;Configuration Persistence Service in R8;Toine Siebelink; diff --git a/docs/files/csv/usecases-old-valid.csv b/docs/files/csv/usecases-old-valid.csv index aa9ecbcef..c10709e9d 100644 --- a/docs/files/csv/usecases-old-valid.csv +++ b/docs/files/csv/usecases-old-valid.csv @@ -1,7 +1,6 @@ Summary;Link;Contacts vFirewall CNF With CDS;:ref:`official doc <docs_vFW_CNF_CDS>`;L.Rajewski, K.Banka -vFirewall with closed loop;:ref:`official doc <docs_vfw>`;B.Freeman, C.Closet, S.Déterme -Scale Out;:ref:`official doc <docs_scaleout>`;B.Freeman, C.Closet, S.Déterme 5G Realtime PM and High Volume Stream Data Collection; :ref:`official doc <docs_realtime_pm>`;M.Przybysz 5G PNF Plug and Play; :ref:`official doc <docs_5g_pnf_pnp>`; M.Przybysz K.Kuzmicki 5G PNF Pre-Onboarding & Onboarding;:ref:`official doc <docs_pnf_onboarding_preonboarding>`;M.Przybysz K.Kuzmicki D.Melia A.Walshe +MDONS extension;:ref:`official doc <docs_CCVPN>`;X.Miao diff --git a/docs/files/csv/usecases.csv b/docs/files/csv/usecases.csv index da4149daa..429d2d394 100644 --- a/docs/files/csv/usecases.csv +++ b/docs/files/csv/usecases.csv @@ -1,5 +1,4 @@ Ref;Summary;Link;Contacts -REQ-342;E2E Network Slicing;:ref:`official doc <docs_E2E_network_slicing>`;Swaminathan S -REQ-345;5G OOF SON;:ref:`official doc <docs_5G_oof_pci>`;Reshmasree c -REQ-347;CCVPN-Transport Slicing;:ref:`official doc <docs_ccvpn>`;X.Miao,G.Agrawal,Shashikanth VH -REQ-325;MDONS extension;:ref:`official doc <docs_CCVPN>`;X.Miao +REQ-440;E2E Network Slicing;:ref:`official doc <docs_E2E_network_slicing>`;Lin Meng +REQ-429;5G OOF SON;:ref:`official doc <docs_5G_oof_pci>`;Swaminathan Seetharaman +REQ-459;CCVPN-Transport Slicing;:ref:`official doc <docs_ccvpn>`;Henry Yu diff --git a/docs/files/integration-project.png b/docs/files/integration-project.png Binary files differdeleted file mode 100644 index d590c842f..000000000 --- a/docs/files/integration-project.png +++ /dev/null diff --git a/docs/files/s3p/honolulu_daily_healthcheck.png b/docs/files/s3p/honolulu_daily_healthcheck.png Binary files differnew file mode 100644 index 000000000..01216aee4 --- /dev/null +++ b/docs/files/s3p/honolulu_daily_healthcheck.png diff --git a/docs/files/s3p/honolulu_daily_infrastructure_healthcheck.png b/docs/files/s3p/honolulu_daily_infrastructure_healthcheck.png Binary files differnew file mode 100644 index 000000000..660902029 --- /dev/null +++ b/docs/files/s3p/honolulu_daily_infrastructure_healthcheck.png diff --git a/docs/files/s3p/honolulu_daily_security.png b/docs/files/s3p/honolulu_daily_security.png Binary files differnew file mode 100644 index 000000000..2efc9c84a --- /dev/null +++ b/docs/files/s3p/honolulu_daily_security.png diff --git a/docs/files/s3p/honolulu_daily_smoke.png b/docs/files/s3p/honolulu_daily_smoke.png Binary files differnew file mode 100644 index 000000000..4192e404d --- /dev/null +++ b/docs/files/s3p/honolulu_daily_smoke.png diff --git a/docs/files/s3p/honolulu_sdc_stability.png b/docs/files/s3p/honolulu_sdc_stability.png Binary files differnew file mode 100644 index 000000000..4d6c4ee2c --- /dev/null +++ b/docs/files/s3p/honolulu_sdc_stability.png diff --git a/docs/files/s3p/honolulu_sdc_stability_resources.png b/docs/files/s3p/honolulu_sdc_stability_resources.png Binary files differnew file mode 100644 index 000000000..dd9333687 --- /dev/null +++ b/docs/files/s3p/honolulu_sdc_stability_resources.png diff --git a/docs/files/s3p/honolulu_so_stability_1_duration.png b/docs/files/s3p/honolulu_so_stability_1_duration.png Binary files differnew file mode 100644 index 000000000..47f625604 --- /dev/null +++ b/docs/files/s3p/honolulu_so_stability_1_duration.png diff --git a/docs/files/s3p/honolulu_so_stability_5.png b/docs/files/s3p/honolulu_so_stability_5.png Binary files differnew file mode 100644 index 000000000..fe8487524 --- /dev/null +++ b/docs/files/s3p/honolulu_so_stability_5.png diff --git a/docs/files/s3p/honolulu_weekly_cpu.png b/docs/files/s3p/honolulu_weekly_cpu.png Binary files differnew file mode 100644 index 000000000..dbf55d272 --- /dev/null +++ b/docs/files/s3p/honolulu_weekly_cpu.png diff --git a/docs/files/s3p/honolulu_weekly_memory.png b/docs/files/s3p/honolulu_weekly_memory.png Binary files differnew file mode 100644 index 000000000..5cbdf04be --- /dev/null +++ b/docs/files/s3p/honolulu_weekly_memory.png diff --git a/docs/index.rst b/docs/index.rst index 9e2a89cbb..eb10f12a8 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -7,12 +7,7 @@ INTEGRATION .. toctree:: :maxdepth: 1 - integration-missions.rst release-notes.rst - integration-repositories.rst + docs_usecases_release.rst integration-resources.rst - integration-labs.rst - integration-CICD.rst - integration-tests.rst - integration-simulators.rst - integration-tooling.rst + integration-missions.rst diff --git a/docs/integration-CICD.rst b/docs/integration-CICD.rst index f7d9f0790..f55bb35ef 100644 --- a/docs/integration-CICD.rst +++ b/docs/integration-CICD.rst @@ -10,16 +10,16 @@ CI/CD .. important:: Integration team deals with 2 different CI/CD systems. - - jenkins CI/CD, CI managed by LF IT and CD by Integration team - - gitlab-ci managed by Integration and OOM team + - Jenkins CI/CD, CI managed by LF IT and CD by Integration team + - GitLab-ci managed by Integration and OOM team Continuous Integration ---------------------- The CI part provides the following features: -- Repository Verification (format of the INFO.yaml) -- Patchset verification thank to json/yaml/python/go/rst/md linters. These Jenkins +- Repository verification (format of the INFO.yaml) +- Patchset verification thanks to json/yaml/python/go/rst/md linters. These Jenkins verification jobs are hosted in the ci-management repository. They can vote +1/-1 on patchset submission. Integration team systematically enables linters on any new repository @@ -35,7 +35,7 @@ The different verification chains are defined in https://jenkins.onap.org/: - integration-terragrunt: https://jenkins.onap.org/view/integration-terragrunt/ - testsuite-robot-utils: https://jenkins.onap.org/view/testsuite-robot-utils/ -The jenkins jobs (jjb) are hosted in https://git.onap.org/ci-management/. +The Jenkins jobs (jjb) are hosted in https://git.onap.org/ci-management/. Continuous Deployment --------------------- @@ -60,21 +60,22 @@ The Jenkins interface was however used to launch the installation of SB-00. This Jenkins script is leveraging resources available in OOM and integration repositories. -The replacement of this CD by a gitlab runner based CD to unify the CD +The replacement of this CD by a GitLab runner based CD to unify the CD management was planned, but finalizing the operation in Guilin was not possible due to performance issues. -Gitlab CD +GitLab CD ......... This CD is leveraging public gitlab-ci mechanism and used to deploy several ONAP labs: - Daily Master: daily run using OOM Master -- Daily Frankfurt: daily run using the last stable version during Guilin Release +- Daily Guilin: daily run using the last stable version during Honolulu Release processing -- Daily Guilin: daily run setup at RC0 (candidate dockers available for integration) +- Daily Honolulu: daily run setup at RC0 (candidate dockers available for integration) - Weekly Master: run once a week with longer tests +- Weekly Honolulu: run once a week with longer tests - Gating: run on OOM, clamp or SO patchset submission. It means a full ONAP deployment on demand based on new patchset declared in gerrit. diff --git a/docs/integration-labs.rst b/docs/integration-labs.rst index fe6fc0f4f..cdf2d7f89 100644 --- a/docs/integration-labs.rst +++ b/docs/integration-labs.rst @@ -4,16 +4,17 @@ .. integration_main-doc: -ONAP Integration Labs -===================== +Integration Labs +================ .. important:: The Integration team deals with several community labs: - The Windriver/Intel lab - The Azure staging lab - - The Continuous Deployment (CD) Labs - The Orange openlab + - The DT lab + - The Nokia dualstack lab Additionally integration contributors may deal with their own lab pushing results in the integration portal (See DT http://testresults.opnfv.org/onap-integration/dt/dt.html) @@ -24,23 +25,16 @@ Windriver/Intel lab The Historical Community Lab ............................ -This lab is the historical lab of ONAP integration team based on Openstack Ocata. -During Guilin, it has been audited and performance issues have been reported -(RAM and CPU). -A reinstallation has to be planned. +This lab is the historical lab of ONAP integration team based on OpenStack Ocata. -The figure hereafter shows all the ONAP project consuming Windriver/Intel lab +The figure hereafter shows all the ONAP projects consuming Windriver/Intel lab resources (April 2020). .. figure:: files/windriver/windriver_servers.png :align: center -In order to avoid disturbing the projects, the reinstallation has been postponed -after Guilin. -A huge cleanup has been done in order to save the resources. -The historical CI/CD chains based on a stand-alone Jenkins VM hosted in Windriver -have been stopped. For Guilin only SB-00 has been kept and re-installed for the -use case support. +This lab is mainly used by the projects for the development. +A staging lab named SB-00 is also available. If you want to use this lab, you need a VPN access. The procedure is described in the `wiki <https://wiki.onap.org/pages/viewpage.action?pageId=29787070>`_. @@ -68,9 +62,21 @@ Orange Openlab -------------- This lab is for community use. It is always provided with the last stable version, -i.e. Frankfurt release during Guilin development time. +i.e. Guilin release during Honolulu development time. Please note that such labs do not provide admin rights and is shared with all the users. It can be used to discover ONAP. See `Orange Openlab access procedure <https://wiki.onap.org/display/DW/Orange+OpenLab>`_ for details. + +DT lab +------ + +The DT lab reported Master daily results in addition of Guilin daily results. + +Nokia lab +--------- + +Nokia setup a lab to support the dual stack IPv4/IPv6 tests. Results are shared +with the community in +`<https://logs.onap.org/onap-integration/daily/onap_daily_nokia_dualstack_master/>`_ diff --git a/docs/integration-missions.rst b/docs/integration-missions.rst index 2cf0b474e..421519859 100644 --- a/docs/integration-missions.rst +++ b/docs/integration-missions.rst @@ -2,7 +2,7 @@ Creative Commons Attribution 4.0 International License. .. _integration-missions: -Integration missions +Integration Missions ==================== .. important:: @@ -17,20 +17,26 @@ Integration missions The different activities may be summarized as follows (proportions are indicative): -.. figure:: files/integration-project.png - +- Community support +- Lab support +- Use case support +- Test development +- Management of daily/weekly CI chains +- Build baseline images +- Automate tests +- Validate the release For each release, the integration team provides the following artifacts: - A daily CI chain corresponding to the release - Staging labs to perform the pairwise testing (when not automated) and support the use case teams -- Baseline JAVA and Python images +- Baseline Java and Python images - oparent library to manage Java dependencies - Test suites and tools to check the various ONAP components - Use-case documentation and artifacts - A testsuite docker included in the ONAP cluster to execute the robot based tests -- Configuration files (scripts, Heat templates, csa files) to help installing +- Configuration files (scripts, Heat templates, CSAR files) to help installing and testing ONAP - Wiki release follow-up tables (blocking points, docker versions,...) diff --git a/docs/integration-repositories.rst b/docs/integration-repositories.rst index ac40067be..d65a06d2e 100644 --- a/docs/integration-repositories.rst +++ b/docs/integration-repositories.rst @@ -19,22 +19,37 @@ Most of the repositories are internal ONAP repositories. │  ├── docker │  │  ├── onap-java11 │  │  └── onap-python + │  ├── ietf-actn-tools │  ├── integration │  ├── seccom │  ├── simulators - │  │  ├── 5G-core-nf-simulator - │  │  ├── dc-simulator - │  │  ├── pnf-simulator - │  │  └── ran-simulator + │  │  ├──5G-core-nf-simulator + │  │  ├──A1-policy-enforcement-simulator + │  │  ├──avcn-manager + │  │  ├──core-nssmf-simulator + │  │  ├──dc-simulator + │  │  ├──masspnf-simulator + │  │  ├──netconf-server + │  │  ├──nf-simulator + │  │  ├──pm-https-server + │  │  ├──pnf-simulator + │  │  ├──ran-nssmf-simulator + │  │  ├──ran-simulator + │  │  └── ves-client │  ├── terraform │  ├── terragrunt │  ├── usecases + │  │  ├── A1-policy-enforcement + │  │  ├── A1-policy-enforcement-r-apps │  │  ├── bbs │  │  └── mdons │  └── xtesting ├── oparent └── testsuite ├── cds + ├── cds-mock-odl + ├── cds-mock-server + ├── cds-mock-ssh ├── heatbridge ├── oom ├── pythonsdk-tests @@ -42,7 +57,7 @@ Most of the repositories are internal ONAP repositories. ├── robot-utils └── testsuite -Please note that integration and teststuite are repositories AND groups hosting +Please note that integration and teststuite are repositories and groups hosting several sub-repositories. Integration @@ -64,11 +79,8 @@ It shall also help identifying, leveraging and adopting existing simulators rather than systematically re-inventing the wheel. .. attention:: - The **new integration sub-repositories** introduced for Guilin are: - - - 5G-core-nf-simulator - - terraform - - terragrunt + The **new integration sub-repositories** introduced for Honolulu are + indicated in **bold**. .. csv-table:: Integration Repositories :file: ./files/csv/repo-integration.csv @@ -96,11 +108,7 @@ Several tooling repositories are associated with the robot tests (heatbridge, robot-python-testing-utils). .. attention:: - For Guilin, 3 new sub repositories have been created: - - - testsuite/cds - - pythonsdk-test - - robot-utils + Honolulu new sub repositories are indicated in **bold**. .. csv-table:: Testsuite Repositories :file: ./files/csv/repo-testsuite.csv diff --git a/docs/integration-resources.rst b/docs/integration-resources.rst index a8dc4b992..4af90c15d 100644 --- a/docs/integration-resources.rst +++ b/docs/integration-resources.rst @@ -2,71 +2,15 @@ Creative Commons Attribution 4.0 International License. .. _integration-resources: -.. integration_main-doc: - -Integration resources +Integration Resources ===================== -.. important:: - The Integration project maintains several community resources: - - - Public Portal: http://testresults.opnfv.org/onap-integration/ - - Test Database & API: http://testresults.opnfv.org/onap/api/v1/projects - - CI/CD logs & artifacts: https://logs.onap.org/onap-integration - - VNF demo artifacts:https://nexus.onap.org/content/repositories/releases/org/onap/demo/vnf/ - - Simulators - - Test frameworks (e.g. python-onapsdk) - -Integration portal ------------------- - -A portal is built to report the status of the different labs collaborating in -Integration, see http://testresults.opnfv.org/onap-integration/ - -.. figure:: files/CI/ONAP_CI_3.png - :align: center - :width: 6.5in - -The code of this web site is shared on a public gitlab project. - -Integration Test database -------------------------- - -The integration team shares a Test Result Database with the OPNFV project. All -the test results of the CD are automatically pushed to this database. -It is possible to retrieve the results through the Test API associated with this -test Database. - -The following information are available: - -- List of pods allowed to push results: http://testresults.opnfv.org/onap/api/v1/pods -- List of projects that declared test cases for CI/CD: http://testresults.opnfv.org/onap/api/v1/projects -- List of test cases per projects: - http://testresults.opnfv.org/onap/api/v1/projects/integration/cases - http://testresults.opnfv.org/onap/api/v1/projects/security/cases -- Results with lots of possible filter combinations: http://testresults.opnfv.org/onap/api/v1/results?last=3 - -It is possible to get results according to several criteria (version, case name, -lab, period, last, CI id,..) -See the `OPNFV test API documentation <https://wiki.opnfv.org/pages/viewpage.action?pageId=2926452>`_. - -Any company running ONAP Integration tests can be referenced to push their results -to this database. -This Database is hosted on a LF OPNFV server. Results are backuped daily. -Integration committers can have access to this server. - -VNF demo Artifacts ------------------- - -VNF demo artifacts are hosted in the demo repositories and published in -https://nexus.onap.org/content/repositories/releases/org/onap/demo/vnf/. - -Communication channels ----------------------- - -The main communication channel for real time support is the rocket chat channel -http://team.onap.eu. -You can also send a mail to onap-discuss AT lists.onap.org -with [ONAP] [Integration] prefix in the title. +.. toctree:: + :glob: -All the different links are reachable from the Integration portal. + integration-repositories.rst + integration-labs.rst + integration-tests.rst + integration-CICD.rst + integration-simulators.rst + integration-tooling.rst diff --git a/docs/integration-s3p.rst b/docs/integration-s3p.rst index 38a76f995..fe0890a1f 100644 --- a/docs/integration-s3p.rst +++ b/docs/integration-s3p.rst @@ -1,108 +1,84 @@ +.. This work is licensed under a + Creative Commons Attribution 4.0 International License. .. _integration-s3p: -:orphan: - -ONAP Maturity Testing Notes ---------------------------- +Stability/Resiliency +==================== .. important:: The Release stability has been evaluated by: - - The Daily Guilin CI/CD chain - - A simple 24h healthcheck verification - - A 7 days stability test + - The daily Honolulu CI/CD chain + - Stability tests + - Resiliency tests .. note: The scope of these tests remains limited and does not provide a full set of KPIs to determinate the limits and the dimensioning of the ONAP solution. CI results -========== +---------- As usual, a daily CI chain dedicated to the release is created after RC0. -A Daily Guilin has been created on the 18th of November 2020. - -Unfortunately several technical issues disturbed the chain: - -- Due to policy changes in DockerHub (new quotas), the installation chain was - not stable as the quota limit was rapidly reached. As a consequence the - installation was incomplete and most of the tests were failing. The problem - was fixed by the subscription of unlimitted account on DockerHub. -- Due to an upgrade of the Git Jenkins plugin done by LF IT, the synchronization - of the miror of the xtesting repository, used daily to generate the test suite - dockers was corrupted. The dockers were built daily from Jenkins but with an - id from the 25th of September. As a consequence the tests reported lots of - failure because they were corresponding to Frankfurt tests without the - adaptations done for Guilin. The problem was fixed temporarily by moving to - GitLab.com Docker registry then by the downgrade of the plugin executed by LF - IT during Thanksgiving break. - -The first week of the Daily Guilin results are therefore not really usable. -Most of the results from the `daily Guilin result portal -<https://logs.onap.org/onap-integration/daily/onap_daily_pod4_guilin/>`_ -are not trustable and may be misleading. -The results became more stable from the the 6th of December. - -The graphs given hereafter are based on the data collected until the 8th of -december. This Daily chain will be maintained during the Honolulu development -cycle (Daily Master) and can be audited at any time. In case of reproducible -errors, the integration team will open JIRA on Guilin. - -Several public Daily Guilin chains have been put in place, one in Orange -(Helm v2) and one in DT (Helm v3). DT results are pushed in the test DB and can -be observed in -`ONAP Testing DT lab result page <http://testresults.opnfv.org/onap-integration/dt/dt.html>`_. +A Honolulu chain has been created on the 6th of April 2021. + +The daily results can be found in `LF daily results web site +<https://logs.onap.org/onap-integration/daily/onap_daily_pod4_honolulu/2021-04/>`_. Infrastructure Healthcheck Tests -................................ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ These tests deal with the Kubernetes/Helm tests on ONAP cluster. -The global expected criteria is **50%** when installing with Helm 2. -The onap-k8s and onap-k8s-teardown providing a snapshop of the onap namespace in -kubernetes are expected to be PASS but two tests are expected to fail: - -- onap-helm (32/33 OK) due to the size of the SO helm chart (too big for Helm2). -- nodeport_check_certs due to bad certificate issuers (Root CA certificate non - valid). In theory all the certificate shall be generated during the installation - and be valid for the 364 days after the installation. It is still not the case. - However, for the first time, no certificate was expired. Next certificates to - renew are: - - Music (2021-02-03) - - VID (2021-03-17) - - Message-router-external (2021-03-25) - - CDS-UI (2021-02-18) - - AAI and AAI-SPARKY-BE (2021-03-17) - -.. image:: files/s3p/guilin_daily_infrastructure_healthcheck.png + +The global expected criteria is **75%**. +The onap-k8s and onap-k8s-teardown providing a snapshop of the onap namespace in +Kubernetes as well as the onap-helm tests are expected to be PASS. + +nodeport_check_certs test is expected to fail. Even tremendous progress have +been done in this area, some certificates (unmaintained, upstream or integration +robot pods) are still not correct due to bad certificate issuers (Root CA +certificate non valid) or extra long validity. Most of the certificates have +been installed using cert-manager and will be easily renewable. + +.. image:: files/s3p/honolulu_daily_infrastructure_healthcheck.png :align: center Healthcheck Tests -................. +~~~~~~~~~~~~~~~~~ These tests are the traditionnal robot healthcheck tests and additional tests dealing with a single component. +Some tests (basic_onboard, basic_cds) may fail episodically due to the fact that +the startup of the SDC is sometimes not fully completed. + +The same test is run as first step of smoke tests and is usually PASS. +The mechanism to detect that all the components are fully operational may be +improved, timer based solutions are not robust enough. + The expectation is **100% OK**. -.. image:: files/s3p/guilin_daily_healthcheck.png +.. image:: files/s3p/honolulu_daily_healthcheck.png :align: center Smoke Tests -........... +~~~~~~~~~~~ -These tests are end to end tests. +These tests are end to end and automated use case tests. See the :ref:`the Integration Test page <integration-tests>` for details. The expectation is **100% OK**. -.. figure:: files/s3p/guilin_daily_smoke.png +.. figure:: files/s3p/honolulu_daily_smoke.png :align: center -An error has been detected on the SDC when performing parallel tests. -See `SDC-3366 <https://jira.onap.org/browse/SDC-3366>`_ for details. +An error has been detected on the SDNC preventing the basic_vm_macro to work. +See `SDNC-1529 <https://jira.onap.org/browse/SDNC-1529/>`_ for details. +We may also notice that SO timeouts occured more frequently than in Guilin. +See `SO-3584 <https://jira.onap.org/browse/SO-3584>`_ for details. Security Tests -.............. +~~~~~~~~~~~~~~ These tests are tests dealing with security. See the :ref:`the Integration Test page <integration-tests>` for details. @@ -111,43 +87,287 @@ The expectation is **66% OK**. The criteria is met. It may even be above as 2 fail tests are almost correct: -- the unlimited pod test is still fail due to only one pod: onap-ejbca. -- the nonssl tests is FAIL due to so and os-vnfm adapter, which were supposed to - be managed with the ingress (not possible for this release) and got a waiver - in Frankfurt. +- The unlimited pod test is still fail due testing pod (DCAE-tca). +- The nonssl tests is FAIL due to so and so-etsi-sol003-adapter, which were + supposed to be managed with the ingress (not possible for this release) and + got a waiver in Frankfurt. The pods cds-blueprints-processor-http and aws-web + are used for tests. -.. figure:: files/s3p/guilin_daily_security.png +.. figure:: files/s3p/honolulu_daily_security.png :align: center -A simple 24h healthcheck verification -===================================== +Resiliency tests +---------------- + +The goal of the resiliency testing was to evaluate the capability of the +Honolulu solution to survive a stop or restart of a Kubernetes control or +worker node. + +Controller node resiliency +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +By default the ONAP solution is installed with 3 controllers for high +availability. The test for controller resiliency can be described as follows: + +- Run tests: check that they are PASS +- Stop a controller node: check that the node appears in NotReady state +- Run tests: check that they are PASS + +2 tests were performed on the weekly honolulu lab. No problem was observed on +controller shutdown, tests were still PASS with a stoped controller node. + +More details can be found in <https://jira.onap.org/browse/TEST-309>. + +Worker node resiliency +~~~~~~~~~~~~~~~~~~~~~~ + +In community weekly lab, the ONAP pods are distributed on 12 workers. The goal +of the test was to evaluate the behavior of the pod on a worker restart +(disaster scenario assuming that the node was moved accidentally from Ready to +NotReady state). +The original conditions of such tests may be different as the Kubernetes +scheduler does not distribute the pods on the same worker from an installation +to another. + +The test procedure can be described as follows: + +- Run tests: check that they are PASS (Healthcheck and basic_vm used) +- Check that all the workers are in ready state + :: + $ kubectl get nodes + NAME STATUS ROLES AGE VERSION + compute01-onap-honolulu Ready <none> 18h v1.19.9 + compute02-onap-honolulu Ready <none> 18h v1.19.9 + compute03-onap-honolulu Ready <none> 18h v1.19.9 + compute04-onap-honolulu Ready <none> 18h v1.19.9 + compute05-onap-honolulu Ready <none> 18h v1.19.9 + compute06-onap-honolulu Ready <none> 18h v1.19.9 + compute07-onap-honolulu Ready <none> 18h v1.19.9 + compute08-onap-honolulu Ready <none> 18h v1.19.9 + compute09-onap-honolulu Ready <none> 18h v1.19.9 + compute10-onap-honolulu Ready <none> 18h v1.19.9 + compute11-onap-honolulu Ready <none> 18h v1.19.9 + compute12-onap-honolulu Ready <none> 18h v1.19.9 + control01-onap-honolulu Ready master 18h v1.19.9 + control02-onap-honolulu Ready master 18h v1.19.9 + control03-onap-honolulu Ready master 18h v1.19.9 + +- Select a worker, list the impacted pods + :: + $ kubectl get pod -n onap --field-selector spec.nodeName=compute01-onap-honolulu + NAME READY STATUS RESTARTS AGE + onap-aaf-fs-7b6648db7f-shcn5 1/1 Running 1 22h + onap-aaf-oauth-5896545fb7-x6grg 1/1 Running 1 22h + onap-aaf-sms-quorumclient-2 1/1 Running 1 22h + onap-aai-modelloader-86d95c994b-87tsh 2/2 Running 2 22h + onap-aai-schema-service-75575cb488-7fxs4 2/2 Running 2 22h + onap-appc-cdt-58cb4766b6-vl78q 1/1 Running 1 22h + onap-appc-db-0 2/2 Running 4 22h + onap-appc-dgbuilder-5bb94d46bd-h2gbs 1/1 Running 1 22h + onap-awx-0 4/4 Running 4 22h + onap-cassandra-1 1/1 Running 1 22h + onap-cds-blueprints-processor-76f8b9b5c7-hb5bg 1/1 Running 1 22h + onap-dmaap-dr-db-1 2/2 Running 5 22h + onap-ejbca-6cbdb7d6dd-hmw6z 1/1 Running 1 22h + onap-kube2msb-858f46f95c-jws4m 1/1 Running 1 22h + onap-message-router-0 1/1 Running 1 22h + onap-message-router-kafka-0 1/1 Running 1 22h + onap-message-router-kafka-1 1/1 Running 1 22h + onap-message-router-kafka-2 1/1 Running 1 22h + onap-message-router-zookeeper-0 1/1 Running 1 22h + onap-multicloud-794c6dffc8-bfwr8 2/2 Running 2 22h + onap-multicloud-starlingx-58f6b86c55-mff89 3/3 Running 3 22h + onap-multicloud-vio-584d556876-87lxn 2/2 Running 2 22h + onap-music-cassandra-0 1/1 Running 1 22h + onap-netbox-nginx-8667d6675d-vszhb 1/1 Running 2 22h + onap-policy-api-6dbf8485d7-k7cpv 1/1 Running 1 22h + onap-policy-clamp-be-6d77597477-4mffk 1/1 Running 1 22h + onap-policy-pap-785bd79759-xxhvx 1/1 Running 1 22h + onap-policy-xacml-pdp-7d8fd58d59-d4m7g 1/1 Running 6 22h + onap-sdc-be-5f99c6c644-dcdz8 2/2 Running 2 22h + onap-sdc-fe-7577d58fb5-kwxpj 2/2 Running 2 22h + onap-sdc-wfd-fe-6997567759-gl9g6 2/2 Running 2 22h + onap-sdnc-dgbuilder-564d6475fd-xwwrz 1/1 Running 1 22h + onap-sdnrdb-master-0 1/1 Running 1 22h + onap-so-admin-cockpit-6c5b44694-h4d2n 1/1 Running 1 21h + onap-so-etsi-sol003-adapter-c9bf4464-pwn97 1/1 Running 1 21h + onap-so-sdc-controller-6899b98b8b-hfgvc 2/2 Running 2 21h + onap-vfc-mariadb-1 2/2 Running 4 21h + onap-vfc-nslcm-6c67677546-xcvl2 2/2 Running 2 21h + onap-vfc-vnflcm-78ff4d8778-sgtv6 2/2 Running 2 21h + onap-vfc-vnfres-6c96f9ff5b-swq5z 2/2 Running 2 21h + +- Stop the worker (shutdown the machine for baremetal or the VM if you installed + your Kubernetes on top of an OpenStack solution) +- Wait for the pod eviction procedure completion (5 minutes) + :: + $ kubectl get nodes + NAME STATUS ROLES AGE VERSION + compute01-onap-honolulu NotReady <none> 18h v1.19.9 + compute02-onap-honolulu Ready <none> 18h v1.19.9 + compute03-onap-honolulu Ready <none> 18h v1.19.9 + compute04-onap-honolulu Ready <none> 18h v1.19.9 + compute05-onap-honolulu Ready <none> 18h v1.19.9 + compute06-onap-honolulu Ready <none> 18h v1.19.9 + compute07-onap-honolulu Ready <none> 18h v1.19.9 + compute08-onap-honolulu Ready <none> 18h v1.19.9 + compute09-onap-honolulu Ready <none> 18h v1.19.9 + compute10-onap-honolulu Ready <none> 18h v1.19.9 + compute11-onap-honolulu Ready <none> 18h v1.19.9 + compute12-onap-honolulu Ready <none> 18h v1.19.9 + control01-onap-honolulu Ready master 18h v1.19.9 + control02-onap-honolulu Ready master 18h v1.19.9 + control03-onap-honolulu Ready master 18h v1.19.9 + +- Run the tests: check that they are PASS + +.. warning:: + In these conditions, **the tests will never be PASS**. In fact several components + will remeain in INIT state. + A procedure is required to ensure a clean restart. + +List the non running pods:: + + $ kubectl get pods -n onap --field-selector status.phase!=Running | grep -v Completed + NAME READY STATUS RESTARTS AGE + onap-appc-dgbuilder-5bb94d46bd-sxmmc 0/1 Init:3/4 15 156m + onap-cds-blueprints-processor-76f8b9b5c7-m7nmb 0/1 Init:1/3 0 156m + onap-portal-app-595bd6cd95-bkswr 0/2 Init:0/4 84 23h + onap-portal-db-config-6s75n 0/2 Error 0 23h + onap-portal-db-config-7trzx 0/2 Error 0 23h + onap-portal-db-config-jt2jl 0/2 Error 0 23h + onap-portal-db-config-mjr5q 0/2 Error 0 23h + onap-portal-db-config-qxvdt 0/2 Error 0 23h + onap-portal-db-config-z8c5n 0/2 Error 0 23h + onap-sdc-be-5f99c6c644-kplqx 0/2 Init:2/5 14 156 + onap-vfc-nslcm-6c67677546-86mmj 0/2 Init:0/1 15 156m + onap-vfc-vnflcm-78ff4d8778-h968x 0/2 Init:0/1 15 156m + onap-vfc-vnfres-6c96f9ff5b-kt9rz 0/2 Init:0/1 15 156m + +Some pods are not rescheduled (i.e. onap-awx-0 and onap-cassandra-1 above) +because they are part of a statefulset. List the statefulset objects:: + + $ kubectl get statefulsets.apps -n onap | grep -v "1/1" | grep -v "3/3" + NAME READY AGE + onap-aaf-sms-quorumclient 2/3 24h + onap-appc-db 2/3 24h + onap-awx 0/1 24h + onap-cassandra 2/3 24h + onap-dmaap-dr-db 2/3 24h + onap-message-router 0/1 24h + onap-message-router-kafka 0/3 24h + onap-message-router-zookeeper 2/3 24h + onap-music-cassandra 2/3 24h + onap-sdnrdb-master 2/3 24h + onap-vfc-mariadb 2/3 24h + +For the pods being part of the statefulset, a forced deleteion is required. +As an example if we consider the statefulset onap-sdnrdb-master, we must follow +the procedure:: + + $ kubectl get pods -n onap -o wide |grep onap-sdnrdb-master + onap-sdnrdb-master-0 1/1 Terminating 1 24h 10.42.3.92 node1 + onap-sdnrdb-master-1 1/1 Running 1 24h 10.42.1.122 node2 + onap-sdnrdb-master-2 1/1 Running 1 24h 10.42.2.134 node3 + + $ kubectl delete -n onap pod onap-sdnrdb-master-0 --force + warning: Immediate deletion does not wait for confirmation that the running + resource has been terminated. The resource may continue to run on the cluster + indefinitely. + pod "onap-sdnrdb-master-0" force deleted + + $ kubectl get pods |grep onap-sdnrdb-master + onap-sdnrdb-master-0 0/1 PodInitializing 0 11s + onap-sdnrdb-master-1 1/1 Running 1 24h + onap-sdnrdb-master-2 1/1 Running 1 24h + + $ kubectl get pods |grep onap-sdnrdb-master + onap-sdnrdb-master-0 1/1 Running 0 43s + onap-sdnrdb-master-1 1/1 Running 1 24h + onap-sdnrdb-master-2 1/1 Running 1 24h + +Once all the statefulset are properly restarted, the other components shall +continue their restart properly. +Once the restart of the pods is completed, the tests are PASS. -This test consists in running the Healthcheck tests every 10 minutes during -24h. +.. important:: -The test was run from the 6th of december to the 7th of december. + K8s node reboots/shutdown is showing some deficiencies in ONAP components in + regard of their availability measured with HC results. Some pods may + still fail to initialize after reboot/shutdown(pod rescheduled). -The success rate was 100%. + However cluster as a whole behaves as expected, pods are rescheduled after + node shutdown (except pods being part of statefulset which need to be deleted + forcibly - normal Kubernetes behavior) -The results are stored in the -`test database <http://testresults.opnfv.org/onap/api/v1/results?pod_name=onap_daily_pod4_master-ONAP-oom&case_name=full>`_ + On rebooted node, should its downtime not exceed eviction timeout, pods are + restarted back after it is again available. -A 6 days stability test -======================= +Please see `Integration Resiliency page <https://jira.onap.org/browse/TEST-308>`_ +for details. -This test consists on running the test basic_vm continuously during 1 week. +Stability tests +--------------- -We observe the cluster metrics as well as the evolution of the test duration. -The test basic_vm is describe in :ref:`the Integration Test page <integration-tests>`. +Three stability tests have been performed in Honolulu: + +- SDC stability test +- Simple instantiation test (basic_vm) +- Parallel instantiation test + +SDC stability test +~~~~~~~~~~~~~~~~~~ + +In this test, we consider the basic_onboard automated test and we run 5 +simultaneous onboarding procedures in parallel during 72h. + +The basic_onboard test consists in the following steps: + +- [SDC] VendorOnboardStep: Onboard vendor in SDC. +- [SDC] YamlTemplateVspOnboardStep: Onboard vsp described in YAML file in SDC. +- [SDC] YamlTemplateVfOnboardStep: Onboard vf described in YAML file in SDC. +- [SDC] YamlTemplateServiceOnboardStep: Onboard service described in YAML file + in SDC. + +The test has been initiated on the honolulu weekly lab on the 19th of April. + +As already observed in daily|weekly|gating chain, we got race conditions on +some tests (https://jira.onap.org/browse/INT-1918). + +The success rate is above 95% on the 100 first model upload and above 80% +until we onboard more than 500 models. + +We may also notice that the function test_duration=f(time) increases +continuously. At the beginning the test takes about 200s, 24h later the same +test will take around 1000s. +Finally after 36h, the SDC systematically answers with a 500 HTTP answer code +explaining the linear decrease of the success rate. + +The following graphs provides a good view of the SDC stability test. + +.. image:: files/s3p/honolulu_sdc_stability.png + :align: center + +.. important:: + SDC can support up to 100s models onboarding. + The onbaording duration increases linearly with the number of onboarded + models + After a while, the SDC is no more usable. + No major Cluster resource issues have been detected during the test. The + memory consumption is however relatively high regarding the load. -Within a long duration test context, the test will onboard a service once then -instantiate this service multiple times. Before instantiating, it will -systematically contact the SDC and the AAI to verify that the resources already -exist. In this context the most impacted component is SO, which was delivered -relatively late compared to the other components. +.. image:: files/s3p/honolulu_sdc_stability_resources.png + :align: center -Basic_vm test -............. + +Simple stability test +~~~~~~~~~~~~~~~~~~~~~ + +This test consists on running the test basic_vm continuously during 72h. + +We observe the cluster metrics as well as the evolution of the test duration. + +The test basic_vm is described in :ref:`the Integration Test page <integration-tests>`. The basic_vm test consists in the different following steps: @@ -171,84 +391,87 @@ The basic_vm test consists in the different following steps: - [SO] YamlTemplateVfModuleAlaCarteInstantiateStep: Instantiate VF module described in YAML using SO a'la carte method. -The test has been initiated on a weekly lab on the 2nd of december. -The results provided hereafter correspond to the period from 2020-12-02 to -2020-12-08. +The test has been initiated on the Honolulu weekly lab on the 26th of April 2021. +This test has been run after the test described in the next section. +A first error occured after few hours (mariadbgalera), then the system +automatically recovered for some hours before a full crash of the mariadb +galera. -.. csv-table:: Basic_vm results - :file: ./files/csv/stability_basic_vm.csv - :widths: 70, 30 - :delim: ; - :header-rows: 1 +:: -.. note:: + debian@control01-onap-honolulu:~$ kubectl get pod -n onap |grep mariadb-galera + onap-mariadb-galera-0 1/2 CrashLoopBackOff 625 5d16h + onap-mariadb-galera-1 1/2 CrashLoopBackOff 1134 5d16h + onap-mariadb-galera-2 1/2 CrashLoopBackOff 407 5d16h - The corrected success rate excludes the FAIL results obtained during the SDNC - saturation phase. - The cause of the errors shall be analyzed more in details. The huge majority of - errors (79%) occurs on SO service creation, 18% on VNF creation and 3% on - module creation. -.. important:: - The test success rate is about 86%. - CPU consumption is low (see next section). - Memory consumption is high. +It was unfortunately not possible to collect the root cause (logs of the first +restart of onap-mariadb-galera-1). - After ~ 24-48h, the test is systematically FAIL. The trace shows that the SDNC - is no more responding. This error required the manual restart of the SDNC. - It seems that the SDNC exceeds its limits set in OOM. The simple manual - restart (delete of the pod was enough, the test after the restart is PASS, - and keep most of the time PASS for the next 24-48h) +Community members reported that they already faced such issues and suggest to +deploy a single maria instance instead of using MariaDB galera. +Moreover, in Honolulu there were some changes in order to allign Camunda (SO) +requirements for MariaDB galera.. -We can observe the consequences of the manual restart of the SDNC on its memory -graph as well as the memory threshold. +During the limited valid window, the success rate was about 78% (85% for the +same test in Guilin). +The duration of the test remain very variable as also already reported in Guilin +(https://jira.onap.org/browse/SO-3419). The duration of the same test may vary +from 500s to 2500s as illustrated in the following graph: -.. figure:: files/s3p/stability_sdnc_memory.png - :align: center +.. image:: files/s3p/honolulu_so_stability_1_duration.png + :align: center -The duration of the test is increasing slowly over the week and can be described -as follows: +The changes in MariaDB galera seems to have introduced some issues leading to +more unexpected timeouts. +A troubleshooting campaign has been launched to evaluate possible evolutions in +this area. -.. figure:: files/s3p/basic_vm_duration.png - :align: center +Parallel instantiations stability test +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -If we consider the histogram, we can see the distribution of the duration. +Still based on basic_vm, 5 instantiation attempts are done simultaneously on the +ONAP solution during 48h. -.. figure:: files/s3p/basic_vm_duration_histo.png - :align: center +The results can be described as follows: -As a conclusion, the solution seems stable. +.. image:: files/s3p/honolulu_so_stability_5.png + :align: center -The memory issue detected in the SDNC may be due to a bad sizing of the limits -and requests in OOM but a problem of light memory leak cannot be exclude. -The workaround consisting in restarting of the SDNC seems to fix the issue. -The issue is tracked in `SDNC-1430 <https://jira.onap.org/browse/SDNC-1430>`_. -Further study shall be done on this topic to consildate the detection of the -root cause. +For this test, we have to restart the SDNC once. The last failures are due to +a certificate infrastructure issue and are independent from ONAP. Cluster metrics -............... +~~~~~~~~~~~~~~~ -The Metrics of the ONAP cluster on this 6 days period are given by the -following tables: +.. important:: + No major cluster resource issues have been detected in the cluster metrics + +The metrics of the ONAP cluster have been recorded over the full week of +stability tests: .. csv-table:: CPU :file: ./files/csv/stability_cluster_metric_cpu.csv - :widths: 20,10,10,10,10,10,10,10 + :widths: 20,20,20,20,20 :delim: ; :header-rows: 1 -.. csv-table:: Memory - :file: ./files/csv/stability_cluster_metric_memory.csv - :widths: 20,10,10,10,10,10,10,10 +.. image:: files/s3p/honolulu_weekly_cpu.png + :align: center + +.. image:: files/s3p/honolulu_weekly_memory.png + :align: center + +The Top Ten for CPU consumption is given in the table below: + +.. csv-table:: CPU + :file: ./files/csv/stability_top10_cpu.csv + :widths: 20,15,15,20,15,15 :delim: ; :header-rows: 1 -.. csv-table:: Network - :file: ./files/csv/stability_cluster_metric_network.csv - :widths: 10,15,15,15,15,15,15 - :delim: ; - :header-rows: 1 +CPU consumption is negligeable and not dimensioning. It shall be reconsider for +use cases including extensive computation (loops, optimization algorithms). The Top Ten for Memory consumption is given in the table below: @@ -258,9 +481,12 @@ The Top Ten for Memory consumption is given in the table below: :delim: ; :header-rows: 1 -At least 9 components exceeds their Memory Requests. And 7 are over the Memory -limits set in OOM: the 2 Opendaylight controllers and the cassandra Databases. +Without surprise, the Cassandra databases are using most of the memory. + +The Top Ten for Network consumption is given in the table below: -As indicated CPU consumption is negligeable and not dimensioning. -It shall be reconsider for use cases including extensive computation (loops, -optimization algorithms). +.. csv-table:: Network + :file: ./files/csv/stability_top10_net.csv + :widths: 10,15,15,15,15,15,15 + :delim: ; + :header-rows: 1 diff --git a/docs/integration-tests.rst b/docs/integration-tests.rst index 62235e04d..0f3c686ca 100644 --- a/docs/integration-tests.rst +++ b/docs/integration-tests.rst @@ -28,9 +28,6 @@ Use Cases The use cases of the last release are described in :ref:`Verified Use cases <docs_usecases_release>`. -The history of the different use cases across the different releases can be -found in :ref:`Use Cases page <docs_usecases>`. - CSIT Tests ---------- @@ -46,15 +43,10 @@ repository for 2 main reasons: - most of the tests are abandoned and not maintained when hosted in a third party repository leading to CI/CD resource waste and misleading test reporting -In Guilin a PoC to help the project to re-insource their functional tests have -been initiated. -See `CSIT wiki page <https://wiki.onap.org/display/DW/Maximizing+Benefits+of+CSIT+in+ONAP+Development>`_ -for details. - -Automatic Tests +Automated Tests --------------- -These tests are run daily/weekly on each new gate (new patchset in OOM, clamp +These tests are run daily/weekly on each new gate (new patchset in OOM, CLAMP or SO). They can be in any language (bash, go, python,...), leveraging any test framework (robotframework, MTS, python-onapsdk). They are all embedded in `xtesting <https://pypi.org/project/xtesting/>`_ dockers. @@ -73,9 +65,11 @@ tests. .. figure:: files/tests/test-dashboard.png + Test dashboard (Guilin version) + All the pages and artifacts are pushed to LF backend: -- Daily chaines: https://logs.onap.org/onap-integration/daily +- Daily chains: https://logs.onap.org/onap-integration/daily - Weekly chains: https://logs.onap.org/onap-integration/weekly - Gating chains: https://logs.onap.org/onap-integration/gating @@ -157,8 +151,8 @@ to adapt then run the security tests on your own system. Note for security tests, integration team follows `SECCOM recommendations and apply waivers granted by SECCOM if needed through xfail lists <https://git.onap.org/integration/seccom/tree/>`_. -Stability Testing ------------------ +Stability/Resiliency tests +-------------------------- Ensuring the stability of ONAP is one of the missions of the Integration team. CI chains and stability tests are performed to help stabilising the release. diff --git a/docs/integration-tooling.rst b/docs/integration-tooling.rst index e69f99054..6c1658e66 100644 --- a/docs/integration-tooling.rst +++ b/docs/integration-tooling.rst @@ -10,12 +10,38 @@ Tooling of this section is to highlight some of them and redirect to their official documentation. These tools can be used for CI/CD, Testing or platform management. - **Upstream tools** are priviledged but when needed specific developments can be done. + **Upstream tools** are privileged but when needed specific developments can be done. Please note that none of these tools are imposed to test developers, in other words, any kind of test is accepted and can be integrated, the list of tools is just indicative. +Integration Project +------------------- + +Integration portal +.................. + +A portal is built to report the status of the different labs collaborating in +Integration, see http://testresults.opnfv.org/onap-integration/ + +.. figure:: files/CI/ONAP_CI_3.png + :align: center + :width: 6.5in + +The code of this web site is shared on a public gitlab project. + + +Communication channels +...................... + +The main communication channel for real time support is the official ONAP +Slack #integration-team chan (https://onapproject.slack.com/). + +You can also send a mail to onap-discuss AT lists.onap.org +with [ONAP] [Integration] prefix in the title. + + Testing ------- @@ -32,7 +58,7 @@ modules) were created especially to deal with OpenStack (see `python-testing-utils project <https://git.onap.org/testsuite/python-testing-utils/>`_). Some GUI tests (using Robotframework Selenium extension) had been initiated but -not maintained, as a consequence there are not integrated in CI/CD. +not maintained, as a consequence they are not integrated in CI/CD. Python-onapsdk @@ -108,41 +134,6 @@ if it is not supported yet. The procedure to start a test is described in `pythonsdk-test README <https://git.onap.org/testsuite/pythonsdk-tests/tree/README.md>`_ -Simulators -~~~~~~~~~~ - -Several simulators are created to support the use cases. - -.. important:: - Before starting the development of a new simulator, please consider the existing - ones, you may fine a simulator that already partially fulfills your needs.. - if so priviledge contributing to the simulator than creating a new one. - -pnf simulator -............. - -The `pnf-simulator <https://git.onap.org/integration/simulators/pnf-simulator/>`_ -can be used for several tasks: - -- Simulate PNF and interact with CDS (reconfiguration, template update) -- Send VES event to the VES collector and trigger closed loops - -A Rest API has been integrated in Guilin, allowing a http control interface of -the simulator. - -See 'README.md <https://gerrit.onap.org/r/gitweb?p=integration/simulators/pnf-simulator.git;a=blob_plain;f=pnfsimulator/README.md;hb=43d113d683ab082f8e2b7ce062e9601e74ffde3a>'__ -for details. - -Please note that this simulator has optional python CLI, see -'README.md <https://gerrit.onap.org/r/gitweb?p=integration/simulators/pnf-simulator.git;a=blob_plain;f=simulator-cli/README.md;hb=43d113d683ab082f8e2b7ce062e9601e74ffde3a>'__ -for details. - -.. note:: - There are several pnf-simulators. This simulator is a legacy simulator. It - was forked and one of the fork is known as Mass PNF simulator (hosted in - integration repository). - - CI/CD ----- @@ -183,3 +174,36 @@ of the test suites in any CI/CD systems and harmonize the inputs and the outputs The official documentation can be found on `xtesting official web site <https://xtesting.readthedocs.io/en/latest/>`_ + +Integration Test database +~~~~~~~~~~~~~~~~~~~~~~~~~ + +The integration team shares a Test Result Database with the OPNFV project. All +the test results of the CD are automatically pushed to this database. +It is possible to retrieve the results through the Test API associated with this +test Database. + +The following information are available: + +- List of pods allowed to push results: http://testresults.opnfv.org/onap/api/v1/pods +- List of projects that declared test cases for CI/CD: http://testresults.opnfv.org/onap/api/v1/projects +- List of integration test cases: + http://testresults.opnfv.org/onap/api/v1/projects/integration/cases +- List of security test cases: + http://testresults.opnfv.org/onap/api/v1/projects/security/cases +- Results with lots of possible filter combinations: http://testresults.opnfv.org/onap/api/v1/results?last=3 + +It is possible to get results according to several criteria (version, case name, +lab, period, last, CI id,..) +See the `OPNFV test API documentation <https://wiki.opnfv.org/pages/viewpage.action?pageId=2926452>`_. + +Any company running ONAP Integration tests can be referenced to push their results +to this database. +This Database is hosted on a LF OPNFV server. Results are backuped daily. +Integration committers can have access to this server. + +VNF demo Artifacts +~~~~~~~~~~~~~~~~~~ + +VNF demo artifacts are hosted in the demo repositories and published in +https://nexus.onap.org/content/repositories/releases/org/onap/demo/vnf/. diff --git a/docs/release-notes.rst b/docs/release-notes.rst index 6d25b47a8..e32597afb 100644 --- a/docs/release-notes.rst +++ b/docs/release-notes.rst @@ -3,17 +3,8 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. http://creativecommons.org/licenses/by/4.0 -Integration Maintenance Release Notes -===================================== - -.. important:: - - The testsuite 1.7.3 image has been produced for the Guilin maintenance release. - This new version includes bug fixes for test cases and healthcheck tests. - See Robot (testsuite) section for details. - -Integration Release Notes -========================= +Integration Honolulu Release Notes +================================== .. csv-table:: Integration Releases :file: ./files/csv/release-integration-ref.csv @@ -23,29 +14,24 @@ Integration Release Notes .. important:: - - Creation of a Guilin Daily CI/CD chain - - Setup of a staging lab on Azure - - Setup of a staging lab on Windriver/Intel lab (with performance audit of the Windriver/Intel labs) - - Creation of java and python baseline images + - Creation of an Honolulu Daily CI/CD chain + - Creation of a dual stack IPv4/Ipv6 chain + - Creation of Java and Python baseline images for Honolulu - Update of oparent (java dependencies) - Update of Seccom waivers and version recommendations - - New security tests (versions, pod limits, nonssl, nodeport certificate verification) - - New automated smoke tests (basic_vm, basic_network, basic_cnf) - - New requirement automated tests (ves-collector, cmpv2) - - Development of a new test framework pythonsdk - - onap_tests framework is now deprecated - - Heavy refactory of CSIT initiated - - Documentation refactoring (official documentation and wiki) + - New security test (tern, internal port certificate verification) + - New automated smoke tests (basic_onboard, basic_cds, dcaemod, basic_clamp, + pnf_macro, basic_vm_macro) + - Update of existing automated tests (5gbulkpm, cmpv2, full) + - Heavy refactoring of CSIT - New repositories (see dedicated section) - Bug fixes Quick Links: - - `Guilin Integration page <https://wiki.onap.org/display/DW/Integration+G+Release>`_ - - `Guilin Integration JIRA follow-up <https://wiki.onap.org/display/DW/Guilin+Docker+version+follow-up>`_ - - `Guilin use case testing status page <https://wiki.onap.org/display/DW/Guilin+Integration+blocking+points>`_ - - `Guilin Integration weather Board <https://wiki.onap.org/display/DW/0%3A+Integration+Weather+Board+for+Guilin+Release>`_ - + - `Honolulu Integration page <https://wiki.onap.org/display/DW/Integration+H+Release>`_ + - `Honolulu Integration JIRA follow-up <https://wiki.onap.org/display/DW/Honolulu+Integration+Blocking+Points>`_ + - `Honolulu Integration weather Board <https://wiki.onap.org/display/DW/0%3A+Integration+Weather+Board+for+Honolulu+Release>`_ Code changes ------------ @@ -53,10 +39,11 @@ Code changes Integration Repo ................. -Version: 7.0.0 (aka Guilin) -^^^^^^^^^^^^^^^^^^^^^^^^^^^ +:Release Date: 2021-04-28 -:Release Date: 2020-11-24 + +Version: 8.0.0 (aka Honolulu) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. csv-table:: Integration Changes :file: ./files/csv/release-integration-features.csv @@ -65,33 +52,24 @@ Version: 7.0.0 (aka Guilin) :header-rows: 1 -Robot (Testsuite) -................. - -Version: 1.7.3 -^^^^^^^^^^^^^^ +Onaptests (pythonsdk_tests) +........................... -:Release Date: 2021-01-11 +Main changes: -Changes for the maintenance release: +.. csv-table:: pythonsdk_tests Changes + :file: ./files/csv/release-pythonsdk-features.csv + :widths: 30,70 + :delim: ; + :header-rows: 1 -- 2020-12-16 [BUGFIX] Rename the renamed Undeploy Policy Keyword 1.7.3 -- 2020-12-15 Add testcases for SON use case -- 2020-12-15 Setup robot linter -- 2020-12-11 Fix undeploy/delete operational policy -- 2020-12-08 Fixes for tcagen2 and repush vFWCL policies -- 2020-12-03 Eliminate man-in-the-middle attack in DFC/SFTP protocol - E2E tests -- 2020-12-01 Change to tcagen2 consul -- 2020-12-01 Update CMPv2 test to use comma as SANs delimiter -- 2020-12-01 Remove Heatbridge for testing -- 2020-11-26 Update k8splugin import declaration in cmpv2 and 5gbulkpm blueprints -- 2020-11-23 Set API_TYPE default to GR_API -- 2020-11-05 PNF registrar - prevent test case fail at general step Submit SDC Software Product +Robot (Testsuite) +................. -Version: 1.7.2 +Version: 1.8.0 ^^^^^^^^^^^^^^ -:Release Date: 2020-11-03 +Main changes: .. csv-table:: Testsuite Changes :file: ./files/csv/release-testsuite-features.csv @@ -103,25 +81,21 @@ Version: 1.7.2 O-Parent ........ -Version: 3.0.2 +Version: 3.2.0 ^^^^^^^^^^^^^^ -:Release Date: 2020-11-24 - .. csv-table:: Oparent Changes :file: ./files/csv/release-oparent-features.csv :widths: 30,70 :delim: ; :header-rows: 1 -Demo Artifacts (HEAT Templates) +Demo Artifacts (Heat Templates) ............................... -Version: 1.7.0 +Version: 1.8.0 ^^^^^^^^^^^^^^ -:Release Date: 2020-11-24 - .. csv-table:: Demo Changes :file: ./files/csv/release-demo-features.csv :widths: 30,70 @@ -133,23 +107,31 @@ The demo artifacts are pushed to https://nexus.onap.org/content/repositories/rel Other Repositories .................. -New Guilin repositories: - -- 5G-core-nf-simulator -- terraform -- terragrunt +New Honolulu repositories: + +- integration/ietf-actn-tools +- integration/usecases/A1-policy-enforcement +- integration/usecases/A1-policy-enforcement-r-apps +- integration/simulators/5G-core-nf-simulator +- integration/simulators/A1-policy-enforcement-simulator +- integration/simulators/core-nssmf-simulator;Core NSSMF Simulator +- integration/simulators/nf-simulator +- integration/simulators/nf-simulator/avcn-manager +- integration/simulators/nf-simulator/netconf-server +- integration/simulators/nf-simulator/pm-https-server +- integration/simulators/nf-simulator/ves-client - testsuite/cds -- pythonsdk-test -- robot-utils +- testsuite/pythonsdk-tests +- testsuite/robot-utils Use Cases and Requirements -------------------------- -See dedicated :ref:`Guilin Use Cases and requirements page <docs_usecases_release>` +See dedicated :ref:`Honolulu Use Cases and requirements page <docs_usecases_release>` -ONAP Maturity Testing Notes ---------------------------- +Maturity Testing Notes +---------------------- :ref:`Maturity testing page <integration-s3p>` diff --git a/docs/usecases/release_automated_usecases.rst b/docs/usecases/release_automated_usecases.rst index a53d0c1b4..aa7369380 100644 --- a/docs/usecases/release_automated_usecases.rst +++ b/docs/usecases/release_automated_usecases.rst @@ -8,8 +8,9 @@ Automated Use Cases ------------------- -These use cases have been run on the Daily Guilin CI chains and are used to +These use cases have been run on the Daily Honolulu CI chains and are used to validate the integration of any new dockers in OOM. +New tests are indicated in **bold**. .. csv-table:: Infrastructure Healthcheck Tests :file: ../files/csv/tests-infrastructure-healthcheck.csv diff --git a/docs/usecases/release_usecases.rst b/docs/usecases/release_usecases.rst index d28a653b3..3d3c43bb5 100644 --- a/docs/usecases/release_usecases.rst +++ b/docs/usecases/release_usecases.rst @@ -5,14 +5,14 @@ :orphan: -Guilin Use Cases -================ +Honolulu Use Cases +================== Description ----------- This session includes use cases and functional requirements which have been -officially verified in Guilin release by the ONAP community. +officially verified in Honolulu release by the ONAP community. For each use case or functional requirement, you can find contact names and a link to the associated documentation. @@ -27,8 +27,8 @@ This documentation deals with Use cases --------- -Guilin New Use Cases -~~~~~~~~~~~~~~~~~~~~ +Honolulu Official Use Cases +~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. csv-table:: :file: ../files/csv/usecases.csv @@ -36,8 +36,8 @@ Guilin New Use Cases :delim: ; :header-rows: 1 -Old Use Cases Tested in Guilin -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Old Use Cases Tested in Honolulu +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. csv-table:: :file: ../files/csv/usecases-old-valid.csv |
