1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
|
.. _integration-s3p:
:orphan:
ONAP Maturity Testing Notes
---------------------------
.. important::
The Release stability has been evaluated by:
- The Daily Guilin CI/CD chain
- A simple 24h healthcheck verification
- A 7 days stability test
.. note:
The scope of these tests remains limited and does not provide a full set of
KPIs to determinate the limits and the dimensioning of the ONAP solution.
CI results
==========
As usual, a daily CI chain dedicated to the release is created after RC0.
A Daily Guilin has been created on the 18th of November 2020.
Unfortunately several technical issues disturbed the chain:
- Due to policy changes in DockerHub (new quotas), the installation chain was
not stable as the quota limit was rapidly reached. As a consequence the
installation was incomplete and most of the tests were failing. The problem
was fixed by the subscription of unlimitted account on DockerHub.
- Due to an upgrade of the Git Jenkins plugin done by LF IT, the synchronization
of the miror of the xtesting repository, used daily to generate the test suite
dockers was corrupted. The dockers were built daily from Jenkins but with an
id from the 25th of September. As a consequence the tests reported lots of
failure because they were corresponding to Frankfurt tests without the
adaptations done for Guilin. The problem was fixed temporarily by moving to
GitLab.com Docker registry then by the downgrade of the plugin executed by LF
IT during Thanksgiving break.
The first week of the Daily Guilin results are therefore not really usable.
Most of the results from the `daily Guilin result portal
<https://logs.onap.org/onap-integration/daily/onap_daily_pod4_guilin/>`_
are not trustable and may be misleading.
The results became more stable from the the 6th of December.
The graphs given hereafter are based on the data collected until the 8th of
december. This Daily chain will be maintained during the Honolulu development
cycle (Daily Master) and can be audited at any time. In case of reproducible
errors, the integration team will open JIRA on Guilin.
Several public Daily Guilin chains have been put in place, one in Orange
(Helm v2) and one in DT (Helm v3). DT results are pushed in the test DB and can
be observed in
`ONAP Testing DT lab result page <http://testresults.opnfv.org/onap-integration/dt/dt.html>`_.
Infrastructure Healthcheck Tests
................................
These tests deal with the Kubernetes/Helm tests on ONAP cluster.
The global expected criteria is **50%** when installing with Helm 2.
The onap-k8s and onap-k8s-teardown providing a snapshop of the onap namespace in
kubernetes are expected to be PASS but two tests are expected to fail:
- onap-helm (32/33 OK) due to the size of the SO helm chart (too big for Helm2).
- nodeport_check_certs due to bad certificate issuers (Root CA certificate non
valid). In theory all the certificate shall be generated during the installation
and be valid for the 364 days after the installation. It is still not the case.
However, for the first time, no certificate was expired. Next certificates to
renew are:
- Music (2021-02-03)
- VID (2021-03-17)
- Message-router-external (2021-03-25)
- CDS-UI (2021-02-18)
- AAI and AAI-SPARKY-BE (2021-03-17)
.. image:: files/s3p/guilin_daily_infrastructure_healthcheck.png
:align: center
Healthcheck Tests
.................
These tests are the traditionnal robot healthcheck tests and additional tests
dealing with a single component.
The expectation is **100% OK**.
.. image:: files/s3p/guilin_daily_healthcheck.png
:align: center
Smoke Tests
...........
These tests are end to end tests.
See the :ref:`the Integration Test page <integration-tests>` for details.
The expectation is **100% OK**.
.. figure:: files/s3p/guilin_daily_smoke.png
:align: center
An error has been detected on the SDC when performing parallel tests.
See `SDC-3366 <https://jira.onap.org/browse/SDC-3366>`_ for details.
Security Tests
..............
These tests are tests dealing with security.
See the :ref:`the Integration Test page <integration-tests>` for details.
The expectation is **66% OK**. The criteria is met.
It may even be above as 2 fail tests are almost correct:
- the unlimited pod test is still fail due to only one pod: onap-ejbca.
- the nonssl tests is FAIL due to so and os-vnfm adapter, which were supposed to
be managed with the ingress (not possible for this release) and got a waiver
in Frankfurt.
.. figure:: files/s3p/guilin_daily_security.png
:align: center
A simple 24h healthcheck verification
=====================================
This test consists in running the Healthcheck tests every 10 minutes during
24h.
The test was run from the 6th of december to the 7th of december.
The success rate was 100%.
The results are stored in the
`test database <http://testresults.opnfv.org/onap/api/v1/results?pod_name=onap_daily_pod4_master-ONAP-oom&case_name=full>`_
A 6 days stability test
=======================
This test consists on running the test basic_vm continuously during 1 week.
We observe the cluster metrics as well as the evolution of the test duration.
The test basic_vm is describe in :ref:`the Integration Test page <integration-tests>`.
Within a long duration test context, the test will onboard a service once then
instantiate this service multiple times. Before instantiating, it will
systematically contact the SDC and the AAI to verify that the resources already
exist. In this context the most impacted component is SO, which was delivered
relatively late compared to the other components.
Basic_vm test
.............
The basic_vm test consists in the different following steps:
- [SDC] VendorOnboardStep: Onboard vendor in SDC.
- [SDC] YamlTemplateVspOnboardStep: Onboard vsp described in YAML file in SDC.
- [SDC] YamlTemplateVfOnboardStep: Onboard vf described in YAML file in SDC.
- [SDC] YamlTemplateServiceOnboardStep: Onboard service described in YAML file
in SDC.
- [AAI] RegisterCloudRegionStep: Register cloud region.
- [AAI] ComplexCreateStep: Create complex.
- [AAI] LinkCloudRegionToComplexStep: Connect cloud region with complex.
- [AAI] CustomerCreateStep: Create customer.
- [AAI] CustomerServiceSubscriptionCreateStep: Create customer's service
subscription.
- [AAI] ConnectServiceSubToCloudRegionStep: Connect service subscription with
cloud region.
- [SO] YamlTemplateServiceAlaCarteInstantiateStep: Instantiate service described
in YAML using SO a'la carte method.
- [SO] YamlTemplateVnfAlaCarteInstantiateStep: Instantiate vnf described in YAML
using SO a'la carte method.
- [SO] YamlTemplateVfModuleAlaCarteInstantiateStep: Instantiate VF module
described in YAML using SO a'la carte method.
The test has been initiated on a weekly lab on the 2nd of december.
The results provided hereafter correspond to the period from 2020-12-02 to
2020-12-08.
.. csv-table:: Basic_vm results
:file: ./files/csv/stability_basic_vm.csv
:widths: 70, 30
:delim: ;
:header-rows: 1
.. note::
The corrected success rate excludes the FAIL results obtained during the SDNC
saturation phase.
The cause of the errors shall be analyzed more in details. The huge majority of
errors (79%) occurs on SO service creation, 18% on VNF creation and 3% on
module creation.
.. important::
The test success rate is about 86%.
CPU consumption is low (see next section).
Memory consumption is high.
After ~ 24-48h, the test is systematically FAIL. The trace shows that the SDNC
is no more responding. This error required the manual restart of the SDNC.
It seems that the SDNC exceeds its limits set in OOM. The simple manual
restart (delete of the pod was enough, the test after the restart is PASS,
and keep most of the time PASS for the next 24-48h)
We can observe the consequences of the manual restart of the SDNC on its memory
graph as well as the memory threshold.
.. figure:: files/s3p/stability_sdnc_memory.png
:align: center
The duration of the test is increasing slowly over the week and can be described
as follows:
.. figure:: files/s3p/basic_vm_duration.png
:align: center
If we consider the histogram, we can see the distribution of the duration.
.. figure:: files/s3p/basic_vm_duration_histo.png
:align: center
As a conclusion, the solution seems stable.
The memory issue detected in the SDNC may be due to a bad sizing of the limits
and requests in OOM but a problem of light memory leak cannot be exclude.
The workaround consisting in restarting of the SDNC seems to fix the issue.
The issue is tracked in `SDNC-1430 <https://jira.onap.org/browse/SDNC-1430>`_.
Further study shall be done on this topic to consildate the detection of the
root cause.
Cluster metrics
...............
The Metrics of the ONAP cluster on this 6 days period are given by the
following tables:
.. csv-table:: CPU
:file: ./files/csv/stability_cluster_metric_cpu.csv
:widths: 20,10,10,10,10,10,10,10
:delim: ;
:header-rows: 1
.. csv-table:: Memory
:file: ./files/csv/stability_cluster_metric_memory.csv
:widths: 20,10,10,10,10,10,10,10
:delim: ;
:header-rows: 1
.. csv-table:: Network
:file: ./files/csv/stability_cluster_metric_network.csv
:widths: 10,15,15,15,15,15,15
:delim: ;
:header-rows: 1
The Top Ten for Memory consumption is given in the table below:
.. csv-table:: Memory
:file: ./files/csv/stability_top10_memory.csv
:widths: 20,15,15,20,15,15
:delim: ;
:header-rows: 1
At least 9 components exceeds their Memory Requests. And 7 are over the Memory
limits set in OOM: the 2 Opendaylight controllers and the cassandra Databases.
As indicated CPU consumption is negligeable and not dimensioning.
It shall be reconsider for use cases including extensive computation (loops,
optimization algorithms).
|
