docs/sections/installation.rst


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441

DCAE mS Installation
====================

The below steps covers manual setup of DCAE VM’s and DCAE service
components.

VESCollector
------------

 

DCAE VES Collector can be configured on VM with ubuntu-16.04 image
(m1.small should suffice if this is only service) and 20Gb cinder
storage

1. Install docker

  sudo apt-get update

  sudo apt install `docker.io <http://docker.io/>`__

1. Pull the latest container from onap nexus

    sudo docker login -u docker -p docker
    `nexus.onap.org <http://nexus.onap.org/>`__:10001

    sudo docker pull
    `nexus.onap.org <http://nexus.onap.org/>`__:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector:1.1

1. Start the VESCollector with below command

    sudo docker run -d --name vescollector -p 8080:8080/tcp -p
    8443:8443/tcp -P -e DMAAPHOST='<dmaap IP>'
    `nexus.onap.org <http://nexus.onap.org/>`__:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector:1.1

     Note:  Change the dmaaphost to required DMAAP ip. To change the
    dmaap information for a running container,  stop the active
    container and rerun above command changing the dmaap IP.

1. Verification

i.  Check logs under container /opt/app/VESCollector/logs/collector.log
        for errors

ii. If no active feed, you can simulate an event into collector via curl

    curl -i  -X POST -d @<sampleves> --header "Content-Type:
    application/json" http://localhost:8080/eventListener/v5 -k

    Note: If DMAAPHOST provided is invalid, you will see exception
    around publish on the collector.logs (collector queues and attempts
    to resend the event hence exceptions reported will be periodic). 

i. Below two topic configuration are pre-set into this container.  When
       valid DMAAP instance ip was provided and VES events are received,
       the collector will post to below topics.

    Fault -
     http://<dmaaphost>:3904/events/unauthenticated.SEC\_FAULT\_OUTPUT

    Measurement
    -http://<dmaaphost>:3904/events/unauthenticated.SEC\_MEASUREMENT\_OUTPUT

VM Init
~~~~~~

To address windriver server in-stability, the below **init.sh** script
was used to start the container on VM restart.  

+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| #!/bin/sh                                                                                                                                                                                    |
|                                                                                                                                                                                              |
| sudo docker ps \| grep "vescollector"                                                                                                                                                        |
|                                                                                                                                                                                              |
| if [ $? -ne 0 ]; then                                                                                                                                                                        |
|                                                                                                                                                                                              |
|         sudo docker login -u docker -p docker nexus.onap.org:10001                                                                                                                           |
|                                                                                                                                                                                              |
|         sudo docker pull nexus.onap.org:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector:1.1                                                                                         |
|                                                                                                                                                                                              |
|         sudo docker rm -f vescollector                                                                                                                                                       |
|                                                                                                                                                                                              |
|         echo "Collector process not running - $(date)" >> /home/ubuntu/startuplog                                                                                                            |
|                                                                                                                                                                                              |
|         sudo docker run -d --name vescollector -p 8080:8080/tcp -p 8443:8443/tcp -P -e DMAAPHOST='10.12.25.96' nexus.onap.org:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector:1.1   |
|                                                                                                                                                                                              |
| else                                                                                                                                                                                         |
|                                                                                                                                                                                              |
|         echo "Collector process running - $(date)" >> /home/ubuntu/startuplog                                                                                                                |
|                                                                                                                                                                                              |
| fi                                                                                                                                                                                           |
+==============================================================================================================================================================================================+
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

This script was invoked via VM init script (rc.d).

ln -s /home/ubuntu/init.sh /etc/init.d/init.sh

sudo  update-rc.d init.sh start 2

 

ThresholdCrossingAnalysis (TCA/CDAP)
------------------------------------

The platform deploys CDAP as cluster and instantiates TCA. For the
manual setup, we will leverage the CDAP SDK docker container to deploy
TCA instances.  To setup TCA, choose VM with ubuntu-16.04 image,
m1.medium size and 50gb cinder volumes.

1. Install docker

     sudo apt-get update

     sudo apt install `docker.io <http://docker.io/>`__

1. Pull CDAP SDK container

sudo docker pull caskdata/cdap-standalone:4.1.2

1. Deploy and run the CDAP container

    sudo docker run -d --name cdap-sdk-2 -p 11011:11011 -p 11015:11015
    caskdata/cdap-standalone:4.1.2

1. Create Namespace on CDAP application

curl -X PUT http://localhost:11015/v3/namespaces/cdap_tca_hi_lo

1. Create TCA app preference file under ~ubuntu as below

+--------------------------------------------------------------------------------------------------------------------------------------------+
| {                                                                                                                                          |
|                                                                                                                                            |
|   "publisherContentType" : "application/json",                                                                                             |
|                                                                                                                                            |
|   "publisherHostName" : "10.12.25.96",                                                                                                     |
|                                                                                                                                            |
|   "publisherHostPort" : "3904",                                                                                                            |
|                                                                                                                                            |
|   "publisherMaxBatchSize" : "1",                                                                                                           |
|                                                                                                                                            |
|   "publisherMaxRecoveryQueueSize" : "100000",                                                                                              |
|                                                                                                                                            |
|   "publisherPollingInterval" : "20000",                                                                                                    |
|                                                                                                                                            |
|   "publisherProtocol" : "http",                                                                                                            |
|                                                                                                                                            |
|   "publisherTopicName" : "unauthenticated.DCAE\_CL\_OUTPUT",                                                                               |
|                                                                                                                                            |
|   "subscriberConsumerGroup" : "OpenDCAE-c1",                                                                                               |
|                                                                                                                                            |
|   "subscriberConsumerId" : "c1",                                                                                                           |
|                                                                                                                                            |
|   "subscriberContentType" : "application/json",                                                                                            |
|                                                                                                                                            |
|   "subscriberHostName" : "10.12.25.96",                                                                                                    |
|                                                                                                                                            |
|   "subscriberHostPort" : "3904",                                                                                                           |
|                                                                                                                                            |
|   "subscriberMessageLimit" : "-1",                                                                                                         |
|                                                                                                                                            |
|   "subscriberPollingInterval" : "20000",                                                                                                   |
|                                                                                                                                            |
|   "subscriberProtocol" : "http",                                                                                                           |
|                                                                                                                                            |
|   "subscriberTimeoutMS" : "-1",                                                                                                            |
|                                                                                                                                            |
|   "subscriberTopicName" : "unauthenticated.SEC\_MEASUREMENT\_OUTPUT",                                                                      |
|                                                                                                                                            |
|   "enableAAIEnrichment" : false,                                                                                                           |
|                                                                                                                                            |
|   "aaiEnrichmentHost" : "10.12.25.72",                                                                                                     |
|                                                                                                                                            |
|   "aaiEnrichmentPortNumber" : 8443,                                                                                                        |
|                                                                                                                                            |
|   "aaiEnrichmentProtocol" : "https",                                                                                                       |
|                                                                                                                                            |
|   "aaiEnrichmentUserName" : "DCAE",                                                                                                        |
|                                                                                                                                            |
|   "aaiEnrichmentUserPassword" : "DCAE",                                                                                                    |
|                                                                                                                                            |
|   "aaiEnrichmentIgnoreSSLCertificateErrors" : false,                                                                                       |
|                                                                                                                                            |
|   "aaiVNFEnrichmentAPIPath" : "/aai/v11/network/generic-vnfs/generic-vnf",                                                                 |
|                                                                                                                                            |
|   "aaiVMEnrichmentAPIPath" :  "/aai/v11/search/nodes-query",                                                                               |
|                                                                                                                                            |
|   "tca\_policy" : "{                                                                                                                       |
|                                                                                                                                            |
|         \\"domain\\": \\"measurementsForVfScaling\\",                                                                                      |
|                                                                                                                                            |
|         \\"metricsPerEventName\\": [{                                                                                                      |
|                                                                                                                                            |
|                 \\"eventName\\": \\"vFirewallBroadcastPackets\\",                                                                          |
|                                                                                                                                            |
|                 \\"controlLoopSchemaType\\": \\"VNF\\",                                                                                    |
|                                                                                                                                            |
|                 \\"policyScope\\": \\"DCAE\\",                                                                                             |
|                                                                                                                                            |
|                 \\"policyName\\": \\"DCAE.Config\_tca-hi-lo\\",                                                                            |
|                                                                                                                                            |
|                 \\"policyVersion\\": \\"v0.0.1\\",                                                                                         |
|                                                                                                                                            |
|                 \\"thresholds\\": [{                                                                                                       |
|                                                                                                                                            |
|                         \\"closedLoopControlName\\": \\"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\\",                     |
|                                                                                                                                            |
|                         \\"version\\": \\"1.0.2\\",                                                                                        |
|                                                                                                                                            |
|                         \\"fieldPath\\": \\"$.event.measurementsForVfScalingFields.vNicUsageArray[\*].receivedTotalPacketsDelta\\",        |
|                                                                                                                                            |
|                         \\"thresholdValue\\": 300,                                                                                         |
|                                                                                                                                            |
|                         \\"direction\\": \\"LESS\_OR\_EQUAL\\",                                                                            |
|                                                                                                                                            |
|                         \\"severity\\": \\"MAJOR\\",                                                                                       |
|                                                                                                                                            |
|                         \\"closedLoopEventStatus\\": \\"ONSET\\"                                                                           |
|                                                                                                                                            |
|                 }, {                                                                                                                       |
|                                                                                                                                            |
|                         \\"closedLoopControlName\\": \\"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\\",                     |
|                                                                                                                                            |
|                         \\"version\\": \\"1.0.2\\",                                                                                        |
|                                                                                                                                            |
|                         \\"fieldPath\\": \\"$.event.measurementsForVfScalingFields.vNicUsageArray[\*].receivedTotalPacketsDelta\\",        |
|                                                                                                                                            |
|                         \\"thresholdValue\\": 700,                                                                                         |
|                                                                                                                                            |
|                         \\"direction\\": \\"GREATER\_OR\_EQUAL\\",                                                                         |
|                                                                                                                                            |
|                         \\"severity\\": \\"CRITICAL\\",                                                                                    |
|                                                                                                                                            |
|                         \\"closedLoopEventStatus\\": \\"ONSET\\"                                                                           |
|                                                                                                                                            |
|                 }]                                                                                                                         |
|                                                                                                                                            |
|         }, {                                                                                                                               |
|                                                                                                                                            |
|                 \\"eventName\\": \\"vLoadBalancer\\",                                                                                      |
|                                                                                                                                            |
|                 \\"controlLoopSchemaType\\": \\"VM\\",                                                                                     |
|                                                                                                                                            |
|                 \\"policyScope\\": \\"DCAE\\",                                                                                             |
|                                                                                                                                            |
|                 \\"policyName\\": \\"DCAE.Config\_tca-hi-lo\\",                                                                            |
|                                                                                                                                            |
|                 \\"policyVersion\\": \\"v0.0.1\\",                                                                                         |
|                                                                                                                                            |
|                 \\"thresholds\\": [{                                                                                                       |
|                                                                                                                                            |
|                         \\"closedLoopControlName\\": \\"ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3\\",                          |
|                                                                                                                                            |
|                         \\"version\\": \\"1.0.2\\",                                                                                        |
|                                                                                                                                            |
|                         \\"fieldPath\\": \\"$.event.measurementsForVfScalingFields.vNicUsageArray[\*].receivedTotalPacketsDelta\\",        |
|                                                                                                                                            |
|                         \\"thresholdValue\\": 300,                                                                                         |
|                                                                                                                                            |
|                         \\"direction\\": \\"GREATER\_OR\_EQUAL\\",                                                                         |
|                                                                                                                                            |
|                         \\"severity\\": \\"CRITICAL\\",                                                                                    |
|                                                                                                                                            |
|                         \\"closedLoopEventStatus\\": \\"ONSET\\"                                                                           |
|                                                                                                                                            |
|                 }]                                                                                                                         |
|                                                                                                                                            |
|         }, {                                                                                                                               |
|                                                                                                                                            |
|                 \\"eventName\\": \\"Measurement\_vGMUX\\",                                                                                 |
|                                                                                                                                            |
|                 \\"controlLoopSchemaType\\": \\"VNF\\",                                                                                    |
|                                                                                                                                            |
|                 \\"policyScope\\": \\"DCAE\\",                                                                                             |
|                                                                                                                                            |
|                 \\"policyName\\": \\"DCAE.Config\_tca-hi-lo\\",                                                                            |
|                                                                                                                                            |
|                 \\"policyVersion\\": \\"v0.0.1\\",                                                                                         |
|                                                                                                                                            |
|                 \\"thresholds\\": [{                                                                                                       |
|                                                                                                                                            |
|                         \\"closedLoopControlName\\": \\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\\",                          |
|                                                                                                                                            |
|                         \\"version\\": \\"1.0.2\\",                                                                                        |
|                                                                                                                                            |
|                         \\"fieldPath\\": \\"$.event.measurementsForVfScalingFields.additionalMeasurements[\*].arrayOfFields[0].value\\",   |
|                                                                                                                                            |
|                         \\"thresholdValue\\": 0,                                                                                           |
|                                                                                                                                            |
|                         \\"direction\\": \\"EQUAL\\",                                                                                      |
|                                                                                                                                            |
|                         \\"severity\\": \\"MAJOR\\",                                                                                       |
|                                                                                                                                            |
|                         \\"closedLoopEventStatus\\": \\"ABATED\\"                                                                          |
|                                                                                                                                            |
|                 }, {                                                                                                                       |
|                                                                                                                                            |
|                         \\"closedLoopControlName\\": \\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\\",                          |
|                                                                                                                                            |
|                         \\"version\\": \\"1.0.2\\",                                                                                        |
|                                                                                                                                            |
|                         \\"fieldPath\\": \\"$.event.measurementsForVfScalingFields.additionalMeasurements[\*].arrayOfFields[0].value\\",   |
|                                                                                                                                            |
|                         \\"thresholdValue\\": 0,                                                                                           |
|                                                                                                                                            |
|                         \\"direction\\": \\"GREATER\\",                                                                                    |
|                                                                                                                                            |
|                         \\"severity\\": \\"CRITICAL\\",                                                                                    |
|                                                                                                                                            |
|                         \\"closedLoopEventStatus\\": \\"ONSET\\"                                                                           |
|                                                                                                                                            |
|                 }]                                                                                                                         |
|                                                                                                                                            |
|         }]                                                                                                                                 |
|                                                                                                                                            |
| }"                                                                                                                                         |
|                                                                                                                                            |
| }                                                                                                                                          |
+============================================================================================================================================+
+--------------------------------------------------------------------------------------------------------------------------------------------+

     Note: Dmaap configuration are specified on this file on
    publisherHostName and subscriberHostName. To be changed as
    required\*\*

1. Copy below script to CDAP server (this gets latest image from nexus
   and deploys TCA application) and execute it

+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
| #!/bin/sh                                                                                                                                                     |
|                                                                                                                                                               |
| TCA\_JAR=dcae-analytics-cdap-tca-2.0.0.jar                                                                                                                    |
|                                                                                                                                                               |
| rm -f /home/ubuntu/$TCA\_JAR                                                                                                                                  |
|                                                                                                                                                               |
| cd /home/ubuntu/                                                                                                                                              |
|                                                                                                                                                               |
| wget https://nexus.onap.org/service/local/repositories/staging/content/org/onap/dcaegen2/analytics/tca/dcae-analytics-cdap-tca/2.0.0/$TCA\_JAR                |
|                                                                                                                                                               |
| if [ $? -eq 0 ]; then                                                                                                                                         |
|                                                                                                                                                               |
|         if [ -f /home/ubuntu/$TCA\_JAR ]; then                                                                                                                |
|                                                                                                                                                               |
|                 echo "Restarting TCA CDAP application using $TCA\_JAR artifact"                                                                               |
|                                                                                                                                                               |
|         else                                                                                                                                                  |
|                                                                                                                                                               |
|                 echo "ERROR: $TCA\_JAR missing"                                                                                                               |
|                                                                                                                                                               |
|                 exit 1                                                                                                                                        |
|                                                                                                                                                               |
|         fi                                                                                                                                                    |
|                                                                                                                                                               |
| else                                                                                                                                                          |
|                                                                                                                                                               |
|         echo "ERROR: $TCA\_JAR not found in nexus"                                                                                                            |
|                                                                                                                                                               |
|         exit 1                                                                                                                                                |
|                                                                                                                                                               |
| fi                                                                                                                                                            |
|                                                                                                                                                               |
| # stop programs                                                                                                                                               |
|                                                                                                                                                               |
| curl -X POST http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/workers/TCADMaaPMRPublisherWorker/stop                                      |
|                                                                                                                                                               |
| curl -X POST http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/workers/TCADMaaPMRSubscriberWorker/stop                                     |
|                                                                                                                                                               |
| curl -X POST http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/flows/TCAVESCollectorFlow/stop                                              |
|                                                                                                                                                               |
| # delete application                                                                                                                                          |
|                                                                                                                                                               |
| curl -X DELETE http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca# delete artifact                                                          |
|                                                                                                                                                               |
| curl -X DELETE http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/artifacts/dcae-analytics-cdap-tca/versions/2.0.0# load artifact                         |
|                                                                                                                                                               |
| curl -X POST --data-binary @/home/ubuntu/$TCA\_JAR http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/artifacts/dcae-analytics-cdap-tcaecho# create app   |
|                                                                                                                                                               |
| curl -X PUT -d @/home/ubuntu/tca\_app\_config.json http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca# load preferences                     |
|                                                                                                                                                               |
| curl -X PUT -d @/home/ubuntu/tca\_app\_preferences.json http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/preferences# start programs      |
|                                                                                                                                                               |
| curl -X POST http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/workers/TCADMaaPMRPublisherWorker/start                                     |
|                                                                                                                                                               |
| curl -X POST http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/workers/TCADMaaPMRSubscriberWorker/start                                    |
|                                                                                                                                                               |
| curl -X POST http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/flows/TCAVESCollectorFlow/startecho# get status of programs                 |
|                                                                                                                                                               |
| curl http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/workers/TCADMaaPMRPublisherWorker/status                                            |
|                                                                                                                                                               |
| curl http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/workers/TCADMaaPMRSubscriberWorker/status                                           |
|                                                                                                                                                               |
| curl http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/flows/TCAVESCollectorFlow/statusecho                                                |
+===============================================================================================================================================================+
+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

1. Verify TCA application and logs via CDAP GUI processes

    The overall flow can be checked here

TCA Configuration Change
~~~~~~~~~~~~~~~~~~~~~~~

Typical configuration changes include changing DMAAP host and/or Policy
configuration. If necessary, modify the file on step #5 and run the
script noted as step #6 to redeploy TCA with updated configuration.

VM Init
~~~~~~

To address windriver server in-stability, the below **init.sh** script
was used to restart the container on VM restart.  This script was
invoked via VM init script (rc.d).

+------------------------------------------------------------------------------------------------------------------------------+
| #!/bin/sh                                                                                                                    |
|                                                                                                                              |
| #docker run -d --name cdap-sdk -p 11011:11011 -p 11015:11015 caskdata/cdap-standalone:4.1.2                                  |
|                                                                                                                              |
| sudo docker restart cdap-sdk-2                                                                                               |
|                                                                                                                              |
| sleep 30                                                                                                                     |
|                                                                                                                              |
| # start program                                                                                                              |
|                                                                                                                              |
| curl -X POST http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/workers/TCADMaaPMRPublisherWorker/start    |
|                                                                                                                              |
| curl -X POST http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/workers/TCADMaaPMRSubscriberWorker/start   |
|                                                                                                                              |
| curl -X POST http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/flows/TCAVESCollectorFlow/start            |
+==============================================================================================================================+
+------------------------------------------------------------------------------------------------------------------------------+

 

This script was invoked via VM init script (rc.d).

ln -s /home/ubuntu/init.sh /etc/init.d/init.sh

sudo  update-rc.d init.sh start 2