1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
|
StndDefined schemas ConfigMap generator
=======================================
## Description
StndDefined schemas Kubernetes ConfigMap generator is a Makefile with shell scripts: 'generate', 'install', 'clean' ane 'utils'.
This Makefile may be used by VES Collector users to generate and install ConfigMap containing schemas
and mapping file for stndDefined validation in VES pod. Additionally script creates file with snippets containing
auto-generated configuration of volumes finally mounted in VES deployment. Process of generation of ConfigMap spec file
is configurable via environment.config file.
Generator files are available in oom/utils ONAP Gerrit repository under external-schema-repo-generator folder.
## Requirements and limitations
### Environment
Target *generate* from Makefile requires stable internet connection to properly clone Git repositories.
Target *install* should be ran on the RKE node, so it can create ConfigMap in the same Kubernetes environment as VES Pod
is installed. It is possible to generate a spec in one environment and move it together with all ConfigMap generator
files to the RKE environment.
### Repository limitations
When running the script branches from selected repository are being downloaded. Time of script execution depends mostly
on repository size and number of schemas. All .yaml files from selected directory in Git repository will be considered
as schemas and attached to ConfigMap spec.
### Generator tool files integration
It is recommended to consider files of this tool as a unity and not split them when moving generator through
environments. Generator tool files that are required are located in `generator` directory:
- Makefile
- install.sh
- generate.sh
- utils.sh
- clean.sh
- environment.config
- bin_packing.awk
## Instruction
### Parameters description
Before running any target from Makefile, configuration in *environment.config* must be properly prepared. Description of
the configurable properties is below.
- **GENERATION_DIRECTORY** - Directory in which generated files will be located.
- **EXTERNAL_REPO_CONFIGMAP_FILENAME_PREFIX** - Prefix for file name of ConfigMap spec, that will be generated.
- **EXTERNAL_REPO_CONFIGMAP_DIRECTORY** - Directory inside *GENERATION_DIRECTORY*, in which generated specs will be located.
- **EXTERNAL_REPO_CONFIGMAP_NAME_PREFIX** - Prefix for Kubernetes name of ConfigMap, that will be generated and installed.
- **SNIPPET_FILENAME** - Filename of snippet with autogenerated content that should be added to VES deployment.
- **REPOSITORY_URL_HTTPS** - URL to remote Git repository which lets cloning repository via HTTPS.
- **REPOSITORY_BRANCH** - Valid OPEN API branch from selected Git repository which contains schemas desired to mount. Script
accepts an array of branches from which schemas will be collected. To pass an array split branch names with space and
cover list in quotation marks. User can put here 5G_API specific repository tag name related to OPEN API schema branch after colon
for example: Rel-16-SA-91:TSG91-Rel16. Having such information script will clone this specific 5G API branch and merge it
with OPEN API schema. If 5G_API specific repository branch after colon will not be set by user, then script will parse
OPEN API if any reference to 5G_API specific repository is present and get branch name for 5G_API from this reference.
Next clone parsed 5G API branch and merge it with OPEN API schema
- **SCHEMAS_LOCATION** - Path to schemas directory on selected Git repo. All YAML files from this repository will be
considered as schema and added to ConfigMap.
- **VENDOR_NAME** - Name of organisation delivering schemas, used only for naming of schemas destination directory in
VES.
### Running commands
#### All
To generate and install ConfigMaps run:
```
make all
```
#### Generation
To run ConfigMaps spec generation as well as snippet file used for mounting ConfigMap to specific Deployment use:
```
make generate
```
#### Installation
To run ConfigMaps installation in Kubernetes use:
**NOTE**: Remember about running this command on RKE node.
```
make install
```
**NOTE**: It is possible that ConfigMap with selected K8S_CONFIGMAP_NAME already exists in Kubernetes.
In such situation existing ConfigMap will be replaced.
**NOTE**: If ConfigMap is bigger than one megabyte it won't be installed
and warning will be displayed during installation.
#### Cleaning
To remove generated files use:
```
make clean
```
## ConfigMap validation
After running the script ConfigMap spec file is generated in selected (be default `genereted`) directory.
Spec files can be manually validated via any text editor. The last generated file includes in spec is schema-map.json file with
mappings of external URLs to prepared local URLs.
To check whether it has been created use command:
```
kubectl -n onap get configmap | grep <CONFIGMAP_NAME>
```
A ConfigMaps with configured name should be visible on the list.
## Mounting ConfigMap into VES Collector
To mount created ConfigMaps in VES, its deployment must be edited. It can be done with:
```
kubectl -n onap edit deployment dep-dcae-ves-collector
```
Snippets with content that should be added to VES deployment are generated in file with name setup in configuration file
under SNIPPET_FILENAME property. No extra configuration in VES is needed when using them.
**NOTE**: When using Vi text editor for deployment edition, correct input mode must be set to keep proper indentation
when pasting snippets. Use ``:set paste `` to turn paste mode on. To close paste mode use ``:set nopaste``
1. Add volumeMounts element
In spec.template.spec.containers[0].volumeMounts add new list element:
**NOTE**: spec.template.spec.containers[0] should be a container with the image:
*nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector::x.x.x*.
It should be the first container, but make sure that the correct container is being edited.
```
volumeMounts:
- mountPath: /opt/app/VESCollector/etc/externalRepo
name: custom-3gpp-schemas
- mountPath: /opt/app/VESCollector/etc/externalRepo/3gpp/rep/sa5/MnS/tree/branch/OpenAPI
name: branch-custom-3gpp-schemas
- ...
```
- mountPath - Directory context for schemas. **NOTE**: must be the same as configuration of VES Collector property
*collector.externalSchema.schemasLocation* in *collector.properties*. This property might be modified via Consul UI,
later after changes in deployment.
- name - Name of ConfigMap volume. For schema map must be the same as set in the 2. step in *name* field.
2. Add volumes element
In spec.template.spec.volumes add a new list element with all desired to mount schemas from ConfigMap in
*items* list. *key* are file names from generated previously spec and *path* is relative path from directory set up
in step 1 as *mountPath*.
Sample spec.template.spec.volumes content:
```
volumes:
- configMap:
defaultMode: 420
name: schema-map-stnd-defined-configmap
name: custom-3gpp-schemas
- configMap:
defaultMode: 420
name: branch-stnd-defined-configmap
name: branch-custom-3gpp-schemas
- ...
```
Fields description:
- name - Name of ConfigMap volume. Consist of branch name and *name* field. Must be same as name in 1. step
- configMap.name - name of installed Kubernetes ConfigMap consist of schema-map or branch name
and K8S_CONFIGMAP_NAME from configuration setting
3. Save and close an editor, K8S will automatically detect changes, terminate old VES Pod and deploy new one with
mounted ConfigMap. Correctness of new VES Pod initialization and mounting ConfigMap can be tracked using
`kubectl -n onap describe pod <VES_POD_NAME>`.
To check if mounted schemas and VES properties configuration are correctly aligned see logs of VES. Each schema which is
present in mapping file, but is not detected by VES will be logged on application startup e.g.:
```
2020-10-01 11:46:55.872 WARN 24 [ Thread-5] o.o.d.s.s.s.e.s.m.s.UrlMapperFactory : Local schema resource missing. Schema file with path /opt/app/VESCollector/etc/externalRepoCustom/3gpp/rep/sa5/MnS/tree/master/OpenAPI/5gcNrm.yaml has not been found.
2020-10-01 11:46:55.872 WARN 24 [ Thread-5] o.o.d.s.s.s.e.s.m.s.UrlMapperFactory : Mapping for publicURL ("https://forge.3gpp.org/rep/sa5/MnS/tree/master/OpenAPI/5gcNrm.yaml") will not be added to validator.
```
To see logs of VES use:
```
kubectl -n onap logs <VES_POD_NAME>
```
|
