diff options
| -rw-r--r-- | external-schema-repo-generator/Makefile | 8 | ||||
| -rw-r--r-- | external-schema-repo-generator/README.md | 175 | ||||
| -rw-r--r-- | external-schema-repo-generator/environment.config | 15 | ||||
| -rwxr-xr-x | external-schema-repo-generator/generate.sh | 227 | ||||
| -rwxr-xr-x | external-schema-repo-generator/install.sh | 58 |
5 files changed, 483 insertions, 0 deletions
diff --git a/external-schema-repo-generator/Makefile b/external-schema-repo-generator/Makefile new file mode 100644 index 0000000..5bc4605 --- /dev/null +++ b/external-schema-repo-generator/Makefile @@ -0,0 +1,8 @@ +include environment.config + +generate: + ./generate.sh $(REPOSITORY_URL_HTTPS) $(REPOSITORY_BRANCH) $(SCHEMAS_LOCATION) $(VENDOR_NAME) $(SPEC_CONFIGMAP_FILENAME) $(K8S_CONFIGMAP_NAME) $(SNIPPET_FILENAME) + +install: + ./install.sh $(SPEC_CONFIGMAP_FILENAME) $(K8S_CONFIGMAP_NAME) + diff --git a/external-schema-repo-generator/README.md b/external-schema-repo-generator/README.md new file mode 100644 index 0000000..1ac06b2 --- /dev/null +++ b/external-schema-repo-generator/README.md @@ -0,0 +1,175 @@ +StndDefined schemas ConfigMap generator +======================================= + +## Description +StndDefined schemas Kubernetes ConfigMap generator is a Makefile with two targets running shell scripts: 'generate' and +'install'. This Makefile may be used by VES Collector users to generate and install ConfigMap containing schemas +and mapping file for stndDefined validation in VES pod. Additionally script creates file with snippets containing +auto-generated configuration of volumes finally mounted in VES deployment. Process of generation of ConfigMap spec file +is configurable via environment.config file. + +Generator files are available in oom/utils ONAP Gerrit repository under external-schema-repo-generator folder. + +## Requirements and limitations + +### Environment +Target *generate* from Makefile requires stable internet connection to properly clone Git repositories. +Target *install* should be ran on the RKE node, so it can create ConfigMap in the same Kubernetes environment as VES Pod +is installed. It is possible to generate a spec in one environment and move it together with all ConfigMap generator +files to the RKE environment. + +### Repository limitations +When running the script branches from selected repository are being downloaded. Time of script execution depends mostly +on repository size and number of schemas. All .yaml files from selected directory in Git repository will be considered +as schemas and attached to ConfigMap spec. + +### Generator tool files integration +It is recommended to consider files of this tool as a unity and not split them when moving generator through +environments. Generator tool files that are required are: +- Makefile +- install.sh +- generate.sh +- environment.config + +## Instruction + +### Parameters description +Before running any target from Makefile, configuration in *environment.config* must be properly prepared. Description of +the configurable properties is below. + +- **SPEC_CONFIGMAP_FILENAME** - Filename name of ConfigMap spec that will be generated. +- **K8S_CONFIGMAP_NAME** - Kubernetes name of ConfigMap that will be generated and installed. +- **SNIPPET_FILENAME** - Filename of snippet with autogenerated content that should be added to VES deployment. + +- **REPOSITORY_URL_HTTPS** - URL to remote Git repository which lets cloning repository via HTTPS. +- **REPOSITORY_BRANCH** - Valid branch from selected Git repository which contains schemas desired to mount. Script +accepts an array of branches from which schemas will be collected. To pass an array split branch names with space and +cover list in quotation marks. +- **SCHEMAS_LOCATION** - Path to schemas directory on selected Git repo. All YAML files from this repository will be +considered as schema and added to ConfigMap. + +- **VENDOR_NAME** - Name of organisation delivering schemas, used only for naming of schemas destination directory in +VES. + +### Running commands + +To run ConfigMap spec generation as well as snippet file used for mounting ConfigMap to specific Deployment use: + +``` +make generate +``` + +To run ConfigMap installation in Kubernetes use: + +**NOTE**: Remember about running this command on RKE node. + +``` +make install +``` + +**NOTE**: It is possible that ConfigMap with selected K8S_CONFIGMAP_NAME already exists in Kubernetes. In such situation +either regenerate spec with new K8S_CONFIGMAP_NAME or remove existing ConfigMap from Kubernetes and install spec again. +To remove ConfigMap from Kubernetes use: +``` +kubectl -n onap delete configmap <CONFIGMAP_NAME> +``` + +## ConfigMap validation +After running the script ConfigMap spec file is generated in current working directory. +Spec file can be manually validated via any text editor. The last file included in spec is schema-map.json file with +mappings of external URLs to prepared local URLs. + +To check whether it has been created use command: + +``` +kubectl -n onap get configmap | grep <CONFIGMAP_NAME> +``` + +A ConfigMap with configured name should be visible on the list. + +## Mounting ConfigMap into VES Collector + +To mount created ConfigMap in VES, its deployment must be edited. It can be done with: +``` +kubectl -n onap edit deployment dep-dcae-ves-collector +``` + +Snippets with content that should be added to VES deployment are generated in file with name setup in configuration file +under SNIPPET_FILENAME property. No extra configuration in VES is needed when using them. + +**NOTE**: When using Vi text editor for deployment edition, correct input mode must be set to keep proper indentation +when pasting snippets. Use ``:set paste `` to turn paste mode on. To close paste mode use ``:set nopaste`` + +1. Add volumeMounts element + + In spec.template.spec.containers[0].volumeMounts add new list element: + + **NOTE**: spec.template.spec.containers[0] should be a container with the image: + *nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector::x.x.x*. + It should be the first container, but make sure that the correct container is being edited. + + ``` + volumeMounts: + - ... + - mountPath: /opt/app/VESCollector/etc/externalRepoCustom + name: custom-3gpp-schemas + ``` + + - mountPath - Directory context for schemas. **NOTE**: must be the same as configuration of VES Collector property + *collector.externalSchema.schemasLocation* in *collector.properties*. This property might be modified via Consul UI, + later after changes in deployment. + + - name - Name of ConfigMap volume. Must be the same as set in the 2. step in *name* field. + +2. Add volumes element + + In spec.template.spec.volumes add a new list element with all desired to mount schemas from ConfigMap in + *items* list. *key* are file names from generated previously spec and *path* is relative path from directory set up + in step 1 as *mountPath*. + + Sample spec.template.spec.volumes content: + + ``` + volumes: + - configMap: + defaultMode: 420 + items: + - key: schema-map.json + path: schema-map.json + - key: SA88-Rel16-faultMnS.yaml + path: 3gpp/rep/sa5/data-models/SA88-Rel16/OpenAPI/faultMnS.yaml + - ... + name: stnd-defined-configmap + name: custom-3gpp-schemas + - ... + ``` + Fields description: + - name - Name of ConfigMap volume. Must be the same as set in the 1. step in *name* field. + - configMap.name - name of installed Kubernetes ConfigMap described in K8S_CONFIGMAP_NAME configuration setting + - configMap.items[].key - name of mounted file from installed previously ConfigMap. + **NOTE**: Not every schema from ConfigMap must be listed in *items* (warning message will be logged on VES startup if + so), but mapping file named as described in MAPPING_FILE_NAME setting is required for correct stndDefined validation + in VES. + - configMap.items[].path: Relative path from *mountPath* from step 1 which describes location of schema location in + VES container. + **NOTE 1**: For correct schemas detection in VES Collector *path* of each schema must be the same as its localURL in + mapping file. Mapping file is included as the last file in generated ConfigMap spec. + **NOTE 2**: For correct mapping file detection in VES Collector its *path* must be the same as in property + *collector.externalSchema.mappingFileLocation* in *collector.properties*. This property might be modified via Consul + UI, later after changes in deployment. + +3. Save and close an editor, K8S will automatically detect changes, terminate old VES Pod and deploy new one with +mounted ConfigMap. Correctness of new VES Pod initialization and mounting ConfigMap can be tracked using +`kubectl -n onap describe pod <VES_POD_NAME>`. + +To check if mounted schemas and VES properties configuration are correctly aligned see logs of VES. Each schema which is +present in mapping file, but is not detected by VES will be logged on application startup e.g.: +``` +2020-10-01 11:46:55.872 WARN 24 [ Thread-5] o.o.d.s.s.s.e.s.m.s.UrlMapperFactory : Local schema resource missing. Schema file with path /opt/app/VESCollector/etc/externalRepoCustom/3gpp/rep/sa5/MnS/tree/master/OpenAPI/5gcNrm.yaml has not been found. +2020-10-01 11:46:55.872 WARN 24 [ Thread-5] o.o.d.s.s.s.e.s.m.s.UrlMapperFactory : Mapping for publicURL ("https://forge.3gpp.org/rep/sa5/MnS/tree/master/OpenAPI/5gcNrm.yaml") will not be added to validator. +``` + +To see logs of VES use: +``` +kubectl -n onap logs <VES_POD_NAME> +``` diff --git a/external-schema-repo-generator/environment.config b/external-schema-repo-generator/environment.config new file mode 100644 index 0000000..074a463 --- /dev/null +++ b/external-schema-repo-generator/environment.config @@ -0,0 +1,15 @@ +# Configuration file for externalSchemaRepoGenerator Makefile +# Remember about escaping special characters, e.g. in REPOSITORY_URL + +# ConfigMap configuration +SPEC_CONFIGMAP_FILENAME=stndDefined-schemas-configmap-spec.yaml +K8S_CONFIGMAP_NAME=stnd-defined-configmap +SNIPPET_FILENAME=ves-snippet.md + +# Source repository configuration +REPOSITORY_URL_HTTPS=https\://forge.3gpp.org/rep/sa5/MnS.git +REPOSITORY_BRANCH="SA88-Rel16 SA89-Rel17" +SCHEMAS_LOCATION=OpenAPI + +# Vendor description +VENDOR_NAME=3gpp
\ No newline at end of file diff --git a/external-schema-repo-generator/generate.sh b/external-schema-repo-generator/generate.sh new file mode 100755 index 0000000..731bcbd --- /dev/null +++ b/external-schema-repo-generator/generate.sh @@ -0,0 +1,227 @@ +#!/bin/sh + +# ============LICENSE_START======================================================= +# OOM +# ================================================================================ +# Copyright (C) 2020 Nokia. All rights reserved. +# ================================================================================ +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============LICENSE_END========================================================= + + +# Arguments renaming +arguments_number=$# +repo_url=$1 +branches=$2 +schemas_location=$3 +vendor=$4 +configmap_filename=$5 +configmap_name=$6 +snippet_filename=$7 + +# Constants +SCHEMA_MAP_FILENAME="schema-map.json" +SUCCESS_CODE=0 +TREE=tree +EXPECTED_1_ARG=1 +EXPECTED_7_ARGS=7 +INDENTATION_LEVEL_1=1 +INDENTATION_LEVEL_2=2 +INDENTATION_LEVEL_3=3 +INDENTATION_LEVEL_4=4 +INDENTATION_LEVEL_5=5 + +# Variables +tmp_location=$(mktemp -d) +valid_branches="" + +# Indents each line of string by adding indent_size*indent_string spaces on the beginning +# Optional argument is indent_string level, default: 1 +# correct usage example: +# echo "Sample Text" | indent_string 2 +indent_string() { + indent_size=2 + indent_string=1 + if [ -n "$1" ]; then indent_string=$1; fi + pr -to $(expr "$indent_string" \* "$indent_size") +} + +# Checks whether number of arguments is valid +# $1 is actual number of arguments +# $2 is expected number of arguments +check_arguments() { + if [ "$1" -ne "$2" ]; then + echo "Incorrect number of arguments" + exit 1 + fi +} + +# Clones all branches selected in $BRANCH from $repo_url +clone_repo() { + for actual_branch in $branches; do + clone_branch "$actual_branch" + done +} + +# Clones single branch $1 from $repo_url. +# $1 - branch name +clone_branch() { + check_arguments $# $EXPECTED_1_ARG + if [ -d $tmp_location/"$1" ]; then + echo "Skipping cloning repository." + echo "Branch $1 has already been cloned in the directory ./$tmp_location/$1" + echo "To redownload branch remove ./$tmp_location/$1." + else + echo "Cloning repository from branch $1" + git clone --quiet --single-branch --branch "$1" "$repo_url" "$tmp_location/$1" 2>/dev/null + result=$? + if [ $result -ne $SUCCESS_CODE ] ; then + echo "Problem with cloning branch $1." + echo "Branch $1 will not be added to spec." + else + valid_branches="${valid_branches} $1" + fi + fi +} + +# Creates file with name $configmap_filename +# Inserts ConfigMap metadata and sets name as $configmap_name +add_config_map_metadata() { + echo "Creating ConfigMap spec file: $configmap_filename" + cat << EOF > "$configmap_filename" +apiVersion: v1 +kind: ConfigMap +metadata: + name: $configmap_name + labels: + name: $configmap_name + namespace: onap +data: +EOF +} + +# For each selected branch: +# clones the branch from repository, +# adds schemas from branch to ConfigMap spec +add_schemas() { + for actual_branch in $valid_branches; do + echo "Adding schemas from branch $actual_branch to spec" + add_schemas_from_branch "$actual_branch" + done +} + +# Adds schemas from single branch to spec +# $1 - branch name +add_schemas_from_branch() { + check_arguments $# $EXPECTED_1_ARG + schemas=$(ls -g $tmp_location/$1/$schemas_location/*.yaml | awk '{print $NF}') + for schema in $schemas; do + echo "$1-$(basename $schema): |-" | indent_string $INDENTATION_LEVEL_1 + cat "$schema" | indent_string $INDENTATION_LEVEL_2 + done +} >> "$configmap_filename" + +# Generates mapping file for collected schemas directly in spec +generate_mapping_file() { + echo "Generating mapping file in spec" + echo "$SCHEMA_MAP_FILENAME"": |-" | indent_string $INDENTATION_LEVEL_1 >> "$configmap_filename" + echo "[" | indent_string $INDENTATION_LEVEL_2 >> "$configmap_filename" + + for actual_branch in $valid_branches; do + echo "Adding mappings from branch: $actual_branch" + add_mappings_from_branch "$actual_branch" + done + + truncate -s-2 "$configmap_filename" + echo "" >> "$configmap_filename" + echo "]" | indent_string $INDENTATION_LEVEL_2 >> "$configmap_filename" +} + +# Adds mappings from single branch directly to spec +# $1 - branch name +add_mappings_from_branch() { + check_arguments $# $EXPECTED_1_ARG + schemas=$(ls -g $tmp_location/$1/$schemas_location/*.yaml | awk '{print $NF}' ) + + for schema in $schemas; do + repo_endpoint=$(echo "$repo_url" | cut -d/ -f4- | rev | cut -d. -f2- | rev) + schema_repo_path=$(echo "$schema" | cut -d/ -f4-) + public_url_schemas_location=${repo_url%.*} + public_url=$public_url_schemas_location/$TREE/$schema_repo_path + local_url=$vendor/$repo_endpoint/$TREE/$schema_repo_path + + echo "{" | indent_string $INDENTATION_LEVEL_3 >> "$configmap_filename" + echo "\"publicURL\": \"$public_url\"," | indent_string $INDENTATION_LEVEL_4 >> "$configmap_filename" + echo "\"localURL\": \"$local_url\"" | indent_string $INDENTATION_LEVEL_4 >> "$configmap_filename" + echo "}," | indent_string $INDENTATION_LEVEL_3 >> "$configmap_filename" + done +} + +create_snippet() { + echo "Generating snippets in file: $snippet_filename" + generate_entries + + cat << EOF > "$snippet_filename" +Snippets for mounting ConfigMap in DCAE VESCollector Deployment +========================================================================= + +## Description +These snippets will override existing in VESCollector schemas and mapping file. + +No extra configuration in VESCollector is needed with these snippets. + +## Snippets +#### spec.template.spec.containers[0].volumeMounts +\`\`\` + - mountPath: /opt/app/VESCollector/etc/externalRepo + name: custom-$vendor-schemas +\`\`\` + +#### spec.template.spec.volumes +\`\`\` + - configMap: + defaultMode: 420 + items: + - key: $SCHEMA_MAP_FILENAME + path: schema-map.json +$schemas_entries + name: $configmap_name + name: custom-$vendor-schemas +\`\`\` +EOF +} + +generate_entries() { + for actual_branch in $valid_branches; do + schemas=$(ls -g $tmp_location/$actual_branch/$schemas_location/*.yaml | awk '{print $NF}') + for schema in $schemas; do + repo_endpoint=$(echo "$repo_url" | cut -d/ -f4- | rev | cut -d. -f2- | rev) + schema_repo_path=$(echo "$schema" | cut -d/ -f4-) + + key="$actual_branch-$(basename "$schema")" + path=$vendor/$repo_endpoint/$TREE/$schema_repo_path + schemas_entries="$schemas_entries- key: $key\n path: $path\n" + done + done + schemas_entries=$(echo "$schemas_entries" | indent_string $INDENTATION_LEVEL_5) +} + +# todo add check of global env whether script should be ran +main() { + check_arguments $arguments_number $EXPECTED_7_ARGS + clone_repo + add_config_map_metadata + add_schemas + generate_mapping_file + create_snippet +} + +main
\ No newline at end of file diff --git a/external-schema-repo-generator/install.sh b/external-schema-repo-generator/install.sh new file mode 100755 index 0000000..2f926b6 --- /dev/null +++ b/external-schema-repo-generator/install.sh @@ -0,0 +1,58 @@ +#!/bin/sh + +# ============LICENSE_START======================================================= +# OOM +# ================================================================================ +# Copyright (C) 2020 Nokia. All rights reserved. +# ================================================================================ +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============LICENSE_END========================================================= + +# Arguments renaming +spec_configmap_filename=$1 +k8s_configmap_name=$2 + +# Constants +MAX_SPEC_SIZE=262144 + +# Alias +alias kubectl_onap="kubectl -n onap" + +# Checks whether ConfigMap $spec_configmap_filename exists +# When file does not exist exits with return code 1 +checkIfSpecExists() { + if [ ! -f "$spec_configmap_filename" ]; then + echo "Spec file $spec_configmap_filename does not exist." + # todo add location of spec with filename + exit 1 + fi +} + +# Uploads ConfigMap spec $spec_configmap_filename to Kubernetes +uploadConfigMap() { + spec_size=$(stat --printf="%s" "$spec_configmap_filename") + if [ "$spec_size" -ge $MAX_SPEC_SIZE ]; then + echo "ConfigMap spec file is too long for 'kubectl apply'. Actual spec length: $spec_size, max spec length: $MAX_SPEC_SIZE" + echo "Creating new ConfigMap $k8s_configmap_name" + kubectl_onap create -f "$spec_configmap_filename" + else + echo "Applying ConfigMap $k8s_configmap_name" + kubectl_onap apply -f "$spec_configmap_filename" + fi +} + +main() { + checkIfSpecExists + uploadConfigMap +} + +main + |