[UTILS] Add new external-schema-repo-generator tool

Added script tool which generates and installs ConfigMap
containing schemas used for event validation. Main purpose
of this script is mounting custom schemas to DCAE VES
Collector, yet it might be used in any component.

Signed-off-by: Michal Banka <michal.banka@nokia.com>
Issue-ID: OOM-2589
Change-Id: Ia82638186d42ba614636b0c258c50e313782da58

5 files changed, 483 insertions, 0 deletions

diff --git a/external-schema-repo-generator/Makefile b/external-schema-repo-generator/Makefile
new file mode 100644
index 0000000..5bc4605
--- /dev/null
+++ b/external-schema-repo-generator/Makefile
@@ -0,0 +1,8 @@
+include environment.config
+
+generate:
+	./generate.sh $(REPOSITORY_URL_HTTPS) $(REPOSITORY_BRANCH) $(SCHEMAS_LOCATION) $(VENDOR_NAME) $(SPEC_CONFIGMAP_FILENAME) $(K8S_CONFIGMAP_NAME) $(SNIPPET_FILENAME)
+
+install:
+	./install.sh $(SPEC_CONFIGMAP_FILENAME) $(K8S_CONFIGMAP_NAME)
+
diff --git a/external-schema-repo-generator/README.md b/external-schema-repo-generator/README.md
new file mode 100644
index 0000000..1ac06b2
--- /dev/null
+++ b/external-schema-repo-generator/README.md
@@ -0,0 +1,175 @@
+StndDefined schemas ConfigMap generator
+=======================================
+
+## Description
+StndDefined schemas Kubernetes ConfigMap generator is a Makefile with two targets running shell scripts: 'generate' and
+'install'. This Makefile may be used by VES Collector users to generate and install ConfigMap containing schemas
+and mapping file for stndDefined validation in VES pod. Additionally script creates file with snippets containing 
+auto-generated configuration of volumes finally mounted in VES deployment. Process of generation of ConfigMap spec file
+is configurable via environment.config file.
+
+Generator files are available in oom/utils ONAP Gerrit repository under external-schema-repo-generator folder.
+
+## Requirements and limitations
+
+### Environment
+Target *generate* from Makefile requires stable internet connection to properly clone Git repositories.
+Target *install* should be ran on the RKE node, so it can create ConfigMap in the same Kubernetes environment as VES Pod
+is installed. It is possible to generate a spec in one environment and move it together with all ConfigMap generator
+files to the RKE environment.
+
+### Repository limitations
+When running the script branches from selected repository are being downloaded. Time of script execution depends mostly
+on repository size and number of schemas. All .yaml files from selected directory in Git repository will be considered
+as schemas and attached to ConfigMap spec.
+
+### Generator tool files integration
+It is recommended to consider files of this tool as a unity and not split them when moving generator through
+environments. Generator tool files that are required are:
+- Makefile
+- install.sh
+- generate.sh
+- environment.config
+
+## Instruction
+
+### Parameters description
+Before running any target from Makefile, configuration in *environment.config* must be properly prepared. Description of
+the configurable properties is below.
+
+- **SPEC_CONFIGMAP_FILENAME** - Filename name of ConfigMap spec that will be generated.
+- **K8S_CONFIGMAP_NAME** - Kubernetes name of ConfigMap that will be generated and installed.
+- **SNIPPET_FILENAME** - Filename of snippet with autogenerated content that should be added to VES deployment.
+
+- **REPOSITORY_URL_HTTPS** - URL to remote Git repository which lets cloning repository via HTTPS.
+- **REPOSITORY_BRANCH** - Valid branch from selected Git repository which contains schemas desired to mount. Script
+accepts an array of branches from which schemas will be collected. To pass an array split branch names with space and
+cover list in quotation marks.
+- **SCHEMAS_LOCATION** - Path to schemas directory on selected Git repo. All YAML files from this repository will be
+considered as schema and added to ConfigMap.
+
+- **VENDOR_NAME** - Name of organisation delivering schemas, used only for naming of schemas destination directory in
+VES.
+
+### Running commands
+
+To run ConfigMap spec generation as well as snippet file used for mounting ConfigMap to specific Deployment use:
+
+```
+make generate
+```
+
+To run ConfigMap installation in Kubernetes use:
+
+**NOTE**: Remember about running this command on RKE node.
+
+```
+make install
+```
+
+**NOTE**: It is possible that ConfigMap with selected K8S_CONFIGMAP_NAME already exists in Kubernetes. In such situation
+either regenerate spec with new K8S_CONFIGMAP_NAME or remove existing ConfigMap from Kubernetes and install spec again.
+To remove ConfigMap from Kubernetes use:
+```
+kubectl -n onap delete configmap <CONFIGMAP_NAME>
+```
+
+## ConfigMap validation
+After running the script ConfigMap spec file is generated in current working directory.
+Spec file can be manually validated via any text editor. The last file included in spec is schema-map.json file with
+mappings of external URLs to prepared local URLs.
+
+To check whether it has been created use command:
+
+```
+kubectl -n onap get configmap | grep <CONFIGMAP_NAME>
+```
+
+A ConfigMap with configured name should be visible on the list.
+
+## Mounting ConfigMap into VES Collector
+
+To mount created ConfigMap in VES, its deployment must be edited. It can be done with:
+```
+kubectl -n onap edit deployment dep-dcae-ves-collector
+```
+
+Snippets with content that should be added to VES deployment are generated in file with name setup in configuration file
+under SNIPPET_FILENAME property. No extra configuration in VES is needed when using them.
+
+**NOTE**: When using Vi text editor for deployment edition, correct input mode must be set to keep proper indentation
+when pasting snippets. Use ``:set paste `` to turn paste mode on. To close paste mode use ``:set nopaste`` 
+
+1. Add volumeMounts element
+
+    In spec.template.spec.containers[0].volumeMounts add new list element:
+
+    **NOTE**: spec.template.spec.containers[0] should be a container with the image:
+        *nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector::x.x.x*.
+        It should be the first container, but make sure that the correct container is being edited.
+
+    ```
+    volumeMounts:
+      - ...
+      - mountPath: /opt/app/VESCollector/etc/externalRepoCustom
+        name: custom-3gpp-schemas
+    ```
+
+    - mountPath - Directory context for schemas. **NOTE**: must be the same as configuration of VES Collector property
+    *collector.externalSchema.schemasLocation* in *collector.properties*. This property might be modified via Consul UI,
+    later after changes in deployment.
+
+   - name - Name of ConfigMap volume. Must be the same as set in the 2. step in *name* field.
+
+2. Add volumes element
+
+    In spec.template.spec.volumes add a new list element with all desired to mount schemas from ConfigMap in
+    *items* list. *key* are file names from generated previously spec and *path* is relative path from directory set up
+    in step 1 as *mountPath*.
+
+    Sample spec.template.spec.volumes content:
+
+    ```
+    volumes:
+    - configMap:
+        defaultMode: 420
+        items:
+        - key: schema-map.json
+          path: schema-map.json
+        - key: SA88-Rel16-faultMnS.yaml
+          path: 3gpp/rep/sa5/data-models/SA88-Rel16/OpenAPI/faultMnS.yaml
+        - ...
+        name: stnd-defined-configmap
+      name: custom-3gpp-schemas
+    - ...
+    ```
+   Fields description:
+   - name - Name of ConfigMap volume. Must be the same as set in the 1. step in *name* field.
+   - configMap.name - name of installed Kubernetes ConfigMap described in K8S_CONFIGMAP_NAME configuration setting
+   - configMap.items[].key - name of mounted file from installed previously ConfigMap.
+   **NOTE**: Not every schema from ConfigMap must be listed in *items* (warning message will be logged on VES startup if
+   so), but mapping file named as described in MAPPING_FILE_NAME setting is required for correct stndDefined validation
+   in VES.
+   - configMap.items[].path: Relative path from *mountPath* from step 1 which describes location of schema location in
+   VES container.
+   **NOTE 1**: For correct schemas detection in VES Collector *path* of each schema must be the same as its localURL in
+   mapping file. Mapping file is included as the last file in generated ConfigMap spec.
+   **NOTE 2**: For correct mapping file detection in VES Collector its *path* must be the same as in property
+   *collector.externalSchema.mappingFileLocation* in *collector.properties*. This property might be modified via Consul
+    UI, later after changes in deployment.
+
+3. Save and close an editor, K8S will automatically detect changes, terminate old VES Pod and deploy new one with
+mounted ConfigMap. Correctness of new VES Pod initialization and mounting ConfigMap can be tracked using
+`kubectl -n onap describe pod <VES_POD_NAME>`.
+
+To check if mounted schemas and VES properties configuration are correctly aligned see logs of VES. Each schema which is
+present in mapping file, but is not detected by VES will be logged on application startup e.g.:
+```
+2020-10-01 11:46:55.872  WARN 24 [       Thread-5] o.o.d.s.s.s.e.s.m.s.UrlMapperFactory     : Local schema resource missing. Schema file with path /opt/app/VESCollector/etc/externalRepoCustom/3gpp/rep/sa5/MnS/tree/master/OpenAPI/5gcNrm.yaml has not been found.
+2020-10-01 11:46:55.872  WARN 24 [       Thread-5] o.o.d.s.s.s.e.s.m.s.UrlMapperFactory     : Mapping for publicURL ("https://forge.3gpp.org/rep/sa5/MnS/tree/master/OpenAPI/5gcNrm.yaml") will not be added to validator.
+```
+
+To see logs of VES use:
+```
+kubectl -n onap logs <VES_POD_NAME>
+```
diff --git a/external-schema-repo-generator/environment.config b/external-schema-repo-generator/environment.config
new file mode 100644
index 0000000..074a463
--- /dev/null
+++ b/external-schema-repo-generator/environment.config
@@ -0,0 +1,15 @@
+# Configuration file for externalSchemaRepoGenerator Makefile
+# Remember about escaping special characters, e.g. in REPOSITORY_URL
+
+# ConfigMap configuration
+SPEC_CONFIGMAP_FILENAME=stndDefined-schemas-configmap-spec.yaml
+K8S_CONFIGMAP_NAME=stnd-defined-configmap
+SNIPPET_FILENAME=ves-snippet.md
+
+# Source repository configuration
+REPOSITORY_URL_HTTPS=https\://forge.3gpp.org/rep/sa5/MnS.git
+REPOSITORY_BRANCH="SA88-Rel16 SA89-Rel17"
+SCHEMAS_LOCATION=OpenAPI
+
+# Vendor description
+VENDOR_NAME=3gpp
\ No newline at end of file
diff --git a/external-schema-repo-generator/generate.sh b/external-schema-repo-generator/generate.sh
new file mode 100755
index 0000000..731bcbd
--- /dev/null
+++ b/external-schema-repo-generator/generate.sh
@@ -0,0 +1,227 @@
+#!/bin/sh
+
+# ============LICENSE_START=======================================================
+# OOM
+# ================================================================================
+# Copyright (C) 2020 Nokia. All rights reserved.
+# ================================================================================
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#      http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============LICENSE_END=========================================================
+
+
+# Arguments renaming
+arguments_number=$#
+repo_url=$1
+branches=$2
+schemas_location=$3
+vendor=$4
+configmap_filename=$5
+configmap_name=$6
+snippet_filename=$7
+
+# Constants
+SCHEMA_MAP_FILENAME="schema-map.json"
+SUCCESS_CODE=0
+TREE=tree
+EXPECTED_1_ARG=1
+EXPECTED_7_ARGS=7
+INDENTATION_LEVEL_1=1
+INDENTATION_LEVEL_2=2
+INDENTATION_LEVEL_3=3
+INDENTATION_LEVEL_4=4
+INDENTATION_LEVEL_5=5
+
+# Variables
+tmp_location=$(mktemp -d)
+valid_branches=""
+
+# Indents each line of string by adding indent_size*indent_string spaces on the beginning
+# Optional argument is indent_string level, default: 1
+# correct usage example:
+# echo "Sample Text" | indent_string 2
+indent_string() {
+  indent_size=2
+  indent_string=1
+  if [ -n "$1" ]; then indent_string=$1; fi
+  pr -to $(expr "$indent_string" \* "$indent_size")
+}
+
+# Checks whether number of arguments is valid
+# $1 is actual number of arguments
+# $2 is expected number of arguments
+check_arguments() {
+  if [ "$1" -ne "$2" ]; then
+    echo "Incorrect number of arguments"
+    exit 1
+  fi
+}
+
+# Clones all branches selected in $BRANCH from $repo_url
+clone_repo() {
+  for actual_branch in $branches; do
+    clone_branch "$actual_branch"
+  done
+}
+
+# Clones single branch $1 from $repo_url.
+# $1 - branch name
+clone_branch() {
+  check_arguments $# $EXPECTED_1_ARG
+  if [ -d $tmp_location/"$1" ]; then
+    echo "Skipping cloning repository."
+    echo "Branch $1 has already been cloned in the directory ./$tmp_location/$1"
+    echo "To redownload branch remove ./$tmp_location/$1."
+  else
+    echo "Cloning repository from branch $1"
+    git clone --quiet --single-branch --branch "$1" "$repo_url" "$tmp_location/$1" 2>/dev/null
+    result=$?
+    if [ $result -ne $SUCCESS_CODE ] ; then
+      echo "Problem with cloning branch $1."
+      echo "Branch $1 will not be added to spec."
+    else
+      valid_branches="${valid_branches} $1"
+    fi
+  fi
+}
+
+# Creates file with name $configmap_filename
+# Inserts ConfigMap metadata and sets name as $configmap_name
+add_config_map_metadata() {
+  echo "Creating ConfigMap spec file: $configmap_filename"
+  cat << EOF > "$configmap_filename"
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: $configmap_name
+  labels:
+    name: $configmap_name
+  namespace: onap
+data:
+EOF
+}
+
+# For each selected branch:
+#   clones the branch from repository,
+#   adds schemas from branch to ConfigMap spec
+add_schemas() {
+  for actual_branch in $valid_branches; do
+    echo "Adding schemas from branch $actual_branch to spec"
+    add_schemas_from_branch "$actual_branch"
+  done
+}
+
+# Adds schemas from single branch to spec
+# $1 - branch name
+add_schemas_from_branch() {
+  check_arguments $# $EXPECTED_1_ARG
+  schemas=$(ls -g $tmp_location/$1/$schemas_location/*.yaml | awk '{print $NF}')
+  for schema in $schemas; do
+    echo "$1-$(basename $schema): |-" | indent_string $INDENTATION_LEVEL_1
+    cat "$schema" | indent_string $INDENTATION_LEVEL_2
+  done
+} >> "$configmap_filename"
+
+# Generates mapping file for collected schemas directly in spec
+generate_mapping_file() {
+  echo "Generating mapping file in spec"
+  echo "$SCHEMA_MAP_FILENAME"": |-" | indent_string $INDENTATION_LEVEL_1 >> "$configmap_filename"
+  echo "[" | indent_string $INDENTATION_LEVEL_2 >> "$configmap_filename"
+
+  for actual_branch in $valid_branches; do
+    echo "Adding mappings from branch: $actual_branch"
+    add_mappings_from_branch "$actual_branch"
+  done
+
+  truncate -s-2 "$configmap_filename"
+  echo "" >> "$configmap_filename"
+  echo "]" | indent_string $INDENTATION_LEVEL_2 >> "$configmap_filename"
+}
+
+# Adds mappings from single branch directly to spec
+# $1 - branch name
+add_mappings_from_branch() {
+  check_arguments $# $EXPECTED_1_ARG
+  schemas=$(ls -g $tmp_location/$1/$schemas_location/*.yaml | awk '{print $NF}' )
+
+  for schema in $schemas; do
+    repo_endpoint=$(echo "$repo_url" | cut -d/ -f4- | rev | cut -d. -f2- | rev)
+    schema_repo_path=$(echo "$schema" | cut -d/ -f4-)
+    public_url_schemas_location=${repo_url%.*}
+    public_url=$public_url_schemas_location/$TREE/$schema_repo_path
+    local_url=$vendor/$repo_endpoint/$TREE/$schema_repo_path
+
+    echo "{" | indent_string $INDENTATION_LEVEL_3 >> "$configmap_filename"
+    echo "\"publicURL\": \"$public_url\"," | indent_string $INDENTATION_LEVEL_4 >> "$configmap_filename"
+    echo "\"localURL\": \"$local_url\"" | indent_string $INDENTATION_LEVEL_4 >> "$configmap_filename"
+    echo "}," | indent_string $INDENTATION_LEVEL_3 >> "$configmap_filename"
+  done
+}
+
+create_snippet() {
+  echo "Generating snippets in file: $snippet_filename"
+  generate_entries
+
+  cat << EOF > "$snippet_filename"
+Snippets for mounting ConfigMap in DCAE VESCollector Deployment
+=========================================================================
+
+## Description
+These snippets will override existing in VESCollector schemas and mapping file.
+
+No extra configuration in VESCollector is needed with these snippets.
+
+## Snippets
+#### spec.template.spec.containers[0].volumeMounts
+\`\`\`
+        - mountPath: /opt/app/VESCollector/etc/externalRepo
+          name: custom-$vendor-schemas
+\`\`\`
+
+#### spec.template.spec.volumes
+\`\`\`
+      - configMap:
+          defaultMode: 420
+          items:
+          - key: $SCHEMA_MAP_FILENAME
+            path: schema-map.json
+$schemas_entries
+          name: $configmap_name
+        name: custom-$vendor-schemas
+\`\`\`
+EOF
+}
+
+generate_entries() {
+  for actual_branch in $valid_branches; do
+    schemas=$(ls -g $tmp_location/$actual_branch/$schemas_location/*.yaml | awk '{print $NF}')
+    for schema in $schemas; do
+      repo_endpoint=$(echo "$repo_url" | cut -d/ -f4- | rev | cut -d. -f2- | rev)
+      schema_repo_path=$(echo "$schema" | cut -d/ -f4-)
+
+      key="$actual_branch-$(basename "$schema")"
+      path=$vendor/$repo_endpoint/$TREE/$schema_repo_path
+      schemas_entries="$schemas_entries- key: $key\n  path: $path\n"
+    done
+  done
+  schemas_entries=$(echo "$schemas_entries" | indent_string $INDENTATION_LEVEL_5)
+}
+
+# todo add check of global env whether script should be ran
+main() {
+  check_arguments $arguments_number $EXPECTED_7_ARGS
+  clone_repo
+  add_config_map_metadata
+  add_schemas
+  generate_mapping_file
+  create_snippet
+}
+
+main
\ No newline at end of file
diff --git a/external-schema-repo-generator/install.sh b/external-schema-repo-generator/install.sh
new file mode 100755
index 0000000..2f926b6
--- /dev/null
+++ b/external-schema-repo-generator/install.sh
@@ -0,0 +1,58 @@
+#!/bin/sh
+
+# ============LICENSE_START=======================================================
+# OOM
+# ================================================================================
+# Copyright (C) 2020 Nokia. All rights reserved.
+# ================================================================================
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#      http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============LICENSE_END=========================================================
+
+# Arguments renaming
+spec_configmap_filename=$1
+k8s_configmap_name=$2
+
+# Constants
+MAX_SPEC_SIZE=262144
+
+# Alias
+alias kubectl_onap="kubectl -n onap"
+
+# Checks whether ConfigMap $spec_configmap_filename exists
+# When file does not exist exits with return code 1
+checkIfSpecExists() {
+  if [ ! -f "$spec_configmap_filename" ]; then
+    echo "Spec file $spec_configmap_filename does not exist."
+    # todo add location of spec with filename
+    exit 1
+  fi
+}
+
+# Uploads ConfigMap spec $spec_configmap_filename to Kubernetes
+uploadConfigMap() {
+  spec_size=$(stat --printf="%s" "$spec_configmap_filename")
+  if [ "$spec_size" -ge $MAX_SPEC_SIZE ]; then
+    echo "ConfigMap spec file is too long for 'kubectl apply'. Actual spec length: $spec_size, max spec length: $MAX_SPEC_SIZE"
+    echo "Creating new ConfigMap $k8s_configmap_name"
+    kubectl_onap create -f "$spec_configmap_filename"
+  else
+    echo "Applying ConfigMap $k8s_configmap_name"
+    kubectl_onap apply -f "$spec_configmap_filename"
+  fi
+}
+
+main() {
+  checkIfSpecExists
+  uploadConfigMap
+}
+
+main
+