Refactor Distributed Analytics project structure

Modified the project structure to improve maintainability and to add future CI and
integration test support.

Change-Id: Id30bfb1f83f23785a6b5f99e81f42f752d59c0f8
Issue-ID: ONAPARC-280
Signed-off-by: Dileep Ranganathan <dileep.ranganathan@intel.com>

13 files changed, 0 insertions, 400 deletions

diff --git a/vnfs/DAaaS/operator/charts/sparkoperator/.helmignore b/vnfs/DAaaS/operator/charts/sparkoperator/.helmignore
deleted file mode 100644
index b7f6f9f1..00000000
--- a/vnfs/DAaaS/operator/charts/sparkoperator/.helmignore
+++ /dev/null
@@ -1 +0,0 @@
-OWNERS
diff --git a/vnfs/DAaaS/operator/charts/sparkoperator/Chart.yaml b/vnfs/DAaaS/operator/charts/sparkoperator/Chart.yaml
deleted file mode 100644
index 86d0c3ab..00000000
--- a/vnfs/DAaaS/operator/charts/sparkoperator/Chart.yaml
+++ /dev/null
@@ -1,5 +0,0 @@
-apiVersion: v1
-appVersion: "1.0"
-description: A Helm chart for Kubernetes
-name: sparkoperator
-version: 0.1.0
diff --git a/vnfs/DAaaS/operator/charts/sparkoperator/README.md b/vnfs/DAaaS/operator/charts/sparkoperator/README.md
deleted file mode 100755
index ba0f05bc..00000000
--- a/vnfs/DAaaS/operator/charts/sparkoperator/README.md
+++ /dev/null
@@ -1,42 +0,0 @@
-### Helm Chart for Spark Operator
-
-This is the Helm chart for the [Spark-on-Kubernetes Operator](https://github.com/GoogleCloudPlatform/spark-on-k8s-operator).
-
-#### Prerequisites
-
-The Operator requires Kubernetes version 1.8 and above because it relies on garbage collection of custom resources. If customization of driver and executor pods (through mounting custom ConfigMaps and volumes) is desired, then the [Mutating Admission Webhook](https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/quick-start-guide.md#using-the-mutating-admission-webhook) needs to be enabled and it only became beta in Kubernetes 1.9.
-
-#### Installing the chart
-
-The chart can be installed by running:
-
-```bash
-$ helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator
-$ helm install incubator/sparkoperator --namespace spark-operator
-```
-
-Note that you need to use the `--namespace` flag during `helm install` to specify in which namespace you want to install the operator. The namespace can be existing or not. When it's not available, Helm would take care of creating the namespace. Note that this namespace has no relation to the namespace where you would like to deploy Spark jobs (i.e. the setting `sparkJobNamespace` shown in the table below). They can be the same namespace or different ones. 
-
-#### Configuration
-
-The following table lists the configurable parameters of the Spark operator chart and their default values.
-
-| Parameter                 | Description                                                  | Default                                |
-| ------------------------- | ------------------------------------------------------------ | -------------------------------------- |
-| `operatorImageName`       | The name of the operator image                               | `gcr.io/spark-operator/spark-operator` |
-| `operatorVersion`         | The version of the operator to install                       | `v2.4.0-v1beta1-latest`               |
-| `imagePullPolicy`         | Docker image pull policy                                     | `IfNotPresent`                         |
-| `sparkJobNamespace`       | K8s namespace where Spark jobs are to be deployed            | `default`                              |
-| `enableWebhook`           | Whether to enable mutating admission webhook                 | false                                   |
-| `enableMetrics`           | Whether to expose metrics to be scraped by Premetheus        | true                                   |
-| `controllerThreads`       | Number of worker threads used by the SparkApplication controller | 10                                     |
-| `ingressUrlFormat` | Ingress URL format | "" |
-| `installCrds`             | Whether to install CRDs                                      | true                                   |
-| `metricsPort`             | Port for the metrics endpoint                                | 10254                                  |
-| `metricsEndpoint`         | Metrics endpoint                                             | "/metrics"                             |
-| `metricsPrefix`           | Prefix for the metrics                                       | ""                                     |
-| `resyncInterval`          | Informer resync interval in seconds                          | 30                                     |
-| `webhookPort`             | Service port of the webhook server                           | 8080                                   |
-
-Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. 
-
diff --git a/vnfs/DAaaS/operator/charts/sparkoperator/templates/_helpers.tpl b/vnfs/DAaaS/operator/charts/sparkoperator/templates/_helpers.tpl
deleted file mode 100644
index 741b500d..00000000
--- a/vnfs/DAaaS/operator/charts/sparkoperator/templates/_helpers.tpl
+++ /dev/null
@@ -1,48 +0,0 @@
-{{/* vim: set filetype=mustache: */}}
-{{/*
-Expand the name of the chart.
-*/}}
-{{- define "sparkoperator.name" -}}
-{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
-{{- end -}}
- {{/*
-Create a default fully qualified app name.
-We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
-If release name contains chart name it will be used as a full name.
-*/}}
-{{- define "sparkoperator.fullname" -}}
-{{- if .Values.fullnameOverride -}}
-{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
-{{- else -}}
-{{- $name := default .Chart.Name .Values.nameOverride -}}
-{{- if contains $name .Release.Name -}}
-{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
-{{- else -}}
-{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
-{{- end -}}
-{{- end -}}
-{{- end -}}
- {{/*
-Create chart name and version as used by the chart label.
-*/}}
-{{- define "sparkoperator.chart" -}}
-{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
-{{- end -}}
- {{/*
-Create the name of the service account to use
-*/}}
-{{- define "sparkoperator.serviceAccountName" -}}
-{{- if .Values.serviceAccounts.sparkoperator.create -}}
-    {{ default (include "sparkoperator.fullname" .) .Values.serviceAccounts.sparkoperator.name }}
-{{- else -}}
-    {{ default "default" .Values.serviceAccounts.sparkoperator.name }}
-{{- end -}}
-{{- end -}}
-{{- define "spark.serviceAccountName" -}}
-{{- if .Values.serviceAccounts.spark.create -}}
-    {{ $sparkServiceaccount := printf "%s-%s" .Release.Name "spark" }}
-    {{ default $sparkServiceaccount .Values.serviceAccounts.spark.name }}
-{{- else -}}
-    {{ default "default" .Values.serviceAccounts.spark.name }}
-{{- end -}}
-{{- end -}}
diff --git a/vnfs/DAaaS/operator/charts/sparkoperator/templates/spark-operator-deployment.yaml b/vnfs/DAaaS/operator/charts/sparkoperator/templates/spark-operator-deployment.yaml
deleted file mode 100755
index fdfc51a2..00000000
--- a/vnfs/DAaaS/operator/charts/sparkoperator/templates/spark-operator-deployment.yaml
+++ /dev/null
@@ -1,79 +0,0 @@
-# If the admission webhook is enabled, then a post-install step is required
-# to generate and install the secret in the operator namespace.
-
-# In the post-install hook, the token corresponding to the operator service account
-# is used to authenticate with the Kubernetes API server to install the secret bundle.
-
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: {{ include "sparkoperator.fullname" . }}
-  labels:
-    app.kubernetes.io/name: {{ include "sparkoperator.name" . }}
-    helm.sh/chart: {{ include "sparkoperator.chart" . }}
-    app.kubernetes.io/instance: {{ .Release.Name }}
-    app.kubernetes.io/managed-by: {{ .Release.Service }}
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: {{ include "sparkoperator.name" . }}
-      app.kubernetes.io/version: {{ .Values.operatorVersion }}
-  strategy:
-    type: Recreate
-  template:
-    metadata:
-      {{- if .Values.enableMetrics }}
-      annotations:
-        prometheus.io/scrape: "true"
-        prometheus.io/port: "{{ .Values.metricsPort }}"
-        prometheus.io/path: {{ .Values.metricsEndpoint }}
-      {{- end }}
-      labels:
-        app.kubernetes.io/name: {{ include "sparkoperator.name" . }}
-        app.kubernetes.io/version: {{ .Values.operatorVersion }}
-      initializers:
-        pending: []
-    spec:
-      serviceAccountName: {{ include "sparkoperator.serviceAccountName" . }}
-      {{- if .Values.enableWebhook }}
-      volumes:
-        - name: webhook-certs
-          secret:
-            secretName: spark-webhook-certs
-      {{- end }}
-      containers:
-      - name: sparkoperator
-        image: {{ .Values.operatorImageName }}:{{ .Values.operatorVersion }}
-        imagePullPolicy: {{ .Values.imagePullPolicy }}
-        {{- if .Values.enableWebhook }}
-        volumeMounts:
-          - name: webhook-certs
-            mountPath: /etc/webhook-certs
-        {{- end }}
-        {{- if .Values.enableMetrics }}
-        ports:
-          - containerPort: {{ .Values.metricsPort }}
-        {{ end }}
-        args:
-        - -v=2
-        - -namespace={{ .Values.sparkJobNamespace }}
-        - -ingress-url-format={{ .Values.ingressUrlFormat }}
-        - -install-crds={{ .Values.installCrds }}
-        - -controller-threads={{ .Values.controllerThreads }}
-        - -resync-interval={{ .Values.resyncInterval }}
-        - -logtostderr
-        {{- if .Values.enableMetrics }}
-        - -enable-metrics=true
-        - -metrics-labels=app_type
-        - -metrics-port={{ .Values.metricsPort }}
-        - -metrics-endpoint={{ .Values.metricsEndpoint }}
-        - -metrics-prefix={{ .Values.metricsPrefix }}
-        {{- end }}
-        {{- if .Values.enableWebhook }}
-        - -enable-webhook=true
-        - -webhook-svc-namespace={{ .Release.Namespace }}
-        - -webhook-port={{ .Values.webhookPort }}
-        - -webhook-svc-name={{ .Release.Name }}-webhook
-        - -webhook-config-name={{ include "sparkoperator.fullname" . }}-webhook-config
-        {{- end }}
diff --git a/vnfs/DAaaS/operator/charts/sparkoperator/templates/spark-operator-rbac.yaml b/vnfs/DAaaS/operator/charts/sparkoperator/templates/spark-operator-rbac.yaml
deleted file mode 100755
index bd5fd3fe..00000000
--- a/vnfs/DAaaS/operator/charts/sparkoperator/templates/spark-operator-rbac.yaml
+++ /dev/null
@@ -1,55 +0,0 @@
-{{- if .Values.rbac.create }}
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
-  name: {{ include "sparkoperator.fullname" . }}-cr
-  labels:
-    app.kubernetes.io/name: {{ include "sparkoperator.name" . }}
-    helm.sh/chart: {{ include "sparkoperator.chart" . }}
-    app.kubernetes.io/instance: {{ .Release.Name }}
-    app.kubernetes.io/managed-by: {{ .Release.Service }}
-rules:
-- apiGroups: [""]
-  resources: ["pods"]
-  verbs: ["*"]
-- apiGroups: [""]
-  resources: ["services", "configmaps", "secrets"]
-  verbs: ["create", "get", "delete"]
-- apiGroups: ["extensions"]
-  resources: ["ingresses"]
-  verbs: ["create", "get", "delete"]
-- apiGroups: [""]
-  resources: ["nodes"]
-  verbs: ["get"]
-- apiGroups: [""]
-  resources: ["events"]
-  verbs: ["create", "update", "patch"]
-- apiGroups: ["apiextensions.k8s.io"]
-  resources: ["customresourcedefinitions"]
-  verbs: ["create", "get", "update", "delete"]
-- apiGroups: ["admissionregistration.k8s.io"]
-  resources: ["mutatingwebhookconfigurations"]
-  verbs: ["create", "get", "update", "delete"]
-- apiGroups: ["sparkoperator.k8s.io"]
-  resources: ["sparkapplications", "scheduledsparkapplications"]
-  verbs: ["*"]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
-  name: {{ include "sparkoperator.fullname" . }}-crb
-  namespace: {{ .Release.Namespace }}
-  labels:
-    app.kubernetes.io/name: {{ include "sparkoperator.name" . }}
-    helm.sh/chart: {{ include "sparkoperator.chart" . }}
-    app.kubernetes.io/instance: {{ .Release.Name }}
-    app.kubernetes.io/managed-by: {{ .Release.Service }}
-subjects:
-  - kind: ServiceAccount
-    name: {{ include "sparkoperator.serviceAccountName" . }}
-    namespace: {{ .Release.Namespace }}
-roleRef:
-  kind: ClusterRole
-  name: {{ include "sparkoperator.fullname" . }}-cr
-  apiGroup: rbac.authorization.k8s.io
-{{- end }}
diff --git a/vnfs/DAaaS/operator/charts/sparkoperator/templates/spark-operator-serviceaccount.yaml b/vnfs/DAaaS/operator/charts/sparkoperator/templates/spark-operator-serviceaccount.yaml
deleted file mode 100755
index 5216f8dd..00000000
--- a/vnfs/DAaaS/operator/charts/sparkoperator/templates/spark-operator-serviceaccount.yaml
+++ /dev/null
@@ -1,11 +0,0 @@
-{{- if .Values.serviceAccounts.sparkoperator.create }}
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: {{ include "sparkoperator.serviceAccountName" . }}
-  labels:
-    app.kubernetes.io/name: {{ include "sparkoperator.name" . }}
-    helm.sh/chart: {{ include "sparkoperator.chart" . }}
-    app.kubernetes.io/instance: {{ .Release.Name }}
-    app.kubernetes.io/managed-by: {{ .Release.Service }}
-{{- end }}
diff --git a/vnfs/DAaaS/operator/charts/sparkoperator/templates/spark-rbac.yaml b/vnfs/DAaaS/operator/charts/sparkoperator/templates/spark-rbac.yaml
deleted file mode 100755
index fa066053..00000000
--- a/vnfs/DAaaS/operator/charts/sparkoperator/templates/spark-rbac.yaml
+++ /dev/null
@@ -1,44 +0,0 @@
-{{- if and (.Values.rbac.create) (ne .Values.sparkJobNamespace "") }}
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  namespace: {{ .Values.sparkJobNamespace }}
-  name: spark-role
-  labels:
-    app.kubernetes.io/name: {{ include "sparkoperator.name" . }}
-    helm.sh/chart: {{ include "sparkoperator.chart" . }}
-    app.kubernetes.io/instance: {{ .Release.Name }}
-    app.kubernetes.io/managed-by: {{ .Release.Service }}
-rules:
-- apiGroups:
-  - "" # "" indicates the core API group
-  resources:
-  - "pods"
-  verbs:
-  - "*"
-- apiGroups:
-  - "" # "" indicates the core API group
-  resources:
-  - "services"
-  verbs:
-  - "*"
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: spark-role-binding
-  namespace: {{ .Values.sparkJobNamespace }}
-  labels:
-    app.kubernetes.io/name: {{ include "sparkoperator.name" . }}
-    helm.sh/chart: {{ include "sparkoperator.chart" . }}
-    app.kubernetes.io/instance: {{ .Release.Name }}
-    app.kubernetes.io/managed-by: {{ .Release.Service }}
-subjects:
-- kind: ServiceAccount
-  name: {{ include "spark.serviceAccountName" . }}
-  namespace: {{ .Values.sparkJobNamespace }}
-roleRef:
-  kind: Role
-  name: spark-role
-  apiGroup: rbac.authorization.k8s.io
-{{- end }}
diff --git a/vnfs/DAaaS/operator/charts/sparkoperator/templates/spark-serviceaccount.yaml b/vnfs/DAaaS/operator/charts/sparkoperator/templates/spark-serviceaccount.yaml
deleted file mode 100755
index bb0e55ea..00000000
--- a/vnfs/DAaaS/operator/charts/sparkoperator/templates/spark-serviceaccount.yaml
+++ /dev/null
@@ -1,12 +0,0 @@
-{{- if .Values.serviceAccounts.spark.create }}
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: {{ include "spark.serviceAccountName" . }}
-  namespace: {{ .Values.sparkJobNamespace }}
-  labels:
-    app.kubernetes.io/name: {{ include "sparkoperator.name" . }}
-    helm.sh/chart: {{ include "sparkoperator.chart" . }}
-    app.kubernetes.io/instance: {{ .Release.Name }}
-    app.kubernetes.io/managed-by: {{ .Release.Service }}
-{{- end }}
diff --git a/vnfs/DAaaS/operator/charts/sparkoperator/templates/webhook-cleanup-job.yaml b/vnfs/DAaaS/operator/charts/sparkoperator/templates/webhook-cleanup-job.yaml
deleted file mode 100755
index d6d9df7c..00000000
--- a/vnfs/DAaaS/operator/charts/sparkoperator/templates/webhook-cleanup-job.yaml
+++ /dev/null
@@ -1,32 +0,0 @@
-{{ if .Values.enableWebhook }}
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: {{ include "sparkoperator.fullname" . }}-cleanup
-  annotations:
-    "helm.sh/hook": pre-delete, pre-upgrade
-    "helm.sh/hook-delete-policy": hook-succeeded
-  labels:
-    app.kubernetes.io/name: {{ include "sparkoperator.name" . }}
-    helm.sh/chart: {{ include "sparkoperator.chart" . }}
-    app.kubernetes.io/instance: {{ .Release.Name }}
-    app.kubernetes.io/managed-by: {{ .Release.Service }}
-spec:
-  template:
-    spec:
-      serviceAccountName: {{ include "sparkoperator.serviceAccountName" . }}
-      restartPolicy: OnFailure
-      containers:
-      - name: main
-        image: {{ .Values.operatorImageName }}:{{ .Values.operatorVersion }}
-        imagePullPolicy: {{ .Values.imagePullPolicy }}
-        command:
-        - "/bin/sh"
-        - "-c"
-        - "curl -ik \
-          -X DELETE \
-          -H \"Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" \
-          -H \"Accept: application/json\" \
-          -H \"Content-Type: application/json\" \
-          https://kubernetes.default.svc/api/v1/namespaces/{{ .Release.Namespace }}/secrets/spark-webhook-certs"
-{{ end }}
diff --git a/vnfs/DAaaS/operator/charts/sparkoperator/templates/webhook-init-job.yaml b/vnfs/DAaaS/operator/charts/sparkoperator/templates/webhook-init-job.yaml
deleted file mode 100755
index a42c3097..00000000
--- a/vnfs/DAaaS/operator/charts/sparkoperator/templates/webhook-init-job.yaml
+++ /dev/null
@@ -1,24 +0,0 @@
-{{ if .Values.enableWebhook }}
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: {{ include "sparkoperator.fullname" . }}-init
-  annotations:
-    "helm.sh/hook": post-install, post-upgrade
-    "helm.sh/hook-delete-policy": hook-succeeded
-  labels:
-    app.kubernetes.io/name: {{ include "sparkoperator.name" . }}
-    helm.sh/chart: {{ include "sparkoperator.chart" . }}
-    app.kubernetes.io/instance: {{ .Release.Name }}
-    app.kubernetes.io/managed-by: {{ .Release.Service }}
-spec:
-  template:
-    spec:
-      serviceAccountName: {{ include "sparkoperator.serviceAccountName" . }}
-      restartPolicy: OnFailure
-      containers:
-      - name: main
-        image: {{ .Values.operatorImageName }}:{{ .Values.operatorVersion }}
-        imagePullPolicy: {{ .Values.imagePullPolicy }}
-        command: ["/usr/bin/gencerts.sh", "-n", "{{ .Release.Namespace }}", "-s", "{{ .Release.Name }}-webhook", "-p"]
-{{ end }}
diff --git a/vnfs/DAaaS/operator/charts/sparkoperator/templates/webhook-service.yaml b/vnfs/DAaaS/operator/charts/sparkoperator/templates/webhook-service.yaml
deleted file mode 100755
index 42c5bc62..00000000
--- a/vnfs/DAaaS/operator/charts/sparkoperator/templates/webhook-service.yaml
+++ /dev/null
@@ -1,19 +0,0 @@
-{{ if .Values.enableWebhook }}
-kind: Service
-apiVersion: v1
-metadata:
-  name: {{ .Release.Name }}-webhook
-  labels:
-    app.kubernetes.io/name: {{ include "sparkoperator.name" . }}
-    helm.sh/chart: {{ include "sparkoperator.chart" . }}
-    app.kubernetes.io/instance: {{ .Release.Name }}
-    app.kubernetes.io/managed-by: {{ .Release.Service }}
-spec:
-  ports:
-  - port: 443
-    targetPort: 8080
-    name: webhook
-  selector:
-    app.kubernetes.io/name: {{ include "sparkoperator.name" . }}
-    app.kubernetes.io/version: {{ .Values.operatorVersion }}
-{{ end }}
diff --git a/vnfs/DAaaS/operator/charts/sparkoperator/values.yaml b/vnfs/DAaaS/operator/charts/sparkoperator/values.yaml
deleted file mode 100644
index bfb03eab..00000000
--- a/vnfs/DAaaS/operator/charts/sparkoperator/values.yaml
+++ /dev/null
@@ -1,28 +0,0 @@
-operatorImageName: gcr.io/spark-operator/spark-operator
-operatorVersion: v2.4.0-v1beta1-latest
-imagePullPolicy: IfNotPresent
-
-rbac:
-  create: true
-
-serviceAccounts:
-  spark:
-    create: true
-    name:
-  sparkoperator:
-    create: true
-    name:
-
-sparkJobNamespace: ""
-
-enableWebhook: false
-enableMetrics: true
-
-controllerThreads: 10
-ingressUrlFormat: ""
-installCrds: true
-metricsPort: 10254
-metricsEndpoint: "/metrics"
-metricsPrefix: ""
-resyncInterval: 30
-webhookPort: 8080