aboutsummaryrefslogtreecommitdiffstats
path: root/test
diff options
context:
space:
mode:
authorPawel Wieczorek <p.wieczorek2@samsung.com>2020-07-16 16:15:06 +0200
committerPawel Wieczorek <p.wieczorek2@samsung.com>2020-07-28 15:06:43 +0200
commit30e199a70b32a6256c2a148eec870800ef1fbefc (patch)
tree148c814d55b920f04a1fa0ce5f4a68896f51ef6f /test
parent3301d5325c59d3e721fd2ec341318c5a0ede0b0c (diff)
Import upstream component version inspection tool
This patch adds utility to check versions of binaries available in Docker containers run on Kubernetes cluster. It has been contributed by: kkkk-k <kkkk.k@samsung.com> Several minor changes were made to comply with ONAP CI linter rules. Issue-ID: INT-1571 Change-Id: Id0e4b557212dec1bf8d2bac580968d69e2cf5595 Signed-off-by: Pawel Wieczorek <p.wieczorek2@samsung.com>
Diffstat (limited to 'test')
-rw-r--r--test/security/check_versions/.gitignore5
-rw-r--r--test/security/check_versions/README.md105
-rw-r--r--test/security/check_versions/env/Vagrantfile35
-rw-r--r--test/security/check_versions/env/configuration/namespaces.yaml45
-rw-r--r--test/security/check_versions/env/configuration/terminated.yaml17
-rw-r--r--test/security/check_versions/env/configuration/versions.yaml112
-rw-r--r--test/security/check_versions/env/requirements-dev.txt9
-rw-r--r--test/security/check_versions/env/requirements.txt6
-rw-r--r--test/security/check_versions/src/k8s_bin_versions_inspector.py691
-rw-r--r--test/security/check_versions/tests/conftest.py12
-rw-r--r--test/security/check_versions/tests/test_gather_containers_informations.py38
-rw-r--r--test/security/check_versions/tests/test_list_all_containers.py52
-rw-r--r--test/security/check_versions/tests/test_main.py84
-rw-r--r--test/security/check_versions/tests/test_sync_post_namespaced_pod_exec.py74
-rw-r--r--test/security/check_versions/tests/test_verify_versions_acceptability.py58
-rw-r--r--test/security/check_versions/tox.ini17
16 files changed, 1360 insertions, 0 deletions
diff --git a/test/security/check_versions/.gitignore b/test/security/check_versions/.gitignore
new file mode 100644
index 000000000..db6444b3c
--- /dev/null
+++ b/test/security/check_versions/.gitignore
@@ -0,0 +1,5 @@
+.pytest_cache/
+__pycache__/
+/env/.vagrant
+/temp/
+/.tox/
diff --git a/test/security/check_versions/README.md b/test/security/check_versions/README.md
new file mode 100644
index 000000000..3934ca77a
--- /dev/null
+++ b/test/security/check_versions/README.md
@@ -0,0 +1,105 @@
+# Kubernetes Binaries Versions Inspector
+
+**Kubernetes Binaries Versions Inspector** (`k8s_bin_versions_inspector`) is a
+python module for verifying versions of CPython and OpenJDK binaries installed
+in the kubernetes cluster containers.
+
+## Commands
+
+### Creating environment
+
+All development and testing process, should be done in prepared virtual machine,
+that is containing development environment for this project. Vagrant plugins,
+that are required to start virtual machine: `vagrant-libvirt`, `vagrant-reload`,
+`vagrant-sshfs`.
+
+```bash
+cd env
+vagrant up
+vagrant ssh
+```
+
+### Install dependencies
+
+To install dependencies for normal usage of script, run this command.
+
+```bash
+pip3 install -r env/requirements.txt
+```
+
+### Code formatting
+
+```bash
+black src tests
+```
+
+### Code static analysis
+
+```bash
+pylint -d C0330 src
+```
+
+### Automatic tests
+
+To running the automated tests is required to have properly configured
+kubernetes cluster, which is in the virtual machine, that is containing
+development environment.
+
+```bash
+PYTHONPATH=src pytest -vv -s tests
+```
+
+### Removing caches
+
+```bash
+find -name __pycache__ -exec rm -Rf {} +
+find -name .pytest_cache -exec rm -Rf {} +
+```
+
+## Acceptable format
+
+Example of the acceptable file format:
+
+```yaml
+python:
+ - 3.6.9
+ - 3.7.3
+java:
+ - 11.0.7
+```
+
+## Paths research
+
+Commands to research for the paths
+of the software binaries in multiple docker images:
+
+```bash
+docker run --entrypoint /bin/sh python:buster -c "which python"
+docker run --entrypoint /bin/sh python:alpine -c "which python"
+docker run --entrypoint /bin/sh python:slim -c "which python"
+docker run --entrypoint /bin/sh python:2-buster -c "which python"
+docker run --entrypoint /bin/sh python:2-alpine -c "which python"
+docker run --entrypoint /bin/sh python:2-slim -c "which python"
+docker run --entrypoint /bin/sh ubuntu:bionic -c "apt-get update && apt-get install -y python && which python"
+docker run --entrypoint /bin/sh ubuntu:bionic -c "apt-get update && apt-get install -y python3 && which python3"
+docker run --entrypoint /bin/sh openjdk -c "type java"
+```
+
+## Todo
+
+List of features, that should be implemented:
+
+- Complete license and copyrights variables.
+- Find a way, to safe searching of the container files from Kubernetes API.
+- Parallelization of executing binaries on the single container.
+- Parallelization of versions determination in multiple containers.
+- Support for determination the old versions of OpenJDK (attribute `-version`).
+- Deleting namespace from cluster in development environment (for example,
+ during cluster reset), cause hanging in namespace terminating state.
+- Find a nicer way to extracting exit code from execution result.
+
+## Links
+
+- <https://github.com/kubernetes-client/python>
+- <https://github.com/kubernetes-client/python/issues/812>
+- <https://success.docker.com/article/kubernetes-namespace-stuck-in-terminating>
diff --git a/test/security/check_versions/env/Vagrantfile b/test/security/check_versions/env/Vagrantfile
new file mode 100644
index 000000000..28abbc504
--- /dev/null
+++ b/test/security/check_versions/env/Vagrantfile
@@ -0,0 +1,35 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+Vagrant.configure("2") do |config|
+
+ config.vm.provider :libvirt do |libvirt|
+ libvirt.default_prefix = "k8s_bin_versions_inspector";
+ libvirt.driver = "kvm";
+ libvirt.cpus = 6;
+ libvirt.memory = 12288;
+ end
+
+ config.vm.box = "generic/ubuntu1804";
+ config.vm.hostname = "k8s-bin-versions-inspector";
+ config.vm.synced_folder ".", "/vagrant", disabled: true;
+ config.vm.synced_folder "..", "/home/vagrant/k8s_bin_versions_inspector", type: :sshfs;
+
+ config.vm.provision "shell", inline: <<-end
+ export DEBIAN_FRONTEND=noninteractive &&\
+ apt-get update &&\
+ apt-get upgrade -y &&\
+ apt-get dist-upgrade -y &&\
+ apt-get install -y python3 python3-pip snap git vim net-tools htop &&\
+ pip3 install --system -r /home/vagrant/k8s_bin_versions_inspector/env/requirements-dev.txt &&\
+ snap install --classic microk8s &&\
+ usermod -a -G microk8s vagrant
+ end
+ config.vm.provision :reload;
+ config.vm.provision "shell", privileged: false, inline: <<-end
+ microk8s reset &&\
+ microk8s config > /home/vagrant/.kube/config &&\
+ microk8s kubectl apply -f /home/vagrant/k8s_bin_versions_inspector/env/configuration
+ end
+end
+
diff --git a/test/security/check_versions/env/configuration/namespaces.yaml b/test/security/check_versions/env/configuration/namespaces.yaml
new file mode 100644
index 000000000..f300cc7da
--- /dev/null
+++ b/test/security/check_versions/env/configuration/namespaces.yaml
@@ -0,0 +1,45 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+ name: ingress-nginx
+
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: kbvi-test-ingress-nginx
+ namespace: ingress-nginx
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: kbvi-test-ingress-nginx
+ template:
+ metadata:
+ labels:
+ app: kbvi-test-ingress-nginx
+ spec:
+ containers:
+ - name: echo-server
+ image: jmalloc/echo-server
+
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: kbvi-test-kube-system
+ namespace: kube-system
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: kbvi-test-kube-system
+ template:
+ metadata:
+ labels:
+ app: kbvi-test-kube-system
+ spec:
+ containers:
+ - name: echo-server
+ image: jmalloc/echo-server
diff --git a/test/security/check_versions/env/configuration/terminated.yaml b/test/security/check_versions/env/configuration/terminated.yaml
new file mode 100644
index 000000000..dd6ce829d
--- /dev/null
+++ b/test/security/check_versions/env/configuration/terminated.yaml
@@ -0,0 +1,17 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: kbvi-test-terminated
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: kbvi-test-terminated
+ template:
+ metadata:
+ labels:
+ app: kbvi-test-terminated
+ spec:
+ containers:
+ - name: python
+ image: python
diff --git a/test/security/check_versions/env/configuration/versions.yaml b/test/security/check_versions/env/configuration/versions.yaml
new file mode 100644
index 000000000..75b7f7b85
--- /dev/null
+++ b/test/security/check_versions/env/configuration/versions.yaml
@@ -0,0 +1,112 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: kbvi-test-python-jupyter
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: kbvi-test-python-jupyter
+ template:
+ metadata:
+ labels:
+ app: kbvi-test-python-jupyter
+ spec:
+ containers:
+ - name: jupyter
+ image: jupyter/base-notebook
+
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: kbvi-test-python-jupyter-old
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: kbvi-test-python-jupyter-old
+ template:
+ metadata:
+ labels:
+ app: kbvi-test-python-jupyter-old
+ spec:
+ containers:
+ - name: jupyter-old
+ image: jupyter/base-notebook:ff922f8f533a
+
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: kbvi-test-python-stderr-filebeat
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: kbvi-test-python-stderr-filebeat
+ template:
+ metadata:
+ labels:
+ app: kbvi-test-python-stderr-filebeat
+ spec:
+ containers:
+ - name: filebeat
+ image: docker.elastic.co/beats/filebeat:5.5.0
+
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: kbvi-test-java-keycloak
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: kbvi-test-java-keycloak
+ template:
+ metadata:
+ labels:
+ app: kbvi-test-java-keycloak
+ spec:
+ containers:
+ - name: keycloak
+ image: jboss/keycloak
+
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: kbvi-test-java-keycloak-old
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: kbvi-test-java-keycloak-old
+ template:
+ metadata:
+ labels:
+ app: kbvi-test-java-keycloak-old
+ spec:
+ containers:
+ - name: keycloak-old
+ image: jboss/keycloak:8.0.0
+
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: kbvi-test-java-keycloak-very-old
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: kbvi-test-java-keycloak-very-old
+ template:
+ metadata:
+ labels:
+ app: kbvi-test-java-keycloak-very-old
+ spec:
+ containers:
+ - name: keycloak-very-old
+ image: jboss/keycloak:2.0.0.Final
diff --git a/test/security/check_versions/env/requirements-dev.txt b/test/security/check_versions/env/requirements-dev.txt
new file mode 100644
index 000000000..1ced42c04
--- /dev/null
+++ b/test/security/check_versions/env/requirements-dev.txt
@@ -0,0 +1,9 @@
+cerberus
+dataclasses
+kubernetes
+pyyaml
+tabulate
+black
+pylint
+pytest
+
diff --git a/test/security/check_versions/env/requirements.txt b/test/security/check_versions/env/requirements.txt
new file mode 100644
index 000000000..e81358f72
--- /dev/null
+++ b/test/security/check_versions/env/requirements.txt
@@ -0,0 +1,6 @@
+cerberus
+dataclasses
+kubernetes
+pyyaml
+tabulate
+
diff --git a/test/security/check_versions/src/k8s_bin_versions_inspector.py b/test/security/check_versions/src/k8s_bin_versions_inspector.py
new file mode 100644
index 000000000..bda7322f5
--- /dev/null
+++ b/test/security/check_versions/src/k8s_bin_versions_inspector.py
@@ -0,0 +1,691 @@
+#!/usr/bin/env python3
+
+# COPYRIGHT NOTICE STARTS HERE
+#
+# Copyright 2020 Samsung Electronics Co., Ltd.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# COPYRIGHT NOTICE ENDS HERE
+
+"""
+k8s_bin_versions_inspector is a module for verifying versions of CPython and
+OpenJDK binaries installed in the kubernetes cluster containers.
+"""
+
+__title__ = "k8s_bin_versions_inspector"
+__summary__ = (
+ "Module for verifying versions of CPython and OpenJDK binaries installed"
+ " in the kubernetes cluster containers."
+)
+__version__ = "0.1.0"
+__author__ = "kkkk.k@samsung.com"
+__license__ = "Apache-2.0"
+__copyright__ = "Copyright 2020 Samsung Electronics Co., Ltd."
+
+from typing import Iterable, List, Optional, Pattern, Union
+
+import argparse
+import dataclasses
+import itertools
+import json
+import pathlib
+import pprint
+import re
+import string
+import sys
+import tabulate
+import yaml
+
+import cerberus
+import kubernetes
+
+
+def parse_argv(argv: Optional[List[str]] = None) -> argparse.Namespace:
+ """Function for parsing command line arguments.
+
+ Args:
+ argv: Unparsed list of command line arguments.
+
+ Returns:
+ Namespace with values from parsed arguments.
+ """
+
+ epilog = (
+ f"Author: {__author__}\n"
+ f"License: {__license__}\n"
+ f"Copyright: {__copyright__}\n"
+ )
+
+ parser = argparse.ArgumentParser(
+ formatter_class=argparse.RawTextHelpFormatter,
+ prog=__title__,
+ description=__summary__,
+ epilog=epilog,
+ add_help=False,
+ )
+
+ parser.add_argument("-c", "--config-file", help="Name of the kube-config file.")
+
+ parser.add_argument(
+ "-s",
+ "--field-selector",
+ default="",
+ help="Kubernetes field selector, to filter out containers objects.",
+ )
+
+ parser.add_argument(
+ "-o",
+ "--output-file",
+ type=pathlib.Path,
+ help="Path to file, where output will be saved.",
+ )
+
+ parser.add_argument(
+ "-f",
+ "--output-format",
+ choices=("tabulate", "pprint", "json"),
+ default="tabulate",
+ help="Format of the output file (tabulate, pprint, json).",
+ )
+
+ parser.add_argument(
+ "-i",
+ "--ignore-empty",
+ action="store_true",
+ help="Ignore containers without any versions.",
+ )
+
+ parser.add_argument(
+ "-a",
+ "--acceptable",
+ type=pathlib.Path,
+ help="Path to YAML file, with list of acceptable software versions.",
+ )
+
+ parser.add_argument(
+ "-d",
+ "--debug",
+ action="store_true",
+ help="Enable debugging mode in the k8s API.",
+ )
+
+ parser.add_argument(
+ "-q",
+ "--quiet",
+ action="store_true",
+ help="Suppress printing text on standard output.",
+ )
+
+ parser.add_argument(
+ "-V",
+ "--version",
+ action="version",
+ version=f"{__title__} {__version__}",
+ help="Display version information and exit.",
+ )
+
+ parser.add_argument(
+ "-h", "--help", action="help", help="Display this help text and exit."
+ )
+
+ args = parser.parse_args(argv)
+
+ return args
+
+
+@dataclasses.dataclass
+class ContainerExtra:
+ "Data class, to storage extra informations about container."
+
+ running: bool
+ image: str
+ identifier: str
+
+
+@dataclasses.dataclass
+class ContainerVersions:
+ "Data class, to storage software versions from container."
+
+ python: list
+ java: list
+
+
+@dataclasses.dataclass
+class ContainerInfo:
+ "Data class, to storage multiple informations about container."
+
+ namespace: str
+ pod: str
+ container: str
+ extra: ContainerExtra
+ versions: ContainerVersions = None
+
+
+def is_container_running(
+ status: kubernetes.client.models.v1_container_status.V1ContainerStatus,
+) -> bool:
+ """Function to determine if k8s cluster container is in running state.
+
+ Args:
+ status: Single item from container_statuses list, that represents container status.
+
+ Returns:
+ If container is in running state.
+ """
+
+ if status.state.terminated:
+ return False
+
+ if status.state.waiting:
+ return False
+
+ if not status.state.running:
+ return False
+
+ return True
+
+
+def list_all_containers(
+ api: kubernetes.client.api.core_v1_api.CoreV1Api, field_selector: str,
+) -> Iterable[ContainerInfo]:
+ """Get list of all containers names.
+
+ Args:
+ api: Client of the k8s cluster API.
+ field_selector: Kubernetes field selector, to filter out containers objects.
+
+ Yields:
+ Objects for all containers in k8s cluster.
+ """
+
+ pods = api.list_pod_for_all_namespaces(field_selector=field_selector).items
+
+ containers_statuses = (
+ (pod.metadata.namespace, pod.metadata.name, pod.status.container_statuses)
+ for pod in pods
+ if pod.status.container_statuses
+ )
+
+ containers_status = (
+ itertools.product([namespace], [pod], statuses)
+ for namespace, pod, statuses in containers_statuses
+ )
+
+ containers_chained = itertools.chain.from_iterable(containers_status)
+
+ containers_fields = (
+ (
+ namespace,
+ pod,
+ status.name,
+ is_container_running(status),
+ status.image,
+ status.container_id,
+ )
+ for namespace, pod, status in containers_chained
+ )
+
+ container_items = (
+ ContainerInfo(
+ namespace, pod, container, ContainerExtra(running, image, identifier)
+ )
+ for namespace, pod, container, running, image, identifier in containers_fields
+ )
+
+ yield from container_items
+
+
+def sync_post_namespaced_pod_exec(
+ api: kubernetes.client.api.core_v1_api.CoreV1Api,
+ container: ContainerInfo,
+ command: Union[List[str], str],
+) -> dict:
+ """Function to execute command on selected container.
+
+ Args:
+ api: Client of the k8s cluster API.
+ container: Object, that represents container in k8s cluster.
+ command: Command to execute as a list of arguments or single string.
+
+ Returns:
+ Dictionary that store informations about command execution.
+ * stdout - Standard output captured from execution.
+ * stderr - Standard error captured from execution.
+ * error - Error object that was received from kubernetes API.
+ * code - Exit code returned by executed process
+ or -1 if container is not running
+ or -2 if other failure occurred.
+ """
+
+ try:
+ client = kubernetes.stream.stream(
+ api.connect_post_namespaced_pod_exec,
+ namespace=container.namespace,
+ name=container.pod,
+ container=container.container,
+ command=command,
+ stderr=True,
+ stdin=False,
+ stdout=True,
+ tty=False,
+ _request_timeout=1.0,
+ _preload_content=False,
+ )
+
+ except kubernetes.client.rest.ApiException:
+
+ if container.extra.running:
+ raise
+
+ return {
+ "stdout": "",
+ "stderr": "",
+ "error": {},
+ "code": -1,
+ }
+
+ client.run_forever(timeout=5)
+
+ stdout = client.read_stdout()
+ stderr = client.read_stderr()
+ error = yaml.safe_load(
+ client.read_channel(kubernetes.stream.ws_client.ERROR_CHANNEL)
+ )
+
+ # TODO: Is there really no better way, to check
+ # execution exit code in python k8s API client?
+ code = (
+ 0
+ if error["status"] == "Success"
+ else -2
+ if error["reason"] != "NonZeroExitCode"
+ else int(error["details"]["causes"][0]["message"])
+ )
+
+ return {
+ "stdout": stdout,
+ "stderr": stderr,
+ "error": error,
+ "code": code,
+ }
+
+
+def generate_python_binaries() -> List[str]:
+ """Function to generate list of names and paths for CPython binaries.
+
+ Returns:
+ List of names and paths, to CPython binaries.
+ """
+
+ dirnames = ["", "/usr/bin/", "/usr/local/bin/"]
+
+ majors_minors = [
+ f"{major}.{minor}" for major, minor in itertools.product("23", string.digits)
+ ]
+
+ suffixes = ["", "2", "3"] + majors_minors
+
+ basenames = [f"python{suffix}" for suffix in suffixes]
+
+ binaries = [f"{dir}{base}" for dir, base in itertools.product(dirnames, basenames)]
+
+ return binaries
+
+
+def generate_java_binaries() -> List[str]:
+ """Function to generate list of names and paths for OpenJDK binaries.
+
+ Returns:
+ List of names and paths, to OpenJDK binaries.
+ """
+
+ binaries = [
+ "java",
+ "/usr/bin/java",
+ "/usr/local/bin/java",
+ "/etc/alternatives/java",
+ "/usr/java/openjdk-14/bin/java",
+ ]
+
+ return binaries
+
+
+def determine_versions_abstraction(
+ api: kubernetes.client.api.core_v1_api.CoreV1Api,
+ container: ContainerInfo,
+ binaries: List[str],
+ extractor: Pattern,
+) -> List[str]:
+ """Function to determine list of software versions, that are installed in
+ given container.
+
+ Args:
+ api: Client of the k8s cluster API.
+ container: Object, that represents container in k8s cluster.
+ binaries: List of names and paths to the abstract software binaries.
+ extractor: Pattern to extract the version string from the output of the binary execution.
+
+ Returns:
+ List of installed software versions.
+ """
+
+ commands = ([binary, "--version"] for binary in binaries)
+
+ # TODO: This list comprehension should be parallelized
+ results = (
+ sync_post_namespaced_pod_exec(api, container, command) for command in commands
+ )
+
+ successes = (
+ f"{result['stdout']}{result['stderr']}"
+ for result in results
+ if result["code"] == 0
+ )
+
+ extractions = (extractor.search(success) for success in successes)
+
+ versions = sorted(
+ set(extraction.group(1) for extraction in extractions if extraction)
+ )
+
+ return versions
+
+
+def determine_versions_of_python(
+ api: kubernetes.client.api.core_v1_api.CoreV1Api, container: ContainerInfo
+) -> List[str]:
+ """Function to determine list of CPython versions,
+ that are installed in given container.
+
+ Args:
+ api: Client of the k8s cluster API.
+ container: Object, that represents container in k8s cluster.
+
+ Returns:
+ List of installed CPython versions.
+ """
+
+ extractor = re.compile("Python ([0-9.]+)")
+
+ binaries = generate_python_binaries()
+
+ versions = determine_versions_abstraction(api, container, binaries, extractor)
+
+ return versions
+
+
+def determine_versions_of_java(
+ api: kubernetes.client.api.core_v1_api.CoreV1Api, container: ContainerInfo
+) -> List[str]:
+ """Function to determine list of OpenJDK versions,
+ that are installed in given container.
+
+ Args:
+ api: Client of the k8s cluster API.
+ container: Object, that represents container in k8s cluster.
+
+ Returns:
+ List of installed OpenJDK versions.
+ """
+
+ extractor = re.compile("openjdk ([0-9.]+)")
+
+ binaries = generate_java_binaries()
+
+ versions = determine_versions_abstraction(api, container, binaries, extractor)
+
+ return versions
+
+
+def gather_containers_informations(
+ api: kubernetes.client.api.core_v1_api.CoreV1Api,
+ field_selector: str,
+ ignore_empty: bool,
+) -> List[ContainerInfo]:
+ """Get list of all containers names.
+
+ Args:
+ api: Client of the k8s cluster API.
+ field_selector: Kubernetes field selector, to filter out containers objects.
+ ignore_empty: Determines, if containers with empty versions should be ignored.
+
+ Returns:
+ List of initialized objects for containers in k8s cluster.
+ """
+
+ containers = list(list_all_containers(api, field_selector))
+
+ # TODO: This loop should be parallelized
+ for container in containers:
+ python_versions = determine_versions_of_python(api, container)
+ java_versions = determine_versions_of_java(api, container)
+ container.versions = ContainerVersions(python_versions, java_versions)
+
+ if ignore_empty:
+ containers = [c for c in containers if c.versions.python or c.versions.java]
+
+ return containers
+
+
+def generate_output_tabulate(containers: Iterable[ContainerInfo]) -> str:
+ """Function for generate output string in tabulate format.
+
+ Args:
+ containers: List of items, that represents containers in k8s cluster.
+
+ Returns:
+ Output string formatted by tabulate module.
+ """
+
+ headers = [
+ "Namespace",
+ "Pod",
+ "Container",
+ "Running",
+ "CPython",
+ "OpenJDK",
+ ]
+
+ rows = [
+ [
+ container.namespace,
+ container.pod,
+ container.container,
+ container.extra.running,
+ " ".join(container.versions.python),
+ " ".join(container.versions.java),
+ ]
+ for container in containers
+ ]
+
+ output = tabulate.tabulate(rows, headers=headers)
+
+ return output
+
+
+def generate_output_pprint(containers: Iterable[ContainerInfo]) -> str:
+ """Function for generate output string in pprint format.
+
+ Args:
+ containers: List of items, that represents containers in k8s cluster.
+
+ Returns:
+ Output string formatted by pprint module.
+ """
+
+ output = pprint.pformat(containers)
+
+ return output
+
+
+def generate_output_json(containers: Iterable[ContainerInfo]) -> str:
+ """Function for generate output string in JSON format.
+
+ Args:
+ containers: List of items, that represents containers in k8s cluster.
+
+ Returns:
+ Output string formatted by json module.
+ """
+
+ data = [
+ {
+ "namespace": container.namespace,
+ "pod": container.pod,
+ "container": container.container,
+ "extra": {
+ "running": container.extra.running,
+ "image": container.extra.image,
+ "identifier": container.extra.identifier,
+ },
+ "versions": {
+ "python": container.versions.python,
+ "java": container.versions.java,
+ },
+ }
+ for container in containers
+ ]
+
+ output = json.dumps(data, indent=4)
+
+ return output
+
+
+def generate_and_handle_output(
+ containers: List[ContainerInfo],
+ output_format: str,
+ output_file: pathlib.Path,
+ quiet: bool,
+) -> None:
+ """Generate and handle the output of the containers software versions.
+
+ Args:
+ containers: List of items, that represents containers in k8s cluster.
+ output_format: String that will determine output format (tabulate, pprint, json).
+ output_file: Path to file, where output will be save.
+ quiet: Determines if output should be printed, to stdout.
+ """
+
+ output_generators = {
+ "tabulate": generate_output_tabulate,
+ "pprint": generate_output_pprint,
+ "json": generate_output_json,
+ }
+
+ output = output_generators[output_format](containers)
+
+ if output_file:
+ output_file.write_text(output)
+
+ if not quiet:
+ print(output)
+
+
+def verify_versions_acceptability(
+ containers: List[ContainerInfo], acceptable: pathlib.Path, quiet: bool
+) -> bool:
+ """Function for verification of software versions installed in containers.
+
+ Args:
+ containers: List of items, that represents containers in k8s cluster.
+ acceptable: Path to the YAML file, with the software verification parameters.
+ quiet: Determines if output should be printed, to stdout.
+
+ Returns:
+ 0 if the verification was successful or 1 otherwise.
+ """
+
+ if not acceptable:
+ return 0
+
+ if not acceptable.is_file():
+ raise FileNotFoundError(
+ "File with configuration for acceptable does not exists!"
+ )
+
+ schema = {
+ "python": {"type": "list", "schema": {"type": "string"}},
+ "java": {"type": "list", "schema": {"type": "string"}},
+ }
+
+ validator = cerberus.Validator(schema)
+
+ with open(acceptable) as stream:
+ data = yaml.safe_load(stream)
+
+ if not validator.validate(data):
+ raise cerberus.SchemaError(
+ "Schema of file with configuration for acceptable is not valid."
+ )
+
+ python_acceptable = data.get("python", [])
+ java_acceptable = data.get("java", [])
+
+ python_not_acceptable = [
+ (container, "python", version)
+ for container in containers
+ for version in container.versions.python
+ if version not in python_acceptable
+ ]
+
+ java_not_acceptable = [
+ (container, "java", version)
+ for container in containers
+ for version in container.versions.java
+ if version not in java_acceptable
+ ]
+
+ if not python_not_acceptable and not java_not_acceptable:
+ return 0
+
+ if quiet:
+ return 1
+
+ print("List of not acceptable versions")
+ pprint.pprint(python_not_acceptable)
+ pprint.pprint(java_not_acceptable)
+
+ return 1
+
+
+def main(argv: Optional[List[str]] = None) -> str:
+ """Main entrypoint of the module for verifying versions of CPython and
+ OpenJDK installed in k8s cluster containers.
+
+ Args:
+ argv: List of command line arguments.
+ """
+
+ args = parse_argv(argv)
+
+ kubernetes.config.load_kube_config(args.config_file)
+
+ api = kubernetes.client.CoreV1Api()
+ api.api_client.configuration.debug = args.debug
+
+ containers = gather_containers_informations(
+ api, args.field_selector, args.ignore_empty
+ )
+
+ generate_and_handle_output(
+ containers, args.output_format, args.output_file, args.quiet
+ )
+
+ code = verify_versions_acceptability(containers, args.acceptable, args.quiet)
+
+ return code
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/test/security/check_versions/tests/conftest.py b/test/security/check_versions/tests/conftest.py
new file mode 100644
index 000000000..7c3e2e171
--- /dev/null
+++ b/test/security/check_versions/tests/conftest.py
@@ -0,0 +1,12 @@
+#!/usr/bin/env python3
+
+import pytest
+
+
+def pod_name_trimmer_fun(pod_name):
+ return "-".join(pod_name.split("-")[:-2])
+
+
+@pytest.fixture
+def pod_name_trimmer():
+ return pod_name_trimmer_fun
diff --git a/test/security/check_versions/tests/test_gather_containers_informations.py b/test/security/check_versions/tests/test_gather_containers_informations.py
new file mode 100644
index 000000000..63401721e
--- /dev/null
+++ b/test/security/check_versions/tests/test_gather_containers_informations.py
@@ -0,0 +1,38 @@
+#!/usr/bin/env python3
+
+import k8s_bin_versions_inspector as kbvi
+import kubernetes
+
+
+def test_gather_containers_informations(pod_name_trimmer):
+ kubernetes.config.load_kube_config()
+ api = kubernetes.client.CoreV1Api()
+ containers = kbvi.gather_containers_informations(api, "", False)
+ data = [
+ (
+ c.namespace,
+ pod_name_trimmer(c.pod),
+ c.container,
+ c.versions.python,
+ c.versions.java,
+ )
+ for c in containers
+ ]
+ sorted_data = sorted(data)
+ assert sorted_data == [
+ ("default", "kbvi-test-java-keycloak", "keycloak", [], ["11.0.8"]),
+ ("default", "kbvi-test-java-keycloak-old", "keycloak-old", [], ["11.0.5"]),
+ (
+ "default",
+ "kbvi-test-java-keycloak-very-old",
+ "keycloak-very-old",
+ ["2.7.5"],
+ [],
+ ), # TODO
+ ("default", "kbvi-test-python-jupyter", "jupyter", ["3.8.4"], []),
+ ("default", "kbvi-test-python-jupyter-old", "jupyter-old", ["3.6.6"], []),
+ ("default", "kbvi-test-python-stderr-filebeat", "filebeat", ["2.7.5"], []),
+ ("default", "kbvi-test-terminated", "python", [], []), # TODO
+ ("ingress-nginx", "kbvi-test-ingress-nginx", "echo-server", [], []),
+ ("kube-system", "kbvi-test-kube-system", "echo-server", [], []),
+ ]
diff --git a/test/security/check_versions/tests/test_list_all_containers.py b/test/security/check_versions/tests/test_list_all_containers.py
new file mode 100644
index 000000000..4178077c3
--- /dev/null
+++ b/test/security/check_versions/tests/test_list_all_containers.py
@@ -0,0 +1,52 @@
+#!/usr/bin/env python3
+
+import k8s_bin_versions_inspector as kbvi
+import kubernetes
+
+
+def exec_list_all_containers(pod_name_trimmer, field_selector):
+ kubernetes.config.load_kube_config()
+ api = kubernetes.client.CoreV1Api()
+ containers = kbvi.list_all_containers(api, field_selector)
+ extracted = ((c.namespace, c.pod, c.container) for c in containers)
+ trimmed = ((n, pod_name_trimmer(p), c) for n, p, c in extracted)
+ result = sorted(trimmed)
+ return result
+
+
+def test_list_all_containers(pod_name_trimmer):
+ result = exec_list_all_containers(pod_name_trimmer, "")
+ assert result == [
+ ("default", "kbvi-test-java-keycloak", "keycloak"),
+ ("default", "kbvi-test-java-keycloak-old", "keycloak-old"),
+ ("default", "kbvi-test-java-keycloak-very-old", "keycloak-very-old"),
+ ("default", "kbvi-test-python-jupyter", "jupyter"),
+ ("default", "kbvi-test-python-jupyter-old", "jupyter-old"),
+ ("default", "kbvi-test-python-stderr-filebeat", "filebeat"),
+ ("default", "kbvi-test-terminated", "python"),
+ ("ingress-nginx", "kbvi-test-ingress-nginx", "echo-server"),
+ ("kube-system", "kbvi-test-kube-system", "echo-server"),
+ ]
+
+
+def test_list_all_containers_not_default(pod_name_trimmer):
+ field_selector = "metadata.namespace!=default"
+ result = exec_list_all_containers(pod_name_trimmer, field_selector)
+ assert result == [
+ ("ingress-nginx", "kbvi-test-ingress-nginx", "echo-server"),
+ ("kube-system", "kbvi-test-kube-system", "echo-server"),
+ ]
+
+
+def test_list_all_containers_conjunction(pod_name_trimmer):
+ field_selector = "metadata.namespace!=kube-system,metadata.namespace!=ingress-nginx"
+ result = exec_list_all_containers(pod_name_trimmer, field_selector)
+ assert result == [
+ ("default", "kbvi-test-java-keycloak", "keycloak"),
+ ("default", "kbvi-test-java-keycloak-old", "keycloak-old"),
+ ("default", "kbvi-test-java-keycloak-very-old", "keycloak-very-old"),
+ ("default", "kbvi-test-python-jupyter", "jupyter"),
+ ("default", "kbvi-test-python-jupyter-old", "jupyter-old"),
+ ("default", "kbvi-test-python-stderr-filebeat", "filebeat"),
+ ("default", "kbvi-test-terminated", "python"),
+ ]
diff --git a/test/security/check_versions/tests/test_main.py b/test/security/check_versions/tests/test_main.py
new file mode 100644
index 000000000..0dff0b230
--- /dev/null
+++ b/test/security/check_versions/tests/test_main.py
@@ -0,0 +1,84 @@
+#!/usr/bin/env python3
+
+import k8s_bin_versions_inspector as kbvi
+import json
+import tempfile
+import yaml
+
+
+def exec_main(pod_name_trimmer, acceptable_data):
+
+ with tempfile.NamedTemporaryFile() as output_temp, tempfile.NamedTemporaryFile() as acceptable_temp:
+
+ with open(acceptable_temp.name, "w") as stream:
+ yaml.safe_dump(acceptable_data, stream)
+
+ result = kbvi.main(
+ [
+ "--quiet",
+ "--output-file",
+ output_temp.name,
+ "--output-format",
+ "json",
+ "--acceptable",
+ acceptable_temp.name,
+ ]
+ )
+
+ with open(output_temp.name, "r") as stream:
+ output_data = json.load(stream)
+ output_extracted = (
+ (
+ item["namespace"],
+ pod_name_trimmer(item["pod"]),
+ item["container"],
+ item["versions"]["python"],
+ item["versions"]["java"],
+ )
+ for item in output_data
+ )
+ output_sorted = sorted(output_extracted)
+
+ assert output_sorted == [
+ ("default", "kbvi-test-java-keycloak", "keycloak", [], ["11.0.8"]),
+ ("default", "kbvi-test-java-keycloak-old", "keycloak-old", [], ["11.0.5"]),
+ (
+ "default",
+ "kbvi-test-java-keycloak-very-old",
+ "keycloak-very-old",
+ ["2.7.5"],
+ [],
+ ),
+ ("default", "kbvi-test-python-jupyter", "jupyter", ["3.8.4"], []),
+ ("default", "kbvi-test-python-jupyter-old", "jupyter-old", ["3.6.6"], []),
+ ("default", "kbvi-test-python-stderr-filebeat", "filebeat", ["2.7.5"], []),
+ ("default", "kbvi-test-terminated", "python", [], []),
+ ("ingress-nginx", "kbvi-test-ingress-nginx", "echo-server", [], []),
+ ("kube-system", "kbvi-test-kube-system", "echo-server", [], []),
+ ]
+
+ return result
+
+
+def test_main(pod_name_trimmer):
+
+ acceptable_data = {
+ "python": ["2.7.5", "3.6.6", "3.8.4"],
+ "java": ["11.0.5", "11.0.8"],
+ }
+
+ result = exec_main(pod_name_trimmer, acceptable_data)
+
+ assert result == 0
+
+
+def test_main_neg(pod_name_trimmer):
+
+ acceptable_data = {
+ "python": ["3.6.6", "3.8.4"],
+ "java": ["11.0.5", "11.0.8"],
+ }
+
+ result = exec_main(pod_name_trimmer, acceptable_data)
+
+ assert result == 1
diff --git a/test/security/check_versions/tests/test_sync_post_namespaced_pod_exec.py b/test/security/check_versions/tests/test_sync_post_namespaced_pod_exec.py
new file mode 100644
index 000000000..50620d3a7
--- /dev/null
+++ b/test/security/check_versions/tests/test_sync_post_namespaced_pod_exec.py
@@ -0,0 +1,74 @@
+#!/usr/bin/env python3
+
+import k8s_bin_versions_inspector as kbvi
+import kubernetes
+
+
+def exec_sync_post_namespaced_pod_exec(pod, command):
+ kubernetes.config.load_kube_config()
+ api = kubernetes.client.CoreV1Api()
+ containers = kbvi.list_all_containers(api, "")
+ container = next(c for c in containers if c.pod.startswith(pod))
+ result = kbvi.sync_post_namespaced_pod_exec(api, container, command)
+ return result
+
+
+def test_sync_post_namespaced_pod_exec():
+ pod = "kbvi-test-python-jupyter"
+ result = exec_sync_post_namespaced_pod_exec(pod, "id")
+ assert result == {
+ "stdout": "uid=1000(jovyan) gid=100(users) groups=100(users)\n",
+ "stderr": "",
+ "error": {"status": "Success", "metadata": {}},
+ "code": 0,
+ }
+
+
+def test_sync_post_namespaced_pod_exec_not_running():
+ pod = "kbvi-test-terminated"
+ result = exec_sync_post_namespaced_pod_exec(pod, "id")
+ assert result == {"stdout": "", "stderr": "", "error": {}, "code": -1}
+
+
+def test_sync_post_namespaced_pod_exec_not_found():
+ pod = "kbvi-test-python-jupyter"
+ command = "/command/not/found"
+ result = exec_sync_post_namespaced_pod_exec(pod, command)
+ assert result["stdout"] == ""
+ assert result["stderr"] == ""
+ assert result["error"]["status"] == "Failure"
+ assert result["error"]["reason"] == "InternalError"
+ assert result["code"] == -2
+
+
+def test_sync_post_namespaced_pod_exec_exit_code():
+ pod = "kbvi-test-python-jupyter"
+ command = ["python3", "--invalid-attribute"]
+ result = exec_sync_post_namespaced_pod_exec(pod, command)
+ assert result == {
+ "stdout": "",
+ "stderr": "unknown option --invalid-attribute\n"
+ "usage: python3 [option] ... [-c cmd | -m mod | file | -] [arg] ...\n"
+ "Try `python -h' for more information.\n",
+ "error": {
+ "status": "Failure",
+ "reason": "NonZeroExitCode",
+ "message": "command terminated with non-zero exit code: error "
+ "executing command [python3 --invalid-attribute], exit code 2",
+ "details": {"causes": [{"message": "2", "reason": "ExitCode"}]},
+ "metadata": {},
+ },
+ "code": 2,
+ }
+
+
+def test_sync_post_namespaced_pod_exec_stderr():
+ pod = "kbvi-test-python-stderr-filebeat"
+ command = ["python", "--version"]
+ result = exec_sync_post_namespaced_pod_exec(pod, command)
+ assert result == {
+ "stdout": "",
+ "stderr": "Python 2.7.5\n",
+ "error": {"status": "Success", "metadata": {}},
+ "code": 0,
+ }
diff --git a/test/security/check_versions/tests/test_verify_versions_acceptability.py b/test/security/check_versions/tests/test_verify_versions_acceptability.py
new file mode 100644
index 000000000..5e2f0d2c8
--- /dev/null
+++ b/test/security/check_versions/tests/test_verify_versions_acceptability.py
@@ -0,0 +1,58 @@
+#!/usr/bin/env python3
+
+import k8s_bin_versions_inspector as kbvi
+import yaml
+import tempfile
+import pathlib
+
+
+def exec_verify_versions_acceptability(containers):
+
+ config = {
+ "python": ["1.1.1", "2.2.2"],
+ "java": ["3.3.3"],
+ }
+
+ with tempfile.NamedTemporaryFile() as temp:
+ with open(temp.name, "w") as stream:
+ yaml.safe_dump(config, stream)
+ acceptable = pathlib.Path(temp.name)
+ result = kbvi.verify_versions_acceptability(containers, acceptable, True)
+
+ return result
+
+
+def test_verify_versions_acceptability():
+
+ containers = [
+ kbvi.ContainerInfo("a", "b", "c", None, kbvi.ContainerVersions([], [])),
+ kbvi.ContainerInfo(
+ "a", "b", "c", None, kbvi.ContainerVersions(["1.1.1"], ["3.3.3"])
+ ),
+ ]
+
+ result = exec_verify_versions_acceptability(containers)
+
+ assert result == 0
+
+
+def test_verify_versions_acceptability_neg_1():
+
+ containers = [
+ kbvi.ContainerInfo("a", "b", "c", None, kbvi.ContainerVersions(["3.3.3"], []))
+ ]
+
+ result = exec_verify_versions_acceptability(containers)
+
+ assert result == 1
+
+
+def test_verify_versions_acceptability_neg_2():
+
+ containers = [
+ kbvi.ContainerInfo("a", "b", "c", None, kbvi.ContainerVersions([], ["1.1.1"]))
+ ]
+
+ result = exec_verify_versions_acceptability(containers)
+
+ assert result == 1
diff --git a/test/security/check_versions/tox.ini b/test/security/check_versions/tox.ini
new file mode 100644
index 000000000..78510e7a3
--- /dev/null
+++ b/test/security/check_versions/tox.ini
@@ -0,0 +1,17 @@
+[tox]
+envlist = black, pylint
+skipsdist = true
+
+[testenv]
+basepython = python3
+deps = -r{toxinidir}/env/requirements-dev.txt
+
+[testenv:black]
+commands = black {toxinidir}/src tests
+
+[testenv:pylint]
+commands = pylint -d C0330,W0511 {toxinidir}/src
+
+[testenv:pytest]
+setenv = PYTHONPATH = {toxinidir}/src
+commands = pytest -vv -s tests