[GITLAB] Initial content for gitlab project chained-ci

Issue-ID: INT-2150

Signed-off-by: Andreas Geissler <andreas-geissler@telekom.de>
Change-Id: Id5dc270b6a7c25189b4e09c4227b806323711d72

18 files changed, 534 insertions, 0 deletions

diff --git a/doc/chained-ci-install.md b/doc/chained-ci-install.md
new file mode 100644
index 0000000..d27939d
--- /dev/null
+++ b/doc/chained-ci-install.md
@@ -0,0 +1,136 @@
+# Creation of an own chained-ci project
+
+In our environment we want to use Chained-CI to automate the ONAP Lab
+installation and testing.
+
+On an own development client host:
+* install ansible
+
+```
+   sudo apt-get install ansible
+```
+
+* Create a Git project (LabInstallation/chained-ci) in git.sdp.telekom.de
+and checked it out.
+
+```
+   git clone git@gitlab.devops.telekom.de:tnap/onapcommunity/integrationproject/onapdeployment/chained-ci.git
+```
+* In my DevEnv I cloned the chained-ci example project:
+
+~~~~
+   git clone https://gitlab.com/Orange-OpenSource/lfn/ci_cd/chained-ci-examples.git
+~~~~
+
+* Copy the example code to chained-ci
+
+~~~~
+   cp -r chained-ci-examples/* chained-ci
+~~~~
+
+* Add submodules in chained-ci and checkin all
+
+~~~~
+   cd chained-ci
+   git submodule add -b alpine-ansible https://gitlab.com/Orange-OpenSource/lfn/ci_cd/chained-ci-roles.git roles
+   git submodule add https://gitlab.com/Orange-OpenSource/lfn/ci_cd/chained-ci-vue.git chained-ci-vue
+
+   git add *
+   ...
+   git commit -a
+   git push
+~~~~
+
+* Extract submodules
+
+~~~~
+   git submodule init
+   git submodule update
+~~~~
+
+* Setup own chain
+
+    * Create entry in ```pod_inventory/infrastructure``` (e.g. pod-test)
+    * Add ```pod_inventory/hostvars/pod-test.yml``` file for the chain-steps and jumphost
+    * Modify the ```pod_inventory/group_vars/all.yml```
+
+* Modifications in ```pod_inventory/group_vars/all.yml```
+    * add image and image_tag in runner section
+    * remove not needed git_projects
+    * add gitlab settings
+    * create project settings
+    * create a trigger token on the target project (in CI/CD settings) and copy token
+    * generate encrypted trigger_token password
+
+         ```
+         echo '...password...' > .vault
+         ```
+
+         ```
+         ansible-vault encrypt_string --vault-password-file .vault '...token...' --name 'trigger-token'
+         ```
+
+    * add encrypted trigger_token to all.yml
+
+
+Example:
+~~~~
+runner:
+  tags:
+    - gitlab-org
+  env_vars:
+    CHAINED_CI_SRC: https://gitlab.devops.telekom.de/tnap/onapcommunity/integrationproject/onapdeployment/chained-ci.git
+  docker_proxy:
+  image: registry.gitlab.com/orange-opensource/lfn/ci_cd/docker_ansible
+  image_tag: latest
+
+gitlab:
+  pipeline:
+    delay: 15
+  base_url: https://gitlab.devops.telekom.de
+  api_url: https://gitlab.devops.telekom.de/api/v4
+  private_token: "{{ lookup('env','CI_private_token') }}"
+
+  git_projects:
+    config:
+      stage: config
+      url: https://gitlab.devops.telekom.de/tnap/onapcommunity/integrationproject/onapdeployment/chained-ci.git
+      branch: "{{ lookup('env','config_branch')|default('master', true) }}"
+      path: pod_config
+    cloud-infra:
+      stage: infra_install
+      api: https://gitlab.devops.telekom.de/api/v4/project/32660
+      url: https://gitlab.devops.telekom.de/tnap/onapcommunity/integrationproject/onapdeployment/cloud-infra
+      trigger_token: !vault |
+        $ANSIBLE_VAULT;1.1;AES256
+        66386364383232303832656238636130373430633539613566646337313164323733333138666163
+        3766623563303133393231663237353633353365633063350a653231393436313961613733643036
+        37393262363864393734323532383662663263663863646233366639633130323731343237653661
+        6265323237306634620a376166616564663135316363333136356135613336646665386532616263
+        64373537346235386438633130656363386633383337626337656234646361366263
+      branch: "{{ lookup('env','cloud-infra_branch')|default('master', true) }}"
+      get_artifacts: config
+      pull_artifacts: 
+      timeout: 300
+      parameters:
+        ansible_verbose: "{{ lookup('env','ansible_verbose') }}" 
+~~~~
+
+* Create .gitlab-ci.yml
+
+~~~~
+   ansible-playbook -vvv --vault-password-file ~/git/LabInstallation/chained-ci/.vault -i ./pod_inventory/inventory ./gitlab-ci-generator.yml
+~~~~
+
+* Checkin all modified files
+
+* Add in chained-ci project CI/CD the following variables:
+
+    ```ANSIBLE_VAULT_PASSWORD``` -> a secret for the Vault password
+    
+    ```CI_private_token``` -> API token to access Gitlab (see User-Settings-Access Tokens)
+
+How to create own project pipelines:
+
+* [DTs Chained-ci pipelines](chained-ci-pipeline.md)
+
diff --git a/doc/chained-ci-intro.md b/doc/chained-ci-intro.md
new file mode 100644
index 0000000..01d74fe
--- /dev/null
+++ b/doc/chained-ci-intro.md
@@ -0,0 +1,61 @@
+# Introduction to Chained-ci
+
+Chained-ci is a solution to chain CI/CD pipelines from different gilab projects.
+It leverages gitlab APIs and features like the artifacts.
+
+![Chained-ci overview](img/chained-CI.png)
+
+As an example if you want to install a solution on an infrastructure, you can
+chain the project pipeline in charge of infrastructure deployment then the
+pipeline in charge of the deployment of the target solution thanks to
+environment variables, configuration parameters provided as artifacts at the
+end of the first pipeline.
+
+Chained-ci offers additional possibilities such as using static artifacts,
+notifying third parties.
+
+## Why Chained-ci
+
+Historically involved in several Open Source projects, it was very hard to
+harmonize CI/CD strategies as each project had its own way to manage it and
+automation culture.
+OPNFV XCI initiative was created to provide a way to create links between CI/CD
+teams of different Open Source Networking communities.
+
+In parallel, internally we started having lots of projects on gitlab.
+The goal was to be able to chain the pipelines of these different gitlab
+projects. Each project remains responsible for its CI/CD strategy but may expose
+artifacts and can be consumed by other projects.
+
+Thanks to Chained-ci, it is possible to easily mix and match pipelines and unify
+CI/CD strategy leveraging gitlab capabilities.
+
+It is possible on the same infrastructure to automate the deployments and
+testing of several projects in parallel.
+
+## Chained-ci project
+
+Chained-ci is composed of several sub-projects:
+
+* [Chained-ci](https://gitlab.forge.orange-labs.fr/osons/chained-ci): the part
+  where the chains will be created and configured
+* [Chained-ci-roles](https://gitlab.com/Orange-OpenSource/lfn/ci_cd/chained-ci-roles):
+  the repository hosting the ansible roles
+* [Chained-ci-tools](https://gitlab.com/Orange-OpenSource/lfn/ci_cd/chained-ci-tools):
+  some tooling to simplify the creation of the chains
+* [Chained-ci-vue](https://gitlab.com/Orange-OpenSource/lfn/ci_cd/chained-ci-vue):
+  a web front end to easily visualise the different chains.
+
+![chained-ci-vue](img/chained-ci-ui.png)
+
+If you click on any element of the chain, you will see the corresponding gitlab
+pipeline. You can also restart the chain in case of failure at the step it
+failed.
+
+## Get started
+
+* [DTs Chained-ci installation](chained-ci-install.md)
+* [DTs Chained-ci pipelines](chained-ci-pipeline.md)
+* [Orange: Create your Chained-ci](./user-guide/configure.md)
+* [Orange: Run your Chained-ci](./user-guide/run.md)
+* [Orange: Examples](./example/example.md)
diff --git a/doc/chained-ci-pipeline.md b/doc/chained-ci-pipeline.md
new file mode 100644
index 0000000..8af10bf
--- /dev/null
+++ b/doc/chained-ci-pipeline.md
@@ -0,0 +1,27 @@
+# Creation of an own Project Chain
+
+Inside of the Chained-CI project installation chains can be created
+to automate the the Openstack tenant, network and VM creation, as
+well as the installation of applications (e.g. ONAP)
+
+Currently the following projects are available supporting chaining:
+
+* cloud-infra -> Creation of Openstack tenant, network and VMs
+* ...
+
+In order to define the pipelines the following steps need to be followed:
+
+* Projects need to be defined in all.yml (see [Chained-CI install](https://git.sdp.telekom.de/LabInstallation/chained-ci/blob/master/doc/chained-ci-install.md "Chained-CI installation"))
+* in the directory chained-ci/pod_config the inventory and pod definitions
+  have to be created
+* in the directory chained-ci/pod_inventory the pipeline definitions
+  have to be created
+
+The execution of the pipeline requires a Jumphost within the target cloud.
+Requirements for the Jumphost:
+* OS: e.g. Ubuntu 18.04
+* Packages: ansible, python-pip, python3-pip, jq
+* Network must be able to reach and resolve the VIM Keystone interface
+* To access the JH a RSA keypair needs to be created without passphrase
+* The public key has to be added to the .ssh/authorized_keys of the 
+  ansible user (e.g. ubuntu)
diff --git a/doc/img/chained-CI.png b/doc/img/chained-CI.png
new file mode 100644
index 0000000..ec096b8
--- /dev/null
+++ b/doc/img/chained-CI.png
diff --git a/doc/img/chained-ci-examples.png b/doc/img/chained-ci-examples.png
new file mode 100644
index 0000000..c771cae
--- /dev/null
+++ b/doc/img/chained-ci-examples.png
diff --git a/doc/img/chained-ci-ui.png b/doc/img/chained-ci-ui.png
new file mode 100644
index 0000000..245f806
--- /dev/null
+++ b/doc/img/chained-ci-ui.png
diff --git a/doc/img/example1.png b/doc/img/example1.png
new file mode 100644
index 0000000..e2ee28e
--- /dev/null
+++ b/doc/img/example1.png
diff --git a/doc/img/example2.png b/doc/img/example2.png
new file mode 100644
index 0000000..2f2fc94
--- /dev/null
+++ b/doc/img/example2.png
diff --git a/doc/img/example3.png b/doc/img/example3.png
new file mode 100644
index 0000000..1eeaddc
--- /dev/null
+++ b/doc/img/example3.png
diff --git a/doc/img/inpod.png b/doc/img/inpod.png
new file mode 100644
index 0000000..dc95989
--- /dev/null
+++ b/doc/img/inpod.png
diff --git a/doc/img/pipeline_creation_1.jpg b/doc/img/pipeline_creation_1.jpg
new file mode 100644
index 0000000..05b5178
--- /dev/null
+++ b/doc/img/pipeline_creation_1.jpg
diff --git a/doc/img/pipeline_creation_2.jpg b/doc/img/pipeline_creation_2.jpg
new file mode 100644
index 0000000..d922476
--- /dev/null
+++ b/doc/img/pipeline_creation_2.jpg
diff --git a/doc/img/pipeline_creation_3.jpg b/doc/img/pipeline_creation_3.jpg
new file mode 100644
index 0000000..1e7a915
--- /dev/null
+++ b/doc/img/pipeline_creation_3.jpg
diff --git a/doc/img/schedule_creation_1.jpg b/doc/img/schedule_creation_1.jpg
new file mode 100644
index 0000000..48987c5
--- /dev/null
+++ b/doc/img/schedule_creation_1.jpg
diff --git a/doc/img/schedule_creation_2.jpg b/doc/img/schedule_creation_2.jpg
new file mode 100644
index 0000000..f447f20
--- /dev/null
+++ b/doc/img/schedule_creation_2.jpg
diff --git a/doc/img/schedule_creation_3.jpg b/doc/img/schedule_creation_3.jpg
new file mode 100644
index 0000000..5ff84ed
--- /dev/null
+++ b/doc/img/schedule_creation_3.jpg
diff --git a/doc/user-guide/configure.md b/doc/user-guide/configure.md
new file mode 100644
index 0000000..4d9e30e
--- /dev/null
+++ b/doc/user-guide/configure.md
@@ -0,0 +1,280 @@
+# Create your chain
+
+You must clone the chained-ci project.
+
+```Shell
+   git clone https://gitlab.forge.orange-labs.fr/osons/chained-ci
+   cd chained-ci
+   git submodule init
+   git submodule update
+```
+
+The chained-ci repository can be shown as follows:
+
+```Ascii
+.
+├── pod_inventory
+│   ├── group_vars
+│   │   └── all.yml
+│   ├── host_vars
+│   │   ├── existing_chained.yml
+│   │   ├── ...
+│   │   └── vpod1.yml
+│   └── inventory
+└── roles
+    ├── artifact_init
+    │   ├── defaults
+    │   └── tasks
+    ├── get_artifacts
+    │   ├── defaults
+    │   └── tasks
+    ├── gitlab-ci-generator
+    │   ├── tasks
+    │   └── templates
+    ├── run-ci
+    │   └── tasks
+    └── trigger_myself
+        └── tasks
+```
+
+The ansible roles are retrieved through the git submodules.
+
+Note: It is recommended to create your branch to create your new chain.
+
+To create you chained-ci chain
+
+* Edit  the **pod_inventory/inventory** file to declare your chain
+* Create a file in **pod_inventory/host_vars** to configure your chain
+* Edit **pod_inventory/group_vars/all.yaml** to reference some variables if
+  needed
+
+## Declare your chain
+
+You must first declare your chain name in the pod_inventory/inventory file.
+Edit the file and add your chain name at the end of the file.
+The chain name shall be self-explicit.
+
+Even if the naming is free, it is recommended to follow the syntax:
+
+`[target][_[infra]]_[[ci_option]]_[pod]]`
+
+e.g.
+
+* `pod2`: deployment of the pod2 infrastructure (GNU/Linux OS)
+* `acumos_pod4`: deployment of acumos on POD4
+* `functest_pod4`: run functest tests on POD4
+* `k8s_pod1`: deployment of kubernetes on bare metal on pod1
+* ...
+* `onap_k8s_daily_pod4`: daily deployment of onap on k8s infra installed on pod4
+
+## Define your chain
+
+The second step consists in creating the ansible role for the new chain.
+
+Create a file in pod_inventory/host_vars.
+
+The file name shall correspond to the chain name you declared in the inventory
+file e.g. acumos_pod4.yaml.
+
+This file describes the chain. It reference the different projects you want to
+chain, indicating the expected artifacts to be re-used from one project to
+another.
+
+```YAML
+---
+jumphost:
+  server: <target POD jumphost IP>
+  user: <ansible user>
+[option section]
+scenario_steps:
+  step 1:
+  ...
+  step 2:
+  ...
+  step N:
+  ...
+```
+
+The chain will be run sequentially (e.g. config then step 1 until step N).
+Each pipeline can however parallelize some stages.
+
+The option section is not mandatory.
+
+### jumphost Definition
+
+A jumphost is associated to a chained-ci even if the different steps can be
+performed through projects hosted in different environements (public gitlab.com
+and internal gitlab). The creator of the chain shall ensure that the projects
+can be invoked from the referenced jumphost.
+
+### Chained-ci options
+
+The possible option fields are:
+
+* environment: this parameter is used and propagated as an environment variable
+  to all the projects involved in the chained-ci
+* longlife_artifact: `true` (not present by default)
+* inpod: `<other_chained_ci_reference>`
+* healthchecks_id: `<uid>`
+
+#### longlife_artifact
+
+The longlife_artifact option is needed to store the artifacts over a long
+period. As other chains may consume the artifacts, if the chained-ci is not run
+frequently, the artifacts could be deleted after a while and become unavailable
+for other chained-ci.
+
+Typically if you install an infrastructure once and run several chains
+requesting the artifacts of the infrastructure installation, it makes sense to
+set this option.
+
+#### inpod
+
+Chained-ci can be seen as matriochka, in fact you may trigger deployments once
+a previous deployment has been completed.
+
+The inpod parameter allows to define a sort of heritage in the chained-ci
+declaration.
+
+![inpod](../img/inpod.png)
+
+#### healthchecks
+
+The healthcheck option is used to reference your chained-ci to
+[healthcheck](https://github.com/healthchecks/healthchecks).
+
+### Chained-ci steps
+
+Each step can be defined as follow:
+
+```YAML
+[step name]:
+  project: [gitlab project reference]
+  get_artifacts:
+    - name: [project name for the artifact]
+      [static_src: true]
+  [extra parameters]
+  [infra]
+```
+
+#### gitlab project reference (mandatory)
+
+It shall correspond to a name of a gitlab project.
+
+#### get_artifacts (optional)
+
+The get_artifacts is to retrieved data (env variables, configuration files)
+from a previous gitlab pipeline from a chained-ci.
+
+These artifacts can be retrieved:
+
+* dynamically (by Default): gitlab API will be called to retrieved the last
+  artifacts corresponding to the project referenced
+* statically: if static_src parameter is set to True, you then must indicate
+  the path to the artifact
+
+By default Chained-ci will call gitlab APIs to retrieved the artifact.
+
+##### limit_to
+
+Please note that it is possible to retrieve only a subset of the files from
+an artifact by using **limit_to**.
+
+```YAML
+get_artifacts:
+      - name: infra_deploy
+        limit_to:
+          - vars/user_cloud.yml: vars/user_cloud.yml
+```
+
+##### in_pipeline=false
+
+It is also possible to indicate that the artifact cannot be retrieved from the
+pipeline by setting in_pipeline=false in the get_artifact section. You then
+have to add the name of the pipeline to fetch and the step in the pipeline.
+
+In the example, we retrieve artifacts from `config` step of last
+`onap_k8s_ic_pod4` pipeline.
+
+```YAML
+ get_artifacts:
+      - name: config:onap_k8s_ic_pod4
+        in_pipeline: false
+        limit_to:
+          - vars/pdf.yml: vars/pdf.yml
+          - vars/idf.yml: vars/idf.yml
+```
+
+#### extra parameters (optional)
+
+You may indicate here additional variables to configure your Chained-ci. These
+variables will be given as environment variables for the run of the project. It
+shall be configured as follows:
+
+```YAML
+extra_parameters:
+      key 1: value 1
+      key 2: value 2
+      ....
+      key N: value N
+```
+
+#### infra (optional)
+
+This option can be specified if you want to use a specific resources of an
+existing infra. This infra shall be declared in the Infrastructure Description
+file.
+
+## Create your static artifacts
+
+It is possible to create a static artifact file. It consists in a zip file
+containing all the files.
+
+For instance if you need to precise an inventory my_jumhost and a set of
+vaiables declared in a foo.yml file, you will need to prepare your file.
+
+```
+├── inventory
+│   └── my_jumphost
+└── vars
+    └── foo.yml
+```
+
+Zip the different files in my_foo.zip and reference zoo in the chained-ci.
+
+```
+get_artifacts:
+  - name: my_foo
+    static_src: true
+```
+
+Note that my_foo must not be in the path once the archive unzipped.
+
+## Gitlab-ci generator
+
+The `.gitlab-ci.yml` of your Chaine-ci project may be a little be tricky to
+generate. That is why a gitlab-ci generator has been introduced.
+
+You can run the gitlab-ci-generator ansible role locally on your machine.
+Based on the chained-ci you created, it will re-generate the file properly.
+
+```Shell
+ansible-playbook -i ./pod_inventory/inventory ./gitlab-ci-generator.yml
+```
+
+`<chain_id>` is the name of the chain you defined in *Declare your chain* step
+
+once you have generated `.gitlab-ci.yml`, you'll need to push your change to the
+gitlab server (via a merge request or not, depending on you push policy) before
+using it.
+
+## chained-ci tools
+
+It provides a set of tools to deal with Chained-ci:
+
+* Get artifacts from a projects
+* set the configuration (ssh key and ssh config)
+* Generate Vault key file
+
+See [README](https://gitlab.com/Orange-OpenSource/lfn/ci_cd/chained-ci-tools)
+for details.
diff --git a/doc/user-guide/run.md b/doc/user-guide/run.md
new file mode 100644
index 0000000..2b77296
--- /dev/null
+++ b/doc/user-guide/run.md
@@ -0,0 +1,30 @@
+# Run your Chained-ci
+
+From the gitlab web page of your Chained-ci project, select the CI/CD menu
+
+![pipeline creation step 1](../img/pipeline_creation_1.jpg)
+
+* Click on the Run Pipeline button
+
+  ![pipeline creation step 2](../img/pipeline_creation_2.jpg)
+
+* Select the acurate branch (Default: master) (orange in image)
+
+  ![pipeline creation step 3](../img/pipeline_creation_3.jpg)
+
+* set variable POD, here POD is the identifier for your chained-CI (yellow in image)
+
+  ![pipeline creation step 3](../img/pipeline_creation_3.jpg)
+
+* Click on Create pipeline i(green in image)
+
+  ![pipeline creation step 3](../img/pipeline_creation_3.jpg)
+
+You can obviously also use the Schedules menu if you want to regularly run
+your chain (cronjob defined).
+
+![schedule creation step 1](../img/schedule_creation_1.jpg)
+
+![schedule creation step 1](../img/schedule_creation_2.jpg)
+
+![schedule creation step 1](../img/schedule_creation_3.jpg)
\ No newline at end of file