diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/chained-ci-install.md | 136 | ||||
| -rw-r--r-- | doc/chained-ci-intro.md | 61 | ||||
| -rw-r--r-- | doc/chained-ci-pipeline.md | 27 | ||||
| -rw-r--r-- | doc/img/chained-CI.png | bin | 0 -> 49416 bytes | |||
| -rw-r--r-- | doc/img/chained-ci-examples.png | bin | 0 -> 75872 bytes | |||
| -rw-r--r-- | doc/img/chained-ci-ui.png | bin | 0 -> 79189 bytes | |||
| -rw-r--r-- | doc/img/example1.png | bin | 0 -> 57539 bytes | |||
| -rw-r--r-- | doc/img/example2.png | bin | 0 -> 60968 bytes | |||
| -rw-r--r-- | doc/img/example3.png | bin | 0 -> 136786 bytes | |||
| -rw-r--r-- | doc/img/inpod.png | bin | 0 -> 35472 bytes | |||
| -rw-r--r-- | doc/img/pipeline_creation_1.jpg | bin | 0 -> 136246 bytes | |||
| -rw-r--r-- | doc/img/pipeline_creation_2.jpg | bin | 0 -> 159171 bytes | |||
| -rw-r--r-- | doc/img/pipeline_creation_3.jpg | bin | 0 -> 141773 bytes | |||
| -rw-r--r-- | doc/img/schedule_creation_1.jpg | bin | 0 -> 75070 bytes | |||
| -rw-r--r-- | doc/img/schedule_creation_2.jpg | bin | 0 -> 176929 bytes | |||
| -rw-r--r-- | doc/img/schedule_creation_3.jpg | bin | 0 -> 179055 bytes | |||
| -rw-r--r-- | doc/user-guide/configure.md | 280 | ||||
| -rw-r--r-- | doc/user-guide/run.md | 30 |
18 files changed, 534 insertions, 0 deletions
diff --git a/doc/chained-ci-install.md b/doc/chained-ci-install.md new file mode 100644 index 0000000..d27939d --- /dev/null +++ b/doc/chained-ci-install.md @@ -0,0 +1,136 @@ +# Creation of an own chained-ci project + +In our environment we want to use Chained-CI to automate the ONAP Lab +installation and testing. + +On an own development client host: +* install ansible + +``` + sudo apt-get install ansible +``` + +* Create a Git project (LabInstallation/chained-ci) in git.sdp.telekom.de +and checked it out. + +``` + git clone git@gitlab.devops.telekom.de:tnap/onapcommunity/integrationproject/onapdeployment/chained-ci.git +``` +* In my DevEnv I cloned the chained-ci example project: + +~~~~ + git clone https://gitlab.com/Orange-OpenSource/lfn/ci_cd/chained-ci-examples.git +~~~~ + +* Copy the example code to chained-ci + +~~~~ + cp -r chained-ci-examples/* chained-ci +~~~~ + +* Add submodules in chained-ci and checkin all + +~~~~ + cd chained-ci + git submodule add -b alpine-ansible https://gitlab.com/Orange-OpenSource/lfn/ci_cd/chained-ci-roles.git roles + git submodule add https://gitlab.com/Orange-OpenSource/lfn/ci_cd/chained-ci-vue.git chained-ci-vue + + git add * + ... + git commit -a + git push +~~~~ + +* Extract submodules + +~~~~ + git submodule init + git submodule update +~~~~ + +* Setup own chain + + * Create entry in ```pod_inventory/infrastructure``` (e.g. pod-test) + * Add ```pod_inventory/hostvars/pod-test.yml``` file for the chain-steps and jumphost + * Modify the ```pod_inventory/group_vars/all.yml``` + +* Modifications in ```pod_inventory/group_vars/all.yml``` + * add image and image_tag in runner section + * remove not needed git_projects + * add gitlab settings + * create project settings + * create a trigger token on the target project (in CI/CD settings) and copy token + * generate encrypted trigger_token password + + ``` + echo '...password...' > .vault + ``` + + ``` + ansible-vault encrypt_string --vault-password-file .vault '...token...' --name 'trigger-token' + ``` + + * add encrypted trigger_token to all.yml + + +Example: +~~~~ +runner: + tags: + - gitlab-org + env_vars: + CHAINED_CI_SRC: https://gitlab.devops.telekom.de/tnap/onapcommunity/integrationproject/onapdeployment/chained-ci.git + docker_proxy: + image: registry.gitlab.com/orange-opensource/lfn/ci_cd/docker_ansible + image_tag: latest + +gitlab: + pipeline: + delay: 15 + base_url: https://gitlab.devops.telekom.de + api_url: https://gitlab.devops.telekom.de/api/v4 + private_token: "{{ lookup('env','CI_private_token') }}" + + git_projects: + config: + stage: config + url: https://gitlab.devops.telekom.de/tnap/onapcommunity/integrationproject/onapdeployment/chained-ci.git + branch: "{{ lookup('env','config_branch')|default('master', true) }}" + path: pod_config + cloud-infra: + stage: infra_install + api: https://gitlab.devops.telekom.de/api/v4/project/32660 + url: https://gitlab.devops.telekom.de/tnap/onapcommunity/integrationproject/onapdeployment/cloud-infra + trigger_token: !vault | + $ANSIBLE_VAULT;1.1;AES256 + 66386364383232303832656238636130373430633539613566646337313164323733333138666163 + 3766623563303133393231663237353633353365633063350a653231393436313961613733643036 + 37393262363864393734323532383662663263663863646233366639633130323731343237653661 + 6265323237306634620a376166616564663135316363333136356135613336646665386532616263 + 64373537346235386438633130656363386633383337626337656234646361366263 + branch: "{{ lookup('env','cloud-infra_branch')|default('master', true) }}" + get_artifacts: config + pull_artifacts: + timeout: 300 + parameters: + ansible_verbose: "{{ lookup('env','ansible_verbose') }}" +~~~~ + +* Create .gitlab-ci.yml + +~~~~ + ansible-playbook -vvv --vault-password-file ~/git/LabInstallation/chained-ci/.vault -i ./pod_inventory/inventory ./gitlab-ci-generator.yml +~~~~ + +* Checkin all modified files + +* Add in chained-ci project CI/CD the following variables: + + ```ANSIBLE_VAULT_PASSWORD``` -> a secret for the Vault password + + ```CI_private_token``` -> API token to access Gitlab (see User-Settings-Access Tokens) + +How to create own project pipelines: + +* [DTs Chained-ci pipelines](chained-ci-pipeline.md) + diff --git a/doc/chained-ci-intro.md b/doc/chained-ci-intro.md new file mode 100644 index 0000000..01d74fe --- /dev/null +++ b/doc/chained-ci-intro.md @@ -0,0 +1,61 @@ +# Introduction to Chained-ci + +Chained-ci is a solution to chain CI/CD pipelines from different gilab projects. +It leverages gitlab APIs and features like the artifacts. + + + +As an example if you want to install a solution on an infrastructure, you can +chain the project pipeline in charge of infrastructure deployment then the +pipeline in charge of the deployment of the target solution thanks to +environment variables, configuration parameters provided as artifacts at the +end of the first pipeline. + +Chained-ci offers additional possibilities such as using static artifacts, +notifying third parties. + +## Why Chained-ci + +Historically involved in several Open Source projects, it was very hard to +harmonize CI/CD strategies as each project had its own way to manage it and +automation culture. +OPNFV XCI initiative was created to provide a way to create links between CI/CD +teams of different Open Source Networking communities. + +In parallel, internally we started having lots of projects on gitlab. +The goal was to be able to chain the pipelines of these different gitlab +projects. Each project remains responsible for its CI/CD strategy but may expose +artifacts and can be consumed by other projects. + +Thanks to Chained-ci, it is possible to easily mix and match pipelines and unify +CI/CD strategy leveraging gitlab capabilities. + +It is possible on the same infrastructure to automate the deployments and +testing of several projects in parallel. + +## Chained-ci project + +Chained-ci is composed of several sub-projects: + +* [Chained-ci](https://gitlab.forge.orange-labs.fr/osons/chained-ci): the part + where the chains will be created and configured +* [Chained-ci-roles](https://gitlab.com/Orange-OpenSource/lfn/ci_cd/chained-ci-roles): + the repository hosting the ansible roles +* [Chained-ci-tools](https://gitlab.com/Orange-OpenSource/lfn/ci_cd/chained-ci-tools): + some tooling to simplify the creation of the chains +* [Chained-ci-vue](https://gitlab.com/Orange-OpenSource/lfn/ci_cd/chained-ci-vue): + a web front end to easily visualise the different chains. + + + +If you click on any element of the chain, you will see the corresponding gitlab +pipeline. You can also restart the chain in case of failure at the step it +failed. + +## Get started + +* [DTs Chained-ci installation](chained-ci-install.md) +* [DTs Chained-ci pipelines](chained-ci-pipeline.md) +* [Orange: Create your Chained-ci](./user-guide/configure.md) +* [Orange: Run your Chained-ci](./user-guide/run.md) +* [Orange: Examples](./example/example.md) diff --git a/doc/chained-ci-pipeline.md b/doc/chained-ci-pipeline.md new file mode 100644 index 0000000..8af10bf --- /dev/null +++ b/doc/chained-ci-pipeline.md @@ -0,0 +1,27 @@ +# Creation of an own Project Chain + +Inside of the Chained-CI project installation chains can be created +to automate the the Openstack tenant, network and VM creation, as +well as the installation of applications (e.g. ONAP) + +Currently the following projects are available supporting chaining: + +* cloud-infra -> Creation of Openstack tenant, network and VMs +* ... + +In order to define the pipelines the following steps need to be followed: + +* Projects need to be defined in all.yml (see [Chained-CI install](https://git.sdp.telekom.de/LabInstallation/chained-ci/blob/master/doc/chained-ci-install.md "Chained-CI installation")) +* in the directory chained-ci/pod_config the inventory and pod definitions + have to be created +* in the directory chained-ci/pod_inventory the pipeline definitions + have to be created + +The execution of the pipeline requires a Jumphost within the target cloud. +Requirements for the Jumphost: +* OS: e.g. Ubuntu 18.04 +* Packages: ansible, python-pip, python3-pip, jq +* Network must be able to reach and resolve the VIM Keystone interface +* To access the JH a RSA keypair needs to be created without passphrase +* The public key has to be added to the .ssh/authorized_keys of the + ansible user (e.g. ubuntu) diff --git a/doc/img/chained-CI.png b/doc/img/chained-CI.png Binary files differnew file mode 100644 index 0000000..ec096b8 --- /dev/null +++ b/doc/img/chained-CI.png diff --git a/doc/img/chained-ci-examples.png b/doc/img/chained-ci-examples.png Binary files differnew file mode 100644 index 0000000..c771cae --- /dev/null +++ b/doc/img/chained-ci-examples.png diff --git a/doc/img/chained-ci-ui.png b/doc/img/chained-ci-ui.png Binary files differnew file mode 100644 index 0000000..245f806 --- /dev/null +++ b/doc/img/chained-ci-ui.png diff --git a/doc/img/example1.png b/doc/img/example1.png Binary files differnew file mode 100644 index 0000000..e2ee28e --- /dev/null +++ b/doc/img/example1.png diff --git a/doc/img/example2.png b/doc/img/example2.png Binary files differnew file mode 100644 index 0000000..2f2fc94 --- /dev/null +++ b/doc/img/example2.png diff --git a/doc/img/example3.png b/doc/img/example3.png Binary files differnew file mode 100644 index 0000000..1eeaddc --- /dev/null +++ b/doc/img/example3.png diff --git a/doc/img/inpod.png b/doc/img/inpod.png Binary files differnew file mode 100644 index 0000000..dc95989 --- /dev/null +++ b/doc/img/inpod.png diff --git a/doc/img/pipeline_creation_1.jpg b/doc/img/pipeline_creation_1.jpg Binary files differnew file mode 100644 index 0000000..05b5178 --- /dev/null +++ b/doc/img/pipeline_creation_1.jpg diff --git a/doc/img/pipeline_creation_2.jpg b/doc/img/pipeline_creation_2.jpg Binary files differnew file mode 100644 index 0000000..d922476 --- /dev/null +++ b/doc/img/pipeline_creation_2.jpg diff --git a/doc/img/pipeline_creation_3.jpg b/doc/img/pipeline_creation_3.jpg Binary files differnew file mode 100644 index 0000000..1e7a915 --- /dev/null +++ b/doc/img/pipeline_creation_3.jpg diff --git a/doc/img/schedule_creation_1.jpg b/doc/img/schedule_creation_1.jpg Binary files differnew file mode 100644 index 0000000..48987c5 --- /dev/null +++ b/doc/img/schedule_creation_1.jpg diff --git a/doc/img/schedule_creation_2.jpg b/doc/img/schedule_creation_2.jpg Binary files differnew file mode 100644 index 0000000..f447f20 --- /dev/null +++ b/doc/img/schedule_creation_2.jpg diff --git a/doc/img/schedule_creation_3.jpg b/doc/img/schedule_creation_3.jpg Binary files differnew file mode 100644 index 0000000..5ff84ed --- /dev/null +++ b/doc/img/schedule_creation_3.jpg diff --git a/doc/user-guide/configure.md b/doc/user-guide/configure.md new file mode 100644 index 0000000..4d9e30e --- /dev/null +++ b/doc/user-guide/configure.md @@ -0,0 +1,280 @@ +# Create your chain + +You must clone the chained-ci project. + +```Shell + git clone https://gitlab.forge.orange-labs.fr/osons/chained-ci + cd chained-ci + git submodule init + git submodule update +``` + +The chained-ci repository can be shown as follows: + +```Ascii +. +├── pod_inventory +│ ├── group_vars +│ │ └── all.yml +│ ├── host_vars +│ │ ├── existing_chained.yml +│ │ ├── ... +│ │ └── vpod1.yml +│ └── inventory +└── roles + ├── artifact_init + │ ├── defaults + │ └── tasks + ├── get_artifacts + │ ├── defaults + │ └── tasks + ├── gitlab-ci-generator + │ ├── tasks + │ └── templates + ├── run-ci + │ └── tasks + └── trigger_myself + └── tasks +``` + +The ansible roles are retrieved through the git submodules. + +Note: It is recommended to create your branch to create your new chain. + +To create you chained-ci chain + +* Edit the **pod_inventory/inventory** file to declare your chain +* Create a file in **pod_inventory/host_vars** to configure your chain +* Edit **pod_inventory/group_vars/all.yaml** to reference some variables if + needed + +## Declare your chain + +You must first declare your chain name in the pod_inventory/inventory file. +Edit the file and add your chain name at the end of the file. +The chain name shall be self-explicit. + +Even if the naming is free, it is recommended to follow the syntax: + +`[target][_[infra]]_[[ci_option]]_[pod]]` + +e.g. + +* `pod2`: deployment of the pod2 infrastructure (GNU/Linux OS) +* `acumos_pod4`: deployment of acumos on POD4 +* `functest_pod4`: run functest tests on POD4 +* `k8s_pod1`: deployment of kubernetes on bare metal on pod1 +* ... +* `onap_k8s_daily_pod4`: daily deployment of onap on k8s infra installed on pod4 + +## Define your chain + +The second step consists in creating the ansible role for the new chain. + +Create a file in pod_inventory/host_vars. + +The file name shall correspond to the chain name you declared in the inventory +file e.g. acumos_pod4.yaml. + +This file describes the chain. It reference the different projects you want to +chain, indicating the expected artifacts to be re-used from one project to +another. + +```YAML +--- +jumphost: + server: <target POD jumphost IP> + user: <ansible user> +[option section] +scenario_steps: + step 1: + ... + step 2: + ... + step N: + ... +``` + +The chain will be run sequentially (e.g. config then step 1 until step N). +Each pipeline can however parallelize some stages. + +The option section is not mandatory. + +### jumphost Definition + +A jumphost is associated to a chained-ci even if the different steps can be +performed through projects hosted in different environements (public gitlab.com +and internal gitlab). The creator of the chain shall ensure that the projects +can be invoked from the referenced jumphost. + +### Chained-ci options + +The possible option fields are: + +* environment: this parameter is used and propagated as an environment variable + to all the projects involved in the chained-ci +* longlife_artifact: `true` (not present by default) +* inpod: `<other_chained_ci_reference>` +* healthchecks_id: `<uid>` + +#### longlife_artifact + +The longlife_artifact option is needed to store the artifacts over a long +period. As other chains may consume the artifacts, if the chained-ci is not run +frequently, the artifacts could be deleted after a while and become unavailable +for other chained-ci. + +Typically if you install an infrastructure once and run several chains +requesting the artifacts of the infrastructure installation, it makes sense to +set this option. + +#### inpod + +Chained-ci can be seen as matriochka, in fact you may trigger deployments once +a previous deployment has been completed. + +The inpod parameter allows to define a sort of heritage in the chained-ci +declaration. + + + +#### healthchecks + +The healthcheck option is used to reference your chained-ci to +[healthcheck](https://github.com/healthchecks/healthchecks). + +### Chained-ci steps + +Each step can be defined as follow: + +```YAML +[step name]: + project: [gitlab project reference] + get_artifacts: + - name: [project name for the artifact] + [static_src: true] + [extra parameters] + [infra] +``` + +#### gitlab project reference (mandatory) + +It shall correspond to a name of a gitlab project. + +#### get_artifacts (optional) + +The get_artifacts is to retrieved data (env variables, configuration files) +from a previous gitlab pipeline from a chained-ci. + +These artifacts can be retrieved: + +* dynamically (by Default): gitlab API will be called to retrieved the last + artifacts corresponding to the project referenced +* statically: if static_src parameter is set to True, you then must indicate + the path to the artifact + +By default Chained-ci will call gitlab APIs to retrieved the artifact. + +##### limit_to + +Please note that it is possible to retrieve only a subset of the files from +an artifact by using **limit_to**. + +```YAML +get_artifacts: + - name: infra_deploy + limit_to: + - vars/user_cloud.yml: vars/user_cloud.yml +``` + +##### in_pipeline=false + +It is also possible to indicate that the artifact cannot be retrieved from the +pipeline by setting in_pipeline=false in the get_artifact section. You then +have to add the name of the pipeline to fetch and the step in the pipeline. + +In the example, we retrieve artifacts from `config` step of last +`onap_k8s_ic_pod4` pipeline. + +```YAML + get_artifacts: + - name: config:onap_k8s_ic_pod4 + in_pipeline: false + limit_to: + - vars/pdf.yml: vars/pdf.yml + - vars/idf.yml: vars/idf.yml +``` + +#### extra parameters (optional) + +You may indicate here additional variables to configure your Chained-ci. These +variables will be given as environment variables for the run of the project. It +shall be configured as follows: + +```YAML +extra_parameters: + key 1: value 1 + key 2: value 2 + .... + key N: value N +``` + +#### infra (optional) + +This option can be specified if you want to use a specific resources of an +existing infra. This infra shall be declared in the Infrastructure Description +file. + +## Create your static artifacts + +It is possible to create a static artifact file. It consists in a zip file +containing all the files. + +For instance if you need to precise an inventory my_jumhost and a set of +vaiables declared in a foo.yml file, you will need to prepare your file. + +``` +├── inventory +│ └── my_jumphost +└── vars + └── foo.yml +``` + +Zip the different files in my_foo.zip and reference zoo in the chained-ci. + +``` +get_artifacts: + - name: my_foo + static_src: true +``` + +Note that my_foo must not be in the path once the archive unzipped. + +## Gitlab-ci generator + +The `.gitlab-ci.yml` of your Chaine-ci project may be a little be tricky to +generate. That is why a gitlab-ci generator has been introduced. + +You can run the gitlab-ci-generator ansible role locally on your machine. +Based on the chained-ci you created, it will re-generate the file properly. + +```Shell +ansible-playbook -i ./pod_inventory/inventory ./gitlab-ci-generator.yml +``` + +`<chain_id>` is the name of the chain you defined in *Declare your chain* step + +once you have generated `.gitlab-ci.yml`, you'll need to push your change to the +gitlab server (via a merge request or not, depending on you push policy) before +using it. + +## chained-ci tools + +It provides a set of tools to deal with Chained-ci: + +* Get artifacts from a projects +* set the configuration (ssh key and ssh config) +* Generate Vault key file + +See [README](https://gitlab.com/Orange-OpenSource/lfn/ci_cd/chained-ci-tools) +for details. diff --git a/doc/user-guide/run.md b/doc/user-guide/run.md new file mode 100644 index 0000000..2b77296 --- /dev/null +++ b/doc/user-guide/run.md @@ -0,0 +1,30 @@ +# Run your Chained-ci + +From the gitlab web page of your Chained-ci project, select the CI/CD menu + + + +* Click on the Run Pipeline button + +  + +* Select the acurate branch (Default: master) (orange in image) + +  + +* set variable POD, here POD is the identifier for your chained-CI (yellow in image) + +  + +* Click on Create pipeline i(green in image) + +  + +You can obviously also use the Schedules menu if you want to regularly run +your chain (cronjob defined). + + + + + +
\ No newline at end of file |
