diff options
22 files changed, 411 insertions, 0 deletions
diff --git a/README.md b/README.md new file mode 100644 index 0000000..f3ba03b --- /dev/null +++ b/README.md @@ -0,0 +1,97 @@ +# Infrastructure as code for OpenStack deployment of ONAP + +## Build your infrastructure with Terragrunt(Terraform) for ONAP + +Preparing ONAP for deployment requires Openstack VMs with Kubernetes and helm installed. +Building underlying infrastructure for ONAP with Openstack GUI or command-line interfaces is not only time-consuming but also prone to mistakes. +By providing Infrastructure as Code for deploying ONAP, building and managing the underlying infrastructure become simpler and easier. +This [link](https://docs.onap.org/en/casablanca/submodules/oom.git/docs/oom_setup_kubernetes_rancher.html#onap-on-kubernetes-with-rancher) shows how to set up the underlying infrastructure with Openstack GUI and Command line tool. + +This Terragrunt(Terraform) code provides the same infrastructure as you would create through the process outlined in the link above. + + +## Directory structure +``` +openlab # Terragrunt scripts to feed configuration into the Terraform modules + └ RegionOne # For multi regions. e.g, us-east-1 + └ stage # Environment specific configuration. e.g, QA/Stage/Prod + └ resource +``` + +Infrastrucuture is organized hierarchically in folders. +The root level folder represents an account for clouds such as Openstack or AWS. +The second and third levels represent the region in a cloud and environment under the region respectively. + +### Preparation +1. You need a cloud storage bucket to store an intermediate state for your infrastructure. The remote state enables your team to work together as well. We tested this code using a Google Storage bucket. You can choose AWS S3 as well as others. + + +2. Openstack is the primary target for this code. We tested this code onto Openstack v3.8 (Ocata) +We deployed VMs and K8s with the scripts and after that we deployed ONAP Frankfurt version with OOM. + +### Usage +#### Set up environment variables for your target cloud. + +1.a You need to export cloud storage credential. +For instance, if you use Google Storage bucket, you can download the credentials from Google UI or the command-line tool. +Go to Google Cloud project's `IAM & Admin` menu and choose the service account associated with the storage. +You can export the credential as a JSON formatted file. Then +`export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credential-file`. +Please, refer to the following [link](https://cloud.google.com/iam/docs/creating-managing-service-account-keys). + +Second, you need to export Openstack credentials. You can use the openstack.rc file downloaded from your Openstack project. +Please, refer to the following [link](https://docs.openstack.org/ocata/user-guide/common/cli-set-environment-variables-using-openstack-rc.html) for details. + +#### Fill in files + - `account.hcl`: Top-level configuration for a cloud account. + - `region.hcl`: The second level configuration for a region within the cloud + - `env.hcl`: The third level configuration for an environment within the region + - `terragrunt.hcl`: files under the compute directory. Since Kubernetes deployment needs 2 types of nodes (control and worker) plus 1 NFS cluster, these files under the compute directory contain the configuration for Kubernentes and NFS nodes + +#### Building all modules for an environment +Move to an environmental level folder, e.g stage. +Then run `terragrunt apply-all` followed by `terraform init` +If you run the command above for the first time and your version of Terraform is <= 0.13-beta, You may end up the error below +`Provider "rke" not available for installation.` +You have to manually download the `rke` provider and copy it under a Terraform's cached plug-in directory. +After copying the rke provider, move to the `rancher` directory and run `terragrunt init`. +Please, refer to the following [link](https://github.com/rancher/terraform-provider-rke#installing-the-provider) for details. + +https://github.com/gruntwork-io/terragrunt-infrastructure-live-example#deploying-all-modules-in-a-region + +#### Updating infrastructure version +Infrastructure may evolve. You can use existing infrastructure as it is or updating the infrastructure to meet a new requirement. +To deploy a different version of infrastructure, you can change a tag of `source` module version. +Please, refer to the below document. +If you like to test a new module (Terraform code) with Terragrunt, you just need to change the source attribute within Terrafrom block in each terragurnt.hcl file. +[link](https://www.terraform.io/docs/modules/sources.html#generic-git-repository) + +#### Using Kubernetes and helm + +Please, refer to [link](https://github.com/gruntwork-io/terragrunt-infrastructure-live-example#example-infrastructure-live-for-terragrunt) + +#### Obtaining your KUBECONFIG +Finally, You need to export Kubernenetes credentials. +This credential is used when Helm service account is created. +For example, `export KUBECONFIG=/path/to/kube_config_cluster.yaml` +In default, `kube_config_cluster.yaml` will be created under `path/to/openlab/RegionOne/stage` directory once you run `terragrunt apply-all` + +## Google Cloud Backend for Terraform +To use the Google Cloud Storage backend for Terraform -- it stores state and manages locking -- you'll need to install the Google Cloud SDK. Follow the instructions here https://cloud.google.com/sdk. +You can do this task with Google Cloud's Web. + +1. Create a service account +gcloud iam service-accounts create `service-account-name` + +2. Binding the service account with a role +gcloud projects add-iam-policy-binding `project id` --member "serviceAccount:service-account-name-above@project-id.iam.gserviceaccount.com" --role "roles/proper-role-such-as-storage-user" + +3. To create a key for the service account created above +gcloud iam service-accounts keys create account.json --iam-account `service-account-name-above@project-id.iam.gserviceaccount.com + +4. Create a storage bucket +gsutil mb -p project-id gs://storage-bucket-name + +## Secrets +How to hide your secret and provide it via a key management tool. Please, refer to the link below. +Refer to https://nderraugh.github.io/ diff --git a/openlab/README.md b/openlab/README.md new file mode 100644 index 0000000..1df82fb --- /dev/null +++ b/openlab/README.md @@ -0,0 +1,3 @@ +# openlab + +Terraform scripts for Openlab(Windriver Lab) diff --git a/openlab/modules/kubernetes/kubernetes/main.tf b/openlab/modules/kubernetes/kubernetes/main.tf new file mode 100644 index 0000000..eccbcd4 --- /dev/null +++ b/openlab/modules/kubernetes/kubernetes/main.tf @@ -0,0 +1,48 @@ +provider "helm" { + version = "~> 0.10.0" + init_helm_home = true + install_tiller = true + service_account = var.service_account + namespace = var.namespace + tiller_image = "gcr.io/kubernetes-helm/tiller:v2.16.6" + + kubernetes { + config_path = "${var.kubernetes_home}/kube_config_cluster.yaml" + } +} + +provider "kubernetes" { + version = ">= 1.10" + load_config_file = true +} + +resource "kubernetes_service_account" "tiller" { + automount_service_account_token = true + + metadata { + name = var.service_account + namespace = var.namespace + } +} + +resource "kubernetes_cluster_role_binding" "tiller" { + metadata { + name = "tiller" + } + + role_ref { + api_group = "rbac.authorization.k8s.io" + kind = "ClusterRole" + name = "cluster-admin" + } + + subject { + kind = "ServiceAccount" + name = kubernetes_service_account.tiller.metadata.0.name + namespace = kubernetes_service_account.tiller.metadata.0.namespace + } + + provisioner "local-exec" { + command = "helm init --service-account ${var.service_account};kubectl -n ${var.namespace} rollout status deployment/tiller-deploy" + } +} diff --git a/openlab/modules/kubernetes/kubernetes/variables.tf b/openlab/modules/kubernetes/kubernetes/variables.tf new file mode 100644 index 0000000..6e26b85 --- /dev/null +++ b/openlab/modules/kubernetes/kubernetes/variables.tf @@ -0,0 +1,5 @@ +variable "kubernetes_home" {} + +variable "service_account" {} + +variable "namespace" {}
\ No newline at end of file diff --git a/openlab/modules/kubernetes/rancher/data.tf b/openlab/modules/kubernetes/rancher/data.tf new file mode 100644 index 0000000..fa38a55 --- /dev/null +++ b/openlab/modules/kubernetes/rancher/data.tf @@ -0,0 +1,15 @@ +data "terraform_remote_state" "control_plane_floating_ips" { + backend = var.backend + config = { + bucket = var.backend_state + prefix = "${var.region}/${var.environment}/compute/control/terraform.tfstate" + } +} + +data "terraform_remote_state" "worker_node_floating_ips" { + backend = var.backend + config = { + bucket = var.backend_state + prefix = "${var.region}/${var.environment}/compute/worker/terraform.tfstate" + } +} diff --git a/openlab/modules/kubernetes/rancher/main.tf b/openlab/modules/kubernetes/rancher/main.tf new file mode 100644 index 0000000..b787f0b --- /dev/null +++ b/openlab/modules/kubernetes/rancher/main.tf @@ -0,0 +1,55 @@ +# If your terraform version is < 0.13-beta, manual installation is needed. +# https://github.com/rancher/terraform-provider-rke +provider "rke" {} + +locals { + control_node = [for ip in (flatten(data.terraform_remote_state.control_plane_floating_ips.outputs.floating_ips)): { + address = ip + role = "control" + }] + worker_node = [for ip in (flatten(data.terraform_remote_state.worker_node_floating_ips.outputs.floating_ips)): { + address = ip + role = "worker" + }] + + all_node = concat(local.control_node, local.worker_node) +} + +resource "rke_cluster" "cluster" { + kubernetes_version = var.kubernetes_version + cluster_name = var.kubernetes_cluster_name + + dynamic nodes { + + for_each = local.all_node + + content { + address = nodes.value.address + user = var.kubernetes_user + role = (nodes.value.role == "control") ? [ "controlplane", "etcd" ] : ["worker"] + } + } + + # You have to have private key on your machine excuting terraform + # An Openstack keypair is generated and stored within the remote state at + # "${var.region}/${var.environment}/keypair/terraform.tfstate" + ssh_key_path = var.ssh_private_key_path + + disable_port_check = false + + depends_on = [null_resource.wait-for-docker] +} + +resource "local_file" "kube_cluster_yaml" { + filename = "${var.kubernetes_home}/kube_config_cluster.yaml" + sensitive_content = rke_cluster.cluster.kube_config_yaml +} + +resource "null_resource" "wait-for-docker" { + provisioner "local-exec" { + # wait untill VM's bootstrapping's done + # If your VMs for Computing node have finished bootstrapping already, + # you may not need this waiting time + command = "sleep 120" + } +} diff --git a/openlab/modules/kubernetes/rancher/output.tf b/openlab/modules/kubernetes/rancher/output.tf new file mode 100644 index 0000000..aa25e43 --- /dev/null +++ b/openlab/modules/kubernetes/rancher/output.tf @@ -0,0 +1,7 @@ +output "ips" { + value = local.all_node +} + +output "kube_config_file" { + value = local_file.kube_cluster_yaml.filename +}
\ No newline at end of file diff --git a/openlab/modules/kubernetes/rancher/variables.tf b/openlab/modules/kubernetes/rancher/variables.tf new file mode 100644 index 0000000..8d63243 --- /dev/null +++ b/openlab/modules/kubernetes/rancher/variables.tf @@ -0,0 +1,17 @@ +variable "backend" {} + +variable "backend_state" {} + +variable "region" {} + +variable "environment" {} + +variable "ssh_private_key_path" {} + +variable "kubernetes_user" {} + +variable "kubernetes_version" {} + +variable "kubernetes_cluster_name" {} + +variable "kubernetes_home" {} diff --git a/openlab/modules/openstack/compute/data.tf b/openlab/modules/openstack/compute/data.tf new file mode 100644 index 0000000..24c6c62 --- /dev/null +++ b/openlab/modules/openstack/compute/data.tf @@ -0,0 +1,36 @@ +data "terraform_remote_state" "keypair" { + backend = var.backend + config = { + bucket = var.backend_state + prefix = "${var.region}/${var.environment}/keypair/terraform.tfstate" + } +} + +data "terraform_remote_state" "network" { + backend = var.backend + config = { + bucket = var.backend_state + prefix = "${var.region}/${var.environment}/network/terraform.tfstate" + } +} + +data "terraform_remote_state" "securitygroup" { + backend = var.backend + config = { + bucket = var.backend_state + prefix = "${var.region}/${var.environment}/securitygroup/terraform.tfstate" + } +} + +data "openstack_images_image_v2" "vm_image" { + name = var.image +} + +data "openstack_compute_flavor_v2" "flavor" { + name = var.flavor +} + +data "openstack_networking_network_v2" "egress_network" { + name = var.network +} + diff --git a/openlab/modules/openstack/compute/main.tf b/openlab/modules/openstack/compute/main.tf new file mode 100644 index 0000000..1e08737 --- /dev/null +++ b/openlab/modules/openstack/compute/main.tf @@ -0,0 +1,27 @@ +resource "openstack_compute_instance_v2" "nodes" { + name = "${var.cluster_name}-${var.node_name}-${count.index}" + image_id = data.openstack_images_image_v2.vm_image.id + flavor_id = data.openstack_compute_flavor_v2.flavor.id + key_pair = data.terraform_remote_state.keypair.outputs.name + network { + name = data.terraform_remote_state.network.outputs.name + } + security_groups = [ data.terraform_remote_state.securitygroup.outputs.name ] + + user_data = var.user_data + + count = var.node_count +} + +resource "openstack_compute_floatingip_v2" "floatingip" { + pool = var.floating_ip_pool + + count = var.node_count +} + +resource "openstack_compute_floatingip_associate_v2" "floatipassociation" { + floating_ip = openstack_compute_floatingip_v2.floatingip[count.index].address + instance_id = openstack_compute_instance_v2.nodes[count.index].id + + count = var.node_count +} diff --git a/openlab/modules/openstack/compute/output.tf b/openlab/modules/openstack/compute/output.tf new file mode 100644 index 0000000..28ae6d0 --- /dev/null +++ b/openlab/modules/openstack/compute/output.tf @@ -0,0 +1,11 @@ +output "floating_ips" { + value = [openstack_compute_floatingip_associate_v2.floatipassociation.*.floating_ip] +} + +output "fixed_ips" { + value = [openstack_compute_instance_v2.nodes.*.access_ip_v4] +} + +output "hostnames" { + value = [openstack_compute_instance_v2.nodes.*.name] +}
\ No newline at end of file diff --git a/openlab/modules/openstack/compute/variables.tf b/openlab/modules/openstack/compute/variables.tf new file mode 100644 index 0000000..871719d --- /dev/null +++ b/openlab/modules/openstack/compute/variables.tf @@ -0,0 +1,18 @@ +variable "cluster_name" { + description = "A name for the cluster" + type = string +} + +variable "backend" {} +variable "backend_state" {} +variable "region" {} +variable "environment" {} + +variable "node_name" {} +variable "image" {} +variable "flavor" {} +variable "network" {} +variable "floating_ip_pool" {} +variable "user_data" {} + +variable "node_count" {} diff --git a/openlab/modules/openstack/keypair/main.tf b/openlab/modules/openstack/keypair/main.tf new file mode 100644 index 0000000..5b1384a --- /dev/null +++ b/openlab/modules/openstack/keypair/main.tf @@ -0,0 +1,6 @@ +resource "openstack_compute_keypair_v2" "key" { + # You cna find a public/private key pair from your remote state storage. + name = "${var.cluster_name}-key" + # In order to generate a new keypair via existing public key + public_key = var.ssh_public_key +} diff --git a/openlab/modules/openstack/keypair/output.tf b/openlab/modules/openstack/keypair/output.tf new file mode 100644 index 0000000..9c042e6 --- /dev/null +++ b/openlab/modules/openstack/keypair/output.tf @@ -0,0 +1,3 @@ +output "name" { + value = openstack_compute_keypair_v2.key.name +}
\ No newline at end of file diff --git a/openlab/modules/openstack/keypair/variables.tf b/openlab/modules/openstack/keypair/variables.tf new file mode 100644 index 0000000..b3c6a92 --- /dev/null +++ b/openlab/modules/openstack/keypair/variables.tf @@ -0,0 +1,6 @@ +variable "cluster_name" { + description = "A name for the cluster" + type = string +} + +variable "ssh_public_key" { } diff --git a/openlab/modules/openstack/network/data.tf b/openlab/modules/openstack/network/data.tf new file mode 100644 index 0000000..fad533f --- /dev/null +++ b/openlab/modules/openstack/network/data.tf @@ -0,0 +1,3 @@ +data "openstack_networking_network_v2" "egress_network" { + name = var.network +}
\ No newline at end of file diff --git a/openlab/modules/openstack/network/main.tf b/openlab/modules/openstack/network/main.tf new file mode 100644 index 0000000..80dd70e --- /dev/null +++ b/openlab/modules/openstack/network/main.tf @@ -0,0 +1,25 @@ +resource "openstack_networking_network_v2" "network" { + name = "${var.cluster_name}-network" + admin_state_up = "true" +} + +resource "openstack_networking_subnet_v2" "subnet" { + name = "${var.cluster_name}-subnet" + network_id = openstack_networking_network_v2.network.id + cidr = "192.168.64.0/24" + ip_version = 4 + gateway_ip = "192.168.64.1" + enable_dhcp = "true" + dns_nameservers = [ "8.8.8.8", "8.8.4.4" ] +} + +resource "openstack_networking_router_v2" "router" { + name = "${var.cluster_name}-router" + admin_state_up = true + external_network_id = data.openstack_networking_network_v2.egress_network.id +} + +resource "openstack_networking_router_interface_v2" "router_interface" { + router_id = openstack_networking_router_v2.router.id + subnet_id = openstack_networking_subnet_v2.subnet.id +}
\ No newline at end of file diff --git a/openlab/modules/openstack/network/output.tf b/openlab/modules/openstack/network/output.tf new file mode 100644 index 0000000..1ca1b0a --- /dev/null +++ b/openlab/modules/openstack/network/output.tf @@ -0,0 +1,3 @@ +output "name" { + value = openstack_networking_network_v2.network.name +}
\ No newline at end of file diff --git a/openlab/modules/openstack/network/variables.tf b/openlab/modules/openstack/network/variables.tf new file mode 100644 index 0000000..86de2f0 --- /dev/null +++ b/openlab/modules/openstack/network/variables.tf @@ -0,0 +1,9 @@ +variable "cluster_name" { + description = "A name for the cluster" + type = string +} + +variable "network" { + description = "A network to the Internet" + type = string +} diff --git a/openlab/modules/openstack/securitygroup/main.tf b/openlab/modules/openstack/securitygroup/main.tf new file mode 100644 index 0000000..af7cbb4 --- /dev/null +++ b/openlab/modules/openstack/securitygroup/main.tf @@ -0,0 +1,10 @@ +resource "openstack_networking_secgroup_v2" "securitygroup" { + name = "${var.cluster_name}-securitygroup" + description = "RKE security group" +} + +resource "openstack_networking_secgroup_rule_v2" "securitygroup_rule" { + direction = "ingress" + ethertype = "IPv4" + security_group_id = openstack_networking_secgroup_v2.securitygroup.id +} diff --git a/openlab/modules/openstack/securitygroup/output.tf b/openlab/modules/openstack/securitygroup/output.tf new file mode 100644 index 0000000..825f4bf --- /dev/null +++ b/openlab/modules/openstack/securitygroup/output.tf @@ -0,0 +1,3 @@ +output "name" { + value = openstack_networking_secgroup_v2.securitygroup.name +}
\ No newline at end of file diff --git a/openlab/modules/openstack/securitygroup/variables.tf b/openlab/modules/openstack/securitygroup/variables.tf new file mode 100644 index 0000000..826fcfb --- /dev/null +++ b/openlab/modules/openstack/securitygroup/variables.tf @@ -0,0 +1,4 @@ +variable "cluster_name" { + description = "A name for the cluster" + type = string +} |