Customized Kubespray Deployment
Tags: kubernetes, kubespray
Published: 2018-10-24 15:23:06

---
title: Customized Kubespray Deployment
published: true
description: kubespray works really well in the default path but the documentation is lacking when you want your infrastructure to be slightly different.
tags: kubernetes, kubespray
cover_image: https://thepracticaldev.s3.amazonaws.com/i/2u6dtlwbesiv0bljo1bi.jpeg
---

# Versions

kubespray | ansible | python | terraform
--- | --- | --- | ---
[this fork](https://github.com/bartlaarhoven/kubespray) | 2.7.0 | 3.6.1 | 0.11.8

⚠️ Warning! Use this fork: https://github.com/kubernetes-incubator/kubespray/pull/3486.

# Motivation

This post serves as documentation for creating a kubernetes cluster from nothing using kubespray on aws with ubuntu images behind a bastion host. I could not find all the documentation that puts all of this together and wanted to write it down for myself the next time I need to do this.

## Steps

1. Clone kubespray and set up some default files

<!-- Uncomment when pull/3486 merges
```sh
git clone https://github.com/kubernetes-incubator/kubespray.git
cp -Rp inventory/sample/ inventory/mycluster
```
-->

```sh
git clone https://github.com/bartlaarhoven/kubespray
cd kubespray
virtualenv ks && . ks/bin/activate
pip install -r requirements.txt
cp -Rp inventory/sample/ inventory/mycluster
```

2. Create an IAM user with admin privileges in some account (TODO probably scope this down?)
0. Create an EC2 key pair
0. Copy the [terraform environment file](https://github.com/kubernetes-incubator/kubespray/blob/7e84de2ae116f624b570eadc28022e924bd273bc/contrib/terraform/aws/credentials.tfvars.example) to `credentials.tfvars` and modify it with the user's key and secret along with the ssh key pair name and the region you'd like the infrastructure to exist in.
0. Customize the [terraform file](https://github.com/kubernetes-incubator/kubespray/blob/7e84de2ae116f624b570eadc28022e924bd273bc/contrib/terraform/aws/terraform.tfvars) with the architecture you'd like, I used 1 master, 1 worker, 1 etcd and left bastions as default. Also modify the inventory file to be `../../../inventory/mycluster/hosts.ini`.
0. Modify the [variables.tf](https://github.com/kubernetes-incubator/kubespray/blob/7e84de2ae116f624b570eadc28022e924bd273bc/contrib/terraform/aws/variables.tf#L23) to be

```hcl
    data "aws_ami" "distro" {
      most_recent = true

      filter {
        name = "name"
    
        values = ["ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-*"]
      }

      filter {
        name = "virtualization-type"
    
        values = ["hvm"]
      }
    
      owners = ["099720109477"]
    }
```

0. Run terraform

```sh
    terraform apply --var-file credentials.tfvars
```

0. Modify the [ansible.cfg](https://github.com/kubernetes-incubator/kubespray/blob/7e84de2ae116f624b570eadc28022e924bd273bc/ansible.cfg) file to use a bastion host by changing the `ssh_args` value to

```
    ssh_args = -F ssh-bastion.conf
```

0. Modify the hosts.ini file. Use the internal DNS names as the `ansible_host` for each of the instances in the private subnet and include the `ansible_user` to be `ubuntu`, for example:

```
    kubernetes-devtest-master0 ansible_host=ip-10-250-205-127.us-west-2.compute.internal ansible_user=ubuntu
```

0. Modify the bastion lines to be the *public* DNS names as the `ansible_host` and include the `ansible_user` to be `ubuntu`, for example:

```
    bastion-0 ansible_host=ec2-22-222-22-22.us-east-2.compute.amazonaws.com ansible_user=ubuntu
```

0. Run ansible-playbook

```
    ansible-playbook -i ./inventory/mycluster/hosts.ini ./cluster.yml -b --become-user=root --flush-cache
```

If you'd like more logs add `-v` or `-vv` up to `-vvvvv`. I also like to pipe this to tee and write the logs to disk or inspection later in case of failure.

Enjoy your new cluster~!

P.S. if anyone knows how to get the code as part of the item so the bulleted stuff works please comment, I'd love to fix the numbering.