Problem with Manual Configuration
While setting up infra for any project on the cloud, VPC(Virtual Private Cloud) is one of the most vital setups we need to do. VPC consists of many moving parts such as Subnet, NAT gateway, Internet Gateway, Route Tables, NACL, Security Group, etc.
Most importantly Bastion Host it's also called Jump Server and used to have an SSH connection to the servers in private VPC or RDP connection so we can say it is a single point of entry from the public network into the private network.
Setting all the above things manually is very much error-prone resulting in a non-functional or exposed VPC. Also managing all these setups and regular changes is a hectic and unsafe task.
Solution
To solve the above problem that comes with manual intervention or setup we can bring Configuration Management into the picture.
Configuration management gives the benefit of Automatic Setup so that there is no human error.
Centralized management and version control [IAAC].
And here, we'll be using Ansible as our configuration management tool that matches all the above requirements. Ansible provides centralized management with ansible-playbook and ansible-galaxy and we can prepare playbooks with YAML as configuration language. We can also version control all the ansible configuration files with Git.
Architecture
Our configuration begins with Playbooks, these playbooks will be using variables that we are going to define in the Vars file and some Ansible Configurations.
These playbooks will be powered by some Ansible Modules.
These modules are going to use Python Boto which will make API calls to our AWS Account
The first thing this Python Boto is going to do is establish a VPC(Virtual Private Cloud) and in VPC we'll be using two Availability Zones.
We'll have public subnets in both zones and these public subnets will route their requests to Internet Gateway via Route Tables. Each public subnet will have its route tables.
We'll also have Private subnets in both zones and private subnets also have their respective route table which routes the request to the NAT gateway. And NAT gateway further takes it back to the Internet Gateway route table of the public subnet.
We'll also have Bastion Host in the Public Subnet which will connect to private EC2 instances that are placed in Private Subnet.
We'll do the entire setup using Ansible.
Flow of Execution
Create an EC2 instance to run Ansible playbooks
Install Ansible
Install Boto
Setup EC2 role for Ansible
Create a project directory
Sample task of creating key-pair to test
Create Variables File for VPC and bastion host
Create a VPC setup playbook
Create a Bastion setup playbook
Create a site.yaml to call both playbooks at once.
Ansible Setup
This is the very first step where we'll install ansible in an EC2 instance which will act just as a controller. We are not going to run any service here so, any lightweight t2.micro will be fine. Here we'll install Ansible and the required dependencies to create and run playbooks for our cloud automation.
We need Python Boto installed in our system to make API calls to our AWS account and create resources accordingly.
Ansible Dry Run
Here, we'll create a key pair for AWS using the Ansible playbook to test if our setup and dependencies are working fine or not.
---
- name: aws_keypair automation
hosts: localhost
connection: local
gather_facts: False
tasks:
- name: sample_key
ec2_key:
name: Sample
region: us-east-1
register: key_out
- name: opening the output in debug mode
debug:
var: key_out
- name: store login key
copy:
content: "{{key_out.key.private_key}}"
dest: ./sample-key.pem
when: key_out.changed
The above is a sample playbook that will create key pair in our AWS account. We can get the modules for creating anything in AWS or any other cloud provider from Ansible Module Documentation.
NB: we need to install the required python-boto dependency for the module which is python3-boto3.
Authentication
While running the playbook with ansible-playbook test-aws.yaml the command we encountered the below error which is an authentication error that implies, currently, the Ansible host machine doesn't have the privilege to create resources in the AWS account.
To overcome this error we can either give the authentication keys in the playbook which is not a safer choice or we can create a role with the necessary permissions and attach the role to the Ansible Host.
And after doing the same we can run the playbook and validate that our setup is done:
We are now able to automate tasks on the cloud using Ansible
Retrieval Operations
register and debug
We can also hold the output from a task to a variable and use it further in the playbook using the register attribute. While creating a keypair we do get a public key and a private key, we can hold that in a variable.
tasks:
- name: sample_key
ec2_key:
name: Sample
region: us-east-1
register: key_out
We can print the variable in the terminal using debug module
- debug:
var: key_out
copy
we can also store that output in a file using the copy module
- name: store login key
copy:
content: "{{key_out.key.private_key}}"
dest: ./sample-key.pem
Now you can see key is created and stored:
Error Handling with Conditions
Now, consider a scenario where we have a lot of tasks and we need to run the same playbook multiple times. In that case, with our first task key-pair will be created and when running the second time we'll be encountered with below error as key pair has already been created
To handle this error we can use the condition in our playbook:
- name: store login key
copy:
content: "{{key_out.key.private_key}}"
dest: ./sample-key.pem
when: key_out.changed
Key-takeaways
While doing cloud automation it uses Python boto for API calls
Every module gives output after the API call in JSON format so it's always necessary to save the output using register so that we can use it for further tasks.
Variables for VPC & Bastion
In this section, we'll create all the necessary variables that will further be used in playbooks.
Variables make the code reusable and organized. Whenever there are any changes we do not need to change the playbook configurations, we just need to edit the variables.
Here we'll create variables for VPC and Bastion host. A bastion host is nothing but our single entry point for our VPC from the public network.
Variable file for VPC
vpc_name: "Vprofile-vpc"
vpcCider: '172.20.0.0/16'
PubSub1Cider: 172.20.1.0/24
PubSub2Cider: 172.20.2.0/24
PubSub3Cider: 172.20.3.0/24
PrivSub1Cider: 172.20.4.0/24
PrivSub2Cider: 172.20.5.0/24
PrivSub3Cider: 172.20.6.0/24
region: "us-east-1"
zone1: us-east-1a
zone2: us-east-1b
zone3: us-east-1c
state: present
Variable file for Bastion Host
bastion_ami: ami-09538990a0c4fe9be
region: us-east-1
MYIP: 117.99.44.80/32
you can refer to my repository for the entire code: https://github.com/ritesh-kumar-nayak/Ansible-aws-vpc
Playbook for VPC
---
- name: vpc setup vprofile
hosts: localhost
connection: local
gather_facts: False
tasks:
- name: Import variables from vpc_setup
include_vars: Variables/vpc_setup
- name: Creating vprofile VPC
ec2_vpc_net:
name: "{{vpc_name}}"
cider_block: "{{vpcCider}}"
region: "{{region}}"
dns_support: yes
dns_hostnames: yes
tenancy: default
state: "{{state}}"
register: vpc_out
- name: debug mode
debug:
var: vpc_out
After executing the above playbook, it has created the VPC
Playbook for Subnets
We'll be creating 1 public and 1 private subnet each, in 3 different zones. And below is the playbook for the same:
Public Subnet
- name: create public subnet-1 in zone-1
ec2_vpc_subnet:
vpc_id: "{{vpc_out.vpc.id}}"
cidr: "{{PubSub1Cidr}}"
state: "{{state}}"
region: "{{region}}"
az: "{{zone1}}"
map_public: yes
resource_tags:
Name: vprofile-pubsub1
register: pubsub1_out
- name: create public subnet-2 in zone-2
ec2_vpc_subnet:
vpc_id: "{{vpc_out.vpc.id}}"
cidr: "{{PubSub2Cidr}}"
state: "{{state}}"
region: "{{region}}"
az: "{{zone2}}"
map_public: yes
resource_tags:
Name: vprofile-pubsub2
register: pubsub2_out
- name: create public subnet-3 in zone-3
ec2_vpc_subnet:
vpc_id: "{{vpc_out.vpc.id}}"
cidr: "{{PubSub3Cidr}}"
state: "{{state}}"
region: "{{region}}"
az: "{{zone3}}"
map_public: yes
resource_tags:
Name: vprofile-pubsub3
register: pubsub3_out
And finally, all 3 public subnets are created
Private Subnet
- name: create private subnet-1 in zone-1
ec2_vpc_subnet:
vpc_id: "{{vpc_out.vpc.id}}"
cidr: "{{PrivSub1Cidr}}"
state: "{{state}}"
region: "{{region}}"
az: "{{zone1}}"
map_public: yes
resource_tags:
Name: vprofile-privsub1
register: privsub1_out
- name: create private subnet-2 in zone-2
ec2_vpc_subnet:
vpc_id: "{{vpc_out.vpc.id}}"
cidr: "{{PrivSub2Cidr}}"
state: "{{state}}"
region: "{{region}}"
az: "{{zone2}}"
map_public: yes
resource_tags:
Name: vprofile-privsub2
register: privsub2_out
- name: create private subnet-3 in zone-3
ec2_vpc_subnet:
vpc_id: "{{vpc_out.vpc.id}}"
cidr: "{{PrivSub3Cidr}}"
state: "{{state}}"
region: "{{region}}"
az: "{{zone3}}"
map_public: yes
resource_tags:
Name: vprofile-privsub3
register: privsub3_out
Now with this, we have created 3 private subnets in the same vpc
Creating Internet Gateway
Now, we need an Internet Gateway for our public subnets. And to do so below is the Ansible task:
- name: Create Internet Gateway
ec2_vpc_igw:
vpc_id: "{{vpc_out.vpc.id}}"
state: "{{state}}"
region: "{{region}}"
resource_tags:
Name: vprofile-IGW
register: igw_out
Now Internet Gateway has been created:
Playbook for Route Table
As we have already created Public Subnets and Internet Gateway for the same, now we'll be creating Route Table which will tell the public subnet to go and access this internet gateway and vice-versa. All the traffic coming to and from the Internet Gateway will be routed through this route table.
This route table will be mapped to the public subnets so that it can route the traffic to the Internet gateway and the traffic from the Internet gateway will be routed back to the subnets.
- name: Creating Route Table
ec2_vpc_route_table:
vpc_id: "{{vpc_out.vpc.id}}"
region: "{{region}}"
tags:
Name: VProfile-PubRT
subnets:
- "{{pubsub1_out.subnet.id}}"
- "{{pubsub2_out.subnet.id}}"
- "{{pubsub3_out.subnet.id}}"
routes:
- dest: 0.0.0.0/0
gateway_id: "{{igw_out.gateway_id}}"
register: pub_RT_out
Route Table has been created and associated with the public subnets
Creating NAT Gateway
We have created 3 private subnets and now we aim to create NAT gateways for those private subnets.
NAT gateway lives in a public subnet however, we have 3 private subnets so ideally we have to create 3 NAT gateways but to save cost we'll create 1 NAT gateway in any one of the public subnets.
- name: Creating NAT Gateway and allocate elastic IP if a nat gateway doesnt yet exist in the
ec2_vpc_nat_gateway:
state: "{{state}}"
subnet_id: "{{pubsub1_out.subnet.id}}"
wait: yes
region: "{{region}}"
if_exist_do_not_create: true
register: nat_out
- name: Printing NAT Gateway Details
debug:
var: nat_out
Now NAT gateway is ready.
Creating Route Table for Private Subnet
As we have created NAT Gateway for private subnets now it needs a routing table that will route the traffic to and from the sunset and NAT gateway.
- name: Set up route table for private subnets
ec2_vpc_route_table:
vpc_id: "{{vpc_out.vpc.id}}"
region: "{{region}}"
tags:
Name: VProfile-PrivRT
subnets:
- "{{privsub1_out.subnet.id}}"
- "{{privsub2_out.subnet.id}}"
- "{{privsub3_out.subnet.id}}"
routes:
- dest: 0.0.0.0/0
gateway_id: "{{nat_out.nat_gateway_id}}"
register: privRT_out
- name: Debug RoutTable output
debug:
var: privRT_out
And now the route table has been created and associated with private subnets redirecting traffic to the NAT gateway.
Retrieve all the Required Ids in a Variable File
Now, we'll store all the IDs of VPCs, Route Tables, Internet Gateway, and NAT Gateway for further use in setting up Bastion Host.
Print IDs on terminal
We can use debug module for this operation with looping as below:
- name: Print all the IDs
debug:
var: "{{item}}"
loop:
- vpc_out.vpc.id
- pubsub1_out.subnet.id
- pubsub2_out.subnet.id
- pubsub3_out.subnet.id
- privsub1_out.subnet.id
- privsub2_out.subnet.id
- privsub3_out.subnet.id
- igw_out.gateway_id
- pub_RT_out.route_table.id
- nat_out.nat_gateway_id
- privRT_out.route_table.id
Now we can see all the ids are printed by looping each variable:
Using IDs at run time
Instead of printing, we can also store the id values in variables and use them at the runtime going forward in the same playbook.
To do so, we'll be using the set_fact module that allows you to dynamically create or update variables based on the results of tasks or other data sources. This is particularly useful when you need to store and use intermediate values or dynamically generated information throughout your playbook.
- name: set facts
set_fact:
vpcid: "{{vpc_out.vpc.id}}"
pubsub1id: "{{pubsub1_out.subnet.id}}"
pubsub2id: "{{pubsub2_out.subnet.id}}"
pubsub3id: "{{pubsub3_out.subnet.id}}"
privsub1id: "{{privsub1_out.subnet.id}}"
privsub2id: "{{privsub2_out.subnet.id}}"
privsub3id: "{{privsub3_out.subnet.id}}"
igwid: "{{igw_out.gateway_id}}"
pubRTid: "{{pub_RT_out.route_table.id}}"
natgwid: "{{nat_out.nat_gateway_id}}"
privRTid: "{{privRT_out.route_table.id}}"
cacheable: yes
Storing variables in an external file
We can also store all the variables in an external variable text file using the copy module so that we can use it in other playbooks instead of limiting it to its native playbook.
We'll use this variable file while creating the Bastion host.
- name: Copying variable to a file for further use
copy:
content: "vpcid: {{vpc_out.vpc.id}}\npubsub1id: {{pubsub1_out.subnet.id}}\npubsub2id: {{pubsub2_out.subnet.id}}\npubsub3id: {{pubsub3_out.subnet.id}}\nprivsub1id: {{privsub1_out.subnet.id}}\nprivsub2id: {{privsub2_out.subnet.id}}\nprivsub3id: {{privsub3_out.subnet.id}}\nigwid: {{igw_out.gateway_id}}\npubRTid: {{pub_RT_out.route_table.id}}\nnatgwid: {{nat_out.nat_gateway_id}}\nprivRTid: {{privRT_out.route_table.id}}\n"
dest: Variables/output_vars
Bastion Host
A bastion host, also known as a jump host or jump server, is a specialized computer on a network that is specifically designed and configured to provide a secure gateway for accessing other systems in a private network. It acts as an intermediary or a point of entry for administrators to securely access servers and devices within a private network from an external or less trusted network, such as the Internet.
The primary purpose of a bastion host is to enhance security by minimizing direct access to critical servers and systems in the private network. Instead of exposing every server to the public internet, administrators connect to the bastion host first and then use it as a gateway to access other resources within the private network.
Setting up Bastion Host
---
- name: Bastion setup
hosts: localhost
connection: local
gather_facts: no
tasks:
- name: bastion setup variables
include_vars: Variables/bastion_setup
- name: import vpc setup variable
include_vars: Variables/output_vars
- name: create vpro ec2 key pair
ec2_key:
name: vpro_key
region: "{{region}}"
register: vprokey_out
- debug:
var: vprokey_out
- name: save private key into file named bastion_key.pem
copy:
content: "{{vprokey_out.key.private_key}}"
dest: "./bastion_key.pem"
mode: 0600
when: vprokey_out.changed
- name: Create Security group for bastion host
ec2_group:
name: bastion_host_SG
description: Alow port 22 from everywhere and all port within SG
region: "{{region}}"
vpc_id: "{{vpcid}}"
rules:
- proto: tcp
from_port: 22
to_port: 22
cidr_ip: "{{MYIP}}"
register: BastionSG_out
- name: creating EC2 instance
ec2:
key_name: vpro_key
region: "{{region}}"
instance_type: t2.micro
image: "{{bastion_ami}}"
wait: yes
wait_timeout: 300
instance_tags:
Name: "Bastion Host"
Project: Vpro-WebApp
Owner: Ritesh
exact_count: 1
count_tag:
Name: "Bastion Host"
Project: Vpro-WebApp
Owner: Ritesh
group_id: "{{BastionSG_out.group_id}}"
vpc_subnet_id: "{{pubsub1id}}"
register: bastionhost_out
Now with the above playbook, we have successfully created a jump server/bastion server
And also you can see it is associated with the respective subnet and VPC :
