Deploy Jenkins agents using EC2 Spot instances

Inspired by Lyft’s Solution, we are leveraging a plugin offered by Jenkins and AWS Auto Scaling service to meet the demand of the jobs served by the Jenkins master node.

What are we aiming for in this POC?

  • We are able to execute jobs in the spot instances
  • It scales instances as needed
  • Prove that Spot instances are helping to cut costs
  • We have confidence to reduce the instance type of master node

Technologies that we will be working with are

High level on how these components will be interacting

How AWS Architecture looks like

Steps to Create this POC

Setup Jenkins Master Node

  1. Inside your AWS Console, click on EC2 Service
  2. Then click on Launch Instances
  3. Select Amazon Linux 2 AMI 64-bit (x86)
  4. Choose t2.micro, Click Next on Configure Instance Details
  5. Choose the VPC you wish to deploy this Jenkins instance. Make sure your VPC has DNS hostnames and DNS resolution enabled
  6. Choose a public subnet inside your VPC that has a route to the internet gateway.
  7. Enable Auto-Assign Public IP
  8. In the user data section — paste the following:
#!/bin/bash
wget -q -O — https://pkg.jenkins.io/debian-stable/jenkins.io.ke
sudo sh -c ‘echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list’sudo yum update -y
sudo yum install java-1.8.0-openjdk -y
sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
sudo rpm — import https://pkg.jenkins.io/redhat/jenkins.io.key
sudo amazon-linux-extras install epel -y
sudo yum install jenkins -y
sudo service jenkins start
  1. Leave others as default, click on Next: Add Storage
  2. Leave storage default
  3. Add any tags you like, Click on Next: Add Tags
  4. Click on Next Configure Security group
  5. Click on Review and Launch
  6. Click on Launch after reviewing your information
  7. Create a key pair that you will use to connect to the instance, select the acknowledge
  8. Click on Launch Instances

Setup Jenkins

  1. After the EC2 instance is finished with initialization and all status checks have been passed, get the public IP address of the instance and open it in a browser in a format like this 3.139.62.228:8080 with a port number of 8080. You should be able to see Jenkins welcome page.
  1. To retrieve this password use this command in your instance:

3. sudo cat /var/lib/jenkins/secrets/initialAdminPassword

sudo cat /var/lib/jenkins/secrets/initialAdminPassword
  1. Copy and paste the output to the Jenkins page, click on continue
  2. Choose Install suggested plugins. Wait till everything is installed
  3. In the “Create First Admin User” page, create your user that will be used to login to the Jenkins console in the future. After, click on save and continue
  4. In the Instance Configuration page, acknowledge and leave the defaults. Click on save and finish.
  5. Click on “Start using Jenkins”

Create an AWS IAM user for EC2-Fleet Plugin

  1. In your AWS console, head to IAM service to create a new user. This user will be used by the EC2-Fleet plugin (which we will install later) that install spot instances for us.
  2. In the IAM service, click on Users on the left panel and click on “Add Users” on the right side.
  3. Type in a username that makes most sense, for example (Ec2-fleet-user).
  4. In the AWS Access type section, highlight the Access Key — Programmatic Access
  5. Click Next on permissions
  6. In the Set Permissions Section, click on “Attach existing policies directly” and click on “Create Policy” to manually create a policy.
  7. Switch to json view and paste the following permissions

8. {

9. “Version”: “2012–10–17”,

10. “Statement”: [

11. {

12. “Action”: [

13. “ec2:*”,

14. “autoscaling:*”,

15. “iam:ListInstanceProfiles”,

16. “iam:ListRoles”,

17. “iam:PassRole”

18. ],

19. “Resource”: “*”,

20. “Effect”: “Allow”

21. }

22. ]

23.}

  1. Click Next on Tags
  2. Click Next on Review
  3. Name the policy that makes most sense, for example (ec2-fleet-permissions-policy)
  4. Click on Create Policy
  5. Head back the tab where you were creating the role. Click on the refresh button, click on “Filter Policies” and choose customer managed.
  6. Choose your policy that you created earlier. If you cant find it, refresh again. Click on “Next: Tags”
  7. Click on Next: Review
  8. Click on Create User
  9. Download the .csv file for later use. Click on Close

Create a launch template and auto scaling group in AWS Console

  1. A launch template includes the information about the instances that auto scaling group need to launch
  2. Head to the EC2 service inside your AWS Console. Click on Launch templates on the left panel. Click on Create launch template
  3. Name the Launch Template for example (EC2-fleet-launch-template)
  4. In the Launch Template Contents section, choose the amzn2-ami-hvm-2.0…x86
  5. Choose t3.small for Instance type
  6. Choose an existing key pair or create a new key pair that will be used to connect to these spot instances
  7. In the network settings, choose VPC and same security group as your Jenkins Server for now
  8. Expand Advanced Details, choose “Request Spot Instances”. Inside the user data, paste the following

9. #!/bin/bash -xe

10.sudo amazon-linux-extras install epel -y

11.sudo yum update -y

12.sudo yum remove java-1.7.0-openjdk

13.sudo yum install java-1.8.0-openjdk -y

14.sudo yum install git -y

15.sudo yum -y update aws-cli

  1. Click on Create Launch template
  2. Next inside the same EC2 service, click on Auto Scaling Groups on the left panel
  3. Click on Auto Scaling Group
  4. Name the Auto Scaling Group — For example (EC2-Fleet-AutoScaling-group)
  5. In the Launch template section — choose the template you have created earlier. Click Next
  6. Choose the VPC where the ec2 instances will be launched
  7. Choose all the private subnets as the availability zones. If your vpc doesnt have atleast 3 subnets, create them and come back to this menu
  8. Click on Next
  9. We will skip the load balancer for now, click on next
  10. Group Size
  11. Desired capacity = 1
  12. Minimum capacity = 1
  13. Maximum Capacity = 4
  14. Skip the scaling policies for now
  15. Click on Next
  16. Skip Notifications for now, click on Next
  17. Skip Tags, Click on Next
  18. Click on Create Auto Scaling Group
  19. You should see one of the instances already spinning up

Install EC2-Fleet Plugin

  1. Head to Jenkins Dashboard. Click on Manage Jenkins on the left panel
  2. Click on Manage Plugins
  3. Choose Available and search for ec2-fleet
  4. Select the plugin and click on install without restart
  5. Click on restart jenkins after installation is complete and no jobs are running
  6. Log back in with your credentials

Setup EC2-Fleet Plugin

  1. Go to Manage Jenkins → Manage Nodes and Clouds → Configure Clouds
  2. Click on the drop down menu “Add a new Cloud”, select Amazon EC2 Fleet
  3. In the AWS Credentials, click on Add drop down menu
  4. Change Kind to AWS Credentials
  5. Change scope to System (Jenkins and nodes only)
  6. Add the ID that specifies the username that holds these credentials
  7. Add the Access Key ID and Secret Access from the csv you have downloaded
  8. Click on Add
  9. Now click on the drop down for AWS Credentials and choose the credentials you have added
  10. Change region to your respective one
  11. EC2 Fleet section should be auto populated with the auto scaling group you have created in your AWS Account
  12. Click on Test Connection
  13. In the Launcher Section
  14. Change Launcher drop down to “Launch Agents via SSH”
  15. Click on “Add” for Credentials
  16. Change Kind to SSH Username with Private Key
  17. Change scope to System (Jenkins and nodes only)
  18. Add “ec2-user” to ID and Username
  19. For Private Key, click on Enter Directly
  20. Paste the contents of your .pem file that was used to create the main jenkins server
  21. Click on Add
  22. Choose the new credentials you created
  23. For Host Key Verification Strategy: Choose Non verifying verification strategy
  24. Choose Private IP option to be accessed internally
  25. Change label to “spot-agents”
  26. Add the following parameters:
  27. Number of executors: 1
  28. Max Idle Minutes before scaledown: 5
  29. Minimum Cluster Size: 1
  30. Maximum Cluster Size: 5
  31. Maximum Total Uses: -1 (default)
  32. Leave the rest default. Click on Save

Add a sample Jenkins Build

  1. In the Jenkins Dashboard, click on New Item on the left panel
  2. Enter an item name “sai-test-build”, choose freestyle project. Click on Ok
  3. Choose Git
  4. Add this repository url: https://github.com/psdilip/python.git
  5. Inside the Build Section — choose Execute Shell and add the following command

6. python3 basics.py

  1. Click on Save

Run the Sample Jenkins Build

  1. In the Jenkins Dashboard, verify that a fleetcloud instance is running inside the “Build Executor Status”
  2. Click on the play button on your sample project to start the build
  3. View the output by clicking the project, clicking on the latest build history and on console output

--

--

--

DevOps Engineer

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Get Historical Forex Rates With A Restful API

Data science… without any data?!

Ink Complete

The Fellowship Whitepaper

Fellowship Logo

How To Do Logging in Python

Not awful outages: Transparency about what’s happening and why

Scalability and Cost Efficiency with AWS

Eleven practical tricks to write Clean Code — Part 1

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sai Dilip Ponnaganti

Sai Dilip Ponnaganti

DevOps Engineer

More from Medium

Using AWS CLI to Create, Users, Groups and S3 Buckets

Docker containers using Ansible on AWS

Getting started with AWS ECR

Using Docker with Ubuntu Via AWS