Let’s build cost effective AWS architecture

Scenario:
We had 43 static databases (no more data changes) running on EC2 instances and having size of around 20TB each, however those databases were required to query only once in a month. Running EC2 instances around the clock with those large EBS volumes does not make sense in terms of cost efficiencies so we thought of doing some cost optimization.

Cost Considerations:
We took the AMIs of those EC2 machines and whenever application team needs to to query the databases, those can be restored from AMIs. This way we need to pay only AMIs snapshot cost for most of the time and then EC2 compute and EBS volumes costs whenever databases will be restored from AMIs for query purpose. As total run time for these EC2 machines is very less in a year, there is no need to buy any reserve instance.

Challenges:
We had kept the EC2 machines in the Private subnet to make it more secure.
Whenever EC2 instances are launched from AMI, they get a new Private DNS/IP and that changes the database connection properties (For ex — tnsentry in case of Oracle).

Solution:
Here we need some kind of fixed dns name for each db host which can point to the new private ip dynamically whenever they are launched from AMI. Our initial thought was that python script will launch the EC2 instances (we created boto3 script to automate the launching of ec2 machines from AMIs) and log all the DB names and private IP details in a log file. Whenever an application needs to query the database, they need to check the log file to look for new Private IP corresponding to a database name. Later we came up, to have a fix DNS using Route53 — Private Hosted Zone.

Solution Workflow:

Architecture

• User logs in a bastion host which is in public subnet and runs the python script. This script launches those 43 EC2 instances from AMIs in the private subnet. This script is based on AWS boto3 module which interacts with AWS endpoints to launch infrastructure. IAM role is attached to the bastion host to have sufficient permissions to do all these infrastructure buildup activities.
• This script will also configure cloudwatch alarms on ec2 machines. These alarms are configured that if no one query the databases (CPU is less than 2% for 24 hours) then ec2 hosts will get terminated.
• Once ec2 comes in running state, it emits cloudwatch event and that event triggers lambda function for each ec2 instance.
• Lambda function gets the private dns of ec2 host and checks the tag on the host to know the database name (AMIs were already tagged with respective DB names).
• Lambda insert/update the private hosted zone record set respectively.

For ex — We can access TEST1 DB host using following alias.
test1.dbsvr.db points to<<TEST1 instance private DNS>>

code snippet

Closing Notes:
There were legal reasons due to which we could not migrate data to other storage. Of course there would be better options to make this more cost effective and simpler.

• Amazon Aurora Serverless — With this option AWS charges for the duration when database is active and we don’t need to pay compute charges if there is no query on the database.
• S3 and Athena — This would be most cost effective way for such a use case where data is static and query capability needed once in a while. In this case, we will reduce the storage cost significantly as S3 is cheaper and then pay Athena data scanned charges.
• Native Logical backups on S3 — This will reduce the storage cost but downside is that restoring those big databases from logical backup will be time taking process.

Contributors: Virender Singla, Rahul Pawar