Managing cloud infrastructure efficiently often involves automating routine tasks to reduce costs and improve operational efficiency. One common task is ensuring that idle or non-essential instances are shut down during off-hours, such as midnight, to save on costs. Automating this task not only saves time but also ensures consistency and accuracy. In this lab, we will explore how to leverage AWS EventBridge Scheduler, AWS Lambda, and Amazon SNS to automatically shut down instances at midnight and send notifications upon successful shutdown. This powerful combination provides a seamless and cost-effective solution for managing your AWS resources.

Introduction

Imagine a world where your cloud infrastructure is smart enough to automatically power down idle instances when they are not needed, and seamlessly power them back up when required. This not only reduces operational costs but also ensures optimal resource utilization. AWS offers robust tools like EventBridge Scheduler, AWS Lambda, and Amazon SNS to help you achieve this automation effortlessly. In this article, we will guide you through setting up an architecture that uses EventBridge Scheduler to trigger a Lambda function at midnight, which will then shut down specified EC2 instances and send notifications upon successful shutdown. This automated approach ensures your instances are not running unnecessarily, saving you money and optimizing your cloud environment.

Before proceeding with this project, you might want to read this comprehensive overview of Amazon Eventbridge.

Overview of the Solution

The architecture we will implement involves the following components:

1. AWS EventBridge Scheduler: To schedule the Lambda function to run at midnight every day.
2. AWS Lambda: To execute the shutdown commands on specified EC2 instances.
3. Amazon SNS: To send notifications upon successful shutdown of instances.
4. AWS Identity and Access Management (IAM): To provide the necessary permissions for the Lambda function to perform the shutdown operation and send SNS notifications.

Key Components

• EventBridge Scheduler: EventBridge Scheduler allows you to set up cron jobs or periodic tasks that trigger based on a predefined schedule. We will configure it to trigger a Lambda function at midnight daily.
• Lambda Function: The Lambda function will contain the logic to identify and shut down the specified EC2 instances. This serverless compute service ensures that the function runs only when needed, optimizing resource usage and cost.
• Amazon SNS: SNS will be used to send notifications to subscribers upon successful shutdown of the instances. This ensures that you are promptly informed of the status of your instances.
• IAM Roles and Policies: Proper IAM roles and policies will ensure that the Lambda function has the necessary permissions to stop EC2 instances and publish messages to SNS securely and efficiently.

Why Use AWS EventBridge Scheduler, Lambda, and SNS?

Automated Scheduling with Precision

AWS EventBridge Scheduler provides a highly precise and reliable scheduling mechanism. It allows you to define cron expressions for triggering actions at specific times, such as midnight, ensuring that your tasks are executed exactly when you need them.

Simplified Management

By using EventBridge Scheduler, you centralize the management of scheduled tasks. This simplifies the process of monitoring and adjusting schedules without modifying the underlying infrastructure or application code.

Seamless Integration with AWS Services

EventBridge Scheduler integrates seamlessly with a wide array of AWS services, including AWS Lambda and SNS. This integration allows for building complex workflows that respond to scheduled events, making it an ideal choice for tasks like shutting down EC2 instances and sending notifications.

Cost Efficiency

Automating instance shutdowns with EventBridge Scheduler, Lambda, and SNS can significantly reduce costs. Instances that are not needed during off-hours can be automatically stopped, ensuring that you only pay for the resources when they are actually in use. Additionally, receiving notifications ensures you are always aware of your infrastructure's status.

Implementation Steps

1. Create the Lambda Function

The Lambda function will be responsible for stopping the EC2 instances and sending a notification via SNS. Here’s a sample Lambda function written in Python:

import json
import boto3
import logging
from botocore.exceptions import ClientError

ec2 = boto3.client('ec2')
logger = logging.getLogger()
logger.setLevel(logging.INFO)
send_sns = False

def lambda_handler(event, context):
    global send_sns
    logger.info('Function {} has started execution.'.format(context.function_name))
    # Collect Instance ids for running instances
    response = ec2.describe_instances()
    ids = []
    for i in response['Reservations']:
        if i['Instances'][0]['State']['Name'] == 'running':
            ids.append(i['Instances'][0]['InstanceId'])
    
    # Stop running instances
    if ids:
      try:
        ec2.stop_instances(InstanceIds=ids, DryRun=False)
        logger.info('Function {} has successfully executed.'.format(context.function_name))
        send_sns = True
      except ClientError as e:
        logger.info('Function {} has failed.'.format(context.function_name))
        print(e)
    else:
        logger.info('function {} could not find any running instances.'.format(context.function_name))
    
    # Send SNS Email 
    if send_sns:
        sns = boto3.client('sns')
        arn = 'arn:aws:sns:us-east-1:123456789012:InstanceShutdownTopic'
        notification = f'{len(ids)} of your EC2 instances with the following Instance ids have been successfully shutdown: {ids}'
        
        response = sns.publish(
            TargetArn = arn,
            Message = json.dumps({'default': notification}),
            MessageStructure = 'json'
            )
        
        return {
            'statusCode': 200,
            'body': json.dumps(response)
            
        }
        

2. Set Up IAM Role and Policy

Create an IAM role with a policy that allows the Lambda function to stop EC2 instances and publish to SNS.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:StopInstances",
        "ec2:DescribeInstances",
        "sns:Publish"
      ],
      "Resource": "*"
    }
  ]
}

Attach this policy to the Lambda execution role.

3. Create an SNS Topic

Create an SNS topic for notifications and subscribe your email to the topic.

aws sns create-topic --name InstanceShutdownTopic

aws sns subscribe --topic-arn arn:aws:sns:us-east-1:123456789012:InstanceShutdownTopic --protocol email --notification-endpoint your-email@example.com

4. Configure EventBridge Scheduler

Set up EventBridge Scheduler to trigger the Lambda function at midnight (UTC).

Using AWS Management Console

1. Open the EventBridge Console:
   • Navigate to the EventBridge console.
2. Create a Rule:
   • Click on "Rules" in the left-hand menu and then "Create rule."
3. Define Rule Details:
   • Enter a name and description for the rule.
   • Set the event source to "Schedule."
4. Define Schedule Pattern:
   • Choose "Cron expression" and enter the cron expression for midnight UTC (0 0 * * ? *).
5. Add Target:
   • In the "Targets" section, choose "Lambda function."
   • Select the Lambda function you created for shutting down the instances.
6. Create Rule:
   • Review the settings and create the rule.

Using AWS CLI:

aws events put-rule \
    --name ShutdownInstancesAtMidnight \
    --schedule-expression "cron(0 0 * * ? *)" \
    --description "Trigger Lambda to shut down EC2 instances at midnight"

5. Test the Setup

Ensure your Lambda function and EventBridge rule are correctly configured. You can manually invoke the Lambda function to verify it stops the specified instances and sends an SNS notification. Check your email to confirm you received the notification.

Aligning with the AWS Well-Architected Framework

The AWS Well-Architected Framework helps cloud architects build secure, high-performing, resilient, and efficient infrastructure for their applications. Our automated instance shutdown solution aligns with several principles of this framework:

Operational Excellence

• Automate Manual Processes: Using EventBridge Scheduler to automate the shutdown of EC2 instances reduces the need for manual intervention, ensuring consistency and reliability.
• Improve Operational Processes: Logging and monitoring via CloudWatch Logs and SNS notifications enhance visibility and provide insights into the automated process.

Security

• Apply the Principle of Least Privilege: The IAM role and policies for the Lambda function are configured to grant only the necessary permissions to perform the shutdown and send notifications, minimizing security risks.
• Manage Authentication and Authorization: IAM roles and policies are used to securely manage access to AWS resources.

Reliability

• Automated Recovery from Failure: By using retries and error handling in the Lambda function, the solution ensures that instances are reliably shut down even in the face of transient errors.
• Monitoring and Notifications: CloudWatch and SNS provide real-time monitoring and notifications, allowing for quick response to any issues that arise.

Performance Efficiency

• Use Serverless Architectures: Leveraging Lambda for the shutdown process ensures that the solution scales automatically with demand, with no need for provisioning or managing servers.
• Optimize Resource Utilization: Automatically shutting down instances during off-hours optimizes the use of AWS resources, reducing waste and costs.

Cost Optimization

• Implement Cost-Awareness: The solution ensures that instances are only running when needed, reducing unnecessary spend on idle resources.
• Right-Sizing Resources: By automating shutdowns, you ensure that resources are appropriately scaled according to usage patterns.

Best Practices for Using EventBridge, Lambda, and SNS

Monitoring and Logging

Ensure that your Lambda function has adequate logging. Use AWS CloudWatch Logs to monitor the execution of your Lambda function. This helps in troubleshooting any issues that arise during the execution.

Security

Implement the principle of least privilege for your IAM roles and policies. Ensure that your Lambda function has only the necessary permissions to stop the EC2 instances and publish to SNS.

Testing

Thoroughly test your setup in a development environment before deploying it to production. This includes testing the Lambda function, the EventBridge rule, and the IAM policies.

Handling Failures

Consider adding error handling and retry logic in your Lambda function to handle any failures during the shutdown process. This ensures that transient issues do not prevent your instances from being stopped and notifications from being sent.

A Final note

By leveraging AWS EventBridge Scheduler, Lambda, and SNS, you can automate the shutdown of EC2 instances at midnight and receive notifications upon successful shutdown, ensuring that your infrastructure is optimized and cost-effective. This solution demonstrates the power and flexibility of AWS's serverless and event-driven services, enabling you to create robust automation workflows with minimal effort. Start implementing this architecture today to streamline your operations, reduce unnecessary cloud costs, and stay informed about your infrastructure status.