The cloud offers scalability and agility, but managing it requires constant vigilance. Logs and metrics are essential for monitoring cloud applications, providing insights into performance, security, and potential issues.

The Challenge: Complexity & Misinterpretation

Many organizations struggle to effectively collect, analyze, and interpret log and metric data. Traditional monitoring tools often lack the scalability and flexibility needed for cloud-based environments. Additionally, differentiating between critical events and harmless informational messages can be tricky.
 

Log Analytics is the process of ingesting streaming log data, aggregating that data, and persisting the aggregated data so that it can be analyzed and visualized.
It is a common big data use case that allows you to analyze log data from websites, mobile devices, servers, sensors, and more for a wide variety of applications such as digital marketing, application monitoring, fraud detection, ad tech, games, and IoT.

Log analytics is the process of collecting, analyzing, and visualizing data generated by IT systems and applications. This data, stored as log files, provides valuable insights into system performance, security, and overall health. By analyzing logs, IT professionals can identify trends, troubleshoot issues, and gain a deeper understanding of how their systems are operating.

Metrics on the other hand, are measures of quantitative assessment commonly used for comparing, and tracking performance or production.

Metrics aggregation in AWS refers to the process of collecting and summarizing data points from various AWS resources into a more manageable format for analysis. This involves consolidating individual data points over a specific time period, allowing you to identify trends, monitor performance, and troubleshoot issues.
CloudWatch is the primary service for metrics aggregation in AWS. It collects data from a wide range of AWS services, including EC2 instances, S3 buckets, and DynamoDB tables. CloudWatch enables you to:

• Visualise Metrics: Plot metrics over time in graphs to identify trends and patterns.
• Set alarms: Configure alarms to notify you when metrics exceed specific thresholds.
• Perform statistical analysis: Calculate statistics like average, sum, and minimum/maximum values for your metrics.
  Metrics aggregation helps you gain insights into the health and performance of your AWS resources. By analyzing aggregated metrics, you can optimize resource utilization, identify potential bottlenecks, and ensure your applications are running smoothly.

In this project, we will create a complete end-to-end metrics collection system that integrates several AWS services.

Our log analytics architecture is similar to the one above from AWS docs.

We have a 3-tier application deployed in our VPC. The web servers are Amazon Elastic Compute Cloud (Amazon EC2) instances deployed in an auto-scaling group.

Here is how it works:

1. The CloudWatch agent is automatically installed on the application servers.

2. The CloudWatch agent will collect logs and metrics and deliver them to CloudWatch.

3. CloudWatch Metrics continuously forwards Metrics to an Amazon Kinesis Data Firehose delivery stream.

4. Amazon Kinesis Data Firehose writes each metric record to Amazon Simple Storage Service (Amazon S3) for durable storage of the raw log data.

Here are the AWS services involved in this solution:

• CloudWatch

• Kinesis Data Firehose

• Simple Storage Service (S3)

It is important to note that while our application is hosted on EC2 instances, our log analytics solution is completely serverless.

Step 1: Setup The 3-tier Application

I have already designed a 3-tier appplication ready to be deployed to AWS using terraform. To learn more about 3-tier applications and how you can design yours, read this article.

The terraform code used to automatically deploy this architecture is on my GitHub page, so to set this up, all we have to do is clone this GitHub repository.
Once you have cloned it, navigate to the terraform directory and run terrform init, then run terraform plan. After that, run terraform apply -auto-approve.
Note: You must configure your own backend in the backend.tf file.

Terraform Init

Terraform Plan

Terraform Apply

Our instance launch template already has cloudwatch agent instaled and running. The instance role has also been modified to include the "CloudWatchAgentServerPolicy" permissions.

Note that it might take around 10 to 15 minutes for terraform to be fully provision the resources and an additional 5 minutes after provisioning is complete before the site becomes active.

The load alancer DNS name will be appear on the terminal after the terraform apply successfully runs.
Visit the load balancer DNS name on your web browser to view the web application.

Once you register/login, you should see something like this:

Step 2: Create A Kinesis Firehose Delivery Stream

Here, we will be creating a kinesis Firehose Delivery stream and configure Cloudwatch to continually send our analytics data into this stream. CloudWatch is the data producer that will send our data into Firehose delivery streams. To learn more about kinesis, checkout my blog post on Getting Kinetic With Amazon Kinesis.

• On the kinesis Data Firehose console, click on "Create delivery stream".

• Under "Source", select "Direct PUT".

• Under "Destination", select "S3".

• Under "Destination settings", click on "Create bucket".

• Make sure your bucket name is unique.

• Leave every other setting as default and click on "Create bucket".

• Back on the Firehose console, select the bucket you just created.

• You can optionally define an S3 bucket error output prefix.

• Click on "create delivery stream".

Step 3: Create A CloudWatch Metric Stream

Here, we have to configure CloudWatch to stream our aplication metrics to the Firehose delivery stream we created in step 2.

• On the CloudWatch Console, click on "Metrics".

• Click on "Create Metric Stream".

• Choose "Custom setup with firehose", also select the firehose stream we created earlier on.

• Scroll down to 'metrics to be streamed".

• Choose "Select metrics" > "Include".

• Choose the "CWAgent".

Note that "CWAgent" will only be available if you have successfully deployed the 3-tier application in your AWS environment.

• Under "select or type metrics", choose "CWAgent: All metrics names".

• Give your stream a name and click on "Create metric stream".

Now we have created a complete end-to-end solution for aggregation, collection and storage of our application metrics.

Step 4: Monitor Metrics Collection

You will have to periodically monitor metrics collection and the behaviour of both the CloudWatch Stream and Firehose delivery stream.

1. To get insights into the Firehose delivery stream:

2. Visit the Firehose console, go to monitoring to view Firehose metrics. These metrics will give you insight into how your solution is performing.

1. To get insights into the CloudWatch Stream:

2. Visit the CloudWatch Console

3. Go to "Metrics" > "Streams" and select your stream. The dashboard shows you performance insights.

4. 

Finally, you can check your S3 bucket to see the aggregated logs.

Here is mine:

Efficient metrics aggregation Solutions on AWS can simplify your Cloud monitoring journey and make it easy to collate and interprete the behaviour of your systems and infrastructure.

Happy Clouding !!!