Amazon CloudWatch is a monitoring service that provides users with insights and other data related to their infrastructure and applications. With these insights, users can then optimize the performance of Amazon Web Services (AWS) applications, manage resources, and monitor the health of their system.
Aggregated logs, metrics, and events are the basis for these CloudWatch insights. Logs are created by everything from your operating systems to the applications running on your AWS instance. As your workload grows, so will your number of log files. When you have more log files stored in your AWS instance than you need, then you have a chance to optimize your cloud costs.
In this guide, we’ll briefly talk about CloudWatch logs and log groups, and then show how you can ensure that log groups have retention periods set to limit AWS costs.
CloudWatch Logs and Log Groups
CloudWatch logs let you use one service to centralize logs from all systems and AWS services. That makes it easier to:
- View logs
- Find specific patterns and error codes
- Filter specific fields
- Archive logs for future use
For example, you can use CloudWatch logs to keep up with errors found in application logs of your Amazon EC2 instances. Specify a threshold for an acceptable error rate and receive a notification once the number of errors passes that boundary. Because CloudWatch logs use existing log data, you can configure this without needing to make any code changes.
CloudWatch Log Groups
All logs are presented as a consistent event flow ordered by the time they occur. You can query and sort CloudWatch logs, group them based on certain fields, use query language to set up customized computations, and present log data within a dashboard.
Log groups are collections of log streams, which are individual sequences of log events from a shared source. Each log group shares access, monitoring, and retention control settings. Users can define log groups and control how to assign streams to a log group.
Finding Log Groups Without a Set Retention Period
You can get the details of an Amazon CloudWatch Log Group to verify it has a retention period set. To do this, run the following command with AWS CLI:
CloudWatch log groups do not have a retention period set by default, so log events are retained forever. When the retention period is set, the log group details contain the attribute "retentionInDays". You can see an example of this output below :
Using jq with AWS CLI you can get the list, for your current region, of CloudWatch log groups without retention period with the following command:
Once you have found the groups that are missing retention periods, next you need to add a retention policy.
Setting Retention Periods for CloudWatch Log Groups
You can set a retention policy that specifies the number of days to retain log events within a log group by using the following syntax in the AWS CLI:
Once you have run this command, you can again run the “describe-log-groups” command to verify that this change has taken place. This small change can instantly start saving your organization money by capping log storage.
Automating this AWS CloudWatch Check with Blink
If you are running AWS CloudWatch, checks like this are important to keep your logs from becoming too costly. While you can run optimizations ad hoc, there’s a better way to get a handle on your AWS costs.
When you create a free Blink account, you can run scheduled, pre-built automations to locate unneeded resources and comply with best practices.
Get started and create your free Blink account today.