Kinesis Optimize Logging

Opportunity Name: Kinesis Optimize Logging

AWS Resource Type: Amazon Kinesis Data Analytics (Managed Service for Apache Flink) application

Opportunity Description:
This Finder identifies Kinesis Data Analytics applications (Apache Flink applications) that are incurring unnecessary CloudWatch costs because their application logging level is set to INFO, the default. At the INFO level, the application writes a much higher volume of log events to CloudWatch Logs than most production workloads need for ongoing monitoring, which drives up CloudWatch ingestion and storage costs without a corresponding increase in operational value.

CloudFix highlights opportunities to reduce these costs by lowering the application's monitoring log level from INFO to ERROR, so CloudWatch only receives log events for actual errors rather than routine informational events.

Criteria for identifying the opportunity:

• The resource is a Kinesis Data Analytics (Apache Flink) application, identified via DescribeApplication.
• The application's monitoring configuration has its log level set to INFO (or another level above ERROR) rather than ERROR.
• The application is not tagged to exclude it from CloudFix recommendations.

Potential Savings (if known):
Savings depend on how much log volume the application generates at the INFO level. CloudFix estimates that error-level logs typically represent a small fraction of total log volume, so lowering the log level to ERROR can reduce the application's CloudWatch logging costs by roughly 95% in most cases.

What happens when the Fixer is executed?
This Finder does not currently have an automatic Fixer. Customers must manually change the application's monitoring log level using the AWS Console or AWS CLI. The process involves:

• Navigating to the Kinesis Data Analytics application in the AWS Console.
• Opening the application's Monitoring configuration.
• Changing the Monitoring log level from INFO to ERROR (this can also be done via the AWS CLI or SDK by updating the application's MonitoringConfiguration).

CloudFix provides the recommended log level change along with the estimated savings for each affected application.

Is it possible to roll back once the fix is applied?
Yes. The monitoring log level can be changed back to INFO (or any other supported level) at any time with no disruption to the application's data processing. This change is fully reversible.

Can CloudFix implement the fix automatically once I accept the recommendation?
No. Customers need to apply the log level change manually or engage CloudFix Professional Services for implementation.

Does this fix require downtime?
No. Changing the application's monitoring log level does not restart or interrupt the running Flink application.

Additional Resources:

• Monitoring Apache Flink Applications on Managed Service for Apache Flink
• Set Up Application Logging in Managed Service for Apache Flink