S3 File Consolidation

Opportunity Name:

S3 File Stitching Finder

AWS Resource Type:

Amazon S3

Opportunity Description:

Amazon S3 buckets often store numerous small files, leading to significant costs associated with frequent requests, especially for use cases such as log storage for Athena queries. By consolidating these small files into larger, logically grouped files, this Finder identifies opportunities to reduce the number of billable requests and optimize S3 costs while maintaining data accessibility and integrity.

This approach can lead to substantial cost savings by reducing the operational expenses of handling a high volume of requests.

Criteria for Identifying the Opportunity:

1. Bucket Identification:
   • Analyze S3 usage patterns, including the number of objects, average object size, and request counts.
2. High Request Cost:
   • Buckets where the cost of "GetObject" requests significantly exceeds storage costs (e.g., request costs are 10x storage costs).
3. Validation:
   • ResourceExist: Validate the presence of relevant S3 buckets.
   • BucketUsedWithGlue: Check for AWS Glue table associations with the bucket.
4. Savings Potential:
   • Estimate savings based on simulated impacts of file consolidation.

Potential Savings:

Up to 90% of "GetObject" request costs can be saved for eligible buckets by consolidating files.

What Happens When the Fixer is Executed?

CloudFix provides recommendations for consolidating small files within identified buckets. Recommendations include a detailed report on eligible buckets, potential savings, and a URL link to the S3 console for easy implementation. File consolidation itself must be performed manually or through external tools such as AWS Glue.

Is It Possible to Roll Back Once CloudFix Implements the Fixer?

Rollback is not applicable as the feature only identifies opportunities and provides recommendations. Users control the implementation of file consolidation.

Can CloudFix Implement the Fix Automatically Once I Accept the Recommendation?

No, this feature is recommendation-only. Implementation requires manual execution by the user or an external automated process.

Does the Fix Require Downtime?

No downtime is required for identifying and recommending file consolidation.

Additional Resources:

• Amazon S3 Documentation
• AWS Glue Documentation