Fixing AWS Glue Crawler Issues with RDS Exported S3 Data by Excluding _SUCCESS Files

Описание: Learn how to resolve AWS Glue Crawler misidentification of S3 exported RDS data files by excluding _SUCCESS files to prevent incorrect table creation.
---
This video is based on the question https://stackoverflow.com/q/79412004/ asked by the user 'Alex' ( https://stackoverflow.com/u/13083700/ ) and on the answer https://stackoverflow.com/a/79412005/ provided by the user 'Alex' ( https://stackoverflow.com/u/13083700/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: AWS Glue Crawler issue with S3 export from RDS

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to drop me a comment under this video.
---
Understanding the Problem: AWS Glue Crawler Misreading S3 Exported Files

If you use an AWS Glue Crawler to scan an S3 bucket containing exported data from Amazon RDS snapshots, you might run into a frustrating issue: the crawler logs warnings about files not matching schema but doesn't treat them as errors. This typically starts happening suddenly without changes in your code or infrastructure.

How the Pipeline Typically Works

Create a snapshot from your RDS database.

Export this snapshot to an S3 bucket.

Use an AWS Glue Crawler to scan the S3 bucket and create tables in the Glue Data Catalog.

The Unexpected Behavior

Instead of recognizing .parquet files correctly as partitions of a table (e.g., table.name), the crawler may:

Create tables named after individual parquet files, such as part-000-1234.parquet.

Create tables using S3 export success flag files like _SUCCESS with appended IDs (e.g., _success_840193).

This happens because the crawler is interpreting _SUCCESS files as data files, which should be ignored.

Root Cause

AWS updated the Glue Crawler behavior so that it no longer automatically excludes certain control files like _SUCCESS present in S3 export folders. Since these files are not data files, they confuse the crawler's schema detection logic.

The Solution: Explicitly Exclude _SUCCESS Files

Terraform Implementation

Add an exclusions pattern to your s3_target in the Glue crawler configuration to ignore _SUCCESS files:

[[See Video to Reveal this Text or Code Snippet]]

AWS Console Implementation

Navigate to your AWS Glue crawler settings.

In the S3 target section, add /_SUCCESS to the Excluded files list.

Why This Matters

Ensuring _SUCCESS files are excluded prevents Glue from mistakenly creating tables with those filenames.

It maintains accurate metadata and schema discovery for your exported RDS data.

Avoids confusing logs and schema mismatches during crawling.

Summary

When using AWS Glue to crawl S3 buckets from RDS exports, always consider excluding control files like _SUCCESS explicitly. This adjustment resolves silent errors and incorrect table creations caused by changes in AWS Glue's crawler handling of such files.



By applying this change, your Glue crawler will correctly identify table partitions and avoid creating erroneous tables based on non-data files.