AWS Glue Best Practices: How to Optimize ETL Jobs for Performance, Cost, and Scalability

With proper configuration, AWS Glue becomes one of the most efficient ETL managed platforms out there. A properly designed AWS Glue ETL job on the right worker type with bookmarking, output partitioning, and idempotency can cost just a few cents per terabyte of data processed. An ill-configured AWS Glue ETL job can be ten times more expensive, unpredictable, and inconsistent, even if it fails to complete processing the entire dataset.

The difference between the two comes down almost entirely to the engineering decisions made when the job is first designed. This guide covers every one of those decisions, and applying AWS Glue best practices consistently, from the first line of code onward, is what separates reliable pipelines from expensive ones.

AWS Glue Architecture Is the Foundation for Making Good Decisions

The key elements of good AWS Glue architecture design are knowledge about how Glue provisions compute resources and calculates costs, how the Spark execution engine works in terms of Glue worker settings, and what AWS Glue does for you and what you should take care of. All engineers who have issues with AWS Glue jobs treat Glue as a PySpark development environment, which is not true and causes them problems.

The AWS Glue Worker Types and How to Choose the Right Compute Unit

Picking the correct worker type is the single highest-leverage decision in AWS Glue job optimization, because it sets the ceiling on both cost and memory headroom for everything that follows.

G.1X is the right default for most jobs. Move to G.2X only when a job fails with an out-of-memory error or handles wide joins, large broadcast tables, or complex window functions. G.4X and G.8X exist for large machine learning or analytics workloads and are rarely needed for standard ETL. Z.2X Flex is the most cost-efficient option for non-time-critical batch jobs, running 45 percent cheaper than G.2X on spot instances that Glue resumes automatically in most cases. G.025X Serverless Micro suits very small jobs and crawlers, but should be avoided for large data volumes or memory-intensive transformations.

The Glue Billing Model You Must Understand

Glue charges per DPU-second for the duration of the job run, billed in one-second increments with a one-minute minimum. One DPU equals four vCPUs plus sixteen GB of memory, which is one G.1X worker. A job with ten G.1X workers running for five minutes costs 10 DPUs multiplied by 300 seconds multiplied by $0.44 divided by 3600, which comes to $0.37. The same job with unnecessary overhead running for fifty minutes instead of five costs $3.67, a tenfold cost difference for the same output.

Several billing components add up beyond core job execution, and understanding each one is the fastest way to spot waste:

• DPU-seconds for job execution are charged per DPU multiplied by seconds active, with the driver and every executor counting toward the DPU total. Minimise this by right-sizing worker count, minimising job duration through code optimisation, and using G.1X rather than G.2X unless memory genuinely requires it.
• Development endpoints, if used, are charged per DPU-hour while the endpoint is running, and charges accrue even when the endpoint sits idle. Shut endpoints down immediately after use and rely on Glue Studio interactive sessions instead, which cost $0.044 per second with a one-minute minimum.
• Crawler DPU-hours consume DPUs at $0.44 per DPU-hour, and crawlers running on large prefixes with many files become expensive fast. Replace crawlers with Partition Projection or the AddPartition API for regularly partitioned data, and only run crawlers when the schema is likely to have changed.
• AWS Glue Data Catalog storage and requests cost $1 per 100,000 objects plus $1 per million requests after the free tier. Use Partition Projection to eliminate per-query GetPartitions calls, and consolidate small tables into fewer larger ones where it makes sense.
• Glue interactive sessions cost $0.044 per second with a one-minute minimum charged per session start. Use the Glue Studio notebook environment and end sessions the moment active development stops.

Glue 4.0 Is the Runtime You Should Be On

As Glue 4.0 came out in 2023 and is the latest version as of 2026, it now uses Apache Spark 3.3 instead of Spark 2.4 in Glue 2.0; hence, Adaptive Query Execution (AQE) is now the default. It has also updated Python to 3.10 from 3.6, thus giving you access to a more up-to-date standard library and much-improved type hints. According to the benchmarks done by AWS, Glue 4.0 is two to four times faster than Glue 2.0, depending on the nature of the workload, but of course, the exact gains depend on the job. In addition to that, Glue 4.0 comes with lakehouse connector support with native Iceberg and Delta Lake reading and writing capabilities, as well as enhanced JDBC pushdown for Glue Studio visual jobs.

Most PySpark code will remain compatible with Glue 4.0 since the API of the Glue library remains the same. However, every job should be tested on Glue 4.0 before production use since some of the APIs that were deprecated in Spark 2.x have been removed in Spark 3.x. AQE is turned on by default and helps to boost performance, but might affect the execution plan. Migrating to Glue 4.0 is highly recommended. This is based on its performance if you still use Glue 2.0 or 3.0 in production as of 2026. The process of migrating to Glue 4.0 is definitely one of the projects where DevOps consulting services could make all the difference for you.

Job Design Fundamentals and the Architectural Decisions That Determine Everything Else

The most important architectural decisions regarding any of your AWS Glue ETL jobs that go into production come before the very first line of Python code is even written. They include whether to use DataFrames or DynamicFrames, how to do incremental ingestion, how to structure the output, and what idempotency guarantees the job gives. Getting it right means getting everything else right; getting it wrong makes everything after that an uphill battle.

DynamicFrames vs DataFrames and When to Use Each

The production pattern that works well in practice is straightforward. Read data into a DynamicFrame only when the Glue connector requires it or the source data is genuinely schema-inconsistent, convert to a DataFrame immediately with toDF() for all transformation logic, and convert back to a DynamicFrame with fromDF() only if the Glue sink connector requires it.

Job Bookmarking Is the Most Important Correctness Configuration

Job bookmarking is the Glue mechanism that tracks which input data has already been processed by a job, so that subsequent runs process only new data rather than reprocessing everything from the beginning. It is the single most impactful configuration for AWS Glue job bookmarks for both job correctness and cost in any incremental ETL pipeline.

The most common Glue data duplication error happens when job bookmarking is disabled and the job appends to the target using append mode rather than overwrite mode. Every job run then reads all source data and appends it to the output, so running the job daily for a month leaves you with thirty copies of the first day's data. The fix is to either enable job bookmarking so only new data gets processed, or use partition overwrite mode so only the output partitions matching the input data window get replaced. The safe pattern is job-bookmark-enable paired with append mode, or job-bookmark-disable paired with dynamic partition overwrite, meaning spark.sql.sources.partitionOverwriteMode is set to dynamic, and the mode is set to overwrite. Never combine job-bookmark-disable with append mode unless the job has explicit deduplication built into the output.

Output Format and AWS Glue Partitioning for Cheap Athena Reads

Getting AWS Glue partitioning right at write time is what determines whether Athena queries against your output stay cheap or start scanning far more data than they need to.

• Use Parquet with Snappy compression as the file format. It is columnar, so Athena reads only the columns a query touches, supports predicate pushdown, and compresses four to six times better than CSV. Writing CSV to the curated zone causes full-table scans in Athena, inflates storage cost, and forfeits predicate pushdown entirely.
• Write with Hive-compatible partition keys such as year=, month=, day=, and source=. Athena auto-discovers partitions this way, the Glue Catalog updates cleanly, and partition pruning can cut Athena scan cost by 70 to 99 percent. Partitioning by a high-cardinality column such as user_id creates millions of tiny partitions and pushes Glue Catalog and Athena metadata overhead to a breaking point.
• Aim for 128 to 512 MB per output file after compression. Athena performs best in that range; too many small files create listing overhead, and very large files limit parallelism. Spark's default behaviour of writing one file per partition often produces thousands of tiny files under 1 MB that degrade Athena performance badly.
• Set spark.sql.sources.partitionOverwriteMode to dynamic so a rerun only overwrites the affected partitions and leaves the rest of the table untouched, keeping reruns idempotent. The static default overwrite rewrites the entire table on every run and silently loses data in partitions not present in the current run's input.
• Use coalesce() or repartition() to control file count before writing whenever you're appending output. Without it, ten GB of output can land in 10,000 files, forcing Athena to open all 10,000 files for any query touching that day.

Performance Optimisation for Making AWS Glue Jobs Run Faster

Glue job performance is determined by three independent variables. The first is the amount of data read, which you minimise by reading only what you need. The second is the efficiency of the transformations, which improves when you lean on Spark's optimiser and avoid Python UDFs where SQL or Spark native functions exist. The third is the number of workers and the parallelism of execution, which should match the workers to data volume with sufficient parallelism. Improving any one of these variables improves job performance. Improving all three through consistent AWS Glue performance optimization is the path to well-performing Glue jobs at scale.

Data Reading Optimisation

Reading less data is the cheapest form of AWS Glue performance tuning available, because every byte you avoid reading is compute, shuffle, and cost you never pay for downstream.

• Apply partition pruning at read time by filtering on partition columns before any transformation, so Spark pushes the filter to the S3 listing and reads only matching partitions. This alone can cut data read by 50 to 99 percent, with a proportional reduction in job time and DPU cost. For example, spark.read.parquet(path).filter(col('year') == '2026').filter(col('month') == '05').
• Apply column pruning by selecting only the columns a transformation needs immediately after reading, for example, df.select('id', 'event_type', 'created_at', 'amount') rather than df.select('*') when only five of fifty columns matter. This reduces memory pressure and shuffle volume proportionally.
• For JDBC sources, push predicates down via the dbtable parameter or the pushDownPredicate option so the filter runs on the database rather than after transfer, reducing data moved to the Glue cluster.
• Read from the specific S3 prefix matching the processing window rather than reading a parent prefix and filtering downstream, which eliminates listing overhead for partitions outside the window entirely.
• Convert source data to Parquet as early in the pipeline as possible. The first conversion costs the same as any other read, but every subsequent read is five to ten times faster, and every downstream job benefits permanently.

Transformation, Optimisation, and Avoiding the Common Performance Killers

Applying AWS Glue Spark optimization at the transformation layer usually delivers the single biggest speed gain of any change on this list, because a handful of common patterns quietly slow every job that uses them.

• Don't use Python UDFs for row-level transformations. Python UDFs make the Spark Catalyst optimizer ineffective and serialize and deserialize rows of data between the JVM and Python interpreters. It results in a slowdown of at least five to ten times compared to the same task done using Spark native functions. You should think twice before writing a transformation as a Python UDF and check if you can implement the logic using Spark SQL functions like regexp_extract, when and otherwise, to_date, explode, and struct, DataFrame API calls, or even a vectorized Pandas UDF.
• Use Adaptive Query Execution, which is enabled by default in Glue 4.0 on Spark 3.3. It automatically handles skewed partitions by splitting large ones, dynamically coalesces small shuffle partitions, and selects the optimal join strategy from runtime statistics. On Glue 2.0 or 3.0, enable it manually with spark.conf.set('spark.sql.adaptive.enabled', 'true').
• Avoid unnecessary sort operations. Spark's sortBy triggers a global sort requiring a full shuffle, and most ETL workloads do not need sorted output. If you need sorted data within a specific partition, use orderBy inside a window function applied to the relevant partition column instead of sorting the entire dataset.
• Broadcast small tables in joins. When joining a large fact table with a small dimension table under 10 to 50 MB, use a broadcast join to send the small table to every executor and skip the shuffle entirely, for example, df_large.join(broadcast(df_small), on='key'). Without the broadcast hint, the join falls back to a sort-merge join that shuffles both tables.
• Cache intermediate DataFrames that get reused. If a transformation result feeds multiple downstream operations, such as two different output tables, cache it with df.cache() or df.persist(StorageLevel.MEMORY_AND_DISK). Without caching, Spark recomputes the transformation from the source for each downstream action, doubling both read and compute costs.

Worker Configuration Optimisation for Right-Sizing the Job

Matching worker count and type to actual data volume is where most AWS Glue optimization work pays off, since over-provisioning is the single most common cause of an inflated Glue bill.

• For a small daily incremental job under 1 GB of input, two to four G.1X workers are enough, or G.025X Serverless, where it's available. Small data doesn't benefit from many workers, and driver startup overhead dominates relative to compute on tiny datasets. A warning sign here is a job using ten workers that sit 95 percent idle, or a two-minute job where a minute and a half is pure startup overhead.
• For a standard daily batch between 1 and 50 GB of input, five to ten G.1X workers are a reasonable rule of thumb, aiming for five to ten GB of input per G.1X worker and a job duration between two and fifteen minutes. A job running past thirty minutes on ten workers needs either more workers or code optimisation, while one finishing under two minutes on ten workers has too many workers for the data volume.
• For a large batch between 50 and 500 GB of input, ten to fifty G.1X workers usually scale proportionally. Watch CloudWatch DPU utilisation and adjust, always starting lower and scaling up. Straggler tasks, where one task takes ten times longer than the others, signal data skew, and adding workers won't fix that; you need to fix the partition distribution instead.
• For memory-intensive jobs with wide joins or large groupBy operations, G.2X workers in the five to fifteen range make sense, since G.2X doubles memory per worker and prevents spill to disk. SparkOutOfMemoryError in the logs is the clearest signal, and upgrading to G.2X is usually cheaper than piling on more G.1X workers to solve an OOM problem.
• For cost-sensitive, non-urgent batch work, Z.2X Flex workers run 45 percent cheaper than G.2X on equivalent capacity, running on spot instances that Glue resumes automatically in most cases. They aren't suited to time-critical jobs or ones with external dependencies waiting on completion.

Auto Scaling Is the Glue Feature Most Teams Underuse

Glue Auto Scaling, enabled through the --enable-auto-scaling job parameter in Glue 4.0, automatically adjusts the number of workers during job execution based on actual resource utilisation. AWS Glue scaling this way is particularly valuable for jobs with variable stage sizes, where an initial read and filter stage uses far fewer resources than a later join and aggregation stage, or where data volume swings significantly between runs.

To enable it, set --enable-auto-scaling to 'true' in the job parameters and configure MinCapacity and MaxCapacity in the job definition rather than NumberOfWorkers, for example, MinCapacity of 2 and MaxCapacity of 20 for a job processing 1 to 50 GB. Glue monitors stage-level metrics under the hood, scaling up when executors are fully utilised and scaling back down when they sit idle. It suits jobs with highly variable stage sizes or unpredictable daily data volume well, but isn't worth it for very short jobs under three minutes, where the auto-scaling overhead doesn't pay for itself. In practice, auto-scaling consistently uses fewer DPU-hours than statically over-provisioning, with a typical 20 to 40 percent cost reduction on variable workloads compared to a fixed worker count.

Cost Optimisation for Reducing AWS Glue Costs Without Sacrificing Reliability

AWS Glue cost at scale is driven by three variables. These are the DPU-seconds consumed per job run, the frequency of job runs, and the overhead of ancillary Glue services such as crawlers, development endpoints, and interactive sessions. Optimising each independently produces additive savings, and disciplined AWS Glue cost optimization can amount to 50 to 80 percent savings on a poorly configured Glue deployment.

The Glue Cost Optimisation Playbook

Running this kind of audit objectively is exactly the sort of work where bringing in outside data engineering consulting support pays for itself quickly, since a fresh set of eyes tends to spot over-provisioning that internal teams have stopped noticing.

• Replacing crawlers with Partition Projection eliminates GetPartitions calls and crawler DPU charges for regularly partitioned tables, cutting crawler DPU costs entirely for covered tables. Crawlers often account for 20 to 40 percent of the total Glue cost in data-lake-heavy environments, and the effort is moderate, involving an Athena table DDL change, followed by query testing, before removing crawlers from the schedule.
• Migrating to Glue 4.0 reduces job duration by two to four times on many workloads thanks to Spark 3.3, which means the same data gets processed faster for fewer DPU-seconds. Expect a 20 to 60 percent DPU-hour reduction on jobs that benefit from AQE and improved join strategies, for moderate effort testing and validating existing jobs on the new runtime.
• Right-sizing worker count by watching CloudWatch DPU utilisation and reducing worker count when utilisation sits consistently below 50 percent typically yields a 10 to 40 percent cost reduction on over-provisioned jobs, which is common when jobs were sized conservatively at the outset. This is low-effort work, adjusting NumberOfWorkers or switching to Auto Scaling.
• Using Z.2X Flex for non-urgent batch work costs 45 percent less than G.2X, and Flex spot instances reduce cost further versus on-demand pricing for jobs without tight SLAs. Most nightly batch loads qualify for a 40 to 50 percent cost reduction with low effort, simply by changing the WorkerType and enabling Flex execution.
• Eliminating development endpoints removes a continuous $0.44 per DPU-hour charge; a five-DPU endpoint running eight hours a day costs $528 a month on its own. Switching to Glue Studio interactive sessions, which are charged per session rather than continuously, eliminates that cost with low effort.
• Converting output to Parquet reduces subsequent read costs in downstream jobs by five to ten times and storage cost by four to six times, while also reducing Athena scan cost. The saving is indirect but often larger than the cost of the conversion job itself, for low-to-medium effort.
• Implementing incremental processing through job bookmarks or explicit time-window parameters, rather than full-table re-reads, can cut DPU-hours by 50 to 90 percent for mature data lakes where daily increments are small relative to total data. This takes moderate effort, implementing the bookmark or time-window logic and validating output idempotency.
• Reducing crawler frequency to weekly or on-deployment runs, rather than hourly, and using S3 event notifications for new partition detection instead of frequent polling, cuts crawler cost by 80 to 95 percent for teams currently running hourly crawlers on large prefixes, with low effort.

The Small Files Problem Is the Hidden Glue Cost Multiplier

Small files are one of the most pervasive and expensive problems in Glue-based data lakes. When a Glue job writes thousands of small Parquet files under 1 MB each instead of a few large files in the 128 to 512 MB range, every downstream job reading that output has to open and read thousands of files individually. The S3 API GET request overhead for thousands of small files can rival the actual data read cost, and Athena's query performance degrades substantially once small files pile up.

• The default Spark shuffle partition count of 200 produces 200 output files after a shuffle, regardless of data volume. For 1 GB of data, that's 200 files of about 5 MB each, which is marginal; for 100 MB of data, it's 200 files of about 500 KB each, which is bad. You'll notice this when the output file count dramatically exceeds the one to ten files a given data volume should need. Fix it by setting spark.sql.shuffle.partitions proportional to data size, roughly one partition per 128 to 512 MB, or by using coalesce(target_file_count) before writing.
• Over-partitioning of output happens when a column with extremely high cardinality, like a second-based timestamp or user_id, generates a file for each partition value. This leads to a million files for a million different users. It will display itself by the number of S3 objects in the output prefix equaling the number of distinct partition values. To resolve the problem, choose partition columns that have low to medium cardinality, for example, date, month, source system, or region, and never use natural keys or high-cardinality IDs.
• Streaming micro-batch writes, where a streaming job writes to S3 in one-to-sixty-second windows, create many small files continuously. S3 object count grows rapidly with file sizes consistently under 1 MB. Running a periodic Glue compaction job, weekly or daily, that reads the small-file prefix and rewrites it as larger Parquet files fixes this, or you can lean on Iceberg's built-in compaction.
• A Glue job failure and restart can leave partial output in place, with the restart writing additional files to the same prefix, so neither run represents a complete dataset. Watch for an unexpected file count in the output prefix or data quality checks catching unexpected row counts. Implementing atomic output, writing to a temp prefix and renaming on success, or using partition overwrite to replace partial output on restart, closes this gap.

The Glue Compaction Job Fixes Small Files After the Fact

A compaction job reads a small-file prefix and rewrites it as target-size Parquet files. The pattern reads the source prefix with spark.read.parquet(), calculates a target file count based on input size divided by a target of roughly 256 MB per file, coalesces the DataFrame to that count with df.coalesce(target_file_count), and writes to a target prefix with df_compacted.write.mode('overwrite'). parquet(args['target_prefix']). After validation, the target prefix gets renamed to the source prefix or swapped in through the Glue Catalog. For Iceberg tables, the built-in OPTIMIZE procedure handles this automatically through a call such as CALL glue_catalog.system.rewrite_data_files(table => "my_db.my_table", strategy => 'sort').

Error Handling and Idempotency for Building Glue Jobs That Fail Gracefully

A Glue job that fails silently, leaves partial output, or produces duplicate data on restart is more dangerous than a Glue job that does not run at all. Production Glue pipelines need explicit error handling at three levels. These are Spark-level errors, such as transformation failures and type mismatches; job-level errors, where an entire job failure triggers a retry; and data-level errors, meaning records that fail business logic validation without raising a Spark exception.

The Three Error Handling Levels

• A Spark job failure is a job-level exception such as an out-of-memory error, a network partition, a driver failure, or an S3 access denied, and it takes the entire job down. Configure MaxRetries in the Glue job for one to three retries on transient failures, and make sure output is idempotent so a retry doesn't create duplicates. These land in CloudWatch Logs, with an SNS notification via an EventBridge rule on the FAILED state and PagerDuty integration.
• A transformation exception happens at the record level, covering a null pointer, a type error, an unparseable date, or anything else that raises a Python exception during transformation. Wrap transformation logic in try/except, route exception records to a quarantine prefix with error metadata attached, and count and alert on quarantine volume. These land in a quarantine prefix tagged with the error type, such as TRANSFORM_EXCEPTION.
• A business rule validation failure covers records that don't raise exceptions but still violate business rules, such as a negative price, a future date, or an orphaned foreign key. Implement a validation step after transformation, separate valid from invalid records, and write invalid ones to quarantine, tagged with the validation rule name.
• A data quality gate failure is a dataset-level check failure, such as too many null values, an unexpected row count, or a schema change. Implement the quality gate before writing to the curated zone; if it fails, halt the job and alert rather than writing partial-quality data. The job halts with a descriptive error message and an SNS notification carrying the quality metric values that triggered it.

Idempotency Is the Property That Makes Retries Safe

An idempotent Glue job produces the same output regardless of how many times it is run with the same input. Idempotency is the property that makes job retries safe and pipeline reruns harmless. A non-idempotent job that gets retried after a partial failure will produce duplicate or inconsistent data.

• Partition overwrite, using spark.sql.sources.partitionOverwriteMode is set to dynamic with mode set to overwrite. This means that rerunning the job for the same date partitions overwrites only those partitions. This suits jobs partitioned by date where a rerun for the same window should replace prior output, but make sure the partition key covers the full time window of the rerun; a job processing three days of data needs to overwrite all three days' partitions.
• Writing to a temp prefix and renaming on success, then deleting the target prefix and copying or renaming from a temp location like s3://bucket/temp/job_name/run_id/gives you atomic output where consumers never read a partial write. Renaming in S3 is actually a copy-plus-delete under the hood, so for large outputs this gets slow and adds API cost.
• For Iceberg tables, MERGE INTO with the natural key is idempotent by definition, since existing records get updated and new records get inserted. This fits dimension table loads and CDC materialisation well, but MERGE is slower than append-only writes, so use it only when upsert semantics are genuinely needed.
• Explicit deduplication before write, using a window function such as row_number() OVER (PARTITION BY natural_key ORDER BY updated_at DESC) = 1 to keep only the latest record per key, helps when bookmarking is disabled and the job may process overlapping input windows. Deduplication requires a shuffle and adds job duration, so partition it by a column that limits the shuffle size.

The Quarantine Pattern Means You Never Silently Drop Bad Records

The worst ETL anti-pattern is silently dropping records that fail transformation or validation. Silent drops mean the bad data never gets investigated, the source system error never gets fixed, and downstream analytics stay subtly wrong in ways that may go unnoticed for weeks. The quarantine pattern ensures bad records are preserved, counted, and alertable.

This translates into a validation flag column for cases when a price is negative, an order date is in the future, or a customer ID is null, followed by splitting the DataFrame into a valid one, dropping the validation flag, and writing to the curated zone, and a quarantine one with only the rows that had issues. In addition, the quarantine write operation includes writing metadata such as the quarantine timestamp and job run ID, partitioning by year, month, and validation_error, and appending to a quarantine path. Compute the quarantine rate by dividing the number of quarantined records by the total number of records, and monitor it as a CloudWatch metric, triggering an SNS alert when the threshold is crossed, for instance, one percent.

If your team doesn't have the bandwidth to build all of this properly the first time, it's often faster to hire dedicated AWS data engineers who have already built quarantine and idempotency patterns like these for other production pipelines.

Monitoring and Observability for Building Visibility Into Glue Job Health

A Glue job that runs without monitoring is a time bomb. Performance degradation, cost escalation, data quality problems, and pipeline stalls all begin as subtle signals in job metrics before they become visible as incorrect data or failed SLAs. Consistent AWS Glue monitoring gives you visibility at the right level of detail for every layer of the Glue pipeline.

The Essential Glue Metrics and CloudWatch Alarms

Good AWS Glue job monitoring rests on a small set of metrics that, tracked consistently, catch almost every production problem before it becomes an incident.

• Job duration, tracked via glue.driver.aggregate.elapsedTime shows total execution time, and a sustained increase points to data volume growth or a performance regression. Alert if a run exceeds twice the baseline p50 duration for that job, and page if it exceeds three times. When it fires, check whether data volume grew, code changed, or AQE got disabled, then add workers or optimise code.
• Job status, tracked through EventBridge Glue Job State Change events, flags success, failure, or timeout. A FAILED or TIMEOUT state should trigger an immediate alert.
• DPU utilisation, comparing allocated workers against actual usage, shows whether the workers you're paying for are doing anything. Alert if average utilisation drops below 30 percent over a full job run, and respond by reducing worker count or enabling Auto Scaling.
• Shuffle spill, tracked via glue.driver.aggregate.shuffleLocalBytesWritten shows data spilling to disk during a shuffle, which signals memory pressure. Alert if the spill exceeds 10 percent of the total shuffle bytes, and respond by moving to G.2X, adding workers to shrink per-worker data volume, or rethinking the partition strategy.
• Quarantine record rate, a custom CloudWatch metric, tracks the fraction of input records rejected by validation. Alert past a 1 percent quarantine rate and page past 5 percent, then investigate source system data quality and notify the upstream data owner.
• S3 write bytes, tracked via glue.driver.aggregate.bytesWritten shows total output volume. Alert on a dramatic deviation from baseline, more than 50 percent above or below the p50, and investigate a source data volume change when it happens.
• Data freshness, a custom metric per table, tracks hours since the last successful update. Alert if a table misses its expected freshness SLA; then check the upstream Glue job for failed runs and escalate to on-call if needed.

The Observability Dashboard and What to Show Per Job

Every production Glue job should have a dedicated CloudWatch dashboard showing the metrics that tell the complete story of job health. At a minimum, a production Glue job's dashboard should include the following.

• A job duration trend as a fourteen-day rolling chart, which reveals seasonal patterns, data volume growth, and performance regressions immediately. If the Wednesday job always takes thirty minutes but this Wednesday took forty-five, the chart shows it at a glance.
• A DPU utilisation trend, which shows whether the job is actually using the workers allocated to it. Utilisation stuck around 20 percent means the job is 80 percent over-provisioned and costing roughly five times more than necessary.
• A records processed versus quarantined trend, a two-line chart tracking valid and quarantined records over time. A rise in quarantine rate with no change in valid record volume points to a source system quality issue.
• S3 bytes read versus written, which shows the compression and filtering effectiveness of each run. Bytes written significantly exceeding bytes read suggests an accidental full-table reread or an append without deduplication.
• Shuffle bytes and spill bytes, where high spill relative to total shuffle is the leading indicator of an OOM error before it actually happens.

Structured Logging for Glue Jobs Makes CloudWatch Logs Useful

The default Glue job logs sent to CloudWatch are verbose, mixed with Spark driver noise, and difficult to query. Implementing structured JSON logging in the job's Python code makes log analysis with CloudWatch Insights genuinely practical.

The pattern is a small log_metric helper that builds a dictionary with a UTC timestamp, an event_type, the job name, the run ID, and any additional keyword arguments, then logs it as a JSON string. You'd call it at each meaningful checkpoint, such as JOB_START with the source prefix and target partition, READ_COMPLETE with records read and elapsed time, VALIDATION_COMPLETE with valid and quarantine counts and rate, WRITE_COMPLETE with records and bytes written, and JOB_SUCCESS with total duration. Once those events are flowing, a CloudWatch Insights query filtering on event_type equal to JOB_SUCCESS and sorting by total_duration_s descending surfaces your slowest jobs in seconds.

CI/CD and Infrastructure as Code for Managing Glue Jobs Like Software

Glue jobs created and modified manually in the AWS console are not reproducible, not version-controlled, and not safely iterable. A change to a production job that causes a regression cannot be immediately reverted; the history of changes isn't visible, and deploying the same job to a new environment requires manual recreation. Managing Glue jobs as code, with version control, CI/CD pipelines, and AWS Glue automation through infrastructure as code, eliminates all three problems.

The Glue Job IaC Structure Using Terraform

A Terraform-managed Glue deployment typically breaks down into a handful of resource types, each covering a distinct piece of the job's lifecycle.

• aws_glue_job manages the job definition itself, covering worker type, count, runtime version, script location, connections, and parameters. Key configuration includes GlueVersion set to '4.0', WorkerType, NumberOfWorkers, MaxRetries, Timeout, and DefaultArguments including --job-bookmark-option, --TempDir, and --enable-metrics.
• aws_glue_trigger manages schedule- or event-based triggers for the job, using a cron expression for scheduled runs, an EventBridge or Glue workflow dependency for event-based runs, or START_ON_DEMAND for manual runs.
• aws_glue_workflow manages a multi-job pipeline with dependencies and parallel execution, defining start triggers, job dependencies, and conditional triggers that only run the next job if the previous one succeeded.
• aws_s3_object manages the Glue job's Python script uploaded to S3, sourced from the Git repository with a versioned S3 key that includes the git commit SHA for traceability.
• aws_iam_role manages the IAM role for job execution under least-privilege principles, granting S3 read on the source, S3 write on the target, Glue Data Catalog access, CloudWatch Logs write, and KMS decrypt where SSE-KMS is in use.
• aws_cloudwatch_metric_alarm manages CloudWatch alarms for the job, covering the FAILED state via EventBridge, a duration threshold, and a custom quarantine metric.
• aws_glue_catalog_database manages Glue Data Catalog databases, typically one per zone per domain, with a location_uri pointing to the relevant S3 prefix.

The Glue Job CI/CD Pipeline

A GitHub Actions pipeline triggered by a pull request against main or a push to main typically runs through four stages. First, lint and test on the PR, running flake8 or ruff for Python linting, pytest with a mocked SparkContext for unit tests on transformation functions, and a Terraform plan validation for non-destructive changes. Second, integration tests on the PR to main, deploy the Glue job to a DEV environment via Terraform, run it against a sample of test data in the DEV S3 bucket, and validate output against row count, schema, and data quality assertions, failing CI if the output doesn't match expectations.

Third, staging deployment on merge to main runs terraform apply against STAGING, uploads the job script to STAGING S3 with the commit SHA in the key, runs the full job against staging-scale data, and compares the resulting CloudWatch dashboard against the previous staging run. Fourth, production deployment after staging approval adds a manual approval gate through a GitHub Environment protection rule, then terraform apply to PRODUCTION, upload of the job script to PRODUCTION S3, and a Git commit tag marking the production deployment version. Rollback is a git revert followed by terraform apply, which redeploys the previous job version in under five minutes.

Testing Glue Job Code With the Unit Testing Pattern

Glue job Python code is standard PySpark. The transformation logic, the functions that process DataFrames, can and should be unit-tested without running a Glue cluster, using a local PySpark session. The key practice is separating transformation logic from Glue infrastructure code, meaning GlueContext and DynamicFrame reads and writes, into pure functions that operate on DataFrames. These pure functions are unit-testable locally.

• Separate transformation logic by defining all business logic as pure functions that take a DataFrame and return a DataFrame, imported from a shared utility module, so the main Glue script only handles reading, calling the transformation functions, and writing.
• Unit test with pytest and AWS Glue PySpark code by creating a SparkSession in pytest fixtures, using findspark for local Spark, passing small in-memory DataFrames to the transformation functions, and asserting on output schema and values.
• Mock Glue infrastructure using unittest.mock to mock GlueContext, the job parameter arguments, and S3 read and write operations in integration tests, so job orchestration logic can be tested without requiring actual AWS access.
• Build test data fixtures covering normal cases, edge cases such as null values, empty partitions, and schema variations, and invalid records that should be quarantined, then assert that the quarantine logic correctly identifies and routes each type.

Schema Evolution, Data Quality, and Pipeline Reliability at Scale

In mature data lakes, the most common causes of failures in Glue pipelines are not related to any issues with infrastructure stability. In fact, it's about schema evolution when changes happen in the source system schema because of adding new columns, renaming fields, or changing data types without notifying the data engineering team working on the downstream process. Thus, a Glue job that used to work successfully for several months will suddenly fail or produce erroneous results that an enterprise-grade AWS Glue data pipeline should be prepared for.

Schema Change Detection and Response Patterns

• When a new column is added, comparing the current schema against the last-known schema stored in DynamoDB or an S3 parameter store detects it. The automated response auto-adds the new column to the curated schema with null values for historical partitions, logs the addition, and notifies via SNS. A human still needs to review whether the column is meaningful, whether it should be renamed, and whether it belongs in downstream analytics models.
• When a column is removed from the source, schema comparison detects the missing column. The automated response substitutes null for the missing column going forward and alerts the data engineering team rather than failing the job outright. A human needs to investigate the source system change and decide whether to hard-fail or keep substituting null.
• A renamed column is trickier. Schema comparison detects the old column missing and a new one present with a similar data distribution, but the system can't reliably tell a rename apart from an add-plus-remove, so it alerts and relies on heuristics. A human needs to manually map the new column name to the old one for backward compatibility.
• A compatible data type change gets handled by attempting a CAST to the expected type, routing cast failures to quarantine, and logging the change, with a human confirming whether the change was intentional and updating downstream type dependencies.
• An incompatible data type change surfaces as a schema validation failure when the cast attempt raises an exception. The automated response routes affected records to quarantine and continues the job with records that pass validation, while a human urgently coordinates with the source system team on whether to accept the new type or reject it.

AWS Glue Data Quality Is the Managed Quality Check Service

AWS Glue Data Quality, which reached general availability in 2023, provides a managed framework for defining and evaluating data quality rules on Glue Data Catalog tables and Glue job outputs without writing custom validation code. Rules are defined in DQDL, the Data Quality Definition Language, and evaluated as part of the Glue ETL job or independently.

• Completeness rules check that required columns have no null values above a threshold, for example, ColumnCompleteness 'customer_id' > 0.99, failing if more than 1 percent of customer_id values are null. On failure, the job halts and alerts, null records get quarantined, and the source system owner gets notified.
• Uniqueness rules check that primary or natural key columns have no duplicates, for example, ColumnUniqueness 'order_id' > 0.999. On failure, investigate the source for duplication and implement deduplication in the ETL job.
• Value range rules check that numeric values fall within expected bounds, for example, ColumnValues 'price' between 0 and 100000 to flag out-of-range prices. On failure, quarantine the out-of-range records and alert the data quality team.
• Row count rules check that a table or processed batch has the expected row count, for example, RowCountMatch 'orders' between 10000 and 1000000, failing if the daily batch falls outside that range. On failure, alert on the deviation and investigate source system changes.
• Schema rules check that all expected columns are present with expected types. For example, ColumnExists 'order_date' and ColumnDataType 'order_date' match 'DATE'. On failure, alert on the schema change and apply the schema change detection response pattern above.

Advanced Patterns for Running AWS Glue at Enterprise Scale

As the number of Glue jobs in a data platform grows from a handful to dozens or hundreds, new operational challenges emerge, including job scheduling and dependency management, job library management, parameter standardisation, and governing who can deploy and modify production jobs. Every one of these patterns exists to keep a growing AWS Glue ETL pipeline portfolio manageable rather than chaotic.

Glue Workflows vs External Orchestration Using Airflow or Step Functions

When it comes to selecting the right AWS Glue workflow orchestration strategy, it depends on how complex your branching logic and coordination between services are. Chains of Glue jobs that operate linearly or run in parallel seldom require anything beyond AWS Glue Workflows, whereas more sophisticated platforms based on AWS Glue together with Lambda, ECS, etc., grow out of Glue Workflows quite fast.

Managing Shared Libraries Across Glue Jobs

Production data platforms with many Glue jobs share common code, such as S3 utilities, CloudWatch logging, schema validation functions, quarantine helpers, and data quality checks. Managing this shared code correctly prevents duplication that creates inconsistency and avoids the deployment headache that appears when shared code changes need to propagate to every consuming job.

• Package shared utility code as a Python wheel, upload it to S3, and reference it via the --additional-python-modules job parameter or the Glue job's ExtraFiles configuration. Versioned wheel files, such as s3://bucket/wheels/shared_utils-1.2.3-py3-none-any.whl, allow controlled rollout and rollback of shared code changes.
• Use --job-parameters for every environment-specific value, such as S3 bucket names, Athena database names, SNS topic ARNs, and quarantine thresholds. Never hardcode environment-specific values in job scripts; this is exactly the pattern that lets the same script deploy to dev, staging, and production by changing only parameters.
• Store JDBC credentials in AWS Secrets Manager and reference them through a Glue Connection object, so the job uses the connection name rather than embedding credentials directly. This centralises credential rotation and keeps credentials out of job scripts entirely.

The Glue Job Production Readiness Checklist

Before promoting any Glue job from development to production, it should satisfy every item on the following checklist. This checklist encapsulates the AWS Glue best practices covered throughout this guide into a concrete pre-production gate that prevents the most common and expensive Glue production failures.

Job configuration

• Worker type is set appropriately, with G.1X for standard ETL, G.2X only if G.1X causes OOM, and Z.2X Flex for non-urgent batch.
• Glue version is set to 4.0, running Spark 3.3 and Python 3.10.
• Job bookmarking is enabled for incremental jobs or explicitly disabled with documented rationale for full-rebuild jobs.
• MaxRetries is set to one to three for transient failures, with idempotent output verified before enabling retries.
• Timeout set to twice the expected maximum job duration, with no unlimited timeout in production.
• Auto Scaling enabled with MinCapacity and MaxCapacity, or a fixed worker count with documented justification.
• TempDir is configured as an S3 path with a lifecycle policy for automatic cleanup.

Data correctness

• Output format is Parquet with Snappy compression, never CSV in the curated or consumption zone.
• Partition columns use Hive-compatible keys with appropriate cardinality, such as date, source, or region, never user_id.
• Partition overwrite mode is dynamic, never static, unless a full-table rebuild is genuinely intended.
• Idempotency is verified, meaning job reruns produce identical output, confirmed with a second run on the same input.
• The quarantine pattern routes bad records to a quarantine prefix and never drops them silently.
• Schema validation defines the expected schema explicitly, with mismatches detected and logged.

Error handling

• A quarantine threshold alarm sends an SNS notification if the quarantine rate exceeds a configured threshold.
• A job failure notification routes the FAILED state through EventBridge to SNS, then to PagerDuty or Slack.
• Structured logging emits JSON log events at job start, read complete, validation complete, write complete, and job end.

Monitoring

• A CloudWatch dashboard tracks job duration trend, DPU utilisation, quarantine rate, and bytes read and written.
• A duration alarm fires if the job exceeds twice its baseline duration.
• A freshness alarm fires if the target table hasn't been updated within its SLA window.

Infrastructure as code

• The job definition lives in Terraform or CDK, in version control, never created manually in the console.
• The job script lives in version control and deploys via the CI/CD pipeline, never uploaded manually.
• The IAM role follows least privilege and has been reviewed and approved by security or the data platform team.

Testing

• Unit tests cover transformation logic with pytest and a local PySpark session.
• An integration test runs the job on a representative sample in the DEV environment with validated output.
• An idempotency test runs the job twice on the same input and confirms the output is identical.
• Edge case tests cover empty input, single-record input, and maximum expected input, and all pass.

AWS Glue at Scale: The Practices That Separate Reliable Pipelines From Expensive Problems

AWS Glue is a powerful managed ETL service that rewards correct configuration with cost-efficient, reliable, and maintainable pipelines. The practices in this guide, covering proper worker sizing, job bookmarking, Parquet output with correct partitioning, idempotent error handling, structured monitoring, and CI/CD deployment, are not advanced optimisations reserved for mature teams. They are the baseline configuration that any production Glue job should have from its first deployment, and they are what AWS Glue best practices actually look like in daily engineering work.

The most expensive Glue problems are almost all preventable with design-time discipline. Think of the job that runs for three hours because it reads an entire table instead of the daily partition, the pipeline that produces duplicates because bookmarking was disabled while the output mode stayed on append, or the production incident caused by a schema change the job had no way to detect or handle. None of these requires sophisticated engineering to prevent. They require applying the patterns described in this guide consistently, treating AWS Glue best practices as the default rather than the exception.

The data engineering teams that run well-managed Glue pipelines at scale have one thing in common. They treat Glue jobs like software. Version-controlled scripts, automated tests, staged deployments, documented error handling, and production monitoring aren't bureaucratic overhead. They are the practices that make a Glue-based pipeline trustworthy enough to build a data platform on, and following consistent AWS Glue best practices from the outset is what makes that trust possible at scale.

Mobisoft Infotech designs and builds AWS cloud data infrastructure, including AWS Glue ETL pipelines, data lake architectures, and analytics platforms for enterprises and scale-ups. Our data engineering practice has designed Glue pipelines processing billions of records daily across healthcare, fintech, retail, and logistics verticals.

Nitin Lahoti

Co-Founder and Director

Read more

Nitin Lahoti is the Co-Founder and Director at Mobisoft Infotech. He has 15 years of experience in Design, Business Development and Startups. His expertise is in Product Ideation, UX/UI design, Startup consulting and mentoring. He prefers business readings and loves traveling.

Read less

Share Article