A nightly pipeline writes curated sales tables. The team wants the Spark job itself to stop and prevent publication when a null-rate or uniqueness constraint fails. Which approach meets that need?
Show answer and explanation
Answer: D
Deequ is an open-source library for declaring data quality constraints, such as completeness and uniqueness, and evaluating them inside a Spark job. When a constraint fails, the job can raise an error before the output is published, which is the gating behavior the team wants. Profiling and post-load Athena checks only report issues after data is already available, and CloudWatch metrics alert people but do not control the job.
Why the other options are wrong
- A. DataBrew profiling describes the data and highlights issues in a report, but it does not block a downstream load and depends on someone reading the output.
- B. Querying the curated table after writing means bad rows are already visible to consumers by the time the check runs, so it cannot prevent the release.
- C. Row count metrics and alarms notify operators, but nothing in the pipeline stops or rolls back the publication, so the gate is still missing.