A nightly AWS Glue Spark job fails with ExecutorLostFailure after running for about 40 minutes. The driver logs show one executor's container being killed, and the failing stage processes a single partition that holds most of the data. Which cause is MOST likely?
Show answer and explanation
Answer: C
ExecutorLostFailure means Spark lost an executor process, and a common reason is that the executor exceeded its memory allocation. When one partition holds most of the rows, a single executor must hold that oversized partition in memory, which can trigger the container kill. The fix is to repartition or salt the skewed key, or to move to a worker type with more memory. Permission, catalog and syntax problems produce their own distinct error messages at predictable points.
Why the other options are wrong
- A. A missing list permission produces an access-denied error from S3, usually at the first read, rather than a lost executor in the middle of a stage.
- B. A deleted catalog table surfaces as an entity-not-found error during job setup, not as a container being killed after 40 minutes of processing.
- D. A syntax error stops the job at parse or planning time with a message naming the offending statement, so it cannot appear as a lost executor after 40 minutes.