A payments company must score card transactions for fraud in real time. Requirements: ingest millions of events per second, run a complex ML model on each event, send immediate alerts while also keeping history for later analysis, and guarantee that each transaction is recorded exactly once. Which architecture should it implement?
Show answer and explanation
Answer: A
The design needs high-throughput ingestion, low-latency model scoring, two output paths, and exactly-once results. Kafka on HDInsight absorbs very high event rates, Databricks Structured Streaming applies the model to incoming data with low latency, and Delta Lake sinks give exactly-once semantics through transactional writes. The hot path raises alerts while the cold path persists history for analysis. Nightly batch and hourly pipelines cannot meet real-time latency, and dedicated pools are not built for streaming ingestion at this rate.
Why the other options are wrong
- B. Nightly Hadoop batch work finishes too late for fraud alerts on live transactions, so it cannot meet a real-time requirement regardless of scale.
- C. Hourly pipelines introduce delays of up to an hour between payment and scoring. That latency violates the immediate alerting requirement, so this design is unsuitable.
- D. Dedicated SQL pools are optimized for warehouse queries rather than high-volume streaming ingestion and continuous scoring, so this design misses the throughput and latency needs.