Operational efficiency and optimization

Cost: token efficiency

Cost on Bedrock on-demand is input tokens plus output tokens, and output tokens cost more per token. Latency also grows with output length.

Long contexts are not free; more context can also reduce quality.

Cost: model selection

• Cost-capability trade-off: evaluate several models on your test set and compare price-to-performance (quality score per dollar), not raw quality.
• Tiered usage: route simple queries to a small, cheap model and complex ones to a large model; cascade by escalating when confidence is low. Bedrock intelligent prompt routing automates this within a model family.
• Right-size: distilled models and small models fine-tuned for one task often beat a large general model on cost with equal quality.
• Pricing modes: on-demand (per token), batch inference (discounted, asynchronous), Provisioned Throughput (hourly, committed), and service tiers that trade price for priority or latency on some models. Match mode to traffic: spiky means on-demand, steady and high means provisioned, offline means batch.
• Cost visibility: application inference profiles with cost allocation tags show spend per application or tenant in Cost Explorer; AWS Budgets and Cost Anomaly Detection alert on unusual spend.

Throughput and capacity

• Quotas apply per model and Region in requests per minute and tokens per minute. Exceeding them returns throttling errors (429). Requests for increases go through Service Quotas. For some models, output tokens count against the quota at a multiple (a burn rate), so setting maxTokens much higher than needed consumes quota.
• Batching: batch inference for offline workloads; batch embedding calls; group small prompts where models allow it.
• Concurrency control: SQS plus Lambda with a maximum concurrency setting keeps you under quotas and smooths bursts.
• Cross-Region inference raises effective throughput and handles bursts.
• Provisioned Throughput gives dedicated model units when you need guaranteed rate; size by tokens per minute at peak and check utilisation so you don’t pay for idle units.
• SageMaker endpoints: auto scaling policies on invocations per instance, concurrency or custom token metrics; scale-in cool-downs to avoid thrashing; warm capacity for slow-loading models.
• Capacity planning: estimate tokens per request × requests per minute at peak, add headroom, and monitor actual prompt and completion size distributions.

Caching

Never cache personalised or sensitive responses in a shared cache without keying on the user or tenant.

Latency and performance

• Streaming (ConverseStream): time to first token drops, so the app feels fast.
• Latency-optimised inference options for supported models and smaller models for time-sensitive paths.
• Parallel requests: run independent sub-tasks (several retrievals, several classification calls) in parallel with Step Functions Parallel or Map states or async code.
• Reduce output length and chain depth; fewer sequential model calls means lower latency.
• Retrieval speed: index tuning (HNSW parameters such as ef_search, m, quantisation to reduce memory), fewer or smaller vectors (lower dimensionality), shard and replica sizing, query preprocessing (rewrite, normalise), hybrid search with custom scoring weights, filtering to shrink the candidate set, and caching frequent queries.
• API profiling: log prompt size, completion size and latency per call; find slow steps with X-Ray; communication patterns such as keeping connections warm, VPC endpoints in the same Region and avoiding cross-Region hops.
• Pre-warm SageMaker and provisioned capacity ahead of known peaks.

Inference parameters

Parameter names and ranges differ per model; use Converse inferenceConfig for the common ones and additionalModelRequestFields for model-specific ones (such as top-k for some models). Validate changes with A/B testing against the same evaluation set before rolling out.

Monitoring and observability

Layers of observability.

What to alarm on. Token usage bursts and cost anomalies (CloudWatch anomaly detection, Cost Anomaly Detection), latency percentiles (p95, p99, time to first token), throttle rate, error rate, guardrail block rate, retrieval empty-result rate, response drift (shifts in length, sentiment, or evaluator scores), and quality scores sampled from production.

Tool and agent monitoring. Track tool call counts, latency, error rates and parameter validation failures per tool; follow multi-agent hand-offs with a trace ID; baseline normal call patterns and alert on deviations (a tool suddenly called 100 times per request signals a loop or abuse).

Vector store operations. Monitor query latency, indexing lag, memory and CPU (OpenSearch metrics, Aurora Performance Insights), index health and shard balance; run automated index optimisation (merge segments, refresh intervals, rebuild after embedding changes) and data quality validation (empty chunks, duplicates, stale documents).

GenAI troubleshooting frameworks. Failures that don’t exist in classic ML: hallucination, inconsistent answers, reasoning errors, prompt drift. Techniques: golden datasets with known answers run on schedule to detect hallucination and regressions; output diffing across runs or versions to measure consistency; reasoning path tracing to find where multi-step logic went wrong; specialised pipelines that sample production traffic, score it with an LLM judge and chart trends.

Common traps

• Provisioned Throughput for unpredictable traffic.
• Caching with high temperature and expecting identical output.
• Semantic cache threshold set too loose.
• Raising maxTokens “just in case” and burning quota and money.
• Placing variable content at the start of a prompt, breaking prompt caching.
• Turning on invocation logging and forgetting the storage cost and sensitivity of what’s stored.
• Measuring only averages; p95 and time to first token matter for users.