AIP-C01 notes: Testing, validation and troubleshooting

This domain is 11% of scored content, the smallest, but its ideas run through the others. Questions ask how to measure whether a GenAI system is good, how to stop a model update from making things worse, and how to find the cause when answers are wrong, truncated or slow. The core difference from classic ML is that outputs are open-ended text, so you need metrics for meaning and behaviour, not just accuracy.

Evaluating GenAI output

Dimensions. Relevance (does it answer the question), factual accuracy or faithfulness (is it supported by the facts or source), consistency (same question, similar answer), fluency and coherence, completeness, tone, safety (toxicity, harmfulness), and format compliance. For RAG add retrieval quality; for agents add task success.

Methods.

Method	Strength	Weakness
Programmatic metrics (exact match, ROUGE, BLEU, BERTScore, regex, JSON schema checks)	Cheap, repeatable	Weak on meaning; need reference answers
LLM-as-a-judge	Scales; scores nuanced qualities such as helpfulness and correctness against a rubric	Judge bias, cost, needs calibration against human scores
Human evaluation	Gold standard for subjective quality	Slow, costly, needs clear rubrics
User feedback	Reflects real use	Sparse and biased toward extremes

Best practice is a mix: automated checks on every change, LLM judges for broad scoring, humans for calibration and edge cases.

Amazon Bedrock Model Evaluation

Automatic evaluation: choose a task type (summarisation, Q&A, classification, general text generation) and built-in or custom datasets from S3; metrics include accuracy, robustness and toxicity, using programmatic scoring.
LLM-as-a-judge evaluation: a judge model scores outputs for metrics such as correctness, completeness, helpfulness, coherence, faithfulness, harmfulness and refusal; you can add custom metrics and use different generator and judge models.
Human-based evaluation: your own team or an AWS-managed team rates responses against criteria like helpfulness or style.
RAG evaluation: assess a knowledge base’s retrieval only (context relevance, coverage) or retrieval plus generation (correctness, faithfulness, citation precision and coverage, completeness). You can evaluate a Bedrock knowledge base or bring your own RAG pipeline’s outputs.
Results go to S3 and the console; jobs can compare multiple models side by side.
Other evaluation tools: SageMaker Clarify and FMEval (open-source library) for models you host.

Comparing models and configurations

Fix the dataset and compare candidates on quality, latency, token cost and business outcome. Report ratios such as latency-to-quality and cost per correct answer.
A/B testing: split live traffic between variants (Prompt Management variants, weighted routing in API Gateway, Lambda aliases with weights, AppConfig feature flags) and compare metrics. Canary testing: send a small share of traffic to the new model or prompt, watch quality and error metrics, then widen or roll back.
Shadow testing: send production requests to the candidate in the background and compare, without user impact.

Continuous quality assurance

Regression suites: a fixed set of inputs with expected properties; run on every prompt, model, guardrail or retrieval change. Track scores over time.
Automated quality gates in CodePipeline: deployment stops if evaluation scores fall below thresholds or hallucination rate rises.
Continuous evaluation of sampled production traffic (from invocation logs) using LLM judges; alert on drops.
Golden datasets with verified answers catch hallucination and semantic drift.

User-centred evaluation

Collect thumbs up/down and ratings through an API Gateway feedback endpoint stored in DynamoDB or S3, free-text comments, and annotation workflows (SageMaker Ground Truth or Amazon A2I style human review) to label samples. Feed results into the next evaluation set and prompt revision.

Evaluating retrieval

Measure relevance scoring (are retrieved chunks actually on topic), context precision and recall (right chunks present, few irrelevant), hit rate or MRR on a labelled query-to-document set, context matching (does the answer rely on the retrieved text), and retrieval latency percentiles.
Compare chunking strategies, embedding models, top-k and reranking on the same query set.

Evaluating agents

Metrics: task completion rate, correct tool selection and parameter accuracy, number of steps and redundant calls, recovery from tool errors, latency and cost per task, and reasoning quality across a multi-step plan. Use Bedrock Agent evaluations (and AgentCore evaluation capabilities), traces and LLM judges over trajectories. Test with scripted scenarios, including adversarial and failure cases such as a tool returning an error.

Reporting

Automated reports and dashboards (CloudWatch, QuickSight, notebooks) comparing models and versions, trend charts of quality, cost and latency, and summaries for non-technical stakeholders tied to business outcomes.

Deployment validation

When a model, prompt, guardrail or index changes:

Run the regression and evaluation suite as a pipeline gate.
Run synthetic user workflows end to end (canaries in CloudWatch Synthetics or Step Functions) against the new version.
Check AI-specific signals: hallucination rate, semantic drift (similarity of new outputs to baseline answers), response consistency, format validity, guardrail block rate.
Deploy gradually (aliases, canary), watch alarms, and keep an automatic rollback path.

Models get deprecated and updated by providers; pin a model version where possible and re-run the suite before migrating.

Troubleshooting

Content handling: context overflow and truncation

Symptoms: ValidationException about input too long, truncated answers, the model ignoring early content (“lost in the middle”), unexpectedly low quality on long inputs.

Count tokens before the call; compare with the model’s context window and your maxTokens.
A response cut off mid-sentence means stopReason is max_tokens: raise the limit or request a shorter format.
Fix: dynamic chunking (chunk size by token count), summarise or map-reduce long documents, retrieve fewer and more relevant chunks, trim history, put critical instructions at the start and end, choose a longer-context model.

FM integration issues

Read the error type: ValidationException (malformed request, wrong body shape for the model, bad parameter, input too long), AccessDeniedException (IAM policy, model access, SCP, VPC endpoint policy, missing guardrail condition), ResourceNotFoundException (wrong model ID or Region), ThrottlingException (quota; backoff, quotas, Cross-Region inference), ModelTimeoutException and ModelNotReadyException (retry with backoff), ServiceQuotaExceededException, ModelErrorException.
Use request validation in API Gateway, structured error logging with request IDs, and replay failing requests from invocation logs.
Common causes: using an InvokeModel body for the wrong provider, calling a model in a Region where it’s not available (use an inference profile), network path blocked (no endpoint or NAT), Lambda timeout shorter than generation time, API Gateway 29-second limit.

Prompt problems

Symptoms: inconsistent format, ignored instructions, verbose or off-topic answers.
Method: change one thing at a time, compare versions side by side on a test set (Prompt Management variants), use few-shot examples, explicit output format and schema, clearer role and constraints, lower temperature for consistency. Keep a prompt testing framework in Lambda and Step Functions that runs the regression suite and records scores.

Retrieval problems

Symptom	Likely cause and fix
Answer ignores obvious document	Chunk too large or split badly; change chunking (hierarchical or semantic); check the document was actually synced and parsed
Exact identifiers not found	Pure vector search; use hybrid search
Irrelevant chunks returned	Wrong embedding model for the domain, too high top-k, no reranker, no metadata filter
Right content, stale answer	Sync not run, deleted documents not removed; fix ingestion triggers
Poor results after changing embedding model	Index and query embeddings differ; re-embed everything
Slow queries	Index parameters, shards, vector dimensionality, filters, instance size
Results got worse over time	Embedding or content drift; monitor relevance scores against a labelled set

Diagnose embedding quality by checking nearest neighbours of known queries, vector norms and dimension mismatches, and empty or tiny chunks from failed parsing (PDF tables, scanned images needing OCR).

Prompt maintenance issues

Prompts rot as data and models change. Use template testing and CloudWatch Logs (Logs Insights over invocation logs) to find prompts that produce confusion or errors; schema validation to detect format drift; X-Ray and trace pipelines to see which template version ran; version everything in Prompt Management and roll back to a known-good version; refine systematically from failure clusters.

Common traps

Using only exact-match metrics to judge free-form text.
An LLM judge from the same model as the generator without calibration.
Evaluating on the data used to tune the prompt.
Looking only at the average score and missing a failing segment.
Blaming the model for retrieval faults; check what context actually reached the prompt.
Changing several variables at once and learning nothing.

Exam checklist

Relevance, factual accuracy, consistency and fluency as the core quality dimensions.
Bedrock Model Evaluation modes: automatic, LLM-as-a-judge, human, RAG (retrieval only and retrieve-and-generate).
A/B, canary and shadow testing; regression suites and quality gates in the pipeline.
Retrieval metrics: relevance, context precision and recall, latency.
Agent metrics: task completion, tool use accuracy, reasoning quality; Bedrock agent evaluation.
Context overflow signs (max_tokens stop reason, ValidationException) and fixes.
Error types and what each means; Region and model ID mistakes; timeouts.
Retrieval symptom-to-cause table: chunking, hybrid search, reranking, sync, re-embedding.