Model Evaluation - SageMaker Debugger, Clarify, and Experiments

Model evaluation on SageMaker is the discipline of answering three questions before a model is allowed to leave the training environment: is it accurate enough, is it healthy enough (no broken gradients, no overfit, no biased outcomes), and is it explainable enough for stakeholders to trust. On the MLA-C01 exam, model evaluation anchors Task 2.3 (Analyze model performance) of Domain 2 (ML Model Development, 26 percent weight). This is an engineering exam, not a statistics exam — questions almost never ask "derive the F1 formula" and almost always ask "given symptoms during training, which SageMaker tool surfaces the root cause" or "which Clarify metric quantifies disparate impact."

This guide is built for the ML engineer perspective. It covers the four classification metrics (accuracy, precision, recall, F1, AUC-ROC) and the three regression metrics (RMSE, MAE, R-squared) that the exam expects you to choose between, the SageMaker Debugger system that catches training pathologies in real time without code changes, the SageMaker Clarify service that handles both pre-training and post-training bias plus SHAP-based explainability, the SageMaker Experiments service that tracks every training run with lineage and comparison views, and the model card discipline that documents intended use and limitations before a model enters the registry. It also covers the most heavily community-cited pain point on this exam — the Debugger vs Model Monitor and Clarify vs Model Monitor distinctions — Debugger and Clarify are training-time tools (Domain 2), Model Monitor is the post-deployment tool (Domain 4), and the exam plants this confusion repeatedly.

What Model Evaluation Means in MLA-C01 Context

Model evaluation in MLA-C01 covers the lifecycle phase between "training run completed" and "model registered for deployment approval." It is not just metric calculation — it includes catching training pathologies (Debugger), measuring fairness and explainability (Clarify), tracking experiments and reproducibility (Experiments), and documenting the model for governance (Model Cards). The community signal is consistent: candidates from data-science backgrounds over-prepare on metric formulas and under-prepare on the operational tooling, while candidates from DevOps backgrounds underestimate how deeply Clarify is tested.

Why Evaluation Is Where Most Production Failures Originate

A model that achieves 95 percent training accuracy can still fail in production if (1) the training loss had subtle gradient issues you did not catch, (2) the model is biased against a protected demographic, (3) the feature attributions are unstable across similar inputs, or (4) the team cannot reproduce the training run six months later when retraining is required. Each of these failure modes has a SageMaker tool dedicated to detecting it before deployment. The MLA-C01 exam tests recognition of which tool catches which failure mode.

The Five Tools That Cover Model Evaluation on MLA-C01

A complete evaluation stack uses five SageMaker tools. SageMaker Debugger captures training tensors and applies built-in rules to detect anomalies during training (vanishing gradients, overfit, poor weight initialisation). SageMaker Debugger Insights profiles CPU/GPU utilisation and IO bottlenecks. SageMaker Clarify computes pre-training and post-training bias metrics and SHAP-based explainability. SageMaker Experiments tracks runs, hyperparameters, metrics, and artifact lineage across iterations. SageMaker Model Cards documents the model's intended use, evaluation results, and known limitations for governance review. Each tool has a distinct purpose, and the exam tests which tool answers which scenario.

Classification Metrics - Choosing the Right Objective

Classification metric choice depends on the cost of false positives versus false negatives, plus class balance. The exam plants scenarios where the wrong metric is the wrong answer.

Accuracy and When It Misleads

Accuracy is (TP + TN) / (TP + TN + FP + FN) — the fraction of all predictions that are correct. It is intuitive but misleads on imbalanced classes. A fraud-detection model where 0.1 percent of transactions are fraud achieves 99.9 percent accuracy by predicting "not fraud" for everything. Accuracy hides the failure. For imbalanced classes, accuracy is the wrong choice.

Precision, Recall, and the F1 Compromise

Precision = TP / (TP + FP) — of the items I flagged as positive, how many actually are. Recall = TP / (TP + FN) — of all the actual positives, how many did I catch. Precision matters when false positives are costly (legitimate transactions blocked, healthy patients receiving treatment). Recall matters when false negatives are costly (fraud not caught, cancer missed). F1 = 2 * (P * R) / (P + R) is the harmonic mean — useful when you need both, especially on imbalanced classes. The exam asks "which metric for spam filtering" — typically precision (false positives are user-visible spam in inbox-equivalent), or "which metric for cancer screening" — typically recall (missing cancer is catastrophic).

AUC-ROC and AUC-PR

AUC-ROC measures the area under the receiver operating characteristic curve — the trade-off between true positive rate and false positive rate across thresholds. AUC-ROC of 0.5 is random guessing; 1.0 is perfect. It is threshold-independent — useful when the deployment threshold is not yet decided. AUC-PR (precision-recall AUC) is preferred for very imbalanced data because ROC can look optimistic when negatives dominate.

Confusion Matrix Reading

The confusion matrix breaks down predictions by class. For a binary classifier, four cells: TP, FP, TN, FN. Multiclass extends to a square matrix. The exam may show a confusion matrix and ask "what is the model getting wrong" — read the off-diagonal cells; large values indicate which classes are being confused with which.

Metric choice is driven by class imbalance and the asymmetric cost of false positives versus false negatives — accuracy is rarely the right metric when classes are imbalanced. For imbalanced binary classification (fraud, anomaly detection, rare-disease screening), F1 or AUC-PR are the right defaults. For balanced multiclass (image classification, sentiment analysis with adequate samples per class), accuracy is reasonable. For ranking and retrieval, precision@k and recall@k matter more than threshold-based metrics. The MLA-C01 exam tests this judgment — a stem describing 1 percent fraud rate and asking "which metric" expects F1 or AUC-PR; an answer choosing accuracy is wrong even though the model achieves 99 percent on accuracy.

Regression Metrics - When Predictions Are Continuous

Regression evaluation has its own metric set with different sensitivities to outliers and scale.

RMSE - Root Mean Squared Error

RMSE = sqrt(mean((y_pred - y_actual)^2)). Squared error penalises large mistakes heavily. RMSE is in the same units as the target, making it interpretable. Use RMSE when large errors are disproportionately costly (e.g., predicting battery life — being off by 10 hours is much worse than being off by 1 hour twice).

MAE - Mean Absolute Error

MAE = mean(abs(y_pred - y_actual)). Linear penalty regardless of error magnitude. MAE is more robust to outliers than RMSE — a single large prediction error does not dominate the metric. Use MAE when you care about typical error and have outliers in the data.

R-squared - Variance Explained

R-squared (coefficient of determination) = 1 - (SS_res / SS_tot). Measures the fraction of target variance the model explains. R-squared of 1.0 is perfect, 0.0 is no better than predicting the mean, negative is worse than predicting the mean. Useful for comparing models on the same dataset, but R-squared can be misleading when added features artificially inflate it on training data.

MAPE - Mean Absolute Percentage Error

MAPE = mean(abs((y_actual - y_pred) / y_actual)) * 100. Useful when relative error matters more than absolute error (forecasting demand at scale). Breaks down when actuals can be zero or near-zero.

SageMaker Debugger - Catching Training Pathologies

Debugger is the runtime monitoring layer for training jobs. It captures internal training state — weights, gradients, activations, losses — and applies rules to detect problems while training is still running.

What Debugger Captures

Debugger hooks into the training framework (PyTorch, TensorFlow, MXNet, XGBoost) and saves tensor data to S3 at configurable intervals. The hooks are non-intrusive — most workloads do not require code changes; the hook is injected by the SageMaker training container. Captured data includes weights, biases, gradients, activations, optimiser state, losses, and custom collections.

Built-in Rules - The 18 Out-of-the-Box Detectors

SageMaker Debugger provides 18+ built-in rules covering common training pathologies. Examples include vanishing_gradient (gradients near zero in deep layers), exploding_tensor (NaN or Inf in any tensor), loss_not_decreasing (training stuck), overfit (validation loss diverging from training), poor_weight_initialization (weight distribution suggests bad init), class_imbalance (training data heavily skewed), dead_relu (large fraction of ReLU outputs zero), and saturated_activation (sigmoid/tanh outputs at extremes). Each rule emits a CloudWatch event when violated; events can trigger Lambda actions or stop the training job.

Debugger Insights - Hardware Profiling

Debugger Insights extends the rule system to system-level metrics — CPU utilisation, GPU utilisation, GPU memory usage, IO throughput, network bandwidth. Insights generates a detailed profiling report at job completion, identifying bottlenecks: low GPU utilisation indicates IO or CPU preprocessing starving the GPU; high CPU and low GPU indicates the data pipeline is the choke point; high IO wait time indicates slow S3 reads. Insights answers the "why is my training slow" question without manual instrumentation.

Debugger Output - The Tensor Storage Pattern

Captured tensors save to the configured S3 path. The default pattern saves at the start of every step (high IO cost) — production configurations limit captures to specific tensors and lower frequencies. The trial component (a sequence of tensors over training time) is the analysis unit; the SMDebug Python library reads trials and supports custom rule development.

Custom Rules

Beyond built-ins, engineers can write custom rules in Python using the SMDebug SDK. A custom rule subclasses Rule, implements invoke_at_step, and returns True when the violation condition is met. Custom rules are deployed as separate processing containers that run alongside the training job. Use cases: domain-specific anomaly detection (e.g., "loss flatlines for 100 steps then spikes" pattern), custom business-metric watching, integration with non-standard frameworks.

Enable SageMaker Debugger built-in rules on every long-running training job — the cost is minimal and the early-stop signal saves hours of wasted compute. Built-in rules like loss_not_decreasing, exploding_tensor, and overfit cost nothing extra to the training job (the rule runs in a separate small container) and emit CloudWatch events the moment the violation is detected. Wire those events to a Lambda that stops the training job and sends an SNS notification — the team learns within minutes that the run is broken instead of discovering it 6 hours later when the job completes with a NaN loss. The MLA-C01 exam tests this pattern as the standard "robust training" answer; not enabling Debugger is rarely the correct choice for production training.

SageMaker Clarify - Bias and Explainability

Clarify is two distinct services in one — bias measurement (fairness) and SHAP explainability — both running as SageMaker Processing jobs, both producing structured reports.

Pre-training Bias Metrics

Pre-training bias metrics analyse the training dataset before any model is fit. Common metrics include:

CI (Class Imbalance) - Difference between size of advantaged and disadvantaged groups.
DPL (Difference in Proportions of Labels) - Difference between positive label proportions across groups.
KL (Kullback-Leibler Divergence) - Distribution divergence between groups.
JS (Jensen-Shannon Divergence) - Symmetric variant of KL.
LP (Lp norm) - General distance metric between distributions.
TVD (Total Variation Distance) - Maximum difference in cumulative distributions.
KS (Kolmogorov-Smirnov) - Maximum CDF difference between groups.
CDDL (Conditional Demographic Disparity in Labels) - Subgroup-conditional disparity.

Pre-training bias signals issues that bias the model regardless of training algorithm — fix the data, not the model.

Post-training Bias Metrics

Post-training bias measures fairness of model predictions, not just labels. Key metrics:

DPPL (Difference in Positive Proportions of Predicted Labels) - DPL on predictions.
DI (Disparate Impact) - Ratio of positive prediction rates between groups; under 0.8 violates the four-fifths rule.
DCO (Difference in Conditional Outcomes) - Subgroup-conditional outcome difference.
AD (Accuracy Difference) - Difference in accuracy across groups.
RD (Recall Difference) - Difference in recall across groups.
CDDPL (Conditional Demographic Disparity in Predicted Labels) - Subgroup-conditional disparity in predictions.

Post-training bias signals that the model itself, not just the data, is producing unfair outcomes.

SHAP Explainability

Clarify computes Shapley values — the game-theoretic attribution of each feature's contribution to a prediction. Two views:

Global feature importance - Average absolute SHAP value across the dataset; tells you which features the model relies on overall.
Local explanation - SHAP values for a single prediction; tells you why this specific input got this specific output.

SHAP is model-agnostic — it works for tree models, deep nets, and ensembles alike. Clarify computes SHAP via Kernel SHAP (model-agnostic, slower) or TreeSHAP (tree-models only, much faster).

Clarify Processing Job Mechanics

Clarify runs as a SageMaker Processing job with a special Clarify container. The job takes a configuration JSON specifying the dataset, label column, sensitive attribute (the protected demographic for bias analysis), prediction column or model endpoint, and the metrics to compute. Output is a JSON report plus visualisations in the SageMaker Studio UI.

SageMaker Clarify is a dual-purpose service — bias measurement (pre-training and post-training) AND SHAP explainability — and both functions run as SageMaker Processing jobs that produce structured reports for governance review. Many candidates conflate Clarify with "fairness only" because that is the marketing tagline, but the SHAP explainability function is equally important on the exam. A stem asking "how do I explain why a specific prediction was made" expects Clarify (SHAP local explanation), not just bias-detection answers. A stem asking "how do I detect if my training data is imbalanced across protected groups" expects Clarify (CI or DPL pre-training metrics). Treat Clarify as the umbrella for both fairness and explainability and you will not lose points on this distinction.

SageMaker Experiments - Tracking Runs and Lineage

Experiments organises training runs into a hierarchy and tracks every parameter, metric, and artifact for reproducibility and comparison.

The Experiment Hierarchy

The data model has three levels. Experiment is the top container — typically one per business problem (e.g., "fraud-detection-v2"). Run (formerly "trial") is one execution within an experiment — one set of hyperparameters. Run components are individual training, processing, and evaluation jobs that compose a run. The hierarchy lets engineers compare multiple hyperparameter configurations within an experiment, and compare entire experiments against each other.

What Experiments Tracks

For each run: hyperparameters supplied to training, metrics emitted during training (typically pulled from CloudWatch), artifacts produced (model files, evaluation outputs), input datasets used, container images, and timing. The lineage is automatic — when a SageMaker Pipeline executes a sequence of training jobs, each becomes a run component with explicit upstream dependencies recorded.

Comparison Views in Studio

SageMaker Studio's Experiments view supports side-by-side comparison: select multiple runs, see hyperparameters in a parallel-coordinates chart, compare training/validation curves overlaid, view confusion matrices side by side. This is the analyst's primary tool for picking the best configuration from an HPO sweep.

Integration with HPO and Pipelines

Automatic Model Tuning (AMT) jobs automatically populate Experiments — every trial in a hyperparameter sweep is a run. SageMaker Pipelines execution similarly populates runs and run components, providing a complete audit trail from raw data through model registration.

Reproducibility via Lineage Tracking

SageMaker ML Lineage Tracking is the underlying lineage graph. Given a deployed model, you can trace backward to the run that produced it, the hyperparameters used, the dataset version, the preprocessing job, the raw S3 source. This audit trail is essential for regulated industries and for incident investigation when a deployed model misbehaves.

SageMaker Model Cards - Governance Documentation

Model cards are structured documentation accompanying a model into the registry, covering intended use, limitations, training data, evaluation results, and ethical considerations.

Required Model Card Sections

A standard SageMaker Model Card includes: model overview (architecture, input/output schema), intended use (business purpose, intended users), training details (algorithm, hyperparameters, dataset), evaluation results (metrics, confusion matrices, fairness measurements from Clarify), known limitations (failure modes, edge cases), ethical considerations, and additional information.

Why Model Cards Matter for MLA-C01

Model cards bridge the gap between technical artifacts and governance review. A risk officer or compliance auditor cannot read a Jupyter notebook; they can read a model card. The card travels with the model package in the registry, so deployment approvals can reference the documented limitations and evaluation results without diving into source code.

Auto-Population from Clarify and Experiments

Model cards can pull evaluation results from Clarify reports and metrics from Experiments runs, reducing manual documentation burden. The integration is not fully automated — engineers still write the prose sections — but the tabular evaluation data populates from the existing tracking systems.

Debugger vs Model Monitor - The Most-Tested Distinction

This is the highest-frequency confusion on the MLA-C01 exam. Internalise the distinction.

Debugger Lives in Domain 2 (Model Development)

Debugger watches training jobs in real time. Its rules look at training-internal state — gradients, activations, losses, hardware utilisation — to catch pathologies during the training run. Debugger output is consumed before the model is registered. If Debugger fires loss_not_decreasing, you stop the training job and re-tune.

Model Monitor Lives in Domain 4 (Monitoring/Maintenance)

Model Monitor watches deployed endpoints in production. Its monitors look at production data — request payloads, predicted outputs, ground truth labels arriving later — to catch drift over time. Model Monitor output is consumed after the model is in production. If Model Monitor fires data-quality drift, you trigger retraining.

Same Underlying Idea, Different Lifecycle Phase

Both tools detect anomalies, but the anomaly types differ. Debugger detects training-time problems (the model never converged, gradients exploded, GPU utilisation is 30 percent). Model Monitor detects production-time problems (the model converged fine but the world has changed and predictions are now drifting). The exam plants stems where the wrong tool is the wrong answer — "production model accuracy declining over weeks" is Model Monitor (Domain 4), not Debugger.

Debugger is a training-time tool (Domain 2 evaluation), Model Monitor is a production-time tool (Domain 4 monitoring) — they solve different problems at different lifecycle phases and the exam tests this distinction repeatedly. Debugger answers "is training healthy" — gradient issues, overfit, hardware bottlenecks while the training job is running. Model Monitor answers "is the deployed model still performing" — feature distribution drift, prediction drift, accuracy decay against ground truth in production. A stem describing "loss diverged during epoch 3" expects Debugger; a stem describing "endpoint accuracy fell from 92 to 78 percent over three months" expects Model Monitor. Choosing Model Monitor for a training-time symptom or Debugger for a production-time symptom is the most common community-cited mistake on the MLA-C01 evaluation domain.

Clarify vs Model Monitor - The Other Frequently Confused Pair

Clarify and Model Monitor also overlap in concept (both measure bias and explainability) but live in different lifecycle phases.

Clarify - Training-Time Bias and Explainability

Clarify runs as a Processing job during model development. It measures pre-training bias on the dataset and post-training bias on a candidate model's predictions on a held-out evaluation set. SHAP explainability runs on the same evaluation set. Clarify's output is consumed before the model is registered.

Model Monitor Bias Drift and Feature Attribution Drift

Model Monitor's bias drift monitor and feature attribution drift monitor are post-deployment versions of Clarify. They run continuously against production traffic, comparing current bias and SHAP distributions against a baseline computed at deployment time. When current diverges from baseline, alarm fires.

Why the Same Engine Powers Both

Both Clarify and Model Monitor's bias/attribution monitors use the same underlying Clarify processing engine. The difference is when and what they evaluate. Clarify is a one-time analysis at training; Model Monitor is a scheduled, continuous analysis at production.

Common MLA-C01 Exam Traps for Model Evaluation

The exam plants specific misconceptions. Recognise them.

Trap 1 - Accuracy on Imbalanced Data

A 99 percent accuracy on a 1 percent positive class is meaningless. F1 or AUC-PR is the right metric.

Trap 2 - RMSE vs MAE Without Considering Outliers

RMSE penalises large errors heavily. If outliers exist and are not fixable, MAE is more representative of typical performance.

Trap 3 - Debugger for Production Monitoring

Debugger is a training-time tool. Production monitoring is Model Monitor.

Trap 4 - Clarify Only Means Fairness

Clarify is fairness AND explainability. Both are tested.

Trap 5 - Disparate Impact Threshold

Disparate Impact below 0.8 (or above 1.25 inverse) violates the four-fifths rule under US EEOC guidelines. The exam may state this threshold.

Trap 6 - SHAP for Feature Selection

SHAP measures contribution to a specific model's predictions, not feature importance for the underlying problem. SHAP can show that the model relies heavily on a noisy feature; that does not mean the feature is genuinely informative.

Trap 7 - Experiments Tracks Code Versions Automatically

It does not. Code versioning is the engineer's responsibility (Git, S3 source URI). Experiments tracks hyperparameters and metrics, not source.

Trap 8 - Debugger Insights Replaces APM

Debugger Insights profiles training-job hardware. It does not replace application performance monitoring for endpoints; that is CloudWatch and X-Ray territory.

Trap 9 - Model Cards Are Optional for All Workflows

In regulated industries (finance, healthcare), model cards are increasingly mandated by internal governance. The exam frames them as best practice, not optional.

Trap 10 - Confusion Matrix Reading

Off-diagonal cells are the errors. Total off-diagonal = total errors. Reading the matrix wrong leads to wrong precision/recall calculations.

Decision Tree - Picking the Right Evaluation Tool

The exam loves "which tool" questions. Internalise this tree.

Symptom: Training loss not decreasing

Tool: SageMaker Debugger built-in rule loss_not_decreasing. CloudWatch event triggers Lambda to stop job.

Symptom: GPU utilisation is 30 percent during training

Tool: SageMaker Debugger Insights. The profiling report identifies whether IO, CPU preprocessing, or sync overhead is the bottleneck.

Symptom: Model performs worse on female applicants than male applicants

Tool: SageMaker Clarify post-training bias metrics (AD, RD, DPPL).

Symptom: Stakeholder asks why a specific loan was rejected

Tool: SageMaker Clarify SHAP local explanation.

Symptom: Need to compare 50 hyperparameter configurations from an HPO run

Tool: SageMaker Experiments comparison view in Studio.

Symptom: Auditor requests provenance of the deployed model

Tool: SageMaker ML Lineage Tracking via Experiments.

Symptom: Risk officer needs documentation of intended use and limitations

Tool: SageMaker Model Cards.

Symptom: Production model accuracy declining over months

Tool: SageMaker Model Monitor Model Quality (Domain 4, NOT Debugger).

Symptom: Training data has 95 percent positive class

Tool: SageMaker Clarify pre-training bias (CI metric) OR SageMaker Debugger class_imbalance rule.

Symptom: Production predictions becoming less explainable over time

Tool: SageMaker Model Monitor feature attribution drift (Domain 4).

FAQ - Model Evaluation, Debugger, and Clarify

Q1 - Which classification metric should I use for fraud detection where 0.5 percent of transactions are fraud?

Use F1 score or AUC-PR (precision-recall AUC), not accuracy. With 0.5 percent positives, accuracy of 99.5 percent is achievable by predicting "not fraud" for everything — useless. F1 balances precision (legitimate transactions wrongly flagged, which damages user experience) and recall (fraud not caught, which damages revenue). AUC-PR captures the precision-recall trade-off across thresholds, useful when the deployment threshold is not yet decided. If the business has a clear cost asymmetry — e.g., one undetected fraud is 100x worse than one false alarm — weight the metric accordingly: prefer recall-weighted F-beta score (F2 emphasises recall, F0.5 emphasises precision) over plain F1.

Q2 - What is the difference between SageMaker Debugger and SageMaker Model Monitor?

Debugger and Model Monitor solve anomaly detection at different lifecycle phases. Debugger runs during training jobs (Domain 2). It captures internal training tensors — weights, gradients, activations, losses — and applies built-in rules to detect training pathologies (vanishing gradients, overfit, exploding tensors, dead ReLU, GPU under-utilisation). Debugger output is consumed before the model is registered for deployment. Model Monitor runs against deployed endpoints (Domain 4). It captures production traffic — request payloads, predictions, ground truth labels arriving later — and compares against baselines to detect drift (data quality drift, model quality decay, bias drift, feature attribution drift). Model Monitor output triggers retraining workflows. The MLA-C01 exam tests this distinction explicitly: training-time symptoms use Debugger, production-time symptoms use Model Monitor. Mixing them up is the most-cited mistake on the evaluation domain.

Q3 - When should I use SageMaker Clarify vs Model Monitor for bias detection?

Use Clarify for one-time bias analysis during model development — pre-training (does the dataset itself encode bias) and post-training (does the trained model produce biased predictions on the held-out evaluation set). Clarify runs as a SageMaker Processing job, produces a structured JSON report, and is integrated into SageMaker Pipelines for automated bias evaluation before model registration. Use Model Monitor Bias Drift for continuous post-deployment monitoring — the deployed model's bias may shift over time as production data distribution changes. Model Monitor schedules the same underlying Clarify analysis on a recurring basis against production traffic, comparing current bias metrics to a baseline established at deployment. Both use the same Clarify computation engine; the distinction is one-time analysis (Clarify in Domain 2) vs continuous monitoring (Model Monitor in Domain 4).

Q4 - How do I explain to a stakeholder why my model gave a specific prediction?

Use SageMaker Clarify SHAP local explanation. SHAP (SHapley Additive exPlanations) computes the contribution of each feature to a specific prediction, accounting for feature interactions in a game-theoretic framework. For tree models (XGBoost, LightGBM, Random Forest), Clarify uses TreeSHAP — fast and exact. For deep learning or other black-box models, Clarify uses Kernel SHAP — model-agnostic but slower. The output is a per-feature contribution score showing which features pushed the prediction toward the predicted class and which pushed against. For global model behaviour, average absolute SHAP values across the evaluation set produces a global feature importance ranking. Both views answer the "why this prediction" question: local SHAP for individual decisions (compliance with GDPR right-to-explanation), global SHAP for overall model behaviour (feature engineering and debugging).

Q5 - My training job ran for 8 hours and produced a model with NaN loss in the final epoch. How do I prevent this in future runs?

Enable SageMaker Debugger built-in rules at the start of training. The relevant rules are exploding_tensor (detects NaN or Inf in any captured tensor), loss_not_decreasing (catches stuck training that would otherwise burn compute), and nan_loss (specific to loss going non-finite). Each rule emits a CloudWatch event when violated. Wire those events to a Lambda function that calls stop_training_job on the offending job and sends an SNS notification to the team. With this setup, the moment NaN appears, training stops within seconds — saving 7+ hours of wasted compute. The Debugger rule processing runs in a separate small container alongside the training job, costing pennies. There is no credible reason not to enable it on production training; the exam treats "robust training without Debugger rules" as suboptimal.

Q6 - I ran an HPO sweep with 100 trials. What is the best way to compare them?

Use SageMaker Experiments in SageMaker Studio. Every trial in a hyperparameter tuning job automatically becomes a run within an experiment, with hyperparameters and objective metric tracked. Open the experiment in Studio, select the runs to compare (you can select all 100 or filter to top-10 by objective metric), and use the comparison view: parallel-coordinates chart shows hyperparameters and their objective metric, line charts overlay training/validation curves, tabular view shows full configurations side by side. For deeper analysis, use the SageMaker Python SDK's ExperimentAnalytics class to load all trials into a pandas DataFrame for custom plotting and statistical analysis. Don't forget that AMT itself selects the best trial automatically based on the objective; Experiments is for the analyst's manual review of why specific configurations performed differently.

Q7 - What goes into a SageMaker Model Card and who reads it?

A complete Model Card has seven sections. Model Overview — name, version, owner, framework, intended use case. Intended Use — business purpose, who should use the model, who should NOT use it. Training Details — algorithm, training data source and statistics, hyperparameters. Evaluation Results — metrics (accuracy, precision, recall, F1, AUC, RMSE — whichever apply), confusion matrix, fairness measurements pulled from Clarify reports. Known Limitations — failure modes, edge cases, performance on subgroups, data drift sensitivity. Ethical Considerations — bias measurements, human oversight requirements, downstream impact. Additional Information — references, related documents, contact information. The audience is governance: risk officers, compliance auditors, downstream application owners, and incident responders investigating a production issue. The model card travels with the model package in the registry, so anyone reviewing a deployment can see the documented intended use and limitations without reading source code. For MLA-C01, recognise model cards as a deliverable of the evaluation phase, not just an afterthought before deployment.