Testing & Evaluation

Run local evaluation jobs against image and text dataset splits, and store the resulting metrics and curve artifacts for comparison.

Evaluation jobs

• Comparison runs — select multiple models over the same dataset to create a batch comparison. Testing groups them under a single comparison_id, so models stack side by side and the trade-offs are readable.
• Task-aware metrics — the UI chooses which metrics to surface based on the task:
  • Classification — accuracy, F1, MCC, and AUC, paired with a confusion matrix.
  • Detection and segmentation — pixel Dice, object IoU matching, precision, and recall.
  • NLP — exact match, ROUGE, and related scores.

Per-item inspection

Aggregate numbers hide the interesting cases. The per-item table renders predictions row by row:

• Ground-truth classes against predicted classes, with raw accuracy scores.
• For vision: label overlap, pixel metrics, and object matching summaries per image.
• For NLP: original text, reference answer, and model prediction in adjacent columns.

Reading a result honestly

Two habits the app is built around:

1. Score on a split the model never saw. A version freeze plus a recorded split is what makes that claim checkable later.
2. Read precision and recall separately. A single headline metric can improve while the failure mode you care about gets worse.