IAT 461 · Data Science for Human-Centered Systems
Was the model right? It depends what you mean by right.
Imagine a tool that reads Twitch chat and classifies each message as abusive or fine. Before we judge the model, you'll try moderating yourself — then watch how human labels become training data, and how no classifier ever copies them perfectly.
Scroll down. Label a few messages first, then follow the diagrams as each idea comes into view.
How would you classify this message?
Try classifying messages yourself
Before any model enters the picture, someone has to decide what counts as abuse. On the left you're on moderation duty: read each message and click Flag as abusive or Allow as fine. There's no answer key yet — just your judgment.
Take your time on the borderline ones. Trash-talk, sarcasm, and context all matter. If two moderators can disagree, imagine training a machine from their labels.
What you just did is how models get built
When you finish, we'll show your calls next to labels from a panel of human reviewers — the same kind of labels used to train a moderation model. The model's job is to reproduce those human decisions on new messages it has never seen.
Abusive — and the model labeled it abusive. A clean catch.
Now watch a model sort the same kinds of message
You've seen how subjective moderation can be. Evaluation asks a sharper question: given a set of reviewer labels we treat as ground truth, how often does the model's call match?
Step through the messages on the left with Next. Each one flows through a small diagram: input (the message) → model (abusive or fine) → outcome (compared to the reviewer label).
Four boxes, and only four
Keep clicking. Notice that every message lands in one of just four boxes: the model can be right in two ways (caught abuse, correctly fine) and wrong in two ways (false alarm, missed abuse).
The same fine-but-labeled-abusive trash-talk and the same hostile-but-missed jab are the kinds of mistakes that decide whether a moderation tool is actually any good.
And it doesn't matter whether the model is a giant language model, a list of banned words, or a coin flip. From the outside it does one thing: it sorts messages into predicted abusive and predicted fine.
Truth on top, the model's call on the bottom
Here's the move that makes evaluation click. Put the truth at the top — abusive on one side, fine on the other. Put the model's classification at the bottom — predicted abusive on one side, predicted fine on the other. Then let every message flow from where it really belongs to where the model sent it.
Messages that flow straight down were judged correctly. Messages that cross over are mistakes. That's the entire story — and it's exactly what the next diagram shows.
Watch the messages sort themselves
Each dot is a message leaving its true group at the top and landing in the bucket the model chose at the bottom. The width of every ribbon is just how many messages took that route.
Two ribbons run nearly straight down — those are the messages the model got right. Two ribbons cross through the middle — those are the two ways it can be wrong.
The only four things that can happen
Cross “what's true” with “what the model said” and you get exactly four outcomes:
- Caught abuse — abusive, and labeled abusive. (True Positive)
- Missed abuse — abusive, but labeled fine. (False Negative)
- False alarm — fine, but labeled abusive anyway. (False Positive)
- Correctly fine — fine, and labeled fine. (True Negative)
Every number you're about to meet is built from just these four counts.
Accuracy: how often, overall, was it right?
The simplest score. Add up everything the model got right — the two straight-down ribbons — and divide by all the messages.
accuracy = (caught + correctly fine) / everything
Here that's a healthy-looking 84%. But hold that number lightly — it quietly mixes two very different kinds of success, and it can hide a lot.
Precision: when it predicts abusive, can you trust it?
Now look only at the predicted abusive bucket — both ribbons feeding it. Some of those were real abuse; some were false alarms on perfectly fine messages.
precision = caught abuse / everything predicted abusive
Precision asks: of everything the model labeled abusive, what fraction deserved it? Low precision means lots of false alarms — innocent users getting muted.
Recall: how much of the abuse did it catch?
Now look only at the actually abusive group at the top — both ribbons leaving it. Some got caught; some slipped through.
recall = caught abuse / all real abuse
Recall asks: of all the abuse that truly existed, what fraction did we catch? Low recall means lots of missed abuse — toxic messages reaching the channel.
Why a 95%-accurate model can be useless
Suppose abuse is rare — say 5 messages in every 100. A lazy model that never labels anything abusive is right about the other 95 every time. That's 95% accuracy while catching exactly zero abusive messages.
Accuracy got fooled because it rewards the easy, common case. Precision and recall don't — they keep their eyes on the abuse, which is the thing we actually care about. That's why you almost never report accuracy alone.
Build your own scenarios
Drag the sliders or pick a preset. The Sankey and confusion matrix stay in sync — tap a metric on either side to see which cells and ribbons it uses.
The whole idea, step by step
- 1Moderation starts with human judgment — and people disagree on borderline messages.
- 2Models learn to reproduce human labels, not some perfect objective truth, and they never match perfectly.
- 3Evaluation compares a model's calls to those labels: abusive or fine × predicted abusive or predicted fine → four outcomes.
- 4Accuracy is the share it got right overall — but it's easily fooled when one class is rare.
- 5Precision asks: when it predicts abusive, is that trustworthy? (watch the predicted-abusive bucket).
- 6Recall asks: of all real abuse, how much did we catch? (watch the abusive group).
- 7You can't max out both at once — choosing the balance is a values call, not a formula.