Week 4 — Models

IAT 461 / 882 · Data Science for Human-Centered Systems · Summer 2026 · Alireza Karduni

Models

Full Metal Alchemist

The Transmutation Circle

  • In Fullmetal Alchemist, alchemists draw a transmutation circle
  • It encodes rules and assumptions about how matter should transform
  • Add materials. Activate. The alchemy executes — precisely, faithfully
  • No judgment. No wisdom. Just the circle.

The Circle Does Exactly What You Drew

  • The construct that emerges is entirely determined by the circle and the materials
  • A flawed circle → a flawed construct
  • The alchemy doesn’t know what you meant — only what you wrote
  • Extraordinary power. Zero discretion.

Data Scientists Draw Circles Too

We call ours models.

What Does a Model Actually Do?

A model takes data and does something useful with it:

  • Summarize — describe patterns in the data
  • Explain — understand why something happens
  • Predict — forecast unseen outcomes
  • Classify — assign categories to new observations

Same general structure every time: inputs → model → output

Equivalent Exchange

  • Whatever wisdom comes out was in the data and assumptions you put in
  • The model doesn’t generate insight from nothing
  • Wrong assumptions → wrong output, faithfully computed
  • Garbage in, garbage out — is just equivalent exchange

Hypotheses and Models Are Not the Same Thing

  • A hypothesis is a verbal conjecture:
    “Screen time harms wellbeing”
  • It is vague. Untestable as stated.
  • A model is a precise, executable construct
  • You need a model to test a hypothesis — but they are not the same thing

Hypotheses and Models Are Not the Same Thing

Hypotheses and Models Are Not the Same Thing

Hypotheses and Models Are Not the Same Thing

Hypotheses and Models Are Not the Same Thing

Hypotheses and Models Are Not the Same Thing

  • The same statistical model can be consistent with completely different hypotheses
  • Rejecting a model does not reject the hypothesis
  • Confirming a model does not confirm the hypothesis
  • The mapping runs in both directions — many-to-many

Failing to reject the null doesn’t mean the null is true

The Implication: Stay Humble

  • Evidence tests a model, not a hypothesis directly
  • Statistical models are lenses, not verdicts
  • Always ask: what other hypotheses is my model consistent with?

All Models Are Wrong

“All models are wrong, but some are useful.”
— George Box

  • No model is the world — all are simplifications
  • The question is never “is this model true?”
  • The question is: “is it useful — and does it know its own limits?”
  • A model confident beyond its design is the most dangerous kind

A Taxonomy of Models

Models Come in Many Flavors

  • Not all models do the same thing
  • Before choosing a model, you need to know what you’re asking it to do
  • Two key questions:
    • What is the goal — explain or predict?
    • Where does the structure come from — theory or data?

The Modeling Space

The Modeling Space

The Modeling Space

The Modeling Space

Explanatory vs. Predictive

  • Explanatory — why does something happen?
    • We care about the relationship between variables
    • Interpretability matters
    • Example: does streaming duration cause viewer drop-off?
  • Predictive — what comes next?
    • We care about accuracy on unseen data
    • The mechanism can be a black box
    • Example: which sessions will end in the next 5 minutes?

The Same Question, Two Models

Same phenomenon. Different questions. Different models.

Structure: Theory-Driven vs. Data-Driven

  • Theory-driven — structure comes from domain knowledge
    • You encode assumptions about how the world works
    • Interpretable by design, but brittle outside its assumptions
    • Example: an SIR model assumes people move through Susceptible → Infected → Recovered
  • Data-driven — structure is learned from data
    • The model finds patterns without being told what to look for
    • Flexible and powerful, but can fail silently
    • Example: a neural net predicting engagement from 200 features

Deterministic vs. Stochastic

  • Deterministic — same input always gives the same output
    • Example: F = ma — given force and mass, acceleration is fixed
  • Stochastic — output is a probability distribution over possible outcomes
    • Example: 70% chance of rain tomorrow
  • Demanding a single point prediction from an uncertain process destroys information

Why Stochastic Models Matter

A model that says “I’m 70% sure” is more useful than one forced to say “yes” or “no.”

Classifier vs. Regressor

The output type determines everything downstream.

  • Classifier — output is a category
    • Is this comment toxic? → Yes / No
    • Which genre does this stream belong to? → Gaming / IRL / Music
  • Regressor — output is a number
    • How many viewers will tune in? → 4,200
    • How toxic is this comment? → 0.73
  • Different output types → different designs, different failure modes, different evaluation metrics

Classifier vs. Regressor

Black Box vs. Interpretable

  • Interpretable — you can read off why the model made a decision
    • Linear regression: each coefficient has a clear meaning
    • Decision tree: follow the branches
    • Random forests:
  • Black box — the model works, but you can’t easily explain how
    • Neural networks: millions of weights, no clear narrative
  • This matters enormously in human-centered systems — can you explain the decision to the person it affects?

Choosing a Model Type

  • There is no universally best model — the right choice depends on:
    • What is the goal? Explain or predict?
    • How much data do you have? Theory fills the gap when data is scarce
    • Does uncertainty matter? Use stochastic models when it does
    • Will someone be affected by the decision? Interpretability may be non-negotiable
  • The taxonomy is a thinking tool, not a checklist

Evaluating Models

Did the Model Actually Work?

  • Building a model is only half the job
  • The harder question: how do we know if it worked?
  • The answer depends entirely on what we asked the model to do
  • Evaluation is not one-size-fits-all

Two Worlds of Evaluation

Before You Run Any Stats: Define the Question

  • Every evaluation starts before the model, not after
  • You need to define:
    • What is the construct you are measuring?
    • How is it operationalized in the data?
    • What would a meaningful result look like?
  • Running a model without this is like measuring with a broken ruler

The Evaluation Chain

Skipping any step contaminates everything that follows.

A Concrete Example

Question: Does increased Twitch stream length lead to more viewers dropping off?

  • Construct: viewer engagement — but what counts as drop-off?
    • Leaving mid-stream? Not returning next session? Both?
  • Operationalization: average viewer retention rate per stream, binned by duration
  • Model: linear regression — retention ~ duration
  • Interpretation: a negative coefficient means longer streams lose viewers on average — but how much matters, and for whom?

Model Fit: How Well Does the Model Explain the Data?

  • For explanatory models, the first question is: how much of the variation does the model account for?
  • We measure this with R² — the coefficient of determination
  • R² = 1 means the model explains all variation in the outcome
  • R² = 0 means the model explains nothing
  • R² = 0.6 means 60% of the variation in the outcome is explained by the predictors

Visualizing Model Fit

The residuals (orange dashes) are the gaps between what the model predicts and what actually happened. Smaller residuals → better fit.

Fit Is Not Enough

  • A model can fit training data perfectly and still be useless
  • If the model is too complex, it memorizes the data instead of learning from it
  • Perfect fit on known data + poor performance on new data = overfitting
  • This is why explanatory fit metrics alone are not sufficient
  • We need a separate strategy for evaluating generalization — how the model performs on data it has never seen

That is what the ML evaluation framework is built to solve.

Evaluating ML Models: Classification

The Core Problem

  • Explanatory models ask: does this fit the data I have?
  • ML models ask something harder: does this work on data I haven’t seen yet?
  • A model that memorizes training data looks perfect — but is useless
  • We need a principled way to test generalization

The Train / Test Split

  • Training set — the model learns from this
  • Test set — held back, never touched during training
  • Performance on the test set tells you how the model does on unseen data

Why Keep a Test Set?

  • A student who memorizes last year’s exam answers will score 100%
  • That tells you nothing about whether they learned
  • The test set is the exam with new questions
  • If you peek at test data during training, you’ve spoiled the exam
  • This is the most common mistake in applied ML

Train / Test / Validation — Three Roles

The validation set is for iterating. The test set is sacred — use it once, at the very end.

Accuracy — The Obvious Metric

  • The simplest question: what fraction of predictions were correct?

\[\text{Accuracy} = \frac{\text{correct predictions}}{\text{total predictions}}\]

  • Easy to compute, easy to understand
  • Example: model predicts viewer will return tomorrow — correct 82 times out of 100 → accuracy = 82%

Why Accuracy Can Lie

  • Imagine a rare disease that affects 1% of the population
  • A model that always predicts “healthy” — no matter what — gets 99% accuracy
  • Is that a good model?
  • It has never correctly identified a single sick person
  • Accuracy is misleading whenever classes are imbalanced

The Confusion Matrix — Axes

True Positives & True Negatives

These are the wins — the model got it right in both directions.

False Positives

A False Positive is a false alarm — the model said positive, but it was wrong. Also called a Type I error.

False Negatives

A False Negative is a missed case — the model said negative, but it was wrong. Also called a Type II error.

The Confusion Matrix — With Numbers

Disease screening example: 1000 patients, disease prevalence 1%

Accuracy = 990/1000 = 99% — but the model found zero sick patients. Every sick person was missed.

Precision — Don’t Cry Wolf

  • Of everything the model predicted as positive, how many actually were?

\[\text{Precision} = \frac{TP}{TP + FP}\]

  • High precision → few false alarms
  • When false alarms are costly: spam filters, content moderation
  • Example: flagging a professor’s email as spam has a real cost

Recall — Don’t Miss Anyone

  • Of everything that was actually positive, how many did the model catch?

\[\text{Recall} = \frac{TP}{TP + FN}\]

  • High recall → few missed cases
  • When missing a case is catastrophic: disease screening, fraud detection
  • Example: missing a cancer diagnosis has a far higher cost than a false alarm

The Precision–Recall Trade-Off

Improving one tends to hurt the other. Which matters more depends entirely on the cost of each type of error in your domain.

F1 Score — Balancing Both

  • Sometimes you need one number that respects both precision and recall
  • The F1 score is their harmonic mean:

\[F_1 = 2 \cdot \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}\]

  • Harmonic mean punishes extreme imbalance — a model with precision=1.0 and recall=0.0 gets F1=0
  • Use F1 when both false positives and false negatives matter
  • Use precision or recall alone when one error type clearly dominates

Evaluating ML Models: Regression

When the Output Is a Number

  • Classification asks: which category?
  • Regression asks: how much?
    • How many viewers will this stream get?
    • How long will a user session last?
    • What will the engagement score be?
  • We can’t use accuracy — there’s no “right or wrong”, only how far off we were
  • We need metrics that measure error

The Residual — The Basic Unit of Error

Residual = Actual − Predicted. Positive means we underestimated. Negative means we overestimated.

Mean Squared Error (MSE)

  • Average all the residuals — but they cancel out (positives and negatives balance)
  • So we square each residual first, then average

\[MSE = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2\]

  • Squaring does two things: removes the sign, and punishes large errors more
  • A single bad prediction can dominate the MSE

Visualizing MSE — Squared Residuals

The area of each orange square is the squared error for that prediction. MSE is the average of all those areas.

Root Mean Squared Error (RMSE)

  • MSE is in squared units — hard to interpret directly
  • If viewers are in thousands, MSE is in thousands squared
  • Take the square root to get back to the original units

\[RMSE = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2}\]

  • RMSE = 8k means predictions are off by roughly 8,000 viewers on average
  • Same scale as the data — directly interpretable
  • Still sensitive to large errors because of the squaring step

Mean Absolute Error (MAE)

  • Instead of squaring, just take the absolute value of each residual

\[MAE = \frac{1}{n}\sum_{i=1}^{n}|y_i - \hat{y}_i|\]

  • Every error counts equally — no extra punishment for large ones
  • More robust when your data has outliers
  • Easier to explain: MAE = 8k means predictions are off by 8,000 viewers on average

MSE vs MAE — The Outlier Problem

One large outlier barely moves MAE — but explodes RMSE. Choose based on whether outliers represent real signal or noise.

Choosing the Right Regression Metric

Metric Penalizes outliers? Units Use when…
MAE No Same as data Outliers are noise; you want robustness
MSE Yes (heavily) Squared Large errors are especially costly
RMSE Yes Same as data Large errors matter; you want interpretability

There is no universally best metric. The choice depends on what kinds of errors hurt most in your domain.