Week 3 — Data & Data Cleaning

IAT 461 / 882 · Data Science for Human-Centered Systems · Summer 2026 . Alireza Karduni

What even is data?

Structured vs Unstructured Data

  • Structured — organized, fits neatly into rows and columns
  • Unstructured — no predefined format; text, images, audio, video
  • Most real-world data is unstructured
  • Data science often starts by imposing structure on unstructured data

Tabular Data

episode show character appearances

Tabular Data

episode show character appearances
EP01
EP02
EP03
EP04

Tabular Data

episode show character appearances
EP01 JJK Gojo 12
EP02 JJK Yuji 8
EP03 OP Luffy 11
EP04 OP Zoro 7

Tidy data is easy to filter, group, and plot. Messy data is not.

Wide Format

One row per subject. Each character gets their own column.

episode Gojo Yuji Luffy Zoro
EP01 12 8 0 0
EP02 4 15 0 0
EP03 0 0 11 7
EP04 0 0 5 14

Easy to read. Hard to analyze. Adding a new character = new column.

Long Format (Tidy)

One row per observation. Every measurement gets its own row.

episode show character appearances
EP01 Jujutsu Kaisen Gojo 12
EP01 Jujutsu Kaisen Yuji 8
EP02 Jujutsu Kaisen Gojo 4
EP03 One Piece Luffy 11
EP03 One Piece Zoro 7
EP04 One Piece Luffy 5

More rows, but now you can group_by("character"), filter("show"), plot anything.

Wide → Long in Python

import pandas as pd

wide = pd.DataFrame({
    "episode": ["EP01", "EP02", "EP03", "EP04"],
    "Gojo":    [12, 4, 0, 0],
    "Yuji":    [8, 15, 0, 0],
    "Luffy":   [0, 0, 11, 5],
    "Zoro":    [0, 0, 7, 14]
})

long = wide.melt(
    id_vars="episode",
    var_name="character",
    value_name="appearances"
)

long[long.appearances > 0]
   episode character  appearances
0     EP01      Gojo           12
1     EP02      Gojo            4
4     EP01      Yuji            8
5     EP02      Yuji           15
10    EP03     Luffy           11
11    EP04     Luffy            5
14    EP03      Zoro            7
15    EP04      Zoro           14

JSON

Flexible, nested, key-value pairs. Common in APIs and web data.

{
  "show": "Jujutsu Kaisen",
  "episode": "EP01",
  "appearances": [
    { "character": "Gojo", "count": 12 },
    { "character": "Yuji", "count": 8 }
  ]
}

Great for nested or hierarchical data. Not great for analysis directly — usually needs flattening first.

XML

Tags wrap content. Verbose but explicit.

<episode show="Jujutsu Kaisen" id="EP01">
  <character name="Gojo" appearances="12"/>
  <character name="Yuji"  appearances="8"/>
</episode>

HTML is just XML with a defined set of tags — your browser reads it the same way.

Markdown

Raw

# Jujutsu Kaisen

**Genre:** Supernatural action

Characters:
- Gojo Satoru
- Yuji Itadori

> "Throughout Heaven and Earth,
> I alone am the honoured one."

Rendered

Jujutsu Kaisen

Genre: Supernatural action

Characters:

  • Gojo Satoru
  • Yuji Itadori

“Throughout Heaven and Earth, I alone am the honoured one.”

Increasingly used as a data format for AI agents and LLM pipelines.

Unstructured Data

No rows, no columns. Just… stuff.

  • Text — emails, articles, subtitles, social media
  • Images — frames, thumbnails, scans
  • Audio — dialogue, soundtrack, sound effects
  • Video — all of the above, across time

Text — an email

from:    producer@toei.co.jp
to:      crew@toei.co.jp
date:    2025-02-03T09:14:00+09:00
subject: One Piece Episode 1123 — Final Cut
length:  187 characters
Team, the final cut for episode 1123 is ready for review.
Luffy's Gear 5 sequence runs 4m 32s.
Please flag any issues before Thursday.

Images — pixels all the way down

Every image is a matrix of numbers — one per channel (R, G, B).

Audio — a waveform

Audio is a sequence of amplitude samples over time — typically 44,100 per second.

Languages & Tools

  • Python — the bread-and-butter of data science
  • R — the statistician’s playground
  • Excel / Google Sheets — still matters

Python 🐍

  • pandas — load, clean, reshape tabular data
  • numpy — fast math on arrays
  • matplotlib / seaborn — visualization
  • scipy.stats — statistical tests
  • scikit-learn — machine learning

Python — loading data

import pandas as pd
import numpy as np

# Simulate a small checkins dataset
df = pd.DataFrame({
    "user_id":      [101, 204, 101, 317, 420],
    "gym_id":       [3, 1, 3, 2, 1],
    "checkin_time": ["2024-01-08 07:32", "2024-01-08 08:15",
                     "2024-01-09 07:28", "2024-01-09 12:04",
                     "2024-01-09 18:45"],
    "duration_min": [45, 60, 50, 30, 75]
})

df.head()
   user_id  gym_id      checkin_time  duration_min
0      101       3  2024-01-08 07:32            45
1      204       1  2024-01-08 08:15            60
2      101       3  2024-01-09 07:28            50
3      317       2  2024-01-09 12:04            30
4      420       1  2024-01-09 18:45            75

Python — wrangling

df.groupby("user_id")["duration_min"].agg(
    visits="count",
    avg_duration="mean"
).sort_values("visits", ascending=False)
         visits  avg_duration
user_id                      
101           2          47.5
204           1          60.0
317           1          30.0
420           1          75.0

python - visualization

import matplotlib.pyplot as plt
plt.bar(df["gym_id"], df["duration_min"])
plt.show()

python - Seaborn

import seaborn as sns
sns.barplot(x="gym_id", y="duration_min", data=df)
plt.show()

R 📊

  • dplyr — filter, mutate, group_by, summarise
  • tidyr — reshaping data (wide ↔︎ long)
  • ggplot2 — layered visualizations
  • readr — fast CSV loading

You can call R from Python via rpy2 — best of both worlds.

R — dplyr pipeline

library(dplyr)

checkins <- data.frame(
  user_id      = c(101, 204, 101, 317, 420, 204, 101),
  duration_min = c(45, 60, 50, 30, 75, 55, 40)
)

checkins %>%
  group_by(user_id) %>%
  summarise(
    visits       = n(),
    avg_duration = mean(duration_min)
  ) %>%
  arrange(desc(visits))
# A tibble: 4 × 3
  user_id visits avg_duration
    <dbl>  <int>        <dbl>
1     101      3         45  
2     204      2         57.5
3     317      1         30  
4     420      1         75

R — ggplot2

library(ggplot2)

checkins$hour <- c(7, 8, 7, 12, 18, 9, 7)

ggplot(checkins, aes(x = hour, fill = factor(user_id))) +
  geom_histogram(binwidth = 1, color = "white") +
  labs(x = "Hour of day", y = "Check-ins", fill = "User") +
  theme_minimal()

Excel & Google Sheets

  • Fastest way to eyeball a new dataset
  • Good for small manual corrections
  • Sorting, filtering, pivot tables — no code needed
  • Google Sheets makes sharing instant
  • Always good to look at raw data if it’s not too big.

Notebook Environments

The deliverable of a data science project should be a notebook — not just code, not just results, not just a report.

  • Code
  • Data
  • Computational results
  • Written analysis

All in one place. All runnable.

Why it matters

  • Reproducible — run it again, get the same result
  • Tweakable — change a parameter, rerun, see what changes
  • Documented — text and visuals live next to the code

Reproducibility

A bad pipeline looks like this:

  1. Load raw data
  2. Do some processing in code
  3. Edit the file by hand
  4. Do more processing

Now try running it again on a new dataset. Good luck.

Jupyter in practice

import pandas as pd

df = pd.DataFrame({
    "user_id": [101, 204, 317],
    "visits":  [24, 19, 12],
    "avg_min": [47.5, 62.1, 38.4]
})

df.describe()
          user_id     visits    avg_min
count    3.000000   3.000000   3.000000
mean   207.333333  18.333333  49.333333
std    108.038573   6.027714  11.955891
min    101.000000  12.000000  38.400000
25%    152.500000  15.500000  42.950000
50%    204.000000  19.000000  47.500000
75%    260.500000  21.500000  54.800000
max    317.000000  24.000000  62.100000

Code, output, and your interpretation — all in the same document.

Collecting Data

Where Does Data Come From?

  • Open / government data — data.gov, census, city open data portals
  • Kaggle — datasets, competitions, notebooks
  • Academic repositories — Google Dataset Search, UCI ML Repository, Harvard Dataverse
  • Domain-specific — IMDB, MyAnimeList, OpenStreetMap, NOAA weather

Proprietary Data

  • Companies like Google, Meta, Netflix have enormous datasets
  • Almost never fully public — business risk, privacy risk
  • What you can get: APIs — controlled, rate-limited access

An API (Application Programming Interface) is a structured way to request data from a service programmatically.

API Example — OpenStreetMap

import requests

url = "https://nominatim.openstreetmap.org/search"
params = {
    "q": "Vancouver, BC",
    "format": "json"
}
r = requests.get(url, params=params)
r.json()[0]
{
  "place_id": 298123,
  "display_name": "Vancouver, BC",
  "lat": "49.2827291",
  "lon": "-123.1207375",
  "type": "city"
}

Twitter API — The Old Days

What researchers and developers could do for free:

  • Streaming API — live 1% sample of all tweets in real time
  • Search API — query recent tweets by keyword, hashtag, user
  • Historical API — access to years of past tweets
  • Used for: sentiment analysis, event detection, social network research

Twitter API — What Changed


Tier Price Tweet reads/month
Free $0 ~100
Basic $200/month 15,000
Pro $5,000/month 1,000,000
Enterprise ~$42,000/month custom

Before 2023 — all of this was free.

Scraping

When there’s no API — you go get the data yourself.

  • Spidering — downloading the right pages
  • Scraping — extracting the content you need
  • Pages are just HTML — structured text your browser interprets

The Lazy Way — Ask an LLM

Paste raw HTML into an LLM and ask it to extract structure:

Here is an HTML table from a webpage. 
Convert it to a clean JSON array with fields: 
rank, title, score, episodes.

<table class="ranking-table">
  <tr><td>1</td><td>Sousou no Frieren</td>
  <td>9.08</td><td>28</td></tr>
  ...
</table>

Works great for small tables. Not practical at scale — token costs add up fast.

BeautifulSoup — The Real Way

import requests
from bs4 import BeautifulSoup

url = "https://myanimelist.net/topanime.php"
headers = {"User-Agent": "Mozilla/5.0"}

r = requests.get(url, headers=headers)
soup = BeautifulSoup(r.text, "html.parser")

titles = soup.select(".anime_ranking_h3 a")

for t in titles[:10]:
    print(t.text.strip())

Once you know the HTML structure, scraping is just finding the right selector.
You can ask an LLM to generate this code for any page — just show it the HTML.

Logging — Capturing Behavior

If you own the system — instrument it.

document.addEventListener("click", (e) => {
  const event = {
    type:      "click",
    target:    e.target.tagName,
    id:        e.target.id,
    timestamp: new Date().toISOString(),
    x:         e.clientX,
    y:         e.clientY
  };
  fetch("/log", {
    method: "POST",
    body:   JSON.stringify(event)
  });
});
{ "type": "click", "target": "BUTTON",
  "id": "play-btn", "timestamp": "2025-01-08T14:32:01Z",
  "x": 412, "y": 309 }

Logging — The Physical World

Cheap hardware, real data. Log everything — storage is nearly free.

Data Cleaning

“Garbage in, garbage out.”

  • Raw data is almost never ready to analyze
  • Cleaning is not optional — bad data produces confident wrong answers
  • Always clean on a copy of the original

Errors vs Artifacts

  • Error — data that is permanently lost or unrecoverable
    • A sensor goes offline for 3 hours
    • A server crashes and logs are gone
  • Artifact — a systematic distortion introduced by processing
    • A scoring system changes how it counts
    • A platform policy change triggers mass user behavior

Artifacts can be fixed. Errors cannot.

The Sniff Test

  • Look at your data before you model it
  • Ask: does this look like what I’d expect?
  • Surprises are either insights or artifacts — figure out which

Most surprises turn out to be artifacts.

Artifact Example — Helldivers 2

In May 2024, Sony announced players would need a PSN account to play.

The game didn’t get worse. The data did.

Outlier Detection

Not all bad data is an artifact. Sometimes one value is just wrong.

47 reviews from players with 0 hours. Do they count?

Outlier Detection

How do we find values that don’t belong?

  • Plot the distribution first — look at your data
  • Use IQR to find values far from the middle
  • Use Z-scores to find values far from the mean
  • Then ask why — don’t just delete

Hours Played — the data

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "game": [
        "Hollow Knight", "Celeste", "Hades",
        "Dead Cells", "Cuphead", "Shovel Knight",
        "Ori", "Axiom Verge", "Balatro", "Slay the Spire 2"
    ],
    "hours_played": [42, 18, 55, 31, 12, 24, 15, 20, 847, 1203]
})

df
               game  hours_played
0     Hollow Knight            42
1           Celeste            18
2             Hades            55
3        Dead Cells            31
4           Cuphead            12
5     Shovel Knight            24
6               Ori            15
7       Axiom Verge            20
8           Balatro           847
9  Slay the Spire 2          1203

Visualizing the Distribution

Most games cluster under 60 hours. Two are way out in the right tail.

IQR — Interquartile Range

Q1 = df["hours_played"].quantile(0.25)
Q3 = df["hours_played"].quantile(0.75)
IQR = Q3 - Q1

lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

print(f"Q1: {Q1}, Q3: {Q3}, IQR: {IQR}")
Q1: 18.5, Q3: 51.75, IQR: 33.25
print(f"Lower bound: {lower:.1f}")
Lower bound: -31.4
print(f"Upper bound: {upper:.1f}")
Upper bound: 101.6
df[df["hours_played"] > upper]
               game  hours_played
8           Balatro           847
9  Slay the Spire 2          1203

IQR — Visual

Anything beyond Q3 + 1.5 × IQR is flagged. Robust to skewed distributions.

Boxplot

Z-Score

from scipy import stats

df["zscore"] = stats.zscore(df["hours_played"])
df[["game", "hours_played", "zscore"]].round(2)
               game  hours_played  zscore
0     Hollow Knight            42   -0.45
1           Celeste            18   -0.51
2             Hades            55   -0.42
3        Dead Cells            31   -0.48
4           Cuphead            12   -0.53
5     Shovel Knight            24   -0.50
6               Ori            15   -0.52
7       Axiom Verge            20   -0.51
8           Balatro           847    1.52
9  Slay the Spire 2          1203    2.40

Z-Score — Visual

|z| > 2 is a common threshold. Sensitive to the outliers themselves — use IQR when data is skewed.

So — outlier or artifact?

  • Balatro and StS2 have 847 and 1203 hours
  • Both flagged by IQR and Z-score
  • But the reason is simple — the game was left running
  • Removing them blindly would be wrong without checking why

Always investigate before you delete.

Missing Values

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "user_id":      [1, 2, 3, 4, 5],
    "age":          [23, np.nan, 31, np.nan, 28],
    "sessions":     [12, 8, np.nan, 15, 6],
    "avg_duration": [45.0, 30.0, 55.0, np.nan, 40.0]
})

df
   user_id   age  sessions  avg_duration
0        1  23.0      12.0          45.0
1        2   NaN       8.0          30.0
2        3  31.0       NaN          55.0
3        4   NaN      15.0           NaN
4        5  28.0       6.0          40.0

Option 1 — Drop rows with missing values

df.dropna()
   user_id   age  sessions  avg_duration
0        1  23.0      12.0          45.0
4        5  28.0       6.0          40.0

Simple. But you lose data — fine if missing values are rare and random.

Option 2 — Fill with the mean

df.fillna(df.mean(numeric_only=True))
   user_id        age  sessions  avg_duration
0        1  23.000000     12.00          45.0
1        2  27.333333      8.00          30.0
2        3  31.000000     10.25          55.0
3        4  27.333333     15.00          42.5
4        5  28.000000      6.00          40.0

Preserves row count. Doesn’t bias the mean. Can hide variance.

Option 3 — Forward fill

df.ffill()
   user_id   age  sessions  avg_duration
0        1  23.0      12.0          45.0
1        2  23.0       8.0          30.0
2        3  31.0       8.0          55.0
3        4  31.0      15.0          55.0
4        5  28.0       6.0          40.0

Useful for time series — assumes the last known value still holds.

Data Compatibility — Name Matching

import pandas as pd

games = pd.DataFrame({
    "title": ["Elden Ring", "elden ring", "Elden Ring™", "ELDEN RING"]
})

# Normalize
games["clean"] = (
    games["title"]
    .str.lower()
    .str.replace(r"[™®]", "", regex=True)
    .str.strip()
)

games
         title       clean
0   Elden Ring  elden ring
1   elden ring  elden ring
2  Elden Ring™  elden ring
3   ELDEN RING  elden ring

Same game, four different strings. Always normalize before joining.

Data Compatibility — Time Zones

import pandas as pd

events = pd.DataFrame({
    "user":      ["Tokyo", "Vancouver", "London"],
    "timestamp": [
        "2024-03-15 22:00:00+09:00",
        "2024-03-15 06:00:00-08:00",
        "2024-03-15 14:00:00+00:00"
    ]
})

events["utc"] = pd.to_datetime(
    events["timestamp"], utc=True
)

events[["user", "utc"]]
        user                       utc
0      Tokyo 2024-03-15 13:00:00+00:00
1  Vancouver 2024-03-15 14:00:00+00:00
2     London 2024-03-15 14:00:00+00:00

Three users. Same moment in time. Always convert to UTC before comparing.

Data cleaning is most of a data scientist’s job