Week 8 - Simple ideas behind unsupervised learning

IAT 461 / 882 · Data Science for Human-Centered Systems · Summer 2026 · Alireza Karduni

Unsupervised Learning

Or how sometimes we don’t have labeled data but we still need to find structure in the data

The Scenario

  • Phigma’s leadership wants to know: which AI models should we pay for?
  • You’re handed a spreadsheet of benchmark scores for a dozen models
  • Reasoning, coding, math, safety, latency, cost, multilingual… the list keeps growing
  • There’s no “correct answer” column — nobody labeled which model is “the best”

A Concrete Starting Point

Model Reasoning Coding
Solis-3 88 85
Solis-3 Mini 84 80
Solis-3 Turbo 90 87
Nimbus Pro 60 92
Nimbus Lite 55 88
Nimbus Max 63 95
Drift-2 45 40
Drift-2 XL 50 45
Drift-1 38 35
  • Nine fictional models, scored on two benchmarks
  • Just by reading the numbers — can you already guess which models are similar?
  • Let’s plot it and see

Plotting the Models

  • Each point is a model
  • Three groups jump out immediately
  • But can we find these groups or clusters automatically when the data is larger?

Too Many Benchmarks, Too Many Models

  • With 2 benchmarks, we just plotted and looked
  • With 15 benchmarks, we can’t visualize anything anymore
  • Some models might be near-twins — scoring almost identically everywhere
  • Today’s tools: group similar models automatically (clustering), and compress many benchmarks into a few meaningful axes (dimensionality reduction)

A Larger Dataset

K-means Clustering

K-means: Starting Point

  • We pick K = 3 and drop 3 random centroids (black X’s)
  • Every model starts unassigned — shown in grey

K-means: Assigning Points

  • Each point is assigned to its nearest centroid

K-means: Updating Centroids

  • Each centroid moves to the average position of its assigned points

K-means: Re-assigning (Iteration 2)

  • Centroids moved, so points near the boundary can switch colors

K-means: Updating Centroids (Iteration 2)

K-means: Re-assigning (Iteration 3)

  • Assignments barely change now

K-means: Converged

  • Centroids stopped moving — this is convergence
  • K-means stops iterating once assignments no longer change

K = 5: Starting Point

  • What if we change K? Let’s try K = 5
  • Now we drop 5 random centroids instead of 3

K = 5: Assigning Points

  • Every point joins its nearest of the 5 centroids

K = 5: Updating Centroids

K = 5: Re-assigning (Iteration 2)

  • Some of the natural families are starting to split

K = 5: Converged

  • 5 clusters, fully converged
  • Notice some colors are splitting what looked like one natural group

K = 3 vs. K = 5

  • Same models, same algorithm — only K differs
  • K = 3 matches the natural model families
  • K = 5 forces extra splits that don’t correspond to anything real
  • K is a hyperparameter — the algorithm doesn’t choose it for us

K-means in with more dimensions

  • What happens when we add more dimensions?
  • We can still use K-means to find clusters, but now we have to visualize in more dimensions

Cluster Profiles

reasoning coding safety
Cluster 0 (orange) 59.9 94.0 87.9
Cluster 1 (blue) 45.7 41.2 52.3
Cluster 2 (green) 86.4 79.9 66.1
Overall average 64.0 71.7 68.8
  • Each row is a cluster’s average score on the original benchmarks
  • Compare each cluster to the overall average — what stands out?
  • Cluster 1 (blue) is high on coding and safety, more moderate on reasoning
  • Cluster 0 (orange) leads on reasoning and coding both
  • Cluster 2 (green) trails on all three

Cluster Profiles: Z-Scores

  • Z-score = how many standard deviations a cluster is above or below the overall average
  • A bar near 0 means that cluster is typical on that benchmark
  • Tall bars (positive or negative) are what make a cluster distinct
  • Cluster 0 (orange): clearly above average on reasoning and coding
  • Cluster 2 (green): below average across the board

How Do We Choose K?

  • Inertia: how tightly packed points are around their centroid — always drops as K increases
  • We’re looking for the elbow — where adding more clusters stops helping much
  • Here, that’s around K = 3
  • Past that point, we’re just splitting real groups apart for a small improvement

How Do We Choose K? Silhouette Score

  • Measures how well-separated clusters are — closer to 1 is better
  • Peaks at K = 2, but is K = 2 the best number?
  • Neither method is perfect, they’re guides

Back to Feature Engineering

Showing just the first few models and benchmarks:

model reasoning coding safety math
0 Solis-3 88.1 87.7 72.4 89.0
1 Solis-3 Mini 91.1 84.8 71.4 89.7
2 Solis-3 Turbo 86.3 82.0 70.8 88.7
3 Nimbus Pro 61.8 91.5 88.5 65.8
4 Nimbus Lite 60.7 92.5 92.2 62.4
5 Nimbus Max 62.8 93.6 91.3 65.8
6 Drift-2 44.5 41.6 54.1 42.0
7 Drift-2 XL 47.2 41.2 54.6 43.3
8 Drift-1 43.0 42.5 52.5 43.1
  • Remember this idea from feature engineering — every model is now a vector of 10 benchmarks
  • We can’t plot 10 dimensions
  • But K-means doesn’t care — it just measures distance, no matter how many dimensions that distance is computed across

K-means in 10 Dimensions

km_hd = KMeans(n_clusters=3, random_state=0, n_init=10).fit(X_hd)
df_hd["cluster"] = km_hd.labels_

counts = df_hd.cluster.value_counts().sort_index()
counts.index = [f"Cluster {i}" for i in counts.index]
counts
Cluster 0    15
Cluster 1    15
Cluster 2    15
Name: count, dtype: int64
  • We ran K-means exactly as before — same .fit() call, just a wider matrix
  • We get cluster labels back, but no picture
  • So how do we know what each cluster means?

Cluster Profiles: One at a Time

  • Orange bars: above the overall average on that benchmark
  • Blue bars: below the overall average
  • Cluster 0: strong multilingual, safety, instruction-following — the reliable assistant group
  • Cluster 1: strong cost efficiency and latency, weak everywhere else — the cheap and fast group
  • Cluster 2: strong reasoning, math, coding — but expensive and slow

DBSCAN: eps as a Hyperparameter

  • eps controls the neighborhood radius — how close two points need to be to be considered neighbors
  • Large eps: almost everything gets absorbed into one cluster, no noise
  • Small eps: more points fall outside any dense region and become noise (grey)
  • Unlike K-means, DBSCAN never forces a point into a cluster — it can say “this point doesn’t belong anywhere”

HDBSCAN

  • HDBSCAN replaces eps with min_cluster_size — how many points a cluster must have to survive
  • Much easier to tune: larger values = more noise points, fewer but tighter clusters
  • Automatically handles clusters of varying density — no fixed radius assumption
  • More robust than DBSCAN across different datasets

From Clustering to Dimensionality Reduction

  • With 2 benchmarks we can plot directly
  • With 3 we can still rotate a 3D chart
  • With 10 benchmarks — we’re stuck
  • Can we find a way to compress many benchmarks into just a few axes without losing too much information?

Principal Component Analysis

  • We start with our 3D benchmark dataset — reasoning, coding, safety
  • PCA will find new axes that capture the most variation in this cloud of points
  • We will walk through each step slowly

Step 1: The Original Data

  • Each point is a model scored on two benchmarks
  • The black X marks the mean — the center of gravity of the whole dataset
  • Before PCA, we move this center to the origin (0, 0)

Step 1: Centering

  • Both panels share the same axis range so the shift is visible
  • Left: original cloud, mean marked with X
  • Right: same cloud, subtracted the mean — now centered at (0, 0)
  • The shape of the cloud is identical — only its position changed

Step 1: Scaling

  • Centering moves the cloud — scaling changes its spread
  • After scaling, each feature has the same standard deviation
  • Without this, benchmarks with larger raw values would dominate the PCA result
  • Now both axes are on equal footing

Step 2: Find the Direction of Most Variation

  • PCA finds the direction along which the data spreads the most — this is PC1
  • PC2 is the direction of second-most variation, constrained to be perpendicular to PC1
  • Each PC is a linear combination of the original benchmarks

PC1: The Direction of Most Spread

  • Each grey line drops a point perpendicularly onto PC1 (blue arrow)
  • The dark dots are where each model lands on PC1
  • PC1 is chosen to maximize the spread of those projected dots
  • Right panel: PC1 projections (blue) are much more spread out than PC2 projections (orange)
  • More spread = more information retained along that axis

What Does Each PC Represent?

  • Each bar is a loading — how much that benchmark contributes to this PC
  • Orange: positive contribution
  • Blue: negative contribution
  • Tall bars tell you what this PC is really measuring
  • What would you name PC1 based on its tallest bars?

Step 4: Project onto the New Axes

  • Each model is now represented by its position on PC1 and PC2 instead of the original 3 benchmarks
  • We went from 3 dimensions down to 2 — without losing much information
  • The relative positions of models are preserved — similar models are still close together
  • Named models already cluster into their families visually

What Does Each PC Represent?

  • Each bar is a loading — how much that benchmark contributes to this PC
  • Orange: positive contribution — higher score pushes a model further along this PC
  • Blue: negative contribution — higher score pushes a model the other way
  • Tall bars tell you what this PC is really measuring
  • What would you name PC1 based on its tallest bars?

Reading the Loadings

  • PC1 loads strongly on all three — it separates overall high-performing models from low-performing ones
  • PC2 contrasts safety against reasoning and coding — models that are safe but not strong technically vs the reverse
  • PC3 captures whatever variation is left after PC1 and PC2
  • PCA didn’t select one benchmark — it found combined patterns across all of them

High-Dimensional Clusters in PCA Space

  • Clusters were found in 10 dimensions — but we can’t see 10 dimensions
  • PCA compresses those 10 benchmarks down to 2 axes for visualization
  • Colors are the K-means cluster labels — the algorithm never saw this 2D view
  • The three families separate cleanly even after compression
  • This is PCA’s most practical use: making invisible structure visible

UMAP

  • UMAP preserves both local and global structure — nearby points stay near, far points stay far
  • Clusters tend to be tight and well-separated
  • Axes are not interpretable — UMAP 1 and UMAP 2 don’t map back to benchmarks
  • Fast enough to run on large datasets

t-SNE

  • t-SNE focuses on preserving local structure — it’s very good at keeping tight clusters tight
  • Global distances are not reliable — two clusters being far apart on screen means nothing
  • Sensitive to the perplexity hyperparameter — results can look very different across runs
  • Slower than UMAP on large datasets

PCA vs. UMAP vs. t-SNE

  • Same data, different way to reduce dimensionality.
  • PCA: linear, axes are interpretable, preserves global structure
  • UMAP: non-linear, tighter clusters, global structure still meaningful
  • t-SNE: non-linear, sharpest local clusters, global distances not meaningful