Guide

Dimensionality reduction and PCA explained

A dataset with two hundred sensor columns sounds rich until you realize ninety of them move together, forty are nearly constant noise, and training a classifier on the full matrix overfits on day one. Dimensionality reduction compresses many correlated features into fewer coordinates that preserve the structure you care about: variance for modeling, neighborhoods for visualization, or latent semantics for embeddings. Principal component analysis (PCA) is the workhorse: a linear transform that finds orthogonal axes ranked by how much variance each explains. This guide walks through PCA step-by-step, shows how to choose the number of components, contrasts linear PCA with nonlinear methods (t-SNE, UMAP, autoencoders), connects reduction to clustering and feature engineering, and lists the scaling and leakage traps that turn a scree plot into a production incident.

Why high dimensions hurt

The curse of dimensionality is not abstract math — it shows up in every over-parameterized pipeline. As feature count grows relative to sample size, distances between points become less meaningful (most pairs look equally far apart), decision boundaries need exponentially more data to generalize, and models memorize noise that cross-validation cannot catch if the same leakage appears in every fold.

Reduction helps when:

Features are redundant — income, log-income, and income-per-household encode overlapping signal.
Visualization is the goal — you need a 2D scatter of ten-thousand-dimensional embeddings.
Downstream cost matters — fewer inputs speed tree ensembles and shrink memory for edge deployment.
Noise dominates — many weak columns dilute the signal a supervised model can learn.

Reduction hurts when rare but predictive dimensions get projected away — a fraud flag that appears in only 0.1% of rows may live in a low-variance component you discard. Always validate downstream task performance, not just explained-variance curves.

PCA step-by-step

Given a centered data matrix X (n rows, p columns), PCA finds orthonormal directions v₁, v₂, ... such that projecting onto v₁ captures maximum variance, v₂ captures the next most (orthogonal to v₁), and so on. Each direction is a principal component; the coordinates along them are the scores.

Center (and usually scale) features. Subtract column means. If units differ wildly (age in years vs income in dollars), standardize to zero mean and unit variance — otherwise PCA chases the largest-scale column.
Compute the covariance matrix (or SVD directly on X). For p features, this is a p × p symmetric matrix of pairwise covariances.
Eigendecompose (or SVD). Eigenvectors are principal axes; eigenvalues λ_i equal variance along component i.
Rank components by descending λ. Component 1 explains λ₁ / Σλ total variance.
Project X onto the top k eigenvectors: X_reduced = X · V_k.

In practice you call sklearn.decomposition.PCA or equivalent — libraries use SVD for numerical stability rather than forming the full covariance matrix when n ≪ p (common in genomics and text).

Explained variance ratio for k components: (Σ_i=1..k λ_i) / (Σ_i=1..p λ_i). A scree plot (eigenvalue vs component index) and cumulative variance curve guide how many k to keep — there is no universal magic threshold.

Choosing how many components

Common heuristics, each with trade-offs:

Elbow on scree plot — keep components before the curve flattens. Subjective but fast.
95% cumulative variance — popular default for preprocessing; may keep noise in high-p datasets.
Kaiser rule — retain components with eigenvalue > 1 when data were standardized. Crude for correlated business features.
Cross-validated reconstruction error — pick k that minimizes held-out MSE when reconstructing X from scores. Ties to autoencoder thinking.
Downstream validation — sweep k, train your classifier or clusterer, pick k with best out-of-fold metric. The only criterion that matters for production.

For visualization only, k = 2 or 3 is fixed — you accept that most variance may be lost. Label the plot with cumulative variance (e.g. "PC1 + PC2 = 43% variance") so readers do not over-interpret separation that might live in discarded components.

Interpreting loadings and biplots

Each principal component is a weighted sum of original features. The loadings (coefficients in v_i) show which columns drive that axis. A PC1 dominated by purchase_amount, session_duration, and page_views might be interpreted as "engagement intensity" — but interpretation is hypothesis-generating, not proof.

A biplot overlays loading vectors on a score scatter: arrows point in the direction where that feature increases most on the 2D plane. Useful for explaining clusters to stakeholders; dangerous when only two components explain a minority of variance.

Sign ambiguity: flipping all loadings on a component by −1 is equivalent. Compare components across runs on the same fitted pipeline, not across refits with different random seeds or row order.

Scaling, sparse data, and PCA variants

PCA is sensitive to preprocessing — document every step in your ML pipeline:

StandardScaler vs none — unscaled PCA on mixed units is meaningless; over-scaled binary flags can dominate.
Sparse high-dimensional text — TF-IDF matrices often use TruncatedSVD (LSA) instead of dense PCA; it works on sparse inputs without densifying.
Kernel PCA — nonlinear mapping via kernels before eigendecomposition; captures curved manifolds but costs O(n²) and is harder to deploy.
Incremental PCA — batch updates for data that do not fit in RAM; critical for streaming logs.
Randomized SVD — approximate top-k components fast when p is huge (image pixels, embedding tables).

Fit scalers and PCA on training data only, then transform validation and inference rows — same train-serve parity rules as any feature transform.

Nonlinear reduction: t-SNE, UMAP, and autoencoders

PCA preserves global linear variance. When clusters sit on a curved manifold (Swiss roll, word embeddings), linear projection fails. Nonlinear methods target local neighborhood structure:

t-SNE — converts pairwise distances to probabilities, minimizes KL divergence in 2D/3D. Beautiful clusters; distances between clusters are not meaningful; O(n²) memory; different seeds give different layouts.
UMAP — graph-based, often faster, better preserves global structure than t-SNE; hyperparameters (n_neighbors, min_dist) change topology; popular for single-cell and embedding dashboards.
Autoencoders — neural network bottleneck learns nonlinear compression; reconstruction loss replaces explained variance; pairs naturally with deep learning stacks and anomaly detection (high reconstruction error = outlier).

Use t-SNE/UMAP for exploration and slides, not as fixed preprocessing fed into a production classifier — the transform is non-invertible, stochastic, and expensive to refit on every batch.

PCA in production workflows

Typical patterns where PCA earns its keep:

Clustering preprocess — run K-means on top-k PCA scores instead of raw fifty-column CRM data; reduces distance metric weirdness.
Multicollinearity removal — linear regression with correlated inputs; PCA or partial least squares stabilizes coefficients.
Image and signal compression — eigenfaces, PCA on spectrograms; lossy but fast.
Exploratory dashboards — PC1/PC2 scatter colored by churn label before investing in a full model.
Whitening before contrastive training — decorrelate augmented batches in some vision pipelines (less common with modern batch norm).

Serialize the fitted PCA object (or ONNX equivalent) alongside the scaler. Monitor explained variance on live batches — if drift shifts covariance structure, stale components silently degrade model quality.

Method comparison table

Method	Linear?	Best for	Production?	Main risk
PCA / TruncatedSVD	Yes	Preprocessing, compression, collinearity	Yes — fast transform	Discards low-variance but predictive features
Kernel PCA	No (kernel)	Curved manifolds, small n	Rare — costly	O(n²), kernel choice arbitrary
t-SNE	No	2D visualization	No — exploratory only	Cluster distance misleading
UMAP	No	Visualization, some graph ML	Limited — refit cost	Hyperparameter sensitivity
Autoencoder	No	Nonlinear compression, anomalies	Yes — with GPU budget	Training complexity, collapse modes

Common mistakes

PCA on raw unscaled mixed units — dollar columns dominate age columns.
Fit on full dataset including test — leakage inflates downstream metrics.
Trust 2D t-SNE cluster gaps as ground truth — confirm with supervised metrics or domain labels.
Too few samples vs dimensions — with n < p, covariance is singular; use TruncatedSVD or regularization.
Discarding components before checking task AUC — variance ≠ predictive power.
Reusing PCA from one population on another — covariance shift breaks loadings; refit or use domain adaptation.

Production checklist

Document centering/scaling — same sklearn Pipeline at train and serve.
Pick k via downstream cross-validated metric, not scree plot alone.
Report cumulative explained variance when showing 2D plots.
Use TruncatedSVD for sparse text/count matrices.
Prefer PCA over t-SNE/UMAP for featurization in production.
Version and store the fitted reducer with model artifacts.
Monitor input covariance drift on live traffic.
Sanity-check loadings with domain experts before naming components.
Compare against baseline without reduction — simpler often wins.
Watch n vs p — regularize or collect more rows before aggressive compression.

Key takeaways

PCA finds orthogonal axes of maximum variance — linear, fast, and deployable.
Scaling and train-only fitting are non-negotiable — otherwise components reflect leakage, not signal.
Explained variance guides compression; task metrics guide science — optimize what you ship.
t-SNE and UMAP are for eyes, not pipelines — beautiful plots, fragile geometry.
Autoencoders extend reduction to nonlinear bottlenecks — at the cost of training and ops complexity.