Machine learning — classic/tabular models, done without lying to yourself

Tabular ML is easy to run and easy to fool yourself with. The deliverable is never "the notebook printed 0.99" — it is an honest estimate of how the model behaves on data it has never seen: a Pipeline that fits every transform on train only, a metric that survives class imbalance, and a DummyClassifier baseline it beats. A number you can't reproduce on a sacred test set you touched exactly once isn't a result — it's a leak you haven't found yet.

Is this the right skill? (decide first)

Your situation	Reach for
Rows of features, predict a column, with trees / linear models / sklearn	machine-learning (this skill)
Images, audio, long text, sequences, or you need a neural net / PyTorch	deep-learning
The table is still dirty (nulls, dupes, mixed types, bad dates)	data-cleaning first — it hands you a validated table
Text/token classification, NER, tokenization, LLM-adjacent NLP metrics	nlp (a TF-IDF + linear/GBDT baseline still lives happily in this skill's pipeline)
KPIs, dashboards, "explain the business"	analytics / business-intelligence
Forecast a dated series forward (revenue next quarter)	forecasting
Building a training corpus of JSONL messages / preference pairs for an LLM	training-data

This skill starts at a clean, validated table (rows × features + a target) and ends at a fitted, honestly-scored model with a test-set number and a baseline it beats. Cleaning is upstream — consume the validated frame data-cleaning produced; don't re-teach it here.

Version reality (verify at author time — this line moves monthly)

Verified 2026-07: scikit-learn current major ~1.9 (1.9.0 shipped 2026-06-02, Python 3.11–3.14) — do NOT pin from memory; check the current stable at scikit-learn.org, the 1.x line ships every few months. GBDTs: XGBoost 3.x and LightGBM 4.x (xgboost 3.3, lightgbm 4.6 current), plus sklearn's own HistGradientBoostingClassifier/...Regressor — a fast native GBDT that eats NaN and (with categorical_features="from_dtype") categoricals with no preprocessing. Pin what you ship (python owns the environment and the pinning); state versions as "~X (verify)", never as frozen fact.

The one rule everything else serves: fit on train only

Every preprocessing step — imputation, scaling, encoding, feature selection, target encoding, resampling — learns parameters from data. Learn them from rows the model is later scored on and the score inflates while production underperforms: that is leakage, the #1 way tabular ML lies. The whole apparatus below — Pipeline, ColumnTransformer, CV, the untouched test set — exists to make "fit on train only" automatic instead of something you remember to do by hand (you won't).

scikit-learn: estimators, Pipeline, ColumnTransformer

Every model is an estimator with the same contract: fit(X, y), then predict(X) / predict_proba(X) (classifiers) / score(X, y). Transformers add transform(X) / fit_transform(X, y). A Pipeline chains transformers + a final estimator into one estimator — so fit fits every step on train, and predict/CV transforms test data with parameters learned on train. That is the leakage guard.

A ColumnTransformer routes different columns down different transformer branches (scale the numerics, encode the categoricals) and stitches the result back together — all still inside the pipeline.

from sklearn.compose import ColumnTransformer, make_column_selector as mcs
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression

numeric = Pipeline([("impute", SimpleImputer(strategy="median")),
                    ("scale",  StandardScaler())])           # scaling matters for LINEAR/SVM/KNN
categoric = Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                      ("onehot", OneHotEncoder(handle_unknown="ignore"))])  # unseen category -> all-zeros, no crash

pre = ColumnTransformer([
    ("num", numeric,   mcs(dtype_include="number")),
    ("cat", categoric, mcs(dtype_include=["object", "category"])),
], remainder="drop")

model = Pipeline([("pre", pre), ("clf", LogisticRegression(max_iter=1000, class_weight="balanced"))])
model.fit(X_train, y_train)          # imputers/scaler/encoder ALL fit on X_train only
model.predict_proba(X_test)          # X_test transformed with train-learned params — no leak

Two load-bearing details: OneHotEncoder(handle_unknown="ignore") so a category unseen in train doesn't crash prediction, and remainder="drop" so unrouted columns don't silently leak through raw; model.set_output(transform="pandas") keeps named-column DataFrames through the pipeline.

Trees skip most of this — scaling is pointless for tree models, and HistGradientBoostingClassifier ingests NaN and categoricals natively, so its pipeline is often just the estimator. Preprocess for the model that needs it, not as a ritual. More patterns (make_column_transformer, FunctionTransformer) → references/pipelines-and-cv.md.

GBDTs are your default on tabular data

Gradient-boosted decision trees are the correct first (and usually last) model for tabular problems. This isn't taste: Grinsztajn, Oyallon & Varoquaux (2022), "Why do tree-based models still outperform deep learning on tabular data?" (arXiv:2207.08815) benchmarked across 45 datasets and found GBDTs beat tuned neural nets on medium-sized tabular data, tracing it to three inductive biases NNs lack: robustness to uninformative features, not being rotationally invariant (so they exploit the meaning of individual columns), and ease of learning irregular / non-smooth target functions. Start with a GBDT; reach for deep-learning on tabular only with a specific reason.

Library	Import	Reach for it when
HistGradientBoostingClassifier	sklearn.ensemble	Default. Fast, zero extra deps, native NaN + categorical.
XGBoost 3.x	xgboost.XGBClassifier	Battle-tested; early_stopping_rounds, rich regularization.
LightGBM 4.x	lightgbm.LGBMClassifier	Fastest on wide/large data; leaf-wise; strong native categoricals.

All three expose the sklearn estimator API, so they drop into the pipeline and CV below unchanged. Don't agonize over XGBoost-vs-LightGBM before you have a baseline and a leak-free CV — split discipline dwarfs the library choice. Tuning knobs (learning_rate, num_leaves/max_depth, early stopping, monotonic_cst) and importance/SHAP → references/gbdt-and-tuning.md.

Leakage-safe feature engineering

Feature engineering is where leakage sneaks back in after the pipeline "protected" you. Rules:

• Any transform that learns from data goes INSIDE the pipeline, so CV re-fits it per fold. A scaler fit on the whole dataset, a SelectKBest run before splitting, an imputer using the global mean — each leaks test statistics into train. The common_pitfalls canonical example: SelectKBest(k=25).fit_transform(X, y) before train_test_split produces a beautiful, meaningless score.
• Target/mean encoding of high-cardinality categoricals must cross-fit. sklearn's preprocessing.TargetEncoder does this: its fit_transform(X, y) uses an internal cross-fitting scheme so each row is encoded from other folds' targets — fit(X, y).transform(X) deliberately differs and would leak. Use fit_transform on train (inside the pipeline); never hand-roll a group-mean encoder.
• No target-derived features. A column computed from the label (or a near-proxy: "was_refunded" when predicting "will_refund") is leakage wearing a feature's clothes. If a feature is impossibly predictive, suspect it.
• Respect time. With any temporal structure, a feature may use only information available at prediction time — no future aggregates, no lifetime values that include post-cutoff rows. Split by time (TimeSeriesSplit), not randomly.

Split + cross-validation: protect the sacred test set

Hold out a test set once, at the very start (train_test_split(..., stratify=y, random_state=0)) and do not look at it until you have a single final model. Every glance — tuning, feature choice, "let me just check" — bleeds information and re-inflates the estimate. Tune and compare with cross-validation on the training portion only; the test set is the one honest number at the end (see the lifecycle below). Score with cross_validate(pipeline, X_tr, y_tr, cv=cv, scoring=[...], return_train_score=True) — a large train-minus-test gap is your overfitting alarm.

Pick the splitter to match the data (cross_validation docs):

• StratifiedKFold — default for classification; preserves class balance per fold (essential when imbalanced). KFold for regression.
• TimeSeriesSplit — any time ordering. Trains on past, tests on future; never shuffles the future into train. A random KFold on time-series data is leakage.
• GroupKFold / StratifiedGroupKFold — when rows cluster (same user/patient/store across many rows). Keep a group entirely in train or test, or the model memorizes the group and CV lies.
• Pass integer random_state to splitters for reproducible folds. Put preprocessing in the pipeline so CV re-fits it every fold — cross_validate(pipeline, ...), never cross_validate(model, X_scaled, ...).

For tuning, wrap CV in GridSearchCV / RandomizedSearchCV / HalvingRandomSearchCV; for an unbiased estimate of the tuning process itself, use nested CV → references/pipelines-and-cv.md.

Metrics: the accuracy trap and what to use instead

Accuracy lies on imbalanced data. At 99% negatives, a model that predicts "negative" always scores 99% accuracy and is worthless. Choose the metric for the task and the cost of each error type (model_evaluation docs):

Task / question	Metric (sklearn.metrics)	scoring string
Ranking quality, threshold-free, balanced-ish	roc_auc_score	"roc_auc"
Imbalanced ranking (rare positive: fraud, disease)	average_precision_score (PR-AUC)	"average_precision"
Cost of false positives high (don't cry wolf)	precision_score	"precision"
Cost of misses high (don't miss a case)	recall_score	"recall"
Balance both, per-class fairness	f1_score (use f1_macro multiclass)	"f1" / "f1_macro"
Multiclass, care about every class equally	balanced_accuracy_score	"balanced_accuracy"
See the actual error breakdown	confusion_matrix, classification_report	—
Regression	r2_score, mean_absolute_error, root_mean_squared_error	"r2", "neg_mean_absolute_error", "neg_root_mean_squared_error"

Prefer PR-AUC (average_precision) to ROC-AUC when positives are rare — ROC-AUC can look great while precision is dismal, since it ignores the negative flood. Feed AUC metrics predict_proba, not hard labels. The default 0.5 threshold is a choice: tune it on validation to hit your precision/recall target. Regression uses root_mean_squared_error now (mean_squared_error(squared=False) is gone). Threshold tuning, calibration, class_weight/resampling → references/metrics-and-imbalance.md.

Anti-patterns — the cardinal sins

Anti-pattern	Do instead
Modeling straight off the raw, dirty table	This skill starts at a clean, validated frame. Run data-cleaning first — nulls, dupes and mixed dtypes are its job, not a modeling problem.
Scaling / encoding / selecting features, then splitting	Leakage (#1 sin). Test statistics are now in train. Split first; put every learned transform inside the Pipeline so CV re-fits per fold.
Celebrating an amazingly predictive feature	Suspect target leakage — a column derived from the label or unavailable at prediction time. Audit provenance before you celebrate.
Reporting 99% accuracy on 1% positives	The accuracy trap. A constant predictor matches it. Report PR-AUC / precision / recall / F1 and a confusion_matrix.
SMOTE-ing the whole dataset because the classes are imbalanced	Resampling before the split, or on the test fold, leaks and evaluates on synthetic rows. Resample inside CV, on the train fold only (imblearn Pipeline), or just use class_weight="balanced".
Shipping on the CV score alone	You never touched a held-out test set, or you peeked at it while tuning. One final, untouched test number — or the estimate is optimistic.
Random KFold on time-series / multi-user data	Future or same-group rows leak into train. Use TimeSeriesSplit / GroupKFold.
Waving off a big train/test gap	Overfitting. Regularize, reduce capacity (max_depth, min_samples_leaf), get more data, or use early stopping. Watch return_train_score.
Going straight to XGBoost with no baseline	Without a DummyClassifier(strategy="most_frequent") / DummyRegressor floor (and a simple linear model), you can't tell if the fancy model adds anything.
Grid-searching 10k combos over all the data	Tuning against the test set is fitting to it. Tune with CV on train, confirm once on test; consider nested CV.
Unset random_state, unpinned versions	Folds and fits stop being reproducible and re-trains stop being comparable. Set an integer random_state on splitters and estimators; pin the versions you ship.

Worked lifecycle (end to end)

from sklearn.dummy import DummyClassifier
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score
from sklearn.metrics import average_precision_score, classification_report

X_tr, X_test, y_tr, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

# 0. BASELINE first — the floor every real model must clear
base = DummyClassifier(strategy="most_frequent")
print("baseline PR-AUC:", cross_val_score(base, X_tr, y_tr, cv=cv, scoring="average_precision").mean())

# 1. default GBDT (native NaN + categoricals -> minimal pipeline). class_weight for imbalance.
clf = HistGradientBoostingClassifier(categorical_features="from_dtype",
                                     class_weight="balanced", random_state=0)
cv_pr = cross_val_score(clf, X_tr, y_tr, cv=cv, scoring="average_precision")
print("model CV PR-AUC:", cv_pr.mean().round(3), "+/-", cv_pr.std().round(3))

# 2. it clears the baseline -> commit, fit on all train, judge ONCE on the sacred test set
clf.fit(X_tr, y_tr)
proba = clf.predict_proba(X_test)[:, 1]
print("TEST PR-AUC:", round(average_precision_score(y_test, proba), 3))
print(classification_report(y_test, (proba >= 0.5).astype(int)))   # threshold is a choice — tune it

Project grounding (02-DOCS + CLAUDE.md)

In a project with a 02-DOCS/ layer (the harness wiki), record the modeling contract in 02-DOCS/wiki/ml/<target>.md, linked from the root CLAUDE.md ## Knowledge map: target definition, split strategy + random_state, CV scheme, chosen metric and why, baseline, pinned versions, and the dated final test-set score. Read it first on every re-train so results stay comparable. No 02-DOCS/? Skip silently — conventions are recorded, never gated.