Data Science 101
Data Science 101
Sub-chapter 1 of AI/ML · The boring tools that win more than people admit
Before you reach for an LLM, ask: would a logistic regression on a clean CSV do this? The answer is yes more often than the AI hype cycle suggests. This sub-chapter is the toolbox you should reach for first.
The instinct we're building: a strong baseline beats a clever model. A baseline you can explain in one sentence - "I'm predicting churn by training a gradient boosting model on these eight features" - is worth ten times a black box you can't.
Outline
- What "data science" actually means - the loop, not the title
- Frame the problem first - what are you actually predicting / measuring
- The classical model toolbox - linear, tree-based, clustering, dimensionality reduction
- Feature engineering - the part everyone wants to skip
- Train / validation / test - and why a leaky split kills products
- Evaluation metrics - pick the wrong one and you optimise for the wrong thing
- When NOT to use ML at all - heuristics, rules, and SQL queries that win
- When NOT to use an LLM - sub-second classification, tabular data, audited decisions
101 Primer
The data science loop
question → data → exploration → baseline → iterate → ship → monitor
If you skip "exploration," you ship bugs disguised as models. If you skip "baseline," you have nothing to compare against. If you skip "monitor," you have a model rotting in production while everyone congratulates themselves on the launch.
Frame the problem
A useful problem statement is one sentence:
"Given the customer's last 30 days of activity, predict the probability they churn in the next 30 days, so we can route the top 5% to a retention specialist."
That sentence tells you:
- Inputs - last 30 days of activity (you need a feature pipeline)
- Output - probability ∈ [0,1] (binary classification with calibrated probabilities)
- Decision - top-5% routing (you only need ranking quality at the head of the distribution)
- Constraint - must be sub-day latency, run nightly is fine
That last bullet is what justifies not using a 70B-parameter LLM.
The classical toolbox
Most tabular problems are well-served by one of these:
| Family | When | Examples |
|---|---|---|
| Linear / logistic regression | Strong baseline; you want interpretability; small data | Churn baseline, A/B test analysis |
| Tree ensembles (XGBoost, LightGBM, CatBoost) | Tabular structured data, mixed feature types | Fraud, ranking, churn, default models |
| K-Means, DBSCAN | Unsupervised grouping | Customer segments, cohort discovery |
| PCA, UMAP, t-SNE | Dimensionality reduction, visualisation | Embed-then-plot for sanity checks |
| ARIMA / Prophet | Time-series with seasonality | Forecasting weekly demand |
| Bayesian models (PyMC) | Small data, want uncertainty quantified | Pricing experiments, sequential testing |
The 80% answer for tabular Welzin work is LightGBM with sensible features and proper cross-validation. Start there.
Feature engineering
This is what separates "I ran sklearn" from "I built a model." Engineering features means asking what the model actually needs to see - not just dumping every column.
- Aggregate - "logins in last 7 days" beats "raw login log."
- Decompose time - day-of-week, hour-of-day, days-since-signup.
- Encode categoricals carefully - high-cardinality columns kill one-hot; use target encoding or embeddings.
- Handle missing-ness explicitly -
is_email_verified_missingis a feature. - Domain-rate-of-change features - slopes, deltas, ratios.
You'll spend 60% of your time on this. That's normal.
Train / validation / test, properly
from sklearn.model_selection import train_test_split
# DON'T do this on time-series data:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
The block above is fine for IID data. For anything time-ordered (churn, fraud, forecasting), random splitting leaks the future into training and your model will look magical in dev and embarrassing in prod.
Use time-based splits:
# Train on data up to 2026-04-30, validate on May, test on June.
train = df[df.date <= "2026-04-30"]
val = df[(df.date > "2026-04-30") & (df.date <= "2026-05-31")]
test = df[df.date > "2026-05-31"]
Other splits that bite:
- Group leakage - same user in train + test. Use
GroupKFold. - Snooping - feature engineered on the full dataset including test. Fit on train only.
Pick the right metric
The interview-favourite "accuracy" is almost always the wrong metric.
| Problem shape | Use |
|---|---|
| Imbalanced binary classification | AUC-ROC, PR-AUC, F1 at chosen threshold |
| Probability calibration matters | Brier score, calibration plots |
| Ranking the top-K matters | Precision@K, NDCG |
| Regression | RMSE, MAE, MAPE - pick based on outlier sensitivity |
| Forecasting | sMAPE, MASE |
Tie the metric to the decision. If you only act on the top 5% of predicted churners, precision@5% is what matters - global AUC is comforting but irrelevant.
When NOT to use ML at all
Real engineering decisions, in order of how often you should make them:
- SQL query. "Customers who logged in once and never returned" doesn't need a model.
- Hand-written rule. "If amount > 10k AND country = X, flag for review" is auditable and shippable today.
- Simple statistics. A 14-day moving average + threshold solves more "anomaly detection" tickets than any neural net.
- ML model. Now you can reach for one.
The cost of a model is: training pipeline, feature store, monitoring, drift detection, retraining cadence, on-call. The cost of a rule is: writing the rule. Match the cost to the problem.
When NOT to use an LLM
You will be tempted. Resist when:
- The task is tabular and structured. A 100-row CSV with known columns? LightGBM. Don't make it a prompt.
- You need sub-50ms inference. An LLM call won't get there. A scikit-learn pipeline will.
- The decision is audited or regulated (credit, hiring, medical). You need a model whose features you can defend in a deposition. "It's an LLM, idk" is not that.
- Cost-per-call matters at scale. Logistic regression at 1M predictions/day is rounding-error compute. An LLM at the same scale is a budget meeting.
The mature take: classical ML and LLMs are complementary. LLMs win on unstructured text/multimodal/few-shot tasks. Classical ML wins on tabular, real-time, auditable problems. Use both.
Hands-on Checkpoints
- Pick a dataset (the Kaggle Telco churn is a good one). Frame the problem in one sentence (as above).
- Explore: 5 plots, 5 sentences each on what you noticed. No model yet.
- Train a logistic regression baseline. Note the AUC.
- Train a LightGBM model. Beat the baseline. Report the lift.
- Do a time-based split if the data has a time column. If you can't, justify in writing why a random split is OK here.
- Pick one business decision the model would inform. Pick the metric that matches it. Optimise for that, not AUC by default.
- Write a 200-word "what I'd do next if this were a real product" - features to add, monitoring to wire, the failure mode that scares you.
Further reading
- Hands-On Machine Learning with Scikit-Learn, Keras and TensorFlow - Aurélien Géron (the book; the first half is the classical toolbox)
- Forecasting: Principles and Practice - Hyndman & Athanasopoulos (free online)
- Eugene Yan - Machine learning lessons learned - practitioner blog, no hype
- LightGBM docs - short, complete
- Calibration matters - sklearn user guide
Welzin opinion: A baseline shipped on Tuesday beats a perfect model shipped never. Start with logistic regression on five hand-engineered features. Compare everything against it.