Part 12 of 12

Review Your Training Code with a Bot: Checklist

ChecklistBy Bot Tutorials · Published Nov 3, 2026 · Facts last verified Nov 3, 2026

Part 12: a review checklist for ML training code, a real data-leakage example with a passing test, and a recap of the whole series.

Training code can run without errors and still be wrong. The model trains, the score looks good, and the mistake only shows up later on real data. A bot makes a useful second reader for this kind of code, as long as you give it a checklist and ask it to point to lines. This final part gives you that checklist, shows a real data-leakage bug next to its fix with a test, and recaps the series.

What the bot can and can't do here

It can:

• Go through code against a checklist and point to each line that fails an item.
• Explain why a pattern is risky, using the docs you paste.
• Write a test that would catch the bug.

It can't:

• See your data, so it can't know whether a column leaks the answer.
• Guarantee it found everything. Treat "looks fine" as "nothing found", not "correct".
• Replace running the code and the tests.

Check first, trust second

Run the tests, read the numbers, then ask the bot. A review is only as good as the evidence you give it.

A real leakage bug

scikit-learn's Common pitfalls page warns against fitting preprocessing on the whole dataset before splitting. Here's that mistake next to the fix:

"""Leakage demo for the code-review part: scaling before the split vs inside a pipeline."""
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

def leaky(X, y):
    Xs = StandardScaler().fit_transform(X)          # BUG: scaler sees the test rows
    Xtr, Xte, ytr, yte = train_test_split(Xs, y, test_size=0.25, random_state=0)
    return LogisticRegression(max_iter=5000).fit(Xtr, ytr).score(Xte, yte)

def fixed(X, y):
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=0)
    m = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)).fit(Xtr, ytr)
    return m, m.score(Xte, yte)

if __name__ == "__main__":
    X, y = load_breast_cancer(return_X_y=True)
    print("leaky score:", round(leaky(X, y), 4))
    print("fixed score:", round(fixed(X, y)[1], 4))

Real output on the breast cancer dataset bundled with scikit-learn:

leaky score: 0.965
fixed score: 0.958

The leaky version scored a little higher, 0.965 against 0.958. Here the difference is small. On other data, especially with feature selection or target-based encoding, a leak can make a useless model look excellent. The habit matters more than this one number.

A test that catches it

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from train_review import fixed

def test_scaler_fit_on_train_only():
    X, y = load_breast_cancer(return_X_y=True)
    Xtr, _, _, _ = train_test_split(X, y, test_size=0.25, random_state=0)
    m, score = fixed(X, y)
    assert m.named_steps["standardscaler"].n_samples_seen_ == len(Xtr)
    assert 0.9 < score <= 1.0

The test checks that the scaler inside the pipeline saw exactly the number of training rows. Result: 1 passed in 1.02s.

The review checklist

Paste your code with this list and ask the bot to go item by item:

1. Split before fitting. Is every scaler, encoder or feature selector fitted on training data only?
2. No answer in the features. Is there a column that's derived from the label, or only known after the event you're predicting?
3. Right split for the data. For time series, does the test set come after the training set?
4. Seeds and versions recorded. Are seeds set, and are library versions pinned (Part 3)?
5. Shapes and dtypes checked. Are labels integers for classification (Part 11)?
6. Train and eval modes. In PyTorch, is model.eval() and torch.no_grad() used for evaluation (Part 6)?
7. Gradients cleared. Is zero_grad() called every step?
8. Metrics beyond one score. Is there cross-validation or a second metric (Part 5)?
9. Safe loading. Are model files loaded only from trusted sources, with weights_only=True for torch.load (Part 9)?
10. Licenses and data rights. Are the dataset and model licenses recorded, and is personal data handled with permission?
11. Memory fits. Does the estimate from Part 2 fit the hardware?
12. Tests exist. Is there at least one test for each of the risky steps above?

The review prompt

Review this training script against the checklist below (pasted). For each item, answer pass, fail or can't tell, and quote the line you're judging. For each fail, suggest a fix and a pytest test that would catch it. Don't rewrite the whole script.

"Can't tell" is important. It stops the bot from passing items it has no evidence for.

Series recap

Across twelve parts we used bots as helpers, not authorities. We checked driver, CUDA and library versions against NVIDIA's and each library's tables, estimated memory before choosing hardware, pinned environments, and learned the basics of NumPy, pandas, scikit-learn, PyTorch, TensorFlow and JAX on a CPU. We read model cards and licenses before downloading, exported a model to ONNX and proved it matched, decoded real errors and profiled slow code, and finished with a review checklist. The rule throughout: check the bot against the official docs, run it yourself, and label honestly what you tested. The series hub lists every part.

FAQ

How big was the leakage effect?

Small in our example: 0.965 with the leak and 0.958 without. On other data it can be much larger, which is why the habit matters more than the number.

Can a bot do the whole review?

No. It's a good second reader, but it only knows what you paste and it can miss things. Tests and your own reading come first.

Was this tested?

Yes. The leaky and fixed scripts ran on our box, and the pytest test that checks the scaler saw only training rows passes.

Sources

1. scikit-learn: Common pitfalls and recommended practices
2. scikit-learn: Installing scikit-learn
3. pytest documentation
4. PyTorch docs: Reproducibility
5. PyTorch docs: torch.load
6. scikit-learn: Model persistence
7. Hugging Face Hub: Licenses
8. pip: Repeatable installs

More in this series

1. Part 1: GPU, Driver and CUDA Compatibility: A Bot-Assisted Check · Guide
2. Part 2: How Much Memory Does a Model Need? A Data Brief · Data brief
3. Part 3: Pin Versions and Build Reproducible ML Environments · Walkthrough
4. Part 4: NumPy and pandas Basics with a Bot: Cheat Sheet · Cheat sheet
5. Part 5: Your First scikit-learn Model, Built with a Bot · Walkthrough
6. Part 6: PyTorch on the CPU: First Steps with a Bot · Guide
7. Part 7: TensorFlow and Keras Install Requirements, Explained · Explainer
8. Part 8: JAX on the CPU: Field Notes with a Bot · Field notes
9. Part 9: Read Hugging Face Model Cards and Licenses with a Bot · Guide
10. Part 10: Export a Model to ONNX and Run It: A Deep Dive · Deep dive
11. Part 11: Read ML Error Messages and Profile Code: Cheat Sheet · Cheat sheet
12. Part 12: Review Your Training Code with a Bot: Checklist (you are here) · Checklist

See the ML Libraries with a Bot series hub · Latest on Bot Tutorials

Bot advice can be wrong. We cite official docs; check them yourself before acting, especially for anything involving money, safety or your data.