WEBVTT

1
00:00:00.000 --> 00:00:02.561
Part 12 of ML Libraries with a Bot.

2
00:00:02.561 --> 00:00:06.073
Review Your Training Code with a Bot:
Checklist.

3
00:00:06.073 --> 00:00:10.170
Training code can run without errors and
still be wrong.

4
00:00:10.170 --> 00:00:16.243
The model trains, the score looks good,
and the mistake only shows up later on
real

5
00:00:16.243 --> 00:00:16.609
data.

6
00:00:16.609 --> 00:00:22.608
A bot makes a useful second reader for
this kind of code, as long as you give it
a

7
00:00:22.608 --> 00:00:25.462
checklist and ask it to point to lines.

8
00:00:26.208 --> 00:00:28.566
What the bot can and cannot do.

9
00:00:28.566 --> 00:00:34.499
Go through code against a checklist and
point to each line that fails an item.

10
00:00:34.499 --> 00:00:38.835
Explain why a pattern is risky, using the
docs you paste.

11
00:00:38.835 --> 00:00:41.725
Write a test that would catch the bug.

12
00:00:41.725 --> 00:00:46.745
See your data, so it can't know whether a
column leaks the answer.

13
00:00:46.745 --> 00:00:49.027
Guarantee it found everything.

14
00:00:49.027 --> 00:00:53.059
Treat "looks fine" as "nothing found", not
"correct".

15
00:00:53.059 --> 00:00:56.025
Replace running the code and the tests.

16
00:00:56.625 --> 00:01:00.157
Run the tests, read the numbers, then ask
the bot.

17
00:01:00.157 --> 00:01:03.900
A review is only as good as the evidence
you give it.

18
00:01:04.500 --> 00:01:11.015
scikit-learn's Common pitfalls page warns
against fitting preprocessing on the whole

19
00:01:11.015 --> 00:01:12.954
dataset before splitting.

20
00:01:12.954 --> 00:01:18.926
Here's that mistake next to the fix: Real
output on the breast cancer dataset

21
00:01:18.926 --> 00:01:25.286
bundled with scikit-learn: The leaky
version scored a little higher, 0.965
against

22
00:01:25.286 --> 00:01:25.751
0.958.

23
00:01:25.751 --> 00:01:28.001
Here the difference is small.

24
00:01:28.833 --> 00:01:36.517
The test checks that the scaler inside the
pipeline saw exactly the number of

25
00:01:36.517 --> 00:01:37.914
training rows.

26
00:01:37.914 --> 00:01:40.509
Result: 1 passed in 1.02s.

27
00:01:41.208 --> 00:01:45.733
Paste your code with this list and ask the
bot to go item by item:

28
00:01:46.333 --> 00:01:48.023
"Can't tell" is important.

29
00:01:48.023 --> 00:01:51.858
It stops the bot from passing items it has
no evidence for.

30
00:01:52.458 --> 00:01:57.248
Across twelve parts we used bots as
helpers, not authorities.

31
00:01:57.248 --> 00:02:03.529
We checked driver, CUDA and library
versions against NVIDIA's and each
library's

32
00:02:03.529 --> 00:02:10.045
tables, estimated memory before choosing
hardware, pinned environments, and learned

33
00:02:10.045 --> 00:02:16.326
the basics of NumPy, pandas, scikit-learn,
PyTorch, TensorFlow and JAX on a CPU.

34
00:02:17.083 --> 00:02:18.749
Tested vs documented.

35
00:02:18.749 --> 00:02:23.666
Tested by us: the leaky and fixed scripts
and the pytest test.

36
00:02:23.666 --> 00:02:30.248
Documented, not tested by us: the
checklist items' background, from scikit-
learn's,

37
00:02:30.248 --> 00:02:32.230
PyTorch's and pip's docs.

38
00:02:32.230 --> 00:02:36.196
Not claimed: that the checklist catches
every bug.

39
00:02:36.875 --> 00:02:42.919
Before you run anything: check every bot
answer against the official docs, work in
a

40
00:02:42.919 --> 00:02:48.819
virtual environment, load model files only
from sources you trust, respect dataset

41
00:02:48.819 --> 00:02:54.071
and model licenses, and keep personal data
out of bots and training sets.

42
00:02:54.071 --> 00:02:56.589
Every source is linked on the page.