WEBVTT

1
00:00:00.000 --> 00:00:02.643
Part 4 of ML Libraries with a Bot.

2
00:00:02.643 --> 00:00:06.374
NumPy and pandas Basics with a Bot: Cheat
Sheet.

3
00:00:06.374 --> 00:00:12.593
Almost every machine learning library sits
on top of NumPy arrays, and most real

4
00:00:12.593 --> 00:00:15.314
data arrives as a pandas DataFrame.

5
00:00:15.314 --> 00:00:21.611
You don't need to know everything about
them, but you do need a handful of ideas:

6
00:00:21.611 --> 00:00:26.120
shapes, dtypes, broadcasting, missing
values and grouping.

7
00:00:26.875 --> 00:00:29.317
What the bot can and cannot do.

8
00:00:29.317 --> 00:00:33.808
Explain why two arrays won't combine,
given their shapes.

9
00:00:33.808 --> 00:00:38.930
Translate "I want the average price per
part" into a pandas line.

10
00:00:38.930 --> 00:00:43.263
Read a traceback and find the column or
shape at fault.

11
00:00:43.263 --> 00:00:44.760
See your DataFrame.

12
00:00:44.760 --> 00:00:49.409
Paste df.head(), df.dtypes and df.shape,
not a description.

13
00:00:49.409 --> 00:00:55.633
Remember which defaults changed between
versions. pandas 3 changed some, as the

14
00:00:55.633 --> 00:00:57.445
dtype line below shows.

15
00:00:57.445 --> 00:01:01.148
Decide what a missing value means in your
data.

16
00:01:01.148 --> 00:01:02.488
That's your call.

17
00:01:03.167 --> 00:01:08.183
Before you change data, look at it: shape,
dtypes and missing counts.

18
00:01:08.183 --> 00:01:12.400
Most bugs are a wrong assumption about one
of those three.

19
00:01:13.000 --> 00:01:19.942
Real output on our box with NumPy 2.5.4
and pandas 3.1.0:

20
00:01:20.542 --> 00:01:26.982
Shape and dtype first. a.shape is (3, 4):
three rows, four columns. a.dtype is

21
00:01:26.982 --> 00:01:31.110
float64, NumPy's default for these random
numbers.

22
00:01:31.110 --> 00:01:38.045
Axis means direction. a.mean(axis=0)
averages down the rows and gives one
number per

23
00:01:38.045 --> 00:01:40.027
column, so four numbers.

24
00:01:40.027 --> 00:01:41.100
Broadcasting.

25
00:01:41.100 --> 00:01:47.623
Adding b with shape (4,) to a with shape
(3, 4) works because NumPy stretches b

26
00:01:47.623 --> 00:01:48.944
across each row.

27
00:01:49.792 --> 00:01:50.906
Check dtypes.

28
00:01:50.906 --> 00:01:57.931
On our box the part column came back as a
pandas StringDtype, not the older object

29
00:01:57.931 --> 00:02:00.844
type, because we're on pandas 3.1.

30
00:02:00.844 --> 00:02:07.956
That's a good example of why bots that
learned on older versions can be wrong
about

31
00:02:07.956 --> 00:02:08.727
defaults.

32
00:02:08.727 --> 00:02:13.354
Count missing values.
df["price"].isna().sum() gave 1.

33
00:02:14.125 --> 00:02:17.903
The second prompt is a good test of a bot.

34
00:02:17.903 --> 00:02:25.009
The right answer explains that
broadcasting aligns trailing dimensions,
so (3,)

35
00:02:25.009 --> 00:02:32.294
doesn't line up with the 4 columns, and
suggests reshaping to (3, 1) or using the

36
00:02:32.294 --> 00:02:33.553
matching axis.

37
00:02:34.333 --> 00:02:35.843
Chained assignment.

38
00:02:35.843 --> 00:02:39.817
Code like df[df.x > 0]["y"] = 1 may not
change df.

39
00:02:39.817 --> 00:02:41.884
Use df.loc[mask, "y"] = 1.

40
00:02:41.884 --> 00:02:45.778
The pandas user guide explains indexing
with loc.

41
00:02:45.778 --> 00:02:47.685
Silent float conversion.

42
00:02:47.685 --> 00:02:53.010
An integer column with a missing value
becomes float in many cases.

43
00:02:53.010 --> 00:02:55.156
Check dtypes after loading.

44
00:02:55.156 --> 00:02:56.030
Wrong axis.

45
00:02:56.030 --> 00:03:02.150
If a mean gives you the wrong number of
values, the axis is probably swapped.

46
00:03:02.750 --> 00:03:08.830
Knowing these words makes bot answers
easier to check, because you can ask
exactly

47
00:03:08.830 --> 00:03:09.868
what you mean.

48
00:03:10.542 --> 00:03:12.530
Tested vs documented.

49
00:03:12.530 --> 00:03:20.199
Tested by us: every line of the script and
every output shown, on NumPy 2.5.4 and

50
00:03:20.199 --> 00:03:21.429
pandas 3.1.0.

51
00:03:21.429 --> 00:03:28.719
Documented, not tested by us: the chained-
assignment advice and the Generator

52
00:03:28.719 --> 00:03:33.169
recommendation, from the pandas and NumPy
docs.

53
00:03:33.958 --> 00:03:38.817
The next part today uses these arrays to
train a first scikit-learn model.

54
00:03:39.417 --> 00:03:45.461
Before you run anything: check every bot
answer against the official docs, work in
a

55
00:03:45.461 --> 00:03:51.360
virtual environment, load model files only
from sources you trust, respect dataset

56
00:03:51.360 --> 00:03:56.613
and model licenses, and keep personal data
out of bots and training sets.

57
00:03:56.613 --> 00:03:59.131
Every source is linked on the page.