WEBVTT

1
00:00:00.000 --> 00:00:02.613
Part 2 of ML Libraries with a Bot.

2
00:00:02.613 --> 00:00:05.227
How Much Memory Does a Model Need?

3
00:00:05.227 --> 00:00:06.226
A Data Brief.

4
00:00:06.226 --> 00:00:12.605
Before you choose a GPU, rent a cloud
machine or download a large model, answer
one

5
00:00:12.605 --> 00:00:15.065
question: will it fit in memory?

6
00:00:15.065 --> 00:00:20.906
The rough answer comes from two numbers,
the model's parameter count and the

7
00:00:20.906 --> 00:00:22.828
precision you load it in.

8
00:00:22.828 --> 00:00:29.130
This brief shows the arithmetic, a small
tested estimator, and how to get a bot to

9
00:00:29.130 --> 00:00:32.128
check your working instead of guessing.

10
00:00:32.958 --> 00:00:35.205
What the bot can and cannot do.

11
00:00:35.205 --> 00:00:38.103
Do the arithmetic and explain each term.

12
00:00:38.103 --> 00:00:43.030
Read a model card and find the parameter
count and stored precision.

13
00:00:43.030 --> 00:00:47.812
Point out what an estimate leaves out,
like activations or caches.

14
00:00:47.812 --> 00:00:50.783
Tell you the exact memory a run will use.

15
00:00:50.783 --> 00:00:54.986
That depends on your code, batch size and
library version.

16
00:00:54.986 --> 00:00:56.363
Benchmark hardware.

17
00:00:56.363 --> 00:01:00.565
Any speed figure it gives you without a
source is a guess.

18
00:01:00.565 --> 00:01:05.275
Know a model's size unless you paste the
model card or file list.

19
00:01:05.875 --> 00:01:13.332
Estimate on paper, then test small on the
hardware you already have, and only then

20
00:01:13.332 --> 00:01:14.424
spend money.

21
00:01:14.424 --> 00:01:21.972
A model that needs 14 GB for weights won't
fit in 12 GB, and no setting will change

22
00:01:21.972 --> 00:01:22.426
that.

23
00:01:23.208 --> 00:01:26.929
Weights take parameters × bytes per
parameter.

24
00:01:26.929 --> 00:01:33.481
Common sizes are 4 bytes for fp32, 2 for
fp16 or bf16, 1 for int8 and half a byte

25
00:01:33.481 --> 00:01:34.937
for 4-bit formats.

26
00:01:34.937 --> 00:01:41.488
For training, Hugging Face's "model
training anatomy" page breaks memory down
for

27
00:01:41.488 --> 00:01:47.312
mixed-precision training with Adam: That's
18 bytes per parameter before

28
00:01:47.312 --> 00:01:48.282
activations.

29
00:01:49.125 --> 00:01:55.727
Real output from our box: The last line
counts the parameters of a real PyTorch

30
00:01:55.727 --> 00:02:02.412
model on our machine: a two-layer network
has 203,530 parameters, which is about

31
00:02:02.412 --> 00:02:03.749
0.81 MB in fp32.

32
00:02:03.749 --> 00:02:09.599
You can check that by hand: 784 × 256 +
256 + 256 × 10 + 10 = 203,530.

33
00:02:09.599 --> 00:02:12.691
The tests: Result: 3 passed in 0.97s.

34
00:02:13.458 --> 00:02:19.886
A 7-billion-parameter model needs about 14
GB just for fp16 weights, and about 126

35
00:02:19.886 --> 00:02:24.275
GB for mixed-precision Adam training
before activations.

36
00:02:24.275 --> 00:02:30.075
That's why people fine-tune large models
with methods that train far fewer

37
00:02:30.075 --> 00:02:33.210
parameters, or use quantized optimizers.

38
00:02:33.210 --> 00:02:39.716
For a 125-million-parameter model, the
weights are a quarter of a gigabyte in
fp16,

39
00:02:39.716 --> 00:02:43.165
and full training fits on far more
machines.

40
00:02:44.000 --> 00:02:49.475
The prompt: Here's the model card and file
list for a model (pasted).

41
00:02:49.475 --> 00:02:55.347
Find the parameter count and stored
precision, quoting the lines you used.

42
00:02:55.347 --> 00:03:02.013
Then estimate weight memory in fp16 and
int8, and training memory using 18 bytes
per

43
00:03:02.013 --> 00:03:06.139
parameter from Hugging Face's training
anatomy page.

44
00:03:06.139 --> 00:03:08.837
List what the estimate leaves out.

45
00:03:08.837 --> 00:03:10.821
Don't give speed figures.

46
00:03:11.500 --> 00:03:18.440
Fits with headroom: if the weights take
under roughly two-thirds of your memory,
try

47
00:03:18.440 --> 00:03:20.423
a small run and measure.

48
00:03:20.423 --> 00:03:25.793
Barely fits: expect out-of-memory errors
once activations arrive.

49
00:03:25.793 --> 00:03:29.594
Lower the batch size or sequence length
first.

50
00:03:29.594 --> 00:03:36.368
Doesn't fit: use a smaller model, lower
precision for inference, or a machine with

51
00:03:36.368 --> 00:03:37.360
more memory.

52
00:03:38.125 --> 00:03:39.801
Tested vs documented.

53
00:03:39.801 --> 00:03:46.108
Tested by us: the estimator, its three
tests, and the parameter count of a real

54
00:03:46.108 --> 00:03:49.062
PyTorch model, all on a CPU-only box.

55
00:03:49.062 --> 00:03:55.448
Documented, not tested by us: the training
memory breakdown, from Hugging Face's

56
00:03:55.448 --> 00:03:59.599
page, and GPU memory behaviour, from
PyTorch's docs.

57
00:03:59.599 --> 00:04:03.032
Not claimed: any speed or benchmark
result.

58
00:04:03.792 --> 00:04:09.369
The last part today pins your library
versions so this week's working setup
still

59
00:04:09.369 --> 00:04:10.539
works next month.

60
00:04:11.208 --> 00:04:17.252
Before you run anything: check every bot
answer against the official docs, work in
a

61
00:04:17.252 --> 00:04:23.152
virtual environment, load model files only
from sources you trust, respect dataset

62
00:04:23.152 --> 00:04:28.404
and model licenses, and keep personal data
out of bots and training sets.

63
00:04:28.404 --> 00:04:30.923
Every source is linked on the page.