Setup & toolkit

tools · 1The pinned set

Pin these. When a lesson shows code, it was written against exactly this set.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip

pip install torch==2.13.0 torchvision==0.28.0 torchaudio==2.11.0
pip install transformers==5.15.0 datasets==5.0.1 accelerate==1.14.0
pip install peft==0.20.0 trl==1.10.0 torchao==0.18.0
pip install jupyterlab matplotlib

python -c "import torch; print(torch.__version__, torch.backends.mps.is_available())"

Install the export tools later, in Lesson 09. They pull in a lot and some of them conflict with each other, so it is better to keep a second virtual environment for exporting than to fight the resolver now.

tools · 2Two breaking changes that will confuse you

tools · 3If you are on a Mac

Apple Silicon is a good machine for this course, with three things to know.

• Memory is shared. There is no separate video memory, so the model, the data and everything else compete for the same RAM. A 16 GB machine can LoRA-fine-tune models of roughly one to two billion parameters in bf16. An 8 GB machine is fine for everything up to Lesson 06.
• No bitsandbytes, no CUDA kernels, no Triton. The 4-bit NF4 path used by most QLoRA tutorials is CUDA-only. Lesson 05 gives you the alternative.
• Some operations still fall back to the CPU. If you hit one, set PYTORCH_ENABLE_MPS_FALLBACK=1. It will be slower but it will run.

export PYTORCH_ENABLE_MPS_FALLBACK=1

tools · 4If you use a free Colab GPU

The free tier gives you an NVIDIA T4 with 16 GB. Two consequences:

• A T4 cannot do bf16. It is a Turing card: fp16 tensor cores yes, bfloat16 no. Use torch.amp.autocast('cuda', dtype=torch.float16) together with a GradScaler. On Apple Silicon and newer NVIDIA cards, use bf16 and skip the scaler.
• Sessions are killed when idle, and there is a weekly cap. Save checkpoints to Google Drive rather than to the session disk, and assume the machine will disappear.

tools · 5A habit worth building now

Before you debug maths, print shapes. Most failures in this course are a tensor of the wrong shape or data on the wrong device, and both are visible in one line.

def peek(name, t):
    print(f"{name:12} {tuple(t.shape)}  {t.dtype}  {t.device}  "
          f"min={t.min().item():.3f} max={t.max().item():.3f}")

peek("images", images)
peek("logits", logits)

And the single most useful test in machine learning: take ten examples and try to overfit them on purpose. A correct training loop drives the loss on ten samples to nearly zero within a minute. If it cannot, stop tuning hyper-parameters — the bug is in your code.