What the course is

The course is free and built around the Hugging Face ecosystem: 🤗 Transformers, Datasets, Tokenizers, Accelerate and the Hub. Chapters 1–4 cover the basics of Transformers. After that it moves on to datasets and tokenizers, then classical NLP tasks, and finally more advanced LLM topics. The stated prerequisites are Python and some basic deep learning knowledge. You don't need a GPU to get value from the early chapters.

Each chapter ends with an ungraded quiz. Chapter 1 also has a short certification exam, hosted in a Hugging Face Space. You need to sign in to take it.

Chapter 1: Transformer models, demystified

The first chapter starts at the highest level: pipeline(). One call wraps preprocessing, the model and postprocessing. With it you can run sentiment analysis, zero-shot classification, text generation, fill-mask, named entity recognition, extractive question answering, summarization and translation. Small details turn out to matter. For example, the mask token depends on the model: [MASK] for BERT and <mask> for RoBERTa. Zero-shot classification also needs candidate_labels.

Then the chapter explains why it works. The ideas that clarified the most for me:

• Self-supervised pretraining. Language models learn without human labels, either by predicting the next token (causal LM) or by filling in masked tokens (masked LM).
• Transfer learning. Pretraining is expensive in both compute and environmental cost. So you start from pretrained weights and fine-tune on a smaller, task-specific dataset.
• Three architecture families. Encoder-only models (BERT, RoBERTa, ModernBERT) use bidirectional attention and suit classification, NER and extractive QA. Decoder-only models (GPT, Llama, Gemma, SmolLM) generate auto-regressively and are the base of modern LLMs. Encoder-decoder models (BART, T5, Marian) handle summarization and translation.
• Architecture vs. checkpoint. The architecture is the blueprint. The checkpoint is the trained weights. That distinction removes a lot of confusion when you browse the Hub.

The section on inference was the most useful part for practical work. Generation has two phases. Prefill processes the prompt and is compute-bound. Decode produces tokens one at a time and is memory-bound. That split explains why time to first token and time per output token behave so differently. It also explains why the KV cache matters: it avoids recomputing past keys and values. The chapter also covers sampling controls (temperature, top-k, top-p, repetition penalties) and why long contexts are costly.

There is also an honest section on bias and limitations. A fill-mask demo suggests gendered professions, and the course notes that fine-tuning does not remove bias inherited from pretraining. That is worth remembering in any domain where fairness matters, such as legal or educational applications.

Chapter 2: Behind the pipeline

Chapter 2 takes pipeline() apart: tokenizer → model (logits) → postprocessing (SoftMax → labels). Key takeaways:

• The tokenizer must match the checkpoint. AutoTokenizer.from_pretrained(checkpoint) and AutoModel.from_pretrained(checkpoint) should point to the same model.
• Models in the library are plain PyTorch nn.Modules. save_pretrained writes a config.json plus model.safetensors.
• Tokenization strategies: word-based (huge vocabularies), character-based, and subword methods (WordPiece, BPE, Unigram). Subword tokenization is the practical compromise.
• Batching needs padding, attention masks and truncation. A model expects a batch dimension, and forgetting it is a classic first error.

The chapter closes with deployment options: TGI (continuous batching, Flash Attention), vLLM (PagedAttention) and llama.cpp for smaller local models. If you run models yourself on a homelab or a VPS, this short section is a good map of the options.

Chapter 3: Fine-tuning a pretrained model

Chapter 3 is where it becomes concrete. You fine-tune a BERT classifier on MRPC from the GLUE benchmark, which asks whether two sentences are paraphrases. Along the way you learn:

• load_dataset("glue", "mrpc") and Dataset.map(batched=True) for fast tokenization of sentence pairs.
• DataCollatorWithPadding for dynamic padding per batch instead of padding everything to the maximum length.
• The Trainer API. TrainingArguments needs only an output_dir. For accuracy you add an evaluation strategy and a compute_metrics function. fp16=True turns on mixed precision.
• A manual training loop with AdamW, a linear scheduler and Accelerate, which shows what Trainer hides.
• Reading learning curves. Loss falls and then converges, while accuracy moves in steps. A large gap between training and validation means overfitting. Two curves that both plateau point to underfitting.

One small detail I liked: when you load a model with a new classification head, you get a warning that some weights are randomly initialised. That is expected. The head hasn't been trained yet.

How I'd recommend starting

1. Read Chapter 1 end to end, even if you know the basics. The architecture overview and the inference section give you shared vocabulary for everything after.
2. Run the code. The notebooks open in Colab. Typing pipeline("sentiment-analysis") and then taking it apart in Chapter 2 builds intuition faster than reading.
3. Do the quizzes. They're short and they catch the subtle misunderstandings. One question made me double-check that SoftMax outputs are between 0 and 1 and sum to 1.
4. Fine-tune something small. Chapter 3's MRPC example runs on a free GPU. Once it works, swap in your own dataset.
5. Treat it as a map, not a finish line. Later chapters cover datasets, tokenizers and more advanced LLM topics. I'm continuing with those.

Closing thought

The course doesn't sell you anything. It explains what's going on under the hood clearly and gives you working code. For anyone building with LLMs, whether you're shipping products or advising on how AI is used responsibly, that grounding pays off quickly.

Course link: Hugging Face LLM Course · Chapter 1 · Chapter 2 · Chapter 3