CSCI 3151 - M38 - Dropout, batch normalization, and training tricks

Описание: This module dives into how deep networks are actually regularized and kept trainable in practice, focusing on three core ingredients: dropout, batch normalization, and a handful of simple but powerful “training tricks.” We start from the failure modes students have already seen—overfitting, unstable optimization, and sensitivity to learning rate or initialization—and frame each new technique as a targeted response: dropout as noisy, ensemble-like regularization that fights co-adaptation; batch normalization as per-batch standardization plus learned scale-and-shift that stabilizes gradient flow; and tools like early stopping, weight decay, gradient clipping, and learning-rate schedules as levers for controlling effective capacity and optimization dynamics.

On the practical side, we build and train PyTorch MLPs with and without these components on small datasets, using ablation experiments to see what each trick actually buys us. Students visualize training and validation curves, compare summary tables of accuracy and loss, and inspect how changes like adding dropout or BatchNorm modify both the speed and stability of learning. Along the way, they learn to distinguish “real” improvements from noise, and to avoid cargo-cult training where tricks are stacked without diagnosing the underlying problem.

By the end of the module, students should be able to (i) explain the roles of dropout and batch normalization in modern deep networks, (ii) interpret training/validation behavior to decide when regularization is helping or hurting, (iii) design small ablation studies to test whether a proposed trick is actually useful for a given model and dataset, and (iv) assemble a sensible, well-justified training recipe rather than treating optimization settings as mysterious magic numbers.

Course module page:
https://web.cs.dal.ca/~rudzicz/Teaching/CS...