Programme

13:00 - 13:45

Coffee break 

Tutorial: The Proportional Depth-Width Scaling Limit of Neural Networks (Mufan Li)

Abstract: We study the scaling limit of neural networks without skip connects, where the depth d and width n approach infinity at a constant ratio d/n. In this limiting regime, we can review each layer of the neural network as a time discretization, and derive a limiting SDE for the feature covariance matrix.

Tutorial: Dynamical Mean Field Theory, Random Matrices and Learning in High Dimensions (Blake Bordelon)

Lunch provided at venue

Research talk: How to train an LLM (Sam Smith)

Abstract: Drawing on the experience of designing and scaling Griffin (https://arxiv.org/abs/2402.19427) and RecurrentGemma, I will introduce some of the key practical concepts behind training large language models. Likely to include: a brief introduction to Transformers, including why MLPs, not Attention, usually dominate computation. A simple mental model of the computational bottlenecks on TPUs and GPUs. How to train models too large to fit in memory on a single device. Scaling laws and hyper-parameter tuning. A detailed discussion of LLM inference. If time permits, I will discuss how to design recurrent models competitive with transformers, their advantages and drawbacks.

On-campus dinner for attendees (registration required)

Research talk

Workshop closure & lunch provided at venue