Publications

A complete citation record is available on Google Scholar.

Research

Selected publications

2025 · NeurIPS

Don't be lazy: CompleteP enables compute-efficient deep transformers

Nolan Dey†, Bin Claire Zhang†, Lorenzo Noci, Mufan Li, Blake Bordelon, Shane Bergsma, Cengiz Pehlevan, Boris Hanin, Joel Hestness

We introduce CompleteP, a parameterization that transfers hyperparameters across depth while keeping every layer in a non-lazy learning regime, improving training compute efficiency by 12-34%.

Paper

Figure 1

2024 · NeurIPS

Understanding and Minimising Outlier Features in Neural Network Training

Bobby He, Lorenzo Noci, Daniele Paliotta, Imanol Schlag, Thomas Hofmann

We study why activation outliers emerge during transformer training and show how architectural and optimization choices can substantially reduce them without slowing convergence.

Paper

Figure 1

2024 · NeurIPS

Super Consistency of Neural Network Landscapes and Learning Rate Transfer

Lorenzo Noci†, Alex Meterez†, Thomas Hofmann, Antonio Orvieto

We find that neural-network loss landscapes can remain remarkably consistent across width, explaining when learning rates transfer from small models to larger ones.

Paper

Figure 1

2024 · ICLR

Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit

Blake Bordelon†, Lorenzo Noci†, Mufan Bill Li, Boris Hanin, Cengiz Pehlevan

We develop a parameterization for residual networks that transfers optimal learning rates across both width and depth.

Paper

Figure 1

2023 · NeurIPS · Spotlight

Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers

Sotiris Anagnostidis, Dario Pavllo, Luca Biggio, Lorenzo Noci, Aurelien Lucchi, Thomas Hofmann

We learn to prune uninformative context tokens dynamically during inference, improving transformer efficiency and interpretability while preserving model quality.

Paper

Figure 1

2023 · NeurIPS

The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit

Lorenzo Noci†, Chuning Li†, Mufan Bill Li†, Bobby He, Thomas Hofmann, Chris Maddison, Daniel M. Roy

We introduce shaped attention, a modification that prevents rank collapse and admits a stable infinite depth-and-width limit.

Paper

Figure 1

2022 · NeurIPS

Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse

Lorenzo Noci†, Sotiris Anagnostidis†, Luca Biggio†, Antonio Orvieto†, Sidak Pal Singh†, Aurelien Lucchi

We connect rank collapse to vanishing query and key gradients, and derive depth-dependent residual scaling that preserves token geometry in deep Transformers.

Paper

Figure 1

2022 · ICML · Oral

How Tempering Fixes Data Augmentation in Bayesian Neural Networks

Gregor Bachmann†, Lorenzo Noci†, Thomas Hofmann

We show that data augmentation creates correlated observations in Bayesian inference and that likelihood tempering corrects the resulting model misspecification.

Paper