Jeffrey Pennington

According to our database, Jeffrey Pennington authored at least 52 papers between 2011 and 2024.

Collaborative distances:

• Dijkstra number of four.
• Erdős number of three.

2024

Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models.

Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron T. Parisi, Abhishek Kumar, Alexander A. Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Fathy Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, Noah Fiedel

Trans. Mach. Learn. Res., 2024

Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability.

Jiri Hron, Laura Culp, Gamaleldin F. Elsayed, Rosanne Liu, Ben Adlam, Maxwell L. Bileschi, Bernd Bohnet, JD Co-Reyes, Noah Fiedel, C. Daniel Freeman, Izzeddin Gur, Kathleen Kenealy, Jaehoon Lee, Peter J. Liu, Gaurav Mishra, Igor Mordatch, Azade Nova, Roman Novak, Aaron Parisi, Jeffrey Pennington, Alex Rizkowsky, Isabelle Simpson, Hanie Sedghi, Jascha Sohl-Dickstein, Kevin Swersky, Sharad Vikram, Tris Warkentin, Lechao Xiao, Kelvin Xu, Jasper Snoek, Simon Kornblith

CoRR, 2024

4+3 Phases of Compute-Optimal Neural Scaling Laws.

Elliot Paquette, Courtney Paquette, Lechao Xiao, Jeffrey Pennington

CoRR, 2024

High dimensional analysis reveals conservative sharpening and a stochastic edge of stability.

Atish Agarwala, Jeffrey Pennington

CoRR, 2024

Training LLMs over Neurally Compressed Text.

Brian Lester, Jaehoon Lee, Alex Alemi, Jeffrey Pennington, Adam Roberts, Jascha Sohl-Dickstein, Noah Constant

CoRR, 2024

Scaling Exponents Across Parameterizations and Optimizers.

Katie E. Everett, Lechao Xiao, Mitchell Wortsman, Alexander A. Alemi, Roman Novak, Peter J. Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jaehoon Lee, Jeffrey Pennington

Proceedings of the Forty-first International Conference on Machine Learning, 2024

Small-scale proxies for large-scale Transformer training instabilities.

Mitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie E. Everett, Alexander A. Alemi, Ben Adlam, John D. Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, Simon Kornblith

Proceedings of the Twelfth International Conference on Learning Representations, 2024

2023

Temperature check: theory and practice for training models with softmax-cross-entropy losses.

Atish Agarwala, Samuel Stern Schoenholz, Jeffrey Pennington, Yann N. Dauphin

Trans. Mach. Learn. Res., 2023

Frontier Language Models are not Robust to Adversarial Arithmetic, or "What do I need to say so you agree 2+2=5?

C. Daniel Freeman, Laura Culp, Aaron Parisi, Maxwell L. Bileschi, Gamaleldin F. Elsayed, Alex Rizkowsky, Isabelle Simpson, Alex Alemi, Azade Nova, Ben Adlam, Bernd Bohnet, Gaurav Mishra, Hanie Sedghi, Igor Mordatch, Izzeddin Gur, Jaehoon Lee, John D. Co-Reyes, Jeffrey Pennington, Kelvin Xu, Kevin Swersky, Kshiteej Mahajan, Lechao Xiao, Rosanne Liu, Simon Kornblith, Noah Constant, Peter J. Liu, Roman Novak, Yundi Qian, Noah Fiedel, Jascha Sohl-Dickstein

CoRR, 2023

Second-order regression models exhibit progressive sharpening to the edge of stability.

Atish Agarwala, Fabian Pedregosa, Jeffrey Pennington

Proceedings of the International Conference on Machine Learning, 2023

2022

Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression.

Lechao Xiao, Jeffrey Pennington

CoRR, 2022

Precise Learning Curves and Higher-Order Scalings for Dot-product Kernel Regression.

Lechao Xiao, Hong Hu, Theodor Misiakiewicz, Yue Lu, Jeffrey Pennington

Proceedings of the Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, 2022

Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions.

Courtney Paquette, Elliot Paquette, Ben Adlam, Jeffrey Pennington

Proceedings of the Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, 2022

Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithm.

Lechao Xiao, Jeffrey Pennington

Proceedings of the International Conference on Machine Learning, 2022

Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling.

Jiri Hron, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein

Proceedings of the International Conference on Machine Learning, 2022

Anisotropic Random Feature Regression in High Dimensions.

Gabriel Mel, Jeffrey Pennington

Proceedings of the Tenth International Conference on Learning Representations, 2022

Online education for data science: Opportunities and challenges.

Jeffrey Pennington, Rose Hartman, Ashwini Davison, Ali Shokoufandeh, Joy Payton, Daniel Chen, André Dietrich

Proceedings of the AMIA 2022, 2022

A Random Matrix Perspective on Mixtures of Nonlinearities in High Dimensions.

Ben Adlam, Jake A. Levinson, Jeffrey Pennington

Proceedings of the International Conference on Artificial Intelligence and Statistics, 2022

2021

Covariate Shift in High-Dimensional Random Feature Regression.

Nilesh Tripuraneni, Ben Adlam, Jeffrey Pennington

CoRR, 2021

Overparameterization Improves Robustness to Covariate Shift in High Dimensions.

Nilesh Tripuraneni, Ben Adlam, Jeffrey Pennington

Proceedings of the Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, 2021

Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit.

Ben Adlam, Jaehoon Lee, Lechao Xiao, Jeffrey Pennington, Jasper Snoek

Proceedings of the 9th International Conference on Learning Representations, 2021

2020

Exact posterior distributions of wide Bayesian neural networks.

Jiri Hron, Yasaman Bahri, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein

CoRR, 2020

Finite Versus Infinite Neural Networks: an Empirical Study.

Jaehoon Lee, Samuel S. Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, Jascha Sohl-Dickstein

Proceedings of the Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020

The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks.

Wei Hu, Lechao Xiao, Ben Adlam, Jeffrey Pennington

Proceedings of the Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020

Understanding Double Descent Requires A Fine-Grained Bias-Variance Decomposition.

Ben Adlam, Jeffrey Pennington

Proceedings of the Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020

Disentangling Trainability and Generalization in Deep Neural Networks.

Lechao Xiao, Jeffrey Pennington, Samuel Stern Schoenholz

Proceedings of the 37th International Conference on Machine Learning, 2020

The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization.

Ben Adlam, Jeffrey Pennington

Proceedings of the 37th International Conference on Machine Learning, 2020

Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks.

Wei Hu, Lechao Xiao, Jeffrey Pennington

Proceedings of the 8th International Conference on Learning Representations, 2020

2019

Disentangling trainability and generalization in deep learning.

Lechao Xiao, Jeffrey Pennington, Samuel S. Schoenholz

CoRR, 2019

A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learning.

Ben Adlam, Jake Levinson, Jeffrey Pennington

CoRR, 2019

Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent.

Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Jascha Sohl-Dickstein, Jeffrey Pennington

CoRR, 2019

Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs.

Dar Gilboa, Bo Chang, Minmin Chen, Greg Yang, Samuel S. Schoenholz, Ed H. Chi, Jeffrey Pennington

CoRR, 2019

Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent.

Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, Jeffrey Pennington

Proceedings of the Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, 2019

A Mean Field Theory of Batch Normalization.

Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, Samuel S. Schoenholz

Proceedings of the 7th International Conference on Learning Representations, 2019

Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes.

Roman Novak, Lechao Xiao, Yasaman Bahri, Jaehoon Lee, Greg Yang, Jiri Hron, Daniel A. Abolafia, Jeffrey Pennington, Jascha Sohl-Dickstein

Proceedings of the 7th International Conference on Learning Representations, 2019

KAMA-NNs: Low-dimensional Rotation Based Neural Networks.

Krzysztof Choromanski, Aldo Pacchiano, Jeffrey Pennington, Yunhao Tang

Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, 2019

2018

Bayesian Convolutional Neural Networks with Many Channels are Gaussian Processes.

Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, Jascha Sohl-Dickstein

CoRR, 2018

The Spectrum of the Fisher Information Matrix of a Single-Hidden-Layer Neural Network.

Jeffrey Pennington, Pratik Worah

Proceedings of the Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, 2018

Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10, 000-Layer Vanilla Convolutional Neural Networks.

Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S. Schoenholz, Jeffrey Pennington

Proceedings of the 35th International Conference on Machine Learning, 2018

Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks.

Minmin Chen, Jeffrey Pennington, Samuel S. Schoenholz

Proceedings of the 35th International Conference on Machine Learning, 2018

Sensitivity and Generalization in Neural Networks: an Empirical Study.

Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, Jascha Sohl-Dickstein

Proceedings of the 6th International Conference on Learning Representations, 2018

Deep Neural Networks as Gaussian Processes.

Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein

Proceedings of the 6th International Conference on Learning Representations, 2018

The emergence of spectral universality in deep networks.

Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli

Proceedings of the International Conference on Artificial Intelligence and Statistics, 2018

2017

A Correspondence Between Random Neural Networks and Statistical Field Theory.

Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein

CoRR, 2017

Nonlinear random matrix theory for deep learning.

Jeffrey Pennington, Pratik Worah

Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 2017

Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice.

Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli

Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 2017

Geometry of Neural Network Loss Surfaces via Random Matrix Theory.

Jeffrey Pennington, Yasaman Bahri

Proceedings of the 34th International Conference on Machine Learning, 2017

2016

Clinical Data Research Network Lessons Learned.

Kevin Murphy, Jeffrey Pennington, Aaron N. Browne, Byron Ruth, Ritu Khare, LeMar Davidson, Patricia K. Morris, Levon H. Utidjian, Charles Bailey

Proceedings of the Summit on Clinical Research Informatics, 2016

2015

Spherical Random Features for Polynomial Kernels.

Jeffrey Pennington, Felix X. Yu, Sanjiv Kumar

Proceedings of the Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, 2015

2014

Glove: Global Vectors for Word Representation.

Jeffrey Pennington, Richard Socher, Christopher D. Manning

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, 2014

2011

Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection.

Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, Christopher D. Manning

Proceedings of the Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, 2011

Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions.

Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, Christopher D. Manning

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, 2011