Nathan Godey

According to our database, Nathan Godey authored at least 7 papers between 2022 and 2024.

Collaborative distances:

• Dijkstra number of five.
• Erdős number of five.

2024

Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck.

Nathan Godey, Éric de la Clergerie, Benoît Sagot

CoRR, 2024

Headless Language Models: Learning without Predicting with Contrastive Weight Tying.

Nathan Godey, Éric Villemonte de la Clergerie, Benoît Sagot

Proceedings of the Twelfth International Conference on Learning Representations, 2024

Anisotropy Is Inherent to Self-Attention in Transformers.

Nathan Godey, Éric Villemonte de la Clergerie, Benoît Sagot

Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, 2024

On the Scaling Laws of Geographical Representation in Language Models.

Nathan Godey, Éric de la Clergerie, Benoît Sagot

Proceedings of the 2024 Joint International Conference on Computational Linguistics, 2024

2023

Is Anisotropy Inherent to Transformers?

Nathan Godey, Éric de la Clergerie, Benoît Sagot

CoRR, 2023

2022

MANTa: Efficient Gradient-Based Tokenization for Robust End-to-End Language Modeling.

Nathan Godey, Roman Castagné, Éric de la Clergerie, Benoît Sagot

CoRR, 2022

MANTa: Efficient Gradient-Based Tokenization for End-to-End Robust Language Modeling.

Nathan Godey, Roman Castagné, Éric de la Clergerie, Benoît Sagot

Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2022, 2022