Quentin Anthony

According to our database, Quentin Anthony authored at least 36 papers between 2019 and 2024.

Collaborative distances:

• Dijkstra number of four.
• Erdős number of four.

2024

Simple and Scalable Strategies to Continually Pre-train Large Language Models.

Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L. Richter, Quentin Gregory Anthony, Eugene Belilovsky, Timothée Lesort, Irina Rish

Trans. Mach. Learn. Res., 2024

Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters.

Vasudev Shyam, Jonathan Pilault, Emily Shepperd, Quentin Anthony, Beren Millidge

CoRR, 2024

Zyda: A 1.3T Dataset for Open Language Modeling.

Yury Tokpanov, Beren Millidge, Paolo Glorioso, Jonathan Pilault, Adam Ibrahim, James Whittington, Quentin Anthony

CoRR, 2024

Zamba: A Compact 7B SSM Hybrid Model.

Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, Beren Millidge

CoRR, 2024

Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence.

Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, Przemyslaw Kazienko, Kranthi Kiran GV, Jan Kocon, Bartlomiej Koptyra, Satyapriya Krishna, Ronald McClelland Jr., Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song, Haoqin Tu, Stanislaw Wozniak, Ruichong Zhang, Bingchen Zhao, Qihang Zhao, Peng Zhou, Jian Zhu, Rui-Jie Zhu

CoRR, 2024

BlackMamba: Mixture of Experts for State-Space Models.

Quentin Anthony, Yury Tokpanov, Paolo Glorioso, Beren Millidge

CoRR, 2024

Infer-HiRes: Accelerating Inference for High-Resolution Images with Quantization and Distributed Deep Learning.

Radha Gulhane, Quentin Anthony, Aamir Shafi, Hari Subramoni, Dhabaleswar K. Panda

Proceedings of the Practice and Experience in Advanced Research Computing 2024: Human Powered Computing, 2024

Comparative Study of Large Language Model Architectures on Frontier.

Junqi Yin, Avishek Bose, Guojing Cong, Isaac Lyngaas, Quentin Anthony

Proceedings of the IEEE International Parallel and Distributed Processing Symposium, 2024

Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference.

Jinghan Yao, Quentin Anthony, Aamir Shafi, Hari Subramoni, Dhabaleswar K. Panda

Proceedings of the IEEE International Parallel and Distributed Processing Symposium, 2024

The Case for Co-Designing Model Architectures with Hardware.

Quentin Anthony, Jacob Hatef, Deepak Narayanan, Stella Biderman, Stas Bekman, Junqi Yin, Aamir Shafi, Hari Subramoni, Dhabaleswar K. Panda

Proceedings of the 53rd International Conference on Parallel Processing, 2024

Demystifying the Communication Characteristics for Distributed Transformer Models.

Quentin Anthony, Benjamin Michalowicz, Jacob Hatef, Lang Xu, Mustafa Abdul Jabbar, Aamir Shafi, Hari Subramoni, Dhabaleswar K. Panda

Proceedings of the IEEE Symposium on High-Performance Interconnects, 2024

Accelerating Large Language Model Training with Hybrid GPU-based Compression.

Lang Xu, Quentin Anthony, Qinghua Zhou, Nawras Alnaasan, Radha Gulhane, Aamir Shafi, Hari Subramoni, Dhabaleswar K. D. K. Panda

Proceedings of the 24th IEEE International Symposium on Cluster, 2024

2023

Continual Pre-Training of Large Language Models: How to (re)warm your model?

Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats L. Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish, Timothée Lesort

CoRR, 2023

RWKV: Reinventing RNNs for the Transformer Era.

Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran G. V., Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Xiangru Tang, Bolun Wang, Johan S. Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Jian Zhu, Rui-Jie Zhu

CoRR, 2023

Emergent and Predictable Memorization in Large Language Models.

Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, Edward Raff

Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, 2023

Accelerating Distributed Deep Learning Training with Compression Assisted Allgather and Reduce-Scatter Communication.

Qinghua Zhou, Quentin Anthony, Lang Xu, Aamir Shafi, Mustafa Abduljabbar, Hari Subramoni, Dhabaleswar K. Panda

Proceedings of the IEEE International Parallel and Distributed Processing Symposium, 2023

MCR-DL: Mix-and-Match Communication Runtime for Deep Learning.

Quentin Anthony, Ammar Ahmad Awan, Jeff Rasley, Yuxiong He, Aamir Shafi, Mustafa Abduljabbar, Hari Subramoni, Dhabaleswar K. Panda

Proceedings of the IEEE International Parallel and Distributed Processing Symposium, 2023

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling.

Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, Oskar van der Wal

Proceedings of the International Conference on Machine Learning, 2023

RWKV: Reinventing RNNs for the Transformer Era.

Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan S. Wind, Stanislaw Wozniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, Rui-Jie Zhu

Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023, 2023

trlX: A Framework for Large Scale Reinforcement Learning from Human Feedback.

Alexander Havrilla, Maksym Zhuravinskyi, Duy Phung, Aman Tiwari, Jonathan Tow, Stella Biderman, Quentin Anthony, Louis Castricato

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023

ScaMP: Scalable Meta-Parallelism for Deep Learning Search.

Quentin Anthony, Lang Xu, Aamir Shafi, Hari Subramoni, Dhabaleswar K. Panda

Proceedings of the 23rd IEEE/ACM International Symposium on Cluster, 2023

2022

GPT-NeoX-20B: An Open-Source Autoregressive Language Model.

Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, Samuel Weinbach

CoRR, 2022

Accelerating MPI All-to-All Communication with Online Compression on Modern GPU Clusters.

Qinghua Zhou, Pouya Kousha, Quentin Anthony, Kawthar Shafie Khorassani, Aamir Shafi, Hari Subramoni, Dhabaleswar K. Panda

Proceedings of the High Performance Computing - 37th International Conference, 2022

Hy-Fi: Hybrid Five-Dimensional Parallel DNN Training on High-Performance GPU Clusters.

Arpan Jain, Aamir Shafi, Quentin Anthony, Pouya Kousha, Hari Subramoni, Dhabaleswar K. Panda

Proceedings of the High Performance Computing - 37th International Conference, 2022

Highly Efficient Alltoall and Alltoallv Communication Algorithms for GPU Systems.

Chen-Chun Chen, Kawthar Shafie Khorassani, Quentin G. Anthony, Aamir Shafi, Hari Subramoni, Dhabaleswar K. Panda

Proceedings of the IEEE International Parallel and Distributed Processing Symposium, 2022

Accelerating Broadcast Communication with GPU Compression for Deep Learning Workloads.

Qinghua Zhou, Quentin Anthony, Aamir Shafi, Hari Subramoni, Dhabaleswar K. Panda

Proceedings of the 29th IEEE International Conference on High Performance Computing, 2022

2021

Cross-layer Visualization and Profiling of Network and I/O Communication for HPC Clusters.

Pouya Kousha, Quentin Anthony, Hari Subramoni, Dhabaleswar K. Panda

CoRR, 2021

Evaluating Multi-Level Checkpointing for Distributed Deep Neural Network Training.

Quentin Anthony, Donglai Dai

Proceedings of the 2021 SC Workshops Supplementary Proceedings, 2021

Scaling Single-Image Super-Resolution Training on Modern HPC Clusters: Early Experiences.

Quentin Anthony, Lang Xu, Hari Subramoni, Dhabaleswar K. D. K. Panda

Proceedings of the IEEE International Parallel and Distributed Processing Symposium Workshops, 2021

Adaptive and Hierarchical Large Message All-to-all Communication Algorithms for Large-scale Dense GPU Systems.

Kawthar Shafie Khorassani, Ching-Hsiang Chu, Quentin G. Anthony, Hari Subramoni, Dhabaleswar K. Panda

Proceedings of the 21st IEEE/ACM International Symposium on Cluster, 2021

2020

HyPar-Flow: Exploiting MPI and Keras for Scalable Hybrid-Parallel DNN Training with TensorFlow.

Ammar Ahmad Awan, Arpan Jain, Quentin Anthony, Hari Subramoni, Dhabaleswar K. Panda

Proceedings of the High Performance Computing - 35th International Conference, 2020

GEMS: GPU-enabled memory-aware model-parallelism system for distributed DNN training.

Arpan Jain, Ammar Ahmad Awan, Asmaa M. Aljuhani, Jahanzeb Maqbool Hashmi, Quentin G. Anthony, Hari Subramoni, Dhabaleswar K. Panda, Raghu Machiraju, Anil Parwani

Proceedings of the International Conference for High Performance Computing, 2020

Accelerating GPU-based Machine Learning in Python using MPI Library: A Case Study with MVAPICH2-GDR.

Seyedeh Mahdieh Ghazimirsaeed, Quentin Anthony, Aamir Shafi, Hari Subramoni, Dhabaleswar K. D. K. Panda

Proceedings of the 6th IEEE/ACM Workshop on Machine Learning in High Performance Computing Environments, 2020

Efficient Training of Semantic Image Segmentation on Summit using Horovod and MVAPICH2-GDR.

Quentin Anthony, Ammar Ahmad Awan, Arpan Jain, Hari Subramoni, Dhabaleswar K. D. K. Panda

Proceedings of the 2020 IEEE International Parallel and Distributed Processing Symposium Workshops, 2020

2019

HyPar-Flow: Exploiting MPI and Keras for Scalable Hybrid-Parallel DNN Training using TensorFlow.

Ammar Ahmad Awan, Arpan Jain, Quentin Anthony, Hari Subramoni, Dhabaleswar K. Panda

CoRR, 2019

Performance Characterization of DNN Training using TensorFlow and PyTorch on Modern Clusters.

Arpan Jain, Ammar Ahmad Awan, Quentin Anthony, Hari Subramoni, Dhabaleswar K. Panda

Proceedings of the 2019 IEEE International Conference on Cluster Computing, 2019