Olatunji Ruwase

According to our database, Olatunji Ruwase authored at least 42 papers between 2004 and 2024.

Collaborative distances:

• Dijkstra number of three.
• Erdős number of three.

2024

Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping.

Guanhua Wang, Chengming Zhang, Zheyu Shen, Ang Li, Olatunji Ruwase

CoRR, 2024

Training Ultra Long Context Language Model with Fully Pipelined Distributed Transformer.

Jinghan Yao, Sam Ade Jacobs, Masahiro Tanaka, Olatunji Ruwase, Aamir Shafi, Hari Subramoni, Dhabaleswar K. Panda

CoRR, 2024

Universal Checkpointing: Efficient and Flexible Checkpointing for Large Scale Distributed Training.

Xinyu Lian, Sam Ade Jacobs, Lev Kurilenko, Masahiro Tanaka, Stas Bekman, Olatunji Ruwase, Minjia Zhang

CoRR, 2024

FastPersist: Accelerating Model Checkpointing in Deep Learning.

Guanhua Wang, Olatunji Ruwase, Bing Xie, Yuxiong He

CoRR, 2024

Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone.

Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Eric Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Masahiro Tanaka, Xin Wang, Rachel Ward, Guanhua Wang, Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang, Ziyi Yang, Donghan Yu, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, Xiren Zhou

CoRR, 2024

Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding.

Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, Zhangyang Wang

CoRR, 2024

FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design.

Haojun Xia, Zhen Zheng, Xiaoxia Wu, Shiyang Chen, Zhewei Yao, Stephen Youn, Arash Bakhtiari, Michael Wyatt, Donglin Zhuang, Zhongzhu Zhou, Olatunji Ruwase, Yuxiong He, Shuaiwen Leon Song

CoRR, 2024

Quant-LLM: Accelerating the Serving of Large Language Models via FP6-Centric Algorithm-System Co-Design on Modern GPUs.

Haojun Xia, Zhen Zheng, Xiaoxia Wu, Shiyang Chen, Zhewei Yao, Stephen Youn, Arash Bakhtiari, Michael Wyatt, Donglin Zhuang, Zhongzhu Zhou, Olatunji Ruwase, Yuxiong He, Shuaiwen Leon Song

Proceedings of the 2024 USENIX Annual Technical Conference, 2024

RecFlex: Enabling Feature Heterogeneity-Aware Optimization for Deep Recommendation Models with Flexible Schedules.

Zaifeng Pan, Zhen Zheng, Feng Zhang, Bing Xie, Ruofan Wu, Shaden Smith, Chuanjie Liu, Olatunji Ruwase, Xiaoyong Du, Yufei Ding

Proceedings of the International Conference for High Performance Computing, 2024

ZeRO++: Extremely Efficient Collective Communication for Large Model Training.

Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Xiaoxia Wu, Connor Holmes, Zhewei Yao, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, Yuxiong He

Proceedings of the Twelfth International Conference on Learning Representations, 2024

2023

SHARP: An Adaptable, Energy-Efficient Accelerator for Recurrent Neural Networks.

Reza Yazdani Aminabadi, Olatunji Ruwase, Minjia Zhang, Yuxiong He, José-María Arnau, Antonio González

ACM Trans. Embed. Comput. Syst., March, 2023

ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Tasks.

Xiaoxia Wu, Haojun Xia, Stephen Youn, Zhen Zheng, Shiyang Chen, Arash Bakhtiari, Michael Wyatt, Reza Yazdani Aminabadi, Yuxiong He, Olatunji Ruwase, Leon Song, Zhewei Yao

CoRR, 2023

DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attention.

Zhewei Yao, Xiaoxia Wu, Conglong Li, Minjia Zhang, Heyang Qin, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He

CoRR, 2023

DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales.

Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, Zhongzhu Zhou, Michael Wyatt, Molly Smith, Lev Kurilenko, Heyang Qin, Masahiro Tanaka, Shuai Che, Shuaiwen Leon Song, Yuxiong He

CoRR, 2023

ZeRO++: Extremely Efficient Collective Communication for Giant Model Training.

Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Connor Holmes, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, Yuxiong He

CoRR, 2023

A Novel Tensor-Expert Hybrid Parallelism Approach to Scale Mixture-of-Experts Training.

Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He, Abhinav Bhatele

CoRR, 2023

A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training.

Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He, Abhinav Bhatele

Proceedings of the 37th International Conference on Supercomputing, 2023

2022

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model.

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, et al.

CoRR, 2022

DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale.

Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, Yuxiong He

Proceedings of the SC22: International Conference for High Performance Computing, 2022

2021

ZeRO-Offload: Democratizing Billion-Scale Model Training.

Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, Yuxiong He

Proceedings of the 2021 USENIX Annual Technical Conference, 2021

ZeRO-infinity: breaking the GPU memory wall for extreme scale deep learning.

Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He

Proceedings of the International Conference for High Performance Computing, 2021

SimiGrad: Fine-Grained Adaptive Batching for Large Scale Training using Gradient Similarity Measurement.

Heyang Qin, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, Yuxiong He

Proceedings of the Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, 2021

2020

ZeRO: memory optimizations toward training trillion parameter models.

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He

Proceedings of the International Conference for High Performance Computing, 2020

DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters.

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, Yuxiong He

Proceedings of the KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2020

2019

LSTM-Sharp: An Adaptable, Energy-Efficient Hardware Accelerator for Long Short-Term Memory.

Reza Yazdani, Olatunji Ruwase, Minjia Zhang, Yuxiong He, José-María Arnau, Antonio González

CoRR, 2019

ZeRO: Memory Optimization Towards Training A Trillion Parameter Models.

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He

CoRR, 2019

Accelerating Large Scale Deep Learning Inference through DeepCPU at Microsoft.

Minjia Zhang, Samyam Rajbhandari, Wenhan Wang, Elton Zheng, Olatunji Ruwase, Jeff Rasley, Jason Li, Junhua Wang, Yuxiong He

Proceedings of the 2019 USENIX Conference on Operational Machine Learning, 2019

2018

Efficient Deep Neural Network Serving: Fast and Furious.

Feng Yan, Yuxiong He, Olatunji Ruwase, Evgenia Smirni

IEEE Trans. Netw. Serv. Manag., 2018

2017

HyperDrive: exploring hyperparameters with POP scheduling.

Jeff Rasley, Yuxiong He, Feng Yan, Olatunji Ruwase, Rodrigo Fonseca

Proceedings of the 18th ACM/IFIP/USENIX Middleware Conference, Las Vegas, NV, USA, December 11, 2017

Optimizing CNNs on Multicores for Scalability, Performance and Goodput.

Samyam Rajbhandari, Yuxiong He, Olatunji Ruwase, Michael Carbin, Trishul M. Chilimbi

Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems, 2017

2016

SERF: efficient scheduling for fast deep neural network serving via judicious parallelism.

Feng Yan, Yuxiong He, Olatunji Ruwase, Evgenia Smirni

Proceedings of the International Conference for High Performance Computing, 2016

2015

Performance Modeling and Scalability Optimization of Distributed Deep Learning Systems.

Feng Yan, Olatunji Ruwase, Yuxiong He, Trishul M. Chilimbi

Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015

Page overlays: an enhanced virtual memory framework to enable fine-grained memory management.

Vivek Seshadri, Gennady Pekhimenko, Olatunji Ruwase, Onur Mutlu, Phillip B. Gibbons, Michael A. Kozuch, Todd C. Mowry, Trishul M. Chilimbi

Proceedings of the 42nd Annual International Symposium on Computer Architecture, 2015

Toward accelerating deep learning at scale using specialized hardware in the datacenter.

Kalin Ovtcharov, Olatunji Ruwase, Joo-Young Kim, Jeremy Fowers, Karin Strauss, Eric S. Chung

Proceedings of the 2015 IEEE Hot Chips 27 Symposium (HCS), 2015

2014

Guardrail: a high fidelity approach to protecting hardware devices from buggy drivers.

Olatunji Ruwase, Michael A. Kozuch, Phillip B. Gibbons, Todd C. Mowry

Proceedings of the Architectural Support for Programming Languages and Operating Systems, 2014

2013

Improving Device Driver Reliability through Decoupled Dynamic Binary Analyses.

Olatunji Ruwase

PhD thesis, 2013

2010

Decoupled lifeguards: enabling path optimizations for dynamic correctness checking tools.

Olatunji Ruwase, Shimin Chen, Phillip B. Gibbons, Todd C. Mowry

Proceedings of the 2010 ACM SIGPLAN Conference on Programming Language Design and Implementation, 2010

2009

Flexible Hardware Acceleration for Instruction-Grain Lifeguards.

Shimin Chen, Michael Kozuch, Phillip B. Gibbons, Michael P. Ryan, Theodoros Strigkos, Todd C. Mowry, Olatunji Ruwase, Evangelos Vlachos, Babak Falsafi, Vijaya Ramachandran

IEEE Micro, 2009

2008

Parallelizing dynamic information flow tracking.

Olatunji Ruwase, Phillip B. Gibbons, Todd C. Mowry, Vijaya Ramachandran, Shimin Chen, Michael Kozuch, Michael P. Ryan

Proceedings of the SPAA 2008: Proceedings of the 20th Annual ACM Symposium on Parallelism in Algorithms and Architectures, 2008

Ditto: a system for opportunistic caching in multi-hop wireless networks.

Fahad R. Dogar, Amar Phanishayee, Himabindu Pucha, Olatunji Ruwase, David G. Andersen

Proceedings of the 14th Annual International Conference on Mobile Computing and Networking, 2008

Flexible Hardware Acceleration for Instruction-Grain Program Monitoring.

Shimin Chen, Michael Kozuch, Theodoros Strigkos, Babak Falsafi, Phillip B. Gibbons, Todd C. Mowry, Vijaya Ramachandran, Olatunji Ruwase, Michael P. Ryan, Evangelos Vlachos

Proceedings of the 35th International Symposium on Computer Architecture (ISCA 2008), 2008

2004

A Practical Dynamic Buffer Overflow Detector.

Olatunji Ruwase, Monica S. Lam

Proceedings of the Network and Distributed System Security Symposium, 2004