Shaohuai Shi

According to our database, Shaohuai Shi authored at least 64 papers between 2010 and 2024.

Collaborative distances:

• Dijkstra number of four.
• Erdős number of four.

2024

Task Scheduling for Efficient Inference of Large Language Models on Single Moderate GPU Systems.

Wenxiang Lin, Xinglin Pan, Shaohuai Shi, Xuan Wang, Xiaowen Chu

CoRR, 2024

ExpertFlow: Optimized Expert Activation and Token Allocation for Efficient Mixture-of-Experts Inference.

Xin He, Shunkang Zhang, Yuxin Wang, Haiyan Yin, Zihao Zeng, Shaohuai Shi, Zhenheng Tang, Xiaowen Chu, Ivor W. Tsang, Yew-Soon Ong

CoRR, 2024

FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression.

Zhenheng Tang, Xueze Kang, Yiming Yin, Xinglin Pan, Yuxin Wang, Xin He, Qiang Wang, Rongfei Zeng, Kaiyong Zhao, Shaohuai Shi, Amelie Chi Zhou, Bo Li, Bingsheng He, Xiaowen Chu

CoRR, 2024

Scheduling Deep Learning Jobs in Multi-Tenant GPU Clusters via Wise Resource Sharing.

Yizhou Luo, Qiang Wang, Shaohuai Shi, Jiaxin Lai, Shuhan Qi, Jiajia Zhang, Xuan Wang

Proceedings of the 32nd IEEE/ACM International Symposium on Quality of Service, 2024

Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules.

Xinglin Pan, Wenxiang Lin, Shaohuai Shi, Xiaowen Chu, Weinong Sun, Bo Li

Proceedings of the IEEE INFOCOM 2024, 2024

Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning.

Zichen Tang, Junlin Huang, Rudan Yan, Yuxin Wang, Zhenheng Tang, Shaohuai Shi, Amelie Chi Zhou, Xiaowen Chu

Proceedings of the 53rd International Conference on Parallel Processing, 2024

Sparse Gradient Communication with AlltoAll for Accelerating Distributed Deep Learning.

Jing Peng, Zihan Li, Shaohuai Shi, Bo Li

Proceedings of the 53rd International Conference on Parallel Processing, 2024

FedImpro: Measuring and Improving Client Update in Federated Learning.

Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xinmei Tian, Tongliang Liu, Bo Han, Xiaowen Chu

Proceedings of the Twelfth International Conference on Learning Representations, 2024

ScheMoE: An Extensible Mixture-of-Experts Distributed Training System with Tasks Scheduling.

Shaohuai Shi, Xinglin Pan, Qiang Wang, Chengjian Liu, Xiaozhe Ren, Zhongzhe Hu, Yu Yang, Bo Li, Xiaowen Chu

Proceedings of the Nineteenth European Conference on Computer Systems, 2024

Performance Analysis and Optimizations of Matrix Multiplications on ARMv8 Processors.

Hucheng Liu, Shaohuai Shi, Xuan Wang, Zoe L. Jiang, Qian Chen

Proceedings of the Design, Automation & Test in Europe Conference & Exhibition, 2024

2023

GossipFL: A Decentralized Federated Learning Framework With Sparsified and Adaptive Communication.

Zhenheng Tang, Shaohuai Shi, Bo Li, Xiaowen Chu

IEEE Trans. Parallel Distributed Syst., March, 2023

Scalable K-FAC Training for Deep Neural Networks With Distributed Preconditioning.

Lin Zhang, Shaohuai Shi, Wei Wang, Bo Li

IEEE Trans. Cloud Comput., 2023

Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models.

Longteng Zhang, Xiang Liu, Zeyu Li, Xinglin Pan, Peijie Dong, Ruibo Fan, Rui Guo, Xin Wang, Qiong Luo, Shaohuai Shi, Xiaowen Chu

CoRR, 2023

Reliable and Efficient In-Memory Fault Tolerance of Large Language Model Pretraining.

Yuxin Wang, Shaohuai Shi, Xin He, Zhenheng Tang, Xinglin Pan, Yang Zheng, Xiaoyu Wu, Amelie Chi Zhou, Bingsheng He, Xiaowen Chu

CoRR, 2023

FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs.

Zhenheng Tang, Yuxin Wang, Xin He, Longteng Zhang, Xinglin Pan, Qiang Wang, Rongfei Zeng, Kaiyong Zhao, Shaohuai Shi, Bingsheng He, Xiaowen Chu

CoRR, 2023

LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning.

Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, Bo Li

CoRR, 2023

Eva: A General Vectorized Approximation Framework for Second-order Optimization.

Lin Zhang, Shaohuai Shi, Bo Li

CoRR, 2023

A Generic Multi-Player Transformation Algorithm for Solving Large-Scale Zero-Sum Extensive-Form Adversarial Team Games.

Chen Qiu, Yulin Wu, Weixin Huang, Botao Liu, Shaohuai Shi, Xuan Wang

CoRR, 2023

FedML Parrot: A Scalable Federated Learning System via Heterogeneity-aware Scheduling on Sequential and Hierarchical Training.

Zhenheng Tang, Xiaowen Chu, Ryan Yide Ran, Sunwoo Lee, Shaohuai Shi, Yonggang Zhang, Yuxin Wang, Alex Qiaozhong Liang, Salman Avestimehr, Chaoyang He

CoRR, 2023

Decoupling the All-Reduce Primitive for Accelerating Distributed Deep Learning.

Lin Zhang, Shaohuai Shi, Xiaowen Chu, Wei Wang, Bo Li, Chengjian Liu

CoRR, 2023

Accelerating Distributed K-FAC with Efficient Collective Communication and Scheduling.

Lin Zhang, Shaohuai Shi, Bo Li

Proceedings of the IEEE INFOCOM 2023, 2023

PipeMoE: Accelerating Mixture-of-Experts through Adaptive Pipelining.

Shaohuai Shi, Xinglin Pan, Xiaowen Chu, Bo Li

Proceedings of the IEEE INFOCOM 2023, 2023

Eva: Practical Second-order Optimization with Kronecker-vectorized Approximation.

Lin Zhang, Shaohuai Shi, Bo Li

Proceedings of the Eleventh International Conference on Learning Representations, 2023

Evaluation and Optimization of Gradient Compression for Distributed Deep Learning.

Lin Zhang, Longteng Zhang, Shaohuai Shi, Xiaowen Chu, Bo Li

Proceedings of the 43rd IEEE International Conference on Distributed Computing Systems, 2023

DeAR: Accelerating Distributed Deep Learning with Fine-Grained All-Reduce Pipelining.

Lin Zhang, Shaohuai Shi, Xiaowen Chu, Wei Wang, Bo Li, Chengjian Liu

Proceedings of the 43rd IEEE International Conference on Distributed Computing Systems, 2023

2022

An Efficient Split Fine-tuning Framework for Edge and Cloud Collaborative Learning.

Shaohuai Shi, Qing Yang, Yang Xiang, Shuhan Qi, Xuan Wang

CoRR, 2022

Nebula-I: A General Framework for Collaboratively Training Deep Learning Models on Low-Bandwidth Cloud Clusters.

Yang Xiang, Zhihua Wu, Weibao Gong, Siyu Ding, Xianjie Mo, Yuang Liu, Shuohuan Wang, Peng Liu, Yongshuai Hou, Long Li, Bin Wang, Shaohuai Shi, Yaqian Han, Yue Yu, Ge Li, Yu Sun, Yanjun Ma, Dianhai Yu

CoRR, 2022

Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning.

Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xin He, Bo Han, Xiaowen Chu

Proceedings of the International Conference on Machine Learning, 2022

EASNet: Searching Elastic and Accurate Network Architecture for Stereo Matching.

Qiang Wang, Shaohuai Shi, Kaiyong Zhao, Xiaowen Chu

Proceedings of the Computer Vision - ECCV 2022, 2022

2021

MG-WFBP: Merging Gradients Wisely for Efficient Communication in Distributed Deep Learning.

Shaohuai Shi, Xiaowen Chu, Bo Li

IEEE Trans. Parallel Distributed Syst., 2021

A Quantitative Survey of Communication Optimizations in Distributed Deep Learning.

Shaohuai Shi, Zhenheng Tang, Xiaowen Chu, Chengjian Liu, Wei Wang, Bo Li

IEEE Netw., 2021

FADNet++: Real-Time and Accurate Disparity Estimation with Configurable Networks.

Qiang Wang, Shaohuai Shi, Shizhen Zheng, Kaiyong Zhao, Xiaowen Chu

CoRR, 2021

Automated Model Design and Benchmarking of 3D Deep Learning Models for COVID-19 Detection with Chest CT Scans.

Xin He, Shihao Wang, Xiaowen Chu, Shaohuai Shi, Jiangping Tang, Xin Liu, Chenggang Yan, Jiyong Zhang, Guiguang Ding

CoRR, 2021

Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters.

Shaohuai Shi, Xianhao Zhou, Shutao Song, Xingyao Wang, Zilin Zhu, Xue Huang, Xinan Jiang, Feihu Zhou, Zhenyu Guo, Liqiang Xie, Rui Lan, Xianbin Ouyang, Yan Zhang, Jieqian Wei, Jing Gong, Weiliang Lin, Ping Gao, Peng Meng, Xiaomin Xu, Chenyang Guo, Bo Yang, Zhibo Chen, Yongjian Wu, Xiaowen Chu

Proceedings of the Fourth Conference on Machine Learning and Systems, 2021

Exploiting Simultaneous Communications to Accelerate Data Parallel Distributed Deep Learning.

Shaohuai Shi, Xiaowen Chu, Bo Li

Proceedings of the 40th IEEE Conference on Computer Communications, 2021

Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Tasks.

Shaohuai Shi, Lin Zhang, Bo Li

Proceedings of the 41st IEEE International Conference on Distributed Computing Systems, 2021

Automated Model Design and Benchmarking of Deep Learning Models for COVID-19 Detection with Chest CT Scans.

Xin He, Shihao Wang, Xiaowen Chu, Shaohuai Shi, Jiangping Tang, Xin Liu, Chenggang Yan, Jiyong Zhang, Guiguang Ding

Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence, 2021

2020

Communication-Efficient Distributed Deep Learning: Survey, Evaluation, and Challenges.

Shaohuai Shi, Zhenheng Tang, Xiaowen Chu, Chengjian Liu, Wei Wang, Bo Li

CoRR, 2020

Communication-Efficient Distributed Deep Learning: A Comprehensive Survey.

Zhenheng Tang, Shaohuai Shi, Xiaowen Chu, Wei Wang, Bo Li

CoRR, 2020

Communication Contention Aware Scheduling of Multiple Deep Learning Training Jobs.

Qiang Wang, Shaohuai Shi, Canhui Wang, Xiaowen Chu

CoRR, 2020

Communication-Efficient Distributed Deep Learning with Merged Gradient Sparsification on GPUs.

Shaohuai Shi, Qiang Wang, Xiaowen Chu, Bo Li, Yang Qin, Ruihao Liu, Xinxiao Zhao

Proceedings of the 39th IEEE Conference on Computer Communications, 2020

FADNet: A Fast and Accurate Network for Disparity Estimation.

Qiang Wang, Shaohuai Shi, Shizhen Zheng, Kaiyong Zhao, Xiaowen Chu

Proceedings of the 2020 IEEE International Conference on Robotics and Automation, 2020

Efficient Sparse-Dense Matrix-Matrix Multiplication on GPUs Using the Customized Sparse Storage Format.

Shaohuai Shi, Qiang Wang, Xiaowen Chu

Proceedings of the 26th IEEE International Conference on Parallel and Distributed Systems, 2020

Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selection.

Zhenheng Tang, Shaohuai Shi, Xiaowen Chu

Proceedings of the 40th IEEE International Conference on Distributed Computing Systems, 2020

Layer-Wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees.

Shaohuai Shi, Zhenheng Tang, Qiang Wang, Kaiyong Zhao, Xiaowen Chu

Proceedings of the ECAI 2020 - 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de Compostela, Spain, August 29 - September 8, 2020, 2020

Benchmarking the Performance and Energy Efficiency of AI Accelerators for AI Training.

Yuxin Wang, Qiang Wang, Shaohuai Shi, Xin He, Zhenheng Tang, Kaiyong Zhao, Xiaowen Chu

Proceedings of the 20th IEEE/ACM International Symposium on Cluster, 2020

2019

Understanding Top-k Sparsification in Distributed Deep Learning.

Shaohuai Shi, Xiaowen Chu, Ka Chun Cheung, Simon See

CoRR, 2019

MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithms.

Shaohuai Shi, Xiaowen Chu, Bo Li

Proceedings of the 2019 IEEE Conference on Computer Communications, 2019

A Convergence Analysis of Distributed SGD with Communication-Efficient Gradient Sparsification.

Shaohuai Shi, Kaiyong Zhao, Qiang Wang, Zhenheng Tang, Xiaowen Chu

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, 2019

A Distributed Synchronous SGD Algorithm with Global Top-k Sparsification for Low Bandwidth Networks.

Shaohuai Shi, Qiang Wang, Kaiyong Zhao, Zhenheng Tang, Yuxin Wang, Xiang Huang, Xiaowen Chu

Proceedings of the 39th IEEE International Conference on Distributed Computing Systems, 2019

Computer-Aided Clinical Skin Disease Diagnosis Using CNN and Object Detection Models.

Xin He, Xiaoming Liu, Zhili Wu, Wu Yu, Xiaowen Chu, Shihao Wang, Shaohuai Shi, Zhenheng Tang, Yuxin Wang, Zhihao Zhao, Jing Dai, Ronghao Ni, Xiaofeng Zhang

Proceedings of the 2019 IEEE International Conference on Big Data (IEEE BigData), 2019

2018

MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithms.

Shaohuai Shi, Xiaowen Chu

CoRR, 2018

Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes.

Xianyan Jia, Shutao Song, Wei He, Yangzihao Wang, Haidong Rong, Feihu Zhou, Liqiang Xie, Zhenyu Guo, Yuanzhou Yang, Liwei Yu, Tiegang Chen, Guangxiao Hu, Shaohuai Shi, Xiaowen Chu

CoRR, 2018

Modeling and Evaluation of Synchronous Stochastic Gradient Descent in Distributed Deep Learning on Multiple GPUs.

Shaohuai Shi, Qiang Wang, Xiaowen Chu, Bo Li

CoRR, 2018

A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning.

Shaohuai Shi, Qiang Wang, Xiaowen Chu, Bo Li

Proceedings of the 24th IEEE International Conference on Parallel and Distributed Systems, 2018

Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs.

Shaohuai Shi, Qiang Wang, Xiaowen Chu

Proceedings of the 2018 IEEE 16th Intl Conf on Dependable, 2018

2017

Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs.

Shaohuai Shi, Xiaowen Chu

CoRR, 2017

Improving the Performance of Fully Connected Neural Networks by Out-of-Place Matrix Transpose.

Shaohuai Shi, Pengfei Xu, Xiaowen Chu

CoRR, 2017

Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units.

Shaohuai Shi, Xiaowen Chu

CoRR, 2017

Supervised Learning Based Algorithm Selection for Deep Neural Networks.

Shaohuai Shi, Pengfei Xu, Xiaowen Chu

Proceedings of the 23rd IEEE International Conference on Parallel and Distributed Systems, 2017

Performance Evaluation of Deep Learning Tools in Docker Containers.

Pengfei Xu, Shaohuai Shi, Xiaowen Chu

Proceedings of the 3rd International Conference on Big Data Computing and Communications, 2017

2016

Benchmarking State-of-the-Art Deep Learning Software Tools.

Shaohuai Shi, Qiang Wang, Pengfei Xu, Xiaowen Chu

Proceedings of the 7th International Conference on Cloud Computing and Big Data, 2016

2011

Mixed Precision Method for GPU-based FFT.

Shuhan Qi, Xuan Wang, Shaohuai Shi

Proceedings of the 14th IEEE International Conference on Computational Science and Engineering, 2011

2010

The GPU-based String Matching System in Advanced AC Algorithm.

Jiang-Feng Peng, Hu Chen, Shaohuai Shi

Proceedings of the 10th IEEE International Conference on Computer and Information Technology, 2010