Shiwei Liu

According to our database, Shiwei Liu authored at least 66 papers between 2019 and 2024.

Collaborative distances:

• Dijkstra number of four.
• Erdős number of four.

2024

Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Models.

Adriana Fernandez-Lopez, Shiwei Liu, Lu Yin, Stavros Petridis, Maja Pantic

CoRR, 2024

(PASS) Visual Prompt Locates Good Structure Sparsity through a Recurrent HyperNetwork.

Tianjin Huang, Meng Fang, Li Shen, Fan Liu, Yulong Pei, Mykola Pechenizkiy, Shiwei Liu, Tianlong Chen

CoRR, 2024

From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients.

Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu, Jiawei Zhao, Yuandong Tian, Zhangyang Wang

CoRR, 2024

Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.

Zhenyu Zhang, Ajay Jaiswal, Lu Yin, Shiwei Liu, Jiawei Zhao, Yuandong Tian, Zhangyang Wang

CoRR, 2024

Composable Interventions for Language Models.

Arinbjorn Kolbeinsson, Kyle O'Brien, Tianjin Huang, Shanghua Gao, Shiwei Liu, Jonathan Richard Schwarz, Anurag Vaidya, Faisal Mahmood, Marinka Zitnik, Tianlong Chen, Thomas Hartvigsen

CoRR, 2024

Dynamic Data Pruning for Automatic Speech Recognition.

Qiao Xiao, Pingchuan Ma, Adriana Fernandez-Lopez, Boqian Wu, Lu Yin, Stavros Petridis, Mykola Pechenizkiy, Maja Pantic, Decebal Constantin Mocanu, Shiwei Liu

CoRR, 2024

MSRS: Training Multimodal Speech Recognition Models from Scratch with Sparse Mask Optimization.

Adriana Fernandez-Lopez, Honglie Chen, Pingchuan Ma, Lu Yin, Qiao Xiao, Stavros Petridis, Shiwei Liu, Maja Pantic

CoRR, 2024

Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusion.

Anke Tang, Li Shen, Yong Luo, Shiwei Liu, Han Hu, Bo Du

CoRR, 2024

OwLore: Outlier-weighed Layerwise Sampled Low-Rank Projection for Memory-Efficient LLM Fine-tuning.

Pengxiang Li, Lu Yin, Xiaowei Gao, Shiwei Liu

CoRR, 2024

Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding.

Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, Zhangyang Wang

CoRR, 2024

Q-Hitter: A Better Token Oracle for Efficient LLM Inference via Sparse-Quantized KV Cache.

Zhenyu Zhang, Shiwei Liu, Runjin Chen, Bhavya Kailkhura, Beidi Chen, Atlas Wang

Proceedings of the Seventh Annual Conference on Machine Learning and Systems, 2024

Sparse Cocktail: Every Sparse Pattern Every Sparse Ratio All At Once.

Zhangheng Li, Shiwei Liu, Tianlong Chen, Ajay Kumar Jaiswal, Zhenyu Zhang, Dilin Wang, Raghuraman Krishnamoorthi, Shiyu Chang, Zhangyang Wang

Proceedings of the Forty-first International Conference on Machine Learning, 2024

Advancing Dynamic Sparse Training by Exploring Optimization Opportunities.

Jie Ji, Gen Li, Lu Yin, Minghai Qin, Geng Yuan, Linke Guo, Shiwei Liu, Xiaolong Ma

Proceedings of the Forty-first International Conference on Machine Learning, 2024

Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity.

Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Kumar Jaiswal, Mykola Pechenizkiy, Yi Liang, Michael Bendersky, Zhangyang Wang, Shiwei Liu

Proceedings of the Forty-first International Conference on Machine Learning, 2024

Junk DNA Hypothesis: Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMs.

Lu Yin, Ajay Kumar Jaiswal, Shiwei Liu, Souvik Kundu, Zhangyang Wang

Proceedings of the Forty-first International Conference on Machine Learning, 2024

CaM: Cache Merging for Memory-efficient LLMs Inference.

Yuxin Zhang, Yuxuan Du, Gen Luo, Yunshan Zhong, Zhenyu Zhang, Shiwei Liu, Rongrong Ji

Proceedings of the Forty-first International Conference on Machine Learning, 2024

AdaMerging: Adaptive Model Merging for Multi-Task Learning.

Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, Dacheng Tao

Proceedings of the Twelfth International Conference on Learning Representations, 2024

NeurRev: Train Better Sparse Neural Network Practically via Neuron Revitalization.

Gen Li, Lu Yin, Jie Ji, Wei Niu, Minghai Qin, Bin Ren, Linke Guo, Shiwei Liu, Xiaolong Ma

Proceedings of the Twelfth International Conference on Learning Representations, 2024

Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs.

Yuxin Zhang, Lirui Zhao, Mingbao Lin, Yunyun Sun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, Rongrong Ji

Proceedings of the Twelfth International Conference on Learning Representations, 2024

FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping.

Ajay Jaiswal, Bodun Hu, Lu Yin, Yeonju Ro, Tianlong Chen, Shiwei Liu, Aditya Akella

Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024

Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning.

Abhinav Bandari, Lu Yin, Cheng-Yu Hsieh, Ajay Jaiswal, Tianlong Chen, Li Shen, Ranjay Krishna, Shiwei Liu

Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024

2023

Don't Be So Dense: Sparse-to-Sparse GAN Training Without Sacrificing Performance.

Shiwei Liu, Yuesong Tian, Tianlong Chen, Li Shen

Int. J. Comput. Vis., October, 2023

Supervised Feature Selection with Neuron Evolution in Sparse Neural Networks.

Zahra Atashgahi, Xuhao Zhang, Neil Kichler, Shiwei Liu, Lu Yin, Mykola Pechenizkiy, Raymond N. J. Veldhuis, Decebal Constantin Mocanu

Trans. Mach. Learn. Res., 2023

The Counterattack of CNNs in Self-Supervised Learning: Larger Kernel Size might be All You Need.

Tianjin Huang, Tianlong Chen, Zhangyang Wang, Shiwei Liu

CoRR, 2023

E2ENet: Dynamic Sparse Feature Fusion for Accurate and Efficient 3D Medical Image Segmentation.

Boqian Wu, Qiao Xiao, Shiwei Liu, Lu Yin, Mykola Pechenizkiy, Decebal Constantin Mocanu, Maurice van Keulen, Elena Mocanu

CoRR, 2023

Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspective.

Can Jin, Tianjin Huang, Yihua Zhang, Mykola Pechenizkiy, Sijia Liu, Shiwei Liu, Tianlong Chen

CoRR, 2023

Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity.

Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Mykola Pechenizkiy, Yi Liang, Zhangyang Wang, Shiwei Liu

CoRR, 2023

Junk DNA Hypothesis: A Task-Centric Angle of LLM Pre-trained Weights through Sparsity.

Lu Yin, Shiwei Liu, Ajay Jaiswal, Souvik Kundu, Zhangyang Wang

CoRR, 2023

Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researchers.

Shiwei Liu, Zhangyang Wang

CoRR, 2023

REST: Enhancing Group Robustness in DNNs Through Reweighted Sparse Training.

Jiaxu Zhao, Lu Yin, Shiwei Liu, Meng Fang, Mykola Pechenizkiy

Proceedings of the Machine Learning and Knowledge Discovery in Databases: Research Track, 2023

Enhancing Adversarial Training via Reweighting Optimization Trajectory.

Tianjin Huang, Shiwei Liu, Tianlong Chen, Meng Fang, Li Shen, Vlado Menkovski, Lu Yin, Yulong Pei, Mykola Pechenizkiy

Proceedings of the Machine Learning and Knowledge Discovery in Databases: Research Track, 2023

Towards Data-Agnostic Pruning At Initialization: What Makes a Good Sparse Mask?

Hoang Pham, The-Anh Ta, Shiwei Liu, Lichuan Xiang, Dung Le, Hongkai Wen, Long Tran-Thanh

Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, 2023

The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter.

Ajay Jaiswal, Shiwei Liu, Tianlong Chen, Zhangyang Wang

Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, 2023

Don't just prune by magnitude! Your mask topology is a secret weapon.

Duc Hoang, Souvik Kundu, Shiwei Liu, Zhangyang Wang

Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, 2023

Dynamic Sparsity Is Channel-Level Sparsity Learner.

Lu Yin, Gen Li, Meng Fang, Li Shen, Tianjin Huang, Zhangyang Wang, Vlado Menkovski, Xiaolong Ma, Mykola Pechenizkiy, Shiwei Liu

Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, 2023

Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Models.

Ajay Kumar Jaiswal, Shiwei Liu, Tianlong Chen, Ying Ding, Zhangyang Wang

Proceedings of the International Conference on Machine Learning, 2023

Graph Ladling: Shockingly Simple Parallel GNN Training without Intermediate Communication.

Ajay Kumar Jaiswal, Shiwei Liu, Tianlong Chen, Ying Ding, Zhangyang Wang

Proceedings of the International Conference on Machine Learning, 2023

Are Large Kernels Better Teachers than Transformers for ConvNets?

Tianjin Huang, Lu Yin, Zhenyu Zhang, Li Shen, Meng Fang, Mykola Pechenizkiy, Zhangyang Wang, Shiwei Liu

Proceedings of the International Conference on Machine Learning, 2023

Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!

Shiwei Liu, Tianlong Chen, Zhenyu Zhang, Xuxi Chen, Tianjin Huang, Ajay Kumar Jaiswal, Zhangyang Wang

Proceedings of the Eleventh International Conference on Learning Representations, 2023

More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity.

Shiwei Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Tommi Kärkkäinen, Mykola Pechenizkiy, Decebal Constantin Mocanu, Zhangyang Wang

Proceedings of the Eleventh International Conference on Learning Representations, 2023

Revisiting Pruning at Initialization Through the Lens of Ramanujan Graph.

Duc N. M. Hoang, Shiwei Liu, Radu Marculescu, Zhangyang Wang

Proceedings of the Eleventh International Conference on Learning Representations, 2023

Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers.

Tianlong Chen, Zhenyu Zhang, Ajay Kumar Jaiswal, Shiwei Liu, Zhangyang Wang

Proceedings of the Eleventh International Conference on Learning Representations, 2023

Data Augmented Flatness-aware Gradient Projection for Continual Learning.

Enneng Yang, Li Shen, Zhenyi Wang, Shiwei Liu, Guibing Guo, Xingwei Wang

Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023

Many-Task Federated Learning: A New Problem Setting and A Simple Baseline.

Ruisi Cai, Xiaohan Chen, Shiwei Liu, Jayanth Srinivasa, Myungjin Lee, Ramana Kompella, Zhangyang Wang

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023

Lottery Pools: Winning More by Interpolating Tickets without Increasing Training or Inference Cost.

Lu Yin, Shiwei Liu, Meng Fang, Tianjin Huang, Vlado Menkovski, Mykola Pechenizkiy

Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence, 2023

2022

A brain-inspired algorithm for training highly sparse neural networks.

Zahra Atashgahi, Joost Pieterse, Shiwei Liu, Decebal Constantin Mocanu, Raymond N. J. Veldhuis, Mykola Pechenizkiy

Mach. Learn., 2022

More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity.

Shiwei Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Mykola Pechenizkiy, Decebal Constantin Mocanu, Zhangyang Wang

CoRR, 2022

Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Training.

Lu Yin, Vlado Menkovski, Meng Fang, Tianjin Huang, Yulong Pei, Mykola Pechenizkiy, Decebal Constantin Mocanu, Shiwei Liu

CoRR, 2022

Achieving Personalized Federated Learning with Sparse Local Models.

Tiansheng Huang, Shiwei Liu, Li Shen, Fengxiang He, Weiwei Lin, Dacheng Tao

CoRR, 2022

Dynamic Sparse Network for Time Series Classification: Learning What to "See".

Qiao Xiao, Boqian Wu, Yu Zhang, Shiwei Liu, Mykola Pechenizkiy, Elena Mocanu, Decebal Constantin Mocanu

Proceedings of the Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, 2022

You Can Have Better Graph Neural Networks by Not Training Weights at All: Finding Untrained GNNs Tickets.

Tianjin Huang, Tianlong Chen, Meng Fang, Vlado Menkovski, Jiaxu Zhao, Lu Yin, Yulong Pei, Decebal Constantin Mocanu, Zhangyang Wang, Mykola Pechenizkiy, Shiwei Liu

Proceedings of the Learning on Graphs Conference, 2022

The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training.

Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, Mykola Pechenizkiy

Proceedings of the Tenth International Conference on Learning Representations, 2022

Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity.

Shiwei Liu, Tianlong Chen, Zahra Atashgahi, Xiaohan Chen, Ghada Sokar, Elena Mocanu, Mykola Pechenizkiy, Zhangyang Wang, Decebal Constantin Mocanu

Proceedings of the Tenth International Conference on Learning Representations, 2022

2021

Efficient and effective training of sparse recurrent neural networks.

Shiwei Liu, Iftitahu Ni'mah, Vlado Menkovski, Decebal Constantin Mocanu, Mykola Pechenizkiy

Neural Comput. Appl., 2021

Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware.

Shiwei Liu, Decebal Constantin Mocanu, Amarsagar Reddy Ramapuram Matavalam, Yulong Pei, Mykola Pechenizkiy

Neural Comput. Appl., 2021

FreeTickets: Accurate, Robust and Efficient Deep Ensemble by Training with Dynamic Sparsity.

Shiwei Liu, Tianlong Chen, Zahra Atashgahi, Xiaohan Chen, Ghada Sokar, Elena Mocanu, Mykola Pechenizkiy, Zhangyang Wang, Decebal Constantin Mocanu

CoRR, 2021

Sparse Training via Boosting Pruning Plasticity with Neuroregeneration.

Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, Decebal Constantin Mocanu

Proceedings of the Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, 2021

Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training.

Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, Mykola Pechenizkiy

Proceedings of the 38th International Conference on Machine Learning, 2021

Selfish Sparse RNN Training.

Shiwei Liu, Decebal Constantin Mocanu, Yulong Pei, Mykola Pechenizkiy

Proceedings of the 38th International Conference on Machine Learning, 2021

Hierarchical Semantic Segmentation using Psychometric Learning.

Lu Yin, Vlado Menkovski, Shiwei Liu, Mykola Pechenizkiy

Proceedings of the Asian Conference on Machine Learning, 2021

2020

Topological Insights in Sparse Neural Networks.

Shiwei Liu, Tim van der Lee, Anil Yaman, Zahra Atashgahi, Davide Ferraro, Ghada Sokar, Mykola Pechenizkiy, Decebal Constantin Mocanu

CoRR, 2020

Topological Insights into Sparse Neural Networks.

Shiwei Liu, Tim van der Lee, Anil Yaman, Zahra Atashgahi, Davide Ferraro, Ghada Sokar, Mykola Pechenizkiy, Decebal Constantin Mocanu

Proceedings of the Machine Learning and Knowledge Discovery in Databases, 2020

Learning Sparse Neural Networks for Better Generalization.

Shiwei Liu

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, 2020

Network Performance Optimization with Real Time Traffic Prediction in Data Center Network.

Fulong Yan, Shiwei Liu, Nicola Calabretta

Proceedings of the European Conference on Optical Communications, 2020

2019

On improving deep learning generalization with adaptive sparse connectivity.

Shiwei Liu, Decebal Constantin Mocanu, Mykola Pechenizkiy

CoRR, 2019

Intrinsically Sparse Long Short-Term Memory Networks.

Shiwei Liu, Decebal Constantin Mocanu, Mykola Pechenizkiy

CoRR, 2019