Computer Science > Computer Vision and Pattern Recognition
[Submitted on 23 Nov 2021 (v1), revised 16 Mar 2022 (this version, v2), latest version 16 Jan 2024 (v4)]
Title:Pruning Self-attentions into Convolutional Layers in Single Path
View PDFAbstract:Vision Transformers (ViTs) have achieved impressive performance over various computer vision tasks. However, modeling global correlations with multi-head self-attention (MSA) layers leads to two widely recognized issues: the massive computational resource consumption and the lack of intrinsic inductive bias for modeling local visual patterns like convolution. To solve both issues seamlessly, we devise a simple yet effective method named Single-Path Vision Transformer pruning (SPViT), to efficiently and automatically compress the pre-trained ViTs into compact models with proper locality added. Specifically, we first propose a novel weight-sharing scheme between MSA and convolutional operations, delivering a single-path space to encode all candidate operations. In this way, we cast the operation search problem as finding which subset of parameters to use in each MSA layer, which significantly reduces the computational cost and optimization difficulty, and the convolution kernels can be well initialized using pre-trained MSA parameters. Relying on the single-path space, we further introduce learnable binary gates to encode the operation choices, which are jointly optimized with network parameters to automatically determine the configuration of each layer. We conduct extensive experiments on two representative ViT models showing our method achieves a favorable accuracy-efficiency trade-off. For example, our SPViT achieves SOTA pruning performance by trimming 52.6% FLOPs for DeiT-B with only 0.3% top-1 accuracy loss. Code is available at this https URL.
Submission history
From: Bohan Zhuang [view email][v1] Tue, 23 Nov 2021 11:35:54 UTC (689 KB)
[v2] Wed, 16 Mar 2022 00:27:35 UTC (620 KB)
[v3] Mon, 15 Aug 2022 00:12:21 UTC (938 KB)
[v4] Tue, 16 Jan 2024 09:18:27 UTC (3,462 KB)
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.