[2107.07467] Only Train Once: A One-Shot Neural Network Training And Pruning Framework

Structured pruning is a commonly used technique in deploying deep neural networks (DNNs) onto resource-constrained devices. However, the existing pruning methods are usually heuristic, task-specified, and require an extra fine-tuning procedure. To overcome these limitations, we propose a framework that compresses DNNs into slimmer architectures with competitive performances and significant FLOPs reductions by Only-Train-Once (OTO). OTO contains two keys: (i) we partition the parameters of DNNs into zero-invariant groups, enabling us to prune zero groups without affecting the output; and (ii) to promote zero groups, we then formulate a structured-sparsity optimization problem and propose a novel optimization algorithm, Half-Space Stochastic Projected Gradient (HSPG), to solve it, which outperforms the standard proximal methods on group sparsity exploration and maintains comparable convergence. To demonstrate the effectiveness of OTO, we train and compress full models simultaneously from

1 mentions: @Maxwell_110
Date:

Referring Tweets

@Maxwell_110
@Maxwell_110 Microsoft 等が提案する pruning 法「Only-Train-Once」📝 t.co/xCD1kmsLPj パラメータを ZIG(図で黄色の定義)に分割し,新たに提案した最適化手法で粗密な重みを求め,pruning している 名の通り,スクラッチから学習・圧縮を同時に行うことが可能で pruning 後の fine-tuning は不要 t.co/5YLY8s6uJE

Related Entries

[1810.11654v3] 3D MRI brain tumor segmentation using autoencoder regularization
Read more [1810.11654v3] 3D MRI brain tumor segmentation using autoencoder regularization
0 users, 1 mentions 2021/08/21 01:37
[1805.12462] On GANs and GMMs
Read more [1805.12462] On GANs and GMMs
0 users, 1 mentions 2022/03/15 22:37
[2108.01099] Shift-Robust GNNs: Overcoming the Limitations of Localized Graph Training Data
Read more [2108.01099] Shift-Robust GNNs: Overcoming the Limitations of Localized Graph Training Data
0 users, 1 mentions 2022/04/21 22:37
[2204.02663v2] Towards An End-to-End Framework for Flow-Guided Video Inpainting
Read more [2204.02663v2] Towards An End-to-End Framework for Flow-Guided Video Inpainting
0 users, 1 mentions 2022/05/05 22:37
GitHub - understandable-machine-intelligence-lab/Quantus: Quantus is an eXplainable AI toolkit for r...
Read more GitHub - understandable-machine-intelligence-lab/Quantus: Quantus is an eXplainable AI toolkit for r...
0 users, 1 mentions 2022/06/14 22:38
Kaggle_meetup_3rd LT ( Sberbank Russian Housing Market ) - Speaker Deck
Read more Kaggle_meetup_3rd LT ( Sberbank Russian Housing Market ) - Speaker Deck
0 users, 1 mentions 2022/07/14 12:11

ML-Newsについて

機械学習の技術に関する情報は流速も早いし、分野も多様でキャッチアップが大変です。Twitterで機械学習用のリストを作っても、普段は機械学習以外の話題が多く流れており、効率的に情報収集するのは困難です。

ML-NewsはSNSを情報源とした機械学習に特化したニュースサイトです。機械学習に関する論文ブログライブラリコンペティション発表資料勉強会などの最新の情報を効率的に収集できます。

機械学習を応用した自然言語処理、画像認識、情報検索などの分野の情報や機械学習で必要になるデータ基盤やMLOpsの話題もカバーしています。
安定したサイト運営のためにGitHub sponsorを募集しています。

お知らせ

  • 2021/12/31: デザインを刷新しました
  • 2021/04/08: 日本語Kaggleのカテゴリを新設しました