[2010.13639] Stochastic Optimization with Laggard Data Pipelinesopen searchopen navigation menucontact arXivsubscribe to arXiv mailings

State-of-the-art optimization is steadily shifting towards massively parallel pipelines with extremely large batch sizes. As a consequence, CPU-bound preprocessing and disk/memory/network operations have emerged as new performance bottlenecks, as opposed to hardware-accelerated gradient computations. In this regime, a recently proposed approach is data echoing (Choi et al., 2019), which takes repeated gradient steps on the same batch while waiting for fresh data to arrive from upstream. We provide the first convergence analyses of "data-echoed" extensions of common optimization methods, showing that they exhibit provable improvements over their synchronous counterparts. Specifically, we show that in convex optimization with stochastic minibatches, data echoing affords speedups on the curvature-dominated part of the convergence rate, while maintaining the optimal statistical rate.

Date:

Related Entries

Neural Networks Play Guess-the-Material to Reduce Latency in the "Tactile Internet" - Hackster.io
Read more Neural Networks Play Guess-the-Material to Reduce Latency in the "Tactile Internet" - Hackster.io
0 users, 1 mentions 2020/01/24 21:51
Kuzushiji Reconnition Competition | Tarin Clanuwat | Kaggle Days - YouTube
Read more Kuzushiji Reconnition Competition | Tarin Clanuwat | Kaggle Days - YouTube
0 users, 0 mentions 2020/01/27 21:50
[2008.05773] Continuous Speech Separation with Conformeropen searchopen navigation menucontact arXiv...
Read more [2008.05773] Continuous Speech Separation with Conformeropen searchopen navigation menucontact arXiv...
0 users, 0 mentions 2020/10/24 03:52
SIGTYP -- March Newsletter
Read more SIGTYP -- March Newsletter
0 users, 1 mentions 2021/03/19 15:51

ML-Newsについて

機械学習の技術に関する情報は流速も早いし、分野も多様でキャッチアップが大変です。Twitterで機械学習用のリストを作っても、普段は機械学習以外の話題が多く流れており、効率的に情報収集するのは困難です。

ML-NewsはSNSを情報源とした機械学習に特化したニュースサイトです。機械学習に関する論文ブログライブラリコンペティション発表資料勉強会などの最新の情報を効率的に収集できます。

機械学習を応用した自然言語処理、画像認識、情報検索などの分野の情報や機械学習で必要になるデータ基盤やMLOpsの話題もカバーしています。
安定したサイト運営のためにGitHub sponsorを募集しています。

お知らせ

  • 2021/12/31: デザインを刷新しました
  • 2021/04/08: 日本語Kaggleのカテゴリを新設しました