[2005.09183] Retrieving and Highlighting Action with Spatiotemporal Referenceopen searchopen navigation menucontact arXivarXiv Twitter

In this paper, we present a framework that jointly retrieves and spatiotemporally highlights actions in videos by enhancing current deep cross-modal retrieval methods. Our work takes on the novel task of action highlighting, which visualizes where and when actions occur in an untrimmed video setting. Action highlighting is a fine-grained task, compared to conventional action recognition tasks which focus on classification or window-based localization. Leveraging weak supervision from annotated captions, our framework acquires spatiotemporal relevance maps and generates local embeddings which relate to the nouns and verbs in captions. Through experiments, we show that our model generates various maps conditioned on different actions, in which conventional visual reasoning methods only go as far as to show a single deterministic saliency map. Also, our model improves retrieval recall over our baseline without alignment by 2-3% on the MSR-VTT dataset.

2 mentions: @CVpaperChalleng
Date: 2020/05/20 02:21

Referring Tweets

@CVpaperChalleng ICIP 2020に採択された論文が arXiv に掲載されました! t.co/p3qlltD2Tb t.co/4XTYthQb5J

Related Entries

Read more [1909.12224] Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer...
0 users, 5 mentions 2019/09/27 17:18
Read more [2001.05634] Self-supervised visual feature learning with curriculumcontact arXivarXiv Twitter
0 users, 2 mentions 2020/01/19 05:20
Read more [2003.06148] PointINS: Point-based Instance Segmentationcontact arXivarXiv Twitter
1 users, 3 mentions 2020/03/16 06:51
Read more [2003.07853] Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentationcontact arXivarXiv ...
0 users, 3 mentions 2020/03/18 03:51
Read more [2004.01961] Neural Architecture Search for Lightweight Non-Local Networkscontact arXivarXiv Twitter
0 users, 3 mentions 2020/04/07 17:21