[2002.02997] DropCluster: A structured dropout for convolutional networkscontact arXivarXiv Twitter

Dropout as a regularizer in deep neural networks has been less effective in convolutional layers than in fully connected layers. This is due to the fact that dropout drops features randomly. When features are spatially correlated as in the case of convolutional layers, information about the dropped pixels can still propagate to the next layers via neighboring pixels. In order to address this problem, more structured forms of dropout have been proposed. A drawback of these methods is that they do not adapt to the data. In this work, we introduce a novel structured regularization for convolutional layers, which we call DropCluster. Our regularizer relies on data-driven structure. It finds clusters of correlated features in convolutional layer outputs and drops the clusters randomly at each iteration. The clusters are learned and updated during model training so that they adapt both to the data and to the model weights. Our experiments on the ResNet-50 architecture demonstrate that our ap

Date: 2020/02/12 00:51

Related Entries

Read more [2009.06732] Efficient Transformers: A Surveyopen searchopen navigation menucontact arXivsubscribe t...
1 users, 15 mentions 2020/09/16 05:21
Read more RTX3080 TensorFlow and NAMD Performance on Linux (Preliminary)
1 users, 19 mentions 2020/09/18 20:21
Read more GitHub - Soongja/basic-image-eda: A simple image dataset EDA tool (CLI / Code)
0 users, 1 mentions 2020/09/19 06:53
Read more GitHub - ariG23498/G-SimCLR: This code for paper G-SimCLR : Self-Supervised Contrastive Learning wit...
0 users, 1 mentions 2020/09/19 08:21
Read more [2009.08576] Pruning Neural Networks at Initialization: Why are We Missing the Mark?open searchopen ...
0 users, 8 mentions 2020/09/21 20:22