Learning Semantic-Visual Embeddings with a Priority Queue

The Stochastic Gradient Descent (SGD) algorithm and margin-based loss functions have been the learning workhorse of choice to train deep metric learning networks. Often, the random nature of SGD will lead to the selection of sub-optimal mini-batches, several orders of magnitude smaller than the larger dataset. In this paper, we propose to augment SGD mini-batch with a priority learning queue, i.e., SGD+PQ. While the mini-batch SGD replaces all learning samples in the mini-batch at each iteration, the proposed priority queue replaces samples by removing the less informative ones. This novel idea introduces a sample update strategy that balances two sample removal criterion: (i) removal of stale samples from the PQ that are likely outdated, and (ii) removal of samples that are not contributing to the error, i.e. their sample error is not changing during training. Experimental results demonstrate the success of the proposed approach across three datasets.