[CVPR 2025] Context-Aware Multimodal Pretraining

Описание: Paper: https://arxiv.org/abs/2411.15099

Authors:
Karsten Roth, Zeynep Akata, Dima Damen, Ivana Balažević*, Olivier J. Hénaff*
(joint senior authorship!).
Work done while first author was at Google DeepMind.

Abstract:
Large-scale multimodal representation learning successfully optimizes for zero-shot transfer at test time. Yet the standard pretraining paradigm (contrastive learning on large amounts of image-text data) does not explicitly encourage representations to support few-shot adaptation. In this work, we propose a simple, but carefully designed extension to multimodal pretraining which enables representations to accommodate additional context. Using this objective, we show that vision-language models can be trained to exhibit significantly increased few-shot adaptation: across 21 downstream tasks, we find up to four-fold improvements in test-time sample efficiency, and average few-shot adaptation gains of over 5%, while retaining zero-shot generalization performance across model scales and training durations. In particular, equipped with simple, training-free, metric-based adaptation mechanisms, our representations easily surpass more complex and expensive optimization-based schemes, vastly simplifying generalization to new domains.