A Mixed Diet Makes DINO
An Omnivorous Vision Encoder

Pre-trained vision encoders like DINOv2 have demonstrated exceptional performance on unimodal tasks. However, we observe that their features are poorly aligned across different visual modalities. For instance, the feature embedding for an RGB image and its corresponding depth map of the same scene exhibit a cosine similarity that is nearly identical to that of two random, unrelated images. To address this, we propose the Omnivorous Vision Encoder, a post-training framework that learns a modality-agnostic feature space. We fine-tune the encoder with a dual objective: first, to maximize the feature alignment between different modalities of the same scene; and second, a distillation objective that anchors the learned representations to a fully frozen teacher. The resulting student encoder becomes "omnivorous" by producing more consistent embeddings for a given scene, regardless of the input modality (RGB, Depth, Segmentation, etc.). This approach enables robust cross-modal understanding while retaining the discriminative semantics of the original foundation model.

Pretrained visual features like DINOv2 are resilient to alterations like image color. But features vary significantly for different sensory views of the same scene, e.g., segmentation maps and depth maps. The model treats these like random, unrelated RGB images (measured by patchwise cosine similarity).

Ideally, we would like an "Omnivorous" representation, which aligns information across visual modalities, and an encoder that outputs this aligned representation regardless of which input modality is observed.

We adopt a parameter-efficient teacher-student framework. The student encoder is initialized from DINOv2. The early layers ($f^*$) are shared and frozen; only the final layers (the adapter $g$) are updated. The teacher's output serves as a stable anchor, preventing catastrophic forgetting during alignment training.

We colorize depth and segmentation maps using a natural color palette derived from the corresponding RGB image, creating "hard positives" that force alignment based on structural content rather than color statistics. We then randomly blend modalities during training, smoothly interpolating between Depth ↔ RGB ↔ Segmentation.

Our training objective combines two complementary losses:

1. Cross-modal alignment loss ($\mathcal{L}_{\text{align}}$): A symmetric InfoNCE loss computed over all modality pairs. Features from the same scene but different modalities are brought together; features from different scenes are pushed apart.

$$\mathcal{L}_{\text{align}} = \frac{1}{3} \sum_{k_1=1}^{3} \sum_{k_2>k_1}^{3}\mathcal{L}_\text{InfoNCE}(m_{k_1}, m_{k_2})$$

2. Anchoring loss ($\mathcal{L}_{\text{anchor}}$): Distills from the frozen teacher, keeping the student's output close to the teacher's pre-trained representation. This prevents "representational drift" or collapse.

$$\mathcal{L}_{\text{anchor}} = \frac{1}{|M|} \sum_{m \in M} (1 - \text{sim}(h_m, h^*_m))$$

The total objective is:

$$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{align}} + \lambda_{\text{anchor}} \cdot \mathcal{L}_{\text{anchor}}$$

Using $\mathcal{L}_{\text{align}}$ alone causes representations to collapse; using $\mathcal{L}_{\text{anchor}}$ alone preserves DINOv2 but doesn't improve alignment. Together, they strike a principled balance.

Omnivorous post-training can improve visual foundation models in three key ways:

1. Representational alignment: Cross-modal retrieval goes from near-random to near-perfect. Different visual modalities of the same scene map to nearby points in feature space.
2. Scene understanding: Downstream semantic and 3D tasks (classification, depth, segmentation, correspondence) are preserved or improved by the multimodal regularization.
3. Transfer to unseen modalities: Task heads trained on RGB generalize zero-shot to segmentation maps and even out-of-distribution modalities like NOCS, where the baseline fails completely.

Our modality-agnostic encoder shows improved generalization to novel visual modalities, paving the way for a more foundational vision model.