Vishaal Udandarao

I am a third year ELLIS PhD student, jointly working with Matthias Bethge at The University of Tuebingen and Samuel Albanie at The University of Cambridge/Google Deepmind. I am also a part of the International Max Planck Research School for Intelligent Systems. I am mainly interested in understanding the generalisation properties of foundation models, both vision-language models (VLMs) and large multi-modal models (LMMs), through the lens of their pre-training and test data distributions. My key research interest-axes are: Data-centric Machine Learning, Robustness/Generalisation to Distribution Shifts, and Foundation Models.

Currently, I am interning in Google Zürich, working with Yongqin Xian, Alessio Tonioni, Federico Tombari, and Olivier Henaff. I am also closely collaborating with Ferjad Naeem, Nikhil Parthasarathy and Talfan Evans.

Previously, I was an MPhil Machine Learning and Machine Intelligence student at The University of Cambridge. My thesis was on Understanding and Fixing the Modality Gap in VLMs. I graduated from IIIT Delhi with a Bachelors in Computer Science in July, 2020.

I am also fortunate to have previously worked with several great mentors: Ankush Gupta (Google Deepmind), Sungjin Ahn (KAIST), Tanmoy Chakraborty (IIT Delhi), Rajiv Ratn Shah (IIIT Delhi), Saket Anand (IIIT Delhi), Rajesh Kumar (Bucknell University), Anubha Gupta (IIIT Delhi) and Jainendra Shukla (IIIT Delhi).

Email  /  CV  /  Google Scholar  /  Twitter  /  Github

We provide practical insights into how to continually pretraining contrastive multimodal models under compute and data constraints.

Our work showcases that the impressive empirical performance of multimodal models like CLIP and Stable Diffusion can be largely attributed to the presence of test concepts within their vast pretraining datasets, thus their reported empirical performance does not constitute "zero-shot" generalization. Quite the contrary, these models require exponentially more data on a concept to linearly improve their performance on tasks pertaining to that concept, highlighting extreme sample inefficiency.

Our work introduces the concept of lifelong benchmarks, enabling effective comparisons of models and reducing overfitting to the biases of a particular dataset. We constructed large-scale lifelong classification benchmarks totalling over 1.5M samples. To facilitate more efficient evaluation, we introduce the Sort&Search method that reduces inference compute costs by 1000x.

CiteME is a benchmark designed to test the abilities of language models in finding papers that are cited in scientific texts.

We introduce "Visual Data-Type Identification": the task of classifying between visual image distortions and styles. On this simple task, we find surprising behaviour of VLMs and LMMs: model scaling does not significantly improve performance. We trace this behaviour back to the LAION-2B dataset and show a simple fine-tuning method to improve performance.

We enhance CLIP's downstream classification performance by (1) curating a support set either by generating synthetic (Stable Diffusion) or retrieving natural (LAION-5B) samples, and (2) observing and fixing a mis-calibration issue with intra-modal distances in CLIP’s embedding space.

A VAE model that learns implicit structure by aligning the latent space with the Valence-Arousal circumplex space. Further, a novel algorithm for mapping categorical and dimensional model labels using annotation transfer across affective facial image datasets is depicted.

A novel bi-modal framework that aims to train two modalities (image and text) in a joint fashion inspired by the Contrastive Predictive Coding (CPC) and Noise Contrastive Estimation (NCE) paradigms which preserve both inter and intra-class relationships in a modality-invariant fashion.

A multi-task learning model which incorporates auxiliary semantics by utilising a weight alignment layer and information exchange layer.

A self-supervised framework to disentangle multiple attributes by exploiting structural inductive biases within images and leveraging contrastive learning paradigms.

An empirical study on the inference of gender, major/minor (computer science, non-computer science), typing style, age, and height from the typing patterns collected from 117 individuals in a multi-device environment.

A meme generation system that uses a trained state-of-the-art transformer-based (GPT-2) model for caption generation by employing an encoder-decoder architecture.

An on-the-fly conceptual network model that incorporates educational semantics and preserves correlations between conceptual entities by applying intelligent indexing algorithms on an inherent concept network so as to improve answer generation.