Vishaal Udandarao

I am a final year ELLIS PhD student, jointly working with Matthias Bethge at The University of Tuebingen and Samuel Albanie at The University of Cambridge/Google DeepMind. My research is funded by a Google PhD Fellowship in Machine Intelligence.

I am very interested in data curation methods to improve multimodal pretraining. During my PhD, I have primarily worked on characterizing and constructing better pretraining datasets for multimodal models.

I previously interned in Google (DeepMind) and Apple, working on data-curation for multimodal models.

I'm currently on the industry job market! Please reach out if you have any open positions in pretraining.

Email  /  CV  /  Google Scholar  /  Twitter  /  Github

We studied three questions fundamental to speech-language pretraining data curation and sampling; our controlled experiments yielded a performant 3.8B SpeechLM that outperforms 3x-larger SpeechLMs.

We provide practical insights into how to continually pretraining contrastive multimodal models under compute and data constraints.

Our work showcases that the impressive empirical performance of multimodal models like CLIP and Stable Diffusion can be largely attributed to the presence of test concepts within their vast pretraining datasets, thus their reported empirical performance does not constitute "zero-shot" generalization. Quite the contrary, these models require exponentially more data on a concept to linearly improve their performance on tasks pertaining to that concept, highlighting extreme sample inefficiency.

Our work introduces the concept of lifelong benchmarks, enabling effective comparisons of models and reducing overfitting to the biases of a particular dataset. We constructed large-scale lifelong classification benchmarks totalling over 1.5M samples. To facilitate more efficient evaluation, we introduce the Sort&Search method that reduces inference compute costs by 1000x.

We enhance CLIP's downstream classification performance by (1) curating a support set either by generating synthetic (Stable Diffusion) or retrieving natural (LAION-5B) samples, and (2) observing and fixing a mis-calibration issue with intra-modal distances in CLIP’s embedding space.