Vishaal Udandarao

I am a second year ELLIS PhD student, jointly working with Matthias Bethge at The University of Tuebingen and Samuel Albanie at The University of Cambridge/Google Deepmind. I am also a part of the International Max Planck Research School for Intelligent Systems. I am mainly interested in understanding the generalisation properties of foundation models, both vision-language models (VLMs) and large multi-modal models (LMMs), through the lens of their pre-training and test data distributions. My key research interest-axes are: Data-centric Machine Learning, Robustness/Generalisation to Distribution Shifts, and Foundation Models.

Previously, I was an MPhil Machine Learning and Machine Intelligence student at The University of Cambridge. My thesis was on Understanding and Fixing the Modality Gap in VLMs. I graduated from IIIT Delhi with a Bachelors in Computer Science in July, 2020.

I am also fortunate to have previously worked with several great mentors: Ankush Gupta (Google Deepmind), Sungjin Ahn (KAIST), Tanmoy Chakraborty (IIT Delhi), Rajiv Ratn Shah (IIIT Delhi), Saket Anand (IIIT Delhi), Rajesh Kumar (Bucknell University), Anubha Gupta (IIIT Delhi) and Jainendra Shukla (IIIT Delhi).

Email  /  CV  /  Google Scholar  /  Twitter  /  Github

We introduce "Visual Data-Type Identification": the task of classifying between visual image distortions and styles. On this simple task, we find surprising behaviour of VLMs and LMMs: model scaling does not significantly improve performance. We trace this behaviour back to the LAION-2B dataset and show a simple fine-tuning method to improve performance.

We enhance CLIP's downstream classification performance by (1) curating a support set either by generating synthetic (Stable Diffusion) or retrieving natural (LAION-5B) samples, and (2) observing and fixing a mis-calibration issue with intra-modal distances in CLIP’s embedding space.

A VAE model that learns implicit structure by aligning the latent space with the Valence-Arousal circumplex space. Further, a novel algorithm for mapping categorical and dimensional model labels using annotation transfer across affective facial image datasets is depicted.

A novel bi-modal framework that aims to train two modalities (image and text) in a joint fashion inspired by the Contrastive Predictive Coding (CPC) and Noise Contrastive Estimation (NCE) paradigms which preserve both inter and intra-class relationships in a modality-invariant fashion.

A multi-task learning model which incorporates auxiliary semantics by utilising a weight alignment layer and information exchange layer.

A self-supervised framework to disentangle multiple attributes by exploiting structural inductive biases within images and leveraging contrastive learning paradigms.

An empirical study on the inference of gender, major/minor (computer science, non-computer science), typing style, age, and height from the typing patterns collected from 117 individuals in a multi-device environment.

A meme generation system that uses a trained state-of-the-art transformer-based (GPT-2) model for caption generation by employing an encoder-decoder architecture.

An on-the-fly conceptual network model that incorporates educational semantics and preserves correlations between conceptual entities by applying intelligent indexing algorithms on an inherent concept network so as to improve answer generation.