Vision-language models have become the default backbone for almost everything multimodal, but treating them as a finished product leaves a lot on the table. This four-part series is a tour of where these models fall short and the surprisingly lightweight ideas that close the gap. It starts in the physical world, where a VLM driving a robot needs reasoning its pre-training never gave it, then moves inside the model to the joint embedding space and shows that its geometry can be corrected rather than retrained. From there it steps back to ask what adaptation actually does to a representation, an analysis that turns out to anticipate much of today’s thinking on tuning foundation models, and finally it reframes the humblest tool in the toolbox, data augmentation, as a precise lever for generalization that remains central even to how modern multimodal LLMs are post-trained. The takeaway is practical: small, well-placed interventions are often what make these systems reliable.
Part 1: Closing the Physical Reasoning Gap in VLMs for Robotic Manipulation
Reference: Learn to Judge, Then Learn to Act: Closing the Physical Reasoning Gap via Rank Prediction Pre-Training (preprint)
This series is about how large-scale vision-language models actually become useful out in the world. The field has converged on VLMs as the default backbone for almost everything multimodal, but treating them as a finished product is a mistake. The first part of this series picks the most demanding test of that backbone, embodied AI, where the model has to do more than describe a scene; it has to drive a physical action. That setting is where the most stubborn, and most consequential, weakness of today’s VLMs becomes visible.
The Physical Reasoning Gap
Web-scale pre-training gives a VLM excellent semantic and spatial understanding. It can name objects, parse layouts, follow instructions. What it never sees during pre-training is physical interaction: no force signals, no contact events, no record of what happens when an action succeeds or fails. The standard VLA recipe then fine-tunes on demonstration trajectories, but behavior cloning rewards imitation over understanding, so even the fine-tuned model can imitate expert actions without acquiring any sense of why those actions worked. The result is brittle on tasks that demand precise interaction, the well-documented physical reasoning gap.
What is striking is that the obvious patches do not close it. Adding more spatial-grounding or embodied-VQA pre-training, both of which improve scene perception, does not reliably translate into better downstream control, and in some cases makes things worse. Better action decoders and chain-of-thought reasoning during policy learning add capacity at the wrong end of the pipeline. The gap lives in what the VLM has learned before VLA fine-tuning even begins.
Judge Before You Act
RPT’s premise is that judging grasps must precede producing them. If a model cannot tell whether a candidate grasp will work, asking it to generate one is asking too much. So instead of pre-training on perception, RPT pre-trains on evaluation: the model is shown a scene and a set of candidate 6-DoF grasp poses, and it has to rank them by physical viability. Near-boundary candidates differ by millimeters or a few degrees, so the only way to score them well is to reason about contact geometry, gripper-object fit, approach feasibility, and stability. Perception alone will not save you.
To make the supervision signal smooth and informative, the ground-truth reward for a candidate grasp is derived from its SE(3) distance to the nearest known successful pose, normalized by the median failure distance so the reward is on a comparable scale across scenes. This gives a continuous quality landscape rather than a binary success/failure label, exactly the fine-grained signal a ranking objective needs.
Training proceeds in two phases. The first is supervised fine-tuning with physical reasoning traces: each candidate is paired with a procedurally generated trace that walks through the approach direction, contact location, and gripper-object fit, grounded in the object’s actual 3D model and pose. These traces are not free-text rationalizations; they encode geometry the VLM cannot read off pixels alone, and they teach the model to map the visual input to a calibrated quality score.
The second phase sharpens the model’s pairwise discrimination. SFT produces good pointwise scores but does not directly optimize for the right ordering of similar candidates, which is precisely what matters near the success boundary. Reinforcement fine-tuning with GRPO targets exactly this: a Thurstone-Gaussian preference model converts pairs of scores into a probability of preferring one pose over another, and a pairwise-agreement reward measures how well the model’s preferences match ground truth. To avoid wasting gradient on easy pairs, comparisons are restricted to each pose’s nearest neighbors in true rank, focusing learning on the boundary cases that SFT cannot resolve. A small calibration term keeps absolute scores from drifting.
Transfer to Action
The hypothesis is that representations learned for ranking will transfer to acting. The transfer interface is deliberately minimal: the RPT checkpoint replaces the base VLM in any downstream pipeline with no architectural changes. In a VLM4VLA-style setup, an action query token is appended to the VLM’s input and decoded into a low-level action chunk via a small MLP head, adding less than one percent new parameters and isolating the effect of the backbone.
The results back the bet. On grasp pose prediction with HouseCat6D, initializing from the full SFT→RFT checkpoint roughly halves both position and orientation error at every model size, with SFT-only and RFT-only each helping partially but neither matching the combined pipeline. On the harder test of sequential action prediction, RPT initialization improves average success rates on SimplerEnv by 3 to 5 points and on LIBERO by 4 to 6 points across model sizes, with the largest gains on LIBERO-Long, the most challenging multi-step suite where physical understanding compounds across steps. The gains stack with downstream advances too: layering hierarchical action reasoning on top of an RPT-initialized model adds another 5 points on LIBERO, confirming that physical pre-training is complementary to architectural improvements rather than substituting for them.
Two qualitative notes are worth holding onto. First, even when an RPT-initialized policy fails, the failure mode is more physically coherent: the gripper approaches the right object and reaches a plausible placement, often running out of time only on the final step, while the base-initialized model fumbles the grasp itself. The improvement is not just in headline success rate but in the texture of what the model does wrong. Second, RPT outperforms perception-oriented pre-training alternatives like Robo2VLM and synthetic embodied VQA on the same downstream benchmarks, and its advantage grows with more pre-training data, which is what we want from a method that is supposed to instill a missing capability.
The takeaway is simple but pointed. The bottleneck for VLM-driven robotic manipulation is not perception, decoder design, or chain-of-thought; it is that the backbone never learned to reason about the physical consequences of an action. Teaching it to judge grasps before producing them, with a learn-to-rank objective and physically grounded reasoning traces, gives the rest of the VLA pipeline something to work with. The next part of the series moves from this embodied setting back toward the joint vision-language embeddings that underlie all of these models, and asks how far those embeddings can be pushed.
Part 2: Pushing the Limits of Multimodal Embeddings
References:
1. Shared Subspace Correction for Zero-Shot Multi-Label Classification with Vision-Language Encoders
2. Exploring the Utility of CLIP Priors for Visual Relationship Prediction
Part 1 took a vision-language model into the physical world and found a missing capability. This part stays inside the model and asks a sharper question: how good is the joint embedding space itself? CLIP and its descendants are sold as general-purpose: encode an image, encode some text, compare them with a cosine similarity, and you have a zero-shot classifier. That recipe is genuinely remarkable, but it hides a brittle assumption. The embedding geometry was shaped by image-caption matching, not by the specific decision you are about to ask it to make. When the task drifts away from that training objective, the geometry quietly stops cooperating. Two papers here probe that limit from different angles, and both fix it by correcting the embedding space rather than retraining the backbone.
The Discrimination Axis Is Pointing the Wrong Way
Consider zero-shot multi-label classification, where an image can carry many labels at once and each concept needs its own independent yes/no decision. The standard recipe builds, for each concept, a positive text prototype (“a photo of a dog”) and a negative one (“a photo without a dog”), and scores an image by which prototype it is closer to. The direction between those two prototypes is the discrimination axis for that concept, the line along which positives are supposed to separate from negatives.
SSC’s starting observation is that this axis is often simply pointing the wrong way. Because contrastive pre-training optimized caption matching rather than per-concept discrimination, the text-derived axis can be directionally misaligned with the direction that actually separates positive from negative images. The effect is severe enough that a humble per-class linear probe frequently beats sophisticated prompt-engineering and co-occurrence methods, none of which touch the axis itself.
The key insight is what happens when you stack those per-concept corrections together. They are not independent quirks; they are correlated, concentrating in a low-rank shared subspace that implicitly encodes how the encoder systematically mis-orients its axes for a given visual domain. The more systematic the misalignment, the lower the rank. On chest radiographs, a handful of shared directions captures almost all the recoverable gain. This means you do not need to learn a separate correction per concept; one small shared subspace can fix all of them at once, and that shared structure is exactly what lets the correction transfer to concepts that had no labels at all.
We can turn this insight into an algorithm. Rather than relying on a fixed SVD estimate, it directly optimizes the target metric, mean average precision, which is non-differentiable, using evolution strategies. Each generation samples one shared direction, builds a population of candidate corrections by perturbing the per-concept scalars along that direction, scores each candidate’s mAP on a small calibration set, and rank-normalizes the results to update the scalars. Each generation contributes a rank-1 update, and the effective rank self-regulates: directions that help many concepts earn strong weights, while useless ones decay to near-zero. The corrected axis for each concept is just the original text axis plus the learned shared correction.
The results hold across four datasets and five encoders. SSC beats zero-shot scoring, state-of-the-art multi-label methods, and per-class linear probes, and it is complementary to patch-level methods like TagCLIP, which recover spatial signal without touching the axis. The most telling case is medical imaging. On MIMIC-CXR chest X-rays, SSC improves every encoder, including ones already fine-tuned for the domain, and it is the only method that improves BiomedCLIP, where an independent per-class probe actually drops below zero-shot. Because the correction lives in a shared subspace rather than a bag of per-class weights, it also keeps working in the low-label regime and even transfers to entirely held-out diagnostic categories, something a per-class probe cannot do at all.
When the Text Prior Cannot Tell Predicates Apart
SSC corrects the axis between two prototypes. CREPE confronts a related failure where the text prior cannot produce useful prototypes in the first place. The task is visual relationship prediction: given a subject and an object in a scene, name the predicate that relates them, as in <person, riding, bike>. You might expect CLIP’s language prior to be perfect for this, since predicates are just words. It is not. CLIP’s text embeddings struggle to distinguish between the different predicates that could relate the same pair of objects. Shown a horse and grass, CLIP rates “in front of”, “growing on”, and “eating” as roughly equally plausible. The prior is too coarse to do the one thing the task demands.
CREPE’s fix is to stop using a fixed text prompt and instead learn a visually grounded one for the region that contains both objects, the union box. It keeps the CLIP backbone frozen and learns a set of context tokens that sit between the subject and object names, while an MLP injects an image-conditioned bias so that the prompt adapts to the specific union image rather than staying generic. Training uses a contrastive objective with a clever source of negatives: for each union image, a pseudo-label is retrieved as the most similar triplet from a vocabulary of all subject-predicate-object combinations, and the learned prompt is pushed to describe the image better than that retrieved triplet. The result is a union-box representation that finally separates predicates, and it drops into any existing relationship-prediction model.
Plugged into standard relationship-prediction models like UVTransE and VCTree, CREPE produces large gains without any extra calibration machinery. It reaches state-of-the-art mean recall at small K, the practically relevant regime, with its mean recall at K=5 actually exceeding the previous best method’s mean recall at K=20. It also lifts the rare, long-tail predicates that frequency-biased methods tend to ignore, and it generalizes to the Unrel benchmark of atypical, unseen relationships (a horse on a sofa, a dog wearing sunglasses) despite never training on them.
Read together, SSC and CREPE make the same point from two directions. A vision-language joint embedding is not a finished classifier; it is a prior whose geometry was tuned for a different objective. When you need it to make a fine-grained decision, separating co-occurring labels, or distinguishing predicates that relate the same objects, the honest move is not to retrain the backbone but to correct the small part of its geometry that is misaligned for your task. The next part turns from correcting embeddings to a deeper question that underlies all of this: what actually happens, representationally, when we adapt these models at all.
Part 3: What Are We Actually Doing When We Adapt a Model?
Reference: A Closer Look at Model Adaptation Using Feature Distortion and Simplicity Bias, ICLR 2023
Here is a question that sounds trivial until you sit with it. You take a strong pretrained encoder, attach it to a new task, and adapt. Everyone does this a dozen different ways: freeze the encoder and train a head (linear probing), unfreeze everything (fine-tuning), or do the head first and then unfreeze (LP+FT). We pick one, watch the accuracy go up, and move on. But what is adaptation actually doing to the representation we worked so hard to pretrain? And if two protocols reach the same accuracy, are the models they produce actually the same? This paper pulls on that thread, and what unravels turns out to anticipate a lot of what we now worry about when adapting foundation models.
The Story Everyone Agreed Upon
The accepted wisdom going in was tidy. Fine-tuning, the story went, is dangerous because it only reshapes the features the in-distribution data touches, and in doing so it warps the directions outside that subspace, exactly the ones you need when the test data drifts. Linear-probe-then-fine-tune was the fix: do the cheap probe first so fine-tuning starts from a sensible place and never wanders far, and you keep the distortion small. Less distortion, better out-of-distribution accuracy. Clean explanation, and it works.
So the natural thing to do is poke it. If limiting distortion is the whole story, then LP+FT should win not just on accuracy but on everything we actually care about in a deployed model. So the paper asks a broader question than the original work did: how do these protocols look when you also grade them on calibration, robustness to corruptions, adversarial robustness, and anomaly detection, the things that decide whether you can trust a model in the wild?
The tidy story springs a leak. No protocol wins across the board. LP+FT does come out best on clean out-of-distribution accuracy and best on average, but plain fine-tuning beats it on adversarial robustness and OOD calibration, and plain linear probing beats it on in-distribution calibration. Controlling distortion buys you generalization, but it quietly does not buy you safety. Something else is going on that distortion alone cannot see.
The Thing Distortion Couldn’t See
That missing factor turns out to be an old acquaintance: simplicity bias, the relentless habit neural networks have of grabbing the easiest feature that fits the data and ignoring everything richer, even when the easy feature is a brittle shortcut. The clever move here is to catch adaptation in the act. Build “dominoes” that glue a complex feature (say a natural image) to a trivially easy one (say a handwritten digit) that is, during training, perfectly predictive. Now you can ask a pointed question: when the model adapts, does it learn the real object, or does it just read the digit?
The answer is unflattering for the protocol everyone reaches for first. Full fine-tuning happily memorizes the digit. It looks brilliant while the shortcut holds and falls off a cliff the moment you decorrelate the easy and hard features, the telltale signature of a model that threw away the encoder’s expressive features to chase the cheap one. Linear probing, precisely because it refuses to touch those pretrained features, resists the temptation far better. Distortion was never the whole story; the protocols differ in how much they surrender to the shortcut, and that is a different axis entirely.
The Quiet Lever
Now two observations collide in a useful way. On tasks close to what the encoder already knows, LP+FT barely moves the encoder during fine-tuning, so whatever the linear probe decided is essentially what you ship. And the linear probe is exactly the part that resists simplicity bias. Put those together and the design problem flips: stop fussing over fine-tuning and fix the probe, because the probe is the lever the whole pipeline pivots on, and it is the cheapest thing in the room to change.
This insight leaves the expensive encoder and the fine-tuning step alone and instead hardens the probe so it reaches for richer features before fine-tuning ever starts, through smoothing the classifier around each input, nudging it off overconfident shortcut solutions, and averaging several probes into a sturdier one. Cheap interventions, applied at the one point that turns out to matter, and they push the adapted model to lean on real signal instead of the shortcut, improving both robustness and generalization.
A side note: As the field moved from adapting standalone encoders to adapting foundation models, the same tensions came roaring back under new vocabulary. How do you fine-tune without trampling the pretrained representation that made the model valuable in the first place? Why do models that ace the in-distribution benchmark fold under distribution shift? How do you adapt while touching as little of the network as possible, the instinct that now drives the entire parameter-efficient fine-tuning literature? This study already circles all three. It said, early and concretely, that accuracy is a misleading scoreboard, that distortion and simplicity bias are two different failure modes you have to track separately, and that the leverage in an adaptation pipeline often sits in its smallest, cheapest component rather than its largest.
Part 4: Augmentation as a Lever for Generalizable Models
References:
1. Attribute-Guided Adversarial Training for Robustness to Natural Perturbations
2. Improving Diversity with Adversarially Learned Transformations for Domain Generalization
3. InterAug: A Tuning-Free Augmentation Policy for Data-Efficient and Robust Object Detection
4. Know Your Space: Inlier and Outlier Construction for Calibrating Medical OOD Detectors
Augmentation is the most unglamorous tool in the deep learning toolbox. Flip the image, jitter the colors, crop a corner, train on the result. It is so routine that it is easy to treat it as a fixed preprocessing step rather than a design decision. The thread running through this final part is that augmentation is actually a precise lever for generalization, and that pulling it well means thinking carefully about three things we usually leave on autopilot: what to perturb, how hard to perturb, and crucially, where to perturb. Four papers, each pulling the lever from a different angle.
What to Perturb: Specify the Shift You Fear
Standard robustness research spent years obsessing over tiny pixel-level perturbations, the imperceptible noise that fools a classifier. AGAT starts from the observation that this is rarely the shift that breaks models in the real world. Real test data differs along semantic attributes: an object rotates, the lighting changes, the background swaps, the camera angle shifts. These are large in pixel space yet leave the label untouched, and the usual norm-bounded defenses simply do not cover them.
The idea is to let the practitioner specify, in advance, the kind of shift they expect, not the exact test data, just a broad characterization in terms of attributes, and then adversarially train against it. A min-max loop generates the attribute-space perturbations that most confuse the classifier, while the model learns to be robust to them. The attribute generator can be a conditional GAN for object-level shifts, a spatial transformer for geometric ones, or a corruption model, so the same recipe covers object changes, geometric transforms, and image corruptions. The point is that augmentation becomes a way to inject prior knowledge about the deployment environment rather than a generic smoothing trick.
How Hard to Perturb: Diversity Is Not Enough
If the test domain is genuinely unknown, as in single-source domain generalization where you train on one domain and must generalize to many, you cannot pre-specify the attribute. The community’s answer was diversity: throw as many transformations at the model as possible. ALT pushes on this and finds it only half right. Diversity is necessary but insufficient. Blindly piling on transformations does not guarantee generalization, because a fixed menu of augmentations cannot model large semantic domain shifts and may not cover the shifts that actually occur.
ALT adds the missing ingredient: hardness. Alongside a diversity module (off-the-shelf augmentations like AugMix or RandConv), it trains an adversary network that, fresh for each batch, learns image transformations specifically designed to fool the current classifier, then enforces consistency between the model’s predictions on clean and transformed images. The adversary manufactures transformations that are both diverse and hard, reaching parts of the transformation space that pre-specified augmentations cannot, and the two modules compose, so adversarial hardness can be layered on top of any diversity strategy.
Where to Perturb: Use the Object’s Context
InterAug pulls the lever on a dimension the others ignore: the spatial extent of the augmentation within an image. For object detection, the obvious choices are to augment the whole scene or to augment only inside the bounding boxes. Both are flawed. Whole-scene augmentation ignores that detection is about objects; box-only augmentation runs into the fact that bounding boxes are sloppy, they include stray background and their tightness varies with the annotator, so restricting manipulation to the box yields inconsistent decision rules for the same object.
The fix is a simple, tuning-free observation: expand each bounding box to an “effective context” that captures the object plus the immediate surroundings that matter, while systematically excluding leakage from neighboring objects, and apply the augmentation there. It plugs into any existing policy (e.g., TrivialAug), needs no architecture changes, and improves both data efficiency and robustness under real-world shifts across detector families. Augmentation, here, becomes a question of respecting the structure of the task, augment the object in its context, not the arbitrary rectangle an annotator drew.
Where to Perturb (in the Representation): Latent vs Pixel Space
The most surprising lever is the one Know Your Space exposes. When you need calibration data for an out-of-distribution detector, you synthesize fake inliers (plausible variations of in-distribution data) and fake outliers (stand-ins for the unknown rest of the world). Everyone debates which augmentation to use. This paper’s striking finding is that the more important question is which space you synthesize in.
The answer that works, for energy-based detectors on medical imaging, is asymmetric. Synthesize inliers in the latent space, where small moves stay on the data manifold and produce tight, realistic variations that hug the in-distribution cluster. Synthesize outliers in pixel space, where you can push samples far and wide to represent the genuinely unfamiliar. Get the spaces right and the detector learns a tight, well-placed boundary; get them wrong and inliers bleed into outliers and the boundary goes slack. The intuition is worth a picture.
The Common Thread, and Where It Leads
Lined up together, these four make augmentation look less like preprocessing and more like a control surface for generalization. AGAT controls what shift to simulate by specifying attributes. ALT controls how hard the augmentations should be, not just how varied. InterAug controls where in the image to apply them by respecting object context. Know Your Space controls which representational space to synthesize in. In every case the lesson is the same: the default, scatter generic augmentations over the whole input and hope, leaves most of the leverage unused, and a small amount of structure about the task or the deployment recovers it.
That framing connects this part back to the rest of the series. Each of these is, at heart, a way to build a model that holds up when the world differs from the training set, the same goal that drove the physical-reasoning pre-training of Part 1, the embedding corrections of Part 2, and the adaptation analysis of Part 3. Augmentation is simply the cheapest lever for it, and these papers show that pulling it deliberately, with intent about what, how hard, and where, is what separates a routine preprocessing step from a genuine tool for generalization.
A side note: And far from being a relic of the deep learning era, this lever is alive and well in how today’s multimodal large language models are built, especially in post-training. The instinct of AGAT, expose the model to the variations you expect rather than hoping it generalizes, now shows up as synthetic data generation and instruction augmentation: paraphrasing prompts, rewriting and re-captioning images, and programmatically composing visual question-answer pairs to cover skills and corner cases the base data misses. The hardness lesson of ALT reappears in how preference and reinforcement-style post-training mine difficult, model-fooling cases, hard negatives and adversarially selected prompts, rather than easy ones, because that is where the learning signal lives. The spatial and representational care of InterAug and Know Your Space echoes in region-level and consistency-based objectives that ground a model on the right part of an image and in the careful construction of positive and negative pairs for contrastive and reward modeling. The vocabulary has changed, from data augmentation to data engines and synthetic post-training corpora, but the underlying move is identical; deliberately manufacture the right variations, with intent about what, how hard, and where, and let the model learn the invariances you actually care about. Augmentation did not get left behind by foundation models; it got promoted to one of the central design problems of building them.