Generative models, for example, GANs and diffusion models in particular, are usually discussed in the context of creating new content: synthesizing faces, generating art, producing music. But there’s an arguably more powerful use for them. Once a generative model has learned what valid data looks like, it becomes a prior, a constraint you can impose on any problem where you need to recover a signal from incomplete or corrupted observations. This series of posts explores that idea across imaging, physics simulation, and audio. The recurring finding is that coupling a learned generative prior with explicit physical or structural constraints produces solutions that are more robust, data-efficient, and trustworthy than either component alone. Self-consistency between the learned prior and the observed data is the lever that makes it all work.
Part 1: When Generative Models Meet the Real World
Reference: Robust Projection onto Image Manifolds with Corruption Mimicking (MimicGAN), IJCV 2020
This series follows one idea through very different formulations and use-cases: a generative model, once it has learned what valid data looks like, becomes a powerful prior you can lean on to solve inverse problems. Not by generating new content, but by constraining a search. The story starts with the most basic version of that idea and the most stubborn obstacle to making it work in practice.
Suppose you have trained a vision generative mode, e.g., a Generative Adversarial Network, that knows what faces look like. It has learned a smooth manifold of face images. Now someone hands you a photo that has been cropped, rotated, or has missing pixels, and asks you to project it back onto that manifold, finding the closest clean face the generator can produce. This projection operation is the foundation that everything else in the series builds on.
The standard tool, Projected Gradient Descent, optimizes a latent vector to minimize the pixel difference between the generated image and the observation. It is simple and elegant, and it breaks the moment the observation carries any corruption. The reason is subtle but fatal: PGD assumes the observation already lives on the image manifold. A cropped face does not. A rotated face does not. Asking for a clean face that pixel-matches a corrupted input is asking the wrong question, and the optimizer happily returns garbage.
The Key Insight: Don’t Fight the Corruption, Mimic It
The fix is to stop pretending the corruption is not there and instead learn what it is, at test time, while solving the projection. Two things are optimized together: a latent vector that represents the projection onto the manifold, and a small surrogate network that approximates the unknown corruption. The optimization alternates between them. First the surrogate updates to better explain how a clean image became the corrupted observation. Then the latent vector updates to find a better clean image given the current estimate of the corruption. The generator stays frozen the entire time, so this is purely a test-time procedure with no retraining.
The surrogate is deliberately shaped around the corruptions that show up in the wild. A spatial transformer layer absorbs geometric shifts like rotation, scale, and translation. Convolutional layers handle intensity and color changes. A masking layer accounts for missing pixels. A skip connection lets the network fall back to the identity when nothing is wrong. Plain PGD turns out to be the special case where the surrogate is forced to be the identity, so relaxing that single assumption buys robustness almost for free.
Why This Matters?
A robust projection is not an end in itself; it is a primitive that unlocks several applications. Adversarial perturbations are just another corruption, so projecting an adversarial example back onto the natural image manifold cleans it, outperforming dedicated defenses. An image that cannot be projected well, even after the corruption is mimicked, is probably anomalous, which gives a handle on anomaly detection. And when test images drift from the training distribution, the same mechanism bridges the gap without paired examples.
The deeper lesson sets up everything that follows. Real observations are messy, and the productive move is not to build fragile systems that assume clean inputs, but to estimate the mess as part of the solution and work backwards through a trusted generative prior. The next part shows that this idea is not really about images at all.
Part 2: Unmixing Sounds Without Labels
Reference: Unsupervised Audio Source Separation using Generative Priors, INTERSPEECH 2020
Part 1 left us with a primitive: project a corrupted observation onto a learned manifold by estimating the corruption at test time. A natural question is whether that primitive is really about images, or whether it captures something more general. The cleanest way to test that is to change the data type entirely and see if the same machinery still works. So we move from pixels to sound.
Audio source separation is one of the oldest inverse problems in signal processing. A recording mixes a voice over piano and drums, and the task is to recover each source on its own. Supervised systems do this well when they have paired data, meaning mixtures alongside their separated sources, but that data is expensive, and the resulting models are brittle. Change the number of sources, the mixing process, or the instruments involved, and the whole thing needs retraining. The opening for a prior-based approach is exactly this brittleness.
The Insight: One Generator Per Source, Optimize at Test Time
The recipe is the same one from Part 1, generalized to several priors at once. Train a separate generative model on each source type using only unlabeled examples of that source. Then, given a mixture, search every source-specific latent space simultaneously for the latent codes whose generated outputs add up to the observed mixture. Each generator constrains its source estimate to lie on the manifold of realistic sounds for that voice or instrument, so the optimization is forced to explain the mixture as a sum of plausible sources rather than arbitrary signals.
The Spectral Domain Trick
One detail makes or breaks the method. Although the generators produce waveforms in the time domain, the loss that drives the optimization has to live in the spectral domain. Small phase differences between estimated and true sources blow up time-domain errors and send the optimizer in the wrong direction, while spectral losses tolerate those phase shifts and still capture perceptual similarity. The full objective combines four terms: a multiresolution spectral loss matching log-magnitude spectrograms at several scales, a source dissociation term that pushes the estimated sources to be spectrally distinct so one does not absorb another, a mixture coherence term tying the reconstructed mixture to the observed one, and a frequency consistency term that keeps frequency bins coherent within each time frame.
The payoff is large. On mixtures of spoken digits, drums, and piano, the approach clearly beats classical unsupervised methods like ICA, PCA, and NMF, with signal-to-interference improvements that are dramatic rather than marginal, and it holds in scenarios where the classical methods collapse. None of this requires a single labeled mixture during training; adding a new source type just means training one more generator.
The takeaway is the one we were testing for. The prior-as-regularizer idea is not tied to images. Train a model that knows what valid solutions look like, then use it to constrain a test-time search, and the modality barely matters. With that established, the rest of the series pushes the idea into harder territory: coupling priors with each other, and with physics.
Part 3: Teaching Models to Check Their Own Work
Reference: Improved Surrogates in Inertial Confinement Fusion with Manifold and Cycle Consistencies, PNAS 2020
Parts 1 and 2 used one or more generators as priors to constrain a search. This part keeps that ingredient but uses it differently, and adds a twist: instead of searching a latent space at test time, a network predicts directly into one, and a second model is trained to keep the first honest. The setting is also more complex, a fast neural surrogate for inertial confinement fusion, where each high-fidelity simulation is enormously expensive. The surrogate’s outputs are multimodal, combining several energy-band images with scalar diagnostics over a high-dimensional input space.
The Problem with Standard Surrogates
A plain surrogate just minimizes prediction error from inputs to outputs. For rich multimodal outputs that approach is fragile, data hungry, and, most importantly, inconsistent with inversion: predict the outputs, try to recover the inputs that produced them, and you get nonsense. Scientists need that backward direction to design experiments, so when forward and backward disagree the whole pipeline becomes untrustworthy.
Manifold and Cycle Consistency
Two ideas fix this. The first, manifold consistency, supplies the prior. An autoencoder is trained on the simulation outputs alone; in learning to reconstruct them it discovers the manifold that all physically valid outputs live on, and its latent space becomes a compact coordinate system for that manifold. This is the same prior the series keeps using, a model that has learned what valid data looks like, just built from an autoencoder instead of a GAN. The surrogate then predicts into that latent space, so every prediction decodes back to a physically coherent output. The prior knows what valid outputs look like; the surrogate only has to find the right spot.
The second idea, cycle consistency, trains a forward model and an inverse model together with the requirement that mapping inputs forward and then back recovers the original inputs.
Cycle consistency is a richer regularizer than squared error: it asks predictions not just to sit near the ground truth but to be physically meaningful, in that some real inverse can recover the inputs behind them. The results follow. Prediction error drops 15 to 25 percent, a surrogate trained on a third of the data matches one trained on all of it, predictions are smoother under input perturbations, and the consistency holds even against inverse models the surrogate never trained against, a sign the regularization captures something real rather than memorizing one pairing.
The shift is worth holding onto. The autoencoder prior still constrains where solutions can live, but now a forward and inverse pairing layers on top, and the two models keep each other honest. The next part carries that exact pairing, in latent space, back into images.
Part 4: Inverting the Uninvertible
References:
Improved StyleGAN-v2 based Inversion for Out-of-Distribution Images (SPHInX), ICML 2022
Adapting Blackbox Generative Models via Inversion (AdvIN), ICML 2023
Everything so far has leaned on a comforting assumption: the thing we want to recover lives on, or near, the manifold the generator already knows. Part 1 even depended on it. But the most useful applications break that assumption. The image we want to invert may be unlike anything the generator was trained on, and the generator itself may be locked behind an API. This part confronts both, and in doing so it reuses the forward-and-inverse pairing from Part 3, now inside the latent space of an image generator.
For example, a StyleGAN trained on aligned face photos has a rich latent space, but hand it a rotated face, a cartoon, or a medical scan and standard inversion falls apart. The latent space was never shaped for those images, and the optimization landscape turns treacherous.
Redesigning the Map, Not the Territory
The first observation is that the latent space itself is not the problem; the way we move through it is. Optimizing directly in the extended style space is non-robust for out-of-distribution images, to the point where tiny perturbations to the solution produce perceptually broken reconstructions. The fix is to optimize a projection head, a small network that maps a sampled prior into the style space, rather than chasing the latent code directly. Critically, the head decouples the codes for different layers of the generator, giving the optimization enough freedom to represent unfamiliar content without the instability of an unconstrained search.
The training trick is to draw a fresh sample from the prior at every iteration. That forces the projection head to produce consistent style codes no matter which sample it sees, a vicinal regularization that makes the recovered solution locally robust instead of a brittle point estimate. This is the same instinct as Part 3’s cycle consistency: demand that a mapping behave well over a neighborhood, not just at a single point.
When You Can’t Even Touch the Generator
The harder, more modern obstacle is access. State-of-the-art generators are increasingly available only through APIs, with no weights and no gradients to fine-tune. Adaptation to a new domain has to happen without touching the model at all. The move is to split adaptation into two clean steps. First, invert a small set of target-domain images onto the generator’s latent space, which can be done server-side. Second, train a tiny sampler, well under one percent of the generator’s size, to draw from the sub-manifold those inverted latents occupy. At inference you sample from the small model, push the result through the frozen generator, and out come target-domain images.
What is striking is how little data this needs. With only a few dozen target images, the approach is competitive with methods that fine-tune the entire generator, and because it operates entirely in latent space it inherits the base model’s semantic structure, so attribute edits like age, expression, and pose still work on the adapted outputs.
Seen together, these two results trace the same arc. When the image is out of distribution, redesign how you navigate the latent space. When the model is a blackbox, stay in latent space entirely and treat the generator as a fixed decoder. In both cases the generator’s hard-won knowledge of natural images is left untouched and simply accessed more cleverly. The final part keeps the generator frozen too, but couples it with something new: the physics of a measurement device.
Part 5: Seeing Through Walls (Almost)
The series has built up a single capability in stages. A generative prior can constrain a search (Parts 1 and 2), two models can constrain each other (Part 3), and a frozen generator can be accessed cleverly even when the target is unfamiliar or the model is a blackbox (Part 4). The last step adds the most demanding partner of all: the physics of a real measurement device, fused with a modern diffusion prior.
Computed tomography reconstructs a cross-section by sending X-rays through an object from many angles. With a full sweep from 0 to 180 degrees, classical methods recover a clean image. But many real systems can only measure a limited wedge of angles, say 0 to 60 degrees. This happens in security scanners that cannot fully encircle luggage, in medical setups where patient positioning limits the sweep, and in industrial inspection of large structures. With only a third of the views, classical reconstruction dissolves into streaks, missing edges, and hallucinated structures.
Why This Is So Hard?
Limited-angle CT is severely ill-posed because the missing angles correspond to entire frequency components that were simply never measured. No amount of clever math recovers information that was never recorded, so a prior is not optional; it is the only way to fill the gaps responsibly. Earlier deep-learning approaches either trained end-to-end networks that overfit to one imaging setup, or used generative priors that had no knowledge of the measurement physics and so drifted away from what the scanner actually observed.
Physics Meets Diffusion
The resolution is to make the prior and the physics take turns. Each step of a diffusion model’s refinement is split in two. A denoising update, conditioned on a quick low-fidelity reconstruction, nudges the image toward the distribution of realistic CT scans. Then a data-consistency update, framed as a proximal step, pulls the image back toward agreement with the actually measured limited-angle data. The diffusion model answers what real CT images look like; the proximal step answers what the scanner truly saw. This is the same don’t-trust-the-network-alone instinct from Part 3, but here the second voice in the conversation is a hard physical constraint rather than a learned inverse.
The gains are substantial, with reconstruction quality improving by more than 3 dB over prior diffusion-based methods across angular ranges, but three qualitative findings matter more than the numbers. A single trained model handles strikingly different domains, working on both medical body scans and checked-in luggage, which is unusual for reconstruction networks that normally need domain-specific training. Because the model is probabilistic, it can draw several distinct reconstructions from the same measurement, and the variance across those samples tracks the true reconstruction error, giving a built-in confidence map even without ground truth. And by reconstructing slices that stack into coherent volumes, it preserves 3D geometry well enough for downstream tasks like segmentation that fail outright on the artifact-laden classical reconstructions.
The Thread, Pulled Tight
Step back and the five parts tell one story. A generative model trained only to capture what valid data looks like becomes a lever for problems it was never explicitly trained to solve: cleaning corrupted images, separating sound, building trustworthy scientific surrogates, adapting locked-down generators, and reconstructing images from impossibly sparse measurements. The constant is a refusal to trust a single learned mapping in isolation. Whether the partner is an estimated corruption, a second source prior, an inverse model, a resampled latent neighborhood, or the physics of a scanner, the winning move is always the same: pin the solution between what the data demands and what a prior knows to be plausible. That balance, more than any single architecture, is what makes these methods robust, data-efficient, and trustworthy.