Exploring the Opportunities and Limitations of Generative Models in Medical Imaging

This blog explores the opportunities and limitations of generative models, including GANs and Diffusion models, in the field of medical imaging. Generative models like ChatGPT have undeniably achieved remarkable success in language modeling and the entertainment industry, where minor errors, omissions, or inaccuracies are less critical and can be easily corrected through human intervention and iterative refinement. The success of these data-driven generative models is anticipated to have a profound impact in the future, as they harness the collective wisdom of large datasets and efficiently tackle time-consuming, routine tasks. However, the requirements in the medical domain are far more stringent, with a heavy emphasis on accuracy and expert interpretation. For example, the expertise of a skilled specialist is far more valuable than the average opinion of a general practitioner, and there are countless patient-specific cases that cannot be adequately captured by collected data through deep learning techniques. Therefore, it would be naive to expect that the success of generative models in other domains will seamlessly translate into the medical field.  For instance, the same challenges that hinder the development of self-driving cars—adapting to the unpredictable mix of vehicles, pedestrians, cyclists, and pets in Manhattan—illustrate why it is so difficult to apply generative models in complex, real-world medical environments. Understanding the risks and limitations of generative models in medical practice is essential before their widespread adoption.  The perspectives shared in this blog are drawn from the insights provided by papers [1], [2], [3], and [4].

Let's begin by exploring the opportunities presented by generative models. In the medical field, supervised learning methods like U-Net have proven to be highly effective in clinical applications. However, in many situations, collecting paired data is either prohibitively costly or exceedingly difficult. As a result, when paired data is unavailable and only unpaired data can be obtained, generative models emerge as the most practical solution. Notably, numerous studies have demonstrated the use of generative models for denoising medical images, though concerns remain about their integration into clinical practice.

Now, let's shift our focus to the limitations of generative models, particularly GANs and Diffusion models, with an emphasis on challenges related to memorization and distortion errors. To make our discussion more concrete, we will specifically examine low-dose dental CBCT systems, where reconstructed images using conventional methods are often affected by noise and artifacts, particularly in the presence of metallic implants, which are common in dental imaging. The pixel dimensions in dental CBCT typically range around 512x512 per slice. The key challenge here lies in achieving effective image enhancement to reduce these artifacts and improve overall image quality.

When using Diffusion Models (DMs) and GANs for image enhancement in dental CBCT, we are concerned about memorization—where the model merely recalls and reproduces interpolations of the training data rather than generalizing from underlying patterns. In such cases, these models may struggle to enhance new CBCT images effectively, as they fail to learn the generalizable properties of CBCT noise, artifact patterns, and image characteristics. Instead, the model produces slight variations of the images it has seen during training, limiting its ability to generalize to new data. To better understand the issue of memorization, let's briefly review the mechanisms of GANs and DMs. Given unpaired training data $\text{Data}_{\text{ref-tr}} = \{x_*^{(1)}, \dots, x_*^{(K)}\}$ (reference MDCT images) and $\text{Data}_{\text{noisy-tr}} = \{x^{(1)}, \dots, x^{(M)}\}$ (noisy dental CBCT images), a generator $G_\theta(\cdot)$ in either GANs or DMs is trained using a loss function based on this data. The key distinction between GANs and DMs is how they learn: GANs rely on adversarial feedback, while DMs explicitly learn to reconstruct the training data step-by-step.

In GANs, the training process involves a minimax game between the generator $G_\theta$ and the discriminator $D$. The discriminator aims to distinguish real MDCT images from generated ones, while the generator's goal is to create images that can deceive the discriminator. The adversarial loss comprises: \[  \text{Discriminator loss} = - \frac{1}{K} \sum_{k=1}^{K} \log D(x_*^{(k)}) - \frac{1}{M} \sum_{m=1}^{M} \log \left( 1 - D(G_\theta(x^{(m)})) \right)\] and   \[ \text{Generator loss} = - \frac{1}{M} \sum_{m=1}^{M} \log D(G_\theta(x^{(m)})) \] In addition to the adversarial loss, the generator can be trained with some fidelity-based losses, such as Perceptual Loss and Structural Similarity Loss (SSIM), to ensure the generated images $G_\theta(x^{(m)})$ closely match the reference images $x_*^{(k)}$ in both appearance and structure.

In contrast, DMs use an iterative denoising process to reconstruct reference images. Starting with noisy inputs, the model progressively removes noise through a series of timesteps to recover the clean reference image $x_*^{(k)}$. At each timestep $t$, the model is trained to predict the noise added to the reference image. The loss function for DMs is defined as: \[ L_{\text{DM}} = \frac{1}{K} \sum_{k=1}^{K} \mathbb{E}_{t, \epsilon} \left[ \| \epsilon - \epsilon_\theta(x_{t,*}^{(k)}, t) \|^2 \right] \]Here, $x_{t,*}^{(k)}$ represents the noisy version of the reference image $x_*^{(k)}$ at timestep $t$, and $\epsilon$ is the actual noise added. The model $\epsilon_\theta(x_{t,*}^{(k)}, t)$ is trained to predict and remove the noise at each step, progressively refining the image until it closely matches the original reference image. This step-by-step denoising process enables DMs to explicitly reconstruct the original reference data, while GANs rely on the adversarial feedback from the discriminator to indirectly guide the generator toward creating realistic images.

Challenges of Memorization over Generalization in Medical Imaging Models

In medical imaging, a key challenge with generative models is their propensity to memorize training data rather than generalize effectively. This memorization can bias models towards recreating common patterns from the training set, limiting their ability to identify or generate data for rare medical conditions. Since rare cases are often underrepresented in training datasets, memorization can cause models to overlook or inaccurately represent these cases, diminishing their diagnostic value. This issue often results in models generating images that largely reflect typical cases, failing to capture the critical variations necessary for recognizing rare pathologies.

A certain degree of memorization is essential for generalization, allowing models to learn and apply patterns from the training data. However, when memorization becomes excessive—where the model begins to replicate specific training examples—it poses significant problems. This is particularly concerning in areas like medical diagnostics, where capturing accurate, patient-specific variations is crucial. Overfitting to the training data in such cases can severely limit the model's ability to generalize to new, unseen cases, undermining its clinical effectiveness.

The primary goal in medical applications is to detect rare diseases and outliers by generating variations that deviate from the training data. However, models like diffusion models and GANs, due to their inherent limitations, often struggle to generate the meaningful variations needed for accurate and reliable medical diagnostics.

GANs are known to be vulnerable to membership inference attacks, as their adversarial training process can sometimes lead the generator to produce data that closely resembles specific training samples, heightening the risk of data leakage. In [1], the authors observed that diffusion models exhibited even higher membership inference leakage than GANs in their experiments.

A related consideration in generative models is the perception-distortion tradeoff. In [2], the authors assert that it is only possible to enhance either perceptual quality (typically assessed through human evaluations using mean opinion scores) or distortion (which refers to the degree of dissimilarity between the generated and original data, often affecting diagnostically important features). Improving one inevitably degrades the other, making it impossible for any algorithm to optimize both simultaneously. This tradeoff is a fundamental limitation of generative modeling, particularly in medical fields where both high visual fidelity and the accurate preservation of crucial details are vital.

Under Construction

Under Construction

Under Construction

Under Construction

Reference

[1] Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramer, F., ... & Wallace, E. (2023). Extracting training data from diffusion models. In 32nd USENIX Security Symposium.

[2] Blau, Y., & Michaeli, T. (2018). The perception-distortion tradeoff. In Proceedings of the IEEE conference on computer vision and pattern recognition 

[3] Hyun, C. M., Baek, S. H., Lee, M., Lee, S. M., & Seo, J. K. (2021). Deep learning-based solvability of underdetermined inverse problems in medical imaging. Medical Image Analysis

[4] Dar, S. U. H., Ghanaat, A., Kahmann, J., Ayx, I., Papavassiliu, T., Schoenberg, S. O., & Engelhardt, S. (2023, October). Investigating data memorization in 3d latent diffusion models for medical image synthesis. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 56-65). Cham: Springer Nature Switzerland.