Posts

Advantages and Limitations of Deep Networks as Local Interpolators, Not Global Approximators

This blog addresses a common misconception in the mathematics community: the belief that deep networks can serve as global approximators of a target function across the entire input domain. I write this post to emphasize the importance of understanding the limitations of deep networks' global approximation capabilities, rather than blindly accepting such claims, and to highlight how their strengths as local interpolators can be effectively leveraged. To clarify, deep networks are fundamentally limited in their ability to learn most globally defined mathematical transforms, such as the Fourier transform, Radon transform, and Laplace transform , particularly in high-dimensional settings. (I am aware of papers claiming that deep networks can learn the Fourier transform, but these are limited to low-dimensional cases with small pixel counts.) The misconception often stems from the influence of the Barron space framework, which provides a theoretical basis for function approximation. Wh...

The Impact of Data-Driven Deep Learning Methods on Solving Complex Problems Once Beyond the Reach of Traditional Approaches

This blog is intended for mathematicians with limited background in physics and computational biology. Recent advancements in data-driven deep learning have transformed mathematics by enhancing—and sometimes surpassing—traditional methods. By leveraging datasets, deep learning techniques are redefining problem-solving and providing powerful tools to tackle challenges once considered impossible. This marks a new paradigm, driven by data, advanced computation, and adaptive learning, pushing the boundaries of what can be achieved. The profound impact of data-driven deep learning was recognized by the 2024 Nobel Prizes in Physics and Chemistry.  The Nobel Prize in Physics honored John Hopfield and Geoffrey Hinton for their groundbreaking contributions to neural networks. Hopfield developed an early model of associative memory in neural networks, known as the Hopfield network , which is based on the concept of energy minimization. The energy function is represented by: \[E(\mathbf{...

Exploring the Opportunities and Limitations of Generative Models in Medical Imaging

This blog explores the opportunities and limitations of generative models, including GANs and Diffusion models, in the field of medical imaging. Generative models like ChatGPT have undeniably achieved remarkable success in language modeling and the entertainment industry, where minor errors, omissions, or inaccuracies are less critical and can be easily corrected through human intervention and iterative refinement. The success of these data-driven generative models is anticipated to have a profound impact in the future, as they harness the collective wisdom of large datasets and efficiently tackle time-consuming, routine tasks. However, the requirements in the medical domain are far more stringent, with a heavy emphasis on accuracy and expert interpretation. For example, the expertise of a skilled specialist is far more valuable than the average opinion of a general practitioner, and there are countless patient-specific cases that cannot be adequately captured by collected data through...

About technological innovation

Throughout my three decades in academia, I've observed a continuous loop of conversations highlighting the vital role of innovation, reform, and the imperative for pioneering work in research and development (R&D). The principle of "High Risk and High Return" has been a central theme, with a strong push for long-term visions over short-term gains. However, this consistent focus has led to a widespread sense of exhaustion, fueled by the concerning observation that most innovative efforts remain trapped within academic circles, barely touching the wider industrial landscape. Academics making breakthroughs often have limited capacity to steer these innovations towards becoming successful commercial products. Industry-academia collaborations struggle to bear fruit due to a variety of subtle and complex reasons, leading numerous promising innovations to become mere line items on a curriculum vitae, devoid of practical implementation. Finding effective R&D policies that...

Utilizing Implicit Neural Representations for Solving Ill-posed Inverse Problems

Image
Recently, the field of medical imaging has witnessed numerous attempts aimed at producing high-resolution images with significantly insufficient measured data. These endeavors are motivated by a variety of objectives, such as reducing data acquisition times, enhancing cost efficiency, minimizing invasiveness, and elevating patient comfort, among other factors. Nevertheless, these efforts necessitate tackling severely ill-posed inverse problems, due to the significant imbalance between the number of unknown variables (needed for desired resolution) and the number of available equations (derived from measured data). For a clearer understanding, let's examine a linear system represented by $\mathbf{A}I = \mathbf{b}_{I} + \mathbf{\epsilon}$ , where $\mathbf{A}$ represents an $m \times n$ matrix with a highly underdetermined scenario ( $m \ll n$ ). This matrix $\mathbf{A}$ serves as a linearized forward model. In this formulation, $I$ is an $n$-dimensional vector representing the imag...

Exploring the Fundamentals of Diffusion Models

Image
This blog post explores the recently popular diffusion models. Since there are many excellent explanations available, I don't intend to duplicate their detailed discussions. Rather, I try to offer a succinct summary that highlights the essential principles and foundational concepts behind diffusion models. Diffusion models belong to the family of generative neural network models, alongside Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs). The term "diffusion" originates from non-equilibrium thermodynamics, wherein particles (images) migrate from high concentration regions (represented by a data manifold filled with structured data) to the low concentration regions (In essence, the vast majority of points within the entire space are overwhelmingly dominated by noisy images). This concept illustrates how diffusion models iteratively refine noise into coherent, structured data by effectively reversing this natural diffusion process. To make the conc...

CT versus MRI: Exploring the Similarities and Distinctions

In this blog post, we take a brief dive into the fundamental principles of Magnetic Resonance Imaging (MRI) and Computerized Tomography (CT), highlighting the inherent differences between the two modalities. While recent strides in AI-driven techniques have enabled simulations of CT from MRI and vice versa, it's essential to recognize the distinct nature of each imaging method. Given their distinct characteristics, one cannot simply replace the other. CT scans offer tomographic (cross-sectional) images based on material linear attenuation coefficients , by directing X-rays through the body from various angles and measuring the decrease in X-ray intensity along linear paths. CT scans outperform MRI in their ability to visualize bone fractures and dense tissues, rendering them indispensable for evaluating traumatic injuries. Despite the ionizing radiation exposure associated with CT scans, they are still utilized for guiding procedures such as biopsies and surgeries, which may not ...