Bringing dim images to life

Taking photos during a night out with friends under low-light conditions can be challenging for smartphone cameras, especially lower-end ones. In dimly-lit environments, your photos could appear “noisy”, blurry and poorly illuminated.

High-end phones and cameras with interchangeable lenses have features that can overcome the challenges of taking photos in low-light conditions, but they tend to be expensive.

Fortunately, these issues can be addressed using the latest advances in artificial intelligence (AI).

Through sophisticated algorithms and deep learning techniques that use neural networks to mimic how the brain learns, AI models have impressive capabilities in enhancing the visual quality and details of images captured on smartphone cameras in dim conditions.

At the Rapid-Rich Object Search Lab in EEE, our team is developing next-generation AI tools that leverage fundamental illumination physics models to achieve this end.

DEEP LEARNING MEETS DARK IMAGES

Traditional signal processing methods, such as histogram equalisation and noise reduction algorithms, have a limited ability to preserve details in low-light images and reduce their grainy appearance when the noise is removed.

Novel AI-based approaches can overcome these limitations by learning the characteristics of images from large datasets taken at low or normal light and the desired enhancements for photos. Through extensive training, AI models can analyse and extract patterns to intelligently enhance low-light images with improved brightness and contrast while reducing noise.

Our research team used these deep learning models to create an AI tool that can enhance low-light images and remove shadows.

Among the deep learning models used, Vision Transformers can recognise and classify images according to their components, while deep reinforcement learning uses neural networks to learn from experience and prioritise desired outcomes.

The AI software agent we developed leverages these models to progressively brighten dark images based on a user’s preference without introducing any obvious image artefacts – a step up from existing methods.

We also showed that Vision Transformers can remove unwanted shadows from an image without loss of visual quality, while using up to 150 times fewer AI model parameters compared to existing methods. With lower processing power and less memory needed, the software is suitable for photo enhancing on smartphones which have limited AI computational capabilities.

Another approach to improve the quality of dimly lit photos is through a deep diffusion model, which uses an AI system trained by adding or removing noise from images.

The deep diffusion model employs a combination of deep learning and diffusion models. It excels in generating high quality and visually appealing content by capturing complex relationships and patterns in the sets of image data used to train it. Two popular AI tools – Stable Diffusion and Midjourney – use the deep diffusion model for producing realistic AI images from text descriptions and prompts.

Beyond creating AI images, the model can enhance smartphone photos taken in the dark when it is equipped with physics-driven deep learning.

Guo Lanqing, a PhD student in my laboratory, developed a physics-driven diffusion AI model – called ShadowDiffusion – that combines the deep diffusion model with illumination physics AI tools. We showed that ShadowDiffusion performed significantly better than existing benchmarks and popular quality metrics for removing shadows in dim images.

ShadowDiffusion can also brighten images taken in low light while keeping noise in the picture low.

A photo taken in dim conditions (left). An existing method called LIME was able to brighten the image but the photo has a substantial amount of noise (middle). Our method uses the ShadowDiffusion AI model we developed and was able to brighten the image while suppressing the noise in the picture (right). Credit: NTU.

 

VIRTUAL STAINING

AI-enhanced imaging has various applications. As a next step, we are looking at how our AI tools can be used for medical imaging.

For instance, staining tissue samples for microscopic analysis is a necessary process to help clinicians diagnose patients. However, staining can be time consuming and costly.

Through our research, we aim to develop an AI software that can address these limitations by virtually staining a sample under microscopic analysis. We envision that such virtual staining can potentially save time, reduce costs and even replace physical staining in the future.

By Wen Bihan

Nanyang Asst Prof Wen Bihan from NTU’s School of Electrical and Electronic Engineering (EEE) is a Senior Member of the Institute of Electrical and Electronics Engineers. He is an elected member of the Institute’s computational imaging and visual signal processing and communications technical committees. His research interests span machine learning, signal processing and computational imaging.

More details of his research can be found in IEEE Transactions on Multimedia (2022), DOI: 10.1109/TMM.2022.3232206; The 37th AAAI Conference on Artificial Intelligence (2023), DOI: 10.1609/aaai.v37i1.25148; IEEE International Conference on Acoustics, Speech and Signal Processing (2023), DOI: 10.1109/ICASSP49357.2023.10094569; and Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023), DOI: 10.48550/arXiv.2212.04711.

The article appeared first in NTU's research and innovation magazine Pushing Frontiers (issue #22, August 2023).