CodeFormer: Unlock the Future of Visual Content Restoration with Blind Face Restoration Technology
Synopsis
This invention presents CodeFormer, which leverages artificial intelligence (AI) for blind face restoration in photos and videos, rejuvenating old content for entertainment, advertising, and preserving historical documents. It uses a Transformer-based network to restore faces with high fidelity, and is adaptable to various degradation levels.
Opportunity
Photographs and films suffer from degradation due to time, wear, and damage. Restoration of visual content is vital in industries ranging from entertainment to advertising. AI, especially computer vision, plays a key role in providing the highly sought-after restoration solutions. CodeFormer offers a blind face restoration solution to rejuvenate old facial images and videos, giving them a new lease on life. This technology can revitalise classic films and TV shows, enhance viewer experiences, extend the commercial lifespan of iconic content, safeguard historical photographs and documents, elevate brand storytelling by revitalising vintage marketing materials, and allow individuals to reconnect with their past by restoring treasured family photos and home videos.
Technology
Blind face restoration refers to the challenging task of restoring degraded facial images and videos. This problem is highly complex due to the loss of information during degradation, resulting in countless possible high-quality restored images and videos from low-quality inputs. CodeFormer, a Transformer-based prediction network, is designed for the reconstruction of damaged or deteriorated faces in images and video frames. CodeFormer excels at restoring faces with a natural appearance that closely resembles the intended target.
What sets CodeFormer apart is its adaptability to various degradation levels. A controllable feature transformation module enables users to fine-tune the balance between restoration fidelity and image quality. To enhance network expressiveness and resilience against degradation, a quantised autoencoder was trained to learn a context-rich codebook. Building upon this autoencoder, a transformer module was trained and fine-tuned for restoration tasks. Additionally, the controllable feature transformation module provides users with the flexibility to tailor the restoration process to their preferences.
Figure 1: Enhancement of the old photo of the 5-th Solvay conference taken in 1927.
Figure 2: Overview of the CodeFormer Technology.
Applications & Advantages
Main applications include:
Restoration of old videos, films and marketing materials in the entertainment and advertising industry.
Restoration of deteriorated faces in high-value historical photos and documents for archival preservation.
Restoration of family photos and home videos for family heritage.
Enhancement of captured photos suffering from compression and blurring.
The advantages of CodeFormer's technology include:
High-quality restoration of degraded facial images and videos, maintaining a natural appearance.
Adaptability to various levels of degradation, allowing for tailored restoration processes.
Use of a quantised autoencoder and transformer module enhances network expressiveness and resilience against degradation.
Flexibility for users to fine-tune restoration fidelity and image quality through a controllable feature transformation module.
Broad application across industries like entertainment, advertising, and archival preservation, enhancing viewer experiences and safeguarding historical content.
Ability to revitalise family photos and home videos, connecting individuals with their past.
References
Shangchen Zhou, Kelvin C. K. Chan, Chongyi Li and Chen Change Loy, Towards Robust Blind Face Restoration with Codebook Lookup Transformer, Advances in Neural Information Processing Systems 35 (NeurIPS 2022), Arxiv.org: https://arxiv.org/pdf/2206.11253.pdf
.tmb-listing.jpg?Culture=en&sfvrsn=6c7b6f1f_1)
