Optical and dimensional limitations of images can be partially overcome today using signal enhancement technologies which typically rely on supervised, deep-learning methods. However, as existing methods are restricted to specific training images and distortion types, they provide poor results for any practical case, unless huge set of data-pairs exist and all share the same exact distortion type. The current invention enables signal enhancement of low-resolution images, by exploiting deviations from expected internal patch recurrences detected within the image itself. More specifically, the approach leverages cross-scale internal repetition of image-specific information, which is trained, at test time, on internal examples extracted solely from the test image. This first unsupervised conventional neural network (CNN)-based super-resolution approach allows for signal enhancement of real-world images acquired under suboptimal, unknown or image-specific conditions and has been shown to outperform state-of-the-art technologies.
Signal enhancement of corrupted input collected under suboptimal conditions, poses multiple challenges, including super-resolution, denoising, deblurring, dehazing and lens artifact correction. While deep-learning techniques have dramatically improved super-resolution performance, they remain limited to specific, high-resolution training data, free of distracting artifacts. For single-image enhancement that does not obey ideal conditions, these limitations demand lengthy and exhaustive training on external databases and strict fulfillment of the training conditions to obtain satisfactory results.
This invention introduces zero-shot super-resolution (ZSSR), which exploits deep-learning methodologies, without requiring any prior image or training data. The internal recurrence of data within the single input image is exploited for on-line training of a small image-specific convolutional neural network (CNN) in examples extracted from the low-resolution image itself. The method bears no recurrence patch-size limitation, enabling adaptation of the CNN to different settings for the same image, and requires no external information or prior training. This unsupervised CNN-based super-resolution method has been shown to substantially outperform externally trained state-of-the-art super-resolution of suboptimal, low-resolution images (Figure A). In cases of ideal imaging conditions, the ZSSR output proved competitive to that of state-of-the-art supervised methods (Figure B).
A) Super-resolution of a suboptimal low-resolution image using the ZSSR approach versus state-of-the-art (SotA) approaches.
B) Super-resolution using ZSSR as compared to state-of-the-art methods, of a low-resolution image generated under ideal, supervised conditions.
Advantages
- Suitable for a wide range of images and data types
- Applicable for images of any size and any aspect ratio
- No pre-training requirement
- Adaptable to images with known or unknown imagining conditions
- Time-effective training
- No requirement of side information/attributes
- No requirement of additional images
Applications
- This invention can be applied in a range of images and distortion types requiring enhancement, including:
- Single video sequences
- Old photos
- Noisy images
- Biological data
- Medical images (e.g., fMRI)
- Audio sequences
The image-enhancement platform has been demonstrated effective for super-resolution, dehazing, watermark removal and image defect elimination (link to project web-page). Ongoing work is optimizing the technology for fMRI image applications. A patent application has been submitted.
This innovation can be of commercial relevance in a wide range of markets reliant upon imaging technologies, including:
- Consumer image processing software programs
- Military and security
- Medical imaging
- Microscopy and other lab imaging
