NVIDIA Is Using Deep Learning To Repair And Reconstruct Images
A scientist team led by NVIDIA has introduced a new method that can be used to repair or reconstruct images with the help of Deep Learning (using Partial Convolutions). The process that this method goes through is known as “Image Inpainting”.
So, this new method can literally help to repair damaged images or even remove objects from an image. NVIDIA posted a Video on their YouTube channel which gives a brief demonstration of the implementation.
In this video NVIDIA showcased how they were removing or recreating certain parts of several photos.
To prepare to train their neural network to perform these tasks, the scientists first generated 55,116 masks of random streaks and holes of arbitrary shapes and sizes for training. They generated nearly 25,000 extra ones for testing as well. These masks were further categorized into six different categories based on sizes relative to the input image, in order to improve reconstruction accuracy.
[Image Credits: NVIDIA]
The researchers mentioned,
“…Previous deep learning approaches have focused on rectangular regions located around the center of the image, and often rely on expensive post-processing”.
So in theory, this new method should be more accurate while being cost effective at the same time.
The scientists trained their neural network by applying the generated masks to images from the ImageNet, Places2 and CelebA-HQ datasets & NVIDIA says that, NVIDIA Tesla V100 GPUs and the cuDNN-accelerated PyTorch deep learning framework were used to accomplish this job.
What my understanding is, by using this method NVIDIA’s neural network is able to detect objects, their characteristics, their surroundings & the characteristics of those surroundings as well. Whenever you need to create/remove an object on/from an image, you mark that area & the network detects the damaged area or extra object. Then for damages areas, it tries to find the prefect match (as a replacement) & replaces the area with the best match it is able to find from a huge online library. For removable objects, it detects the properties of the object & it’s surroundings and replaces the objects with a similar area with the same characteristics of it’s surroundings.
Something to be exited with is that, in future this can be implemented as a feature in regular photo editors. So, in that way editing or recreating photos will be a lot easier compared to what we have today.