In March 2020, a paper named Neural Radiance Fields, NeRF appeared. With this technique, we could take a bunch of input photos and train a neural network to learn them, and then synthesize new, previously unseen views of not just the materials in the scene but the entire scene itself. We can learn and reproduce entire real-world scenes from only a few views by using neural networks. However, it had some limitations, such as trouble with scenes with variable lighting conditions and occlusions.
Now a follow-up paper has come named Deformable Neural Radiance Fields (D-NeRF). The goal here is to take a selfie video and turn it into a portrait that we can rotate around freely. This is something that the authors call a nerfie.
The original NeRF technique cannot perform this task, as it doesn’t do well at all with moving things. And here is where the deformable part of the name comes into play. This new technique D-NeRF outputs a clean reconstruction. It performs well even on tough cases with beards, all kinds of hairstyles, etc. It can even work for glasses and computes the proper reflection and refraction off of the lens. It also has a feature, using which we can even zoom out and capture the entire body of the test subject.
The authors associate a latent deformation code (ω) and an appearance code (ψ) to each image. They then traced the camera rays in the observation frame and transform samples along the ray to the canonical frame. It is done using a deformation field encoded as an MLP that is conditioned on the deformation code ω. The template NeRF module is queried using the transformed sample location (x, y, z0), the viewing direction (θ, φ), and the appearance code ψ as inputs to the MLP.
For more detail visit: https://nerfies.github.io/
References
- Deformable Neural Radiance Fields paper: https://arxiv.org/abs/2011.12948
Author
Shubham Bindal
Applied Singularity 