Castle in the Sky

Dynamic Sky Replacement and Harmonization in Videos. Through the power of neural network-based learning algorithms today it is possible to perform video to video translation. For instance, it goes, a daytime video, and out comes a nighttime version of the same footage. 

This work ‘Castle in the Sky’ proposes a vision-based method for video sky replacement and harmonization, which can automatically generate realistic and dramatic sky backgrounds in videos with controllable styles. This method runs in real-time and is free of user interactions and the authors decompose this artistic creation process into a couple of proxy tasks such as sky matting, motion estimation, and image blending.

Take a look at this video. The authors changed the sky and put a spaceship in there, which is already amazing. The spaceship is not stationary but moves in harmony with the other objects in the video. Moreover, since the sky has changed and the lighting situation has changed with it, the colors of the remainder of the image also have to change.

We can do so much more with this. For instance, put a castle in the sky, make it an extra planet. Let’s see if it is able to recolor the image after changing its surroundings. How well does it do when the background is changed to a dynamic one like a thunderstorm? This new method handles this case as well. Click here for the video.

So before we look under the hood to see how all this is done, let’s first list our expectations

1. We expect that it has to know what pixels to change to load a different sky model.
2. It should know how the image is changing and rotating over time. 
3. Some recoloring also has to take place.

Now let’s have a look and see how the model architecture is able to fulfill our expectations.

1. It has a sky matting network and this network finds the parts of the image where the sky is, so this network does the work to fulfill our Expectation-1.
2. For Expectation-2, there is the motion estimator that computes the optical flow of the image, this tracks the movement of the sky over time.
3. And there is the recoloring module as well to do the recoloring (if required).

So this can do not only sky replacement but detailed weather and lighting syntheses are also possible.

By looking at the results, listing our expectations, and then examining the architecture of the neural network, we can evaluate the research work with some confidence.

Author

Shubham Bindal

References

1. Castle in the Sky: Dynamic Sky Replacement and Harmonization in Videos: https://arxiv.org/pdf/2010.11800.pdf