Sky replacement for a music video
February 2021
I was contacted to do some VFX sky replacement for this video. I had a week to edit over 50 shots and replace the sky to make it seem more interesting and other worldly. Apparently it was a last minute decision by the record label to make the music video stand out, so the production company reached out to me to help out. This really isn’t my usual work as I mainly focus on 3D animation but i’m always up for a challenge. I wouldn’t normally document a job like this, but I thought the way I tackled it within the week deadline was worth showcasing.
Using machine learning to automate the process
There is a really incredible script called SkyAR. It lets you import a source video and sky texture and it replaces the sky in the video for the image you provided, with correct colour grading and motion tracking.
However, I quickly found out that the script would only export video up to 720p whereas the raw footage from the music video I was working with was 3k x 2k (a lot bigger than 720p). This presented a big big problem, as if I had to rotoscope the sky by hand it would have taken me a lot longer than a week for 50 shots. So I knew I had to come up with a solution using the script.
I came up with a pretty ingenious solution of using the script to generate a luma matte (black and white image) that adobe after effects could use to isolate the sky. White being area to remove and black being ignore. I fed the script a white image as the sky texture and fiddled with the scripts settings to shade the footage completely black. That way, it wont really matter what the resolution is, as when I scale it up and use it as a matte it will just affect the original raw footage. There was still the issue with it being only 20fps on every exported video from the script, but I quickly saw there was a jpg sequence setting which allowed me to get an image sequence instead of a video file. That way I can determine how many fps it is in my editing software. It felt really good figuring all that out as I wouldn’t be able to finish the job otherwise :) me smart.
I made some space-y(ie?) looking sky textures in photoshop to replace the sky in the footage. As I only used SkyAR to generate a matte for me, I wasn’t able to take advantage of the fact it also tracks the skybox to the footage making it a believable part of the shot. I tracked all the footage in after effects instead. Some shots were fine as not a lot of camera movement, others were so shakey and had blurred backgrounds that I had to manually track them. (which is as bad as it sounds.)
I needed to provide all the shots and accompanying luma mattes for the colourist to colour grade the footage together.
So I basically had to rinse and repeat that whole process 50 times in a week.