What Happens When You AI Upscale a Video 51X?

Описание: Ever wondered what happens if you render out a 4K video in standard definition, then used AI upscaling to bring it back to 4K, only to re-render it out to do the entire process over and over again?
What do 51 passes of AI upscaling and enhancement do to a piece of video footage?

0:00 - Original resolution footage, 1 pass AI stabilization. No AI upscaling. Light grain added.
1:11 - 1 AI upscaling pass to 4K using iris model. Add grain. All default settings. Iris model auto selected.
2:22 - 2 AI upscaling passes to 4K using iris model.
4:42 - 4th AI pass, I got impatient and bumped the 'recover detail' slider up to 50 from the default of 20.
5:54 - On the 5th and 6th pass, I was still feeling impatient and turned the 'grain' up to 6 from the default of 5.
11:44 - 10 AI upscaling passes to 4K using iris model. For pass 10 and beyond, 'recover detail' was bumped up to 75 - 85. Grain remains at the default of 5.
17:35 - 15 AI upscaling passes to 4K using iris model.
23:26 - 20 AI upscaling passes to 4K using iris model.
29:18 - 25 AI upscaling passes to 4K using iris model.
35:09 - 30 AI upscaling passes to 4K using iris model.
41:00 - 35 AI upscaling passes to 4K using iris model.
46:52 - 40 AI upscaling passes to 4K using iris model.
52:43 - 45 AI upscaling passes to 4K using iris model.
58:34 - 50 AI upscaling passes to 4K using iris model. Grain was turned down to 2.
59:45 - 51 and final AI upscaling pass to 4K using iris model. Grain turned down to 2.

I recorded a number of QHD video files, then edited the clips into a sequence of crossfading abstract colours and forms quickly moving in and out of focus. I am working under the assumption that AI models are trained on the kinds of things normal people make, and I wanted to give it something it wasn't designed for, something it would struggle with. I exported each stage at 768x432, which is close enough to standard def while maintaining the QHD aspect ratio. I used a high bitrate so compression artifacts wouldn't steal the show, I wanted the focus to be on what the AI models were introducing, not what h.264 was taking away. I introduced some fake digital noise or grain with each step to further prevent compression jaggedness in areas of saturated colour. I also wanted the noise to introduce a bit of random into the image so texture wouldn't be obliterated, since the AI upscaling includes noise reduction.

I was originally intending on doing 50 AI passes, but when I was editing, I realized I was 15 seconds short of hitting the 1 hour mark, so I gave it one more pass to push beyond the 60 minute mark and make this a full length feature film!

This artwork is inspired by the piece "I Am Sitting In a Room" by Alvin Lucier. While Lucier wrote about the sound characteristics of the room taking over the sound and obliterating his original words, I always felt this overlooked the role of the technology itself in shaping the final sound. A microphone is going to emphasize, attenuate, and otherwise shape the sound it captures. The pre-amp will introduce some noise while filtering others. The analogue tape it is being recorded to will have it's own quirks and limitations. A speaker playing back the recording will further shape the sound.

Let's take the room out of the equation and do the whole process digitally, where the real world can't creep in, and it will only be the technology shaping the final results.

Source material is video footage of crumpled aluminum foil with three RGB LED portable lights shining on it.
Shot on a Nikon Z6 with the 105mm /2.8 MC
Video edited and rendered out in Vegas 19.
AI upscaling done in Topaz Video AI v6.2.2
AMD 780M doing the GPU AI work.

_________________________________________

http://kyleclements.com