To your eyes solely: bettering Netflix video high quality with neural networks | by Netflix Know-how Weblog | Nov, 2022

  1. Video preprocessing, which encompasses any transformation utilized to the high-quality supply video previous to encoding. Video downscaling is essentially the most pertinent instance herein, which tailors our encoding to display resolutions of various gadgets and optimizes image high quality beneath various community circumstances. With video downscaling, a number of resolutions of a supply video are produced. For instance, a 4K supply video shall be downscaled to 1080p, 720p, 540p and so forth. That is usually performed by a standard resampling filter, like Lanczos.
  2. Video encoding utilizing a standard video codec, like AV1. Encoding drastically reduces the quantity of video information that must be streamed to your machine, by leveraging spatial and temporal redundancies that exist in a video.
  • A discovered strategy for downscaling can enhance video high quality and be tailor-made to Netflix content material.
  • It may be built-in as a drop-in resolution, i.e., we don’t want some other modifications on the Netflix encoding facet or the consumer machine facet. Tens of millions of gadgets that assist Netflix streaming robotically profit from this resolution.
  • A definite, NN-based, video processing block can evolve independently, be used past video downscaling and be mixed with totally different codecs.
Structure of the deep downscaler mannequin, consisting of a preprocessing block adopted by a resizing block.
Left: Lanczos downscaling; proper: deep downscaler. Each movies are encoded with VP9 on the similar bitrate and had been upscaled to FHD decision (1920×1080). You could must zoom in to see the visible distinction.
  • The neural community structure was designed to be computationally environment friendly and in addition keep away from any unfavourable visible high quality influence. For instance, we discovered that only a few neural community layers had been ample for our wants. To scale back the enter channels even additional, we solely apply NN-based scaling on luma and scale chroma with a normal Lanczos filter.
  • We carried out the deep downscaler as an FFmpeg-based filter that runs along with different video transformations, like pixel format conversions. Our filter can run on each CPU and GPU. On a CPU, we leveraged oneDnn to additional scale back latency.
A top-down view of integrating the deep downscaler into Cosmos.