Had the pleasure of working with Jason Garrett-Glaser (better known as DarkShikari) circa 2011 on a patch for x264 to better utilize more cores (slice-threaded lookahead). At the time he was working with Sony to optimize their PlayStation game streaming service, and this had the potential to lower encoding latency (critical to game streaming) even on same-core systems by spreading the frame over more cores to do more parallel work, and doing more intelligent scheduling and convergence after that tile-based work was reconverged into a frame. At the time we were using servers with 48 physical cores to do transcoding, a bit of a rarity in a single NUMA space back then and there were no instances to test this on so I flamegraph traced a few encodes at very high thread parallel counts and send back the instrumented profiles (from a custom compiled version) showing the core work being done and he used that to optimize the motion estimation code which worked on the first shot to perfectly level out cpu usage and increase encoding speed across ~24 cores (was previously topping out around 8 and running one core hot iirc). I got a look into how exotic and complex this project was, particularly in terms how low level and advanced things like motion estimation, quantizing and perceptual compression were... it was and still is sort of magic how many smart techniques are baked into something that we barely pay attention to. They were, even at that time, working on x265.
His blog, Diary of an x264 developer, was a great read too for anyone wanting to get a vantage point on how crazy that codebase was.
Using multiple CPUs for encoding goes way back. The early (1994) C-Cube MPEG-1 and MPEG-2 real-time encoders for DirecTV were built with 8 chips running in parallel (out of necessity since they were only 60 MHz processors) to encode 704x480. Each chip encoded a slice of the image and an auxiliary bus was used to send reconstruction video between chips so that motion vectors could cross chip image boundaries.
Here's a picture of the VME bus development board. It was a two board sandwich to support 10 chips in parallel (only one board pictured).
The first real-time HDTV encoders for terrestrial digital television (ATSC) were built by Harris using an IBM encoder chip. It took 6 chips to encode 1920x1080. The IBM chips didn't have an auxiliary bus, so you could see motion vectors limited to chip image boundaries (in this case, 6 tiles arranged like so).
Yeah at this time intel was just releasing their quick sync tech and a few vendors were putting 4 intel cpus on a pcie board and using it for video encoding! Matrox also had a considerable piece of this business doing video compression and editing/compositing using bespoke processors and boards.
Despite being more "power efficient" - both nvidia and intel's hardware solutions left a lot to be desired in terms of final quality for the same bitrate, compared to x264 that seemed to just put out substantially better quality video for the same bitrate even at the same h264 profile levels. There are still lots of x264 design elements, some even verbatim ported, into modern av1 and hevc encoders.
As someone doing a lot of DVD transcoding at the time, I remember being excited for Quick Sync on my new Sandy Bridge build - only to find that, while fast, it had absolutely _horrendous_ quality. I revisited it on my previous main Plex server build, which I think was a Haswell, and it was still unusable for transcoding compared to x264.
It took Nvidia until the encoder they bundled into 20xx series cards and Intel until the 6th or 7th generation of Quick Sync, but their hardware encoders are _finally_ at a point you can get decent quality at reasonable bitrates. Even so, x264 (and x265!) beat the pants off of them if one can spare the time and electricity; they're still that far ahead.
Very cool. Took a similar approach to optimizing openutv by looking at flamegraphs. UTV is a fork of openrv which is an older release of the commercial RV. Like many things it uses ffmpeg and also native readers and writers for certain formats. Allowed me to get it to 8k@60fps buttery smooth playback with hardware acceleration. If you are still in the video/imaging space would love your feedback on UTV. openutv.com
Intel has supported native hardware decoder for AV1 since Tiger Lake. That’s quite old. You could buy a used NUC with Tiger Lake. Even when new a NUC with 1115G4 or 1135G7 isn’t expensive. I found one with damaged WiFi for $20. https://www.ebay.com/itm/366767313411 You can buy it now.
I went to the link to buy a computer (a NUC11TNKi3) with this capability for $20, but all it would let me do is place a bid on it; I cannot buy it now.
Cheaper to get a long fiber optic HDMI cable and connect your TV to your PC GPU. That's what I do. You get the best uncompressed video (typ 4K 120Hz or 8K 60Hz), zero lag so it can be used for stuff other than video, and don't have to maintain another box
Lots of media devices that don't have AV1 decoders tend to have issues decoding high resolution AV1 in real time so you either get pauses to buffer or dropped frames.
Sorry, but YouTube began thermal throttling on a Firestick the moment they started forcing AV1 regardless of the presence of on-device hardware support.
If the common hardware customers already have fails when you attempt to use a newer codec (but works fine otherwise), that can't be said to be a mainstream codec yet.
Your particular model is not representative of "the common hardware" as a whole. And it can be mainstream even when a significant fraction of devices can't use it. AV1 is a third of Netflix streams and most YouTube streams, that's more than enough to qualify as mainstream.
Even if half the devices were broken, it would still be more than a third of streams and mainstream.
A lot less than half of devices are broken when playing YouTube, I assure you.
If they stopped forcing it, it would still be the best choice most of the time. It doesn't "have to be" forced to be a major choice. The forcing changes the exact percentage but not the overall picture.
Nvidia Shield is hardly the most popular anything. Roku has 43% of the market and Amazon has 35%.
Would be interesting to see if we could segment third-party from TV-embedded (see quote), but the metrics from the article seem to reflect actual consumer usage, as most consumers just use what comes in the TV they buy nowadays.
> This reflects a consolidated approach and eliminates yet another device in the home, and it’s catching on, as just 34% consider streaming media players as their primary viewing device. Meanwhile, 56% said the same for smart TVs.
Very fair. Looks like Amazon is a bit better with AV1 support starting in 2021 with the Fire TV Stick 4k Max - 1st Gen[0], with a single absense of it on the seemingly uber-budget friendly $35 "Fire TV Stick HD Alexa Voice Remote (2024)".
That's still a sliver of the market, but the more royalty-free* codecs we use, the better.
>> The NVIDIA Shield TV (Pro and Tube) is widely considered the best Android TV streaming device for self-hosted media servers in 2026. While the Apple TV 4K wins on UI polish and the Onn 4K Pro wins on price, the Shield wins on something most other devices can't match: lossless audio bitstream passthrough. TrueHD, TrueHD Atmos, DTS-HD MA, DTS:X - the Shield passes them all to your AV receiver untouched.
>> Combined with hardware decode for H.264, H.265, VP9, and AV1, plus an 8-year-old Tegra X1+ chip that still gets driver updates, the Shield is a genuine "set it and forget it" Jellyfin client.
Nvidia shield is a device that simply works and passes the audio untouched to my sound receiver and I enjoy it __everyday__.
Nvidia hasn't released a new model of the Shield since 2019. The current Apple TV model, Google TV model, Roku model, Fire TV, and Onn models support AV1.
The Shield is dead though and a long-time abandoned, right?
I remember the Shield going from fantastic to a horrible mess software wise over the span of a single Android generation (maybe 2019 or 2020), constant ARC weirdness, etc.
Shield is a 10 year old device at this point. It's essentially the 2017 hw that they've refreshed a couple of times with minor changes. Wait for the actual shield next for proper support.
AV1 decode works well enough even on quite old devices, and it has more favorable quality and size, individually or when considering both. I prefer it to h264 (or vp9 or even h265) at this point, and there isn't a particular need for hardware decode to benefit from it. On Youtube (probably the biggest and most accessible source of av1 video), when comparing size and quality on different codecs (h264/vp9/av1), av1 is either gonna have much smaller size while keeping some level of quality, or have better quality at similar size, or even have both be better.
Firesticks are quite popular. I'd like to see what percentage in the wild actually does decode AV1 in hardware, and without stuttering or buffering problems. Even my newer devices sometimes choke on h265.
The first Apple chips with AV1 support only came out in 2023. So the majority of Apple hardware out there doesn't support it. It will be a long time till support is at >95%.
At any point in time there's only a couple video codecs you should be using. When we can repeatedly shave off 30% of video bandwidth, there's no reason to stop making one or two new ones every decade.
Edit: And you can also look at the video codec landscape as having two major standards that put out new versions sometimes. The MPEG/h.26x series and the VPx/AVx series. It's not quite that simple but it's close.
There is no need to have a final one. It's no problem to keep supporting older codecs and add new ones in new products, as long as there's a "common ground" of widely supported codecs.
I don’t think we’re far off a “final” video codec, in the sense that jpg and mp3 and zip are effectively “final” image and sound and data codecs. Sure, there are newer codecs with higher compression foot a given quality level, but at some point it’s good enough to not bother upgrading, especially when the new version costs exponentially more for negligible gain.
…maybe THIS one will be The Final Codec… doodoodoo dooo
jpg: webp, avif, jxl. The first two are adopted because of non-negligible gains, the latter because of features missing in jpg.
mp3: Both AAC and opus have non-negligible gains and both are fairly widely used.
zip[1]: lz4, zstd, and lzma2 dominate at least gzip's implementation of deflate: For any given setting of gzip, there is a setting for at least one of those three that is both faster and smaller.
1: Not a codec; the zip format specifies a dozen methods for encoding. I'm assuming you mean Deflate, which is the most common compression method associated with zip.
> jpg and mp3 and zip are effectively “final” image and sound and data codecs
In popular consciousness, maybe. When people think image they thing jpg. But most websites you use aren't delivering images as jpg. Most sites care about saving money on bandwidth and their site loading fast, and because of that they tend to use webp and avif, which are much smaller for the same quality.
Re: mp3, Opus is a pretty big improvement. In my experience I've found that Opus is pretty reliably transparent at 96kbit/s, whereas LAME (which really squeezes mp3 for all it's got) needs ~130kbit/s
I think encode costs are higher (although I'm struggling to find any actual data) but in my experience this is totally negligible in practice.
Of course, don't go converting all your mp3s to Opus, but if you're running a music streaming server or whatever, Opus is pretty sweet... and probably close to "as good as it gets" until/unless we figure out some crazy new tricks.
Yes, "neural" or "learned" video codecs are considered the future. They already have competitive compression ratios, but last thing I heard was that they are still much slower than conventional codecs.
One uncommonly used feature of H.264 is support for lossless video. It's a bit hard to use due to tooling (not easy to discover and select the feature), and is sometimes incompatible with web browsers, but you can get actual lossless video using a mainstream video codec. You can verify that it really is lossless, and doesn't have any conversion step where it's quantized to 8-bit YUV.
"x264vfw" is one tool which supports this format, even inside of an AVI file. So older software that uses Video For Windows can create those files.
FYI, both VP9 and AV1 support lossless as part of their base profiles, so lossless files using those codecs should be compatible with all implementations (at least in theory). Though those profiles only support 4:2:0, so it depends on your use case if that really counts as lossless.
I struggle to think of a situation where the blur of chroma subsampling is acceptable but very high quality or prores isn't good enough, I need lossless.
There's a distinction to be made between lossless (compression) and lossless (bit depth/gamma/yadda yadda); typically the format conversion isn't considered part of the "lossy/lossless" distinction.
Definitely not something to be forgotten if you're expecting to compress lossless RGB, although this is pretty rare outside of niche-ish use cases (afaik game capture is basically the only place where people are likely to be using RGB and not YCbCr? could be wrong!)
Also (if you round correctly) there is no generation loss in the format conversion. You can round-trip between them as many times as you want and you only get the loss of the initial RGB->YCbCr.
I've attempted to get researchers to use lossless H.264 to encode CT scans. They typically use directories full of tiffs. Even with intra-slice compression (i.e. tiff compression) they are pretty huge. But H.264 will let you recover the original images to the individual bit, and save a huge amount of space thanks to inter-slice similarities.
I’d imagine the main drawback of using H.264 here is that you’d still have to decode to get the frames back. This is similar to how even though many consumer grade cameras will output natively to h.264, if you want to edit the videos from it you’d usually have to convert it to prores or something first (the editor would usually do this automatically behind the scenes)
I’m not sure if lossless h.264 is I only or not, but if it uses inter-frame compression then it’s difficult to randomly access arbitrary frames. For any arbitrary frame you’d have to find the nearest key frame, then interpolate from there based on the inter-frame compression metadata.
It's acceptable to use the nearest key frame when randomly accessing arbitrary frames.
The real problem is that you'd have to rewrite the entire app stack to use H.264 or AV1 for storing the image stack generated by CT.
CT images are 16-bit monochrome, and the pixel values have real physical meaning (Hounsfield Units). So you'd have to prove that your compression tool can reproduce the original images 1:1. The H.264 encoder would also need to be modified to support 16-bit pixels.
And you'd need to modify the post-processing tools to accept your new file format. Otherwise users will still have to convert it back to DICOM or some other commonly accepted format before they can use their post-processing software.
So unless you're a CT vendor, you'll have a hard time pushing this into a hospital.
Lots of hardware requires specifically h.264 as a transit; many of the MPEG standards are built around it. So this lets you get MPEG-PS, or lossless BDs, or some hardware/specialized software decoders for machines that otherwise couldn’t keep up (like set top boxes).
~370 MB/sec of raw data.[...]This is next to impossible to deal with.[...]You can't move it anywhere fast. Even SSDs have trouble dumping this straight from RAM to Disk
Always interesting to see what happened in ten years --- 370MB/s is well below what the slowest NVMe SSD can do today, and easily achieved over a 5Gbps Ethernet connection.
On average, yes. But without hiccups: only very few NVMe drives can do this over really long runs without stalls. I've been working a lot with high rate SDR earlier this year and it took a lot of testing of different drives to find one that did not have this issue. I was quite surprised because you'd think that the underlying mechanism, if it works at all should be able to work sustainably but once every 10 minutes or so there would be a (brief) stall which would cause the SDR to start dropping data because it wasn't serviced fast enough. The only one that didn't do this was the 'Lexar SSD NM1090 PRO 4TB', which was surprising because it was one of the cheaper ones that I tried.
I really like the idea that compression is intelligence, there are so many tricks an interesting ideas related to it. A bunch of them are described in the article, but you can imagine a long tail of many more inscrutable tricks chained together to get really incredible shrinkage in data.
Hutter prize (I think mentioned in a 3b1b video somewhere else in this thread) is a great example. Given this was written before advent of LLM, I would love to see someone throw openAI/Anthropic levels of compute at a similar problem to see what the swarm can produce.
I remember reading an article on here by one of the founders of MPEG about why he was leaving MPEG after several decades, and part of it was that modern AI is indistinguishable from compression and he wanted to explore new compression algorithms using the recent explosion of knowledge
The newest submissions to Hutter Prize made crazy jumps by just replacing the LSTM with a transformer and training it.
35'000$ for a prompt and some training is a nice chunk of cash.
Next steps are replacing the encoders and all the other human-biased compression parts by just raw data and training, probably another 20-30% is up for grabs.
H.264 High Profile is close to Patent Free. [1] I often wonder how much more efficiency could we get if we were to increase block size outside of spec.
WebP stands on its own merits but it's profoundly annoying that as of Pop OS 22.04 I still have to open them up in Firefox instead of Eye of Gnome. Compatibility issues like this are littered everywhere. This laziness on the part of supporting WebP is a significant driver of continued use of PNG.
Yes, it's very disheartening. We're approaching two decades of WebP, after all. At this point it's not even worth implementing its support because JXL is superior in every use case.
Not really, lossless WEBP still decodes much faster than lossless JXL, so if you need to decode the images faster, you use WEBP. Yes, you could also speculatively decode the JXL images to hide the decode time.
For lossy, neither is in a good place, since lossy AVIF beats both lossy JXL and lossy WEBP. Lossless AVIF is a joke and barely competes with PNG.
JXL decoding can still be optimized and had significantly less time for it, so let's not be so sure if that statement holds. Last update brought much improvement at just the area you mention.
And AVIF compression ratio beats JXL only conditionally. E.g. my photo library had AVIF at 29% and JXL at 25% the input size at the same perceptual quality (SSIM2=83). Sometimes it's on par. Sometimes AVIF wins. JXL wins almost always. Not to mention you can losslessly, reversibly transcode lossy JPG files to JXL, getting significant size reduction for free.
I think the case for JPEG XL is not that it's the best at everything, but that it is both quite good overall while also supporting basically every obscure image format feature under the sun. It's the potential USB of image formats.
> while also supporting basically every obscure image format feature under the sun
Except for gain maps :)
I don't really mind because I don't use them anyway. I prefer to use the client's tonemapper. But they are a useful feature that's not supported by (chrome's implementation of) jxl
Open a bug report on Eye of Gnome. WebP has no issues opening on all of the native MacOS tooling. Since this is an open source program it should be fairly easy to submit a PR to bring in webP support.
With JPEG XL finally being allowed into mainstream web browsers, we’ll hopefully see more native platform support and be able to slowly phase every older codec out, since JPEG XL does very well at more or less everything (except for animation, compared to video codecs).
I hope so, but I disagree with the size of the "except": JXL performs better than APNG and GIF, and still supports transparency and lossless encode. Video codecs struggle here. So even with animation it still depends!
There's also a case for AVIF in some use cases (for this image it performs poorly). But if I had to choose one, I'd choose JXL.
WebP is limited to 16383x16383. I've run into this limit a handful of times and I would expect it to become more of an issue as screen pixel densities increase.
This reminds me that I still haven't watched `part 2` of a very nice video from 3Blue1Brown - "Compression is Intelligence". YouTube suggestions, definitely failed me with this one.
In the end, YouTube still loses frames above 1080p, even if you have fiber-optic internet, the ultimate CPU and GPU, and loads of RAM, just because YouTube's code is garbage.
I've shared this article many times. I work in space and the vast majority of the time, new people insist that all images should be raw. After all, you go to great expense to get somewhere in the name of science, you don't want to discard any information. And that's true in some cases but typically not for a whole mission or even a whole image if you can crop a specific part. Usually 100 or 1000 images instead of one is more valuable even if the compression is lossy.
That's just image compression though. Very few space-grade cameras support video compression. In a lot of cases it works well even though you think you're after a series of still images.
It’s mostly mission dependent as you mentioned, if the value of some images are different than others, compressing several pictures and selecting the best to get in raw is a good strategy, and you can also crop them to get interesting areas as mentioned.
It can also depends on the downlink bandwidth available and power/energy budget, so the optimal strategy will differ.
It can be misleading at times though, for example some interesting feature may happen to be just a compression artifact. A cloud looking like some strange human-made object for example, happened to us one time.
I work with video sites and I'd never understood why an hour-long video could be so small. Saving only what changes between frames explained it all. It also explains why videos of rain or confetti look pixelated.
It is time to make a standard neural network for video, which guesses the next frame and/or applies super-resolution like DLSS does, so the necessary information to transmit is reduced to the minimum necessary for DLSS to reproduce the image without loss.
I remember reading this article when it was new, and I left fully agreeing, but more broadly: modern video compression is magic.
Like, it's baffling to me how much we can compress high-quality video nowadays. HEVC and AV1 and VP9 are all also very cool and they still sort of break my brain. I understand how they work but it doesn't feel like it should compress as well as it does.
I've done AV1 encodes where I was convinced the conversion must have failed in some way for it to be that small, scrubbed through it and it all looks fine, watched through the entire thing to confirm that wasn't the case, just waiting for the shoe to drop, but then to find out, no, it's really just that good.
I wish the video presented had been one with, say, 5x higher bitrate, as it would have sufficed to make the point without presenting a video with very glaring compression artifacts.
I'm surprised that 10 years later, the article still teaches an incorrect conceptual understanding of "quantization".
The context heavily implies that quantization is "discarding information" generally (e.g. clipping, masking). The wikipedia article he links in the footnote disagrees, and so do I:
> quantization is the process of mapping input values from a large set (often a continuous set) to output values in a (countable) smaller set, often with a finite number of elements.
Quantization is a function over values, or the image itself. Clipping and masking, by contrast, are functions over the space/frequency dimensions, i.e. the pixel indices ("preimage"). A rule like "apply a circular mask" or "drop the first quadrant" does not depend on the values!
The most extreme example of quantization is a threshold filter. Regardless of location, output "1" if the pixel is brighter than X, or "0" otherwise. This depends only on the value, not on the location. He also misses the opportunity to point out that chroma subsampling is another application of quantization, this time to chroma values (drop the low-order bits).
Codecs like H.264 and JPEG use spatially-varying quantization rules to effectively discard more information at high frequencies, so his explanation of H.264 is spot on, but a non-expert encountering quantization for the first time would probably come away with the wrong idea. For an article explicitly targeted to non-experts that goes out of its way to introduce the term, it's a pretty important oversight.
Consider me confused. Isn’t quantizing a value, e.g. reducing the number of bits in an int, a form of discarding information from that value? Isn’t your comparison just about discarding information on another level (discarding entire values)?
I think the GP's point is, if you don't know what quantization is, the article gives the impression it consists of discarding some of the values, which is where the compression comes from. This is a good "0-th order" intuition, but quantization is a softening of this; instead of fully retaining or fully discarding a value, there is a sliding scale of how much information to discard. If you already know what quantization is, it's easy to see how discarding is a special case, but the article gives the impression discarding = quantization.
Exactly correct. Quantizing is like rounding. This is the same notion as in quantum mechanics: the values aren’t any real number (a huge input space), but rather integer multiples of some constant (a small output space). Quantization is “rounding” to the nearest allowed output.
To build on the example used in the article, you could quantize an image by dropping the lower half of the bits of every pixel value. This discards half the information by mapping the larger input image space (e.g. 8 bits per pixel) to a smaller output image space (e.g. 4 bits per pixel), effectively scaling/rounding the original values.
The fact that higher frequencies can be quantized more intensely with relatively little image quality loss is a completely separate idea from quantization itself. In this context, high frequency masking as described in the article could be accurately described as a spatially dependent quantization to zero-bit output values. But the quantization part is the mapping, not the shape of the mask, and I think the article is misleading about the distinction.
In real codecs, there usually isn’t a hard boundary like a mask, but instead an empirical table of how many bits to keep based on the perceptual value of the information at each frequency.
The definition you quoted explains why quantization discards information: multiple distinct input values map to the same output value. Once that happens, you can no longer determine the original input from the output alone. The distinctions between those inputs are the information that has been lost.
I’m not saying that quantization doesn’t lose information. Instead, I’m saying that every step to lose information is not quantization. Functions on the preimage (clipping, masking) are not quantization.
Transmission cost of an interpolated frame: 0 with DLSS. But adding a little extra information could help reduce artifacts.
Sure, DLSS uses game engine information, such as motion vectors and depth, which is unavailable for video. But neural networks can estimate motion directly from video frames and interpolate without that information. Providing additional data to fix artifacts would cost little.
There are some interesting advances in neural video encoders and decoders - I suspect it will look a lot like the neural models in DLSS 5 for compression -> decompression/reconstitution. There have been some lossless text-compression neural models built that seem to surpass even high level compression algorithms so with time those could get there.
Well, that was disappointing. Everything in that article is generic to any video compression codec, even MPEG. I thought this would discuss why H.264 compresses so much more than predecessors, but not a word about that. Should have been titled "video compression is magic".
For personal use, if those are your only two choices, then go with H.265 for higher efficiency, or H.264 for wider compatibility. If other codecs are an option, go with AV1 for higher efficiency. It’s currently the best and already pretty widely supported.
The main reason to use H.264 in 2026 is legacy device support. Some devices cannot play H.265, but H.264 is playable almost anywhere, as a 20+ year old codec.
I was surprised that Twitter only lets you upload H.264 still, in 2026.
Twitter's video story is not their strongest. I used to run an account that posted a lot of video content, and their compression quality was so mixed that I ended up writing a little mobile app that would transcode videos to Twitter's advertised optimal format (720p 30fps H.264 5Mbps + AAC LC) before uploading them.
Even then, it was a crapshoot whether or not the video would look like a pixel collage, and I'd need to delete the Tweet, tweak the video a bit, and try again.
It depends on how you want to play or share the media.
E.g. I remember having lots of issues sharing smaller video files encoded with H.265 via both WhatsApp and Messenger. iMessage doesn't support H.265 (or at least it didn't work for me at the time).
Patents aside, h265 from my reading is almost universally better than h264. My understanding which could be wrong is that for a given quality h265 will always be smaller, and particularly for >1080p it's recommended. Hardware support has been around since at least the Shield days (2019) so that shouldn't be an issue. Only use h264 if you are needing to hw-decode on really old hardware.
AV1 is the future/present: Fully open standard, better than h265. Only downside is hardware support: You need hardware made in the last few years.
"The H.264 video, 300 frames long is 175KB. A single frame of that video in PNG is 1015KB. It looks like we're storing 300 times the amount of data in the video. But the file size is a fifth. So H.264 would seem to be 1500x as efficient as PNG. How is this even possible? All right, what's the trick?"
Mostly the trick is they're comparing lossy to lossless compression which doesn't even make any sense.
I don’t think they are confused either just don’t think it’s a good way to introduce the point.
When good expositors choose motivating questions, say like 3 blue 1 brown, they don’t usually make apples to oranges comparisons. The best examples motivate while making sense before and after you learn.
Sorry, but it's only apple to oranges if you already know the answer. That answer, explained in the next section, is very clearly intended for someone who didn't already know it:
> This concept of throwing away bits you don't need to save space is called lossy compression. H.264 is a lossy codec - it throws away less important bits and only keeps the important bits.
> PNG is a lossless codec. It means that nothing is thrown away. Bit for bit, the original source image can be recovered from a PNG encoded image.
Actually I had the same thought, but this is obviously an article for people who don't know the difference between lossless and lossy compression. Ok, you might argue that it's not a good fit for HN, but going by the many times it seems to have been posted here, some people disagree.
It makes tons of sense to compare those numbers. It's a big part of how you benchmark lossy compression. I guess in some sense they should use BMP instead of PNG but it's arguable.
Sure that’s true. But if something makes sense for a technical benchmark that’s not a guarantee it’s the best way to introduce a concept. I just think other approaches could have been more clear for this case.
Had the pleasure of working with Jason Garrett-Glaser (better known as DarkShikari) circa 2011 on a patch for x264 to better utilize more cores (slice-threaded lookahead). At the time he was working with Sony to optimize their PlayStation game streaming service, and this had the potential to lower encoding latency (critical to game streaming) even on same-core systems by spreading the frame over more cores to do more parallel work, and doing more intelligent scheduling and convergence after that tile-based work was reconverged into a frame. At the time we were using servers with 48 physical cores to do transcoding, a bit of a rarity in a single NUMA space back then and there were no instances to test this on so I flamegraph traced a few encodes at very high thread parallel counts and send back the instrumented profiles (from a custom compiled version) showing the core work being done and he used that to optimize the motion estimation code which worked on the first shot to perfectly level out cpu usage and increase encoding speed across ~24 cores (was previously topping out around 8 and running one core hot iirc). I got a look into how exotic and complex this project was, particularly in terms how low level and advanced things like motion estimation, quantizing and perceptual compression were... it was and still is sort of magic how many smart techniques are baked into something that we barely pay attention to. They were, even at that time, working on x265.
His blog, Diary of an x264 developer, was a great read too for anyone wanting to get a vantage point on how crazy that codebase was.
[1]https://github.com/mstorsjo/x264/blob/master/doc/threads.txt
(edit: was Gaikai that was acquired by Sony in 2012 and is the basis for PlayStation Now)
Using multiple CPUs for encoding goes way back. The early (1994) C-Cube MPEG-1 and MPEG-2 real-time encoders for DirecTV were built with 8 chips running in parallel (out of necessity since they were only 60 MHz processors) to encode 704x480. Each chip encoded a slice of the image and an auxiliary bus was used to send reconstruction video between chips so that motion vectors could cross chip image boundaries.
Here's a picture of the VME bus development board. It was a two board sandwich to support 10 chips in parallel (only one board pictured).
https://www.w6rz.net/scorpio.jpg
The first real-time HDTV encoders for terrestrial digital television (ATSC) were built by Harris using an IBM encoder chip. It took 6 chips to encode 1920x1080. The IBM chips didn't have an auxiliary bus, so you could see motion vectors limited to chip image boundaries (in this case, 6 tiles arranged like so).
Yeah at this time intel was just releasing their quick sync tech and a few vendors were putting 4 intel cpus on a pcie board and using it for video encoding! Matrox also had a considerable piece of this business doing video compression and editing/compositing using bespoke processors and boards.
Despite being more "power efficient" - both nvidia and intel's hardware solutions left a lot to be desired in terms of final quality for the same bitrate, compared to x264 that seemed to just put out substantially better quality video for the same bitrate even at the same h264 profile levels. There are still lots of x264 design elements, some even verbatim ported, into modern av1 and hevc encoders.
As someone doing a lot of DVD transcoding at the time, I remember being excited for Quick Sync on my new Sandy Bridge build - only to find that, while fast, it had absolutely _horrendous_ quality. I revisited it on my previous main Plex server build, which I think was a Haswell, and it was still unusable for transcoding compared to x264.
It took Nvidia until the encoder they bundled into 20xx series cards and Intel until the 6th or 7th generation of Quick Sync, but their hardware encoders are _finally_ at a point you can get decent quality at reasonable bitrates. Even so, x264 (and x265!) beat the pants off of them if one can spare the time and electricity; they're still that far ahead.
Today I learned about Intel Quick Sync. Thanks. I retired from video compression in 2011, so I've been out of the loop for a long time.
https://en.wikipedia.org/wiki/Intel_Quick_Sync_Video
Very cool. Took a similar approach to optimizing openutv by looking at flamegraphs. UTV is a fork of openrv which is an older release of the commercial RV. Like many things it uses ffmpeg and also native readers and writers for certain formats. Allowed me to get it to 8k@60fps buttery smooth playback with hardware acceleration. If you are still in the video/imaging space would love your feedback on UTV. openutv.com
Ten years later, 2026 magic: AV1
https://engineering.fb.com/2026/06/22/video-engineering/adop...
And soon AV2
https://av2.aomedia.org/
That's all well and good, but let me know when I can buy a used $50 NUC or pi or such that has hw decoding of the same for my TV front ends.
Intel has supported native hardware decoder for AV1 since Tiger Lake. That’s quite old. You could buy a used NUC with Tiger Lake. Even when new a NUC with 1115G4 or 1135G7 isn’t expensive. I found one with damaged WiFi for $20. https://www.ebay.com/itm/366767313411 You can buy it now.
I went to the link to buy a computer (a NUC11TNKi3) with this capability for $20, but all it would let me do is place a bid on it; I cannot buy it now.
It looks like they normally sell for around $200: https://www.ebay.com/sch/i.html?_nkw=NUC11TNKi3&LH_Sold=1
Cheaper to get a long fiber optic HDMI cable and connect your TV to your PC GPU. That's what I do. You get the best uncompressed video (typ 4K 120Hz or 8K 60Hz), zero lag so it can be used for stuff other than video, and don't have to maintain another box
Do you really care if your NUC uses an extra 2 watts to do software decoding?
Lots of media devices that don't have AV1 decoders tend to have issues decoding high resolution AV1 in real time so you either get pauses to buffer or dropped frames.
Yes but if it says "NUC" I doubt you'll have that issue.
And maybe ten more and enough hardware will support it that it will be mainstream.
AV1? It's mainstream already. Lots of hardware supports it and you don't need hardware support for many other cases.
Sorry, but YouTube began thermal throttling on a Firestick the moment they started forcing AV1 regardless of the presence of on-device hardware support.
>forcing AV1 regardless of the presence of on-device hardware support
I'm posting from a machine without AV1 acceleration. I disabled AV1 in Firefox's about:config and Youtube falls back to VP9.
Then that particular model doesn't support it properly. I don't think that changes my point.
If the common hardware customers already have fails when you attempt to use a newer codec (but works fine otherwise), that can't be said to be a mainstream codec yet.
Your particular model is not representative of "the common hardware" as a whole. And it can be mainstream even when a significant fraction of devices can't use it. AV1 is a third of Netflix streams and most YouTube streams, that's more than enough to qualify as mainstream.
It's on most of YouTube streams because Google forces it on devices that lack hardware support, even if it breaks the functionality of the device.
Mainstream support doesn't have to be forced.
Even if half the devices were broken, it would still be more than a third of streams and mainstream.
A lot less than half of devices are broken when playing YouTube, I assure you.
If they stopped forcing it, it would still be the best choice most of the time. It doesn't "have to be" forced to be a major choice. The forcing changes the exact percentage but not the overall picture.
Mainstream? The most popular hardware streaming device (Nvidia shield) does not even support it, so I wouldnt call it mainstream.
Nvidia Shield is hardly the most popular anything. Roku has 43% of the market and Amazon has 35%.
Would be interesting to see if we could segment third-party from TV-embedded (see quote), but the metrics from the article seem to reflect actual consumer usage, as most consumers just use what comes in the TV they buy nowadays.
> This reflects a consolidated approach and eliminates yet another device in the home, and it’s catching on, as just 34% consider streaming media players as their primary viewing device. Meanwhile, 56% said the same for smart TVs.
https://www.cepro.com/news/roku-amazon-lead-streaming-media-...
Interesting you bring up Roku as a main comparison, as only the Roku Ultra device has AV1 support and that is via DASH only:
https://developer.roku.com/dev/docs/media
So if you are saying Roku has 43% of the market, I would consider that as support for my point that AV1 is not that widely supported.
Very fair. Looks like Amazon is a bit better with AV1 support starting in 2021 with the Fire TV Stick 4k Max - 1st Gen[0], with a single absense of it on the seemingly uber-budget friendly $35 "Fire TV Stick HD Alexa Voice Remote (2024)".
That's still a sliver of the market, but the more royalty-free* codecs we use, the better.
0: https://developer.amazon.com/docs/device-specs/device-specif...
*: https://arstechnica.com/gadgets/2026/03/av1s-open-royalty-fr...
> Nvidia Shield is hardly the most popular anything.
This is so wrong..
> From Jellyfin blog: https://jellywatch.app/blog/jellyfin-nvidia-shield-tv-setup-...
>> The NVIDIA Shield TV (Pro and Tube) is widely considered the best Android TV streaming device for self-hosted media servers in 2026. While the Apple TV 4K wins on UI polish and the Onn 4K Pro wins on price, the Shield wins on something most other devices can't match: lossless audio bitstream passthrough. TrueHD, TrueHD Atmos, DTS-HD MA, DTS:X - the Shield passes them all to your AV receiver untouched.
>> Combined with hardware decode for H.264, H.265, VP9, and AV1, plus an 8-year-old Tegra X1+ chip that still gets driver updates, the Shield is a genuine "set it and forget it" Jellyfin client.
Nvidia shield is a device that simply works and passes the audio untouched to my sound receiver and I enjoy it __everyday__.
EDIT: <trying to fix formatting>
the jellyfin quotes are saying "best", which is significantly different from "most popular", which is what judge2020 said.
Nvidia hasn't released a new model of the Shield since 2019. The current Apple TV model, Google TV model, Roku model, Fire TV, and Onn models support AV1.
The Shield is dead though and a long-time abandoned, right?
I remember the Shield going from fantastic to a horrible mess software wise over the span of a single Android generation (maybe 2019 or 2020), constant ARC weirdness, etc.
I would not consider the ancient Nvidia shield the "most popular" streaming device.
You can’t be serious, I bet 9 out of 10 owners of streaming devices haven’t even heard of the Nvidia Shield.
Shield is a 10 year old device at this point. It's essentially the 2017 hw that they've refreshed a couple of times with minor changes. Wait for the actual shield next for proper support.
My $100 NUC has AV1 hardware decoder support.
Maybe a device from 2019 is a bad data point
Niche luxury products aren't exactly mainstream either
I guess it depends on your definition of the words niche and luxury.
av1_nvenc exists therefore it's mainstream :)
Isn't the latest shield hardware 7 year old at this point? Only one year newer than AV1 itself
AV1 decode works well enough even on quite old devices, and it has more favorable quality and size, individually or when considering both. I prefer it to h264 (or vp9 or even h265) at this point, and there isn't a particular need for hardware decode to benefit from it. On Youtube (probably the biggest and most accessible source of av1 video), when comparing size and quality on different codecs (h264/vp9/av1), av1 is either gonna have much smaller size while keeping some level of quality, or have better quality at similar size, or even have both be better.
Firesticks are quite popular. I'd like to see what percentage in the wild actually does decode AV1 in hardware, and without stuttering or buffering problems. Even my newer devices sometimes choke on h265.
I had an iphone 13 and av1 playback was pretty bad, could technically be done with some apps but never worked well. New iphone it works quite well.
The first Apple chips with AV1 support only came out in 2023. So the majority of Apple hardware out there doesn't support it. It will be a long time till support is at >95%.
I’ve found that YouTube just uses it anyway in software decoding. 4k AV1 video seems to use about 10% cpu on my M1.
2018 AV1 demo
https://news.ycombinator.com/item?id=16796127
We need one final video codec.
Otherwise we will have this:
https://xkcd.com/927/
At any point in time there's only a couple video codecs you should be using. When we can repeatedly shave off 30% of video bandwidth, there's no reason to stop making one or two new ones every decade.
Edit: And you can also look at the video codec landscape as having two major standards that put out new versions sometimes. The MPEG/h.26x series and the VPx/AVx series. It's not quite that simple but it's close.
There is no need to have a final one. It's no problem to keep supporting older codecs and add new ones in new products, as long as there's a "common ground" of widely supported codecs.
I don’t think we’re far off a “final” video codec, in the sense that jpg and mp3 and zip are effectively “final” image and sound and data codecs. Sure, there are newer codecs with higher compression foot a given quality level, but at some point it’s good enough to not bother upgrading, especially when the new version costs exponentially more for negligible gain.
…maybe THIS one will be The Final Codec… doodoodoo dooo
jpg: webp, avif, jxl. The first two are adopted because of non-negligible gains, the latter because of features missing in jpg.
mp3: Both AAC and opus have non-negligible gains and both are fairly widely used.
zip[1]: lz4, zstd, and lzma2 dominate at least gzip's implementation of deflate: For any given setting of gzip, there is a setting for at least one of those three that is both faster and smaller.
1: Not a codec; the zip format specifies a dozen methods for encoding. I'm assuming you mean Deflate, which is the most common compression method associated with zip.
> jpg and mp3 and zip are effectively “final” image and sound and data codecs
In popular consciousness, maybe. When people think image they thing jpg. But most websites you use aren't delivering images as jpg. Most sites care about saving money on bandwidth and their site loading fast, and because of that they tend to use webp and avif, which are much smaller for the same quality.
Re: mp3, Opus is a pretty big improvement. In my experience I've found that Opus is pretty reliably transparent at 96kbit/s, whereas LAME (which really squeezes mp3 for all it's got) needs ~130kbit/s
I think encode costs are higher (although I'm struggling to find any actual data) but in my experience this is totally negligible in practice.
Of course, don't go converting all your mp3s to Opus, but if you're running a music streaming server or whatever, Opus is pretty sweet... and probably close to "as good as it gets" until/unless we figure out some crazy new tricks.
I feel as though AI is going to blow these all out of the water soon.
And then we'll get AI-designed hardware codecs for them.
Or the AI will be the codec - intelligence is compression.
Yes, "neural" or "learned" video codecs are considered the future. They already have competitive compression ratios, but last thing I heard was that they are still much slower than conventional codecs.
One uncommonly used feature of H.264 is support for lossless video. It's a bit hard to use due to tooling (not easy to discover and select the feature), and is sometimes incompatible with web browsers, but you can get actual lossless video using a mainstream video codec. You can verify that it really is lossless, and doesn't have any conversion step where it's quantized to 8-bit YUV.
"x264vfw" is one tool which supports this format, even inside of an AVI file. So older software that uses Video For Windows can create those files.
FYI, both VP9 and AV1 support lossless as part of their base profiles, so lossless files using those codecs should be compatible with all implementations (at least in theory). Though those profiles only support 4:2:0, so it depends on your use case if that really counts as lossless.
I struggle to think of a situation where the blur of chroma subsampling is acceptable but very high quality or prores isn't good enough, I need lossless.
4:2:0 is not lossless. 4:4:4 with quantizing down to 8-bit YCbCr isn't lossless either. But 10-bit 4:4:4 is reversible back to 8-bit RGB without loss.
There's a distinction to be made between lossless (compression) and lossless (bit depth/gamma/yadda yadda); typically the format conversion isn't considered part of the "lossy/lossless" distinction.
Definitely not something to be forgotten if you're expecting to compress lossless RGB, although this is pretty rare outside of niche-ish use cases (afaik game capture is basically the only place where people are likely to be using RGB and not YCbCr? could be wrong!)
Also (if you round correctly) there is no generation loss in the format conversion. You can round-trip between them as many times as you want and you only get the loss of the initial RGB->YCbCr.
Animation production (2D or 3D) can also have raw RGB frames.
Ah, I hadn't considered that but that totally makes sense!
x265 supports lossless as well (though I believe it's not part of the H.265 standard). It's what I use for video capture.
What is the purpose of lossless H.264 vs. ProRes or the like?
Is it similar to lossless audio were there's not a ton of optimization you can do?
I've attempted to get researchers to use lossless H.264 to encode CT scans. They typically use directories full of tiffs. Even with intra-slice compression (i.e. tiff compression) they are pretty huge. But H.264 will let you recover the original images to the individual bit, and save a huge amount of space thanks to inter-slice similarities.
They are still using directories of tiff, though.
I’d imagine the main drawback of using H.264 here is that you’d still have to decode to get the frames back. This is similar to how even though many consumer grade cameras will output natively to h.264, if you want to edit the videos from it you’d usually have to convert it to prores or something first (the editor would usually do this automatically behind the scenes)
I’m not sure if lossless h.264 is I only or not, but if it uses inter-frame compression then it’s difficult to randomly access arbitrary frames. For any arbitrary frame you’d have to find the nearest key frame, then interpolate from there based on the inter-frame compression metadata.
It's acceptable to use the nearest key frame when randomly accessing arbitrary frames.
The real problem is that you'd have to rewrite the entire app stack to use H.264 or AV1 for storing the image stack generated by CT.
CT images are 16-bit monochrome, and the pixel values have real physical meaning (Hounsfield Units). So you'd have to prove that your compression tool can reproduce the original images 1:1. The H.264 encoder would also need to be modified to support 16-bit pixels.
And you'd need to modify the post-processing tools to accept your new file format. Otherwise users will still have to convert it back to DICOM or some other commonly accepted format before they can use their post-processing software.
So unless you're a CT vendor, you'll have a hard time pushing this into a hospital.
Lots of hardware requires specifically h.264 as a transit; many of the MPEG standards are built around it. So this lets you get MPEG-PS, or lossless BDs, or some hardware/specialized software decoders for machines that otherwise couldn’t keep up (like set top boxes).
prores is not lossless. It's also an intraframe codec so compression is not great.
It depends what you mean by "not great". Prores's compression is far better than say H.264, if you actually want usable footage for editing.
you can easily record lossless h264 via obs and any ffmpeg-backed player will play it just fine
the days where the OS matters for video playback are long gone
~370 MB/sec of raw data.[...]This is next to impossible to deal with.[...]You can't move it anywhere fast. Even SSDs have trouble dumping this straight from RAM to Disk
Always interesting to see what happened in ten years --- 370MB/s is well below what the slowest NVMe SSD can do today, and easily achieved over a 5Gbps Ethernet connection.
Incidentally, H.264 patents have either expired or will be expiring soon: https://meta.wikimedia.org/wiki/Have_the_patents_for_H.264_M...
On average, yes. But without hiccups: only very few NVMe drives can do this over really long runs without stalls. I've been working a lot with high rate SDR earlier this year and it took a lot of testing of different drives to find one that did not have this issue. I was quite surprised because you'd think that the underlying mechanism, if it works at all should be able to work sustainably but once every 10 minutes or so there would be a (brief) stall which would cause the SDR to start dropping data because it wasn't serviced fast enough. The only one that didn't do this was the 'Lexar SSD NM1090 PRO 4TB', which was surprising because it was one of the cheaper ones that I tried.
I really like the idea that compression is intelligence, there are so many tricks an interesting ideas related to it. A bunch of them are described in the article, but you can imagine a long tail of many more inscrutable tricks chained together to get really incredible shrinkage in data.
Hutter prize (I think mentioned in a 3b1b video somewhere else in this thread) is a great example. Given this was written before advent of LLM, I would love to see someone throw openAI/Anthropic levels of compute at a similar problem to see what the swarm can produce.
https://arxiv.org/html/2404.09937v1
I remember reading an article on here by one of the founders of MPEG about why he was leaving MPEG after several decades, and part of it was that modern AI is indistinguishable from compression and he wanted to explore new compression algorithms using the recent explosion of knowledge
The newest submissions to Hutter Prize made crazy jumps by just replacing the LSTM with a transformer and training it.
35'000$ for a prompt and some training is a nice chunk of cash.
Next steps are replacing the encoders and all the other human-biased compression parts by just raw data and training, probably another 20-30% is up for grabs.
Previous discussions:
4.nov.2016 https://news.ycombinator.com/item?id=12871403 219 comments
17.mar.2022 https://news.ycombinator.com/item?id=30710574 219 comments
24.may.2019 https://news.ycombinator.com/item?id=19997813 180 comments
14.jun.2024 https://news.ycombinator.com/item?id=40681306 205 comments
It's still magic
H.264 High Profile is close to Patent Free. [1] I often wonder how much more efficiency could we get if we were to increase block size outside of spec.
[1] https://meta.wikimedia.org/wiki/Have_the_patents_for_H.264_M...
Worth pointing out that PNG is also just awful and should probably die like BMP died. WebP was already available at the time this article was written:
PNG: 1039993 bytes - baseline
optimized*: 541539
WebP: 398974 - lossless mode, of course.
JPEG XL: 310778 - lossless, quick, and easy.
*after monumental compute effort; still probably some room for improvement but the effort is ridiculous.
WebP stands on its own merits but it's profoundly annoying that as of Pop OS 22.04 I still have to open them up in Firefox instead of Eye of Gnome. Compatibility issues like this are littered everywhere. This laziness on the part of supporting WebP is a significant driver of continued use of PNG.
That's crazy; gwenview (The KDE image viewer) has supported webp for quite some time.
Yes, it's very disheartening. We're approaching two decades of WebP, after all. At this point it's not even worth implementing its support because JXL is superior in every use case.
Not really, lossless WEBP still decodes much faster than lossless JXL, so if you need to decode the images faster, you use WEBP. Yes, you could also speculatively decode the JXL images to hide the decode time.
For lossy, neither is in a good place, since lossy AVIF beats both lossy JXL and lossy WEBP. Lossless AVIF is a joke and barely competes with PNG.
JXL decoding can still be optimized and had significantly less time for it, so let's not be so sure if that statement holds. Last update brought much improvement at just the area you mention.
And AVIF compression ratio beats JXL only conditionally. E.g. my photo library had AVIF at 29% and JXL at 25% the input size at the same perceptual quality (SSIM2=83). Sometimes it's on par. Sometimes AVIF wins. JXL wins almost always. Not to mention you can losslessly, reversibly transcode lossy JPG files to JXL, getting significant size reduction for free.
I think the case for JPEG XL is not that it's the best at everything, but that it is both quite good overall while also supporting basically every obscure image format feature under the sun. It's the potential USB of image formats.
> while also supporting basically every obscure image format feature under the sun
Except for gain maps :)
I don't really mind because I don't use them anyway. I prefer to use the client's tonemapper. But they are a useful feature that's not supported by (chrome's implementation of) jxl
Try this, as it should support animated webp features in eom. =3
sudo apt install eom webp-pixbuf-loader libimlib2-dev libwebp-dev pkg-config libwebp-dev libgdk-pixbuf2.0-dev libgtk-3-dev
Open a bug report on Eye of Gnome. WebP has no issues opening on all of the native MacOS tooling. Since this is an open source program it should be fairly easy to submit a PR to bring in webP support.
Eye of Gnome can open webp. You need webp-pixbuf-loader package installed.
Classic
With JPEG XL finally being allowed into mainstream web browsers, we’ll hopefully see more native platform support and be able to slowly phase every older codec out, since JPEG XL does very well at more or less everything (except for animation, compared to video codecs).
I hope so, but I disagree with the size of the "except": JXL performs better than APNG and GIF, and still supports transparency and lossless encode. Video codecs struggle here. So even with animation it still depends!
There's also a case for AVIF in some use cases (for this image it performs poorly). But if I had to choose one, I'd choose JXL.
WebP is limited to 16383x16383. I've run into this limit a handful of times and I would expect it to become more of an issue as screen pixel densities increase.
For completeness, my background job of the highest imaginable PNG compression yielded optimized: 528313 bytes. Took only several minutes...
This reminds me that I still haven't watched `part 2` of a very nice video from 3Blue1Brown - "Compression is Intelligence". YouTube suggestions, definitely failed me with this one.
In the end, YouTube still loses frames above 1080p, even if you have fiber-optic internet, the ultimate CPU and GPU, and loads of RAM, just because YouTube's code is garbage.
I've shared this article many times. I work in space and the vast majority of the time, new people insist that all images should be raw. After all, you go to great expense to get somewhere in the name of science, you don't want to discard any information. And that's true in some cases but typically not for a whole mission or even a whole image if you can crop a specific part. Usually 100 or 1000 images instead of one is more valuable even if the compression is lossy.
That's just image compression though. Very few space-grade cameras support video compression. In a lot of cases it works well even though you think you're after a series of still images.
It’s mostly mission dependent as you mentioned, if the value of some images are different than others, compressing several pictures and selecting the best to get in raw is a good strategy, and you can also crop them to get interesting areas as mentioned.
It can also depends on the downlink bandwidth available and power/energy budget, so the optimal strategy will differ.
It can be misleading at times though, for example some interesting feature may happen to be just a compression artifact. A cloud looking like some strange human-made object for example, happened to us one time.
I work with video sites and I'd never understood why an hour-long video could be so small. Saving only what changes between frames explained it all. It also explains why videos of rain or confetti look pixelated.
It is time to make a standard neural network for video, which guesses the next frame and/or applies super-resolution like DLSS does, so the necessary information to transmit is reduced to the minimum necessary for DLSS to reproduce the image without loss.
I remember reading this article when it was new, and I left fully agreeing, but more broadly: modern video compression is magic.
Like, it's baffling to me how much we can compress high-quality video nowadays. HEVC and AV1 and VP9 are all also very cool and they still sort of break my brain. I understand how they work but it doesn't feel like it should compress as well as it does.
I've done AV1 encodes where I was convinced the conversion must have failed in some way for it to be that small, scrubbed through it and it all looks fine, watched through the entire thing to confirm that wasn't the case, just waiting for the shoe to drop, but then to find out, no, it's really just that good.
One year left before the "last major patent" expires.
Wait is this true? Will I finally be able to use h264 on da Vinci resolve for free?
I am right now. Though perhaps not with GPU acceleration.
I wish the video presented had been one with, say, 5x higher bitrate, as it would have sufficed to make the point without presenting a video with very glaring compression artifacts.
I'm surprised that 10 years later, the article still teaches an incorrect conceptual understanding of "quantization".
The context heavily implies that quantization is "discarding information" generally (e.g. clipping, masking). The wikipedia article he links in the footnote disagrees, and so do I:
> quantization is the process of mapping input values from a large set (often a continuous set) to output values in a (countable) smaller set, often with a finite number of elements.
Quantization is a function over values, or the image itself. Clipping and masking, by contrast, are functions over the space/frequency dimensions, i.e. the pixel indices ("preimage"). A rule like "apply a circular mask" or "drop the first quadrant" does not depend on the values!
The most extreme example of quantization is a threshold filter. Regardless of location, output "1" if the pixel is brighter than X, or "0" otherwise. This depends only on the value, not on the location. He also misses the opportunity to point out that chroma subsampling is another application of quantization, this time to chroma values (drop the low-order bits).
Codecs like H.264 and JPEG use spatially-varying quantization rules to effectively discard more information at high frequencies, so his explanation of H.264 is spot on, but a non-expert encountering quantization for the first time would probably come away with the wrong idea. For an article explicitly targeted to non-experts that goes out of its way to introduce the term, it's a pretty important oversight.
Consider me confused. Isn’t quantizing a value, e.g. reducing the number of bits in an int, a form of discarding information from that value? Isn’t your comparison just about discarding information on another level (discarding entire values)?
I think the GP's point is, if you don't know what quantization is, the article gives the impression it consists of discarding some of the values, which is where the compression comes from. This is a good "0-th order" intuition, but quantization is a softening of this; instead of fully retaining or fully discarding a value, there is a sliding scale of how much information to discard. If you already know what quantization is, it's easy to see how discarding is a special case, but the article gives the impression discarding = quantization.
Exactly correct. Quantizing is like rounding. This is the same notion as in quantum mechanics: the values aren’t any real number (a huge input space), but rather integer multiples of some constant (a small output space). Quantization is “rounding” to the nearest allowed output.
To build on the example used in the article, you could quantize an image by dropping the lower half of the bits of every pixel value. This discards half the information by mapping the larger input image space (e.g. 8 bits per pixel) to a smaller output image space (e.g. 4 bits per pixel), effectively scaling/rounding the original values.
The fact that higher frequencies can be quantized more intensely with relatively little image quality loss is a completely separate idea from quantization itself. In this context, high frequency masking as described in the article could be accurately described as a spatially dependent quantization to zero-bit output values. But the quantization part is the mapping, not the shape of the mask, and I think the article is misleading about the distinction.
In real codecs, there usually isn’t a hard boundary like a mask, but instead an empirical table of how many bits to keep based on the perceptual value of the information at each frequency.
The definition you quoted explains why quantization discards information: multiple distinct input values map to the same output value. Once that happens, you can no longer determine the original input from the output alone. The distinctions between those inputs are the information that has been lost.
I’m not saying that quantization doesn’t lose information. Instead, I’m saying that every step to lose information is not quantization. Functions on the preimage (clipping, masking) are not quantization.
This reminds me of that old nursery rhyme. Billy was a chemist's son, but Billy is no more, what he thought was H2O was H.264
It is tradition to re-read this every year. Along with the m1 blackmagic one.
Ref.:
https://news.ycombinator.com/item?id=38099633
I wonder if an AI can make a better codec, and if we would think of it as "magic".
Transmission cost of an interpolated frame: 0 with DLSS. But adding a little extra information could help reduce artifacts.
Sure, DLSS uses game engine information, such as motion vectors and depth, which is unavailable for video. But neural networks can estimate motion directly from video frames and interpolate without that information. Providing additional data to fix artifacts would cost little.
There are some interesting advances in neural video encoders and decoders - I suspect it will look a lot like the neural models in DLSS 5 for compression -> decompression/reconstitution. There have been some lossless text-compression neural models built that seem to surpass even high level compression algorithms so with time those could get there.
The fact that you said 5 worries me.
Socrates was certainly a pioneer of conveying information efficiently
Well, that was disappointing. Everything in that article is generic to any video compression codec, even MPEG. I thought this would discuss why H.264 compresses so much more than predecessors, but not a word about that. Should have been titled "video compression is magic".
Article is dated (2016).
The only question I have:
Should I use H.264 or H.265?
For personal use, if those are your only two choices, then go with H.265 for higher efficiency, or H.264 for wider compatibility. If other codecs are an option, go with AV1 for higher efficiency. It’s currently the best and already pretty widely supported.
The main reason to use H.264 in 2026 is legacy device support. Some devices cannot play H.265, but H.264 is playable almost anywhere, as a 20+ year old codec.
I was surprised that Twitter only lets you upload H.264 still, in 2026.
Twitter's video story is not their strongest. I used to run an account that posted a lot of video content, and their compression quality was so mixed that I ended up writing a little mobile app that would transcode videos to Twitter's advertised optimal format (720p 30fps H.264 5Mbps + AAC LC) before uploading them.
Even then, it was a crapshoot whether or not the video would look like a pixel collage, and I'd need to delete the Tweet, tweak the video a bit, and try again.
It depends on how you want to play or share the media.
E.g. I remember having lots of issues sharing smaller video files encoded with H.265 via both WhatsApp and Messenger. iMessage doesn't support H.265 (or at least it didn't work for me at the time).
H.264 (if you want to support 20 year old devices) or av1. Don't use h.265. The patent situation around it is a total mess.
h264, if you had a reason to use something else you wouldn't need to ask
That all depends on your use case.
How are you generating video?
How are you playing it back?
H.264 for 1080p and less. H.265 for 2160p.
Patents aside, h265 from my reading is almost universally better than h264. My understanding which could be wrong is that for a given quality h265 will always be smaller, and particularly for >1080p it's recommended. Hardware support has been around since at least the Shield days (2019) so that shouldn't be an issue. Only use h264 if you are needing to hw-decode on really old hardware.
AV1 is the future/present: Fully open standard, better than h265. Only downside is hardware support: You need hardware made in the last few years.
that's like asking "should I use an audi or a tesla"
"The H.264 video, 300 frames long is 175KB. A single frame of that video in PNG is 1015KB. It looks like we're storing 300 times the amount of data in the video. But the file size is a fifth. So H.264 would seem to be 1500x as efficient as PNG. How is this even possible? All right, what's the trick?"
Mostly the trick is they're comparing lossy to lossless compression which doesn't even make any sense.
Yes, as they explain shortly after, in the next section "shedding weight".
They're not actually confused here. It's a writing style, where you pose an interesting problem/scenario, then explain it.
I don’t think they are confused either just don’t think it’s a good way to introduce the point.
When good expositors choose motivating questions, say like 3 blue 1 brown, they don’t usually make apples to oranges comparisons. The best examples motivate while making sense before and after you learn.
Sorry, but it's only apple to oranges if you already know the answer. That answer, explained in the next section, is very clearly intended for someone who didn't already know it:
> This concept of throwing away bits you don't need to save space is called lossy compression. H.264 is a lossy codec - it throws away less important bits and only keeps the important bits.
> PNG is a lossless codec. It means that nothing is thrown away. Bit for bit, the original source image can be recovered from a PNG encoded image.
Actually I had the same thought, but this is obviously an article for people who don't know the difference between lossless and lossy compression. Ok, you might argue that it's not a good fit for HN, but going by the many times it seems to have been posted here, some people disagree.
It makes tons of sense to compare those numbers. It's a big part of how you benchmark lossy compression. I guess in some sense they should use BMP instead of PNG but it's arguable.
Sure that’s true. But if something makes sense for a technical benchmark that’s not a guarantee it’s the best way to introduce a concept. I just think other approaches could have been more clear for this case.
Discussed elsewhere here: h.264 supports lossless storing.