This is not a vector extension (like Intel AVX/AVX-512 or Arm SVE), but a matrix extension (like Intel AMX or Arm SME or the "tensor" operations of NVIDIA GPUs).
Some of the latest generations of Intel server CPUs with P-cores already have the AMX matrix extension, which can be used to implement fast AI inference.
AMD has not implemented AMX yet, and probably they will not implement it, because this new "AI Compute Extension", which has been defined by Intel and AMD together, is an alternative/extension to AMX (ACE inherits some parts of AMX, but not all). It appears that the fate of Intel AMX will be the same as that of the original Apple undocumented AMX extension, which was replaced by the SME extension defined together with the Arm company (like Intel AMX will be replaced by ACE defined together with AMD).
Matrix extensions are more efficient for AI inference than vector extensions, because they reduce the ratio between memory accesses and computation operations.
However, I would like to have not only a matrix extension for AI, but also a matrix extension for all numeric formats until FP64, like in Arm/Apple SME or in the NVIDIA and AMD "datacenter" GPUs.
While ACE inherits a part of AMX, it adds features that are intended to simplify the writing of programs that mix AMX instructions with AVX10 a.k.a. AVX-512 instructions, for operating on the same data.
One thing that stuck out to me is that deals with a lot more data formats, in particular, low-precision formats like FP4, FP6 and FP8. Manipulating those formats can take a lot of annoying effort; in general, x86 (until AVX-512, at least) has unconvincing support for so-called "lane-crossing" instructions that move data across 16-byte boundaries within a vector. So you can imagine unpacking, e.g., tightly packed 7-bit data to 8-bit data is a real slog.
I can already immediately think of a use case for vunpackb in some of the stuff I'm working on, where we'd like to efficiently unpack weights from the high half of a vector.
Separately, adding all signed–unsigned variants of the VNNI dot product instructions is a welcome (albeit niche) change. There was an annoying divergence here between major ISAs: x86 added vpdpbusd which computed a dot product between u8 and i8, while ARM added vdotq, which computes a dot product either between u8 and u8 elements, or i8 and i8. So for broad compatibility, you generally had to restrict one of your inputs to [0,127]. This difference shows in the design of (for example) WASM relaxed SIMD, where the result of wasm.dot.i8x16.i7x16.add.signed is implementation-defined if you exceed the [0,127] range. ARM later added mixed-sign variants, and now x86 consummates it.
The fraction of the installed base of x86 CPUs that support AMX is very small (i.e. only a part of the recent Intel server CPUs support AMX, while the other Intel server CPUs, all Intel consumer CPUs and all AMD CPUs do not support AMX).
CPUs with ACE will in most cases replace CPUs that did not support AMX, so all the registers specified by ACE, but not by AVX10 a.k.a. AVX-512 are new.
It has been available for several years in all the Zen 4 and Zen 5 CPUs and it will be available in the Zen 6 CPUs that should be launched early next year (the launches of both the new Intel and AMD CPUs have been delayed for next year because of the memory price problem, which discourages computer upgrades).
A large fraction of the Intel server CPUs support AVX-512.
All the CPUs that will be launched by Intel from now on will support AVX-512. The CPUs launched by Intel in the first months of this year, i.e. Panther Lake, Wildcat Lake and Clearwater Forest, are the last Intel CPUs without AVX-512 support.
The only market segment where Intel still outsells AMD is in laptop CPUs, especially in corporate laptops, so indeed for now AVX-512 is still not supported in most new laptop and mini-PC CPUs.
If CPU launches are being delayed until next year due to high memory prices, I wish I shared the CPU manufacturers' optimism the memory prices will be lower next year.
Please define new. Also, I think AMD uses very similar cores in server and client. So, disabling AVX512 may be an Intel thing (my guess is that so they can easily move threads between E & P cores).
They didn't disable it at first on their client CPUs, and it resulted in code randomly crashing depending on whether ifunc resolvers first ran on a big core or a little core.
It's pretty surprising that multiple CPU vendors have run into issues like this (some more than once, fucking Samsung), when it's pretty much the first thing that anyone on the toolchain side of thing asks when they hear about heterogenous cores on a CPU.
It would be possible, but Microsoft has already accumulated many decades of being extremely reluctant to make even the slightest changes in Windows in order to accommodate new CPU features.
Since 1990 and until now, the Intel-AMD ISA and the hardware architecture of the Intel-AMD CPUs have accumulated a large number of very undesirable features, which increase the cost of hardware and reduce the security of computers, for which the only justification was the fact that Microsoft refused to update Windows so they expected that the hardware vendors must do whatever it takes to keep their CPUs compatible with the legacy operating systems.
These workarounds have moved inside the CPUs many functions that properly belong to the operating system, like remote management and voltage-frequency control.
And do you know about Linux? After cursory search I couldn't find anything as well. In this case it seems very expensive, if not impossible without the huge penalty, to implement such a workaround in the hardware.
Heterogeneous compute aware kernel scheduler seems like the most natural approach to me.
The kernel can't do anything here, unless your answer is "threads can't ever migrate between types of cores, so cross your fingers on which core you spawn on". The kernel doesn't know what flavor of ISA userspace is expecting, because userspace doesn't know what flavor it wants until it actually gets to run: symbol multiversioning lets you compile a function for multiple versions of an instruction set and pick the best one at runtime.
You could imagine the kernel being clever and handling the first SIGILL in hardware and then marking the process as big-core-only, but you can't handle this generally since you can have new instructions get encoded as nop (I believe the AArch64 PAC instructions do that?). You could also have an ifunc resolver intentionally call the instruction and expect and handle the signal.
The solution is to make your hardware actually work, and not claim to support instructions that do not actually exist on all of your cores.
When an invalid instruction exception occurs, it is very easy to identify whether the invalid instruction was an AVX-512 instruction, because those start with a prefix different from any other instructions.
After that happens, the thread can be marked as requiring AVX-512 and then it can be scheduled only on P-cores.
Moreover, one can define a new flag for the ELF header of executable files, specifying the AVX-512 requirement, which will be set by the compiler, depending on the compilation options, and then the OS knows from the beginning how to schedule a process.
Of course, the best solution is still to implement a uniform ISA. If a lower performance is accepted, the execution of the AVX-512 instructions can be implemented at a negligible additional cost over the previous x86_64 ISA.
What has scared the E-cores designers into not implementing AVX-512 was not the execution part, but the instruction decoding part. The instruction decoder for x86_64 is already horrendously complicated when many instructions must be decoded concurrently in a single clock cycle and adding the need to handle the completely distinct encoding of the AVX-512 makes it even more complicated.
Nonetheless, the competition from AMD has eventually forced Intel to give up in their attempt of reducing production costs by saving die area with a simpler instruction decoder in the E-cores, so the E-cores of Nova Lake will have the same ISA as its P-cores.
> When an invalid instruction exception occurs, it is very easy to identify whether the invalid instruction was an AVX-512 instruction, because those start with a prefix different from any other instructions.
You're missing the point: the processes executing AVX-512 do not have any particularly strong requirement for AVX-512 instructions, they're just optimistically using them to improve performance because CPUID claimed they were supported. It would be pathologically bad behavior to control scheduling behavior with this, because in practice it would result in things like glibc deciding to use AVX-512 for string functions resulting in every program that uses memcpy (i.e. every program) not being allowed to be scheduled on the little cores.
And what happens when E-core decoder encounters the AVX-512 instruction set? Does this mean that E-cores will have to implement some sort of (downsized) vectorized execution engine too? Otherwise, I don't see how this would work. Migrating on the HW core level would be too complicated.
I was fully expecting a set of parody instructions such as TSB (take a step back), SAT (stop and think), TIT (think it through) and the defined but not implemented KAH (kill all humans), alongside a set of interrupts for when it is about to break any of the 3 laws.
So how does this differ from available sse / avx instructions already in most x64 machines?
This is not a vector extension (like Intel AVX/AVX-512 or Arm SVE), but a matrix extension (like Intel AMX or Arm SME or the "tensor" operations of NVIDIA GPUs).
Some of the latest generations of Intel server CPUs with P-cores already have the AMX matrix extension, which can be used to implement fast AI inference.
AMD has not implemented AMX yet, and probably they will not implement it, because this new "AI Compute Extension", which has been defined by Intel and AMD together, is an alternative/extension to AMX (ACE inherits some parts of AMX, but not all). It appears that the fate of Intel AMX will be the same as that of the original Apple undocumented AMX extension, which was replaced by the SME extension defined together with the Arm company (like Intel AMX will be replaced by ACE defined together with AMD).
Matrix extensions are more efficient for AI inference than vector extensions, because they reduce the ratio between memory accesses and computation operations.
However, I would like to have not only a matrix extension for AI, but also a matrix extension for all numeric formats until FP64, like in Arm/Apple SME or in the NVIDIA and AMD "datacenter" GPUs.
ACE is a proliferation of AMX -- palette 2.
So AMD is actually going to implement the basics of AMX -- whether they also do palette 1 is up to them.
While ACE inherits a part of AMX, it adds features that are intended to simplify the writing of programs that mix AMX instructions with AVX10 a.k.a. AVX-512 instructions, for operating on the same data.
One thing that stuck out to me is that deals with a lot more data formats, in particular, low-precision formats like FP4, FP6 and FP8. Manipulating those formats can take a lot of annoying effort; in general, x86 (until AVX-512, at least) has unconvincing support for so-called "lane-crossing" instructions that move data across 16-byte boundaries within a vector. So you can imagine unpacking, e.g., tightly packed 7-bit data to 8-bit data is a real slog.
I can already immediately think of a use case for vunpackb in some of the stuff I'm working on, where we'd like to efficiently unpack weights from the high half of a vector.
Separately, adding all signed–unsigned variants of the VNNI dot product instructions is a welcome (albeit niche) change. There was an annoying divergence here between major ISAs: x86 added vpdpbusd which computed a dot product between u8 and i8, while ARM added vdotq, which computes a dot product either between u8 and u8 elements, or i8 and i8. So for broad compatibility, you generally had to restrict one of your inputs to [0,127]. This difference shows in the design of (for example) WASM relaxed SIMD, where the result of wasm.dot.i8x16.i7x16.add.signed is implementation-defined if you exceed the [0,127] range. ARM later added mixed-sign variants, and now x86 consummates it.
this also adds new registers to operate on (more state) - 1KB more state at least (512b x 16)
It reuses AMX registers, so I think the only new state is the block scale register (1024 bits)?
The fraction of the installed base of x86 CPUs that support AMX is very small (i.e. only a part of the recent Intel server CPUs support AMX, while the other Intel server CPUs, all Intel consumer CPUs and all AMD CPUs do not support AMX).
CPUs with ACE will in most cases replace CPUs that did not support AMX, so all the registers specified by ACE, but not by AVX10 a.k.a. AVX-512 are new.
From the ISA perspective, bonzini is right: only BSR0/SCALADATA is new.
AVX512 isn't available on most new CPUs, I'm guessing ACE will only be available on server CPUs for at least a couple of years at launch?
AVX-512 is available in most new CPUs.
It has been available for several years in all the Zen 4 and Zen 5 CPUs and it will be available in the Zen 6 CPUs that should be launched early next year (the launches of both the new Intel and AMD CPUs have been delayed for next year because of the memory price problem, which discourages computer upgrades).
A large fraction of the Intel server CPUs support AVX-512.
All the CPUs that will be launched by Intel from now on will support AVX-512. The CPUs launched by Intel in the first months of this year, i.e. Panther Lake, Wildcat Lake and Clearwater Forest, are the last Intel CPUs without AVX-512 support.
The only market segment where Intel still outsells AMD is in laptop CPUs, especially in corporate laptops, so indeed for now AVX-512 is still not supported in most new laptop and mini-PC CPUs.
If CPU launches are being delayed until next year due to high memory prices, I wish I shared the CPU manufacturers' optimism the memory prices will be lower next year.
> AVX512 isn't available on most new CPUs
Please define new. Also, I think AMD uses very similar cores in server and client. So, disabling AVX512 may be an Intel thing (my guess is that so they can easily move threads between E & P cores).
They didn't disable it at first on their client CPUs, and it resulted in code randomly crashing depending on whether ifunc resolvers first ran on a big core or a little core.
It's pretty surprising that multiple CPU vendors have run into issues like this (some more than once, fucking Samsung), when it's pretty much the first thing that anyone on the toolchain side of thing asks when they hear about heterogenous cores on a CPU.
Would it not be possible to solve the problem of "thread migration to an incompatible core" on the kernel scheduler level?
It would be possible, but Microsoft has already accumulated many decades of being extremely reluctant to make even the slightest changes in Windows in order to accommodate new CPU features.
Since 1990 and until now, the Intel-AMD ISA and the hardware architecture of the Intel-AMD CPUs have accumulated a large number of very undesirable features, which increase the cost of hardware and reduce the security of computers, for which the only justification was the fact that Microsoft refused to update Windows so they expected that the hardware vendors must do whatever it takes to keep their CPUs compatible with the legacy operating systems.
These workarounds have moved inside the CPUs many functions that properly belong to the operating system, like remote management and voltage-frequency control.
And do you know about Linux? After cursory search I couldn't find anything as well. In this case it seems very expensive, if not impossible without the huge penalty, to implement such a workaround in the hardware.
Heterogeneous compute aware kernel scheduler seems like the most natural approach to me.
The kernel can't do anything here, unless your answer is "threads can't ever migrate between types of cores, so cross your fingers on which core you spawn on". The kernel doesn't know what flavor of ISA userspace is expecting, because userspace doesn't know what flavor it wants until it actually gets to run: symbol multiversioning lets you compile a function for multiple versions of an instruction set and pick the best one at runtime.
You could imagine the kernel being clever and handling the first SIGILL in hardware and then marking the process as big-core-only, but you can't handle this generally since you can have new instructions get encoded as nop (I believe the AArch64 PAC instructions do that?). You could also have an ifunc resolver intentionally call the instruction and expect and handle the signal.
The solution is to make your hardware actually work, and not claim to support instructions that do not actually exist on all of your cores.
When an invalid instruction exception occurs, it is very easy to identify whether the invalid instruction was an AVX-512 instruction, because those start with a prefix different from any other instructions.
After that happens, the thread can be marked as requiring AVX-512 and then it can be scheduled only on P-cores.
Moreover, one can define a new flag for the ELF header of executable files, specifying the AVX-512 requirement, which will be set by the compiler, depending on the compilation options, and then the OS knows from the beginning how to schedule a process.
Of course, the best solution is still to implement a uniform ISA. If a lower performance is accepted, the execution of the AVX-512 instructions can be implemented at a negligible additional cost over the previous x86_64 ISA.
What has scared the E-cores designers into not implementing AVX-512 was not the execution part, but the instruction decoding part. The instruction decoder for x86_64 is already horrendously complicated when many instructions must be decoded concurrently in a single clock cycle and adding the need to handle the completely distinct encoding of the AVX-512 makes it even more complicated.
Nonetheless, the competition from AMD has eventually forced Intel to give up in their attempt of reducing production costs by saving die area with a simpler instruction decoder in the E-cores, so the E-cores of Nova Lake will have the same ISA as its P-cores.
> When an invalid instruction exception occurs, it is very easy to identify whether the invalid instruction was an AVX-512 instruction, because those start with a prefix different from any other instructions.
You're missing the point: the processes executing AVX-512 do not have any particularly strong requirement for AVX-512 instructions, they're just optimistically using them to improve performance because CPUID claimed they were supported. It would be pathologically bad behavior to control scheduling behavior with this, because in practice it would result in things like glibc deciding to use AVX-512 for string functions resulting in every program that uses memcpy (i.e. every program) not being allowed to be scheduled on the little cores.
And what happens when E-core decoder encounters the AVX-512 instruction set? Does this mean that E-cores will have to implement some sort of (downsized) vectorized execution engine too? Otherwise, I don't see how this would work. Migrating on the HW core level would be too complicated.
Thank $ALL_DIETIES that the TCG wasn’t involved!
What's TCG in this context, and why's that good? I'm out of the loop on this.
The Trusted Computing Group.
They make Simple things Complicated.
I was fully expecting a set of parody instructions such as TSB (take a step back), SAT (stop and think), TIT (think it through) and the defined but not implemented KAH (kill all humans), alongside a set of interrupts for when it is about to break any of the 3 laws.