This was my ~weekend project. It's a PEX88096 switch card in the CPU x16 slot of an AM4 motherboard, running four R9700s and two 3090s at x8 each. A switch was really the only way I could see to hang this many GPUs off a consumer board.
My favorite result: a P2P copy across the four R9700s moved 54.7 GB/s in total, about twice the ~28 GB/s the x16 uplink carries one way, so that traffic never leaves the switch.
The kernel was less cooperative. Current kernels (7.0.6 on this box at the moment) size the switch's bridge window too small for four 32 GB BARs, so two of the R9700s just came up without them. I ended up writing a small module that lays out the BARs before amdgpu loads to get around it.
It was a very fun adventure and I have a few more posts coming later this week about some rabbit holes I fell into along the way.
This was my ~weekend project. It's a PEX88096 switch card in the CPU x16 slot of an AM4 motherboard, running four R9700s and two 3090s at x8 each. A switch was really the only way I could see to hang this many GPUs off a consumer board.
My favorite result: a P2P copy across the four R9700s moved 54.7 GB/s in total, about twice the ~28 GB/s the x16 uplink carries one way, so that traffic never leaves the switch.
The kernel was less cooperative. Current kernels (7.0.6 on this box at the moment) size the switch's bridge window too small for four 32 GB BARs, so two of the R9700s just came up without them. I ended up writing a small module that lays out the BARs before amdgpu loads to get around it.
It was a very fun adventure and I have a few more posts coming later this week about some rabbit holes I fell into along the way.