Packed-twin-inference doubles inference speed on MI50

github.com

Technique achieves 38.1 tokens/s on a single AMD MI50, up from 19.4, by running multiple computations side-by-side. Similar to speculative decoding but exploits unused compute without an extra model.