AWS SageMaker HyperPod introduces disaggregated prefill and decode for LLM inference

Separates prefill and decode phases onto different GPU pools via EFA RDMA to eliminate interference. Improves throughput by up to 40% for long prompts and concurrent requests.