Researchers Zekun Zhao and team introduced Parallel Prompting, a way to answer many questions about the same long text in one go. It treats each question as a separate stream within a single request, so the model reads the document only once per step. Implemented in the SGLang system with a few dozen lines of code, the technique runs on CodeLlama‑7B and shows large speed gains. On an Nvidia A100 GPU it is 3.7 × faster, and on an RTX A6000 it is about 6 × faster than the standard SGLang setup.
Why it matters
It lets developers and data‑labeling teams process many queries per document much quicker, reducing GPU costs.
