A team led by Soumya Sharma released a new benchmark, CIMEMORIES DIALOGUE, with 4,410 two‑agent conversations to test privacy leakage. They ran the benchmark on seven open‑weight models from the Qwen‑3 and Llama‑3.1 families. The study shows that models that perform better on tasks also leak more private information, especially when using chain‑of‑thought prompting. Leaks grew up to three times when conversations extended from one turn to six turns.
Why it matters
Enterprises using LLM agents may face higher privacy risks as they adopt more capable models.
