Advocate, Skeptic and Judge agents score essays without task-specific training.
Automated essay scoring usually depends on supervised models trained for one prompt and one rubric, and a single LLM asked to grade an essay tends to drift toward the middle of the scale and gives little explanation for its score. This line of work, which I lead as first author in the Digital Learning Lab at UC Irvine, asks whether structured interaction between several LLM agents can produce trait-level scores that are both more reliable and easier to inspect.
MADEST (2025) compared three multi-agent designs for trait-level essay scoring: a supervisor that delegates sub-traits to rater agents, rater–reviewer collaboration, and debate between an Advocate (strengths), a Skeptic (weaknesses) and a Judge, with first-token log-probabilities used as a confidence signal when the Judge weighs the two sides (Keramati & Warschauer, 2025). In that evaluation the debate design outperformed the other two, and its agreement with human raters was on a par with the agreement between the two human raters. I presented the comparison as a contributed talk in the NeurIPS 2025 Education Program (Keramati & Warschauer, 2025).
MADRAG (NLP4DH 2026, co-located with ACL) adds retrieval-augmented grounding: before deciding, the Judge compares the essay with retrieved, rubric-aligned scored exemplars (Keramati et al., 2026). On ASAP essay sets 7 and 8, MADRAG substantially outperforms prompt-based LLM baselines and approaches supervised systems without any fine-tuning. Ablations show that retrieval drives the calibration gains, while debate mainly improves higher-level traits such as Ideas and Organization. A qualitative error analysis in the paper catalogues the remaining failure modes, for example a Judge captured by one debater’s framing, which motivated the confidence-diagnostics work described in the next project.
The debate approach also informs the Multi-Agent Review feature of PapyrusAI, the lab’s AI writing coach, and the scorer’s evaluation is preregistered on the Open Science Framework (osf.io/7qpwg). MADRAG was designed and evaluated with Shiyuan Zhou, Sharad Mehrotra and Mark Warschauer.
The MADRAG pipeline: a supervisor assigns rubric traits to debating teams; for each trait an Advocate and a Skeptic argue, and a Judge grounded in retrieved scored essays issues the score (figure from the NLP4DH 2026 paper).
Presents MADRAG, a training-free analytic essay scoring framework that combines multi-agent debate with retrieval-augmented exemplar grounding. The system improves calibration, reduces middle-score bias, and achieves strong trait-level scoring performance that is competitive with supervised essay scoring approaches.
@inproceedings{keramati2026madrag,title={{MADRAG}: Multi-Agent Debate with Retrieval-Augmented Generation for Training-Free Analytic Essay Scoring},author={Keramati, Ali and Zhou, Shiyuan and Mehrotra, Sharad and Warschauer, Mark},booktitle={Proceedings of the 6th International Conference on Natural Language Processing for the Digital Humanities},pages={325--345},year={2026},month=jul,publisher={Association for Computational Linguistics},doi={10.18653/v1/2026.nlp4dh-1.30},}
A novel approach to automated essay scoring using multi-agent systems. By leveraging debate between specialized agents, the framework provides a more reliable and nuanced evaluation, surpassing traditional single-agent methods and bringing scoring closer to human-level agreement.
@misc{keramati2025madest,title={{MADEST}: Multi-Agent Debate Essay Scoring Triangulation},author={Keramati, Ali and Warschauer, Mark},howpublished={Zenodo},note={First-year Ph.D. poster},year={2025},month=sep,publisher={Zenodo},doi={10.5281/zenodo.17196206},}
Selected as one of 11 contributed talks in the Education Track of the NeurIPS 2025 Education Program.
Introduces multi-agent system architectures for essay scoring, comparing approaches such as supervisor, collaboration, and debate. The work highlights how multi-agent debate can improve scoring reliability and move automated essay evaluation closer to human-level agreement while making advanced AI concepts more accessible for educators and assessment professionals.
@inproceedings{keramati2025essayscoring,title={Application of Multi-Agent Systems for Essay Scoring},author={Keramati, Ali and Warschauer, Mark},booktitle={NeurIPS 2025 Education Program (Education Track), San Diego},note={Contributed talk},year={2025},month=dec,}
