Teacher-in-the-loop scoring in PapyrusAI
Bringing multi-agent essay scoring into a classroom writing coach, with teachers in control.
PapyrusAI is the Digital Learning Lab’s generative-AI writing environment. Students work through planning, drafting, revising and reflecting in guided conversations that start from research-based prompts; instructors decide which prompts are available and can modify them, and the design principle is to support students without doing the work for them. The platform was built by a UCI team with National Science Foundation support and is now being developed and evaluated by the IES-funded WRITE AI Center.
Automated scores only help a classroom if teachers can check and correct them before students see them. In a CSCL 2026 paper I proposed a teacher-in-the-loop multi-agent framework for transparent essay scoring and feedback: agents score each rubric dimension and explain their reasoning, and teachers review, edit and approve scores and feedback before anything is released to students (Keramati & Warschauer, 2026).
Within PapyrusAI and the WRITE AI Center, I work on essay scoring and feedback that builds on the multi-agent debate method from MADRAG (Keramati et al., 2026), and on evaluating the AI models and prompts behind the platform. The public project page describes the feature as Multi-Agent Review: “A multi-agent system provides scoring and feedback on student writing, in alignment with instructor rubrics.”
The scorer’s validation against human ratings is preregistered on the Open Science Framework: the debate-scorer evaluation on public essay sets (osf.io/7qpwg) and, with Tamara Tate, a human–AI agreement study on authentic essays from undergraduate writing-intensive courses (osf.io/5c9nk). I also contributed to a longitudinal study of AI–human score agreement in primary-grade writing, presented at the NCME AIME-Con 2026 conference.
Links: PapyrusAI at the WRITE AI Center · papyrusai.org · WRITE AI news
