Ali Keramati

UC Irvine

akerama1 [at] uci.edu

I am a Ph.D. candidate in Education Data Science at the University of California, Irvine, and a graduate student researcher in the Digital Learning Lab, directed by Mark Warschauer. I hold an M.A. in Education from UC Irvine and a B.Sc. in Computer Engineering, with a minor in Educational Sciences, from the University of Tehran.

My research combines natural language processing and multi-agent AI with educational research to study how digital learning systems can support teaching and learning, focusing on reliable assessment, learner simulation, and the evaluation of AI agents. I designed MADRAG, a training-free essay scorer in which Advocate and Skeptic agents debate an essay and a Judge agent calibrates its score against retrieved, rubric-aligned examples. I study whether token-level confidence reveals reasoning quality in multi-agent debate (ACL 2026 SRW, GEM 2026) and how well large language models simulate students’ programming errors (EDM 2026).

In summer 2026 I was an applied scientist intern at Microsoft Copilot Studio, where I co-developed ThinkingBox-Bench, a benchmark of 507 business workflows that tests whether tool-using agents succeed reliably across repeated attempts. At UC Irvine I work on PapyrusAI, the lab’s AI writing coach, within the IES-funded WRITE AI Center, and I have served as a teaching and learning assistant.

Outside research, I love the arts: painting is my creative outlet, music keeps me focused, and I have explored acting and film direction with Tazeh Theater. I also played soccer for Esteghlal F.C. U17 and still join friends on the field whenever I can.

news

Oct 2026	Co-authored paper on AI–human writing score agreement in primary grades presented in a paper session at NCME AIME-Con 2026 in Pittsburgh.
Sep 2026	Serving as Learning Assistant for EDUC 217 AI and Digital Learning, a graduate course at UC Irvine taught by Prof. Mark Warschauer (Fall 2026).
Aug 2026	Our preprint One Success Isn’t Reliability: ThinkingBox, from my Microsoft internship, is on arXiv (co-first author).
Jul 2026	Three first-author papers at ACL 2026 in San Diego: The Confident Liar (Student Research Workshop), MADRAG (NLP4DH), and Early-Token Confidence (GEM).
Jun 2026	Co-first-authored paper Simulating Students’ Java Programming Errors with Large Language Models published at EDM 2026 in Seoul.

selected publications

ACL SRW
The Confident Liar: Diagnosing Multi-Agent Debate with Log-Probabilities and LLM-as-Judge

Ali Keramati, Justin Cheok, Jacob Horne, and Mark Warschauer

In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), Jul 2026

Abs DOI arXiv Bib PDF

Diagnoses failure modes in multi-agent debate systems by combining token-level log-probability signals with LLM-as-judge evaluation. The work shows how confidence calibration can surface cases where agents are confidently wrong, improving the reliability of multi-agent reasoning for evaluation tasks.
@inproceedings{keramati2026confidentliar, title = {The Confident Liar: Diagnosing Multi-Agent Debate with Log-Probabilities and {LLM}-as-Judge}, author = {Keramati, Ali and Cheok, Justin and Horne, Jacob and Warschauer, Mark}, booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)}, pages = {1361--1375}, year = {2026}, month = jul, publisher = {Association for Computational Linguistics}, doi = {10.18653/v1/2026.acl-srw.121}, }
NLP4DH
MADRAG: Multi-Agent Debate with Retrieval-Augmented Generation for Training-Free Analytic Essay Scoring

Ali Keramati, Shiyuan Zhou, Sharad Mehrotra, and Mark Warschauer

In Proceedings of the 6th International Conference on Natural Language Processing for the Digital Humanities, Jul 2026

Abs DOI arXiv Bib PDF

Presents MADRAG, a training-free analytic essay scoring framework that combines multi-agent debate with retrieval-augmented exemplar grounding. The system improves calibration, reduces middle-score bias, and achieves strong trait-level scoring performance that is competitive with supervised essay scoring approaches.
@inproceedings{keramati2026madrag, title = {{MADRAG}: Multi-Agent Debate with Retrieval-Augmented Generation for Training-Free Analytic Essay Scoring}, author = {Keramati, Ali and Zhou, Shiyuan and Mehrotra, Sharad and Warschauer, Mark}, booktitle = {Proceedings of the 6th International Conference on Natural Language Processing for the Digital Humanities}, pages = {325--345}, year = {2026}, month = jul, publisher = {Association for Computational Linguistics}, doi = {10.18653/v1/2026.nlp4dh-1.30}, }
EDM
Simulating Students’ Java Programming Errors with Large Language Models

Ali Keramati^*, Jie Cao^*, Iman Mohammadi, Mark Warschauer, and Yang Shi

In Proceedings of the 19th International Conference on Educational Data Mining, Jun 2026

Abs DOI arXiv Bib PDF Supp

Investigates whether large language models can realistically simulate the programming errors that students make when learning Java. By generating authentic error patterns, the work supports the development of educational tools, automated tutoring, and large-scale evaluation of programming-education interventions.
@inproceedings{keramati2026javaerrors, title = {Simulating Students' {Java} Programming Errors with Large Language Models}, author = {Keramati, Ali and Cao, Jie and Mohammadi, Iman and Warschauer, Mark and Shi, Yang}, booktitle = {Proceedings of the 19th International Conference on Educational Data Mining}, pages = {532--540}, year = {2026}, month = jun, publisher = {International Educational Data Mining Society}, doi = {10.5281/zenodo.21040050}, }
arXiv
One Success Isn’t Reliability: ThinkingBox, a Sandbox and Benchmark for Agents in Stateful Business Workflows

Zhuochun Li^*, Youngmin Ko^*, Ali Keramati^*, Tuhin Kundu, Liang-Chun Tsai, Nicola Ferri, Mirco Milletari, Jiaxiang Liu, Susana Palmaz Lopez Pelaez, Yuepeng Wang, Vadim Smolyakov, Xiang Jiang, Kjartan Ólafsson, and Tommy Guy

arXiv preprint arXiv:2608.19741, Aug 2026

Abs arXiv Bib Blog Code

Recent agent benchmarks increasingly ground evaluation in executable environments, from code repair to web navigation, app APIs, and function calling. Yet completing consequential work beyond code requires more than producing a plausible response or valid tool call: agents must gather missing information over multiple turns, follow domain policies, coordinate dependent tools, and realize the correct persistent state transition without collateral effects. In this paper, we introduce ThinkingBox, a sandbox for tool-agent-user interaction that provides isolated MCP-compatible tool sessions, complete execution traces, and outcome evaluation over terminal backend state. Built on this sandbox, ThinkingBox-Bench contains 507 policy-conditioned workflows across business scenarios, including retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support. Each attempt is evaluated by task-specific executable checks that accept valid trajectories while rejecting wrong, missing, or extra effects; designated tasks additionally check required properties of the final response. Our experiments reveal that even the strongest proprietary and open-weight models show steep reliability drops: Claude Opus 5 falls from 66.50% pass@1 to 47.53% when all 20 attempts must succeed, and Kimi-K3 from 57.37% to 17.60%. Moreover, many failed trials terminate cleanly after valid state-changing actions, so response- or tool-call-level signals poorly proxy end-to-end completion. ThinkingBox-Bench reveals a large gap between occasionally finding a successful trajectory and reliably completing stateful business tasks. We release both ThinkingBox and ThinkingBox-Bench.
@article{li2026thinkingbox, title = {One Success Isn't Reliability: {ThinkingBox}, a Sandbox and Benchmark for Agents in Stateful Business Workflows}, author = {Li, Zhuochun and Ko, Youngmin and Keramati, Ali and Kundu, Tuhin and Tsai, Liang-Chun and Ferri, Nicola and Milletari, Mirco and Liu, Jiaxiang and Palmaz Lopez Pelaez, Susana and Wang, Yuepeng and Smolyakov, Vadim and Jiang, Xiang and {\'O}lafsson, Kjartan and Guy, Tommy}, journal = {arXiv preprint arXiv:2608.19741}, year = {2026}, month = aug, archiveprefix = {arXiv}, }
NeurIPS
Application of Multi-Agent Systems for Essay Scoring

Ali Keramati and Mark Warschauer

In NeurIPS 2025 Education Program (Education Track), San Diego, Dec 2025

Contributed talk

Oral Abs Bib HTML Video Website

Selected as one of 11 contributed talks in the Education Track of the NeurIPS 2025 Education Program.

Introduces multi-agent system architectures for essay scoring, comparing approaches such as supervisor, collaboration, and debate. The work highlights how multi-agent debate can improve scoring reliability and move automated essay evaluation closer to human-level agreement while making advanced AI concepts more accessible for educators and assessment professionals.
@inproceedings{keramati2025essayscoring, title = {Application of Multi-Agent Systems for Essay Scoring}, author = {Keramati, Ali and Warschauer, Mark}, booktitle = {NeurIPS 2025 Education Program (Education Track), San Diego}, note = {Contributed talk}, year = {2025}, month = dec, }