cv
Education, research experience, teaching, service and awards. The PDF icon downloads my CV; transcripts are on the education page.
Contact Information
| Name | Ali Keramati |
| Professional Title | Ph.D. Candidate in Education Data Science, University of California, Irvine |
| akerama1@uci.edu | |
| Location | Irvine, California |
Professional Summary
Ph.D. candidate in Education Data Science at the University of California, Irvine (Digital Learning Lab), with an M.A. in Education from UC Irvine (2026) and a B.Sc. in Computer Engineering from the University of Tehran. I combine natural language processing and multi-agent AI with qualitative and quantitative educational research to study how digital learning systems can support teaching and learning. My research spans reliable assessment, learner simulation, and the evaluation of AI agents. Experience includes PapyrusAI and the IES-funded WRITE AI Center at UC Irvine and an applied-scientist internship at Microsoft. First or co-first author of 7 peer-reviewed publications; Program Committee member, EDM 2026.
Experience
-
2025 - Present Irvine, CA
Graduate Student Researcher
Digital Learning Lab, University of California, Irvine
Director: Prof. Mark Warschauer
- Develop and evaluate multi-agent LLM methods for rubric-based essay scoring and feedback, including comparisons with human ratings and analysis of model reasoning; first author of the ACL 2026 Student Research Workshop, NLP4DH 2026, GEM 2026, CSCL 2026, and ICLR 2026 workshop papers; co-first author of the EDM 2026 paper.
- Work on PapyrusAI, the lab’s AI writing coach, including within the IES-funded WRITE AI Center: teacher-reviewed multi-agent essay scoring and feedback, and evaluation of AI models and prompts.
- Ongoing: methods to localize uncertainty in LLM agents’ generated SQL and Python and to decide when an agent should stop, resample, or ask for clarification.
-
2025 - Present Irvine, CA
Teaching Assistant and Learning Assistant
University of California, Irvine
- EDUC 217 AI and Digital Learning (graduate course), Fall 2026: Learning Assistant (instructor Prof. Mark Warschauer).
- EDUC 30 21st Century Literacies, Spring 2026: Teaching Assistant (instructor Dr. Viet Vu).
- EDUC 173 Cognition and Learning in Educational Settings: Teaching Assistant, Fall 2025 (instructor Prof. Shayan Doroudi) and Winter 2026 (instructor Prof. Drew Bailey).
-
2021 - Present Research collaboration (remote)
With Prof. Robyn M. Gillies (University of Queensland) and Prof. Mohammad Reza Keramati
-
2026 - 2026 Redmond, WA
Applied Scientist Intern
Microsoft
Microsoft Copilot Studio
- Built evaluation and training pipelines for multi-turn, task-oriented AI agents: LLM-as-judge evaluators, simulated users, and sandbox-derived rewards for reinforcement-learning post-training.
- Contributed to ThinkingBox and co-developed ThinkingBox-Bench, an open-source agent sandbox and reliability benchmark, building its user simulator and the LLM judges that evaluate agent behavior; co-first author of the accompanying arXiv preprint.
-
2023 - 2024 Tehran, Iran
Research Assistant, School of Electrical and Computer Engineering
University of Tehran
- Cognitive Systems Lab (Prof. Majid Nili Ahmadabadi), May 2023 – Dec 2024: contributed to an AI-powered framework for personalized learning at scale, including a chatbot for open learner modeling (arXiv preprint, 2024).
- In collaboration with the National Brain Center (Prof. Fatemeh Sadat Mirfazeli), Jul – Sep 2024: co-developed a LLaMA-3 chatbot that administers the Faux Pas Recognition Test of theory of mind for individuals with autism, with rehabilitative guidance and automated evaluation under human oversight (ICCKE 2024).
- Technology Enhanced Learning Lab (Prof. Fattaneh Taghiyareh), Mar – Aug 2023: built an IEEE LOM metadata extractor for slides, documents and PDFs, and contributed to iTAG (CSICC 2025).
- Natural Language Processing group (Profs. M. J. Dousti and H. Faili), Feb – Dec 2024: research on large language models, retrieval-augmented generation, and parameter-efficient fine-tuning.
-
2020 - 2024 Tehran, Iran
Teaching Assistant
School of Electrical and Computer Engineering, University of Tehran
- Head TA: Discrete Mathematics, Internet Engineering, and Software Engineering. Supervising TA: Data Structures and Algorithms.
- TA: Reinforcement Learning, Engineering Probability and Statistics, Algorithm Design, Database Design, and Electric Circuits.
Education
-
2025 - Present Irvine, CA
Ph.D. in Education Data Science
University of California, Irvine
- Expected completion: Jun 2029.
- GPA: 3.98/4.00.
- Coursework: Neural Networks and Deep Learning; Deep Generative Models; Brain-Inspired Learning Systems; Neural Architecture of Language; Language Models in Cognitive Science; Learning Analytics Fundamentals; Learning Analytics Practicum; Applied Regression Analysis; Research Epistemologies and Methodologies; Qualitative Research Methods in Education I and II; Early Childhood Education Policy.
- Unofficial transcript
-
2026 Irvine, CA
M.A. in Education
University of California, Irvine
-
2019 - 2024 Tehran, Iran
B.Sc. in Computer Engineering
University of Tehran
School of Electrical and Computer Engineering
- GPA: 18.51/20. Minor in Educational Sciences, Faculty of Psychology and Education (completed Jul 2023; GPA: 19.31/20).
- Transcripts: Computer Engineering (English) · Educational Sciences minor (English) · more on the education page
Selected Research Contributions
- Reliable automated assessment and feedback. Automated essay scoring usually depends on task-specific training and gives little explanation for its scores. As first author, I designed and evaluated MADRAG (NLP4DH 2026): an Advocate and a Skeptic agent debate an essay’s strengths and weaknesses, and a Judge agent calibrates its score against retrieved, rubric-aligned scored examples, without fine-tuning. MADRAG outperforms prompt-based baselines and approaches supervised systems. Related first-authored work adds teacher-in-the-loop feedback (CSCL 2026) and shows that LoRA fine-tuning of an 8-billion-parameter model, ordered by essay structure, outperforms a general LLaMA-70B baseline on conclusion scoring (arXiv preprint).
- Measuring reasoning quality in multi-agent LLM systems. Multi-agent LLM systems are usually judged only by their final answers. As first author, I designed the debate frameworks and confidence analyses in the ACL 2026 Student Research Workshop and GEM 2026 papers. In rubric-based essay scoring, token-level confidence detected the critical reasoning failures flagged by an LLM judge with an AUROC of 0.804 for the constructing agent versus 0.634 for the auditing agent (ACL 2026 SRW); on two ASAP essay sets, confidence over the first few generated tokens was the strongest predictor of LLM-judged reasoning quality, outperforming full-sequence statistics (GEM 2026).
- Reliability benchmarks for tool-using AI agents. Agent benchmarks often credit a single successful run. At Microsoft, I contributed to ThinkingBox, a sandbox that grades agents by executable checks on the final system state, and co-developed ThinkingBox-Bench, 507 policy-conditioned business workflows, building the user simulator and the LLM judges that evaluate agents; co-first author of the paper (arXiv preprint). Across 20 repeated attempts per task, the top-scoring model’s success fell from 66.5% on a single attempt to 47.5% when it had to succeed on all 20.
- Simulated learners. Collecting representative student errors for a new programming task normally takes extensive classroom deployment. As co-first author, I designed the Self-Refine student simulator and the syntax-tree and tree-edit-distance evaluation in the EDM 2026 paper, comparing five LLMs and three prompting strategies against 74,080 authentic Java submissions. In blinded expert annotation, 164 of 196 erroneous submissions generated by the selected model (Claude Sonnet 4 with Self-Refine) were judged human-written (83.7%), although the synthetic and authentic error distributions differed in shape.
Projects
-
2026 ThinkingBox and ThinkingBox-Bench
Microsoft, open source · contributor to the sandbox; co-developer of the benchmark (user simulator, LLM-judge evaluators); co-first author of the accompanying arXiv preprint
- Agent sandbox with isolated Model Context Protocol (MCP) tool sessions and complete execution traces; benchmark of 507 policy-conditioned workflows in five business domains (v1.0, August 2026).
- github.com/microsoft/thinkingbox · github.com/microsoft/thinkingbox-data
-
2025 - Present PapyrusAI
Digital Learning Lab, UC Irvine · graduate student researcher
- AI writing coach that uses guided dialogue and personalized feedback to help students develop ideas, organize arguments, and revise their writing. Instructors can customize learning activities and monitor student engagement to support writing instruction and AI literacy.
- PapyrusAI project page · public frontend repository
Publications
-
2026 The Confident Liar: Diagnosing Multi-Agent Debate with Log-Probabilities and LLM-as-Judge
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)
Ali Keramati, Justin Cheok, Jacob Horne, Mark Warschauer
-
2026 MADRAG: Multi-Agent Debate with Retrieval-Augmented Generation for Training-Free Analytic Essay Scoring
Proceedings of the 6th International Conference on Natural Language Processing for the Digital Humanities (NLP4DH 2026), co-located with ACL 2026
Ali Keramati, Shiyuan Zhou, Sharad Mehrotra, Mark Warschauer
-
2026 Early-Token Confidence Predicts Reasoning Quality in Multi-Agent LLM Debate
Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM), co-located with ACL 2026
Ali Keramati, Justin Cheok, Jacob Horne, Mark Warschauer
-
2026 Simulating Students' Java Programming Errors with Large Language Models
Proceedings of the 19th International Conference on Educational Data Mining (EDM 2026)
Ali Keramati†, Jie Cao†, Iman Mohammadi, Mark Warschauer, Yang Shi (†equal contribution)
-
2026 A Teacher-in-the-Loop Multi-Agent Framework for Transparent Essay Scoring and Feedback
Proceedings of the 19th International Conference on Computer-Supported Collaborative Learning (CSCL 2026)
Ali Keramati, Mark Warschauer
-
2026 The First Tokens Matter: Early Confidence Signals for Evaluating LLM Reasoning
ICLR 2026 Workshop on Logical Reasoning of Large Language Models
Ali Keramati, Justin Cheok, Jacob Horne, Mark Warschauer
-
2024 Students' Reflection on the Effect of Collaborative Learning on the Learning Environment and Academic Achievement in Online Reflective Platforms
Reflective Practice, 25(2), 117–131
Ali Keramati, Mohammad Reza Keramati, Mohammad Hossein Arefian
Presentations
- AI-Human Writing Score Agreement in Primary Grades: A Longitudinal Investigation. Young-Suk G. Kim, Tamara P. Tate, Ali Keramati, Julie Jung, Max Lu, Justin Cheok. Paper session, NCME Artificial Intelligence in Measurement and Education Conference (AIME-Con 2026), Pittsburgh, PA, October 2026.
- Application of Multi-Agent Systems for Essay Scoring. Ali Keramati, Mark Warschauer. Contributed talk (one of 11 in the Education Track), NeurIPS 2025 Education Program, San Diego, CA, December 2025. Recording
Peer Review
2026: Program Committee Member, 19th International Conference on Educational Data Mining (EDM 2026)
2026: Reviewer, Journal of Educational Data Mining
2026: Reviewer, Fifth Workshop on Generation, Evaluation and Metrics (GEM 2026), co-located with ACL 2026
2026: Reviewer, International Society of the Learning Sciences (ISLS) 2026 Annual Meeting (ICLS and CSCL)
2025: Reviewer, NeurIPS 2025 Education Program
Service and Leadership
2026: Student Volunteer, ACL 2026, San Diego, CA
2025–2026: Graduate Ambassador, The People's Coalition at UCI (TPC)
2020: President, University of Tehran ACM Student Chapter
Honors and Awards
-
2026 ACL 2026 Diversity and Inclusion Subsidy
Association for Computational Linguistics
-
2025 Selected participant, LearnLab Summer School
Carnegie Mellon University
Computer Science Education Research track; program fee covered by the NSF-funded SPLICE program.
Skills
AI methods: LLM evaluation (LLM-as-judge; confidence and log-probability analysis), multi-agent systems and debate, retrieval-augmented generation, parameter-efficient fine-tuning (LoRA) with 4-bit quantization, reinforcement learning
Research methods: Learner and user simulation, learning analytics, regression, qualitative coding, topic modeling
Programming: Python, SQL, Java, C/C++, JavaScript, R
ML and LLM tools: PyTorch, Hugging Face Transformers, PEFT, TRL, vLLM, LangChain, LangGraph, Model Context Protocol (MCP)
Data and retrieval: pandas, scikit-learn, FAISS, Elasticsearch, PostgreSQL, Neo4j
Education research tools: LearnSphere, pyBKT, ATLAS.ti, Dedoose