Conference, workshop and journal papers, a book chapter, preprints and talks, newest first. Use the box below to filter by title, author, venue or year.
Diagnoses failure modes in multi-agent debate systems by combining token-level log-probability signals with LLM-as-judge evaluation. The work shows how confidence calibration can surface cases where agents are confidently wrong, improving the reliability of multi-agent reasoning for evaluation tasks.
@inproceedings{keramati2026confidentliar,title={The Confident Liar: Diagnosing Multi-Agent Debate with Log-Probabilities and {LLM}-as-Judge},author={Keramati, Ali and Cheok, Justin and Horne, Jacob and Warschauer, Mark},booktitle={Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},pages={1361--1375},year={2026},month=jul,publisher={Association for Computational Linguistics},doi={10.18653/v1/2026.acl-srw.121},}
Presents MADRAG, a training-free analytic essay scoring framework that combines multi-agent debate with retrieval-augmented exemplar grounding. The system improves calibration, reduces middle-score bias, and achieves strong trait-level scoring performance that is competitive with supervised essay scoring approaches.
@inproceedings{keramati2026madrag,title={{MADRAG}: Multi-Agent Debate with Retrieval-Augmented Generation for Training-Free Analytic Essay Scoring},author={Keramati, Ali and Zhou, Shiyuan and Mehrotra, Sharad and Warschauer, Mark},booktitle={Proceedings of the 6th International Conference on Natural Language Processing for the Digital Humanities},pages={325--345},year={2026},month=jul,publisher={Association for Computational Linguistics},doi={10.18653/v1/2026.nlp4dh-1.30},}
Evaluating reasoning quality in multi-agent LLM systems is challenging, especially for open-ended tasks without reference answers. We investigate whether intrinsic confidence signals, token-level log-probabilities from decoding, can predict reasoning quality as assessed by LLM-as-judge evaluation. Using a debate-based essay scoring framework, we compare confidence proxies against rubric-based judge scores across two ASAP essay sets. We find that early-token confidence, particularly within the first few generated tokens, is consistently the strongest predictor of reasoning quality, outperforming full-sequence statistics. Analysis of log-probability trajectories shows that the opening phase of generation is the most heterogeneous and therefore most informative. We also observe a systematic asymmetry between agent roles, with stronger alignment between confidence and quality for supportive reasoning than for adversarial critique. These results suggest that early decoding dynamics provide a lightweight and effective signal for estimating reasoning reliability in multi-agent LLM systems.
@inproceedings{keramati2026earlytoken,title={Early-Token Confidence Predicts Reasoning Quality in Multi-Agent {LLM} Debate},author={Keramati, Ali and Cheok, Justin and Horne, Jacob and Warschauer, Mark},booktitle={Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},pages={653--667},year={2026},month=jul,publisher={Association for Computational Linguistics},doi={10.18653/v1/2026.gem-main.60},}
Investigates whether large language models can realistically simulate the programming errors that students make when learning Java. By generating authentic error patterns, the work supports the development of educational tools, automated tutoring, and large-scale evaluation of programming-education interventions.
@inproceedings{keramati2026javaerrors,title={Simulating Students' {Java} Programming Errors with Large Language Models},author={Keramati, Ali and Cao, Jie and Mohammadi, Iman and Warschauer, Mark and Shi, Yang},booktitle={Proceedings of the 19th International Conference on Educational Data Mining},pages={532--540},year={2026},month=jun,publisher={International Educational Data Mining Society},doi={10.5281/zenodo.21040050},}
Proposes a teacher-in-the-loop multi-agent framework for essay scoring and feedback: a supervisor agent routes each rubric trait to specialized worker agents that score it and explain their reasoning, while the teacher monitors, reviews, and curates the scores and feedback that students receive.
@inproceedings{keramati2026teacherloop,title={A Teacher-in-the-Loop Multi-Agent Framework for Transparent Essay Scoring and Feedback},author={Keramati, Ali and Warschauer, Mark},booktitle={Proceedings of the 19th International Conference on Computer-Supported Collaborative Learning (CSCL 2026)},pages={770--772},year={2026},month=jun,publisher={International Society of the Learning Sciences},doi={10.22318/cscl2026.913900},}
Examines whether early token-level confidence signals can predict the reasoning quality of large language models in multi-agent debate systems. The findings show that signals from the first few generated tokens are especially informative for estimating reasoning reliability, offering a lightweight way to monitor and evaluate multi-agent LLM performance.
@inproceedings{keramati2026firsttokens,title={The First Tokens Matter: Early Confidence Signals for Evaluating {LLM} Reasoning},author={Keramati, Ali and Cheok, Justin and Horne, Jacob and Warschauer, Mark},booktitle={ICLR 2026 Workshop on Logical Reasoning of Large Language Models},year={2026},month=apr,}
Explores how instructors perceive and respond to AI-assisted cheating through an exploratory analysis of online forum discussions. The work surfaces the range of attitudes, concerns, and proposed responses among educators as generative AI becomes widespread in academic settings.
@inproceedings{mohammadi2026cheating,title={Mapping Instructors' Attitudes Towards Cheating with {AI}: An Exploratory Analysis Using Forum Analysis},author={Mohammadi, Iman and Keramati, Ali and Rismanchian, Sina},booktitle={Proceedings of the 20th International Conference of the Learning Sciences (ICLS 2026)},pages={2092--2096},year={2026},month=jun,publisher={International Society of the Learning Sciences},doi={10.22318/icls2026.578118},}
Proposes a new way to analyze large language model reasoning traces in multi-agent systems by treating disagreement as a meaningful analytic signal. The work shows how semantic similarity between agent reasoning can help detect coding ambiguity, improve qualitative analysis workflows, and strengthen human–AI collaboration in educational research.
@inproceedings{tajik2026disagreement,title={Disagreement as Data: Reasoning Trace Analytics in Multi-Agent Systems},author={Tajik, Elham and Borchers, Conrad and Shahrokhian, Bahar and Simon, Sebastian and Keramati, Ali and Pal, Sonika and Sankaranarayanan, Sreecharan},booktitle={Proceedings of the 16th International Learning Analytics and Knowledge Conference (LAK26)},pages={933--939},year={2026},month=apr,publisher={ACM},doi={10.1145/3785022.3785101},}
@inproceedings{mohammadi2026voices,title={Voices from the Classroom: An Analysis of Online Forum Discussions on Generative {AI} in Everyday Teaching},author={Mohammadi, Iman and Keramati, Ali and Cao, Jie and Shi, Yang},booktitle={Companion Proceedings of the 16th International Conference on Learning Analytics \& Knowledge (LAK26)},pages={151--153},year={2026},month=apr,}
arXiv
One Success Isn’t Reliability: ThinkingBox, a Sandbox and Benchmark for Agents in Stateful Business Workflows
Zhuochun Li*, Youngmin Ko*, Ali Keramati*, Tuhin Kundu, Liang-Chun Tsai, Nicola Ferri, Mirco Milletari, Jiaxiang Liu, Susana Palmaz Lopez Pelaez, Yuepeng Wang, Vadim Smolyakov, Xiang Jiang, Kjartan Ólafsson, and Tommy Guy
Recent agent benchmarks increasingly ground evaluation in executable environments, from code repair to web navigation, app APIs, and function calling. Yet completing consequential work beyond code requires more than producing a plausible response or valid tool call: agents must gather missing information over multiple turns, follow domain policies, coordinate dependent tools, and realize the correct persistent state transition without collateral effects. In this paper, we introduce ThinkingBox, a sandbox for tool-agent-user interaction that provides isolated MCP-compatible tool sessions, complete execution traces, and outcome evaluation over terminal backend state. Built on this sandbox, ThinkingBox-Bench contains 507 policy-conditioned workflows across business scenarios, including retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support. Each attempt is evaluated by task-specific executable checks that accept valid trajectories while rejecting wrong, missing, or extra effects; designated tasks additionally check required properties of the final response. Our experiments reveal that even the strongest proprietary and open-weight models show steep reliability drops: Claude Opus 5 falls from 66.50% pass@1 to 47.53% when all 20 attempts must succeed, and Kimi-K3 from 57.37% to 17.60%. Moreover, many failed trials terminate cleanly after valid state-changing actions, so response- or tool-call-level signals poorly proxy end-to-end completion. ThinkingBox-Bench reveals a large gap between occasionally finding a successful trajectory and reliably completing stateful business tasks. We release both ThinkingBox and ThinkingBox-Bench.
@article{li2026thinkingbox,title={One Success Isn't Reliability: {ThinkingBox}, a Sandbox and Benchmark for Agents in Stateful Business Workflows},author={Li, Zhuochun and Ko, Youngmin and Keramati, Ali and Kundu, Tuhin and Tsai, Liang-Chun and Ferri, Nicola and Milletari, Mirco and Liu, Jiaxiang and Palmaz Lopez Pelaez, Susana and Wang, Yuepeng and Smolyakov, Vadim and Jiang, Xiang and {\'O}lafsson, Kjartan and Guy, Tommy},journal={arXiv preprint arXiv:2608.19741},year={2026},month=aug,archiveprefix={arXiv},}
arXiv
The Order Matters: Sequential Fine-Tuning of LLaMA for Coherent Automated Essay Scoring
Automated Essay Scoring (AES) systems must judge interdependent discourse elements (e.g., lead, claim, evidence, conclusion), yet most approaches treat these in isolation, harming coherence and generalization. We investigate task-aware fine-tuning of LLaMA-3.1-8B for AES using parameter-efficient LoRA with 4-bit quantization and compare three training curricula: (i) Sequential (progressively fine-tuning on lead, then position, then claim, then evidence, then conclusion), (ii) Independent (task-specific models), and (iii) Randomized (shuffled multi-task). Experiments on the PERSUADE 2.0 corpus show that modeling task dependencies matters: Sequential fine-tuning yields the strongest overall results, including F1 scores of 65% (evidence) and 87% (conclusion) and corresponding accuracies of 63% and 85%, surpassing Independent training and outperforming a general-purpose LLaMA-70B baseline on conclusion despite its far larger capacity. Randomized training improves position scoring (57% F1) but is less consistent elsewhere. These findings indicate that (1) curriculum design aligned with discourse structure can materially improve AES, and (2) small, task-optimized models can be competitive with substantially larger Large Language Models (LLM), offering a practical path to scalable, cost-effective assessment. We release templates and implementation details to facilitate reproduction and future work on curriculum design for educational NLP.
@article{keramati2026order,title={The Order Matters: Sequential Fine-Tuning of {LLaMA} for Coherent Automated Essay Scoring},author={Keramati, Ali and Warschauer, Mark},journal={arXiv preprint arXiv:2606.10327},year={2026},month=jun,archiveprefix={arXiv},}
@article{keramati2026leaders,title={Developing Future Leaders: Practicing Reflective Leadership Skills in {Iranian} Primary Schools Within a Cooperative Learning Environment},author={Keramati, Mohammad Reza and Ansarizadeh, Fahimeh and Keramati, Ali},journal={Reflective Practice},volume={27},number={2},pages={177--190},year={2026},publisher={Taylor \& Francis},doi={10.1080/14623943.2025.2541423},}
@incollection{keramati2026chapter,title={Fostering Essential 21st-Century Skills Through Cooperative Learning},author={Keramati, Mohammad Reza and Keramati, Ali},booktitle={Teaching and Learning 21st-Century Skills: Collaboration and Communication in Formal and Informal Educational Settings},editor={Gillies, Robyn M.},pages={283--298},year={2026},publisher={Springer Nature Singapore},doi={10.1007/978-981-95-5997-8_16},}
Selected as one of 11 contributed talks in the Education Track of the NeurIPS 2025 Education Program.
Introduces multi-agent system architectures for essay scoring, comparing approaches such as supervisor, collaboration, and debate. The work highlights how multi-agent debate can improve scoring reliability and move automated essay evaluation closer to human-level agreement while making advanced AI concepts more accessible for educators and assessment professionals.
@inproceedings{keramati2025essayscoring,title={Application of Multi-Agent Systems for Essay Scoring},author={Keramati, Ali and Warschauer, Mark},booktitle={NeurIPS 2025 Education Program (Education Track), San Diego},note={Contributed talk},year={2025},month=dec,}
A novel approach to automated essay scoring using multi-agent systems. By leveraging debate between specialized agents, the framework provides a more reliable and nuanced evaluation, surpassing traditional single-agent methods and bringing scoring closer to human-level agreement.
@misc{keramati2025madest,title={{MADEST}: Multi-Agent Debate Essay Scoring Triangulation},author={Keramati, Ali and Warschauer, Mark},howpublished={Zenodo},note={First-year Ph.D. poster},year={2025},month=sep,publisher={Zenodo},doi={10.5281/zenodo.17196206},}
Automatic metadata tagging of educational content to enhance Intelligent Tutoring Systems (ITS). Leveraging NLP, machine learning, and domain-specific rules, it efficiently extracts both general and pedagogical metadata, improving personalized learning recommendations.
@inproceedings{shahhoseini2025itag,title={{iTAG}: Easy, Rapid, Automatic Intelligent Tagging for Educational Contents},author={Shahhoseini, Narges and Taghiyareh, Fattaneh and Keramati, Ali},booktitle={2025 29th International Computer Conference, Computer Society of Iran (CSICC)},pages={1--5},year={2025},month=feb,publisher={IEEE},doi={10.1109/CSICC65765.2025.10967458},}
Identifies challenges in implementing cooperative learning (CL) in Iran’s primary schools, including traditional teaching methods, stakeholder readiness, and lack of facilities. Solutions include improving curriculum, fostering positive attitudes, and enhancing teacher training for effective CL implementation.
@article{keramati2024cooperative,title={Challenges and Solutions of Implementing Cooperative Learning in Classroom Environments: Lived Experiences of Primary School Teachers in {Iran}},author={Keramati, Mohammad Reza and Gillies, Robyn M. and Ansarizadeh, Fahimeh and Keramati, Ali},journal={International Journal of Early Years Education},volume={33},number={1},pages={120--137},year={2025},publisher={Taylor \& Francis},doi={10.1080/09669760.2024.2406374},}
Implemented Theory of Mind assessment for individuals with Autism by developing a LLaMA-3 chatbot to administer the Faux Pas Recognition Test. The system utilizes a multi-phase pipeline that combines interactive rehabilitation dialogues with adaptive hints, an automated scoring phase using an LLM-as-judge, and generates clinician-ready reports to enhance social cognition and improve faux pas detection.
@inproceedings{fallah2024theoryofmind,title={Automating Theory of Mind Assessment with a {LLaMA-3}-Powered Chatbot: Enhancing Faux Pas Detection in Autism},author={Fallah, Avisa and Keramati, Ali and Nazari, Mohammad Ali and Mirfazeli, Fatemeh Sadat},booktitle={2024 14th International Conference on Computer and Knowledge Engineering (ICCKE)},pages={365--372},year={2024},month=nov,publisher={IEEE},doi={10.1109/ICCKE65377.2024.10874775},}
arXiv
AI-Powered Digital Framework for Personalized Economical Quality Learning at Scale
Mrzieh VatandoustMohammadieh, Mohammad Mahdi Mohajeri, Ali Keramati, and Majid Nili Ahmadabadi
AI-powered digital learning framework designed for scalable, personalized, and cost-effective education. Framework integrates AI-based learner modeling, a personalized recommender system, and AI-assisted support for both learners and facilitators, aiming to address challenges in educational access, soft skills development, and large-scale implementation.
@article{vatandoust2024aipowered,title={{AI}-Powered Digital Framework for Personalized Economical Quality Learning at Scale},author={VatandoustMohammadieh, Mrzieh and Mohajeri, Mohammad Mahdi and Keramati, Ali and Nili Ahmadabadi, Majid},journal={arXiv preprint arXiv:2412.04483},year={2024},month=nov,archiveprefix={arXiv},}
Explores the impact of collaborative learning (CL) on the learning environment and academic achievement in online reflective platforms (ORPs). The mixed-methods approach reveals that CL enhances student engagement, promotes a supportive learning atmosphere, and improves academic performance, with positive reflections from students on the collaborative and interactive nature of the online learning experience.
@article{keramati2024reflection,title={Students' Reflection on the Effect of Collaborative Learning on the Learning Environment and Academic Achievement in Online Reflective Platforms},author={Keramati, Ali and Keramati, Mohammad Reza and Arefian, Mohammad Hossein},journal={Reflective Practice},volume={25},number={2},pages={117--131},year={2024},publisher={Taylor \& Francis},doi={10.1080/14623943.2024.2305868},}
Research background shows that university teachers in the field of education have different definitions of cooperative learning with some not differentiating among group work, collaborative learning, and cooperative learning. The main purpose of the current study is not to distinguish these three concepts from each other, but to identify the perceptions of university academics in Australia and Iran. The current study intends to present a clearer understanding of cooperative learning, highlighting the similarities and differences that may exist between the two countries. This was a qualitative study involving 23 university teachers who were interviewed on their understanding of cooperative learning. Thematic analysis was used to analyse the in-depth interview data which were coded and organized using Atlas/ti software. Participants in both Australia and Iran defined cooperative learning as an interactive procedure. They used similar terms such as active learning, exchange of knowledge, and individual accountability. Australian participants emphasized the shared goal, defining the roles and planning, while Iranian participants paid more attention to the management aspects of cooperative learning. This cross-cultural study identifies new insights of cooperative learning that may need to be considered in order to develop the quality of teaching and learning in university classrooms.
@article{keramati2023concept,title={Concept of Cooperative Learning in {Australia} and {Iran}},author={Keramati, Mohammad Reza and Gillies, Robyn M. and Ansarizadeh, Fahimeh and Keramati, Ali},journal={Iranian Journal of Comparative Education},volume={6},number={4},pages={2693--2708},year={2023},publisher={Comparative Education Society of Iran},doi={10.22034/ijce.2023.387792.1482},}
