Joyce Horn Fonteles

Multimodal AI and human–AI collaboration for understanding how people learn together.

AI researcher · software architect · educator

PhD candidate in Computer Science, Vanderbilt University · NSF EngageAI Institute

I build AI systems that help people make sense of human interaction, systems that experts can inspect, correct, and come to trust.

I'm a Computer Science PhD candidate at Vanderbilt University, advised by Prof. Gautam Biswas, and a researcher in the NSF EngageAI Institute. My work combines large language models, computer vision, and speech analysis with human-in-the-loop annotation to interpret how students learn together in real classrooms: what they say, where they look, how they move. I design the analysis pipelines, the evaluation methods that tell us when an LLM's reading of a classroom can be relied on, and the interfaces that put those results in front of researchers and teachers.

I have spent fifteen years in industry: architecting healthcare and fintech platforms, leading the engineering and database teams behind 60+ judicial systems in the Brazilian Amazon, and training the public servants who took those courts from paper to digital. That shapes how I do research. I care about data that holds up in production, and about tools that real people actually adopt.

Human-in-the-loop

Human–AI partnership

A framework for interpreting learning behaviors from classroom recordings in which AI models propose interpretations, experts verify and correct them, and authority, labor, and inspectable evidence are allocated across every stage of the analysis pipeline.

Projects
GEM-STEP: multimodal analysis of embodied science learning2022 – present

A modular framework that combines computer vision, audio processing, and LLM-based late fusion to classify self- and socially-shared regulation behaviors, refined over time with chain-of-thought prompting and active learning from human feedback.

LLMs and embodied computational thinking2024 – present

AI tools that turn students' embodied exploration into executable code through natural-language-to-code workflows, with prompt engineering and multilingual guardrails for culturally aware, pedagogically aligned interactions. Curriculum co-design and pilots in the U.S. and Israel.

Explainable AI

Interfaces people can interrogate

Review tools that keep the evidence next to every AI judgment, so researchers and teachers can validate it, override it, and learn from it, built on privacy-preserving pipelines where identifiable K–12 data is processed locally.

Projects
AI-enhanced timeline for multimodal interaction analysis2023 – present

An interface that lays gaze, movement, speech, and AI-detected behaviors side by side on a shared timeline, so learning scientists can check what the models found against the video and correct it.

LLM evaluation

Knowing when to trust an LLM's reading

Methods for evaluating LLMs and vision-language models as qualitative analysts of multimodal classroom data: where they agree with expert coders, where they diverge, and how evaluation design changes the answer.

Projects
Evaluating LLMs as analysts of classroom interaction2025 – 2026

A method for measuring how well large language models perform multimodal analysis of embodied classroom learning, compared against expert human interpretation.

Key papers
Validity & trust

Evidence that travels with every result

Validity, reliability, reproducibility, and uncertainty evidence carried through each pipeline stage, so that a questionable interpretation can be traced to the step that produced it, and experts' trust in the analysis can be measured rather than assumed.

Earlier work:Conducting particles: gesture-driven music visualization (2008 – 2021)

Real-time 3D particle systems mapped to musical structure and controlled with Leap Motion and Kinect, evaluated with musicians and conductors, plus a rhythm game for hand-therapy exercises tested with users with motor challenges.

Timeline chart loaded.
Today
At this point...
0 items active

Research

Graduate Research Assistant

Vanderbilt University · NSF EngageAI Institute

Multimodal learning analytics: deep learning and LLMs that identify and interpret student behavior from gaze, movement, speech, and gesture; interactive tools for expert validation; lead on GEM-STEP multimodal AI; co-lead of the University of Haifa collaboration.

Multimodal AIHuman-in-the-loopLLMsLearning analytics

Graduate researcher, advised by Dr. Maria Andréia Formico Rodrigues

Universidade de Fortaleza (UNIFOR)

Real-time gesture-driven systems (Leap Motion, Kinect) that visualize musical structure; usability studies with musicians, conductors, and users with motor challenges.

HCIMusic computingComputer graphics

Undergraduate researcher (Scientific Initiation)

Universidade de Fortaleza (UNIFOR) · CNPq

3D computer graphics and music computing for data visualization.

Computer graphicsMusic computing

Education

Ph.D. in Computer Science

Vanderbilt University

Intelligent open-ended learning environments for embodied learning, using AI and multimodal learning analytics to study students' learning behaviors.

Advisor: Prof. Gautam Biswas
Multimodal AIEmbodied learning

M.Sc. in Applied Informatics

Universidade de Fortaleza (UNIFOR)
Advisor: Prof. Maria Andréia Formico Rodrigues
Thesis: Creation and Evaluation of a Gesture-Controlled Conducting System for the Visualization of Musical Structures
HCIVisualizationMusic computing

B.Sc. in Computer Science

Universidade de Fortaleza (UNIFOR)
Advisor: Prof. Maria Andréia Formico Rodrigues
Thesis: A Particle System Controlled by Musical Computing
Computer graphicsMusic computing

Industry

Head of Project Implementation (Software Development Team Lead)

Pará State Court of Appeals

Led the development team through analysis, fixes, and modernization of dozens of court systems, and through planning, testing, and deployment of new ones. Product Owner and Scrum Master across projects.

LeadershipSoftware engineering

Head of the Database Division

Pará State Court of Appeals

Led the DBA team administering Oracle, SQL Server, PostgreSQL, and MySQL databases behind 60+ judicial systems; performance and data-integrity work.

LeadershipData infrastructure

Training Analyst

Lanlink (Pará State Court of Appeals)

Created e-learning content and taught hands-on courses that moved public servants across the Amazon region from paper-based to digital judicial processes.

Teaching & training

Lead Engineer

IMMI Financial Solutions

Led mobile (Swift, Android) and web (Node.js, React, MongoDB) development for financial platforms; agile delivery and client requirements across Brazil and the U.S.

LeadershipSoftware engineering

Technology Consultant

LEAD – Dell Technologies

Architecture guidance for a hybrid mobile app for accessible distance learning; taught Dialogflow, React Native, and Redux to development teams.

Software engineeringTeaching & trainingAccessibility

Solutions Architect

FFIT – Innovation in Health and Education

Designed and led mobile and web applications (Node.js, Ruby on Rails, Angular, Ionic, React Native) with reusable components and microservices; Product Owner and technical liaison with clients.

Software engineering

Systems Analyst

Unimed Fortaleza

Business analyst and full-stack developer for the main health-services portal (Java, JasperReports, PL/SQL); Scrum Master for the internal agile team.

Software engineering

Development Intern

Caixa Econômica Federal

Built an ASP system delivering Severance Fund (FGTS) data to clients; server maintenance and backups.

Software engineering

Publications

Making Interpretations Accountable: Propagating Validation Evidence Through a Human–AI Multimodal Learning Analytics Pipeline

LAK 2027
Validity & trustHuman-in-the-loopMultimodal AI

Expanding Access to Collaborative Computational Thinking through the ReCode Loop and Role-Differentiated AI Agents

Learning and Individual Differences
AI agentsLLMsEmbodied learning

A Human–AI Partnership Framework for Multimodal Analysis in Embodied Learning Environments

AIED 2026 – International Conference on Artificial Intelligence in Education
Human-in-the-loopMultimodal AI

Do Emotion Recognition Models Generalize to Classrooms? Robustness and Fairness Analysis

CV4Edu Workshop @ CVPR 2026
Computer visionAffective computing

When Modalities Disagree: Multimodal Sentiment Conflict as an Affective Signal in Collaborative Learning

ACII 2026 Workshops and Demos
Affective computingMultimodal AICollaborative learning

A Multimodal Study of Emotion Regulation in Student Dyads During Computational Model Building

ISLS 2026
Affective computingCollaborative learning

Multimodal Engagement Analytics for Collaborative Learning: A Human-in-the-Loop Approach

EARLI SIG: AI in Learning, Teaching and Education 2026
Human-in-the-loopCollaborative learning

A Human–AI Multimodal Analysis of Collaborative Knowledge Integration through Embodiment and Reflection

British Journal of Educational Technology
Human-in-the-loopMultimodal AICollaborative learning

Analyzing Embodied Learning in Classroom Settings: A Human-in-the-Loop AI Approach for Multimodal Learning Analytics

Learning and Instruction
Human-in-the-loopMultimodal AILearning analyticsExplainable AI

A Novel Approach to Evaluating the Effectiveness of Large Language Models for Multimodal Analysis of Embodied Learning in Classrooms

LAK 2026 – International Learning Analytics and Knowledge Conference
LLM evaluationMultimodal AI

A Theory of Adaptive Scaffolding for LLM-Based Pedagogical Agents

AAAI 2026 – Proceedings of the AAAI Conference on Artificial Intelligence
AI agentsLLMs

SyncFlow: A Scalable Platform for Multimodal Learning Analytics

US-RSE 2025
Data infrastructureMultimodal AI

Personalizing Student–Agent Interactions Using Log-Contextualized Retrieval-Augmented Generation (RAG)

arXiv:2505.17238
AI agentsLLMs

Exploring Agentic Multimodal Late Fusion with LLMs for Embodied Learning

MMLA Workshop @ EDM 2025
AI agentsMultimodal AILLMs

Exploring the Design of Pedagogical Agent Roles in Collaborative STEM+C Learning

CSCL 2025 – International Conference on Computer-Supported Collaborative Learning
AI agentsCollaborative learning

Interaction Analysis Practice Within Computational Trends: Promises and Challenges

ICLS 2025 – International Conference of the Learning Sciences
Human-in-the-loop

Using Collaborative Interactivity Metrics to Analyze Students' Problem-Solving Behaviors during STEM+C Computational Modeling Tasks

Learning and Individual Differences
Collaborative learningLearning analytics

A Multimodal Approach to Support Teacher, Researcher and AI Collaboration in STEM+C Learning Environments

British Journal of Educational Technology
Multimodal AICollaborative learning

Designing an AI-Enhanced Timeline for Monitoring Multimodal Interactions in Embodied Learning Environments

ICCE 2024 – 32nd International Conference on Computers in Education
Explainable AIVisualizationHuman-in-the-loop

A First Step in Using Machine Learning Methods to Enhance Interaction Analysis for Embodied Learning Environments

AIED 2024 – Artificial Intelligence in Education (LNCS)
Human-in-the-loopMultimodal AIExplainable AI

Promoting Equitable Learning Outcomes for Underserved Students in Open-Ended Learning Environments

IDC 2024 – 23rd ACM Interaction Design and Children Conference
HCI

Exploring Artificial Intelligence Supported Interaction Analysis

ICLS 2024 – International Conference of the Learning Sciences
Human-in-the-loop

3D Gaze Tracking for Studying Collaborative Interactions in Mixed-Reality Environments

ICMI Companion '24 – 26th International Conference on Multimodal Interaction
Computer visionCollaborative learning

Multimodal Methods for Analyzing Learning and Training Environments: A Systematic Literature Review

ACM Computing Surveys
Multimodal AI

Analyzing Students Collaborative Problem-Solving Behaviors in Synergistic STEM+C Learning

LAK 2024 – 14th Learning Analytics and Knowledge Conference
Collaborative learningLearning analytics

ChimeraPy: A Scientific Distributed Streaming Framework for Real-time Multimodal Data Retrieval and Processing

IEEE BigData 2023 – IEEE International Conference on Big Data
Data infrastructureMultimodal AI

Identifying Gaze Behavior Evolution via Temporal Fully-Weighted Scanpath Graphs

LAK23 – 13th International Learning Analytics and Knowledge Conference
Computer visionLearning analytics

User Experience in a Kinect-Based Conducting System for Visualization of Musical Structures

Entertainment Computing
HCIMusic computing

Gesture-Controlled Interactive Musical Game to Practice Hand Therapy Exercises and Learn Rhythm and Melodic Structures

IEEE SeGAH 2018 – 6th International Conference on Serious Games and Applications for Health
HCIAccessibilityMusic computing

Interactive Musical Game with a Gesture-Controlled Virtual Puppet

SVR 2017 – 19th Symposium on Virtual and Augmented Reality
HCIMusic computing

Visual and Gesture-Driven Interaction Using the Leap Motion to Conduct a 3D Particle System: Evaluation and Analysis of an Orchestral Performance

Journal on Interactive Systems
HCIMusic computingComputer graphics

Visual and Interactive Performance of Particles Conducted by the Leap Motion

SVR 2015 – XVII Symposium on Virtual and Augmented Reality
Computer graphicsMusic computing

Apresentando o Tratbem: Suporte Social, Informacional e Psicológico ao Tabagista

XV Workshop de Informática Médica – XXXV Congresso da Sociedade Brasileira de Computação
HCI

Real-Time Animations of Virtual Fountains Based on a Particle System for Visualizing Musical Structures

SVR 2014 – XVI Symposium on Virtual and Augmented Reality
Computer graphicsMusic computing

Creating and Evaluating a Particle System for Music Visualization

Journal of Visual Languages & Computing
Computer graphicsVisualizationMusic computing

Honors & grants

AI4Ed Summer Fellow (second selection)

AI Institutes Virtual Organization (AIVO) · funded by Google.org

Fellowship connecting researchers across the NSF National AI Research Institutes on AI in education.

Student Horizon Travel Grant

LIVE, Vanderbilt's AI and learning sciences incubator

Competitive award for presenting research at an international conference.

Red Hat Certified Developer in Cloud-native Applications

Red Hat

AI4Ed Summer Fellow

AI Institutes Virtual Organization (AIVO) · funded by Google.org

Fellowship connecting researchers across the NSF National AI Research Institutes on AI in education; presented at the AI4Ed summit.

Faculty Seed Grant ($10,000), co-author

University of Haifa

Collaborative research on LLMs and embodied computational thinking.

Peabody Global Research Grant ($7,500), co-author

Vanderbilt Peabody College

International collaboration with the University of Haifa on AI-driven embodied learning.

Certified PBB Practitioner

Caroli.org

2nd Place, Best Full Paper (Application track)

19th Symposium on Virtual and Augmented Reality (SVR 2017)

Certified ScrumMaster®

Scrum Alliance

Scientific Initiation Scholarship

CNPq (Brazil's National Council for Scientific and Technological Development)

Undergraduate research scholarship from Brazil's National Council for Scientific and Technological Development.

Service & mentoring

Co-organizer, CV4Edu workshop: Computer Vision × Education

CVPR 2026

Bridging vision, HCI, and learning sciences around domain robustness, construct-aligned inference, and privacy-aware multimodal evaluation.

Supervising an undergraduate researcher

LLMs as late-fusion mechanisms for classification and explainable AI.

Supervising a foreign master's student

LLM-based agents for computational thinking support.

Mentoring an undergraduate intern

Transcription and speaker diarization models for classroom audio.

Reviewer

Computers & Education · CVPR 2026 · LAK 2025, 2027 · AIED 2025 · EDM 2025 · ICCE 2025 · ISLS 2024, 2025 · EAAI 2025, 2027

Mentoring two undergraduate students

Hand tracking and gesture recognition with Leap Motion; educational game development in Unreal Engine.