From Multimodal Sensing Data to Actionable Health Insights
Build and inspect a personal LLM health agent in your browser using the public GLOBEM sensing sample. Write tools, follow the agent’s tool calls, and test whether a change to its instructions helps. An AI helper supports you as you work; you check the answers against the data.
This tutorial took place on October 12, 2026. Slides and exercises remain open in AgentLab.
Thank you to the 50+ participants who joined us on October 12, 2026 at the Shanghai International Convention Center! The afternoon brought together an opening talk, invited talks, hands-on agent building in the browser, and a panel on evaluation and safety.
“Why have I been sleeping poorly this week?”
Wearable and smartphone records can show when sleep, activity, or phone use changed. An LLM agent can call tools to explore those records, but its explanations still need checking. In this hands-on tutorial, you will build an agent over the public GLOBEM sample, inspect how it reaches an answer, and compare answers before and after changing a tool or skill. The hands-on runs in your browser, with nothing to install.
Start with a health question.
Connect sensing data and tools.
Compare the answer with the evidence.
No prior LLM or agent experience required. We start with the basics (model, tool, agent and skill), then build an agent, write tools for it, and test whether a rewritten skill helps.
Follow the slides alongside editable Python cells. Ask the AI helper about the lesson, a blank, or an error: it uses the tutorial, your current code and recent outputs to offer explanations and hints. You run and check any suggested changes. You can also add your own cells, inspect variables, and browse examples or function source.
Open AgentLab for the slides, Python exercises and AI helper. Model calls and the AI helper require a seat code provided at the tutorial.
A short-format, hands-on instantiation of Student–AI Collaborative Inquiry (SACI). Read the accepted paper (PDF).
SACI pairs a student and AI agent as co-investigators. The student contextualizes, interrogates, and reflects on claims; the agent retrieves and computes evidence, forms hypotheses, and surfaces counter-evidence. Responsibility remains with the student and instructor. This tutorial applies the model to a public GLOBEM sample: participants question the agent’s answers against the data, and when the model proposes a change to its own instructions, the participant judges it.
October 12, 2026 · 14:00–17:30 · Shanghai time (UTC+8). Room 5A, Shanghai, China. Includes 80 minutes of guided hands-on practice and a 30-minute conference coffee break.
Introduction to the tutorial and the afternoon’s activities.
Georgia Institute of Technology · 20-minute talk followed by 5 minutes of Q&A.
Chinese Academy of Sciences.
The University of Hong Kong.
The University of Hong Kong.
Explore the public GLOBEM sample and how an agent connects a model with tools.
Conference coffee break.
Write tools, improve an agent’s skill, and check the change against the data.
With Dr. Thomas Plötz, Dr. Teng Han, Dr. Chenshu Wu, and Dr. Edith C. H. Ngai. Discussion of response quality, alignment with users’ goals, failure modes, and ethical considerations. Includes audience Q&A and closing remarks.
Dr. Thomas Plötz will give the opening talk.
Opening speaker
School of Interactive Computing, Georgia Institute of Technology
Dr. Thomas Plötz is a Professor in Georgia Tech’s School of Interactive Computing. His research develops machine-learning methods for interpreting wearable and other sensor data to assess human behavior in everyday settings.
Invited speaker
Institute of Software, Chinese Academy of Sciences
Dr. Teng Han’s recent work includes health-related HCI, such as electrotactile studies of pain modulation and interaction support for people with visual impairments, and VR studies of cognitive fatigue, state mindfulness, and transient physiological discomfort. His broader research spans wearable haptics and multisensory human–machine interaction.
Invited speaker
The University of Hong Kong
Dr. Chenshu Wu develops sensing AI for healthcare, eldercare, and homecare, including contactless and non-intrusive monitoring. His recent work explores online cardiac monitoring from audio-visual streams, self-supervised prediction of adverse events in type 1 diabetes, and low-cost fall detection for older adults in home settings.
Invited speaker
The University of Hong Kong
Dr. Edith C. H. Ngai is an Associate Professor at The University of Hong Kong. Her research connects wearable and mobile sensing with AI for health, including visual analytics of multimodal health data, smartphone-based digital phenotyping, and LLM-powered dietary monitoring and personalized analysis.
The tutorial is built to be accessible to participants from non-LLM backgrounds while staying meaningful for those with prior experience.
The hands-on uses the official public sample of the GLOBEM dataset: 10 college students with a Fitbit and a phone-sensing app, about seven weeks each. Observed missing values stay missing, and no credentialed data access is required.
A team spanning multimodal wearable sensing, LLM-based health intervention, conversational agents and clinical workflow evaluation, and human-centered AI for healthcare.
Slides and exercises are available now. Model calls and the AI helper require the seat code provided at the tutorial.
The browser tutorial includes slides, editable Python exercises, reference solutions and an AI helper. Participants can export their work as a notebook and download the public sample and lab package.
The hands-on uses the official public GLOBEM sample, not the full credentialed PhysioNet release. Observed missing values remain missing.
