UbiComp/ISWC 2026 · Half-day Tutorial · Shanghai, China

From Multimodal Sensing Data to Actionable Health Insights

Prototyping Your Personal LLM Health Agent

Build a working personal LLM health agent over a synthetic multimodal sensing dataset, turning raw sensor streams into personalized, actionable health insights. Then ground every claim, catch confounds, and evaluate it for faithfulness and safety.

Oct 11, 2026 (tentative)
Shanghai, China
Half-day (3.5h), hands-on
Bring a laptop · Python ≥ 3.10
Registration TBA Starter repository TBA

“Why have I been sleeping poorly this week?”

Most consumer wearables and health dashboards can show you the data behind that question, but they cannot answer it. LLM agents — language models that call functions, run analyses, and visualize results — offer a path to turn raw multimodal sensing streams into grounded, personalized answers. In this half-day, hands-on tutorial, you will build a working personal LLM health agent from a minimal scaffold, over a synthetic, curated multimodal sensing dataset (sleep, heart rate, activity, GPS, screen time, EMA).

Diagram of the half-day tutorial workflow: a participant asks why they have been sleeping poorly; a health-agent scaffold accesses fully synthetic multimodal data and uses retrieval, analysis, and visualization tools; six modules run from introduction through evaluation, safety, and wrap-up.
The personal LLM health agent stack participants build during this half-day tutorial, across six modules from introduction to wrap-up. Open the figure to view it at full size.

What you'll do — and leave with

No prior LLM or agent experience required. We open with a concise primer, then build, evaluate, and stress-test an agent together.

What you will do

  • Build health-specific tools (data retrieval, analysis, visualization) and register them with an LLM agent.
  • Wire up an agent loop and iterate it on real questions over wearable, smartphone, and self-report data.
  • Examine grounding, confounds, safety handling, and ethical considerations in a moderated panel with invited guests.

What you will leave with

  • A working LLM health agent running locally on the provided dataset.
  • A runnable codebase and design template you can adapt to your own research.
  • A practical checklist for evaluating LLM agents — minimal faithfulness checks, confounds and causal humility, and deterministic safety handling (refusal, emergency escalation, crisis support).
  • A roadmap for extending the scaffold in your own work.

The learning model behind the tutorial

A short-format, hands-on instantiation of Student–AI Collaborative Inquiry (SACI). Read the accepted paper (PDF).

Accepted · UbiComp/ISWC 2026 Education Forum Students and AI agents co-investigate multimodal sensing data through mutual interrogation, under instructor scaffolding. Explore the SACI learning model Hide the SACI learning model

Student–AI Collaborative Inquiry SACI

SACI pairs a student and AI agent as co-investigators. The student contextualizes, interrogates, and reflects on claims; the agent retrieves and computes evidence, forms hypotheses, and surfaces counter-evidence. Responsibility remains with the student and instructor. This tutorial applies the model to synthetic multimodal sensing data, with instructor-led scaffolds for refusal, red-teaming, and research governance.

The broader Student–AI Collaborative Inquiry model: instructor scaffolding and ethical safeguards around mutual interrogation between a student and AI agent over the student's multimodal sensor data, followed by five learning phases.
The SACI paper’s model figure. This tutorial uses a synthetic teaching dataset rather than participants’ personal data. Open the figure to view it at full size.
Paper From AI-as-Shortcut to AI-as-Partner: Student-AI Collaborative Inquiry for Health-Agent Education Zhihan Jiang & Xuhai Xu · UbiComp Companion ’26

Schedule

A half-day session of 3.5 hours, alternating short conceptual lectures, guided coding, and a moderated panel. Every module ships with a starter notebook, reference solution, and an optional advanced exercise.

Invited Speakers

Invited speakers will contribute to the tutorial talks and our moderated Module 5 panel on evaluation and safety. Additional speakers will be announced here as they are confirmed. Speakers are listed alphabetically by family name.

Teng Han Invited speaker

Teng Han

Institute of Software, Chinese Academy of Sciences

Dr. Teng Han’s recent work includes health-related HCI, such as electrotactile studies of pain modulation and interaction support for people with visual impairments, and VR studies of cognitive fatigue, state mindfulness, and transient physiological discomfort. His broader research spans wearable haptics and multisensory human–machine interaction.

Xin Liu Invited speaker

Xin Liu

Google Research

Dr. Xin Liu’s research spans ubiquitous and mobile computing, machine learning, and health, with a focus on foundation models for wearable and consumer health data. His recent work includes scaling multimodal wearable-sensor models and building personal health agents that reason across consumer-device and health-record data.

Chenshu Wu Invited speaker

Chenshu Wu

The University of Hong Kong

Dr. Chenshu Wu develops sensing AI for healthcare, eldercare, and homecare, including contactless and non-intrusive monitoring. His recent work explores online cardiac monitoring from audio-visual streams, self-supervised prediction of adverse events in type 1 diabetes, and low-cost fall detection for older adults in home settings.

Prerequisites & what to bring

The tutorial is built to be accessible to participants from non-LLM backgrounds while staying meaningful for those with prior experience.

Prerequisites

  • Intermediate Python: comfortable reading and modifying a Jupyter notebook, installing packages, and debugging API calls.
  • No prior LLM or agent experience required — we open with a concise primer.
  • Familiarity with wearable or behavioral data is helpful but not required; cleaned sample datasets are provided.

What to bring

  • A laptop with Python ≥ 3.10 and Jupyter installed.
  • API key or local model recommended — for the full LLM-based path, bring either a key for a supported provider or a compatible local model (for example, via Ollama). If neither is available—or setup fails—the deterministic scripted fallback supports the guided tool-building and agent-wiring exercises, but it does not reproduce open-ended LLM reasoning.
  • Detailed setup instructions, including a one-command install script, are sent two weeks before the tutorial.

The dataset

You'll work with a fully synthetic teaching dataset whose schema and feature types are inspired by the GLOBEM behavioral dataset, paired with additional synthetic records covering modalities not represented in GLOBEM. No credentialed data access is required during the tutorial, and we provide a data-privacy checklist you can adapt for your own studies.

Sleep Heart rate Physical activity GPS / location Screen time EMA self-reports Context

Organizers

A team spanning multimodal wearable sensing, LLM-based health intervention, conversational agents and clinical workflow evaluation, and human-centered AI for healthcare.

Zhihan Jiang

Zhihan Jiang

Postdoctoral Researcher

Columbia University

zj2445@cumc.columbia.edu
Will Ke Wang

Will Ke Wang

Postdoctoral Researcher

Columbia University

kw3215@cumc.columbia.edu
Blue (Georgianna) Lin

Blue (Georgianna) Lin

Postdoctoral Researcher

Columbia University

gl2981@cumc.columbia.edu
Brenna Li

Brenna Li

Postdoctoral Researcher

Stanford University

brennali@stanford.edu
Xuhai “Orson” Xu

Xuhai “Orson” Xu

Assistant Professor

Columbia University · Google Research

xx2489@columbia.edu

FAQ

Do I need prior LLM or agent experience?
No. We open with a concise primer, and every module ships a starter notebook with fill-in-the-blanks plus a reference solution. Intermediate Python is the only real prerequisite.
Do I need an API key? Will it cost money?
Not necessarily. For the full LLM-based path, bring either an API key for a supported provider or a compatible local model. If neither is available—or setup fails—an interface-compatible scripted fallback supports the guided exercises, but it does not reproduce open-ended LLM reasoning. Any hosted-provider cost depends on the provider and account; the scripted fallback itself has no API cost.
What do I need on my laptop?
Python ≥ 3.10 and Jupyter. We send detailed setup instructions, including a one-command install script, two weeks before the tutorial.
Do I get to keep the code and data?
Yes. The full stack is released as an open-source repository under the MIT License, together with the synthetic teaching dataset, so you can adapt both for your own research.
Is this clinical or medical training?
No. The tutorial builds research prototypes for sensor-data sensemaking. Outputs are exploratory, not validated medical guidance, and any deployment on humans (including self, family, or research participants) requires appropriate IRB review.

Registration & resources

Links go live closer to the conference. Check back, or email us to be notified.

Registration To be announced on the conference website
Starter repository To be announced
Conference UbiComp/ISWC 2026 · Oct 11 (tentative) · Shanghai
Contact zj2445@cumc.columbia.edu
Related tutorial Multi-agent AI systems with sensing data — a complementary UbiComp/ISWC 2026 tutorial