UbiComp/ISWC 2026 · Half-day Tutorial · Shanghai, China

From Multimodal Sensing Data to Actionable Health Insights

Prototyping Your Personal LLM Health Agent

A hands-on tutorial. Build a working personal LLM health agent from a minimal scaffold, over a synthetic, curated multimodal sensing dataset — and turn raw sensor streams into grounded, personalized answers.

Oct 11–15, 2026
Shanghai, China
Half-day (3.5h), hands-on
Bring a laptop · Python ≥ 3.10
Registration TBA Starter repository TBA

“Why have I been sleeping poorly this week?”

Most consumer wearables and health dashboards can show you the data behind that question, but they cannot answer it. LLM agents — language models that call functions, run analyses, and visualize results — offer a path to turn raw multimodal sensing streams into grounded, personalized answers. In this half-day, hands-on tutorial, you will build a working personal LLM health agent from a minimal scaffold, over a synthetic, curated multimodal sensing dataset (sleep, heart rate, activity, GPS, screen time, EMA).

The personal LLM health agent stack participants build during the tutorial, across six modules from introduction to wrap-up.
The personal LLM health agent stack you build across the tutorial's six modules.

What you'll do — and leave with

No prior LLM or agent experience required. We open with a concise primer, then build, wire, and stress-test an agent together.

What you will do

  • Build health-specific tools (data retrieval, analysis, visualization) and register them with an LLM agent.
  • Wire up an agent loop and iterate it on real questions over wearable, smartphone, and self-report data.
  • Examine evaluation, safety, and ethical considerations in a moderated panel with invited guests.

What you will leave with

  • A working LLM health agent running locally on the provided dataset.
  • A runnable codebase and design template you can adapt to your own research.
  • A practical checklist for evaluating LLM agents on faithfulness, safety, and user-alignment.
  • A roadmap for extending the scaffold in your own work.

Schedule

A half-day session of 3.5 hours, alternating short conceptual lectures, guided coding, and a moderated panel. Every module ships with a starter notebook, reference solution, and an optional advanced exercise.

Keynote & Panel

Module 5 features a moderated panel on evaluation and safety, joined by invited guests. Keynote and panel speakers will be announced here as they are confirmed.

Keynote

Keynote

To be announced

Keynote speaker

Panel on Evaluation & Safety

Panelists

To be announced

Invited guests from evaluation, safety, and clinical AI

Prerequisites & what to bring

The tutorial is built to be accessible to participants from non-LLM backgrounds while staying meaningful for those with prior experience.

Prerequisites

  • Intermediate Python: comfortable reading and modifying a Jupyter notebook, installing packages, and debugging API calls.
  • No prior LLM or agent experience required — we open with a concise primer.
  • Familiarity with wearable or behavioral data is helpful but not required; cleaned sample datasets are provided.

What to bring

  • A laptop with Python ≥ 3.10 and Jupyter installed.
  • Either (a) an API key for one of the supported providers (a small free-tier credit option will be available), or (b) the ability to run a small open-weights model locally (e.g., via Ollama).
  • Detailed setup instructions, including a one-command install script, are sent two weeks before the tutorial.

The dataset

You'll work with a synthetic, de-identified teaching dataset whose schema and feature types are inspired by the GLOBEM behavioral dataset, paired with additional synthetic records covering modalities not represented in GLOBEM. No credentialed data access is required during the tutorial, and we provide a data-privacy checklist you can adapt for your own studies.

Sleep Heart rate Physical activity GPS / location Screen time EMA self-reports Context

Organizers

A team spanning multimodal wearable sensing, LLM-based health intervention, conversational agents and clinical workflow evaluation, and human-centered AI for healthcare.

Zhihan Jiang

Zhihan Jiang

Columbia University

zj2445@cumc.columbia.edu
Will Ke Wang

Will Ke Wang

Columbia University

kw3215@cumc.columbia.edu
Blue (Georgianna) Lin

Blue (Georgianna) Lin

Columbia University

gl2981@cumc.columbia.edu
Brenna Li

Brenna Li

Stanford University

brennali@stanford.edu
Xuhai “Orson” Xu

Xuhai “Orson” Xu

Columbia University

xx2489@columbia.edu

FAQ

Do I need prior LLM or agent experience?
No. We open with a concise primer, and every module ships a starter notebook with fill-in-the-blanks plus a reference solution. Intermediate Python is the only real prerequisite.
Do I need an API key? Will it cost money?
Either bring an API key for one of the providers we support (a small free-tier credit option will be available for participants without one), or run a small open-weights model locally via Ollama. Both paths work for the full tutorial.
What do I need on my laptop?
Python ≥ 3.10 and Jupyter. We send detailed setup instructions, including a one-command install script, two weeks before the tutorial.
Do I get to keep the code and data?
Yes. The full stack is released as an open-source repository under the MIT License, together with the synthetic teaching dataset, so you can adapt both for your own research.
Is this clinical or medical training?
No. The tutorial builds research prototypes for sensor-data sensemaking. Outputs are exploratory, not validated medical guidance, and any deployment on humans (including self, family, or research participants) requires appropriate IRB review.

Registration & resources

Links go live closer to the conference. Check back, or email us to be notified.

Registration To be announced on the conference website
Starter repository To be announced
Conference UbiComp/ISWC 2026 · Oct 11–15 · Shanghai
Contact zj2445@cumc.columbia.edu