What Can We Learn from Sketching Through Data in Virtual Reality?

Teaser image for the project: An image of trajectories of particles of a flotation simulation visualized as blue-green-yellow ribbons.

Dense 3D motion data is difficult to analyze with 2D tools— clutter and occlusion often hide important patterns, while automated methods like clustering offer limited flexibility for exploration. Shape-based similarity matching addresses this by highlighting trajectories with similar geometry, reducing clutter and guiding attention to relevant motion [1]. Building on prior work showing that VR sketching effectively captures conceived shapes in design tasks [2], we use it here as an intuitive way to define trajectory queries. We explored this approach in analysis sessions with a domain expert in flotation.

Takeaways from Our Analysis Session

Immersive sketching supports visual hypothesis exploration — simple planar sketches often serve as a starting point to retrieve complex 3D motion patterns, which can then be refined by sketching 3D paths or selecting trajectory segments from the results.

Videos

Preview Video

Exploration Scenario

Downloads

Research Paper

2-page conference paper (PDF)

Download Paper

Poster

Full-size poster (PDF)

Download Poster

Your feedback is welcome and valued

Franziska Kahlert

Phd-student

Chair of Computer Graphics and Visualization at TUD Dresden University of Technology and ScaDS.AI

View University Profile →

Diving deeper into the approach

Challenging Data

An image of trajectories of particles of a flotation simulation visualized as blue-green-yellow ribbons.

An image of trajectories of bubbles of a flotation data set visualized as blue-green-yellow colored ribbons together with spheres representing the bubbles.

Dataset Description

The 3D motion dataset is created through a numerical flotation simulation [3] with 560 time steps over 0.5 seconds leading to 4 million particle and 135 bubble trajectories.

Typical Analysis Tasks

Task 1: Characterize typical bubble motion.
Task 2: Identify common particle motion patterns.
Task 3: Locate regions of similar particle motion.
Task 4: Analyze particle-bubble interactions.

Our Immersive Tool

We developed an immersive analysis tool for exploring 3D trajectory data using sketch-based retrieval. Users draw motion patterns in space using a VR controller, and the system retrieves similar segments from a dataset based on shape similarity.

A screenshot of the analysis tool showing colored ribbons for the trajectories and spheres for the particles and bubbles.

Visualizing trajectories as colored ribbons and particles as textured spheres to illustrate orientation.

A screenshot of the analysis tool showing a sketched trajectory in grey and the VR controller while sketching.

Sketching motion patterns directly in 3D space using VR controllers.

A screenshot of the analysis tool showing the VR controller selecting a green colored segment of a trajectory through a wireframe indicator.

Selecting a segment from sketched or existing trajectories to create a query.

A screenshot of a retrieved trajectory segment colored in green with the not matching part of the trajectory colored in grey.

Retrieving trajectories with symbolic encoding and edit-distancebased matching [1].

A Real Exploration Scenario from Our Use Case

The expert began with a planar C-shaped sketch, based on a 2D drawing of their expectation that particles would move around rising bubbles (T4). The initial retrieval confirmed a few matching patterns.

A figure showing a 2D illustration of the expectation of particle movements around bubbles resulting in lines around a circle with some ending outside and some on the circle.

Expectations illustrated in 2D

An image showing a C-shaped sketch colored in purple in the immersive environment.

Planar sketch for C-shaped path around bubble

An image showing few retrieval results for the C-shaped sketch colored in purple.

Retrieval result for C-shaped path around bubble

To explore more complex motion, the developer—more experienced with 3D input—refined the planar sketch into a spiral-like path around the bubble utilizing the full 3D space. This revealed additional instances and even regions with multiple occurrences (T3). Time-encoded visualization showed that these motions were indeed caused by rising bubbles but were obscured in the dense dataset.

An image showing a 3D sprial-like sketch colored in brown in the immersive environment.

3D Sketch for spiral-shaped path around bubble

An image showing multiple retrieval results for the 3D spiral-like sketch colored in brown in the immersive environment.

Retrieval result for spiral-shaped path around bubble

An image showing a detail of the retrieval results of the 3D spiral-like sketch colored in brown in context of the trajectories of the bubbles which are represented by spheres.

Detailed views of retrieval result with bubble trajectories.

An image showing a detail of the retrieval result for the 3D spiral-like sketch using the color to encode time on the trajectories of the bubbles and particles together with spheres representing bubbles.

Detailed views of retrieval result with time encoding.

An image showing the detail of the flotation dataset without any filtering applied thus occluding the previously visible retrieval results.

Detailed views of retrieval result with unfiltered dataset.

This shows how 3D sketching uncovers spatial patterns missed with planar input.

More Retrieval Results

3D Spiral inticating turbulences.

V-shape with long descent representing falling of particle and getting caught in wake of the bubble.

V-shape with long ascent indicating particle getting caught in the wake of the bubble traveling a longer than expected distance in the wake before colliding.

What We Already Learned

Sketching revealed interpretable motion types: C-shapes around bubbles, spirals indicating turbulence and V-shapes for wake entry
Results aligned with domain expectations, supported hypothesis exploration and revealed new insights
The expert mainly used planar sketches, likely reflecting familiarity with 2D motion illustrations, and refined queries with segments from retrieved trajectories

What Still Needs Work

Improve the precision of mid-air sketching
Add adaptability to the retrieval algorithm (e.g. direction of motion)
Improve result interpretation by visualizing similarity scores

References

[1] F. Kahlert and S. Gumhold. Partial Matching of Trajectories with Particle Orientation for Exploratory Trajectory Visualization. 2020. https://doi.org/10.2312/vmv.20201193

[2] S. Monty et al. Analysis of Immersive Mid-Air Sketching Behavior, Sketch Quality, and User Experience in Design Ideation Tasks. 2024. https://doi.org/10.1109/ISMAR62088.2024.00041

[3] B. Tiedemann and J. Fröhlich. Collision dynamics of particles and bubbles in gravity-driven flotation: A DNS investigation. 2023. https://doi.org/10.1002/pamm.202300290