One frame per second
Person masks, source identities and sound-event labels at a one-second annotation interval.
156 dev recordings · 74 eval recordingsSPATIAL AUDIO × PANORAMIC VISION
Audio-Visual Person Segmentation 360
Audio Research Group
Tampere University, Tampere, Finland
THE TASK
Audio-visual instance segmentation (AVIS) segments and tracks individual sound-producing objects in videos. Models must segment visible persons while they produce sound, suppress their masks during silence even when they remain visible, and preserve their identities when sound production resumes.
Each visible sound-producing person receives one mask per timestamp and retains the same identity across sound-event classes; simultaneous event labels are allowed.
Regions of the same person split across the panoramic seam are annotated as a single instance mask.
A person track links the annotated masks of the same person throughout one recording.
THE DATASET
AVPS-360 is built on STARSS23, a dataset of indoor sound scenes improvised by participants following general instructions and recorded in 23 rooms with a spherical microphone array and a co-located 360° camera. It provides synchronised four-channel first-order Ambisonics, equirectangular video at 1920 × 960 pixels and 29.97 frames per second, and spatiotemporal sound-event annotations.
The dataset captures intermittent sound production in real-world interactions dominated by speech, alongside laughter, clapping, and footsteps.
We keep the STARSS23 partition: its development set becomes our training split and its evaluation set our test split, so no room or recording session is shared between the two.
Equirectangular 360° recordings
AVIS / YTVIS JSON structure
One person can have multiple event labels
Occlusion. For heavily occluded persons whose visible regions cannot be reliably annotated, we retain sound-event labels and source identities but omit masks. These persons are treated as non-targets at those timestamps.
Comparison with video instance segmentation and audio-visual segmentation datasets.
| Dataset | Task | Domain | Video | Audio | #Videos | Anno. FPS | Len. | Total | #Frames | Masks | Cls | Track |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| YouTube-VIS 2019 | VIS | Common | perspective | – | 2,883 | 6 | 4.61 s | 3.69 h | – | 131k | 40 | ✓ |
| YouTube-VIS 2021 | Common | perspective | – | 3,859 | 6 | 5.03 s | 5.39 h | – | 232k | 40 | ✓ | |
| OVIS | Common | perspective | – | 901 | 5–6 | 12.77 s | 3.20 h | – | 296k | 25 | ✓ | |
| AVSBench-O | AVS | Common | perspective | mono | 5,356 | 1 | 5.0 s | 7.4 h | 12,972 | 12,972 | 23 | × |
| AVSBench-S | Common | perspective | mono | 12,356 | 1 | 7.8 s | 26.8 h | 82,972 | – | 70 | × | |
| RefAVS-Bench | Common | perspective | mono | 4,002 | 1 | 10.0 s | 11.1 h | 40,020 | – | 51 | × | |
| AVISeg | AVIS | Common | perspective | mono | 926 | 1 | 61.4 s | 15.8 h | 56,871 | 94,074 | 26 | ✓ |
| AVTrack | Human | perspective | mono | 871 | 1 | 54.0 s | 13.1 h | 47,202 | 36,216 | 1 | ✓ | |
| FAIR-Play (BiAVIS) | Common | perspective | binaural | 1,364 | 1 | 1.0 s | 3.7 h | 13,304 | 19,164 | 8 | × | |
| BiAVIS-Bench | Common | perspective | binaural | 870 | 1 | 1.0 s | 2.2 h | 8,052 | 8,656 | 6 | × | |
| AVPS-360 (1 Hz, ours) | Human | panoramic | FOA | 230 | 1 | 163.7 s | 10.46 h | 37,718 | 31,678 | 1 | ✓ | |
| AVPS-360 (10 Hz, ours) | Human | panoramic | FOA | 230 | 10 | 163.7 s | 10.46 h | 376,537 | 260,015 | 1 | ✓ |
Preview sample videos with person masks annotated at 1 Hz and 10 Hz. Different mask colours represent different people, and each person retains the same colour throughout a sample. In the 360° view, first-order Ambisonics (FOA) audio is rendered binaurally and stays aligned with the scene as you look around. Wear headphones for spatial playback.
10 annotated frames per second · 100 ms annotation interval
Conversation and movement · colours distinguish person instances
Audio preview: omnidirectional W channel.
Development and evaluation labels at 1 Hz and 10 Hz, built on the same STARSS23 recordings.
Person masks, source identities and sound-event labels at a one-second annotation interval.
156 dev recordings · 74 eval recordingsPerson masks, source identities and sound-event labels at a 100 ms annotation interval.
156 dev recordings · 74 eval recordings| Release | Split | Recordings | Frames | Person masks | Person tracks |
|---|---|---|---|---|---|
| 1 Hz | Dev | 156 | 25,424 | 20,734 | 340 |
| 1 Hz | Eval | 74 | 12,294 | 10,944 | 168 |
| 10 Hz | Dev | 156 | 253,899 | 171,268 | 340 |
| 10 Hz | Eval | 74 | 122,638 | 88,747 | 168 |
Development and evaluation annotations at both 1 Hz and 10 Hz are available together in the same Zenodo record.
Download foa_dev.zip, video_dev.zip, foa_eval.zip and video_eval.zip from STARSS23. AVPS-360 provides the additional annotations; recording filename stems connect the two resources.
ACKNOWLEDGEMENTS & CITATION
Please cite the AVPS-360 paper and STARSS23 for the source recordings. Follow the licence terms of each resource.
@misc{sun2026avps360,
author = {Sun, Yifu and Sudarsanam, Parthasaarathy and
Virtanen, Tuomas and Politis, Archontis},
title = {{AVPS-360}: A Spatial Audio-Visual Benchmark for Instance
Segmentation and Tracking of Sound-Producing Persons},
year = {2026},
eprint = {2610.40480},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2610.40480}
}