SPATIAL AUDIO × PANORAMIC VISION

AVPS-360

Audio-Visual Person Segmentation 360

Yifu Sun · Parthasaarathy Sudarsanam ·
Tuomas Virtanen · Archontis Politis

Audio Research Group
Tampere University, Tampere, Finland

GROUND-TRUTH PERSON MASKS / 10 Hz PREVIEW
230recordings
10.46 hof panoramic recordings
4 channelsfirst-order Ambisonics audio
1 & 10 Hzannotation releases

THE TASK

Task definition

Audio-visual instance segmentation (AVIS) segments and tracks individual sound-producing objects in videos. Models must segment visible persons while they produce sound, suppress their masks during silence even when they remain visible, and preserve their identities when sound production resumes.

01

Sound-producing persons

Each visible sound-producing person receives one mask per timestamp and retains the same identity across sound-event classes; simultaneous event labels are allowed.

02

Panoramic instances

Regions of the same person split across the panoramic seam are annotated as a single instance mask.

03

Recording-level identities

A person track links the annotated masks of the same person throughout one recording.

THE DATASET

The AVPS-360 dataset

AVPS-360 is built on STARSS23, a dataset of indoor sound scenes improvised by participants following general instructions and recorded in 23 rooms with a spherical microphone array and a co-located 360° camera. It provides synchronised four-channel first-order Ambisonics, equirectangular video at 1920 × 960 pixels and 29.97 frames per second, and spatiotemporal sound-event annotations.

The dataset captures intermittent sound production in real-world interactions dominated by speech, alongside laughter, clapping, and footsteps.

We keep the STARSS23 partition: its development set becomes our training split and its evaluation set our test split, so no room or recording session is shared between the two.

VIDEO1920 × 960

Equirectangular 360° recordings

ANNOTATIONSMasks + identities

AVIS / YTVIS JSON structure

EVENTS5 sound classes

One person can have multiple event labels

Female speechMale speechClappingLaughterFootsteps

Occlusion. For heavily occluded persons whose visible regions cannot be reliably annotated, we retain sound-event labels and source identities but omit masks. These persons are treated as non-targets at those timestamps.

Dataset comparison

Comparison with video instance segmentation and audio-visual segmentation datasets.

Comparison of segmentation datasets. Len. is the mean recording duration; #Frames counts annotated frames; Cls counts object classes; Track indicates persistent instance identities.
DatasetTaskDomainVideoAudio#VideosAnno.
FPS
Len.Total#FramesMasksClsTrack
YouTube-VIS 2019VISCommonperspective–2,88364.61 s3.69 h–131k40✓
YouTube-VIS 2021Commonperspective–3,85965.03 s5.39 h–232k40✓
OVISCommonperspective–9015–612.77 s3.20 h–296k25✓
AVSBench-OAVSCommonperspectivemono5,35615.0 s7.4 h12,97212,97223×
AVSBench-SCommonperspectivemono12,35617.8 s26.8 h82,972–70×
RefAVS-BenchCommonperspectivemono4,002110.0 s11.1 h40,020–51×
AVISegAVISCommonperspectivemono926161.4 s15.8 h56,87194,07426✓
AVTrackHumanperspectivemono871154.0 s13.1 h47,20236,2161✓
FAIR-Play (BiAVIS)Commonperspectivebinaural1,36411.0 s3.7 h13,30419,1648×
BiAVIS-BenchCommonperspectivebinaural87011.0 s2.2 h8,0528,6566×
AVPS-360 (1 Hz, ours)HumanpanoramicFOA2301163.7 s10.46 h37,71831,6781✓
AVPS-360 (10 Hz, ours)HumanpanoramicFOA23010163.7 s10.46 h376,537260,0151✓

Sample videos

Preview sample videos with person masks annotated at 1 Hz and 10 Hz. Different mask colours represent different people, and each person retains the same colour throughout a sample. In the 360° view, first-order Ambisonics (FOA) audio is rendered binaurally and stays aligned with the scene as you look around. Wear headphones for spatial playback.

Ground-truth annotations
10 Hz

10 annotated frames per second · 100 ms annotation interval

Conversation and movement · colours distinguish person instances

Audio preview: omnidirectional W channel.

Download annotations

Development and evaluation labels at 1 Hz and 10 Hz, built on the same STARSS23 recordings.

AVPS-3601 Hz

One frame per second

Person masks, source identities and sound-event labels at a one-second annotation interval.

156 dev recordings · 74 eval recordings
AVPS-360-HFR10 Hz

10 frames per second

Person masks, source identities and sound-event labels at a 100 ms annotation interval.

156 dev recordings · 74 eval recordings
Development and evaluation annotation statistics
ReleaseSplitRecordingsFramesPerson masksPerson tracks
1 HzDev15625,42420,734340
1 HzEval7412,29410,944168
10 HzDev156253,899171,268340
10 HzEval74122,63888,747168

Development and evaluation annotations at both 1 Hz and 10 Hz are available together in the same Zenodo record.

Get the source audio and video

Download foa_dev.zip, video_dev.zip, foa_eval.zip and video_eval.zip from STARSS23. AVPS-360 provides the additional annotations; recording filename stems connect the two resources.

STARSS23 recordings

ACKNOWLEDGEMENTS & CITATION

Citation

Please cite the AVPS-360 paper and STARSS23 for the source recordings. Follow the licence terms of each resource.

@misc{sun2026avps360,
  author    = {Sun, Yifu and Sudarsanam, Parthasaarathy and
               Virtanen, Tuomas and Politis, Archontis},
  title     = {{AVPS-360}: A Spatial Audio-Visual Benchmark for Instance
               Segmentation and Tracking of Sound-Producing Persons},
  year      = {2026},
  eprint    = {2610.40480},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  url       = {https://arxiv.org/abs/2610.40480}
}