Google AI Launches SANPO: Multi-Attribute Video Dataset to Enhance Advanced Visual Scene Understanding
-
Google AI recently launched the SANPO dataset, aimed at helping AI models better understand outdoor human-centric scenes. The importance of this dataset lies in the fact that it includes not only real-world data but also synthetic data, along with rich annotations and multi-attribute features. The release of this dataset will contribute to solving problems related to human-centric scene understanding beyond the field of autonomous driving.
The name SANPO derives from its functionalities, encompassing 'Scene understanding, Accessibility, Navigation, Path planning, and Obstacle avoidance.' It consists of real-world and synthetic data, with the real-world data including videos from two stereo cameras, supporting multi-view approaches. The dataset also includes 11.4 hours of video captured at 15 frames per second, featuring dense annotations.
To protect privacy, Google researchers adhered to local, city, and state-level laws during data collection and ensured the removal of any personal information, such as faces and vehicle license plates, before sending the data for annotation.
To address imperfections in video capture, such as motion blur and human scoring errors, the researchers introduced SANPO-Synthetic, a high-quality synthetic dataset designed to match real-world conditions. SANPO-Synthetic includes 1,961 sessions recorded using virtualized Zed cameras, with balanced distributions of head and chest positions.
Another feature of the SANPO dataset is the annotation of panoramic instance masks for a portion of the real-world and synthetic data, assigning a category and ID to each pixel. Unlike real-world data, SANPO-Synthetic contains more instances per frame. Additionally, SANPO is the only dataset that includes both panoramic segmentation and depth image data.
Researchers trained two state-of-the-art models using the SANPO dataset for depth estimation and panoptic segmentation. They found this dataset particularly challenging for dense prediction tasks, with synthetic data outperforming real-world data in accuracy—primarily because real-world environments are more complex while synthetic data offers more precise segmentation annotations.
The introduction of the SANPO dataset fills a gap in the field of human-centric scene understanding. It combines real-world and synthetic data with rich annotations, multi-attribute features, and a unique integration of panoptic segmentation and depth information. Additionally, the researchers' privacy commitments enable this dataset to support other researchers in developing visual navigation systems for the visually impaired and advancing the field of visual scene understanding.
SANPO blog introduction: https://blog.research.google/2023/10/sanpo-scene-understanding-accessibility.html