Crowd-Guided Ensembles: How Can We Choreograph Crowd Workers for Video Segmentation?
Publication
ACM CHI Conference on Human Factors in Computing Systems
Authors
Alexandre Kaspar, Geneviève Patterson, Changil Kim, Yagız Aksoy, Wojciech Matusik, Mohamed Elgharib
Abstract
In this work, we propose two ensemble methods leveraging a crowd workforce to improve video annotation, with a focus on video object segmentation. Their shared principle is that while individual candidate results may likely be insuffi- cient, they often complement each other so that they can be combined into something better than any of the individual results—the very spirit of collaborative working. For one, we extend a standard polygon-drawing interface to allow workers to annotate negative space, and combine the work of multiple workers instead of relying on a single best one as commonly done in crowdsourced image segmentation. For the other, we present a method to combine multiple automatic propagation algorithms with the help of the crowd. Such combination requires an understanding of where the algorithms fail, which we gather using a novel coarse scribble video annotation task. We evaluate our ensemble methods, discuss our design choices for them, and make our web-based crowdsourcing tools and results publicly available.