Anyone who has spent enough time publishing some kind of artistic tools in computer graphics—even in computer vision conferences—has probably gotten reviews that praised a paper’s ideas and results, but required a user study. Or the reviewers didn’t like the paper’s methods or results, but still said the authors should add a user study. And so, maybe you added user studies to your next paper solely because you believed reviewers would require it.

As an author, reviewer, and PC member who works in more “artistic” subjective subfields, I have experienced or witnessed cases like these many times in the past few years. So often, the user study seems unnecessary or even useless, just a waste of everyone’s precious time.

I suspect many studies published in graphics and vision are performative, not informative: we do them not because we want to learn something, but solely because reviewers expect to see a study. So we put on a show, a performance.

Some subfields, including animation and tone-mapping, do have long traditions of rigorous, worthwhile studies, as does the field of HCI. For example, I think Forrester Cole’s work on making and perceiving line drawings are quite significant, high-quality perceptual studies. This post isn’t about those lines of work. This post is about the performative user studies that have become common since the advent of crowdworking in 2008. Crowdworking made gathering numerical scores easy, by allowing researchers to treat human individuals as algorithmic data sources.