TY - GEN
T1 - Adversarial Imagery
T2 - Extended Abtracts of the 2026 CHI Conference on Human Factors in Computing Systems, CHI 2026
AU - Valente, Pedro
AU - Domingues, João
AU - Correia, Nuno
AU - Magalhaes, Joao
AU - Nóbrega, Rui
N1 - Publisher Copyright:
© 2026 Copyright held by the owner/author(s).
PY - 2026/4/13
Y1 - 2026/4/13
N2 - As generative AI models become increasingly sophisticated, effective human-centered evaluation methods are essential for responsible development. We present AdversarialImagery, a gamified annotation tool designed to collect human assessments of AI-generated images through competitive gameplay. Our system features three game modes (plus timed variants) that mirror different real-world scenarios: single image classification (G1), direct comparison between real and AI-generated (G2), and multi-image assessments (G3). The tool incorporates gamification elements, including rankings, qualitative feedback, and achievements, to sustain engagement while maintaining annotation quality. Through a formative study (N = 22) and public deployment (N = 129, 7,760 labels), we demonstrate that tutorial design significantly impacts accuracy and reveal systematic patterns: human features are consistently easier to detect, while comparison modes reduce deception rates by up to 60%. This work validates gamification as a viable approach for large-scale generative AI evaluation and provides practitioners with an empirically validated flexible tool for human-centered evaluations of image-to-image generative AI.
AB - As generative AI models become increasingly sophisticated, effective human-centered evaluation methods are essential for responsible development. We present AdversarialImagery, a gamified annotation tool designed to collect human assessments of AI-generated images through competitive gameplay. Our system features three game modes (plus timed variants) that mirror different real-world scenarios: single image classification (G1), direct comparison between real and AI-generated (G2), and multi-image assessments (G3). The tool incorporates gamification elements, including rankings, qualitative feedback, and achievements, to sustain engagement while maintaining annotation quality. Through a formative study (N = 22) and public deployment (N = 129, 7,760 labels), we demonstrate that tutorial design significantly impacts accuracy and reveal systematic patterns: human features are consistently easier to detect, while comparison modes reduce deception rates by up to 60%. This work validates gamification as a viable approach for large-scale generative AI evaluation and provides practitioners with an empirically validated flexible tool for human-centered evaluations of image-to-image generative AI.
KW - AI Evaluation
KW - Games with a Purpose
KW - Generative AI
KW - Human-Centered AI
UR - https://www.scopus.com/pages/publications/105038097604
U2 - 10.1145/3772363.3799274
DO - 10.1145/3772363.3799274
M3 - Conference contribution
AN - SCOPUS:105038097604
T3 - Conference on Human Factors in Computing Systems - Proceedings
BT - CHI 2026 - Extended Abtracts of the 2026 CHI Conference on Human Factors in Computing Systems
A2 - Oliver, Nuria
A2 - Shamma, David A.
A2 - Candello, Heloisa
A2 - Cesar, Pablo
A2 - Lopes, Pedro
A2 - Artizzu, Valentino
A2 - Draxler, Fiona
A2 - Lopez, Gustavo
A2 - Reinschluessel, Anke V.
A2 - Tong, Xin
A2 - Toups Dugas, Phoebe O.
PB - ACM - Association for Computing Machinery
Y2 - 13 April 2026 through 17 April 2026
ER -