DEEP NEURAL NETWORKS FOR SOCIAL VISUALS: STUDYING CLIMATE COMMUNICATION ON YOUTUBE

Abstract

Visual politics is becoming increasingly salient online. The qualitative methods of the research tradition do not expand to complex media ecologies, but advances in deep neural networks open an unprecedented path to large-scale analysis on the basis of actual visual content. However, the analysis of social visuals is challenging, since social and political scenes are semantically rich and convey complex narratives and ideas. This paper examines validity conditions for integrating deep neural network tools in the study of digitally augmented social visuals. It argues that the complexity of social visuals needs to be reflected in the validation process and its communication: It is necessary to move beyond the conventionally dichotomous approach to neural network validation which focuses on data and neural network respectively, to instead acknowledge the interdependency between data and tool. The final definition of good data is not available until the end of the process, which itself relies on a tool that needs good data to be trained. Themes change during the process not just because of our interaction with the data, but also because of our interactions with the tool and the specific way in which it mediates our analysis. An upshot is that the conventional approach of performance assessment – i.e., counting errors – is potentially misleading in this context. We explore our argument experimentally in the context of a study that addresses climate communication on YouTube. Climate themes such as polar bear in arctic landscapes and elite people/events present tough cases of social visuals.