Semantic Clustering for Visual Data

Abstract

Social media have gone through an overall visual turn. From the text-first nature of former Twitter or the first era of Facebook, we now face platforms that are visual first (if not visual only) both in terms of design and usage. This poses new challenges for researchers that aim at understanding this growing amount of data from a computational or quantitative perspective. Methods developed within the domain of computer vision were developed for tasks (e.g., object recognition, image segmentation), that are of not always of immediate use in research dealing with users or social practices and have thus proved to be of little use. To address the limitations shown by current CNN-based approaches, we propose and evaluate a Visual LLM-based semantic clustering methodology that can capture subtle social and cultural meanings within images, going beyond mere visual or spatial similarities.