TRANSFORMATIVE TOOLS, EMERGING CHALLENGES: EMPIRICAL AND PRACTICAL EXPERIENCES WITH LARGE LANGUAGE MODELS FOR TEXT CLASSIFICATION AND ANNOTATION IN COMMUNICATION STUDIES

Resumen

Advancements in Large Language Models have been showing important research opportunities within the field of communication studies. It offers the capacity to conduct large-scale content classification and annotation with low computational expertise and reduced manual coding efforts, potentially allowing more possibilities for researchers in social sciences to explore understudied topics (Bail, 2023; Chang et al., 2024). Because of its functioning and vast domains and language training, LLMS also potentially unlocks more generalizable, complex, and diverse analyses across various communication materials than previous computational tools and approaches (Chang et al., 2024). These materials encompass a wide spectrum, ranging from journalistic content to the digital discourse of political actors and social media conversation threads. At the same time, LLMs also raise important concerns with potential biases, data privacy, models’ transparency, environmental impact, and power imbalances (Jameel et al., 2020; Fecher et al., 2023). Although highly discussed recently, as a recent topic LLMs still need deeper theoretical elaboration and dialogue between empirical investigations specifically for communication scholars (Gil de Zúñiga et al., 2024; Guzman and Lewis, 2020). Our panel assembles a collection of case studies that harness LLMs to tackle text classification and annotation tasks related to media and communication problems, issues, and topics. These research papers engage in an exploration of: (a) pipeline structuring: diverse methodologies for structuring effective pipelines tailored to this form of analysis; (b) tools and models comparison: comparisons of the various LLMs tools and models available for text classification and annotation, highlighting their strengths and weaknesses; (c) optimal variables and tasks: identifying the variables and tasks where LLMs demonstrates exceptional performance and reliability; (d) limitations: discussions on the existing limitations of these tools, including limitations related to specific tasks, variables, languages and data formats; (e) prompt development: strategies for developing, adapting and adjusting prompts that allows better results for specific tasks; and (e) ethical and political dimensions: an examination of the ethical and political considerations inherent in the deployment of LLMs in communication research. This panel puts together valuable efforts of different research groups across the world to not only use, but also reflect on the use of LLMs in Communication studies. They show important avenues for the field to think about different approaches to validity, ethics and truthful cooperation between humans and computational models without erasing the challenges of doing so, and the disagreements - not only between humans, but between humans and their computational language models too.