Ilmu Komputer & AI editorial
Open AccessOA2023
Qualitative Research Methods for Large Language Models: Conducting Semi-Structured Interviews with ChatGPT and BARD on Computer Science Education
This study evaluates whether qualitative content analysis methods can be applied to interviews with large language models (ChatGPT and BARD) on computer science education. The authors find that while such methods are applicable, the high variability of model responses necessitates careful consideration of contextual factors, and they provide guidelines for conducting such interviews.
Andreas Dengel; Rupert Gehrlein; David Fernes; Sebastian Görlich; Jonas Maurer; H. H. Pham; Gabriel Großmann; Niklas Dietrich genannt Eisermann· Informatics· 2023· DOI 10.3390/informatics10040078
The core problem
Large language models (LLMs) such as ChatGPT and BARD are increasingly used for language translation, text generation, and human-like conversation. These models are trained on vast amounts of data encompassing diverse opinions and perspectives, which raises the possibility of a new qualitative research approach: interviewing LLMs to gain insights into public opinions, potentially mimicking interviews with large groups of subjects. However, it remains unclear whether qualitative content analysis methods can be effectively applied to interviews with these models. This paper investigates the applicability of qualitative content analysis to interviews with ChatGPT in English, ChatGPT in German, and BARD in English, using the relevance of computer science in K-12 education as an exemplary topic. The authors aim to foster understanding of LLMs' abilities and limitations and to provide guidelines for conducting and analyzing interviews with LLMs.
Innovation
The authors conducted semi-structured interviews with three LLM configurations: ChatGPT in English, ChatGPT in German, and BARD in English. The interviews focused on the relevance of computer science in K-12 education. The responses were analyzed using qualitative content analysis methods. The study examined how contextual factors influence the responses and whether established qualitative research methods are applicable to LLM interviews. The authors also documented the process to derive guidelines for conducting and analyzing interviews with LLMs.
Introduction
Large language models (LLMs) such as ChatGPT and BARD are increasingly used for language translation, text generation, and human-like conversation. These models are trained on vast amounts of data encompassing diverse opinions and perspectives, which raises the possibility of a new qualitative research approach: interviewing LLMs to gain insights into public opinions, potentially mimicking interviews with large groups of subjects. However, it remains unclear whether qualitative content analysis methods can be effectively applied to interviews with these models. This paper investigates the applicability of qualitative content analysis to interviews with ChatGPT in English, ChatGPT in German, and BARD in English, using the relevance of computer science in K-12 education as an exemplary topic. The authors aim to foster understanding of LLMs' abilities and limitations and to provide guidelines for conducting and analyzing interviews with LLMs.
Why it matters
Based on the results, the authors formulated guidelines for conducting and analyzing interviews with LLMs. They suggest that qualitative content analysis methods can be applied, but with careful consideration of contextual factors that may affect responses. The guidelines aim to aid researchers in conducting more nuanced and insightful interviews. However, from an overall view, the authors generally do not recommend using interviews with LLMs for research purposes due to their highly unpredictable results. Instead, they suggest using LLMs as exploration tools for gaining different perspectives on research topics and for testing interview guidelines before conducting real-world interviews. This approach can help researchers refine their methods and explore diverse viewpoints without relying solely on LLM outputs for definitive conclusions.
Who should read this
Pembaca Research Digest dan praktisi rumpun ini.
Opening member content…