Evaluating Generative AI Tools for Qualitative Analysis
PI: Tim Brick (Human Development and Family Studies)
College of HHD has guaranteed tuition for CHHD students at 25% time on-project (and 25% ICDS/RISE) only.
Other Senior Team Members:
Sunny Bai, Assistant Professor, Ballmer Institute, University of Oregon; Sebrina Doyle-Fosco, Assistant Professor of Human Development and Family Studies, PSU; Christopher Skurka Assistant Professor of Media Studies, Bellisario College of Communications. The PI will serve as the mentor for the junior researcher.
The goal of this project is to develop a pipeline that can leverage zero-shot and few-shot learning with Retrieval Augmented Generation (RAG) in Large Language Models (LLMs) to partially automate qualitative coding of conversational transcript data. Manual coding of video data remains a primary means of data processing in the behavioral science. The complex nature of many of the coded behaviors (e.g. “providing emotional support”) makes it difficult to process using conventional methods. Initial work has shown some success applying LLMs to speed this type of coding, but little systematic work has been done to evaluate its effectiveness in different domains. By leveraging a suite of available data sets across different domains and coding schemata, this project will provide a proof-of-concept for a grant to develop assistive, private tools for processing data that would otherwise require expensive manual coding, and an evaluation of effectiveness of several LLMs to several domains to identify where work is most needed. Importantly, this project will also identify areas where LLMs are insufficient for this type of approach.
The objectives of this project are: (1) Local / Private LLM configuration: create a Roar Collab pipeline for applying zero-shot and few-shot prompts to data sets across different publicly available Large Language Models. (2) Data Preprocessing: Create a pipeline to ingest transcript data and coding scheme data and process it into appropriate inputs (e.g. vector encodings, RAG database, prompt sets, etc.) for the LLM pipeline. If time allows, this pipeline may be extended with LLM tools for audio transcription and diarization and/or video processing. (3) Evaluate performance of LLMs on a variety of tasks on different data sets relative to human coding results and predictive outcomes. A number of data sets are available with different levels and complexities, as well as different predictive outcomes. Available datasets include:
Adolescent risk factors: Transcripts of parent-adolescent interactions coded for emotional and informational support and linked to a variety of child outcomes.
Educational Interactions: Transcripts of student-teacher interactions coded for a variety of educator characteristics.
Focus groups: Data from focus groups examining information search and presentation as it relates to adoption of novel pharmaceuticals.
The anticipated outcomes of this project include a pipeline for ingestion and processing of transcript data for automated qualitative coding; evaluations of LLM-based qualitative coding prompts across several domains; and identification of domains in which LLMs appear to be most promising for qualitative coding work.
A list of specific areas of computational and/or data science expertise or skills that the current team is particularly interested in recruiting to support the project: Text processing, generative AI / RAG operation, prompt engineering. At least one programming language appropriate to genAI operation (probably Python, but others accepted).
Any other requirements or expectations of potential ICDS Junior Researchers: Tuition support only available for students in the College of Health and Human Development. Parts of the project may be available with a smaller time commitment (e.g. less time or summer 2026 only) and no tuition support for students from other colleges.
A list of specific objectives for work supported by this call: generating proof-of-concept and pilot data to support a NIH proposal; at least one paper submitted to a quantitative methods journal; likely also 1-2 collabroative substantive publications in behavioral science / education domains.
At least one medium to long-term goal: A collaborative proposal to NIMH call for Innovative Mental Health Services Research Not Involving Clinical Trials (PAR-25-283; Submission Feb. or June 2026) or CTSI Limited Competition: High Impact Specialized Innovation Programs in Clinical and Translational Science (PAR-25-156; Submission May 2026).
A short statement (1 sentence to 1 paragraph) explaining the connection of the project to ICDS’s mission: The project falls right into the ICDS mission of advancing AI in research and education.
A paragraph summarizing team member’s recent and/or planned engagement with ICDS: Timothy Brick is an ICDS co-hire and regularly participates in ICDS activities and committees, such as monthly lunches, faculty search committee, and annual ICDS symposia, and a faculty representative for ICDS and CHHD on the Chief Information Security Officer’s advisory board. He was recently appointed by the SSRI as a liaison with the ICDS for HPC development.