AI-Enhanced Dynamic Assessment of ESL Academic Writing
PI: Matthew E. Poehner
While the rapid growth of Generative AI has come to be viewed by many as a threat to academic integrity and teaching and learning, the proposed project, AI-Enhanced Dynamic Assessment of ESL Academic Writing, conceptualizes it as a resource to transform instruction and assessment of international students for whom English is a second language (L2). Like their peers who are native speakers of English, international students are increasingly turning to AI to produce written texts, but they also employ it to correct language errors and provide translations. The situation is rendered more complex by the fact that L2 English international students also must take the Test of English as a Foreign Language (TOEFL), a high-stakes language examination required for admission at most North American universities. Meeting the demands of the TOEFL, which emphasizes academic writing (e.g., producing essays that develop logical, evidence-based arguments after reading or listening to brief informational texts), is a rigorous process that many students spend months preparing for, often by enrolling in programs such as the Intensive English Communication Program (IECP) at Penn State. With the wide range of learner backgrounds and experiences with English, students and instructors alike often struggle to determine the precise needs and difficulties of individuals.
A response to this situation that has met with success is a form of diagnostic language evaluation known as Dynamic Assessment (DA). Informed by Vygotsky’s Sociocultural Theory in psychology, DA provides expanded insight into learner cognitive processes and the problems underlying their language functioning by intervening in the assessment process to offer support (referred to as mediation) when learners experience difficulties. The extent of mediation that learners require (e.g., reminders, a model, feedback) and how learners respond to it (i.e., improving their assessment performance) informs the diagnosis of their emerging linguistic abilities and provides a focus for more tailored instruction. While DA has become well established in the field of applied linguistics, a disadvantage is that procedures are often timeintensive, requiring one-to-one tester-learner administrations. The proposed project is to develop an AI-enhanced DA system that students can access to receive updated diagnoses of their L2 English writing progress, including areas needing improvement.
Project PI Professor Matthew Poehner is a specialist in the Sociocultural Theory of psychology and the diagnostic assessment of language development; Professor Xiaofei Lu, Project Investigator, is a specialist in computational and corpus linguistics. To date, with volunteer support from graduate students in the Department of Applied Linguistics, effort has focused on the feasibility of training a GPT model to identify problems with student writing and employing ChatGPT-4o as a resource to mediate student efforts to revise their essays. An RA supported through the ICDS Junior Researcher Program will enable development of the AI-DA model and a piloting/validation study in preparation for a planned proposal for external funding. Specific tasks at this stage in which the Junior Researcher/RA will be involved include: fine-tuning a base GPT model to assess specific areas (sub-constructs) of student academic writing—such as language use, rhetorical organization, and integration of source material—in line with TOEFL rubrics; conducting prompt engineering within the ChatGPT interface to shift the model’s suggestions and feedback from overt corrections to guiding learner reflection and efforts to revise, an important feature of DA diagnosis; supporting analysis of data to be collected during a planned piloting of the system (including rubric-generated scores of student writing and discourse analysis of student engagement with ChatGPT-4o); and contributing to subsequent system refinement following the pilot.
This work is in preparation for the mid-/long-term goal (anticipating for fall of 2027, depending on calls for proposals) of applying for external funding that will extend the project to include (a) availability of the AI-DA system to ESL learners at targeted institutions and at varied levels of language study/proficiency and (b) training for ESL teachers to interpret assessment results and plan instruction specific to profiles of individual learners. The Educational Testing Services (author of the TOEFL) and the Spencer Foundation have both been identified as potential funding sources.
As can be seen, the AI-DA project is interdisciplinary, residing at the intersection of linguistics, cognitive psychology, and education. Although no ICDS-affiliated centers are relevant to the proposed project, it connects directly to the ICDS Hub/Area of AI. Moreover, the project’s focus on international students, ESL, the TOEFL exam, and AI aligns with the ICDS’s mission to advance multidisciplinary approaches to data science research that addresses issues of societal import.
The AI-DA project calls for a level of effort of 25% time for an RA for 2 semesters, reflecting approximately 8 hours per week devoted to the project and 2 hours per week for ICDS engagement. If the successful candidate is in the Department of Applied Linguistics, their remaining 25% would be covered by the department to support other research projects in the Center for Language Acquisition. Both Professors Poehner and Lu agree to serve as mentors to the ICDS Junior Researcher during their time working on the project, and it is anticipated that when external funding for the larger project is secured, the ICDS Junior Researcher will be invited to continue as part of the research team.
Successful applicants will have a background in applied linguistics or a closely related field, with particular interest in and professional experience with L2 assessment and instructional technologies to support language education. A background that includes working with Generative AI in educational settings is highly desirable. As it is anticipated that the successful applicant will be a doctoral student in Applied Linguistics, their tuition will be funded through that department. Both Professors Poehner and Lu are also tenure-line faculty in the Department of Applied Linguistics.
To date, Professors Poehner and Lu have not yet engaged with the ICDS. However, given the rapidly growing interest in Generative AI in the field of applied linguistics and the impact it is already having on L2 assessment and teaching, their research agendas are expanding to include such work, as evidenced by their recent professional conference presentations. In addition, Poehner has only recently been appointed as Director of the Center for Language Acquisition, and he has included AI-focused research as a priority in the Center’s strategic planning. Given these developments, the Junior Researcher Program will serve as a catalyst for the Center’s involvement with ICDS activities.