Algorithmic Affidavits and Automation Bias: Empirical Evaluation of Generative AI in Police Report Writing
PI: Dana Calacci (IST)
Principal Investigator (PI)
Dr. Dana Calacci, assistant professor in the College of Information Sciences and Technology (IST) and an ICDS Co-hire, studies the socio-technical and legal impacts of datafication and AI on communities, especially worker groups. Through collaborations like the Workers Algorithm Observatory, which she co-directs, she designs and deploys technologies with communities that aim to answer their most pressing questions about the impact of AI, new platforms, and surveillance on their lives. Dana received her PhD from MIT’s Media Lab in 2023, and a B.S. in computer science from Northeastern University in 2015. She also has experience as a startup co-founder and a mixed-media artist. Her writing and work has appeared or been featured in NPR’s Radiolab, Gizmodo, Wired, Reuters, The Atlantic’s CityLab, the New York Times, and other major publications.the New York Times, and other major publications.
Dr. Calacci will serve as mentor for the Junior Researcher.
Other Senior and Junior Team Members
Nasser Eledroos – Project Co-lead Nasser Eledroos is a public interest technologist focused on racial equity in technology and justice systems. He currently serves as Policy Strategist for Color Of Change, where he leads efforts to design and implement technology policy to protect Black people’s rights in the U.S. at the Federal and State levels.
Previously, Nasser was Managing Director at Northeastern University School of Law’s Center for Law, Information and Creativity (CLIC). He is a Senior Fellow with Atlantic Fellows for Racial Equity at Columbia University and the Nelson Mandela Foundation. He holds a Global Fellow with The Atlantic Institute at Rhodes Trust, University of Oxford, where he is working on the future of Extended Reality (XR) social justice storytelling.
His prior roles include Senior Staff Technologist at the Suffolk County District Attorney’s Office, and Technology Fellow at the ACLU of Massachusetts. At the DA’s Office, he developed data-driven approaches to reduce racial disparities in the criminal legal system under the direction of former-D.A. Rachael Rollins. At the ACLU, he investigated surveillance systems and addressed misconduct in the Massachusetts drug lab scandal.
Nasser has served on the board of the Muslim Justice League (2022-2024), and the Steering Committee for the Ford Foundation’s Public Interest Technology alumni network (2019-2021, 2024).
Tianqi Kou – Graduate Researcher & Candidate for Junior Researcher Tianqi Kou is a final PhD candidate in the Penn State College of IST and is co-advised by Dr. Dana Calacci and Dr. Fred Fonseca. Tianqi specializes in AI ethics and history and feminist philosophy of science, with recent work in the politics of replicability.
Maya Nagiub – Undergraduate Researcher Maya Nagiub is a current third-year student at Penn State University. Maya will graduate from the College of Information Sciences and Technology in May 2026 with a B.S. in Applied Data Sciences with an application focus in Cybersecurity.
Team’s History of Interdisciplinary Engagement
As demonstrated by the publication history of the PI and other team members, the research team has extensive experience leading projects that cross discipline boundaries and engage expertise from diverse fields such as information science, law and law enforcement, critical algorithm studies, social sciences, human computer interaction (HCI), AI, and data stewardship and rights.
ICDS Engagement
As an ICDS co-hire, Dr. Calacci regularly engages with ICDS, including through providing content for the ICDS Staff Newsletter (March 2025) and collaborating with and engaging services from the Research Innovations with Scientists and Engineers (RISE) team and Project Management Office (PMO). Recently, Dr. Calacci lectured as part of the AI for Social Impact” series presented by the Center for Socially Responsible AI (CSRAI). Dr. Calacci’s talk, titled “How AI is Reshaping Your Paycheck: Personalized Wages in the Inference Economy,” discussed how companies are using AI to change how they set pay.
Ability to Meet Expectations for PI/Team Sponsoring Junior Researcher
Dr. Calacci, Nasser, and other team members are prepared to meet all expectations outlined in the ICDS Junior Researcher Call.
Project Description and Objectives
Project Description
This project investigates the full lifecycle of hype surrounding generative AI technologies in policing—from exaggerated capability claims by vendors, to procurement by law enforcement agencies, to courtroom use of AI-generated evidence. The first aim is to map how marketing narratives and techno-solutionist logics shape the adoption of these tools and obscure their risks. We will analyze documents (e.g., press releases, contract bids, RFPs, internal reports) and conduct interviews with stakeholders (e.g., vendors, oversight bodies, law enforcement), using thematic analysis and other qualitative methods to identify hype patterns and perceptions of AI across law enforcement.
The second aim is to empirically evaluate the technical performance of specific generative AI tools used in police report writing. Using open-source and commercial systems, we will reconstruct and benchmark automated report generation pipelines. These will be stress-tested across linguistic, demographic, and environmental variables to quantify hallucination rates, transcription errors, and bias—major concerns of law enforcement and the public at large.
Together, these threads will help make the relationship between performance claims by AI providers and their real-world abilities clear, contributing important evidence to a national debate on AI tools in policing infrastructure.
Specific Objectives for Work
Hype Lifecycle
● Map the communication and adoption pipeline for 2–3 AI policing tools (e.g., Axon’s “Draft One”).
● Collect and analyze artifacts: vendor press materials, RFPs, internal department memos, court filings, etc.
● Conduct interviews with stakeholders across the AI policing “supply chain.”
● Use document and discourse analysis to trace how inflated claims are produced, circulated, and institutionalized.
Empirical Evaluation
● Build AI pipelines simulating vendor tools using models like Whisper and GPT-4.
● Benchmark performance across dialects (AAVE, accented English) and noisy audio environments.
● Measure hallucination rates, speaker misattribution, and rights-related omissions in generated reports.
● Compare observed performance to marketing claims to identify gaps.
Medium to Long-term Goal(s)
● Develop a generalizable framework for identifying and intervening in AI hype cycles in the public sector.
● Create public benchmarks and reproducible audits for generative AI used in high-risk institutional settings.
● Produce policy-relevant outputs (e.g., white papers, amicus briefs) on evidentiary standards and procurement reform.
● Lay the groundwork for a major external proposal. This is likely to be a non-profit funder, such as the Mozilla Foundation, Ford Foundation, etc).
● Publish empirical and theoretical findings in venues such as FAccT, CSCW, or CHI
Realistic Alignment of Effort/Objectives and Realistic Expectations of Junior Researchers
This is a large and ambitious project, and the Junior Researchers’ role will be scoped. The Junior Researcher will contribute ~10 hours/week over one semester (25% RA), focused on one of or a combination of: (a) qualitative data collection, including interviews or artifact collection; (b) data scraping and preprocessing collected data for the hype analysis or (b) scripting and evaluating model performance in the empirical pipeline. These tasks will be scoped to yield preliminary data and reproducible code/products within one term, with outputs feeding directly into upcoming publications and grant proposals. Both threads provide meaningful opportunities for technical and interdisciplinary training while advancing ICDS’s mission in AI accountability and computational social impact. They also provide scoped, clear research projects for an early-career researcher.
Alignment with Junior Researcher Call & ICDS Mission
ICDS Mission
This project aligns directly with ICDS’s mission of advancing data-driven, computationally intensive research for public impact. We plan to engage with ICDS within this project by sharing findings through ICDS events and online (talk, blog post, symposium, etc). We also hope to contribute a curated dataset or other artifacts that can aid reproducibility and other research, which could be hosted on ICDS infrastructure. Model evaluation can be done on ROAR. We will encourage the Junior Researcher to participate in ICDS seminars, events, and workshops, including the annual symposium.
Relevant ICDS Hub/Area
This work falls within the ICDS hub/area of Artificial Intelligence.
ICDS Center Affiliation
This work is not directly affiliated with a specific ICDS center.
Interdisciplinary Basis of Project
This project combines perspectives from computer science (NLP, ASR, model evaluation), law and policy (Fourth Amendment, evidentiary standards), and communication/STS (hype analysis, vendor narratives, and institutional decision-making). The research team includes expertise from information science, law, critical algorithm studies, and former law enforcement professionals (Nasser Eledroos, ex-CTO at the Boston DA’s office).
Funding
Level of Effort & Tuition Funding Plan
The Junior Researcher will dedicate 25% effort to this project during the academic year and 100% effort to the project during Summer 2026. All tuition will be covered from the PI’s department startup funds.
Intended Pay Grade
The intended graduate student pay grade for the Junior Researcher is Grade 19.
Expectations of Junior Researcher
Expertise or Skills Sought
The Junior Researcher should possess experience in one or more of the following areas.
Hype Lifecycle / Discourse Analysis:
● Familiarity with qualitative coding frameworks (e.g., grounded theory, content analysis)
● Computational text analysis (e.g., topic modeling, TF-IDF, clustering) using Python or R
● Document scraping and preprocessing (e.g., PDFs, web content, OCR tools)
● Prior work or coursework in science & technology studies (STS), critical data studies, or communication research is a plus
Text-to-speech AI Evaluation:
● Natural language processing (NLP), particularly with transformer-based models (e.g., GPT-4, Claude, etc.)
● Some experience with automatic speech recognition (ASR) and familiarity with tools like Whisper or similar
● Python-based data science pipelines, including libraries like HuggingFace and scikit-learn
● Demonstrated text mining skills
● Experience with bias auditing or model evaluation is a plus
● Comfort with reproducible workflows and version control (Git)
General Expectations
For the duration of the Junior Researcher project, the Junior Researcher will allocate their time between research to support their dissertation, the ICDS Junior Researcher work outlined in this proposal, and regular engagement with ICDS (e.g. ICDS Symposium, talk series, lunches, and other events).
As outlined in the Junior Researcher Call, the Junior Researcher will submit a mid-semester project report and a ~1 page written report describing progress towards the proposed objectives and goals for each project/opportunity within two weeks of the end of their ICDS-supported appointment.
To the best of their ability and as schedules allow, the Junior Researcher should plan to participate in a one-hour meeting of Dr. Calacci’s lab group once per week.