EMBEDDING INTELLIGENCE IN 3D MODELING WORKFLOWS : AN APPROACH FOR LANGUAGE LANGUAGE MODELS TO ASSIST USERS IN 3D MODELING USING NATURAL LANGUAGE
PI: Felicia Ann Davis (Arts and Architecture)
1 Project Team
This proposal is submitted by a senior interdisciplinary faculty team with expertise spanning design computing, artificial intelligence, and human-computer interaction. The project will be overseen by Dr. Felecia Davis, who will serve as Principal Investigator (PI) and mentor, with support from Dr. José Duarte and Dr. Yubo Kou.
1.1 Principal Investigator and Mentor
Dr. Felecia Davis
Associate Professor, School of Architecture and Landscape Architecture Stuckeman Center for Design Computing, College of Arts and Architecture Director, SOFTLAB
Dr. Davis is a nationally recognized leader in computational design and responsive material systems. Her work integrates architecture, textiles, and computation to explore how design can mediate interaction and perception. As Director of SOFTLAB, she leads research focused on multimodal human-computer interaction and adaptive environments. Dr. Davis will oversee project reporting, guide research development, and serve as the primary mentor for the ICDS Junior Researcher.
1.2 Senior Team Members
Dr. José Pinto Duarte
Professor, School of Architecture and Landscape Architecture Stuckeman Chair in Design Innovation Director, Stuckeman Center for Design Computing Affiliate Faculty, Departments of Architectural Engineering and Engineering Design
Dr. Duarte is an internationally respected scholar in computational design and generative systems. His research contributions span shape grammars, rule-based design, and mass customization in architecture. As Director of the Stuckeman Center for Design Computing, he has led several interdisciplinary research initiatives that apply algorithmic strategies to urban and architectural challenges. His role on the team strengthens the project’s foundation in design theory and computational modeling frameworks.
Dr. Yubo Kou
Assistant Professor, College of Information Sciences and Technology Haile Family Early Career Professor Department of Information Sciences and Technology
Dr. Kou’s research bridges human-computer interaction (HCI), social computing, and computer-supported cooperative work (CSCW). His work explores user engagement, safety, and collaboration in digital environments, with applications in gaming, learning, and online communities. His expertise enhances the project’s user-centered evaluation strategy and contributes methodological depth to the design and assessment of human-AI interaction systems.
2 Relevant ICDS Areas and Affiliated Centers
Artificial Intelligence (AI), Computational Sciences, Data Science
2.1 Relevant ICDS-Affiliated Centers
Center for Artificial Intelligence Foundations and Scientific Applications (CENSAI): This research contributes to CENSAI’s mission by advancing foundational AI techniques, including tool learning, preference optimization, and human-AI collaboration. These methods are applied in architectural and spatial computing contexts. Center for Socially Responsible Artificial Intelligence (CSRAI): This research emphasizes accessibility, inclusivity, and ethical tool design in support of novice users and individuals with disabilities. These values align closely with CSRAI’s commitment to developing AI applications that promote social good and responsible innovation.
3 Level of Effort and Tuition Funding
The ICDS Junior Researcher will be appointed at 50 percent effort (half-time) for two academic semesters. This level of effort reflects the depth, rigor, and interdisciplinary scope of the expected outcomes. It exceeds the recommended minimum of 25 percent and ensures sustained engagement with the project, allowing adequate time for technical development, iterative refinement, and collaborative mentorship. This appointment also supports active dissemination of results through conference presentations, workshops, and ICDS-affiliated initiatives. The Junior Researcher’s tuition will be fully funded by the Department of Architecture.
4 Project Description
This research proposes the development of an AI-augmented interface that enables users to interact with 3D modeling applications through natural language. The project seeks to reimagine how designers engage with complex software systems by embedding large language models (LLMs) within the modeling environment. By allowing users to express design intentions through text or voice, and translating those inputs into executable modeling actions, the system aims to reduce interface complexity and expand access to digital tools.
The research addresses a significant gap in current AI and computational design research: the lack of applied methodologies for adapting LLMs to perform domain-specific modeling tasks. While instruction fine-tuning and tool-calling are common in other technical fields, their structured application in creative modeling workflows remains limited. This proposal defines a flexible research direction to align general-purpose language models with the logic, tools, and semantics of 3D modeling environments. The intended outcome is a working prototype that interprets user intent, selects or calls appropriate tools, and generates accurate and executable outputs.
Rather than prescribing a fixed implementation, the proposal invites ICDS Junior Researchers to contribute original approaches toward this goal. Contributions may include dataset development, implementation of fine-tuning pipelines, or the creation of evaluation methods to assess usability and performance. This research direction encourages critical engagement with the broader question of how language-based interfaces can reshape intelligent systems in creative practice.
4.1 Areas of Expertise Sought
The faculty team seeks an ICDS Junior Researcher with strong foundational skills in artificial intelligence and a demonstrated interest in applying AI methods to design and modeling workflows. The project welcomes applicants from diverse backgrounds including computer science, information sciences, architecture, and design computing.
Minimum expectations include:
(i) Proficiency in Python programming, including experience with Hugging Face Transformers or similar frameworks
(ii) Experience designing and preprocessing training datasets for code generation or domain-specific instruction tuning
(iii) Familiarity with fine-tuning methods and parameter-efficient adaptation strategies such as LoRA
(iv) Understanding of 3D modeling workflows or scripting environments
(v) Interest in user-centered evaluation and mixed-method usability testing
4.2 Project Objectives
The ICDS Junior Researcher will contribute to one or more of the following core objectives:
(i) Scraping and structuring data from Rhino’s Python API documentation for dataset creation
(ii) Generating instruction-response pairs using GPT-based models to simulate user queries and generate scripts
(iii) Implementing supervised fine-tuning with parameter-efficient methods such as LoRA
(iv) Benchmarking model performance on scripting tasks using accuracy, validity, and error metrics
(v) Designing and conducting usability studies with architecture students
(vi) Preparing the project’s technical findings for submission to peer-reviewed conferences (e.g., ACADIA, CAADRIA, AAG)
4.3 Additional Expectations
The ICDS Junior Researcher is expected to actively engage with the faculty team through regular meetings, collaborative reviews, and technical planning discussions. One hour per week will be reserved for structured group meetings. The exact meeting time will be determined based on the availability of all faculty members and the Junior Researcher’s class schedule.
5 Relevance to ICDS’s Interdisciplinary and Computational Research Goals
This research directly aligns with the mission of the Institute for Computational and Data Sciences by advancing interdisciplinary research at the intersection of architecture, artificial intelligence, and human-computer interaction. It applies advanced computational methods, including large language model adaptation and tool integration, to a problem of broad scientific and societal relevance: enabling more inclusive and intuitive interaction with design technologies. By synthesizing domain expertise in computational design with emerging AI techniques, the project embodies ICDS’s commitment to catalyzing cross-disciplinary research that drives innovation and impact.
6 Team Members’ Engagement with ICDS
The research team has a demonstrated record of engagement with ICDS. Dr. Felecia Davis served on the ICDS Coordinating Committee from 2021 to 2023, where she contributed to proposal review and strategic planning. Dr. José Duarte also served on the committee in an earlier term. Both faculty members have played active roles in shaping ICDS research priorities and supporting its interdisciplinary mission.