Sample Research Project

The following is a sample project proposal for an independent research experience that we have developed after your initial proposal.

Note

This specific type of project discusses the possible usage of actual student data and would require Institutional Review Board (IRB) approval. A project including human subjects and/or their data would need to have to be proposed the semester prior to the start of the project in order to allow time for the IRB process.

Generative AI for Scalable Feedback in Professional Computing Education

Instructor: Lucas P. Cordova, Ph.D.
Email: lpcordova@willamette.edu
Term: Summer 2025 (11 Weeks Undergrad or 14 Weeks Grad)
Credit Hours: 4

Project Focus

This independent graduate research experience focuses on the application of large language models (LLMs), specifically GPT-4, to improve scalability and quality in formative feedback for professional and workplace-integrated computing education. The student will evaluate generative AI’s role in augmenting instructional feedback at scale—particularly for code correctness, style, and documentation—and produce dissemination-ready deliverables suitable for academic or industry-facing venues.

Target Outcome

The project will culminate in the production of a conference-ready paper, industry white paper, or field-specific dissemination resource, with a potential submission target including venues such as SIGCSE, IEEE EDUCON, or practitioner conferences related to software engineering education and workplace learning.

Research Objectives

  • Evaluate the utility of generative AI feedback systems for professional computing learners
  • Compare GPT-4 feedback to human reviewer standards (e.g., TAs, instructors, peer mentors)
  • Investigate how prompt engineering and rubric alignment affect feedback quality and relevance
  • Identify limitations or ethical challenges in applying LLMs in high-stakes or career-critical environments

Background Readings

  • “Large Language Models as Programming Assistants” (Chen et al.)
  • “Automated Feedback in Computer Science Education” (Piech et al.)
  • “The Effect of AI Feedback on Learning in Online Programming Environments” (recent CHI or L@S papers)
  • Industry white papers on AI-based developer tools (e.g., GitHub Copilot, Amazon CodeWhisperer)

Preliminary Assignment

Reconstruct a baseline automated feedback pipeline using GPT-4 on real or simulated student code submissions. Annotate and evaluate generated feedback using a rubric derived from professional coding standards and educational best practices.

Timeline and Deliverables

The project is based on conducting research during the Summer semester and will be divided into 11 weeks for undergraduate students or 14 weeks for graduate students. The timeline below outlines the major phases of the project, including key milestones and deliverables. A graduate version of the timeline will include additional weeks for literature review, dataset definition, and analysis.

Weeks 1–2: Contextualization and Literature Review

  • Identify a target dissemination venue (conference, journal, white paper audience)
  • Review literature on AI-generated feedback in education and industry
  • Explore empirical methods used in LLM studies and feedback evaluation

Weeks 3–4: Dataset Definition and Evaluation Criteria

  • Select or simulate a dataset of professional-quality code submissions
  • Define rubrics and evaluation metrics (clarity, correctness, tone, etc.)
  • Set up annotation tools or benchmarking criteria

Weeks 5–7: Implementation and Feedback Generation

  • Design and refine GPT-4 prompts for feedback generation
  • Run experiments across different coding contexts and prompt strategies
  • Collect and log feedback output, including anomalies or inconsistencies

Weeks 8–9: Analysis and Interpretation

  • Conduct thematic analysis and comparative evaluation of feedback
  • Identify trends, strengths, and weaknesses in generative output
  • Evaluate against human feedback if available

Weeks 10–11: Synthesis and Dissemination

  • Draft paper, poster, or industry report with figures and analysis
  • Prepare slides and present to faculty, at TechBytes, or a relevant audience
  • Revise deliverables based on feedback and prepare for submission

Learning Outcomes

  • Design and execute an applied research study with professional relevance
  • Gain experience in field-appropriate dissemination and communication
  • Develop evaluation frameworks for AI-in-the-loop tools in computing education
  • Contribute to the understanding of responsible AI use in real-world learning contexts

Final Products

  • A dissemination-ready deliverable (e.g., paper, white paper, presentation)
  • Annotated dataset and reproducible experimentation code
  • Summary presentation for presenting to faculty and students at TechBytes and/or practitioner audience