Examining Student Use of Generative AI Through User-Input Data in an Engineering Statics Course

1. Introduction

Generative artificial intelligence (GenAI) can produce a variety of content forms through its use of machine learning algorithms and reliance on large datasets (Bahn & Strobel, 2023). Large language models (LLMs) are a form of GenAI that excel at interpreting and generating text. The widespread usage of GenAI chatbots–including but not limited to OpenAI’s ChatGPT, Anthropic’s Claude, Google’s Gemini, and Microsoft’s Copilot— has impacted a number of industries spanning from finance to healthcare, and most notably in the context of this paper, education (Mishra & others, 2025). Specifically, at the university level, the sudden emergence of such technology has caused an immediate need to create or modify academic policy surrounding the usage of GenAI (Pisica & others, 2023). Researchers, educators, and administrators all seem to provide various opinions on how GenAI should be correctly used, or whether it should be used at all.

Students appear to use GenAI models in a variety of ways: to answer content questions, to improve their work before submitting, to save class schedules and create to-do lists, etc. (Fauzi & others, 2023). Simultaneously, however, students can leverage GenAI models in ways which violate academic integrity policies, either knowingly or unknowingly. GenAIs are particularly disruptive in academia due to their ability to replicate human writing, with some experts fearing that “…students as well as researchers can start outsourcing their writing to ChatGPT” (Eke, 2023). The emergence of GenAI chatbots has been described as a “dual-edged sword” for higher education, in which these tools can both enhance learning while concurrently undermining the authenticity of students’ work (Cotton et al., 2024). A survey conducted in 2025 of over 1,000 undergraduate students in the United Kingdom found that nearly 92% use AI in some form, a marked increase from 66% of students in 2024. Of these students, 88% reported using GenAI for assessment preparation (Freeman, 2025). Furthermore, a study of engineering students at an R1 university in the United States found a statistically significant increase in GenAI usage between 2023 and 2024, with approximately 45% of students classified as regular or “superusers” of LLM-based chatbots (Ovi & others, 2025). The nuanced uses of this technology create a need for research on how students interact with GenAI in academia.

This paper investigates exploratory work based on user-input data—rather than self-reported data—to a single GenAI chatbot. This work supports broader studies on students’ perceptions, motivations, and sentiments towards a variety of GenAI platforms (Defrancisis & others, 2025; Wyszynski & others, 2025). One concern when studying GenAI use, particularly regarding academic integrity, is the honesty of students in their self-reported accounts. Students may be reluctant to self-identify as actively engaging in academic dishonesty, which introduces social desirability bias into survey-based studies (Ovi & others, 2025). Recent studies analyzing actual user-input data–rather than self-reported data–have begun to address this limitation. For example, the StudyChat dataset collected and annotated student interactions with a ChatGPT-based tool in a university-level artificial intelligence course, demonstrating the value of user-input data in an academic setting (Tomasino & others, 2025). Harnessing user-input data provides a new avenue for studying student usage of GenAI in a more quantitative manner. By analyzing student prompts to a GenAI chatbot called Ace (an LLM chatbot embedded within Top Hat’s Learning Management System), this study explores the following research questions:

1. How are college engineering students using Ace?

2. How do deadlines and course events affect student Ace usage?

3. Do ethical and effective AI usage interventions have an impact on solution seeking through Top Hat’s Ace?

The researchers hypothesize that engineering students use Ace to receive virtual assistance when in-person help is unavailable, that semester deadlines and course exams may lead to increased Ace usage, and that ethical and effective AI usage interventions lead to a decrease in solution-seeking behavior, such as copying and pasting homework questions directly into the chatbot.

2. Literature Review

The integration of GenAI tools into higher education has generated much interest and a rapidly growing body of literature, much of which relies solely on self-reported data from students about their usage. While these studies provide valuable insight and information related to student perceptions, attitudes, and habits, they are all limited by social desirability bias and the inability to capture actual behavioral patterns. The following subsections present current literature on student GenAI usage, concerns about academic integrity, interventions to promote ethical AI usage, and the emerging use of user-input data to quantify student-chatbot interactions.

2.1. Student GenAI Usage in Higher Education

Many studies in the current literature rely on self-reported data from students to study GenAI in universities. Almassaad et al. conducted a cross-sectional survey (\(n=859\)) to investigate student utilization and perceptions of GenAI, and found that 78.8% of students frequently use AI tools (Almassaad & others, 2024). Students commented positively on the ability to receive instant feedback from AI tools, though they expressed concerns about academic integrity and the unreliability of information. Another study (\(n=2,555\)) found that 51% of students used GenAI for academic purposes, and 54% of respondents reported they were supportive of a peer using GenAI as an aid to complete an academic assignment (Johnson & others, 2024). A large-scale study of over 72,000 undergraduate students in China found that over 60% used GenAI during the 2023-2024 academic year, with academic use exceeding personal use. In this study, there were reported pros and cons of GenAI use: respondents noted increased cognitive engagement but also experienced reduced active learning opportunities and lower motivation to learn (Yao & others, 2025). Similarly, a study of 498 European students found that the willingness to adopt GenAI was driven by perceptions of usefulness and estimated that approximately 10% of students used GenAI on assessments such as exams (Reiter et al., 2025). As a result, concerns of academic integrity violations are not isolated.

2.2. Academic Integrity and GenAI

The potential for GenAI to facilitate students engaging in academically dishonest behavior have been widely documented. Cotton et al. noted that ChatGPT can generate coherent text that may be indistinguishable from student writing, effectively multiplying the risks associated with traditional contract cheating, for example, with the use of Chegg (Cotton et al., 2024). A systematic review of the literature pertaining to academic integrity since the introduction of ChatGPT has identified four key recommendations for institutions: 1.) the development of ethical AI usage guidelines; 2.) targeted educator training; 3.) student training on responsible AI usage; and 4.) clear institutional policy development (Evangelista, 2025). Interestingly, a survey of high school students found limited evidence that the introduction of chatbots was the primary factor in behavior related to cheating, suggesting that behavior was more context- and circumstance-driven (Lee & others, 2024). Attempts to drive positive adoption of GenAI tools have been focused on interventions and training rather than prohibiting and punishing usage.

2.3. Ethical Interventions in Education

Some studies have begun to evaluate the effectiveness of ethical AI interventions on student behavior. Much of the research into ethics and AI has been focused on knowledge and awareness outcomes rather than changes in student behavior. A study on the online explicit-reflective learning module for 90 science and engineering students found significant improvements in ethical knowledge and awareness following an intervention; behavioral outcomes, however, were not measured (Qiao & others, 2024). A recent review of ethics education related to AI noted that although there have been numerous efforts at both the university and national levels to embed AI ethics into the curriculum, the impact of these interventions on student behavior remains largely unexplored (Barkhuff & others, 2025).

2.4. User-input Data

The ability to collect user-input data is very limited at this point in time. Although difficult to collect, the actual recording of student prompts into GenAI chatbots represents an advanced methodological approach to studying student GenAI usage. The StudyChat dataset, which was collected from a university-level AI course, includes over 16,851 student interactions (i.e., “utterances”) over 2,214 conversations. These interactions were characterized to reflect common student behaviors (Tomasino & others, 2025). The analysis of these interactions found that the mean conversation length was 7.6 interactions with students showing a range of behaviors, from answer-seeking to conceptual inquiries.

The present study stems from many of the prior studies building upon the ethical interventions, the user-input data methodology, and the examination of student usage patterns. This work incorporates a similarity-detection-based framework to solution-seeking behavior and also introduces a quasi-experimental design comparing the effectiveness of interventions. To the authors’ knowledge, no prior study has combined user-input data analysis with an ethical and effective AI usage intervention framework in an undergraduate engineering course.

3. Methodology

This study relies on data collected from one semester of a large-enrollment engineering Statics and Mechanics of Materials course. Three sections of the course (1020, 1030, and 1040) were run during the same semester, all taught using a digital interactive textbook hosted on Top Hat, “Statics and Mechanics of Materials: An Example-based Approach” (Barry & others, 2020). A total of 215 students were enrolled across all three sections (\(1020, \ n=83; \ 1030, \ n=70; \ 1040,\ n=62\)). All course sections were taught by two instructors who previously authored the textbook and developed the course curriculum; one instructor taught two sections (1020 and 1030), and the other taught one section (1040). The demographic information of the participants is presented in Table 1. All participants were undergraduate students.

On the Top Hat platform, students could access course announcements, textbook readings, assignments, and an online chatbot called Ace (Top Hat Support, 2025). Ace can generate human-like replies to student prompts using a stateless API call to OpenAI’s GPT API. Ace can be activated by students on any page of the textbook and on course assignment pages using either a mobile or web browser. For each user, Ace operates independently; each prompt context consists of only the course material on the page the student is actively viewing and the chat history (if it exists) of the particular student engaging with Ace. No mechanism for cross-user retrieval, shared memory, or persistent session state exists. All model weights are fixed at deployment, and Ace does not learn in real time. Before the start of this voluntary study, students were informed that they were not required to use Ace or any other GenAI and that their interactions with the chatbot would not be considered in their academic standing.

At the beginning of the semester, both the 1030 and 1040 sections received in-class instruction on ethical and effective AI use. The 1020 group did not receive any intervention measures and was used as a control group for comparison. The ethical AI module was assigned to both the 1030 and the 1040 sections and was designed with the following learning objectives for students:

• Understand the definition of AI and AI ethics and their relevance in engineering.

• Discuss basic principles of ethical AI usage in the context of engineering.

• Recognize proper and improper usages of AI within engineering.

Within the module, students were provided with excerpts from the National Society of Professional Engineers (NSPE) position statement No. 03-1774 on Artificial Intelligence, the United Nations Educational, Scientific and Cultural Organization (UNESCO) Recommendation of the Ethics of Artificial Intelligence, International Business Machines’ (IBM) Principles for Trust and Transparency, and a case study on IBM’s Watson (Artificial Intelligence NSPE Position Statement No. 03-1774, 2023; IBM’s Principles for Trust and Transparency, 2024; Strickland, 2019; United Nations Educational, Scientific and Cultural Organization, 2022). Students were required to answer five multiple-choice questions related to AI usage in engineering after reading through the module’s content. The questions, which assessed student understanding of AI definitions, AI ethics, UN-ESCO recommendations, NSPE position statements, and IBM’s fairness principles, are presented in Figure 1. A total of 52 students (\(74.28\%\)) completed this module in the 1030 section, and 47 students (\(75.81\%\)) completed the module in the 1040 section. Students had four possible attempts to answer each question correctly, and both sections achieved a \(100\%\) correctness score.

Figure 1.Multiple choice assessment questions administered as part of the ethical AI usage module assigned to the 1030 and 1040 course sections. Students had four attempts per question; both sections achieved 100% correctness (Artificial Intelligence NSPE Position Statement No. 03-1774, 2023; IBM’s Principles for Trust and Transparency, 2024; Strickland, 2019; United Nations Educational, Scientific and Cultural Organization, 2022).

An effective Ace usage lecture was delivered to only the 1030 section. The lecture provided students with knowledge of how to use Ace effectively, along with an in-class demonstration. Students were shown how to write effective prompts to Ace and were informed of the chatbot’s Socratic nature. Students were also shown how to use specific features within Ace, such as Summarization, Practice, and References. Finally, students were instructed on reading basic LaTeX math operations, which often appear in Ace’s outputs. The interventions presented to the course sections are summarized in Table 2. Prior knowledge of course content was not investigated in any of the interventions.

Throughout the semester, all student prompts to Ace across the 1020, 1030, and 1040 sections of the course were recorded, timestamped, and anonymized; data analysis was driven by the technology acceptance model (Davis, 1985). User-input data was provided directly to the researchers by Top Hat via a research agreement prior to the start of the study. Only the researchers had access to the data, which was stored on a local Network Attached Storage (NAS). This device had two layers of protection: first, it was on the researcher’s University network, which required university login credentials to access; second, the NAS was hidden on the network, and access was granted only to the researchers via a unique password.

The researchers were able to read the students’ prompts to Ace, as well as the chatbot’s reply. Each student was assigned a unique identifier, allowing conversations to be grouped. Using the ratio command from the rapidfuzz library in Python, each student prompt to Ace was assigned a similarity score based on the Levenshtein Distance to every homework question assigned in the course, with the goal of determining whether students were directly asking Ace to complete their assignments; a total of 572 homework questions were included in the comparison bank (RapidFuzz 3.14.3, 2025). The Levenshtein distance is a well-established metric in language processing in which the minimum number of single-character edits required to transform one string (i.e, the input) to another (i.e., the problem statement) is computed (Levenshtein, 1966). This metric has found widely accepted use in plagiarism detection and fuzzy string matching applications (Nesmachnow & Maulini, 2008). Only the maximum similarity score among all the questions was considered for each student prompt. All student prompts that had a similarity score of at least 75 with a course homework question were extracted. Extracted prompts were secondarily validated by the researchers to confirm that the student was solution-seeking and to avoid false positives. Table 3 shows the results from the data parsing, including the counts and proportion of students engaged in solution-seeking behavior for all three sections.

Further statistical analysis was conducted in R, using a Pearson’s chi-squared test of independence to determine whether the counts of the number of students engaged in solution-seeking were significantly different across the three intervention groups. This test was selected due to the categorical nature of the response variable (i.e., solution-seeking via copying and pasting directly into Ace) and its ability to be applied to samples with unequal variances and sizes. The collected data satisfied the assumptions of Pearson’s test, given that the observations were independent counts in mutually exclusive categories and that the expected counts were all greater than five across all sections (McHugh, 2013). The test was run under the null hypothesis that the distribution of solution-seeking was equal across all groups and that there was no association between the GenAI intervention and the observed solution-seeking behavior. An alpha value of \(\alpha = 0.05\) was used as the basis for this test.

A subset of student prompts was thematically coded using the methods outlined in “Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research” (Creswell & others, 2015). The subset consisted of all prompts from the student who submitted the most prompts in each section, as well as all prompts submitted by the three students who submitted the fewest in each section. The subset was sampled from an upper and lower extremes perspective of users to include both superuser and minimal-user interaction styles, and was not considered to be a representative sample of all prompts submitted to Ace throughout the term; it was comprised 227 prompts submitted by 12 students over the course of the semester. Two researchers coded each prompt individually using a modified coding schema developed in a prior study and presented in Table 4 below (Wyszynski & others, 2025). The assigned codes were compared, and a joint set of codes was determined. If disagreements arose, an arbitrator decided on the final code. The percentage of agreement between the codes prior to arbitration was \(85.4\%\), with a final inter-rater reliability score of \(94.71\%\).

4. Results and Discussion

This section presents and discusses the results of an analysis of student prompts to the generative AI chatbot, Ace. Student use of Ace was examined, and prompts to Ace were thematically coded to identify trends in GenAI interactions. Temporal patterns of Ace activity are also analyzed to assess the influence of assignment deadlines and course events on GenAI use. Finally, the effectiveness of GenAI instructional interventions is evaluated through an analysis of the number of students engaged in solution-seeking behavior across course sections with differing interventions.

4.1. GenAI Usage

Of the 215 students involved in the study, 194 (\(90.23\%\)) used Ace at least once during the semester. From the 1020 section, \(85.54\%\) of students used Ace; in the 1030 section, \(95.71\%\); and in the 1040 section, \(90.32\%\). This high rate of usage is consistent with data reported in the literature; recent studies have shown GenAI usage rates ranging from 60% to 90% in higher education (Freeman, 2025; Ovi & others, 2025). The high usage rates in the 1030 and 1040 sections suggest that the Ethical AI Usage Module increased student awareness and adoption of the tool. Additionally, the 1030 section had the highest usage rate, further indicating that the Effective Ace Usage Lecture promoted usage.

The number of responses recorded for each student in all three sections was recorded, and the quartiles and interquartile ranges for the number of Ace prompts submitted during the semester are shown in Table 5. The 1030 section has the largest spread of the middle 50% of values recorded compared to the other ranges from the 1020 and 1040 sections, which are approximately equal. In the 1030 section, counts of Ace chats show several outliers. An upper bound based on the third quartile and interquartile range indicates that approximately six superusers are in the 1030 section, likely contributing to the elevated quartile values. The presence of these six superusers in the 1030 section parallels findings from previous research, where a subset of students was found to engage with GenAI tools more frequently than their peers (Ovi & others, 2025). It is plausible that the Effective Ace Usage Lecture contributed to the emergence of superusers by increasing comfort and proficiency with the tool.

4.2. Thematic Coding

Based on the thematic coding, students most often used Ace to develop their conceptual understanding of course content, a finding consistent with prior research (Almassaad & others, 2024; Sousa & Cardoso, 2025; Wyszynski & others, 2025). In the context of this engineering course, many students also used Ace to complete assigned Statics problems. Ace was often asked to list the steps required to complete relevant calculations, to clarify equation set-ups and problem statements, and to explain how to apply formulas in the context of statics problems (see Methodology (M) in Table 4). Students often prompted Ace with, “What equation should I set up?” and/or “How would I apply this formula?” when seeking assistance with coursework. Due to Ace’s embedded nature within the course textbook, students also used the chatbot to parse sections of their book in search of definitions, equations, and reference values. Some students asked for specific equations–for example, “Give me Young’s modulus equation”–and others asked Ace to find relevant relations–“Relate length to axial strain.” Ace was also used to help students with unit conversions, using the reference values provided in the text. Among the student prompts to Ace that were thematically coded, some students used Ace for solution-seeking. In these instances, students either directly copied and pasted course content into the chatbot or relied on Ace to solve an assigned problem.

Looking at Table 4 in greater detail, Methodology and Conceptual Understanding (\(33.92\%\) and \(28.19\%\), respectively) account for over 62% of all coded student prompts, suggesting the Ace usage was predominantly to support student learning. The relatively low frequency of Find Solution prompts (4.85%) indicates that behavior that directly violated ethics and academic integrity accounted for a small share of overall interactions. Error Checking and Reference Textbook (\(11.45\%\) each) further substantiate the idea that students used the chatbot to verify their work or pull equations, descriptions, etc., rather than find an answer to a question. Interestingly, the Unreliable code (\(6.19\%\)) being higher than Find Solution suggests that students maintained a level of critical thinking when using Ace; they scrutinized Ace’s response and noted instances when the chatbot provided incorrect or irrelevant responses. These patterns are broadly consistent with prior findings where students use GenAI tools to broaden their understanding and enhance their work product rather than to generate a solution (Ovi & others, 2025). Further analysis was conducted on when students interacted with Ace to better understand their GenAI usage behavior.

4.3. Temporal Usage Patterns

To assess whether assignment deadlines and course events affected the frequency of prompts to Ace, timestamps of student-Ace interactions were tracked throughout the semester. Figure 2 shows the plot of the aggregated number of unique Ace users across all course sections on a given day during the academic term. A trend emerges when considering the days of the week on which students use Ace. Throughout the first three months of the semester, usage peaks on the early days of the week, then tapers off as the week continues. These peaks continue until all Ace usage becomes minimal in December. Across all days of the week, students most often submitted prompts to Ace on Sundays and Mondays, aligning with assigned reading and homework due dates, respectively.

Figure 2.Ace usage across all three course sections plotted over the duration of the academic semester.

Peaks in Ace usage in Figure 2 may be associated with several course events. Notably, a peak in usage occurring on September 8\({\mathstrut}^{\mathrm{th}}\) aligns with the date of the ethical AI usage module assigned to both the 1030 and 1040 course sections. Similarly, the mode of use on September 17\({\mathstrut}^{\mathrm{th}}\) aligns with an in-class introduction to a GenAI research study (separate from this one) presented during the lecture. These interventions likely drove early student engagement with Ace, and potentially illustrate the technology acceptance model at play, since at this stage of introduction Ace has arguably the highest level of perceived usefulness and perceived ease of use to students, driving attitudes towards adopting this technology (Albayati, 2024). Over the course of the semester, however, the number of Ace users not only stabilizes but also steadily declines. One possible explanation for this decline may be a lowered perception of usefulness, as students may have observed Ace functioning unreliably, which may have ultimately led them to taper or stop their Ace usage entirely. This declining trend may also reflect the novelty effect wherein initial curiosity drives early engagement that gradually diminishes as the tool becomes routine (Yao & others, 2025).

The course midterm exam was administered to all three sections on October 17\({\mathstrut}^{\mathrm{th}}\). A rise in the number of Ace users occurred on October 13\({\mathstrut}^{\mathrm{th}}\), with a 150% increase over the previous day. Similarly, prior to the second midterm exam on November 21\({\mathstrut}^{\mathrm{st}}\), another small rise is observed in the number of Ace users. It appears reasonable to conclude that course events, such as assignment deadlines and exams, affect students’ use of Ace throughout the semester.

Figure 3.Quantity of students using Ace aggregated by hour of interaction with overlayed office hour availability.

In addition to the date of Ace interaction, the time at which students prompted Ace most frequently was also examined in determining the impact of course deadlines on student Ace usage. Figure 3 depicts a clear bimodal distribution in the number of students using Ace by hour of the day. Students appear to most often prompt Ace during the evening between 4 pm and 10 pm, and in the early morning between 1 am and 3 am. Homework and lecture video questions in the course were due at midnight, while in-class worksheets and assigned reading questions, both of which were graded for completion, were due at the beginning of lectures. Course lecture times appear to correspond with a reduction in the number of Ace users, with lectures for all three course sections being held from 11 am to 1:50 pm. The drop in users between 7 am and 10 am may align with the typical sleeping schedule of a college engineering student. However, increased evening use may suggest that students increase Ace interactions prior to homework deadlines.

The bimodal distribution is also consistent with the hypothesis that Ace served as a virtual assistant when in-person resources, such as office hours, tutoring centers, and responses from the course email, were unavailable. The concentration of usage in the late evening and early morning hours corresponds to when traditional academic support structures are closed, reinforcing the potential role of GenAI as an accessible, on-demand learning support mechanism.

4.4. Solution-seeking Behavior

Solution-seeking behavior (i.e., copying and pasting homework questions directly into Ace) was investigated to determine whether GenAI ethical and/or effective use interventions could reduce cases of academic integrity violations. The contingency table in Table 6 summarizes the results of data parsing using similarity scores generated between student prompts and a bank of course homework questions. Pearson’s Chi-square test of independence was conducted to assess whether the counts of solution-seeking users differed across course sections. The test resulted in a chi-statistic value of \(\chi^2_2 = 3.02\), and a p-value of \(0.22\) (\(\alpha = 0.05\)). From this analysis, it appears that there is no significant association between course section and the number of students engaged in solution-seeking, suggesting that the distribution of students engaged in solution-seeking interactions with Ace did not differ significantly across the three groups with varying ethical and effective GenAI usage interventions.

There may be several factors that could explain the lack of a statistically significant finding. Notably, the Ethical AI Usage Module–while achieving a 100% correctness score on a knowledge-based assessment–may not have been sufficient to influence actual student behavior; prior research into ethics education has shown knowledge gains do not directly correlate to behavioral changes (Qiao & others, 2024). Furthermore, the Levenshtein threshold of 75, although good at capturing direct “copy-and-paste” behavior, does not capture more sophisticated forms of solution seeking. Students could paraphrase the questions or omit large portions of the problem statement. Students engaging in this behavior would circumvent detection, reducing observed differences between groups.

5. Conclusion

This study examined student usage of the GenAI chatbot Ace in a large-enrollment, university-level, Statics and Mechanics of Materials course by exploring user-input data. Through the analysis of timestamped and anonymized student prompts, this work addressed three research questions about how students use Ace, the influence of deadlines and course events on usage, and the impact of ethical and effective AI usage interventions on solution-seeking behavior.

In regard to the first research question, “How are college engineering students using Ace?” the vast majority of students (90.23%) used Ace at least once during the semester. Thematic coding of 227 user prompts from a 12-student subset–selected from the over 5,000 total prompts–revealed that students primarily use Ace for methodological inquiries (33.92%) and aiding in conceptual understanding (28.19%). Solution-seeking behavior accounted for only 4.85% of the thematic codes, and the Levenshtein similarity analysis corroborated this finding: only 121 of the over 5,000 student prompts had a similarity metric greater than 75, and only 74 were classified as solution-seeking. Less than 1.5% of prompts were clearly identified as solution-seeking. These findings suggest that most students used ACE as a learning support tool rather than as a means of cheating.

The second research question, “How do deadlines and course events affect student Ace usage” was supported by temporal analysis. Student engagement with ACE closely followed assignment deadlines. The bimodal hourly usage distribution, with peaks in the evening prior to midnight homework deadlines and in the early morning corresponding with reading assignment and lecture video deadlines, suggests that students used Ace when in-person academic support was unavailable. This finding highlights the potential of GenAI to extend the availability of learning support beyond traditional academic support (i.e., office hours).

The third research question, “Do ethical and effective AI usage interventions have an impact on solution seeking through Top Hat’s Ace?” yielded a null result. A Pearson’s chi-squared test showed no statistically significant difference in solution-seeking behavior across the three intervention groups (\(\chi_2^{2} = 3.02\), \(p = 0.22\)). While the 1040 section had the lowest proportion of solution-seeking behavior (11%), they also only had the Ethical AI Usage Module, not the Effective Ace Usage Lecture. This finding suggests that a single-exposure ethical and/or effective AI usage intervention may be insufficient at altering student behavior.

5.1. Study Limitations

The current study has several identified limitations, beginning with issues in generalizability. This study was conducted at a single institution and within a single course. In addition, the total sample population and section-specific course populations are also admittedly small. The thematic coding conducted analyzed a total of 227 prompts to Ace, and while this qualitative analysis yielded high-quality data and insight into students’ GenAI usage, these prompts were only composed by a subset of 12 students. With over 5,000 unique prompts, more data analysis is required to expand this sample and to provide more generalizable trends.

Beyond sample size concerns, the quasi-experimental design introduces confounding variables related to the section-level differences. From a methodological standpoint, the Levenshtein distance-based detection metric, although good at capturing verbatim copying of homework questions prompted into the chatbot, does not capture prompts that were paraphrased or omitted large portions of the problem statement.

Another important limitation is that the study had no provisions for analyzing the use of GenAI tools outside of the course; in a 2025 study of the same course at this institution, ChatGPT was found to be used by narrowly more students surveyed than Ace (\(n=158\)), suggesting that the conversations collected in this research constitute only a fraction of a much larger dataset (Wyszynski & others, 2025). Additionally, students’ prior knowledge of course content was not investigated as part of this research.

5.2. Implications and Future Work

Despite the aforementioned limitations, this study makes several contributions. Foremost, this work demonstrates the feasibility and value of user-input data as a complement to self-reported data when studying student GenAI usage, offering a methodology that can be replicated in our institution’s courses and at other institutions that use Top Hat. Future work may involve coupling user-input data with student self-reported data to provide a comprehensive account of the broad uses of GenAI in the classroom, and potentially examine cases of discrepancies in self-applied codes. The comparison of user-input prompts to GenAI with self-reported reflection may provide further insight into the students’ emotional and motivational dimensions required to fully understand behavior.

Additionally, the temporal usage pattern analysis identified deficiencies in academic support and can be used when designing academic support structures. For example, instructors may consider altering the times at which assignments are due when they are unavailable for support during the nights and weekends.

The null finding regarding the ethical and effective use of AI interventions contributes to the developing literature on their effectiveness and illustrates the need for alternative approaches. The interventions described in this study were single implementations; one avenue for future work may be to develop a GenAI ethics and education curriculum that is revisited throughout the term and continually built upon in class. Consistent ethics and effective use of education may shape students’ attitudes and perceptions of the usefulness of such technology and potentially lead to a non-null finding.