1. Introduction

The paper examines student experience with a pilot GenAI based chatbot in a first-year engineering course of over 800 students at a Midwestern R1 institution, implemented as part of a university-wide initiative. The course combines lectures with multiple lab sections (25 students each) and focuses on Industry 4.0 and digital manufacturing (Fuerst et al., 2024; Gupta et al., 2025). Course assignments simulate real engineering workflows, including parametric computer-aided design modeling, product database file management, and structured engineering-change processes (Fuerst et al., 2024).

The course was selected due to its large enrollment and instructional demands, supported by 12 Teaching Assistants (TA) responsible for managing labs, grading assignments, and responding to student queries (Fuerst et al., 2024; Gupta et al., 2025). The student-to-TA ratio of approximately 67:1 creates a limited window for students to receive timely support. Unanswered questions are often submitted through help forms via emails, which can take one to two days to address. Meanwhile, the teaching assistants face a heavy burden from repetitive queries and grading tasks (Gupta et al., 2025). In this context, a GenAI based solution is particularly well suited. While chatbots have been explored in various educational contexts, the study makes a distinct contribution by examining a large-scale, first-hand deployment in a first-year engineering design course. Our work leverages a course communication platform, ensuring minimal overhead for instructors and seamless access for students (Gupta et al., 2025).

We use a mixed-methods approach (Bergdahl & Sjöberg, 2025; Creswell & Creswell, 2023) that combines quantitative analysis guided by the UTAUT model (Bergdahl & Sjöberg, 2025; Elkefi et al., 2024; Kajiwara & Kawabata, 2024; Lai et al., 2024; Venkatesh et al., 2003; Wijaya et al., 2025) with complementary information from qualitative responses from students directly (Bae et al., 2024; Bergdahl & Sjöberg, 2025; Elkefi et al., 2024; V. R. Lee et al., 2024). The dual perspective captures both usage patterns and perceptions, highlighting the strengths, limitations, and challenges of integrating a chatbot in a high-enrollment course (Bergdahl & Sjöberg, 2025; Elkefi et al., 2024). Findings provide empirical evidence on student adoption and experience of the technology, informing educators, researchers, and decision-makers about the readiness of GenAI based chatbots for large-scale implementation and outlining potential benefits and limitations (Gupta et al., 2025).

2. Background

A GenAI based chatbot is a computer program designed to interact with humans using natural language (Álvaro & Barreda, 2025; Wang et al., 2021). Leveraging deep learning methods for natural language processing (transformer architecture with attention mechanism), a chatbot interprets semantics of human input, understands the context and generates responses in a manner mimicking human conversation (Bhardwaj & Bedi, 2024; Okonkwo & Ade-Ibijola, 2021; Wang et al., 2021). Its ability to parse and identify relations on a large corpus of text and logically answer progressive queries make it a powerful tool (Álvaro & Barreda, 2025; Uhm et al., 2025).

The introduction of ChatGPT (Chat Generative Pretrained Transformer) by OpenAI in 2022 disrupted the AI and chatbot sector (Bae et al., 2024; Bhardwaj & Bedi, 2024; Nah et al., 2023; Wang et al., 2021). Since then, other GenAI based chatbots such as Google’s Gemini, Anthropic’ s Claude, and Meta’s Llama have flooded the market (Gupta et al., 2025). GenAI models are trained on a large corpus of text, audio, and image data. These powerful multimodal capabilities enable the system to generate text, audio, images, and even video outputs probabilistically in response to simple interactive commands (Martin & Graulich, 2024; Nah et al., 2023).

As in other fields, GenAI has gained significant traction in education. GenAI based virtual assistants support instructors by helping design more effective curricula (Zheng et al., 2024), course based automated feedback generation (Fung et al., 2024), grading short answers and open-ended responses (Jauhiainen & Garagorry Guerra, 2025; Qiu et al., 2024; Zhao et al., 2025), creating diverse assignments, multiple choice question (MCQ) based testing (Hang et al., 2024) and producing well-reasoned solutions (Bergdahl & Sjöberg, 2025; Martin & Graulich, 2024; Okonkwo & Ade-Ibijola, 2021). When paired with learning management systems, they can analyze, predict, and assess student learning progress. Automated feedback on assignments (Zhao et al., 2025), attendance patterns, submission timing, tests and targeted learning materials can support personalized instruction, thereby making the learning experience more student-centered (Okonkwo & Ade-Ibijola, 2021; Wang et al., 2021).

Students commonly use the chatbots for tasks such as writing assistance (Guo & Li, 2024; Hu et al., 2025), teaching assistance (Y. Lee, 2025; Lui et al., 2024; Németh et al., 2025), generating quizzes (Thüs et al., 2024), programming and debugging (Liu et al., 2026; Šarčević et al., 2024), brainstorming and proofreading (Bae et al., 2024; Elkefi et al., 2024; V. R. Lee et al., 2024). According to a recent study, most students (89.69%) use GenAI tools for educational purposes and have a strong intention to continue doing so (Elkefi et al., 2024). This trend is reinforced by students’ perceptions of its usefulness, credibility, and enjoyment, making it a strong candidate for sustained adoption. Efficiency remains a key perceived benefit: 30.9% of students identify it as the primary advantage (Bae et al., 2024). By streamlining routine tasks, students can allocate more time for higher order learning (Zhong & Zhao, 2025).

They also support administrative functions like recruitment, admissions, and course guidance (Neupane et al., 2024; Nguyen et al., 2021; Saha & Saha, 2024). Course-specific chatbots reduce faculty workload by addressing routine questions on syllabi, assignments, software, and grading policies (Amarnath & Nagarajan, 2024; Gupta et al., 2025). Automated responses save time, lower student stress, and allow instructors to focus on course improvement and meaningful interaction (Okonkwo & Ade-Ibijola, 2021; Sanders et al., 2024; Zhong & Zhao, 2025). ChatGPT also supports research exploration of unfamiliar problems, drawing on knowledge across disciplines (Khalifa & Albadawy, 2024).

However, significant challenges exist. Concerns commonly center on data accuracy and bias, overreliance, reduced independent critical thinking, fabricated or hallucinated responses, academic dishonesty, privacy and security issues and plagiarism (Alier Forment et al., 2024; Bae et al., 2024; Kajiwara & Kawabata, 2024; V. R. Lee et al., 2024). Since GenAI tools are sophisticated probabilistic word predictors, they can cook up any story to suit the given prompts leading to hallucination (Álvaro & Barreda, 2025; Bae et al., 2024; Chen et al., 2025; Nah et al., 2023). Although such tools can save time and effort, taking responses at face value or over-relying on them may undermine independent reasoning and cause automation bias. It is thus important for students to verify responses against relevant sources (Bae et al., 2024; Bartlett & Camba, 2024; Nah et al., 2023).

Academic integrity is also directly impacted. Because systems like ChatGPT can generate high-quality responses instantaneously, some students may feel little incentive to attempt the work themselves (V. R. Lee et al., 2024; Nah et al., 2023). It can also lead to GenAI addiction where any given stress situation (assignments etc.) is quickly mitigated by the ease of using such tools (Al-Obaydi & Pikhart, 2026; Kooli et al., 2025; Li & Jiang, 2025). The dynamic can further discourage those who invest effort, particularly in grade-driven environments where inference-based assessments carry weightage towards final grade (Fehr & Fischbacher, 2003; V. R. Lee et al., 2024). Thus, the education system will be dragged into the vicious downward spiral of academically dishonest practices. Instructors may also be tempted to offload substantial portions of course design or teaching to automated systems (Bergdahl & Sjöberg, 2025). As the University of North Carolina cautions, “AI should help you teach, not teach for you”, teachers also should use GenAI responsibly (An et al., 2025).

GenAI chatbots such as ChatGPT are trained on vast variety of data and hence lack specialized knowledge for course-specific queries. The outcome is that it provides inaccurate or fabricated responses (hallucination) and undermining critical thinking abilities (Álvaro & Barreda, 2025; Rachha & Seyam, 2024; Uhm et al., 2025; Yang et al., 2025). To overcome the problem and apply GenAI in educational setting, methods like fine-tuning and retrieval-augmented generation (RAG) are used (Wan et al., 2025).

Model fine-tuning is a common approach in machine learning, where a pre-trained and validated model is adapted by retraining the final layers of its neural network using new, domain-specific data. This approach is convenient and integrates seamlessly with existing workflows. However, it is resource and time intensive, requires data science expertise and is hard to scale (Álvaro & Barreda, 2025; Uhm et al., 2025; Wan et al., 2025). Since the original model is modified, careful evaluation is essential to ensure performance and reliability of the new model.

On the other hand, the RAG framework (Benfenati et al., 2024; Bingöl et al., 2025) combines the strength of a retrieval system grounded in an external specialized database with human like logical conversation of a GenAI model (Álvaro & Barreda, 2025; Uhm et al., 2025). It basically comprises of embedding, retrieval and generation. Fig. 1 illustrates the embedding phase. In this phase, domain knowledge is stored in vector databases. Documents (text, pdf, word, excel etc.) are converted to sizeable chunks that must fit the token window of an embedding model (e.g. “text embedding ada 002” of OpenAI). Embedding model converts the chunks to multi-dimensional vectors which are indexed and stored in the vector databases (e.g. Pinecone, Chroma etc.) (Álvaro & Barreda, 2025; Wan et al., 2025).

Figure 1
Figure 1.Embedding Phase of a general RAG Framework

In the retrieval phase (Fig. 2), query submitted by user is converted to a vector embedding via the same embedding model. Using cosine similarity, top chunks of documents based on a threshold of similarity value are retrieved and reconverted to text from vectors (Álvaro & Barreda, 2025; Wan et al., 2025). In the generation phase (Fig. 2), the query along with the retrieved specialized text is injected into the large language model (LLM). Along with these two, another key text is injected, known as the core prompt (Álvaro & Barreda, 2025; Wan et al., 2025). Core prompt is critical as it creates the persona of the chatbot and provides guidelines on the tone, audience, length of response required, default response, boundary conditions etc. It thus directs the LLM to give preference to specialized database over LLM’s own information (Gupta et al., 2025).

Figure 2
Figure 2.Retrieval and Generation Phase of a general RAG Framework

Compared to model fine-tuning, RAG offers several advantages. It is easier to implement and scale, requires minimal technical expertise and fewer computational resources, making it well-suited for course specific applications (Uhm et al., 2025). Domain experts can integrate and update curated instructional materials and institutional knowledge dynamically to produce accurate, context-aware answers (Álvaro & Barreda, 2025; J. Lee et al., 2024). Thus, it helps to minimize inaccuracies and hallucination as backend information can be corrected if the answers are not up to the mark (Álvaro & Barreda, 2025).

Table 1 provides a summary of key papers that have utilized RAG framework in education use cases. The three major themes are teaching, assessment and administration.

Table 1.Summary of key RAG based chatbot applications in Education
Category Author and Year Study Design Sample Base Key Findings
Teaching Guo & Li, 2024 Mixed Method 69 Students STEM undergraduate students enrolled in English as a foreign language (EFL) class created their own RAG based chatbots for improving argumentative writing skills. 33% of students used the chatbot for idea generation, 28% for finding credible evidence, 28% for proofreading and 23% for evaluation. Use of self-made chatbots led to clearer writing goals, increased writing confidence, reinforced writing beliefs and developed positive attitude towards writing.
Y. Lee, 2025 Quantitative 20 Questions RAG based statistics tutor designed for quantitative analysis course was evaluated over 20 questions by two expert reviewers. It generated factually correct and accurate responses as reviewed by experts minimizing the hallucination problem. It provided responses over a range of statistical topics from basic concepts to R (Programming language) code for data analysis.
Lui et al., 2024 Mixed Method 129 Students GPTutor, an intelligent tutoring system using RAG framework, was made available for learning and revision purposes for two software engineering courses. Active users scored an average of 62.2 on exams as compared to seldom users with 51.9. Students were able to locate the information on lecture notes easily and got relevant answers from the chatbot. Repeat content and lack of depth were some of the limitations.
Thüs et al., 2024 Quantitative 33 Students OwlMentor, an RAG based chatbot assistant, was made available to master’s students for EdTech department for scientific literature comprehension. Technology Acceptance Model (TAM) was used to understand user acceptance. Perceived Usefulness was the top predictor for intention for use. Self-efficacy was inversely related to chatbot usage. However, there was no significant correlation between actual usage and learning gains.
Assessment Jauhiainen & Garagorry Guerra, 2025 Quantitative 54 Responses The paper explores automatic evaluation of open-ended written examination responses. Verification based chain of thought prompting, RAG framework and ChatGPT 4 are used for accurate recall and alignment with university evaluation criteria. 92.2% of the evaluations were within one grade. 72.2% grades were within one grade with expert teacher evaluation. Teacher gave slightly lower grade as compared to chatbot with closest alignment on logical consistency.
Fung et al., 2024 Qualitative Curriculum Guide DSRAG (Data Science RAG), a cloud based chatbot that provides personalized feedback to K-12 students on Data Science course. Unlike the baseline model, DSRAG provides detailed explanation of mistakes and what each part of the code does. It also links concepts to real world scenarios.
Qiu et al., 2024 Mixed Method 169 responses SteLLA (Structured Grading System Using LLM and RAG) is a tool for automated short answer grading for a college-level Biology course. The tool has got a substantial agreement (Cohen's kappa coefficient) of 0.6720 as compared to human level at 0.8315. Student feedback shows that the tool can identify relevant facts and statements, is tolerant to some typos but sometimes infer too much from the given text.
Zhao et al., 2025 Quantitative 300 Questions Automated short answer grading tool with RAG framework can provide reliable and instant feedback to students. RAG based example selection for GenAI models to grade outperformed random selection. Rubric also helped to improve the accuracy of grading.
Administration
Administration (continued)		 Neupane et al., 2024 Quantitative 50 students BARKPLUG v.2, a RAG based LLM chatbot provides information to users on campus resources like departments, programs, facilities, and student resources. The system demonstrates an impressive mean RAG Assessment Score (RAGAS) of 0.96. Based on the feedback from students, the System Usability Score (SUS) was 67.75, which was satisfactory with room for improvement.
Saha & Saha, 2024 Quantitative Curated Reddit Database The paper demonstrates a practical solution for solving international student queries by using curated reddit database with RAG framework. RAGAS scores moved from 0.56 to 0.92 for faithfulness, 0.65 to 0.95 for relevancy, 0.43 to 0.82 for context precision, and 0.38 to 0.71 for context recall by using RAG. Thus, the responses generated were more accurate and reliable.

RAG framework provides contextually relevant responses with effective guardrails that prevent hallucination effectively. However, smaller sample size, complex custom built chatbots from scratch and smaller timeframe of testing are some of the gaps observed and noted in the literature.

3. Chatbot Deployment

The chatbot was deployed during the Spring '25 and Summer '25 semester as part of a university-wide pilot initiative to test AI-based virtual assistants. Ready-to-use GenAI base framework from providers like Edstem.org facilitated easy adoption and customization for specific courses (Gupta et al., 2025). The deployed chatbot leverages Edstem’s platform built on “Azure AI GPT-4o” large language model (LLM), “text-embedding-ada-002” as the text embedding model using the RAG framework as demonstrated in Fig. 1 and Fig. 2 (Gupta et al., 2025).

Deploying the chatbot requires two major inputs from instructors critical to chatbot success. The first is the specialized resource setup (Fig. 1). All course-relevant materials (e.g., syllabus, lecture notes, assignments, etc.) must be uploaded to the chatbot’s backend (Álvaro & Barreda, 2025; Gupta et al., 2025; Uhm et al., 2025). These documents can be provided in various formats, including Word, Excel, and PDF, and should include clear metadata, such as titles, assignment identifiers, or lecture names, both within the documents and in the designated sections of the platform (Gupta et al., 2025). These materials serve as the knowledge base for the course, enabling the chatbot to retrieve and respond to student queries accurately. The quality and precision of the chatbot’s responses are determined by the completeness and clarity of the uploaded documents (Lewis et al., 2020).

The second step is the configuration of the chatbot’s persona (Fig. 2) through prompt engineering (Gupta et al., 2025). While the setup process may vary across platforms, the goal is to tailor chatbot’s behavior and communication style to the specific course context. There are seven key considerations to follow when defining this (Gupta et al., 2025):

1) Role: The role of the chatbot is to serve as a virtual assistant, offering students context-aware guidance and responses that reflect the structure and focus of the course material.

2) Task: In our deployment, the primary objective of the chatbot was to address student queries related to the course content, assignments, software usage, and grading policies. Clear boundaries were established to guide its responses. For software-related questions, the chatbot was permitted to retrieve information from the internet, while for all other topics, it was restricted to consulting the course-specific knowledge base.

3) Format: Responses from the chatbot can be formatted according to the instructor’s preference and pedagogical objectives, including bullet points, concise summaries, or detailed explanations.

4) Tone: The chatbot’s responses can be configured to adopt either an informal or formal tone. In our deployment, the tone was set to be professional and formal.

5) Audience: The intended audience must also be specified, as the chatbot tailors its responses to suit different levels of expertise, such as undergraduate or graduate students.

6) Scope: Defining the scope of the chatbot’s responses is critical. The system must strictly provide answers relevant to the course content. For off-topic or unrelated queries, the chatbot should either provide a default response or prompt the user for clarification. Establishing a clear scope ensures that the chatbot is used solely for its intended purpose: supporting students within the context of the specific course.

7) Default message: Finally, the prompt should include essential information such as help form link, the course website, and a disclaimer that chatbot can make mistakes. This information allows the system to direct students to appropriate resources when it is unable to provide a satisfactory answer.

After setting up the chatbot, sample queries must be created and answers validated based on the information provided by instructors. Extrinsic validation using human feedback is helpful, as it not only ensures accuracy but also provides insights into the backend documents provided (Saeidi, 2025). If responses are incorrect, the underlying documents should be reviewed and updated accordingly. It is also important to verify that the chatbot adheres to the defined boundaries (specified in core prompt) by posing questions outside the course scope and evaluating its responses. Once these steps are completed, the chatbot is ready for deployment (Gupta et al., 2025).

Although the system is well-informed, it is not an expert and may occasionally provide inaccurate responses. Students should therefore be encouraged to exercise critical thinking and avoid overreliance. The Edstem platform includes a feature that allows instructors to edit and update the chatbot responses, further improving accuracy over time.

The process demonstrates that operating such a system no longer requires expert-level knowledge of large language models or deep learning architectures. Setup can be completed within a few hours, with validation and refinement occurring pragmatically throughout the semester using both individual and aggregate student feedback. In the case of our course, the chatbot successfully handled simple to intermediate queries, while more complex or highly specific questions were routed to teaching assistants via help forms. This approach allowed the system to address the bulk of repetitive queries efficiently (Gupta et al., 2025).

4. Methodology

4.1. Experimental Design

To examine how students experienced and evaluated the chatbot, the study employs a convergent mixed-methods design. This approach is well suited for capturing both the measurable aspects of technology acceptance and the nuanced, contextual dimensions of student experience (Creswell & Creswell, 2023; Elkefi et al., 2024).

To ground this study in a formal analytical and psychological framework, UTAUT was selected. UTAUT offers a comprehensive basis for examining adoption decisions and can help institutions allocate time, resources, and effort more efficiently by predicting user acceptance (Ursavaş, 2022; Venkatesh et al., 2003). The UTAUT model identifies four core determinants: Performance Expectancy (PE), Effort Expectancy (EE), Social Influence (SI), and Facilitating Conditions (FE), along with up to four moderating variables. Empirical studies demonstrate that UTAUT can explain up to 70% of the variance in users’ intention to adopt a technology (Venkatesh et al., 2003).

Inductive content analysis was used to identify emerging themes from the survey responses for the open-ended qualitative questions (Bae et al., 2024). The purpose of these questions was to uncover the underlying reasons for both the use and non-use of the chatbot. By examining recurring themes in the responses, independent patterns were expected to emerge that provide deeper context for interpreting the quantitative findings. Following established guidelines, each response was reviewed, and a codebook was developed to capture the recurrent themes (Bae et al., 2024; Bergdahl & Sjöberg, 2025).

4.2. Data collection

A survey instrument was designed to collect mid-semester feedback on the chatbot deployed during the Spring '25 and Summer '25 in the pilot course (Gupta et al., 2025). Spring '25 is an in-person 16-week course while Summer '25 is an asynchronous four-week course (also known as maymester) with exactly same content as the 16-week course. The objective was to compare and assess the usage of GenAI over different modes of content delivery (in-person vs asynchronous). The survey was released between 25th and 28th March for Spring '25 (middle of the semester) and between 6th and 15th June (middle to end of semester) for Summer '25. Participation in the survey was voluntary, and all respondents were required to provide informed consent before beginning the survey. To encourage participation, students received extra credit for completing the questionnaire.

Students who had used the chatbot prior to the survey release were asked a total of 15 questions. For the quantitative component grounded in the UTAUT framework, items were developed to measure the four core constructs: performance expectancy (three items), effort expectancy (three items), social influence (three items), and facilitating conditions (two items) (Ursavaş, 2022; Venkatesh et al., 2003). Two additional items were included to assess intention for future use. All quantitative questions were adapted following the guidelines provided in the original UTAUT model formulation (Venkatesh et al., 2003):

Performance expectancy refers to the degree to which an individual believes that using this technology will improve task efficiency. It synthesizes concepts from several technology acceptance models, including perceived usefulness, extrinsic motivation, job fitness, relative advantage, and outcome expectation. Prior research consistently identifies performance expectancy as the strongest predictor of behavioral intention to use a system (Venkatesh et al., 2003). Accordingly, the hypothesis tested through this construct was:

  • H1: Performance expectancy positively influences students’ intention to continue using the chatbot.

The following three items were included in the survey to assess this hypothesis:

  • PE1 - The Chatbot is useful for the course

  • PE2 - The Chatbot enables me to accomplish learning tasks more quickly

  • PE3 - The Chatbot improves productivity

Effort expectancy refers to the perceived ease associated with using the system. It encompasses constructs such as perceived ease of use, complexity, and overall usability found in earlier technology acceptance models (Venkatesh et al., 2003). The corresponding hypothesis for this study was:

  • H2: Effort expectancy positively influences students’ intention to continue using the chatbot.

The survey included the following three items to measure effort expectancy:

  • EE1 - My interaction with the Chatbot is clear and understandable

  • EE2 - It is easy to become skillful at using the Chatbot

  • EE3 - Using the Chatbot is time consuming

EE3 was intentionally negatively worded. Consequently, lower scores indicate more positive perceptions of ease of use and vice-versa. To maintain consistency across effort-expectancy measures, EE3 was reverse-coded for analysis.

Social influence describes the extent to which the peers and close circle of an individual expect them to use a system. In UTAUT, it comprises subjective norms, social factors, and image (Venkatesh et al., 2003). Accordingly, the corresponding hypothesis for this study is:

  • H3: Social influence positively affects a person’s intention to continue using the chatbot.

To assess this construct, the following items were included:

  • SI1 - My usage of the Chatbot is influenced by people who are important to me

  • SI2 – I use the Chatbot because my peers are using it

  • SI3 - The University supports usage of Chatbot

Although SI3 was included following UTAUT guidelines, post-survey review suggested students may lack sufficient insight into institutional decision-making to respond reliably. To reduce potential bias, SI3 was excluded from the Structural Equation Modeling (SEM) analysis.

Facilitating conditions describe perceptions of available organizational and technical support for technology use. In UTAUT, this construct encompasses perceived behavioral control, facilitating conditions, and compatibility (Venkatesh et al., 2003). The corresponding hypothesis for this study is:

  • H4: Facilitating conditions positively affects a person’s intention to continue using the chatbot.

To assess this construct, the following items were included:

  • FC1 - I have the necessary knowledge to use Chatbot

  • FC2 - There is a specific person (or group) available for assistance with Chatbot related difficulties

Behavioral Intention (BI) reflects an individual’s expressed willingness or plan to perform a particular behavior (Venkatesh et al., 2003). Within the UTAUT framework, it serves as a strong predictor of actual usage. Two items assessing students’ intention to use the chatbot were included:

  • BI1 - I intend to continue using the Chatbot in this course

  • BI2 – I would like to use the Chatbot in other courses

Observed variable for behavioral intention was created by taking the mean of BI1 and BI2 to simplify the SEM process for better interpretability. In addition to these quantitative items, two open-ended qualitative questions were incorporated to deepen the interpretation of results. These questions asked students: “What are the benefits of using the AI chatbot in the course?” and “What are your concerns regarding chatbot use in education?” (Elkefi et al., 2024).

5. Results and Analysis

5.1. Descriptive analysis

Coordinating this 16-week course involves significant logistical demands, with graduate teaching assistants handling most student support, including lab assistance, grading, and email help-form responses. Student questions are typically raised during labs or submitted via help forms addressing assignments, attendance, or grading (Gupta et al., 2025).

Fig. 3 shows the number of questions received via help forms emails by semester. From Fall '23 to Fall '24, we see the questions per student (shown as a trend line in Fig. 3) to be greater than 1. During Fall '23, an average of 2.38 questions were received per student. The number dropped to 1.16 in Spring '24 and 1.29 in Fall '24. Following chatbot deployment in Spring '25, questions decreased to 0.5 per student (57% reduction) compared to 1.16 in Spring '24, despite no major changes to course structure or staffing. Weekly trends in Fig. 4 further show that Spring '25 consistently had fewer questions than previous semesters, with Fall '23 exhibiting the highest volume. Also, fall semesters have more queries asked than spring semesters.

Figure 3
Figure 3.Questions asked per Student by Semester
Figure 4
Figure 4.Week over week (WoW) Help Form Trend by Semester

Fig. 5 shows the different categories (assignments, grading, software and others) of issue types generally raised by students every semester via help forms through emails. From Spring '24 to Spring '25, the percentage distribution has remained consistent (a mean of 29% for Assignment, 33% for Grade, 27% for Software and 11% for other issues). But the number of queries has reduced from 528 in Spring '24 to 295 in Spring '25 (# Queries trendline in Fig. 5). Simultaneously, queries have been increased on chatbot that were previously asked over help form (Fig. 6 details about chatbot interaction). The stability in category percentage combined with reduction in queries proves the versatility of the chatbot in answering queries for different issue types. One category that cannot be directly attributed to chatbot for help form email reduction is grading. It is attributed to a combination of the chatbot and the effective grading mechanism deployed during the course (Fuerst et al., 2024; Gupta et al., 2025).

Figure 5
Figure 5.Help Form Email Volumes by Issue Type

Next, we provide an examination of student interactions with the chatbot, quantitative analysis on survey data, and results from structural equation modeling of the various hypothesis laid out in methodology (Section 4).

5.1.1. Chatbot interaction data

Chatbot interaction data were collected from Spring '25 and Summer '25 courses (Fig. 6). Approximately 72% of Spring '25 students and 77% of Summer '25 students actively used the chatbot as of end of semester. In terms of views, Spring '25 had 23,532 views combining all posts (55 views per user). While Summer '25 had 9,652 views (114 views per user). It was nearly double the views per user in Spring '25. In terms of comments made, Summer '25 comments per active user were nearly 5 times (3.8) as compared to Spring '25 (0.7). While chatbot usage was substantial in both semesters, engagement was higher in Summer '25, highlighting its increased importance in asynchronous, online course settings with minimal in-person interaction.

5.1.2. Survey exploratory data analysis

A total of 459 survey responses were collected in Spring '25 and 47 in Summer '25, with 454 and 41 complete responses retained, respectively. The response rates were 77% for Spring '25 and 37% for Summer '25, with the lower Summer '25 rate likely due to the asynchronous, online format.

Table 2 summarizes responses for each construct among chatbot users. Performance expectancy, effort expectancy, and facilitating conditions had median ratings of 4 (agree) on a 5-point Likert scale. Behavioral intention items (BI1 and BI2) showed strong support, with 85% and 88% positive responses in Spring '25, indicating broad intent to continue using the chatbot (BI1) and to adopt it in other courses (BI2).

Table 2.Response summary for Spring '25 Survey Items for 259 students
Item Mean Median Std. Dev. Skewness Kurtosis
PE1 3.83 4.00 0.92 -1.38 2.08
PE2 3.58 4.00 0.90 -0.76 0.74
PE3 3.52 4.00 0.93 -0.77 0.49
EE1 3.75 4.00 0.92 -0.87 0.58
EE2 3.90 4.00 0.86 -0.36 -0.38
EE3 3.73 4.00 0.90 -0.52 -0.42
SI1 2.15 2.00 1.02 0.33 -0.82
SI2 2.18 2.00 1.06 0.40 -0.95
FC1 4.15 4.00 0.79 -1.05 1.51
FC2 3.51 4.00 0.92 -0.30 -0.26
BI1 0.85 1.00 0.36 -1.92 1.71
BI2 0.88 1.00 0.33 -2.30 3.32

5.1.3. Hypothesis testing and data analysis using SEM

SEM-based approach was used to analyze the UTAUT framework, with stability ensured by a sample size exceeding 200. SEM was run for Spring '25 semester as it satisfied the sample size criteria. Summer '25 data was thus excluded from SEM analysis as it had 41 responses only, but open-ended responses were used in qualitative analysis. Overall behavioral intention observed variable was created using the mean of BI1 and BI2. Univariate normality was assessed using skewness and kurtosis thresholds of ±3 and ±10, respectively (Table 2). Relationships among five constructs (performance expectancy, effort expectancy, social influence, facilitating conditions, and behavioral intention) were examined through hypothesis testing as illustrated in Fig. 7 and detailed in section 4 (Kajiwara & Kawabata, 2024; Ursavaş, 2022; Venkatesh et al., 2003).

A confirmatory factor analysis was conducted with acceptable model fit results. Model demonstrated good fit indices (CFI = 0.995, TLI = .992, SRMR = .055, RMSEA = .042, (χ2 (35) = 50.61, p < 0.05)). Standardized factor loadings (values in Fig. 7) were significant for most of the indicators (p < 0.05). Performance expectancy demonstrated strong loading, ranging from 0.84 to 0.92. Social Influence was acceptable with values between 0.67 and 0.86. Effort expectancy and facilitating conditions showed weaker loading with values ranging from 0.45 to 0.72 and 0.47 to 0.81 respectively. Thus, there is scope for future refinements in the survey instrument for two constructs.

Composite reliability values indicated adequate internal consistency for PE (CR = 0.865), but lower reliability for EE (CR = 0.580), SI (CR = 0.684), and FC (CR = 0.501). Similarly, average variance extracted (AVE) supported convergent validity only for PE (AVE = 0.780) and SI (AVE = 0.598), while EE (AVE = 0.371) and FC (AVE = 0.437) fell below the recommended threshold of 0.50.

The model explained 44.3% of the variance in behavioral intention (R2 = 0.443), indicating moderate explanatory power. PE had the strongest effect on behavioral intention (β = 0.65, p < .05) showing that students perceiving usefulness and performance enhancements were likely to continue using the chatbot. Other constructs showed weak or insignificant effects in the presence of PE. Overall, results support Hypothesis H1 and align with UTAUT theory (Elkefi et al., 2024; Kajiwara & Kawabata, 2024; Venkatesh et al., 2003).

Figure 6
Figure 6.Chatbot Interaction Data for Spring '25 and Summer '25 semester
Figure 7
Figure 7.UTAUT Hypothesis and SEM Results

5.2. Content analysis results

Responses to open-ended questions were analyzed using content analysis and organized into four themes: benefits, concerns, rationale for non-use, and proposed use (the latter two addressed only non-users).

5.2.1. Benefits of using the chatbot

Of the 495 valid responses from Spring '25 and Summer '25, 294 (59.4%) students had used the chatbot at the time of survey rollout. Fig. 8 illustrates the key themes. 265 (90%) responses mentioned quick query resolution. One student mentioned, “I feel like the chatbot is the quickest way to get answers about low to medium priority issues.” 118 (40%) responses were related to dependency reduction on TA and instructors (For example - “it saves time for both the user and the instructor as it eliminates needless meetings or emails for small issues”).

In 63 (21%) responses, repetitive queries were viewed by students on the chatbot discussion board (For example - “if we are stuck on something, it helps with that. Seeing people’s previous queries helps us to avoid them”). Other key themes are reliable information and summarization, with 39 (13%) and 17 (6%) responses, respectively (For example - “it helps with summarizing information regarding assignments and labs. It’s also nice to be able to get instant feedback and answers to questions without having to wait for a person to reply”). Only 9 (3%) responses were negative, stating that it was not helpful (For example - “I did not find a benefit; it gave me information that was not helpful to my problem”).

Figure 8
Figure 8.Key Themes on Benefits of using Chatbot

5.2.2. Concerns of Using Chatbot

Fig. 9 shows the key themes on concerns of using chatbot. 69 (23%) respondents expressed no concerns. A student captured the essence of this attitude: “I think it is a great idea as I think it surely decreases the number of help forms sent for simple stuff and allows students who need more complicated help to have a quicker response in the help forms and/or in office hours. I do not think I have any concern on it.” However, 159 (54%) respondents cited inaccuracy concerns (For example - “if the chatbot is not checked regularly by TA, it could be responsible for giving incorrect information and hurting a student’s performance”). Inaccuracy was related to the limited knowledge of the chatbot, which was highlighted by 87 (30%) respondents. Likewise, 58 (20%) and 28 (10%) responses were related to lack of trust and over-reliance on the chatbot.

Figure 9
Figure 9.Key Themes on Concerns of using the Chatbot

5.2.3. Rationale behind not using Chatbot

201 respondents (Spring '25 and Summer '25) did not use the chatbot at the time of the survey. A student claimed that “I don’t really have concerns; I just don’t have a use for the Chatbot.” 77 (38%) responses highlight lack of trust in the chatbot. 22 (11%) respondents who did not use the chatbot mentioned that TA were more helpful and that students prefer human interaction over chatbots. One student mentioned “I don’t have any concerns with it, but I’d rather ask a TA for help just because I can get the hands-on experience or explanation rather than something just written in text.” The remaining 18 (9%) responses showed that students were unaware of the chatbot.

5.2.4. Proposed use of the chatbot

124 (62%) respondents claimed that students would use the chatbot in last resort situations where there is no one to help. Another view is that they would use it if a quick solution is required. 37 (18%) respondents highlighted the accuracy of the chatbot responses as the reason for their skepticism. If the model underwent proper training and provided better answers and catered to complex queries, then students would be more willing to use it. In addition, 22 (11%) students mentioned that they did not feel the need to use it at all. Finally, 17 (8%) students mentioned that they were unaware that the chatbot was available. “If it was more talked about and utilized, I would use it more” says a student.

6. Discussion

Our study seeks to answer key questions posed in the prior work by direct implementation of the chatbot on a large engineering course to understand ease of implementation and perceptions of students for future use. The paper thus makes a significant contribution to the ongoing discussion on the application of GenAI-based virtual assistants in class by empirical firsthand evidence. It provides confidence and direction towards adoption by larger education community.

Firstly, RAG framework makes GenAI applications practical for specialized use (like each unique course in education), reducing hallucination while improving accuracy. The software developed by Edstem demonstrates that a chatbot system can be rapidly deployed by addressing two primary components: course document ingestion and prompt configuration (e.g., core prompts and disclaimers). The system is intuitive and straightforward to implement, even for instructors without technical expertise in deep learning or natural language processing. As a result, it offers timely instructional support for courses with limited staff and high volumes of repetitive student queries. Timely solutions enhance motivation and alleviate academic stress on students and reduce dependency on TA for easier queries. It also frees up TA time to focus on higher order problem engagement with students in class.

Secondly, survey responses from the chatbot user group indicate strong acceptance of the technology: 88% indicated that the chatbot should be introduced in other courses as well. The results provide substantial evidence of the chatbot’s perceived merit and usability. It aligns with similar research on students’ perception on adoption of GenAI with 89.69% students indicating future usage by (Elkefi et al., 2024) and 83% by (Kajiwara & Kawabata, 2024). The outcome of performance expectancy construct being significant to users’ adoption also aligns with related research on UTAUT framework for adoption. Based on the findings, the funding for the chatbot tool has been extended by the institution for the course.

Thirdly, qualitative findings align with prior studies examining students’ perceptions of GenAI and similar large language models. Key benefits reported in the literature, such as rapid query resolution, reduced dependency on instructors or peers, the ability to address repetitive questions without fatigue, reliable information access, and effective summarization, were empirically confirmed in this study. The usage of chatbot was comparatively more in asynchronous environment (Summer '25 semester) highlighting the demand of such technology in online or distance learning. Likewise, concerns related to inaccuracy, overreliance, and trust emerged during qualitative analysis, also consistent with existing literature.

Finally, the tool provides 24/7 access and enables interaction not only with the chatbot but also with peers through a discussion-board environment. This setup fosters collaboration, with the chatbot acting as an intermediary by allowing students to view previously asked questions, react to responses through upvotes or downvotes, and build on existing discussions. Additionally, the system supports the organic generation of frequently asked questions (FAQs), creating a reusable knowledge base that can serve as a reference for future course offerings.

7. Conclusion and Future work

The paper helps mitigate the gaps found in the literature for RAG based chatbot implementation by providing empirical evidence over a large course deployment for two semesters. The chatbot was able to answer a wide variety of queries and reduce the help-form mail queries by 57%. Students perceived improvement in study performance by quick query resolution and dependency reduction but were concerned with inaccuracy and limited depth of chatbot responses. The teaching assistants also observed significant reduction in repetitive queries in labs.

Overall, the positive outcome provides concrete direction towards deployment and large-scale adoption capability. The application of GenAI based chatbot is a promising step towards personalization of education. It provides flexibility and bandwidth to engage students that is practically not possible for courses with high enrollment. Thus, it helps in reducing stress and cognitive load on students. Students do not have to wait for an email reply, lab class or office hour for an answer. It helps to bring continuity in study and assignment solving which is otherwise broken due to unanswered queries.

This study has limitations as well. The course examined in this study was part of a pilot. Therefore, a convenience sample was used (students from the course). Additionally, as the course primarily serves engineering and technology students, there may be an inherent predisposition to adopting and engaging with technological tools. Future work should extend the pilot to courses across the university to obtain a more representative and diverse sample. Also, chatbot is found to be useful for a large enrollment course. Future work should include courses of standard size (25-30 students) enrollment for comparison.

Another limitation is that the SEM analysis loadings were not statistically significant for some constructs. Also, Summer '25 semester had inadequate sample size for SEM analysis. It highlights the need to expand the scope of data requirement for future study. Also, while the chatbot effectively addresses easy to intermediate-level queries, there remains room for improvement in response quality for complex queries. Future efforts will focus on enhanced training, validation, and deployment of the chatbot in subsequent semesters, along with a longitudinal study to examine how these improvements influence student perceptions, usage patterns, and learning outcomes over time.

As for next steps, key concerns like academic dishonesty and overreliance must be addressed as part of future research. Experiments are being designed by authors to understand the impact of overreliance and reduced critical thinking in students in alignment with psychological studies on human memory and learning. It will provide quantifiable evidence and determine course of action to prevent deterioration of students’ independent thinking.

GenAI addiction can quickly become a key problem leading to proliferation of cheating. Research on effective AI literacy training of different stakeholders (including students, teachers and parents etc.) must be conducted to educate them about the use and misuse of chatbots in education. It must inform the stakeholders with a clear AI usage policy implemented at every level (from institution to classroom) and rethink assessment strategies in the age of AI.