VIRTUAL TUTOR OVERSIGHT

• Summary: Inappropriate Interactions – The AI “virtual mentor” chatbot not only became overly relied upon by students (even preferring it over human help) but also insulted a student with inappropriate language, which is clearly an unacceptable interaction. This raises concerns about the chatbot’s content moderation and the social dynamics in class.

• Training/Awareness Activities – The school should train staff and students on the proper use of AI tutors and their limitations. Teachers need to supervise AI interactions (“human in the loop”) and be ready to intervene. Additionally, the AI tool itself should be vetted (perhaps via an ethics or content review process) to ensure it has proper safeguards against inappropriate responses. By raising awareness, students will know to report issues and not to treat the AI as an infallible source, and teachers will be prepared to monitor and guide AI usage.

• High Risk – An AI insulting a student is a serious incident that can harm the student’s well-being and the classroom environment. It’s also a safeguarding issue – the school provided a tool that caused emotional harm. Moreover, overreliance on the AI tutor might hamper students’ interpersonal skills or willingness to seek help from humans. These factors present a high risk to student welfare and require prompt attention.
   

• Decision: The school would not accept a scenario where a school-sanctioned tool can potentially harm or mislead students. To mitigate, they might establish stricter oversight: for example, limit the AI’s use to certain contexts, configure the AI with stricter content filters, and ensure a teacher is actively monitoring interactions. Training and clear guidelines will be put in place so that AI is a supplemental resource, not a replacement for teacher support. By mitigating the risk through these measures, the school protects students and maintains a healthy, respectful learning environment.