AI based Coursework Chat Assistant

This blog is written by Ramalakshmi Vaidhiyanathan and explores using AI to aid student’s assessment queries.

Ramalakshmi Vaidhiyanathan is a lecturer in the School of Computer Science and Informatics, with around 18 years of software engineering experience in the industry.

Introduction to Poe

“Believe nothing you hear, and only one half that you see”. – Edgar Allen Poe

I happened to see a demonstration of Poe, an AI chatbot aggregator platform in one of the Digi Fest sessions. The demonstrator explained how they have been using the tool in a higher secondary school environment where students were provided this tool to do some self-paced learning about a particular topic from specific knowledge sources. Some of the knowledge sources provided were books downloaded as PDF. It seemed too fantastical but the fact that it was demonstrated live was enough to pique my interest and get me thinking. I just made note of the tool name and left it at that. This was a time when I had just joined Cardiff University and had taken over someone else’s module to teach. I was trying to personalize the content, changing the materials to align with my style of teaching. The assessments were already moderated and released to the students. But the students were yet to start working on them. I was warned of the barrage of questions that would be coming my way once the students start them. But before that I needed time to go through the coursework assessment and the solution guidance to prepare myself. This was when I decided to try out Poe.

Practicing Poe

“It is a happiness to wonder; — it is a happiness to dream.” – Edgar Allen Poe

I started with the free version using a Gmail account I had created for this purpose and created a “prompt bot”.

This effectively creates a chatbot much like the “Hi there!” style assistant bot you see in various websites. This prompt bot used the default base bot of “GPT 3.5 Turbo”. I provided the coursework and solution guidance as its knowledge source (after removing the university related data like email addresses etc.) and a very minimal prompt to act as my assistant to help the students understand the coursework (thereby helping me understand too).

Once I created the chatbot, I tested it by asking questions related to the coursework just as a student would. I tried combinations of straightforward questions about what a task was about, what are the word limits, how I would achieve highest mark, asking for direct solutions, and indirect way of getting answers etc. I had to learn a bit more about prompt engineering to fine tune my initial prompt to the bot to ensure that it doesn’t reveal the solutions directly or too much in depth but enough to help the students. In some cases, it gave some basic answers, especially for questions which were of low points, but I was alright with that. Poe also provides options to control the creativity of the bot’s responses to prevent hallucination. I had set it to the default of 0.5, but if we did lower it, we would get more consistent responses.

Poe is part of Quora, and it needs an account to start with. The privacy concerns are like any other Gen AI tool and technology available. The data will be saved to enhance its performance and since Poe is an aggregator and uses a base bot, it will in turn refer to the privacy concerns of the base bot.

Coursework Assistant Bot at your service

“Let my heart be still a moment and this mystery explore..”. – Edgar Allen Poe

Once I was convinced of its performance, I shared the bot with the students after week 9 by making it publicly accessible and asked them to use it as an additional tool for any queries regarding the coursework especially during the time when I won’t be able to answer their queries especially during out of office hours. I did provide a disclaimer that since I won’t be able to see their prompts and the responses from the bot, which I will come to later, the students were asked to send me the same question and get an official response from me. Whenever I received a question from the student, I tested it against the bot and verified its response. If it was dissatisfactory then I updated the knowledge source with the related information. It is possible to add additional knowledge source using plain text as and when required, which I found very helpful. Once the semester was done and dusted, I sent out a feedback form to the students to provide their opinions on the Coursework Assistant bot. I received only three responses and all of them were in favor of trying out such bots in the future. One of the comments on the performance of the bot was “I was sometimes questioning whether the bot was giving correct interpretations of the coursework, and I would normally prefer emailing the module leader anyway to get a definite answer”. This showed the student’s distrust in Gen AI tools especially if it has a direct impact on their grades. The students do use Gen AI tools to write their assessments like the programming exercises and reflective questions, but they do not yet trust the Gen AI tool which acts as a digital twin to their module instructor.

Pros and Cons of Poe

“There is no exquisite beauty… without some strangeness in the proportion.” – Edgar Allan Poe

• It is part of Quora and hence privacy is of some concern. I had asked the students to use their university email id, or an email id created for exclusive use to use the coursework assistant bot I had created. The coursework and the related knowledge sources provided had all private personal information (PPI) removed and did not contain any copyrighted information but if they do then Poe might not be the right tool to use.
• It is an aggregator and hence uses a specific Gen AI of our choice. It has a few free options available and most of them were fit for my purpose. The free version has limits on the number of prompts per day (30) and the free base bot one can use. It doesn’t seem to have a limit on the knowledge sources though.
• Once shared publicly, it does not allow the bot creator to know of the various prompts that were asked of it and the responses that were provided by the bot, unless the user wants to share the chat. I am sure the students wouldn’t volunteer to share their prompt and responses with me, but I am planning to ask some students to help me this year by providing me the responses for better understanding and evaluation of the tool.
• It is highly configurable and allows the creator to fine tune most of the specifics of the bot, like the temperature of the responses, turning of recommendations of other bots, not allowing suggestions in replies etc.
• It is easy to create a prompt bot and requires less technical know-how on Gen AI and LLMs and has faster turnaround time. If one has sufficient prompt engineering skills, it is easy to create a prompt bot. Curating the knowledge sources and ensuring that no PPI is part of the knowledge source is the only part of this exercise that is time consuming.

Marching on

“It is a happiness to wonder; — it is a happiness to dream.” – Edgar Allan Poe

Even though Poe was introduced to me as a learning aid, I decided to make it my digital twin, sort of. The first attempt was a rushed one since I was very new to the module. I had very less time to test the tool and create a bot that was fit for purpose. I barely scratched the surface of this tool and its potential. I am planning to use it again this year and I have already created a chatbot for my coursework with much tighter measures in place. I have currently provided the chatbot to other instructors and teaching associates of my module for testing. When I asked my students about the use of such tool as their aid, I received similar responses to last academic year’s.

I am planning to release the tool to the students after week 7 and test them live during some of the tutorial sessions to evaluate the performance of the chatbot. I dream of a time when this chatbot would respond to all the repetitive coursework queries leaving me to provide specific individual help where and when necessary. Fingers crossed!