AI Teaching Assistant Pro
I am in the process of evaluating a suite of AI tools from Contact North, Ontario, for students, instructors and administrators.
Contact North offers a suite of six AI tools for teachers/instructors. In my previous post, I evaluated one of these tools, a tool for generating multiple choice questions and answers. In this post I will evaluate their tool for generating essay questions and a rubric for answers.
Generate essay questions with a scoring rubric on any subject
This tool allows you to enter a topic for an essay or to upload a file, choose the number of essay questions you want, the academic level (Introductory, Intermediate, Advanced), and a choice of language, with the option of adding a scoring rubric.
As well as entering a topic, you can also upload a text on which the tool will sets questions and rubrics. (The file though needs to be less than 30 MB, which will preclude loading a whole book).
What I did
I set ‘introductory level’.
- I set as one topic ‘Alexandre Dumas’ The Count of Monte Christo.’
- I uploaded Section 1.2 ‘The Skills Needed in a Digital Age‘ from my online textbook, Teaching in a Digital Age.
For both activities, I received a set of three essay questions and a set of marking rubrics for each question.
The Count of Monte Cristo
The questions generated for ‘The Count of Monte Cristo’ were:
- Discuss the theme of revenge in The Count of Monte Cristo. How does Edmond Dantès’ quest for vengeance shape his character and the narrative of the story?
- Analyze the role of fate and free will in The Count of Monte Cristo. How do these concepts influence the decisions made by the characters throughout the novel?
- Examine the significance of friendship and loyalty in The Count of Monte Cristo. How do these values affect the relationships between characters and the overall plot?
The Skills Needed in a Digital Age
- Discuss the essential skills needed for effective communication in a digital age, particularly focusing on the role of social media. How do these skills differ from traditional communication methods?
- Analyze the importance of developing soft skills in the context of a knowledge-based society. What challenges do educators face in teaching these skills effectively?
- Evaluate the role of technology in developing critical thinking and problem-solving skills in students. How can educators effectively integrate technology into their teaching practices to enhance these skills?
Rubrics
There were scoring rubrics provided for each question, as follows:
- Excellent (9-10): a thorough analysis, insightful examples, strong grasp of the topic
- Good (7-8): discusses the importance of specific themes, and identifies some challenges. It includes relevant examples but may lack depth in analysis or specificity.
- Fair (5-6): The response touches on the importance of the topic but provides limited discussion on the challenges. Examples may be vague or not directly related to the topic.
- Poor (3-4) The response discusses the topic but lacks clarity and depth. There is minimal discussion of the challenges, and examples are largely absent.
- Inadequate (1-2) The response fails to adequately address the question, providing little to no relevant information or examples or the challenges.
It should be noted that the scoring rubrics were very similar for both topics, although the subject matter was very different. In other words, these are general rather than specific rubrics.
My evaluation
1. Target group (Scale 0-5)
Is it clear who should make use of these tools and for what purpose?
Yes, this is clearly a tool for any teacher or instructor who wants to set essay questions. Teachers or instructors though will need to define the topic or choice of uploaded document at the level appropriate for their students.
I give this a score of 4 out of 5 on this criterion.
2. Ease of use (Scale: 0-10)
- Is it easy to find/log in? Yes, just click and enter your topic or upload an existing document (this may mean copying and pasting into Word or pdf).
- Is it easy to generate questions? Yes, once you have set level and the topic or provided an uploaded file – just click and see
- Does it provide the necessary information quickly? Yes, extremely fast. You get the rubrics at almost the same time as the questions.
- Is it easy to make use of the questions and answers it provides? You will need to check carefully the questions offered and especially the rubric to make sure they are fit for purpose and actually match what you have been teaching (see (3) below.)
I give ease of use a score of 9 out of 10.
3. Validity and specificity of questions and answers (Scale: 0-10)
- How valid and relevant were the questions, given the topic? If you have read ‘the Count of Monte Cristo’, you will be able to judge as well as I can whether the questions meet this criterion. I did study the book for my high school final exam (and re-read it recently), and I would say the questions do target appropriate topics but I would like to hear from specialists in French literature on this one. However, I did have a problem with the questions on my own book. I say very little about social media in this section – they are just one example I give of many 21st century skills and I don’t really compare them to other communications media. Also I don’t discuss the relationship between technology and skills until much later in the book. I had a feeling that the questions were derived more from the general literature than the specific text I uploaded.
- Were the questions at an appropriate level of difficulty? Since these questions were set at an introductory level, I thought they were rather challenging. When I reset the one on skills at an advanced level, I found they were more specific to the actual chapter than the introductory level response.
- Were the rubrics helpful? Not really, other than giving some very general criteria for scoring. These could be adjusted for any question, but would not provide particularly helpful guidelines to an instructor wanting to assess whether students really got to the ‘meat’ of a question.
- Does it provide a range of possible acceptable rubrics where this is appropriate? No, it provides a very generic template of rubrics.
- Does it provide relevant follow-up questions or activities for either students or instructor? No.
I am giving this 6 out of 10. In general the questions were good, but the rubrics were not specific enough to the actual questions. I would recommend that instructors and teachers use the tool for generating questions, check that the questions are specific to what has been taught in the class, and develop their own rubrics for marking.
4. Likely learning outcomes (Scale: 0-10)
- provides accurate/essential assessment on the topic/question (0-3 points) I am giving this 2 out of 3. The questions do test some of the key issues in the topic but may lack some validity to specifics in the teaching.
- helps with testing key concepts or principles within the study area/topic (0-3 points) I am giving this 3 out 3. The questions do support analysis and critical thinking, but the instructor will still need to make a personal assessment of the essay.
- enables/supports critical thinking about the topic (with: max 5 points) or without (max 3 points) good feedback. I give this a three. The questions do require critical thinking to answer and test higher order learning but the rubrics are not specific enough to be useful. (However, it’s good for teachers to have to think for themselves about what rubrics they want to use when marking essays).
- motivates the instructor (1 point). Yes, I think instructors will find this helpful, especially if they use the questions as a starting point, then refine the questions to fit specifically their teaching on this topic.
Total score: 9 out of 10
5. Transparency (Scale: 0-5)
Where do the questions and rubrics come from? Who says? Does it provide references, facts or sources to justify the questions and the scoring rubric it provides? What confidence can I have in the information provided?
As usual with AI tools, it is not completely transparent where the questions come from. For instance, are they based solely on the document provided or do they range across the whole Internet discussion on this topic? I suspect the latter, especially for the rubrics, which means instructors should check carefully the match between question and what was actually taught (although it may be a useful guide for future teaching). I give this a score of 1 out of 5.
6. Ethics and privacy
I don’t see any issues here. It is a useful tool for guiding instructors on essay assessment, but they will need to show due diligence to check the validity of the questions against their own teaching. The essay questions seemed to me to be valid within the subject domain and would enable accurate assessment of learning by the instructor, but they should at most use the rubrics as a rough guide to scoring.
These tools require no data from either students or instructors.
A warning though. Students will be able to use general AI LLMs such as ChatGPT to search the answers to the essay questions and it will be difficult to detect, so you may need to ask your students to give examples that are likely to be unique to your course, if that is possible.
I give this a score of 4 out of 5.
7. Overall satisfaction (Scale: 0-10)
I like this tool, so long as it is used by teachers as a guide to essay assessment and they evaluate the questions against their actual teaching objectives. The rubrics though are not particularly helpful apart from the five categories of scoring. It would be best for teachers to develop their own scoring rubrics within the five ranges of score. I give 7 out of 10 for overall satisfaction
Overall evaluation
I give this a total score of 40 out of 60 – roughly 67%. This is a useful tool in providing guidance on appropriate essay questions. It allows for testing higher levels of student learning, but it does need to be used with care, as a support rather than a replacement for teacher or instructor assessment. Also the tool lends itself to cheating by students. Overall though I do recommend its use, with care.
Over to you
Have you used this tool? How useful was it to you? What are the drawbacks? Please use the comment box at the end of this blog post.
Up next
I will be looking at the tool for developing a course syllabus – and a heads up that I think this is a fantastic tool for teachers and instructors.
Dr. Tony Bates is the author of eleven books in the field of online learning and distance education. He has provided consulting services specializing in training in the planning and management of online learning and distance education, working with over 40 organizations in 25 countries. Tony is a Research Associate with Contact North | Contact Nord, Ontario’s Distance Education & Training Network.
