This insightful guest post by Dr Joanne Chia and the Director of the LCC, Dr Angela Frattarola, offers an inside look at their team’s journey in developing the GenAI-enabled Waai app. Their goal is to support student writing and thinking in the age of ChatGPT through strategic AI integration. By examining user needs, engineering smart prompts, and optimising functionality, they aim to create an AI tool that promotes authentic student voices. This post provides a fascinating case study of the pedagogical and technical considerations involved in designing educational AI to facilitate discovery, inquiry, and self-directed learning. For instructors exploring mindful AI integration, it opens up thought-provoking conversations on harnessing technology to enhance thinking and engagement.
Background
Three LCC colleagues were chatting about Mentimetre one day and its then new asynchronous feature. Such a Mentimetre sequence “interacts” with students by offering set explanations based on student responses to various questions. However, the application lacks branching, something that can only be found in gamified frameworks. It also does not allow users to navigate to different parts of the sequence. It was these limitations that inspired us to apply for an EdeX grant, with the goal of creating an app that would allow students to receive immediate feedback on their ideas. While there are several “writing buddy” apps out there that check grammar and language usage, we were hoping to create something that could assist students throughout the writing process, from brainstorming ideas to final editing. We envisioned an app in which students could upload their writing throughout the semester. It would prompt students on brainstorming techniques, self-evaluation, and at times, draw on AI to help students develop their ideas.
Before we began designing the app, we surveyed about 240 students. This survey showed that students would be excited to try an app that might assist them through the researching and writing of papers. It was surprising to us, however, to also learn that students valued their own voices over the seeming convenience of LLMs like ChatGPT. In the survey, students repeatedly expressed concern about losing their own voices if they relied on an AI writing tool.
Target audience
We decided that the students who most needed such an app would be the wide range of students taking CC0001: Communication and Inquiry in an Interdisciplinary World. This is a general writing class that all NTU students take as part of the Interdisciplinary Collaborative Core. Engineering students, possibly the largest group of students taking CC0001, are often taught to value and follow a sequence. We tried to cater to this learning approach by using a design thinking framework throughout the app. Humanities students, while very much acquainted with writing, tend to write from a fixed thesis statement rather than from an open-ended question. CC0001 was designed to defamiliarise the writing process for students, requiring that they first engage with a specific lived experience, record and analyse their observations, and then formulate a conceptual question that, while not definitively answerable, requires research. This design, which roots student writing in their experiences and individual inquiry, has become even more relevant with the threat of ChatGPT. Moreover, this movement away from thesis statements and into self-directed inquiry is something of a cultural shift for many students regardless of discipline. The web app, thus, aspires to support students through this challenging shift.
Pedagogical framework
CC0001 assignments are scaffolded, meaning that students only get to the final essay by following a series of steps that begins with fieldwork and ends with an exploratory argumentative essay. The course is premised on the idea that writing is a tool for thinking, and we strive to give each student the experience of discovering their argument through the messy process of researching, drafting, and revising.
Design Thinking as a non-linear process has been adapted as a framework for our app. While students are not problem-solving in the way typically seen in design thinking, they are experimenting with different approaches to form an insightful argument that responds to an unanswerable research question of their own design. As with design thinking, this process begins with empathising with an audience.
Incorporating AI into the app
The tagline for the app is “For the hardest questions, ask Waai.” The pairing of AI with design thinking is an interdisciplinary approach that has not really been explored. The ideation stage for the app was probably the hardest part of the process, taking a few months before we could pin down what the app pages would even look like. In a way, we were following a design thinking process ourselves.
The structure of the app follows the sequence of assignments planned in CC0001 so that students, while testing out the app asynchronously, would be able to follow a similar sequence of tasks. At the same time, a side panel or menu based on the different design thinking stages allows students to move forward and backward in their brainstorming process.
Based on the coursebook guidelines written for both teachers and students, pre-designed prompts were created that would mediate students’ engagement with AI. We discovered that the prompts were the heart of the app and that was where we wanted to control what kind of information students get. For instance, if students were to receive a list of ideas to consider, this might limit or overly influence their ability to generate their own ideas. Instead, we wanted the AI to guide students to just brainstorm ideas or concepts. The prompts therefore would, in theory, help isolate these ideas and concepts. App users would be able to choose from a dropdown list of prompts to aid their brainstorming. When designing these prompts, establishing backend prompts was equally instrumental to the success of the AI tool in the app. Our app developer, Yajat, a Year 2 student from the School of Computer Science, was central to coming up with the initial backend prompts through, perhaps ironically, the help of ChatGPT.
The AI chatbot named “Nudgey” was designed from scratch by our UI/UX designers, Azra and Alicia, Year 2 students from ADM. Its drop-down menu allows users to accumulate the “chats” they had with the chatbot, based on the prompts they had chosen, in order to revisit different parts of the process. Although it has a “save” function that stores all information provided by users, from their areas of interest to their source summaries, in a future version of the app, we hope to allow users to “pin” these chats. This way, students could select which chats they would like to carry forward with them as they continue their ideation journey.
Prompt Engineering
The following table offers a sample of “Nudgey” AI prompt templates available in relation to the first graded assignment students submit for the module:
|
Prompts for “Introduce Op-Ed Page” |
|
|
Frontend prompts |
Backend prompts |
| “Help me uncover abstract concepts from my AEIOU observations.” | “Abstract ideas are ways of thinking based on general ideas rather than on real things and events. Your goal is to review the AEIOU observations provided, understand their context as a writer, and offer three possible abstract concepts to analyse.” |
| “Help me uncover useful definitions of concepts relating to my AEIOU observations.” | “Definitions are statements that express the essential nature of something. Your goal is to review the AEIOU observations provided, understand their context as a writer, and offer three possible definitions to analyse.” |
| “Identify any patterns or trends in my observations that I might have missed.” | “A trend is something that happens for a reason, while a pattern is the observation of things happening in a specific sequence that is predictable in some manner. Your goal is to review the AEIOU observations provided, understand their context as a writer, and offer three possible patterns or trends to analyse.” |
| “Give me phrases that can emphasise the five senses from the AEIOU observations.” | “Phrases are a small group of words standing together as a conceptual unit, typically forming a component of a clause. Your goal is to review the AEIOU observations provided, understand their context as a writer, and offer three possible phrases to analyse.” |
The vocabulary used follows that of the inquiry process for CC0001 that guides students toward an original topic. Students are asked to introduce their Op-Ed authentically through the use of vivid description that contains sensory details. Students are also asked to zoom in on a seemingly insignificant detail to avoid pursuing an obvious topic. As a result of this initiative, either a pattern or break in pattern may be discovered that could lead to a genuine problem or puzzle.
Our prompt templates were engineered to avoid the spectre of “AIgiarism” as well as scenarios typically discovered if we otherwise put the writing into ChatGPT and asked it to simply analyse it. We want to avoid having the AI evaluate the student’s writing in a way that might contradict the teacher’s assessment. One possibility is that ChatGPT will come back with a whole list of possible ideas for students to pursue which could suppress their own voice. A connected possibility is that ChatGPT would give a lot of information rather than help students focus on a particular detail.
The prompts were initially tested by the team, and it was discovered that the feedback that came back from the AI reflected greater engagement with the ideas of students themselves rather than with ideas more broadly connected to a particular topic. As such, the prompts also help students zoom in on a particular topic in the form of a suggestion that they could either take on or leave behind. For instance, the prompt could help narrow down a topic on the value of photography to one on the influence of the visual medium on social media platforms. This may or may not be a better direction to take depending on how saturated the topic would already be. Moreover, this process of analysis leads students to identify particular concepts that they would need to consider as they proceed to think of a research question they could ask in their Op-Ed.
A discovery in our initial testing phase was that the AI feedback received for the same prompt with the same writing input could vary across different occasions. This is not caused by the learning of the AI in our case as we are subscribed to and using the Azure OpenAI service which does not use or train any of the app data or user data. At the same time, the probabilistic nature of the AI that emerges suggests that the “hallucinations” that might occur for an LLM could work to our advantage. In our case, these “hallucinations” do not pose a challenge; rather, they ensure randomness or creativity in terms of the feedback students get. This is useful as students would be able to receive even more customised feedback.
Implications for teaching writing
This app is highly experimental and would require conclusive results to determine if it actually enhances the writing and thinking process. It is certainly not the first writing app to attempt to incorporate AI to achieve formative goals. Our vision, however, goes beyond perfecting a writing piece to generating and bringing ideas to life.
The reception so far from CC0001 test users from the current Semester 1 2023-2024 batch is mixed – some find it amazing and feel that it should not be given free of charge, while others find it more restrictive than directly interacting with ChatGPT. While this app does not hope to enhance any existing LLM, nonetheless, it hopes to investigate the connections between writing and thinking in the context of a rapidly developing AI-centred world that is here to stay.
|
Waai Research Team Yajat Gulati (Year 2, SCSE) Putri Azra Besim Kukuljac (Year 2, ADM) Alicia Ng Ying Xuan (Year 2, ADM) Choo Shuen Ming (Year 4, SoH) Dr Angela Frattarola (Director, LCC) Asst Prof Lisa Winstanley (ADM) Dr Joanne Chia (Lecturer, LCC) |
