I have a Siberian Husky named Rollo who is trained in scent work – think bomb detection dogs without combustibles! When scent work dogs are first being trained, the odor is paired with a reward. Over time the dog learns that finding the target odor (and telling the handler) leads to something good – that reward. During training, the handler must be very intentional about what gets rewarded because the dogs learn to alert on exactly what we pay them for.
A common challenge in scent work training is when handlers try to train an alert behavior, such as a sit, called a trained response. This behavior follows the finding of odor. If the dog has a trained response, handlers will often rely on the dog giving this response to indicate that the dog has found odor. There are two things that happen when a handler is overreliant on the trained response. If the handler is not careful, they can end up rewarding the trained response and not the finding of odor. And more importantly, the focus becomes on the trained response and the process the dog goes through to find the odor becomes lost. The handler can no longer read the dog’s change of behavior when they are in an odor cone, when they have cataloged (identified but not yet told the handler) odor or even when they are problem solving the movement or convergence of odors.
I argue higher education faces a surprisingly similar problem. In education, grades are the rewards our students are trained to seek. Rubrics, participation policies, and assessments all communicate what we value. AI, however, complicates this relationship. When students can use AI to generate increasingly polished products, the artifact itself becomes a less reliable indicator of learning. The question is no longer what we reward, but whether the thing being rewarded actually represents student thinking and learning.
For me and my own teaching practice, AI has created a challenge given that I teach a fully asynchronous course introducing students to concepts in social media marketing. The final assignment in this class is to create a social media marketing plan for an organization of the students’ choosing. Whether agentic or not, AI is perfectly capable of adequately producing such a product.
When I tested agentic AI against my own course materials, the results were mixed. Some activities exposed clear limitations, but others produced something more concerning.
The AI produced perfectly acceptable drafts and the final marketing plan. While these submissions weren’t exceptional, they checked the boxes. The challenge is that the marketing plan itself may be the educational equivalent of the trained response: a visible outcome that can occur without revealing much about the thinking that produced it.
What I discovered wasn’t unique to a social media marketing course. Nearly every discipline is facing a similar challenge. AI can generate essays, presentations, discussion posts, coding solutions, and a growing number of other academic artifacts. If our assessments primarily reward polished outputs, then AI has exposed a weakness that may have existed all along. The question is no longer whether students can produce a deliverable. The question is whether we can design learning experiences that make their thinking visible.
These experiments forced me to consider that fundamental question – what am I actually trying to assess? Was the goal of the class to produce a marketing plan, or was I more interested in the students’ ability to think like a marketer? If it was the latter, then I needed to find ways to reward the process instead of the product.
For years, many of us have assessed the educational equivalent of the “sit.” We grade the essay, the discussion post, the presentation, or the marketing plan because those artifacts are tangible and relatively easy to evaluate. What is often much harder to observe are the behaviors that actually indicate learning: the false starts, the revisions, the questioning of assumptions, the defense of ideas, and the moments when understanding begins to take shape. The final artifact becomes the alert behavior, while the thinking that produced it remains largely invisible. AI did not create that reality. It simply made it impossible to ignore.
When my dog finds odor, I want to reward the detection, not simply the “sit” that follows. As educators we face a similar choice. If AI can produce a polished output, then I need to find a way to stop treating that output as evidence of learning. So, for me, the output, that social media marketing plan, was no longer the thing that told me if the students were learning. What I needed to do was to fundamentally redesign the class to allow me to assess the human work (and struggle) behind the product. I needed to find a way to assess the students’ evaluation of ideas, decision making, choice defense, assumption revision and context adaptation. In other words, their critical thinking. The challenge was figuring out how to do this in a fully online, asynchronous course where I had no ability to observe students’ thinking in real time.
As an example, the first assignment in the course asks students to create a bio statement for the organization they will be working with that is aligned with both the organization’s mission and adjectives the student has chosen to represent the organization’s “personality”. Historically students struggled with this activity either recycling the organization’s mission statement, copying and slightly modifying existing marketing language or generating long and unfocused statements. Despite feedback on the drafts, students rarely revised and seemed to miss making the connections I wanted them to make.
In a face-to-face class we would have time in class discussion and group work for me to witness the students’ thought process, their ability to defend the choices they had made and to help them make those connections. This felt like an insurmountable obstacle in the online space and one that I felt students were off-loading to AI.
If I wanted to assess thinking rather than just outputs, I needed a mechanism that would work in the online environment and could surface student reasoning throughout the assignment process. Enter an AI agent.
Using a Custom AI Bot to Make Student Thinking More Visible
To help my students make these connections without this real-time guidance, I created an AI-powered alignment coach. The coach is designed to understand my assignment requirements and will not do the assignment for the students. Instead, as the student presents their ideas, it asks questions, challenges assumptions and pushes students to clarify their thinking.
What surprised me most was not that the final products were better – even though they were – it was that for the first time ever, I was able to have insight into my online students’ thinking process. Because I had access to the transcripts of their conversations with the agent, I was able to “see” them argue and defend their choices, reject suggestions they disagreed with and more importantly revise and test their ideas refining their work through multiple iterations. While their submitted drafts were more polished and had greater depth, what was even more valuable was visibility into the process.
What the AI coach ultimately gave me was visibility into iteration. Previously I only saw the final version students submitted. Now I can see how an idea evolved. I can see the questions students asked, the suggestions they rejected, the assumptions they revised, and the compromises they made. In many cases the most significant learning occurred not in the final answer but in the changes that happened between the first and fifth draft.
Students can certainly attempt to game these interactions; this challenge is not unique to AI. Students have always been capable of submitting work completed with minimal effort. What these conversations provide is substantially more evidence of student reasoning than the traditional submission alone. Rather than reducing visibility into student thinking, the interaction increased visibility.
Ultimately, it also meant that my reward structure had to change. If I wanted students to engage in reflection, revision, and decision-making, then those behaviors needed to count toward the grade. Rather than evaluating only the final marketing plan, I began placing greater emphasis on the conversation, iteration, and evidence of reasoning that led to it.
One of the most exciting parts of teaching is watching a student move from a simple understanding of a concept to a more sophisticated one. In a face-to-face classroom, we witness these moments through conversations, questions, and spontaneous dialogue. Online learning rarely provides the same visibility. Even online discussions lack the richness of these in-person interactions.
The alignment coach wasn’t the only redesign added to the course to make student thinking more visible. I also replaced a traditional peer-feedback discussion, which often produced superficial comments, with a collaborative persona-building activity. Students collectively created audience personas that were later transformed into AI agents that they could interact with throughout the quarter. The result was not only a resource they could use later in the project, but also richer conversations about audience needs, assumptions, and strategy.
What emerged from these redesigns was an unexpected insight. AI wasn’t simply complicating assessment or creating opportunities for students to off-load their learning. It was creating new opportunities to observe learning that had previously been hidden.
This doesn’t mean that AI replaces human interaction, but it does mean that in online asynchronous environments we can use it to help us create and capture the parts of learning that matter most. Most importantly, if we are serious about preparing students for a world where AI can generate almost any artifact, then perhaps the most important question we can ask is the same one every dog trainer asks – what exactly are we rewarding?
Reframing this question helps make our courses more AI resilient and more importantly it helps us make sure that our classes feel more relevant and authentic. In scent work, experienced handlers learn to watch for the subtle changes in behavior that tell them a dog is working odor. The final alert matters, but the alert is only the end of a much richer process.
Learning works the same way. The essay, project, discussion post, or marketing plan may be the alert, but the thinking behind it is the odor trail. AI is forcing us to look more closely at that trail. And that may be one of the most valuable opportunities this technology offers educators.
