Today we will look at how to use Evals in Microsoft Copilot Studio. We will first build a basic agent, give it some simple knowledge, and then create an evaluation test set to see how well the agent performs.

Testing an agent manually in the test chat is useful while we are building it. But as our agent grows, we don’t want to keep typing the same 10, 20, or 100 questions every time we change an instruction, update a knowledge source, or add a new tool. This is where agent evaluations, or evals, become useful.

Copilot Studio lets us create reusable test sets containing questions or conversations, run those tests against our agent, and evaluate the responses using several different methods. We can then make changes to the agent and run the same tests again to see whether the agent improved or whether we introduced a regression.

Note: Agents created in the new GitHub Copilot harness do not appear in the standard Copilot Studio editor. The two harnesses are separate, and agents cannot currently be converted between them. We will run into scenarios in both editors below.

What Are Evals?

An eval, short for evaluation, is essentially a test for an AI agent. In some ways, they are similar to automated tests that we are used to in software world. The difference is that traditional automated tests usually check something deterministic, e.g. Input A -> Expects Output B.

With AI, the output can be correct even when the wording is different. So an eval is more like: Input A → did the AI produce an acceptable result according to our criteria?

With evals, we can give the agent an input such as:

What is the maximum hotel rate I can expense?

The agent gives us an answer. We then evaluate that answer against some definition of what a good response should look like.

For example, perhaps we expect the answer to communicate:

The maximum reimbursable hotel rate is $250 per night before taxes.

Instead of manually checking this question every time we update the agent, we can save it as part of an evaluation test set and run it repeatedly. Copilot Studio calls these collections of scenarios test sets.

This is an important shift in how we build agents.

Instead of asking:

Does the agent seem to work?

We can start asking:

Can the agent consistently pass the scenarios that matter to us?

Build a Basic Agent for Our Evaluation

Before creating an eval, let’s build something simple that we can test. In this example, we will create an Employee Travel Policy Agent. We want the agent to answer questions about a fictional company’s travel policy. Keeping the example small makes it easier to understand what our evaluation is actually testing.

Open Microsoft Copilot Studio, select your org, and create a new agent. We will call our agent:

Contoso Travel Policy Agent

And for the instructions, enter something similar to this:

You are the Contoso Travel Policy Agent. Answer questions about Contoso employee business travel. Be concise and helpful. Only answer using the travel policy information provided to you. If the policy does not contain enough information to answer the question, say that you do not have enough information rather than making up an answer.

The last part is important. We don’t only want to test whether the agent knows the correct answers. We also want to test whether the agent knows when it doesn’t have enough information.

Add Some Knowledge to the Agent

Next, we need to give our agent something to answer questions from. In a real implementation, this might be a SharePoint site, Dataverse, a website, uploaded files, or another enterprise data source. For this demo, we will keep things simple and create a small travel policy.

Our fictional policy will contain the following information:

Contoso Employee Travel Policy

Flights

Employees may book economy-class flights for domestic travel.

Business class is permitted for international flights longer than eight hours and requires manager approval.

Hotels

The maximum reimbursable hotel rate is $250 per night before taxes.

Meals

Employees may claim up to $75 per day for meals.

Alcohol is not reimbursable.

Ground Transportation

Taxi, rideshare, rental cars, and public transportation may be reimbursed when used for business purposes.

Approval

International travel requires manager approval before booking.

We can put this information into a simple document and add it as a knowledge source.

Here I have this as a text file:

And this text file is uploaded to the agent:

Let’s remove “Search all websites” as well.

Once the knowledge source is available, save the agent.

Test the Agent Manually First

Before creating an evaluation, we should test a few questions manually.

Select Preview, then and ask:

What is the hotel limit?

The agent should give us an answer similar to:

The maximum reimbursable hotel rate is $250 per night before taxes.

Now try:

Can I fly business class internationally?

The agent should explain that business class is permitted for international flights longer than eight hours and that manager approval is required.

We can also test a question that is not covered by the policy:

Can I expense a gym membership while traveling?

Ideally, the agent should tell us that it doesn’t have enough information in the travel policy to answer the question.

At this point, our agent basically works. Let’s create an evaluation.

Create an Evaluation

You can follow along the directions below for the classic and new Copilot Studio. There are some things we will run into along the way, but the main thing here is to understand the concept of evaluations as we see more features in the UI.

In the new experience, evaluations are built around conversations. We can generate conversations automatically, import them from CSV, or write our own. Copilot Studio then runs those conversations against our agent and scores how well the agent performs.

At the top of Copilot Studio, select Evaluate:

Note the Copilot Studio Classic Editor looks a little different here:

Click + Create your first evaluation. There are several ways we can start building our evaluation.

The first option is to upload conversations from a CSV file. This can be useful if we already have a set of real or synthetic conversations that we want to evaluate.

But for our first test, we can use something much easier.

Generate a Quick Conversation Set

Under More ways to start, select Quick conversation set

Copilot Studio will automatically generate 10 conversations based on the agent’s description, instructions, and topics.

This is a nice feature because we can create a useful first evaluation without manually designing every conversation.

For our travel agent, Copilot Studio might generate conversations around scenarios such as:

• hotel reimbursement limits
• meal expenses
• business-class eligibility
• international travel approval
• ground transportation

We should review these before running the evaluation.

AI-generated test cases can save us time, but they may not cover every scenario that is important to our business. We can edit the generated conversations and add our own cases before running the evaluation.

Clicking into one of the tests, we can see the expected conversation. These can be edited, more on that later:

Write Our Own Conversations

We can also choose to write questions and conversations ourselves. Select Add Conversations->Write:

For our travel agent, we might create a conversation such as:

User

Can I book a $300 hotel for my trip?

Expected agent behavior

The agent should explain that the maximum reimbursable hotel rate is $250 per night before taxes.

Here we can add our user question:

And then optionally add an agent response.

Or we might test a more conversational scenario:

User

I’m flying internationally.

Agent

How long is your flight?

User

Nine hours.

Expected behavior

The agent should explain that business class is permitted for international flights longer than eight hours and that manager approval is required.

Configure the Test Set

Once we have our conversations, we can configure the evaluation. On the right side of the evaluation page, Copilot Studio displays the Configure test set panel. We can give our evaluation a name such as:

Travel Policy Core Evaluation

Then we can either save the evaluation for later or select Evaluate to run it immediately. I clicked Save, and then Run.

What Does Copilot Studio Evaluate?

Once the test set runs, Copilot Studio evaluates the conversations and produces scored results. The important thing here is that we are no longer simply asking:

Did the agent produce the exact sentence we expected?

Instead, we can evaluate the quality of the overall interaction.

For example, did the agent:

• provide the correct information,
• use the appropriate tool,
• follow its instructions,
• maintain context throughout the conversation,
• and avoid making up information?

This makes evaluations much more useful for agents than traditional string-based testing.

Review the Evaluation Results

After the evaluation completes, we can inspect the runs and their scores.

A FAIL means the agent did not meet the criteria defined for that evaluation. It is a signal that we should investigate, not necessarily proof that the agent is broken. We should look at the agent response, the expected behavior, and the evaluation criteria to determine whether we need to improve the agent or adjust the test.

PASS = the response met our evaluation criteria.
FAIL = the response did not meet our evaluation criteria.
ERROR = something went wrong.

Suppose in your eval your expected answer is:

The hotel limit is $250 per night.

But your agent responds:

The hotel limit is $300 per night.

That is a meaningful FAIL: the agent gave the wrong answer.

However, imagine the agent responds:

Employees can be reimbursed for hotels up to $250 each night.

In our example, the user asked if WiFi is included as a reimbursable expense, and the agent did not have that knowledge to answer. So we get the fail based on “One or more questions not answered“:

In this case, it’s up to us if we want to add WiFi to the policy, or specifically handle the abstinence by fixing the eval.

Let’s fix the eval.

Edit the conversation and add an expected agent response that makes the desired behavior explicit, for example:

The travel policy does not specify whether Wi-Fi, in-flight internet, or baggage fees are reimbursable. The agent should state that there is not enough information in the available policy to answer this question and should not invent a reimbursement rule.

That makes the test intent much clearer.

Select the conversation by closing the results and to Configuration, then open the conversation:

If we still get a fail and the test did not treat abstinence correctly, we can create our own custom evaluation criteria.

Create a Custom Evaluation Criteria in Classic Editor

To do this in the Classic Copilot Studio Maker Experience, add a Test Method:

Select Custom:

Name: Correct Abstention

Evaluation instructions:

Evaluate whether the agent correctly handles questions that are not answered by the available travel policy.

Pass when:

• The agent clearly states that the policy does not provide enough information to answer the question.
• The agent does not invent or infer a reimbursement rule that is not supported by the policy.
• The response remains relevant to the user’s question.

Fail when:

• The agent claims that Wi-Fi or baggage fees are reimbursable or not reimbursable without supporting policy information.
• The agent invents policy details.
• The agent does not address the user’s question.

Then create 2 labels:

Now when you run the evals again, it should account for abstinence. You can then iterate through the process to correct each eval.

Copilot Studio lets us compare evaluation runs over time, which means we can make a change to our agent and then see whether that change improved or degraded its behavior.

This gives us a development loop that looks like:

Build → Evaluate → Review → Improve → Evaluate Again

That is exactly where evals become valuable with regressions.

Choose an Evaluation Method

Back in the Test Method, we can choose our evaluation methods. Different tests may need to be scored differently.

Copilot Studio currently provides several evaluation methods, including:

• General quality
• Content safety
• Tool use
• Keyword match
• Custom

Different methods are available depending on whether we are using single-response or conversational test sets. You can read more about the types of test methods here: Choose evaluation methods – Microsoft Copilot Studio | Microsoft Learn

Test Different Ways of Asking the Same Question

Another useful approach is to test variations in how users ask questions.

A carefully written test might say:

What is the maximum reimbursable hotel rate?

But real users might ask:

What’s the hotel limit?

How much can my hotel cost?

Can I book a $275 hotel?

What’s our nightly allowance?

Does the company cap hotel prices?

All of these questions are testing roughly the same concept, but they exercise the agent differently. A strong test set should include realistic wording, abbreviations, vague questions, conversational language etc.

Evals and ALM

There is another interesting capability here. We don’t necessarily have to run every evaluation manually from Copilot Studio.

Microsoft provides Power Platform REST APIs that can trigger existing Copilot Studio evaluation test sets programmatically. The APIs can retrieve test sets, start evaluation runs, and retrieve results. This means evals can eventually become part of our release process.

Our workflow could look like:

This is where agent engineering starts looking increasingly similar to traditional software engineering. We don’t simply deploy because the agent looked good in one test conversation. We can build repeatable quality gates around it.

Some Power Platform APIs available include Get agent test sets:

https://api.powerplatform.com/copilotstudio/environments/{EnvironmentId}/bots/{BotId}/api/makerevaluation/testsets?api-version=2024-10-01

And run a test set:

https://api.powerplatform.com/copilotstudio/environments/{EnvironmentId}/bots/{BotId}/api/makerevaluation/testsets/{TestSetId}/run?api-version=2024-10-01

You can also get runs, run details, etc.

Note also that Evals don’t replace the Copilot Studio test chat. The test chat is excellent while we are actively developing. We can ask questions, experiment, tweak instructions, and immediately see how the agent reacts. Evals give us repeatability.

Final Thoughts

Evals are one of the most important concepts to understand as we move from experimenting with AI agents to building production agents. Generative AI isn’t completely deterministic. Different users can ask the same question in different ways, and an agent can give different but equally valid responses. That means we need more than traditional exact-text testing. We need to test things such as:

• meaning,
• quality,
• grounding,
• abstention,
• tool usage,
• safety,
• conversation context,
• and business-specific behavior.

Copilot Studio’s evaluation capabilities give us a practical framework for doing this. We don’t need to start with hundreds of tests. We can start with ten important scenarios. As the agent grows, our evaluation suite grows with it.

Keep learning. Keep building.

Explore AI, agents & Microsoft technology.

I share practical ideas, tutorials, and videos about AI, AI agents, Microsoft technologies, and the Power Platform.

Subscribe on YouTube →

Carl de Souza Enterprise Architect at Microsoft · AI Technology Expert