Introducing Simulations in Serval: Test your agent like you test your code

IT teams are trusting AI agents with more of their work, from answering questions to granting access and onboarding employees. But every edit to an agent's skills can change how it handles requests, and teams need a way to know that a change works as expected without breaking anything that used to work.

Software teams solve this with tests that run after every change, but IT teams haven't had an equivalent for AI agents. We built Simulations so IT teams can test Serval's Help Desk Agent, the AI agent that resolves employee requests end to end, reliably and at scale.

How do Serval Simulations work?

Take a laptop request form that changes based on who's asking. A workflow looks up the requester's employment type in the HR system, and contractors answer two extra questions, their sponsor and their contract end date. Contractors are employed by an outside agency, so their requests should go to their sponsor for approval instead of a manager. Before publishing the automation, IT wants to make sure contractor requests reach the right approver. Testing by hand would mean filing real hardware requests, so the team runs Simulations on tickets that employees never see.

1. 1.The team writes a scenario, or has Catalyst draft one from a past ticket or a spreadsheet of test cases. In one scenario, a contractor asks for a laptop for client work. The team then sets the expected result separately, which is that the request includes the contractor's sponsor and contract end date, and goes to the sponsor for approval.
2. 2.Another AI model plays the contractor as the simulated requester.
3. 3.The Help Desk Agent works the ticket. Nothing reaches your real systems, so no approval or order goes out. A model called the mocker replies the way your workflow would, like "Order submitted for approval."
4. 4.A third model, the judge, compares what the agent said and submitted with the expected result. The run fails if the request goes out without a sponsor or goes to a manager for approval.

What happens when a Simulation fails?

One scenario covers one situation, so from past tickets, Catalyst built a suite for laptop requests that included scenarios for two labels: a contractor listed in the HR system as "Contractor," and an older contractor still listed as "Contingent worker." The expected result is the same for both. The suite results show the pass rate, the scenarios that failed, and how the pass rate changes over time.

The "Contractor" scenario passes. The "Contingent worker" scenario fails: the form treats that contractor as a regular employee, never asks for a sponsor, and sends the request to a manager instead.

From the failed run, the team selects Investigate with Catalyst. Catalyst reads the judge's reasoning and the conversation, and finds the cause: the workflow was built to recognize only "Contractor," HR's new label, but older contractor records still say "Contingent worker." After the team updates the workflow to recognize both labels, older contractors can request laptops correctly, and the suite passes again.

Can I test changes before they go live?

Yes. When IT edits the skills or workflows the agent uses for laptop requests, they can run the laptop request suite against the unpublished drafts, while employees keep getting the published version. Teams can also require a suite to pass before a change is published.

Why can you trust the results?

The Help Desk Agent works the same way in a Simulations run as it does in production. When you test a draft, the agent has to find it just as it would any published skill. If the agent can't find your new skill in a Simulations run, it wouldn't find it in production either, so you know to improve the skill before you publish it.

The judge checks whether the agent reached the right outcome, and it doesn't require one exact conversation. The agent might ask for the sponsor before the device or after it, and the judge accepts either order.

Access is never granted during a Simulations run, and workflows only run against your real systems if you set them to live and confirm it.

We also use Simulations ourselves. Before we ship a new version of the Help Desk Agent, we run it against our own internal Simulations to check that it still reaches the right outcome. We first built Simulations for an enterprise customer that needed to test hundreds of historical tickets before rolling the agent out.

Get started

Simulations are available in Serval today. Read our docs to create your first scenario, or book a demo to see simulations in action.

Frequently asked questions

How do you test an AI help desk agent before it reaches production?

In Serval, you create scenarios that describe an employee's request and the result you expect. The simulated requester plays the employee, and the Help Desk Agent handles the request on a ticket employees never see. A judge then decides whether the agent reached the expected result.

Do Simulations change anything in my real systems?

Not unless you choose to. Access requests are always simulated, so no access is granted and no approver is notified. Workflows are simulated by default, and one only runs against your real systems if you set it to live and confirm it.

Can I turn past support tickets into test cases?

Yes. You can create a scenario directly from a ticket, or have Catalyst draft scenarios from a spreadsheet of past tickets. You can then group scenarios into suites and rerun them after every change.