Testing an AI Agent That Controls CAD Software - Live in 72 Hours

Cosmon's Nexus platform lets engineers use AI prompts to perform actions inside CAD and CAE applications. With customer POCs approaching, Cosmon needed test coverage in place quickly. ThinkSys joined the engagement and began testing within 72 hours.

Testing Nexus required us to verify two things: whether the AI interpreted the prompt correctly and whether the requested engineering action was actually completed in the target application.

We built a compatibility matrix covering 6 CAD/CAE applications, approximately 12 application versions, and 4 OS configurations. The test setup combines WinAppDriver and Selenium for automation with Promptfoo for evaluating AI behaviour across prompts and expected outcomes.

Testing has identified around 110 defects so far, including approximately 30 related specifically to AI behaviour rather than conventional functional issues.

Building AI that takes actions in other software? Let's talk about how you'd test it.

Book a call

At a Glance

Meet Cosmon

Cosmon builds Nexus, a platform that lets engineers use natural-language instructions to perform tasks across CAD, CAE, and related engineering applications. Instead of carrying out each step manually in the underlying software, an engineer describes the required change and Nexus executes the corresponding actions.

Our QA work covers two parts of the platform:

• Nexus Connector V1:  Testing browser and connector workflows, including integrations with SolidWorks, Ansys, Abaqus, and COMSOL.
• Nexus Desktop V2:  Functional and integration testing, alongside the development of an automation framework to increase regression coverage as the product evolves.

Testing a system like Nexus requires more than checking whether an AI-generated response sounds correct. The output is an action inside engineering software. If a prompt is interpreted incorrectly, the result could be the wrong geometry, parameter, material, or other change to the engineer's model.

The Challenge

1. A fixed POC deadline

Cosmon had customer proofs of concept approaching and needed meaningful QA coverage before those sessions took place. There wasn't time to complete a conventional onboarding phase and then begin testing.

Environment setup, product discovery, test planning, and execution had to happen in parallel. The immediate priority was to find issues that could affect the upcoming POCs while building enough product knowledge to expand coverage quickly.

2. A large compatibility surface

Nexus needs to work across multiple CAD and CAE applications, application versions, operating systems, and configurations. Each combination introduces its own interfaces, dependencies, workflows, and potential failure points.

A feature working correctly with one version of SolidWorks, for example, doesn't establish that the same workflow will behave correctly with another supported application or environment. Compatibility therefore had to be tested systematically rather than treated as a one-time integration check.

3. Testing inside Windows desktop environments

A significant part of Nexus operates within Windows desktop applications. That introduces variables that don't normally appear in browser-only testing, including application installation and state, OS configuration, permissions, local dependencies, and interactions between Nexus and the engineering software running on the machine.

Coverage had to account for these environment-specific failures as well as the functionality of Nexus itself.

4. Verifying the engineering result, not just the integration

A successful connection to SolidWorks, Ansys, Abaqus, or COMSOL doesn't prove that a Nexus workflow worked correctly.

When Nexus receives an AI-driven instruction, the test needs to follow that instruction through to the target engineering application and verify the resulting change. A command can complete without an obvious technical error and still produce the wrong engineering result.

5. Testing non-deterministic AI behaviour

AI-driven workflows can't always be validated with the exact-match assertions used in conventional functional testing.

A prompt can produce different wording across runs while remaining valid. Conversely, a plausible response doesn't prove that the correct downstream action was taken.

Testing therefore needed to evaluate the prompt and model behaviour alongside the resulting application state, with separate criteria for determining whether the AI response and the executed action were acceptable.

6. Regression coverage that could grow with the product

Every additional application, version, configuration, and Nexus workflow increases the number of combinations that may need to be checked before a release.

Relying entirely on manual regression would make that coverage increasingly difficult to maintain. Cosmon needed automation that could take over repeatable checks while the QA team continued to focus manual testing on new functionality, integration behaviour, and AI scenarios that required deeper investigation.

Testing Live Within 72 Hours

Cosmon had customer POCs approaching, so the first priority was getting useful test coverage in place quickly. ThinkSys began testing within 72 hours of the engagement starting.

Rather than completing discovery before execution began, we ran the two in parallel. The first testing cycles focused on the workflows required for the upcoming POCs. What we learned from those runs then shaped the broader environment setup, compatibility coverage, test planning, and automation work.

That approach continued as the engagement expanded. When a new build arrives, the team divides the scope by application, version, and test area, then runs those streams in parallel. Issues are reported within the same build-testing window, giving Cosmon time to investigate and address them before the next release cycle.

Testing the Full Chain: From Prompt to Engineering Result

Testing Nexus doesn't stop at checking the model's response. The test has to follow the instruction through every system involved:

User Prompt → AI/LLM Response → Agent/Connector → CAD/CAE Application → Engineering Result

A response can be technically valid and still produce the wrong result. Nexus might identify the correct operation but apply it to the wrong object, modify the wrong parameter, or behave differently across supported CAD/CAE applications.

For that reason, we validate the complete workflow. A test sends a prompt to Nexus, follows the execution into the target application, and verifies what actually changed there.

Depending on the workflow, that can mean checking whether the correct application opened, the expected model or file was used, the requested operation completed, geometry was created or modified correctly, and the resulting CAD state matches the intended outcome.

The AI response is useful evidence during the test, particularly when diagnosing a failure, but it isn't enough to determine whether the workflow passed. The final state inside the engineering application determines whether the requested task was completed correctly.

What ThinkSys Built

1.A structured test path matrix

We created a client-approved matrix to define coverage across the variables that can affect Nexus: operating systems, CAD/CAE applications and versions, system configurations, integration scenarios, agent workflows, AI/LLM models, functional workflows, and negative and edge-case scenarios.

The current scope covers around 6 CAD/CAE applications, roughly 12 application versions, and 4 OS configurations.

The matrix also provides a repeatable basis for smoke and compatibility testing. When a supported CAD/CAE application releases a new version or an environment dependency changes, the team can identify which combinations and workflows need to be rerun instead of deciding coverage from scratch for each release.

2. Establishing expected behaviour before automating it

We validated critical scenarios manually before moving them into automation.

That was particularly important for Nexus because the expected result can depend on the connected engineering application, its version, the current application state, and the surrounding system configuration. Automating a workflow before establishing what correct behaviour looks like would make the test repeat an assumption rather than validate the product.

Manual testing continues where it provides more value, including new functionality, exploratory testing, compatibility scenarios, new integrations, AI-driven behaviour, edge cases, and usability.

3. Automating the repeatable regression layer

For Windows desktop workflows, we use WinAppDriver to automate scenarios such as application launches, UI interactions, connector and agent operations, state checks, and repeatable regression paths.

Browser coverage uses Selenium with Java and TestNG, structured around the Page Object Model with shared components to keep common interactions in one place as the suite expands.

We automate scenarios when the expected result is established, the workflow is repeated often enough to justify automation, and the result can be verified reliably. Scenarios that are still changing or depend heavily on exploratory judgment remain manual until automation provides a clear benefit.

4. Why WinAppDriver is part of the stack

WinAppDriver is no longer under active Microsoft development, so its use is a deliberate choice rather than a default.

For the current Nexus workflows, it provides the Windows desktop automation capabilities needed to interact with and validate the applications under test. We account for its maintenance status when deciding what belongs in the automation suite and how the framework should evolve.

For new desktop automation requirements, we also evaluate alternatives such as Appium's Windows driver and FlaUI against the application, required interactions, technology stack, and long-term maintenance needs before selecting an approach.

5.  AI/LLM validation with Promptfoo

Testing AI behaviour needs a different assertion model. We validate expected behaviour and outcome rather than relying on exact text matching, defining evaluation criteria, expected outputs, and acceptable variations to determine whether a response is correct.

Promptfoo evaluates prompt and response behaviour, expected outputs, model comparisons, response consistency, negative scenarios, and AI-driven workflows.

The pattern looks like this:

prompts:

  - "Create a 50mm fillet on the selected edge"

tests:

  - assert:

      - type: contains-any

        value: ["fillet", "round"]

      - type: javascript

        value: |

          // The real assertion: did the CAD application change?

          return checkCadState({ feature: 'fillet', radius: 50 });

Note: The text assertion is a sanity check. The state assertion is the test. 

Results

Testing has identified around 110 defects, including approximately 30 related to AI behaviour. These included incorrect agent actions, problems with prompt execution, and cases where AI-driven behaviour differed from the expected result.

That distinction matters because an AI workflow can fail even when the surrounding application is functioning correctly. The interface may work, the integration may remain connected, and the agent may complete an action, but the action itself can still be wrong.

AI-behaviour issues account for roughly 27% of the defects identified so far, reinforcing the need to evaluate the model's behaviour and the resulting engineering state alongside conventional functional testing.

What the framework enables: the matrix and automation suite expand as Cosmon adds engineering applications, versions, AI models, and workflows. The baseline evaluation suite runs whenever a model or prompt changes, so behavioural regressions surface by comparison rather than by someone noticing. 

What This Engagement Taught Us

Testing an AI agent means testing what it changes

Checking the model's response is only part of the test when an AI system can take actions in other software.

For Nexus, a response can look correct while the wrong operation is performed inside the CAD/CAE application. That makes the resulting application state a critical part of the assertion. We need to know what the model said, what the agent attempted, and what actually changed in the engineering application.

Non-deterministic output needs different assertions

The same prompt doesn't always need to produce identical text to be correct. What matters is whether the response stays within the expected behaviour and leads to the correct result.

For AI-driven scenarios, we define what must be true, what variation is acceptable, and what would constitute a failure. Exact matching can still be useful for deterministic parts of the workflow, but it isn't sufficient as the primary validation method for variable model output.

Model and prompt changes need a baseline

Changing a model, prompt, or related AI logic can improve one scenario while degrading another.

A baseline evaluation suite gives the team a consistent set of scenarios to rerun after those changes. Instead of judging a handful of new responses in isolation, results can be compared against previously established behaviour to identify where performance improved, remained stable, or regressed.

Automate where the expected result is stable

Repeatable compatibility checks, application-state validation, and established regression scenarios are good candidates for automation.

New integrations, unfamiliar failure modes, changing workflows, and AI behaviours that haven't yet been characterised benefit from manual investigation first. Once the team understands the expected behaviour and can verify it reliably, those scenarios can move into automation where it makes sense.

The order of the work matters when the deadline is fixed

Cosmon had customer POCs approaching, so waiting for discovery, environment setup, and test planning to finish before beginning execution would have delayed useful feedback.

We started with the workflows that mattered to those POCs and ran discovery alongside testing. That meant accepting some inefficiency early in the engagement, but it also meant Cosmon had QA coverage on priority workflows within 72 hours.

The broader compatibility matrix and automation framework were then built using what those early testing cycles had already taught us about the product.

Building an AI agent that takes action in other software? Tell us what it connects to, and we'll help you work out how to test the complete workflow.

Book a call

Frequently Asked Questions

How do you test an AI agent that performs actions in external software?

We test beyond the model response and verify what actually happened in the connected application.

For Nexus, that means sending a prompt, following the agent's execution, and checking whether the requested operation was completed correctly inside the CAD/CAE application. The response helps us understand what the model intended to do, but the resulting application state tells us whether the workflow actually worked.

How do you validate that an AI response produced the correct downstream action?

We validate the stages separately so a failure can be traced through the workflow:

Prompt → AI response → Agent/connector → External application → Expected result

The model might interpret the prompt correctly while the connector executes the wrong operation. The connector might call the correct operation but target the wrong object. Separating these stages helps identify where the behaviour diverged from what was expected.

How do you test LLM output when the same prompt gives different answers?

We define the behaviour that must remain consistent rather than requiring identical wording on every run.

That can include required information, prohibited responses, expected actions, acceptable variation, and the resulting state of the connected application. Two responses can be worded differently and still pass if both satisfy the criteria and produce the correct outcome.

Exact matching is still useful where the output itself is deterministic.

How do you regression-test when the model or prompts change?

We maintain a baseline evaluation suite containing representative prompts, expected behaviours, and acceptance criteria.

When a model, prompt, retrieval strategy, or related AI logic changes, the same scenarios can be rerun and compared with the baseline. This makes it possible to see where behaviour improved, remained stable, or regressed instead of judging a few responses manually after each change.

How do you test CAD software integrations?

Connectivity is only the starting point.

Depending on the workflow, we verify application launch, model and file operations, parameter changes, geometry creation or modification, and the resulting state inside the CAD/CAE application.

A technically successful connection can still produce an incorrect engineering result, so integration testing needs to verify the requested operation rather than simply confirming that the systems communicated.

How do you build a compatibility test matrix across operating systems, application versions, and configurations?

We identify the variables that can change application behaviour and map the supported combinations across them.

For Cosmon, the current matrix covers around 6 CAD/CAE applications, roughly 12 application versions, and 4 OS configurations, along with relevant configurations, dependencies, integrations, and workflows.

The matrix provides a defined basis for smoke and compatibility testing. When an application version or dependency changes, the team can identify the affected combinations and determine what needs to be rerun.

How quickly can a QA team ramp up on a complex product?

It depends on how much product knowledge and environment preparation are required before meaningful testing can begin.

For Cosmon, ThinkSys started testing within 72 hours because customer POCs were approaching. We did that by running product discovery and testing in parallel and focusing the first cycles on the workflows needed for those POCs.

That approach makes sense when early feedback is more valuable than completing the entire test structure upfront. For an established product where comprehensive baseline coverage is the immediate goal, a more deliberate discovery and planning phase may be appropriate.

What is Promptfoo, and how are you using it?

Promptfoo is an open-source evaluation framework for testing and comparing LLM outputs against defined scenarios and criteria.

For this engagement, it provides a repeatable way to evaluate AI behaviour across baseline prompts and identify changes when prompts or models are updated.

It complements the functional and integration testing rather than replacing it. Promptfoo can help determine whether the model behaved as expected; the rest of the test chain verifies whether that behaviour produced the correct result inside the engineering application.

What tools can be used to automate Windows desktop applications?

Windows desktop automation requires tools that can interact with native application controls rather than browser DOM elements.

For the Nexus engagement, we use WinAppDriver for repeatable Windows desktop workflows. Other options include Appium's Windows automation capabilities and FlaUI, with the appropriate choice depending on the application technology, required interactions, existing automation stack, and maintenance requirements.

Browser automation tools such as Playwright and Cypress are designed primarily for web applications and aren't direct replacements for native Windows UI automation.

Is WinAppDriver still maintained?

WinAppDriver's maintenance status needs to be considered when choosing it for new automation work.

It remains part of the Nexus automation stack because it supports the Windows desktop interactions required for the current workflows. For new requirements, we assess it alongside alternatives such as Appium's Windows automation options and FlaUI rather than assuming the same tool is appropriate for every desktop application.

The decision depends on the application under test, the interactions that need to be automated, compatibility requirements, and the expected lifetime of the automation.

How do you decide what to automate and what stays manual?

We look at how often a scenario runs, how stable its expected behaviour is, and whether the result can be verified reliably.

Repeatable regression paths, compatibility checks, and established application-state validations are strong candidates for automation.

New functionality, exploratory testing, unfamiliar integrations, usability, and AI behaviour that hasn't yet been characterised usually benefit from manual investigation first.

For an AI-driven product, that boundary isn't permanent. Once a scenario is understood and its expected behaviour can be defined reliably, it can become a candidate for automation.