Uncovering the Real Cost of In-House End-to-End Testing

What it takes

Tests for 80% coverage

250

Time to create 250 tests

63 Business days

Test runs

5,000 / mo

(250 tests running 1x per day)

100 test failures

100 test runs

Test failures

375 / mo

Test maintenance

563 hrs / mo

QA team to hire

4 QAEs, 1 SDETs

Team cost

~$480,000 / year

4 QAE(s)
1 SDET(s)
Running your tests in parallel

10 tests in parallel (25 batches)

Time spent

500hrs / year

Run cost

~$15,600/ year

Total cost of end-to-end testing in-house
~$495,600 / year

Step 1

Determine how many tests you need

10

Team size is a good indicator of your app’s testing footprint.

0 yrs

The age of an application affects the number of tests it needs due to factors such as codebase stability, accumulated complexity, and tech debt.

High
80 %

Aiming for 80% E2E test coverage ensures critical paths are well-tested, significantly reducing the risk of undetected bugs.

Step 2

Test creation & management

2 hrs

2 hours per test case on average. This includes outlining, coding, creating test data, and verifying it works.

1.5 hrs/test

~1.5 hours on average. This includes re-running the test, reproducing the failure, debugging, PR reviews, and re-deploying the test code.

Edit fail rate
90000/year

As of January 2022, the base salary for titles like Senior QA Engineer or Senior Test Automation Engineer in the US was $90,000 before benefits and other comp.

By default, we recommend the number of Software Development Engineers in Test (SDETs) based on industry standards.

An SDET enhances your QAE team by developing automated testing tools and frameworks, increasing testing efficiency, coverage, and reliability.

120000/year

As of January 2022, the base salary for titles like Senior Software Engineer in Test and SDET in the US was $120,000 before benefits and other comp.

Step 3

Runs setup

1

You should run the suite at least daily, or more if you deploy continuously.

Tests run simultaneously across multiple resources. This method can reduce total execution time significantly but might incur higher costs due to the need for more resources.

130

Major vendors charge ~$130/test/mo for up to 25 parallel runs and ~$100/test/mo for 26 or more.

10

Max: 625

Example: If you have 200 tests, you might run 4 batches of 50.