Build vs buy AI testing: in-house stack or vendor platform.

What an in-house agentic test stack actually involves

A realistic in-house stack in 2026 combines: an LLM-orchestration layer (LangGraph for stateful workflows, or simpler bespoke orchestration for narrower scope); LLM API access (Anthropic, OpenAI, or a self-hosted model for sensitive workloads); browser-automation primitives (Playwright directly, or Stagehand on top of Playwright for higher-level verbs, or Browser Use for Python-first workflows); test infrastructure (Playwright runners in CI, a result-storage layer, a dashboard); and the integration plumbing that wires these together.

None of these pieces is hard individually. The integration work is real. A capable engineer can prototype the stack in a week; making it production-grade (handling flakes, scaling parallelism, managing browser sessions, integrating with the team's CI) takes months. The honest framing is that in-house is buildable but not free.

What vendor platforms deliver that in-house does not

Mature self-healing locator engines. Vendor platforms have years of training data from thousands of customers. The self-healing reliability is meaningfully better than what an in-house team can achieve in the first year of building.

Operational support. When a test fails at 2am during a production deploy, the vendor platform has a support team. The in-house stack has whoever happens to be on call.

Device farms. Mobile testing especially benefits from vendor-managed device farms. Building an in-house device farm is rarely cost-positive.

Vendor-side scale. Test execution at scale (thousands of parallel browser sessions) is operationally complex. Vendors absorb this complexity; in-house teams have to solve it themselves.

Compliance evidence. SOC 2 attestation, HIPAA readiness, PCI-DSS scope documentation. Vendors maintain these as part of being a vendor; in-house stacks need their own compliance work.

Cost math at 5 engineers

A 5-engineer company building an in-house agentic test stack to compete with a vendor platform is a bad use of time. The opportunity cost (engineers not shipping product) is the dominant line. Even if the in-house stack is technically capable, the company is small enough that vendor pricing is small in absolute terms and the build cost is large relative to the team.

What works at 5 engineers: pay for a vendor platform if the testing problem is acute; lean on free-tier or low-cost developer-side AI tools (Cursor, Claude Code, GitHub Copilot) for unit-test generation; skip the deeper investment until the company is larger.

Cost math at 50 engineers

At 50 engineers, the calculation gets more interesting. A vendor platform contract can be five to six figures annually; building an in-house stack requires 1 to 2 engineer-FTEs of sustained investment, which is comparable cost in raw terms but with different value profiles.

The vendor wins on operational maturity, support, and time-to-value. The in-house stack wins on customisation, control over data flow, and avoiding the per-vendor relationship overhead. Most 50-engineer companies are better served by a vendor platform unless they have a specific reason to build (regulated data handling that vendors cannot satisfy, an unusual application architecture, or a strategic bet on testing as a competitive moat).

The hybrid pattern is common at this scale: vendor platform for end-to-end (Mabl, Testim, QA Wolf), in-house LLM-assisted workflows for unit-test generation (Cursor or Copilot plus team conventions). This gets the best of both at modest total cost.

Cost math at 500 engineers

At 500 engineers, vendor pricing scales meaningfully (low-to-mid six figures annually for an enterprise contract on a mature platform), and the strategic question is whether testing infrastructure is a capability worth owning. For companies where it is (FAANG-scale operations, defence contractors, healthcare-adjacent platforms with unique compliance needs), an in-house build can compete on total cost and deliver real strategic differentiation.

For companies where testing infrastructure is not strategic, vendor platforms still win on total cost when the surrounding work (support, training, vendor relationship management) is accounted for. The honest framing is that scale alone does not justify build; strategic intent does.

The pieces of an honest build decision

If a team is seriously considering build, the questions to surface first:

What specific capability is vendor platforms not delivering? If the answer is generic ("they cost too much" or "they are not flexible enough"), the build is unlikely to deliver enough to justify itself. If the answer is specific ("we need on-prem deployment with FedRAMP attestation, no vendor can provide it"), the build is more defensible.

Who will own the in-house stack in three years? Builds that succeed have a designated team of multiple engineers committed for the long run. Builds that start with one excited engineer and no succession plan reliably wither.

What is the total cost of ownership over three years? Include engineer-FTE on the build, the LLM API costs at projected scale, the infrastructure costs, the on-call burden, and the opportunity cost of those engineers not working on product. Compare against the vendor-platform total over the same period.

What is the exit path if the build does not work out? A vendor contract is exit-able. An in-house stack that the team has come to rely on is harder to dismantle. Plan for the possibility before committing.

Why hybrid is the right answer for most teams

Vendor for the operationally hard parts (end-to-end with self-healing, mobile device farms, visual regression infrastructure). In-house for the high-customisation, low-operational-burden parts (unit-test generation, code-review assistance, internal-tool testing). This split lets the team get the vendor maturity where it matters and the in-house flexibility where it pays off, without committing to either extreme.

See QA Wolf vs testRigor for the managed-vs-platform end-to-end framing, Diffblue vs Qodo Cover for the unit-test angle, and Playwright MCP vs Stagehand for the underlying agentic-browser primitives.

Frequently asked questions

Can a small team realistically build their own AI testing stack?

Realistically no, in the sense of replacing a vendor platform. A small team can integrate LLM-based test generation into existing Playwright or Cypress workflows using off-the-shelf libraries (LangGraph, LangChain, plus the Anthropic or OpenAI SDKs) and get meaningful productivity gains. Building a full platform comparable to Mabl or testRigor takes years of focused investment that small teams cannot fund.

What changed in 2024 to 2026 that makes in-house stacks viable for some teams?

Three things: open-source LLM-orchestration libraries matured (LangGraph, AutoGen, CrewAI); foundation-model APIs became cheaper and more capable; browser-automation primitives became LLM-friendly (Playwright MCP, Stagehand, Browser Use). The combined effect is that a team with strong engineering can assemble a useful agentic test stack from open-source pieces, which was not realistic two years ago.

Where do vendor platforms beat in-house?

Mature flake-management, self-healing locator engines with years of training data, vendor-side device farms, vendor-side support relationships, and the operational maturity that comes with thousands of customers. None of these are quick to replicate. Vendor platforms still beat in-house on these vectors and will for the foreseeable future.

What is the right team size threshold for in-house?

There is no clean threshold. The honest framing is that in-house pays off when test infrastructure is a strategic capability for the company (Google, Microsoft, Meta-scale operations) or when there is a specific workflow the vendor platforms cannot address. For a typical 50-engineer company, vendor platforms are almost always cheaper and faster than building.

Hybrid: vendor for end-to-end, in-house for unit-test generation?

Common pattern in 2026. Teams use a vendor platform for end-to-end (where the operational complexity is real) and an in-house LLM-assisted workflow for unit-test generation (where the operational complexity is low and the customisation value is high). The hybrid pattern is the right pragmatic answer for many teams.