CaseRun vs Xray, Zephyr and TestRail — measured
No adjectives. Bulk-operation timings we measured, prices from the public Marketplace API, and review scores anyone can check. Where a competitor is cheaper or bigger, this page says so.
Why the big apps feel slow
Xray, Zephyr and most established apps store every test case as a Jira issue. That has real benefits (JQL, issue links) and one structural cost: bulk work becomes thousands of REST round-trips. Closing a test run of 500 cases means 500 issue updates, one network call at a time. That is not a bug the vendors can patch out — it is where the data lives.
CaseRun stores tests in a relational database that Atlassian provisions per installation (Forge SQL). A 500-case status change is one SQL statement. In our measurements: 87ms for 500 updates (round-trips take on the order of 17 seconds), 5,000 cases added in 220ms, and a case list that stays at <1s with 5,000+ tests in it. Nothing leaves your Jira site, and there is no vendor server to trust or wait on.
Annual price, 100-user Jira site
Marketplace app pricing follows your Jira user tier. Public prices, September 2026:
| App | Annual (100 users) | Public rating | Cloud installs |
|---|---|---|---|
| Xray | $7,280 | 4.3★ (552 reviews) | 25,454 |
| Zephyr Scale | $6,810 | 4.1★ (492) | 15,301 |
| Zephyr Essential | $5,990 | 3.9★ (900) | 10,211 |
| QMetry | $4,180 | 4.7★ (200) | 2,180 |
| CaseRun | $2,400 | new listing | new listing |
| AIO Tests | $1,980 | 4.9★ (114) | 3,451 |
Honest notes: AIO Tests is cheaper than CaseRun and very well reviewed — if price is the only criterion, look at it too. Xray and Zephyr are the category leaders with years of features CaseRun does not try to match. CaseRun's case is narrow: the day-to-day work — recording results in bulk, importing, listing — runs in milliseconds, and it costs a third of the leaders. Sites with up to 10 users pay nothing (platform rule).
What users of the incumbents say
We read 2,946 public Marketplace reviews across this category before building CaseRun. The most repeated complaint theme was not missing features — it was waiting: minutes to add a case, bulk updates that crawl, pages that time out. Even the highest-rated app's few negative reviews mention slow bulk updates. The ratings in the table above are public; the pattern is easy to verify.
What CaseRun does — and does not do
| In the box | Not in the box |
|---|---|
| Test cases with steps, folders, priorities, labels; test cycles; result recording (single and bulk); reports; CSV import from Xray, Zephyr Scale/Squad, TestRail, qTest; CSV export in the same shape; CI result ingest (JUnit XML from any runner — setup for 8 runners); Jira issue links. | BDD/Gherkin syntax; running the tests themselves (CaseRun ingests results, it is not a test runner); Jira Data Center or Server (cloud only). |
Switching
Export CSV from your current tool and drop it in — column mapping is automatic, including Xray's one-row-per-step format. The way out is symmetrical: CaseRun exports the same CSV shape it imports, so you are not locked in.
CaseRun is built and documented on this site; the Atlassian Marketplace listing is linked from the home page. Questions: support@caserun.dev.