If you build on AppExchange, you don’t ship to one Salesforce org. You ship into hundreds of orgs you will never see. Each has its own data, profiles, flows and other packages, and Salesforce upgrades all of them three times a year.
So the real question for an ISV is simple. How do you know your managed package still works in an org you don’t control? The short answer: keep a small golden smoke suite that runs on every build, a regression suite per feature, and one run on Salesforce’s preview sandboxes before each seasonal release. Then make every test check the data, not just the screen.
I’ve spent the last 18 months doing this with Salesforce ISVs. Apsona is one of them, and they let us share real numbers. The rest stay anonymous. Here’s what keeps breaking, what you can fix yourself this week, and where TestZeus does the heavy lifting.
The short version
ISV testing is harder than customer testing. Your code runs in someone else’s org, on someone else’s calendar.
The most common setup we see is still a spreadsheet and a busy QA team. One ISV ran about 400 manual cases across 11 products every month. Another had roughly 2,600 cases and was 90 to 95% manual.
Script tools break when the UI changes. One ISV dropped a scripted tool because every UI change meant re-scripting.
Green tests that only check the screen are the riskiest kind. Check the data.
Apsona went from a Google Sheet in January to a 94.4% smoke pass rate in October, and caught a real product bug along the way.
Why ISV testing is a different sport
A customer tests one org. They know its data, its users and its quirks. An ISV tests a package that has to survive every customer’s quirks at once.
Four clocks run against you at the same time. Salesforce ships three major releases a year. You ship your own sprints. Your customers keep changing their own orgs. And when you push an upgrade, the subscriber doesn’t have to do anything. Per Salesforce’s own docs, the only sign a customer gets is a higher version number on the package detail page, and the developer who started the push has to resolve any upgrade that fails.
That last part matters. With a push upgrade, your customer becomes your test environment if you skip the testing. Nobody wants that job.
Problem 1: The regression suite lives in a spreadsheet
When we started with Apsona in January 2026, their regression suite was a master Google Sheet. About 400 test cases across 11 products. Document Mail Merge alone had 114. Every monthly sprint release, the QA team ran sanity on all 11 products over two days: critical features on day one, the rest on day two.
Another ISV, a survey and forms product built 100% natively on Salesforce, had it bigger. About 2,600 test cases, each 10 to 50 lines long. A QA team of 9 to 10. Work that was 90 to 95% manual. Automation had been in their QA owner’s annual goals for two years running.
A spreadsheet works while the product is small. Then every release adds rows, and nobody adds days.
What you can do without any tool:
Rank features by revenue and by blast radius. At Apsona, Document Mail Merge and Apsona for Salesforce came first, because that’s where the money and the users are.
Split the sheet into two boxes: regression (what already works must keep working) and new features. They need different rhythms.
Stop adding new cases to the sheet. Write new feature tests straight into your automation tool, or into Jira as Gherkin, so you never convert them twice.
Problem 2: Scripts cost more to maintain than to write
Script-based tools are fine on day one. The bill arrives on day ninety.
The survey ISV learned this the hard way. They trialed a script-based Salesforce testing tool and dropped it. Their reasons: the license was expensive, and every UI element change meant re-scripting. For a product that changes its UI every release, that’s a treadmill.
Lightning makes it worse. Generated ids change, and components move from Aura to LWC. A test tied to the page structure breaks even when the feature works perfectly.
What you can do without any tool:
Test intent, not layout. A test should say “create a survey with 10 questions and submit it”, not “click the third button in the second div”.
If you write locators, use roles, labels and test ids. Our free locator healer scores locators for stability if you want a quick check.
Put a summary on page one of every report: what ran, what failed, why. If only one person can read your reports, that person is your bottleneck.
Problem 3: Green tests that only check the screen
A test can pass while the customer still gets the wrong answer.
Apsona’s head of development put it simply in one of our reviews: customers only see the output. Apsona’s customers generate donation receipts and reports from Salesforce data. If the numbers inside are wrong and someone acts on them, a green UI test means nothing. So the team kept pushing their tests past the screen and into the data.
The survey ISV built their whole suite around this idea. About 70 to 80% of their tests follow one pattern: build a survey, answer it as a respondent, then go to the Survey Result and Question Result records in Salesforce and confirm every answer landed correctly. That’s a test that checks data, not pixels.
What you can do without any tool:
End every important test with a data check. Query the record with SOQL, or open the generated file and read the values.
Add “boring” input cases to your smoke suite: spaces, long names, special characters, pasted text.
Write component-level unit tests for custom LWC inputs. They catch input bugs faster than any UI test.
Problem 4: Same test, different org, different result
Every ISV runs at least three environments. At Apsona it’s QA, stage and production. A test that passes in one and fails in the next usually isn’t a product bug. It’s hardcoded data, an environment URL baked into a test, or a report left behind by the last run.
Apsona fixed this with test data. In April they started moving from static, hardcoded values to dynamic test data. Apsona’s QA team called it a really nice feature: no more hardcoding values in the test, just a variable, and fresh data every month instead of the same data every run.
What you can do without any tool:
Keep environment values (URLs, users) at the environment level and test data in variables. Never inside the test steps.
Give every test a setup and a teardown. If a test creates a report, it deletes it.
Keep a dedicated test org that mirrors production. At Apsona that’s the partner org, which is effectively production minus one.
Problem 5: Login, email and OTP flows break first
A life sciences ISV we worked with in late 2025 had about 1,500 test scripts and wanted an 80 to 90% pass rate. Their biggest blocker wasn’t their own app. Email and OTP authentication scenarios caused about 90% of their test failures. Custom components were the other big headache.
Auth is also moving under your feet. Since Spring ‘26, Salesforce disables the creation of new connected apps by default and points new integrations to External Client Apps. If your test setup or install guide still says “create a connected app”, it’s already out of date.
What you can do without any tool:
Use a dedicated test inbox per environment and read OTPs and emails programmatically. Apsona does this with a dedicated test account, so email features no longer need a manual check.
Run tests as a named integration user with a stable profile, not a person’s login.
Move test integrations to External Client Apps now, in a sandbox first.
Problem 6: Release month eats the team
ISV testing load isn’t flat. It spikes.
The survey ISV releases three to four times a year. They told us they might run very little for two months, then 2,000 to 3,000 scenarios in a release month. Apsona runs monthly sprints and packs regression and bulk testing into the last week of one month and the first week of the next.
If your regression suite only runs in release month, release month becomes the first time you learn what broke three weeks ago.
What you can do without any tool:
Run the smoke suite on every build, so release month is a confirmation, not a discovery.
Run your suite on Salesforce’s preview sandboxes before each seasonal release. Our Salesforce release calendar shows the dates for your instance.
Push upgrades to your own orgs first, then stagger customers in batches. Salesforce recommends both.
Problem 7: Agents are next, and they don’t package cleanly
Apsona is building Agentforce agents for reporting: a Q&A agent, a report builder and a report explainer. Along the way their team ran into something many ISVs have reported on the Trailblazer Community. New agents now get built with Agent Script, and the agent bundle itself can’t go into a managed package yet. Supporting pieces like Apex classes, flows and actions can ship in the package, but the agent is set up in each customer org after install.
That changes testing. You are no longer testing one agent. You are testing every customer’s copy of it, configured after install, with their data.
What you can do without any tool:
Test the actions and Apex behind the agent as normal code, with normal unit tests.
Write down 20 to 30 real user questions per topic, including rude, vague and off-topic ones, and run them after every agent change.
Check what the agent did, not only what it said. A polite answer with the wrong record updated is a fail.
Our free Agentforce coverage planner builds that test list for you.
A practical playbook for ISV regression testing
Here’s the layered setup we now recommend to every ISV. None of it needs TestZeus. Each layer catches a different kind of break.
Apex unit tests. You already need 75% code coverage to upload a managed package. Treat that as the floor, not the goal.
Install and upgrade tests. Install the new version in a fresh scratch org and upgrade an old one. Most packaging bugs show up here.
Golden smoke suite. Tag 10 to 15 cases that must pass every single day: inline edit, data validation, file output, tricky inputs. Run them on every build.
Feature regression suites. One suite per feature, in a set order, before every release. Start with one use case per feature, then widen.
Preview and post-push checks. Run the smoke and regression suites on Salesforce’s preview sandboxes before each seasonal release. Run smoke again in a production-like org after you push.
Where TestZeus fits
You can build all of the above with scripts and grit. Most ISVs we meet have tried. Here’s where we take over the parts that eat the most time.
ISV problem | What TestZeus does |
|---|---|
Regression in a spreadsheet | Write tests in plain English or Gherkin. Import existing scripts, record in the browser, or generate from Jira, docs and your code repo. |
Script maintenance | The agent reads the page the way a person does, so an Aura to LWC move usually doesn’t mean rewriting tests. Test Blocks turn a recording into deterministic steps you can replay and edit one step at a time. |
Green tests that check the screen | Validate with SOQL queries, Apex, API calls, emails and generated files, not only the UI. |
Environment drift | Swap the environment at suite run time. Keep URLs at environment level and use dynamic test data. |
Release-month spikes | Run suites in parallel or in order, on a schedule, in cloud browsers. Tag a feature and run all its cases in one click. No local setup. |
Reports nobody can read | An AI summary on top, a step-level timeline and video below, and root cause analysis with suggested fixes when you connect your code. |
Agents | Agent Harness tests Agentforce text and voice agents with multi-turn adversarial runs, run-as profiles, and SOQL and tool-call checks. |
Security matters when your test tool touches customer-like data. TestZeus is SOC 2 Type II and ISO 27001 certified, can mask data before it reaches an LLM, and can run on-prem.
Apsona: from a spreadsheet to feature suites
Apsona sells data management, reporting and document mail merge on AppExchange. About 92% of their customers are nonprofits, so every donation receipt has to be right.
In January, Apsona’s QA team set quarterly goals for automating with TestZeus. By October, they had met their Q1 and Q2 goals and most of Q3.
Q1: one use case per feature. The team started small and proved TestZeus worked on every product before going wide.
Q2: every feature covered. By May, Document Mail Merge alone had 18 automated use cases. Email checks moved to a dedicated test account, so the mail merge email feature no longer needed manual testing. A single tag now selects all 18 cases for a run, instead of picking them one by one.
Q3: Fast Mode and a use-case library. Most cases now run in Fast Mode, and the team has a library of use cases organized by feature.
August: working in parallel. Apsona’s QA team told us the engagement had been very good. They started from scratch, and could now automate in one tab while testing the next sprint’s tickets in another.
October: the best cycle yet. 120 cases in scope and 91 executed. Smoke passed 94.4% across 54 runs. Sanity passed 79.3% across 87 runs. Fast Mode covered 89% of executed runs. By the team’s own review, the cycle was faster and more reliable than September’s.
The suite also did its real job: it caught a confirmed product bug. In the Tabular View, Contact type sorted ascending instead of descending. The suite found it, and the team flagged it in the October review.
Apsona’s team says they are now more confident running full suites. The Q4 goal is an ordered suite for every feature by year end, rolled out from November 2 through December 29. From next year, the plan is to run each feature’s suite in one go instead of clicking through cases one at a time.
The takeaway
ISV testing isn’t about running more tests. It’s about running the right 15 every day, the right 120 every release, and checking the data every time.
Start with the playbook above. Then bring your ten most important test cases to a TestZeus demo and see them run on your own sandbox. Or start with our free tools.
FAQ
How should a Salesforce ISV test a managed package before a release?
Run Apex unit tests, then install and upgrade tests in a scratch org, then a golden smoke suite and your feature regression suites. Before each Salesforce seasonal release, run the same suites on a preview sandbox.
Are Apex tests enough for ISV regression testing?
No. Salesforce requires 75% Apex code coverage to upload a managed package, but Apex tests don’t check the UI, LWC behavior, generated files or what happens in a real customer org. You need end-to-end tests on top.
What should go in a smoke suite?
10 to 15 cases that cover your money features and your most fragile inputs. At Apsona that means inline edit, data validation and file output.
How do push upgrades change testing?
With a push upgrade, customers don’t do anything, so they won’t catch a break before it reaches them. Test the upgrade path in your own orgs first, then push in batches.
Can an AI testing agent test a managed package?
Yes, if it checks data and not just screens. Look for SOQL and file validation, environment switching, dynamic test data and readable reports. Then run it on your own top 10 cases before you decide.
// Start testing //
