A new AI tool shows up most weeks. Some of them are genuinely good. Most are a demo, a wrapper, or a thing that will be gone by next quarter. We cannot try all of them, and neither can you, so we run three quick checks first. Anything that fails a check does not get our time.
These are not deep evaluations. They are the gate before the evaluation. They take a few minutes, and they decide whether a tool is worth a real look. Further down we run them on two tools we have used and publish the result, so a small team can copy it as an adoption checklist.
Check one: can it run locally and keep your data?
The first question is where the work happens. Does the tool run on a machine we control, or does it ship our data to someone else’s server to function?
This is not purity for its own sake. A tool that runs locally keeps the data in the building, costs nothing per call, and does not stop working when a vendor has a bad day. A tool that only works by sending everything to a cloud endpoint inherits all the dependence we try to avoid, which is the whole reason we own our AI stack instead of renting it. We are not absolutist about it, but the burden of proof sits with the cloud-only tool. If it cannot run on our hardware and hold our data, it had better be doing something nothing local can. The test is concrete: pull the network cable. A tool that keeps working has passed check one, and a tool that goes dark was never really yours.
Check two: is it actively maintained and openly licensed?
The second question is whether the tool will still be here in a year, and whether we are allowed to depend on it.
Maintenance is easy to check. Look at the project. When was the last release? Are issues being answered, or is the repo a ghost town with a glossy homepage? An AI tool that has not shipped in a long while is a tool you are adopting at the moment it stops being supported.
Licensing is the other half. We read the actual license before we build anything real on top of a tool. Open weights is not the same as open source, a distinction we pull apart in why open weights is not open, and “free to try” is not the same as “free to use the way we intend to.” A clause we skip now is a wall we hit later. The license tells us, in a short document, exactly what we are allowed to do.
Check three: does it earn its place over what we already run?
The third question is the one most people skip. Even a good tool has to beat the tool it would replace.
We already run a stack that works: a writing model and a coding model on a single Apple M4 Pro with 48 GB of unified memory, served through Ollama. A new tool does not get adopted because it is new or interesting. It gets adopted because it does a real job better than what is already there, or does a job nothing in the stack can do at all. “Marginally nicer” is not a reason to add a dependency, retrain a habit, and carry one more thing that can break. The bar is not “is this good.” The bar is “is this worth displacing what already works.”
The real cost of a new tool is rarely the sticker price. It is the switching cost: the habit you retrain, the integration you wire, the failure mode you now have to learn. A tool that is ten percent better and doubles that surface area is a bad trade, and most exciting launches are exactly that trade in disguise.
Where do the three checks fit our evaluation process?
These three checks are not the whole process. They are the entrance to it. A tool that passes moves onto the path we use for everything: research first, then tested in real conditions, then in-production only once it has actually earned a place in real work. It is the same ladder behind how we evaluate local tools for the Atlas. Most tools never make it past research, and that is the point. The three checks keep the failures cheap, so the real evaluation is spent on tools that might actually last.
A worked example helps. A tool lands that promises faster local transcription. Check one passes, it runs offline. Check two is shaky: the last release was nine months ago and the issues sit unanswered. That single fact ends it before we ever install it. The few minutes spent reading the repo saved a week of building on something already abandoned.
Evaluating AI tools with the three checks: OpenCode and OpenJarvis scored
Here are the three checks run on two tools we have installed and used on our test machine: OpenCode, the coding agent we use daily, and OpenJarvis, the Stanford assistant we tested and removed. Licences and release dates were read from each project’s GitHub repository on 17 October 2026. The rest comes from our own testing in OpenCode vs OpenJarvis and OpenJarvis on a Mac.
| Check | OpenCode | OpenJarvis |
|---|---|---|
| One: runs locally, keeps data | Pass, with a caveat. It runs against a local model, but the desktop app starts Sentry reporting at launch and we found no off switch | Pass after a change. Answers locally; usage counters go to PostHog by default, and an undocumented switch stops them |
| Two: maintained, openly licensed | Pass. MIT; latest release v1.19.3 on 16 October 2026 | Licence passes, release pace to watch. Apache 2.0; latest tagged release desktop-v1.0.2 on 25 May 2026, though code was still being pushed on 17 October |
| Three: earns its place | Pass. It does the coding work every day | Fail. The assistant slot was already filled by our Hermes agent |
| Result | Adopted | Not adopted |
OpenJarvis did not fail for being bad. It failed check three, the one most teams skip. OpenCode passed all three and still carries a mark on check one, which is the honest result: a pass is not a clean bill.
To use this as an adoption checklist in a small team, give each check one named person, write the answer and the date in a shared table like the one above, and record the source for each cell. Recheck check two every quarter, because a release date is the fastest fact in the table to go stale.
When should you override the checklist?
Checklists can make you miss things. A tool that fails check one today, by being cloud-only, might still be the right call for a specific job where nothing local comes close. A young project that fails the maintenance check might be exactly the early bet worth making. The three checks are a default, not a law. They exist to save attention, not to replace judgment, and we override them on purpose when a tool clearly warrants it.
But most tools do not warrant the override. Most of them quietly fail one of the three checks, and the few minutes it takes to find out are the cheapest minutes in the whole evaluation.
The volume only grows from here. As the release pace of local AI tools climbs, a cheap, fast filter matters more, not less, because the cost of chasing every shiny launch compounds while the three checks stay the same few minutes. A team that guards its attention this way spends it on the handful of tools that actually last, and skips the churn that eats everyone else.
Take the next AI tool that catches your eye. Run it through the three checks before you run it at all. The minutes you spend are the cheapest in the whole evaluation, and they are the ones that protect every hour after.
