Aigarth runs as 12 services. Each one is its own package. The way we test them is the most underrated part of the platform. This post is about how that test architecture works, why the no-dev-server rule matters, and the one piece of infrastructure (a dual-mode pg-mem harness) that makes the whole thing fast.
The number
As of this week, the platform has more than 600 vitest cases across 12 services. The breakdown:
- services/ann: 224 cases. The biggest test surface. Includes the Organism routes, the lineage and memory primitives, the retrain cron, the integration test harness.
- services/training: 48 cases. The training orchestrator, the recipe registry, the retrain guardrail.
- services/work: 38 cases. The new Work Runtime. The Work Item envelope, the scheduler scoring, the verifier V01-V03 adversarial cases, the ULID generator.
- services/marketplace: 16 new cases for the Organism listings. Plus the older tissue listing tests.
- services/tissue, services/identity, services/billing, services/compute, services/dataset, services/gateway, services/economy, services/qubic, packages/trinary, packages/aigarthpool, packages/observability: the rest. Smaller per-service, but they add up.
Total: well over 600 cases. The number will be a little higher by the time you read this, because the Work Runtime is new and we are still filling in coverage.
The no-dev-server rule
The most important rule in the test architecture: tests do not run against a live platform stack. No pnpm dev. No pnpm stack:dev. No Postgres running. No NATS running. The tests run against the build, not the platform.
The reason is simple. A test that needs a running stack is a test that is flaky. A flaky test is a test that someone eventually disables. A disabled test is a test that does not exist. So we banned the dev server from the test path entirely.
The cost of this rule: every test has to bring its own database (or its own mock of one). The benefit: the same test that passes in CI passes on a developer laptop, with zero setup. The tests are not infrastructure-dependent. They are code.
The three test patterns we use
Three patterns, picked by what the code is doing.
1. Pure function tests (no DB, no HTTP)
For logic that does not touch the database, the test is just a function in, function out. The Work Runtime's scheduler scoring and the verifier decision logic are both pure functions. Their tests run in tens of milliseconds. The whole Work Runtime test suite runs in under a second.
This is the cheapest pattern. When the test is pure, the test is also the documentation. You can read the test and understand the contract without running anything.
2. Unit tests with a mocked DB (in-memory)
For logic that touches the database, the test uses a mock Drizzle client. The Organism CRUD routes in services/ann use this pattern. The mock returns canned data; the test asserts that the right query was called with the right arguments. These tests run in low hundreds of milliseconds per case. The 37 HTTP-level cases in services/ann run in about 30 seconds total.
The mock is a hand-written stub, not a generated one. It is small (a few hundred lines) and the tests reference the same fixtures the production code uses. When the schema changes, the mock changes too. The cost of keeping the mock in sync is real, but smaller than the cost of running a real database in CI.
3. Integration tests with a real schema (pg-mem)
For the few tests that genuinely need a real database (the recursive CTE in the lineage, the FK on parent_id, the slug UNIQUE constraint, the CHECK on memory kind), the test uses a pg-mem-backed Postgres emulator. The harness is in services/ann/src/tests/integration/setup.ts. It runs every migration in services/ann/drizzle against a fresh in-memory database per test file, and the test runs against the real schema.
pg-mem is not perfect. It does not implement every Postgres feature (no GIN trigram indexes, no collations). For the few features it does not support, we either fall back to JS-level checks in the test, or we skip the test and document the gap. We do not silently skip. The skip is always visible in the test output.
The integration suite is opt-in via pnpm --filter @aigarth/ann test:integration. It does not run on every save. It runs on every push. The full integration suite (11 cases) runs in about 10 seconds. The pg-mem setup is fast enough that we can afford it on every CI run.
The dual-mode harness
The harness in services/ann/src/tests/integration/setup.ts has two modes. pg-mem is the default. It runs in any environment, no setup. postgres is opt-in, set via INTEGRATION_DB_MODE=postgres. It connects to a real Postgres at the existing docker-compose URL, creates a fresh database per test file, and runs the same migrations.
Why two modes. The pg-mem mode is the fast path. It runs in CI on every push. The postgres mode is the confidence check. We run it before every release. The two modes exercise the same test code, so the only thing that changes between them is the database underneath. If a test passes in pg-mem and fails in postgres, we know pg-mem is missing a feature we need. If a test passes in both, we know the schema is right.
The harness also handles one annoying thing: the self-referential FK on the organisms table (parent_id and root_id both reference organisms.id). pg-mem's eager FK check fires before the row is added, which breaks the founder-insert case. The harness disables the eager check on the self-referencing tables and the test re-implements the check in JS. Not pretty, but it works. And the workaround is documented in the code.
The 4-minute CI loop
The CI runs in parallel across the services, courtesy of Turborepo. The slowest single test run is services/ann at about 44 seconds. The other services finish in 10-30 seconds. The whole pipeline runs in about 4 minutes wall clock, end to end.
We could go faster. We do not need to. A 4-minute loop is fast enough that a developer will not context-switch while waiting for CI. It is slow enough that we cannot afford to run a real database on every push. The dual-mode harness is the compromise.
What we would do differently
Two things.
- The pg-mem + Postgres dual-mode harness is in services/ann. It should be a shared package so every service can use it. The Work Runtime would benefit from a real-DB integration test for the 5-table schema. Today it has only the pure-function tests. We will likely extract the harness in the next refactor.
- The CI cache is not warm. A cold CI run takes 4 minutes; a warm run (after the first push) takes about 90 seconds. The first push of the day still feels slow. We can fix this with a better Turbo remote cache, but the engineering effort is not worth the win for a team our size.
What is next in this series
Post 04 will cover the Falsification Audit: how we decide what is shipped and what is thrown away. The Audit is the most underrated document in the v0.2 evolution PEP. It is also the one we are most often asked about.