Innovation lab
Bench notesQuestions we are still working on.
Open questions we work on, not a product roadmap. Each one is a failure mode we have hit, and what we now do about it on every build.
01The bench
Six failure modes, and the fix.
- retrievalquery[1][2]AI · Natural language
Why do good models still give bad answers?
Most disappointing AI features fail at retrieval rather than at the model, so that is where the work goes.
- Tuned chunking, embeddings and reranking over your content
- Citations back to source documents on every answer
- Guardrails and a graceful hand-off to a human
- harnessbaselinecandidateheld-outAI · Machine learning
How do you know it actually got better?
We start by establishing whether a model is the right answer at all, then train or fine-tune on your data with a held-out evaluation set so improvements are measured rather than felt.
- A baseline and evaluation harness before any training
- Versioned models with reproducible training runs
- Drift monitoring and a documented retraining trigger
- forecastAI · Predictive analytics
What does a forecast actually promise?
Forecasting built on your historical data with the uncertainty made visible, so decisions are made on a range rather than a single confident-looking number.
- Backtested accuracy reported against real historical periods
- Confidence intervals surfaced in the interface, not hidden
- Scheduled retraining as new data arrives
- tokenscolor.brandDesign · Systems
Why does the build drift from the design?
A token-driven system — colour, type, spacing, radius, elevation — with a component library built on top, so a change to a token updates everywhere rather than in nineteen places.
- Design tokens shared between Figma and the codebase
- Documented component library with usage guidance
- Contribution guidelines so the system survives us leaving
- releasetestbuildstagingprodrolled backBackend · CI/CD
What happens when a release is wrong?
Every merge runs the test suite, builds an artefact and deploys to staging automatically; production is one reviewed click away with a rollback that takes seconds.
- Automated test, build and staging deploy on every merge
- One-click production release with instant rollback
- Environment parity so staging genuinely predicts production
- isolationtenant atenant bSaaS · Multi-tenancy
What stops one customer reading another?
Tenant isolation enforced at the data layer rather than trusted to application code, so one customer can never read another’s rows even if a query is written wrong.
- Row-level isolation enforced in the database itself
- Per-tenant usage accounting for billing and limits
- Migration tooling with a tested rollback path
02The rule of the bench
Research is only useful if we say what it found.
- Written change requests
No surprises in writing
Scope changes adjust timeline and cost in writing before any work starts. We do not silently absorb scope creep, and we do not silently bill for it either.
- Honest technical reads
We will tell you not to
When we audit an inherited codebase and the honest answer is "this needs a rewrite", we say so — even when the smaller engagement would have been easier to sell.
- Tests and docs included
Built to be handed over
Unit and integration coverage on critical paths, plus architecture and setup documentation, delivered as part of the work rather than quoted as an extra.
Hit one of these yourself?
Tell us where it broke. An engineer replies.