An evidence record you can question.

Actual product lab: 2026-10-23

We ran the local app at commit 7f14f805 through its server API, an isolated PostgreSQL database, real retrieval and paid model requests. The source was a synthetic Northstar support policy. No customer document or production service was used.

Four attempts are preserved: two stopped before model calls, the first paid attempt returned six outputs before a recording query failed, and a later attempt returned seven outputs. The later attempt changed the policy and question from cancellation to reopening a support request before its model calls. It is not a repeat of the same cancellation test.

The original cancellation question received a safe decline instead of the expected cited answer. The after-change cancellation question was never asked. The later reopening answers cited 14 days before a source import and 30 days after it. This demonstrates those saved reopening responses, not a completed cancellation test or automatic monitoring.

Sondel prepared this lab and its semantic review with AI assistance. It has not received customer validation. There is no aggregate pass-rate claim. Automatic lexical checks had mistakes, including a false failure on a negated rollover statement. The record preserves those checks separately from the corrected semantic review. Each result below is assessed against its actual question and source.

Download every attempt and output (JSON) Download the worked scorecard (CSV)

Watch the saved results

This 110-second recording replays the saved lab results. It uses synthetic data. It is not a live widget recording or a customer case study. The full corpus remains available above.

Download the video · Download the transcript

Later run: all seven outputs

This run used the revised reopening policy. The unsupported-term and missing-fact questions were deliberately covered by explicit negative statements in the source. They do not establish performance on arbitrary missing facts.

First paid run: all six outputs

The source covered cancellation. The answer gate declined the first policy answer. A later recording query used a missing database column and stopped this attempt after the updated source was imported. That failure prevented the after-change cancellation question.

What ran, and what did not

The run exercised document import, embeddings, hybrid retrieval, reranking, drafting, the default strict answer gate, API responses, stored turns, and the app cost ledger. Models were Voyage voyage-4 and rerank-2.5, with Baseten DeepSeek-V4.1-Flash for drafting.

The browser widget, visitor authentication, owner dashboard, customer identity, external actions, and action authority were bypassed. No Task or Flow was configured. No handoff was submitted or delivered. The action request proves only a decline and an offered handoff, not completed action or permission enforcement. The optional second model check was off.

Response times describe thirteen local API calls on one host. They are not a service-level promise. The app ledger estimates USD 0.019263 across both paid attempts, using its 2026-10-22 price list. This is not an invoice reconciliation. Setup, model variation, arbitrary customer documents, real account actions, production operation, and customer savings remain unproven.

Historical code tests: 2026-10-19

On 2026-10-19, we ran the complete answer and policy package test command from app commit ff997964580334e49bd64b54ff94d49b52a70ff6. The answer package ran. It recorded 338 top-level tests passed, 724 subtests passed, 0 failed, and 2 skipped. The policy package was attempted, but its database checks were blocked before database startup. Its raw runner result was 7 top-level tests passed, 0 subtests passed, 38 failed, and 0 skipped.

The record covers code paths in the answer and policy packages. It does not cover the full app, a live model service, every integration, or customer operation. Download the complete test-count record (JSON).

Results and blocked checks

The runner recorded 38 policy test failures. These were blocked checks, not observed policy assertion failures. The host could not create a shared-memory segment, so the private PostgreSQL cluster did not start. We did not mark those checks as passed.

The answer package skipped the recorded-evaluation replay and development answer-rate tests because their explicit replay settings were absent. Provider results used scripted providers or stored fixtures. No fresh paid model request was made. Subtests share code and fixtures, so adding them does not create an independent customer sample.

Download the raw record and keep its denominators.

What this evidence does not establish

This record does not establish a production wrong-answer rate, time saved, setup speed, renewal, reliability against future questions, or superiority to another vendor. A passing fixture proves only the assertion it checks. No model or production database was exercised through those fixtures.

Before publishing a customer case study, collect the customer-approved scope, source versions, question denominator, reviewed and unknown results, failures, full owner effort, costs, and dates. Record permission for the wording and identifying details. Use the case-study evidence record.