Principles:
* Claude / Codex / Cursor etc are great data agents - you just need context to enable them to answer business questions correctly.
* Much of the important context lives in analysts heads'
* Context must be maintainable (short, human readable - markdown and yaml)
* Added context should be load bearing, not repeat of information that is available in another source of truth - and evaluations prove the value of the context
* Companies should own their own context, and must be shareable with any model
* Context should be version controlled (and github works as a context store).
What the library does:
* Performs an interview with your analysts (context-interview skill), one domain at a time, similar to what I have done as a consultant. The interview is informed by your query logs.
* The AI develops evaluation questions at the same time as building the context - and at the end of each domain interview asks you to run an evaluation with and without the context. The evaluation will actually check the results against your dashboards if you allow it. We have shown increases in accuracy from 87% to 95% by adding this context in our imaginary open-source company, Shorelane.
* Provides a set of skills to use the context that follow best practices for self service analytics:
* Write a plain english plan for the business user to approve
* Execute the plan
* Verify that the execution followed the plan, and provide a simple uncertainty score.
* Enable a adversarial review of the answer if their are doubts.
Context repo written in markdown and yaml after you finish the interview. We use a single github repo, divided into domains. The interview can be shared and restarted by checking in/out of github.
A set of skills to share with your business team
Optional: We offer a fast MCP server to serve the Github-based context to your team.
The AI performs an interview with that analyst using their AI of choice, then the analyst can hand the interview off, through Github, for another analyst to build a second domain's context (if there are multiple analysts).
Automated: Through github actions, you can hook your data processing repo (like dbt) to this repo, so that when a change occurs upstream, it reruns the evaluations like a regression suite. If there are failures, you can add your AI API key (currently only Anthropic, but on the list to expand) and it will suggest changes to the context.
The reactive mechanism - where the business starts asking questions they never have before. I am building observability tools (seen nodaldata.io) specifically around that process. This is where new context typically needs to be added.
The third way context goes stale is that the business quietly changes a definition - “active customer” now means something different, without a dbt change or a "new question" being asked. Including dashboard reconciliation in occasional regression tests could surface drift when a dashboard trusted number and the eval number diverge.
Also, since every definition carries an approver name and a confirmed date, there may be an "interview refresh process" (maybe every 6 months) where analysts re-interview for stale documentation.
In addition to the observability tools that highlight new questions, I try to instill sampling into self service analytics to continually monitor performance of the system.
Have you seen any other techniques that could work?