From Embeddings to Evidence: Making AI Systems Easier to Trust
Embeddings are often treated as the foundation of modern AI applications. They allow us to represent the meaning of text numerically, search for similar information, and retrieve context for a language model. For many RAG systems, that is the basic recipe: chunk documents, embed them, retrieve the most relevant passages, and use them to generate an answer.
But building these systems taught me that embeddings are only the beginning. Finding information that is relevant is not the same as proving that an answer is supported. Retrieving information is also not the same as helping someone get useful work done.
At AAA, I built a chatbot for a financial services team that needed to answer questions about a new credit card program. The information was important, so the team needed to know exactly where each answer came from, and they needed a way to verify it themselves.
We used vector search to chunk and embed our sources of truth, then retrieve the most relevant passages for each question. But the retrieved text was only part of the solution. We also preserved the source documents and linked answers directly back to the corresponding Google Drive files.
That made the chatbot’s answers inspectable. Someone could open the underlying document, review the original information, and decide whether the response was accurate. We also integrated the chatbot into Google Workspace and added it to existing Google Chat rooms. This kept the system inside the team’s normal workflow and made verification collaborative. Teammates could confirm that an answer was correct or point out a better source or interpretation.
The important lesson was that the embedding model did not establish trust on its own. It helped locate the relevant information. Trust came from source transparency and human review.
At Ibotta, the problem changed. We were no longer focused only on answering questions. We wanted an AI system that could help people get work done.
A simple RAG chatbot was not enough for that. The system needed to work inside existing workflows, minimize permission barriers, and take actions that users could easily review and adjust. To support that, we built scripts and predefined workflows that empowered an agent to complete different kinds of work and edit documents across a project workspace.
One example was creating a project closeout document. Instead of asking someone to search through an entire project folder, the workflow could examine the project’s retrospective, meeting notes, weekly updates, Gantt chart, communications plan, task tracker, and other documents created throughout the project lifecycle.
With workflows for extracting information from those documents, the system could organize the project history and use it to draft a closeout: what happened, what went well, and what could have been better. The user still reviewed and edited the result, but they no longer had to spend hours locating and organizing every relevant document themselves.
This approach also helped remove common bottlenecks. Users did not need to know the perfect prompt, find the right template, or understand exactly how to structure the document. The workflow defined those pieces ahead of time. Users could provide the information they had, and the agent could take it from there.
The system’s actions remained user-driven, visible, and reversible. It worked with existing files and project folders, created drafts and documents in familiar places, and did not require the AI to operate outside the boundaries of ordinary work.
Across both projects, I learned that embeddings are only the beginning. They can help an AI system find relevant information, but relevance is not the same as trust, and retrieval is not the same as useful work.
At AAA, trust came from linking answers to their source documents and keeping teammates involved in verification. At Ibotta, usefulness came from building workflows that could organize scattered project information and create meaningful deliverables while keeping every action reviewable and reversible.
The real work is designing the system around the model—so people can understand what it used, decide whether it is correct, and remain in control of what happens next.