Productionising The Vibes

One stream of work that has really picked up recently are projects where we work with founders and teams to improve their tested and validated prototypes and Minimum Viable Products (MVPs). This means we take these working concepts and transform them into production-ready product that are able to handle significant visitors or customers without spiralling AI costs and without outages during heavy loads. Often there is a strong AI element to the product, and that's why we get brought in.

Initially, the vibe coded product or feature (something built quickly using AI coding tools) is working well, but the team has reached a limit to improvement. Increasingly, we're also being brought in to wrestle escalating AI costs. This is often a stressful scenario, where the team has Product Market Fit (PMF) but the cost of servicing customers is steadily increasing with no end in sight.

In this piece we cover the essential elements we add to benchmark and drive performance and reliability increases across AI projects. Without having these building blocks in place it's impossible to understand where effort should be spent to improve your product and if you're actually making any long-term sustained improvement.

Documentation

We start with a thorough code review and document as we go along. Project documentation should live close to the code and should be accessible by users and agents. Often this means it should be contained within the code base, but we've seen it also work where it's connected to Notion. The key here is to keep the documentation concise, up-to-date and relevant to the current state of the project.

If the documentation lives with the code, we keep a small, always-current set of documents next to it: a roadmap of the work in progress, running notes worth holding on to, and the plans and specifications for whatever is being built right now. Anyone picking the project up, whether a new developer or an AI agent, can get up to speed from these quickly.

We initialise these documents and keep them current through the life of the project. Once a particular task or unit of work is done, we then delete the plans and specifications and migrate this information to either README.md or one of the documents in the docs/ folder. The test here is to avoid filling this folder up with all the step by step information from a unit of work. Instead add a light summary of what each ticket or unit of work contains. Like with the code outside this folder, keeping it well maintained means it's easier for agent and developer to understand the project when they pick it up for the next bit of work.

Recommended tools: Git, Notion

Evals

For us Evaluations (Evals) are the secret sauce on any AI project. Evals are ways for measuring whether an AI feature or product is performing as intended. It allows every prompt or agentic loop to be benchmarked and so you can validate quantitatively if updates to your prompt or model improve or degrade performance. Crucially because you can run these before your product gets shipped to real users, you can be confident that you aren't giving them a worse product with every update.

On most projects we start by building an evaluation pipeline that can be used on day 1 to benchmark performance. This drives conversation with clients to identify the following key criteria:

1. What does success look like? What is a bad response?
2. What are representative data sets or inputs?
3. Run tests on these inputs to see how the AI product performs.
4. How can these inputs vary in real-life performance? Introduce these variations and see how it works again.
5. Optimise and iterate: tweak prompts or model choice and understand how this affects outcomes.

We do the above for each prompt or Agent node because you need to test each component in isolation to understand its performance in isolation. It's also then useful to set up unique evaluations for the entire product or feature to understand how the system works together.

In practice we check outputs in three ways: automatically with code, against a set of known-good examples, and with a second AI acting as a judge where the output is hard to score.

Recommended tools: Langfuse, Logfire, Braintrust

Observability

Observability gives you insight into how your Agent or AI feature is operating. Without the detailed tracing offered by observability tooling you are often flying blind. Evals form a part of your observability suite and it is useful to see your traces and outputs all in the one platform. Think of observability as the instrumentation that wraps around your AI product or feature and shows you what's happening inside each call out to an agent and why.

The core of AI observability is application tracing. The platform should show you structured logs of every request, the prompt, context retrieval, tool calls, the model's response, token usage, and latency. You can see how all of this information is useful and how you can then get real visibility into the operation of your platform.

All of the recommended platforms record these elements:

• Operation: A unit of work that describes typically every step in generating a response from an LLM. Typically this could be LLM calls, tool calls, etc.
• Trace: A connected multi-step set of operations representing the timeline to go from an initial interaction to a final response.
• Metric: A value measured over time, such as latency or cost
• Log: A time stamped record of a single event.

To implement observability in a project, we'll need to add a small amount of tracking code so the platform can record every AI call, what it cost, and how long it took.

Once instrumented, your observability platform of choice will capture and display the elements described above and allow you to observe how your AI product or feature operates at a granular level. This instrumentation data is more than just a capture of historic interactions. It should also be used to continuously improve your product. A trace could be scored by another LLM, annotated by a stakeholder or be captured and used to update your evaluation dataset. It represents an opportunity to close the loop. The detailed token and cost information provided by these platforms also allow you to carry out optimisation to reduce token consumption and therefore costs.

Recommended tools: Langfuse, Logfire, Braintrust

Product Tracking

In some instances you may want to understand how users are interacting with your wider product, beyond the AI feature at its core. This requires user behaviour tracking across the product. The mechanics are almost identical to the observability implementation, but the question is different. Here you are measuring the user experience: page load times, clicks and taps, form completions, and the points where people drop out of a flow. It gives you a customer-centric view of product usage that sits alongside the AI-centric view your observability already provides.

This matters most once an AI feature is working. Observability gives you insight that your prompt is fast and cheap and hopefully accurate. Product tracking tells you whether anyone reaches it, uses it, and comes back to use it again. We have seen teams tune an agent for weeks, then discover that most users had abandoned the flow two steps earlier for reasons that had nothing to do with AI.

Implementation looks very similar to the instrumentation piece, and it will often mean a second layer of tracking code alongside your traces. You capture similar events twice, once for the engineers debugging the AI and once for the product and data teams reading behaviour. We accept the duplication because the two audiences ask different questions and work in different tools. To keep it manageable, we share a single session or user identifier across both, so a product event such as "abandoned checkout" can be traced back to the AI operations behind it. It's at this point that product tracking and observability stop being separate exercises. You can put a token cost against a user journey, and a user outcome against a token cost.

Recommended tools: PostHog, Amplitude

Wrap Up

All of these elements are used to help teams better understand the current performance of their product and how we can work together to optimise and improve both performance and cost. Implementing these features in a product helps turn a vibe coded prototype into a real product that is reliable and cost-efficient to run. If you'd like some help getting your project to the next step, do reach out, we'd be happy to help.