Skip to content

Blog

  • in AI, Data
  • 10 min read

The One Where the Documents Answer Back

A close-up of a man with pale, clouded eyes looking straight at the camera

On the last Friday of every month, my team gets together (remotely) for a hack day. We have complete freedom to build anything we are interested in. The only catch is that you should have a vision of how what you build will improve the company. The goal is to build a proof of concept (P.O.C.) to show the rest of the team by the end of the day. Then, everyone votes on which P.O.C. impressed them, and the winner gets £50 to donate to charity or claim as a voucher at any store.

I've been interested in the idea of a company brain since I first heard it in a Y Combinator video (YC is a well-known U.S. startup accelerator). It speaks to a problem I face every day as a data analyst: a lack of context. I build data pipelines and run analyses, but I rarely see the decisions that came before the work, so I rely on my manager to remember and pass them on. That doesn't scale, for him or for me. If meetings, emails and documents were saved somewhere an AI could search, it could hand me that context before I start the next task.

My conceptualisation for this idea was a RAG pipeline (explained below). What I built was a small app where you uploaded a PDF or Word document, asked a question in plain English, and it answered with numbered references, like [1] and [2]. Click a number to see the exact passage it used.

This post has two halves: what I built for the team, and what it would take to make it something a team can rely on. The second half is really where things get interesting.

  • in Data, AI
  • 7 min read

The One Where the Row Counts Matched

A still from Inception: two men in suits at a bar

Last week on Obi the Explorer, I told you why pausing a Fivetran connector doesn't put any money back in your pocket, and what you inherit when you own the pipeline yourself. If you missed it, you can catch up here. Today's question comes before all of that: how do you know the replacement actually works?

I'll save you some time. Matching row counts won't be enough.

The One Where I Built a Thinking Partner

Four office workers huddled together in a bright, empty corridor, in a still from Severance

I watch an unhealthy amount of YouTube videos. However, my only saving grace is that they're videos about technology, business, sports and geopolitics. So, I'd say I'm getting more useful information than not. I'm evaluating myself, so take this with a grain of salt.

Someone makes a good point about pricing, the latest technology, a development project, or how to talk to small-business owners, and I think, "That's useful." Then I close the tab. A week later, I remember learning something, but not what. I often don't take the time to think through how what I've learned can be applied to my benefit.

So I built a small system to fix that. It has two parts: a tiny MCP server that fetches a video's transcript, and a notebook of plain markdown files where an AI assistant and I think each video through against what I'm actually trying to decide. On the first day, I used it properly by processing eight videos.

The MCP server is about 200 lines of Python. The notebook is the part that matters. This post covers both, what broke along the way, and what I still can't tell you.

  • in Data, AI
  • 8 min read

The One Where You Leave a Managed Connector

Jamie Lee Curtis and Lindsay Lohan in Freaky Friday

Last week on Obi the Explorer, I told you we replaced three Fivetran connectors with pipelines we run ourselves on Airflow, and that a connector quietly does seven jobs on every run. If you missed it, you can catch up here. That post ended by teasing this one, so here we are.

The seven jobs were only the start of what we took on. This week is about the rest of the inheritance and why pausing a connector doesn't put any money back in your pocket.

  • in Data, AI
  • 12 min read

The One Where We Replaced Fivetran

Cynthia Erivo as Elphaba, in a wide black hat, in Wicked

Last time on Obi the Explorer, the agent learned to do everything at once. Four channels, live kitchen conditions, and lookups running side by side: a pattern called Parallel Delegation. I explained it with a host who, the moment a guest walks in, asks the maître d' about them and radios the kitchen to ask how busy it is, both at once. Draw that on a whiteboard, and you get a DAG (directed acyclic graph): steps that fan out, join back together and never loop back. That DAG lived in a diagram, but this week I'd like to show you one that lives in production.

For the last few months, I led the replacement of three of our Fivetran connectors with pipelines we run ourselves on Airflow. Those three made up 96% of our Fivetran usage. Not the most glamorous sentence I've ever written, I know. Stick with me, though, because there's a lot that goes into this that I was not really prepared for. Replacing Zendesk was horrible.

This is part one of a weekly series I'll be dropping over the next eight weeks. I'll take the project apart, looking at what broke, what I'd do differently, and how we proved it worked before we switched anything off.

The One Where The Supplier Fixes Their Own Invoice

Leonardo DiCaprio as Frank Abagnale in a pilot's uniform, surrounded by flight attendants, in Catch Me If You Can

Recently, I've been building an end-to-end invoice review app. Not the most glamorous sentence I've ever written, I know. Stick with me, because it turned into one of the more interesting lessons I've had about where AI belongs in a system and, more importantly, where it doesn't.

The One Where Everything Happens At Once

Evelyn Wang with her third eye open, papers flying around her in Everything Everywhere All at Once

Last time on Obi the Explorer, the agent got a critic. One LLM wrote the recommendation, a second one judged it, and nothing went out until it passed. I closed by wondering how much of a good host is really just knowing what's going on in the room, and whether an agent can ever know that from the outside.

This week, we find out. This is the last iteration, so I'll land it properly.

If you've worked front-of-house, you'll know there's never just one door. The phone rings while an email comes in, someone's waiting at the desk, and there's a message sitting in the Instagram inbox from last night. Assuming your place is trendy like that.

Meanwhile, the kitchen is telling you it's slammed, and you're quoting wait times based on whatever you last heard (Yeah, it gets chaotic, but it's rewarding at the end of the day). So you will notice that in the same restaurant there are four different conversations, and the customer gets a different experience depending on which one they pick.

The One With The Critic

Remy on Linguini's head, reflected in the kitchen mirror in Ratatouille

Last time on Obi the Explorer, the agent made contact with the outside world. It started telling the kitchen what to do, and I spent most of the previous post explaining why it should check the fridge first. I closed on a question I couldn't quite answer. If the agent learns to check its own work, who checks the checker?

This week, we find out. Kind of.

So last week, I shouted out my anniversary. It was great. Had a lovely dinner and went to a magician's bar afterwards for drinks. As in, while you're at your table, a magician comes and does a whole bunch of tricks. Like your own private show. Try new things, but know enough about the person you're with to decide which risks to take. Alright, that's all for now.

The One Where We Make Contact

Louise Banks and a heptapod pressing hands against the glass in Arrival

Last time on Obi the Explorer, the agent learned. It remembered regulars, kept track of what they liked, and wrote it all down once each conversation was over. I closed on Ava from Ex Machina, and the idea that the dangerous part isn't knowing someone. It's acting on it without asking.

This week, the agent finally acts. Not on you, thankfully. On the kitchen.

If you've watched The Bear, you'll know the sound. An order gets called, and the whole line shouts "yes, chef!" before anyone has touched a pan. Once an order is fired, it's happening. Ingredients come out, prep starts, and someone's evening now revolves around that dish. You can't really un-fire it.

That's the moment this post is about.

  • Let's have a virtual coffee together!

    Want to see if we're a match? Let's have a chat and find out. Schedule a free 30-minute strategy session to discuss your AI challenges and explore how we can work together.

    Book Free Intro Call