Skip to content

Data

  • in AI, Data
  • 10 min read

The One Where the Documents Answer Back

A close-up of a man with pale, clouded eyes looking straight at the camera

On the last Friday of every month, my team gets together (remotely) for a hack day. We have complete freedom to build anything we are interested in. The only catch is that you should have a vision of how what you build will improve the company. The goal is to build a proof of concept (P.O.C.) to show the rest of the team by the end of the day. Then, everyone votes on which P.O.C. impressed them, and the winner gets £50 to donate to charity or claim as a voucher at any store.

I've been interested in the idea of a company brain since I first heard it in a Y Combinator video (YC is a well-known U.S. startup accelerator). It speaks to a problem I face every day as a data analyst: a lack of context. I build data pipelines and run analyses, but I rarely see the decisions that came before the work, so I rely on my manager to remember and pass them on. That doesn't scale, for him or for me. If meetings, emails and documents were saved somewhere an AI could search, it could hand me that context before I start the next task.

My conceptualisation for this idea was a RAG pipeline (explained below). What I built was a small app where you uploaded a PDF or Word document, asked a question in plain English, and it answered with numbered references, like [1] and [2]. Click a number to see the exact passage it used.

This post has two halves: what I built for the team, and what it would take to make it something a team can rely on. The second half is really where things get interesting.

  • in Data, AI
  • 7 min read

The One Where the Row Counts Matched

A still from Inception: two men in suits at a bar

Last week on Obi the Explorer, I told you why pausing a Fivetran connector doesn't put any money back in your pocket, and what you inherit when you own the pipeline yourself. If you missed it, you can catch up here. Today's question comes before all of that: how do you know the replacement actually works?

I'll save you some time. Matching row counts won't be enough.

  • in Data, AI
  • 8 min read

The One Where You Leave a Managed Connector

Jamie Lee Curtis and Lindsay Lohan in Freaky Friday

Last week on Obi the Explorer, I told you we replaced three Fivetran connectors with pipelines we run ourselves on Airflow, and that a connector quietly does seven jobs on every run. If you missed it, you can catch up here. That post ended by teasing this one, so here we are.

The seven jobs were only the start of what we took on. This week is about the rest of the inheritance and why pausing a connector doesn't put any money back in your pocket.

  • in Data, AI
  • 12 min read

The One Where We Replaced Fivetran

Cynthia Erivo as Elphaba, in a wide black hat, in Wicked

Last time on Obi the Explorer, the agent learned to do everything at once. Four channels, live kitchen conditions, and lookups running side by side: a pattern called Parallel Delegation. I explained it with a host who, the moment a guest walks in, asks the maître d' about them and radios the kitchen to ask how busy it is, both at once. Draw that on a whiteboard, and you get a DAG (directed acyclic graph): steps that fan out, join back together and never loop back. That DAG lived in a diagram, but this week I'd like to show you one that lives in production.

For the last few months, I led the replacement of three of our Fivetran connectors with pipelines we run ourselves on Airflow. Those three made up 96% of our Fivetran usage. Not the most glamorous sentence I've ever written, I know. Stick with me, though, because there's a lot that goes into this that I was not really prepared for. Replacing Zendesk was horrible.

This is part one of a weekly series I'll be dropping over the next eight weeks. I'll take the project apart, looking at what broke, what I'd do differently, and how we proved it worked before we switched anything off.

How to Count a Person Exactly Once

Naruto's shadow clones filling a forest clearing

It's still just one Naruto.

Last week on Obi The Explorer, I covered a data pipeline built on Fivetran that turns every Typeform survey into a single reusable set of models, rather than a fresh CSV and a fresh script every time. You can read all about it here. Everything was rosy until a survey came through and exposed weaknesses in the modelling. This post explains why counting people isn't the same as counting answers, what had to change in the models to fix it, and how getting it right ended up mattering well outside the warehouse.

Why We Stopped Rebuilding Survey Analysis From Scratch Every Time

Phil Connors reporting from Gobbler's Knob in Groundhog Day

Our own Groundhog Day.

At my current company, we run patient feedback surveys, clinical intake forms, and stigma studies through Typeform. Think teams across multiple markets and functions. The same way any company interested in providing its customers with the best possible experience would. For the uninitiated, Typeform is a form builder that lets businesses collect data people want to share. It's easy to use, customisable, and it's great. (Not a sponsorship wink wink).

  • in dbt, AI, Data
  • 7 min read

The Seven Gates We Put Between an AI Agent and Our dbt Models

How our data team stopped trusting a coding agent to remember the process.

TL;DR: An AI agent will write you a dbt model that compiles, passes every test, and is wrong. Adding more instructions doesn't fix this, because the real gap is memory, not knowledge. We moved the process out of the agent's head and into the repo: seven phases, each a gate that must be verifiably passed before the next can start. Below is the whole workflow, what triggered it, and what it's caught so far.

  • Let's have a virtual coffee together!

    Want to see if we're a match? Let's have a chat and find out. Schedule a free 30-minute strategy session to discuss your AI challenges and explore how we can work together.

    Book Free Intro Call