I Let an AI Agent Do My Bookkeeping (Mostly)

I've tried to keep a budget plenty of times. It usually lasts about six weeks. The app is fine. What kills it is the upkeep: hundreds of transactions a month, half of them labeled "TST* SOMETHING 0423," a Target order that's really four different kinds of spending, and a card payment that looks like spending even though it isn't.

So I stopped trying to be disciplined and set up a system where I don't have to be. An AI agent does the tedious part, and I just answer questions in chat.

The setup

Everything runs at home, and it's all fairly boring:

• Lunch Money is my budgeting app. It pulls in transactions from my checking, savings and credit card accounts through Plaid.
• Hermes, an open-source agent framework, runs in a Docker container on my Unraid home server. It uses Claude as the model.
• A small Lunch Money MCP server that I maintain. It lets the agent list transactions, categories, accounts and recurring bills, and it can update categories, split transactions and trigger a bank sync. When I need a new capability, I add a tool for it.
• Craft, my notes app, also connected through MCP. The agent writes summaries and research there, and leaves a handoff doc when I switch chat apps.
• Discord (and before that Telegram) is how I talk to it. I mostly send short messages from my phone.

The rule I care about most: the agent doesn't write anything to my finances until I approve it. It can read whatever it wants, but any change gets shown to me first.

What I actually use it for

"How's our spending this month?" That's the most common question. The agent pulls month-to-date transactions, totals them by category, and points out what skews the number. This month, for example, the first property tax installment was more than 80% of all spending, so it gave me the total with and without the tax. It also leaves out card payments and transfers so nothing gets counted twice, and it tells me which ones it left out.

Categorizing in rounds of five. Uncategorized transactions come to me in small batches, each with a suggested category, a confidence level and a short reason ("categorized as Groceries 14 times before"). I reply "all," or "all except 2; 2 = Gifts." It also keeps a running file of my payee rules. Costco needs a receipt. The Shell Cellar is a shop, not a gas station. Uber Eats counts as Restaurants, but Uber rides count as Ride Share.

Receipts. I can drop a PDF receipt into the chat. Even when it's just a scanned image, the agent reads the line items, groups them by category, checks that they add up to the card charge, and splits the transaction. It also noted which items are FSA-eligible.

Catching things I'd miss:

• A refund filed as a "transfer" instead of in the original purchase's category. That meant the purchase still counted as spending, but the refund that cancelled it didn't.
• A Target gift card promo that showed up as two mystery charges. Once I sent it a screenshot of the item discounts, the totals made sense.
• A subscription I'd cancelled. It knows the service is cancelled and tells me if a charge shows up anyway.
• Cards that aren't linked. The agent knows that for some cards, all it sees is the payment to the card and not what I bought, and it says so instead of pretending its numbers are complete.

Forecasting, not just reporting. "Can I cover everything until payday?" gets a running balance through the next paycheck: mortgage, HOA dues, insurance, card payoffs. The low point is flagged, and if it gets too thin, it suggests moving money from savings. For the winter utility bills, it compared year over year and then adjusted for something the history doesn't show: I started charging an EV at home this year.

Why it works

1. It remembers what I prefer. My categorization rules, which accounts aren't linked and how my cash flow usually goes all live in files the agent reads before each task. I don't have to explain things twice.
2. It's honest about what it can't see. It tells me when it only read the existing data versus actually syncing with the banks, when a pending charge might still change, and when something is just a guess. That's what makes me trust the numbers.
3. It's fast to use. Most of my replies are one word. Ten minutes a week keeps the books cleaner than any budgeting attempt I've made before.
4. It runs on my own server. The agent, my notes integration and the finance connector all run on my own hardware. The only thing that leaves my house is the model calls.

What I'd tell someone trying this

• Start read-only. Ask questions for a couple of weeks before you let the agent write anything.
• Make it ask before every change. It's slower, but you'll never have to unpick a bad batch edit.
• Write rules down as they come up. Every correction ("Walmart is General Shopping") should go into a file the agent reads later.
• Expect it to have gaps. Unlinked cards, cash and gift cards are all invisible to it. A good agent tells you that up front.

I still check the numbers. I just don't spend my evenings categorizing coffee anymore.