How to build your own real-time assistant for important conversations

Audio comes in on two tracks and is transcribed in the cloud in real time. The model analyses the conversation in “ticks”: every N seconds, or when a pad is pressed.

Stack: Python + FastAPI, everything local on a Windows laptop; the only external services are Soniox (transcription) and Anthropic (analysis). There is also an offline mode: upload a recording and a plan and get a timeline with on-plan stretches in green and deviations in red.

Control lives on an ordinary music pad console, the HXW / Avatar EMP16: 16 velocity-sensitive pads in a 4×4 grid, 4 endless knobs, 4 faders, an OLED screen, USB or Bluetooth (specs). You can hit a pad or move a fader without looking away from the other person and without keyboard noise: that is the whole reason it’s there.

Pads: four layers of tagging, one row per layer. What I felt; how the other person behaved; what I think they felt; what is happening with the meeting. Each press lands as a tag on the timeline at that exact moment. The “Analyze now” pad triggers an out-of-turn analysis.

Knobs and faders: processing parameters. Knobs select discrete things: model, preset mode, spending cap. Faders smoothly change four values, covered in the next section. The analyser reads them on every tick, so a turn takes effect from the next analysis, not from the next meeting.

Two practical points

• Binding by learning. Every unit sends its own MIDI numbers, and you can’t know them in advance. You pick a role on screen, turn the knob you want, and the binding is saved. Pads are calibrated the same way, with the first press.
• The controller runs in a separate process. If it disconnects or crashes, you lose tags, not the meeting recording (see “What to watch out for”). The system also works without the controller: the same pads and knobs are on screen, but then you have to look at the monitor.

The main finding: the cost of a meeting is driven by how often the model analyses, not by the size of the dialogue window. This goes against intuition: it feels like “sending lots of text” is what’s expensive. But text is cheap, while every extra model call drags along the system prompt, the answer and overhead.

What you change on the fly

The order of impact was confirmed by measurements on real meetings. Hourly cost is shown as a guide: what matters is not the numbers themselves but the spread between a knob’s extreme positions.

The controls sit on the panel in this order, by strength of impact. In a conversation it works like this: an important stretch starts, I raise the pace; small talk starts, I lower it. A text field is useless for this, while a fader is the same gesture as tagging with a pad. When a control moves, the forecast on screen updates instantly: you see the cost of a decision before you make it.

Preset modes

A preset sets all controls at once; after that you can move them by hand.

Three non-obvious things the measurements showed

1. The answer cap is the main valve. The model writes “as much as allowed”, not “as much as needed”: in a long meeting most analyses hit the cap, and the answer becomes the most expensive line in the bill. Lower the cap until the analyses get poorer; test it by running the same recording on two settings.
2. Memory makes every next analysis more expensive. Facts and open questions accumulate, and by the end of a meeting one tick costs several times more than at the start. Trim it with a window of recent facts, not an on/off switch: memory is what makes the analysis smarter.
3. Russian text costs roughly three times more than English: it splits into tokens less efficiently. That’s why back-of-the-envelope estimates always come out more optimistic than the bill.

No controls needed here, but these numbers had to be tuned on live meetings.

Measurement rule: don’t judge cost from runs shorter than 15 minutes. On a short test a noticeable part of the bill goes to writing the prompt cache, which doesn’t have time to pay off, and the hourly cost comes out twice too high.

The main rule: a failure of one part must not cost the meeting. The controller, the second track, the analysis, the database link: any of them may fail, but the recording must finish.

• A silent failure is worse than a loud one. The worst breakdowns look like normal operation on screen. The system must distinguish “wait” (service overloaded) from “go fix it” (out of credit, wrong key) and say so loudly.
• The controller runs in a separate process that dies together with the server. MIDI drivers on Windows can crash the whole process in a way you can’t catch. And only one process can hold a MIDI input: a forgotten old listener will grab the controller. If the device “isn’t visible”, check the cable first.
• The other person arrives on the output, not in the microphone. Capture the output of any device, including Bluetooth headsets, and keep the two tracks separate.
• A track can die mid-meeting. The other person moves from the call into the room, and all speech starts being attributed to you. No fresh audio on the track, no judgment about the role.
• Don’t mix two speaker labelings word by word. The transcription service cuts the lines, and the track physics decides once which voice is you. Token-by-token substitution tears monologues to shreds.
• The model’s answer is data, not a guarantee. It sometimes returns a list as a string. Normalize everything external to the expected types at the boundary, or the assistant goes silent until the end of the meeting.
• Give the model memory, not the tail of the transcript. Without accumulated facts and questions it re-asks what has already been said. Memory plus only the new chunk of dialogue: both cheaper and smarter.
• Nothing periodic next to the recording. Polling devices on a timer is a road to a crash. Check only when there is a reason, with an increasing pause.
• Cost is part of the interface. The spend counter and forecast are always on screen. An unaccounted cost line means “expensive or not” is decided on the wrong number.
• Show what goes into the model. A diagnostic panel (“this tick, on these words, produced this output and cost this much”) lets you spot a breakdown during the meeting, not after.
• The screen is a function of the event stream. Then replaying a meeting comes for free and never diverges from what you saw live.
• Don’t invent numbers or people. If the database is unavailable, the meeting is recorded under a temporary name and linked later. An ambiguous participant is chosen by a human, not the system.

The biggest gap between expectation and reality is where quality breaks and where the cost comes from.

Minimum kit: a Windows laptop, Python 3.11+, API keys for a speech recognition service and a language model, any MIDI controller with pads and faders (you can skip it at first).

1. Start offline. Call recording + plan → transcript → review of “where we followed the plan, where we drifted”. This shows whether you need help at all before you deal with real time.
2. Capture two tracks separately. The microphone is you, the computer output is the other person. Don’t mix to mono: you’d force the model to guess what is physically known for certain.
3. Add live analysis with memory. On every tick: the plan, accumulated facts and only the new chunk of dialogue; the answer strictly follows a schema. Validate answer types at the boundary: the model sometimes returns the wrong thing.
4. Keep parameters in session state, not in a config file. Pace, depth, window, memory and model are read on every tick. Add a cost panel right away: spent so far, cost per tick, forecast per hour. Without it the controls are turned blind.
5. Connect the MIDI controller in a separate process. Pads are tags, faders are parameters; binding by learning. The controller process must die together with the server.
6. Record the event stream, not just the text. Then any meeting can be replayed exactly as the host saw it, and the same recording can be run on two control settings.
7. Link to a people memory, non-critically. Mine is CoFlow: calls only at the start and the end, and if the database is unavailable the meeting still runs and is recorded locally.

And above all: record other people only with their consent.

If you are really interested, I can share the source code: a version you can start building your own from. Write to me and tell me which meetings you want such an assistant for.

Ask for the source code

I’m Pavel Mihutski, RND.energy, Valencia. For 20 years I’ve been building and rebuilding businesses; now I research how people make decisions. I have been analysing communication and building ERP and CRM systems since 2010. Artorix and CoFlow are the current versions of tools I have built and rebuilt many times. 9 industries, 130+ projects.

For me Artorix is not a product but a working tool. I use it for interviews and negotiations inside the Founder Sprint: ten working days with a founder on one main question. Where money is being lost, why everything depends on them, what will break as the company grows. From the company’s real data, including conversations with the team and call recordings, I build an honest picture of what is actually happening and where to go next. You get a grounded answer, an action plan with owners and a way to check the result.

If you want an assistant like this but don’t want to build it yourself, setting it up for your meetings is separate paid work.

How the Founder Sprint works. A first 20-minute conversation: via rnd.energy, LinkedIn or Telegram.

© Pavel Mihutski · RND.energy. This guide is for personal use: it describes the approach and the lessons, not the source code. Republishing, passing it on to third parties or commercial use only with the author’s written consent. The source code is not public: I share a version in person, on request.