LLM in the Loop

I used to work with an AI enthusiast. Let’s call him Dave.

Dave was an ideas guy. He saw opportunities everywhere and wanted to move fast. But Dave was also extremely busy. His workflow usually consisted of typing three raw bullet points into ChatGPT, asking it to bulk them out into a full proposal, and firing it across to the team.

The resulting documents were painful to read: dense, generic, and padded out with fluff. AI slop.

My solution? I’d paste Dave’s document into ChatGPT and ask it to summarize the three core points hidden inside the noise.

Take three ideas, use AI to inflate them into 2,000 words, and then use AI to summarise them back to three core points. It felt absurd. Surely there’s a more direct way to communicate?

Generic AI Output is a Feature, Not a Bug

Why does AI output sound so generic? It’s a fundamental part of how an LLM works.

At its core, an LLM takes your prompt and generates the next most likely word (or token). Then, it takes your prompt and that first word, and generates the next most likely word. It repeats this process until it reaches a natural stopping point.

It doesn’t think. It doesn’t come up with an answer before figuring out how to phrase it, it just generates the most likely text. That is why it sounds so generic: it generates the average of everything it has been trained on. Mediocre, generic, and unoriginal.

Human in the Loop

In software engineering, we often talk about keeping the “Human in the Loop”. The AI might generate a block of code, but the senior engineer checks it against strict criteria: Does this meet the actual requirements? Does it handle edge cases? Is it secure, scalable, and readable?

The human does the critical thinking. The LLM does the bulk of the heavy lifting.

LLM in the Loop

In executive thinking and writing, the balance shifts. I don’t want an LLM creating generic slop based on my novel thought. Instead, I take control of the writing, and I use the LLM as a researcher and editor.

Writer Sam Thomas Davies highlights this in his essay Everyone Writes With AI. Almost Nobody Edits With It. His framework involves writing your own initial draft to capture real human insight, then using an LLM as a ruthless editor for tone, clarity, and logical gaps. Similarly, tech executive Don Woodlock points out in his video How to Use AI Without Letting It Think for You that LLMs excel at generating options and broadening your scope (divergence), but humans must handle the exploration of ideas, and final convergence and decision making.

A real-world example

An MD mentioned what keeps him up at night is getting more high intent traffic onto his website. He was spending a non-trivial amount on a digital marketing agency, but didn’t know whether he was focusing on the right things. Digital marketing isn’t my primary domain, but vendor oversight is, so I ran a diagnostic to see where the gaps are.

I mentioned to my LLM thinking partner that I was going to look into this, and it immediately (un)helpfully drafted an email for me to send to the MD (edited for brevity, it was long!):

Dear MD,
I have discussed your digital marketing questions with my network (lie), and 
they have recommended the following actions for you:

  1. Invest in PPC (good advice, but unsubstantiated)
  2. Targeted outreach on LinkedIn (sales, not marketing)
  3. Automate customer re-orders (wrong problem)

I recommend speaking with these local agencies (two agencies I have not worked
with).

Best,
Rob

Safe to say, that output went in the bin.

Instead, I used the LLM as a research assistant and editor.

• Targeted Research: I used the model to quickly come up with realistic search terms and deep dive into how current Google technologies work, such as shopping / product and location results.
• Human Synthesis: With this information, I dug into the code, used tools to confirm my findings, and framed my recommendations.
• Editorial Polish: I wrote the first draft of the report itself, based on my findings and recommendations. Then, I asked LLM to critique it, asking to find weak spots, fix the flow, and highlight poor phrasing.

The result? In less than three hours, I diagnosed the problems, decided on the recommended fixes, prioritised them on commercial impact, structured a board-level review, and produced a clear deliverable. In the past, that research and drafting process would have easily taken twice as long and the final document wouldn’t have read nearly as cleanly.

The framework

To move fast, use LLM as a tool with defined strengths, and keep the task hierarchy clear:

1. Ideation (Human): Connecting different ideas, and deciding what matters, is a deeply human task. LLMs can spit out ideas, but they’re always generic and average.
2. Research (Human + LLM): LLMs are great at accelerating technical research, summarising complex topics, and mapping findings to options. However, all facts must be checked by a human; LLMs generate text that sounds plausible, but don’t check facts.
3. Drafting (Human): This is where your experience, knowledge, and perspective bring your original thinking to life. Write the first draft yourself to preserve your authentic voice and avoid sounding generic and bland.
4. Editing (Human + LLM): Tell the LLM which style you are going for, and ask it to critique the draft without sparing your feelings. It will give you pointed feedback, find logical fallacies and unresolved loops, refine difficult sentences, and suggest structural improvements.

LLMs are extraordinary productivity tools, but they are accelerators, not thinkers. Thinking is for humans, but keep the LLM in the loop.

References & Notes

YouTube: Don Woodlock - How to Use AI Without Letting It Think for You

Words Into Works: Sam Thomas Davies - Everyone Writes With AI. Almost Nobody Edits With It

(Technical side note: Yes, describing modern LLMs like ChatGPT or Gemini purely as token predictors oversimplifies modern agentic setups, tool use, and search wrappers. But underneath the agents, the core mechanism remains generative text prediction, not human cognition.)