Imagine you are preparing a brief on an unfamiliar subject, and you ask a junior employee to help with the research. They return in record time with a beautifully structured presentation filled with polished arguments and confident recommendations, references in tow.

Then you discover how they got there.

They did a cursory search on the topic, read the first few seemingly relevant articles and got straight to work on the presentation. They never checked how trustworthy those sources were. Or traced claims back to their origins. Or looked for contradictory evidence.

Not exactly the trusted aide you thought you had.

Yet increasingly, this resembles our experience working with AI assistants.

How AI Assistants Find Their Answers

The modern AI assistants we use are no longer just LLMs working from what they learnt during training. Many now search external sources (often the web) for information relevant to our question, bring that information into context, and then use the LLM to synthesise a response. This general approach of pairing an LLM with retrieved information, commonly known as retrieval-augmented generation (RAG), has made answers from AI assistants far more current, grounded and traceable.

But retrieval has a fundamental challenge – finding a highly relevant article is not the same as finding an accurate and reliable one. It could be written by someone who’s not an expert on the topic. It could be hearsay, opinion, speculation, or simply something repeated often enough to look established. In other words, good retrieval cannot be done blindly, and needs to be paired with judgements on how much each piece of information deserves our trust.

Enter the AI Snake Oil Salesman

A friend of mine recently tested this in a rather mischievous way. He published an article containing deliberately false information about a pretty niche subject. Lo and behold, when he asked an AI assistant about this subject, the false claims came back to him, now incorporated into an extraordinarily fluent and confident-sounding answer.

The worrying part isn’t just that the AI assistant got it wrong – it’s how right the wrong answer feels.

Now, this isn’t a new phenomenon – we humans have routinely fallen prey to snake oil salesmen with slick presentations light on accuracy. Part of the reason is that it’s mentally taxing to weigh and judge every piece of information that comes our way based on its actual content (let alone investigate the evidence behind it).

Instead, our brains rely on a variety of shortcuts when deciding what to trust, one of which being how easy it feels to process something. Clear language, coherent sounding arguments and familiar concepts can all make the information easier to digest. That feeling of ease can, in turn, influence how credible or truthful we perceive the information to be. Psychologists even have a term for it – processing fluency.

Now if processing fluency can influence our judgement, then LLMs present an interesting problem: fluency is, in many ways, a by-product of how they are built. LLMs have great linguistic fluency, producing clear sentences and polished structures that feel natural to any native speaker. In addition, LLMs demonstrate domain fluency, speaking like an expert of the domain by using vocabulary, concepts and expected arguments commonly associated with it.

In essence, AI assistants can reproduce many of the surface signals we humans traditionally associate with expertise without necessarily having done the work that gave those signals their credibility in the first place.

What Are AI Companies Doing?

To be fair, AI companies aren’t oblivious to this problem. ChatGPT, Claude and Gemini increasingly offer differentiated modes such as Search, Thinking and Deep Research that actually try to do the work and check for credibility, spending more time retrieving and comparing sources before producing an answer. These are all good, but there seem to be some gaps in these developments.

First, differentiated modes that do more work are good, but the default mode still sounds way too confident overall. We could easily have an answer that contains one claim that’s well established, another based on incomplete evidence, and yet another drawn largely from anecdotal comments on Reddit or a random forum. Yet all three may be presented with the same fluency and in the same confident tone.

Secondly, the differentiated modes generally describe how much thought or effort we want the AI assistant to employ. Presumably, the additional effort is intended to produce a more reliable answer. But ultimately, effort is not what I care about as a user. I’m more interested in how much I can trust the answer.

Designing for Reliability

I believe there’s an opportunity here for more nuanced UX design: How might we design AI interactions and interfaces that give users a better sense of how much they can trust different parts of an answer?

I believe Perplexity is on the right track in answering this design challenge by experimenting with labels indicating whether sources used within their responses are academic, governmental or trusted. I would also like to propose a few other relatively simple interaction ideas:

• Make differences in reliability visible – rather than presenting an answer as one continuous block of equally authoritative prose, the interface could distinguish between claims that are well established, based on limited evidence, contested or primarily AI inference
• Include discussion of reliability in the response – instead of saying “X causes Y”, the LLM’s response could be “there’s fairly strong evidence that X causes Y” or “the evidence here is limited, but my interpretation is…”
• Allow users to interrogate reliability locally – instead of switching the entire conversation into Deep Research, allow the user to select a claim and essentially say I need this part to be right – triggering deeper verification for that claim

Before That Happens

Before such interfaces arrive, one thing we can do today is to explicitly ask the assistant to communicate reliability as part of its response. For example, we might add to our prompt: “For each claim, include a brief assessment of how well supported it is.” If this is something you need frequently, these expectations could also be encoded into persistent instructions, projects or skills.

In some sense, this approach echoes how you might need to improve the performance of the feckless junior employee we encountered at the start of this essay – by asking him not just to “show his work”, but to tell you how much confidence you should place in each point, and why.

LLMs have made polished language, coherent arguments and expert vocabulary extraordinarily cheap. When sounding like an expert becomes essentially free, we need to figure out how to tune our interactions to look beyond how good an answer sounds and ask how well supported it actually is.