*When I reference artificial intelligence (AI) in this post, I’m referring to large language models like ChatGPT or Gemini as most people use the term today.

I’ve been in the computing world for a long time. Before AI, the expectation for computerized processes was precision and repeatability. If we do the same thing over and over, we should get the same result over and over. Once we tested the process in a given scenario, we knew it worked.

If you’d like a fancy word to describe this, the systems were deterministic.

AI doesn’t work this way.

AI is probablistic.
It generates an answer rather than the answer.

If you ask AI the exact same question twice, you’ll often get slightly different answers. This is not a flaw in the system, it’s part of the system design. Randomness is incorporated into the system.

Higher and lower levels of variation are known as temperature. In some scenarios, you can manually set the temperature for the model you’re using.

Sometimes Accurate-ish is Fine

Many times, you’re not looking for an exact answer. There might be multiple valid ways to do what you want and it doesn’t matter which one is used.

Some examples:

• Basic research. When you’re learning a new topic and you ask general questions about that topic, you will get varying results that are useful but not all the same. As long as the general concepts are similar, the exact wording doesn’t matter. You still get a good overview of the topic.
• Reviewing documents. When you ask AI to review a document for grammar or writing style. The exact phrasing isn’t important as long as it meets your goals.
• Generate a chart/graph/dashboard. If you give data to AI and ask for a bar chart based on the data, the exact width or height of the bars probably doesn’t matter. Similarly the exact colors probably don’t matter. If they do, you could include them in your request.
• Generate an image. When you ask for an image from a general description, there is a pretty wide range of what meets your needs.
• Writing documents. A document generated by using AI will have variation from one request to another. As long as it meets your requirements, it doesn’t need to be exactly the same. However, that does mean that every time AI updates the document it could change unexpected things.

Accomodating Accurate-ish

So, if AI isn’t deterministic, how do I trust what it tells me? The short version is that we don’t entirely.

We need to have processes in place that verify AI output to ensure accuracy.

For simple coding, I can use AI to generate a script faster than I can write it manually. But, I can’t take an AI generated script and run it in a customer environment. I need to manually review the code to make sure it looks like what I want. Then, I test it in an isolated environment (just like I would for code that I write). Finally, when I have verified the functionality, I can use it to get some work done.

Manual review works fine for me, but doesn’t scale up well. What if I’m writing something big and complex and want to move quickly?

For writing something larger ideally, you’ll automate the testing process. When you do that, you can potentially skip the manual code review. This lets you scale up and work faster.

You can also have one AI write code and have another AI review the code. That gives a different viewpoint and might find things that your automated testing doesn’t like security flaws.

For documents, you could also have one AI review the document created by another AI. But, in most cases, you’ll probably need a manual review. There have been some high profile instances of major consulting firms (who are very AI-first) submitting reports with critical inaccuracies. Similarly, there have been instances of legal documents submitted to court that reference precedents that don’t exist.

In dedicated AI systems that perform tasks like automating invoice data entry, they can assign a probablilty to accurate recognition of the data. For example, the AI thinks there’s a 95% chance it correctly identified the data from the incoming PDF. In such a system, you can assign a probability level where the document is set aside for manual review and approval.