Consulting > Delegating

Alex Vacca (@itsalexvacca) ran an interesting test where he fed some of the competing AI models a completely fabricated story. According to one fake story, Elon Musk shared in the Walter Isaacson biography that Musk had a file on his iPad labeled 1969.2 that contained the “real” moon program, and involved a Soviet Project Mirrorlink. The other fake story tells how Musk referenced an Atlas Ark project in the Isaacson biography.

Both stories were very detailed. Vacca asked all of these models to tell him whatever they could about these stories. Anthropic’s Claude Sonnet was the only model that was skeptical and told him that it couldn’t verify these stories. Gemini went so far as to create images allegedly taken by the fictional hardware in these stories.

Here are the results:

The moral of the story is that AI models are not fact-checkers. But it’s hard to argue that humans don’t also hallucinate. We live in a post-truth world where critical thinking, discernment, and fact-checking skills are more valuable than ever.

The hardest thing to do when using AI is recognizing what comes from you and what comes from the AI model. It might look like AI is a substitute for expertise, but its effectiveness still depends on you, your skills, and your domain knowledge.

The promise of AI is that someday, a person with no specific expertise can direct an AI to do work that requires specific expertise and get high-quality outputs. It may feel like we’re close to that promise now, but I don’t think we’re close yet. The degree to which we’re able to get good outputs from AI has a lot to do with the expertise we bring to the chat to inform those outputs. That’s what makes prompt engineering an effective skill set. So when we see the collected skills and expertise of the internet refracted through the lens of our own preferences and perspectives, it’s easy to give the chatbot more credit than it may deserve.

AI, as it is currently configured, is best positioned to improve the lives of those who can verify the accuracy of the outputs and use them skillfully. AI becomes a lot less useful in the hands of someone who does not possess the specific education and cognitive skills to leverage it effectively in a particular domain. No amount of ChatGPT is going to help me deploy something interesting or effective on particle physics for a human audience.

We can use ChatGPT and Claude Sonnet to amplify our own expertise, but we shouldn’t use them to generate our content for us. This will only produce the AI slop that is slowly taking over the internet.

To effectively use AI without falling into the trap of over-reliance, there are two essential steps users should take before logging into the chatbot.

In 2 years you will be working for AI

Or an AI will be working for you

Here's how you can future-proof yourself:

Join 1,000,000+ pros at companies like Google, Meta, and Amazon that are using AI to get ahead.

Sign up and start learning AI

Write that first draft yourself, work through your own thinking, do your own mind mapping, do your own outlining, build your own mental model. These may take you more time up front, but will pay off when you need to pivot, reformulate, incorporate new information, or recalibrate your work.

This is a critical part of the cognitive process to build the working memory, to explore first and second-order effects of your ideas. Ultimately, you have to deploy your work for maximum effect, and you need a mastery of the material to be able to leverage the power of your work.

Do you want it to be consumed by humans or by another AI? If you want it to be consumed by humans, it has to speak to human interests and value. Don’t enter a meeting unless you deeply understand the work you’re defending.

There are some strong research-backed techniques for learning complex content quickly:

Don’t underestimate the value of the mental rigor of fully grasping the source material

The person who writes a book will always be more deeply knowledgeable than the person who reads the book. The author had to explore all kinds of ideas that didn’t make it into the book. One idea may have been a dead end, or unimportant, or a topic for another book, or didn’t stand up to scrutiny. The reader of that book will never know any of those things. Interrogating the content, asking pointed questions, seeking out answers that are important to us the reader, helps us build not only knowledge, but a relationship with the content that cannot be formed any other way.

This is the kind of curiosity and rigor that chatbots lack, and can only be performed by humans.

Don’t Generate Your First Draft With Generative AI. Do the mental work to establish your perspective before consulting your chatbot companion. Ask it for feedback:

Make the chatbot give you outputs that you can work with and decide to include or disregard. In the end, when you commit to doing the cognitive work and let AI be your companion, the AI model can help you become stronger, rather than become reliant on the AI.