What Happens to Hand-Crafted Content in the Age of AI?

Since the introduction of ChatGPT in 2022, the world of content generation has changed. In every Top Dog customer engagement, we inevitably get asked, “What do you think of AI content generation and ChatGPT? Can’t we just have ChatGPT create all our content?” While we definitely see a place for ChatGPT and other large language models (LLMs), there is still a huge need for hand-crafted content. Let’s discuss why that is.

How LLMs Work

For us to understand the strengths and limitations of LLMs, we first need to be able to understand how they work, at least at a high level.

The short answer is that an LLM is basically a super-sophisticated version of the next-word-prediction that your average smartphone performs when you’re sending a text to your friend. Given a prompt supplied by the user, the LLM is asked to generate the next word. And the next. And the next. Eventually, it generates sentences, paragraphs, and thousand-word articles. It does so according to a neural network that has been trained on a large corpus of text (and now images, but let’s stick to text for this section).

Yes, it’s much more sophisticated than what you phone does, much more, but it’s the same idea. The neural network is of a special configuration, called a “transformer network,” that allows it to maintain a context for the text generation. In other words, it maintains a bit of a memory, and the memory grows the more you interact with it in a session. The transformer network will use both the user’s prompts as well as its own answers to help it generate the next round of text.

Researchers are discovering that the larger the network and the larger the set of training data, the more sophisticated the AI responses and the more it seems to “understand” (though we’ll talk about the limitations of “understanding” in just a bit). Open AI, the company behind ChatGPT, has released a series of LLMs, starting with GPT-1 in 2018. The network in GPT-1 contained 117 million parameters. Subsequent GPT releases increased the size of the network by several orders of magnitude.

• GPT1: 117 million parameters
• GPT2: 1.5 billion parameters
• GPT3.5: 175 billion parameters
• GPT4: Rumored to be 1.76 trillion parameters, though Open AI has kept this information private. This number, 1.76 trillion, would be an order of magnitude increase from GPT3.5, which in turn was two orders of magnitude increase from GPT2.

In addition to a gigantic network size, the best LLMs are trained on a similarly gigantic corpus of text. Effectively, these models ingest nearly every web page, book, and Wikipedia article they can get their hands on. Consequently, they are able to generate text on a wide variety of subjects.

Strengths of LLMs

LLMs can generate cogent text on many subjects, according to the training that they have been given. This makes them quite general purpose and applicable to a variety of human knowledge domains.

With respect to content marketing, LLMs can quickly describe or summarize existing industries that are well documented in the public sources of information that LLMs have been trained on.

This makes them ideal for low-value, “filler” content that is needed to round out a full content marketing strategy.

LLMs also:

1. Generate lots of text quickly. Where a 1000-word blog post might take a human a few hours to research and write, an LLM can generate it nearly instantaneously.
2. LLMs generate text with good spelling, grammar, and punctuation. They make fewer basic mistakes than humans. That said, they are not perfect, and again, they can be influenced by the spelling, grammar, and punctuation of their training corpus.

Limitations of LLMs

While LLMs are truly revolutionary and will impact business and society in profound ways, it’s easy to be fooled by their achievements into believing that they really “think” and “understand.” Indeed, there is a debate raging among AI scientists as to whether AIs really “understand” things, and if so, to what level. Most would agree that LLMs “understand” what they are writing at some level. If they did not, they would produce gibberish from one sentence to the next. Clearly, they “understand” in enough ways that matter for responding to basic prompts in the real world. For many applications, that might be enough. But they frequently make mistakes or “hallucinate” (yes, that term is used among the AI researchers studying the phenomenon). Google’s Bard includes this disclaimer at the bottom of its screen: “Bard may display inaccurate info, including about people, so double-check its responses.”

Gary Marcus, a scientist and overall critic of generative AI, writes a blog that highlights many limitations of LLMs and AI systems. Gary argues that LLMs actually don’t have a very deep understanding of anything, which is why they make the mistakes they do.

Gary provides evidence of the limitations frequently in his blog. A recent post highlighted an AI-generated image that was posted on Twitter/X. It contains errors that would not be there if the AI truly understood what it was doing. While this example is graphical and not textual, it makes the point quite succinctly, proving that a picture is truly worth a thousand words.

Marcus says that he’s identified at least 10 problems with the image. Some obvious ones include the horn through the man’s head and the six fingers on his left hand. Like Marcus, I’ll let you search for the others.

In addition to “hallucinations,” LLMs have other issues, some of which are specific to the use of LLMs in content marketing and others which are more general.

1. LLM’s have very coarse settings for tone. Google’s Bard, for instance, has “More Casual” and “More Professional” settings. But do LLMs understand humor, for instance, and do they understand when using it might be in bad taste? While a human can ask for content written in a particular style or even from a particular author, LLMs often miss the mark in their responses.
2. LLMs aren’t privy to confidential information. LLMs are trained on publicly available data. If you’re a stealth-mode startup, or creating a new product category with a revolutionary entry product, LLMs won’t know anything about those.
3. LLMs don’t know anything about your buyer personas or how to write for them. The phrase “know your audience” is critically important for content marketers. Ideally, we’re writing content that targets a specific set of buyer personas. How technical is your buyer? What do buyers in this market segment expect of their suppliers? These are things the LLM will struggle with.
4. LLMs don’t know anything about your buyer’s journey, sales process, and funnel stages. As with buyer personas, as content marketers, we’re ideally targeting specific content to help a prospective buyer take action at a specific place in the buyer’s journey. What you write for a buyer who is early in the journey and just gathering high level information is different than what you write to help a buyer take the final purchase decision.
5. As with your own products, LLMs don’t know the capabilities of your competitors products. They specifically don’t know how to position your product against those other products.
6. Sometimes, LLMs generate text that is a long, direct copy of text in their own training documents, almost as if they were reciting a memorized passage. In December, 2023, for instance, the New York Times sued Open AI and Microsoft over copyright issues. Marcus covers some of the issues and shows examples in another must-read blog. This happens with both text and images. So, imagine an industry news site suing you when your LLM-generated text includes verbatim copies of content from their articles.

To round out this section about the limitations of LLMs, I also thought it would be an interesting exercise to ask an LLM about the limitations of using LLMs for content marketing. I prompted Google’s Bard with: “What limitations do LLMs have for doing content marketing?”

Bard responded with:

While LLMs (Large Language Models) offer exciting possibilities for content marketing, they also have some limitations that need to be considered:

Lack of understanding and expertise:

• Limited knowledge of specific industries: LLMs are trained on vast amounts of text data, but they may not have deep understanding of specific industries or domains. This can lead to content that lacks nuance and relevance. * Difficulties with factual accuracy: LLMs can sometimes generate text that is factually incorrect or misleading, especially on complex topics. This requires careful fact-checking and human oversight.

Creativity and originality:

• Formulaic and repetitive content: LLMs can struggle to generate truly original and creative content. They often rely on patterns and templates learned from their training data, which can lead to repetitive or generic outputs. * Difficulty with humor and emotional engagement: LLMs might struggle to understand and generate content that is humorous or emotionally engaging, which are crucial elements for effective marketing.

Technical limitations and biases:

• High computational cost: Training and running LLMs can be expensive, especially for large-scale content creation. * Potential for bias: LLMs can inherit biases from the data they are trained on, leading to discriminatory or offensive outputs. Careful selection and filtering of training data is crucial. * Limited understanding of user intent and context: LLMs may not fully grasp the specific needs and preferences of the target audience, leading to content that misses the mark.

Overall, LLMs are a powerful tool for content marketing, but they should not be seen as a replacement for human creativity and expertise.

Here are some key takeaways:

• Use LLMs for specific tasks: They excel at generating different text formats, summarizing information, and providing basic outlines.
• Maintain human oversight: Fact-check, edit, and add the human touch to ensure quality and relevance.
• Focus on specific needs: Define your target audience, content goals, and brand voice clearly to guide the LLM’s output.
• Combine with other tools: Use LLMs alongside analytics, design tools, and SEO optimization for a well-rounded approach.

By understanding the limitations and leveraging the strengths of LLMs, you can create effective content marketing strategies that enhance your brand communication.

First, it’s not bad. It’s an interesting response in that it cites some of the same limitations that I listed, above. But some of my limitations are not listed. Does Bard really understand how content marketing works? Does it understand the purpose of content marketing and does it actually write with strategic intent to influence outcomes with customers? That seems a bit of a stretch.

Where to Use AI-Generated and Hand-Crafted Content

So, after internalizing all the strengths and limitations of LLMs, what next? Does that mean that AI-generated content has no place in content marketing? No, not at all. But we need to be respectful of LLM limitations and ensure that we’re leveraging the technology where we we can deal with those limitations.

For example, if you needed a basic market segment “explainer” piece, LLMs could probably do a reasonable job. The ideas that underpin the market segment would be in public content around the web and the LLM would have ready access to those. The piece doesn’t really target a very specific buyer persona at a specific location in the buyer’s journey. In this case, the combination of an LLM generating a first draft and a human editor revising that draft and checking the content to screen out any “hallucinations” would probably result in a solid piece on a quick time scale, for a low cost.

On the other hand, if you’re looking for a specific piece that is going to help your company simplify a complex technology, target a specific buyer and the end of the buyer’s journey, and try to influence them to move forward with a purchase, the LLM-generated text will not be of much help. Further, if the product in question has not been released and is quite revolutionary, the LLM won’t have been trained on it and won’t be able to add much.

Given all this, a simple rule of thumb might be: First, focus on human-generated content to create targeted pieces that help your buyers understand your unique value proposition and differentiation. These are the pieces that will move your sale forward, and they should be hand-crafted for this outcome. Then, use AI/LLM-generated content to round out your content library. But don’t expect the AI/LLM-generated content to perform well or generate many referrals or likes. AI/LLM-generated content will typically be flat and dull.

Finally, note that a large document might use both AI and hand-crafted content. If the document includes a few paragraphs on general market topics, for instance, those would be appropriate to generate via an AI. But you should use hand-crafted content for those sections that are specifically trying to move the buyer forward. Blend the two in the right ratios according to your goals.

Conclusion

There is a place for both AI/LLM-generated content and hand-crafted content as a part of your content-marketing strategy. First, focus your strategy on hand-crafted content that moves your sales forward. Then, fill in gaps with generic AI/LLM-generated content.

Top Dog Content Marketing is skilled at working with our customers to find the right balance between AI/LLM-generated and hand-crafted content. We look forward to discussing this with you.