ChatGPT Fails: 13 Common Mistakes and Errors You Need to Know About

I’ve written about ways to use ChatGPT for SEO and content marketing related tasks like keyword research, creating title tags, and local SEO.

I use it at least once a day for various tasks. The more you use the platform, the more useful it will be for you to understand its limitations.

The founder of OpenAI (the company that built ChatGPT) called the tool “incredibly limited” and a “horrible product” and warned users about trusting them:

ChatGPT is incredibly limited, but good enough at some things to create a deceptively large impression.

it is a mistake to rely on it for anything important right now. it is a glimpse of progress; we have a lot of work to do on robustness and veracity.

—Sam Altman (@sama) December 11, 2022

Several publications published content with errors on meaningful YMYL (“Your Money or Your Life”) topics like personal finance and health.

This article discusses the most common technical errors and common errors when using ChatGPT and how to resolve them.

Common ChatGPT Errors

If you’ve been using ChatGPT for “a long time” (i.e. since late 2022), you’ve probably seen the prompt below at some point.

Obviously, a new software product in an emerging technology area that is massively popular will encounter problems.

If you’re making ChatGPT part of your daily workflow, it helps to know the common issues you may encounter.

ChatGPT also presents issues familiar to all websites, such as 404 not found errors, 5xx internal server errors, 403 access denied errors, and Cloudflare errors such as 1020 and 524 errors.

Often these can be resolved by clearing your cache, using (or disabling) a VPN, or making sure you are entering the web address correctly.

And there are also several more platform-specific errors:

1. API issues

Often times, users may hit specific character limits, token limits, or functionality limits with the ChatGPT API.

2. Character limits

The number of characters you can enter in ChatGPT and the number of characters ChatGPT responses can contain are limited.

They vary depending on your version (for example, GPT-3.5 has an input limit of 2048 and GPT-4 is 4096) and in the web interface versus the API.

3. Connection timeout on long replies

If ChatGPT processes a long and complex response, it will sometimes time out.

You can see this when you ask the platform to perform functions like writing code or analyzing and responding to long inputs.

4. Unfinished Answers

This is not an error per se, but often if you ask ChatGPT to write some code or a long text, it will stop halfway.

You can type continue and it will exit in the next prompts in these cases.

5. Limited Posts

ChatGPT, at the time of writing, limits the number of messages you can use per period, even for paid accounts.

Similarly, there is specific rate limiting for different GPT versions, OpenAI tools, and account types.

6. Connection issues

Depending on the level of access required or the duration of the session. ChatGPT may log you out or you may not have access to certain features with certain account statuses.

Most of these problems can be solved by:

• Rework your prompts to accommodate character limits.
• Make sure you’re signed in and can access the feature you’re trying to use with your account level.
• Ask ChatGPT to regenerate responses.
• Starting a new discussion.

---

---

Common ChatGPT Errors

ChatGPT makes many mistakes, including:

• Mistaking verifiable facts.
• Detailing a process he “supposedly” used to generate an answer that was impossible given the limitations of the platform.
• Generation of broken code.
• Broken formatting or skipped statements in output.
• And more.

Other users have detailed instances of the lying platform to solve a CAPTCHA and problems with simple math related to age And distance.

Some of this has improved with GPT-4, but not all. Here is a prompt and output using the latest version of ChatGPT:

While it’s fun to do “gotcha” prompts and catch ChatGPT saying incorrect or goofy things, it’s more beneficial to be aware of the areas where ChatGPT can be wrong – the “error categories”. “, If you want.

Understanding these will make you both a better prompt engineer and a better output editor. Let’s go over the most common categories of errors that I know of.

7. Confident but Fake

The problem of “hallucinations” is an important problem for ChatGPT and other generative AI solutions.

ChatGPT will answer all sorts of questions it doesn’t know the answer to convincingly. This does not render the correct answer.

The example screenshot above shows that it does not use any qualifying language or hedging. You get an unequivocal decisive answer.

8. Not understanding the number of words and characters

ChatGPT is terrible at estimating and understanding character count accurately.

If you are doing SEO and content related work, this is a major limiting factor. You may be looking to write title tags, meta descriptions, or content that needs to meet a word count:

It’s actually quite close. I’ve had cases where the tool was much further away.

9. Errors in reasoning

I searched for “logic puzzles” and grabbed the first list of “easy logic puzzles” I found.

ChatGPT fixed the first three puzzles, then the fourth:

10. Coding errors

There are now several real-life examples of people using ChatGPT to build websites, apps, and browser extensions or fix coding issues.

That said, ChatGPT can also generate code that doesn’t work. I encountered the following issues while using ChatGPT to help code:

• The code just doesn’t work properly. (Actions not performed when users use a tool.).
• The calculations built by the ChatGPT code are incorrect.
• Using the wrong API. (Often it’s because it’s not aware of the API you want to use because the API was released or updated after the training data period for GPT, which is is over at the end of 2021. Despite this, ChatGPT does not let you know. )
• Ignoring or “forgetting” instructions and project requirements or constraints.

Again, it is imperative to always carefully check the code you deploy and test the output of anything ChatGPT helps.

Some users have also reported that GPT-4 or even legacy mode (rather than the new default mode, Turbo) performs better for coding tasks.

11. Obsolete Information

Training data for ChatGPT may be updated at some point, and various plugins and tools have started to integrate web crawling and real-time data.

However, the web version of ChatGPT currently can’t explore the web and is not aware of information dating from before the end of 2021.

This is important, especially when associated with the “confident but false” problem.

12. Short Memory

There is a limit to the number of characters in a chat that ChatGPT will “remember” so it will frequently make errors due to losing the context of your first prompts or its first responses.

13. Formatting errors

As with coding, if you ask ChatGPT to perform functions such as creating a table or formatting text a certain way, it will sometimes make errors or ignore specific instructions.

Avoiding and Fixing ChatGPT Errors

So how can you get the most out of ChatGPT while dealing with platform issues and limitations?

Focusing on these key areas can help you avoid many of the potential drawbacks of using ChatGPT in your day-to-day tasks:

Be aware of the limits

Staying up to date with specific product limitations such as (eg character limits for prompts and responses, training data date for GPT, etc.) will help you create prompts that generate good responses and better quality assurance results.

OpenAI does a good job of updating this information with new releases. Some items here are things they’ve explicitly said they’re dealing with.

Carefully review and edit outputs

This should be fairly obvious, but you should carefully edit any text and QA code before publishing or deploying it.

Because of ChatGPT’s compelling trust, I strongly recommend that you have a domain expert review the content and code before deployment.

It is relatively easy to be fooled by some content and claims made by ChatGPT, especially if a topic is not your area of ​​expertise.

But if you have deep experience on a subject, it’s easier to catch misleading or inaccurate statements.

Layer tools and processes above the API

OpenAI has deployed plugins and made available an API specific to ChatGPT.

You can solve some of the limitations of the platform by using different tools created by others or by creating your own.

Now that you know the potential pitfalls of ChatGPT, you can go ahead and establish some safeguards for using the tool – taking advantage of its benefits while avoiding many of the downsides.