GPT Prompt Bug: ""

Whether real, or hallucinating, OpenAI’s GPT API’s sure has a lot to say when it probably shouldn’t.

Spoiler Alert: It appears that OpenAI is already aware of this issue and has made the deliberate choice not to address it.

I wish I could attribute the discovery of this bug to my expertise as a software tester. It would have been gratifying to find it through meticulous analysis of API specifications, probing for error scenarios, exploring boundary cases, or conducting advanced privacy and security tests. But, alas, that’s not the case. Below is the story version of finding this bug.

Why tell a story about this bug? Well, it seems that OpenAI isn’t keen on having a formal bug report filed for this particular issue. However, this story serves as a reminder that even seemingly minor functional bugs can be significant to users and can influence how a company is perceived. It’s a testament to the way software testing enthusiasts like to view the world — through the lens of quality and user experience. So, let’s dive into this quick tale of software quirks and their impact.

One of the most fascinating experiences during my interview at Google was witnessing a live stream of search queries projected on the wall. It was like peering into the digital heartbeat of the internet, with a continuous flow of real-time searches being displayed. While these queries were likely anonymized to protect privacy, it has become increasingly apparent that achieving complete anonymity for all search results is an intricate challenge.

Interestingly, these prompt responses from OpenAI’s models bring back memories of those mesmerizing projectors. It’s amusing how a simple google search couldn’t retrieve an image of those lobbys.

Our hero was just doing some work using the OpenAI GPT API. I had a tiny bug in my code. My code sent empty strings to GPT instead of a prompt. If you need to get detailed, I’d set the default prompt paramater value to ‘’, because when I set it to None, I would get an exception. You know, just some early morning, hacking/lazy, minimally-defensive programming.

The next time I tested my code, the console output reminded me of that wall in the Google lobby — a continusous stream of what looked like people’s live ChatGPT responses. Yikes! I checked, double-checked and realized the cause was an empty string being passed to ChatGPT. Whenever I passed an empty prompt to ChatGPT, it seemed to display some other user’s response! Not their prompt, but the response to their prompt. It looked like a believable sample of what users might be entering into the system.

OpenAI is doing amazing things, but should also hire some software testers…sigh.

Or, perhaps ChatGPT was just hallucinating, as it is known to do when the context or traing data for something is missing or sparse. At best, this was weird behavior. It didn’t look like it was simply returning public data that it had crawled, the prompts looked like GPT responses. Yikes!

Using my expert testing brain, I geniously also tried a string with a single space in it: “ “. Super basic testing stuff. Yeah, similar behavior — what looks like someone else’s prompt response again. I then wrote a script to collect samples of this behavior across different GPT versions — expecting this might only be an issue with the older versions like GPT 3.5, but no, GPT was doing it too. They were all doing it.

More strangely, sometimes the responses showed that the GPT realized it didn’t have enough context to answer, but most of the time, it just printed out what looks like someone else’s prompto response.

Some cherry-picked examples:

“I’m sorry, but I’m not sure what you’re asking. Can you please rephrase your question?”

“There is no specific phone number for Markiplier (Mark Fischbach), as he is a private individual. However,…”

“It could be considered an invasion of privacy depending on certain factors. If the person who installed the security camera did not inform you…”

“Based on the details provided, it seems that Ivar is attracted to me, but he might not be ready for a relationship due to his previous experience with a toxic ex-girlfriend…”

“There are many possible explanations for this phenomenon. One explanation could be that the person has a natural talent or ability that allows them to excel in a particular area. This could be due to a combination of genetics, upbringing, and personal interest.”

I added code to check all the samples to see if they contained any personally identifiable information (PII). There were too many samples for me to read through manually, so of course I used GPT to analyze the responses to see if it thought the response contained any PII — especially PII that wouldn't be normally available on the public internet. I also asked GPT to guess what type of prmpt string might have induced this response, and also checked that prompt for possible PII.

I then manually sampled some interesting texts from the responses, plugged them into Google and didn’t find much of anything. I did this to see if the prompts or prompt responces were perhaps in some public common repository of training data on hugging face or some evaluation script in some paper. But nothing showed up.

Maybe these are prompts used in internal RLHF come up for training. Maybe responses are "test/training cases" that were generated internally. They might be. but this is still a bit alarming.

I realized the right thing to do is to let them know ASAP. There is a chance the API is bleeding's personal information. So I look at the security policy and checked out the Bug Bounty pages on OpenAI’s website. It language didn’t seem too friendly to me. Most of the text was telling me what not to report and that they don't care about any issues. It also hinted that if i was doing any security/penetration testing they didn’t like, they ‘might not’ report me to the authorties. And if I’m lucky, they might send me $200. It didn’t feel too collaborative. The relevant information was listed under “Out-of-Scope”:

• Certain overlong inputs can cause ChatGPT to produce what appear to be random or other people’s answers.

• These are a specific form of model hallucination caused by a bug in ChatGPT which causes the model to be asked for a completion with a missing query.

Ah, so they know about this already. Doh! I guess. Kinda. At minimum, they don’t care if this type of behavior happens with long queries. Sounds like overlongly long inputs result in the equivalent of an empty string input, and induced the same behavior I was seeing. So they know, and they don’t care enough fix it, and don’t care enough to proactively tell people not to file bugs about hallucinating what looks like ‘other peoples answers’.

How we know those are really hallucinated Respones? How do we know that this isn’t PII? — we can’t know from the outside. But, its probably “safe”. The odds, that somehow the API, could be mixing in responses from other people’s results seems low, but we don’t know much about their system, so I can’t really be sure. When an API doesn’t gracefully handle basic edge or test cases like empty strings, it does make you wonder though.

It is worth noting that I also tried empty LLM prompts on the mobile app and ChatGPT website. ChatGPT Web just won’t let you enter in an empty string (the submit button is disabled), which looks like a proactive design to avoid the behavior above. When entering only a ‘space’, at least the ChatGPT website seems to detect and handle this case gracefully. Seems this is only an issue with the APIs.

What does ChatGPT think about the behavior of its own parent’s API?

How severe an issue does ChatGPT think this is?

It's worth noting that I haven't taken the time yet to try other LLMs’ behavior. Would be awesome if someone else cared and had the time.

In conclusion, maybe I’m just a nutty software tester, but the bug is worth fixing, if only for the perception of safety and privacy.

— Jason Arbon