Hidden characters in copied text: how to find and remove them

Sometimes two pieces of text look identical, but a program says they are different. A password that you typed correctly is rejected. A search for a word finds nothing, even though it is plainly on the page. A line of code that looks perfect throws an error. In many of these cases the cause is a character you cannot see. The free Invisible Character Cleaner finds hidden Unicode characters in pasted text, lists them, and gives you a cleaned version. This guide explains what they are, where they come from, what trouble they cause and how to deal with them.

What invisible characters are

Unicode, the standard that lets computers handle text from every language, includes characters that take up no visible space or look like something else. They exist for good reasons: controlling how text joins, which direction it runs, or where a line may break. The trouble starts when they end up in places where they are not expected.

The most common ones are:

Character	Code point	What it does
Zero-width space	U+200B	Marks a place where a line may break, with no visible width
Zero-width non-joiner	U+200C	Stops two letters from joining in scripts that join letters
Zero-width joiner	U+200D	Asks two characters to join, used in some scripts and in emoji sequences
Byte-order mark, or zero-width no-break space	U+FEFF	Appears at the start of some files to mark the encoding
Word joiner	U+2060	Prevents a line break at its position
Soft hyphen	U+00AD	Shows a hyphen only if the word has to be split across lines
Left-to-right mark, right-to-left mark	U+200E, U+200F	Set the direction of surrounding text
Non-breaking space	U+00A0	Looks like a normal space but never allows a line break

The non-breaking space deserves a special mention, because it is not invisible in the way the others are. It looks exactly like an ordinary space. But to a program it is a different character, and a b with a non-breaking space is not equal to a b with a normal space.

You can see the effect in a few lines of code

Take the word helloworld and put a zero-width space between the two halves. On screen it reads hello​world, with no visible gap. In Python:

text = "hello​world"
print(len(text))              # 11
print(text == "helloworld")   # False
print([hex(ord(c)) for c in text if ord(c) > 127])   # ['0x200b']

The string has 11 characters instead of 10, it is not equal to helloworld, and the hidden character is U+200B. A non-breaking space behaves the same way:

print("a b" == "a b")    # False

That is exactly how an invisible character breaks a login form, a database lookup or a comparison in your code.

Where they come from

• Copying from web pages. Websites sometimes insert zero-width characters or soft hyphens to control line breaks or to mark text.
• Word processors and chat apps. Rich-text editors and messengers may add non-breaking spaces, directional marks or other formatting characters when you paste.
• PDF and e-book text. Text extracted from a PDF often contains soft hyphens, odd spaces and broken line endings.
• Files saved with a byte-order mark. Some editors write U+FEFF at the very start of a file. Programs that do not expect it may treat it as part of the first word or the first column name.
• Right-to-left text. Mixed English and Arabic or Hebrew text may include direction marks so that punctuation appears in the right place.
• Deliberate marking. Some people use invisible characters to fingerprint text, so that a copy can be traced. This is one reason to clean text you did not write.

What goes wrong

Code that will not run. A zero-width space or a non-breaking space in a source file can cause a syntax error that points to a line which looks fine. If you copy code from an article or a chat and it fails mysteriously, hidden characters are a prime suspect.

Forms and logins. A password or username with a hidden character at the end will not match what the server stored. Pasting credentials from a document is a common way to introduce one.

Searching and sorting. Searching a document for a word fails if the word contains a hidden character. In a spreadsheet, two cells that look the same may refuse to match in a lookup, or a list may sort strangely.

Database imports. A CSV file whose first column name begins with a byte-order mark has a column name that does not match the one you expect. The data is fine, but the name carries an invisible prefix.

Duplicate detection. Two identical-looking entries are treated as different because one contains a hidden character.

Layout. A non-breaking space can stop a line from wrapping where you want, producing a long line that overflows its box on a small screen.

How to use the Invisible Character Cleaner

1. Open the Invisible Character Cleaner.
2. Paste the text into the input box.
3. The tool lists every hidden character it finds and where it is.
4. Copy the cleaned text from the output box. Hidden characters are removed, and unusual spaces are replaced with a normal space.

Because the tool reports each character it found, you also learn what was in your text. That is useful: if there are twenty zero-width spaces in a paragraph, they were probably put there deliberately and you may want to know where the text came from.

Cleaning with care

Not every invisible character is junk. Removing them blindly can change the meaning of some text.

• Languages that rely on joiners. In some scripts, the zero-width non-joiner and joiner control how letters connect. Removing them can change how a word is displayed. If you work with such languages, check the result.
• Emoji sequences. Many emoji, such as ones with skin-tone modifiers or family groups, are built from several characters joined with a zero-width joiner. Removing the joiner splits them into separate pictures.
• Right-to-left text. Directional marks may be needed to display mixed text correctly.

For plain English text, code, passwords and data fields, cleaning is almost always safe and helpful. For text in other languages, scan the output.

Prevention

Paste as plain text whenever you can. Most editors have a "paste without formatting" shortcut, and it removes a lot of rich-text baggage. Configure your code editor to show invisible characters, or to highlight unusual ones; many have an option for it. Use a linter or a formatter in projects, since many report unexpected characters. When you accept data from users, normalise it by trimming spaces and, where it makes sense, removing zero-width characters on the server. In PHP, the multibyte strings lesson shows how to work with Unicode text safely, and the regular expressions lesson covers patterns for stripping characters.

Related tools

If two versions of text should be identical, compare them with the Text Diff Checker, which highlights differences by line or by word. Count characters and words with the Word Counter; a count that is higher than you expect can be another sign of hidden characters. To change the capitalisation or naming style of cleaned text, use the Case Converter.

Quick answers

Can I see these characters in my editor? Some editors can show them if you turn on a "render whitespace" or "show invisibles" option. Others show nothing, which is why a cleaning tool helps.

Is a normal space invisible too? A normal space is the ordinary character U+0020 and is expected almost everywhere. The tool replaces unusual spaces with it.

Will cleaning change my visible text? For ordinary text, no. You will see no difference, only fewer surprises when a program reads it.

Does it work on code? Yes. Paste the code, clean it, and copy it back. Re-check the result in your editor.