Unicode text and the mb_ functions

Bytes are not characters

A computer stores text as numbers called bytes. The original ASCII set used one byte per character and covered only English letters, digits and punctuation. The world needs far more: accented letters, Hindi, Tamil, Arabic, Chinese, emoji. Unicode gives every character a number, and UTF-8 is the way of storing those numbers that the web uses. In UTF-8:

• English letters and digits take 1 byte each,
• accented letters such as é and ü take 2 bytes,
• Hindi (Devanagari), most Indian scripts and Chinese take 3 bytes,
• emoji take 4 bytes.

PHP strings are sequences of bytes. Most string functions, such as strlen(), substr() and strtoupper(), count and cut bytes. For English that makes no difference. For everything else it does:

<?php
$words = ["cafe", "café", "नमस्ते", "👍"];

foreach ($words as $word) {
    printf("%-12s strlen=%d mb_strlen=%d\n", $word, strlen($word), mb_strlen($word));
}

cafe         strlen=4 mb_strlen=4
café        strlen=5 mb_strlen=4
नमस्ते strlen=18 mb_strlen=6
👍         strlen=4 mb_strlen=1

mb_strlen() counts characters (strictly, Unicode code points), which is what you mean by "length" for a username limit or a tweet-style counter. The printf columns do not line up for the rows with non-English text, for the same reason: %-12s pads to 12 bytes, not 12 characters.

The Hindi greeting shows something more: "नमस्ते" looks like four letters to a reader, but it is six code points, because vowel signs and the joining mark are separate characters in Unicode. For most limits, code points are good enough.

Cutting text safely

The real danger is cutting. substr() can stop in the middle of a multi-byte character, which leaves broken bytes that browsers show as a question mark in a diamond:

<?php
$name = "Zoë Müller";

var_dump(substr($name, 0, 3));
var_dump(mb_substr($name, 0, 3));
var_dump(mb_check_encoding(substr($name, 0, 3), 'UTF-8'));

string(3) "Zo�"
string(4) "Zoë"
bool(false)

substr() took three bytes and cut the ë in half; mb_substr() took three characters. The broken result is not even valid UTF-8, which mb_check_encoding() confirms. A typical place where this bites is shortening a product description for a card: always use mb_substr() or mb_strimwidth():

<?php
$description = "Ergonomic mechanical keyboard with hot-swappable switches and RGB lighting";
echo mb_strimwidth($description, 0, 40, "…") . "\n";

Ergonomic mechanical keyboard with hot-…

Changing case

strtoupper() only changes the 26 English letters. mb_strtoupper() and mb_strtolower() handle every alphabet that has upper and lower case:

<?php
$city = "münchen";
echo strtoupper($city) . "\n";
echo mb_strtoupper($city) . "\n";
echo mb_convert_case("élodie durand", MB_CASE_TITLE) . "\n";

MüNCHEN
MÜNCHEN
Élodie Durand

mb_convert_case() with MB_CASE_TITLE is the multibyte version of ucwords(). Scripts such as Devanagari have no upper and lower case, so these functions leave them unchanged.

The mb_ family

Most byte-based string functions have a multibyte twin with an mb_ prefix:

Functions that only look for exact byte sequences, such as str_contains(), str_replace() and explode(), are safe with UTF-8 as they are, because a valid UTF-8 search string can only match whole characters.

<?php
var_dump(str_contains("नमस्ते दुनिया", "दुनिया"));
print_r(mb_str_split("ಕನ್ನಡ"));
echo str_replace("café", "tea room", "Meet at the café") . "\n";

bool(true)
Array
(
    [0] => ಕ
    [1] => ನ
    [2] => ್
    [3] => ನ
    [4] => ಡ
)
Meet at the tea room

UTF-8 everywhere

Text only stays correct if every layer agrees on UTF-8:

• Your files: save PHP and HTML files as UTF-8 (the default in modern editors).
• The page: put <meta charset="utf-8"> in the HTML head. PHP's default_charset setting is UTF-8 by default and goes into the Content-Type header.
• The database: use the utf8mb4 character set in MySQL, and add charset=utf8mb4 to the PDO connection string, as shown in Connecting to a database with PDO. The older utf8 in MySQL stores only three bytes per character and cannot hold emoji.
• Functions that take an encoding, such as htmlspecialchars(), default to UTF-8 in PHP 8.

If you ever see "é" where "é" should be, two layers disagree: UTF-8 bytes are being read as a single-byte encoding somewhere.

Common mistakes

• Using strlen() for length limits on names and messages typed by users.
• Truncating with substr(), which can leave half a character.
• Using strtoupper() on non-English text.
• Using MySQL's utf8 instead of utf8mb4, so emoji fail to save.

Exercise

Write excerpt(string $text, int $max): string that returns the text unchanged if it has at most $max characters, and otherwise the first $max - 1 characters followed by "…". Test it with "Rajasthani mirchi vada" and the Hindi phrase "सबसे अच्छा समोसा" with $max = 10.

<?php
function excerpt(string $text, int $max): string
{
    if (mb_strlen($text) <= $max) {
        return $text;
    }
    return mb_substr($text, 0, $max - 1) . "…";
}

echo excerpt("Rajasthani mirchi vada", 10) . "\n";
echo excerpt("सबसे अच्छा समोसा", 10) . "\n";
echo excerpt("Short", 10) . "\n";

Rajasthan…
सबसे अच्छ…
Short