k3n_dvemoji for TYPO3: Emoji Support & UTF-8mb4 Migration

Emojis turning into question marks are not a display problem but a database misconfiguration

When an editor enters an emoji in the TYPO3 backend and the frontend shows a question mark or an empty box instead, the problem lies neither in the browser nor in TYPO3. It lies in the MySQL or MariaDB database, which is configured with the utf8 character set instead of utf8mb4. The difference: utf8 stores at most 3 bytes per character, while emojis need 4. k3n_dvemoji is a TYPO3 extension for this problem that is published neither in the TER nor on Packagist or GitHub, so how exactly it solves it cannot be verified. Since TYPO3 v9.5, however, the installer itself sets up utf8mb4 for new installations, so a conversion is only needed for older or misconfigured installations.

TYPO3 administrators and agencies that look after older installations therefore need diagnosis and a fix in one: finding tables with the wrong character set and converting them, a process that is error-prone when done by hand.

Typical use cases involve older installations, migrations and international content

The most common scenario is a TYPO3 v8 or v9 installation that has been running for years and whose database is still configured for utf8 rather than utf8mb4. An estimated 30% of all TYPO3 installations set up before 2018 have this problem, because the installer used utf8 as its default back then. As long as editors only use Latin characters, nobody notices. As soon as a social media post with an emoji is embedded on the website, or an editor puts an emoji in a headline, the output breaks. The fix is to convert the affected tables and columns to utf8mb4.

The second scenario is moving a database between servers. When a TYPO3 database is migrated from an old server to a new one, a faulty mysqldump command can lose the character set. A dump with —default-character-set=utf8 instead of utf8mb4 truncates every 4-byte character. The character set and the import then have to be corrected afterwards.

The third scenario is international websites in Asian languages. Common Chinese, Japanese and Korean characters take three bytes in UTF-8, but rare and historical ideographs from CJK Extension B onwards take four. Without utf8mb4, those characters cannot be stored and come out garbled. Mathematical symbols, musical notation and various special characters from the Unicode range above U+FFFF are affected as well, which matters for university websites and academic portals.

The technical architecture analyzes and converts database tables at column level

A character set migration works on two levels: analysis and conversion. The analysis scans all TYPO3 tables and checks the character set of every column. The result is a report showing which tables and columns are still on utf8 (3 bytes) instead of utf8mb4 (4 bytes). A typical TYPO3 system has 100 to 300 tables, and often 20 to 50 of them use the wrong character set.

The conversion produces ALTER TABLE statements for every affected column. That sounds trivial but has pitfalls: indexes on varchar columns in utf8mb4 can exceed the maximum MySQL index length (767 bytes with InnoDB in the older Antelope file format). That is why these limits have to be checked in advance and the index definitions adjusted accordingly.

In addition, the TYPO3 configuration has to declare utf8mb4 as the character set, up to v11 in LocalConfiguration.php and from v12 in config/system/settings.php: ['DB']['Connections']['Default']['charset'] = 'utf8mb4' and ['DB']['Connections']['Default']['defaultTableOptions']['collation'] = 'utf8mb4_unicode_ci'. Older configurations still use the key ['tableoptions']['collate'], which TYPO3 v13.4 deprecates. Without this configuration, TYPO3 keeps creating new tables in utf8.

Common problems are index length limits, missing backups and inconsistent columns

Problem one: the MySQL index length limit. With InnoDB in the older COMPACT row format (the default up to MySQL 5.6), the maximum index length is 767 bytes. A varchar(255) in utf8mb4 needs 1,020 bytes of index space. The conversion fails with “Specified key was too long”. The fix: switch ROW_FORMAT to DYNAMIC (the default from MySQL 5.7) or reduce the varchar length of the affected columns.

Problem two: no backup before the conversion. An ALTER TABLE on large tables (100,000+ rows) locks the table and can cause outages on production databases. The fix: run the conversion outside business hours and take a full backup beforehand. For large tables, Percona’s pt-online-schema-change enables a lock-free conversion.

Problem three: inconsistent columns after a partial conversion. If only parts of the database were converted, JOINs between utf8 and utf8mb4 columns can cause performance problems, because MySQL has to perform an implicit character set conversion. Gosign checks every table for consistency after each conversion.

From TYPO3 v9.5 on, the extension is redundant if the database is configured correctly

Since TYPO3 v9.5, the installer sets up utf8mb4 as the character set for new installations on MySQL. A fresh installation of v9.5 or later therefore configures the database correctly out of the box, and k3n_dvemoji is not needed. A conversion is still needed for installations that are migrated from v7, v8 or v9 to v12 or v13 and whose database has never been converted. Gosign takes care of this automatically in every TYPO3 migration: an AI-assisted database scan, a complete conversion and a consistency check are part of the standard upgrade process. The scan analyzes all tables and columns in under 5 minutes, generates the required ALTER TABLE statements and runs them only after a full backup, so the previous state can be restored if anything fails. The result is a database with consistent utf8mb4 support throughout, and no extension is needed in production.