Blank messages in an iPhone text export: what causes the gaps
A conversation that exports with a scattering of blank messages in it, each with its time and its sender but no words, may not be a conversation with empty messages in it at all. It can be a reading problem, and one known cause sits in the database itself: from iOS 16, the text of a message can be missing from the column where a tool reading it the old way looks for it.
Where did the text of the message go?
Into a second column. The Messages database in an iPhone backup, sms.db, has a message table
with a text column and an attributedBody column. From iOS 16 the text column can be empty
while the words are in attributedBody; Magnet Forensics’ notes on iOS
16 show it
for edited messages, whose text is null while the blob holds the latest wording. The blob is
stored in typedstream, the format of Apple’s older NSArchiver class. It is not a property list,
so a plist reader will not open it, but readers for typedstream itself exist:
pytypedstream in Python and
crabstep in Rust are two of them.
A program that reads text and stops returns those rows without their words. It does not fail on
them, so nothing warns, and the export still has every row in it. That is why the symptom is
scattered gaps rather than a missing conversation.
Why are only some messages blank and not all of them?
Because it happens row by row. The text column is filled on some messages and empty on others in
the same thread, so a tool that reads only text gets part of a conversation right and misses the
rest. ChatExport reads in this order: text when the column is not NULL, otherwise the words it
extracts from attributedBody, and otherwise nothing, which is also what a message that carried
only a photo looks like, because it had no words to begin with.
Is this the same thing as a missing message?
No, and the difference matters if the document is going anywhere official. A missing message is absent from the file. A blank message is present, with its timestamp, its sender and its position in the thread intact, and only its words gone. When the words are in the blob, the second one is fixed by exporting again with a tool that reads it. The first one is a question about the backup rather than about the export, and which messages a computer backup actually contains covers it.
How do you prove this is your cause, and fix it?
By counting the same kind of row twice: once in your export of one conversation, and once in that
conversation’s database, among the rows whose text is NULL while attributedBody is not. Then
compare the two numbers with the rule under step 3. The steps below do both counts, then the fix.
1. Count the blank messages in the export you already have
Count only the rows that show a time and a sender and nothing else: no words, no attachment, no photo. Skip reaction lines as well. The query in step 3 leaves out the same kinds of row, so the two numbers measure one thing. In a CSV, filter the column that holds the message text for empty cells, then drop the rows whose attachment column is not empty. In a PDF or an HTML file an empty row has nothing to search for, so scroll through the same conversation and count them by eye.
2. Find sms.db in the backup and copy it, with its WAL file
Inside a backup, files are stored under hashed names, and the Messages database is the file named
3d0d7e5fb2ce288813306e4d4636395e047a3d28, with no extension. In a current backup it sits in a
subfolder named 3d, after the first two characters of its name. To reach the backup folder, open
the Apple Devices app with the phone connected, go to the General tab, click Manage Backups,
right-click the backup and choose Show in Explorer; then search that folder for the name.
Since iOS 17.4 a backup can also hold the database’s write-ahead
log, sms.db-wal, which is stored
under its own hashed name, cd47480f213dba9bc38ee792775d17e3f5a73a59. SQLite only reads that log
when it sits beside the database under the same name plus
-wal, so copy both files into an empty folder of their own and
rename the copies sms.db and sms.db-wal. Work only on those copies, so nothing you do can touch
the backup itself.
If the backup is encrypted, these files are encrypted with the backup password, and a SQLite tool
cannot read them as they are. The check then needs an unencrypted backup or a decrypted copy of the
file. One route to the copy is iOSbackup, a Python
library whose getFileDecryptedCopy() method writes a decrypted copy of one file from the backup,
given the password.
3. Count the rows whose words are in the blob
Open the copy read-only with the SQLite command-line tool, for
example sqlite3 -readonly sms.db run inside that folder. First find the conversation by its phone
number, address or group name:
SELECT ROWID, chat_identifier, display_name FROM chat;
Then put that conversation’s ROWID in place of 42 here:
SELECT COUNT(*)
FROM message m
JOIN chat_message_join cmj ON cmj.message_id = m.ROWID
WHERE cmj.chat_id = 42
AND m.text IS NULL
AND m.attributedBody IS NOT NULL
AND COALESCE(m.associated_message_type, 0) = 0
AND COALESCE(m.item_type, 0) = 0
AND NOT EXISTS (
SELECT 1 FROM message_attachment_join maj WHERE maj.message_id = m.ROWID
);
It leaves out reactions, system lines and every message with an attachment, the same rows step 1 skipped. It tests for NULL and not for an empty string, because NULL is the case in which ChatExport reads the blob. Then read the two numbers this way:
- The query returns at least as many rows as you counted. Every blank row you counted can be explained by words sitting in the blob, and exporting again with the same tool will not change the result.
- It returns fewer, but more than zero. This is part of the cause. The rest is something else, and the next section names two candidates.
- It returns zero. This is not your cause.
4. Export the same conversation with ChatExport or another tool that reads the blob
The fix is on the reading side. The backup on your disk already holds the words, so nothing needs to be taken from the phone again, which is worth knowing before anybody deletes and repeats a backup in the hope that it helps.
5. Check the gaps are gone in the same places
Compare the same stretch of the conversation, not the total length.
What else leaves a row without words?
Two kinds of message change their row in a way a reader can mistake for this problem. Both are described by Magnet Forensics for iOS 16.
What happens to a message that was unsent?
Its row stays, but both text and attributedBody are emptied to NULL, so the words are no longer in
that row and exporting again cannot bring them back. The query in step 3 does not count it, because its
blob is empty. ChatExport prints “message unsent by sender” in place of the words when the row
carries the unsent marker it reads: the date_retracted column, or, with Show edit history
switched on, a marker inside message_summary_info. A row with no words, no attachment and no
unsent, deleted or edit marker is left out of a ChatExport export rather than shown blank.
What happens to a message that was edited?
Its text is NULL and its current wording is in attributedBody, so it is one case of the cause
above and the query in step 3 counts it. The earlier wordings are in another blob,
message_summary_info. ChatExport shows the current wording, marks it “(edited)” when the row
carries an edit date or earlier wordings, and writes out those earlier wordings only when Show
edit history is switched on.
What can a tool still get wrong here?
It can take the wrong words out of a blob, or none. How often depends on how the tool reads the format. ChatExport does not implement a full typedstream decoder: it looks for the message string right after a fixed marker in the blob, and when the bytes there do not hold a readable string it falls back to the longest readable run of characters in the whole blob. So on an unusual blob our export can show a fragment that is not the message. When even the fallback finds nothing, the message has no words, and if it has no attachment either, it is left out of the export.
One sign of a wrong decode is worth knowing whatever tool made the export. U+FFFD, the replacement character, is the one Unicode defines to replace “an incoming character whose value is unknown or unrepresentable”, so finding it in a message is a sign that something read bytes it could not decode and kept the result. ChatExport does not print it from a blob: the first read discards a string that contains it and moves to the fallback, and the fallback treats it as a break between runs, so it never ends up in the text that is kept.
Our own notes on the format, written while building this program and published under CC BY 4.0, are at ChatExportKnowledge, our notes on Apple’s message format. For the wider question of getting a conversation off the phone in the first place, the full guide to exporting text messages from an iPhone to a computer covers the routes, what each one carries, and what the finished document says about its own source.