Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

From the Twitter help center: "Note about DM fails: Your service provider may split SMS messages greater than 160 characters into multiple messages. In this case, the second message will post as a normal Tweet because it doesn't begin with d+username, as the first message did. Please be careful to ensure that your messages are under 160 characters."

Twitter doesn't have multipart SMS reassembly? Twitter advertises that the size limit for direct messages is now 10,000 characters.[1] But apparently you can only send big from their app, or when advertising ("sponsored tweets"), not from SMS.

Multipart SMS reassembly isn't all that hard; every smartphone does it. Messages are split into 153 byte fragments with a binary header, with a max of 255 fragments. At Twitter's scale message parts may come in to different front end machines, which means having to reassemble fragments across multiple processes/machines. Most SMS gateways don't reassemble fragments on inbound because of that scaling problem, but Nexmo and Twilio will pass through the fragment numbers so you can do it. (This is a new Twilio feature in beta. Ask Twilio to turn it on if you need it.) Outbound, SMS gateways almost always handle multipart. Have to be able to send big ads in bulk, after all.

[1] http://time.com/3918810/twitter-direct-messages-length/



While the ordering of messages is part of the protocol, it's not a required part - there's no guarantee that a phone will correctly indicate the number of parts of the message in the header(s).

Given that, I'd rather say "this feature will fail in this way" than "sometimes, depending on your phone and carrier, your long SMSs may be split into multiple parts, the second of which will posted as a public message".


For GSM, number of parts is a required item in the multipart message header.[1] It's field 5 of the UDH.

Some phones have had trouble with reassembly. There's an old Android bug which manifests as a new incoming message being reassembled with parts of an old one, sometimes an old one from months ago.[2][3] That indicates a broken reassembly implementation. It's easy to see how that can happen. The "unique ID" on multipart messages is only 8 bits (16 bits with an alternate header format). That number is generated by the calling phone. Unmatched fragments should be discarded (or delivered as an error) after a few minutes. Android keeps unmatched SMS fragments for the life of the phone, resulting in the occasional reuse of long-forgotten message fragments. Worse, when this happens, when the correct fragment comes in, it's saved as an orphan. So once there's been an orphaned fragment, 256 long messages after that, there will be another bogus reassembly using the old fragment, with the correct fragment being saved as a new orphan. Some users changed their phone number to get around this bug.

Google's reaction to years of bug reports on the problem was to close the bug report as "Obsolete" without fixing it.[3]

To work around this botch, there's an app: "SMS Multi-Part Cleaner". Really.

[1] https://en.wikipedia.org/wiki/Concatenated_SMS [2] https://code.google.com/p/android/issues/detail?id=17769 [3] https://code.google.com/p/android/issues/detail?id=28697


"For GSM, number of parts is a required item in the multipart message header.[1] It's field 5 of the UDH. "

number != order.

As said, sms's are not guaranteed to be received in a particular order, nor is the order recoverable (because the clocks are not guaranteed right, etc, so even sorting by time, ...)


Read the spec. The header on each multipart message fragment has the number of fragments, the fragment number (counted from 1, for some reason), and a semi-unique ID generated by the sending phone. You also get the source phone number along with the message. This is sufficient for correct reassembly.

The semi-unique ID is normally only one byte, and so old unmatched fragments can match new messages. Android keeps old unmatched fragments in a database for the life of the phone, which is insane. (We drop them after 5 minutes in our system. A more conservative approach would be to wait until a successful message from the same phone, indicating communications have been reestablished, and a few minutes after that, purge old fragments from the same source.)


I've read the spec. In fact, i'll go further. I've built multiple election systems that use SMS to send data around :)

Every carrier tech i've talked to tells me the same thing - the order of messages is not guaranteed nor is it recoverable.

So while it's not that i don't believe you, it's that "in practice, it seems everyone thinks and does otherwise".

"This is sufficient for correct reassembly." Except, as you prove later, it isn't when the unique ids can match other messages :)


If you have your own system which doesn't send UDH headers, and you can't reassemble, it's your problem. SMS messages are not ordered, but then can contain ordering info. The major smartphones all send UDH headers, which have enough data for reassembly. At least if you don't keep old fragments for insanely long periods.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: