Sunday, February 17, 2013

Finding and Reverse Engineering Deleted SMS Messages


Recovering deleted SMS messages from Android phones is a frequent request I get. Luckily, there are several places and ways to recover these on an Android phone.  After working a case that involved manually carving hundreds of juicy, case making messages, I collaborated with cheeky4n6monkey on a way to automate the process.  A huge thank you to Adrian, because I think the only way to truly appreciate the script is to do the manual work first.

That being said, in my last post Dude, Where's my Data I explored the importance of knowing what your automatic tools are doing and digging deeper as there may be critical information these tools are not parsing.   Harlan Carvey contributed a great comment which I think sums it up nicely: “Tools provide a layer of abstraction over the data itself, often hiding the data from the analyst who is not curious.”

I am not trying to give these tools a bad rap.  In fact, I use my "all in one" tools every day. However, by understanding the raw data, you can leverage these tools to help you find and understand critical data not automatically provided.

Recently I used Cellebrite to understand the structure of SMS messages, which I could then apply to SMS fragments found in unallocated space and the mmssms.db-journal file.  Although Cellebrite recovers deleted messages, it does not do so from areas outside of the SMS database (to my knowledge).  Of course, these "other places" contained the most important data for my case.

In this post, I am going to cover some common locations in the file system to recover deleted text messages.  Additionally, because the SMS structure can vary across Android devices, I am going to show how I deconstructed the SMS message, and then applied the information to SMS messages found in unallocated space.  I am sure there is more than one way to skin this cat, some may even be better; this is just the way I did.

For this example, I used a Samsung GSM SGH-T959V Galaxy S.  Even if you don't do Mobile forensics, the principles of this example can be applied to determine structured data found in unallocated space.


Where the Messages are hiding

When working with cell phones, several types of acquisitions may be taken:  logical, file system and physical.  A logical acquisition is usually the information as the end user sees it.  Text messages, call logs etc.  It does not include deleted data.  A file system acquisition is the next step up. It provides access to the files system, but not unallocated space.  A physical acquisition is a bit by bit copy of the flash memory and thus, includes unallocated space. For more information on these three types of acquisitions, check out this page on Mobile Forensics on Wikipedia.

For recovering deleted text messages a physical extraction is the best.  However, there are several locations in a file system extraction that can yield deleted text messages: the SMS Database, the SMS journal file and a log database.

SMS Database

Text messages are stored in an SQLite database named mmssms.db typically under the location /Root/data/com.android.providers.telephony/databases/. These SQlite databases retain deleted data. If you are using a program like Cellebrite, it will "automatically" recover deleted text messages from this database. However, I also manually check for fragments using a Hex viewer, or an SQLite Viewer like Oxygen Forensics SQLite Viewer. They offer a 30 day free trial if you want to play around with it. This SQlite Viewer show blocks of deleted data:



SMS Journal File
The mmssms.db-journal file is a roll back journal file written to by the SQLite Database. If this file exists, it will be in the same directory as the mmssms.db file.  It can contain numerous deleted text messages. Since cheeky4n6 monkey helped develop a script to parse this, he has done an excellent write up on the format and structure which I will post a link to once its up.  I won't go into too much detail here, except to say the text messages contain the same structure as in the SMS database.

logs.db
This file appears to be a database that logs various activities on the phone such as calls and SMS messages.  The field holding the SMS messages appears to contain the first 50 or so characters of a text message.  Because it is an SQLite database, you can view deleted data as explained above.

Unallocated Space
Glorious unallocated space - my favorite location to find deleted text messages.  I have found hundreds of deleted text messages here through various keyword searches.  If you are lucky, you can carve a whole mmssms.db SQLite database from unallocated space as explained here by Richard Drinkwater.  This will allow you to open the complete file with an SQLite viewer and view information such as the phone number, date, sent date, status and folder.  If not, this is where leveraging your tools can help you parse this information manually.

Determining the SMS structure

Case Setup:
Let's say a bowl of Trix has been eaten, and you need to determine who did it and when.  A keyword search for "trix" brings you to the following message in unallocated space (phone numbers have been changed for this example):



 As you can see from the example above, the phone number and SMS body are readily apparent.  However, just by looking at the ASCII data on the right, it is not possible to tell if this was a sent or received message. In this case, that is critical information.  If the message was sent, the suspect whose phone you're examining ate the Trix.  If this text message was received from somebody, the sender ate the Trix.  We also need to figure out what the date of the message was.

The steps I use to determine the structure of the SMS are as follows: View the SMS database to see the schema and the format of the data. Do some pattern matching using existing SMS messages and apply the pattern to deleted messages.

First, I use one of my 'all in one' tools to see if it was able to parse the existing messages on the phone. Below is a text message in the existing inbox of the phone (picture from Cellebrite):



By viewing the mmssms.db file in Hex view and locating the text message above, I can begin to see how this data is stored in a raw format. I can see information such as the phone number, body and SMSC (the phone numbers are normally 10 digits long, but messages from T-Mobile come through with a phone number of 456). I know the date information must be hiding in here somewhere:


By viewing the Schema of the mmssms.db SQLite database table that holds the SMS messages and viewing the message within the database, I can gather more information about this message:

Schema 



Message


 By looking at the database, I can see that the date is stored in Epoch, and there are some flags that look like they might correspond to whether the message was sent or received – the fields named “read” and “type”.

By using all of this information, I can begin to figure out the structure.  The order of the fields in the raw data follows the order of the SMS schema. For clarity's sake, I have only included the most common fields in the picture below:


Address:
The Phone number associated with the message.

Date Field
To determine how the date is stored, I viewed this value in the database and noted it was stored in Epoch.  I tried converting the Hex value 01 3C 5D BC 32 76 to decimal which is 1358782280310. This value matches the value in the database. Converting to UTC yields Mon, 21 January 2013 15:31:20 UTC, which matches the value displayed by Cellebrite.

Read/Type
By using my all in one tool, I looked through the existing SMS Messages and cross referenced these values to what I was seeing in the database to establish what the flag values of "read" and "type" mean.
    Read:
        00 = Message was unread
        01 = Message was read
    Type (Aka"folder"): 
        01 = Inbox
        02 = Sent

Now I can apply the same structure to the message found in unallocated space and parse it manually:


 Date: Hex value 01 39 22 20 57 66 to decimal which is 1344897308518. Converting to UTC yields Mon, 13 August 2012 22:35:08 UTC

Type:  02 which means the message Sent.

So now I have some valuable information about this deleted text message. This message was sent by the user of the phone indicating they ate the Trix, and the message was sent on August 13, 2012 at 3:35PM AZ time.

I should also mention that within Cellbrite, you can view an existing message in the mmssms.db file in Hex view.  Cellebrite color blocks data it has parsed, and when you hover over a block, it will show you what values it parsed. I wanted to show the more "manual" way first in case some other tools do not do this, or if for some reason the database was not parsed, which I have ran into. Please note, it does not do this for messages in unallocated space.

 
Depending on the phone, the SMS structure may be totally different. Also, because the information is stored in a SQLite database and deleted fragments may be located outside of an SQLite database, there does not seem to be a "header" or "magic number" to carve for.

Here are three SMS schemas from three different Android Phones. Some different fields of interest have been circled. The same method used above could be used to determine how these values are stored "in the raw" to parse deleted messages found in unallocated space.


Samsung CDMA SPH-D720 Nexus S




Samsung GSM SGH-T839 Sidekick 4G

Samsung GSM SGH-T959V Galaxy S

 
As you can see from the three phones above, each phone has a different way of storing the SMS messages which can make it difficult to write a script to "recover all deleted text messages from all Android phones" -I have left that up to the monkeys - well at least one cheeky one.

Addendum [added 02/18/2013]
Please make sure and read the comment posted by Brian below. He makes a great point.  For any of the above to work, the data generated by the forensic tool/software needs to be validated against the actual device. I had done that in this case, but failed to mention it.

[added 02/25/2013]
Adrian's script is now available, make sure and check it out.

36 comments:

  1. Mari,

    Great stuff! I really like how you dig into the structure of the data that you're looking for, and then use that to pull out "deleted" data that simply isn't gone yet! Great work!

    ReplyDelete
  2. Mari,
    You successfuly converted a fairly complex forensic trick into an easy to understand and informative read. Thank you for taking the time to write this up and share it with me. I enjoyed it. Carlos C.

    ReplyDelete
  3. Mari,

    This is a good post on the process of carving data, but I would suggest having access to the same make/model and compare the "flags" of known data against your output, rather than using your tool to get the references. Your process is sound, but if your tool mistakenly correlates a "01" to being "read" when in fact it was "sent", all of your work is for nothing.

    It all comes back to trust but verify, and I would always recommend using the actual device and the data from that device rather than the output of a tool when drilling down to figure out the data storage/correlation.

    Brian

    ReplyDelete
  4. Brian,
    Brian,

    Great comment, I absolutely agree. In this particular case I was able to verify the data with the phone, but did not add that to the blog. Thanks for pointing that out :-)

    All of our automated tools needs to be verified against known data whether it's cell phones or Internet History. This also helps translate how the software writer is interpreting the raw data as well. For example, for this phone the database calls a flag "type" while the software output refers to it as "folder".

    ReplyDelete
  5. I performed as reported by your guide, I took my file and uploaded mmssms.db on program winhex, and with this program, I find my deleted messages but can not figure out the correct value in all those numbers as part contains the date of sending or receiving messages, i do not undersand correct value hexadecimal for translating in data epoch
    I can not understand the exact point the exact value that can give me the date
    what free program can display in hex ​​values​​, date, time, month, day and year?
    with winhex, it is difficult to find, the exact value where the data resides, I tried many values, converted to hexadecimal, but these values ​​I have never led to a correct date
    hexadecimal values, translating them into unix epoch that translates well the values ​​of this guide, but I can not find in my post the hex point where the data resides

    any of you know a free program that color, as in driving, sorting, and divided, and in color, what is the address, date, and body as in the guide?


    i wait you help please

    ReplyDelete
  6. please i ask your help in some text messages, i can not retrieve the date with that method why?

    in some sms can retrieve the date, month, day, year and time, taking the value after 01 of the 12 numbers and decode in decimal, after it is passed unix, and I get the correct date

    but in some cases is not present the value 01 and other values ​​that make in order to get the date, this happens also to messages that are not erased, the string to get the date does not come out in no attempt in some cases, why does it happen?

    why? what is wrong? This method of data recovery does not always work?

    ReplyDelete
  7. Hello,

    One way to determine where the date value may be is to have the same model phone to do some testing on. You could send the same text message at different times to determine which set of numbers is changing. This could help establish which set of Hex values makes up the date.

    ReplyDelete
    Replies
    1. thanks for the reply :) I wanted to ask what is the name of the program you used the most useful guide you wrote.
      The program that you used the company Cellebrite, it has a great feature to highlight the hex values ​​right of residence of the date and time.
      what is the specific name of the program you used?
      I tried Phyisical Analyzer the company Cellebrite and does not have this function to classify the values
      what other program you used for classification? This program exists in the trial version?
      where can a find a free program that performs this function highlight the values​​, an area where information such as date time?
      i wait you reply thanks
      and very thank you for your guide :)

      Delete
  8. one last question with this method can be explained in your guide, in some cases it is not possible to recovery the date and time the messages?

    or I can always retrieve it when they are the messages?

    ReplyDelete
  9. I used Cellebrite Physical Analyzer. It will only highlite values that it knows how to parse. For instance, it does not highlight dates in unallocated space because it did not parse that data.

    The only other program I can think of that may help you is the SQLite viewer I mentioned from Oxygen Forensics. There is a feature that will let you sweep hex values and it will show the possible dates.


    ReplyDelete
  10. Can the average person do this in order to retrieve deleted texts messages from an Android Samsung Exhibit? If so, how do I go about getting software to do this?

    ReplyDelete
  11. I would take a look at the book Android Forensics by Andrew Hoog http://amzn.com/1597496510. He discusses various tools and methods you can use to get a physical extraction of the phone.

    Most of the common software/hardware packages out there are expensive ($4,000-$10,000) If you need the evidence for court, I would recommend contacting a forensic examiner to conduct the work as specific steps and procedures need to be followed to ensure evidence integrity.

    ReplyDelete
  12. My husband sent me a picture of him in his uniform at basic training. Instead of locking the message I accidentally deleted it and have been kicking myself ever since. How can I get it back!! I have a galaxy note 1

    ReplyDelete
  13. Does this work on htc evo 3d?

    ReplyDelete
  14. Hi Can I Make my Own Sms Sarver plz if Any Know Let Know

    ReplyDelete
  15. Hey thanks for this post please let me know does it work on Samsung Galaxy S3

    ReplyDelete
  16. hey! does this require root?

    ReplyDelete
  17. This post makes the assumption that you have a forensic tool like Cellebrite, MPE+, XRY etc to take a forensic image of a cell phone.

    If you do not have access to these tools, a Google for "forensic image android phone" may lead you in the right direction.

    You can also use the adb from the Google Android SDK to connect to the phone and file system to access the database.

    I believe most of these methods require root access.

    ReplyDelete
  18. With this, would there be any possible way to retrieve a deleted MMS, such as a video or picture?

    Thanks

    ReplyDelete
  19. Does the _id and thread_id increment for each new SMS and conversation? And does it start with 1?

    ReplyDelete
  20. How did you acquire a physical image of the device? Once rooted you could easily get a dd image but the rooting process + installing tools will overwrite some free space... How would you do it without loosing anything?

    ReplyDelete
  21. I use a forensics tool called Cellebrite to acquire a physical image. There are also less expensive tools such as MPE+ and Oxygen Forensics that can take a physical image.

    ReplyDelete
  22. Any open source alternatives? Thanks Mari

    ReplyDelete
    Replies
    1. Try Santoku : https://viaforensics.com/resources/tools/santoku/

      Not quite sure what the imaging capabilities are, but it is an open source tool for mobile forensics.

      Delete
  23. And how far back can you go to retrieve deleted texts? Weeks, months, a year? Thanks, Mari.

    ReplyDelete
  24. Its hard to say. I have had phones where I have found deleted messages going back years, and some only days. It depends on various factors such as the make/model phone, how much the phone has been used, available memory etc...

    ReplyDelete
  25. OMG! This is all gibberish to me. I guess my texts are gone forever!
    My job could be on the line and the proof of innocence was a text thread I deleted.
    waahh!

    ReplyDelete
  26. Not all is lost! I would recommend hiring a professional forensic cell phone examiner to recover your messages. Google "Cell Phone Forensic Companies", or contact me at my company (BitWise Forensics) for assistance.

    ReplyDelete
    Replies
    1. Hello Mari, been trying to contact you to no avail via email and the phone on your website.

      Delete
    2. Please email me at arizona4n6 at gmail.com and I will be happy to answer you.

      Delete
  27. Just running into this now, Mari. Great post!

    ReplyDelete
  28. i need help step by step in retreiveing deleted sms messages from my galaxy exzibit phone

    ReplyDelete
  29. Hello...
    Thank you for your educational blog makes a complete newbie like me understand:) I have a question..I have a 30 day free trial of Ufed from Cellebrite. I have performed an extraction but there is very little analysed data..only locations. There are no call logs or contacts or sms new or deleted. It extracts from usb mass storage.is this correct. I tried an extract from ufed downloader but its says waiting for ufed then error cannot connect to device. check cable is connected properly. I have a Sony xperia c5303. DrFone for android can recover 90 deleted messages. I just wondered why there are no records at all in Ufed extraction. Any help would be very much appreciated.
    Thank you

    ReplyDelete
  30. Great post! There is some really helpful information on this blog. Thank you so much for sharing this!

    ReplyDelete
  31. I know I'm late to this post, but still an awesome post I read all the way through. Still great info, even from 2013. I wonder though, if because the schema is different on every phone, there must be a script on the phone that deciphers every phone's schema to a readable text, right?

    ReplyDelete