Unpacking Android backups

One of the less known new features introduced in ICS is the ability to backup a device to a file on your computer via USB. All you have to do is enable USB debugging, connect your phone to a computer and type the adb backup command in a shell. That will show a confirmation dialog on the phone prompting you to authorize the backup and optionally specify a backup encryption password. It looks something like this:


This doesn't require rooting your phone and lets you backup application data, both user installed and system applications (APK's), as well as shared storage (SD card) contents. There are some limitations though: it won't backup apps that have explicitly forbidden backups in their manifest, it won't backup protected (with DRM) apps and it won't backup some system settings such as APN's and WiFi access points. The transfer speed is limited by ADB channel speed (less than 1MB/s), so full backups can take quite some time. There is also a rather annoying bug in 4.0.4 where it will backup shared storage even if you don't request it. With all that said, it's a very useful tool, and will hopefully see some improvements in the next Android version.

The backup command is fairly flexible and lets you specify what apps to backup, whether to include system apps when doing a full backup, and whether to include shared storage (SD card) files. Here's a summary of the available options as displayed by adb's usage:

adb backup [-f ] [-apk|-noapk] [-shared|-noshared] [-all] [-system|-nosystem] [<packages...>]

 - write an archive of the device's data to <file>. If no -f option is supplied then the data 
 is written to "backup.ab" in the current directory.

 (-apk|-noapk enable/disable backup of the .apks themselves in the archive; 
 the default is noapk.)
 (-shared|-noshared enable/disable backup of the device's shared storage / SD card contents; 
 the default is noshared.)
 (-all means to back up all installed applications)
 (-system|-nosystem toggles whether -all automatically includes system applications; 
 the default is to include system apps)
 (<packages...> is the list of applications to be backed up.  If the -all or -shared flags 
 are passed, then the package list is optional.  Applications explicitly given on the command 
 line will be included even if -nosystem would ordinarily cause them to be omitted.)

The restore command is however quite limited -- there are no options, you can only specify the path to the backup file. One of the features most noticeably lacking is conditional restore: restores are all or nothing, you cannot restore only a subset of the apps (packages), or restore only the contents of the shared storage. Supporting this will require modifying the firmware, but you can extract only the needed data from the backup and copy it manually. Copying apps and app data to your device requires root access, but extracting and copying external storage files such as pictures and music can be done on any stock ICS device. And if you create a backup file containing only the files you need to restore, you wouldn't need root access at all. This post will present the format of Android's backup files and introduce a small tool that allows you to extract and repackage them as needed.

SDK API's for using Android's backup architecture were announced as far back as Froyo (2.2), but it has probably been available internally even before that. As introduced in Froyo, it uses a proprietary Google transport to backup application settings to the "cloud". ICS adds a local transport that lets you save backups to a file on your computer as well. The actual backup is performed on the device, and is streamed to your computer using the same protocol that adb pull uses to let you save a device file locally. When you execute the adb backup command a new Java process (not an activity or service) will be started on your device and it will bind to the system's BackupManagerService and requests a backup with the parameters you specified. BackupManagerService will in turn start the confirmation activity shown above, and execute the actual backup if you confirm (some more details including code references here). You have the option of specifying an encryption password, and if your device is already encrypted you are required to enter the device encryption password to proceed. It will be used to encrypt the archive as well (you can't specify a separate backup encryption password).

After all this is done, you should have a backup file on your computer. Let's peek inside it. If you open it with your favourite editor, you will see that it starts with a few lines of text, followed by binary data. The text lines specify the backup format and encryption parameters, if you specified a password when creating it. For an unencrypted backup it looks like this:

ANDROID BACKUP
1
1
none

The first line is the file 'magic', the second the format version (currently 1), the third is a compression flag, and the last one the encryption algorithm ('none' or 'AES-256').

The actual backup data is a compressed and optionally encrypted tar file that includes a backup manifest file, followed by the application APK, if any, and app data (files, databases and shared preferences). The data is compressed using the deflate algorithm, so, in theory, you should be able to decompress an unencrypted archive with standard archive utilities, but I haven't been able to fine one compatible with Java's Deflater (Update: here's how to convert to tar using OpenSSL's zlib command: dd if=mybackup.ab bs=24 skip=1|openssl zlib -d > mybackup.tar). After the backup is uncompresed you can extract it by simply using tar xvf mybackup.tar. That will produce output similar to the following:

$ tar tvf mybackup.tar
-rw------- 1000/1000      1019 2012-06-04 16:44 apps/org.myapp/_manifest
-rw-r--r-- 1000/1000   1412208 2012-06-02 23:53 apps/org.myapp/a/org.myapp-1.apk
-rw-rw---- 10091/10091     231 2012-06-02 23:41 apps/org.myapp/f/share_history.xml
-rw-rw---- 10091/10091       0 2012-06-02 23:41 apps/org.myapp/db/myapp.db-journal
-rw-rw---- 10091/10091    5120 2012-06-02 23:41 apps/org.myapp/db/myapp.db
-rw-rw---- 10091/10091    1110 2012-06-03 01:29 apps/org.myapp/sp/org.myapp_preferences.xml

App data is stored under the app/ directory, starting with a _manifest file, the APK (if requested) in a/, app files in f/, databases in db/ and shared preferences in sp/. The manifest contains the app's version code, the platform's version code, a flag indicating whether the archive contains the app APK and finally the app's signing certificate (called 'signature' in Android API's). The BackupManagerService uses this info when restoring an app, mostly to check whether it has been signed with the same certificate as the currently installed one. If the certificates don't match it will skip installing the APK, except for system packages which might be signed with a different (manufacturer owned) certificate on different devices. Additionally, it expects the files to be in the order shown above and restore will fail if they are out for order. For example, if the manifests states that the backup includes an APK, it will try to read and install the APK first, before restoring the app's files. This makes perfect sense -- you cannot restore files for an app you don't have installed. However BackupManagerService will not search for the APK in the archive, and if it is not right after the manifest, all other files will be skipped. Unfortunately there is no indication about this in the device GUI, it is only shown as logcat warnings. If you requested external storage backup (using the -shared option), there will also be a shared/ directory in the archive as well, containing external storage files for each shared volume (usually only shared/0/ for the first/default shared volume).

If you specified an encryption password, things get a little more interesting. It will be used to generate an AES-256 key using 10000 rounds of PBKDF2 with a randomly generated 512 bit salt. This key will be then used to encrypt a randomly generated AES-256 bit master key, that is in turn used to encrypt the actual archive data in CBC mode ("AES/CBC/PKCS5Padding" in JCE speak). A master key checksum is also calculated and saved in the backup file header. All this is fairly standard practice, but the way the checksum is calculated -- not so much. The generated raw master key is converted to a Java character array by casting each byte to char, the result is treated as a password string, and run through the PBKDF2 function to effectively generate another AES key, which is used as the checksum. Needless to say, an AES key would most probably contain quite a few bytes not mappable to printable characters, and since PKCS#5 does not specify the actual encoding of a password string, this produces implementation dependent results (more on this later). The checksum is used to verify whether the user-specified decryption password is correct before actually going ahead and decrypting the backup data: after the master key is decrypted, its checksum is calculated using the method described and compared to the checksum in the archive header. If they don't match, the specified password is considered incorrect and the restore process is aborted. Here's the header format for an encrypted archive:

ANDROID BACKUP
1
1
AES-256
B9CE04167F... [user password salt in hex]
9C44216888... [master key checksum salt in hex]
10000         [number of PBKDF2 rounds]
990CB8BC5A... [user key IV in hex]
2E20FCD0BB... [master key blob in hex]

The master key blob contains the archive data encryption IV, the actual master key and its checksum, all encrypted with the key derived from the user-supplied password. The detailed format is below:

[byte] IV length = Niv
[array of Niv bytes] IV itself
[byte] master key length = Nmk
[array of Nmk bytes] master key itself
[byte] MK checksum hash length = Nck
[array of Nck bytes] master key checksum hash

Based on all this info, it should be fairly easy to write a simple utility that decrypts and decompresses Android backups, right? Porting relevant code from BackupManagerService is indeed fairly straightforward. One thing to note is that it uses SYNC_FLUSH mode for the Defalter which is only available on Java 7. Another requirement is to have the JCE unlimited strength jurisdiction policy files installed, otherwise you won't be able to use 256 bit AES keys. Running the ported code against an unencrypted archive works as expected, however trying do decrypt an archive consistently fails when checking the master key checksum. Looking into this further reveals that Android's PBKDF2 implementation, based on Bouncy Castle code, treats passwords as ASCII when converting them to a byte array. The PKCS#5 standard states that 'a password is considered to be an octet string of arbitrary length whose interpretation as a text string is unspecified', so this is not technically incorrect. However since the 'password' used when calculating the master key checksum is a randomly generated value (the AES key), it will obviously contain bytes not mappable to ASCII characters. Java SE (Oracle/Sun) seems to treat those differently (most probably as UTF-8), and thus produces a different checksum. There are two ways around this: either use a Bouncy Castle library with the Android patches applied, or implement an Android-compatible PBKDF2 function in our decryption code. Since the Android Bouncy Castle patch is quite big (more than 10,000 lines in ICS), the second option is clearly preferable. Here's how to implement it using the Bouncy Castle lower level API's:

SecretKey androidPBKDF2(char[] pwArray, byte[] salt, int rounds) {
  PBEParametersGenerator generator = new PKCS5S2ParametersGenerator();
  generator.init(PBEParametersGenerator.PKCS5PasswordToBytes(pwArray), 
                 salt, rounds);
  KeyParameter params = (KeyParameter)
     generator.generateDerivedParameters(PBKDF2_KEY_SIZE);

   return new SecretKeySpec(params.getKey(), "AES");
}

This seems to do the trick, and we can now successfully decrypt and decompress Android backups. Extracting the files is simply a matter of using tar. Looking at the archive contents allows you to extract certain files that are not usually user accessible, including app databases and APK's without rooting your phone. While this is certainly interesting, a more useful scenario would be to restore only a part of the archive by selecting only the apps you need. You can do this by deleting the ones you don't need, repacking the archive and then using adb restore with the resulting file. There are two things to watch out for when repacking though: Android expects a particular ordering of the files, and it doesn't like directory entries in the archive. If the restore process finds a directory entry, it will silently fail, and if files are out of order, some files might be skipped even though the restore activity reports success. In short, simply tarring the unpacked backup directory won't work, so make sure you specify the files to include in the proper order by creating a backup file list and passing to tar with the -T option. The easiest way to create one is to run tar tvf against the decompresed and decrypted original backup. Once you create a proper tar file, you can pack it with the provided utility and feed it to adb restore. Another thing you should be aware of is that if your device is encrypted, you need to specify the same encryption password when packing the archive. Otherwise the restore will silently fail (again, error messages are only output to logcat). Here's how to pack the archive using the provided shell script:

$ ./abe.sh pack repacked.tar repacked.ab password

Full code for the backup pack/unpack utility is on github. Keep in mind that while this code works, it has very minimal error checking and might not cover all possible backup formats. If it fails for some reason, expect a raw stack trace rather than a friendly message. Most of this code comes straight from Android's BackupManagerService.java with (intentionally) minor modifications. If you find an error, feel free to fork it and send me a pull request with the fix.

Comments

Martin Sandford said…
This comment has been removed by the author.
Nikolay Elenkov said…
Untar, modify, then tar again and pack using the provided utility. What files are you modifying?
Martin Sandford said…
This comment has been removed by the author.
David Schlenk said…
I've been hit with the bug where I thought I had a full backup but so far haven't been able to get anything terribly useful from it. I've tried the Java code, which seems to properly extract the system partition before I get a ZipException, but the SD card portion I haven't had any luck getting. I've also given the perl scripts a go but they don't seem to work for me either. The stacktrace from your Java app is:
Exception in thread "main" java.lang.RuntimeException: java.util.zip.ZipException: invalid stored block lengths
at org.nick.abe.AndroidBackup.extractAsTar(AndroidBackup.java:181)
at org.nick.abe.Main.main(Main.java:35)
Caused by: java.util.zip.ZipException: invalid stored block lengths
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:147)
at java.io.FilterInputStream.read(FilterInputStream.java:90)
at org.nick.abe.AndroidBackup.extractAsTar(AndroidBackup.java:160)
(Line numbers might be slightly different as I attempted to do a bit of troubleshooting on the code. )
My guesses for the error: my data file's size - it's about 10GB; on the xda post about the perl utility, there's some talk about the first part of the file actually being compressed but the second part isn't - perhaps that's the case for me. When the Java app dies, the file that is created does seem to be a valid tar other than it doesn't contain all the data.

So far have tried the Java utility on my mac and the perl scripts on my mac and a Ubuntu 8.04 box.
Nikolay Elenkov said…
Sounds like you've hit the external/shared storage bug indeed. Unfortunately the extractor tool doesn't handle this automatically. What you can do is (carefully) extract the shared storage part from the archive with a binary editor, and that should leave you with a valid backup for /data and /system. Then you should be able to use the Perl script to convert the extracted part to a tar file, so that you can restore /sdcard. I think the XDA post has some hints about this.
Anubis said…
This comment has been removed by the author.
Anubis said…
This way is much faster:

dd if=mybackup.ab bs=24 skip=1|openssl zlib -d > mybackup.tar
Nikolay Elenkov said…
Yes, I mention this in the article, but it only works for unencrypted backups.
Anubis said…
And if is password encrypted, which will be the command?

I've checked your github, but my java installation is not in "/usr/local/jdk1.7.0_04/bin/java" but in "/usr/bin/java"
Nikolay Elenkov said…
The path in the script is just a sample, edit to fit your environment. Run the shell scripts without parameters to see usage.
Nikolay Elenkov said…
If you meant what the openssl, etc. standard command is, there isn't a direct one. You have to read the header and do some processing to derive the decryption password. It could probably be done in a shell script using some combination of sed, grep, awk and openssl, but it won't be pretty. If you don't want to use Java, check out the Perl scripts on XDA, or use your favorite scripting language to port the decryption code.
Anubis said…
Apparently, the tar generated by either the java method or the dd and zlib one, seems to generate invalid tar streams.

Just check it trying to delete a file from the tar backup:

$ tar -v -f backup.tar --delete apps/com.adobe.flashplayer/a/com.adobe.flashplayer-1.apk
tar: Deleting non-header from archive
tar: Deleting non-header from archive
tar: Deleting non-header from archive
tar: Deleting non-header from archive
tar: Deleting non-header from archive
tar: Deleting non-header from archive
tar: Exiting with failure status due to previous errors

$ tar --list -f backup.tar
tar: Skipping to next header
apps/com.boatbrowser.free/r/app_plugins/com.adobe.flashplayer/.adobe/Flash_Player
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

Are you experiencing the same issue?
Anubis said…
example issue

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=554433
Nikolay Elenkov said…
Not but, I haven't used the --delete option either. In any case, you need to delete not only the apk, and preserve the file order to able to restore. What works for me is unpacking, deleting the file and re-packing. See the last part of the article.
Anubis said…
I have tried unpacking the tar, deleting the corresponding folder and repacking the tar with -T option, but some entries appear twice in the new tar, for example this:

apps/com.android.chrome/r/app_sslcache/www.google.com.443
apps/com.android.chrome/r/app_InvalidationClient/chromesync#1078188958

The text file used with -T weights 495 KB, while getting the list of the tar results in 3.7 MB, so something went wrong.

Did you ever experience this issue?
Nikolay Elenkov said…
I'd suggest creating an minimal backup to experiment with (a couple of packages, etc.) so you can keep track of your changes.
Anubis said…
How to detect what apps / files are being specified not to be saved to adb backup? Where are those settings?
Anubis said…
Which exact flavour and version of Tar are you using to pack the tar archive?
Nikolay Elenkov said…
$ tar --version
tar (GNU tar) 1.26
Anubis said…
This comment has been removed by the author.
Thomas said…
Nice write-up. I sure like your abe.sh utility. It proved much faster than dd due to the block size being limited to 20 bytes.
Nikolay Elenkov said…
Thanks, glad it work for you.

The Java code is using a much larger buffer and compression speed should be comparable to zlib's (I think Java SE uses native code for this), so in general it should perform better for larger backups (I haven't done an actual benchmark though).
Reki Nagato said…
Well, your project on GitHub seem down. Would you like to share the unpack utility for windows?
Nikolay Elenkov said…
Github seems OK to me. Temporary problem?

You can clone the code like this:

git clone http://github.com/nelenkov/android-backup-extractor.git

As this is written in Java, there is no Windows-specific version.
Nikolay Elenkov said…
I've added an Ant build file that lets you create an all-in-one jar you can run with:

java -jar abe.jar
Reki Nagato said…
But I just get an error:
build:
[javac] Compiling 2 source files to /home/reki/android-backup-extractor-master/build
[javac] /home/reki/android-backup-extractor-master/src/org/nick/abe/AndroidBackup.java:216: cannot find symbol
[javac] symbol : constructor DeflaterOutputStream(java.io.OutputStream,java.util.zip.Deflater,boolean)
[javac] location: class java.util.zip.DeflaterOutputStream
[javac] finalOutput = new DeflaterOutputStream(finalOutput, deflater,
[javac] ^
[javac] 1 error

BUILD FAILED
/home/reki/android-backup-extractor-master/build.xml:15: Compile failed; see the compiler error output for details.
Nikolay Elenkov said…
Do read the blog post first. Use Java 7.
Easy said…
This comment has been removed by the author.
Easy said…
I struggled getting the restore to work properly and after many BackupManager exceptions, the procedure below allowed me to backup an application and data, modify files in that backup and restore it successfully. (Of special importance is the way the TAR archive is recreated after adding/replacing/removing files.)

1. Backup application and data on the source:

adb -d backup -f filename.ab -apk javapackagename

2. Before restoring the application, remove it from the target.

adb uninstall javapackagename

3. If you want to modify the backup contents use abe.jar as follows to extract the backup file to

a TAR file:

java -jar abe.jar unpack filename.ab filename.tar

4. Extract the TAR file to a place of your choice. In this example this is the current directory:

tar -xvf filename.tar

5. Now replace any files you want to replace in the apps folder created. For example, I replaced the database files in apps/javapackagename/db. (filename and filename-journal)

6. Create a new TAR file containing the files. Note how each file is added individually AND notice that the apps directory is not preceeded by ./ but is just using the directory name.

tar -cvf filename_new.tar apps/javapackagename/_manifest apps/javapackagename/a/application.apk apps/javapackagename/db/database-journal apps/javapackagename/db/database

7. Repack the TAR file sothat it can be used by Androids BackupManager:

java -jar abe.jar pack filename_new.tar filename_new.ab

8. Restore application and data to the target:

adb -d restore filename_new.ab
hyperair said…
Actually, I noticed some pretty high CPU usage coming from dd as well, arising from the block size. I used this instead: … | (dd bs=24 skip=1 count=0; cat) | …

That uses dd to skip over the first 24 bytes, and allows cat to take over funnelling the stream later.
boyanboev said…
A cool way to check the dd progress:

$ pgrep -l '^dd$'
8789 dd
$ watch -n 10 kill -USR1 8789
Unknown said…
Thanks for the information on the file format.
I've created some tools for my specific needs. I am sharing in case it is useful to someone.
Check it out at:
https://github.com/pedrox/android-backup-utils
How in the world do I start this software or whatever you call it. I have no clue what I'm doing and I've read this thing twice and still don't understand. Also I'm trying to open an encrypted backup.
Ok just kidding I got it to run but now I get this (I made my password stars because it's stupid and embarrassing)

Dakotas-iMac:android-backup-extractor-20130526-bin Kameechewa$ java -jar abe.jar unpack /Users/Kameechewa/Downloads/android-backup-extractor-20130526-bin/backup.ab /Users/Kameechewa/Downloads/android-backup-extractor-20130526-bin/unpacked.tar ********
Strong AES encryption disabled

Magic: ANDROID BACKUP
Version: 1
Compressed: 1
Algorithm: AES-256
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.lang.RuntimeException: java.security.InvalidKeyException: Illegal key size
at org.nick.abe.AndroidBackup.extractAsTar(AndroidBackup.java:320)
at org.nick.abe.Main.main(Main.java:58)
... 5 more
Caused by: java.security.InvalidKeyException: Illegal key size
at javax.crypto.Cipher.checkCryptoPerm(Cipher.java:1024)
at javax.crypto.Cipher.implInit(Cipher.java:790)
at javax.crypto.Cipher.chooseProvider(Cipher.java:849)
at javax.crypto.Cipher.init(Cipher.java:1348)
at javax.crypto.Cipher.init(Cipher.java:1282)
at org.nick.abe.AndroidBackup.extractAsTar(AndroidBackup.java:244)
... 6 more
Nikolay Elenkov said…
You need to install the unlimited strength JCE policy so that Java can use larger key sizes, as used by the backup encryption.
Ok I did that and now I'm getting

Strong AES encryption enabled
Magic: ANDROID BACKUP
Version: 1
Compressed: 1
Algorithm: AES-256
IV: 21DA85E43F9D27BCC7427DFD54D96675
MK: C4B3378557E7FE6C8D5BE7D4B62F6B242F4CB53E6F945F865C826AFF6426DB84
MK checksum: 781096C69E1D0FF654FABBEC04AE63F3BB5191E0BD096237068F7C2502FAD90A
key bytes: C4B3378557E7FE6C8D5BE7D4B62F6B242F4CB53E6F945F865C826AFF6426DB84
salt bytes: 87245C34DC7DE48ED5B5DF066CDB6BD3D9C3511A13B67A5FDCC4708278DC87D0955C3609A9561040AE6496BF857561065AFC47443E0A499985D87E32498003CC
MK as string: [ᅣᄈ7ナW￧￾lヘ[￧ᅯᄊ/k$/Lᄉ>oヤ_ニ\ツj￿d&ᅴト]
Key format: RAW
Calculated MK checksum: 3DC8D274D31AE7CE748C70F86ECBCDA231D79F74D64BE33603037ACF003A96A5
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: Invalid password or master key checksum.
at org.nick.abe.AndroidBackup.extractAsTar(AndroidBackup.java:320)
at org.nick.abe.Main.main(Main.java:58)
... 5 more
Caused by: java.lang.IllegalStateException: Invalid password or master key checksum.
at org.nick.abe.AndroidBackup.extractAsTar(AndroidBackup.java:286)
... 6 more
Frau OMG! said…
hi i am trying to unpack my .ab backup in ubuntu. but after typing in terminal:
sh abe.sh unpack nexus4.ab
i got this:
root@marcars-desktop:/home/marcars/Android/android-backup-extractor-master# sh abe.sh unpack nexus4.ab
Error: Could not find or load main class org.nick.abe.Main
I'm having this problem too
Hi,

I have the backup file which has following 4 lines in starting: -

ANDROID BACKUP
1
1
none

I am unable to extract it with the the Android Backup Extractor. Could you please help me out?
I am using a Mac computer. Would it be possible if I send over my file for extraction?

Thanks in anticipation!!!
alan carr said…
This comment has been removed by the author.
alan carr said…
This comment has been removed by the author.
alan carr said…
I'm getting the error as in Dakota's second post. Running Kubuntu 12.04 with adb 1.0.31 for Linux. Downgraded from Android 4.4.2 to 4.3 on Nexus 7 2012. Trying to restore from adb or Ti Backup does not work.
Nikolay Elenkov said…
Can't be sure what the cause is, but there is a bug in 4.4 that might trigger this. Get latest code from Github and try with that.
Ronak Patel said…
i'm getting error like this..

13663200 bytes read
13719500 bytes read
13732300 bytes read
13788600 bytes read
13857700 bytes read
Exception in thread "main" java.lang.RuntimeException: java.util.zip.ZipExceptio
n: invalid stored block lengths
at org.nick.abe.AndroidBackup.extractAsTar(AndroidBackup.java:192)
at org.nick.abe.Main.main(Main.java:35)
Caused by: java.util.zip.ZipException: invalid stored block lengths
at java.util.zip.InflaterInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at org.nick.abe.AndroidBackup.extractAsTar(AndroidBackup.java:173)
... 1 more
Nikolay Elenkov said…
The file is probably corrupted.
Benjamin C said…
This comment has been removed by the author.
Nikolay Elenkov said…
You should contact Helium support to find out if the backup files it produces are in the same format as native Android backups. You can also look at the header at the file to see how if it looks the same as what is described in this post.
Benjamin C said…
This comment has been removed by the author.
Benjamin C said…
This comment has been removed by the author.
G-MaN said…
I have a Nexus 5, I've just updated to 4.4.3 and the adb backup version was changed to 2! The backup extractor prints the message "Don't know how to process version 2" Any chance for you to update your extractor please?
G-MaN said…
Looking on github through the BackupManagerService.java history, this change introduced the new version: "Adapt to underlying changes in the PBKDF2 implementation"
G-MaN said…
well, given that the change seems to be only because of cryptographic reasons and my backup not being encrypted I hex-edited the ab file and changed '2' to '1' and the extractor works.
Nikolay Elenkov said…
Yes, I saw that change. I will update the program to accept ver 2 when I get a chance. For now, you can simply change the version to 1 manually (as you did), it should work even if the backup is encrypted, because the program tries both algorithms if the first try fails. Or, if not encrypted, use the OpenSSL method, it simply skips the header, so the version doesn't matter.
G-MaN said…
Thanks for the suggestions and for this tool, too :)
Nikolay Elenkov said…
I've updated the code to support backups with version 2. Use the 'pack-kk' command to create ver to backups (maybe I should change it to pack-v2 or smth).
G-MaN said…
This is great, thanks!
Benjamin C said…
This comment has been removed by the author.
Nikolay Elenkov said…
Build the extractor tool from the latest code Github if your backups are encrypted, otherwise just use OpenSSL to decompress.
Benjamin C said…
This comment has been removed by the author.
Nikolay Elenkov said…
This is not the latest version. The latest version handles backups with ver=2, but you have to build it from source. Why do you want to create an uncompressed backup with adb? There is no such option, so it can't be done without modifying the Android build (ROM).
Benjamin C said…
This comment has been removed by the author.
Nikolay Elenkov said…
So apparently Helium intentionally changes a few bytes to make your life harder. See
http://sourceforge.net/projects/heliumbackupextractor/
Benjamin C said…
This comment has been removed by the author.
Benjamin C said…
This comment has been removed by the author.
Nikolay Elenkov said…
This refers to Android source code (BackupManagerService.java). It simply means that the tar file inside the backup is not compressed. Helium is probably changing something at runtime or they have their own implementation of the BackupService. Not idea why they would turn off compression though. You can't achieve the same thing without writing code and messing with Android internals.
Benjamin C said…
This comment has been removed by the author.
Andrei Paval said…
I have this problem too... I updated the Android sdk and abe but I still get the error...
Benjamin C said…
This comment has been removed by the author.
ubunter said…
On Ubuntu openssl is not compiled with zlib support, so the command:

dd if=backup.ab bs=24 skip=1|openssl zlib -d > mybackup.tar

leads to "openssl:Error: 'zlib' is an invalid command.". But there is standalone command you can use instead:

zlib-flate -uncompress

=> dd if=backup.ab bs=24 skip=1 | zlib-flate -uncompress > mybackup.tar
Sam Segers said…
If openssl is configured without zlib support (debian like systems), you can also use "zlib-flate -uncompress"
MrSharpie said…
Unfortunately am in the same position after using adb to take a "full backup" of my nexus 4 before proceeding to wipe/unlock.

Have scoured the XDA forums but only post I found with hints to recover backups with the bug was this
http://forum.xda-developers.com/showpost.php?p=29072968&postcount=4
tarfix.pl doesn't work for me, and I don't have the nous to be able to follow the high level instructions by binaryhero

Any chance you have some pointers over how exactly to extract the shared storage part from the archive with a binary editor?
MrSharpie said…
Unfortunately am in the same position after using adb to take a "full backup" of my nexus 4 before proceeding to wipe/unlock.

Have scoured the XDA forums but only post I found with hints to recover backups with the bug was this
http://forum.xda-developers.com/showpost.php?p=29072968&postcount=4
tarfix.pl doesn't work for me, and I don't have the nous to be able to follow the high level instructions by binaryhero

Any chance you have some pointers over how exactly to extract the shared storage part from the archive with a binary editor?
^_^ said…
Hello, thanks a lot for your application. I have two questions:

1. After modifying/editing/deleting files from an extracted android backup using the method in your readme file, do I have to update the manifest file before packing and restoring the modified backup to my device? If yes, how do I modify or update the manifest file? As I saw, it just contains some numbers. What are those numbers?

2. For apps that disallow backups, how do I change the "allowBackup" in the manifest so that I can backup using adb again? Where is the manifest file in the apk? Is it possible to modify this?

Pleaseeeee please answer. Thanks a lot. :)
Nikolay Elenkov said…
You can find detailed guides about repacking backups on XDA, etc., but it's generally tricky because order and format needs to be preserved. IIRC, it should work even if you don't modify the manifest, but do test on a recent Android version.

You cannot change allowBackup unless you have root, but once you do, you can access data directly anyway.
^_^ said…
Hello,

I searched everywhere. Repacking is done using the same method as I did by creating a list of the tar file, editing the list file without changing order and converting it back to ab using star or pax. But there is no information anywhere about the manifest file. What do I do with it? I just want to know the way to update the manifest file. I want to know even if backups work without updating the manifest.

What is IIRC?

By the way, the restore worked with the above method on lollipop. I just want to know how to update the manifest. Please tell me.

I used apktool to unpack an apk and it was converted to readable format. I can edit the androidmanifest.xml now. I can change the allowBackup to true. So, my question is, will the app work with adb if I install the modified apk? I haven't got a device to test all these. So I'm asking. :) Please answer. Thank you. :)
^_^ said…
Why don't you answer? Did I say something wrong? :( I couldn't find how to edit the _manifest file in the internet. That's all I want to know. Please answer.
Nikolay Elenkov said…
I can't give you detailed instructions, check Android source code for details. These might be helpful, check their READMEs: http://sourceforge.net/projects/adb-split/ and http://sourceforge.net/projects/adbextractor/
^_^ said…
Yes, I read the READMEs before. It's written.
- When adding or removing files for an app you may need to edit _manifest so adb restore accepts the changes.

But how do I edit? I can easily understand if you say it. Please help a 17 year old girl. Computer science is not my subject. Thanks. :)
Nikolay Elenkov said…
The manifest format is as described in the blog post. If you want to mess around with Android internals, you need to do some work yourself, you won't find guides for everything. Being young actually makes this much easier :)

Here's the format, as it appears in Android source code:

// Manifest format. All data are strings ending in LF:
// BACKUP_MANIFEST_VERSION, currently 1
//
// Version 1:
// package name
// package's versionCode
// platform versionCode
// getInstallerPackageName() for this package (maybe empty)
// boolean: "1" if archive includes .apk; any other string means not
// number of signatures == N
// N*: signature byte array in ascii format per Signature.toCharsString()
^_^ said…
Thank you. I didn't need a guide, just a hint 'cause I don't know anything technical. I'll use my intelligence to experiment with it. I've to find out how to find those values now, especially the signature part. I have to find out how to obtain the hash of the key so that the manifest can be modified if necessary. If you already know it, please share it. :) Thanks.
Jérôme Poulin said…
Using "tail -c +25" instead of dd is also much more efficient.
^_^ said…
Hello, I'm getting this error when I try to pack with star:
---------------------------
star.exe - Application Error
---------------------------
The application was unable to start correctly (0xc000007b). Click OK to close the application.
---------------------------
OK
---------------------------

If I try pax, it says ustar header too small. I couldn't find a solution. Please help. Thanks.

It has worked for me before. I don't know what's the problem.
Jos S said…
Trying to unpack a Sony Z2 encrypted backup. Any idea why it fails and what to do?
This backup is encrypted, please provide the password
Password:
Exception in thread "main" java.lang.RuntimeException: java.security.InvalidKeyException: Illegal key size
at org.nick.abe.AndroidBackup.extractAsTar(AndroidBackup.java:203)
at org.nick.abe.Main.main(Main.java:35)
Caused by: java.security.InvalidKeyException: Illegal key size
at javax.crypto.Cipher.checkCryptoPerm(Cipher.java:1039)
at javax.crypto.Cipher.implInit(Cipher.java:805)
at javax.crypto.Cipher.chooseProvider(Cipher.java:864)
at javax.crypto.Cipher.init(Cipher.java:1396)
at javax.crypto.Cipher.init(Cipher.java:1327)
at org.nick.abe.AndroidBackup.extractAsTar(AndroidBackup.java:118)
... 1 more
Nikolay Elenkov said…
You need the unlimited strength policy files for JCE, read the blog post for details.
jd said…
This comment has been removed by the author.
jd said…
Hi I was also wondering how you convert from tar to adb using version 3. Many thanks.
Tim Small said…
Very useful thanks!

I found repacking and restoring um challenging...

Using tar -T ended up including the directories, so that every file was in the tar archive twice - this is because the original listing made with tar -t included both directories and files. Fix was to manually delete the files from the listing (tar -tv | grep '^d' will show just the directories).

Then for some reason just using GNU tar (1.27.1 from Debian) didn't seem to create archives that my phone (Moto G 4G XT1039 Android 4.4.4) was willing to restore, so I used the FreeBSD tar command instead and that worked (from the Debian bsdtar package). The final incantation was:

java -jar abe.jar unpack backup.ab - | tar xpf - --numeric-owner

java -jar abe.jar unpack backup.ab - | tar tf - > appsfiles-orig

java -jar abe.jar unpack backup.ab - | tar tvf - | grep '^d' > appsfiles-orig-directories





bsdtar cvplf - -T appsfiles-orig-nodirs --numeric-owner | java -jar /home/tim/compile/android-backup-extractor/abe.jar pack-kk - modified.ab

adb restore modified.ab


I don't think it was necessary to force-stop the application in question (the purpose of the exercise was to edit it's database so that I could move a bunch of files from internal to sdcard storage), BICBW...
Nikolay Elenkov said…
Yes it is rather tricky and possibly version-dependent. See this site, they have repackaged mine and other tools, their README includes an extensive guide to repacking:

http://sourceforge.net/projects/adbextractor/
pmcgraw0 said…
since my openssl didn't have a zlib command, i found zlib-flate -uncompress worked instead.

Popular posts from this blog

Password storage in Android M

Decrypting Android M adopted storage