yashsingh1/xdaforums huggingface dataset:
[General Accessories] LA_:
cat mi.zlb | python -c "import zlib,sys;sys.stdout.write(zlib.decompress(sys.stdin.read()))" > mi.tar
Traceback (most recent call last):
File "<string>", line 1, in <module>
zlib.error: Error -5 while decompressing data: incomplete or truncated stream

I think there could be 2 possible reasons:

1) the command line I use is incorrect. I am not sure what

cat mi.zlb | openssl zlib -d > mi.tar 2>&1 | tee -a log

2>&1 | tee-a log means.

2) the mi.ab file I've got is corrupted/not full.

Coming back to the script. Why doesn't it wait for backup completion? It waits for button press on the phone, but doesn't wait for file copying.

I commented lines related to file deletion, so I've got another copy of mi.ab. Then I commented backup line, so the script used mi.ab I saved before. But it is failed to extract the file (strange, but there is no zlib error any more):

unpacking backup file
tar: Unrecognized archive format
tar: apps/com.xiaomi.hm.health/db/origin_db-journal: Not found in archive
tar: apps/com.xiaomi.hm.health/db/origin_db: Not found in archive
tar: Error exit delayed from previous errors.

mi.ab file size is 266815, mi.zlb - 266791, but mi.tar is 1681 only.

And, my app version is 1.2.312. Is it OK? Haven't found app version requirements.

Upd. Finally I've managed to extract the data. I did the following:

1) run command manually thru Terminal

/path/to/adb backup -f mi.ab -noapk -noshared com.xiaomi.hm.health

In result I got mi.ab file (file size is 812611, not 266815 as it was before)

2) run commands

tail -c +25 mi.ab > mi.zlb 2>&1 | tee -a log
cat mi.zlb | python -c "import zlib,sys;sys.stdout.write(zlib.decompress(sys.stdin.read()))" > mi.tar

Somehow 'openssl zlib' didn't generate correct mi.tar file

3) extracted files origin_db and origin_db-journal manually (tar was returning given above Unrecognized archive format error) and moved them to db folder.

4) then the script generated extract.csv and extract.js files.

Btw, looks like there no xdg-open tool on OSX. Also mi_data.html is part of analytics package. If user downloaded just extract tool, then this line will fail.

To address issue 1 I've added the following code to the script:

read -rsn 1 -p $"Press any key once backup is finished... "

Tar started to work without any modifications (not sure what was the problem before) and xdg-open I replaced just with open (see http://stackoverflow.com/questions/10006958/open-an-html-file-with-default-browser-using-bash-on-mac).

Thanks for your script and help!
[General Accessories] xmxm:
Thank you for updates on your progress. I do not have option to test these changes, but I've tried to update run.sh script based on your feedback. Here is a list of modifications that I've made to it based on your findings:

1. Added new settings for ADB path (by default it points to .)

2. Added pause after ADB backup - apparently Windows version does not seem to require it.

3. Added comment stating alternative command for openssl zlib - I could parametrize it if needed, but I do not know if it is worth it since I do not know if Python comes preinstalled.

4. Added another comment stating that xdg-open could be replaced with open command - If you know how this could be checked on the fly, I'd appreciate any information.

5. Added checking for actual mi_data.html existance before trying to open it.

Few additional changes were synced from latest updates to Windows batch:

1. run ADB kill-server after ADB operations complete.

2. Implement settings and support for ADB over Wifi.

If you could test new version and let me know if it works, I would really appreciate it and would merge it into the next version.
[General Accessories] joshmosh:
Good morning,

I am trying to use your analytics with the miband app on my iPhone. There is a file on the iPhone which looks very similar or even identical to origin_db. When I bin\sqlite3 \dborigindb \db\miband.sql I get the error messages

no such table date_data

no such table: android_metadata

Any hints what is in these tables ?

Sorry for posting something sort of off topic

Cheers

Joshi
[General Accessories] LA_:
Looks like there is a problem with new script with newly added ADB over Wi-Fi functionality -

Extract Started
02-27-15 13:44
ok
Cannot find database files. Non-rooted phone? Attemting backup approach
Press Backup My Data button on device...
* daemon not running. starting it now on port 5037 *
* daemon started successfully *
adb: unable to connect for backup
...

Also, there is some syntax problem with open line:

./run.sh: line 84: syntax error in conditional expression
./run.sh: line 84: syntax error near `-a'
./run.sh: line 84: ` [[ -f ./mi_data.html -a $OpenHTML == 'Y' ]] && open mi_data.html'
[General Accessories] xmxm:
First we need to see if this file is in SQLite format. You can try to open it with some applications that open SQLite databases, for example I am using DB Browser: https://github.com/sqlitebrowser/sqlitebrowser/releases. If you can open it in such an application, the second step would be to look at the table names - it is possible that iPhone version has different table naming. You would need to look for the table that contains one row for every day of activity and it would likely have one of its columns named "Date". Then we would need to see if that table has structure and formatting similar to Android version.

Alternatively, you can PM me the file and I could look at it.
[General Accessories] joshmosh:
Thank you for your kind offer.

Yes, it is a sqlite structure. I can open it with a sqlite browser. It consists of just one table named data. There are a number of fields named id, date, summary, indexes etc

There is one line per day. The entries under summary are json structures.

I will send you the file as PM.

Thank you very much in advance for your help, I appreciate it !

Cheers

Joshi

PS: may I ask you to post your findings to this thread ? Maybe it can help others with a similar problem as well. Thank you.
[General Accessories] joshmosh:
I have not found a hint how to attach files to a PM, so I attach it here, sorry.

Joshi
[General Accessories] xmxm:
Ok, it is a SQLite file and the table structure is very similar to Android version, although order of data elements is slightly different. I'll work on creating a data extract code for your file and will post it in the next couple of days for you to test.
[General Accessories] xmxm:
Here is a first take on iOS specific file. Put this file along with user.sqlite it in db\ directory and run following command:

bin\sqlite3.exe db\user.sqlite < db\miband_ios.sql

Could you please review resulting files and check if data matches reading on your phone? In particular, I'd like to know if Fall Asleep time and Awake time (columns 5 and 6 in .csv file) are in sync with what you see on your phone.

Also, could you please share information on how you pulled this sqlite file from you iPhone, so I could add this information to instructions page?
[General Accessories] joshmosh:
Than you very much !!!

Extraction went well. Please see PM for the questions you answered. Later this day I will check contents of .csv file with data on my iphone. In addition I will write up how to extract both files (data.sqlite and lua .sqlite) from the iphone. Both files can be copied with a pretty standard ios-method: iTunes sync.

Thanks a lot and have a nice day

Joshi
[General Accessories] joshmosh:
This is a method how to acces the sqlite file(s) on an iPhone.

1. Get a copy of itools. You can get it from various sources, for example cnet and others. Google is your friend :o

2. Plug in your iPhone to your PC

3. open itools

4. Click on Applications in top row

5. on the left you will find name of your iPhone and below "Applications". Click on Applications

6. Locate your application Mi Fit and click it.

7. On the right you will find a button labelled "browse". Click it.

8. A window will open. Doubleclick on "Documents"

9. Click on users.sqlite, then on "Export" above.

10. A window will open asking you where to export this file. Export it to a location of your choice on your PC

11. Click OK

Thats it. It works for me on my (jailbroken) iPhone. I have no idea if this will work on a non-jailbroken one, but I think so.

Unfortunately I have not yet found a method to automate it.
[General Accessories] mritsch:
raw data investigations

I did some investigations on the raw Mi Band data stored in date_data section of the SQL Light database. This raw data consists of 4320 bytes for each day. If you look at the data and consider its length, you may come also to the assumprion, that 3 bytes are stored/overwritten in this 4320 byte structure each minute. The raw structure starts at 0:00 am with three bytes and so on.

That was my guess and I wanted to verify it and evaluate what is in that 3 byte structure each minute. Therefore I extracted the raw data and pasted it into an excel sheet (attached to this post).

When you look at the excel sheet you may see, that the third byte of the three byte structure holds the counted steps of this minute. The sum up the steps gathered by the third byte of the three byte structure and the output of the total steps presented in the Mi Band app is equivalent (screenshots of the Mi Band app also attached). Also the step timeline is equivalent to the output of the app.

My interpretation on the other two bytes is not finalized: my guess is that the first byte is a status byte. Definitively it is 7E when the time it belongs to is in the future. Furthermore the byte changes when the sleep period changes (5 corresponds deep sleep, whereas when (more) 4`s are noted light sleep is presented in the app). Maybe it is helpful for others investigating the bytes. I color coded my light/deep sleep sections in the excel sheet as well.

The second byte from my perception is raw unfiltered sensor data. But I am not confident.

Maybe we jointly can reverse the information stored in this raw structure and enable therefore detailed results for each day and not only for the last night or the steps of the current day. From my perception, it is feasible.

mritsch
[General Accessories] joshmosh:
Please find a German localization in the attachment.

thank you very much indeed for all your efforts and your really good work

Joshi
[General Accessories] xmxm:
Well, it seems that iOS database does not have application locale setting that is needed for analytics report. Please place attached app_locale.js into directory with all reports.
[General Accessories] xmxm:
Thank you. I've added German localization to the version 5.60 of analytics tool along with app_locale.js so it would work with data extracted from iOS.
[General Accessories] xmxm:
Hi,

Thank you very much for your investigation. I actually did my own investigation some time ago and also saw that the length of binary data of 4320 bytes corresponds to 1440 minutes but in my case I saw third byte being 0 for many initial records and ignored it completely, not checking it throughout the file.

I did some research on the first byte and came to similar conclusions - values 5 corresponds to sleep with almost no movement and 4 corresponds to some movement in sleep, 1 most likely correspond to walking, some higher values might correspond to walking as well (or maybe running) and 7E corresponds to no data. I assumed that Byte #2 represents acceleration activity and that number of steps could be retreived from there and wasted much time trying to decode it until I switched to other tasks.

Converting sleep to values to Light/Deep sleep looks like a bit complicated task - switches between light/deep modes almost always happen during value 4, however it does not always switches on value 4 - it seems to do some averaging (or runs some high pass filter). For example if the current mode is deep sleep (most values are 5) and then there are multiple 4's coming through, the mode would switch to light sleep at the first 4 and will stay at light sleep until it discovers a long sequence consisting of all 5's.

I could not come with a good implementation of such filter using SQLite language and especially was not able to match Mi Band application logic, so I tried a different approach: I started to average data on each hour - I defined a threshold and if amount of 4's in that hour was below theshold, I would declare that hour to be Deep Sleep, otherwise I would declare it to be light sleep.

This is where I got stuck, because I thought that 3rd byte was always 0 and tried to extract number of steps from step #2, which was not giving me good match to what Mi Band application was showing. I later got pulled away with analytics and Google Fit syncrhonization, but now, thanks for you doing the same investigation and finding that byte #3 is representing steps, I think that we could work on decoding raw data properly. I am attaching a sql code to extract raw data from blob for the last day and parse it into separate bytes, so I hope it would be useful if you want to do any additional investigation. If you would like to get the data for all past days, just comment out

and b.id = (select max(id) from date_data)

clause at the end of script, however be aware that it would produce huge file.

I also need to come up with a way to properly represent this huge set of data. I was initially thinking of adding an extra column to CSV file which would be a string of 24 characters. Each character would represent and averaged activity over that hour: Deep or Light sleep or Walking, where amount of steps for walking would be represented by values from 0 to 9, so it would look something like

ddLddL...127....3....Ldd

but I am not sure if this is a good way to represent the data. If you have any ideas, please let me know.
[General Accessories] joshmosh:
Wonderfull, thanks a lot !

Everything now works well. There a two minor quirks I found in my localization (some fields are a tad too long). Later this day or this evening (my timezone) I will send an update.

Thank you again and have a nice day

Joshi
[General Accessories] xmxm:
Thanks to the boost that mritsch post provided on step counts data, I've now updated data extraction package with experimental raw extraction functionality. It is disabled by default, but can be enabled by setting ExtractRaw parameter to Y in run.bat. Please note that it could produce quite a big file and that total daily count of steps may still be slightly off, but for now I think it should be good enough.
[General Accessories] joshmosh:
xiaomi in their "wisdom" :( decided not to provide this (in my view) very interesting data in their iPhone version - sh**

I noticed that in my German localization there is quite some room for improvement. I will try to come near the high standards xmxm has set, so bear with me. II need a bit more time for testing. Thank you for your patience.

Joshi
[General Accessories] xmxm:
iOS version has this data too. I've updated post #1 with version 0.2 of iOS extraction tool that now contains raw data script specific to iOS. You would need to change ExtractRaw flag to Y in runIOS.bat to enable raw data extraction.
[General Accessories] xmxm:
Sorry, I do not have unix environment so I am coding blindly here. Could you please try this version? If you still get error in backup data phase, could you please completely remove lines 21-27 and try again?
[General Accessories] joshmosh:
Very good, thank you. I will give it a try.

Please find attached a new version of German localization, only a very minor update. I also noticed that there seems to be no localization for the terms "Sleep Goal" and "Activity Goal". This is of course a very minor issue.

Have a nice day and thank you for keeping up the good work

Joshi
[General Accessories] joshmosh:
Thank you mritsch and xmxm for providing the raw data stuff. This looks extremely interesting ! A hourly, daily, weekly, monthly profile of your activities and sleep periods, with minute granularity. Oh boy, this is going to be fun, fun, fun ! If I only had more time - already three fun projects in parallel and now number four ...

Thanks a lot

Joshi

---------- Post added at 08:37 AM ---------- Previous post was at 08:04 AM ----------

Me again - I am really excited with this raw stuff :)

xmxm, wouldn't it be even more useful to extract the raw data into a sqlite db instead (or in addition) to .csv ? The sheer amount of data could be handled (sorted, accumulated etc) much better with sqlite tools. Would it be possible to fill the raw data in a sqlite database ? Not the binary, but the converted text. I could of course take the csv file and fill it into a sqlite db, but perhaps it is easier and more elegant to do it right at the source.

I hope I am asking not too much ...

Thank you very much in advance

Cheers

Joshi
[General Accessories] xmxm:
You can edit miband_ios_raw.sql and remove word "temp" in the first line - this would persist blobdata table inside original sqlite database.
[General Accessories] joshmosh:
Great, thank you. Now this will give me quite a bit to chew on :)

Cheers

Joshi
[General Accessories] joshmosh:
Good morning,

I have tried to make more sense of it, but there are still questions:

time and steps is pretty obvious, but what is the meanig of the field RawActivity and RawSensor ? Any hint ?

In addition, the temporary sqlite DB is not what I meant in my previous posting. I would like to have a new sqlite database with fields that are equivalent to the fields in the raw csv file:

Fieldname Type

time DATETIME

activity TEXT

steps INTEGER

rawact INTEGER

rawsensor INTEGER

To replace .mode csv by .mode sql did not work. I have to admit that my sqlite knowledge leaves much room for improvement.

Onece again, thank you so much for your help.

Cheers

Joshi
[General Accessories] xmxm:
I am not sure what are you planning to do with this table - if you are planning to pull the data somehow from SQL, you could use the last query from miband_raw.sql which returns the data in format you specified. Anyways, here is another script - it would populate resulting data in the format you specified into table called dataraw in the same database.

As for RawActivity and RawSensor, these two columns are representing raw data as it came from the sensor. RawActivity is the column from which exact information on activity (Walking/Running/Light Sleep/Deep Sleep) could be derived, however the exact logic is still not fully known. RawSensor might represent raw acceleration data reported by Mi Band, but it is yet unknown what we could do with this data.
[General Accessories] joshmosh:
Since I am not yet sure what I really want to do with it, I just want to have the flexibility which is provided by a database.

Thanks a lot for your help. I guess this is what I think that I need.

Cheers

Joshi
[General Accessories] olivercervera:
Hello,

I synced successfully data from smartphone to desktop, however I cannot get it synced in Google Fit.

What I get is the following, however I tried several times...

The token does work as I can use fit listds and returns a list of values, but the sync returns this error.
[General Accessories] xmxm:
It might mean that there is something in data that is causing json file to be generated incorrectly. Could you please PM me your database file or extract.csv?
[General Accessories] olivercervera:
Sure, sending PM now. :)
[General Accessories] cliq:
can I please know why there is no data from all sleep displayed in the stream tab on fitnesssyncer.com?? Check pic
[General Accessories] cliq:
can you tell me how I can get this desktop application to work simila way?
[General Accessories] xmxm:
Desktop application can use "adb backup" to run a backup for MiBand app and pull the backup over to PC. Generally, backup operation does not require root and works on many devices. Tasker profile is trying to copy files over from MiBand data directory and you need to have root privileges to do so.
[General Accessories] xmxm:
I cannot say from this screenshot - there are quite a few things to check:

1. Was it ever syncing sleep data properly before and then just stopped or it never worked?

2. If it never worked before, have you configured a separate sync profile for sleep data?

3. Do you see columns 2,18,20,21,22,23 being populated properly in your extract.CSV file?
[General Accessories] mritsch:
sleep phase extraction

Yesterday evening I took a look at the sleep phase extraction processing that is used by the app to summarize the raw activity data (5 for non REM sleep and 4 for REM sleep) to specific sleep phases. Therefore I investigated the Mi Fit APK file. From my perception the corresponding class is located in cn\com\smartdevices\bracelet\analysis\DataAnalysis.java

I do not want to share that Java file here because of legal and IPRs.

When you extract the APK file and do some decompiling you can retreive that mentioned file by yourself (Virtuous Ten Studio may help). The given procedures executed by the original app are not easily transferable to the SQL tool chain xmxm utilized.

mritsch
[General Accessories] Bonoboo:
del
[General Accessories] SotongDJ:
I can't sync data to google fit

C:\WINDOWS\system32>cd C:\Users\David\Desktop\miband_fit_02

C:\Users\David\Desktop\miband_fit_02>fit

MiBand sync to Google Fit v0.2

Open (I can't post link) in your browser and paste Token v

alue here:

Token:ya29.L(My fit auth key)0Ng

1324438 is empty, creating it.

Error in fit\post_segment.log file: "Bad Request".

Reason: "invalidArgument"

C:\Users\David\Desktop\miband_fit_02>

post_segment.log:

{

"error": {

"errors": [

{

"domain": "global",

"reason": "invalidArgument",

"message": "Bad Request"

}

],

"code": 400,

"message": "Bad Request"

}

}
[General Accessories] xmxm:
For anyone experiencing this type of error, please try downloading attached zip file and unpacking it to your fit\ directory - it should overwrite 1324438.json with a new version which should (hopefully) fix the issue.
[General Accessories] olivercervera:
It worked, great! Many thanks!
[General Accessories] xmxm:
Thank you for confirming this. I will add it to the next release of Fit Sync
[General Accessories] xmxm:
Google Fit Sync package updated to v0.3 in post #1. Changes include:

Updated 1324438.json - should fix sync issues that some users had.

New FitTools.bat - this utility allows retreiving various information from your Google Fit account:

listds - Show all data sources

listdata - Show data for specific data source

deletedata - Delete data for specific data source. Note that you might not be able to delete data from data sources not created by Mi Fit Sync application

backupdata - Create backup directory and extract definitions and data from all data sources in your Google Fit account.

loaddata - Loads data from .json file. Note that currently you can only load data to data sources created by Mi Fit Sync application. This means that there is no full restore datasets functionality - you could only restore some datasets (created by Mi Fit Sync app) through this option.

Please note that I will not be able to provide support on FitTools.bat - any use of it would be is at your own risk!
[General Accessories] xmxm:
Thank you very much for finding the code segment that does sleep analysis. I've looked into sleepAnalyze and it seems to have a lot of phases to do data processing. Since jad/fernflower do not recover variable names during decompilation, tracing the logic is not easy. I'll keep looking to see if I could extract at least some logic from that code.
[General Accessories] Devo7v:
This is great work and I would absolutely use an Xposed module/app if you ever created one. I'm not sure if it's possible but I think it would be amazingly useful if you could get some sort of API so that other apps could make use of the app. I don't think Xaomi will ever put out a public API so other apps can control the band, but it looks like you might be on the way to doing just that. Keep up the great work.

EDIT: I'm not sure if this would be helpful, but it looks like somebody is working on opening up the firmware and you might find some useful information. (https://github.com/KashaMalaga/XiaomiMiBand)