yashsingh1/xdaforums huggingface dataset:
[General Accessories] xmxm:
Thank you for your excellent analysis and description of calculation logic!

I've tried to implement this logic (except filtering) into updated miband_raw script. To test it you'd need to unzip miband_raw.sql and health.sql into db\ directory and then edit health.sql and update Height and Weight values to your height in centimeters and weight in kilograms and then run data extract.

You can run direct raw conversion without pulling data from smartphone by running following command:

bin\sqlite3 db\origin_db <db\miband_raw.sql

This run would create both extract_raw.csv and extract_raw_summary.csv which contains Run DIstance and Run Calories data aggregated by day - so you can compare it with daily data on your Mi Fit program. On my data, numbers would not match exactly but are still pretty close, which I think is a great progress.
[General Accessories] xmxm:
I've updated miband_extract to version 5.90 in post 1. The new version includes updates to Raw data extraction logic - with the help of information provided by Osid, the application can now parse walk distance and calories and run Distance and calories from raw data. Values in activity description column is updated to distinguish between Walking and Running activities.

As a reminder, to enable Raw data extraction, edit run.bat and change ExtractRaw to Y and also set Height and Weight to match the values you've set in Mi Band application on your smartphone.

I've also added logic to create one more file - extract_raw_summary.csv - this file contains daily sums for colums calculated from raw data so you can compare values from this file with values in regular extract.csv to see how close these metrics from raw data are matching summary values provided by Mi app. For my data, the values in extract_raw_summary.csv and extract.csv are pretty close (although not matching exactly), so I am happy with this result.

If you see big discrepancy occuring for your files, please let me know and send me some sample data for troubleshooting.
[General Accessories] corpcd:
Hi,

I'm testing the app with a Meixu mx4 with Flyme, rooted, usb debugging on under Linux Mint 17.1

I use the ForceBackup=Y option though the mobile is suposed to be rooted. The SDPATH is set to sdcard, which exists in the device. I've also tried /sdcard/Download to be sure I have rights to modify that folder.

When i run commands at terminal, i create the file mi.tar with no problems, I can open that file normally with file manager, but looks like the script doesnt recognize it as a tar file.

When i use the ForceBackup=N option I get another error about -c option to adb is not valid (the line that is sent to the phone with adb "su -c ...").

Any clue on what's happening?
[General Accessories] mritsch:
Log file

Hi all, I have another source of data in my device memory directory of my device (Moto X 2nd gen) (file named mili_log.txt in the folder \sdcard ).

In that file there is

- detailed sleep phases as filtered as in the app

EXAMPLE:

2015-03-16 20:20:15.267 SleepInfo

BaseInfo:

this.startDate:Sun Mar 15 23:19:00 MEZ 2015

this.startOnBedDate:Sun Mar 15 23:18:00 MEZ 2015

this.stopDate:Mon Mar 16 06:27:00 MEZ 2015

this.stopOnBedDate:Mon Mar 16 07:05:00 MEZ 2015

this.minutesOfRem:122

this.minutesOfNonRem:306

this.sleepCount:428

this.awakeCount:0

this.awakeNum:0

this.userSleepStart:-2147483648

this.userSleepEnd:387

this.nosleepReason:0

this.hasSleep:true

this.stageSleep:11

StageSleepInfo:

this.start:1398,this.stop:1398,this.mode:7

this.start:1399,this.stop:1457,this.mode:5

this.start:1458,this.stop:1473,this.mode:4

this.start:1474,this.stop:1540,this.mode:5

this.start:1541,this.stop:1577,this.mode:4

this.start:1578,this.stop:1676,this.mode:5

this.start:1677,this.stop:1719,this.mode:4

this.start:1720,this.stop:1776,this.mode:5

this.start:1777,this.stop:1802,this.mode:4

this.start:1803,this.stop:1826,this.mode:5

this.start:1827,this.stop:1864,this.mode:7

- steps info

EXAMPLE

2015-03-16 20:20:15.264 StepsInfo

BaseInfo:

this.steps:5779

this.time:68

this.runtime:4

this.distance:3969

this.runDistance:299

this.walkDistance:3670

this.calories:216

this.runCalories:16

this.walkCalories:200

this.activeMinutes:72

this.continusActive:13

this.sSteps:144

this.activeList:8

StageStepInfo:

this.time:0,this.steps:0,this.count:1,this.distance:0

this.time:0,this.steps:0,this.count:0,this.distance:0

[..]

this.time:0,this.steps:0,this.count:0,this.distance:0

this.time:46,this.steps:23,this.count:1,this.distance:14

this.time:47,this.steps:434,this.count:5,this.distance:274

this.time:48,this.steps:17,this.count:1,this.distance:10

this.time:49,this.steps:104,this.count:1,this.distance:65

this.time:50,this.steps:100,this.count:1,this.distance:62

this.time:51,this.steps:10,this.count:1,this.distance:6

this.time:52,this.steps:97,this.count:1,this.distance:60

this.time:0,this.steps:0,this.count:0,this.distance:0

this.time:0,this.steps:0,this.count:0,this.distance:0

this.time:0,this.steps:0,this.count:0,this.distance:0

this.time:0,this.steps:0,this.count:0,this.distance:0

this.time:57,this.steps:186,this.count:2,this.distance:118

this.time:58,this.steps:45,this.count:1,this.distance:28

this.time:0,this.steps:0,this.count:0,this.distance:0

this.time:0,this.steps:0,this.count:0,this.distance:0
[General Accessories] corpcd:
I found the problem... or problems...

First, openssl must be installed with zlib support in ubuntu, by default, it isnt, so I had to install version 1.0.2 with zlib option.

Second, there was an error with WITH in miband.sql query. The WITH command was introduced in version 3.8.3, in oficial repositories the lastest release is 3.8.2 version, so I had to instal libsql 3.8.6 (i386 and amd64) and then sql 3.8.6

With those updates, now the script works.
[General Accessories] xmxm:
Thank you, I started to write you an answer on original post, but then saw this one. I'll update pre-requirements with this information and minimal versions requirements.
[General Accessories] xmxm:
Thank you - this is quite an interesting find. The file on my phone only contains sleep information for last 3 days (so it does not have all the historic information), but it may still be useful to validate sleep decoding logic if we come with it.
[General Accessories] corpcd:
To your comments regarding Linux requeriments:

Actually, I think I did read somewhere that zlib support begun with openssl 1.0, so the requeriment should read openssl 1+ with zlib support.

Also, there are alternatives to openssl to extract that zlib file, you mention python suggesting to comment the openssl line to use the python alternative, but no update has been made to the 5.90 version of the script (at least, the linux version), so no way of using that alternative. I havent investigated myself in detail how to use python to decompress zlib, though.

Great work!
[General Accessories] fletcheroony:
Syncing with HTC One M8

I am have a Verizon HTC One M8 running 5.0 (no root access). When I try to extract the data, I get the backup screen on my phone and click yes, but no extract.csv file is generated.

Any idea what I might be doing wrong?
[General Accessories] signaleleven:
Damaged DB

Hello,

during a phone reinstall, the DB got somehow damaged.

2 days are now empty (or with weird data) and the output of the program (i use bonobo's tasker task, but nevermind) shows data only until the 16th of march.

Is there a way to repair the DB? I attach my DB, the affected days are 17 and 18 of march.
[General Accessories] cliq:
Thank u!

1. It has never synced before. This was my first time using this

2. no, see 1

3. Yes they all populate nicely

Any help?
[General Accessories] Osid:
You can repair DB only if you can recreate ActivityData.

How? If you have logfile mili_log.txt you can extract them from data property of JSON Uploadata (base64 encoded)

Try to search for lines in log with

xxxx-xx-xx BleTask <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

xxxx-xx-xx BleTask ============================stop sync data================================

xxxx-xx-xx BleTask <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

After those there's JSON with data property

Keep in mind that you must reconstruct also indexs.

Otherwise you reconstruct them for ActivityData logged during sync of bracelet.

Lines are in this form:

xxxx-xx-xx xxxx xxx currentDataLen: 27,currentDataTotalLen:27

xxxx-xx-xx xxxx MiLiProfile <hex ActivityDatas>

When you have correct byte[] of activityData, you must update column data of date_data table in DB for the corresponding day and rerun DataAnalysis.

You can accomplish this using Xposed and Mi Fit app.
[General Accessories] signaleleven:
Ouch. That's the deal breaker for me. I use bonobo's script that monitors wake up time in tasker, and that wipes mili_log every morning.

I guess I will have to wipe the db and start from scratch then :(

I opened the DB with a sqlite client, hoping that I could just put "zeroes" on those days, but it's a bit more complex that I thought, and I have limited time.

Thanks for the info though, I will make so that my mili_log.txt is backed up every day before wiping!
[General Accessories] xmxm:
Are you running on Windows or Linux/OSX?

Check the log file and see if any errors are being listed there.

For Windows: Go to run.bat and look for "del mi.ab", "del mi.zlb", "del mi.tar" lines and remove them. Run sync again and see if these files are generated.

For Linux/OSX, ensure that you have minimal needed versions of sqllite and openssl installed - see details in post #1
[General Accessories] xmxm:
I just went to fitnessync and removed all profiles and created sleep profile from the scratch and I see my sleep time uploaded:

Please go to my account, and click on Edit for your sleep Source profile. There is a bug in the interface so it will show wrong columns, so you would need to change Type in dropdown from Sleep to Activity and then again to Sleep. Check that your columns settings are matching mine:

If it still does not work, you can PM me your extract.csv and I will try to see if there is anything wrond with the data in it.
[General Accessories] xmxm:
I could think of 2 options. Both options require your phone to be rooted and being able to back up your Mi Fit databases just in case. I use Titanium Backup for backups, but there are other tools as well. Make sure that you are familiar with backup and restore process before proceeding.

Please note that this is just recommendations - I did not test it with my data, so it may or may not work for you (that's why you would need backups).

First option - just delete empty rows on your smartphone (needs to be rooted):

1. Backup your existing Mi Fit databases.

2. Install aSQLiteManager (free from Play Store).

3. From inside aSQLiteManager, open /data/data/com.xiaomi.hm.health/databases/origin_db database (your phone has to be rooted for this).

4. From list of tables select date_data table. Once table opened, click on Data button

5. Find rows you want to delete (you may use PgUp/PgDn buttons). Once you found row you want to delete, click on "Edit" label to the left of the row and then click "Delete" on the next dialog box.

6. Exit the app saving the database if needed.

7. See DB problem is solved. If not, restore from the backup.

Second option - resetting DB but loading old data:

1. Backup your existing Mi Fit databases and copy original origin_db file to your desktop.

2. Use SQLite browser to export all database into SQL file. I am using DB Browser for SQLite: https://github.com/sqlitebrowser/sqlitebrowser. When using it, go to File -> Open database to open your baked up origin_db database and then use File -> Export -> Database to SQL file... to save all the data into SQL text file.

3. Open exported SQL file from step 2 in the notepad. Delete all the lines from beginning of the file up until the first line that says

INSERT INTO `date_data` VALUES ...

(in my case that is line 10).

4. Scroll down until the last line that inserts into 'date_data' and delete everything after it.

5. within inserts to 'date_data', find lines that you want removed and delete them.

6. Save updated file.

7. Wipe and reintialize your DB on smartphone and confirm that new database gets populated properly.

8. Once it is working properly, copy new origin_db file to the desktop.

9. Open origin_db in SQLite browser and use File-> Import -> Database from SQL file. Select file from step 6. Select No when asked whether to create new database file.

10. Once import is complete, you would need to upload updated origin_db and origin_db-journal to your phone and place them to /data/data/com.xiaomi.hm.health/databases/ directory.
[General Accessories] xmxm:
I forgot to update linux script in 5.90 with newest version - since I have no easy way to debug Linux shell scripting, .sh are usually lagging behind Windows .bat versions. Please try this file - I've also added support for Raw data extraction to it. If that works properly, I will include it in 5.91 version of extraction package.
[General Accessories] corpcd:
Python command seems to work well, maybe you should use another variable "ALTZLIB=N" (or Y).

With the new script now it is mandatory to specify the "android-sdk-linux/platform-tools" path in the ADBPath variable. If you have it in the bashrc file is not necessary, but it's ok.

Also there is an error in the raw extraction routine:

echo $"INSERT INTO _PersonParams (Height,Weight) VALUES ( "$Height","$Weight");" 2>&1 | tee db\health.sql

Should be:

echo $"INSERT INTO _PersonParams (Height,Weight) VALUES ( "$Height","$Weight");" 2>&1 | tee ./db/health.sql

What is that file for? It is not used anymore after its creation.
[General Accessories] Honusnap:
HI all,

I recommend to add "-d" after each adb command ... :D

"http://http://mefit-run1-857.appspot.com/" => Remove http://

Work good, need to check datas.

I got problem with the Mi Fit application, syncronisation with the Band show that some datas are missing, last 3 days i got no sleep time WHILE your extract see sleep time ... like i supposed it's not the band that is faulty but their application.
[General Accessories] xmxm:
Than you for your comments: I've removed extra http:// in fit.bat. I will see how -d parameter could be accomodated - I doubt it would not work with ADB over WiFi.
[General Accessories] xmxm:
INSERT INTO _PersonParams is used to pass Height and Weight parameters from batch file to Raw data extraction SQL. It really needs to be done once (or values could be edited directly in SQL scripts), but I wanted to keep all parameters in a single place.
[General Accessories] corpcd:
Oh, I see, health.sql is called by miband_raw.sql query.
[General Accessories] offwhite:
thanks
[General Accessories] kaefert:
Hi there!

I've got an mi band now to, and I'm would like to get "raw data" from it. "raw data" in the sense as the most detailed I can get. So at least the 10 minute windows that the app shows should be possible I guess.

I am a linux desktop user and I am running the mi band app on my rooted sony xperia z.

I started my experiments with the run.sh from miband_extract_v590.zip

I've made a few adoptions and came as far as to get data with daily statistics, and I've found that the sqlite database "user-db" contains the detected "training-sessions".

Now I've found that you uploaded a new run.sh a few posts ago and you try to load adb from ./adb now --> just use adb without any path, any modern distribution has the adb tools as a package and in the run path.

Also the backup stuff doesn't seem to work for me, but by setting ForceBackupMode=N I can get the database.

Next problem I see is: my sqlite interpreter says your sql script "miband.sql" has an error:

Error: near line 88: near "with": syntax error

Next thing: "tee db\health.sql" doesn't do what you hoped, the backslash is simply interpreted as escaping the h, therefore it ends up in "dbhealth.sql". You want a slash instead

update: okey I see now the "with" syntax error is my fault, my sqlite is too old...

3.8.2 2013-12-06 14:53:30 27392118af4c38c5203a04b8013e1afdb1cebd0d

update2: I've downloaded a newer sqllite binary from here:

https://www.sqlite.org/2015/sqlite-shell-linux-x86-3080803.zip

and with this I am able to make your script produce a extract_raw.csv file with 1 minute windows! really nice! I guess for true "raw" sensor data one would need to hack the mi band firmware and modify it to send it via bluetooth and also hack together an receiver for that data on the android end.

I've uploaded my latest variant of your run.sh script as an attachment to this post.

Thanks for your great work! really nice how much data I could get with so little effort!

I wonder - do you know how to interpret the column "RawSensorData" that has one integer value between 0 and 156 for each minute (at least for me so far)?

And or maybe the "RawActivity" which gives me between 0 and 38 so far?
[General Accessories] jblazea50:
Thanks for doing this.

Quick question, would I need to extract the data daily, put it in the dropbox folder, and add new source to fitnesssyncer?
[General Accessories] renton82:
Same here! The only way is to use "grant new permission", but "awakenings" still remains to 0! Why???

If i open mi_data.html all is diplayed correctly...
[General Accessories] kaefert:
Where can find the sleep report that is shown in your 3rd and last picture from the first post of this thread?

UPDATE: found it. its selectable in mi_data_byDoW.html by choosing it in the dropdown box (sorry for not looking harder before posting the question)
[General Accessories] edd astorian:
Hi,

I encounter a problem with FitnessSyncer and Google Fit I followed to tutorial in the beginning of the topic) .

In FitnessSyncer, the datas in graphs are multiplied by 10 (I use the metric system) but the FitnessSyncer csv seem ok.

I have every datas days in Google Fit but no data for them.

Known issue ?

Do you have a clue for that ?

Thank you.
[General Accessories] johnny1176:
Hello,

i have a non rooted Huawei P7 and cannot use the adb backup function.

So i tried to make a backup with helium (some mentioned this somewhere in the forum).

i get the backup but the mi.ab file doesn´t work.....

here is what i get:

unpacking backup file

miniz.c version: 9.1.15

Mode: d, Level: 9

Input File: "mi.zlb"

Output File: "mi.tar"

Input file size: 6358480

Total input bytes: 6358480

Total output bytes: 6357504

Success.

bin\tar: Hmm, this doesn't look like a tar archiv

bin\tar: Skipping to next file header

bin\tar: apps/com.xiaomi.hm.health/db/origin_db*:

bin\tar: 2: Not found in archive

bin\tar: Error exit delayed from previous errors

maybe someone has a idea what to test?

thanks for you help :o

cu

Alexander
[General Accessories] xmxm:
I am back now. Sorry, I had few busy weeks at my work and was not able to work on the app. I've created a new shell script for linux which should combine all these recommendations and others that I've received directly - you can try and see if it works better for you. If it works for everyone, I will include it in the next build.
[General Accessories] xmxm:
It is difficult for me to troubleshoot FitnessSyncer problems as I've seen in the past that files that were not loaded correctly for other people worked fine when I've loaded it in my fitnesssyncer account. I can try following: please PM me your CSV file along with screenshots showing incorrect data (or data missing). I'll try to load it through my fitnesssyncer account and see if I could reproduce the issue.
[General Accessories] xmxm:
I cannot promise that I could fix FitnessSyncer problems but you can PM me your CSV file and I'll try to load it through my fitnesssyncer account and see if I could reproduce the problem.
[General Accessories] xmxm:
Generally speaking, yes. It is up to you how frequently you want to put new file to dropbox.

Please note that some people on the forum are having some issues with fitnesssyncer, which I was not able to reproduce with my account, so your mileage may vary.
[General Accessories] xmxm:
Have you tried setting ForceBackupMode to Y in batch file?

I am not sure which format Helium is using (it may be not be the same as native android backup) - if you PM me your backup file, I'll have a look at it.
[General Accessories] xmxm:
RawSensorData is the raw data - my guess would be that it is proportional to the amount of movement that accelerometer tracked during a minute, however I do not know exact logic on how to interpret it. You can try decompiling original MI program (there were links to specific classes that do data parsing few pages back in the thread) and see if there is anything in the code that may give any hints.

As for RawActivity, here is heuristic that I have used in my scripts:

0 - Idle (most likely)

4 - Sleep (likely deep sleep)

5 - Sleep with movement (not exactly light sleep as light sleep phases are calculated as combination of deep sleep with high number of "sleep with movement" events

15+ - Running (this is how my logic is currently interpreting it)

126 - No data (these elements are filtered out during raw extraction)
[General Accessories] agent00f:
I have the following problem, phone is motoX 2013 on 5.0.2

miband_extract_v590>run

The system cannot find the file specified.

The system cannot find the file specified.

* daemon not running. starting it now on port 5037 *

* daemon started successfully *

error: device not found

error: device not found

error: device not found

error: device not found

Press "Backup My Data" button on device...

* daemon not running. starting it now on port 5037 *

* daemon started successfully *

adb: unable to connect for backup

Extraction failed

The system cannot find the file specified.

The system cannot find the file specified.

--------------

log file:

Extract Started

Sun 04/12/2015

Renaming

ADB SU copy to sdcard

ADB pull

Cannot find database files. Non-rooted phone? Attemting backup approach

unpacking backup file

miniz.c version: 9.1.15

Mode: d, Level: 9

Input File: "mi.zlb"

Output File: "mi.tar"

Input file size: 0

inflate() failed with status -5!

bin\tar: apps/com.xiaomi.hm.health/db/origin_db*: Not found in archive

bin\tar: Error exit delayed from previous errors

deleting temp files

The system cannot find the path specified.

Still cannot find files. Restoring original files

=========

log_adb:

The system cannot find the file specified.

The system cannot find the file specified.

The system cannot find the file specified.
[General Accessories] xmxm:
Looks like adb cannot connect to your phone. Are you able to connect through ADB manually, like adb shell ?
[General Accessories] agent00f:
Thanks. Debug/adb drivers never installed correctly for the phone.
[General Accessories] netyang:
Thank you very much for your work. It work great!.

How many days the data can keep in extract.csv files?

Thank you.
[General Accessories] netyang:
By the way, I think the sleep in minutes should use column 3 in the FinessSyncer settings.
[General Accessories] bews:
I've tried to use the sleep settings for FitnessSyncer from the 3rd post but I just can't seem to get it to sync sleep at all. Does anyone here have it working? If so, would you care to share your settings with me please?
[General Accessories] Bonoboo:
I discovered, that raw data stored only locally and not syncing with cloud.

If logout database wiped and after log in there no raw data for previous days.

Seems previously it synced, but now - no.