[Android General] smtddr:
UPDATE#11

Research has taught me that the mmc thing is a kernel module (specifically linux/source/drivers/mmc/card/block.c) and if I want to update it, I need to modify the kernel image. Looking around, it appears that nobody really does that... what they do instead is simply compile from source using the config from the phone. So, I got boot.img then using mkboot command split the boot.img file into ramdisk and kernel. Using binwalk, found where the gzip part of the kernel was and gunzipped it, giving me an uncompressed kernel. Searching this uncompressed kernel image again with binwalk, located another gzip within. gunzipped that and I got the Kernel config. Comment at the top said "Linux/arm 3.10.28 Kernel Configuration", so I went to kernel.org and downloaded the source of kernel 3.10.28. In the downloaded linux source's directory, I copied the kernel-config I got from the kernel image and placed it in this dir as ".config" so the kernel would compile with the right options. I left everything else as default when asked. Wouldn't build because of some line containing __devinit but various googling for the error and I discovered some kernel devs actually submitted a patch to remove it, so I removed it from my source. Then it failed to compile because of some missing firmware blobs. PR1593801-s3203_n_dsx8232_JTOUCH.img and PR1593801-s3203_n_dsx8232_TTOUCH.img.

What I did then, was create a 250 byte file containing only the number "8" over and over again, then another file containing the number "9" over and over. Named them the above JTOUCH and TTOUCH images respectively and compiled the kernel. I then used a hexeditor to examine where in the uncompressed kernel image those 8s and 9s ended up. First, I noticed that the 2 files were concatenated together with no compression or encryption or padding or delimiting bytes in between. Then I noticed all the function names & bytes that appeared just before the 8s and just after all the 9s. I compared it to the kernel image from my phone and was able to deduce the general area of the 2 firmwares. I then notice a block of function names that didn't match anything else in the file, a block of functions starting with "msm8x16_wcd_*" then suddenly a block of functions starting with "wcd_mbhc_*". I concluded to extract this area of the kernel image and split on those function names to create the firmware images. The cool thing here is, even if I'm wrong on the split since they're concatenated together with no delimit mark... it didn't really matter where I chose to split them as long as I just don't misjudge the start of the first firmware and end of the 2nd. Or I could be wrong about this and somewhere else in the kernel the offset and length of the firmware is stored and referenced during bootup.

So then I "make clean" and rebuilt the kernel.

ARCH=arm SUBARCH=arm CROSS_COMPILE=arm-linux-gnueabi- make

For this you gotta be sure you have arm-linux-gnueabi-gcc on your machine.

Then using mkbootimg --kernel /path/to/newly/built/zImage --ramdisk /path/to/old/ramdisk/extracted/from/boot.img/ramdisk.gz --dt /path/to/old/extracted/dt.img, created a boot.img containing the newly compiled kernel and the old ramdisk & dt.img

.....and..... it would have been amazing if this had worked, but of course it failed to boot, because I have no idea how to generate another dt.img that this phone needs and apparently using the old one from the boot.img I got doesn't work. I don't even get a chance to "adb shell logcat" or "adb shell dmesg" to see what went wrong. The phone goes into a fast reboot cycle. The while BLU logo screen appears for about a second then the screen goes blank and phone reboots, over and over. Maybe BLU has custom kernel modifications for the phone, who knows. I would have like it to boot up even if wifi, camera and all kinds of stuff was broken.

UPDATE#12

The size of the firmware is indeed stored in the kernel. I did a bunch of tests changing the size of the 2 fake imgs and I kept finding the little-endian representation of the sizes next to each other, always matching and just about in the same spot. i.imgur.com/smahbf4.png, so now I'm trying to find this same area in the real kernel. I've also noticed that I was sorta wrong about the no delimiters between the firmwares. Sometimes there is, sometimes there isn't. Through many tests increasing/decreasing the length of the function names that appear before my fake firmware as well as changing the size of the firmware itself, the kernel appears to be maintaining some kind of 4-byte-alignment. There is always 2 nulls after the function name and then the first firmware starts, and the beginning of the firmware must always be at an offset divisible by 4. The compile process add/removes padding zeroes just before the function name to maintain these rules. Even when the 2nd firmware starts, if it's not a place divisible by 4 then zeroes get padded between the first firmware and the 2nd one to force the 2nd firmware to start at a place divisible by 4.

This was annoying at first, but I now realize that these rules significantly narrow down exactly where the firmware will be in the real kernel image and I can sorta verify my guesses by finding the sizes in the binary that match. I've also noticed that the area containing the image sizes seems to have the value 0xC0 at every 4th byte, as you can see from the image. I suspect this area of the image is some kind of table-of-contents for all the files in the image.

UPDATE#13

So, after a bunch of attempts at booting the kernel and the phone rebooting immediately. I began to suspect that perhaps the kernel is signed in someway and some SHA1/CRC/etc didn't match so the phone bailed out without even trying to boot. To test this theory, I opened up the original zImage-format kernel image extracted from the phone... went to the center of the file and changed 3 bytes(that were not zero) arbitrarily to something else. My thinking here is this should be enough to fail any kind of kernel-signing process but not enough to completely ruin the boot up process. I was happy to see that the phone still proceeded to boot up even with those 3 bytes changed. I didn't use the phone enough to find out exactly what I broke by altering, but this at least made me confident that the entire image isn't somehow signed which would mean there's no hope of me getting anything to boot on it besides the one it came with. Then I went to try some other ways of creating the zImage. First, I used binwalk on the original zImage to tell me when the gzip archive starts for extracting the kernel image. I used dd to create a file that containing all bytes _before_ the gzip header and called that file zImage_header_bytes.bin. I then took the arch/arm/boot/Image file from my own kernel build process, gzipped it, and appended it to the zImage_header_bytes.bin file, then made a boot.img from it. Phone didn't boot. Then, I noticed that my make file has a "Image" and "zImage" target. So what I did then is "make zImage", then deleted the uncompressed Image, then ran "make zImage" again. Noticed that the build process must first create an Image then do whatever it does to make "zImage". So, I did this again but I took the original uncompressed kernel image and copied it arch/arm/boot/Image, then typed "make zImage" again. The result was a zImage file that was bigger than the one the build-process normally made which told me it used the original uncompressed Image file to create the zImage. I then tried making a boot.img out of this and... it still failed to boot. I then went back to my original kernel extraction process:

The trailing garbage message reminded me that I actually threw away some bytes when retrieving the uncompressed image so now I'm working on figuring out the "footer" file, such that I can take my custom uncompressed image, gzip it and put the original header & footer on it. Though, if that were the case then I would have expected my trick of slipping in a different Image into the kernel build process to be made into zImage... would have given it the correct header & footer and should have booted up.... I dunno. Still trying. I'm convinced that, at the very least, I should be able to compile from source the same kernel that's already running on the phone and get the phone to boot up. Maybe it'll crash/freeze and I'll never get a chance to enter my pin, but I should at least be able to get past the initial white BLU logo and into the animated colorful video BLU logo where "adb shell" becomes available and allow me to look at dmesg & logcat for further errors to work on.

UPDATE#14

android.googlesource.com/kernel/msm.git/+/android-msm-dory-3.10-kitkat-wear , so I downloaded this kernel because it seemed much closer to the kernel already on the device. It has files that the kernel.org one does not. e.g., msm8916-sim.dts & msm8916-smp2p.dtsi because in my phone's settings screen the processor info says MSM8916. Also, going into the sound directory and running "find . -name '*.c' -exec grep -E msm8x\|wcd {} \; | grep static" reveals pretty much all the function names that I see the extracted kernel occupying the firmware blob area. I now strongly suspect that those firmware blobs are more or less the result of compiling the files in sound/soc/codecs. So I went ahead and built this kernel. A couple of errors about missing header files, but it's really that they're in a different folder. So I had to copy around 3 or 4 .h files. Then there was a complaint about a multiple declaration of a function, I simply appended a "1" to the function name in .c file defining the function a 2nd time. At the end, there was a complaint: "drivers/net/wireless/wcnss/wcnss_wlan.c:808: undefined reference to `wcnss_rf_read_reg'", I don't know what to do about that so I just commented out and changed the code around there so it wasn't called. I'm sure that brakes wifi, but my goal was to just boot the phone up even if wifi is broken. I can fix that later. So I eventually got my zImage, and I used it and the old dt.img to build a custom boot.img and ....... this time it took the phone much longer before giving up and rebooting! It was like it was just about to load the animated-coloful-logo. It's not the kernel size either, this custom zImage and the resulting boot.img are both smaller than my other custom_boot.img where I only alter the ramdisk contents... and that one does boot up the phone just fine. This makes me think that the phone progressed further in the start-up process before running into a fatal error. The fact that so much msm8196 stuff is in this kernel makes me think it has a much better chance at working. It even has a target like this:

ARCH=arm SUBARCH=arm CROSS_COMPILE=arm-linux-gnueabi- make msm8916_defconfig

and unlike the kernel.org tar files, this one has arch/arm/boot/dts/qcom/msm8916*

I actually might try copying all the extra files from android.googlesource.com kernel into the plain vanilla one. The coloful animated logo has sound, so maybe trying to load the sound related stuff is why it crashed.

UPDATE#15

More progress! android-msm-angler-3.10-marshmallow-dr , doesn't crash at all. What happens is the while BLU logo screen appears then, very slowly fades to dark from the center out as if someone physically broke the screen. Like a black square slowly fades in at the center of the screen and grows larger until the whole screen is very dark greyish/black. "adb devices" and "fastboot devices" cannot detect the device. I have to hold the power button down for 10 seconds to force a power-down. This is good news because that means my attempts to boot a custom kernel are working. I might not know the exact configuration needed, but it's not a kernel-signing problem and it's not a problem with how I'm compiling and creating my zImage. The kernels are loading and executing, they just don't do the right thing. It wouldn't compile though without a few changes, I had to comment out the "tp_log_debug" and "tp_log_err" calls in hw_tp_common.c and in direct-io.c I had there was a function call that returned a value the code never used, "cmpxchg(&sb->s_dio_done_wq, NULL, wq)", the compiler gave a warning about it and then said something about some warnings will become errors due to compile flags somewhere. I just changed that code to do something harmless:

if(cmpxchg(&sb->s_dio_done_wq, NULL, wq)) {
wq = wq;
}

That way the return value of cmpxchg is being used in the if-statement and the "wq = wq" doesn't actually change anything. I just used a variable, "wq", that was declared earlier in the function. Oh and disable anything like CONFIG_EXT3 because stuff related to it gave compile errors. As far as I can tell from running the "mount" command in adb-shell, this phone only uses vfat, ext4 and "fuse". So yeah, there's hope! This kernel is 3.10.73 according to its Makefile.... I still really wish I could generate a dt.img from this source code. That dtbTool never works for me. Keeps saying "0 unique dtb" or something. I'm also getting a better idea of why I seem to be having better luck with these, h t t p android.googlesource.com/kernel ...the "msm" section has a description indicating it's for Qualcomm chipset which my BLU phone is definitely telling me in the Settings screen. My guess is BLU took this base kernel and made some changes perhaps. I don't see a 3.10.28-msm on googlesource.com. That would probably be the best thing to try.

UPDATE#16

More progress again! Now trying stuff with "android-msm-seed-3.10-marshmallow". This the only kernel were I only have to make a small one-line code change.

./kernel/sched/fair.c:static inline int select_best_cpu(struct task_struct *p, int target, int reason, int sync)

The compile failed because a declaration of this function was missing the "sync" parameter. Everywhere else in the file it had the sync value but I had to add it there. And in ./arch/arm/mach-msm/Kconfig the section "config PHYS_OFFSET" kept rewriting the .config PHYS_OFFSET to 0x00200000 even when I changed it to 0x80000000 to match the img_info I got from mkboot extracting the original boot.img. I had to add the line "default "0x80000000" if ARCH_MSM8916" so it would compile with the correct base address.

Also, Found this tool: /github.com/mypalmike/csplitb , that allows me to extract dtb files out of the dt.img that I got from mkboot pulling files out of the original boot.img. So now that I have a file called msm8916-0000.dtb in a dir called "dtbfiles", the command mkbootimg_tools/dtbToolCM -2 -o custom_dt.img -s 2048 -p k/android-msm-seed-3.10-marshmallow/scripts/dtc/ dtbfiles/ will produce a dt.img for the current kernel I'm compiling(3.10.49) and then I created a custom boot.img out of all this to attempt booting up the phone. I should note here it was important to use dtbToolCM, not the regular dtbTool. The regular will make a dt.img but when that's use to make a boot.img then "fastboot boot custom_boot.img", it'll complain "Failed remote: dtb not found". Only the dtbToolCM does it so that complaint doesn't occur. So after all this... I still get the growing fade-to-black square... but now I got a kernel that compiled with very minimal modifications and a dt.img that I believe matches the new kernel I'm trying to run. Now I just gotta think about what else I can look into. The phone doesn't have to work perfectly, just boot up enough that adb-shell works so I can look at logcat/dmesg for other error messages to work on.

Stay tuned!

UPDATE#17

More progress yet again! So I found out that the exact version of gcc used for a particular version of android are kept as static binaries on googlesource.com. Because binwalk on the original boot.img->kernel->extracted_gunzipped_kernel showed me the linux header and gcc 4.7, I decided to download that toolchain's tarball from "android.googlesource.com/platform/prebuilts/gcc/linux-x86/arm/arm-eabi-4.7/" to compile from now on. So I kept getting that fade-to-black screen. I looked carefully at my .config. Simply copying the .config I extracted from the boot.img into the kernel-source root works, but it asks me a ton of questions and rewrites stuff. I finally noticed one thing that looked important to me and was set by the new kernel "CONFIG_AUTO_ZRELADDR=y". The .config from the boot.img left this unset. When I changed it to "=n", the build failed with arm-eabi-4.7/bin/arm-eabi-ld:--defsym:2: syntax error. I reran the "make zImage" but this time like:

ARCH=arm SUBARCH=arm CROSS_COMPILE=../../arm-eabi-4.7/bin/arm-eabi- make zImage V=1

That V=1 makes it print out the exact commands it's running to do stuff, so I saw the problem:

../../arm-eabi-4.7/bin/arm-eabi-ld -EL --defsym _kernel_bss_size=1312864 --defsym zreladdr= -p --no-undefined -X -T arch/arm/boot/compressed/vmlinux.lds arch/arm/boot/compressed/head.o arch/arm/boot/compressed/piggy.gzip.o arch/arm/boot/compressed/misc.o arch/arm/boot/compressed/decompress.o arch/arm/boot/compressed/string.o arch/arm/boot/compressed/hyp-stub.o arch/arm/boot/compressed/lib1funcs.o arch/arm/boot/compressed/ashldi3.o -o arch/arm/boot/compressed/vmlinux

See how zreladdr has no value set to it? A search for zreladdr in all of the kernel source showed me arch/arm/mach-msm/Makefile.boot had a hardcoded list of various ZRELADDRs for different chipsets but MSM8916, for my phone, was missing. I googled "MSM8916 zreladdr" and found various Makefile.boot that did have MSM8916, set as 0x80008000. Great! So I added that value to my Makefile.boot and ran the make-command again, it built the zImage without a problem! ....but still, fade-to-black-graphic-corruption. I also toyed around with changing the ZRELADDR randomly and it definitely had an effect. If I make it 0x00008000 the phone would crash & reboot immediately. If I made it 0xA0000000 the phone would hang. When it's 0x80008000, it would do the fade-to-black. One of these 3 things would happen for random values of ZRELADDR. This really made me think my problems are related to having an incorrect ZRELADDR for this new kernel. From reading about it, I learned ZRELADDR is where the kernel gets copied to after it's decompressed somewhere else in memory. Corruption can happen if the place it's being copied to overlaps with other important memory. So I started thinking that maybe the value 0x80008000 doesn't work for this phone for whatever reason. Again I felt the need to prove to myself that this kernel is actually running. Since everyone out there seems to have it set to 0x80008000 I decided to leave the value as that and run make menuconfig, go into kernel-hacking and I noticed a "CONFIG_BOOT_PRINTK_DELAY", that'll slow down the each message being printed by the kernel by N milliseconds. N being what you give on the kernel cmdline, e.g. "boot_delay=250". If my kernel did get uncompressed and started running, then putting a boot_delay=250 should definitely delay when my screen fades to black. I went ahead an enabled the delay, added to boot.img-creation process the 250 millisecond delay and again attempt to run it. To my delight, the phone did take much longer before the fade-to-black occurred! Then I set the boot_delay=0 and tried booting the exact same custom_boot.img again. This time the fade-to-black was immediate. Excellent, so this kernel is getting unpacked and starts to run... prints out some messages... then something goes wrong. At this point, I'm sure professionals have a UART cable to do a serial-connection and actually see what the messages are. I'm sure something very helpful is in there, but I don't have such a cable.

I'm still thinking of what to do.... I feel like I'm close. Even if I don't ultimately figure this out I've gained a ton of knowledge in this quest. :)

Hopefully I'll be back with another update!

UPDATE#18

Further down the rabbit hole! So when I have display problems on my Linux PC, I usually have to do something like video=vesa on the kernel cmdline temporarily while I try to get some kind of proprietary video-driver-binary-blob to load. I just noticed that /proc/cmdline has more stuff in it than what was supplied when I assembled the bootimg using mkbootimg.

The only thing that the mkboot reported after extracting stuff from the original boot.img stops after androidboot.bootdevice. That's also the only stuff I give mkbootimg when combining the zImage, ramdisk and dt.img into customboot.img. Everything starting at androidboot.emmc is coming from... I have no idea. But the one thing that really caught my attention was qcom,mdss_dsi_otm1284a_720p_video! I never put any kind of value like that in my custom-kernel. Maybe that's the problem? To verify it, I ran the strings command on the uncompressed original kernel and sure enough the string was in that kernel image, but not in mine. Then, I searched the ramdisk and dt.img. The dt.img file also has the string in it! While looking around to learn more about dt.img, I discovered the command "dtc -I dtb -O dts msm8916-0000.dtb > ./msm8916.dts" will give me the human readable source; and it works the other direction too. So now I can go from dt.img-->.dtb--->dts and back again! I looked at the source and there was a huge section label "qcom,mdss_dsi_otm1284a_720p_video" with all kinds of stuff that definitely looked like it's describing how to control the screen. Hmm, so if the kernel is asking for a dt-entry that doesn't exist maybe the screen gets messed up? I know for sure my kernel doesn't have that string in it so probably whatever it's doing is wrong. I changed the name of this entry in the dts, then compiled it back into a dt.img and booted up the original boot.img hoping that now the name is changed, the original kernel wouldn't find it and the screen would fade to black. That would make me feel confident that the problem I was having is related to kernel & dt.img not matching screen-mode. Unfortunately, even with the name change the device booted up properly and the /proc/cmdline still showed the same normal-named video-mode. "Hmm..." I thought, then I noticed the width & height values. I changed the height from the original value(1280) to like 640. That worked! After the white-BLU-logo, at about the time the screen would fade to black for my kernel... original kernel started the animated-logo but it was half cut-off at the bottom by a big blue square and when the Android-UI showed up, all the icons and everything were shrunk down to fit in the top-half of the screen! OK THEN! So even though I changed the name, the kernel still found it. Next experiment, completely delete the entry from the dt.img. I did that...and the result was the screen faded to black after the white-BLU logo, just like my custom kernel does! So now I'm feeling pretty sure that my custom-kernel is requesting a video-mode not in the dt.img. The only place I see in the "make menuconfig" to supply this kind of info is CONFIG_CMDLINE, but the config file I extracted from the original boot.img does not use that. I then noticed an option for creating a "zImage-dtb" so I tried that but what it does is literally appends the .dtb file to the end of the zImage. I see the data in hexedit, but the kernel I got from the phone has that strings _AFTER_ it's been uncompressed. So I was expecting the dtb to be inserted into the Image AND THEN compressed into zImage-dtb. I tested it and zImage-dtb still doesn't boot my phone. Still looking around for another way to do this. If I can just push this custom-kernel to boot up enough for adb to kick-in, I can start actually looking at errors from dmesg, /proc/kmsg and logcat.

UPDATE#19

Step by Step!!! So after compiling my kernel and careful comparing of what I see in my hexeditor, I tracked down the file BLU-devs hardcoded that "qcom,mdss_dsi_otm1284a_720p_video" string in. drivers/video/msm/mdss/mdss_mdp.c . When I added a variable holding that string near the top of "static int mdss_mdp_get_pan_cfg(struct mdss_panel_cfg *pan_cfg)", my compiled kernel looked just like theirs in the same hex area. Maybe IDApro could disassemble this kernel and show me clearly what's going on, but I don't have that. What I do have is a fade-to-black screen. I thought to myself, what if I could put some code in here that'll stop the screen from fading out? Then I'd have an idea of what lines of code the kernel reached. I first wanted to do an infinite-loop, but looking at init/main.c I saw a thread started. I don't want any other threads interfering; I want everything to just halt. Google'd how to cause a kernel-panic and found, in hindsight is obvious, that causing a segfault will kill the whole process. Someone gave an example and I put it into my function:

static void screen_stay_on() {
int *p = 0;
printk("%d", *p); //invalid memory access, will cause segfault.
}

I tested this code right in the init function in the mdss_mdp.c and sure enough, the screen didn't fade out. It just stayed at the white-BLU logo. Excellent!!! I then moved screen_stay_on() into all the error-checking parts of the code, one-by-one, many-many-many recompiles and "fastboot boot custom_boot.img" for a few hours. Eventually I narrowed it down to this:

rc = of_property_read_u32(pdev->dev.of_node, "qcom,max-mixer-width", &mdata->max_mixer_width);
if (rc) {
pr_err("device tree err: failed to get max mixer width\n");
screen_stay_on();
return -EINVAL;
}

Okay!!!! So if it called my function then I know for sure the error message above must have been sent to the UART-console. Remember a few updates earlier I said I can decompile the dt.img->dtb->dts to actually see its source code? Well I checked the source and sure enough, "qcom,max-mixer-width" was missing! I google'd msm8916 qcom,max-mixer-width and found other dtsi(differnet from dts) with just about all the same values I have and qcom,max-mixer-width = <2048>;. So I just went ahead and added that value right above other values that the kernel was checking for. Recreated the dt.img and tried to boot again. The screen faded to black! So I solved that error!!!!! Now as it turns out, after moving my screen_stay_on() code to all error-handling within mdss_mdp.c I can now say for certain that no errors occur in that file. The main function in here is static int mdss_mdp_probe(struct platform_device *pdev), and by the time that function reaches the end it has called all the other functions in the file and they all must have succeeded without error, so I put the screen_stay_on() in the error-handling at the end and the screen still fades out, so probing for the screen is working. Also, in mdss_mdp_get_pan_cfg I put:

if(strcmp("dsi:0:qcom,mdss_dsi_otm1284a_720p_video", pan_name) == 0)
screen_stay_on();

The code did some processing beforehand that appears to remove the "1:" at the beginning, so by doing this and seeing that the screen didn't fade out informed me that the correct video-mode string was being sent. I guess it's in the bootloader because I didn't put it in the cmdline when creating the boot.img and I removed my variable containing that value from the code. This conclusion is further enforced in that nowhere in the kernel-source can I find a call to "mdss_mdp_probe", so I guess the bootloader is what called it. Now, the fact that this drivers/video/msm/mdss/, is in the "videos" folder and my kernel-config file has CONFIG_FB_MSM=y and CONFIG_FB_MSM_MDSS=y seems to indicate that if I slowly work my way through all the .c files in msm and mdss, I'll eventually succeed in getting the device to start up enough for adb-shell. I think this because based on timing, the screen seems to be the last thing before the animated screen shows up and the moment that appears(actually even like a split second before) adb-shell starts working. Stay tuned! :D