yashsingh1/xdaforums huggingface dataset:
[G1 Android Development] donaldj83:
im shure this has already been answered but exactly what is the formula to go from hex to MB for example

is it hex converted to dec

dec / 1024 = mb

then the reverse would be

MB* 1024 = dec converted to hex
[G1 Android Development] xaueious:
Just to confirm, after modifying the size before '@' symbol, we need to modify the offset...
[G1 Android Development] lbcoder:
THAT DEPENDS!

If you modify the size of a partition, then you need to modify the OFFSET of the NEXT partition.

In the case of the SYSTEM partition, you do NOT modify the OFFSET unless you are for whatever reason, modifying the size of the BOOT partition, which you should NOT be doing. At least not now.

Offset of any partition = offset+size of PREVIOUS partition.
[G1 Android Development] xaueious:
Okay :) Thanks
[G1 Android Development] zim2dive:
thanks! (I figured I'd start by understand Stock SPL, get those sizes/offsets correct, and then produce equivalent #'s for Danger SPL... so I'm half way there :) )
[G1 Android Development] lbcoder:
Note: It may NOT be necessary to specify offsets in some cases.

As I understand the system, you only NEED to specify the offset IF there is a GAP between the end of the previous partition and the start of this one.

I.e., we have a stock partition layout like this:

mtdparts=msm_nand:256k@0x24c0000(misc),5120k@0x26c0000(recovery),2560k@0x2bc0000(boot),69120k@0x2e40000(system),69120k@0x71c0000(cache),76544k@0xb540000(userdata)

This is equivalent to doing this:

mtdparts=msm_nand:256k@0x24c0000(misc),5120k@0x26c0000(recovery),2560k(boot),69120k(system),69120k(cache),76544k(userdata)

*** I *think*. Unless it does something weird, like trying to fill in the first available space that's big enough, but I really doubt that it does that. I think that it just sticks the next partition's offset at the NEXT available space. In any case, specifying the offsets makes it clear and readable, so I recommend doing that.
[G1 Android Development] lbcoder:
A good way to TEST your layouts is to fastboot a recovery image with "-c". This allows you to perform a non-destructive test and take a look at the partition table. You can see your sizes and names at /proc/mtd, and you can view the partition table in the output of dmesg.
[G1 Android Development] zim2dive:
Based on a few posts I found for DangerSPL (ie. http://xdaforums.com/showthread.php?p=5533977) I *believe* the equivalent DangerSPL table would be

Block k M bytes(D) bytes(H) StartD EndD StartH EndH
misc 256 0.25 262144 40000 38535168 38797312 24C0000 2500000
recover 5120 5 5242880 500000 40632320 45875200 26C0000 2BC0000
boot 2560 2.5 2621440 280000 45875200 48496640 2BC0000 2E40000
system [COLOR="Red"]92160[/COLOR] 90 94371840 5A00000 48496640 142868480 2E40000 8840000
cache [COLOR="Red"]30720 [/COLOR]30 31457280 1E00000 142868480 174325760 [COLOR="Red"]884[/COLOR]0000 A640000
userdat [COLOR="Red"]91904[/COLOR] 89.75 94109696 59C0000 174325760 268435456 [COLOR="Red"]A64[/COLOR]0000 10000000

Which I think would yield this DangerSPL "equivalent" (the parts in red are all that change.. start addr for system doesn't change, only its size)

mtdparts=msm_nand:256k@0x24c0000(misc),5120k@0x26c0000(recovery),2560k@0x2bc0000(boot),[COLOR="Red"]92160[/COLOR]k@0x2e40000(system),[COLOR="Red"]30720k@0x8840000[/COLOR](cache),[COLOR="Red"]91904k@0xa640000[/COLOR](userdata)

I don't know the history of DangerSPL.. was it considered an ideal partition ratio, or it was just something else that was available that fit the need (b/c it wasn't known how to fine tune to this level?)
[G1 Android Development] lbcoder:
By my math, your numbers look fine.

The reason for the deathspl's partition layout is that it happens to be what HTC selected for MAGIC. Nobody's changed them.

In fact, the only SPL I am aware of that has a customized partition table is the russian hacked 1.33.2003 SPL, which was hacked to match 1.33.2005.
[G1 Android Development] Firerat:
great thread :)

ref the partition sizes

I would be tempted to keep the official /cache small, but large enough for recovery , 5mb should be plenty if not overkill

then set system to something suitable, say 75mb for a CM5.0.8, giving a little room for some customisation, and the rest data

then employ a Bind mount for cache, same as I did for cachehack

this bind mount could be scripted, so that it goes to sdext or in the case of an MT3G on data

have to say I am looking forward to playing around with this, but duty calls, darts match in 30 min
[G1 Android Development] lbcoder:
That's a great idea -- it would keep the cache/waste small while still allowing the market to download and install big applications.
[G1 Android Development] Firerat:
:)

I actually went for 90mb system in the end, didn't realise just how large it had got in CM-5.0.8

Happy to say its working great

Every 0.1s: adb shell df -h Tue Jun 22 13:21:48 2010

Filesystem Size Used Available Use% Mounted on
tmpfs 47.7M 0 47.7M 0% /dev
tmpfs 4.0M 0 4.0M 0% /sqlite_stmt_journals
/dev/block/mtdblock3 90.0M 87.7M 2.3M 97% /system
/dev/block/mtdblock5 114.8M 82.9M 31.8M 72% /data
/dev/block/mtdblock4 457.4M 118.2M 314.7M 27% /cache
/dev/block/mmcblk0p2 457.4M 118.2M 314.7M 27% /sd-ext
/dev/block/mmcblk0p2 457.4M 118.2M 314.7M 27% /cache
/dev/block//vold/179:1
7.0G 6.4G 582.7M 92% /sdcard

I did a Bind mount for cache ( to /sd-ext/cache/ )

the Command line I used was

mtdparts=msm_nand:256k@0x24C0000(misc),5120k@0x26C0000(recover),2560k@0x2BC0000(boot),92160k@0x2E40000(system),5120k@0x8840000(cache),117504k@0x8D40000(userdata)

NB , that needs to be used on both your ROM's Kernel and the Recovery.img ( so you can flash ;) )

Edit : just read some kernel help, "-" can be used as a 'wild card' for size

so, to keep compatible with mt3g ( who will get an extra 75mb (ish) on data )

mtdparts=msm_nand:256k@0x24C0000(misc),5120k@0x26C0000(recover),2560k@0x2BC0000(boot),92160k@0x2E40000(system),5120k@0x8840000(cache),-@0x8D40000(userdata)

Initally I tried just a nandrestore, but things were a little screwy, so I fastboot erased system ( with -w flag) , flashed CM5.0.8, then did

mount -a
cd /data
unyaffs /sdcard/nandroid/<serial>/<backup>/data.img

now, here is the strange bit

I can fastboot boot a patched 5.0.8Final kernel and it works fine, but it refuses to boot when flashed.

this isn't an issue with the cmdline, it is the kernel as I can get 5.0.8test1 kernel to boot just fine when flashed.

guess I'll have to have a look at what changed between 2.6.33.4 and 2.6.34
[G1 Android Development] lbcoder:
Would this be the same issue catching you as here: http://xdaforums.com/showpost.php?p=6879430&postcount=90 ?
[G1 Android Development] Firerat:
yeap

at the moment I can't make any sense of it, but as I said it works fine if I fast boot it, but will not work when flashed be it via fastboot or recovery

just done a diff on 2.6.33 and 2.6.34, hoping something will jump out at me
[G1 Android Development] lbcoder:
Don't be diff'ing from 2.6.33 to 2.6.34.

I'm running TEST 3 right now with a 2.6.34 kernel.

According to zim2dive here: http://xdaforums.com/showpost.php?p=6761102&postcount=31 it works for TEST 4 also, so the change that broke it is between TEST4 and FINAL.
[G1 Android Development] Firerat:
ok , kewl

I actually missed all but test1 :)

Edit:

looks like its a problem with my nand, I noticed the kernel was a little larger so I compiled CM2.6.34 with bluetooth as modules, and bingo smaller kernel works when flashed...
[G1 Android Development] lbcoder:
Update:

I've got an issue going in the CM issue tracker located here: http://code.google.com/p/cyanogenmod/issues/detail?id=1684

This issue covers both the kernel issue as well as the general implementation in terms of the possibility of actually integrating this functionality into future versions of CM.

** Steve brought up an interesting point about migrating from stock partition table to custom that got me thinking about this in terms of recovery implementation.

I have thought up a scheme to allow this to be implemented "more or less" using a STANDARD recovery, and to allow a much greater flexibility than simply being able to "update from stock" -- also to change as the need exists, i.e. as AOSP keeps getting bigger, so may the system partition. More than that, I believe that I have come up with a scheme to modify recovery partitions for full advanced functionality withOUT having to actually hard-code the partition table into it -- by having it actually READ the partition table from the boot partition. Also a way to save the system and userdata partitions when performing a radio or spl update.
[G1 Android Development] lbcoder:
Interesting.

Are you certain that it is actually a problem with your nand and not a more general "just too big" problem? I know that the latest CM kernel still generally FITS within the allocated boot partition space, but this doesn't necessarily mean that there isn't some weird addressing conflict based on the particular SPL.... you might want to TEST your nand to be sure.

Also: Are you certain that it was actually the SIZE and not simply the fact that you recompiled it?

Or possibly something that was changed in CM since 5.0.8 that fixed whatever issue it was?

I guess the easiest way to test this would be on a different phone with the same radio/SPL as you have.
[G1 Android Development] Firerat:
I was going to flash danger to check if it was an SPL thing,

initially I compiled the kernel 'straight' from the 'stock' config, it was only once I reduced the kernels size by moving the bluetooth to modules that it worked

I'll play around with it some more later,

tbh, I would prefer that it was an SPL issue at least I know I can get around it.

but its not that big a deal if it is my nand failing, my Tmo contract is up next week so I'm in the market for a new phone.
[G1 Android Development] zim2dive:
So you are fastbooting a recovery image with a modified partition table ... so the (unmodified) installs have room to put all the normal stuff on /system?

Then _before_ reboot (which would be bad at this point?) you flash a modified kernel with a boot.img that has the new partition sizes?

Slick!
[G1 Android Development] Firerat:
yeap

tbh, seeing as this works so well, I might just flash the modified recovery

but we do need to arrive and some consensus on the partition sizes

might be able to get away with a much smaller cache, it will only be used in recovery

oh, and seeing as I now have 114mb data, the DangerSPL users will be very keen to be doing this as well
[G1 Android Development] lbcoder:
Yep, that's exactly what's happening.

But just to clarify one thing, it isn't so much a "modified kernel" as it is a "modified boot image". Though he's custom compiled the kernel, it is for a completely separate issue that he had (which may or may not affect anyone else in the entire world). The partition table can be passed to ANY kernel on the command line, and the command line is set within the boot image OUTSIDE of the actual kernel.

Note that you can use ANY recovery you prefer -- doesn't have to be modified -- by passing the mtdparts= command line using the "-c" parameter to fastboot. The modified recovery image is required mainly in case you want to FLASH it.

FYI: I've got this installed now in this form. It works perfect. Minor weirdness that happened -- I *updated* from T3 nonedanger to this (by restoring the data.img from nandroid backup), and during first boot for some reason the userdata climbed from ~50 MB right up to ~95 MB. Not sure why yet. A wipe would certainly solve this, but I'm going to see about cleaning it. Yes, there is some that can be saved simply by clearing the /data/gapps directory (still full).

Ah, nevermind.... its because the dalvik-cache was in /cache for nonedanger, so all the dalvik-cache was recreated in /data.

Anyway, with the gapps gone, got a nice 33.6 MB free in /data.
[G1 Android Development] lbcoder:
I'm thinking that it might not be a bad plan to make a few things a little more dynamic regarding the recovery. Remember that the boot partition is what ends up controlling the live system's partition table....

If mtd was built as a MODULE (for recovery), then recovery could load the partitions *as needed*, which means as specified in the update script.... so IF the install image needs a custom layout, then rmmod mtd; modprobe mtd mtdparts=...;. For other recovery functions, just need to modprobe mtd using whatever is set in the INSTALLED boot image.

I have a bunch of stuff over on the CM issue tracker: http://code.google.com/p/cyanogenmod/issues/detail?id=1684
[G1 Android Development] Firerat:
:)

yeap, I put the command line in the recover.img, because I have seen enough people who don't have fastboot

and yeah, its the dalvik-cache, if it does get tight we can bindmount it to sd-ext ( thinking about it I will have to 'fix' my all in one patch, it will see us as MT3Gs now ;) )
[G1 Android Development] lbcoder:
One very cool thing I just learned about the yaffs2 filesystem... (because I forgot to fastboot -w before installing)... when there is no yaffs2 header at the beginning of the partition, it scans through it from the front until it finds the first yaffs2 header and mounts at that offset. So when your partition grows backwards, you don't gain the space, but IT STILL WORKS!!!
[G1 Android Development] lbcoder:
HEH HEH, FROYO's out....

We may now be NEEDING this mod. :)

And by "we", I include those with DEATHSPL.
[G1 Android Development] Firerat:
lets not forget MT3G :)
[G1 Android Development] lbcoder:
A thought re dalvik cache.....

The dalvik caches for all the SYSTEM apps and framework on CM5.0.8 add up to a whopping 29.5 MB.

If we ODEX the system apps and framework, we can break 29.5 MB off the data partition. Note that when an apk is odex-ed, the classes.dex file is removed, so for a minimal increase in the system, we end up with a MASSIVE reduction in the userdata.... the only thing this breaks is stuff like theming... but if you use sensible odex scripts like the one linked to at the bottom, then you could THEME FIRST, then odex.

http://android.modaco.com/content/htc-hero-hero-modaco-com/301910/odex-script-for-unodexd-roms/

Note: The script can easily be updated to also odex the framework.

Now re odexed files... I'm not certain if you can mix and match.... i.e. if you odex one do you need to odex them all? Or do you just need to make sure that you have a matching apk/odex pair for each package you odex?

Another idea to massively save space:

If the USER applications can be odexed.... would have to account for this in application updates, i.e. make re-odexing part of the application installation process. Or this might really screw up when it comes to signatures...

Thoughts?
[G1 Android Development] jubeh:
would it be feasible to reduce /cache to minimum size and bind mount it instead to /data, that way, with firerat's configuration, we'd have 90, 128k, 119 with /cache mounted to /data-cache. it would be used during installs and have as much space available as it needs rather than try to guess (and waste space) at 5 mb.

We don't know when downloads might get larger.
[G1 Android Development] Firerat:
yes, but remember recovery uses cache to issue commands and keep a log, you can't have it too small, but yeah I thought 5mb was overkill

the script I included will bind mount to /sd-ext/cache, but if thats not mounted then it will bind mount to /data/cache

thinking about it, I'll write a script which can take the sizes for system:cache and output command line, and even patch the boot.img for you

it will require linux ( I don't have windows to convert/test it )
[G1 Android Development] goldenarmZ:
What splendid timing. :D
[G1 Android Development] lbcoder:
You want timing? Froyo launched with an EARTHQUAKE that rattled east-central CANADA and the north-east US.... where it NEVER quakes.
[G1 Android Development] Firerat:
Attached is script ( + bins ) to create the command line, and optionally 'patch' a boot image.

currently it requires linux, but I think I have the bins to get the Phone to patch the boot.img

unzip the files into a PATH, and chmod them to 750

then, with your boot.img ( or recovery.img ) in your current directory

MTD-Helper.sh 90 1 boot.img

use:
MTD-Helper.sh <system size (mb)> <cache size (mb)> [ boot.img ]
e.g.
MTD-Helper.sh 90 1 boot.img - modifies boot.img to include a MTD Partition map
System	90mb
Cache	1mb
data	remaining mb
~~~~~~~~~
MTD-Helper.sh 90 1 - simply prints out the command line
e.g.
Kernel command line :
mtdparts=msm_nand:256k@0x24C0000(misc),5120k@0x26C0000(recovery),2560k@0x2BC0000(boot),92160k@0x2E40000(system),1024k@0x8840000(cache),-k@0x8940000(data)

script is at

http://pastebin.com/TAgHCkkg
[G1 Android Development] asb123:
I realise that this means devs will now no longer need to cut anything out of roms, but does this mean that roms will be faster? (no/less compression, more zipalining or something ect..)
[G1 Android Development] Firerat:
lol

zipaline

^^^ some 'devs' say its all zipaligned to bamboozal newbs into donating

these days its automatic ( and has been for a very long time )

the dev that says all apps are zipalined is a fraud, he can't shell script to save his life * and is only a file pusher/donate tart

* I saw his kernel switch and fell about laughing, he was for some strange reason coping a kernel boot.img to /sdcard, flashing it after not checking the copy worked and rebooting

hehe, ok so sdcard was full, what then?

And, errm why not just flash the boot.img direct from /data/ ?

funny as rubber duck

whats more in another ROM he failed to correctly execute a script via a2sd, that toggled 3d support, lmfaoaktfmwhgafc

hehe

I wish he had got his little "I'll run this kewl 3d toggle script firerat has done via a2sd" thing right, it would have been a proper rofl when people complained about 3d being turned off every other other boot :D

hehe muppet

guess I should chill on ridiculing the donate king, but by some miracle England managed to stay in the world cup, so nah fingers up, I can pull my bow string back ( no not been drinking strongbow, just bitter ).

anyway, it depends some say sd is faster than internal memory ( well one so far )

personally I find that when I keep as much as I can on internal memory I get better performance, but maybe I have a bad class 6

but you know, if class 6 is faster than internal memory why do we not just put everything on it? its not difficult to do ( I already did it a while back but it was poor, then again I might have a bad class6 )

actually, if class6 is faster than internal, why do the wm2andriod guys look for ways to get things on their internal memory?

as I said only one "'dev'" has said that internal is slower than SD, but he is just a file pusher

On a serious note, this is basically greater flexibility

the only thing cache is really used for is market downloads and OTA updates

since we skip OTA and can use some other 'place' for market.

In the past with NoneDanger we could use it for dalvik-cache. Danger.. well it was useless, might as well make it do something useful ;)

anyway, I've thought of a way to make a flashable zip or two to get this working no matter if its 'officially' supported or not

quite simple, one flash makes your recovery compatible, reboot to recovery, you then flash your rom + extras and then a further flash patches the boot.img, ;)
[G1 Android Development] speedysilwady:
its been a while since ive been on, lifes been busy but wow yet again lbcoder and firerat revolutionizing ROM fitment onto our G1's, this is even better news than the cachehack! good luck to the both of you!
[G1 Android Development] Firerat:
I just scripted it

http://pastebin.com/TAgHCkkg
[G1 Android Development] Firerat:
#### ~~~ beta 2 ~~~ ####

#####################

EDIT:2010-07-04

update

New thread created

http://xdaforums.com/showthread.php?p=7061471

these flies will no longer be updated, they are here for historic reasons

#####################

This is super easy ( I hope )

But its also a bit beta, so if your not sure and don't know how to fastboot yourself out of trouble wait until its confirmed stable.

two flashable zips attached

one is recovery, the second is boot

These tiny little flashable zips are to patch your existing system

the idea is

flash the recovery patcher

its script runs, extracts the recovery.img, repacks it with MTD Part hack cmdline and flashes the recovery

you then reboot to recovery

Wipe and flash your ROM ( + extras )

flash the boot patcher

its script runs, extracts the boot.img, repacks it with MTD Part hack cmdline and flashes the boot

your done,

while you have the modified recovery you just need to do the boot patcher

i.e. flash your ROM, flash the boot patcher

by default the script will install a mapping of, system=90mb cache=5mb

You can override this with a txt file on the root of the sdcard

/sdcard/mtdpartmap.txt

a very simple format

one line of text

e.g.

90 5

for system 90 mb cache 5 mb

100 2

would be system 100mb cache 2mb

Edit: 2010-07-03 - cache less than 2mb causes errors in recovery

the remainder will be data..

additionally the boot patcher will install a script to bind mount /cache with /sd-ext/cache ( or /data/cache in the event that /sd-ext is not mounted )

Edit : 2010-06-25

why did no one tell me its not working :)

fixed the scripts, they *will* actually write the recover/boot img now :)

Edit : 2010-06-29

keep forgetting to add this ( it was in an earlier post, but some of you may come here direct )

after you have done a nandbackup, get your phone into fastboot mode , and on pc

fastboot erase system -w

that will clean everything ( system , data and cache )

since they are blank no 'junk' from cache ( like a bit of a dex file ) will suddenly appear in system

then patch flash the recovery

once rebooted you could then try a nandrestore , then patch flash boot

if you get stuck on the g1 ( with no logcat ) try flashing the attachment in

http://xdaforums.com/showpost.php?p=6959829&postcount=94

############################################

Edit : 2010-07-01

New attachments ..-v1

These fix a bug where the data partition didn't get the proper name

additionally the bind mount script creates dalvik-cache dir(s) as required.

############################################

Please use the -v1 versions from now on

############################################

Edit : 2010-07-03 - for Devs and chefs

http://xdaforums.com/showthread.php?p=7046457#post7046457

Patch your ROM/Kernel zip to auto detect the MTD Partition layout used by recovery

users will not need to 'manually' Patch boot ;)

############################################

Will have to consolidate all this in one post soon,

probably in the General Android Forum since this should work on most

if not all Android devices

############################################
[G1 Android Development] scholbert:
Hi,

what a useful and evolutionary step within G1 development.

I always believed that this is possible somehow, untill i read lbcoder's thread a few days ago.

First i wondered about the missing mtd partition table in the kernel sources.

At the time i went through SPL disassembly and had a look at the initial boot scripts,

was pretty sure that the table must have been handed over as kernel command line from the bootcode in some way.

But no idea how to override it at this point... Anyway i didn't had the time to digg deeper...

Just to complete the information in this thread:

http://xdaforums.com/showthread.php?t=542688

Where it all began ;)

@Firerat:

Great work!!

Have fun!

scholbert
[G1 Android Development] lbcoder:
It didn't begin there since that stuff died out before anybody took the next logical step.

It began HERE: http://forum.cyanogenmod.com/topic/1095-why-several-partitions-at-all/