yashsingh1/xdaforums huggingface dataset:
[Windows Mobile Software Development] ervius:
final version (I Hope!!!!! :) )

http://www.sendspace.com/file/q6dqes

-d decompile a package with MNGE files translating them into modules

-c compile from modules into packages to MNGE Files!!!

* solved some iusses with some MNGE files

* added imageinfo.txt file creation into modules just created from MNGE File!!!

bye!
[Windows Mobile Software Development] Ameet:
:( But still no answer for my question :( What is MNGE and how to find out which modules are these MNGE thingies :(
[Windows Mobile Software Development] ervius:
I' m only building the compiler/decompiler, don't know the pratical use of them.
[Windows Mobile Software Development] Ameet:
I meant which modules or files your tool is to be used for? :D Tried it on every possible file, it just crashes :D
[Windows Mobile Software Development] Da_G:
Hi Ameet,

I covered it in my detailed post, but to recap;

.cab.pkg contains files to replace inside the IMGFS partition on device

Some files can be stored in device as modules (which we know)

We use imageinfo.bin/s000/s001 format to store modules.

Microsoft's .cab.pkg format uses a different format, to acheive same result, file remains as a file (instead of coredll.dll becoming coredll.dll\imageinfo.bin, coredll.dll\s00x) - but file has new header, if you open in hex editor, instead of MZ header, has MNGE header - this file will be treated by ULDR as a module and installed to IMGFS as a module.

In order to utilize the Update Loader system properly, we'll need the ability to load up modules, hence the need for this converter. (which i believe this last version now works 100%!)

Hope that answers your q! :)
[Windows Mobile Software Development] Ameet:
Pretty much clearer now :) Thanks bro :) Will wait and watch the progress and shall put in my 2c in between :D
[Windows Mobile Software Development] ervius:
well, and at end tool works well ;)

download here and try!

http://www.sendspace.com/file/2sfjms

bye!!!
[Windows Mobile Software Development] Ameet:
Indeed works. Tried it on CIF.dll from OEMDrivers :D But it doesnt create the module back from the MNGE file after compiling. Attached is the CIF.dll file converted to MNGE (attached is only the header form the file) But once converted to MNGE header, the file cannot be converted back to module (or is it not required to do?)

Time to cook and check :D. But then this question still remains, how does one avoid the kitchen from stripping relocs from ROM?

Another question is which file are meant to be convereted to MNGE header or all files and modules?

DSMBuilder is a good tool to insert the versioning info in DSM files, someone posted a tool for editing certificate in this thread, will that work? :D
[Windows Mobile Software Development] elektrownik:
I think that it is important also: .pkg file must go to Windows/Start Menu folder, as I understand it is impossible to install it from other dir
[Windows Mobile Software Development] l3v5y:
I'm guessing BuildOS (the really old versions) and not combining DSMs will allow that...
[Windows Mobile Software Development] Ameet:
I guess Ervius's BuildOS would do the trick :D Correct me if I'm wrong :)
[Windows Mobile Software Development] Da_G:
I've done my first successful ImageUpdate deployment :)

I used a slightly modified stock ROM for testing.. on vacation now, will post all the particulars when i get back

.dsm are very important to imageupdate, very slight problem with .dsm and it won't work!
[Windows Mobile Software Development] Ameet:
AWESOME News brother :D

Eagerly waiting for you to come back soon :D:D:D:D
[Windows Mobile Software Development] 12aon:
Amazing news man good (well and bad for us) time to go on a holiday, can't wait to see this in action, 12
[Windows Mobile Software Development] Ameet:
Great :D

Enjoy the vacation. Till then we'll need to find a tool to edit the DSM files and to create / edit personal certification :) I dont have the tools to create or edit the ".pvk .spc" files :(

What happens in this scenario? (just for illustration)

ROM cooked by chef A

Flashed by user X

Update in IMFS released by chef A signed with his personal cert

Update installed by user X on ROM cooked by chef A

(can this scenario work?) - This update installed by user Y on ROM cooked by chef B?

ROM cooked by chef B

Flashed by user Y

Update in IMGFS release by chef B signed with his personal cert

Update installed by user Y on ROM cooked by chef B

(can this scenario work?) - This update installed by user X on ROM cooked by chef A?
[Windows Mobile Software Development] Da_G:
Hi,

In this case, cert match would fail for both scenarios, unless chef b is also cooking in chef a's certs, and the other way around

defaultcerts.dat allows multiple certs, so a chef could easily allow this if he chose

however, he would have to share initial .dsms used for cooking too, so that guid, interface, match up, with version incremented

I've now successfully stepped through entire OS upgrade through ImageUpdate, worked great!

I did get several errors, created about 14 super packages, 12 had differing errors (one was out of space error), other 2 completed with no errors

will have to do some more research into the errors as the os came out on the other side with no issues

and, of course, i verified that all my user data remained intact.. untouched :)
[Windows Mobile Software Development] v5HMeca:
So there should be a universal GUID database going (containing Opera, HTC apps, etcetera) and create an XDA-Dev cert.

That will allow for a commonground for chefs to basically let other chefs push updates. If they don't want other chefs to update their stuff, they can just omit the XDA cert.

edit-

On that note, maybe it wouldn't be a bad idea to somehow incorporate this idea into Joel's "OpnMarket."
[Windows Mobile Software Development] Ameet:
I thought so :D Maybe as suggested above, the universal GUID might work out ;)

Did you manage to figure out which files or modules are required to be converted to MNGE header? And which kitchen did you use to cook the ROM (and for which device) :)?
[Windows Mobile Software Development] CRACING:
Platform rebuilder and its kitchen don't work since it removes all dsm ( and rgu of course ). So I think buildOS will workout.

BTW; Bepe is working with a new kitchen which is gui this time ( Screenshot ). Hope he knows about Image Update too...

Best Regards
[Windows Mobile Software Development] Chainfire:
Awesome stuff - bookmarking :)
[Windows Mobile Software Development] elektrownik:
Hello, I'm n00b in WM cert's so I have question: Can this be used withoud cooking ROM ? I'm thinking about iPAQ 21X which can't be updated becouse it was manufactured by Foxconn not by HTC so it doesn't use flash.bin, it need bldiag.nb0 and hpimage.bin, but it also have ULDR, and I was able to dump ROM, so now can I take cert from this dump and sign .pkg file ? Or I should create my own cert and use it for signing ? Or it is impossible without cooking new ROM ?
[Windows Mobile Software Development] Arktronic:
Da_G, did you happen to test whether .provxml files work as they're supposed to with image updating? As in, mxip_* and mxipcold_* are only executed on cold boot, and mxipupdate_* is executed on cold boot AND on update?
[Windows Mobile Software Development] Da_G:
If you have no way to adjust DefaultCerts.dat in IMGFS, you have to have a private key for one of the certificates in there.. otherwise .cab.pkg will fail verifysignature check

@trancecoder: sry, wasnt paying attention to filesystem at all, only testing signature checks etc, overall viability

when I get back home will test that for sure if no one has gotten to it

btw; seems the out of space error i got was accurate, OEM chooses amount of space to reserve for ImageUpdate, for htc raphael gsm it's 9MB, once you pass that remaining updates will fail

So there's a balancing game to be done between how much space we reserve for updates, allow for user read/write, etc - more allocated ImageUpdate space means more updates can be applied in the future.. but less overall storage space available to the user.
[Windows Mobile Software Development] v5HMeca:
Can I get some clarification on something?

I was talking to someone via PM, and I he mentioned that the ULDR is limited to 3MB (or some such small number) and he says that the packages that it can implement are also limited to 3MB.

See, I was thinking that ULDR was just another loader, like an IPL or SPL, and it has the ability to load large packages and send them to the OS environment, much like the SPL is capable of doing with ROMs, which in a sense, is also a "large package" (except the SPL is destructive insofar as it doesn't allow the current OS environment to remain in tact).

So now I'm turning to you folks for some clarification. Help?
[Windows Mobile Software Development] Da_G:
Correct, ULDR is about 3mb, but that's not related to the limit on package size, there is a limit as i discussed in previous post, but it's set by OEM, unrelated to ULDR size..
[Windows Mobile Software Development] v5HMeca:
Thanks for clarifying.

Oh, OK. So I just read the post you were talking about with the "balancing game." Is there a way to circumvent this? Since UDLR has access to the OS file structure, it would be cool if it could just extract packages there and then shift them around as needed rather than requiring what basically amounts to as a reserved "temp" area. The limitation, in that scenario, is making sure that there is enough free space on the device to extract the package. Changing the size of the ULDR's "temp" area as needed (thus making it dynamic) would be an alternative.

But seeing as I'm not much of an expert in this field, I don't know how viable either of these options are.
[Windows Mobile Software Development] Da_G:
this would require shifting the user data, then dynamically resizing the partition, prett y sure uldr is not capable of this, but i will look into it :)
[Windows Mobile Software Development] ervius:
and funally, mnge2module works well also with some "malformed"MNGE_Packages" try to see, PRB compile them without any error!!!

download here! ;)

http://www.sendspace.com/file/dhkr6s

bye!!!
[Windows Mobile Software Development] magicalan:
i read it all still dont know what it is ...does it work in all resolutions
[Windows Mobile Software Development] Ameet:
Da_G. How do we create a personal certificate to be used and how is it used? I mean do we have signer and how to apply that signer to sign the nbh file?

Ervius. I sincerely request you make this new kitchen based on the old style cooking, not many people have still got used to the new PRB style app.reg and the new style folders ;)
[Windows Mobile Software Development] ervius:
is difficult, I'm working on, but hv files creation became very difficult!!!
[Windows Mobile Software Development] ervius:
the oldstyle kitchen is ready, and create correctly default and user.hv, but rom doen't boot then.....

I compared also with old buildos, and default.hv is larger than the created by oldstyle in this kitchen....

strange.....

maybe I need some more knowledge about cereg.dll. who can help me!?!?!?
[Windows Mobile Software Development] l3v5y:
I really like the new style PRB roms... But if imageupdate won't work on them, we'll have to go back to older things...
[Windows Mobile Software Development] Ameet:
The reason why I requested for the old style kitchen was that Ervius's new visual kitchen is too too too complex for people like me :D (especially someone who are cooking just for 1 device :D)

My 2 questions are still unanswered :(

1. How do edit / create personal key & how to sign the files with that?

&

2. Which modules / files are meant to be converted to MNGE header?
[Windows Mobile Software Development] ND4SPD:
Really interesting stuff going on! Amazing as always Da_G!

I know that Da_G has been able to create get the ImageUpdate system to work, and he has posted steps, but if someone can make a concise way of creating an imageupdate package, I'm willing to integrate that feature into my ground up rebuild of the old style kitchen (not using buildos/packagetools)

It'll be an easy and intuitive to use kitchen and this seems like a great bit to integrate into the kitchen.

I've already got it to reprocess all the DSM's to match the proper format when the ROM builds. This will allow the imageupdate's to actually work. I just don't know anything about the signing or the packaging of the software and that what is needed to deploy the package.
[Windows Mobile Software Development] Da_G:
@ameet:

msft's signtool.exe program is floating around xda, should pop up on a search, the usage is signtool.exe -v c:\path\to\the.pvk -s c:\path\to\the.spc -t http://a.timestamping.server.like.verisigns file-to-sign.cab.pkg

as to what files get mnge header - if it's stored in imgfs as a module (imageinfo.bin, s00x), then it becomes an MNGE. Essentially, MNGE is the "proper" format for storing a module when it's not in the IMGFS (like on your hard drive), as defined by msft standards. :)

as to the anatomy of a package, that will be a fairly long post, i'll cover it from start to finish when i get back from my vacation on friday :)

but to quickly sum it up - package is essentially a .cab archive with .cab.pkg extension, signed and timestamped, containing .dsm defining : version info, files belonging to package, dependancy info, upgrade/full install flag, name of package - dsm guid must match an existing guid on system and version info must be incremented if package is to replace existing one on device

package also contains the files referenced in the .dsm, along with an optional .rgu for registry entries, and an optional .provxml for file copy/move/registry/metabase operations

it would look like

foobar.cab.pkg

---guid.dsm

---[optional]guid.rgu

---[optional]mxip_ or mxip_update_xxxx.provxml

---somefile.ext

---anotherfile.ext

---morefiles.ext

then deployment is simply getting the package on the device and executing it in file explorer as you would a .cab :) the update agent is already associated with .cab.pkg .cab.pku .cab.pks

and now i wander off to some buffet
[Windows Mobile Software Development] Ameet:
Understood this except for one thing :D. What is the timestamping server thing? Do we need that? :(

So all the modules in IMGFS (or in OEM, SYS, IXP folders) should be converted to MNGE header, right?

Enjoy the buffet :D :)
[Windows Mobile Software Development] Chainfire:
Just flying through this, am I correct in assuming that:

- One could make this cab.pkg file with a NEW dsm

- Upon opening the file in Windows (does it use WCELoad or whatever the normal CAB opener is?), it will reboot the system and flash it to imgfs?

- How about XIP, is that updatable?

- Has it been tried with a certificate uploaded to the priviliged execution store instead of using one from the default cert set?

I'm asking about that because over the past two years I've had various cool ideas that required XIP updates and whatnot, or be cooked in. I'd rather have them as .cab.pkg installable by the masses.
[Windows Mobile Software Development] Ameet:
can some kind soul post the signtool.exe? unless if it is to be avoided by msft :( I found many link but none to download this tool :(
[Windows Mobile Software Development] Chainfire:
Its in the DriverWiz package around here somewhere (shhh!). Only place I know to get it apart from Visual Studio.
[Windows Mobile Software Development] Da_G:
Hi,

If I didn't timestamp, I received warnings from signtool (about not being timestamped) and the package would not pass signature checking..

This is the timestamp server url I used in testing, which worked great:

http://timestamp.verisign.com/scripts/timstamp.dll

note timstamp.dll is not a typo :)

@chainfire:

yes, xip is fully updatable using this method, i successfully updated my msxipkernel from 21725 -> 21728, as far as the image update tool is concerned it's just treated as another partition, nothing special needs to be done, updated same as any other package :) oemxipkernel would be updatable in the same manner. in order to add new things to the NK partition I think you would need to issue an update to one of the other .dsms there not associated with files (like msxipkernelltk)

haven't tried updating the other cert store but from what i understand it's only defaultcerts.dat that's used for the imageupdating operation.

the association is with updatebin.exe for .cab.pkg etc., it's handles initial validation, rebooting into uldr, and passing the files on to uldr

yes, you can make a package with a new .dsm and will be treated as a new package, added to the imgfs (subject again to the 9mb limit that htc uses in its roms)

i have not yet been able to get a package with a new .dsm working (errors parsing the .dsm and/or extraction errors)

i need to get with one of the creators of the .dsm tools and update them with full specs on the Device Side Manifest so we can get a 100% compatible .dsm going (unless nd4spd has this going already, it seems?) - ill be doing that when i get back on friday

didn't make it to the buffet yet but i hit the bar and smiths grocery store, brought me back a 12 pack of labatts and a 6 pack of fat tire, lol :)

edit: by the way, verisign has some good info on signing and timestamping irt: platform builder here that applies:

https://knowledge.verisign.com/support/code-signing-support/index?page=content&id=AR190

also, it contains a link to the platform builder sdk download from microsoft, which contains signtool.exe, and some development certs (pvk cer spc combo) :)
[Windows Mobile Software Development] pidsw:
I am really glad that this information is being shared by da_g, I would like to see what it can do to help the flexibility of the platform goiung forward.

To answer somone elses question on this thread - you can use imageupdate with a locked, OEM Bootloader, but you have to get the right certs.

You can provision your phone to accept universal Test Certs.

You can update your bootloader, but I am not sure if this is possible in CE 5.x, it is supported in CE 7.x. I have a feeling that if we get this working well will have CE 7.x (alpha) builds availble pretty soon.

The main hurdle will be the OEM's - you will not be able to take advantage of a lot of the features of CE 7.x (light much faster graphics) if there are no drivers. The OEMs are still scrambling to get 6.5 drivers out, and I have yet to see any that make use of the new lower level interfaces provided.

Most go through DDI (partially accllerated on 7201A) or gles_cm (partial acceleration again, lots of software overhead).

d3dm and ddraw just get routed throught ddi and gles_cm respectively, which defeats their purpose..
[Windows Mobile Software Development] ND4SPD:
Yeah, I've rebuilt a package editor that manages the Device Side? (thought it was Software) Manifest. I've got a quickly thrown together version that can give me all the details about the .dsm files. Tell me what exactly the DSMs need to have for them to work with the update manager and I'll release a preliminary tool to manage the files.

*HINT* For all those C# coders out there, take a look at the pkgcommon.dll file with Reflector.NET (google it). It should provide you with a way to manage a DSM file.

Meanwhile, I've got an initial screenshot of my new kitchen. There's still many things to implement (like this Update Manager/Deployment to Device).

Though you're probably only interested in the first screen. That's where it will be managing deployment. The ROMs are automatically processed into the correct DSM format when they are dumped/when new packages are added.

I just need to know what other things need to be added to the dsm info as well.
[Windows Mobile Software Development] ND4SPD:
I noticed you're from Seattle. Do you have a little something to share (WM7)? :D Great news about stock bootloader, no more need (possibly?) for HardSPL. Good to know. Also, about CE 7.x, will a lot of stuff be WPF/Silverlight/some new form of XAML-based?
[Windows Mobile Software Development] Chainfire:
So what are you saying, we can import our own certs and use this for updating on a stock device? Or that we need HTC's (or whomevers) certs?
[Windows Mobile Software Development] Chainfire:
@Da_G

Thanks for the added info. I'm sure I'll need your help when push comes to shove :D

In the meantime, while you guys are still figuring this out, there was a GUID database that needed creating, I believe? Perhaps this is something I can be of assistance with? Tell me what needs storing and I can spend some time setting that up.
[Windows Mobile Software Development] Soul_Est:
As for getting the correct DSMs into the ROM we could use DSMBuilder: http://xdaforums.com/showthread.php?t=412272
[Windows Mobile Software Development] Da_G:
dsm builder and dsm analyzer both need a little work to be 100%, i hope to talk to their authors when i get back from vacay :) and nd4spd i talked to already :)
[Windows Mobile Software Development] xda2_haseeb:
A few questions!!

First of all guys!! GR8 work especially you Da_G :) And i would request you guys to prepare a tutorial for chefs on how-to cook keeping this technology in mind :) it be great, i would say it would be better if it be a Step by Step Tutorial :D

Secondly i have a few questions :o

1. As said b4 ULDR need some temp space in the ROM right?? to update a SYS (abt 90MB) will it need 90MB temp?? would it be difficult to achieve this much free temp??

2. Are the *.cab.pkg files compressed too or are they just of normal size, i mean will a *.cab.pkg file of a 90MB SYS be of 90MB or less??

Thanks in advance!! and sorry if im being a stupid :(
[Windows Mobile Software Development] inertiax3:
Hello all.. has anyone sucesfully G'relocked/WMRelocked MNGE-converted modules? For some reason, WMReloc shows "***" in the log next to these modules and the ROM doesen't boot? I can't try any more experiments since I've lost my phone (stupid me..) but was wondering what do you use to rellocate these.. thanx
[Windows Mobile Software Development] hilaireg:
I can help on this one ...

I had to stop using DSM Builder as my kitchen builds would cease to function.

Agreed, I sent a PM to the author of DSM Analyzer; no response as of yet - also included the URL of this thread in my PM.

DSM Analyzer: http://xdaforums.com/showthread.php?t=444186

@ND4SPD:

Any plans to make your DSM Tool Windows XP compatible?

@Da_G:

Are there package dependencies in there somewhere? I would expect that some applications requiring updates may have dependencies on other applications being updated first - or being present.

Cheers,
[Windows Mobile Software Development] ND4SPD:
Sure, mine has the same functionality as the DSM analyzer that you linked to, just looks a bit diferent. As long as you have .net framework 3.0 or 3.5 you should be fine.

I'll clean it up and post it today.
[Windows Mobile Software Development] xda2_haseeb:
Then please do... THNX :)
[Windows Mobile Software Development] CRACING:
NOPE, its still under testing and researching and it isn't completed yet so how is it possible to write a tutorials at present? :p

I know many ppls are waiting for this and I also know da_g and many other experts are doing hardwork and great job, so we must wait and watch until its done. :)

Sorry, if I said loud!!!

Best Regards
[Windows Mobile Software Development] xda2_haseeb:
I know my frnd tht its under testing but the way to cook such ROMs is known, atleast a tutorial with knwo info shuld be made so other chefs can test it too as the knowledge is increased the tutorial could be modified according to it :) hope u got wht i said :D
[Windows Mobile Software Development] CRACING:
I agree you but 75% members will be asking silly, nonsense and etc questions which makes the thread bigger and etc. :(

BTW: da_g has written enough infos to understand for chef.

Lets wait for ameet or da_g to reply.... :)
[Windows Mobile Software Development] Ameet:
I just replied :D:D

Actually I'm too low in this knowledge to reply anything of much importance in this project :D I'm still trying to figure out which files / modules are to be converted to MNGE header :D

I can help Da_G in writing the tutorial too if required :)
[Windows Mobile Software Development] CRACING:
cool :D

Thanks for the reply.

Best Regards