yashsingh1/xdaforums huggingface dataset:
[Windows Mobile Software Development] keyx:
1. so after running imageupdate creator with the dump folder, i got the packages folders and the original dump folder now has those "_PACKAGE_FILES" folders, do I build the rom using that dump folder with imgfsfromdump? if that's the case isn't everything became file?I don't see any module left?

2. I suppose i also need to run the image update creator with the XIP folder, do i need the xip also has the file? the dump by xipport does not create file only module , i.e S000), do i have to use dumprom instead?

3. rebuild xip with the new dsms , do you mean rebuilding the xip after running the imageupdate on the xip folder? and how do you rebuild it? I only know how to use xip port to port xip.
[Windows Mobile Software Development] l3v5y:
Awesome! Will build a Diamond ROM to test, though now I need to make a certificate for my own OEMs... :s
[Windows Mobile Software Development] ND4SPD:
I'm building an Image Update ROM today. I'll have some finalized instructions with clarification later today.
[Windows Mobile Software Development] l3v5y:
I tried the latest release of your ImageUpdate creator... It's strange, I assume it's making packages but at the end of the "Create Packages from Flat" I have all the packages with no DSM/RGU containing file versions of the modules (not files that were files in the dump), all the files that were files, and all the XMLs created are treated like:

Any hints?

EDIT:

Adding the public folder from the earlier release fixes things, I've now got a stack of cab.pkg, problem is, I now don't have a dump to cook with... :p

For some of the packages in my dump, I have a _PACKAGE_FILES directory, but not all... Is this supposed to happen?

(I do sort of have a dump, but it's got a lot of packages in, lots of bsms xmls and csvs, and it's got no modules...)

Oh, and as far as I can tell, patching default certs did nothing at all (I have an identical file after it...)
[Windows Mobile Software Development] ND4SPD:
Sorry - I forgot to mention that you probably should keep a second copy of the full ROM dump. The problem is that once I hand over all the bsms, pkds, and csvs to pkgcommon.dll to create the packages - it creates those _PACKAGE_FILES directories that it builds the .cab.pkg's from. As well, all of the modules are converted to the MNGE format so that the ULDR partition can handle them.

As for the defaultcerts.dat patching, I'll check on that. However, open up the defaultcerts.dat in notepad before and after patching to see if the new cert was added in. When I did so, I did find the new cert for me but I could be delusional :p
[Windows Mobile Software Development] l3v5y:
Will try with a copy of the dump.... :)

Is there any chance your program could copy the dump to a separate location work on it, return the cab.pkgs and then delete the copy of the dump?
[Windows Mobile Software Development] ND4SPD:
Guess I could do that. It would take a very long time with some ROMs though and that's why I hesitated at first to do that.
[Windows Mobile Software Development] l3v5y:
My current workflow (or what I'm thinking) is along the lines of:

Dump ROM using oldstyle kitchen, modify things to my liking, rebuild ROM with DSMs preserved. That gives the base ROM.

For updates, rebuild ROM with whatever changes are wanted, and then dump the ruu_signed.nbh to a raw dump use Image Update Creator on it and get a set of cab.pkgs. It's a couple of extra steps, but I'm going to look at the code for Image Update Creator and try and make it without a GUI (so command line only) which would make it much easier to use in a batch file.
[Windows Mobile Software Development] l3v5y:
Hehe... I suppose... I've built a batch file that uses XCOPY to create a temp dir for usage with Image Update Creator.
[Windows Mobile Software Development] ND4SPD:
If you want a command line only version - just take the entry point from the event handler and use that as the main method. Then just copy the rest of the code - i'm pretty sure you can use the background worker as well in the command line version
[Windows Mobile Software Development] ND4SPD:
Fixed up the code that patches the DefaultCerts.dat

Changes are on xdaimageupdate.codeplex.com
[Windows Mobile Software Development] l3v5y:
Excellent... I'm going to build a test Leo ROM for ImageUpdate (since you said that it's WM6.5, so I'm guessing the 6.1 ULDR won't like it).
[Windows Mobile Software Development] Da_G:
Hey guys, sorry i've been out of the loop with this. My time is really short lately - but if you want to contact me directly about it, send me a PM on here or an IM on MSN/AIM/Gtalk :)
[Windows Mobile Software Development] l3v5y:
With the -conservative flag, could we theoretically set an amount of free space to remain for updates, whilst reducing the IMGFS for some gains? That way we can do compression etc on the original files, remove things etc and gain space, but still have ImageUpdate.
[Windows Mobile Software Development] l3v5y:
I've built a ROM which has the default certs patched, and everything for ImageUpdate ready, but using my own cab.pkg files it won't work. I get something like validation failed, though it doesn't actually give any detail of why.

It also seems that FWUPDATE doesn't delete cab.pkgs on the microSD card, but does in \My Documents. Perhaps this is of interest?
[Windows Mobile Software Development] ND4SPD:
Yes, if I'll patch it so that you can enter in the amount of free space.

Can you post the UpdateValidator.log file?
[Windows Mobile Software Development] l3v5y:
I can't find a log file, though as an error code I get 80180015, with Update ID: Unknown. If I can get any more info to you, let me know.
[Windows Mobile Software Development] ND4SPD:
Create a folder called Release in the root of your device. When you run image update, it should create a log file there.

also, I was looking through the imgfs tools code and found that you might need to use my patched version of ImgfsFromDump to create an image update rom. There is a small flag in the ROM that gives it the ability to load up the image updates into it's free sectors that has been disabled in Tadzio's versions. I've added in a small conditional statement that will check for image update builds and enable that flag. Will post soon
[Windows Mobile Software Development] ND4SPD:
I'm still trying to figure it out a bit, but here's another step I didn't mention: Cab all the .cab.pkg files into a .cab file and rename the extension .cab.pks. The .cab.pks would include both the MSXIPKERNEL/MSXIPKERNELLTK from the XIP and IMGFS .cab.pkg files. This is a super package and if you run this instead of individual packages, it will be able to lookup dependencies so that the image update doesn't fail.

Hopefully I should have my ImgfsFromDump mod that allows for image updating.
[Windows Mobile Software Development] Sleuth255:
Is this something we want in our ROMs? It's scary what an Operator could do to our phones with this and OTA delivery....
[Windows Mobile Software Development] Da_G:
No one says you have to leave the OEM certs in your cert store, don't allow them to update your device :)
[Windows Mobile Software Development] l3v5y:
So to do a very minor patch, I have to run an update of the whole ROM for dependencies? That doesn't seen to great...

Can I look at the dependencies of the cab.pkg I want to update and include those cab.pkgs as well?

I chose the CommonNonEA package purely because it's one of the simplest packages in the ROM! :S
[Windows Mobile Software Development] ND4SPD:
No, but if that package has any dependencies, then you'd need to. I'm just trying to cover all bases for testing.
[Windows Mobile Software Development] l3v5y:
As far as I can tell, CommonNonEA has no dependencies. Are the shadow packages related to this?

Using DSM Editor, it says the "Package Flags" is canonical, should I change that to update, or will that break things?

I can't find a complete set of packages form a ROM more recent than 21869 so unless you know of one, I may not be able to cover all the dependencies!
[Windows Mobile Software Development] mazjed:
any how to pleas

hi every body

**** the weed has screwed up my mined

i couldn't understand any thig

would some buddy pleas make a how to update a new build pleas :)

yes i feel stoned
[Windows Mobile Software Development] ND4SPD:
*BUMP*

Was hoping people could test out the program and leave some feedback either here or on the codeplex site. That way, I can fine tune the program to act differently for different COM builds so that there aren't any errors in the Image Update build.

I want to finalize this code so I can release the Image Update ROM building side of the software that is for the most part complete. However, all of that software is useless if this part is not always outputing valid package definitions to build the image update packages with.
[Windows Mobile Software Development] l3v5y:
I modified the Imgfs tools to create ImageUpdate compatible ROMs, but I don't have my HD2 with me, and you said the WM6.1 update system was different so my Diamond won't play nicely with it.
[Windows Mobile Software Development] ND4SPD:
Thanks for the response - it's cool. Guess I just wanted to see if there was still any other interest for getting Image Update working and to make the tools better, because my software is still very basic and rudimentary.
[Windows Mobile Software Development] hdubli:
can u please share the tools, i will try on my hd2 :)
[Windows Mobile Software Development] l3v5y:
With creating cab.pks files, is it a cab file (just packaged up) or a CAB installer (with _setup.xml)?
[Windows Mobile Software Development] ND4SPD:
i think it's just a cab package file renamed to .cab.pks
[Windows Mobile Software Development] l3v5y:
Cool.

Tried that, got another error.

Anyone know what any of the error codes actually mean?
[Windows Mobile Software Development] l3v5y:
I got annoyed at loading up a hex editor to look at the IMGFS info, so I wrote a simple application that displays the FS_BOOT_SECTOR of an IMGFS file. Drag the file over my tool and you'll get a command line display.

Maybe that's helpful.

What annoys me is that I ran a few Leo IMGFS files through it, and all had the dwImageUpdateModeFlag set to 0. Somewhere I saw it should be non-zero to enable ImageUpdate. Is that true?

Also, anyone have an ImageUpdate capable ROM already? (I know Da_G does) Some pointers would be good!
[Windows Mobile Software Development] ND4SPD:
yeah, it needs to be set to 1 - i never released my update to the imgfsfromnb, but i did change that

i saw it in the source code for the imgfs tools - maybe that's where you saw it?
[Windows Mobile Software Development] l3v5y:
That could be where I saw it...

I changed that in IMGFSFromDump, but I'm still not getting ImageUpdate to work. Perhaps I could have a look at the tools you've got working?

Does anyone have a list of what the error codes mean?
[Windows Mobile Software Development] ND4SPD:
[edit] i'll upload when i find my source code :P

sure - uploading now...
[Windows Mobile Software Development] l3v5y:
Am I correct in thinking replacing

if(ret<CurrSize)

with

if(false)

or similar will make the ROM uncompressed?

Once that's done, I'd guess you could streamline the code a little by removing all of the compression stuff?
[Windows Mobile Software Development] ND4SPD:
don't know about that - i largely ignored that part of the code :D

just recompiled the code and uploaded it here
[Windows Mobile Software Development] l3v5y:
I've edited IMGFSFromDump to accept two new parameters, dwFreeSectorCount and dwHiddenSectorCount. Both are added onto the end of the IMGFS, using the current sector size (on the HD2 it's 0x800).

They need to be hex, so IMGFSFromDump imgfs.bin imgfs-new.bin 0x3E80 0x40 is what I'm using now. It also sets dwUpdateModeFlag to 1.

If you don't specify dwHiddenSector count it uses whatever was set in the input IMGFS.

Now, if only someone can tell me what these things mean! :D
[Windows Mobile Software Development] l3v5y:
FINALLY!!

With some help from Da_G, I've managed to get a package to update.

It was a package with no files in, but still...

Now, module->MNGE?
[Windows Mobile Software Development] ND4SPD:
Woah! How did you get it to update? Details please! :D

btw, the image update creator program that I built converts modules to MNGE format when it creates the .cab.pkg file
[Windows Mobile Software Development] l3v5y:
I'm using Ervius module-> MNGE, and a script that makes and signs cab.pkgs. It doesn't generate DSMs, which is where my current issues are...

One issue I've met is that DSMs must not be read only. Da_G pointed that out.

Also, not entirely sure why, but 21869 and 21872 are apparently the same for ImageUpdate! :O

I made a cab.pkg of CommonEA, which isn't in the WWE ROM, and that worked!

Is there a simpler way to make a valid DSM than looking at your ImageUpdate builder and going from there? What I hope to do is build a light, stripped out ROM and then have everything as addon cab.pkgs, but I need to be able to make my own DSMs.

I'll post my edited IMGFSTools at some point. What they do is set the image update flag to true, and add a customizable ammount of free space at the end of the IMGFS.
[Windows Mobile Software Development] ND4SPD:
yeah - i've got an addon for the tool that just edits the DSMs that doesn't invalidate them. Other than the free space/image update flag, is there anything else you did? otherwise - my version will work as well :D
[Windows Mobile Software Development] l3v5y:
The biggest issue I had was picking a package that didn't have dependencies. CommonEAis good because It's not there normally, and has no dependencies! Will make a video at some point to prove it works!
[Windows Mobile Software Development] Raptordin:
I think this would be a huge development when it finally materializes.

Grats.
[Windows Mobile Software Development] :
Yes but is CommonEA in your current ROM. Last I understood, that stand for Common Eastern Asia and then CommonNonEA is Non Eastern Asia

So if you are updating the package that is not there in the first place, is it just writing the file to there or it is actually updating it?

Next thing that I am working on is a way to deliever these updates OTA. I have started to look into the server requiremtns and it seems to be that it needs to be a OMA DM server. I am playing with these trying to get it going.

If anyone knows what file needs to be patched to recgonize the new server. Could you please let me know.

I will start to play with this Image update stuff and I am sure we will be able to get it working in due time.
[Windows Mobile Software Development] l3v5y:
I've built an MNGE->module converter, that seems to work on all the modules I've tried, and I used the net cf package in the first post as well, and putting the MNGEs from there through my code generate a new MNGE that was identical to the original.

I need to add some kind of error handling, so that my code picks up if a dll isn't an MNGE, but that should be fairly easy...
[Windows Mobile Software Development] :
very nice, got a sample application? now we need to get the dsm working i guess
[Windows Mobile Software Development] janneman22:
how about the progress? can i use it a ROM?
[Windows Mobile Software Development] l3v5y:
I've been working on DSMs and ImageUpdate etc for a bit today...

I've written an application that can generate a valid DSM (it's build in to libnb.dll for now) for files (contains no certificate data), though there's a field in the file entry (what the DSM uses for files) that I don't know what it does...

typedef struct _fileentry
{
DWORD dwNameLength;
DWORD dwFlags;
DWORD dwNameOffset;
DWORD dwExeBase;
DWORD dwFileSize;

} FileEntry,*PFileEntry;

dwFlags is what I'm not sure about. There seem to be two values I've seen in existing DSMs, 0x10000003 and 0x10000001. I'm not sure what they mean, but I guess it's something to do with update/remove. If anyone can shed some light on that, it'd be very helpful!

Back to the progress...

With a DSM I've made, I've got a script that will churn out a cab.pkg, and those run successfully, and persist after a hard reset. The issue is, if a file in the cab.pkg over rights one that's already there (in a previous cab.pkg) the update fails, but if the file doesn't exist in the new cab.pkg, it's not removed.

So, cab.pkgs work, can be inserted in to ROMs, and can persist an update, the issue is just the exact specifics of the dwFlags value in FileEntry (note, the struct is my own construction based on lots of swearing at a hex editor).

[YOUTUBE]tecgRhy8GGg[/YOUTUBE]
[Windows Mobile Software Development] l3v5y:
Right...

With the DSM flags, I've now got somewhere.

0x10000001 is for normal files

0x10000003 is for files that need an action (provxmls and rgus).

That still doesn't help with updating existing files, so anyone have any more info?
[Windows Mobile Software Development] l3v5y:
Thanks to a quick search, this post and an HTC hotfix, I now have a cab.pku.

The file starts with "DMNG", which I guess means Diff MNGE or something. The format is (after a quick check) not the same as an MNGE.

There's an RGU file, and that starts with "DFBN", and also doesn't make a whole lot of sense! :D

The DSM includes all the files that the original package had, though there are a few new flags I've not seen before.

0xA0000001 is on the MNGE that's being updated

0x20000001 on what I think is an MNGE

0x90000003 is what the RGU that's being updated has.

So, if anyone has any ideas, I'd love to hear them! The cab.pkus are attached.

EDIT:

The 0x20000001 is indeed a flag for modules (MNGEs in cab.pkgs), I just checked with the original packages, now attached.

EDIT2:

The MNGEs have the dwExeBase set (all the ones I've seen are set to 0x10000000)
[Windows Mobile Software Development] BionicSniper:
So the current hurdle is to decode the meanings of the dsm flags? What is the status of a centralized server?
[Windows Mobile Software Development] sublimity:
I have a device Win CE and I need to access.

I have only cab.pku

1) exctract

2) open in hex view some exe file and see header : 444D 4E47 : DMNG

how restore original file ?
[Windows Mobile Software Development] Ameet:
AWESOME initiative brothers (& sisters too) :D

I'd like to suggest a few things though :) Maybe I could be wrong

If we can have this thread break down all the information to the detailed base i think it will be helpful for non software people like me to understand :) I mean for e.g. what is MNGE, how is the ULDR connected to the ROM update system without flashing.. so forth and so on :)

Will surely link up this thread to my tutorials, this will be another major update in the ROM cooking and flashing now :D
[Windows Mobile Software Development] B4PJS:
Maybe too simple a suggetstion for notifying a user of updates is to cook an email account into any ROM that you create that checks say once every 24hrs with either a link to the updated packages or just telling the user to run the update client. Of course you would need to block sending emails from this address and use some form of authentication/filters so that only the chef could send the emails.
[Windows Mobile Software Development] aiiro:
Being a complete dumbuser regarding everything that concerns PPC hardware, chips, ROMs, flashing and all the mighty abbreviations used in this forum (like RUU, RGU, DSM, ULDR, SPL, OMA etc. etc.) and focusing mainly on usability for average user, I'd still like to input some of the ideas what I had after reading this thread.

For the average user, flashing isn't much of an option taking into account the abovementioned. There isn't even a decent tutorial for flashing non-HTC devices like for the v1520 that I have. So, usability-wise, the cab.pkg aspect that is discussed here, would be very effective to use in the following context:

1. A desktop PC application that is somewhat like a crossbreed between File Explorer and CVS/SVN system and allows changing any ROM file.

2. All the transactions involving cab.pkg format conversion, certificates handling, updating the actual files etc. would be handled in the belly of the app, for user it looks like the app is simply able to change ROM files on the fly. Of course, on mobile device probably it is still needed to install the package.

3. The "update server" could simply be emulated by the app itself, provided that we would change the update server IP in the mobile device to the one ActiveSync is using (169.smth.smth.smth or whatever it was)

Et cetera. I hope I was able to transfer the main idea, which is allowing changing or editing of any ROM file in an easy way by an universal app on any device without flashing it. Of course the app would need additional actions like dumping the original ROM etc, but that's already doable anyway so it doesn't really matter.

Maybe I am an idiot but considering the information in the thread I think such an app would be possible to build.

If I am an idiot, sorry for wasting Your time :)
[Windows Mobile Software Development] DaveShaw:
Very nice and detailed. :)

Want me to move this to post #2?

EDIT:

@aiiro

The applications for this will come in once (if) we get it working, I'm sure different folks will find different uses to please the users. But fisrt, let's get it running :)

Dave
[Windows Mobile Software Development] Da_G:
@DaveShaw:

Yes please!

Thanks :)
[Windows Mobile Software Development] DaveShaw:
Done (and added 2 reserved posts for you :))

Dave
[Windows Mobile Software Development] aiiro:
@Dave:

OK, thanks! At least I found out I am not an idiot (this time) - I sometimes wonder about that :)

OK, gonna be patient now and not create content-empty posts anymore.
[Windows Mobile Software Development] Ameet:
Which means the ULDR should also not be removed from the ROM?
[Windows Mobile Software Development] Da_G:
Yes, the ULDR needs to remain to support this functionality (as it is the ULDR that handles the merging of .cab.pkg's with the live filesystem)
[Windows Mobile Software Development] cmonex:
yeah ULDR needs to remain for this to work;

also must not strip relocs from roms (bepe's kitchen does it for example). ULDR needs them.

l2tp seems right about MNGE format, that was my guess too. :)
[Windows Mobile Software Development] Ameet:
But how do one avoid this from happening? :(
[Windows Mobile Software Development] B4PJS:
Any chance that this thread can be stickied, as I believe that it is a very important thread (VIT ;))
[Windows Mobile Software Development] Ameet:
Know there are many mods here too but I took the liberty of sticking this thread :D
[Windows Mobile Software Development] l2tp:
well, i have studied MNGE structure and probably i will write MNGE2Module and Module2MNGE conveter tonight..

if anybody is interested of course...
[Windows Mobile Software Development] ervius:
while you study... ;)

try this unidirection tool MNGE2Module!

here:

http://www.sendspace.com/file/7sr18x

it only decompile MNGE to module into package, while for compile MNGE file is easyer, if works, 10 minutes to add also "-c" (compile fropm module to MNGE) will be inserted! ;)

report if works well!!!

bye!
[Windows Mobile Software Development] v5HMeca:
Did you do a hash check? That would be the best way to tell if there is actually a difference between the two.
[Windows Mobile Software Development] ervius:
new version, decompile and compile again MNGE Files

try it and report if works well!!!

here:

http://www.sendspace.com/file/77y5ia

bye!
[Windows Mobile Software Development] Ameet:
Can someone tell me how to avoid this from happening during cooking? :(

Great, but can you elaborate on what is MNGE and how to find out which modules are to be used with this tool :)
[Windows Mobile Software Development] okekok:
I agree that this an important thread and am sorry I can't contribute to the really skilled work but if there is something unskilled (like helping to maintain databases?) just PM me and I'm willing.