yashsingh1/xdaforums huggingface dataset:
[Windows Mobile Software Development] ervius:
for posted netcf example, the optimum is:

cabarc.exe - 20 N new_pack.cab.

pkg package\*.*

so, header (-s 20 reserve space for sign!), is same size, but how and with , sign it, I don't know!

shure, but now maybe we have to fight against right sign code!?!? ;)

bye!
[Windows Mobile Software Development] Da_G:
Yes, I think the ROM will need be cooked with additional certs, in \SYS\Metadata\DefaultCerts.dat - these appear to be the certs that are being checked against. So we can replace with SDKCerts, sign .cab.pkgs with that, should be good!
[Windows Mobile Software Development] ervius:
ok, go to work then, I ', yet thiniing about oldstyle buildos with this all new features of visualkitchen without platformrebuilder if someone wants use oldstyle ;) (maybe I'm at my goal!)
[Windows Mobile Software Development] DaveShaw:
Can we not look at removing the signing check the same way cmonex did for the kernel? Or is that the same signing check?

Just so I am getting this right? I could cook a ROM with a custom DaveShaw https update server IP and then provide automatic updates to my ROM, bug fixes, new build release, normal CABs, etc. all using Windows Update from my website??

That would be damn useful, no more re-flashes :)

Dave
[Windows Mobile Software Development] l2tp:
probably to create MNGE from module folder we just need

copy /b imageinfo.bin + S000 + S001 + ... module.dll

and add MNGE header to the beginning of the file.

but maybe i am wrong and this will not work in all cases.

at least when i converted in this way dll module from 21725 to MNGE and compared it to the same file which was originally in MNGE format, there were 0 differences.
[Windows Mobile Software Development] inertiax3:
I have an updated WM65 NK.EXE (stored as a file with MNGE header, specific for my device) and for the love of god don't know how else to update the existing XIP with the new NK.EXE.. so I guess ImageUpdate would take care of that for me :)
[Windows Mobile Software Development] Diewi:
Take a look at post 14 and 16 of this thread. There you'll find the answer.
[Windows Mobile Software Development] Da_G:
IRT this post:

http://xdaforums.com/showpost.php?p=3976205&postcount=6247

Not sure how htcRIE handles the free space in imgfs, imgfsto/fromdump has "bigstorage" mode that shrinks imgfs partition to fit, and -conservative to stop that behavior (which we want, because we want to keep the free space buffer)
[Windows Mobile Software Development] anryl:
so the kitchen would be like this ?

CREATE DUMP

@echo off
cd temp
DEL *.* /S /Q
RD DUMP /S /Q
copy ..\OSORIG\os.nb os.nb
..\tools\imgfs\nbsplit -kaiser os.nb
..\tools\imgfs\imgfsfromnb os.nb.payload imgfs.bin -conservative
..\tools\imgfs\imgfstodump imgfs.bin
CD ..

MODIFY DUMP AND INSERT NEW XIP (XIPPORT)

INSERT CERT ? WHERE ?

CREATE ROM

@echo off
cd temp
..\tools\wmreloc -gdump
..\tools\imgfs\imgfsfromdump imgfs.bin imgfs-new.bin
..\tools\imgfs\imgfstonb imgfs-new.bin OS.nb.payload OS-new.nb.payload -conservative
..\TOOLS\IMGFS\NBMerge -kaiser OS-new.nb
..\tools\htcrt.exe /buildrom "..\raphael.htcrtproj" "%cd%\ruu_signed.nbh"
cd..

AND FOR PKG

cabarc N %1.cab.pkg %1\*.*
signcode -spc TestCert_Privileged.cer -v TestCert_Privileged.pvk -t http://timestamp.verisign.com/scripts/timstamp.dll %1.cab.pkg

:confused:
[Windows Mobile Software Development] Da_G:
Cert goes to defaultcerts.dat (belongs to metadata package)

for the last part, cabarc line needs -m NONE to indicate no compression, and -a SHA1 to use SHA1 hashing algorithm rather than MD5, otherwise, yes, looks good :)
[Windows Mobile Software Development] Da_G:
I've posted the first fully functional .cab.pks for the imageupdate test ROM over in the raphael section,

Let's see how things look,

I've tested it on my end and it appears to be working fully, it remains after a hard reset :)

The tools are coming together! A fully operational proper .dsm editor is the next key step :)
[Windows Mobile Software Development] anryl:
nice pks just installed one pkg OK out off it :D

now installing the whole pks :cool: succesfull
[Windows Mobile Software Development] rjmjr69:
This is so exciting keep up the great work you guys and gals are amazing thanks so much for your hard work
[Windows Mobile Software Development] da1stprince:
wow....the possibilities!
[Windows Mobile Software Development] ervius:
in my 3d of visualkitchen, I've posted a 1.3 version that with the support of dsm_editor, by dos prompt, shoud build an old_style rom, ready to be used to update with .pkg files, try it and try to update some rom parts with .pkg files, and report if works well....

(uldr and conservative mode have to be untouched from original .payload file to permitt that all work well!)
[Windows Mobile Software Development] Da_G:
great work ervius, i will test as soon as i get home from my trip :)

a little more info on how uldr handles the imgfs update - i did some analysis on the dumped imgfs after cab.pkg installation - after which i dumped the uldr and did some analysis there -

it appears uldr comes with a special read/write version of the imgfs driver - uldr is loaded entirely into ram (so that the filesystem can be completely untouched by anything other than the imgfs driver)

it is capable of all normal functions you would expect a filesystem driver to do, including deletion of files :) so it should be possible to use uldr to also delete things from imgfs (although i am not sure how at this moment)
[Windows Mobile Software Development] elektrownik:
I'm stupid, but cant we use uldr imgfs driver with wm to make files r/w from os ?
[Windows Mobile Software Development] ervius:
dsm_editor 1.7 some bugs solved and right files/modules flags read & write ;)

DA_G, try it (also to build old_style roms to repalce with existing into \tools folder!)

download here:

http://www.sendspace.com/file/c66tlc

bye!
[Windows Mobile Software Development] ND4SPD:
Nice job ervius! My tool that rebuilds the dsms from scratch should be up some time tomorrow - I've still got to finish consolidating the software.

I'm trying your kitchen now :D
[Windows Mobile Software Development] wootty2000:
So reading between the lines, would the following be possible.

Take a dump of a vendor ROM

Find a similar, but slightly older HTC ROM

Create the deltas, and make a image update

Install any other ROM via HardSPL and play

Have a real fault with the phone that requires going back to the vendor (say keyboard failure) under warranty or insurance

Install the HTC ROM and perform the update made earlier to "return" the ROM back to the vendor's version.

I ask because, when Orange UK released the Tytn2 I really wanted to play with cooked ROMs but was afraid of having a fault with the phone and Orange rejecting the warranty because I had flashed it.

It took Orange 6 to 9 months (if I remember right) to release an update. Only then was I happy of using cooked ROMs.

As it happened, the did have a keyboard failure and was able to flash the Orange ROM back before sending it back on the insurance
[Windows Mobile Software Development] SGregory:
When I try to "install" netcf.cab.pkg, I have unsuccessful status with result code 80180015. Someone knows what could be wrong?
[Windows Mobile Software Development] inertiax3:
@ervius

Please, have you further developed your MNGE->MODULE conversion tool? The modules that I get with the last version of your tool are not relocable for some reason :(
[Windows Mobile Software Development] cmonex:
I have one such device (was in production too). but it is not HTC. it's pretty cool, but the OEM ****ed up other things performance wise.

I wouldn't put the bootloader in nor flash though. less reliable than NAND with ECC. (of course it is better than raw nand but who uses raw nand in these devices?!?!)

as for XIP'ing vs performance, nor flash is still slower to read than ram, so the best is a device with a load of ram that then gets entire OS loaded into it. I have another device that only has nor and its bootloader has options to load OS as XIP in nor or load it as XIP in ram.

the latter option is the default, as it makes the device screaming fast. XIP in nor isnt as fast as that.

of course if you were to implement this, the OS could be stored decompressed in ram (well, it can be compressed imgfs too, if you want, but decompressed image is even faster but eats more ram of course). and to make boot times better, have an internal backup battery thingie, that can keep the ram powered when you soft reset, or change batteries, or this OS would always have to get reloaded and that'd make boot very slow. it could of course still have persistent storage, that is not related to how the *read-only* OS image itself is stored.

well I guess we could talk a lot more about how to optimize performance assuming we don't mind the costs for flash and ram chips. that'd be a new thread though :D
[Windows Mobile Software Development] cmonex:
no, would be too much pain, uldr itself already has to do a lot of painful work.
[Windows Mobile Software Development] cmonex:
I'd like to know these answers myself too! so I think uldr should only be used for things like a rom has some small bugs left in it and you don't want people to hard reset just to install a permanent fix to those...

btw I dont think uldr handles XIP, it's just IMGFS updater, no? maybe I'm wrong and it can update XIP itself too (though these WM5+ XIPs dont have chains)
[Windows Mobile Software Development] inertiax3:
Da_G has managed to update the MSXIPKernel part of the XIP so I guess it also handles the XIP..
[Windows Mobile Software Development] mbarvian:
since .cabs are based on _setup.xml, could we also remove files using these .cab.pkgs?
[Windows Mobile Software Development] Da_G:
IIRC, ULDR has support for 3 types of partition

RAMIMAGE (NK partition, XIP)

ROMIMAGE (used in CE)

IMGFS (OS partition)

Can update all 3 :)

When ULDR writes RAMIMAGE, it copies entire image off the flash to memory, relocs etc, and then writes new image back to flash

So fragmentation, etc. isn't an issue - just free space :)
[Windows Mobile Software Development] stepw:
Neat-o! So it's important to maintain power while ULDR erases/overwrites XIP with an image from RAM.
[Windows Mobile Software Development] stepw:
More RAM, less battery lifetime. A non-rechargeable backup battery will not survive long if it has to provide power to RAM while main battery is offline.
[Windows Mobile Software Development] pidsw:
I've seen a few NOR/NAND Hybrid devices in the works, the should be more common in high end devices next year.

Regarding storage, this may be a niave question, but the memory allocated to potential image updates is declared in a texture file for the memory layout of a device BSP. Is this fixed in hardware, or just a decision based on the OS? If it is fixed in hardware, it will really hurt existing devices on WM7 as many things have been moved around to improve performance.

Aside: MS recommends that OEMS put all frequently used DLL's into modules as they use virutal memory RAM more efficiently than Files and that are faster to load and slightly faster to execute. Part of the execution benefit based on the assumption that the "rebuilding" that imageupdate and cooking does helps to minimize TLS swapping and other virtual memory overhead which can be quite a perf loss on many devices which require software interrupts to update their TLS (VM->Physical map)

We often get swapping of system dlls when we have the potential for ~120 megs of accesseble memory (much of it read only however) if the "big hammer" sections are filed. This should not happen, and hurts responsivness which is arguably one of the most measures of performance on a phone.

I noticed that custom roms have far fewer modules, than the latest HTC WM6.5 drops - is there a reason for this that I do not know about?
[Windows Mobile Software Development] anoopnayak:
Need some inputs on Language packages

Hi,

I am not sure I am asking the query in right place but my quesry is related to Image update so thought to get some inputs from you. I have set of language packages for a perticular language and when I pass the input file to the Update application it installs all the language packages and restarts but after that I get a UI saying all tha packages are installed properly. Even after installing the language packs the device bootsup in the base language (0409). What might be the reason for the same? It would be great if I can get some inputs on the same as.

thanks

Anks
[Windows Mobile Software Development] chavonbravo:
Try changing values of these keys:

[HKEY_LOCAL_MACHINE\MUI]

"SysLang"=dword:00000409

[HKEY_CURRENT_USER\MUI]

"CurLang"=dword:00000409
[Windows Mobile Software Development] drellisdee:
Exactly how do you add another cert to the defaultcerts.dat?
[Windows Mobile Software Development] ND4SPD:
open cmd window:

type: copy /b DefaultCerts.dat+"cert file name".cer DefaultCerts.dat

If you get an error, it's probably because DefaultCerts.dat need read-only protection taken off.
[Windows Mobile Software Development] xda2_haseeb:
Can i have a step by step guide of what should i do to get Image Update System working?? Can i also know how-to make a certificate, and i guess adding procedure is given in the post above, am i right?? I also want to know how-to create *.cab.pkg files?? THNX!!
[Windows Mobile Software Development] ND4SPD:
I really wish I could help with that, but I can't even get a proper Image Update ROM flashed onto my phone without any errors. However, once you have a ROM that can handle Image Update, here's what you need to do:

Re-build all the dsms properly - apparently you don't need it to check against the certs in the dsms, so that's fine. You can use Ervius's DSM Editor to do so.

Update the DefaultCerts.dat to have the cert that you are going to sign all of the .cab.pkg files with. This will be a .cer file.

Using cabarc, build all of the .cab.pkg (you can specifiy the output file to end with .cab.pkg instead of just .cab)

Sign all of the .cab.pkg files with the .spc and the .pvk files that go along with the .cer file.

Figure out all of the dependencies of a specific package and then cabarc all of the required .cab.pkg files into a .cab.pks file - you don't need to sign this .cab.pks. If you don't have the time or don't want to bother figuring out the dependencies - you should probably just put all of the .cab.pkgs you generated into one big .cab.pks file. If there aren't any dependencies, you can just copy the specific .cab.pkg file onto the device and then run it. It's the same for the .cab.pks file - once you have it, copy it to the device and then run it from there. The Update program should automatically attempt to install at that point.

If you create a Release folder in the root of your device, you'll also be able to get a UpdateValidator.log file that will tell you exactly where you got an error if something did go wrong.
[Windows Mobile Software Development] xda2_haseeb:
Thnx!! its really informative but seems to be too much difficult :p so ill try sticking to normal ROMs for now :D
[Windows Mobile Software Development] chavonbravo:
I'm wondering how all this would work for updating a device w/out hardspl and an with the original rom.

1) Is it possible to edit existing .dsm's inside the rom, like Transciber package for exampe?

2) If so, could one edit it to be empty (in other words, remove Transcriber), and still have the .dsm properly signed?

3) Is it possible to extract certificate that is used in rom and use it on new .dsm? I doubt this is possible.

4) So if that can't be done, can DefaultCerts.dat be overwritten in ROM to include your own certificate? Or maybe there's some sort of protection to prevent this.

Reason I ask is for a TG01, which doesn't have any known update or flash program yet, so maybe image update is the only viable option for now.
[Windows Mobile Software Development] ND4SPD:
As far as I know, it's not possible to edit the stuff in the ROM. According to Da_G the TFAT partition (the user partition) which contains the windows folder, "shadows" the IMGFS partition. In other words, even if you changed something in the windows folder, it wouldn't change the actual ROM since that is located in the IMGFS partition, not the TFAT partition.

Image Update is in fact designed to run without HardSPL or any such thing. Because it doesn't go check against the SPL or the IPL to make sure it's from an authentic source, it checks the certificates in the individual Image Update packages against the DefaultCerts.dat. In your case, the ROM will be signed with Microsoft's Firmware Certificates and Toshiba's own certificates. Unless you can get access to Toshiba's own .pvk files for the certificate they files are signed with, this will not be a viable option for you.

There is however, another thing you should explore, but only if you are experienced. Itsme's tools can live read/write to the IMGFS partition. This way, you can update the ROM to contain the proper certs/dsms that will allow you use Image Update to update the system. This is very risky, and you can easily brick your phone with the slightest mistake

I'll need to check once more, but I don't think that WM6.5 supports Image Update packages that deletes the software. You can use an update provxml in that specific package that will delete those files from the ROM.

Good luck!
[Windows Mobile Software Development] chavonbravo:
You speak of pdocwrite, right? If so, I imagine the old and new defaultcert.dat files should be same size, and I'd have to find the offsets where the old is with pdocread, and then write a same file size in that offset with pdocwrite?
[Windows Mobile Software Development] ND4SPD:
Yeah, i'm talking about the pdocwrite/pdocread tools. I think what you said would work. The way I'm going to try it is by dumping the whole rom with pdocread, making the changes, and then writing it back with pdocread again. Just remember that the new defaultcerts must be the exact same as the old one or you're going to have some errors.
[Windows Mobile Software Development] chavonbravo:
For that to work, I'd need a 440 byte certificate to replace the existing Toshiba one. How can you go about creating a certificate that you know will be that size? Every one I've seen is bigger.

edit: had to play quite a bit with makecert.exe options until i got the needed 440byte size, so no probs. Just gotta try it now.
[Windows Mobile Software Development] chavonbravo:
Well, don't think that'll work. I've tried, and get the error:

CopyFileToTFFS(imgfsnew.bin:0, 0, 05140000)

ERROR: ITWriteDisk - The media is write protected.

So IMGFS is write protected from itsme tools as well. Maybe there's a flag in registry that can be set to change that, but I doubt it.
[Windows Mobile Software Development] ND4SPD:
did you try dumping the rom to your device and then writing that back to the device or did you just replace specific scertificate.
[Windows Mobile Software Development] chavonbravo:
The whole thing. Can't really do only a part as imgfs is compressed, so you can't just copy/paste the part you need.
[Windows Mobile Software Development] chavonbravo:
Here's a sample of an official Toshiba ImgUpd file. Wouldn't work on my tg01 because the update file is for an older version released in Japan. Have the log in it as well. Just if it interests someone.
[Windows Mobile Software Development] chavonbravo:
Just wondering what kind of verification is done on the .cab.pkg from ULDR side. I figure the verification could be maybe bypassed on IMGFS by overwriting the necessary FWUPDATE files with patched version. Any ideas?
[Windows Mobile Software Development] ND4SPD:
I think it's just the same cert checking - you sign cab/dsm with pvk and spc and then it checks against the x509 cer file. If you could patch that, that would be great!

btw, I was looking through the source code for the IMGFS tools that tadzio made and noticed that the capabilities for making image update ROMS was removed. In fact - tadzio rights in a comment that if the value of a certain header is set to a non-0 number, image update will work. However, I don't know enough to make a smart change to it without risking my device. Anyone who know's their C++ should take a look at the source code to tadzio's tools.
[Windows Mobile Software Development] Da_G:
I started that project this weekend actually :)

I'm not basing on tadzio/mamaich's code though, starting from scratch.. I don't want to use the os.nb.payload file.. waste of space.. can just have it dump the partitions from the .nb and re-create it when it's done :)

Hopefully it doesnt take me too long! I'm shooting for september for a decently working beta :)

I plan to integrate all the good stuff we'd need for Image Update like relocating modules in OS/NK, defining free space reserved in OS/NK, etc :)
[Windows Mobile Software Development] jwzg:
Do the switches (i.e. -conservative and -bigstoragemove) determine whether the ULDR partition is in place or simply the size of the imgfs?
[Windows Mobile Software Development] Da_G:
in -conservative mode imgfstools retains any free sectors at the end of the imgfs (i.e. space reserved for image update)

in -bigstorage mode it removes that free space, 'reclaiming' it for TFAT :)
[Windows Mobile Software Development] B4PJS:
Is there any way of getting -conservative to work with visual kitchen new style roms? I changed -bigstoragemode to -conservative and it just errorred :(
[Windows Mobile Software Development] Da_G:
I'm not sure, never tried..

You have to stay within the size of the original imgfs partition, to use -conservative i believe..

Also the "new style" erases all .dsm and strips the reloc section from all modules, so it wouldn't work at all for Image Update, since both are essential :)
[Windows Mobile Software Development] jwzg:
Can you post the old-style tools to use on older devices such as the Hermes? (edit: there are no tools before Tadzio's of which I'm aware that do NOT produce bad NAND blocks, so nevermind)

I was able to get the size of imgfs to the original in Bepe's old Hermes kitchen w/Tadzio's tools using the -conservative switch. I have 44.65MB of storage memory available instead of 47-48MB using -bigstoragemove, and three partitions show up when nbmerge has run (I assume these are imgfs, TFAT and uldr). When the ROM is dumped and pulled apart, all of the numbers show up correctly in PKGTool, but the files all show up in the SYS directory. How did the numbers show up right? I simply used the unaltered DSM's from the 3.29 Kaiser ROM I ported. I even used the DSM from the Kaiser OEMDrivers folder and just changed the GUID on the RGU file to match. (I don't make my ROM's with separate packages like most people do). Because of this, we know the DSM's are not completely altered, but that's as far I can go. I'm not a coder, so guys, have at it.
[Windows Mobile Software Development] jwzg:
Yes, the imgfs size must be kept within original parameters. Also, os.nb.extra must be present in the temp folder when nbmerge is run.
[Windows Mobile Software Development] ND4SPD:
I've patched the ImgfsFromDump.exe tool to include the free space for image update so that I don't get that error any more using bigstorage (-conservative handle fails for the Diamond).

I'll post it soon! I'm visiting grandparents in India right now and they don't have an internet connection - i have to go to an internet cafe to post
[Windows Mobile Software Development] jwzg:
I think that Ervius might be on to getting this fixed with the visual kitchen (see here), but post the ImgfsFromDump and the switch needed to make it work.
[Windows Mobile Software Development] ervius:
maybe there is a bug also on dsm_editor....

with new style ext_packages it assumes as "files" the name of package instead of real "package_name"

have to investigate about!
[Windows Mobile Software Development] ND4SPD:
yeah...kind of forgot to put a switch lol i'll do that now