yashsingh1/xdaforums huggingface dataset:
[Windows Mobile] tobbbie:
After getting Typhoon and Hurricane old style kitchen off the bench, I get back to have my old faithful Tornado cooked with OSBuilder :-). It has quite advanced since 1.2.089 and my first cooking has already given an impressive advance in free RAM (need to investigate why) and observed (not yet measured) speed compared to the old batch kitchen.

Some of the speed I attribute to the automatic setting of

"

Disable Demand paging for modules:

gwes.exe

"

Compared to RAM and performance wealthy device like the HD, the Tornado is really limited (Available RAM: 52.72 MB, OMAP 850 200MHz; Free RAM with 4MB PP usually 32MB - but have seen 35 with OSBuilder - edit: last Batchbuild delivers the same, so no advantage here) and needs an optimized approach.

How can I find out which other modules are subject to potential paging activities?

Which other strategies may help to either speed up performance or free up RAM? Usually you never get both the same time - that is why I am puzzled to see that happen with my current build.

My assumptions:

For Speed:

Use XPR instead of LZX compression. I have however not seen noticeably improvement here.

Increase PP size. Above 4 MB seems to deliver little to no value for the RAM and performance capacity of the device. Usually not more than 3 big programs running (like Pocket Payer, JBed, Vito AudioNotes, GPSdash or Sportypal).

prevent paging. Need to try that Kernel Flag thing - but I doubt that it will be usable this way for Tornado. I remember there was also a registry setting for that - not sure if that ever worked. Lot of RAM needed though.

Convert files to modules. I think I have not fully understood the concept of modules vs. files. Especially having .exe as modules seems useless as they cannot stay in slot0 but get an own slot for their process. I know this is a central topic of ROM cooking (and re-location) and there are also some myths attached to it. I would like to discuss this in a separate thread - so if you know one where to go, give me link :-)

For free RAM:

Decrease PP size. Below 3 MB it gets difficult here as the performance suffers noticeably.

Prevent loading of device-drivers that are not needed. Should not happen as only device-drivers that are needed for HW on-board should be in ROM. But sometimes useful for ROMs that shall fit several variants of a device (with/without Camera or WLAN).

Limit filesystem caching

Limit glyph caching

Limit replacement cache

Avoid UPX compression - here I have not seen negative effects yet.

Any more tips for cooking for such a limited device?!
[Windows Mobile] ultrashot:
All modules :)

Disabling demand paging for the most common used modules is better than disabling demand paging at all. As for me, I disable it for almost all oem drivers.

The only good way to determine if your Page Pool is big enough is CeLog usage.

In terms of performance I can only give basic ideas:

1) Demand paging = module won't get its own place in Page Pool. Module will be loaded for unlimited usage, usage without unloading of less used blocks.

2) IMGFS uncompressing = module or file won't be compressed, so paging-in will take less time.

3) If you have default KernelFlags=2, change it to 0.

4) Free up virtual memory (hmm, valid only if you build WM6.5)

a) r/w sharing

b) reversmod some used files.

c) cleanup resource files.

5) Delete CePerf calls.

...and all these ideas are (suddenly) implemented in OSB. They're effective.
[Windows Mobile] tobbbie:
Well - is there an indication (which part of the module, also related to the r/w sharing) as to which parts of a module are suspect to paging?

Understood - but excluding too many will raise RAM demand or not? What will happen to the memory that these "excluded from demand paging" modules need? Will it be taken from the PP and only the remaining PP is for "on demand paging" or is it taken from RAM/VM and the PP is untouched (in fact effectively bigger as there are less modules left comepting for the same PP in case of paging need)?

Will try that if I get it to work on my smartphone.

You mean that "Exclude from Demand paging" would do that, or? So the answer to my question above? -> means more RAM needed because these "excluded from demand paging" modules get their own memory and don't compete in the PP?

Yes, understood, but will that gain be noticeable? I mean these compressions (LZX, XPR) are optimized for short decompression time. Possibly an academic option, or?

What does this do? I have seen that the "Advanced Options" allow to set Kernel Flags but currently these settings are not used. Can they be set elsewhere as well?

Why does that only help with WM6.5? Currently my free memory with 258 modules in Slot0 is 30272 KB - that should be OK, or?

I called the tool, but nothing was listed there - so nothing to clean up here?

Thanks for taking care :-)
[Windows Mobile] ultrashot:
Parts? Well, you can't adjust paging of certain parts.

These modules' mem won't go to Page Pool, they won't be partially loaded/unloaded after some time, they won't add extra activity to imgfs.dll.

Yes to the first question and yes to the second.

It solely depends on device performance. Some will benefit, some won't. My two devices get a visible boost.

Read on MSDN

For example, in stock Leo ROM KF = 2 - more secure kernel mode theoretically, but in practice only slower.

Even on WM6.5 that has two more slots for modules there is a small risk of slot overloading. Overloading will lead to opposite effect, making you having much less memory. On WM6.1 risk is much bigger, reversmodding is slightly less effective. In WM6 (or was it WM5?) even VM align differs (64KB against 4KB), so my advices aren't valid for it...

Real slot0 VM free space can be estimated by using DevHealth & DevHealthAnalyzer, but I can't say for sure it is correct for WM6.1.

Right.
[Windows Mobile] tobbbie:
I mean how can you determine if a file/module has big parts of "pageable" memory? I think this "paging" is not limited to modules or how a file is loaded to memory, it just has certain "parts/sections" (or how you may call it correctly) that can be paged and others that cannot. The overhead created by this paging activity is linked to the amount of memory that needs to be paged in/out and to it's frequency. So best gain is when you can avoid large page-able areas that are paged frequently. I guess the OS has a special logic to optimize here for the available page-pool (which constitutes the area of which the RAM is taken).

For which files can this property of "exclude from demand paging" be set at all? Only modules (e.g. by flags in the relevant sections), all executable ROM files (so also non-modules) - even all files that may be loaded via OS?

Understood - so they just get the RAM they need and keep it as long as loaded. With "no extra activity to imgfs.dll" you mean that the OS has no need to re-read sections/parts of the module from the ROM back to RAM (page-in) because no parts can be discarded. So I conclude that this "exclude from demand paging" is working only for modules (via setting dedicated flags?) - right?

Well this gain is linked to removing overhead of de-compression from imgfs-read. So it would help for modules where parts are paged frequently or for big files. In the end it is a trade in of ROM size with performance. You have to take into account the time it takes to read the bytes from internal memory, then to decompress these and put them to RAM.

I have no big modules left where I expect a gain, but my ROM has many files compressed with UPX. I wonder if UPX-ed files are still compressed in IMGFS and how the read-decompress-execute sequence is handled. Even UPX-ed files have compressible parts (e.g. resources or icons) and large areas that cannot be compressed. It will be very difficult to test for improvements here - so I will not play with that imgfs compress option for now.

Thanks for the link - I have read it. My "problem" is to find out which option my Kernel has set. Where can I find that in OSB? I think that my old OS (nk.exe is from WM5, so WinCE 5.1) is not using these advanced security restrictions on loading modules - I have disabled other security checks in nk.exe as well for performance reasons already.

I have played with this in the past - however I have not experienced problems with free VM yet (in the small range of RAM I have anyway).

Another question to the built-in tools: the list I get with "Cleanup Resource modules/files" shows a lot of elements, some with imports empty, most with imports "1" and few with imports "2" marked in red. What is the purpose of this tool and what does it check? Why are the "2" imports marked red?
[Windows Mobile] tobbbie:
If you have a setup that has no ULDR, it will not add this empty ULDR. This could probably be helpful if you want to migrate to a newer XIP that cannot deal with a missing ULDR partition.

I have not noticed a problem for my old Tornado (WM5 nk.exe) using that setting. I would also not expect any impact here - what is the expected gain by grouping the modules at the beginning of the imgfs partition? Imgfs is no physical harddisk where the saved time to position the magnetic heads could gain performance. Are there any low-level overheads attached when the modules are scattered over the imgfs area? Why recommending not to use it for older devices and pre 6.5 kernels?
[Windows Mobile] ultrashot:
You aren't right - paging is applied only to modules. Files can't be paged.

Yes, right conclusion.

of course, they are.

OSB shows xip info on start. Something like that:

Шаблон - ROM\ROM.tpl:
Начало MBR :0x00000000
Размер сектора :0x0800
Разделы:4
1: BOOT-UL 0x00001000-0x0031FFFF CHS-OK
2: XIP 0x00320000-0x0083FFFF CHS-OK
3: IMGFS 0x00840000-0x0085FFFF CHS-OK
4: FATFS 0x00860000-0x3BDBFFFF CHS-OK

Регионы FLASH:3
0: XIP
1: RDONLY
2: FILESYS

XIP (Шаблон\<XIP>):
DEVICE : HTC-Leo
ROM Date : 30 августа 2010 г.
ROM Version : 3.14.04666
PP(Mb) : 15 (offset: 0x002EF384)
KERNEL : Native WM6.5

dllFirst : 0x01F901FD (Lo=0x01F90000; Hi=0x01FD0000)
dllLast : 0x02000000
physFirst : 0x80000000
physLast : 0x804DB448
modules : 41
RAMStart : 0x804DC000
RAMFree : 0x805CB000
RAMEnd : 0x8E600000
CopyEntries : 7
CopyOffset : 0x802424DC
ProfileLength : 0x00000000
ProfileOffset : 0x00000000
files : 14
[COLOR="Red"][B]KernelFlags : 0x00000002[/B][/COLOR]
FSRamPercent : 0x00000004
DrivglobStart : 0x00000000
DrivglobLength : 0x00000000
CPUType : 0x01C2 (ARMV4I)
MiscFlags : 0x0002
pExtensions : 0x80002AEC
TrackingStart : 0x00000000
TrackingLength : 0x00000000

It isn't an advanced security - in fact it is easily overrideable (OEMs like to override it very much). All WinCE 5.x versions should support it.

well, we all have the same small range of VM (32 MB) ;)

Purpose is to wipe out all unneeded code sections from resource files/modules. They don't need it, but some of them (only OEM/3rd party resource dlls - MS's is better in saving VM) have it. Libraries with two imported functions are marked as red because 2 is still too little for library with functionality, but it can theoretically have some little function set.
[Windows Mobile] tobbbie:
Well not any files, of course, but I thought so far that "paging" refers to an OS activity of mapping virtual and physical memory. See here: http://en.wikipedia.org/wiki/Paging. I can imagine that the WinCE with its concept of having fixed located modules in dedicated VM slots also may have a dedicated mechanism how VM/RAM memory allocation is dealt with for just these modules - separate from the normal RAM/VM handing of the OS for all other executable files.

So I think that referring to the generic term of "paging" that this is done in WinCE memory management also for any other running executable, just that for this not the "Page Pool" is taken but the OS memory management deals with it.

Discussing further down on FSRAMPERCENT I came across the term "swapping" - possibly I need to dig deeper here on the difference of paging and swapping. I am a little rusty on these terms.

So the question is if marking such UPX-ed files as "don't compress" in imgfs (like e.g. other hardly compressible like WMA, JPEG or PNG) will get a good enough relief so that performance gets a gain. On the other hand it would be bad practice if decompression "in vain" would eat lot of CPU here. I guess that compressed files in imgfs are stored in blocks each marked if compressed or not - so no decompression if not compressed. You should know how these compressed imgfs records are written. Is really a 300kB UPX-ed file (e.g. the WMV libraries of WMPlayer) uselessly "compressed" and then without effect "decompressed" again from imgfs?

Are there any objective means to test for such performance improvements?

Not sure if I have ever seen this - will check when I am at home again.

Also here I need to see if that was listed for my device - can't remember that. Want to dig a little deeper here as to what this % share of filesystem in the 1st, 2nd, 3rd 2MB block and the rest of memory shall be good for. MSDN tells that the minimum would be "0x00000020" so your setting "0x00000004" may not be used as you have set it. Why this separation of 3 x 2MB blocks and the "rest of memory"? Would that FS Share be reserved (and not available for other purposes) or just taken if needed? If so what is "if needed"? How does that FS-Share relate to higher level of FS memory allocations (cache)?

OK, so the strategy is to do code cleanup for anything that has to come from the imgfs. The list supplies a set of suspected candidates but this may not be complete (so even 3 imports may be useless) and each has to be checked individually if it really works after cleanup. It could be helpful here to get the potential gain of memory in the list (e.g. the size suspected to be deleted). This guides you to the most promising candidates. Can't remember having seen this if it is already there.

Also other "cleanup" activities could have preview on potential gain, so the "merge all packages to one" or the "strip relocation sections" (btw. it is strip, not stripe - if I get the meaning right).

Thanks for being so helpful! I hope not to bore the others with these deep dives.
[Windows Mobile] ultrashot:
well, we speak about demand paging algorithm, CE5 implementation.

In IMGFS each 4KB-block has its own info (decompressed size, compressed size) and it means UPX-compressed files are compressed twice (first by upx algorithm, then compressed file is compressed by imgfsfromdump's xpr/lzx/xph algorithm). Same routine when you want to decompress it. Same for jpg and other compressed-by-default files - no real need to compress them one more time.

0x00000004 is set by HTC, not by me. You shouldn't pay too much attention to this value, WM seems to be regulating this value itself without our help.

Let's say - *.mui files should be always cleaned. They should never have a code. As for other resource dlls - yes, you should check them manually.

<here is a paragraph about IDA help>

you really should download CE5 Platform Builder (google it) and take a look at kernel source code :)
[Windows Mobile] ceesheim:
here let me help you a bit:

CE5
[Windows Mobile] CRACING:
LOL :D

And hey bro, nice to see you in Windows Mobile Developments. Any plan on WM dev? :p

Thanks...

Best Regards
[Windows Mobile] AndrewSh:
I think Cees here is just to countenance his friends (Barin, ultra, Ondra, HD2O & me) :)
[Windows Mobile] ultrashot:
From user side - yes. But OSB can set which sections ImgFsFromDump should compress and which no. ImgFsFromDump then just builds final image keeping in mind which sections should be compressed. If cecompr_nt.dll thinks one of 4KB-blocks shouldn't be compressed, it just returns given size as result. If decompressedSize=compressedSize, kernel won't call decompressing routine :)

wealthy... X1, just like Blackstone and others, aren't that wealthy as you think. They have a lot of performance problems caused by poor drivers and other oem things. Even network isn't good on X1 (shame on HTC - they developed it). We're trying to take everything out of these devices and my advices are a big part of performance tweaks I use besides, of course, driver patch work which is essential on X1 (again shame on HTC).

Well, it isn't that much related to certificate checking. What CertPatcher does? It patches VerifyBinary function making kernel think module/file has highest priviledges.

What about full kernel mode? If it is enabled, there is no extra time used while calling API functions or using SetKMode/SetProcPermissions routines. It should lead to permanent performance gain.

Every little bit helps. The more VM you have overall, the more it will take for device to run out of memory. It improves performance indirectly.

MS wrote a lot of articles about VM - and as they spend a lot of time on this topic, it means it is very important. Bad that MS hasn't ported WM to CE6 kernel - it would have been interesting to have issue-free (in terms of VM) WM.
[Windows Mobile] Nixeus:
Hello :)

Thanks a lot for this new release :)

I have a little error in my build process now

Any idea please ?

Thanks a lot,

Best regards,

Nixeus
[Windows Mobile] tobbbie:
So it means that non-compressible parts of a file are just stored unchanged - good (and expected). I see the point that OSB can do clever things on the module parts. Performance benefit depends on how frequently these uncompressed areas are read. I can understand that OSB options for imgfs are intended to tweak module access from there - as this may repeat over time many times just depending on the need to have these modules loaded. I think however that for already compressed files (and I talk about UPX-ed exe or dll in the build) it may not supply such a big gain as for modules. Looking at size of a build (and yes, the Tornado has not much here either...) it is best to have executable files UPX-ed and still run through imgfs compression as there also the icons/resources (left untouched by UPX in my settings) are treated. Finally it is a trade of build-size with performance.

That sounds interesting - thanks for the insight, I will try that.
[Windows Mobile] Nixeus:
Hope we have the XPH compression soon :)
[Windows Mobile] AndrewSh:
Hope dies last :)
[Windows Mobile] Nixeus:
Waiting WAiting :D :D
[Windows Mobile] tobbbie:
What about cleaning up the resource files from icons that are linked to certain DPI which are not applicable in your build? Some are already in DPI specific packages, but some may be generic in the way to just include all. No clue if that link of DPI and used icons is easy to determine and if it would be safe to remove them without effect on the module's usability.
[Windows Mobile] -=Barin=-:
OSBuilder V1.2.153 with XPH support is ready.
[Windows Mobile] AndrewSh:
Here is the cecompr for new compression method http://xdaforums.com/showpost.php?p=17570836&postcount=1
[Windows Mobile] HD2Owner:
thanks alot barin/ultrashot. great work!
[Windows Mobile] AndrewSh:
Changelog for OS Builder V1.2.153 FULL (15.09.2011) - now with XPH support

1. While registry preview or rom-cooking: if reg settings are not tied to a key (after removing the key directives), a comprehended syntax error indicating the line is displayed

2. Reloc engine is modified – now Reloc and Recmod is supported for WP7 Modules (REVERSMOD – still under development) for all new fixup types (3 - IMAGE_REL_BASED_HIGHLOW, 5 - IMAGE_REL_BASED_ARM_MOV32A and 7 - IMAGE_REL_BASED_ARM_MOV32T)

3. Added support for XPH compression - see the settings in Build Dump and ROM.

! ATTENTION!

To enable XPH compression – you would also replace cecompr.dll for that library in XIP.

It’s cecompr.dll ported from Mango WP7.1. Guy in charge – ultrashot (who doubts?) Thanx to him for fixing ordinals in the import table for WM6.5 compatibility.

Warning! This library contains the ARM v7 instructions, so it will only work on devices that support these instructions ( particularly on devices with Snapdragon CPU on board - like HTC HD2)
[Windows Mobile] Nixeus:
Thanks a lot for this new release and all these new features ! Testing it !
[Windows Mobile] CRACING:
Thanks for the new version.

I have replaced the cecompr.dll to XPH enabled cecompr.dll in OEMXIPKernel and also selected XPH in OsBuilder Settings but still I see XPH is not supported. :confused:

Like this:

XIP (cecompr.dll):

XPR: is supported

LZX: is not supported

XPH: is not supported

Am I missing something? or its correct and I can go forward cooking ROM?

BTW; I can smell the taste of WP7 ROM being cooked on OsBuilder upcoming release. :D

Great work guys, Keep it Up

Thanks...

Best Regards
[Windows Mobile] AndrewSh:
CRACING, leave the XIP folder untouched

Just put new cecompr somewhere and define the path to it in Settings/XIP of OsBuilder. Then - define XPN compression in Settings/Build Dump and Rom

See the example on my screenshots.
[Windows Mobile] CRACING:
Hmm this idea seems better.

BTW; I have cooked the ROM and at the end it says Building IMGFS partition file under XPH compression. Flashed and works g8. :)

Thanks...

Best Regards
[Windows Mobile] erickwill:
Thanks for your awesome work my friend!
[Windows Mobile] ai6908:
Anyone using xph compression with rhod-w? I couldnt get a bootable ROM with that compression setting!
[Windows Mobile] Jackos:
It's because Rhodium devices use different processor architecture. It's not possible to use ARM v7 instruction-set on a ARM v6 device.
[Windows Mobile] AndrewSh:
You have Qualcomm MSM7600 and I wrote the warning at the last line of Changelog:

Warning! This library contains the ARM v7 instructions, so it will only work on devices that support these instructions ( particularly on devices with Snapdragon CPU on board - like HTC HD2)
[Windows Mobile] ai6908:
Ah thanks Andrew! Well you know some people (me) cant read! LoL..:D
[Windows Mobile] ffboy2009:
Volume key problem?

THX for this great kitchen tool which makes non-htc deviece's rom cooking more easy and improve WM performance.

when i upgrade OSB to latest 1.2.153 FULL version,i found a problem occur in my TG01,my Volume key won't work at all...

without changeing anything it always came this problem

when i changed SYS version 29021&29022,OEM uses official,it also get this problem

i wonder if anyone can help solve this?

here's my build logs,hope u people can see and help me!

Regards
[Windows Mobile] ai6908:
I finally decided to switch to OSB, but there is one bug that I cannot figure out how to solve. For some reason, messaging enhancement does not let my emails open up if cooked in.

If the exclude that pkg, then emails open just fine. I have tried different versions right from the time sprint came out with a ROM, to latest OBOE ROM and it hasnt fixed a thing.

If I install the html files and the dll files separate by a cab over the same cooked pkg, everything works fine and dandy!

Has anybody run into this kinda issue before?
[Windows Mobile] AndrewSh:
I suspect - you have to do something with build order. Maybe Messages Enhancements rgu is overwritten by some other package containing the same registry keys but with different values. Check the build order and make registry preview to be sure that no **** happen to correspondent rgu meanings. You can change building order by modifying packages cif and sof. Try to move Messages Enhancement closer to the end in the order list.
[Windows Mobile] tobbbie:
This is one of the really advanced options of OSB. While in older kitchen you had to deal with .provxml or even later configuration options, you can set the sequence of the registry build in the order you want right away!

There are so many things to clean up when you move from an old kitchen...