yashsingh1/xdaforums huggingface dataset:
[Windows Mobile Software Development] 6Fg8:
:) compressonator was my "color meter" during the reverse process

the results are pretty close. I did a series of testpaints (greyscale, colorshades), and couldnt find any problems so far.

I'll build that into an app so you can try on your own.
[Windows Mobile Software Development] Chainfire:
Cool, already trying to apply your formula in my own QTC viewer... :)

(eventually all this stuff should be in CFC, and CFC will be available as DLL as well)
[Windows Mobile Software Development] 6Fg8:
ah, one shouldnt be too quick. There's 2 more things to mention:

amplyfing colors is actually done this way (the other formula wont reach whitelevels):

color = factor * (color + 1 ) - 1

where factor is 8 for R1,G1,B1,R2,B2, and factor is 4 for G2

And the second: alpha *= 16
[Windows Mobile Software Development] D-MAN666:
alpha /= 15 for [0..1] range

and alpha *= 17 for [0..255] range :)
[Windows Mobile Software Development] Chainfire:
Seems to work well for images like the moon, but normal graphics are still not good, at least for me.
[Windows Mobile Software Development] 6Fg8:
could you attach a "normal graphic"? Just curious ...
[Windows Mobile Software Development] eidolen:
Hi Chain,

I wanted to know if it was possible to just do a flip and not scale with CFC so I can skip the QVGA scale part but still do the new compress after all the pngs are modified. Am I missing something?

Eidolen
[Windows Mobile Software Development] Chainfire:
I just figured something... The pixels that render really bad have the leftmost bit of key1 set... the bit we ignore in your formula! Trying to figure out what the deal is with these pixels.

I will in a sec... just reading this now, when I wanted to post this info.

You are right, CFC does not include flipping without scaling... Next version?
[Windows Mobile Software Development] 6Fg8:
Right, i didnt test with ATC. Already wondering they wasted a bit :)
[Windows Mobile Software Development] 6Fg8:
of course, D-MAN, you're right :)
[Windows Mobile Software Development] D-MAN666:
Hey, plz someone point me where i'm dumb :)

var colorChunk1: Number = chunk.readUnsignedShort();
var colorChunk2: Number = chunk.readUnsignedShort();

var r1: Number = (((((colorChunk1 >> 10) & 0x1F) + 1) << 3) - 1);
var g1: Number = (((((colorChunk1 >> 5) & 0x1F) + 1) << 3) - 1);
var b1: Number = ((((colorChunk1 & 0x1F) + 1) << 3) - 1);
var r2: Number = (((((colorChunk2 >> 11) & 0x1F) + 1) << 3) - 1);
var g2: Number = (((((colorChunk2 >> 5) & 0x3F) + 1) << 2) - 1);
var b2: Number = ((((colorChunk2 & 0x1F) + 1) << 3) - 1);

var keyColor1: RGBColor = new RGBColor(r1, g1, b1);
var keyColor2: RGBColor = new RGBColor(r2, g2, b2);

Produces wrong key colors for me :(

Update:

it was all about sex. byte sex. works as a charm now :D
[Windows Mobile Software Development] D-MAN666:
but now it screws up the black level :)

The factor itself s/b modified, i.e. 255/63 for G2 and 255/31 for other components
[Windows Mobile Software Development] Chainfire:
im programmatically comparing the outputs, this indeed is MUCH better!
[Windows Mobile Software Development] 6Fg8:
its only valid for color > 0

how do you modify the factor in your implementation?
[Windows Mobile Software Development] D-MAN666:
I not quite getting what do you mean :)
[Windows Mobile Software Development] Chainfire:
Even including that adjustments, D-MAN's formula creates a better match with the PNG's created by The Compressonator.

In fact, if you discard the blocks with the leftmost bit set, and allow for a single value difference per pixel (for diffirent rounding), they're mostly correct now... but not perfect.

EDIT: Note that pixels that still do not match seem to mostly be edges of things... wonder if The Compressonator does some anti-aliasing or smth...
[Windows Mobile Software Development] 6Fg8:
puh ... confusing, might be too many hours on the computer ...

what formula?

Regarding the color amplyfing, the formula

color = factor * (color + 1 ) - 1

is only valid for color > 0
[Windows Mobile Software Development] D-MAN666:
This way the color space in not uniform (plain math) :) (0 -> 0, 1 -> 15, 2 -> 23, ...) 15 between 0 & 1, and 8 between 2 & 3 and further. wtf? :)

The problem is what you did an

almost right

guess re weights. But they're not 8 and 4, though it' would look quite "sexy". What we need is to convert 5 or 6 bit range into 8 bit range.

So weights will be (255/31 ~= 8.2258064516129032258064516129032) and (255/63 ~= 4.047619047619047619047619047619).
[Windows Mobile Software Development] Chainfire:
Re: the awkward pixels (not the ones with the left bit of key one set, but others), the difference seems to be 7, 8 or 9 99% of the time. Perhaps your math thoughts can figure that out :)
[Windows Mobile Software Development] 6Fg8:
Yes, correct. Last time i handled such stuff no floating point arithmetic was available :D
[Windows Mobile Software Development] D-MAN666:
btw, re alpha: you can get a speedup by replacing alpha *= 17 (floating point) with alpha = (alpha << 4) & alpha (bitwise) :)
[Windows Mobile Software Development] 6Fg8:
reminds me of some bitfreaking times back in goold old ASM :)
[Windows Mobile Software Development] D-MAN666:
Chainfire, it could be float2byte related problem. How do you convert? Rounding or flooring?

6Fg8, :D
[Windows Mobile Software Development] Chainfire:
I only round at the very last moment, so that should never make more than 1 in difference... I tried rounding at very places in between (round, floor, ceil) but the difference remains approx the same.
[Windows Mobile Software Development] Chainfire:
Some more findings:

if set(leftmostbit key1) {
if mixdown = 0 {
color = 0
}
if mixdown = 2 {
color = key1
}
if mixdown = 3 {
color = normal-formula
}
}

Can't seem to figure out mixdown = 1 though...

EDITED: Added mixdown = 2
[Windows Mobile Software Development] eidolen:
When someone has time to shift gears a sec was hoping you could help with a question. I have been doing testing on the new files and ran into a "slight" problem. (Yes, same file)

I can not find the exact resolution/formula to have the exact image come out, the way it was made. Making images @ 512x256, 256x128, etc works but it distorts the image. The actual image that appears on the device is 480x216. I have tried using that resolution and a few others but it doesn't seem to want anything other than factors of 64. (Otherwise white rectangle) I even tried placing a 480x216 image on a 512x256 rgba palette at location 0.0 but still no joy.

I guess it's possible not to be able to do a true 1:1 reproduction but if there is a way, I figured you guys would know. Seems like a math ratio thing that I'm missing.

Eidolen
[Windows Mobile Software Development] Chainfire:
alternate formula solved!

when the leftmost bit of key1 (k1k in my code) is set, the following rules should be applied depending on the mixdown values (pixels in my code).

if mixdown = 3 use the normal formula we already know and love to hate, if not:

case pixels[i] of
0: begin
r := 0;
g := 0;
b := 0;
end;
1: begin
// if < 0, value = 0
r := Max(0, Round((k2r / -4) + k1r);
g := Max(0, Round((k2g / -4) + k1g);
b := Max(0, Round((k2b / -4) + k1b);
2: begin
r := Round(k1r);
g := Round(k1g);
b := Round(k1b);
end;
end;

or into a single line per color-type:

r := Max(0, Round(((k2r * (2 - pixels[i])) / -4) + k1r)) * Min(pixels[i], 1);
g := Max(0, Round(((k2g * (2 - pixels[i])) / -4) + k1g)) * Min(pixels[i], 1);
b := Max(0, Round(((k2b * (2 - pixels[i])) / -4) + k1b)) * Min(pixels[i], 1);

making the code thus:

a := alpha[i] * 17;

if (k1k = 0) OR (pixels[i] = 3) then begin
r := Round( ((k2r * pixels[i]) / 3) + ((k1r * (3 - pixels[i])) / 3) );
g := Round( ((k2g * pixels[i]) / 3) + ((k1g * (3 - pixels[i])) / 3) );
b := Round( ((k2b * pixels[i]) / 3) + ((k1b * (3 - pixels[i])) / 3) );
end else begin
r := Max(0, Round(((k2r * (2 - pixels[i])) / -4) + k1r)) * Min(pixels[i], 1);
g := Max(0, Round(((k2g * (2 - pixels[i])) / -4) + k1g)) * Min(pixels[i], 1);
b := Max(0, Round(((k2b * (2 - pixels[i])) / -4) + k1b)) * Min(pixels[i], 1);
end;

EDIT: Note, obviously, when pixels == 3, r = Round(k2r), etc
[Windows Mobile Software Development] 6Fg8:
OMG ... i wonder if there's a simpler rule to that :D

well the night is long, and i've still a day off :)
[Windows Mobile Software Development] D-MAN666:
Looks like a second mixdown at the first glance...
[Windows Mobile Software Development] Chainfire:
Hahaha!

BTW for reference, please note that my comparer allows a difference of exactly 1 between r,g,b values between the compressonator PNG and the output due to possible rounding stuff. If you take that into account, the 'alternate' formula is now 100% correct.

The 'normal' formula is still off a bit sometimes, I want to solve that too.

And then, we can get working on what the best way would be to actually _encode_ files to this format :)
[Windows Mobile Software Development] 6Fg8:
ha, great stuff. Implemented the new rulez and ... wow :)

now my favorite wallpapers are reeeaaaaly shiny

@Chainfire thx for sharing

@D-MAN666 without your basic work we wouldnt be were we are
[Windows Mobile Software Development] 6Fg8:
Hm thats a completely different story. I tend to implement some "call compressonator" in background for that. Dont wanna invent the wheel twice, so why not take whats there? Ok, might be kind of illegal, but i wouldnt consider that legal what we're doing here :D
[Windows Mobile Software Development] D-MAN666:
Yeah, choosing key colors, making use of first bit rule, etc is not a straightforward task.

UPDATE: Re encoding: if you're eager to implement something now :), i can tell you the way we encode QTC in Manila Editor. At least, it provides decent results, and after that we can compare our output with compresonator output to polish it up.
[Windows Mobile Software Development] eidolen:
Just wanted to report I am running a full VGA CFC compressed theme and it's working like a dream.

Some quick details.

307 files compressed.

6.94 mb before

1.07 mb after

No scaling was done at all but I did compress the 8 RGB files with Compressonator using the ATC RGB 4Bit setting, though it will work with alpha as well.

Just unreal that it could be compressed that much with no discernable quality loss.

Hats Off,

Eidolen
[Windows Mobile Software Development] 6Fg8:
My next step will be to implement a solution to provide import functionality for QTC files. If i can be of any help, you know how to reach me :)
[Windows Mobile Software Development] eidolen:
I think I stumbled onto something. I realized there was a discrepancy in the amount of files between my theme and what I converted using cfc. After doing a directory compare it turns out there was around 50 files that cfc was skipping over. I then ran the process again with just these files, using the dual folder method for RGB-RGBA. Only 1 RGB file was not converted and the rest were RGBA.

Though I did learn some things, I was not able to fix it. The one glaring similarity with all the files it skips/crashes on, is that the height is larger than the width. (Supposed to be) I can make any file load into the Compressonator instead of crashing by setting the height the same size or larger than the width, but if it is set smaller, the program complains or crashes hard.

I know Manila Editor works with these files so I'm sure it's nothing. Sorry I couldn't determine more.

Eidolen
[Windows Mobile Software Development] Chainfire:
Hmm, just figured out how to do encoding, it's remarkebly simple for the normal formula, not quite sure on the best way for alternate yet, though.

@eidolen

Just wait a bit 'til my QTC <-> PNG converter is done... Will not be done properly yet, but perhaps next post I will attach a test version - no time to finish it up properly before sunday.
[Windows Mobile Software Development] 6Fg8:
Someone stumbled over that?

http://developer.amd.com/gpu/compress/Pages/default.aspx

Looks like a dll which provides everything Compressonator does. Has definition for ATI_TC_FORMAT_ATC_RGBA_Explicit. Interface description is looking good too, i'll give that a try.
[Windows Mobile Software Development] Chainfire:
Ah, it's been updated! I had checked it out once but at that time it didn't support the texture formats we want.
[Windows Mobile Software Development] 6Fg8:
BTW Compressonator 1.50 supports ATC compression with the ATICompressor.dll, seems to be the reason why they removed CTES.dll
[Windows Mobile Software Development] Chainfire:
Attached is CFC 0.4b1, which can now directly decode QTC's to PNG's. Still working on encoding, but doubt I will finish that today.
[Windows Mobile Software Development] Chainfire:
Heh yeah I made a mistake in my thinking somewhere about my way to encode. Please share how Manila Editor does it :)
[Windows Mobile Software Development] D-MAN666:
Okay, that's pretty straightforward.

First, choosing key colors: we go through all 16 (or less) non-transparent (with alpha > 0) pixels in chunk collecting 2 "qtc" colors (closest key1 and key2) for each pixel. next we find farthest key1 and key2 colors (for wider colorspace), using the following formula to calculate "distance" between colors in RGB colorspace (it takes into account how eye percepts colors):

private function countDistance(color1: RGBColor, color2: RGBColor): Number {
return 30*Math.pow(color1.r-color2.r, 2)+59*Math.pow(color1.g-color2.g, 2)+11*Math.pow(color1.b-color2.b, 2);
}

The final part is simple: we just need to iterate through mixdown values to find the best mixdown for each pixel (this could be done w/o loop w/ math only, but the formula is too complex).

This provides nice results, but there're obvious problems, like when chunk is single-coloured, both key1 and key2 will be the same, though closest result may be with slightly different keycolors + some mixdown.

This is quick explanation, so please fell free to ask questions. :)
[Windows Mobile Software Development] Chainfire:
That's a bit how I started out myself. Oh well, weekend time :)

EDIT: Got a cool idea to try though... but will not have time til sunday
[Windows Mobile Software Development] eidolen:
The new cfc is a beautiful thing. I ran it on a full manila package, (1,011 files) and it never skipped a beat. It produced 376 "upright" png files that look great. Makes me want to touch myself.

Not that it matters anymore but I found that if I named all the files I was having trouble with to CTES instead of ATC, they worked perfectly. Strange.

Thanks again for the awesome work. I can delete like 30 directories now and theming just got way easier.

Cheers,

Eidolen
[Windows Mobile Software Development] chavonbravo:
Chainfire, it would be great if we could use cfc to batch convert theme not only from vga to qvga, but wvga to qvga.

Perhaps a new option flag could be added to denote the difference from vga or wvga in the command:

cfc -nsf outpng1 outpngscale1
cfc -nsf outpng2 outpngscale2

This would be great for porting Touch HD's manila3d to qvga devices like kaiser and polaris. It should simply a matter of changing the percentage of vertical heighth change from 50% to 40% of original.

Perhaps even from wvga to vga for people that want touch hd's manila on diamond/raphael.
[Windows Mobile Software Development] Chainfire:
Cool. Nice find about that CTES thing btw, wonder if that's what I originally did to modify those 'strange' files when I did way back when (it annoys me that I don't recall how I did it).

Glad it's working out for you!

Well, using a different scale should be no problem. Thing is, to actually use the HD Manila on the (Q)VGA I think there would need to be a different libgles_mn version that patches input coordinates again... unless of course you use the (Q)VGA Manila executables, but then you'd also have to modify Mode9 again. And I fear there will be aspect ratio issues when just doing this. As for patching 'Wide' Manila to 'Normal' Manila, a part of the input can be patched easily as it's done with the proxy DLL, but there is one point in Manila where the conversion is hardcoded. We patched this by patching in a "shr 1" from VGA to QVGA, but as the relation between WVGA and (Q)VGA includes a factor 1.25 that will not work. Hmm...

Please let me know your thoughts on how this is currently done... Must note that I haven't kept up with HD Manila versions or their ports.

@all

Dang I wish I had my laptop with me (in the shop for repairs) so I could muck with the encoding... I think I figured the trick out (yes, again!)
[Windows Mobile Software Development] pcarvalho:
a QWVGA version of manila will be released by htc... could this be used to improve qsize versions?

hope this is true as we would only have to worrie about Y aka the bottom bar :)

http://www.boygeniusreport.com/2009/01/07/htc-iolite-gets-outed-by-retailer/

http://www.theunwired.net/?item=unveiled-expansys-previews-the-htc-iolite-windows-mobile-professional-smartphone

my biggest wish would be the driver to be directly transferable to QVGA, that would be awsome :D
[Windows Mobile Software Development] w04g005:
Your second link says it's a 2,8" QVGA device. I have never heard about a QWVGA device with 2,8" but if theres really Touchflo3d installed maybe we can port it!! especially to solve the standby bug
[Windows Mobile Software Development] w04g005:
Hey guys i found another source: (unfortunately german but it's REAL)

http://www.ppc-welt.info/community/showthread.php?t=139544
[Windows Mobile Software Development] pcarvalho:
if that is correct...then forget about all i posted...its got TF2D on it :(
[Windows Mobile Software Development] Chainfire:
CFC 0.4b2

Here is 0.4 beta 2.

Only new addition since 0.4b1 is the ability to directly encode PNG's to QTC's. Quality seems to be great so far.

It's not nearly as fast as The Compressonator, but it can convert all Manila QTC's to PNG and back to QTC in about 3 minutes on my computer (The Compressonator takes 9 seconds to do this ;)). But then again, CFC does pick the right format for you automatically, so that safes a bit of work.

As far as my testing goes, results are very comparably with The Compressonator in quality, while CFC sometimes even beats it :eek: (then again, the reverse also happens)

Note that this version does not include the alternate encoding method yet (0.4b3 ?). It seems that method is best served by pixel blocks having either one (very precise results) or two colors (precise results) and full black. Will go on with this :)

Please let me know of any issues you encounter when encoding.
[Windows Mobile Software Development] D-MAN666:
Nice job!

I wonder what method did you use for encoding ;)

Also, have you compared binary output from QFC and Compressonator? I just wonder how frequently do key colors differ
[Windows Mobile Software Development] Chainfire:
Well I did a lot of binary checking with the original QTC and Compressonator's PNG output of it. Doing some comparison of the 'wanted color' results and the key colors used lead me to the first clues as how to 'guess' good results. Then some thinking lead me to the rest. All in all, CFC checks on average 24 color key combinations per color-type per pixel, then checks the results, assigns them a score (with a color-type-weighted formula like you posted before) and picks the one with the least deviance. You could call it 'educated-guess-brute-force encoding' ;) Originally I thought to just start out with green, as it's the most 'important' color, but it seems in actual encoding, even with the weighted scores, about 50% of the time going from R and sometimes even B delivers the best results. Exactly how I do it will be disclosed once I finish the QTC encoding with the alternate method and the content of the first number of posts in this topic.

I have not been able to find a neat formula. I'm not sure if the Compressonator even uses one - it would have to be a 'fuzzy' one in that case. Right now I suspect (somewhat backed by the binary data I checked) its just more optimized (my code is completely un-optimized) and doesn't check as many candidates.

So far from comparing the rendered output of original QTC's and the QTC's I generated from the PNG's I generated from the original QTC's, the results are largely the same. It seems however Compressonator and whatever HTC originally used if something else, clear a pixels different base-colors to 0 when the alpha is 0. I don't do this but pick another used color instead (whenever a pixel isn't visible, not necessarily just alpha == 0) to reduce the number of colors the encoder has to match. However, to encode them to the alternate method you

would

set them to 0. That's the only thing that caught my eye from the comparisons. Perhaps the Compressonator cuts a corner there, not sure.
[Windows Mobile Software Development] Chainfire:
Just a small note to clarify, I say fuzzy logic must be involved if it does use a formula, as as far as I can see, you cannot reduce the variables to a solvable matrix - as usual when you reduce data size. (Then again, it's been a while since I had matrix calculus @ university and the last time I really used it to write code (my PAR2 lib) is some time ago as well)
[Windows Mobile Software Development] Chainfire:
CFC 0.4 released!

All posts on the first page updated with new information, CFC updated to 0.4 including some new options/switches (see first post). Encoding to QTC now also includes the 'alternate' method.

Enjoy!
[Windows Mobile Software Development] Chainfire:
CFC 0.41 released

0.41 added... There was a small bug in CFC compression in 0.4, it didn't always set PayloadSize correctly, which could create errors with padding/trimming.

Whoops!

(talking to myself here, but whatever :))
[Windows Mobile Software Development] Chainfire:
Cfc 0.42 + gui

See the title ;)
[Windows Mobile Software Development] cpt.bert:
When will a cab with TF3D be available?

tnx
[Windows Mobile Software Development] showaco:
Great work, must say that my themes are quicker and more responsive being compressed with cfc. Can't believe how many updates I missed(today):)
[Windows Mobile Software Development] AF241:
The GUI version, absolutely lovely and easy to use. The converted images are very close to original, better than the preview. At the moment the background for the preview screen is white which makes white icons appear empty, I think it would be better if you change it to some odd grey or something. Thank you for your hard work.
[Windows Mobile Software Development] eidolen:
U R A Machine!
[Windows Mobile Software Development] Chainfire:
CFC 0.45 and CFC GUI 0.45.12 released!

CFC 0.45 and CFC GUI 0.45.12 released!

CFC (core) changes:

- Heavily modified encoding algorithm. It is often slower but the quality should be much better.

CFC GUI changes:

- No longer freaks out of the wrong file attributes are set on some files

- Added background color selection

- Added tool to patch a complete Manila package for CFC compatibility and optimization

Modifications to the encoding algorithm

As stated above, I have heavily adjusted the encoding algorithm.

The old algorithm worked fairly well when an image didn't have many different colors per pixel block, but the quality dropped fast as an image got more 'complex'. This did not show with many of images that originally came from Manila, due to the nature of the encoding. However, the 'complexity' of the images got higher as they got rescaled, and the problem was very visible there. It is also visible when you try to put a completely home-brew image into Manila.

Thus the algorithm was heavily adjusted. As said, it is much slower, but quality should be much higher. Exactly how slow it is depends on the 'complexity' of the image you are trying to convert. For example, it converted the complete original Manila PNGs (first converted out of QTCs) back to QTCs in a minute on my PC, which is twice as fast as the previous version. However, it took 4 times as long (8 minutes) to convert the QVGA rescaled PNGs back to QTC.

What I want from you

I need to know if the new encoding algorithm works well (and if you can compare, better than the previous version) for you. If this is a keeper, I can remove the traces from the old algorithm from my code (which is annoying me) and will try to optimize this algorithm more for speed. Thank you.