yashsingh1/xdaforums huggingface dataset:
[Touch HD General] Mach2003:
But with a 16kb table size, a bit too big for ModDriver's use. I was wondering if we should use the 10 degree table as base, then for certain rectangles use a much smaller 5 degree table or even smaller rectangles the 2 degree table.

for example, with made up rectangles:

if( (dLat > -4.0) && (dLat < -3.0) && (dLon > 34.0) && (dLon < 37.0)
return Use2DegreeTable( 430 );
if( (dLat > -8.0) && (dLat < -2.0) && (dLon > 30.0) && (dLon < 39.0)
return Use5DegreeTable( 820 );
return Use10DegreeTable();

I wonder if we could get <2m error for the populated world with only a few 5 and 2 degree tables, or if the exception list would be too big. Using your mapping system, could you estimate the number and size of the 5 and 2 degree tables we would need? Do you need new csv files or are the 5 and 10 you have properly random?
[Touch HD General] cybermaus:
I'll check those other tables and see if I can come up with some squares. You can probably already see where they'll be: Andes, Caribbean, extended Mediterranean and Indonesia/Philippines. Maybe also California and Japan, as they are very close to the rim.

But I think you mean, within 5 meters? The green you see now is 2<Err<5 so to get within 2 meters, we need to even improve on this table.

Alternatively, how difficult is is to put the altitude correction is a separate DLL that can be unloaded? After all, we only need one sample per 10 minutes? Per hour, Per program run? Probably the only people who would travel far and fast enough are in an airplane.

Myself, I know nothing of DLL loading, so if it is a bad idea, do tell me.
[Touch HD General] Mach2003:
In my case (in a system run driver dll), it's best not to have to load another library at run time, although I do have to load the Samsung SDK library at run time, I have to do special code to make sure it only loads after the system is fully booted, and not during the init phase. Also under WinMo, there could be cases when there is no room to load the library, even though plenty of RAM is available, I'd rather deal with this at startup, then sporadically durring run time.

About how far does a user have to move, generally, to need another seperation value, and still be within the <5 meter target (at say 140kph max)? I currently am re-running the seperation routine (if GPS type says it's needed), on every pass (once a second).
[Touch HD General] cybermaus:
How about this idea:

Go for a full 1 degree table, but store it in a byte-file, not a static array.

Move back the the single dimension addressing scheme, where we have a master index value, but of the file rather than an array. At program open, use that index to read the 4 surrounding matrix values into 4 variables and close the file.

Next, just use those 4 variables every 100 samples or so to update the separation. So even if the user travels fast, we will calculate a new interpolation every 100 samples, but based on the variables, not by rereading the file. Only if the user crosses a degree line, re-read the 4 variables.

If the user travels exactly along a full-degree line, random walking over it all the time, its a file read every 100 seconds max. Or you could put the 9 closest values instead into memory instead of 4. He would have to travel half a degree at least then before he triggers a re-read, which at non-polar latitudes is between 50Km and 30Km at least. (if I calculate correct)

Or, keep it at 2 degrees, but make the matrix into 16-bit short values, so we do not have the .5M starting error in the matrix values to begin with.

While I understand we do not want to keep a 16K table in memory, are we worried about a 32 or 64K file?

BTW: I will not be able to map those other files, I have to work tonite, and I was otherwise engaged this evening.
[Touch HD General] Mach2003:
This is idea is very interesting, we could even keep a 1 degree table with two byte data on the card without bugging anyone (128kb). We could even load a slightly larger grid into memory, say 10 degrees x 10 degrees (200 bytes at short size), this would cover a lifetime for most users :), in any case it would keep reading slow card memory to an absolute minimum.

I'll run off a 1 degree table as signed 16 bits values, and see if the 5000 test generates any readings over 5 meters.
[Touch HD General] Mach2003:
A 1 degree byte table only gave 1 reading over 5 meters of error, 33 that were between 2 and 5 meters. No point in moving to a word array.
[Touch HD General] cybermaus:
Just realized, to also keep processing to a minimum, not 100 or 1000 but 256 samples is probably the best natural cycle:

if ( dSepSaved<>0 ) {
cSampleCount++;
if ( cSampleCount=0 ) { return dSepSaved;}
if ( abs(dLon-dLonSaved)<0.01 and abs(dLat-dLatSaved)<0.01) ) { return dSepSaved;}
}

This will keep any floating point calculations to a minimum.

If will even keep file lookups caused by someone being precisely along a whole degree to a minimum.

(0.01 degree is an oblong with 2Km long side)

And those may be over the reaaaly steep sections, like the pacific rim, ie, over the ocean. Can you run me a 50000 sample table, and give me only the values over 2M?. Purely out of interest.
[Touch HD General] Mach2003:
I'm not sure if the test code with floating point calls (fabs) and math are going to be much more efficent than just calculating the seperation on each pass. Especially if the array is (and most likely will be) still in memory.

I am testing a 10 x 10 degree load into memory now. If a user get to any edge, it will load a new 10x10 (user in the center), and not again until user travels 5 degrees in any direction.

I ran test code that outputs only >2 meter errors to a GPX waypoint file, and looked at them (all 35 of them ;) on Mapsource. Using my "populated earth" filter, only one point was on land. GPX is also supported in Google Earth.

Yes, using the GPX file format, I can run a large number of samples, even remove the populated earth filter, and only log the error points. Let me finish the 10x10 code, then I'll run it.
[Touch HD General] cybermaus:
Aha, so 35/50000, you can claim a 99.94% better than 2M midpoint accuracy. Or in fact, a 1/50000 is a 99.998% better than 2M midpoint accuracy over land :)
[Touch HD General] cybermaus:
Probably true, I was still thinking 4 variables rather than a loaded sub-matrix. But the 256 cycle counter would still save CPU.
[Touch HD General] Mach2003:
That was a 5000 sample, not a 50,000 sample.
[Touch HD General] Mach2003:
50,000 random samples, lat and lon randomized to 0.01 degrees.

4 Equal Quadrants sampled with 12,500 each (+/- 90,0, 0,180).

AltTest_1.gpx, all points 2 or more meters in error.

AltTest_1Land, I manually removed points that were not near to (or on) land.

Land Error Counts (meters of error):

<2, 49,463 points

2+, 259

3+, 184

4+, 69

5+, 14

6+, 6

7+, 4

8+, 1

9 or more, 0
[Touch HD General] cybermaus:
Nice.

Thats 99% better than 2M midpoint over land (98.926 is 99 right;)).

Also 99% better than 3M (99.444)

And 99.95% better than 5M

I think these are very good results.

I will complete the gpsVP program with the 10 degree EGM2008 matrix you generated. The reason is, it is intended that Windows Mobile 2003 SmartPhone edition compatibility is maintained, so the table needs to be small. And there is little need for me to rewrite all the work you did, so if they want better precision, they can use your Mod driver (if/when)
[Touch HD General] Mach2003:
If you are using your original code, double check coordinates with N90.0 and E180.0, as when I was debugging I had a whole "line" of bad results because we were using a row/column that did not exist in the table (the +1 row/col). I did rotate my file data so that it has lats across and lons down, and may have introduced the bug that way.

I also added bound and range code to ensure the table would not be read past it's limits. Inserted this just at the top of EGM96Geoid.

dLat = Bounds( dLat, -90.0, 89.99 ); // table does not have a row of data after +90, so I need a slightly lower value, -90 is fine.

// lons are technically valid from +180 to +360.
if( dLon >= 180.0 )	// there is no col after +180.00
dLon -= 360.0;	// 182 is the same location as -178
if( dLon < -180.0 )
dLon += 360.0;
[Touch HD General] cybermaus:
Hmm, during debug I tested the +-90 +-180 scenarios', and while technically it sometimes came out of bounds for 3 out of 4 values, the compiler returned 0 for the out of bounds values, and because of the delta's, it gave all the weight to the single remaining correct value, which turned out correct.

Also, when would a GPS feed value +182, or even 180? I was thinking it would be a pretty rare occurrence to even see it feed +179.999999, and it would jump to -179.999999 after that. So I did see, but ignored the bounds issue.

Anyway, while I was very happy with my code as little as 2 weeks ago, now I know it is 2nd rate compared to the strategy we cooked up in this thread. And since your driver is/will be free, I am not going to rebuild it in a similar way. I'll just iron out the stupid errors (I will check the bounds again for the mid-bounds, not just the far corners), complete the settings page, and use your driver myself :)

Unless of course if you provide a copy of your code, but that would be a dangerous thing to do, I am bound to have remarks, and we'll spend another week debating.
[Touch HD General] Mach2003:
I'll give you my code for this module, no problem. Let me clean it up a bit and run it though the ARM compiler first (so far I am only running it on PC) and then I'll post it.

I had 20 or so "over 5's" on the International date line, and another 20 or so at the south pole, when I noticed that the edges of the table were being used. Reading memory "after" the table can cause a run time exception, if that memory is not in the owner process code space (especially bad for a system owned driver, the memory can be owned by device.exe, but belong to another driver).
[Touch HD General] cybermaus:
Could that be an artifact of the randomizer (with 0.01 increments, you'll get whole numbers roughly once every hundred, so exactly 180 is actually possible)

Yes, I do know that, and had briefly considered to pad the table with 3 zero values at each end, but I did not bother, due to my believe it would not happen, see above.
[Touch HD General] cybermaus:
Thanks. but no rush. Lets have the code tested and matured a bit. there's bound to be a minor code change in the first week or testing it.
[Touch HD General] Mach2003:
You might as well have some 'fun' too;

This module is still debug, many efficiency changes are needed, but vars are easy to see and understand. I inserted, in comment, the table creation code functions, these are NOT ready to be compiled, just for reference.

I am using and testing with the attached 1 degree table. The code *should* work with any integer table size.

What would your GPS show as longitude anywhere exactly on the antimeridian (International Date Line, +-180 degrees), and would it matter if you were heading east or west? I understand that getting to the South Pole is very rare for humans, but it is 'possible'.

All this aside, we are reading data from a port, using, what we have proven to be, buggy GPS driver output. I'll keep the bounds in my code.
[Touch HD General] cybermaus:
Thanks. Debugging is always fun;)

Well, I am guessing, traveling west: -179.99978; -179.99993; +179.99998; +179.999961; I guess if you stand still, the random walk would hit 180.00000 at some point. But theoretically, it should never read 180.00001. But yes, assuming GPS drivers are correct may be asking for trouble.

Its probably get funnier at the poles, because at the exact pole, you are at the same time at all longitudes [-180.00000..180.00000]

Oh, I was not trying to say you should not, I agree I was sloppy. My college professor would fail me for assuming sane input. He would then fail me again for allowing a potential 'out of bounds'. And if I argued, he would fail me a 3rd time for knowing it and still allowing it.

PS: I never failed any of the IT classes, I was a good boy at school, the sloppyness came later.
[Touch HD General] Mach2003:
I might be a bit older than you, I can't even remember going to school.
[Touch HD General] cybermaus:
Possible, but not likely. I could also simply have a better memory. For example, I remember working with microVAX, at work
[Touch HD General] Mach2003:
Test Version of ModDriver

The attached file has the Altitude corrections installed. It corrects it even when the GPS data has a seperation value, assuming our table data is better than the one built-in the unit.

Other unrelated mods included:

- Fixed Cell tower position when neither GPS nor Cell tower information is valid, first few seconds of startup

- Stopped writing to myLocation when service is not installed (key does not exist)

- Fixed ioctl calls to ModDriver, full support for WaitCommEvent calls now.

- Might have now eliminated the "error reporting on" issues.

Readers of this thread may try it and report if there is any problem with altitude (or the other mods).
[Touch HD General] cybermaus:
At first glance, all seems to be working. 44.5M separation, nicely close to the 44.25 of EGM2008, and better than the chipsets 47.2

But when I force it to use COM6 (GPSID port) as input rather than MSAPI, the COM4 output separation alternates between 44.5M from driver and 47.2M from chipset. It does so for no apparent reason, the altitude does not jump any corresponding 2.7M

I attached the log csv from MSAPI and COM6 (GPSID) and a log from NMEA on COM4 (GPSMod). The logs were made from the same time period, and from GPS cold start, so the first samples are without fix. And even after a fix, my chipset takes a while to become stable

As you can see, while both MSAPI and GPSID ports are stable with the 47.2 chipset value, the NMEA data generated by your program swaps between chipset and calculated values. This only seems to happen when I make GSPID/COM6 the preferred port, when MSAPI is the preferred one, the COM4 data is stable at 44.5 calculated value

----

I also see that even though you install in device, the table is stored in /Storage Card/Mach2003. Probably good, but some users may be in the habit of clearing that folder as it is a log/debug folder. Maybe /Storage Card/Program Files/Mach2003 would be better for this permanent file. And /Program Files/Mach2003 if device install is selected and no storage card is availible at that moment.

I assume that if you cannot find the RAW table, it reverts to passing through the chipset value?

----

About the code:

Why

dLat = Bound( dLat, -90.0, 89.99 );

in one place, and

if( dLon >= 180.0 )

dLon -= 360.0;

if( dLon < -180.0 )

dLon += 360.0;

in another?

and should it not be

dLat = Bound( dLat, -90.0, 89.999999 );
[Touch HD General] Mach2003:
It would be most helpful to see the ModDriver.txt log from the devices \Mach2003 folder, with logging enabled plus enchanced. I can then see both input and output to see why it is failing.

I want to install the data file to card (if it exists) even when driver is installed to device (as it should always be). I just used an existing routine (as you know) to find and open a file on any storage device, it uses Mach2003 folder. I'll look into changing the folder.

It should, but honestly, I have not realy looked at that yet. I am still returning a seperation of 0.0 when the function call fails.

Because the longitude wraps around in a circle, but the latitude does not. Column 180 data, can be read from column -180, with column -179 right beside it. Or col -181 data can be read from column 179, with 180 right beside it.
[Touch HD General] cybermaus:
Well, 89.99999999999999999999 would be a bit overkill, because the NMEA string only supplies 6 digits.

I know, the MSAPI makes it into a float first, and theoretically we have no idea of the internal precision of the driver, but 0.01 is still 1Km for longitudes, even at the poles, so seeing 89.990009 would be quite possible, and still 1km from the pole. (though I admit pole travelers would probably bring "real" GSM equipment)

Would Bound( dLat, -90.0, +90.0); not work anyway, because the matrix does have the border values at all ends?

I'll go collect that extra logging now.
[Touch HD General] cybermaus:
Oh wait: I now suddenly see what you are saying. Yes, in that sense, I agree it does not matter, 89.99 is good enough.
[Touch HD General] Mach2003:
Bounds call will replace any latitude 89.99 or greater to force the table to use row 89 and 90 instead of 90 and 'table overflow'. Row 90 is the last row, so no we can't use row[90] and row[90+1].

Row 90 contains all 15's. Row 89 contains values ranging from 12 to 16.

c[0]=12, c[1]=12, c[2]=15, c[3]=15.
Adjacent columns are almost always the same value, we can make lonitude any exact degree for this example:
dMid1 = (0.000d000 * (12 - 12])) + 12 = 12
dMid2 = (0.0000000 * (15 - 15))) + 15 = 15;
dSep = (0.99 * (15 - 12)) + 12; = 14.97, we round to a single dp: 15.0

or

dSep = (0.9999999 * (15 - 12)) + 12; = 14.99999, we round to a single dp: 15.0

One value or the other makes no difference to the code, size, speed, or outcome. I intentionally left it low so that the fmod() function would not possibly round it back up to 90, you never know what the FP processing code would do with it.

Edit I re-read and saw what you were talking about, :( (Still the result is the same)
[Touch HD General] cybermaus:
Nuf said about bounds......

I collected some logs.

The first go was really weird. The device was still running from yesterday (though without GPS) and I collected a log that gave me 17M separation but correct coordinates? Don't ask me why. I was unable to collect the proper logs though, but I attach what I have anyway.

Next I restarted the device, and did it again. This time alternation between 25 and 27. For some reason, I am unable to collect csv logs. Is that because of the extended, or did I forget overnight what I have to do to get these?

I do have the extended log from the /Mach2003, and the NMEA from the generated output though. Again alternating.
[Touch HD General] Mach2003:
Sorry, it looks like we cross posted...

I have a bug when using WaitCommEvent, each time a port is opened to read ModDriver, the timeout is cut in half, if smooth is enabled. I have corrected it here, but not yet tested it properly. This might explain your weird results. Eventually the system bogs down so bad a soft reset is required.

Track Logging creates the GPX and CSV logs on the card, regardless of the program logs (or extended).

I'll look at your logs now...
[Touch HD General] Mach2003:
I have no error trap in the seperation routine, when the original GPS signal is invalid, I don't alter it, thus you get original seperation value, I'll change that.

Your COM6 GPS often does not send the sats in view information sentences GPGSV. I wait for these to arrive (or timeout waiting), this currently causes the original seperation to be used (see above). At first thought, I don't see a way around getting your GPS to work with ModDriver properly, but I'll come up with something... I have to wait for the GSV to arrive, as all the other tested GPS units send this as the last few sentences of the message.

Next step, fix the insertion of seperation in all messages, except if the seperation routine fails (the table is not readable). It is a good thing your COM6 is failing, or we might not have noticed this for a long time...
[Touch HD General] cybermaus:
I guess so :(

Well, it still works with the MSAPI, so don't bend over backwards to fix it. Didn't that rant rpettinger found had sentence sequence inconsistancy as one of his levels of suck?

Edit: I seem to recall that the Sirf had a utility with which you could set the frequency of sentences, and the 1 every 5 seconds was the default for satellite info. Maybe I do have a real Sirf, but I have been unable to get SirfTech to talk to it through the GPSID. (I'd like to enable WAAS/EGNOS)
[Touch HD General] Mach2003:
Is COM6 your internal or an external port?

2.51

- Fixed WaitCommEvent dividing time in half for each open port call

- Created a struct for each open port in ioctl

- Fixed seperation

- When table can't be loaded

- When Com times out

Try this copy to see if seperation works in all the noted cases now (even with COM6, although it will still timeout, seperation should still be correct)
[Touch HD General] cybermaus:
COM6 is my 'virtual' internal port, provided by the GPSID. Your program suggest COM4 for its output.

COM3 has an internal label of Modem_GPS, but it does not provide useful data, and given that COM1 & COM2 are labeled Modem_GSM and Modem_Blue, I am thinking it is a proprietary way that is used to talk to the baseband chip on these clones.

I do not currently have any Bluetooth devices, I used to have a TomTom mark II (SirfIII based) but I gave that to a friend. I also still have a USB dongle with a SirfII, but that cannot connect to PPC. I also have a GARMIN with SirfIII chip, but loaned that to another friend.

But I do seem to recall that both Sirf devices (not sure about the Garmin) and this one too if it is a Sirf, send out the GSV only once every 5 cycles.

Testing new program......

Edit: Works good in both MSAPI and COM6 cases. Nice constant 44.5M separation.

I did reinstall selecting 'use defaults from cab', but it still had my altered port order values.
[Touch HD General] Mach2003:
See if your Sirf COM port likes this copy better, I tried to pass messages without GSV sentences through, but have no way to test it, as both my devices always have GSV.
[Touch HD General] cybermaus:
What am I looking for specifically?

I know what you mean with the GSV messages and the timeout, but the previous 251 build already had a good result in the client program. Or did it loose 4 out of 5 samples? It did look a little more stable in its random walk.....

Edit: Also, its still unconfirmed its a Sirf. You have to be careful assuming claims are true with these clones
[Touch HD General] Mach2003:
The log file you send last night, had your COM6 wait for the GSV messages even though there were none due to arrive. This caused late inbound readings, forcing ModDriver to add another (about 1 second) to the Lag calculation instead of using the "new" GPS value.

You might not have noticed, unless you were driving with COM6 and a mapping program, and then you might have seen jumpy position depending on how steady your speed was, and if you were going at the same heading or not.

Your ModDriver.txt log file should show valid inbound readings within a few milli-seconds of every second. Don't confuse the ModDriver outbound logs, nor a flat NMEA log, becuase ModDriver will provide a valid outbound message even when an inbound message is late.
[Touch HD General] cybermaus:
Ok, tested that version, which I've named build 252, based only on the fact you named the previous one 251.

I included the output of it, both when your driver is connected to COM6 and to MSAPI, both for 251 and 252. I also logged some NMEA when loggin directly from COM6, without your driver in-between.

What I think I see is they all work good from the client program from a fix point of few.

But your newer 252 version does not send GSV info when it does not get this as input. The older 251 did regenerate empty GSV, regardless whether it had them as input, causing the client program to report a valid fix from 0 out of 0 satellites.

When your program is connected to MSAPI, it was already not sending empty GSV info in both versions.

All in all, I think the newer one is better, but please go through the NMEA to see if it looks good to you.

----

What I also suddenly notice is that my chipset only provides 4-digit position information. Thus 51.XXXX rather than 51.XXXXXX. Your program makes that into 51.XXXX00, which is fine, but I am unhappy with my GPS reporting on an 11 meter resolution latitudinal and COS(51)*11 meter longitudinal

That sucks. I know GPS has 11 meter error bars easily, but it should report a proper midpoint. This is going to mean tracks are not only going to be wrong, but they will look distinctly sawtooth like.

Well, not your problem anyway.