yashsingh1/xdaforums huggingface dataset:
[Touch HD General] cybermaus:
I think I know what happened: The first log I made, the HD had *no changing* in its altitude reading for all 20 or so samples. So I decided to run again, while walkin up and down the stairs. But gpsVP was also running keeping your driver active etween sessions. So is it possible that during your second reading, it already had the values from the first?

Ok. The matrix is good enough. I'm can live with a 2m structural error, better than the 47m structural error I had before, and we have the normal GPS error to accept anyway.

Yes, but do not assume the error is structural. Around the globe, it could go either way. I can double-check to see if the interpolation is correct, but I would not expect much improvement without making the matrix more fine-grained.

Of course, I was intending to also allow a user supplied local value, for those focused on absolute perfection. (Or in case my matrix turned out to be very wrong)

EDIT: Log attached, lat/lon cleaned out, so no time pressure.

This time it behaves as you said, well done.

An extra remark though (sorry, I am one of those guys who always has some remark, I cannot help myself): You may notice how wildly things are fluctuating for the first 10 or so values after getting a fix. I still advocate skipping the first 10, and taking the next 10. And consequently, to carry over the value from previous run, so the first 20 samples are corrected by the old value, avoiding a potential altitude jump at sample 20.

But: Well done. If/When you incorporate this code, your driver will have an additional benefit. Not sure how many people are ignoring and unaware of altitude, but you must make a few people happy, rpettipher and me at the vary least.
[Touch HD General] Mach2003:
Having ModDriver running would not affect GpsTest results at all. Mod Driver (at least on your device) has not been updated. GpsTest 'Forgets' all saved information on restart, nothing is saved except to RAM.

FYI: I think your interpolation may be funny when negative numbers are involved. I am close to 49.88 North, and 119.39 West, but it looks like you interpolate from -11 to -10 (west/10), but should, perhaps, interpolate from -11 to -12.

In any case I totally agree that 2 meters is way close enough for this!!!
[Touch HD General] cybermaus:
I'll doublecheck the calculations, focussing on negative numbers.

See edit with log above
[Touch HD General] Mach2003:
Your latest log actually makes sense now! You are mistaken (liar liar pants on fire), your msapi is sending Altitude ABOVE ellipsoid, not seperation. This time you get enough variance to detect that the Alt84 field is changing, and I correctly assess that you have AAE, and fix it for you. Prove me wrong, show me a ModDriver.txt MsApi file where your ALt84 (F11 in the logs) value is exactly the same for a long time, while Alt (F9 in the logs) is changing! (you can get my email address from the donate link in my signature)

Yes that is the plan, store the TYPE of each port, and re-check it only on soft reset. This was a proof of theory concept so far. But even with erratic altitudes, the difference between Alt and Alt84 will be the same value (or Alt84 will always be the same value if it pans out that you REALLY do get seperation in Alt84).
[Touch HD General] cybermaus:
:mad: :confused: :eek:

I am blind. My eyes see what my mind wants it to see, and I never bothered to doublecheck. And apparently I need to buy new pants.
[Touch HD General] cybermaus:
Anyway, bouncing back.

I indeed found an error for negative index and put a correction for it.

I also rethought the interpolation formula, it should also behave slightly better

Please try it out:

Index: EGM96Geoid.c
===================================================================
--- EGM96Geoid.c	(revision 469)
+++ EGM96Geoid.c	(working copy)
@@ -129,6 +129,7 @@
void EGM96Geoid(double dLat, double dLon, double *dAlt, double *dSep)
{
double dLatDelta, dLonDelta, dIdx;
+	double dMid1, dMid2;
int iLatIdx, iLonIdx, iIdx;

switch (iEGM96Mode) {
@@ -143,15 +144,23 @@

// Array index of position
dLatDelta = modf (dLat/10, &dIdx);
-	iLatIdx = dIdx;
+ iLatIdx = dIdx;
+	if (dLat<0) {
+	iLatIdx--;
+ dLatDelta++;
+	}
dLonDelta = modf (dLon/10, &dIdx);
iLonIdx = dIdx;
+	if (dLon<0) {
+	iLonIdx--;
+ dLonDelta++;
+	}
iIdx = (iLonIdx + 18) * 19 + iLatIdx + 9;

// Calculate interpolated altitude
-	*dSep =
-	((geoid[iIdx] - geoid[iIdx+1] + geoid[iIdx+20] - geoid[iIdx+19])
-	* (dLonDelta * dLatDelta / 100)) + geoid[iIdx];
+	dMid1 = dLatDelta * (geoid[iIdx+1 ] - geoid[iIdx ]) + geoid[iIdx ];
+	dMid2 = dLatDelta * (geoid[iIdx+20] - geoid[iIdx+19]) + geoid[iIdx+19];
+	*dSep = dLonDelta * (dMid2 - dMid1 ) + dMid1;

// Apply correction
*dAlt -= *dSep;
[Touch HD General] Mach2003:
Should'nt it be:

+ if (dLon<0) {
+	iLonIdx-=19;
+ dLonDelta++;
+ }
[Touch HD General] cybermaus:
No the 18 offset is later. I checked it with debug in all 4 quadrants.

Edit: Just to be sure, I;ll run a debug around the 179degree scenario's too....

Edit2: I'm sticking with this one. It even turn op the correct values for Exactly -180.0000000,-90.0000000 and +180.0000000,+90.0000000, though I doubt someone standing on the poles will ever see those exact figures.
[Touch HD General] cybermaus:
BTW, do you want me to test something, because be wrapping up my Touch HD this evening (Europe time) to send it to its new owner.
[Touch HD General] rpettipher:
cybermaus, do I understand that your touch HD is in fact behaving in the same way as mine for the MSAPI values?
[Touch HD General] cybermaus:
Well, I have half a mind of trying to save face and claiming its behavior changed, due to installing some drivers or something, but I did not actually install any drivers over the last week. So yes, I must be a blind fool and it behaves the same as yours. Despite the 6.1 vs 6.5 issue

Still, I am at a different geography, and we did just change the interpolation formula's.
[Touch HD General] rpettipher:
No problem - at least here at 500m it is pretty obvious which figure is which - it must be a bit confusing around sea level. Don't you get a bit worried wandering around at -8 metres?
[Touch HD General] Mach2003:
I am home now, but don't have anything ready to test, so go ahead and ship it.
[Touch HD General] cybermaus:
****, yes. I do sometimes get 0 AltSL values. Normally they hover around +5 (here at my desk), but they do sometimes vary as low as -5

Maybe due to a coincidence, I got a temporarily stable +47 value for Alt84, and thus I "recognized" it as my separation value. That could explain how I've been so blind.
[Touch HD General] Mach2003:
@cybermaus

I tested your new interpolation routine for areas near me, and did not like the results much, perhaps I am in one of those "high slope" areas.

I re-worked your code using an alternate idea. Basically I find the closest index value, average the next points in both lat and lon directions to create a "center" point, and then calculate the distance the subject lat and lon are from the index point, and the center point, and use that for the interpolation value.

Notes:

- I use a slightly different paramater list than you (float <= (lat, lon)

- I converted your matrx to a two dimensional array for simplicity, ARM supports two doimensional arrays with a single instruction anyway.

- I ran 1000 random numbers (all in Canada) through both yours and my routines, and captured the results. The biggest difference was over 9 meters. I then took 10 samples with the largest difference and ran them through the NGA website, and found that, for the 10 samples I tried, I was closer (within 1.5 meters) of the website.

Attached the C module with the new code. If you see anything in there that I assumes incorrectly, or have any comments on my methods, please do report. You are, of course, free to use my code as you see fit, anywhere.
[Touch HD General] Mach2003:
Here is the latest copy of my little test application.

- My Interpolation of the matrix is included.

- Seperate detection for COM and MSAPI

- I no longer use a count of readings for testing, I just wait until one of the altitude values changes, and then assess what type it is based on that reading alone (compared to the previous reading). Since I discard un-fixed readings from the test, the first good reading with a changed number should indicate it's type.

- I no longer support the phantom msapi that has seperation under Alt84, we have no examples of this type anymore.

I think we will get correct altitudes in the log files, no matter what port or device is used. I still am not retaining type after exit of test application, so each time you try it, type is re-assessed (try it many times, under different circumstances).

EDIT: I found a bug in the COM type detection, fixed it, and attached the updated copy now.
[Touch HD General] cybermaus:
I am humbled. Thanks for the return code, I will study it. Can't wait to see what the value for my region is.

Sounds like a good method. Probably easier too, as you do not need to remember all these readings.

That decision might come back to bite you. My shame aside, we only have a sample of 5 devices, and the mail rpettinger found does indicate there was at least one such a version at some point in time. Also you can determine the case without doubt and without endangering decision making for the other cases, and we have a good and simple solution for the case.

[EDIT] On the other hand, probably no one is really interested in Alt84, only in a correct AltSL, so even if the case does exists out there somewhere, they probably will not drop by to complain.
[Touch HD General] Mach2003:
Sorry for the confusion, MSAPI ports that have true altitude and seperation will pass through my code un-altered, as desired. So in effect that type is still supported. If some device has MSAPI where altitude to the ellipsoid and seperation value were present, my code would fail that device, by passing it through un-altered, when it should be changed to true altitude.
[Touch HD General] cybermaus:
I striked through the "humbled" portion above. I've been running your and my code, and do not see any clear winner.

But first, how were you unhappy with the result at your location. Your routine is giving -16.17, mine is -16.14. NGA is giving -16.21

Is it possible there has been a mistake in copying the patched code, so the code you have of me is not working as it should?

Running randomly around the world, picking out those that are clearly different between your and mine:

(I started with >3 different, but those are too many so I moved to >9)

Lat	Lon	NGA	You	Me
-21.98	-176.76	53.11	46.63	41.84
40.80	135.35	27.46	33.91	30.04
34.41	31.64	11.38	18.46	24.87
19.62	84.60	-64.56	-70.50	-64.95
14.76	115.80	22.63	36.05	21.41
39.76	44.92	20.32	20.87	6.61
37.53	45.84	18.47	-9.07	0.47 !!!
-34.45	145.39	11.66	20.01	9.34
65.58	-74.51	-23.30	-7.49	-17.56

As you can see, it is not so clear which is better, but it is clear both are off more than we thought. Ignoring the !!! (matrix typo?) we still have >5 meter errors.

Maybe we should go to a 5 degree matrix. Or should we accept as is?

EDIT: No, the !!! is not a matrix typo. Sorry.
[Touch HD General] Mach2003:
Fair enough, so neither interpolation is better than the other.

We'd have to compare a lot more numbers with the true values in order to assess overall average accuracy in order to determine if the matrix needs to be expanded. Have you downloaded the 1x1 second or the 2.5x2.5 second tables from NRG site?
[Touch HD General] cybermaus:
You lost me. NRG? 1x1 second? Surely we are not going to make a 1 second matrix? And who is NRG?

---

Interesting titbit: For my location (51.63N 4.37E) NGA is giving 44.10M, but my Sirf? chipset is giving me 47.2, and if I remember correct, my HD (trough MSAPI) was also giving me 47.2.

The matrix was actually filled from http://sps.unavco.org/geoid/ but at least that is also giving me the same 44.096 as NGA.
[Touch HD General] cybermaus:
Lets see:

If we would go for a better matrix, we may need to segment the earth. After all, above the cancers, 10 degree longitude is much smaller than near the equator. Latitude distance is the same all over.

We could do a 2.5 degree resolution for latitudes and longitudes, which would give a 46x91=4186 value matrix. (a 1 degree one would be 65341 values)

If we would segment it in 3 sections 40 degrees from the poles, with only 10 degree for longitudes, we would get two 5x91=455 and a 33x91=3003 value matrix. Total 3913 values.

Am I calculating that correct? Hardly seems worth the trouble?

Ok, so 4186 1-byte values. That means writing a program to retrieve them, because I am not going to fill that in manually like I did last time. But would it be worth it?

EDIT:

We probably want to use the F77 for XP program here

http://earth-info.nga.mil/GandG/wgs84/gravitymod/egm96/egm96.html

At first for checking our current solutions, then maybe to build a better matrix
[Touch HD General] rpettipher:
For information, I am including the response I got from the Ozi Explorer man when I asked if he calculated the geoid separation himself:

I must admit that I have failed to find it on the web, so it does not help much, but then I have never found the data tables that you use either.

However I did find this, which I found quite amusing:

http://esr.ibiblio.org/?p=801
[Touch HD General] Mach2003:
I have the 2.5x2.5 seconds tables and application from NGA EGM2008 site

I have it functioning here, and ran 5000 random world points through both your and my routines plus the NGA windows software.

Your routine provides 594 points that are -+ 5 meters from the factual result, Mine gave 993 points -+ 5 meters. I have not looked at the data any further yet.

I can produce a table on *any* resolution electronically now.
[Touch HD General] cybermaus:
All the information is here

They aren't so much tables as formulas. The original code is written in Fortran, which I have not programmed since my days on the good old VAX systems. The routine applies a 360n function over a 15minute resolution table. Luckily, there are precompiled binaries for DOS, Windows and UNIX. Those binaries give the exact same results as those web-sites, so I am betting its the same binaries that are used as CGI.

I am currently making a console program that randomly takes a few thousand points on Earth, passes it to the NGA binary for the correct result, also passes it to our routines, and then counts the errors we make and how big those errors are.

With a small alteration, that same program can then make a 2.5degree table for us, so I do not need to type in the values manually like last time.

EDIT: Never mind, you're faster then me.....

EDIT2: Given that my routine is better, but also does not use so many multiplications and roots, I think its the better one, but maybe on a 2.5 degree resolution? Though the calculations may not matter so much if you consider you probably only need one calculation every 10 minutes or every 1000 readouts or so.
[Touch HD General] Mach2003:
That looks almost the same as the EGM2008 link I posted, except the EGM96 links have a DOS application as well as the windows one (and are using data 12 years older).

And no, the data downloaded is a pre-computed table in 2.5 minute (0.041666... degrees) resolution. The Windows application interpolates points within the 2.5 minute resolution.

EDIT: Never mind, again, I kust saw you "edits"
[Touch HD General] Mach2003:
Now THAT was a good read, so much of it describes my life since the beggining thoughts of GpsModDriver!
[Touch HD General] cybermaus:
So EGM2008 puts me at 44.25. Slightly higher than the 44.10 of EGM96, but still below the 47.2 of Sirf? and GPSone, putting these last 2 still beyond the 2 meter mark we arbitrarily talked about before.
[Touch HD General] Mach2003:
I just redid your table using EGM2008 (2.5m), and an automated program. And ran the two functions side by side again with 5000 random positions. It does not make any difference what function we use (yours or mine) the amount of errors and size are very close to the same. While scanning the list (attached csv) I saw a few real ugly ones -+28 meters, but even they are few and far between.

I think we can use your algorthm, and the 2008 table at 10 degrees, and be done with it. Let me know what you think!

Added the 5000 point data as a zipped csv file.

static const signed char geoid[19*37] = {
-30,-52,-62,-46,-16,20,48,50,35,21,13,4,-7,-12,-6,1,2,3,15,
-30,-53,-63,-43,-19,6,23,26,21,15,12,10,-5,-11,6,10,1,2,15,
-30,-53,-60,-37,-18,1,6,9,11,16,11,8,-9,-13,7,15,-1,0,15,
-30,-52,-57,-32,-16,-7,-1,-1,5,13,1,-7,-16,-21,0,14,-1,0,15,
-30,-48,-52,-30,-17,-12,-9,-8,-3,1,-11,-23,-28,-31,-11,11,-3,-1,15,
-30,-43,-44,-27,-16,-13,-13,-11,-9,-13,-29,-39,-40,-37,-18,0,-8,-2,15,
-30,-39,-38,-24,-10,-13,-10,-5,-11,-23,-39,-47,-42,-23,-16,-16,-15,0,15,
-30,-35,-32,-22,-11,-10,-8,-2,-8,-20,-29,-33,-30,-18,-18,-31,-25,6,15,
-30,-29,-27,-17,-8,-7,-4,-4,-12,-14,-11,-7,-23,-25,-24,-42,-27,4,15,
-30,-23,-21,-11,-3,-1,1,-1,-9,-4,2,-11,-27,-33,-35,-47,-26,9,15,
-30,-24,-8,-4,5,7,8,8,-1,15,1,-20,-32,-34,-39,-42,-18,12,15,
-30,-25,2,9,12,22,36,35,28,14,-10,-48,-52,-34,-26,-23,3,7,15,
-30,-22,5,20,13,13,18,20,4,-13,-41,-47,-41,-27,-13,5,24,19,15,
-30,-19,0,20,2,-3,3,-8,-19,-26,-43,-33,-18,1,24,29,29,28,15,
-30,-17,2,21,4,-7,-8,-7,-12,-19,-17,-10,17,32,44,49,45,34,15,
-30,-15,1,24,11,6,3,-6,-10,2,3,17,30,59,63,64,56,37,15,
-30,-10,3,21,21,21,10,0,4,14,17,25,34,52,62,61,60,32,15,
-30,-6,10,17,27,23,15,12,13,19,33,31,44,49,59,56,56,34,15,
-30,-5,14,15,25,18,22,17,11,17,24,31,34,51,45,48,49,36,15,
-30,-2,17,20,26,25,26,23,13,9,22,27,28,45,48,41,42,35,15,
-30,1,18,25,33,29,33,22,5,-17,2,14,26,34,40,19,30,29,15,
-30,4,20,29,38,33,31,7,-12,-11,-5,9,16,39,27,16,20,24,15,
-30,5,22,34,45,39,14,-10,-27,-28,-10,2,11,31,11,11,11,18,15,
-30,5,29,33,45,41,15,-11,-32,-49,-32,-31,-18,-18,-12,6,3,13,15,
-30,5,30,33,38,28,15,-10,-37,-63,-58,-42,-15,-29,-19,-4,-3,8,15,
-30,6,17,29,40,23,6,-21,-61,-89,-91,-61,-38,-41,-33,-21,-10,2,15,
-30,6,18,25,27,13,-9,-40,-76,-103,-97,-68,-37,-55,-44,-30,-13,2,15,
-30,-3,11,9,13,-2,-25,-48,-64,-63,-63,-59,-34,-66,-42,-34,-12,1,15,
-30,-13,-8,-3,-2,-21,-38,-46,-26,-7,-25,-38,-34,-55,-42,-34,-14,3,15,
-30,-25,-21,-14,-15,-33,-39,-26,-1,34,11,-13,-24,-28,-30,-28,-9,2,15,
-30,-34,-27,-25,-22,-34,-22,4,38,60,52,22,8,2,-4,-17,-8,2,15,
-30,-42,-39,-31,-22,-27,-14,24,52,76,60,38,30,23,15,0,-5,1,15,
-30,-50,-43,-33,-18,-15,13,46,69,72,60,49,42,38,23,12,0,0,15,
-30,-51,-55,-30,-17,-2,32,57,77,62,45,41,19,17,20,16,3,2,15,
-30,-50,-56,-35,-14,5,34,57,62,49,35,21,3,-2,6,14,3,2,15,
-30,-51,-58,-45,-10,19,44,64,51,32,25,9,-7,-11,3,9,3,1,15,
-30,-52,-62,-46,-16,20,48,50,35,21,13,4,-7,-12,-6,1,2,3,1,
};
[Touch HD General] Mach2003:
LOL, should we be correcting the GPS type's that HAVE seperation in the data?
[Touch HD General] cybermaus:
Wow, EGM96 and EGM2008 in this set have 51 points with >10meters, and a few above >40M difference in-between themselves!!! And the max absolute error in our routines is 55 not 28, with a good 140 or so >10M and half >2M

.

Of course, 140 points is only 3% Or am I reading the table wrong, what is NRC?

Sorry to say, but with most devices following probably something similar to 96, and such differences to 2008, maybe we should.

But only if you feel like it, because to be quite honest, already too much time went into this thing, which started (for me) as a method to solve my own personal problem (a configurable fixed offset would have been soooo much easier).

In fact, your program, even without any altitude awareness, already corrects 2 out of 3 peoples problem (in a sample set of 3 :)), so I think already a lot of people (of the subset aware of the problem) will be happy as is, so if you want to end this thing, do so and don't feel bad about it. Also realize, as soon as you write into your specs that it has altitude correction, rest assured you'll get post from people complaining the altitude is still often incorrect, because all we are doing is correcting the mid-point of the vertical random walk.

But if you do still have energy; Since you have the program ready (I stopped coding as soon as I read you had it, I am a slower programmer anyway), can you run another 5000 samples with a 2.5degree table? To see if we can get within 2M of 2008?
[Touch HD General] cybermaus:
Attached the html page (just open local on your PC) with all 172 samples >10 error geocoded on Google.

I wanted to see if they by luck would fall all over the ridges in the ocean. It turns out they ae clustered, but over Andes, Caribian and Central Asia
[Touch HD General] Mach2003:
I've made tables at 10, 5, and 2 degrees so far. (My software allows any granulation, and either builds the table or the 5000 point sample. I am using your interpolation, except changed to allow a different row and column size due to granulation.

At 5 degree, the number of >10 goes down to 239, most of then within 5 degrees North of the equator. At 2 degrees only 1 > 10 (csv attached).

"NRC" should be "NGA", here in Canada the National Research Councel is responsible for all these sorts of things :(, it is the output from the 2.5 minute table downloaded from the site I quoted above, using the Windows software from the same site. EGM94 is computed using your routine, Err is EGM94 - NRC, <2, <5 < 10 >10, are counts. Bottom of count rows have a sum.

The egm2008_2d.zip contains the 2 degree matrix as a header file. This way the table granulation is independant of the code module. I included the small code changes I made to your routine (as a comment).

EDIT: Oh I almost forgot, these 5000 points are not random for the whole world!!! I used 5 ranges that cover only the populated areas of the planet, and randonized 1000 points in each of those areas.

EDIT2: The bounding rectangles: N11, N60, W51, W128: N10, S56, W31, W82: N36, S38, W18, E60: N70, N/S0, E10, E98: N58, S49, E97, E175.
[Touch HD General] Mach2003:
Here is the 5 degree results and header file.

Would you have to modify your code to use a 2.5 degree matrix, I'm not sure I can get a non integer in there. I'd also have to modify my table builder code to use a non-integer number ((int)x += 2.5 = 2).

I also added a "sizeof(geoid)" line to the botton of the header file.

EDIT: Your code looks fine at 2.5, mine fails

Sizeof( geoid) comment is useless, using last matrix not new one.
[Touch HD General] cybermaus:
Geocoding the 2-degree matrix errors >2M into Google Maps (just open html locally on PC)

Edit: Going to do some real work now, the new day has started, need to make a living.
[Touch HD General] Mach2003:
I had a typo in the grid list I gave you, fixed by editing the post.

LatLo*100, LatHi*100, LonLo*100, LonHi*100, NumSamples

DoRange( 1100, 6000, -12800, -5100, 1000 );
DoRange( -5600, 1000, -8200, -3100, 1000 );
DoRange( -3800, 3600, -1800, 6000, 1000 );
DoRange( 0, 7000, -1000, 9800, 1000 );
DoRange( -4900, 5800, 9700, 17500, 1000 );
[Touch HD General] cybermaus:
Hmm, that did not explain the odd samples south of the azores, nor the lack of errors in the Andes and Indonesia. So I did a map of *all* samples (all 5000, regardless of error). I won't add the html page (took 20 minutes to process) but here is a screenshot.

I think your randomizer needs tweaking. And the boundary control.
[Touch HD General] Mach2003:
Yup, I had switched to a different rand() function recently. This 2d result uses the original rand function.
[Touch HD General] cybermaus:
Much better. And it shows what we expected, that this is quite good.