Are you familiar with Automa? I tried to get my organization to purchase it for automated testing, but approval for that ran aground on the fact that Automa doesn't have a US distributor. (As you may infer, I'm a gov't contractor.)

I was hoping that your company offerred a product similar to Automa, but it looks like your business model is different to that.

Anyway I ended up having to re-implement my Automa-using Python scripts using a different technology (TestStack.White and C#) and I 100% concur with your analogy about TCP over IP and building on unreliable foundation. It's also analogous to the sensor fusion problem of AI: picking out what to do when different methods of interacting with the automation layer report conflicting results. (And you have multiple ways because you need the redundancy to detect and deal with problem #1 - automation interfaces are incredibly unreliable.)

For example between IT-mandated invasive antivirus software injecting shims in the OS and/or screwy win32 code in the program-under-test, or bugs in TestStack.White, (I've found indications of all three causes) there is some kind of randomness or ambiguity in the Window handles you get when you ask for certain controls on a dialog. You might ask for the OK button and get the button 50% of the time and 50% of the time get a handle which turns out to be the textbox above it.

So I've taken a belt-and-suspenders approach and built a database of all of the controls I expect to find and their screen positions, a folder full of screenshot snippets which I intend to make my test automation program compare, and use that to augment what Windows UIA is telling me is on-screen. Due to the problem of receiving the wrong window handle (on this app) when I ask TestStack.White to find a control at a certain position or by name, I resort to just cross-collating the entire list of controls on the window with the best match from the database. However even this is 4x harder than it should be, because for some reason the positions and sizes in the database (obtained by a previous attempt with a different automation tool) differ by small pixel amounts from what TestStack.White reports. So I had to implement a fuzzy match.

And then the entire object-oriented architecture of my automation utility program collapsed on itself like a house of cards, because now I have 2 or 3 different derived classes to represent "the same" conceptual object: if you code "button.Click()" should it click the button with the handle found by TestStack.White? Or click the coordinate from the database? Or click where the screenshot matched? Or do nothing because this is a mock object? Worse, I now have a constellation of 1..N OOP instances associated with each conceptual object: maybe the database says I should have a button, but I can't find a handle for it thus I cannot create a ButtonFromHandle but no matter I can still click on the center coord of ButtonFromData. But if I have both I really want to default to one (which?) that is most reliable. Object-oriented programming design methodologies are completely inadequate to deal with the case of "some number between 1 and N instances actually represent one conceptual object and should be treated like a sheaf of overlaid transparencies". Yes, it can be done, but the design turns into something that makes sense from that aspect but makes no sense from the API-user's point of view: you just want to click "the" button you don't want to do "button.Implementations.First().Click()" or some such. Then add in multiple layers of mocks for unit testing the automation tool and you get N*M combinations...

Not sure where I was going with this, I guess I'm just excited to find someone who's worked on the same (or similar) problem.