Level crossings aren't an issue. It's easy to detect the presence of large objects blocking a level crossing if the barriers fail to come down and signal to the train to stop before it reaches the crossing.

Drivers tend to earn much more money than guards do because the work is more skilled. If all you're doing is opening doors then you can be a min wage worker who is easily replaceable and thus not very unionisable. So there is benefit even in systems that still need guards.

The real issues are deeper. Yes, union opposition is a big one, but there are technical problems too. Full moving block signalling i.e. trains that track how far they are from other trains and know to slow down or speed up depending on distance was being discussed in the 1990's but the only implementations anywhere are on metro lines. For above-ground heavy rail there are no implementations anywhere in the world, as far as I know.

As observed by the OP this cannot be entirely a technical issue. If Google can make a driverless car that can navigate California, a driverless train is surely far easier. The real problem is that train automation is not a problem that attracts the best engineers. Google can hire out entire academic departments of the top machine vision researchers in the world to work on self driving cars because self driving cars are something that appeals to the whims of billionaires and techies alike. Also because the market size is effectively limitless.

But when was the last time you heard about an automated train startup? When did you hear about someone leaving their job at Facebook to go work on train automation? Trains do not appeal to rich billionaires or American tech workers in the same way that cars do, so funding for them comes entirely from the capex budgets of mostly government funded rail operators, and is spent on a handful of large engineering conglomerates. There are I think only about 4 companies in the world that can make automated train systems of any kind. Each engagement is a massive activity that always requires very large and expensive customisations to the base system. Because there are so few competitors they're huge public contracts and it only takes one or two to be ruled out or refuse to bid and you're down to a single potential supplier.

Projects like this can fail and when they fail they fail very expensively and with huge political fallout. They are the epitome of huge expensive government IT projects. The Jubilee line automation project started installing equipment in 2006 and didn't activate until 2011, with loads of line closures and problems that made major news.

There are also some more direct technical issues. Trains have huge stopping distances. They need to know where other trains are blockages are far in advance of being able to physically see them. They also spend a lot of time in tunnels. Therefore camera based approaches like what self driving cars use are not so trustworthy. This means upgrades to the track so all trains can know where they are at all times, and to get the benefits means reliable communication from a central control room rather than all trains autonomously deciding things like how fast to go (remember they cannot see slow trains on the track ahead). These things in turn mean line closures and engineering works, which are very expensive. And remember there's no money for upgrades because they're all government subsidised.

There's an article about some of the issues and the smoother rollouts London is seeing these days here:

https://www.railengineer.uk/2015/05/08/lu-northern-line-goes...