Network with your fellow employees and consider threatening to leave. The basic dynamic of work, especially in a large corporation, is to extract value from you, not to maximize your utility. Each manager down the chain is in turn trying to maximize their executive power to climb up the chain. Developing your career as a valued employee is distinctly secondary to the strategic considerations each manager faces in trying to maximize their own benefit. If you're doing a good job right now, the manager wants to keep you on his team rather than give you away to someone in IT, with whom the business department is in competition for influence in the C-suite. Your manager doesn't derive any great benefit from giving you away to another department; the nice virtuous feeling of maximizing the collective good does nothing to advance his political position or help him meet his targets next quarter after you've gone. It just presents him with the annoying problem of trying to replace you, which is actual work.

You should read Smith & Bueno de Mesquita's The Logic of Political Survival, or the simpler version presented for a more general audience The Dictator's Handbook. Corporations are not little meritocratic democracies, they're zero-sum dictatorships with nice architecture. There are politics in academia, of course, but ultimately it's an unselfish environment dedicated to the maximization and democratization of knowledge. The business world offers similar looking campuses and promotes collegiality among employees etc., but to a very different end.

Since you're a data scientist, why not (either at home or during, cough, 'spare' time at work) look into the work above and start running agent-based simulations examining the structure of authoritarian hierarchy and informational asymmetries? Imagine an internal economy in which there's a floating exchange rate between information and authority, and two flavors of information: external and internal. You could og course look for emergent behaviors but since your firm already has an org chart and that's probably available somewhere on the corporate servers then you should probably use that as a real-world benchmark for your, ah, researches.

It would be most interested to hear of your findings via gmail. Protip: people in the business department like talking about The Big Picture (saying this in terms of meeting collective corporate mission goals, meaning the drama of climbing the internal corporate ladder), and think of knowledge workers like your good self as borderline autistic for not being able to see the obvious path of going into management and getting other people to do the grunt work while they wield actual power.

On the plus side their relative lack of technical comprehension means you enjoy great flexibility in exactly how you structure your time and what results you produce, because managements' understanding of complexity and difficulty largely comes from you int he first place. Basically, your manager has power over you by and has no intention of cluing you in on his personal career strategy/politics, and you're confused because you know his stated explanations are bullshit.

It is perfectly OK for you to play this game back by pretending to accept his bullshit and strategically exploiting the resources available to you for your own gain as long as you give him a good story about what you're working on now. The downside is you won't get paid as much as if you got the promotion. But corporate doesn't care about you; the promotion is just a carrot to persuade you to work harder where you currently are. Put on a display of humility and drone-like belief in the wisdom of your manager (which I assure you he will be delighted to believe), then express your eagerness to pursue and aggressive new target that will redound to his glory. (Use the word 'aggressive' in describing it, this is a holy term among professional managers and will briefly cause his brain to stop working properly.) Emphasize the difficulty and novelty of the task you are about to take on and persuade him that you're ready to prove yourself again, all of which will appeal to his ego. Your goal here should be to devote about 50% of your work time to actually achieving whatever it is you have promised to do, and to apply the rest of your time to the improvement of your own strategic position. Make both considerations look like part of the same project and give him overly detailed technical explanations of how clever it is until he stops listening and gives you the go-ahead.