> So, my question: Is there a way to get Agile right? or are we doomed to fail in the same ways that has been described by countless articles discussing the downfalls of Agile?

Yes. Don't be dogmatic. Pick an initial set of processes/practices that match more or less what you already do, want to do (based on a reasonable belief of utility and practicality), or have done (with success) in the past. As you move forward, periodically reexamine what you do and change when something isn't working well (in Scrum this is the Sprint retrospective, which seems to be the first thing cut from every Scrum-derived process I've ever been part of or near to "save time"). Experiment not just with your code and systems, but with your processes and organizational structures themselves. If you only apply things like the ideas of Extreme Programming to your programs, you're only doing half (actually less) of what Agile aims for.

Sterman in his Business Dynamics book has a nice set of diagrams starting on page 15, talking about feedback loops/systems for learning.

Attempting to draw them as ASCII art, not that skilled at this:

Simplest feedback model:

Real world
^ \
/ \
/ v
Decisions<--Information feedback

This is a very common thing, people make a decision, try it, get info, and make new decisions. What they don't do is extend this feedback to their decision making model. He extends this model on page 16 with:

Real world
^ \
/ \
/ v
Decisions<--Information feedback
^
\
\
Strategy, Structure,
Decision Rules
^
\
\
Mental Models
of Real World

Again, note that the feedback does not go back to the mental models which feed into the decision making strategies and, ultimately, the decisions. This is the dogmatic approach to any process. You select it, and you run it regardless of its real applicability or utility but based on a belief that you don't update (because it's dogmatic). Doesn't matter if this is Agile (big-A) or big design up front (BDUF) or anything else, if you don't reconsider your models then you will not be agile, you will be stuck. This doesn't mean things won't work, you may have settled into a local optimum (on accident or on purpose). Maybe things are done this way because they've always been done this way, but 30 years ago someone did think about the business and selected something that worked (deliberately) and things haven't changed much so it happens to still work well. But once it fails, you have no deliberate way of adapting to the new reality (which is a key to being agile, responsiveness).

The final model he shows on page 19 brings that feedback to not just the decisions but also the models (this is the hard one to put into ASCII art):

Real world
^ \
/ \
/ v
Decisions<----Information feedback
^ ^|
| ||
| |v
Strategy, Structure,<----Mental Models
Decision Rules of Real World

This, to me, is the critical thing to agility and if you focus just on the manifesto and not the Agile Industrial Complex (and fuck SAFe and others like them), you will see that this is present (though less explicitly). Paying attention to feedback, trying new things, being adaptive. That is the essence of agility and Agility.

=======================

Regarding using a shared spread sheet or something else for managing your work, that's all about scale. A 4-person team can use a spreadsheet for status management, it may not be the best, but it will totally work. Scale up to 4 teams, each with 10+ people, possibly geographically separated, and you'll need something else. But take one 4-person team or 4-teams that are all located in the same space, you could manage your status with sticky notes on a whiteboard and probably be as successful as with the spreadsheet or the fanciest project management software (and maybe more successful in some cases because of the relative ease and flexibility sticky notes on a whiteboard afford compared to the fixed, or more fixed, workflows of most PM systems). What you record, how often you review and consider things, those are going to be more critical.