The "food quality" argument (as it is put above) or really any argument claiming that caloric accounting is only a part of the picture basically say:

It's a lot more complicated than that. Even if is caloric accounting is true in the sense that you could control weight gain/loss via caloric surpluses/deficits (pretty hard to deny) this does not account for why caloric surpluses are so common.

A daily surplus of just 100 calories per day (~4% of daily total) will earn you an extra kg every couple of months (7000 calories per kg), 5-6 per year. Being off by in your "calculations" by this much will take you from underweight to obese in a few years. How then, do the many people who don't fluctuate their weight wildly manage to balance out their caloric input/output without precisely measuring their food. In fact, even highly disciplined people who weigh and record every bite could be off by 100 calories just because of inaccuracies in labeling. If your lemons are more caloric than the average lemon, you're screwed.

In the context of intentional and highly disciplined rapid changes in body size (like dieters or body builders) the caloric accounting method works. Add/remove 500 calories per day to add/remove about a pound per week, all else equal.

It's obviously "true," but just as obviously is not a complete picture. The relevance is contextual and practical. IE, body builders and crash dieters should and do reference it. A parent trying to make sure their kids are healthy does not.

We don't really understand metabolism enough to have real science based answers to our questions. That's why I think all these accusations of "bro science" are silly. There is no alternative. Referencing caloric accounting is attractive because it can be lab tested and it works in a lab, but labs control for other factors (like apetite). That doesn't mean it's a complete picture. It's relevant only to the extent that you can mimic lab conditions in your life.

Just like caloric deficit is true when isolated, so are other ideas that get thrown around. Hyper-palatable processed foods. Sugar addiction. Carbs. Insulin responses and appetite. Glycemic indexes. Gut flora. The carb/insulin people are interested in how food affects appetite.

I'll give you an example just to demonstrate how little we know. Say you take a morbidly obese individual and have them lose 100kg of weight, they have a capacity to regain that weight in a very short period much faster than anyone who has never been obese. A regular person usually cannot gain 25kg in a year. Biggest losers do it regularly

The same is true of body builders (power lifters, rugby players, etc). If they lose their 25 kg of "excess" muscle, they will often be able to regain much faster than anyone starting from scratch. IE take two 75kg guys with 15% body fat. One was 90kg with 10% body fat 5 years ago. Have them both exercise and diet hard, racing to put on 10kg of muscle. Use trainers to "control" for experience as much as possible. The ex body builder will leave the other guy in the dust.

I've heard this called "cellular memory," "muscle memory" (which is already takes, so rude) and such. But naming it does not mean we understand it. We don't have any kind of understanding of how or why this works. We just don't know.

This is why diet debates, especially among HN-like intellectual crowds can be annoying. Everyone has their tidbit of "scientifically proven" knowledge, but the truth is that we're in the dark ages. It's like geology before plate tectonics or biology before darwinian evolution.