I'm currently losing weight with a weight loss specialist. They had an intro presentation and went over some of the reasons we have such an obesity epidemic and why it's so hard to keep weight off. Some of the key points they mentioned:(I'm paraphrasing here because I don't have my notes on me.)

-

There is a hunger hormone called Ghrelin. The amount that your body produces is based on the highest weight attained that you kept for at least a year. When you're at that weight, your Ghrelin levels are at about the same as someone who is 150 pounds lighter that you who is also at their max weight ever attained. However, once you've lost weight your body starts producing more Ghrelin which makes you hungrier. This takes decades to reset from whatever your bodies weight target from 'highest weight held for at least a year' to your new lower weight so basically your Ghrelin levels will be higher than average for decades after losing weight(unless you gain it back). There is also another hormone that was recently discovered to has an effect on appetite but I can't remember its name and we didn't know much about it yet.

-

Regarding gaining weight, there are three satiety signals that are used to help you not overeat. One is a feeling of fullness that only lasts for ~15 minutes but kicks in within a bite or two of calories needed to maintain weight so if you have this signal you may only gain 3 pounds max in a year as long as you don't wait 20 minutes and then go back for seconds. The second satiety signal was loss of savor(food just stops tasting good) and lasts for a couple(~4?) hours. The last was feeling nauseous(not the same as eating so much you couldn't fit another bite and can't move, we can all get that), and was triggered by eating a couple bites past the fullness signal level(if you only have this signal you may gain ~15-20 lbs in a year if I'm remembering correctly). The problem is that most people only have one or two of these signals and many don't have any of them. If you've never had to diet to lose weight and have always had a good weight level then congratulations, you probably have at least 1 or 2 of these and maybe all three. If you've had several diets and have lost hundreds of pounds in aggregate over several diets over a decade plus then chances are you probably don't have any of the 3 signals. Everyone else probably falls in the middle where they only have 1 or maybe 2 satiety signals.

-

If we were able to determine what gene/set of genes or other processes determined if you had these signals and were able to give others who don't have them the ability to receive these same signals we would be able to keep new people from gaining weight. If we found a way to reset Grehlin levels for people who have lost weight(hopefully a pill, maybe through gene modification via crispr) then we'd be able to help people who have lost weight to keep it off long term. Regarding sugars, etc. yes, it makes it infinitely harder to diet when there are so many high calorie options out there, but there are deeper underlying medical reasons out there than just that.

-

Basically, the only people who have been able to be successful at keeping weight off long term are those who either exercise 2+ hours PER DAY(less time can help with strength, etc but not with weight loss/maintenance because you'll just be hungrier and end up eating more calories to make up for the ones you burned) or those who are perpetually on a diet for the rest of their lives and work hard to re-lose any pounds gained after a trip, etc. The good news is we're making some progress in weight loss research(but still have a long ways to go) and there are decent appetite suppressants, etc. that can help you lose weight and keep it off if you go to a Dr. that specializes in this and that this isn't an impossible to solve willpower problem, but a medical one that can be fixed. The bad news is you'll probably be on a diet for the rest of your life or until we solve the above questions on satiety signals and hunger hormones.

-

tldr: it's a medical problem, not a willpower problem, and won't be fully solved until we treat it like such and invest the resources into finding a cure. Until then we're just treating the symptoms.