I don't have access to Kahneman's full paper to scrutinize (yet - I'm asking around) but it seems possible to me that they wanted to place themselves in opposition to the general practices of education at the time. Notice that since his work began, education has changed drastically, several times over. Several examples of approaches tried are the classical lecture being jotted down to study at home; study groups with two-way communication; exam preparation; test preparation; collaborative group work; coaching; a specialist could go on.

Bear in mind that 50 years ago the SAT was still in flux and a lot of it was being experimented with and new. Given that it was one of the biggest, and most repeatedly renewed concerns for researchers like Kahneman, there's little to doubt he's not only an expert on the SAT, but also knows as many of its downsides as anyone.

Therefore, I believe this paper is a sort of "lessons learned" story which shows that his approach is better than the form of education being undertaken by schools.

⁂

I find it ESPECIALLY curious that neither the article nor anyone in this whole thread (and have I tried reading most of it) has commented on the process, and consequences, of being caught on trick questions like bat and ball.

It is my understanding that our intelligence has evolved through use in situations where its impact was more or less immediately visible, and where this feedback could be acted upon.

Example: shaping tools. Is the flint stone sharp? No. Mash it against rocks. Is it sharp now? It's a bit sharper, but not sharp enough. Mash it against rocks some more. Is it sharp enough now? OK, you're done shaping your spear head.

Example: hunting. Approach the prey. It runs away. You don't notice why, you hadn't taken wind into account when looking for clues. No feedback, therefore you couldn't act upon it.

Example: hunting. Approach the prey. It runs away. You notice it did after wind turned and gave away your position. You got feedback from the grass and leaves moving in the wind. Next time you'll be able to act upon this.

Example: trying to shake fruit off a tree. You find a low-hanging branch and try to shake it. First you try this way, then that way, and finally you find the best way to get fruit without making too much fall down.

The last example extends to any sort of experimenting, tinkering, happy-hacking.

It is however notable that the bat and ball question does not test that. There's a question, and you give an answer. There's no feedback before it becomes final, and once it does that is the clear cut-off. This represents the stone-cold, immovable, monolithic machinery displayed by many technical subjects, such as science, mathematics, and some forms of computer programming, but also strategy, and some forms of art. There is no iterative process, you get one try, based on which you can in no way build a tangible mental model of how something works and what parameters of your thinking you need to adjust in order to better yourself. One very stupid example is when someone in a job interview asks you about standard library function names and argument orders. Either you remember, or you don't. I'll call this feedback the feedback gap.

Bear in mind I have mentioned "happy-hacking" above and "computer programming" below. In fact, they're both computer programming. The difference? If I'm presented with a python program where I can use an iterative process, a repl, and its help() command that immediately gives me access to documentation, then I can very easily build up a mental model of what's going on. Exceptions and errors give me constant, constructive feedback which comes immediately. This immediacy is extremely important and even a slight delay makes the learning process slower. Additionally, if some things aren't available as immediate feedback, I can find out. For example, when trying to get at the fruit, I saw immediately where the fruit was in the tree branches. When typing out python, I don't have this, I don't see what the functions are, so I need to use help(). That works well enough. Some people like intellisense for that. Works well too. It all fills the feedback gap.

There's a similar difference in ease of progress between experimental mechanics and similar physics, versus branches where experimentation cannot happen. That's feedback gap again.

It is my belief that this sort of immediate feedback is needed in other technical subjects, especially mathematics and physics. Approaches such as theorem provers are helpful in mathematics, but they're nowhere near being complete, and nowhere near the utility and immediacy of a repl. I am fairly sure there are other ways in which the feedback gap can be filled. Perhaps different methodology, or differently structured theories, can give us more immediate feedback? Perhaps mathematical systems in which theorems are easier to tentatively prove or disprove can become more successful in breeding new results?

Or perhaps the theories are not to blame, but we need more tools. For example, my abstract geometry teacher kept reminding us that we needed to come up with such quick checks. Non-linearities were always useful. His favourite was the binary distance function, which was 0 for two identical points and 1 for different points. A lot of stupid theorems can be disproved by checking some examples with this.

Can someone else comment on any such tools?

Writing this comment definitely came with an insight or two for me. If you read it, thanks for going on this trip with me.