I wonder how many years of probabilistic thinking it would take to drive it back into your head?

What's your prior for p1 = P(jars contain jelly and something else | you're told "six jars have jelly"), and your prior for p2 = P(jars contain only jelly | you're told "six jars have jelly")? Are they the same for you? For me, they're quite different: p1 << p2. Why is that? As a first guess, I'm very unused to seeing jars containing jelly plus other crap, even though I have seen those strange peanut butter + jelly combo jars. (http://www.americansweets.co.uk/ekmps/shops/statesidecandy/i...) Furthermore I'm unused to being told "a jar contains X" when a jar in fact contains "X and Y (and Z and ...)".

The Majority use probabilistic reasoning (even if they often suck at it) in every day life. Rational agents (which humans approximate) use probabilistic reasoning, see Cox's Theorem and the work of others. This is (part of) why calling out a logical fallacy in an argument is often a mistake -- a lot of arguments aren't small (or large) trades of deductive proofs, they're probabilistic arguments where each side is trying to update the other side with supposedly new information. Many logical fallacies can turn into probabilistic theorems. For example, "arguments from authority" can become valid in a probabilistic framework. How? While [expert said X is true] --> [X is true] is a fallacious deduction (unless you assume as a premise that whatever expert says is true, then it's just a tautology...), probabilistically our everyday experience suggests P(x is true | expert said x is true) > P(x is true | random nobody said x is true). We can find plenty of cases where experts are wrong, but in general that pattern seems to hold, and we're more willing to accept counter-intuitive truths from experts than from people on the street.

To address one of your examples: [We're informed 'a few people in the crowd acted out against the police'] --> [Not all of them acted out] is indeed an invalid logical deduction without more premises, but let's treat this probabilistically. (Fair warning: I tried my best to make sure there are no errors in my math but I've been up all night writing signal processing code...)

Let x="Not all of the crowd acted out",

let y="We're informed 'a few people in the crowd acted out against the police'",

and let c=our general background context="We know an incident occurred with the crowd and the police".

We have a prior belief about x: P(x | c) = P("Not all of the crowd acted out" | "we know an incident occurred").

We receive a new piece of information, y: "We're informed 'a few people in the crowd acted out against the police'". Now let's perform a Bayesian update:

P(x | y and c) = P(x | c) * P(y | x and c) / P(y | c)

Clearly if P(y | x and c) > P(y | c), then our updated belief about "Not all of the crowd acted out" will go up, and the inference [y] --> [x] is valid. (You would interpret the --> not as "therefore" but "increases our confidence in". [Being informed the phrase 'a few people acted out'] increases our confidence in ['not all people acted out'].)

I think it's clear that P(y | x and c) is in fact greater than P(y | c). If you don't, then continue with this next section. Using marginalization we can break down P(y | c) independent of x, then use the product rule to break it down even further: P(y | c) = P(y and x | c) + P(y and not(x) | c) = P(y | x and c) * P(x | c) + P(y | not(x) and c) * P(not(x) | c)

So, is P(y | x and c) >? P(y | x and c) * P(x | c) + P(y | not(x) and c) * P(not(x) | c), subtract and factor:

P(y | x and c) * (1 - P(x | c)) >? P(y | not(x) and c) * P(not(x) | c), by the sum rule:

P(y | x and c) * p(not(x) | c) >? P(y | not(x) and c) * P(not(x) | c)

P(y | x and c) >? P(y | not(x) and c)

Reminding what all the variables are again (temporarily dropping off the c for brevity):

P("told 'a few' acted out" | "not all acted out") >? P("told 'a few' acted out" | "all acted out!")

I don't know about you, but I think it's very unlikely to be told 'a few' acted out if in reality everyone acted out. QED.~

To put my whole comment another way: try writing an AI and sick it on this test. If it doesn't use probabilistic algorithms and instead insists when it comes to question #2 that "I cannot know the answer other than it's between 0 and 8, the question isn't explicit enough!", it's going to perform very poorly. Human language is not crisp, treating it as if it were is a bad idea, that's why we use crisp math when we want to be crisp and if we can do automated proofs so much the better.

On your last remark, I find it amusing that if I take your "it's not explicit enough!" complaint seriously, I'm not sure whether you think the correct answer is supposed to be "proton" or "neutron" or "electron". That's because I'm not sure whether you mean "least negative" to mean to "furthest to the right from negative epsilon (i.e. 'most positive')", implying the proton, or "smallest member of the negative reals excluding 0", implying the electron, or "smallest member of the negative reals and 0", implying the neutron. If I wrote the question I think I would have meant "proton" as the answer. So given that the answer is proton, however, the error the student most likely made is just a parse error, failing to properly change "least negative" into "most positive". Similar to answering "yes" to the question "the sky is not not green?", or asserting -3 * -3=-9. It's a different sort of trick on the student than what you insist question #2 could be tricking the student. In #2, the trick would be "Aha, but we never said the jars were entirely jelly! Don't insert words that aren't there!" In this charge question, the trick is "Aha, you didn't parse 'least negative' into 'most positive'! Read carefully!"