trentmkelly/LessWrong-43k huggingface dataset:
Link: Interview with Vladimir Vapnik
by Daniel_Burfoot — 2009-07-25T13:36:52.175Z

I recently stumbled across this remarkable interview with Vladimir Vapnik, a leading light in statistical learning theory, one of the creators of the Support Vector Machine algorithm, and generally a cool guy. The interviewer obviously knows his stuff and asks probing questions. Vapnik describes his current research and also makes some interesting philosophical comments:

V-V: I believe that something drastic has happened in computer science and machine learning. Until recently, philosophy was based on the very simple idea that the world is simple. In machine learning, for the first time, we have examples where the world is not simple. For example, when we solve the "forest" problem (which is a low-dimensional problem) and use data of size 15,000 we get 85%-87% accuracy. However, when we use 500,000 training examples we achieve 98% of correct answers. This means that a good decision rule is not a simple one, it cannot be described by a very few parameters. This is actually a crucial point in approach to empirical inference.

This point was very well described by Einstein who said "when the solution is simple, God is answering". That is, if a law is simple we can find it. He also said "when the number of factors coming into play is too large, scientific methods in most cases fail". In machine learning we dealing with a large number of factors. So the question is what is the real world? Is it simple or complex? Machine learning shows that there are examples of complex worlds. We should approach complex worlds from a completely different position than simple worlds. For example, in a complex world one should give up explain-ability (the main goal in classical science) to gain a better predict-ability.

R-GB: Do you claim that the assumption of mathematics and other sciences that there are very few and simple rules that govern the world is wrong?

V-V: I believe that it is wrong. As I mentioned before, the (low-dimensional) problem "forest" has a perfect solution, but it is not simple and you cannot obtain this solution using 15,000 examples.

Later:

R-GB: What do you think about the bounds on uniform convergence? Are they as good as we can expect them to be?

V-V: They are O.K. However the main problem is not the bound. There are conceptual questions and technical questions. From a conceptual point of view, you cannot avoid uniform convergence argum

...
Freaky Fairness
by cousin_it — 2009-07-25T08:14:32.185Z

Consider this game:

where the last payoff pair is very close to (3,2). I choose a row and you choose a column simultaneously, then I receive the first payoff in a pair and you receive the second. The game has no Nash equilibria in pure strategies, but that's beside the point right now because we drop the competitive setting and go all cooperative: all payoffs are in dollars and transferable, and we're allowed beforehand to sign a mutually binding contract about the play and the division of revenue. The question is, how much shall we win and how should we split it?

Game theory suggests we should convert the competitive game to a coalitional game and compute the Shapley value to divide the spoils. (Or some other solution concept, like the "nucleolus", but let's not go there. Assume for now that the Shapley value is "fair".) The first step is to assign a payoff to each of the 2N = 4 possible coalitions. Clearly, empty coalitions should receive 0, and the grand coalition (me and you) gets the maximum possible sum: 6 dollars. But what payoffs should we assign to the coalition of me and the coalition of you?

Now, there are at least two conflicting approaches to doing this: alpha and beta. The alpha approach says that "the value a coalition can get by itself" is its security value, i.e. the highest value it can win guaranteed if it chooses the strategy first. My alpha value is 2, and yours is 2+ϵ2. The beta approach says that "the value a coalition can get by itself" is the highest value that it cannot be prevented from winning if it chooses its strategy second. My beta value is 3+ϵ1, and yours is 3.

Astute readers already see the kicker: the Shapley value computed from alphas assigns 3-ϵ2/2 dollars to me and 3+ϵ2/2 dollars to you. The Shapley value of betas does the opposite for ϵ1. So who owes whom a penny?

That's disturbing.

Aha, you say. We should have considered mixed strategies when computing alpha and beta values! In fact, if we do so, we'll find that my alpha value equals my beta value and your alpha equals your beta, because that's true for games with mixed strategies in general (a result equivalent to the minimax theorem). My security value is (10+4ϵ1)/(4+ϵ1), and yours is (10-ϵ2)/(4-ϵ2).

This still means the signs of the epsilons determine who owes whom a penny. That's funny because, if you plot the game's payoffs, you will see that the game isn'...
Many Reasons
by steven0461 — 2009-07-25T05:09:28.208Z

I'm here to teach you a new phobia. The phobia concerns phrases such as "for many reasons".

Rational belief updating is a random walk without drift. If you expect your belief to go up (down) in response to evidence, you should instead make it go up (down) right now. If you're not convinced, read about conservation of expected evidence, or the law of iterated expectations.

If evidence comes in similar-sized chunks, the number of chunks in the "for" direction follows a binomial distribution with p=.5. Such a distribution can output most of the pieces being in the same direction, but if the number of pieces is high, this will happen quite rarely.

So if you can find, say, ten reasons to do or believe something and no reasons not to, something is going on.

One possibility is it's a one in a thousand coincidence. But let's not dwell on that.

Another possibility is that the process generating your reasons, while unbiased, is skewed. That is to say, it produces many weak reasons in one direction and a few strong reasons in the other, and it just happened not to produce such a strong reason in your case. And so we have many empirical reasons to think the Sun will rise tomorrow (e.g., it rose on June 3rd 1978 and February 16th 1260), and none that it won't. But this does not seem to describe cases like "what university should I choose", "should I believe in a hard takeoff singularity", or "is global warming harmful on net".

Another possibility (probably a special case of the previous one, but worth stating on its own) is that what you're describing as "many reasons" is really a set of different manifestations of the same underlying reason. Maybe you have a hundred legitimate reasons for not hiring someone, including that he smashes furniture, howls at the moon, and strangles kittens. If so, the reason underlying all these may just be that he's nuts.

Then there's the last, scariest, most important possibility. You may be biased toward finding reasons in one direction, so that you will predictably trend toward your favorite belief. This means you're doing something wrong! Luckily, thinking about why you thought the phrase "for many reasons" caused you to find out.

In sum, when your brain speaks of "many reasons" all going the same way, grab, shake, and strangle it. It may just barf up a better, more compressed way of seeing the world, or confess to confirmation bias. ...
Celebrate Trivial Impetuses
by Alicorn — 2009-07-24T22:36:05.049Z

There is a flipside to the trivial inconvenience: the trivial impetus. This is the objectively inconsequential factor that gets you off your rear and doing something you probably would have left undone. It doesn't have to be a major, crippling akrasia issue. I'm not talking so much about finishing your dissertation or remodeling your house, although a trivial impetus could probably get you to make some progress on either. I'm talking about little things that make your life a little better, like trying a new food or permitting a friend to drag you along to a gathering of people and pizza.

An illustrative anecdote: the first time I tried guacamole, I was out with my family at a restaurant and my parents decided to order some. The waiter came out with a little cart with decorative little bowls full of ingredients and a couple of avocados, and proceeded to make guacamole right there with all the finesse of one of those chefs at a hibachi restaurant. He then presented us with the dish of guacamole and a basket of chips.

If my prior reasons for avoiding guacamole had been related to concerns about its freshness or possible arsenic content, this would have been a non-trivial reason to try the new food, but they weren't - I was just twelve, and it was green goop. But on that day, it was green goop that someone had made right in front of me like performance art! I simply had to have some! It was delicious. I have enjoyed guacamole ever since. I would almost certainly have taken years longer to try it, if ever I did, had it not been for that restaurant's habit of making each batch of guacamole fresh in front of the customer.

Not all trivial impetuses have to be so random and fortuitous. Just as you can arrange trivial inconveniences to stand between you and things you should not be doing, you can often arrange trivial impetuses to push you towards things you should be doing. For instance, I often get my friends to instruct me to do things when I'm having trouble getting moving: sometimes all it takes to get me to stop dithering and start making the pasta salad I agreed to bring to a party is someone agreeing when I say, "I should make pasta salad now". Or "I should go to bed now", or "I should probably pay that bill now".

Does anyone have any other ideas for trivial impetuses that could be...
Are calibration and rational decisions mutually exclusive? (Part two)
by Cyan — 2009-07-24T00:49:14.505Z

In my previous post, I alluded to a result that could potentially convince a frequentist to favor Bayesian posterior distributions over confidence intervals. It’s called the complete class theorem, due to a statistician named Abraham Wald. Wald developed the structure of frequentist decision theory and characterized the class of decision rules that have a certain optimality property.

Frequentist decision theory reduces the decision process to its basic constituents, i.e., data, actions, true states, and incurred losses. It connects them using mathematical functions that characterize their dependencies, i.e., the true state determines the probability distribution of the data, the decision rule maps data to a particular action, and the chosen action and true states together determine the incurred loss. To evaluate potential decision rules, frequentist decision theory uses the risk function, which is defined as the expected loss of a decision rule with respect to the data distribution. The risk function therefore maps (decision rule, true state)-pairs to the average loss under a hypothetical infinite replication of the decision problem.

Since the true state is not known, decision rules must be evaluated over all possible true states. A decision rule is said to be “dominated” if there is another decision rule whose risk is never worse for any possible true state and is better for at least one true state. A decision rule which is not dominated is deemed “admissible”. (This is the optimality property alluded to above.) The punch line is that under some weak conditions, the complete class of admissible decision rules is precisely the class of rules which minimize a Bayesian posterior expected loss.

(This result sparked interest in the Bayesian approach among statisticians in the 1950s. This interest eventually led to the axiomatic decision theory that characterizes rational agents as obeying certain fundamental constraints and proves that they act as if they had a prior distribution and a loss function.)

Taken together, the calibration results of the previous post and the complete class theorem suggest (to me, anyway) that irrespective of one's philosophical views on frequentism versus Bayesianism, perfect calibration is not possible in full generality for a rational decision-making agent.
AndrewH's observation and opportunity costs
by Scott Alexander — 2009-07-23T11:36:34.233Z

In his discussion of "cryocrastination", AndrewH makes a pretty good point. There may be some better things you can do with the money you'd spend on cryonics insurance. The sort of people who are into cryonics would probably accept that donating it to the Singularity Institute is probably, all in all, a higher utility use of however many dollars. Andrew's conclusion is that you should figure out what maximizes utility and do it, regardless of how small a contribution is involved. He's right, but I want to use the same example to push a point that is very slightly different, or maybe a little more general, or maybe the exact same one but phrased differently.
Consider an argument frequently made when politicians are discussing the budget. I frequently hear people say it would cost between ten and twenty billion dollars a year to feed all the hungry people in the world. I don't know if that's true or not, and considering the recent skepticism about aid it probably isn't, but let's say the politicians believe it. So when they look at (for example) NASA's budget of fifteen billion dollars, they say something like "It's criminal to be spending all this money on space probes and radio telescopes when it could eliminate world hunger, so let's cut NASA's budget."
You see the problem? When we cut NASA's budget, it doesn't immediately go into the "solve world hunger" fund. It goes into the rest of the budget, and probably gets divided among the Congressman Johnson Memorial Fisheries Museum and purchasing twelve-thousand-dollar staplers.
The same is true of cryocrastination. Unless you actually take that money you would have spent on cryonics and donate it to the Singularity Institute, it's going into the rest of your budget, and you'll probably spend it on coffee and plasma TVs and famous statistician trading cards and whatever else.
I find myself frequently making this error in the following way: a beggar asks me for money, and I want to give it to them on the grounds that they have activated my urge to help people. Then think to myself "I can't justify giving the money to this beggar when it would help many more people if I gave it to a responsible charity." So I say no, and forget all about it, and never give the money to anyone. Even though (from a charity point of view) I know of a superior alternative to giving the money to the beggar, I would still be better off jus...
The Nature of Offense
by Wei Dai — 2009-07-23T11:15:54.647Z

Recently, an extended discussion has taken place over the fact that a portion of comments here were found to be offensive by some members of this community, while others denied their offensive nature or professed to be puzzled by why they are considered offensive. Several possible explanations for why the comments are offensive have been advanced, and solutions offered based on them:

to be thought of, talked about as, or treated like a non-person (Alicorn)

analysis of behavior that puts the reader in the group being analyzed, and the speaker outside it (orthonormal)

exclusion from the intended audience (Eliezer)

Each of these explanations seems to have an element of truth, and each solution seems to have a chance of ameliorating the problem. But even though the discussion has mostly died down, we appear far from reaching an agreement, and I think one reason may be the lack of a general theory of the phenomenon of "offense", in the sense of giving and taking offense, that we can use to explain what has happened, so all of the proposed explanations and solutions feel somewhat arbitrary and unfair.

(I think this article has it mostly right, but I’ll give a much shorter account since I can skip the background evo psych info, and I’m not being paid by the word. :)

Let’s consider what other behavior are often considered offensive and see if we can find a pattern:

use of vulgar language (where it's not customarily used)

failing to address someone by their honorary titles

not affording someone their customary privileges

to impugn someone’s beauty, intelligence, talent, morality, honor, ancestry, etc.

making a joke at someone’s expense

What do all these have in common? Hint: the answer is quite ironic, given the comment that first triggered this whole fracas.

most people here don't value social status enough and (especially the men) don't value having sex with extremely attractive women that money and status would get them

As you may have guessed by now, I think the answer is status. Specifically, to give offense is to imply that a person or group has or should have low status. Taking offense then becomes easy to explain: it’s to defend someone’s status from such an implication, out of a sense of either fairness or self-interest. Let’s go back to the three hypotheses I collected and see if this theory can cove

...
Are calibration and rational decisions mutually exclusive? (Part one)
by Cyan — 2009-07-23T05:15:45.853Z

I'm planning a two-part sequence with the aim of throwing open the question in the title to the LW commentariat. In this part I’ll briefly go over the concept of calibration of probability distributions and point out a discrepancy between calibration and Bayesian updating.

It's a tenet of rationality that we should seek to be well-calibrated. That is, suppose that we are called on to give interval estimates for a large number of quantities; we give each interval an associated epistemic probability. We declare ourselves well-calibrated if the relative frequency with which the quantities fall within our specified intervals matches our claimed probability. (The Technical Explanation of Technical Explanations discusses calibration in more detail, although it mostly discusses discrete estimands, while here I'm thinking about continuous estimands.)

Frequentists also produce interval estimates, at least when "random" data is available. A frequentist "confidence interval" is really a function from the data and a user-specified confidence level (a number from 0 to 1) to an interval. The confidence interval procedure is "valid" if in a hypothetical infinite sequence of replications of the experiment, the relative frequency with which the realized intervals contain the estimand is equal to the confidence level. (Less strictly, we may require "greater than or equal" rather than "equal".) The similarity between valid confidence coverage and well-calibrated epistemic probability intervals is evident.

This similarity suggests an approach for specifying non-informative prior distributions, i.e., we require that such priors yield posterior intervals that are also valid confidence intervals in a frequentist sense. This "matching prior" program does not succeed in full generality. There are a few special cases of data distributions where a matching prior exists, but by and large, posterior intervals can at best produce only asymptotically valid confidence coverage. Furthurmore, according to my understanding of the material, if your model of the data-generating process contains more than one scalar parameter, you have to pick one "interest parameter" and be satisfied with good confidence coverage for the marginal posterior intervals for that parameter alone. For approximate matching priors with the highest order of accuracy, a different cho...
The Price of Integrity
by Aurini — 2009-07-23T04:30:48.788Z

Related Posts: Prices or Bindings?

On the evening of August 14th, 2006 a pair of Fox News journalists, Steve Centanni and Olaf Wiig were seized by Islamic militants while on assignment in Gaza City. Nothing was heard of them for nine days until a group calling themselves the Holy Jihad Brigades took credit for the kidnappings. They issued an ultimatum, demanding the release of Muslims prisoners from American jails within a 72 hour time frame. Their demands were not met.

But then a few days later the journalists were allowed to go free... but not before they’d been forced into converting to Islam at gunpoint, and had each videotaped a statement denouncing U.S. and Israeli foreign policy.

The war raged on.

A couple of kidnapped journalists is nothing new (certainly not three years after the fact) and aside from the happy ending this particular case wouldn’t worth mentioning if not for a unique twist that occurred after they returned home. A fellow Fox News contributor, Sandy Rios, openly criticized the two men; she said that no true Christian would convert – falsely or otherwise – merely because they were threatened with death. As she later explained to Bill Maher:*

My point was that Christians – I don’t know what their faith is – but I’m talking about Christians who responded to the story and said that they would have done the same thing...

Christ followers can’t do that. We don’t have that freedom. We have to profess Christ no matter what... Christianity is, by its very nature, radical. It is not normal or natural to lay down your life for a friend. It is not natural or normal to say ‘I will not deny my faith even if you do cut my head off.'

I agree with her, and admire her courage for sticking with her convictions. If you buy into Christianity’s metaphysical claims, then bearing false witness to your faith ought be considered a serious crime; not only does it show a pathological attachment to life (when eternal bliss lies just around the corner) furthermore, it completely ignores the core premises of Christianity, as well as the death of its founder. This very question split the early Church: whether or not those who'd become apostates [had renounced Christ] due to persecution at the hands of the Romans could ever be fo...
Murder
by KatjaGrace — 2009-07-23T02:52:37.000Z

People are murderers if they kill other people. They are not murderers if they let other people die when they can cheaply prevent it. For instance I am not a murderer if I spend a couple of hundred dollars on clothing rather than sending it to a decent charity, even if the predicted result is that one more person will die.

People don’t want to be murderers, but they don’t mind letting people die, except those who are close to them. People also don’t like or respect murderers, but they don’t mind others letting people die, except people who are close to them. People don’t like being murdered or being allowed to die equivalently, regardless of whether those involved are close to them. It is interesting that people’s treatment of others’ lives correlates so with how third party observers deal out like and respect, rather than how the person whose life is at stake feels about it. It is commonly assumed that killing people is bad because we care about the person who gets killed. This might be what we think about when we are condemning murderers, but it doesn’t predict our actions at all well.

Humans aren’t evil here in the same sense that we think of someone who kills for a pair of jeans as evil. It’s not purposeful. Most people believe that they do care about other people’s lives, because they have great trust in their emotions to tell them when something bad is happening. They never check this. But what do you do when you find your emotions do not tell you this at all? One response is to spend yonks trying to justify things like physical distance and action vs. omission as being morally relevant while taking credit for being wonderfully deep. This is evil.