trentmkelly/LessWrong-43k huggingface dataset:
a visual explanation of Bayesian updating
by Jan Christian Refsgaard — 2021-05-08T19:45:46.756Z
Link: https://www.badprior.com/blog/a-visual-explanation-of-bayesian-updating/

As a teaser here is the visual version of Bayesian updating:

But in order to understand that figure we need to go through the prior and likelihood!

You find me standing in a basketball court ready to shoot some hoops. What do
you believe about my performance before I take a shot?. There are no good Null
hypothesis here unless you happen to have a lot of knowledge about the average
human basket ball performance!, and even so, why do you care whether I am
significant different from the average?, You can fall back to the new
statistics
which is almost as good as the Bayesian approach, it but does not answer what you
should believe before I take a shot.

The Beta distribution is a popular prior for binary events, when the two parameter (α and β) are equal to 1, it is uniform. Since you my dear reader have no concept about my basket skills you assume a θ comes from a Beta(1,1) distribution, formally:

p(θ)∼Beta(1,1)

Where θ is my probability of scoring, the distribution looks like this:

Completely Uniform, a great prior when you are totally oblivious.

I take a shot and miss (z=0), the likelihood of a miss looks like this:

(if you are extra currious, you can brush up on the math behind all the binary distributions here)

Notice that:

p(z=0∣θ=0)=1, the likelihood that I always miss is 1

p(z=0∣θ=0.5)=0.5, the likelihood that I miss half the time is 0.5

p(z=0∣θ=1)=0, the likelihood that I always hit is 0, which is
obvious as I can't score all the time if I just missed.

Notice that these likelihoods and not probabilities, but how likely the data are for different values of θ, so it is twice as likely:

p(z=0∣θ=0)p(z=0∣θ=0.5)=10.5=2

That the data z=0 was generated by θ=0 compared to θ=0.5.

Bayesian Updating Math

Here is Bayes theorem for the Bernoulli distribution with a Beta prior, where
the parameter z is 1 when I score and 0 otherwise:

p(θ|z)=p(z∣θ)p(θ)p(z)

For technical reason p(z), the probability of the data, is difficult to
calculate, it is however 'just a normalization constant' because it does not
depend on θ which is my scoring probability, thus we can simply drop it
and get an unnormalized posterior:

p(θ|z)∝p(z∣θ)p(θ)

An unnormalized posterior is simply a density function that does not sum to 1,
when we plot it, it looks 'correct' except we have screwed up the numbers on
the y axis.

Visual Bayesian Updating

So now we have a 'square' prior p(θ)∼Beta(1,1) and we have a triang...
Pre-Training + Fine-Tuning Favors Deception
by Mark Xu — 2021-05-08T18:36:06.236Z

Thanks to Evan Hubinger for helpful comments and discussion.

Currently, to obtain models useful for some task X, models are pre-trained on some task Y, then fine-tuned on task X. For example, to obtain a model that can summarize articles, a large language model is first pre-trained on predicting common crawl, then fine-tuned on article summarization. Given the empirical success of this paradigm and the difficulty of obtained labeled data, I loosely expect this trend to continue.

I will argue that compared to the paradigm of training a model on X directly, training on Y then fine-tuning on X increases the chance of deceptive alignment. More specifically, I will argue that fine-tuning a deceptive model will produce a deceptive model and fine-tuning a non-deceptive model is more likely to produce a deceptive model than training from scratch.

From Does SGD Produce Deceptive Alignment?:

More specifically, we can single out three types of models that optimize for the base objective:

internally aligned models optimize for the base objective because it is equal to the model's terminal objective

corrigibly aligned models optimize for the base objective because their terminal objective is a robust pointer to a model of the base objective. Note that these are called corrigible because they are amenable to corrections of the model of the base objective (though not necessarily corrections to their pointer).

deceptively aligned models optimize for the base objective for instrumental reasons

Assume that pre-training has produced a model that optimizes for the pre-training loss and is one of the above types. The arguments are weaker if pre-training is imperfect. If the model is not one of the above types, then these arguments do not apply.

We can analyze the three cases separately:

If pre-training has produced an internally aligned model, then we are beginning fine-tuning with a mesa-optimizer that has a misaligned proxy objective. The probability of fine-tuning producing deception is controlled by how far the pre-training loss is from the fine-tuning loss and how far the current capabilities are from deception.

If pre-training has produced a corrigibly aligned model, then we begin fine-tuning with a mesa-optimizer that has a misaligned pointer. The probability of fine-tuning producing deception is controlled by how easy it is to shift the pointer and how far the current c

...
Migraine hallucinations, phenomenology, and cognition
by Richard_Kennaway — 2021-05-08T15:56:37.985Z

I have several times in my life experienced migraine hallucinations. I call them that because they look exactly like what other people report under that name.

I'll come back to those.

If I look at someone, and hold up my hand so as to block my view of their head, I do not experience looking at a headless person. I experience looking at a normal person, whose head I cannot see, because there is something else in the way.

Why is this? One can instantly talk about Bayesian estimation, prior experience, training of neural nets, constant conjunction, and so on. However, a real explanation must also account for situations in which this filling-in does not occur. One ordinary example is the pictures here. I see these as headless men, not ordinary men whose heads I cannot see.

Migraine hallucinations provide a more interesting example. If you've ever had one, you might already know what I'm going to say, but I do not know if this experience is the same for everyone.

If I superimpose the hallucination on someone's head, they seem to have no head. I don't mean that I cannot see their head (although indeed I can't), but that I seem to be looking at a headless person. If I superimpose it on a part of their head, it is as if that part does not exist. Whatever the blind spot covers, my brain does not fill it in. Whatever my hand covers, my brain does fill in, not at the level of the image (I don't confabulate an image of their face), but at some higher level. I know in both cases that they have a head. But at some level below knowing, the experience in one case is that they have no head, and in the other, that they do. My knowledge that they have a head does nothing to alter the sensation that they do not.

It is quite disconcerting to look at myself in a mirror and see half my head missing.

Those who have never had such hallucinations might try experimenting with their ordinary blind spots. I am not sure it will be the same. The brain has had more practice filling those in, and does not have to contend with the jaggies.

From this I cannot draw out much in the way of conclusions about vision and the brain, but it provides an interesting experience of the separation between two levels of abstraction. When we look at the world and see comprehensible objects in it, our brain did that before it ever came into our subjective experience. When the mechanism develops a fault, it presents...
Interview with Christine M. Korsgaard: Animal Ethics, Kantianism, Utilitarianism
by Erich_Grunewald — 2021-05-08T11:44:39.521Z
Link: https://www.erichgrunewald.com/posts/interview-with-christine-m-korsgaard-animal-ethics-kantianism-utilitarianism/

Christine M. Korsgaard was kind enough to answer a few questions of mine. Here's an excerpt:

ERICH: Many animal welfare advocates seem to be utilitarians, possibly due to the influence of Peter Singer. But you, of course, are not a utilitarian. Why is it not a convincing moral philosophy, in your view?

CHRISTINE: Because I believe that everything that is good must be good-for someone, some creature, I don’t believe it makes sense to aggregate goods across the boundaries between creatures. Of course, if you say “I can do something that’s good for Jack, or I can do something that’s good for Jack and also good for Jill,” everyone thinks that the second option is better, and that makes it look as if aggregation makes sense – the more good, the better. The problem only shows up when you have to do some subtracting in order to maximize the total. If Jack would get more pleasure from owning Jill’s convertible than Jill does, the utilitarian thinks you should take the car away from Jill and give it to Jack. I don’t think that makes things better for everyone. I think it makes it better for Jack and worse for Jill, and that’s all. It doesn’t make it better on the whole.

Of course, behind this there is a deeper problem. Utilitarians think that the value of people and animals derives from the value of the states they are capable of – pleasure and pain, satisfaction and frustration. In fact, in a way it is worse: In utilitarianism, people and animals don’t really matter at all; they are just the place where the valuable things happen. That’s why the boundaries between them do not matter. Kantians think that the value of the states derives from the value of the people and animals. In a Kantian theory, your pleasures and pains matter because you matter, you are an “end in yourself” and your pains and pleasures matter to you.
Crowdfunding Vaccine Development
by River — 2021-05-08T04:10:42.805Z

Imagine that at the beginning of the pandemic, March 2020, you were given the opportunity to buy a place in the queue to get the vaccine. Imagine knowing nothing more than you actually did in March 2020. You don’t know how long it will take to develop a vaccine, or which companies will do it, or exactly how effective it will be, or what if any side effects it will have. You don’t know what kind of pandemic rules will be in place where or for how long. How much would you pay for that place in the vaccine queue? My personal answer, given that I had not much income at the time, is $100, but figure out for yourself what your answer is.

Got a number in mind?

Yes?

Good.

Now lets compare it to what is needed. Operation Warp Speed spent $12.4 billion. A dose of Pfizer/BioNTech costs $19.50, and lets assume conservatively that 300 million doses will be given in the US. That’s $18.25 billion getting spent on vaccines for 150 million people, or $121.67 per person. Was your number above that? Do you think the average number of those 150 million people was above that? I think so. My number specific to March 2020 was only a little below that, and at other times in my life would have been much higher. I’m sure a few particularly well off people would have paid $10k or $100k to be at the very front of the queue. If I’m right that the average amount those 150 million people would pay for a place in a vaccine queue is above $121.67, then crowdfunding vaccine development, with the promise of distributing vaccines in descending order of crowdfunding contribution amount, would have provided at least as good an incentive as the world we actually have.

Why does this matter? If you think the current system will continue to work fine in the future, it doesn’t. But if, like Zvi, you think the Biden administration has just actively destroyed the ability of the current system to develop vaccines in response to future pandemics, then you should care about alternatives. If, like me, you think it was never a good idea for the government to tell N - 1 companies to wait 20 years before they can manufacture life-saving vaccines (this is what a patent is), then you should care about alternatives.
D&D.Sci May 2021: Monster Carcass Auction
by abstractapplic — 2021-05-07T19:33:19.920Z

You are an apprentice to Carver, the most successful butcher in your tiny, snow-swept village. Today, for the first time since you joined her, she is sending you to buy carcasses at the daily Auction.

(The (first-price, sealed-bid) Monster Carcass Auction began as a collective effort by local shopkeepers to divert Adventurers from trying to sell them random corpses, but has since become an integral part of the village economy, as well as the population’s main protein source.)

Carver thinks you should trust your instincts and bid however feels right. It’s an approach that’s served her well thus far – the record you’ve been compiling of her bids and subsequent sales attests to that, among other things – but you suspect a more data-driven approach would work better. And if you do well enough on this expedition, that might suffice to prove it to her.

You make sure to arrive at the very end of the event, like your boss always does; this means you’ll lose any tie-breakers – matching bids are resolved in favour of whoever bid first – but also means your rivals will have already put in their bids, so none of them will be able to change their bidding strategy to account for Carver’s absence.

The lots available are as follows:

Lot	Species	Days Since Death
#1	Yeti	0
#2	Snow Serpent	2
#3	Snow Serpent	1
#4	Winter Wolf	1
#5	Yeti	5
#6	Winter Wolf	1
#7	Snow Serpent	1
#8	Snow Serpent	5
#9	Winter Wolf	3
#10	Winter Wolf	7
#11	Winter Wolf	8
#12	Snow Serpent	8
#13	Winter Wolf	2

(As usual, this is all the information given to bidders; the original organizers took the term ‘blind auction’ a little too literally, and by the time anyone realized, the practice of hiding almost everything about the lots had become a tradition.)

You and your employer are risk-neutral, and don’t care how much or little time and effort you spend butchering. You brought 400 silver pieces. How much will you bid for each lot?

Notes:

Payments are collected in lot order; if you’re unable to pay your bid by the time a given lot comes up, you lose your claim to that lot but incur no penalty.
Your records are in no particular order, but the glacial pace of life in your village suggests there are no time trends to account for.

I’ll be posting an interactive letting you test your decision, along with an explanation of how I generated the dataset, sometime next Friday. I’m giving you a week, but the task shouldn’t take more than a few hours; use Excel, R, Python, Ou...
Why quantitative finance is so hard
by lsusr — 2021-05-07T19:29:41.933Z

This is not financial advice.

Quantitative finance (QF) is the art of using mathematics to extract money from a securities market. A security is a fungible financial asset. Securities include stocks, bonds, futures, currencies, cryptocurrencies and so on. People often use the techniques of QF to extract money from prediction markets too, particularly sports betting pools.

Expected return is future outcomes weighted by probability. A trade has edge if its expected return is positive. You should never make a trade with negative expected return. It is not enough just to use expected return. Most peoples' value functions curve downward. The marginal value of money decreases the more you have. Most people have approximately logarithmic value functions.

A logarithmic curve is approximately linear when you zoom in. Losing 1% of your net worth hurts you slightly more than earning 1% of your net worth helps you. But the difference is usually small enough to ignore. The difference between earning 99% of your net worth and losing 99% of your net worth is not ignorable.

When you gain or lose 1% of your net worth, the expected change to the logarithm of your wealth is a tiny -0.01%. When you gain or lose 99% of your net worth the expected change to the logarithm of your wealth is -400%.

log(1.01)+log(0.99)≈−0.0001log(1.99)+log(0.01)≈−4

This is called a risk premium. For every positive edge you can use the Kelly criterion to calculate a bet small enough such that the you edge exceeds your risk premium. In practice traders tend to use fractional Kelly.

Minimum transaction costs are often constant. It is not sufficient for your edge to merely exceed your risk premium. It must exceed your risk premium plus the transaction cost. Risk premium is defined as a fraction of your net worth but transaction costs are often constant. If you have lots of money then you can place larger bets while keeping your risk premium constant. This is one of the reasons hedge funds like having large war chests. Larger funds can harvest risk-adjusted returns from smaller edges.

Getting an Edge

The only free lunch in finance is diversification. If you invest in two uncorrelated assets with equal edge then your risk goes down. This is the principle behind index funds. If you know you're going to pick stocks with the skill of a monkey then you might as well maximize diversification by picking all the stock...
Life and expanding steerable consequences
by Alex Flint — 2021-05-07T18:33:39.830Z

Financial status: This is independent research. I welcome financial support to make further posts like this possible.

Epistemic status: I believe this is a helpful lens through which to view the significance of AI in a way that is not fundamentally about intelligence.

In this world, there are two types of objects: objects whose steerable consequences diminish over time, and objects whose steerable consequences expand over time.

Consider a small rock on a table. Suppose I move that rock a little to the left, and consider the ways that this action might affect the future. The rock might have been holding down some papers, and those papers might now be blown about by a gust of wind. Or someone might walk into the room and, seeing the rock being out of place, walk over and move it back. In fact the rock exerts a gravitational effect on every other object in the universe, and the tiny movement of the rock will have consequences that ripple out for the life of the universe.

But although these consequences are real, the rock cannot be used by us to produce a predictable large-scale effect on the world very far into the future — say, on the timescale of decades. The consequences of moving the rock become too unpredictable for us to reason about. Even if we are allowed to move the rock to any point in the universe, we cannot really use this power to effect any useful control over the future, at least not without involvement from humans. As we consider the causal fallout of moving the rock we quickly hit a wall of foggy uncertainty, and so in this sense rock cannot be used on its own to steer the future.

But consider now the action of introducing a living organism to the surface of Mars. Suppose that some scientists have chosen or engineered a particular kind of mold that will thrive in the environmental conditions present on Mars. Suppose that we move an object of the same size as the rock, only now the object is a mold specimen together with an initial food source, and we move it from some laboratory on Earth to the surface of Mars. Although the physical size of this initial specimen might be quite small, this action could have consequences that eventually affect the entire surface of Mars.

Furthermore, some of these consequences are quite predictable. We can predict that the mold will reproduce. We can predict that the specimen will spread outwards from its initial loc...
Zero to One: A Minimum Viable Review
by dominicq — 2021-05-07T17:45:34.903Z

This is Peter Thiel’s matrix:

In his book, Zero To One, and in this talk at SXSW, Thiel essentially explains that where we are as a society on this matrix defines how we act and what we do. Every quadrant is a religion, and each religion has a doctrine:

Indefinite pessimism: Things will get worse, but we don’t know how exactly. Best to eat, drink, and be merry.
Definite pessimism: Things will get worse, and we know how. Best to save money and prepare for the worst. Winter is coming.
Indefinite optimism: Things will get better, but we don’t know how exactly. Best to do what works now and keep options open.
Definite optimism: Things will get better, and we know how. Best to plan big projects and work on making them a reality.

The crux of the matter is the role of luck. Imagine an axis: on one end, you have someone believing that luck played no part in their success – regardless of the circumstances, they believe that they would have arrived at the same outcome. And on the other end, you have someone who believes that luck was all it was: if any of the million small variables changed, the outcome would have been drastically different.

What is luck? Baby don’t hurt me…

There’s a useful classification system for types of luck that I found in the James Austin > Marc Andreessen > Naval Ravikant pipeline, and it goes something like this:

Blind luck
Luck from hustling
Luck from preparation
Luck from your unique character

I got my current job because the company contacted me, so that was luck. But they contacted me because I had contacted them 2 years ago, and I had sent over 100 job applications at that point, refining my resume and interviewing skills. It was still luck – it’s not like I just decided I’d get that job and then it happened – but it was a different kind of luck than just being contacted by the company without any effort at all.

But you were born in the right country, in the right time, in the right family, went to the right school, and so on! Are you not forgetting about all these other types of luck?

If you take a very long step backwards and look at the universe, all that you get is a bunch of atoms that move according to the laws of physics. Whatever happens had to have happened. This is true, but is also completely useless, because luck starts having meaning only when you go many levels above the level of atoms.

People who attribute success to luck or abilit...