trentmkelly/LessWrong-43k huggingface dataset:
Beware over-use of the agent model
by Alex Flint — 2021-04-25T22:19:06.132Z

This is independent research. To make it possible for me to continue writing posts like this, please consider supporting me.

Thank you to Shekinah Alegra for reviewing a draft of this essay.

Outline

A short essay intended to elucidate the boundary between the agent model as a way of seeing, and the phenomena out there in the world that we use it to see.

I argue that we emphasize the agent model as a way of seeing the real-world phenomenon of entities that exert influence over the future to such an extent that we exclude other ways of seeing this phenomenon.

I suggest that this is dangerous, not because of any particular shortcomings in the agent model, but because using a single way of seeing makes it difficult to distinguish features of the way of seeing from features of the phenomenon that we are using it to look at.

The phenomenon under investigation

Yesterday I wrote about the pitfalls of over-reliance on probability theory as a sole lens for looking at the real-world phenomena of machines that quantify their uncertainty in their beliefs. Today I want to look at a similar situation with respect to over-reliance on the agent model as a sole lens for looking at the real-world phenomena of entities that exert influence over the future. Under the agent model, the agent receives sense data from the environment, and sends actions out into the environment, but agent and environment are fundamentally separate, and this separation forms the top-level organizing principle of the model.

And what is it that we are using the agent model to see? Well let’s start at the beginning. There is something actually out there in the world. We might say that it’s a bunch of atoms bouncing around, or we might say that it’s a quantum wavefunction evolving according to the Schrodinger equation, or we might say that it is God. This post isn’t about the true nature of reality, it’s about the lenses we use to look at reality. And one of the things we see when we look out at the world is that there are certain parts of the world that seem to exert an unusual amount of influence over the future. For example, on Earth there are these eight billion parts of the world that we call humans, and each of those parts has a sub-part called a brain, and if you want to understand the overall evolution of the cosmos in this part of the world then you can get a long way just in terms of pure...
"Who I am" is an axiom. 
by dadadarren — 2021-04-25T21:59:10.566Z

I guess most of us, in moments of existential crisis, have asked the question of "Why am I me?". It is not about grammar: "because 'I' and 'me' are both first-person singular pronouns." Nor is it a matter of tautology: "of course Bill Gates is Bill Gates. duh" It's the urge to question why among all things exist, I am experiencing the world from this particular being's perspective. How come I am not a different person, or animal, or some other physical entity.

Such a question does not have any logical explanation. Because it is entirely due to subjectivity. I inherently know this human is me because I know the subjective feeling of that person. Pain can be logically reduced to nerve signals, colors by wavelength. Yet only when this human is experiencing them will I have the subjective feeling. Explaining "Who I am" is beyond the realm of logic. It is something intuitively known. An axiom.

So rational thinking should have no answers to questions such as "the chances of me being born as a human". I'm a human is a fact that can only be accepted, not explained. However, when framed in certain ways, similar questions do seem to have obvious answers. For example, "There are 7.7 billion people in the world, about 60% Asian. What's the probability of me being an Asian ?" Many would say 60%.

That's because it is uneasy to have no answer. So we often subconsciously change the question. Here "I/me" is no longer taken as the axiomatic, subjectively defined perspective center. Instead used as a shorthand for some objectively defined entity, e.g. a random human. By using an objective reinterpretation, the question falls back into the scope of logical thinking. So it can be easily answered: since 60% of all people are Asian, then a randomly selected person should have a 60% chance of being one.

Mixing the subjectively defined "I" to objectively defined entity seldomly causes any problem. Anthropic reasoning is an exception. Take the Doomsday Argument for example. It is true a randomly chosen human would be unlikely to be the very early ones. If it turns out so, the Bayesian update to doom-soon would be valid too. However, the argument uses words such as "I" or "we" as something primitively understood: the intuitive perspective center. In this case, probabilities such as "me being born as the first 5% of all human" have no answer at all. Because "who I am" is something that ca...
Would robots care about Meaning and Relating?
by Raemon — 2021-04-25T21:17:15.575Z

A few weeks ago, Vaniver gave a talk discussing meaning and meaningfulness. Vaniver had some particular thing he was trying to impart. I am not sure I got the thing he intended, but what I got was interesting. Here is my bad summary of some things I got from the talk, and some of the discussion after the talk (in particular from Alex Ray). No promises that either of them endorse this.

Epistemic status: I am not very confident this is the right frame, but it seemed at least like an interesting pointer to the right frame. "

WTF is Meaning™?

Humans seem to go around asking questions like "What makes life meaningful? What is 'The Meaning of Life?'. What is my purpose? What is the point of it all?"

What is the type-signature of a "Meaning", such that we'd recognize one if we saw it?

When asking a question like this, it's easy to get lost in a floating series of thought-nodes that don't actually connect to reality. A good rationalist habit around questions like this is to ask: "Do we understand this 'meaning' concept well enough to implement it in a robot? Could a robot find things meaningful? Is there a reason we'd want robots to find things meaningful? What sort of algorithms end up asking "what is the meaning of life?"

Here is a partial, possible answer to that question.

Imagine a StarCraft playing robot.

Compared to humans, StarCraftBot has a fairly straightforward job: win games of StarCraft. It does a task, and then it either wins, or loses, and gets a boolean signal, which it might propagate back through a complex neural net. Humans don't have this luxury – we get a confused jumble of signals that were proxies for what evolution actually cared about when it programmed us. We get hungry, or horny, or feelings of satisfaction that vaguely correlate with reproducing our genes.

StarCraftBot has a clearer sense of "what is my purpose."

Nonetheless, as StarCraftBot goes about "trying to get good at StarCraft", it has to make sense of a fairly complex world. Reality is high dimensional, even the simplified reality of the StarCraft universe. It has to make lots of choices, and there's a huge number of variables that might possibly be relevant.

It might need to invent concepts like "an economy", "the early game", "micro", "units", "enemy", "advantage/disadvantage." (disclosure: I am neither an ML researcher nor a Starcraft pro). Not only that, but it needs some way to na...
For mRNA vaccines, is (short-term) efficacy really higher after the second dose?
by Sam Marks — 2021-04-25T20:21:59.349Z

For the Pfizer and Moderna vaccines, efficacy is commonly reported as something high (like 95%) after two doses, and something lower (in the 50-80% range) after only one dose. However, I think there's good reason to doubt that this 50-80% number is capturing the right thing, and there's good reason to believe that the mRNA vaccines are ~95% effective starting 12 days after the first dose. This is important because if your immunity is as high 12 days post-first-dose as it will be 12 days post-second-dose, then any precautions that you are waiting until you are fully vaccinated to get rid of can instead be shed 3-4 weeks earlier.

(Note that getting a second dose might still cause your immunity to wane more slowly in the long term; that is, second doses might still increase long-term efficacy.)

Overall, my current best guess, with confidence ~65%, is that the mRNA vaccines are equally effective 12 days after the first dose as they are 12 days after the second dose.

Below I'll explain the cases for and against the claim that the second dose boosts immunity. In short, the phase 3 data strongly suggests that it doesn't, and real-world data from Israel strongly suggests that it does. I'm posting this as a question because I hope that someone who knows more about this, or who has seen data that I haven't seen, can help figure this out.

Why might we think that second doses don't boost efficacy?

The short answer: because this is what the data from the phase 3 trials straightforwardly says. Here are the key graphs; note the lack of apparent effect from the second dose.

Incidence of Covid cases in the Pfizer phase 3. Note that dose 2 was given on day 21. Source.

From the Moderna phase 3. Dose 2 was given on day 28. Source.

So what's up with the commonly reported 50-80% post-first-dose efficacy? The issue is that this number captures efficacy starting on the day you receive the vaccine (or sometimes 7 days later). As you can see from the graphs, is too soon for the vaccine to have any effect at all. Rather, immunity starts appearing in the data starting sharply on day 12, so what we really want to know is the efficacy in the time period between day 12 and the second dose.

(Note that since there's an incubation period between exposure and developing symptoms, the efficacy actually starts before day 12, perhaps around day 7. Shouldn't there be some noise around the day the vaccine ...
Is there a good software solution for mathematical questions?
by ChristianKl — 2021-04-25T18:03:09.594Z

I have a programming problem that I want to optimize. To optimize it I have to calculate the minimum value that's possible for one variable given a set of equations (where some but not all variables are known at runtime). Is there any good automatized system that does that job for me and that gives me a formula given my constraints?
The Fall of Rome, III: Progress Did Not Exist
by LukeOnline — 2021-04-25T05:54:44.406Z

The Malthusian Trap is a famous idea in regards to population pressure. Agricultural progress is linear, population growth is exponential. Inevitably, the population size exceeds the carrying capacity of its farmland. This results in starvation, disease and (civil) war, which reduces the population size. Afterwards, the population can grow again and the cycle repeats.

It's a relatively horrible perspective, and it makes agricultural developments very valuable. Innovate quickly enough, and you can prevent death and war.

Malthus came up with this idea in the 18th century. It led to well-intentioned attempts to spread 'modern' agricultural methods around the world - which did not work, much to the chagrin of many Europeans.

In the 20th century, the Danish economist Ester Boserup came along. She investigated methods of agricultural intensification and concluded that historically, they were terrible. We modern people are used to technological advances being awesome. Modern cars are safer, more efficient and more comfortable than cars from 50 years ago. Modern PCs are cheaper, smaller and more powerful than PCs from 20 years ago. Technology improves our lives.

But that's not true for pre-industrial agricultural tech. The most primitive agricultural method is slash-and-burn agriculture. Take a random piece of forest and burn it. The ashes will make the ground highly fertile. Throw some seeds in there, wait a summer, and crops will grow without much effort. You might be able to use the same plot of land for a couple of years, but then you'll have exhausted the soil. On to the next piece of forest!

The first farmers in an uninhabited area will be able to utilize this method. It'll take roughly thirty years for the forest to regrow, allowing farmers to repeat the cycle. But it does mean that each farmer will need a lot of land.

As the population grows, the land available to each farmer decreases. More crops need to be produced per square meter. Agricultural intensification becomes necessary. Human labor will need to replenish the soil and to optimize yields. The soil will have to be ploughed, manure needs to be spread, weeds need to be removed, perhaps the soil will have to be drained of excess water, or irrigation systems need to be built and maintained.

This is a lot of work. And humans are lazy efficient. So agricultural intensifi...
Best empirical evidence on better than SP500 investment returns?
by bluefalcon — 2021-04-25T05:13:43.613Z

I am not smart enough to be a real rationalist, and I am really lazy. But I am smart enough to understand and implement rationalist advice and I suspect y'all are a better source of advice than normal investors. What is the highest level of returns I can achieve with a buy and hold strategy of some kind, across any asset class, and what is the empirical evidence for this? Also, I am self-employed, in a field where the 99th percentile earners make 7 figures per year and I make only low six figures. So how should I assess investment in an asset vs trying to invest in my career? I am not quite risk-neutral but pretty close to it. My expenses are well below my income; I'm done saving for retirement; further investing is just a game at this point. But I don't know how to play it.
Malicious non-state actors and AI safety
by keti — 2021-04-25T03:21:31.098Z

Here, I discuss the possibility of malicious non-state actors causing catastrophic suffering or existential risk. This may be a
very significant, but neglected issue.

Consider the sort of person who becomes a mass-shooter. They're malicious.
They're willing to incur large personal costs to cause large amounts of
suffering. However, mass-shootings kill only an extremely small proportion
of people. But if they could AGI, they would have the potential to cause
vastly greater amounts of suffering. So that is what some may try to do.

It wouldn't surprise me if they'd succeed. To do so, they would need to do two things: acquire the information
necessary to create AGI, and be the first one to use it to either destroy
or take over the world. Both of these sound pretty possible to me.

A lot of artificial intelligence capability and alignment research is
public. If the information necessary to create and/or control AGI is
public, then it would be easy for a malicious actor to obtain it.

If the information is private, then a malicious non-state actor could
potentially join the organization to attain access to it. They could try
to join the company organization to gain access to it. If they act like
decent people, it may be extremely difficult to detect malicious
tendencies in them.

Even if they can't join the company or organization, they could still
potentially just steal the information. Of course, the organization could
try to protect their information. But malicious actors trying to take over
the world may act quite differently than regular cyber criminals, so it's
not clear to me that regular information security would defend against
them. For example, such an actor would potentially try physically breaking
into the places with the information as well as coercing people for it.
Both of these would normally be too dangerous for a regular cyber
criminal, so the organization might not sufficiently anticipate the
threat.

If a malicious actor can acquire the information necessary for creating
AGI, I wouldn't be surprised if they would be able to destroy or take over
the world before anyone else could.

First, the malicious actor would need a sufficiently large amount of computational resources. Computers in botnets can be rented out
extremely cheaply, and vastly more cheaply than actually purchasing the
computers. Or they could hack into massive numbers of computers on...
Estimating COVID cases & deaths in India over the coming months
by GeneSmith — 2021-04-24T21:35:21.267Z

Does anyone have a good method to estimate the number of COVID cases India is likely to experience in the next couple of months? I realize this is a hard problem but any method I can use to put bounds on how good or how bad it could be would be helpful.

I'm trying to convince some people who work in global health to send the US stockpile of unused and unapproved Astra Zeneca vaccines to the hardest hit parts of India. My naive calculation indicates sending 25,000,000 doses (basically our whole stockpile) would save at least 17,000 lives using fairly conservative assumptions. But that was based on an estimate that 20% of the total population would be infected and a third of those would be infected after the vaccines we send are injected and become effective.

The 20% is based on the rate of US infections and the 1/3rd is basically a random guess about how many infections there would be left between vaccination and the end of the pandemic.
Identity in What You Are Not
by Gordon Seidoh Worley — 2021-04-24T20:11:49.480Z

We often talk about identity as if it is what we are. Yet, strangely, most of the trappings of identity are things that are not, strictly speaking, us.

Some examples of ways people identify themselves:

I'm a rock/punk/folk/etc. music fan
I'm an artist/writer/musician/intellectual/etc.
I'm really into trains/horses/anime/etc.
My astrological sign/personality type/love language/etc. is...
My style is preppy/grungy/emo/etc.
I'm a generally happy/sad/angry/etc. person

Yet there's also a sense in which none of these things actually are us, but rather are things we do, associate with, or like. This suggests there's some confusion about what identity is, so let's explore what's up with it and maybe we can sort this out.

What is identity?

One everyday notion of identity is something like "who we are" or "how we tell each other apart". We might think of it as our personal coordinates in thingspace.

Another is that identity is the thing that makes me me and you you. If you're religious, maybe you think of this as your soul. If you're a romantic, you might call it your authentic self. Or maybe you just think of it as the self experiencing the self, the thing it is like to be you and not someone else.

Let's call this first kind of identity personal identity since it identifies the person and the second kind self identity since it's related to how you conceive of yourself.

I claim that we regularly conflate these two kinds of identity, this causes confusion, and furthermore that neither ultimately gets at what we want to mean by "identity". But before we get to all that, let's start by looking at how identity forms.

How does identity form?

When we are born we don't seem to know much about ourselves, as in we literally seem to lack the cognitive ability to form categories to construct a map of the world that includes ourselves. Over time, though, this changes, sometimes slowly and sometimes rapidly. By the time we are toddlers we have something like a clear identity that people can use to tell us apart from others and that we can use to distinguish ourselves.

As toddlers, though, we're not very good at making map-territory distinctions due to limitations of our present level of cognitive development. Thus we easily mix up personal identity and self identity and think they are one and the same. Some classic examples of things I remember toddlers saying that illustrate this type of confusi...
Naturalism and AI alignment
by Michele Campolo — 2021-04-24T16:16:16.389Z

Edit 1: Allergic to naturalism and other realist positions? You can still benefit from reading this, by considering ideal observer theory instead. I am claiming that something like an ideal-observer AI can be built, and that there is a non-trivial chance such an agent becomes aligned (after being given enough knowledge about the physical world).
Edit 2: The best objection I've received so far is "I am sceptical of this approach"; others find it at least plausible. If you know why this approach does not work, please leave a comment. If you think the orthogonality thesis renders all the realism-inspired approaches to alignment useless, please explain why/how.

Posted also on the Effective Altruism Forum.

Introduction

The problem of creating machines that behave ethically is inherently multidisciplinary; hence, it is often attacked with ideas coming from different fields, including subfields of ethics such as metaethics.

This post is made of two parts and its main focus is AI alignment, but it might be of interest also to philosophers. Part I shows how AI can contribute to metaethics, through empirical experiments that might settle some philosophical debates. Part II explains why the metaethical position known as naturalism could be crucial to the design of aligned AI.

The second part builds on ideas developed in the first one, so AI researchers might not want to entirely skip the first part.

Part I

Metaethics and empirical evidence

Readers from the AI field might not be familiar with metaethics, so I will briefly introduce the core concepts here. When two philosophers are discussing whether kids ought not to jump on their parents’ bed, the debate usually fits the domain of applied or normative ethics. Metaethics is instead concerned with second-order questions regarding morality, such as: “When people claim stealing is wrong, are they conveying their feelings, or are they stating something they believe to be true?” and “Is morality subjective or objective?”

The problem with these questions is that they risk being practically unsolvable. If the claim “morality is subjective” doesn’t imply any empirical observation which would be less likely under the claim “morality is objective”, and vice versa, the entire discussion reduces to a collection of irrefutable statements.

Some philosophers, such as Baras (2020), are actually comfortable with this situation: they see metaethics...