trentmkelly/LessWrong-43k huggingface dataset:
Optimizers: To Define or not to Define
by J Bostock — 2021-05-16T19:55:35.735Z

Introduction

The definition of "optimizer" has been a sort of rationalist pastime for a many years now. Many different ones have been suggested, and many have fallen down. Most have been in terms of the behaviour of the system in question, which is probably not what we really want, as we might want to tell if something is an optimizer without having to observe it in action, developing an optimizer detector so-to-speak.

Obviously as LWers we are not searching for some "true essence" of the meaning behind the word, we are really searching through different descriptors which are good at slicing through thingspace, and identifying what we already call optimizers. If we pick sensible descriptors, they ought to correlate with one another, and if we do enough of this, we might find that some cause one another. Finding the root causal nodes that make some things optimizers would be very useful. That would be about as close to a definition as we could ask for.

A New Description

I think that a key part of optimization is observation of the optimized domain, which from an outside perspective means there is mutual information between the optimizer and the optimized domain. The other key component is that the optimizer acts on the external world in a way which depends on this mutual information. To really be an optimizer it should also fit existing criteria like compressing possible futures. We can formalize this mathematically as such:

We have an optimizing

a

gent A and an e

x

ternal optimization domain X. To make most of the maths work later we ought to think of them as distributions over optimizers and domains.

Observation is a function which depends on X and maps one distribution of A to another: (In principle it can be defined in terms of individual A and X values, but the function can be extended to probability distributions over A and X the same way other functions can)

ObsX:Aold→Anew

It also ought to have the property of increasing mutual information between A and X, in order to be a decent learning process.

1: I(A;X)<I(ObsX(A);X)

The next part is an output (action) function:

ActA:Xold→Xnew

Which should have the property of compressing domain-space:

2: H(X)>H(ActA(X))

And we should also add another condition, that the more an optimizer learns, the better it is at optimizing. First let us define an "improvement func...
Against Against Boredom
by Logan Zoellner — 2021-05-16T18:19:59.909Z

I'm trying to clarify some feelings I had after reading the post Utopic Nightmares. Specifically, this bit:

But in a future world where advancing technology’s returns on the human condition stop compensating for a state less than perfect hedonism, we can imagine editing boredom out of our lives

I would like to describe a toy moral theory that--while not exactly what I believe--gets at why I would consider "eliminating boredom" morally objectionable.

Experience maximizing hedonism

Consider an agent that perceives external reality through a set of sensors S0,S1,...Sn . It uses these sensors to build a model of external reality and estimate it's position in that reality at a point in time as a state Mt. It also has a number of actions At,i available to it at any given time.

The agent estimates the number of reachable future states V(t)=E(|{Mis.t∃pathMt−>..−>Mi|}) and "chooses" its actions so as to maximize the value of R(t) for some future time t. Obviously if the agent is dead, it cannot perceive or affect its future state, so it estimates V(dead)=1.

Internally the agent is running some kind of hill-climbing algorithm, so it experiences a reward after choosing an action Ai,t at time t of the form R(t)=V(t)−V(t−1). In this way, the agent experiences pleasure when it takes actions that increase V(t) and pain when it takes actions that decrease V(t) and over time the agent learns to take actions that maximize V(t).

Infinite Boredom

Now consider the infinite boredom of Utopic Nightmares. In this case the agent reaches a local maximum for V(t) and R(t) is now constant (and equal to zero). But of course there is no reason why R(t) need be zero when V(t) is constant. There's no reason why we couldn't have instead used R(t)c=V(t)−V(t−1)+c for our hill-climb. The agent would experience endless bliss (for positive values of c) or endless suffering (for negative values of c). Human experience suggests that our personal setting for c is in fact significantly negative (as humans suffer greatly from boredom).

What might be the value of using a large negative value for c? Consider, perhaps, the case of Simulated Annealing where the algor...
Why Glen Weyl is a technocrat
by ChristianKl — 2021-05-16T18:01:16.945Z

In 2019 Glen Weyl wrote the article Why I Am Not A Technocrat where he asserted that while he isn't a technocrat the rationality community is technocratic. I think he suffers from a case of projection because he fails to understand the problem of technocracy.

Weyl presents the challenge of one between the humanities on the one side and people believing in technology on the side. He sees himself on the side of the humanities and thinks that doesn't put in on the technocrat side.

In a reply to Scott,

Weyl writes

about him finding it surprising that Scott sees the technocratic response to COVID-19 a positive example. When talking about why it surprises him, he writes about him co-developing the Harvard

Covid-Roadmap

that was a lot worse then Audrey Tang's response in Taiwan.

Scott wrote about how the rationalist community with it's way of thinking did well on COVID-19 not that the technocrats in power, which partly includes Weyl, got things right. It seems that Weyl understands that his response was awful, but fails to understand why he's the kind of person who reacted with an awful technocratic response that is different from the rationalist response of our community.

What did the Covid-Roadmap get wrong? In the key points of the roadmap to tackle Covid it doesn't include any of the phrases:

science, experiment, trial, probability, uncertainty, knowledge, education, ventilation, mask, drug, vaccine, cost-benefit analysis, FDA, QALY, utility, work-from-home, distancing, bureaucracy

The roadmap calls for upholding bureaucracy by saying:

programs established by states and administered by local health authorities—can and should be fully aligned with civil liberties, due process, non-discrimination, data and health privacy protections, and health ethics.

From the Harvard perspective health ethics likely means the bioethics people who are the reason we don't have challenge trials. Managing to both avoid speaking about the need for science will at the same time advocating regulation that make it harder to do the necessary science is quite an accomplishment for Harvard ethicists.

As far as developing additional technology, the only new technology that they think should be developed is track-and-trace apps. They argue that it's important that any track-and-trace apps follows all the bureaucratic norms they can think of while considering adoption of the app unimportant eno...
Power Outage Chances
by jefftk — 2021-05-16T16:40:02.389Z

On yesterday's post about generators, Chris asked whether power
outages are worth planning for. That's a good question: very roughly,
you can think of whether it is worth spending

$X to
handle an emergency with probability

P as whether you'd
be willing to spend

$X/P if the emergency were certain.
So: what's

P?

Let's start by looking at national blackout statistics. The Energy
Information Administration has power companies report data on
many things, including reliability starting in 2013. Taking
"SAIDI With MED" (System Average Interruption Duration Index including
Major Event Days), weighting by "Number of Customers" (lines of
service), and excluding utilities that didn't report these values, I
see (sheet):

In some ways this is an overestimate of what I care about: it includes
both short and local outages that, while annoying, are not much of an
issue. I am much more concerned about longer
regional outages: if everyone in the Boston Area was suddenly out
of power at the same time there wouldn't be anywhere near enough
generators / out-of-area hotel rooms / etc to go around.

On the other hand, it's also an underestimate: the worst power outages
have been from major storms—Hurricanes (Isabel, Ike,
Sandy) and
winter storms (CA,
TX)—and
accidents—Northeast,
Southwest.
The 2013 - 2019 period looks like it had fewer of these than typical?
Additionally, with the grid becoming more interconnected and more
computerized the risk of a large regional outage seems likely larger?

How likely would you estimate a >3d outage is for the average
person in the US over the next 10y?

Comment via: facebook
Christenson's "Epistemology of Disagreement: The Good News"
by Aidan_Kierans — 2021-05-16T03:58:58.062Z

Epistemic status: I’m moderately confident in the positions I endorse here, and this series is the product of several months’ research, but it only skims the surface of the literature on these questions.

Bookkeeping: This post is part of a short series reviewing and commenting on papers in epistemology and philosophy of science concerning research norms. You can find the other posts in this series here and here. The sources used for these posts were suggested to me by my professor as a somewhat representative sample of the work on these subjects. The summaries and views expressed are my own unless otherwise stated. I have read the papers in the bibliography; I have not read the papers in the “See Also” section, but they are relevant to the discussion, and I encourage anyone interested to give them a shot. Many of the papers mentioned in this series are publicly available on philpapers.org.

Introduction

One fairly common situation that may present opportunities for [epistemic] improvement is that of discovering that another person’s belief on a given topic differs markedly from one’s own. And it is this sort of opportunity that I want to concentrate on here. How should I react when I discover that my friend and I have very different beliefs on some topic? Thinking about belief in a quantitative or graded way, the question concerns cases in which my friend and I have very different degrees of confidence in some proposition P. Should my discovery of her differing degree of belief in P lead me to revise my own confidence in P? (Christenson, pp. 187-188).

David Christenson argues in "Epistemology of Disagreement: The Good News" that when you disagree with your “epistemic peer(s)” about an issue, it's rational to move your belief somewhat toward what your peer(s) believe unless you have reasons to doubt their judgement(s) besides your confidence in your own. This is fairly uncontroversial if you have reason to believe your peer’s judgement is more reliable for whatever reason, but more disputed when it comes to disagreements where you and your peer are equally likely to make the correct judgement. By Christenson’s account, this means it would be epistemically rational to endorse controversial positions in fields like philosophy (which have relatively little consensus) less often or with less confidence than people currently do.

In the literature on disagreement, an epi...
Thinking About Generators
by jefftk — 2021-05-16T02:40:03.535Z

Every so often I think about what we would do in an extended power
outage. Generally this comes up when a crisis is in the news, so not
a good time to buy things. During the

February
Texas Storm I set myself a reminder for three months later to
think about generators. So: here I am.

We currently have two sources of standard 120V power in a blackout:

Battery: 1 kWh of lead-acid.

Solar: 2 kW of solar

(2 kW is the capacity of the

SPS;
the panels themselves are rated for a total of 5 kW.)

They do have downsides, though. The battery has a limited capacity,
and the solar requires the sun to be shining, the panels to be clear,
and the sun to be in the right part of the sky. They complement each
other somewhat, using the battery to time-shift, but there are many
blackout situations (ex: winter storm) where once the battery is
depleted the solar wouldn't give us anything.

A fossil fuel generator is potentially a good fit here. While it
doesn't have the long-term properties of solar, fuel is cheap enough
that you can have a capacity far larger than you would want to spend
on batteries.

For example, a generator can turn a five gallon can of gas into ~15
kWh. That's already much more than my battery, and each additional
can of gas is far cheaper than a large battery.

On the other hand, gas is a pain. It turns to varnish in a
few months and (especially if it has some ethanol) it pulls in water
from the air, so even if you're using fuel stabilizer you want to
rotate it. It's not safe or legal to store in your basement, and the
temperature fluctuations of storing it outdoors aren't great for it
either. Storing gas in your car's gas tank could be good, since it
will get automatically rotated, but most cars now have anti-siphon
filters.

Propane seems more attractive: it doesn't go bad, you can store it
outdoors if you keep it out of direct sun, and while it's less
efficient than gasoline a 20lb propane tank still gets you ~10 kWh.

You can get inverter generators that can take both gasoline and
propane (ex: $535),
which seems like a good combination? Thoughts?

Comment via: facebook
On "Epistemic Trespassing" and Intellectual Modesty
by Aidan_Kierans — 2021-05-16T00:40:56.087Z

Epistemic status: I’m moderately confident in the positions I endorse here, and this series is the product of several months’ research, but it only skims the surface of the literature on these questions.

Bookkeeping: This post is part of a short series reviewing and commenting on papers in epistemology and philosophy of science concerning research norms. You can find the other posts in this series here and here. The sources used for these posts were suggested to me by my professor as a somewhat representative sample of the work on these subjects. The summaries and views expressed are my own unless otherwise stated. I have read the papers in the bibliography; I have not read the papers in the “See Also” section, but they are relevant to the discussion, and I encourage anyone interested to give them a shot. Many of the papers mentioned in this series are publicly available on philpapers.org.

Introduction

Linus Pauling, the brilliant chemist and energetic proponent of peace, won two Nobel Prizes—one for his work in chemistry, and another for his activism against atomic weapons. Later, Pauling asserted that mega-doses of vitamin C could effectively treat diseases such as cancer and cure ailments like the common cold. Pauling was roundly dismissed as a crackpot by the medical establishment after researchers ran studies and concluded that high-dose vitamin C therapies did not have the touted health effects. Pauling accused the establishment of fraud and careless science. This trespasser did not want to be moved aside by the real experts. (Ballantyne, p. 367).

Experts drift over a highly-visible boundary line and into a domain where they lack either the relevant evidence or the skills to interpret the evidence well. But they keep talking nonetheless. Experts on a public stage are cast in the role of the ‘public intellectual’ or ‘celebrity academic’. … So what do you have to say about philosophy, Neil deGrasse Tyson? And what about arguments for the existence of God, Professor Dawkins?. (Ballantyne, p. 369).

The above cases outline Nathan Ballantyne’s meaning when he refers to “epistemic trespassing”; Linus Pauling doesn’t understand medicine, and public scientists like Neil deGrasse Tyson and Richard Dawkins tend to misunderstand what philosophy is and/or how to use it (according to philosophers). But epistemic trespassing isn’t a practice limited to public figure...
Research Motivations and "The Division of Cognitive Labor"
by Aidan_Kierans — 2021-05-16T00:00:41.106Z

Epistemic status: I’m moderately confident in the positions I endorse here, and this series is the product of several months’ research, but it only skims the surface of the literature on these questions.

Bookkeeping: This post is part of a short series reviewing and commenting on papers in epistemology and philosophy of science concerning research norms. You can find the other posts in this series here and here. The sources used for these posts were suggested to me by my professor as a somewhat representative sample of the work on these subjects. The summaries and views expressed are my own unless otherwise stated. I have read the papers in the bibliography; I have not read the papers in the “See Also” section, but they are relevant to the discussion, and I encourage anyone interested to give them a shot. Many of the papers mentioned in this series are publicly available on philpapers.org.

Introduction

By 1804, the phlogiston theory was dead. Thirty years earlier, the same theory had been favored by almost every chemist in Europe. If the chemical revolution was resolved on the basis of reason and evidence, then it appears that there must have been some moment between 1774 and 1804 when the balance of evidence finally tipped against the phlogiston theory and in favor of Antoine-Laurent Lavoisier's "new chemistry."

Imagine that the objective degree of confirmation of the phlogiston theory just prior to noon on April 23, 1787, was 0.51, that of the new chemistry 0.49. At noon, Lavoisier performed an important experiment, and the degrees of confirmation shifted to 0.49 and 0.51, respectively. Allowing for a time lag in the dissemination of the critical information, we can envisage that there was a relatively short interval after noon on April 23, 1787, before which all rational chemists were phlogistonians, and after which all were followers of Lavoisier. (Kitcher, p. 5).

Throughout "The Division of Cognitive Labor," philosopher Philip Kitcher envisions scenarios like the above in which the evidence for some theory A begins to slightly outweigh that for some competing theory B (after a period in which B's evidence outweighed A's). Kitcher contends that while any given scientist would be rational to believe A over B at this point (and investigate narrower hypotheses on the basis of that belief), if we were to allocate the research efforts for the community...
Suppose $1 billion is given to AI Safety. How should it be spent?
by hunterglenn — 2021-05-15T23:24:26.927Z

What are the current bottlenecks in AI Safety progress, and if they were solved with money, what would be the next bottlenecks?

Do current researchers need more money? Do we need to catalyze the creation of more researchers? Do we need to alter public opinion? Something else?

If you had $1 billion to spend on AI Safety, what would you do with it all, from start to finish?
[Event] Weekly Alignment Research Coffee Time (05/17)
by adamShimi — 2021-05-15T22:07:02.339Z

Just like every Monday now, researchers in AI Alignment are invited for a coffee time, to talk about their research and what they're into.

Here is the link.

And here is the everytimezone time.

Note that the link to the walled garden now only works for AF members. Anyone who wants to come but isn't an AF member needs to go by me. I'll broadly apply the following criteria for admission:

If working in a AI Alignment lab or funded for independent research, automatic admission
If recommended by AF member, automatic admission
Otherwise, to my discretion

I prefer to not allow people who might have been interesting but who I'm not sure will not derail the conversation, because this is supposed to be the place where AI Alignment researchers can talk about their current research without having to explain everything.

See you then!
The Purpose of Purpose
by Gordon Seidoh Worley — 2021-05-15T21:00:20.559Z

I often make reference to the category I call variously purpose, care, concern, meaning, norms, or telos (I'll just use "purpose" from here on out to point to it).

I've not really explained what I mean by it in great detail.
That's a problem because it's the ground of reasoning.

It's the thing that bridges the gap created by the problem of the criterion.

It's also intimately tied up with what we value, since depending on what you point at with "value" it's either that purpose underlies values or the value assessment mechanism creates purpose.

I think understanding values is essential to addressing AI alignment.

So I'm writing this post both so I can reference it and so I can work out more precisely what I mean.

So what is purpose?

Often a thing we are trying to achieve.

That's not quite it, though, because that doesn't distinguish purpose from related concepts like desires and goals.
Also doesn't necessarily have to have an object to be achieved.
For example, I might have the purpose of compassion, and that purpose has an existence beyond specific instances of having compassion for a particular being.

Desires

I think what people usually mean by desire, wants, needs, etc. is something less abstract and more concrete than purpose.
Desires might be manifestations of purpose towards specific ends.
For example, perhaps I desire cake, but my purpose is to stay alive, which instrumentally requires consuming calories so metabolic processes can serve the purpose of keeping me alive.

Goals

Sometimes we use "purpose" as a synonym for "goal".
But I think there's a meaningfully different category we can point to with "goal" that we should differentiate from the one I'm pointing to with "purpose".
A goal typically can be achieved or attained and held on to.
There's something that can be measured and if it satisfies some criterion then the goal is met.

Put another way, missions can be accomplished.

A purpose may not have an end.
For example, I can just keep on caring about staying alive so long as I'm alive, but at some point I'll satisfy (or not) my goal of living for 100 years.

The other terms in the category also help point the way.

Care/concern

Heidegger talks about what I'm calling "purpose" here as Sorge.
"Care" or "Concern" are probably the best translations, although "Sorge" is cognate with English "sorrow" and "worry" is also a reasonable translation, so it contains this extra nuance of los

...