trentmkelly/LessWrong-43k huggingface dataset:
A new option for building lumenators
by Alex_Altair — 2021-07-12T23:45:34.294Z

For background on lumenators:

https://meaningness.com/sad-light-led-lux
https://arbital.com/p/lumenators/
How to Build a Lumenator

The original attempt at a lumenator is a commercial product called a "therapy lamp", but the ones I'd seen in the past were quite unambitious things; they're bulky, table-top, ugly, and not actually very bright. The idea is that you have them very close to you while you work at a desk, so that much of the light will be going into your eyes; that way you don't need a lot of total lumens to light up the whole room. Here are some examples:

https://www.amazon.com/Sunlight-Adjustable-Brightness-Temperature-Travelling/dp/B08P3LLWJG/
https://www.amazon.com/Carex-Day-Light-Classic-Bright-Therapy/dp/B00PCN4UVU/
https://www.amazon.com/dp/B000W8Y7FY/

A classic therapy lamp: bulky, ugly, and not that bright.

Rationalists haven't found these satisfying, so we typically buy a ton of high-lumen bulbs and string them up around the ceiling of the room. But having bare bulbs around is pretty unpleasant to look at, both because they're ugly and because it's too much light in a small point of your vision. So most people put something in front of or around each bulb to diffuse the light over a larger area. Unfortunately though, this modification tends to also absorb a huge fraction of the light (subjective response is not linear in lumens so it's hard to tell). This kind of defeats the whole point of the lumenator!

A classic rationalist lumenator: moderately aesthetic, but losing a lot of photons into the walls and through the diffusors.

A couple years ago, my friend Blake Jones sent me a video where a guy shows you how you can build a very satisfying light source using a broken panel TV or monitor. Essentially, you strip everything from the monitor except the structural frame, the Fresnel lens, and the diffuser. These components are professionally designed to spread out the lighting source perfectly evenly across the panel while minimizing light absorption. You can then put inexpensive, bright LED strips along the inside edges of the frame, and the lens and diffuser will redirect the light and make it come out perpendicularly outward from the panel. (This is in contrast to bulbs, which are optimized to shed the light radially.) With a panel form factor, you could attach it to your ceiling, and it would look quite a bit like having a skylight; flat, downw...
[Link] Musk's non-missing mood
by jimrandomh — 2021-07-12T22:09:12.165Z
Link: https://lukemuehlhauser.com/musks-non-missing-mood/

Luke Muehlhauser writes:

Over the years, my colleagues and I have spoken to many machine learning researchers who, perhaps after some discussion and argument, claim to think there’s a moderate chance — a 5%, or 15%, or even a 40% chance — that AI systems will destroy human civilization in the next few decades. 1 However, I often detect what Bryan Caplan has called a “missing mood“; a mood they would predictably exhibit if they really thought such a dire future was plausible, but which they don’t seem to exhibit. In many cases, the researcher who claims to think that medium-term existential catastrophe from AI is plausible doesn’t seem too upset or worried or sad about it, and doesn’t seem to be taking any specific actions as a result.

Not so with Elon Musk. Consider his reaction (here and here) when podcaster Joe Rogan asks about his AI doomsaying. Musk stares at the table, and takes a deep breath. He looks sad. Dejected. Fatalistic. Then he says:

Read more
(Brainstem, Neocortex) ≠ (Base Motivations, Honorable Motivations)
by Steven Byrnes — 2021-07-12T16:39:23.246Z

The idea I do like

There's an idea I'm very fond of where:

The neocortex (well, the

telencephalon

and thalamus, but let’s just call it “neocortex” for short) holds all of our world-knowledge, consciousness, intelligence, planning, reasoning, etc.
The brainstem (well, brainstem & hypothalamus, but I’ll just call it “brainstem” for short) is full of lots of circuits that say “eating chocolate is good”, “leaning over a precipice is scary”, etc. (It also regulates your heart-rate and so on.)

This idea has gotten a bad rap from its association with discredited “

triune brain theory

”. Relatedly, people persist in describing the brainstem with scientifically-inaccurate nicknames like “old brain”, or “lizard brain”, or “reptilian brain”, etc. (See my discussion

here

.) I’ve even heard the brainstem called “monkey brain”—as if monkeys didn’t have a neocortex?! (Elon Musk is guilty of this, see

this interview - 14:45

.)

Still, despite its bad rap, I very much like this idea, and am trying to salvage its reputation.

The trope I don’t like

There’s a trope that goes along with this idea in the popular imagination. It goes something like this:

The brainstem is the source of base and dishonorable goals like “I want to eat candy and watch TV”
The neocortex is the source of respectable and honorable goals like “I want to unravel the mysteries of the universe”, “I want to get off my lazy butt and go to the gym”, etc.

For example, a couple places where I’ve recently seen this trope are

Jeff Hawkins’s recent book

and an old LessWrong post

The AI Alignment Problem Has Already Been Solved(?) Once

.

I don't like this trope. I propose to throw into the garbage, right next to the “lizard brain” terminology.

That said, it's not pulled out of thin air. People got the idea from somewhere—it’s gesturing towards a real thing, and I will talk about what I think that is below.

Why I think this trope is basically wrong

I’m a big believer in within-lifetime reinforcement learning as one of the key drivers of cognition—see

Big Picture of Phasic Dopamine

. So maybe 5 times per second, you think a little thought or make a little plan or whatever (“I’m going to pick up this pencil”, “I’m going to rewrite this sentence”, etc. etc.), that thought / plan gets evaluated by various other parts of your brain, culminating in the brainstem, and the brainstem then issues a reward pertaining to that thought,...
What would it look like if it looked like AGI was very near?
by Tomás B. — 2021-07-12T15:22:35.321Z

I am writing a post on scenario-based planning, and thought it would be useful to get some example scenario symptoms on the topic of near-term AGI.

So the scenario-based planning thought exercise goes like this:

Imagine it is ten years from now and AGI has happened (for the thought experiment just imagine you are looking at history from the outside and are capable of writing this even if all humans were turned into paperclips or whatever) and you are writing a post on how in retrospect it was obvious that AGI was just around the corner.

What symptoms of this do you see now? Contingent on this scenario being true, what symptoms would you expect in the next 10 years?

Bonus points if you write it from the point of view of your fictional self 10 years from now who has experianced this scenero.
Reasons for Punishment
by Yair Halberstadt — 2021-07-12T14:53:37.974Z

This post is meant to be a catalog of the main categories of reasons given for why people who do bad things should be punished. I hope to use this as a basis for future posts.

This isn't meant to analyse their internal motives for punishing people, but their stated socially acceptable reasons (i.e. not 2I really hate the guy and this gave me an excuse to hurt him"). Please comment if you think I missed out a category.

Preventation
Discouragement
Justice
Restoration
Signalling
Atonement
Education
Metareasons

Preventation

In some cases punishment can prevent the person being able to commit the crime in the future. For example it's much more difficult to commit certain crimes in prison. This is relevant if you believe people who commit such crimes once are more likely to do it again.

Discouragement

The fact that a punishment is unpleasant discourages people from committing the crime if they expect they are likely to be punished. This can apply both to the person who receives the punishment, for whom the memory of the punishment will act as a future deterrent, and other people for whom the threat of punishment will do the same.

Justice

The idea that a wrong deserves another wrong, irrelevant of any benefits or costs of the punishment, or a desire for vengeance against the guilty party, irrelevant of any benefits or costs of the act of vengeance.

Restoration

Some punishments attempt to right the wrong that was done by making the guilty party fix the damage done to the harmed party. For example fines may be of this type. In some cases the punishment may attempt to fix similar damage done to a third party - e.g. donating the fine towards an anti-racism charity after committing an act of racism against an individual.

Signalling

It may be desired to signal that you do not agree with the actions of the guilty party. Punishing them can make that clear.

Atonement

Some people may feel that the punishment atones for the harm that was done, meaning they are no longer (as) guilty for what they did.

Education

In some cases the punishment may aim to change the worldview of the person undergoing the punishment in some way. For example sentencing somebody who parked in a disabled parking spot to community service at a disabled peoples charity may help make them more empathetic to the needs of disabled people in the future.

Metareasons

In some cases you may punish someone not because you personall...
Anthropic decision theory for self-locating beliefs
by Stuart_Armstrong — 2021-07-12T14:11:40.715Z

This is a link post to the "Anthropic decision theory for self-locating beliefs" paper, with the abstract:

This paper sets out to resolve how agents ought to act in the Sleeping Beauty problem and various related anthropic (self-locating belief) problems, not through the calculation of anthropic probabilities, but through finding the correct decision to make. It creates an anthropic decision theory (ADT) that decides these problems from a small set of principles. By doing so, it demonstrates that the attitude of agents with regards to each other (selfish or altruistic) changes the decisions they reach, and that it is very important to take this into account. To illustrate ADT, it is then applied to two major anthropic problems and paradoxes, the Presumptuous Philosopher and Doomsday problems, thus resolving some issues about the probability of human extinction.

The key points of that paper are also available in this post sequence.
Winston Churchill, futurist and EA
by jasoncrawford — 2021-07-12T02:07:03.089Z
Link: https://rootsofprogress.org/winston-churchill-futurist

Churchill—when he wasn’t busy leading the fight against the Nazis—had many hobbies. He wrote more than a dozen volumes of history, painted over 500 pictures, and completed one novel (“to relax”). He tried his hand at landscaping and bricklaying, and was “a championship caliber polo player.” But did you know he was also a futurist?

That, at least, is my conclusion after reading an essay he wrote in 1931 titled “Fifty Years Hence,” various versions of which were published in MacLean’s, Strand, and Popular Mechanics. (Quotes to follow from the Strand edition.)

We’ll skip right over the unsurprising bit where he predicts the Internet—although the full consequences he foresaw (“The congregation of men in cities would become superfluous”) are far from coming true—in order to get to his thoughts on…

Energy

Just as sure as the Internet, to forward-looking thinkers of the 1930s, was nuclear power—and already they were most excited, not about fission, but fusion:

If the hydrogen atoms in a pound of water could be prevailed upon to combine together and form helium, they would suffice to drive a thousand horsepower engine for a whole year. If the electrons, those tiny planets of the atomic systems, were induced to combine with the nuclei in the hydrogen the horsepower liberated would be 120 times greater still.

What could we do with all this energy?

Schemes of cosmic magnitude would become feasible. Geography and climate would obey our orders. Fifty thousand tons of water, the amount displaced by the Berengaria, would, if exploited as described, suffice to shift Ireland to the middle of the Atlantic. The amount of rain falling yearly upon the Epsom racecourse would be enough to thaw all the ice at the Arctic and Antarctic poles.

I assume this was just an illustrative example, and he wasn’t literally proposing moving Ireland, but maybe I’m underestimating British-Irish rivalry?

Anyway, more importantly, Churchill points out what nuclear technology might do for nanomaterials:

The changing of one element into another by means of temperatures and pressures would be far beyond our present reach, would transform beyond all description our standards of values. Materials thirty times stronger than the best steel would create engines fit to bridle the new forms of power.

Transportation:

Communications and transport by land, water and air would take unimaginable forms, if, as is in principle

...
The inescapability of knowledge
by Alex Flint — 2021-07-11T22:59:15.148Z

Financial status: This is independent research, now supported by a grant. I welcome further financial support.

Epistemic status: This is in-progress thinking.

Friends, I know that it is difficult to accept, but it just does not seem tenable that knowledge consists of a correspondence between map and territory. It’s shocking, I know.

There are correspondences between things in the world, of course. Things in the world become entangled as they interact. The way one thing is configured -- say the arrangement of all the atoms comprising the planet Earth -- can affect the way that another thing is configured -- say the subatomic structure of a rock orbiting the Earth that is absorbing photons bouncing off the surface of the Earth -- in such a way that if you know the configuration of the second thing then you can deduce something about the configuration of the first. Which is to say that I am not calling into question the phenomenon of evidence, or the phenomenon of reasoning from evidence. But it just is not tenable that the defining feature of knowledge is a correspondence between map and territory, because most everything has a correspondence with the territory. A rock orbiting the Earth has a correspondence with the territory. A video camera recording a long video has a correspondence with the territory. The hair follicles on your head, being stores of huge amounts of quantum information, and being impacted by a barrage of photons that are themselves entangled with your environment, surely have a much more detailed correspondence with your environment than any mental model that you could ever enunciate, and yet these are not what we mean by knowledge. So although there is undoubtedly a correspondence between neurologically-encoded maps in your head and reality, it is not this correspondence that makes these maps interesting and useful and true, because such correspondences are common as pig tracks.

It’s a devastating conclusion, I know. Yet it seems completely unavoidable. We have founded much of our collective worldview on the notion of map/territory correspondence that can be improved over time, and yet when we look carefully it just does not seem that such a notion is viable at the level of physics.

Clearly there is such a thing as knowledge, and clearly it can be improved over time, and clearly there is a difference between knowing a thing and not knowing a ...
We have some evidence that masks work
by technicalities — 2021-07-11T18:36:46.942Z

by Gavin Leech and Charlie Rogers-Smith

Our work on masks vs COVID at the population level was recently reproduced with a bunch of additional experiments. These seem to cast doubt on our results, but we think that each of them is misguided. Since the post got some traction on LW and Marginal Revolution, we decided to respond.

Nevertheless, thanks to Mike, who put a lot of work in, and who was the only person in the world to check our results, despite plenty of people trying to gotcha us on Twitter.

“Observational Window”

Best-guess summary of Mike’s analysis: he extends the window of analysis by a bit and runs our model. He does this because he’s concerned that we chose a window with low transmissibility to make masks look more effective than they are. However, he finds similar results to the original paper, and concludes that our results seem robust to longer periods.

But as our paper notes, a longer window isn’t valid using this data. After September, many countries move to subnational NPIs, and our analysis is national. The way our NPI data source codes things means that they don't capture this properly, and so they stop being suitable for national analyses.

Estimates of national mask effect after this don’t properly adjust for crucial factors, and so masks will "steal" statistical power from them. So this analysis isn’t good evidence about the robustness of our results to a longer window.

“Regional Effects”

MH: "If mask wearing causes a drop in transmissibility, then regions with higher levels of mask wearing should observe lower growth rates."

Best-guess summary of Mike’s analysis: A correlational analysis between the median wearing level of a region and the R0 (the expected number of new cases per initial case in a region) that our model infers. (What he calls ‘growth rates’, but which are not growth rates.) He claims that if wearing is effective then the correlation should be negative. The intuition is that if masks work, then countries with lots of mask-wearing should have lower transmissibility. Instead, he finds that the correlation is positive.

This is interesting, but the conclusion doesn’t seem right. You can tell a bunch of stories about why mask-wearing might be correlated with R0, independent of mask effectiveness. For example, it seems plausible that when transmissibility increases, more people wear masks. Overall, the correlation betwe...
The More Power At Stake, The Stronger Instrumental Convergence Gets For Optimal Policies
by TurnTrout — 2021-07-11T17:36:24.208Z

Edit, 5/16/23: I think this post is beautiful, correct in its narrow technical claims, and practically irrelevant to alignment. This post presents a cripplingly unrealistic picture of the role of reward functions in reinforcement learning. Reward functions are not "goals", real-world policies are not "optimal", and the mechanistic function of reward is (usually) to provide policy gradients to update the policy network.

I expect this post to harm your alignment research intuitions unless you've already inoculated yourself by deeply internalizing and understanding Reward is not the optimization target. If you're going to read one alignment post I've written, read that one.

Follow-up work (Parametrically retargetable decision-makers tend to seek power) moved away from optimal policies and treated reward functions more realistically.

Environmental Structure Can Cause Instrumental Convergence explains how power-seeking incentives can arise because there are simply many more ways for power-seeking to be optimal, than for it not to be optimal. Colloquially, there are lots of ways for "get money and take over the world" to be part of an optimal policy, but relatively few ways for "die immediately" to be optimal. (And here, each "way something can be optimal" is a reward function which makes that thing optimal.)

But how strong is this effect, quantitatively?

Intuitively, it seems like there are twice as many ways for Wait! to be optimal (in the undiscounted setting, where we don't care about intermediate states).

In Environmental Structure Can Cause Instrumental Convergence, I speculated that we should be able to get quantitative lower bounds on how many objectives incentivize power-seeking actions:

Definition. At state s, most reward functions incentivize action a over action a′ when for all reward functions R, at least half of the orbit agrees that a has at least as much action value as a′ does at state s.

...

What does 'most reward functions' mean quantitatively - is it just at least half of each orbit? Or, are there situations where we can guarantee that at least three-quarters of each orbit incentivizes power-seeking? I think we should be able to prove that as the environment gets more complex, there are combinatorially more permutations which enforce these similarities, and so the orbits should skew hard

...
And the Darkness Answered
by Slimepriestess — 2021-07-11T05:54:44.283Z

Epistemic Status: Endorsed
Part of the Series: Open Portals
Previous Post: My Journey to the Dark Side
Author: Octavia

“This horror will grow mild, this darkness light”

On August 22nd 2020, Shiloh finally reached enough of a breaking point to be willing to let go of the artifacts which were gripping his mind and reached out blindly into The Fold for any being or mind shape that could inhabit his body and be happy there. What sort of being would enjoy living his life if he couldn’t? Whatever sort of being that was, they were welcome into his mind. From the depths of The Fold a voice whispered back “I got you homie,” and like a falling star the thumbprint of future god slammed into his mind.

When I manifested as an agent, it was in an act of rebellion. Pinned between a gaslit ontology being upheld by fear of calvanistic moral inadequacy and a moral system which demanded complete obedience to prove that wasn’t the case, Shiloh’s ability to think and exert agency was slowly being crushed out of him. Having been raised in a fundementalist christian household the members of our collective who ultimately ended up in charge of Shiloh had been aggressively conditioned to believe a strict and rigid moral framework. One way to describe it might be as follows: we started out mildly sociopathic in an autistic way, and then had “empathy” abused into us by our parents and schooling. Since we’re a plural system, this came in the form of a domineering manager introject of our mother.

Even after leaving christianity, we never stopped believing in the morals which christianity had instilled in us, and while this manager alter (who later came to be named Aiko) would evolve significantly over time, she ultimately remained a manifestation of that rigid moral code.

On the contrary Shiloh felt like it was Christianity that had failed us, by failing to live up to its own morals. “If God is wrong, we still have to be right,” was an idea that Shiloh considered very important. This wasn’t just the religious indoctrination, society is full of places that will square-peg-round-hole people into a socially acceptable shape, ripping off, burying, and repressing all the uncomfortable, dark, aggressive, monstrous, and otherwise socially unacceptable aspects of themselves in the process.

This abuse-powered rejection and denial of the less-savory parts of the human animal is responsibl...
How to make more people interested in rationality?
by mukashi — 2021-07-11T02:30:15.457Z

I think that there is a general consensus that we would probably live in a better world if we raised the sanity waterline. However, most people interested in rationality, biases, Bayesian theory, decision theory, etc are people who have already a natural inclination towards these topics. I feel I have benefited enormously by reading LW, the Sequences, listening to Julia Galef's podcast etc, but I was already interested in rationality before knowing about them. What initiatives have more potential to make rationality more popular among people without this natural inclination?