trentmkelly/LessWrong-43k huggingface dataset:
AI as a Civilizational Risk Part 5/6: Relationship between C-risk and X-risk 
by PashaKamyshev — 2022-11-03T02:19:46.847Z

The general model presented in the previous parts suggests that the US and potentially other parts of the world will suffer some form of civilizational collapse, potentially as early as 10 years from now. In the absence of decisive action to limit the spread of narrow behavioral modification AIs, such collapse is inevitable. Social media optimizational pressure points only towards the destruction of social cohesion. The collapse of civilization is a big deal, however, this isn't the same as an even more serious subject of existential risks.

I want to discuss the relationship between civilization and existential risk, specifically the P(x-risk | unaligned AGI) by 2070. I would argue that this probability is higher than 35%. The exact quantity depends significantly on the definition of "reduction in human potential."

Foom Speed

Understanding the pathway that civilization takes towards AGI is essential in predicting the most likely safety research that might or might not happen by default. It is also essential to understand the public perception of safety. Suppose my theory of slow civilizational decline is correct. In that case, sooner or later, people need to start connecting the overuse of behavioral modification, persuasion, and scam AIs as one of the core causes of civilizational collapse.

The speed with which an AGI could improve itself is an interesting question. People have described models of "fast foom," such as foom or a rapid, second-level improvement. In this scenario, it is plausible to go from something more intelligent than humans in programming to something significantly more intelligent than humans or all other optimization in the world combined.

I tend to doubt this view. I suspect that two factors will limit Foom's speed. One is the quickness with which AI can acquire real-world feedback. If real-world feedback is vital for self-improvement, this could certainly be a human timeline improvement over weeks. The other question worth considering is: "how much of previous AI improvement was bottlenecked by humans in the first place?". Suppose we are in a situation where humans try to improve AIs by running very long and complex algorithms and then coming back several days to check on them. Then taking humans out of that loop will not necess...
Is there a good way to award a fixed prize in a prediction contest?
by jchan — 2022-11-02T21:37:45.111Z

Let's say we're having a prediction contest where we want people to pre-register their probability estimates for the outcomes of a set of discrete events (e.g. the candidates for various races in an upcoming election). We have a fixed sum of, say, $100 that we must award to the winner(s) in some way. We are not allowed to award any more or less than that. How can we determine the award in such a way as to incentivize people to honestly report their estimates?

The naïve approach would be to do the following:

Score everyone's predictions according to some proper scoring rule

Award the $100 to whoever has the highest score.

However, I'm pretty sure this doesn't work. A proper scoring rule incentivizes honest estimates if the goal is to maximize your total score; but in this case, maximizing your score is not the goal. Under this system, I'm incentivized to make estimates that may have a suboptimal expected score as long as the expectation is high-variance, because that will increase the probability that I get the top spot.

Can you figure out a system that avoids this problem?

(Distributing the prize randomly or in partial shares is allowed. However, destroying money or creating extra money is not.)
"Are Experiments Possible?" Seeds of Science call for reviewers
by rogersbacon — 2022-11-02T20:05:17.334Z

Are Experiments Possible?

By [redacted]

Abstract: Randomized controlled experiments are a cornerstone of many fields, especially when trying to establish a cause-effect relation. Conversely, the quality of evidence arising from natural experiments and observational data is often called into question, to the point that the motto no causation without manipulation was coined. This presupposes a distinction between the experimenter E and the system S being manipulated, which is lost if the system under consideration is extended to E + S, to include the experimenter. Seen from the outside, deliberate manipulation is just one of the many phenomena that occur naturally in E + S. The question then becomes: what are the characteristics that a (closed) dynamical system must possess for it to give rise to natural randomized experiments? Are randomized experiments even possible e.g. in a deterministic system ruled by Hamiltonian mechanics? Here I engage with this issue in an extremely simplified setting, with the hope of setting the stage for more general work on the subject.

- -

Seeds of Science is a new journal (funded through Scott Alexander's

ACX grants program

) that publishes speculative or non-traditional articles on diverse scientific topics. Peer review is conducted through community-based voting and commenting by a diverse network of reviewers (or "gardeners" as we call them).

We just sent out an article for review - "Are Experiments Possible?" - that may be of interest to those in the LW community with expertise in physics so I wanted to see if anyone would be interested in joining us a gardener and reviewing the article. It is free to join and anyone is welcome (we currently have gardeners from all levels of academia and outside of it). Participation is entirely voluntary - we send you submitted articles and you can choose to vote/comment or abstain without notification (so it's no worries if you don't plan on reviewing very often but just want to take a look here and there at what kinds of articles people are submitting). Another unique feature of the journal is that comments are published along with the article after the main text.

To register, you can fill out this google form. From there, it's pretty self-explanatory - I will add you to the mailing list and send you an email that includes the manuscript, our publication criteria, and a simple review...
Humans do acausal coordination all the time
by Adam Jermyn — 2022-11-02T14:40:39.730Z

I used to think that acausal coordination was a weird thing that AI’s might do in the future, but that they certainly wouldn’t learn from looking at human behavior. I don’t believe that anymore, and think there are lots of examples of acausal coordination in everyday life.

Examples people do

Voting

The political science argument against voting goes:

The probability that my vote tilts the election is tiny, so the expected value to me is tiny, so it’s not worth my time to vote.

And the standard rebuttal to this is:

If everyone thought that way democracy would fall apart, so I should vote.

More precisely, I expect that the people who justify voting on these grounds are doing the following reasoning:

I’m like the people who share my views in my tendency to vote. That means that there is a correlation between my decision to vote (or not) and the tendency of other people in my coalition to vote (or not). Lots of people who share my views vote, enough to make our collective vote worth my while. So I should vote, so that we all vote and we all win.

This is an example of acausal coordination! The pro-voting position amounts to reasoning about what other actors with correlated thought processes will do and picking the option which, if each actor does the same reasoning and comes to the same conclusion, leads to a better outcome.

Recycling/Carbon Footprint Reduction

The usual argument against recycling/reducing your personal carbon footprint goes:

I only have control over my environmental impacts. The effect of choosing to reduce these is tiny, so there’s no point in bearing even small costs to do it.

And the standard rebuttal is:

If everyone decided to reduce their footprint/recycle/etc. we’d have the problem(s) solved.

Again, the rebuttal is fundamentally an argument about how to acausally coordinate with other people to achieve a collective goal. Whether or not I recycle is logically connected to whether or not other people who share my reasoning recycle. There are lots of those people, which makes me want to recycle so that they recycle so that we collectively help the environment a significant amount.

Dieting

Why do people feel (psychologically) bad when they go over the limits on their diet? I don’t think it’s because they screwed up once, I think it’s because they view their commitment to a diet as a coordination effort between their present and future selves. Specifically, the ...
Far-UVC Light Update: No, LEDs are not around the corner (tweetstorm)
by Davidmanheim — 2022-11-02T12:57:23.445Z
Link: https://twitter.com/davidmanheim/status/1587757208120557568

I wrote a tweetstorm on why 222nm LEDs are not around the corner, and given that there has been some discussion related to this on Lesswrong, I thought it was worth reposting here.
People interested in reducing biorisk seem to be super excited about 222nm light to kill pathogens. I’m also really excited - but it’s (unfortunately) probably a decade or more away from widespread usage. Let me explain.

Before I begin, caveat lector: I’m not an expert in this area, and this is just the outcome of my initial review and outreach to experts. And I’d be thrilled for someone to convince me I’m too pessimistic. But I see two and a half problems.

First, to deploy safe 222nm lights, we need safety trials. These will take time. This isn’t just about regulatory approval - we can’t put these in place without understanding a number of unclear safety issues, especially for about higher output / stronger 222nm lights.

We can and should accelerate the research, but trials and regulatory approval are both slow. We don’t know about impacts of daily exposure over the long term, or on small children, etc. This will take time - and while we wait, we run into a second problem; the Far-UVC lamps.

Current lamps are KrCl “excimer” lamps, which are only a few percent efficient - and so to put out much Far-UVC light, they get very hot.

https://link.springer.com/article/10.1134/1.1448635

This pretty severely limits their use, and means we need many of them for even moderately large spaces.

They also emit a somewhat broad spectrum - part of which needs to be filtered out to be safe -

https://pubmed.ncbi.nlm.nih.gov/33465817/

- further reducing efficiency. Low efficiency, very hot lamps all over the place doesn’t sound so feasible.

So people seem skeptical that we can cover large areas with these lamps. The obvious next step, then, is to get a better light source. Instead of excimer lamps, we could use LEDs! Except, of course, that we don’t currently have LEDs that output 222nm light.

(That’s not quite true - there are some research labs that have made prototypes, but they are even less efficient than Excimer lamps, so they aren’t commercially available or anywhere near commercially viable yet, as I’ll explain.)

But first, some physics!
The wavelength of light emitted by an LED is a material property of the semiconductor used. Each semiconductor has a band-gap which corresp...
Housing and Transit Thoughts #1
by Zvi — 2022-11-02T12:10:00.575Z

Housing is the Balsa-style area where there has been the most progress lately. There seems to be growing consensus that it is vital we build more housing in places that people want to live. California is making serious attempts to force localities to do exactly that, here in New York we are making at least some progress, and across the country the YIMBY movement is making progress.

The bulk of this post is a roundup of the housing (and related transit) stories about such struggles over the past month. Going forward, it seems far more helpful to have this story and its links and other resources stored and told in large chunks, and to give those not interested a way to skip it.

This post is not intended to be my in depth exploration of Balsa’s future agenda for housing. However, I do think it makes sense to first take this opportunity to flesh out what I said about this in the FAQ, and outline what I am thinking in broad terms.

On Getting Houses Built Where People Want to Live

This is what I wrote in the Balsa FAQ:

As a concrete example, the most encouraging state level win recently has been the YIMBY/housing legislation coming out of California. It is not a coincidence, in Zvi’s opinion, that this is happening in America’s largest state, where the state is almost its own federal level (California would be one of the world’s top 10 economies on its own, etc) to take on the locals, where everyone enjoys the benefits of housing built elsewhere. If California was split into several regional states things would be a lot harder.

There are constitutional barriers that complicate any Federal effort to get more housing built where people want to live, but everyone getting to enjoy the benefits of everyone else building all at once gives us hope that turning YIMBY into a Federal cause could make things much easier.

Several people pushed back a bit on this, pointing out that California is the biggest state but that it also has the most severe housing crisis, so we don’t need ‘big state’ as an explanation for why things are happening there.

I agree that this has also been a contributing factor.

I’d also continue to emphasize that local and state work on housing is both vital and great, and seeing real results. This even shows some signs of the Cascade of Hope effect, where YIMBY success emboldens people to do more YIMBY and also to think perhaps other achiev...
Mind is uncountable
by Filip Sondej — 2022-11-02T11:51:52.050Z

I'd like to share a simple thought experiment that has some pretty strange implications. It seems to show that the world cannot always be sharply divided into separate experiences.

We can make the analogy, to how the world cannot always be divided into separate organisms. Usually it can, but there are cases, like conjoined twins, where someone would say it's one organism, someone else that they are two organisms, and someone else that it's something in between.

The experiment lies on two postulates:

P1. Mind-body continuity

It's the idea that tiny physical changes should correspond to tiny mental changes. And that should work for arbitrarily tiny ones.[1] So you couldn't completely change the content of your experience, just by moving one atom in one of your neurons.

An infinitesimal physical change, will cause an infinitesimal change of that system's behavior. So if you rejected mind-body continuity, you'd have to admit that some non-infinitesimal mental change can have an infinitesimal (practically zero) effect on the physical world. So basically you'd be forced into near epiphenomenalism.

P2. Splitting the brain in half creates two separate experiences

This is illustrated nicely by the experiments with split brain patients (like this one). For example, if you had a split brain and closed your eyes, you couldn't tell if your right hand and left hand are touching the same surface. Those two experiences are separate[2] - they aren't unified into one, so you cannot compare them.

(It’s not relevant here if they should be treated as two persons. I only claim that they are two experiences.)

Experiment

Imagine that you undergo a brain-splitting surgery. You are anesthetized so you don't feel pain, but in this experiment, you're fully awake. And this surgery is very slow - they cut only one nerve fiber at a time.

At the beginning, you have one unified experience, but at the end you have two. So one possibility is that there was some exact point when the split occurred - some critical fiber that needed to be cut. But that would violate the mind-body continuity that we assumed. So the only alternative is that we gradually went from one to two experiences. That there was some "in-between" state.

What does it feel like?

So, how would it feel to be in such a state? Here is a real-life example of brain-conjoined twins where we may say that such "partially co...
AI Safety Needs Great Product Builders
by goodgravy — 2022-11-02T11:33:59.283Z

In his

AI Safety Needs Great Engineers

post, Andy Jones explains how software engineers can reduce the risks of unfriendly artificial intelligence. Even without deep ML knowledge, these developers can work effectively on the challenges involved in building and understanding large language models.

I would broaden the claim: AI safety doesn’t need only great engineers – it needs great product builders in general.

This post will describe why, list some concrete projects for a few different roles, and show how they contribute to AI going better for everyone.

Audience

This post is aimed at anyone who has been involved with building software products: web developers, product managers, designers, founders, devops, generalist software engineers, … I’ll call these “product builders”.

Non-technical roles (e.g. operations, HR, finance) do exist in many organisations focussed on AI safety, but this post isn’t aimed at them.

But I thought I would need a PhD!

In the past, most technical AI safety work was done in academia or in research labs. This is changing because – among other things – we now have concrete ideas for how to construct AI in a safer manner.

However, it’s not enough for us to merely have ideas of what to build. We need teams of people to partner with these researchers and build real systems, in order to:

Test whether they work in the real world.
Demonstrate that they have the nice safety features we’re looking for.
Gather empirical data for future research.

This strand of AI safety work looks much more like product development, which is why you – as a product builder – can have a direct impact today.

Example projects, and why they’re important

To prove there are tangible ways that product builders can contribute to AI safety, I’ll give some current examples of work we’re doing at Ought.

For software engineers

In addition to working on our user-facing app,

Elicit

, we recently open-sourced our Interactive Composition Explorer (

ICE

).

ICE is a tool to help us and others better understand

Factored Cognition

. It consists of a software framework and an interactive visualiser:

ICE visualiser

On the back-end, we’re looking for better ways to instrument the cognition “recipes” such that our framework stays out of the user’s way as much as possible, while still giving a useful trace of the reasoning process. We’re using some meta-programming, and having good C...
Why is fiber good for you?
by braces — 2022-11-02T02:04:35.579Z

Fiber is defined as the parts of plants that we can eat but not digest. We're always told to eat more of it. Why should we eat things we can't digest?

There are many papers to point to. Fiber consumption is strongly associated with less cancer, less heart disease, and less death (recent meta-analysis here). Most of these findings come from observational studies, where people report their diets to researchers. Studies in which fiber consumption is experimentally controlled tend to find fewer benefits, so people who eat more fiber may already be healthier for other reasons.

Sidestepping the causal question, we could still ask: what are the plausible mechanisms? How does fiber do this?

Below is a list of potential benefits from an authoritative-seeming (~1,000 cites) paper in the journal Nutrients. I'm struck by how long the list is. I'm still inclined to follow the conventional advice--it feels right and if something was severely harmful about fiber we'd know from the observational data. But I do wonder: with such diverse and complex impacts on the body, could there be various negative effects too?

Reasons fiber could decrease cancer

Dietary fiber (DF) is “fermented to produce short chain fatty acids” in the large intestine, “which have anti-carcinogenic properties”
Since “DF increases fecal bulking and viscosity, there is less contact time between potential carcinogens and mucosal cells”
“DF increases the binding between bile acids and carcinogens”
DF increases antioxidants
DF decreases estrogen (which could cause cancer)
Reasons fiber could decrease heart disease

“[S]oluble fibers have been shown to increase the rate of bile excretion therefore reducing serum total and LDL cholesterol”
“[S]hort chain fatty acid production [(see point 1)], specifically propionate, has been shown to inhibit cholesterol synthesis”
DF “regulate[s] energy intake [I believe through a satiety effect; you feel more full] thus enhancing weight loss or maintenance of a healthier body weight”
DF “decrease[s] pro-inflammatory cytokines such as interleukin-18 which may have an effect on plaque stability”
Reasons fiber could decrease diabetes
The authors focus on cancer and heart disease protection because these effects have been approved by the FDA. The paper also mentions this protection against diabetes:

“Dietary fibers have been show

...
Information Markets
by eva_ — 2022-11-02T01:24:11.639Z

Epistemic status: Exploratory. This post would be shorter and contain more math if I'd thought about it more first.

I don't like prediction markets, as currently described. They're similar to ordinary stock markets, which economists say are supposed to be efficient, but don't look it. People say "If you think the markets are wrong then you can bet on them and make a profit" but I don't actually expect that to be true, because markets don't only contain sincere attempts to optimise prices. They also contain schemes to extract money from others without adding information, or to cheat in forbidden ways without getting caught, and similar nonsense, and so honest trading strategies have to be not only beneficial but also inexploitable, or you'll just end up paying people to rob you. Most of this gets much worse in a prediction market, especially a market that is being used to inform decisions that people care about, and where who knows how many people have who knows how much private information behind their opaque betting strategies. I don't expect the libertarian zero-regulation "let the market blindly solve all our problems" will actually produce something anyone should trust here.

Other facts I dislike about prediction markets:

They aren't fair in any technical sense over the value of information provided.
Can't accurately collate private information known to different participants.
Implement EDT instead of LDT if followed blindly, which they don't provide any alternative to doing.
Don't legibly convey why they reached the conclusions they reached.
Pay information ransoms to people who intentionally create uncertainty in the first place, which is a perverse incentive.
Make adding information a race to the microsecond even though there's usually no strong time pressure on the part of a subsidiser / people who want the prediction.
Include rational agents who don't think others are lying, but are betting money, even though that's clearly a zero-sum game.

I think I can fix most of these, albeit at the cost of proposing something that is substantially less pure and simple as "everyone can bet about what will happen". To distinguish them, I'll call the thing I'm trying to describe an Information Market, although it fills the same niche.

Shareability levels of Information:

Public Information: Everyone has it; everyone knows everyone has it; it's basically part of the commo...