trentmkelly/LessWrong-43k huggingface dataset:
A Hypothetical Takeover Scenario Twitter Poll
by Zvi — 2023-04-24T14:00:00.789Z

I ran an experimental poll sequence on Twitter a few weeks back, on the subject of what would happen if a hostile ASI (Artificial Superintelligence) tried to take over the world and kill all humans, using only ordinary known techs.

The hope was that by exploring this, it would become more clear where people’s true objections and cruxes were. Surely, at least, one could establish that once the AI was fully an agent, fully uncontrolled, had a fully hostile goal, was definitively smarter and more capable than us and could scale, that we were obviously super dead? Or, if not, one could at least figure out what wrong intuitions were blocking that realization, and figure out what to do about that.

This post is an experiment as well. I wouldn’t say the results were a full success. I would say they were interesting enough that they’re worth writing up, if only as an illustration of how I am thinking about some of these questions and potential rhetorical strategies. If you’re not interested in that, it is safe to skip this one.

Oh No

Consider the following scenario:

There come into existence an ASI (Artificial Super-Intelligence), defined here roughly as an AI system that is generically smarter than us, has all the cognitive capacities and skills any human has displayed on the internet, can operate orders of magnitude faster than us, and that can copy itself, and which is on the internet with the ability to interact and gather resources.

That ASI decides to do something that would kill us – it wants a configuration of atoms in the universe that doesn’t involve any humans being alive. The AI does not love you, the AI does not hate you, you are composed of atoms it can use for something else and the AI’s goal is something else. Or maybe it does want everyone dead, who knows, that is also possible.

The ASI kills us.

Many people, for a variety of reasons many of which are good, do not expect step one to happen any time soon, or do not expect step 2 to happen any time soon conditional on step 1 happening.

This post is not about how likely it is that steps 1 and 2 will happen. It is about:

Q: If steps 1 and 2 do happen – oh no! – how likely is it under various scenarios that step 3 happens, that we all die?

A remarkably common answer to this question is no, we will be fine.

I don’t understand this. To me you have so obviously already lost once:

The AS

...
Credible, costly, pseudonymity 
by M. Y. Zuo — 2023-04-24T13:35:12.718Z

An interesting idea, that's stuck around in my thoughts but escaped elaboration until the recent issue of an influx of new users became prominent, is the idea of pseudonymity.

Pseudonymity is important because although a real-name system works fine for stopping most low-effort or spam-like posts, not every potential participant on LW would want to use their real names.

However, there becomes the problem that the moderators can't review an arbitrary number of new accounts due to practical constraints.

And without this review process, there would be too much low-quality content (from what I understand) flooding the front page. Which would decrease the credibility of the account that wrote it, but also slightly that of other pseudonymous accounts by association since most readers will have lingering suspicions of the possibility of someone creating multiple accounts.

So a way to resolve multiple issues at once is to charge for new pseudonymous accounts.

It ensures moderator time spent reviewing will be compensated, greatly increases the bar for attempted spamming or trolling, and increases the credibility of new pseudonymous users.

As every reader will understand, if sufficient announcements are made beforehand, that whoever behind the account spent real money on establishing it, thus they would be unlikely to behave recklessly or create multiple pseudonym accounts unless they were highly motivated.

The exact details of implementation will likely have to be more closely examined before it could be practically added.

I'm unsure if this issue has been raised before on LW, but I found the logic compelling enough to write this post anyways to see if anyone has suggestions
On Artifice and Intelligence
by  — 2023-04-24T13:26:15.617Z
Link: https://medium.com/@shlomi.sher/on-artifice-and-intelligence-f19224281bee

"GPT is tasked with imitating intelligence, in any way it can. Fed an enormous trove of data, and endowed with enormous computing power, it often produces a convincing imitation."

In this article, the author suggests some strategies that may help to answer the following question: "In order to imitate intelligence, did GPT become intelligent? Or is it imitating intelligence in some other way?"
AGI ruin mostly rests on strong claims about alignment and deployment, not about society
by Rob Bensinger — 2023-04-24T13:06:02.255Z

Dustin Moskovitz writes on Twitter:

My intuition is that MIRI's argument is almost more about sociology than computer science/security (though there is a relationship). People won't react until it is too late, they won't give up positive rewards to mitigate risk, they won't coordinate, the govt is feckless, etc.

And that's a big part of why it seems overconfident to people, bc sociology is not predictable, or at least isn't believed to be.

And Stefan Schubert writes:

I think it's good @robbensinger wrote a list of reasons he expects AGI ruin. It's well-written.

But it's notable and symptomatic that 9/10 reasons relate to the nature of AI systems and only 1/10 (discussed in less detail) to the societal response.

https://www.lesswrong.com/posts/eaDCgdkbsfGqpWazi/the-basic-reasons-i-expect-agi-ruin

Whatever one thinks the societal response will be, it seems like a key determinant of whether there'll be AGI ruin.

Imo the debate on whether AGI will lead to ruin systematically underemphasises this factor, focusing on technical issues.

It's useful to distinguish between warnings and all-things-considered predictions in this regard.

When issuing warnings, it makes sense to focus on the technology itself. Warnings aim to elicit a societal response, not predict it.

https://www.lesswrong.com/posts/gEShPto3F2aDdT3RY/sleepwalk-bias-self-defeating-predictions-and-existential

But when you actually try to predict what'll happen all-things-considered, you need to take the societal response into account in a big way

As such I think Rob's list is better as a list of reasons we ought to take AGI risk seriously, than as a list of reasons it'll lead to ruin

My reply is:

It's true that in my "top ten reasons I expect AGI ruin" list, only one of the sections is about the social response to AGI risk, and it's a short section. But the section links to some more detailed discussions (and quotes from them in a long footnote):

Four mindset disagreements behind existential risk disagreements in ML
The inordinately slow spread of good AGI conversations in ML
Inadequate Equilibria

Also, discussing the adequacy of society's response before I've discussed AGI itself at length doesn't really work, I think, because I need to argue for what kind of response is warranted before I can start arguing that humanity is putting insufficient effort into the problem.

If you think the alignment problem itself is...
For alignment, we should simultaneously use multiple theories of cognition and value
by Roman Leventov — 2023-04-24T10:37:14.757Z

This post is a follow-up to "A multi-disciplinary view on AI safety research". I elaborate on some arguments behind this view.

TL;DR: please skim section headings and bolded sentences in the text.

Computationally tractable mathematical models of alignment are bound to be biased and blind to certain aspects of human values

No single mathematical model of human values that has orders of magnitude fewer degrees of freedom than an actual human will adequately capture the complexity of value because humans are complex systems and, therefore, cannot be reduced to a much simpler model.

If the model is sufficiently complex to robustly capture human values, such as whole-brain emulation, then the ethical concerns and S-risks of actually using these models for alignment appear because the model itself may suffer.

Many mathematical theories of human cognition or frameworks for computing (inferring) human values are considered as the basis for alignment, as well as process theories of alignment that implicitly rely on a particular mathematical theory even if it doesn’t infer (humans’ or AIs’) values explicitly, such as the shard theory (RL-based), or the latest Beren Millidge’s computational anatomy of human values, or cooperative inverse reinforcement learning, or Bayesian models and approaches, or various linguistic process theories of alignment that I expect to become very hot this year due to the astonishing success of LLMs. However, since all these theories are collapsing the complexity of humans (or else they are equivalent to full human simulations), they are all bound to be incomplete. Moreover, all these theories are bound to be biased (this is a form of inductive bias if you wish), that is, to be relatively blind to specific kinds of human values, or to specific aspects of human nature that we can see as somehow related to (or producing) “values”.

In other words, human values are not only complex in the sense that they are very elaborate. Crucially, human values are also not capturable within a single mathematical framework or ontology for describing them.

From “solving the alignment problem” to engineering the alignment process

The main implication of the above thesis is that we should abandon the frame that we should “solve” the alignment problem and seek a smart theory that will “crack” this problem.

I feel that a fair amount of unproductive debates and unproductive...
Power laws in Speedrunning and Machine Learning
by Jsevillamol — 2023-04-24T10:06:35.332Z
Link: https://arxiv.org/abs/2304.10004

Anticipating the future of AI necessarily requires anticipating results in performance. Ideally, we would like to understand the dynamics of improvements. This would help us preempt capabilities and understand how plausible sudden improvements are.

However, this problem is notoriously difficult. Among other reasons, Machine Learning benchmarks use many different metrics for measuring performance. And the history of improvements in all domains is limited, spanning around a dozen improvements in the longest-running benchmarks.

To circumvent these problems, we follow Sevilla (2021) and study video game speedrunning. Using data from speedrun.com, we investigate a previously noted regularity in world record progressions - an astounding fit to a power law pattern.

Figure 1: Plot of the ratio of improvement of world records against the number of previous world records in 25 speedrunning categories. We see a pattern characteristic of a power law.

Exploiting this regularity, we develop a random effects model for predicting the size of successive record improvements. We show that this model is significantly better than a baseline of predicting no improvement, and has a performance comparable to a model fit to the whole dataset ex-post. We combine this model with a simple model for predicting the timing of records to solve an outstanding problem from Sevilla (2021) - beating a baseline of predicting no improvement when forecasting new records out to some time horizon.

After studying the model in a data-rich environment, we adapt it to the more difficult case of predicting benchmarks in Machine Learning. Due to the lack of longitudinal data, we fail to provide definitive evidence of a power-law decay in Machine Learning benchmark improvements. However, we still show that it improves over a constant prediction baseline.

Assuming a power-law decay, we show that the model suggests two interesting patterns for machine learning:

Machine Learning benchmarks aren’t close to saturation. They exhibit a pattern of improvements more aggressive than what we will see once the benchmarks are close to their irreducible loss.
Sudden large improvements are infrequent but aren’t ruled out. According to the model, improvements over one order-of-magnitude in size above recent improvements happen once every fifty times.

Figure 2: Visual representation of two implications of the power-law model for p

...
The Brain is Not Close to Thermodynamic Limits on Computation
by DaemonicSigil — 2023-04-24T08:21:44.727Z

Introduction

This post is written as a response to jacob_cannel's recent post Contra Yudkowsky on AI Doom. He writes:

EY correctly recognizes that thermodynamic efficiency is a key metric for computation/intelligence, and he confidently, brazenly claims (as of late 2021), that the brain is about 6 OOM from thermodynamic efficiency limits

[...]

EY is just completely out of his depth here: he doesn't seem to understand how the Landauer limit actually works, doesn't seem to understand that synapses are analog MACs which minimally require OOMs more energy than simple binary switches, doesn't seem to understand that interconnect dominates energy usage regardless, etc.

Most of Jacob's analysis for brain efficiency is contained in this post: Brain Efficiency: Much More than You Wanted to Know. I believe this analysis is flawed with respect to the thermodynamic energy efficiency of the brain. That's the scope of this post: I will respond to Jacob's claims about thermodynamic limits on brain energy efficiency. Other constraints are out of scope, as is a discussion of the rest of the analysis in Brain Efficiency.

The Landauer limit

Just to review quickly, the Landauer limit says that erasing 1 bit of information has an energy cost of kTlog2. This energy must be dissipated as heat into the environment. Here k is Boltzmann's constant, while T is the temperature of the environment. At room temperature, this is about 0.02 eV.

Erasing a bit is something that you have to do quite often in many types of computations, and the more bit erasures your computation needs, the more energy it costs to do that computation. (To give a general sense of how many erasures are needed to do a given amount of computation: If we add n-bit numbers a and b to get a+bmod2n, and then throw away the original values of a and b, that costs n bit erasures. I.e. the energy cost is nkTlog2.)

Extra reliability costs?

Brain Efficiency claims that the energy dissipation required to erase a bit becomes many times larger when we try to erase the bit reliably.

The key transition error probability α is constrained by the bit energy:
α=e−EbkBT
Here's a range of bit energies and corresponding minimal room temp switch error rates (in electronvolts):

α=0.49, Eb=0.02eV

α=0.01, Eb=0.1eV

α=10−25, Eb=1eV

This adds a factor of about 50 to the energy cost of erasing a bit, so this would be quite significant if true...
Value Learning – Towards Resolving Confusion 
by PashaKamyshev — 2023-04-24T06:43:29.360Z

Part 1 - Keep calm and use revealed preferences. Originally posted on substack

In this post, I want to talk about the problem of value learning and specifically highlight a large amount of confusion around the problem and the term “values” generally. I hope to offer a paradigm that somewhat helps mitigate the confusion. This post is extremely difficult primarily because it requires a certain amount of detachment from many cultural convictions, including and especially “rationalist” convictions, to understand that certain cultures (current or future) may feel pretty different about issues X Y or Z.

Value learning is a subproblem of AGI alignment. It is not the whole problem, there are many other subproblems, such as decision theory or ontological crises. It is a sufficiently large subproblem. There seems to not have been much philosophical progress on this question in the last 10 years. I would even argue based on this sequence that there is probably more confusion now than before.

The question of “what humans value” might seem simple at first. We want to live happy and fulfilling lives, impact the world around us, be free from needless suffering, raise kids, and look forward to a good future for them. How hard can it be to get the AGI to put all of this into its utility function? How hard can it be to make the AGI care about the exact same things? I argue that while difficult in some parts, the confusion around “human values” has given the wrong perception of the difficulty in both directions. That is, it might be easier than the doomers think in some parts, but there are difficulties hiding elsewhere (discussed in part 2).

This confusion can be dissolved somewhat, and I hope to present a new paradigm that will jump-start the discussion in a more coherent framework. A big danger that the absence of philosophical clarity brings is the false rejection of promising paradigms, such as inverse reinforcement learning, which presents the most likely correct starting paradigm for a "good enough” AGI (which is different from “perfect AGI”). It is extremely dangerous to use bad and imprecise (an un-falsifiable) philosophical arguments to reject promising AI ideas.

The simple answer is that if you want to learn approximately well enough about what humans value, look at how humans behave and what variables they are increasing with th...
Summaries of top forum posts (17th - 23rd April 2023)
by Zoe Williams — 2023-04-24T04:13:31.797Z

We've just passed the half year mark for this project! If you're reading this (whether you're a regular reader or this is your first post), please consider taking this 5-10 minute survey - all questions optional. If you listen to the podcast, we have a separate survey for that here. This will directly influence our decisions of whether and how to continue this project next year, and we appreciate everyone who takes the time to fill it out.

Back to our regularly scheduled intro...

This is part of a weekly series summarizing the top posts on the EA and LW forums - you can see the full collection here. The first post includes some details on purpose and methodology. Feedback, thoughts, and corrections are welcomed.

If you'd like to receive these summaries via email, you can subscribe here.

Podcast version: Subscribe on your favorite podcast app by searching for 'EA Forum Podcast (Summaries)'. A big thanks to Coleman Snell for producing these!

Object Level Interventions / Reviews

AI

12 tentative ideas for US AI policy (Luke Muehlhauser)

by Lizka

Linkpost for

this list

of ideas by Luke Muehlhauser, which they tentatively think would increase the odds of good outcomes from transformative AI. These include:

Software export controls.
Requiring hardware security features on cutting-edge chips.
Tracking stocks and flows of chips and licensing big clusters.
Requiring a license to develop frontier AI models (which are then subject to info security and testing and evaluation requirements).
Fund specific genres of alignment, interpretability, model evaluation and info sec R&D.
Create a narrow antitrust safe harbor for AI safety & security collaboration.
Require certain kinds of AI incident reporting.
Clarify the liability of AI developers for concrete AI harms.
Create means for rapid shutdown of large compute clusters and training runs.

See also this

List of lists of government AI policy ideas

and post on

FLI (Future of Life Institute) report: Policymaking in the Pause

, both by Zach Stein-Perlman.
Policy recommendations from the

FLI policy brief

were:

Mandate robust third-party auditing and certification.
Regulate access to computational power.
Establish capable AI agencies at the national level.
Establish liability for AI-caused harms.
Introduce measures to prevent and track AI model leaks.
Expand technical AI safety research funding.
Develop standards for i

...
Do LLMs dream of emergent sheep?
by Shmi — 2023-04-24T03:26:54.144Z

This is somewhat speculative, but borne out by empirical data: LLMs developed an analogue of human System 1 and System 2, and their version of "System 2" is also slow, effortful and unreliable, just like ours.

https://twitter.com/emollick/status/1650324834960678912

This is an unusual emergent behavior, by all accounts. It is not like there is a shortage of accurate math in the training data. Maybe distilling these particular rules from the data is harder?

Interestingly enough, augmentation works for LLMs the same way it works for humans: you can get a math plugin, such as Wolfram Alpha, which enhances the model's abilities temporarily, but does not improve its "native" abilities with prolonged use. The moment the plugin (or a human's calculator) is "disconnected" the model (or the human) is back to its previous "self", not having learned much.

Maybe reasoning is harder in general? Or maybe it is an artifact of the training data? Is something like a System 2 an emergent feature? Does it mean that there is a "ceiling" of sorts for logical reasoning by the current crop of models, and we should not expect them to, say, code better than a human, write better poetry, prove theorems and discover new laws of nature? Would it have some implication for super-intelligent machines? I must admit, I am quite confused.
Not using a priori information for Russian propaganda
by EniScien — 2023-04-24T01:14:22.781Z

(The first version was written extremely poorly and received a lot of bad marks, so I hesitated to publish it for a long time, now this is the second version)

I live in Russia, so I regularly encounter samples of Putin's propaganda, and apparently, due to the effect of familiarity, I overestimate it credibility.

One of the characteristic features is that she does not seek to convince you that she is telling the truth, she seeks to convince you that no one is telling the truth, because such a thing simply does not exist, there are only views that are beneficial to various sides of propaganda.

The propaganda of your Motherland and the propaganda of the Anglo-Saxons, who throughout the history of our country have been trying to destroy it, because they are jealous.

And from this follows the tactics of attempts to persuade in specific situations. For example, if a house was destroyed in Ukraine, then you will be told that it is the Ukrainians who are bombing themselves in order to put the noble Russian soldiers in a bad light, and themselves as victims.

And there is often no evidence in either direction, and therefore it can very easily begin to seem that, in general, both versions are equally likely, and now you are simply supporting the enemies of the motherland for no reason.

However, this is a mistake, it does not take into account a priori information, which says that the probability of regularly bombing yourself after the troops of another country invaded you is extremely small, but the probability of bombing by the invading countries is extremely high, both in terms of statistics and with point of view of assessing the complexity of the hypothesis.