trentmkelly/LessWrong-43k huggingface dataset:
five ways to say "Almost Always" and actually mean it
by Yudhister Kumar — 2023-04-22T10:38:20.456Z
Link: https://www.ykumar.org/almost-always/

In English

A boring, colloquial way.

"Almost all the eggs are gone!" (as half a dozen remain)

Not-Finite-ness

A slightly-less-boring mathy way.

"Almost all prime numbers are odd!"

There are infinitely many primes. There is exactly one even prime number (2). Infinity minus one is... infinity.

"Almost all natural numbers are larger than one-hundred thousand quadrillion quadrillion vigintillion! (10^83)"

There are infinitely many natural numbers. There are infinitely many natural numbers larger than one-hundred thousand quadrillion quadrillion vigintillion. No practical difference between 10^83 and 1 (other than that one is an upper bound on the number of atoms in the universe).

You formalize this with sets: if you have some (infinite) set, a subset whose complement has a finite cardinality encompasses almost everything in the set. We call this cofiniteness.

Probability ~One

A way to quantify surety.

"Almost all the people will lose money at the casino!"

I'm not sure what the exact rates are, but I'd bet money on this being true.

"Almost all numbers are composite!"

It is well known that the number of prime numbers below Nis approximately N/lnN. In the limit as N goes to infinity, the ratio of primes to non-primes numbers goes to approximately zero. So if you choose a random positive integer, I would bet my life savings that it's composite.

"Almost all graphs are asymmetric."

An intuitive explanation: if you take all possible combinations of nodes and vertices and let the number of each tend towards infinity, you should expect chaos to triumph over order. (sorry Ramsey). This does depend on a certain definition of symmetry, however, and a clearer statement would be "almost all graphs have only one automorphism."

The Lebesgue Measure is Not-Zero

A straightforward-yet-strange way for real numbers.

The Lebesgue measure is what you get when you try to generalize length, area, and volume to n-dimensions. It forms the basis for our current understanding of integration, and helps us figure out how big stuff is.

Something with zero volume basically doesn't exist anyway.

"Almost all real numbers are irrational!"

Well yes, all the rationals have measure zero. All countable sets have measure zero, and the rationals are countable.

(Measures are nice: they're a neat generalization of physical scales (mass, volume, etc.) to arbitrary mathematical objects. They're particularl...
P(doom|superintelligence) or coin tosses and dice throws of human values (and other related Ps).
by Muyyd — 2023-04-22T10:06:47.701Z

TL;DR P(doom|superintelligence) is around 99% because AI researchers now don't have a precise theory of human values and precise understanding of inner structure of current AI, so they can’t encode one into the other. And while future AI may have different architecture and development process than LLMs, it also may be some variation of the same with all complications included.

What "doom" here is means? It means "certain death". And P(doom|superintelligence) means "probability that superintelligent AI will change the world in a way that will be incompatible with human survival".

Apparently there is no unified human value theory so i am going to use descriptions from clearerthinking.org and this post (value hierarchy seems distinct and well put). And to calculate P(doom|superintelligence) I will take "Immediate survival needs", all 5 of them. With 0.5 probability of success for each of them (because they are distinct). But where does "0.5" came from?

Level I: Immediate Survival

1. Oxygen – an

open airway

(between our lungs and the outside world) and a constant supply of air that is at least 19.5% oxygen (and partial pressure of oxygen of less than 1.4 atm)

2. Functioning – freedom from severe/acute injury, bodily damage, and organ failure

3. Safety – no immediate threats from the environment, including from dangerous human or nonhuman animals

4. Temperature – protection from hypothermia or hyperthermia (i.e., one’s core body temperature needs to stay within 95-104 degrees Fahrenheit at all times)

5. Hydration -at least a few liters of water every three days

It comes from aggregation of 2 schools of thoughts: natural ethics emergence and alien actress.

Some people think that morality is naturally emergent phenomena and AI will simply attain it by getting smarter, some people even think that a particular version of human ethics will emerge (out of human-prodused training data) that they like. And we don’t have to worry about things like AI alignment because thing will sort themself out so P(doom|superintelligence) is very small.

Some people think that there is a "shoggoth" pretending to be "nice and helpful", that we can have somewhat meaningful conversation with it about "justice" and "diversity" but we have no idea if this entity upon attaining (or making) superintelligence still be as "nice and helpful" as it is appear to be now. And this people have different P(doom|s...
LessWrong moderation messaging container
by Raemon — 2023-04-22T01:19:00.971Z

This is a shell post for various comments which I'm using as the source-of-truth for some moderation messaging (currently, what a new user sees when they go to make their first post or comment).

It seemed good to make it public so people could comment on it if they wanted. I'm particularly unsure about how to handle things like "not being familiar with background material." I think it's easy to overdo and make it feel very burdensome, but also man it sure is annoying having a bunch of people showing up missing a lot of context.

(You can comment on this post, the two initial comments are just used as the text-source for some places in our codebase)
Neural network polytopes (Colab notebook)
by Zach Furman — 2023-04-21T22:42:37.671Z
Link: https://colab.research.google.com/github/zfurman56/polytopes/blob/main/Polytopes.ipynb

The polytope theory of neural networks (also known as the "spline theory of deep learning") seeks to explain (ReLU) neural networks based on their piecewise linear regions. This gives a helpful intuition for how neural networks approximate functions, as well as a potential avenue for interpretability research. For anyone who wants to engage directly with the topic, I put together a small Colab notebook, with interactive 2D and 3D visuals.
Readability is mostly a waste of characters
by vlad.proex — 2023-04-21T22:05:34.907Z

Based on this research, I have created a free web app for compressing GPT prompts at gptrim.com. gptrim is also a Python library. Contributions are welcome.

Readability and information

The other day I was trying to feed a large text to ChatGPT, and I grew frustrated with the limits of the chat message size. After some brainstorming with Bing, I had an insight.

Not all tokens in a text convey information. Some are there to make it easier for humans to parse text. Consider spaces: they are mostly used to tokenize words for the human visual cortex. ButifIwrotewithoutspacesyouwouldprobablystillunderstandme.

There are very few cases where a missing space would cause ambiguity. The nicest one ChatGPT could come up with was:

"Iamnowhere" could be interpreted as "I am nowhere" or "I am now here," which have opposite meanings.

Even in such cases, the ambiguity would probably be solved by context.

What about punctuation? Again, I suspect there are very few cases where removing punctuation would create an ambiguity that cannot be solved by context. ChatGPT:

"Let's eat grandma" vs. "Let's eat, grandma." Without the comma, it could be misinterpreted as a request to eat the grandmother.

Can we go further? We could apply word stemming. This would significantly reduce our character count. How much information would we sacrifice? Try to read this snippet (full text below).

a machin learn research write me in respons to yesterday ’ s post , say : i still think gpt - 2 is a brute - forc statist pattern matcher which blend up the internet and give you back a slight unappet slurri of it when ask . i resist the urg to answer “ yeah , well , your mom is a brute - forc statist pattern matcher which blend up the internet and give you back a slight unappet slurri of it when ask .” but i think it would have been true. (source)

A machine learning researcher writes me in response to yesterday’s post, saying:

I still think GPT-2 is a brute-force statistical pattern matcher which blends up the internet and gives you back a slightly unappetizing slurry of it when asked.

I resisted the urge to answer “Yeah, well, your mom is a brute-force statistical pattern matcher which blends up the internet and gives you back a slightly unappetizing slurry of it when asked.”

But I think it would have been true.

I find that "caveman Scott" is still very much readable, especially if I know ...
Thinking about maximization and corrigibility
by James Payor — 2023-04-21T21:22:51.824Z

Thanks in no small part to Goodhart's curse, there are broad issues with getting safe/aligned output from AI designed like "we've given you some function f(x), now work on maximizing it as best you can".

Part of the failure mode is that when you optimize for highly scoring x, you risk finding candidates that break your model of why a high-scoring candidate is good, and drift away from things you value. And I wonder if we can repair this by having the AI steer away from values of x that break our models, by being careful about disrupting structure/causal-relationships/etc we might be relying on.

Here's what I'd like to discuss in this post:

When unstructured maximization does/doesn't work out for the humans
CIRL and other schemes mostly pass the buck on optimization power, so they inherit the incorrigibility of their inner optimization scheme
It's not enough to sweep the maximization under a rug; what we really need is more structured/corrigible optimization than "maximize this proxy"
Maybe we can get some traction on corrigible AI by detecting and avoiding internal Goodhart

When does maximization work?

In cases when it just works to maximize, there will be a structural reason that our model connecting "x scores highly" to "x is good" didn't break down. Some of the usual reasons are:

Our metric is robustly connected to our desired outcome. If the model connecting the metric and good things is simple, there's less room for it to be broken.
Examples: theorem proving, compression / minimizing reconstruction error.
The space we're optimizing over is not open-ended. Constrained spaces leave less room for weird choices of x to break the correspondences we were relying on.
Examples: chess moves, paths in a graph, choosing from vetted options, rejecting options that fail sanity/legibility checks.
The optimization power being applied is limited. We can know our optimization probably won't invent some x that breaks our model if we know what kinds of search it is performing, and can see that these reliably don't seek things that could break our model.
Examples: quantilization, GPT-4 tasked to write good documentation.
The metric f is actively optimized to be robust against the search. We can sometimes offload some of the work of keeping our assessment f in tune with goodness.
Examples: chess engine evaluations, havin

...
Would we even want AI to solve all our problems?
by So8res — 2023-04-21T18:04:11.636Z

Status: more of the basics, that I find myself regularly repeating.

When people from the wider population talk to me about politics or global warming or various other large global issues (as they're prone to do), I am prone to noting that most problems would be easier to fix if we had superintelligent friends at our backs.

A common objection I get goes something like:

So you think you can get AI to solve all the world's problems? Wouldn't that kinda suck, though? Wouldn't a problem-free life be kinda terrible? Isn't overcoming obstacles part of the fun of life?

To which my answer is: that's completely right. If we aren't careful, artificial intelligence could easily destroy all the value in the future, e.g. by aggressively solving everyone's problems and giving everyone everything they ask for, in a way that sucks all the joy and meaning out of life.

Value is fragile. The goal is not to alleviate every ounce of discomfort; the goal is to make the future awesome. My guess is that that involves leaving people with real decisions that have real consequences, that it involves giving people the opportunity to screw up, that it involves allowing the universe to continue to be a place of obstacles that people must overcome.

But it also probably involves putting an end to a bunch of the senseless death, and putting an end to bad things happening to good people for no reason, and putting an end to the way that reality can punish a brief lapse of judgment with disproportionate, horrific, and irreversible consequences.

And it probably involves filling the universe with much more fun and joy and laughter. (And not just by building a bunch of speakers that play laugh tracks on a loop, but by filling the universe with the sorts of things that cause joyful laughter among healthy people, plus the healthy people to experience those things and laugh joyfully.)

The problem of making the future awesome is tricky and subtle. The way to get AI to help in the long term, is not to tell them exactly what our current best guess of what's "good" consists in, in a long list of current philosopher's best guesses, and then hope that modern philosophers got it exactly right. Because modern philosophers haven't got it exactly right; and that whole approach is doomed to failure.

The hope is that we can somehow face our AI and point at all the things we care about, and say "we don't know how to d...
The Commission for Stopping Further Improvements: A letter of note from Isambard K. Brunel
by jasoncrawford — 2023-04-21T17:42:22.587Z
Link: https://rootsofprogress.org/isambard-brunel-on-engineering-standards

On May 24, 1847, a bridge over the Dee River in Chester, England, collapsed. A passenger train plunged into the river; five people were killed and nine seriously injured.

The subsequent investigation blamed the bridge’s cast iron girders. Cast iron, like concrete but unlike wrought iron or steel, is strong in compression but weak in tension, and it is brittle, meaning that it breaks all at once, rather than deforming. The wrought iron trusses evidently were not enough to strengthen the girder.

Etching of River Dee bridge disaster. Illustrated London News, June 12, 1847

In response to the disaster, a Royal Commission on the Application of Iron to Railway Structures was created in August of that year, “to inquire into the conditions to be observed by engineers in the application of iron in structures exposed to violent concussions and vibration”—that is, to set up standards and requirements, or as they were known in France at the time, règles de l’art.

Isambard Kingdom Brunel. Wikimedia

In their investigation, the Commission solicited the opinion of one of the most eminent engineers of the age, Isambard Kingdom Brunel. But his response was, presumably, not what they expected.

Brunel begins his letter by saying that he is sorry they asked for his opinion, because of “my doubts of the advantage of such an enquiry, and my fears of its being, on the contrary, productive of much mischief, both to science and to the profession.” (Brunel’s son, writing his biography, says that he called them “The Commission for Stopping Further Improvements in Bridge Building.“) But since they did ask, he felt it necessary to state his full and honest views.

While he was happy to give his engineering opinion to the commission, he warned that

… the attempt to collect and re-issue as facts, with the stamp of authority, all that may be offered gratuitously to a Commission in the shape of evidence or opinions, to stamp with the same mark of value statements and facts, hasty opinions and well-considered and matured convictions, the good and the bad, the metal and the dross … this, I believe, always has rendered, and always will render, such collections of miscalled evidence injurious instead of advantageous to science…

He argued that there was no way the Commission could get better information than an engineer could on his own, but that in addition they would receive a lot of useless opinions, whi...
Should we publish mechanistic interpretability research?
by Marius Hobbhahn — 2023-04-21T16:19:40.514Z

TL;DR: Multiple people have raised concerns about current mechanistic interpretability research having capabilities externalities. We discuss to which extent and which kind of mechanistic interpretability research we should publish. The core question we want to explore with this post is thus to which extent the statement “findings in mechanistic interpretability can increase capabilities faster than alignment” is true and should be a consideration. For example, foundational findings in mechanistic interpretability may lead to a better understanding of NNs which often straightforwardly generates new hypotheses to advance capabilities.

We argue that there is no general answer and the publishing decision primarily depends on how easily the work advances alignment in relation to how much it can be used to advance capabilities. We recommend a differential publishing process where work with high capabilities potential is initially only circulated with a small number of trusted people and organizations and work with low capabilities potential is published widely.

Related work:

A note about differential technological development

,

Current themes in mechanistic interpretability research

,

Thoughts on AGI organization and capabilities work

,

Dan Hendrycks’s take

, etc.

We have talked to lots of people about this question and lost track of who to thank individually. In case you talked to either Marius or Lawrence about this, thank you!

The basic case for publishing

Let’s revisit the basic cases for publishing.

Alignment is probably hard. To get even close to a solution, it likely requires many people working together, coordinating their work and patching together different approaches. If the work isn’t public, this becomes much harder and thus a solution to alignment becomes less likely.
More people can engage with the work and build on it. Especially people who want to get into the field or are less connected might not be able to access documents that are only shared in small circles.
It gives more legitimacy to the work of organisations and individuals. For people who are not yet established in the field, publishing their work is the most obvious way to get noticed by an organisation. For academics, publications are the most relevant resource for their careers and organisations can generate more legitimacy by

...
500 Million, But Not A Single One More - The Animation
by Writer — 2023-04-21T15:48:22.117Z
Link: https://youtu.be/ljmifo4Klss

This video is an animation of 500 Million, But Not A Single One More, by @jai

Some notes for the curious about the cult of Sopona, the god of Smallpox of the Yoruba people

When I was getting feedback on the storyboard from Jai, he mentioned that there is an actual god of Smallpox: Sopona. We decided to use it to lightly inspire our design by adding a mask similar to the ones found in wooden statues representing the god.

Apparently, in Nigeria there were cults of Sopona that strongly opposed vaccination. At the bottom-left paragraph at this link, there's an interesting story. Dr. O. Sapara, a Nigerian physician, infiltrated "the sopono cult", described as a large secret society in western Nigeria that used to cause smallpox outbreaks to generate clients to "cure" and to follow-up on blackmails. Sapara infiltrated the society and helped the government ban the cult as an illegal organization. After the cult was banned worship of the God continued.

Jai rightly mentioned the priests of the cult as a striking example of the most hollow humans allying with the enemy.
Talking publicly about AI risk
by Jan_Kulveit — 2023-04-21T11:28:16.665Z

In the past year, I have started talking about AI risk publicly - in mainstream newspapers, public radio, national TV, some of the most popular podcasts. The twist, and reason why you probably haven't noticed is I'm doing this in Czech. This has a large disadvantage - the positive impact is quite limited, compared to English. On the other hand, it also had a big advantage - the risk is very low, because it is very hard for memes and misunderstandings to escape the language bubble. Overall I think this is great for experiments with open public communication .

Following is an off-the-cuff list of notes and suggestions. In my view the debate in Czech media and Czech social networks is on average marginally more sensible and more informed than in English so far, so perhaps part of this was successful and could be useful for others.

Context: my views

For context, it's probably good to briefly mention some of my overall views on AI risk, because they are notably different from some other views.

I do expect

1. Continuous takeoff, and overall a large amount of continuity of agency (note that continuous does not imply things move slowly)

2. Optimization and cognition distributed across many systems to be more powerful than any single system, making a takeover by a single system possible but unlikely

3. Multiagent interactions to matter

4. I also do expect the interactions between the memetics and governance and the so-called "technical problem" to be strong and important

As a result I also expect

5. There will be warning shots

6. There will be cyborg periods

7. World will get weird

8. Coordination mechanisms do matter

This perspective may be easier to communicate than e.g. sudden foom - although I don't know.

In the following I'll usually describe my approach, and illustrate it by actual quotes from published media interviews I'm sort of happy about (translated, unedited). Note that the specific ways how to say something or metaphors are rarely original.

Aim to explain, not to persuade

Overall I usually try to explain stuff and answer questions, rather than advocate for something. I'm optimistic about the ability of the relevant part of the public to actually understand a large part of the risk at a coarse-grained level, given enough attention.

So even though we invented these machines ourselves, we don't understand them well enough?

We know what's going on there at the mic

...