trentmkelly/LessWrong-43k huggingface dataset:
Is the meaning of words chosen/interpreted to maximize correlations with other relevant queries?
by tailcalled — 2022-10-20T10:03:19.931Z

A: Is there any water in the refrigerator?
B: Yes.
A: Where? I don’t see it.
B: In the cells of the eggplant.

The above might be a communication failure; A might have been looking for some water to drink, and you can't drink the water in the cells of an eggplant, so it is not water that is suitable for the purposes of A's quest. We might pin the failure to communicate on B; the contents of the cells of the eggplant is not usually considered water.

But water has a core essence, a property that gives it all its other properties: H2O. (Or H2O in its liquid state, or similar; it doesn't really matter as the eggplant also contains H2O in its liquid state.) It literally is the central node in a network of all the properties of water. And there's similar cases where you would consider the water in the cells of the eggplant to be water; for instance you might say that the eggplant is mostly made up of water if you want a rough guess of its density.

I would tend to say that this shows that the meanings of words is contextual; it depends on the situation. But in what way, exactly, does it depend on the situation? Here's a proposal:

You guess a variety of facts that the people communicating might be interested in knowing about, as well as a variety of observables that are available, as well as people's epistemic state with respect to those observables, and then you select the meaning of your words so that they maximize the correlations with these, within the range of ambiguity that the words can refer to.

The idea being that this maximizes opportunities for relevant communication, as well as for deduction. This proposal is vague and not necessarily well thought-out; I would be interested in general discussion, pointers to other work, examples or counterexamples, etc..
How to Write Readable Posts
by David Hartsough — 2022-10-20T07:48:12.419Z
Link: https://davidhartsough.com/writings/readable/

TL;DR: Broaden and deepen your understanding of your reader, and assume they need your post to be more clear and concise in order to understand your ideas.

But readability isn't just "clarity and brevity"; let's talk about assumptions for approachability and ways to step into the mind of your reader.

Outline

Prose On Writing and Revising and the reasoning behind "readability"

List of Readable Writing Tips and helpful assumptions to make of your reader

The Best Advice on writing and revising

Breakdown of Principles on readable and approachable writing

Collection of References and resources

Quick Conclusion

Relevance to EA

I ended up writing an entire prequel post for this post that explains why I was inspired to write this on the EA Forum in the first place and why I want to make the EA Forum more readable and approachable. You can read this preface here: Toward a more approachable and accessible EA Forum

(Just For Fun) Foreword

Most posts begin with a quote by some "wise" old person (who probably already said most of what the author is hoping to say in their post). It seems fun, so I made up this "proverbial" quote and hope you read it in the raspy voice of an old, wise person:

It is not a matter of merely writing and publishing; one must edit and revise as well.

So too is it not only a matter of what can be read; rather, one must ask: "What will be read?"

On Writing and Revising

Write to be Read

When you write a post, your end goal is for your readers to understand you and your ideas.

Presumably, these are the steps you hope to achieve with your writing process:

You write a post about ideas.

Others read your post.

Others understand your ideas.

(Others do something with those ideas.)

But dang, even making it to step 2 is a challenge!

How can you ensure that anyone will even read your writings, nevertheless begin to understand the ideas you've written about?

Understand to be understood

Upon closer inspection, you need to know: (1) who your reader is, and then (2) how to write posts that will be readable and understandable to them.

Now this process involves more nuanced steps:

Ensure you understand your readers. ("Who is reading my writing?")

Ensure your readers understand you. ("Will they understand my writing?")

If you understand your readers, then you can write in a way that your readers understand you. Thus, good writing: (1) can be read and w...
Notes on "Can you control the past"
by So8res — 2022-10-20T03:41:43.566Z

The following is a (lightly edited version of a) series of notes I sent Joe Carlsmith about his essay,

Can you control the past?

. It's addressed to Joe, but it seems worth publishing here while I'm on the topic of decision theory. I’ve included some of his comments, and my replies, below.

I only recently skimmed

Can you control the past?

, and have a couple notes that you may or may not be interested in. (I'm not under the impression that this matters a ton, and am writing this recreationally.)

First: this is overall a great review of decision theories. Better than most I've seen. Nice.

Now, onto some more substansive points.

Who am I?

I think a bunch of your sense of oddness about the "magic" that "you can write on whiteboards light-years away" is stemming from a faulty framing you have. In particular, the part where the word "you" points to a single physical instantiation of your algorithm in the universe. I'd say: insofar as your algorithm is multiply instantiated throughout the universe, there is no additional fact about which one is really you.

For analogy, consider tossing a coin in a quantum-mechanical universe, and covering it with your hand. The coin is superpositioned between heads and tails, and once you look at it, you'll decohere into Joe-who-saw-heads and Joe-who-saw-tails, both of whom stem from Joe-who-hasn't-looked-yet. So, before you look, are you Joe-who-saw-heads or Joe-who-saw-tails?

Wrong question! These two entities have not yet diverged; the pasts of those two separate entities coincide. The word "you", at the time before you split, refers to ~one configuration. The time-evolution splits the amplitude on that configuration between ~two distinct future configurations, and once they've split (by making different observations), each will be able to say "me" in a way that refers to them and not the other, but before the split there is no distinction to be made, no extra physical fact, and no real question as to whether pre-split Joe "is" Joe-who-will-see-heads versus Joe-who-will-see-tails.

(It's also maybe informative to imagine what happens if the quantum coin is biased. I'd say, even when the coin is 99.99999% biased towards heads, it's still the case that there isn't a real question about whether Joe-who-has-not-looked-at-the-coin is Joe-who-will-see-heads versus Joe-will-see-tails. There is a question of to wha...
Rhythmic Baby Toys
by jefftk — 2022-10-20T01:50:05.327Z

In another

debatably educational
application of my

rhythm stage setup, Nora
really likes playing with my bass pedals. She especially likes them
when I have them set in a particular way:

When you tap them they make a noise

When you tap them with a consistent beat they make them more
complex noise.

If your beat isn't consistent enough they go back to the
simpler noise until your tempo is clear again.

Here's a demo:

This has me wondering whether some sort of toy like this could be good
at teaching kids a sense of consistent rhythm? The initial noise is
enough to make the toy fun, but then if you can be consistent enough
you get something more fun. You could probably start with Somewhat
relaxed threshold for "consistent enough", and then tighten it
over time as they got better?

Comment via: facebook
What Does AI Alignment Success Look Like?
by Shmi — 2022-10-20T00:32:48.100Z

Suppose you are put in stasis and wake up 10 years after the FOOM. You are trying to figure out if the AI Alignment project succeeded or not. How can you tell? Not vaguely, concretely. What metrics indicate success, and what metrics indicate failure? The following are potential examples, based on various sci-fi AI tropes. No need to discuss each one separately, there are millions more, the idea is to delineate failures from success in the general case.

Earth has been turned into a mega-brain with no visible humans around.
The Universe around the Solar System has disappeared (or turned into an apparently solid shell of unknown composition), but humans are still around, living in abundance of anything they want or need and seemingly content to stay that way.
Every human gets their own (real or simulated) universe to play with. Some end up creating trillions of creatures whom they torture for fun.
Humanity lives inside a giant simulation.
AI development is under a strictly enforced interdict.
Anything remotely alive-looking is digimon-like creatures, who proclaim themselves human.
The universe appears empty except for one black hole that encodes humanity (in some form) in its horizon microstates.

Any links discussing this would be appreciated, too.
Scaling Laws for Reward Model Overoptimization
by leogao — 2022-10-20T00:20:06.920Z
Link: https://arxiv.org/abs/2210.10760

TL;DR: Reward model (RM) overoptimization in a synthetic-reward setting can be modelled surprisingly well by simple functional forms. The coefficients also scale smoothly with scale. We draw some initial correspondences between the terms of the functional forms and the Goodhart Taxonomy. We suspect there may be deeper theoretical reasons behind these functional forms, and hope that our work leads to a better understanding of overoptimization.

Some other results:

We compare two different methods of optimization (RL, best-of-n); RL consumes much more KL distance than best-of-n for the same amount of optimization.
We show that using KL distance between the initial and optimized policies is not a reliable measure of optimization power when comparing different methods. We also find that penalizing based on the KL distance in RL does not change the KL distance--gold reward frontier in our setting.
We find some very preliminary evidence that at our scales, scaling the policy does not substantially increase the amount of optimization pressure placed on the RM. Further study of this effect could be relevant to some models of inner optimization.
With a few additional assumptions, our functional form also makes some predictions about iterated RLHF (that it will reduce Extremal Goodhart but not Regressional Goodhart).
This setup only captures the effect of overoptimizing a learned RM relative to using the ground truth directly. In particular, this setup does not directly capture any mismatch between the ground truth labels and the human intent, which plausibly contains a majority of the difficulty of outer alignment

If you're interested in the intersection of alignment theory and empirical research, we're hiring! We want to gain insight on things like Goodhart's Law, ELK, and inner alignment via experiments on large language models. Shoot me (leogao) a DM if you're interested.

Select figures from the paper
What is Consciousness?
by belkarx — 2022-10-19T21:14:59.442Z

The word consciousness gets thrown around a lot on these forums, but what does it really mean? The first, easy, shallow definition to come up on The Internet is "the quality or state of being aware especially of something within oneself". So when do humans become "aware"? Looking back on your own experiences as a child, would you really call yourself "conscious"? What is the relation between wakefulness and consciousness? What is consciousness?

I'm trying something distinct from the typical long-winded highbrow posts seem here: bullet pointing data[1], with admittedly subjective conclusions and some questions/wonderings for more active engagement and personal opinion-forming with the topic. Partially wikipedia-style, given the sheer density of footnotes, but much more speculative, with tendrils extending into various broader topics.

Self-awareness is associated with an area of the brain called the DMN (default mode network)

Maybe it can be selectively blocked
Or selectively electrically stimulated (what effects would that have on someone's life?)

When do we become self-aware?

Children have a DMN that is much more weakly connected.[2]What is the critical point for self-awareness and how can it be stimulated earlier?

Conscious vs Self-aware

A nomenclature issue - these are often used to describe the same idea, but consciousness is often associated with the variable "in a sleep state". Can it be quantified by just checking frequencies of brain waves?

In that case, is being in REM sleep conscious? Is the word "consciousness" sufficient?

Is lucid dreaming conscious? (more research needs to be done on this - how exactly does a lucid dreaming state differ from REM)

I propose [3]a two-pronged definition, each part individually specified and quantified (in the past[4], complexity of brain activity has been used to measure consciousness - that is effective when consciousness is artificially induced through anesthetics but less valuable practically):

degrees of wakefulness - measured through frequencies of EEG waves, or in the future, if REM sleep should not count as wakefulness, biosensing[5] of various neurotransmitters such as glycine or specific proportional combinations of histamine, norepinephrine, and serotonin[6]
degrees of self-awareness - measured by connectiveness of DMN

Different proportions of these data points can make up distinct, discrete "types" of

...
What to do if a nuclear weapon is used in Ukraine?
by Valentin2026 — 2022-10-19T18:43:58.486Z

Tegmark, Samotsvety, and Metaculus provide probability estimations for the use of nuclear weapons in Ukraine, and the start of the full-scale nuclear wars (kaboom and KABOOM in the terminology of Tegmark). kaboom will certainly be a warning sign before KABOOM, so it makes sense to think beforehand about the action plan in the case of kaboom.

Individual plans

I am aware of two potential strategies of survival during the nuclear war. First, you can hide in a shelter, and for preparation I recommend to read "Nuclear war survival skills" . Second, you can try to be in a place minimally affected by nuclear war. In my opinion, the second strategy is better in a long run, since the life in the bombed territory will be most likely far worse than on the untouched one, even if today this untouched territory is a relatively poor country. Thus, as soon as the warning event (little kaboom, using nuclear weapons against Ukraine) happens, I would move for some period of time to one of the countries that are unlikely to be bombed. Here is my quick and dirty estimate of which countries are potentially good for this purpose. Suppose a full scale Russia-NATO war. Then:
-Obviously Russia and NATO countries are not good (a couple of small NATO countries like Montenegro may survive but it is a risky decision). Potentially NATO alliances like Israel are also dangerous, but so far Russian-Israel relationships are quite good so I would consider it much less likely.
-Tegmark argues that China will be struck by US so that it would not become the dominant power after the war, referring to this declassified official US strategy. Thus, China seems to be very risky option as well.
-If China indeed is struck by USA, Australia is not safe because of AUKUS. If, moreover, China uses the same strategy of getting rid of potential dominators, India could be under attack.
Thus, I think that countries and regions that are not likely to be struck include:
Latin America
Africa
New Zealand and Pacific (Australia has some risk)
Asia except: China, maybe India and Pakistan, and maybe countries of CSTO
Non-NATO European countries (the safest bet would be Switzerland).
Also, a remote part of an attacked country may still work if there is no military base there.
Now, of course, there's the question of fallout and nuclear winter. Will this country or re...
If I asked for an explanation of a perfect Utopia, could you give one?
by Akkira — 2022-10-19T17:56:47.909Z

Given a handful of requirements, what are some options to creating a perfect (or close to) Utopian civilization? This is purely for my own entertainment purposes, and I feel its a fun, timeless topic to discuss among the users of LessWrong. I highly encourage criticism of others responses and answers, as well as the criticism of the requirements/restraints I have stated below.

The requirements are simple.

Increase happiness, pleasure, and well-being as much as possible, while decreasing suffering, displeasure, and unwell-being as much as possible.
No limit can be put on growth as a species and society. We must be able to expand, and increase population.
No AI or "Basilisk" like entity can be in control of society. A human person(s) must be in charge, if any.
Solutions to problems must be stated now. Using the advancement of technology and philosophy in the future as a potential solution is not an option. For example, stating: "Humanity will be more advanced in the future, and able to solve this problem at that point in time" Is not a solution.
This Utopia takes place in this current timeline, in whatever current time you are seeing this post, meaning you cannot change the past to influence your Utopia. However, Explaining your plans for the future of your Utopia is highly recommended.
Measures must be taken to eliminate, or reduce the risk of societal collapse.

Morals and ethics are not in play, the less moral and ethical, the better.
Should we push for requiring AI training data to be licensed?
by ChristianKl — 2022-10-19T17:49:55.644Z

We frequently speak about AI capability gain being bad because it shortens the timeframe for AI safety research. In that logic, taking steps to decrease AI capability would be worthwhile.

At the moment the large language models are trained with a lot of data without the company, that trains the language model, licensing the data. If there would be a requirement to license the required data, that would severely reduce the available data for language models and reduce their capabilities.

It's expensive to fight lawsuits in the United States. Currently, there are artists who feel like their rights are violated by Dalle 2 using their art as training data. Similar to how Thiel funded the Gawker lawsuits, it would be possible to support artists in a suit against OpenAI to require OpenAI to license images for training Dalle 2. If such a lawsuit is well-funded it will be much more likely that a precedent for requiring data licensing gets set which would slow down AI development.

I'm curious about what people who think more about AI safety than myself think about such a move. Would it be helpful?
Hacker-AI and Digital Ghosts – Pre-AGI
by Erland Wittkotter — 2022-10-19T15:33:47.103Z

Software vulnerabilities are mainly found by chance. We all know hacking is labor-intensive. This will invite and thrive the use of AI. It is assumed that it is just a matter of time until attackers create tools with advanced “Hacker-AI” to accelerate the detection and understanding of hard-/software vulnerabilities in complex/unknown technical systems.

This post suggests that Hacker-AI will become a dangerous consequential cyberwar weapon - capable of decapitating governments and establishing persistent global supremacy for its operator. Two features amplify the motivation for creating and deploying this Hacker-AI: stealth distribution/deployment as an undetectable digital ghost and that it could be irremovable, i.e., the first Hacker-AI could be the only one.

Here we focus solely on significant problems related to attacks or threats. These threats we define as actual or potential harm or damage to humans and/or their property. Threats and attacks are intentional and significant for their victims immediately or in the future. We give every significant problem a headline that catches the gist of the underlying issues so we can refer to these problems easily later.

We don’t assume extraordinary abilities from an AGI. The drivers for this development are technically skilled humans/organizations who seek AI tools to accomplish their goals faster and less labor-intensive.

Problems/ Vulnerabilities in our IT ecosystem

The following 10 problems, issues, or vulnerabilities do not follow a particular order. It is assumed that they all contribute to the danger of Hacker-AI. Preventing Hacker-AI implies confronting these issues technically.

“Software is invisible”. Software consists of (compiled) instructions that run directly or via an intermediary layer on CPUs. It’s stored in files invisible in strict practical or operational terms. We can only make it visible indirectly with assumptions on other software, i.e., that their complex interplay of instructions is reliable, not modified or corrupted. We then see compiled software data encoded in binary characters. However, seeing that code tells us nothing about code’s purposes. Instead, we depend on announcements on what the software is supposed to do, and we (usually) get what is expected. But how can we be sure that there is no additional hidden (malicious) sleeper code or backdoors in the software?

Malicious code could be ...
The reward function is already how well you manipulate humans
by Kerry — 2022-10-19T01:52:30.317Z

We have many, many parables about dangers arising from super intelligent or supremely capable AI that wreaks ruin by maximizing a simple reward function. "Make paperclips" seems to be a popular one[1]. I worry that we don't give a more realistic scenario deep consideration: the world that arises from super capable AIs that have "manipulate humans to do X" as the reward function.

This is a real worry because many human domains already have this as a reward function. The visual art that best captures human attention and manipulates emotion is what is sought out, copied, sold, and remembered. The novels that use language most effectively to manipulate human emotion and attention are what sells, gather acclaim, and spread through the culture. Advertisements that most successfully grab human attention and manipulate emotion and thought leading to purchase get the most money. I'm sure anyone reading this can easily add many more categories to this list.

An ML model that can successfully manipulate humans at a super human level in any of these examples would generate great wealth for it's owner. I would argue that there is considerable evidence that humans can be manipulated. The question is, how good are humans at this manipulation? Is there enough untapped overhead in manipulative power that an AI could access to perform super human manipulation?

Fiction explores the concept of super human entertainment. "The Entertainment"[2] of Infinite Jest, and Monty Python's "Funniest Joke in the World"[3] both present media that can literally kill people by extreme emotional manipulation. In the real world it seems video games are already entertaining enough to kill some people.[4] Could an AI create content so engaging as to cause dehydration, starvation, sleep deprivation, or worse? I can perhaps too easily imaging a book or show with a perfect cliff-hanger for every chapter or episode, where I always want to read or watch just a little bit more. With a super human author this content could continue such a pattern for ever.

In a more commercial vein, imagine there is a company that sells ads, and also has the best AI research in the world. This company trains a model that can create and serve ads such that 90% of people who view the ad, buy the product. This would lead to a direct transfer of wealth, draining individuals bank accounts and enriching the ad compa...