trentmkelly/LessWrong-43k huggingface dataset:
Why do some people try to make AGI?
by TekhneMakre — 2022-06-06T09:14:44.346Z

Why do some people invest much of their energy trying to discover how to make AGI?

Who's trying to discover how to make AGI? Academic researchers and their students, academic institutions, small and large companies and startups with AI teams with some members working on speculative things, independent lone researchers, privately funded research groups, independent / volunteer / hobbyist researchers and collaborations.

Are they really? A lot of this is people working on narrow AI or on dead-end approaches without learning anything, intentionally or not. Some people explicitly say they're trying to discover how to make AGI.

For the people investing much of their energy trying to discover how to make AGI, why are they doing that?

Plausible reasons:
-- Coolness / prestige (it's cool to be an AGI researcher; it's fun to be in the club; it's cool to be one of the people who made AGI)
-- Money (salary)
-- Need a job.
-- The problem is interesting / fun.
-- It would be interesting / fun to see an AGI.
-- AGI seems generally very useful, and humans having useful things enables them to get what they want, which is usually good.
-- AGI is the endpoint of yak-shaving.
-- There's something they want to with an AGI, e.g. answer other questions / do science / explore, make money, help people, solve problems.
-- They want there to be an agent more intelligent than humans.
-- They want everthing to die (e.g. because life contains too much suffering).
-- They want every human to die.
-- They want to disrupt society or make a new society.
-- They want power over other people / the world.
-- They want nerds in general to have power as opposed to whoever currently has power.
-- They want security / protection.
-- It fits with their identity / self-image / social role.
-- They're in some social context that pressures them to make AGI.
-- They like being around the people who work on AGI.
-- They want to be friends with a computer.
-- To piss off people who don't want people to work on AGI.
-- To understand AGI in order to understand alignment.
-- To understand AGI in order to understand themselves.
-- To understand AGI in order to understand minds in general, to see a mind work, to understand thought.
-- They believe in and submit to some kind of basilisk, whether a Roko's basilisk (acausal threats from a future AGI) or a political / moral-mazes basilisk (the AGI is the future new CEO / re...
Miriam Yevick on why both symbols and networks are necessary for artificial minds
by Bill Benzon — 2022-06-06T08:34:12.023Z

Miriam Yevick was a mathematician who corresponded with physicist David Bohm in the 1950s and went on to publish a very interesting article on the formal structure of perception and cognition: Holographic or fourier logic [1]. The abstract:

A tentative model of a system whose objects are patterns on transparencies and whose primitive operations are those of holography is presented. A formalism is developed in which a variety of operations is expressed in terms of two primitives: recording the hologram and filtering. Some elements of a holographic algebra of sets are given. Some distinctive concepts of a holographic logic are examined, such as holographic identity, equality, contaminent and “association”. It is argued that a logic in which objects are defined by their “associations” is more akin to visual apprehension than description in terms of sequential strings of symbols.

In 1978 she commented on an article by John Haugeland, The nature and plausibility of Cognitivism, and spells out the implications of her idea for cognition ([2] p. 253):

The author here points out a distinction between two modes of understanding our environment: the first identifies objects by quasi-linguistic representations; the other apprehends objects by means of nonarticulate skills. This dichotomy; which is undoubtedly related to the complexity of the concrete objects to be recognized or manipulated, was projected as follows by von Neumann (1966, pp. 51-54): “certain objects are such that their description is more complex than the object itself.”

We can explicate this proposition on a theoretical level in the domain of optical patterns. (See Yevick, 1975op. cit.). Such patterns or objects are thin, white regions on a black background. These can be simple (regular), like the outlines of rectangles; or complex, like the outlines of Chinese characters or random-like motions. The following holds true: a complex object requires a long (sequential, quasi-linguistic) description but yields a sharp recognition (auto-correlation) spot under holographic filtering; hence it is identified most readily by holographic recognition, or holistically. A simple object requires a short (quasi-linguistic) description but yields a diffuse recognition spot; hence it is identified most readily by quasi-linguistic representation or description.

Description and holographic recognition thus appear as two (c

...
Optimization and Adequacy in Five Bullets
by james.lucassen — 2022-06-06T05:48:03.852Z
Link: https://jlucassen.com/optimization-adequacy-five-bullets/

Context: Quite recently, a lot of ideas have sort of snapped together into a coherent mindset for me. Ideas I was familiar with, but whose importance I didn't intuitively understand. I'm going to try and document that mindset real quick, in a way I hope will be useful to others.

Five Bullet Points

By default, shit doesn't work. The number of ways that shit can fail to work absolutely stomps the number of ways that shit can work.
This means that we should expect shit to not work, unless something forces it into the narrow set of states that actually work and do something.
The shit that does work generally still doesn't work for humans. Our goals are pretty specific and complicated, so the non-human goals massively outnumber the human goals.
This means that even when shit works, we should expect it to not be in our best interests unless something forces it into the narrow range of goal-space that we like.
Processes that force the world into narrow, unlikely outcome ranges are called optimization processes - they are rare, and important, and not magic.

Main Implications

The biggest takeaway is look for optimization processes. If you want to use a piece of the world (as a tool, as an ally, as evidence, as an authority to defer to, etc), it is important to understand which functions it has. In general, the functions a thing is "supposed to have" can come wildly apart from the things that it's actually optimized to do. If you can't find a mechanism that forces a particular thing to have a particular useful property, it probably doesn't. Examples:

"Society" designates certain people/organizations/processes as authorities - sources of reliable evidence. It turns out that a lot of these are not in fact reliable, because nothing forces them to be. Nobody fires the news anchor when they get a prediction wrong.
A sort of internal version is checking when to use explicit modeling and when to trust intuition. Intuitions seem to come with a built-in feeling that they are truth-tracking and should be trusted, even in cases where you know they don't have the evidence budget to make this possible. To a certain extent, you can check manually if an intuition is actually optimized to be truth-tracking.
The air conditioner thing.

The obvious first step when looking for optimization processes: learn to recognize optimization processes. This is the key to what Yudkowsky calls an adequacy argum...
Why agents are powerful
by Daniel Kokotajlo — 2022-06-06T01:37:07.452Z

[Written for Blog Post Day. Not super happy with it, it’s too rambly and long, but I’m glad it exists.]

Here are some questions I think this theory of agency can answer:

What are agents?
Why should we expect AI agents to be useful for various important tasks?
Why should we think agentic mesa-optimizers may arise for some data+reward-signals that weren’t designed explicitly to produce them?
Why should we think humans+AI tools will eventually be outcompeted by AI agents?
Why should we expect, on priors, mildly superhuman AI agents to be powerful enough to take over the world, if they wanted to?

What are agents?

Earlier in “

Agents as P

2

B chain reactions

” I defined agents as successful P2B feedback loops—a learner algorithm and a planner algorithm hooked up together, the planner plans to plan better and thus outputs actions that result in getting more useful data into the learner, more resources for the planner to plan with… then the process repeats… Like how “fire” is a successful feedback loop in which heat+plasma+fuel come together to produce more heat+plasma, and envelop more fuel, if any is nearby.

Then I asked: Is the planner algorithm strictly necessary here? Could you get something similar to a P2B feedback loop, but built around something other than a planner? I explored this question indirectly in “

Gradations of Agency.

” The answer seems to be “sorta.” You can have e.g. a Level 3 system that doesn’t do any planning and yet behaves remarkably similar to a P2B feedback loop in many contexts, due to imitating the success of others. Such a system would tend to underperform P2B in contexts where there aren’t good examples to imitate.
We could stick to the original definition and say: Agents are P2B feedback loops, so, level 4 and above.

But I think it’s probably better to be a bit more galaxy brain and say: the key thing is convergent instrumental resource feedback loops and that thinking about P2B is a stepping stone which helps us see why such phenomena exist and why the really big ones tend to have planners sitting in the middle directing them, and the really really big ones tend to have level 6+ algorithms sitting in the middle directing them…

By analogy, heat+plasma+wood+oxygenl chain reactions aren’t the key thing. The key thing is chain reactions that convert some sort of fuel into thermal and kinetic energy plus more of the chain reaction.&n...
Epistemological Vigilance for Alignment
by adamShimi — 2022-06-06T00:27:43.956Z

This post is part of the work done at Conjecture.

Nothing hampers Science and Engineering like unchecked assumptions.

As a concrete example of a field ridden with hidden premises, let's look at sociology. Sociologist must deal with the feedback of their object of study (people in social situations), their own social background, as well as the myriad of folk sociology notions floating in the memesphere. You might think that randomized surveys and statistics give you objective knowledge of the sociological world, but these tools also come with underlying assumptions — that the phenomenon under study must not depend on the fine structure of the social network, for example. In general, if you don’t realize this, you will then confidently misinterpret the results without considering the biases of your approach — as in asking kids to sort their play activities into three categories you defined in advance, and then seeing this as a “validation” of the classification.

How to avoid these mistakes? Epistemological vigilance, answer Pierre Bourdieu, Jean-Claude Chamboredon, and Jean-Claude Passeron in "Le métier de sociologue". They borrow the term from French philosopher of science Gaston Bachelard, to capture the attitude of always expliciting and questioning the assumptions behind notions, theories, models, experiments. So the naive sociologists err because they fail to maintain the restless epistemological vigilance that their field requires.

Alignment, like sociology, demands a perpetual questioning of unconscious assumptions. It’s because the alignment problem, and the way we know about it, goes against some of our most secure, obvious, and basic principles about knowledge and problem-solving. Thus we need a constant vigilance to keep them from sprouting again unnoticed and steering our work away from alignment.

In this post I thus make explicit these assumptions, and discuss why we have to be epistemologically vigilant about them.[1] Taken separately, none of these call to vigilance is specific to alignment — other fields fostered it first. What makes alignment unique is the combined undermining of all these assumptions together. Alignment researchers just can't avoid the epistemological struggle.

Here is my current list:[2]

Boundedness: the parameters of the problem are bounded, and such bounds can be approximated.

Reasons for vigilance: we can’t find (yet) any bo

...
AGI Ruin: A List of Lethalities
by Eliezer Yudkowsky — 2022-06-05T22:05:52.224Z

Preamble:

(If you're already familiar with all basics and don't want any preamble, skip ahead to

Section B

for technical difficulties of alignment proper.)

I have several times failed to write up a well-organized list of reasons why AGI will kill you. People come in with different ideas about why AGI would be survivable, and want to hear different obviously key points addressed first. Some fraction of those people are loudly upset with me if the obviously most important points aren't addressed immediately, and I address different points first instead.

Having failed to solve this problem in any good way, I now give up and solve it poorly with a poorly organized list of individual rants. I'm not particularly happy with this list; the alternative was publishing nothing, and publishing this seems marginally more dignified.

Three points about the general subject matter of discussion here, numbered so as not to conflict with the list of lethalities:

-3. I'm assuming you are already familiar with some basics, and already know what '

orthogonality

' and '

instrumental convergence

' are and why they're true. People occasionally claim to me that I need to stop fighting old wars here, because, those people claim to me, those wars have already been won within the important-according-to-them parts of the current audience. I suppose it's at least true that none of the current major EA funders seem to be visibly in denial about orthogonality or instrumental convergence as such; so, fine. If you don't know what 'orthogonality' or 'instrumental convergence' are, or don't see for yourself why they're true, you need a different introduction than this one.

-2. When I say that alignment is lethally difficult, I am not talking about ideal or perfect goals of 'provable' alignment, nor total alignment of superintelligences on exact human values, nor getting AIs to produce satisfactory arguments about moral dilemmas which sorta-reasonable humans disagree about, nor attaining an absolute certainty of an AI not killing everyone. When I say that alignment is difficult, I mean that in practice, using the techniques we actually have, "please don't disassemble literally everyone with probability roughly 1" is an overly large ask that we are not on course to get. So far as I'm concerned,

if you can get a powerful AGI tha

...
Teaching all things to all people in all ways
by Anonymous — 2022-06-05T21:38:45.220Z

Last month, I accidentally unloaded some of my beliefs on a friend with regards to how a number of issues that are reacted to by members in our society (US) appear to be fundamentally related to education. Generally, I feel that some appear to distinguish their general right of freedom of speech that they share with others as something entirely different when compared to "information". These, however, are the same and the way in which we are educated is directly tied to the way that we interpret information. This is fundamental because information is how we distinguish reality. To clarify, when I refer to the first amendment I'm not referring to the freedom to "tweet", host a blog, all-CAPS at individuals in a forum, etc. I'm generally referring to any and all possible material that can be used to learn all of the things that are ubiquitous to an individual.

To start on why I believe that information is worth reconsidering how we think about (especially in regards to how it works with capitalism)...We are currently in the information age, and our number one source of information is in a state of anarchy. If we interpret the first amendment in the way that I described it, then this is actually a good thing™. If we attempt to enforce order on this medium, then another market (that some refuse to acknowledge) will always appear thus tying this somewhat into capitalism. We all likely believe that "some" things should not be learned, but we need to always keep in mind that if we "censor" information, it doesn't make it go away. It only drives up the value of said information elsewhere. Also, there's two types of censorship to consider, one is the restriction of information, the other is the excess of information which has the same effect as censorhip. I definitely believe that the resolution of these apsects can be handled by some aspects of capitalism and its overall structure of values without needing to enforce/censor if given the right perspective (which I will get into in a later piece).

Regardless, as a society it appears that we are practically at a point where individuals can have a completely different perception of reality from another, and it seems as if it can escalate towards near-hostilities when another attempts to resolve it altruistically. This seems to be because of a difference in values or perhaps it's simply when individuals are asked to percei...
Noisy environment regulate utility maximizers
by Niclas Kupper — 2022-06-05T18:48:43.083Z

This post was written as part of the AISC under supervision of John Wentworth. I started my PhD simultaneously with this project and admit that I underestimated the necessary effort to excel at both. I still had a great time, learned a lot and met a bunch of interesting people. I never hit upon the "one cool thing", but I hope this post still has some useful takeaways.

I would like to thank John Wentworth for his supervision and Lucius Bushnaq for our brainstorming sessions. I would also like to thank the AISC team for organizing the event. Finally, I'd like to thank Lauro Langosco for recommending AISC to me.

Consider some agent maximizing some utility. We usually model this agent to be in some environment. It can be reasonable to assume that the environment we find ourselves in is noisy. There are two (mostly) equivalent ways this can happen: either our observations contain some noise, or our actions are performed noisily.
Even if the environment is not noisy we might still gain some useful properties from training an agent with (some kinds of) noise. There is mild evidence suggesting it helps with robustness, see e.g. Open AI's robot hand or this paper. I want to investigate a particular formalism of how to treat noise. From this formalism I hope to gain
- A definition / understanding of noise resistance
- Protection against distributional shifts
- Better generalization
- Other theoretical insights

How I got here

The project started with the hope to expand the idea of Utility Maximization as compression. This is related to the Telephone Theorem and the Generalized Heat Engine. While exploring these ideas I played around with various models, and particularly focused on ones where compression works nicely. Not unsurprisingly boolean functions were particularly insightful, as they are both easy to do explicit calculations with and more easily relate to compression. This lead me to the theory of boolean functions and noise sensitivity, which I then viewed as a utility function, although many interpretations are possible / useful.

Formal setup

Consider some finite space A. We consider two classes of functions:
1. Boolean functions taking bits as input and returning bits as output i.e. u:{−1,1}A→{−1,1}.
2. Generalized boolean functions which instead returns a real number i.e. u:{−1,1}A→R.

In the rest of the post I will not always specify the type o...
Expected Value vs. Expected Growth
by tom-pollak — 2022-06-05T18:48:35.868Z

If you wager your entire bankroll every time you place a bet, your bankroll will eventually be 0.

Expected Value

Expected value is the average amount we receive if we repeat a bet an infinite number of times.

E[X]=∑ixiP(xi)

Where:

xi - the percentage of stake won or lost from outcome $i$
P(xi) - the probability of that outcome.

For a typical sports bet, $x_i$ can be found with the equation:

xi=S(Oi−1)B0

Where:

S - stake
Oi - decimal odds of outcome $i$
B0 - initial bankroll before bet

This calculates the relative increase in our bankroll from each possible outcome. For a losing scenario, xi=−SB0 as we just lose our stake.

Using EV

This can be used as a performance metric for a bet – if one bet has an expected value of 5% and another 10% we would tend to prefer the latter to the former. However, this ignores relative likelihoods on the different outcomes.

E.g. If a bet has a 0.0000001% likelihood but pays out 11,000,000% the expected value is 10%, but this is not an attractive bet for most people. You certainly wouldn't expect a 10% return on your investment.

So we need a system dependent on the player's current bankroll.

Enter Expected Growth

The fundamental issue is that, while expected value calculates the long-term value of infinitely repeating a bet, in any real scenario we can only bet once, and only get one outcome.

This is the problem with the previous example, as despite being positive EV, you could only bet fractions of a penny if you have any hope to be paid out before your bankroll hits 0. So while you have an expected value of 10%, the _expected outcome_ is that you lose the wager almost every time.

We can formalize this:

EG[X]=−1+∏i(1+xi)P(xi)

xi is the money won/lost relative to your entire bankroll, as opposed to expected
value, which is relative to your stake.

This is expected growth, and it uses the expected outcomes of an event rather than the
value. For any given case, expected growth will be less than the expected value. This
lends itself to the idea of a finite bankroll, so that no matter how positive EV a bet
is, betting too much can cause your bankroll to shrink.

If you wager your entire bankroll every time you place a bet, your bankroll will eventually be 0.

Let us take this as an example for EG, any outcome that causes you to lose your entire bankroll (xi=−1) will imply EG=−1, no matter the probability of th...
Northboro Air Quality Measurement
by jefftk — 2022-06-05T17:50:02.060Z

Yesterday I played the

Northboro contra dance with

Kingfisher, and I made the
same CO2 measurements I did

at BIDA a few weeks ago.
This is a very different environment:

30 vs 80 attendees

Eight big windows with no fans vs two doors with big barrel fans

Maybe 1/3 the size

Here's what I measured, compared to last time:

The overall shape is very similar: rising as dancers arrive and start
dancing, falling during the break, rising again after the break, and
then falling at the end. BIDA's break was probably slightly later,
and it took a little longer for the dancers to leave the home at the
end of the night.

Because atmospheric CO2 is ~400ppm, this chart makes the two cases
look a bit more similar than they really are. Let's subtract 400 ppm
from each and look at CO2 levels above baseline:

Looking at the area under the curve, a dancer at Northboro likely
experienced about 40% of the risk of one at BIDA (448 vs 1,116 ppm
hours of excess CO2).

Putting fans in the windows would likely cut this even further,
especially if you have them blow across
the hall and match them to the prevailing wind.

Comment via: facebook