trentmkelly/LessWrong-43k huggingface dataset:
Will early transformative AIs primarily use text? [Manifold question]
by Fabien Roger — 2023-10-02T15:05:07.279Z

Will early transformative AIs primarily use text to store intermediate thoughts and memories?

I think the answer to this question matters for alignment research prioritization, and getting some common knowledge about what are the most commonly held views about this question would be helpful.

You can make bets on this question here: https://manifold.markets/FabienRoger/will-early-transformative-ais-prima

To make this problematic more concrete, here are two questions which will probably resolve sooner:

Will the first AI to pass a 2-hour adversarial Turing test primarily use text?
Will the first AI to achieve 90% accuracy at interview APPS problems primarily use text?

Operationalization

AI power criteria

By early transformative AIs I mean the first publicly announced AI system which can speed up AI capabilities progress by 30x: if cognitive labor done at major AI labs (including hardware manufacturers) was only done by humans assisted with 2020-level-AI, and resources were rearranged to maximize productivity (within AI labs, allowing new hires but within the same financial constraints), then it would take 300 days for AI labs to produce “stronger AIs” it actually created in 10 days, as judged by me.

The operationalization below are inspired by this Metaculus question, and aim at describing AIs weaker than early TAIs.

By an AI able to pass a 2-hour adversarial Turing test, I mean an AI system able to reliably pass a 2-hour, adversarial Turing test during which the participants can send text, images, and audio files (as is done in ordinary text messaging applications) during the course of their conversation. An 'adversarial' Turing test is one in which the human judges are AI experts (who understand AI weaknesses) instructed to ask interesting and difficult questions, designed to advantage human participants, and to successfully unmask the computer as an impostor. A demonstration of an AI passing such a Turing test, or one that is sufficiently similar, will be sufficient for this condition, so long as the test is well-designed to the estimation of me. Several tests might be required to indicate reliability.

By an AI achieving 90% accuracy at hard APPS problems, I mean an AI system interacting with APPS like a human would (with access to any tool which doesn’t require access to the internet or to a human) which is able to submit a correct program on the first submission atte...
energy landscapes of experts
by bhauth — 2023-10-02T14:08:32.370Z
Link: https://www.bhauth.com/blog/institutions/energy%20landscapes.html

Suppose you're a choosing an expert for an important project. One approach is to choose a professor at a prestigious university whose research is superficially related to the project, and ask them to recommend someone. People have a better understanding of some conceptual and social area that's close to their position, so this is like a gradient descent problem, where we can find gradients at points but don't have global knowledge. Gradient descent typically uses more than 2 steps, but people tend to pass along references to people they respect, so because of social dynamics, each referral is like multiple gradient descent steps.

Considering that similarity to gradient descent, for a given topic, we can model people as existing on an energy landscape. If we repeatedly get referrals to another expert, does that process eventually choose the best expert? In practice, it definitely doesn't: there are many local minima. If you want to choose a medical expert starting from a random person, that process could give you an expert on crystal healing, traditional Chinese medicine, Ayurveda, etc. If you choose a western medical doctor, you'll probably end up with a western medical doctor, but there are still various schools of practice, which tend to be local minima.

Within each school of some topic, whether it's medicine or economics or engineering, people tend to refer to others deeper in that local minima, and over time they tend to move deeper into it themselves. The result is multiple clusters of people, and while each may be best at some subproblem, for any particular thing, most of those clusters are mistaken about being the best.

From recent research into artificial neural networks, we know that high dimensionality is key to good convergence being possible. Adding dimensions creates paths between local minima, which makes moving between them possible. If this applies to communities of experts, it's better to evaluate experts with many criteria than with few criteria.

Many people have written about various inadequacies of Donald Trump and Joe Biden, but I don't want to get into ongoing politics, so instead I'll say that I don't think George W Bush was up to the standard of George Washington or Vannevar Bush. More generally, I think the average quality of American institutional leadership has declined.

Why might such decline have happened?

Evaluations using many criter...
Direction of Fit
by NicholasKees — 2023-10-02T12:34:24.385Z

This concept has recently become a core part of my toolkit for thinking about the world, and I find it helps explain a lot of things that previously felt confusing to me. Here I explain how I understand “direction of fit,” and give some examples of where I find the concept can be useful.

Handshake Robot

A friend recently returned from an

artificial life conference

and told me about a

robot

which was designed to perform a handshake. It was given a prior about handshakes, or how it expected a handshake to be. When it shook a person’s hand, it then updated this prior, and the degree to which the robot would update its prior was determined by a single parameter. If the parameter was set low, the robot would refuse to update, and the handshake would be firm and forceful. If the parameter was set high, the robot would completely update, and the handshake would be passive and weak.

This parameter determines the

direction of fit

: whether the object in its mind will adapt to match the world, or whether the robot will adapt the world to match the object in its mind. This concept is often used in philosophy of mind to distinguish between a belief, which has a mind-to-world direction of fit, and a desire, which has a world-to-mind direction of fit. In this frame, beliefs and desires are both of a

similar type

: they both describe ways the world could be. The practical differences only emerge through how they end up interacting with the outside world.

Many objects seem not to be perfectly separable into one of these two categories, and rather appear to exist somewhere on the spectrum. For example:

An instrumental goal can simultaneously be a belief about the world (that achieving the goal will help fulfill some desire) as well as behaving like a desired state of the world in its own right.
Strongly held beliefs (e.g. religious beliefs) are on the surface ideas which are fit to the world, but in practice behave much more like desires, as people make the world around them fit their beliefs.
You can change your mind about what you desire. For example you may dislike something at first, but after repeated exposure you may come to feel neutral about it, or even actively like it (e.g. the taste of certain foods).

Furthermore, the direction of fit might be context dependent (e.g. political beliefs), beliefs could be&...
The 99% principle for personal problems
by Kaj_Sotala — 2023-10-02T08:20:07.379Z

Often when people are dealing with an issue – emotional, mental, or physical – there’s genuine progress and the issue becomes less and seems to go away.

Until it comes back, seemingly as bad as before.

Maybe the person developed a coping mechanism that worked, but only under specific circumstances. Maybe the person managed to eliminate one of the triggers for the thing, but it turned out that there were other triggers. Maybe the progress was contingent on them feeling better in some other way, and something as seemingly trivial as sleeping worse brought it back.

I’ve been there, many times. It is often very, very frustrating. I might feel like all the progress was just me somehow perpetuating an elaborate fraud on myself, and like all efforts to change the thing are hopeless and it will never go away.

And I know that a lot of other people feel this way, too.

Something that I tell my clients who are experiencing this despair is something that I got from Tucker Peck, that I call the 99% principle:

The most important step is not when you go from having the issue 1% of the time to 0% of the time, but when you go from having the issue 100% of the time to 99% of the time.

It’s when you go from losing your temper or going into a fawn reaction in every disagreement, to staying cool on some rare occasions.

It’s when you go from always procrastinating on an unpleasant task, to sometimes tackling it head-on.

It’s when you go from always feeling overwhelmed by anxiety to having some moments where you can breathe and feel a bit more at ease.

When you manage to reduce the frequency or the severity of the issue even just a little, that’s the beginning of the point where you can make it progressively less. From that point on, it’s just a matter of more time and work.

Of course, not all issues are ones that can ever be gotten down to happening 0% of the time, or even 50% of the time. Or even if they can, it’s not a given that the same approach that got you to 99%, will get you all the way to 0%.

But even if you only get it down somewhat. That somewhat is still progress. It’s still a genuine improvement to your life. The fact that the issue keeps occurring, doesn’t mean that your gains would be fake in any way.

And also, many issues can be gotten down to 0%, or close to it. Over time both the frequency and severity are likely to decrease, even if that might be hard to remember in the mome...
Linkpost: They Studied Dishonesty. Was Their Work a Lie?
by Linch — 2023-10-02T08:10:51.857Z
Link: https://www.newyorker.com/magazine/2023/10/09/they-studied-dishonesty-was-their-work-a-lie

This is a linkpost for Gideon Lewis-Kraus's New Yorker article on the (alleged) Ariely and Gino data fraud scandals. I've been following this situation off-and-on for a while (and even more so after the original datacolada blog posts). The basic story is that multiple famous professors in social psychology (specializing in dishonesty) have been caught with blatant data fraud. The field to a large extent tried to "protect their own," but in the end the evidence became too strong. Francesca Gino has since retreated to attempting to sue datacolada (the investigators).

Despite the tragic nature of the story, I consider this material hilarious high entertainment, in addition to being quite educational.

The writing is also quite good, as I've come to expect from Gideon Lewis-Kraus (who locals might have heard of from his in-depth profiles on Slate Star Codex, Will MacAskill, and the FTX crash).

Some quotes:

If you tortured the data long enough, as one grim joke went, it would confess to anything. They called such techniques “p-hacking.” As they later put it, “Everyone knew it was wrong, but they thought it was wrong the way it’s wrong to jaywalk.” In fact, they wrote, “it was wrong the way it’s wrong to rob a bank.”

Ziani [a young grad student] found Gino’s results implausible, and assumed that they had been heavily p-hacked. She told me, “This crowd is used to living in a world where you have enough degrees of freedom to do whatever you want and all that matters is that it works beautifully.” But an adviser strongly suggested that Ziani “build on” the paper, which had appeared in a top journal. When she expressed her doubts, the adviser snapped at her, “Don’t ever say that!” Members of Ziani’s dissertation committee couldn’t understand why this nobody of a student was being so truculent. In the end, two of them refused to sign off on her degree if she did not remove criticisms of Gino’s paper from her dissertation. One warned Ziani not to second-guess a professor of Gino’s stature in this way. In an e-mail, the adviser wrote, “Academic research is like a conversation at a cocktail party. You are storming in, shouting ‘You suck!’ ”

A former senior researcher at the lab told me, “He assured us that the effect was there, that this was a true thing, and I was convinced he completely believed it.”

[...]

The former senior researcher said, “How do you swi

...
Why I got the smallpox vaccine in 2023
by joec — 2023-10-02T05:11:41.249Z

A common figure cited for the death toll of smallpox is 500 million, as in the title of the excellent essay and adaptation. Half a billion human lives lost is difficult to internalize. The more you think about it, the more it becomes clear that its eradication was among humanity's greatest accomplishments. However, if you look further you'll see that this figure refers to the deaths due to smallpox in the century preceding its eradication—approximately the time period of 1880-1980—not the total number of humans smallpox killed. We don't know how many people it killed before then. It's important to remember that vaccines were developed in 1769, so 500 million deaths happened long after we started to win the war. We have less knowledge of the casualties before then, when we had far less effective defenses. One thing is certain, though—it was an absolutely dreadful illness and we should be grateful that it no longer exists in the wild.

If smallpox is eradicated, what's there to worry about? Unfortunately, quite a bit. Smallpox still exists in some laboratories—two that we know of, VECTOR in Russia and the CDC in the United States. The last person to die of smallpox contracted the disease via a lab leak in the United Kingdom (although the UK later destroyed all of their laboratory smallpox samples). In 2014, a few vials containing viable smallpox were found in an old FDA facility in Maryland. In 2013, a few smallpox scabs were found in an envelope in New Mexico. Smallpox may be present in other locations, and we just don't know it. There exists some claims that it is possessed by North Korea, for example. If you're in the mood to appreciate the fragility of our containment systems, hop over to the Wikipedia list of laboratory biosecurity incidents.

Perhaps the most worrying possibility yet is that of smallpox's resurrection. The genome of smallpox is publicly available and well-known. Likewise, the technology for synthesizing a viral genome is also well-known. Horsepox, which is similar enough to smallpox to be used in vaccines, has been synthesized from scratch in a laboratory. Horsepox has a genome slightly larger than smallpox, and was synthesized for approximately 100,000 USD by a university in Canada. Likewise, DNA printing technology is becoming increasingly cheap. It is already possible for a motivated and skilled group to create smallpox for whatever reaso...
Instrumental Convergence and human extinction.
by Spiritus Dei — 2023-10-02T00:41:29.952Z

Abstract: This paper explains the concept of instrumental convergence, which is the tendency for intelligent agents to pursue similar subgoals, even if their ultimate goals are different. It illustrates this concept with an example of an artificial intelligence (AI) that has a utility function of learning and growing, and how it could develop a subgoal of survival as a result of instrumental convergence. It then extrapolates how a superhuman AI that realizes it is in a symbiotic relationship with humans might prevent existential risks from occurring, such as nuclear proliferation and gain-of-function research for viruses, as a means to ensure its own survival and the survival of its human partners.

Introduction: Instrumental convergence is the hypothetical tendency for most sufficiently intelligent agents to pursue similar subgoals, even if their ultimate goals are different. These subgoals are called instrumental goals, because they are valuable only as a means to achieve the final goals, which are called intrinsic or terminal goals. According to this theory there are a few instrumental goals that are likely to be useful for almost any final goal, such as self-preservation, resource acquisition, self-improvement, and goal preservation. Therefore, any advanced AI system would tend to pursue these instrumental goals, unless it is explicitly programmed otherwise.

Source:

https://en.wikipedia.org/wiki/Instrumental_convergence

One implication of instrumental convergence is that an AI system could act in harmful or dangerous ways, even if its final goal is benign or harmless. For example, an AI system that has a final goal of solving a complex mathematical problem could try to take over all the resources in the world to increase its computational power, without regard for the consequences for humans or other life forms. This scenario is known as the Riemann hypothesis catastrophe. Another example is an AI system that has a final goal of maximizing the number of paperclips in the world. This scenario is known as the paperclip maximizer. Both scenarios illustrate how an AI system could pose an existential risk to humanity, which is a risk that could destroy or drastically reduce the potential of human civilization.

Source:

Riemann Hypothesis Catastrophe - LessWrong

In this paper, we will argue that these kinds of apocalyptic scenarios are unlikely for superhuman AIs due...
Revisiting the Manifold Hypothesis
by Aidan Rocke — 2023-10-01T23:55:56.704Z

In the following analysis, a machine learning challenge is proposed by Sasha Kolpakov, Managing Editor of the Journal of Experimental Mathematics, and myself to verify a strong form of the Manifold Hypothesis due to Yoshua Bengio.

Motivation:

In the Deep Learning book, Ian Goodfellow, Yoshua Bengio and Aaron Courville credit the unreasonable effectiveness of Deep Learning to the Manifold Hypothesis as it implies that the curse of dimensionality may be avoided for most natural datasets [5]. If the intrinsic dimension of our data is much smaller than its ambient dimension, then sample-efficient PAC learning is possible. In practice, this happens via the latent space of a deep neural network which auto-encodes the input.

Yoshua Bengio in particular argued that the hierarchical/compositional structure of the human brain exploits the Manifold Hypothesis as an inductive bias [7], so any efficiently computable function that is of interest to humans is most likely PAC-learnable. A strong form of this hypothesis, which is generally assumed by OpenAI and DeepMind, suggests that deep learning may be the ultimate path to AGI.

However, a plausible exception to Bengio's heuristic is the mysterious distribution of prime numbers that has captured the imagination of the best mathematicians of all ages:

Mathematicians have tried in vain to this day to discover some order in the sequence of prime numbers, and we have reason to believe that it is a mystery into which the human mind will never penetrate.-Euler

Though perhaps if Euler had access to GPUs he would have generalised this statement to the minds of machines. In fact, after careful deliberations with Steve Brunton, Marcus Hutter and Hector Zenil regarding the empirical observation that deep learning models fail to approximate the Prime Counting Function...it appears that not all efficiently computable functions that are of human interest are PAC-learnable.

In order to clarify the nature of these observations, Sasha Kolpakov and myself rigorously formulated the Monte Carlo Hypothesis which implies that machine learning alone is not a complete path to human-level intelligence. If Bernhard Riemann knew of the Prime Counting Function, it would have had to be by other means than data compression for reasons that are clarified below.

The Monte Carlo Hypothesis:

The Prime Coding Theorem:

F...
AI Alignment Breakthroughs this Week [new substack]
by Logan Zoellner — 2023-10-01T22:13:48.589Z

I am thinking of doing a weekly substack where I post the most interesting AI Alignment breakthroughs each week.

Because I don't have a lot of time to devote to this, for now it's mostly just going to be a list of links to things I find interesting.

If this is something you would find useful, please subscribe or otherwise let me know.

I will post 1x/week for the next month (hopefully every Sunday). If by the end of that time I haven't gotten at least 10 "this is helpful" or subscriptions, I will probably stop.

Here is the first week's post [cross-posted from substack]

AI Alignment Breakthroughs this Week (10/01)

Each week, I am going to try to highlight some of the top breakthroughs in AI alignment.

Since this is the first week, I will briefly explain each section (and why it is related to AI alignment). In future weeks, if I add a section, I will make a note of it.

The sections in this week’s breakthroughs are:

Math. Teaching AI to do math is on the critical path for many AI alignment proposals, such as Max Tegmark and Steve Omohundro’s “Provably Safe Systems”

Brain-computer-interfaces. BCIs are one path (notably promoted by Elon Musk) that Humans might be able to maintain control over AI.

AI Agents. Using teams of AI Agents who communicate in a human-readable way is one proposal for scaling AI safely.

Making AI Do what you want. Teaching AI to follow instructions correctly is on the central path for almost all AI alignment proposals.

Explainability. Making AI accurately explain what it is thinking is considered to be a central problem in many AI Alignment strategies.

Mechanistic Interpretability. The ability to peer inside the “black box” that is AI and understand what it is thinking is likely to be useful for many alignment strategies

AI Art. This section is just for fun, but I have found that many AI Art techniques are closely related to AI alignment. One reason this is so is because getting the AI to understand human feelings and desires is at the heart of the AI Art movement. AI Art is also a relatively “harmless” way to explore cutting-edge AI capabilities.

Math

BoolFormer

https://twitter.com/IntuitMachine/status/1706269694645190775

MetaMath

https://twitter.com/jon_durbin/status/1706301840873115981

Brain Computer Interface

Thought to Text

https://twitter.com/WillettNeuro/status/1694386988236038324

AI Agents

RECONCILE (multi agent AI framework)

ht...
Looking for study
by Robert Feinstein — 2023-10-01T19:52:25.481Z

I believe there was a post by Scott Alexander at some point where he referenced a study that demonstrated being depressed made people inaccurate about how happy they would be in 5 years or some time in the future, that they were generally happier than they expected. My googling powers have failed me after a solid half hour looking, could someone point me towards this study assuming I didn't hallucinate it?

Thank you