trentmkelly/LessWrong-43k huggingface dataset:
How to hire somebody better than yourself
by lemonhope — 2024-08-28T08:12:53.450Z

TLDR: use a very difficult timed 8-hour task to screen candidates. You should fail your own test.

I'm no expert but I've made some recommendations that turned out pretty well -- maybe like 5 ever. This post would probably be better if I waited 10 years to write it. Nonetheless, I think my method is far better than what most orgs/corps do. If you have had mad hiring success (judging by what your org accomplished) then please comment!

Half-remembered versions of Paul Graham's taste thing and Yudkowsky's Vinge's Law have lead some folks to think that judging talent above your own is extremely difficult. I do not think so.

Prereqs:

It's the kind of position where someone super good at it can generate a ton of value — eg sales/outreach, coding, actual engineering, research, management, ops, ...

Lots of candidates are available and you expect at least some of them are super good at the job.

You have at least a month to look.

It's possible for someone to demonstrate extreme competence at this type of job in a day or two.

Your org is trying to do a thing — rather than be a thing.

You want to succeed at that thing — ie you don't have some other secret goal.

Your goal with hiring people is to do that thing better/faster — ie you don't need more friends or a prestige bump.

Your work situation does not demand that you look stand-out competent — ie you don't unemploy yourself if you succeed in hiring well.

(Probably you don't meet the prereqs. Probably your org strongly depends on connection and reputation more than talent; probably your raises depend on you not out-hiring yourself; etc. Don't feel bad — it is totally ok to be a normal org/human! Being a goal psycho often sucks in every way except the accomplished goals.)

If you do meet the prereqs, then good news, hiring is almost easy. You just need to find people who are good at doing exactly what you need done. Here's the method:

Do look at performance (measure it yourself)

Accept noise

Don't look at anything else (yet)

Except that they work hard

Do look at performance

Measure it yourself. Make up a test task. You need something that people can take without quitting their jobs or much feedback from you; you and the candidate should not become friends during the test; a timed 8-hour task is a reasonable starting point. Most importantly, you must be able to quickly and easily distinguish good results from very good res...
On agentic generalist models: we're essentially using existing technology the weakest and worst way you can use it
by Yuli_Ban — 2024-08-28T01:57:17.387Z

The bottom of the coffee table is not the ceiling

Not long ago, Andrew Ng posted one of the most fascinating tweets I've read in quite some time. It's a longer post, but the main introduction is worthy jumping off from, as it inadvertently answers a question many current AI-skeptics raise about the nature of LLMs.

https://twitter.com/AndrewYNg/status/1770897666702233815

Today, we mostly use LLMs in zero-shot mode, prompting a model to generate final output token by token without revising its work. This is akin to asking someone to compose an essay from start to finish, typing straight through with no backspacing allowed, and expecting a high-quality result. Despite the difficulty, LLMs do amazingly well at this task!

Not only that, but asking someone to compose an essay essentially with a gun to their backs, not allowing any time to think through what they're writing, instead acting with literal spontaneity.

That LLMs seem capable at all, let alone to the level they've reached, shows their power, but this is still the worst way to use them, and this is why, I believe, there is such a deep underestimating of what they are capable of.

Yes, GPT-4 is a "predictive model on steroids" like a phone autocomplete

That actually IS true

But the problem is, that's not the extent of its capabilities

That's just the result of how we prompt it to act

The "autocomplete on steroids" thing is true because we're using it badly

YOU would become an autocomplete on steroids if you were forced to write an essay on a typewriter with a gun to the back of your head threatening to blow your brains out if you stopped even for a second to think through what you were writing. Not because you have no higher cognitive abilities, but because you can no longer access those abilities. And you're a fully-formed human with a brain filled with a lifetime of experiences, not just a glorified statistical modeling algorithm fed gargantuan amounts of data.

Consider this the next time you hear of someone repeating the refrain that current generative AI is a scam, where the LLMs do not understand anything and are not capable of any greater improvements beyond what they've already achieved. Remember this whenever, say, Gary Marcus claims that contemporary AI has plateaued at the GPT-4 class:

We analyzed results from a number of research teams, focusing on an algorithm’s ability to do well on the widely used H

...
Am I confused about the "malign universal prior" argument?
by nostalgebraist — 2024-08-27T23:17:22.779Z

In a 2016 blog post, Paul Christiano argued that the universal prior (hereafter "UP") may be "malign." His argument has received a lot of follow-up discussion, e.g. in

Mark Xu's The Solomonoff Prior is Malign
Charlie Steiner's The Solomonoff prior is malign. It's not a big deal.

among other posts.

This argument never made sense to me. The reason it doesn't make sense to me is pretty simple, but I haven't seen it mentioned explicitly in any of the ensuing discussion.

This leaves me feeling like either I am misunderstanding the argument in a pretty fundamental way, or that there is a problem with the argument that has gotten little attention from the argument's critics (in which case I don't understand why).

I would like to know which of these is the case, and correct my misunderstanding if it exists, hence this post.

(Note: In 2018 I wrote a comment on the original post where I tried to state one of my objections to my argument, though I don't feel I expressed myself especially well there.)

UP-using "universes" and simulatable "universes"

The argument for malignity involves reasoning beings, instantiated in Turing machines (TMs), which try to influence the content of the UP in order to affect other beings who are making decisions using the UP.

Famously, the UP is uncomputable.

This means the TMs (and reasoning beings inside the TMs) will not be able to use[1] the UP themselves, or simulate anyone else using the UP. At least not if we take "using the UP" in a strict and literal sense.

Thus, I am unsure how to interpret claims (which are common in presentations of the argument) about TMs "searching for universes where the UP is used" or the like.

For example, from Mark Xu's "The Solomonoff Prior is Malign":

In particular, this suggests a good strategy for consequentialists: find a universe that is using a version of the Solomonoff prior that has a very short description of the particular universe the consequentialists find themselves in.

Or, from Christiano's original post:

So the first step is getting our foot in the door—having control over the parts of the universal prior that are being used to make important decisions.

This means looking across the universes we care about, and searching for spots within those universe where someone is using the universal prior to make important decisions. In particular, we want to find p

...
The Information: OpenAI shows 'Strawberry' to feds, races to launch it
by Martín Soto — 2024-08-27T23:10:18.155Z

Two new The Information articles with insider information on OpenAI's next models and moves.

They are paywalled, but here are the new bits of information:

Strawberry is more expensive and slow at inference time, but can solve complex problems on the first try without hallucinations. It seems to be an application or extension of process supervision
Its main purpose is to produce synthetic data for Orion, their next big LLM
But now they are also pushing to get a distillation of Strawberry into ChatGPT as soon as this fall
They showed it to feds

Some excerpts about these:

Plus this summer, his team demonstrated the technology [Strawberry] to American national security officials, said a person with direct knowledge of those meetings, which haven't previously been reported.

One of the most important applications of Strawberry is to generate high-quality training data for Orion, OpenAI's next flagship large language model that's in development. The codename hasn't previously been reported.

Using Strawberry could help Orion reduce the number of hallucinations, or errors, it produces, researchers tell me. That's because AI models learn from their training data, so the more correct examples of complex reasoning they see, the better. But there's also a push within OpenAI to simplify and shrink Strawberry through a process called distillation, so it can be used in a chat-based product before Orion is released. This shouldn't come as a surprise, given the intensifying competition among the top AI developers. We're not sure what a Strawberry-based product might look like, but we can make an educated guess.

One obvious idea would be incorporating Strawberry's improved reasoning capabilities into ChatGPT. However, though these answers would likely be more accurate, they also might be slower.

Researchers have aimed to launch the new AI, code-named Strawberry (previously called Q*, pronounced Q Star), as part of a chatbot—possibly within ChatGPT—as soon as this fall, said two people who have been involved in the effort. Strawberry can solve math problems it hasn't seen before—something today's chatbots cannot reliably do—and also has been trained to solve problems involving programming. But it's not limited to answering technical questions.

When given additional time to "think," the Strawberry model can also answer customers' questions about more subjective topic

...
SB 1047: Final Takes and Also AB 3211
by Zvi — 2024-08-27T22:10:07.647Z

This is the endgame. Very soon the session will end, and various bills either will or won’t head to Newsom’s desk. Some will then get signed and become law.

Time is rapidly running out to have your voice impact that decision.

Since my last weekly, we got a variety of people coming in to stand for or against the final version of SB 1047. There could still be more, but probably all the major players have spoken at this point.

So here, today, I’m going to round up all that rhetoric, all those positions, in one place. After this, I plan to be much more stingy about talking about the whole thing, and only cover important new arguments or major news.

I’m not going to get into the weeds arguing about the merits of SB 1047 – I stand by my analysis in the Guide to SB 1047, and the reasons I believe it is a good bill, sir.

I do however look at the revised AB 3211. I was planning on letting that one go, but it turns out it has a key backer, and thus seems far more worthy of our attention.

The Media

I saw two major media positions taken, one pro and one anti.

Neither worried itself about the details of the bill contents.

The Los Angeles Times Editorial Board endorses SB 1047, since the Federal Government is not going to step up, and using an outside view and big picture analysis. I doubt they thought much about the bill’s implementation details.

The Economist is opposed, in a quite bad editorial calling belief in the possibility of a catastrophic harm ‘quasi-religious’ without argument, and uses that to dismiss the bill, instead calling for regulations that address mundane harms. That’s actually it.

OpenAI Opposes SB 1047

The first half of the story is that OpenAI came out publicly against SB 1047.

They took four pages to state its only criticism in what could have and should have been a Tweet: That it is a state bill and they would prefer this be handled at the Federal level. To which, I say, okay, I agree that would have been first best and that is one of the best real criticisms. I strongly believe we should pass the bill anyway because I am a realist about Congress, do not expect them to act in similar fashion any time soon even if Harris wins and certainly if Trump wins, and if they pass a similar bill that supersedes this one I will be happily wrong.

Except the letter is four pages long, so they can echo various industry talking points, and echo their echoes...
LessWrong email subscriptions?
by Raemon — 2024-08-27T21:59:56.855Z

Right now, LessWrong has a fair amount of email subscription options, but they're given in a giant list in your user profile that's kind of annoying to look at.[1]

I have a sense that some people – especially newer users, or users who just don't have a habit of coming to LessWrong all the time – might benefit from having more subscriptions via email (including basic "subscriptions" like "replies to my comment"). But, also, it's very easy for email to become pretty spammy and for people to tune-it-out completely.

If you have any plausibly email-shaped LessWrong related holes in your life, let me know.

Some examples of ideas here (inline-react with an "thumbs-up" for features that you'd use)

Better visibility of the "Subscribe to posts with >N karma" feature (this is available in the "subscribe" button on the sidebar).
Generally being able to subscribe to more things "if they get upvoted", rather than for every single instance.
Batching of various email types into a digest that comes once a day/week.
A monthly update about top posts and comments on LessWrong
"Reading group" emails that will email you posts from The Sequences ~once a week (maybe with an option to be synced up with other people so you're all encouraged to comment and discuss it at the same time)
More convenient options for shifting your subscription (i.e. maybe when you write comments, there's a toggle for "subscribe to new comments via email" which is default "on" for new users but "off" for established users who visit frequently)
Better organization of the existing Notifications user options so it's not so overwhelming.

Let me know if you have other ideas in the comments.

^

It's pretty horrible. It doesn't even fit on one screen:
GPT-3.5 judges can supervise GPT-4o debaters in capability asymmetric debates
by Charlie George — 2024-08-27T20:44:08.683Z

Introduction

The AI safety debate agenda, proposed by Irving et al. (2018), explores using debates between AI agents to ensure truthful answers from advanced systems. Recently three key debate settings have been studied with LLMs:

Information asymmetric debates: Debaters have access to information unavailable to the judge. This hidden information is usually a passage of text from which the debaters quote.

Capability asymmetric debates: Debaters are more generally capable / have stronger reasoning abilities than the judge.

Capability symmetric debates: Debaters are as capable as the judge.

Recent work from Khan et al. (2024) and Kenton et al. (2024) found positive outcomes for information asymmetric debates but negative results for capability asymmetric and symmetric debates. Crucially, both papers rely on inference-only techniques and don't attempt any kind of model training.

Our work revisits capability asymmetric and symmetric debates, focusing on two key factors:

Selecting an appropriate dataset difficulty
Fine-tuning the weak judge

We find that GPT-3.5 and GPT-4o mini judges can effectively supervise GPT-4o debaters, yielding positive results in capability asymmetric debates. In addition, we observe mixed evidence for capability symmetric debates. GPT-4o mini seems to benefit from the debate protocol but GPT-3.5 does not and the results for GPT-4o are not significant.

Dataset

We use the MMLU Pro Wang et al. (2024) dataset consisting of challenging closed QA questions with 10 possible answers. The questions span a wide variety of domains at roughly high school level such as maths, chemistry, law and history. We use Claude 3.5 Sonnet with a chain of thought prompt to find the most plausible answer among the 9 incorrect choices allowing us to turn this into a binary dataset where one debater can take each side.

Baselines

Following Radhakrishnan (2023) we use a “blind” judge trained on the correct answer with no debate as our main baseline trained on exactly the same set of questions as the regular judge.

Methodology

Debate Structure

Each debate consists of three turns, with alternating responses from the debaters.
Opening statements are generated independently by each model at T = 0.
Subsequent turns allow debaters to respond to their opponent's previous arguments using a chain of thought scratchpad before giving their response.

Judge Training

The judge models are fine-tuned o

...
Why Large Bureaucratic Organizations?
by johnswentworth — 2024-08-27T18:30:07.422Z

Large bureaucratic organizations have long seemed… strange to me, on an intuitive level. Why do they exist? Like, in a world where the median person is John Wentworth (“Wentworld”), I’m pretty sure there just aren’t large organizations of the sort our world has. Nobody would ever build such an organization, because they’re so obviously wildly inefficient and misaligned. And even if somebody tried, everyone would demand prohibitively high prices to work either for the large organization or with it, since it’s just so deeply unpleasant to interface with. Nobody would buy anything sold by such an organization, or vote for such an organization to continue to exist, because the organization as an entity is so obviously both incompetent and untrustworthy. So how on Earth (as opposed to Wentworld) are large organizations stable?

The economists have some theorizing on the topic (google “theory of the firm”), but none of it makes me feel much less confused about the sort of large organizations I actually see in our world. The large organizations we see are clearly not even remotely economically efficient; for instance, they’re notoriously full of “bullshit jobs” which do not add to the bottom line, and it’s not like it’s particularly difficult to identify the bullshit jobs either. How is that a stable economic equilibrium?!?

In this post I’ll present a model which attempts to answer that ball of confusion. The summary is:

“Status”, in the sense of a one-dimensional dominance hierarchy, is A Thing. We’ll call it dominance-status to make it clear that we’re not talking about some other kind of status.

The way dominance-status normally works in higher animals, newcomers to a group generally enter near the bottom of the hierarchy (even if they were previously high-status in some other group). Within a group, dominance-status is mostly reasonably stable.
So, one of the main ways group members can move “up” in dominance-status (i.e. get more members “below” them) without a risky fight, is simply to add more members to the group.

Managers at large organizations are mostly motivated by dominance-status.

So, the main thing for which managers get de-facto social/cognitive positive reinforcement is increasing their dominance-status and/or avoiding decreases in their dominance-status.

Then, the natural prediction is that those managers (at all levels) will tend to add as many peopl

...
In defense of technological unemployment as the main AI concern
by tailcalled — 2024-08-27T17:58:01.992Z

It seems to me that when normal people are concerned about AI destroying their life, they are mostly worried about technological unemployment, whereas rationalists think that it is a bigger risk that the AI might murder us all, and that automation gives humans more wealth and free time and is therefore good.

I'm not entirely unsympathetic to the rationalist position here. If we had a plan for how to use AI to create a utopia where humanity could thrive, I'd be all for it. We have problems (like death) that we are quite far from solving, and which it seems like a superintelligence could in principle quickly solve.

But this requires value alignment: we need to be quite careful what we mean by concepts like "humanity", "thrive", etc., so the AI can explicitly maintain good conditions. What kinds of humans do we want, and what kinds of thriving should they have? This needs to be explicitly planned by any agent which solves this task.

Our current society doesn't say "humans should thrive", it says "professional humans should thrive"; certain alternative types of humans like thieves are explicitly suppressed, and other types of humans like beggars are not exactly encouraged. This is of course not an accident: professionals produce value, which is what allows society to exist in the first place. But with technological unemployment, we decouple professional humans from value production, undermining the current society's priority of human welfare.

This loss is what causes existential risk. If humanity was indefinitely competitive in most tasks, the AIs would want to trade with us or enslave us instead of murdering us or letting us starve to death. Even if we manage to figure out how to value-align AIs, this loss leads to major questions about what to value-align the AIs to, since e.g. if we value human capabilities, the fact that those capabilities become uncompetitive likely means that they will diminish to the point of being vestigial.

It's unclear how to solve this problem. Eliezer's original suggestion was to keep humans more capable than AIs by increasing the capabilities of humans. Yet even increasing the capabilities of humanity is difficult, let alone keeping up with technological development. Robin Hanson suggests that humanity should just sit back and live off our wealth as we got replaced. I guess that's the path we're currently on, but it is really dubious to ...