trentmkelly/LessWrong-43k huggingface dataset:
Existential AI Safety is NOT separate from near-term applications
by scasper — 2022-12-13T14:47:07.120Z

Stephen Casper,

scasper@mit.edu

. Thanks to Alex Lintz and Daniel Dewey for feedback.

This is a reply but not an objection to a recent post from Paul Christiano titled

AI alignment is distinct from its near term applications

. The post is fairly brief and the key point is decently summed up by this excerpt.

I worry that companies using alignment to help train extremely conservative and inoffensive systems could lead to backlash against the idea of AI alignment itself. If such systems are held up as key successes of alignment, then people who are frustrated with them may end up associating the whole problem of alignment with “making AI systems inoffensive.”

I have no disagreements with this claim. But I would push back against the general notion that AI [existential] safety work is disjoint from near term applications. Paul seems to agree with this.

We can develop and apply alignment techniques to these existing systems. This can help motivate and ground empirical research on alignment, which may end up helping avoid higher-stakes failures like an AI takeover.

This post argues for strongly emphasizing this point.

What do I mean by near-term applications? Any challenging problem involving consequential AI and society. Examples include:

Self-driving cars
Recommender systems
Search engines
AI weapons
Cybersecurity
Unemployment
Bias/fairness/justice involving systems that work with humans or human data
Misuses of media generators including text and image generators

I argue that working on these problems probably matters a lot for three reasons. The second and third of which are potential matters of existential safety.

Non X-risks from AI are still intrinsically important AI safety issues.

There are many important non X-risks in this world, and any altruistic-minded person should care about them. For the same reason we should care about health, wealth, development, and animal welfare, we should also care about making important near-term applications of narrow AI go well for people.

There are valuable lessons to learn from near-term applications.

Imagine that we figured out ways to make near-term applications of AI go very well. I find it incredibly hard to imagine a world in which we did any of these things without developing a lot of useful technical tools and governance strategies that could be retooled or built on for higher-stakes...
What is the correlation between upvoting and benefit to readers of LW? 
by banev — 2022-12-13T14:26:09.664Z

Do posts with more upvotes actually provide more (real) value to community members? Is there any evidence to support this hypothesis? How this could be measured at all?

Regarding confirmation bias: personally, as a LessWrong member, are you more likely to click on an upvote for posts you agree with or for posts you disagree with? Or to the post interesting, but not questioning your assumptions and your map of the reality? To be honest? Could you count your upvotes/downvotes during last, I don't know, year, and check the ratio?

And do the posts you initially agree with really make you less wrong?

There are two different criteria for comments (like and agree), but only one for posts (like). Perhaps there is room for improvement here.

Below is the letter from (supposedly) admins (italic emphasis is mine):

P.S. to admins: No, I will not pay attention to what content gets upvoted, sorry. Also I will not "invest in improving my output" specifically to moderators of LW, as I already do it on a constant basis.

Quick note to say that your recent posts and comments haven't been up to LessWrong's standard, and I've rate limited you to posting once per day. To get a better sense of the site's norms, I'd recommend paying attention to what content gets upvoted before writing again. Here are some common issues that are worth specific attention (not that any of these necessarily apply to you):

- quality of reasoning
- ignoring of well-known existing arguments or evidence
- poor formatting, punctuation, or grammar
- low clarity
- inappropriate/hostile tone
- appearing on the site over a "pet topic"

PS: If in a few months you have invested in improving your output, you can message us and share evidence of your progress and we can re-evaluate.
Limits of Superintelligence
by Aleksei Petrenko — 2022-12-13T12:19:15.378Z

Surprisingly, I haven’t seen anything written about the ultimate limits of intelligence. This is a short post where I try to ponder this question.

Superintelligent agents are often portrayed as extremely powerful oracles capable of predicting long-term outcomes of actions and thus steering the future toward their goals. One problem with this understanding is that the future also contains the agent itself, its future actions, and their consequences. Since no system can have a perfect model of self there should be some maximum level of precision and quality of future prediction, unless this future does not depend on the agent’s reaction to events or actions of any other agents of comparable capacity.

This should essentially put some kind of hard limit on the capabilities of any agent, although it might not be possible to define it mathematically like the Bekenstein bound (limit of information capacity) or Landauer's principle (limit of computation density). I also have a gut feeling that this self-referential problem limits max-intelligence much more restrictively than Landauer's or any other physics-based bound.

I don’t think this definition of intelligence limit has a lot to do with AI safety: the result of any runaway self-improvement loop is still going to run circles around the combined brainpower of humanity. Unless there is a way to significantly lower this bound by making future prediction harder? For example, a world with many agents of comparable stature might be much more difficult to predict, thus limiting the potential destructive capabilities of any one agent.

If anyone knows any literature that covers this topic, please let me know in the comments! Also, if any theologists are present, I’m wondering how this kind of limit would apply to theoretical models of God. Another immovable object - unstoppable force paradox?
Last day to nominate things for the Review. Also, 2019 books still exist.
by Raemon — 2022-12-13T08:53:21.114Z

Two quick reminders:

One: You have about one more day to nominate posts! Posts need at least two positive upvotes to make it to phase 2, so check out the voting dashboard to upvote things that seem important to you.

Two: Although the 2020 books won't be shipped until sometime in January (alas), the 2019 books are still available on Amazon, and if you order them now they'll arrive in time for Christmas.

That is all. Happy reviewing.
AI alignment is distinct from its near-term applications
by paulfchristiano — 2022-12-13T07:10:04.407Z

I work on AI alignment, by which I mean the technical problem of building AI systems that are trying to do what their designer wants them to do.

There are many different reasons that someone could care about this technical problem.

To me the single most important reason is that without AI alignment, AI systems are reasonably likely to cause an irreversible catastrophe like human extinction. I think most people can agree that this would be bad, though there’s a lot of reasonable debate about whether it’s likely. I believe the total risk is around 10–20%, which is high enough to obsess over.

Existing AI systems aren’t yet able to take over the world, but they are misaligned in the sense that they will often do things their designers didn’t want. For example:

The recently released ChatGPT often makes up facts, and if challenged on a made-up claim it will often double down and justify itself rather than admitting error or uncertainty (e.g. see here, here).
AI systems will often say offensive things or help users break the law when the company that designed them would prefer otherwise.

We can develop and apply alignment techniques to these existing systems. This can help motivate and ground empirical research on alignment, which may end up helping avoid higher-stakes failures like an AI takeover. I am particularly interested in training AI systems to be honest, which is likely to become more difficult and important as AI systems become smart enough that we can’t verify their claims about the world.

While it’s nice to have empirical testbeds for alignment research, I worry that companies using alignment to help train extremely conservative and inoffensive systems could lead to backlash against the idea of AI alignment itself. If such systems are held up as key successes of alignment, then people who are frustrated with them may end up associating the whole problem of alignment with “making AI systems inoffensive.”

If we succeed at the technical problem of AI alignment, AI developers would have the ability to decide whether their systems generate sexual content or opine on current political events, and different developers can make different choices. Customers would be free to use whatever AI they want, and regulators and legislators would make decisions about how to restrict AI. In my personal capacity, I have views on what uses of AI are more or less beneficial and what ...
Take 10: Fine-tuning with RLHF is aesthetically unsatisfying.
by Charlie Steiner — 2022-12-13T07:04:35.686Z

As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day for 25 days. Or until I run out of hot takes. This take owes a lot to the Simulators discussion group.

Fine-tuning a large sequence model with RLHF creates an agent that tries to steer the sequence in rewarding directions. Simultaneously, it breaks some nice properties that the fine-tuned model used to have. You should have a gut feeling that we can do better.

When you start with a fresh sequence model, it's not acting like an agent, instead it's just trying to mimic the training distribution. It may contain agents, but at every step it's just going to output a probability distribution that's been optimized to be well-calibrated. This is a really handy property - well-calibrated conditional inference is about as good as being able to see the future, both for prediction and for generation.

The design philosophy behind RLHF is to train an agent that operates in a world where we want to steer towards good trajectories. In this framing, there's good text and bad text, and we want the fine-tuned AI to always output good text rather than bad text. This isn't necessarily a bad goal - sometimes you do want an agent that will just give you the good text. The issue is, you're sacrificing the ability to do accurate conditional inference about the training distribution. When you do RLHF fine-tuning, you're taking a world model and then, in-place, trying to cannibalize its parts to make an optimizer.

This might sound like hyperbole if you remember RL with KL penalties is Bayesian inference. And okay; RLHF weights each datapoint much more than the Bayesian inference step does, but there's probably some perspective in which you can see the fine-tuned model as just having weird over-updated beliefs about how the world is. But just like perceptual control theory says, there's no bright line between prediction and action. Ultimately it's about what perspective is more useful, and to me it's much more useful to think of RLHF on a language model as producing an agent that acts in the world of text, trying to steer the text onto its favored trajectories.

As an agent, it has some alignment problems, even if it lives totally in the world of text and doesn't get information leakage from the real world. It's trying to get to better trajectories by any means necessary, even if...
Are lawsuits against AGI companies extending AGI timelines?
by SlowingAGI — 2022-12-13T06:00:31.504Z

More precisely: Are copyright lawsuits against companies developing large language model slowing down the creation of AGI?

For example, there's this ongoing lawsuit: https://githubcopilotlitigation.com/.

You could also imagine this kind of lawsuits extended to other groups whose data are used, such as artists.
EA & LW Forums Weekly Summary (5th Dec - 11th Dec 22')
by Zoe Williams — 2022-12-13T02:53:29.254Z

Supported by Rethink Priorities

This is part of a weekly series summarizing the top posts on the EA and LW forums - you can see the full collection here. The first post includes some details on purpose and methodology. Feedback, thoughts, and corrections are welcomed.

If you'd like to receive these summaries via email, you can subscribe here.

Podcast version: prefer your summaries in podcast form? A big thanks to Coleman Snell for producing these! Subscribe on your favorite podcast app by searching for 'EA Forum Podcast (Summaries)'. More detail here.

Top / Curated Readings

Designed for those without the time to read all the summaries. Everything here is also within the relevant sections later on so feel free to skip if you’re planning to read it all. These are picked by the summaries’ author and don’t reflect the forum ‘curated’ section.

Some observations from an EA-adjacent (?) charitable effort

by patio11

The author founded VaccinateCA, which ran USA vaccine location infrastructure for 6 months. They think this saved many thousands of lives, at a cost of ~$1.2M.

Learnings from this include:
1. Enabling trade as a mechanism for impact:

Many actors (eg. Google, the White House, pharmacists) wanted residents vaccinated, but weren’t working together.
By getting data via calling pharmacies and convincing a few people at Google to show it, they were able to do something very impactful without being hired to.

2. Engaging with the system:

Many people in policy are skeptical of tech.
501c3 charity status is extremely useful for reputation and funding.
Carefully planned comms created initial positive press, which in turn was a key reason Google and government stakeholders engaged with them.

3. Factors in the team’s success:

Money to commit early and boldly.
Social capital to call in favors in the tech community / get resources.
Good understanding of the infrastructure of the project (call centers and information flows of IT systems).
Ability to write good software quickly.
PR experience and talent.
A network with lots of high agency people willing to try things.
People management skills (they didn’t have this, but it would have been very helpful)

While this was not an “EA Project” the author expects it wouldn’t have happened if EA hadn’t drawn their attention to expected value math, and is grateful for this.

Why development aid is a really exciting field

by ...
Alignment with argument-networks and assessment-predictions
by Tor Økland Barstad — 2022-12-13T02:17:00.531Z

Here is a simplistic diagram of an argument-network. Real-world examples would typically be larger, more interconnected, and not shaped like trees.

Here are some of the concepts that will be touched upon in this post:

Argument networks: Imagine rigorous and extensive argumentation being organized into a network, where each node contains one "piece" of the argument. If you agree with the contents of each node, you have agreed with every step of the argumentation as a whole.
Reviewer-assessment predictions: The techniques described in this post suppose that we have systems that can make accurate predictions for how various human reviewers will answer questions. Some questions could be aimed at ascertaining the degree to which humans agree with the argumentation itself. Other questions could try to ascertain the degree to which the argumentation fits various descriptions (e.g. whether certain specific standards and rules are upheld). The systems that predict what human reviewers will answer don't need to be perfect, but a certain minimum level of accuracy and precision would be needed.
Testing of reviewer-assessment predictions: A positive thing with predictions of reviewer-assessments is that accuracy can be tested. However, there are some security concerns relating to this.
Argument-network score-functions: Argument networks would receive a score. The algorithm that calculates the score would take a variety of things into account. One of the important factors would be the level of agreement that is predicted for the individual nodes. A good score-function might forbid types of arguments (or ways of presenting arguments) that make it easy to systematically fool human reviewers. It could make it so that there are only a few "types" of arguments that are allowed, with only one legal way of presenting each "type".
Wiggle room (relative to score-function): Given some specific score-function, are AIs able to make argument-networks that argue in favor of contradictory claims, in such a way that both argument-networks get a high score? If so, that would indicate that the score-function has wiggle room.
Score-function exploration: Human operators may define a "space" of possible score-functions, and get help from AIs to search within that space for score-functions that have positive attributes (low wiggle room, etc). When there is wiggle room, AIs should be

...
Revisiting algorithmic progress
by Tamay — 2022-12-13T01:39:19.264Z
Link: https://arxiv.org/abs/2212.05153

How much progress in ML depends on algorithmic progress, scaling compute, or scaling relevant datasets is relatively poorly understood. In our paper, we make progress on this question by investigating algorithmic progress in image classification on ImageNet, perhaps the most well-known test bed for computer vision.

Using a dataset of a hundred computer vision models, we estimate a model—informed by neural scaling laws—that enables us to analyse the rate and nature of algorithmic advances. We use Shapley values to produce decompositions of the various drivers of progress computer vision and estimate the relative importance of algorithms, compute, and data.

Our main results include:

Every nine months, the introduction of better algorithms contributes the equivalent of a doubling of compute budgets. This is much faster than the gains from Moore’s law; that said, there's uncertainty (our 95% CI spans 4 to 25 months)

Pareto frontiers for training models to achieve the performance of well-known models over time.

Roughly, progress in image classification has been ~45% due to the scaling of compute, ~45% due to better algorithms, ~10% due to scaling data

Attribution of progress to algorithmic progress, compute scaling and data scaling between model pairs based on Shapley decomposition. “NS” indicates that there was no scaling of the relevant input between these models. Numbers may not all add up to 100 due to rounding.

The majority (>75%) of algorithmic progress is compute-augmenting (i.e. enabling researchers to use compute more effectively), a minority of it is data-augmenting

Shares of algorithmic progress that is compute- vs. data-augmenting.

In our work, we revisit a question previously investigated by Hernandez and Brown (2020), which had been discussed on LessWrong by Gwern, and Rohin Shah. Hernandez and Brown (2020) re-implement 15 open-source popular models and find a 44-fold reduction in the compute required to reach the same level of performance as AlexNet, indicating that algorithmic progress outpaces the original Moore’s law rate of improvement in hardware efficiency, doubling effective compute every 16 months.

A problem with their approach is that it is sensitive to the exact benchmark and threshold pair that one chooses. Choosing easier-to-achieve thresholds makes algorithmic improvements look less significant, as the scaling of compu...
An exploration of GPT-2's embedding weights
by Adam Scherlis — 2022-12-13T00:46:17.898Z

I wrote this doc in December 2021, while working at Redwood Research. It summarizes a handful of observations about GPT-2-small's weights -- mostly the embedding matrix, but also the LayerNorm gain parameters -- that I found while doing some open-ended investigation of the model. I wanted to see how much I could learn by studying just those parameters, without looking at the attention layers, MLP layers, or activations.

This is still mostly unedited. Feedback and questions are very welcome.

Embedding matrix glossary

Latent space – 768-channel-dim space that activations between blocks (and in skip connections) live in.
Token space – 50,257-token-dim space of one-hot token vectors
[Token] embedding matrix – W of shape [768, 50257] (this is the transpose of `model.lm_head.weight`)
Embedding vector – Any 768-dimensional column of W
Embedding space – Latent space

Preferred channel basis from LayerNorm

This section involves more math than the rest; you can skip it without missing much. "Preferred basis" means the same thing as "privileged basis" as defined by Anthropic here.

Most of GPT-2’s architecture – the attention layers, fully-connected layers and residual connections – don’t impose a preferred basis on the latent space.

(The skip connections give a canonical map between the latent spaces in different layers, so that it makes sense to talk about “the” latent space globally.)

The LayerNorm layers actually do privilege a basis, though, for two reasons.

LayerNorm(x) is defined as follows, where w and b are learned parameters:

yi=∑j(Idij−1i1j)xj(subtract mean; 1 is an all-ones vector)
zi=yi/√∑jyjyj (normalize variance)
LayerNorm(x)i=wizi−bi (apply bias and gain parameters)

The first line picks out part of a preferred basis – a preferred vector (and the subspace orthogonal to it), namely the all-ones vector 1.

The second line doesn’t pick out a basis, because the stdev of the components of y is just its L2 norm and normalizing a vector is independent of (orthonormal) basis. (It does impose a preferred inner product, but we had that already: GPT-2’s unembedding matrix is the transpose of the embedding matrix, and W^T W is a matrix of inner products.)

The third line is the interesting one. The gain vector w is multiplied elementwise with z, which breaks basis-independence. (You could also write this as D_ij z_j where D is a diagonal matrix, D_ii = w_i.)...