trentmkelly/LessWrong-43k huggingface dataset:
Announcing Dialogues
by Ben Pace — 2023-10-07T02:57:39.005Z

As of today, everyone is able to create a new type of content on LessWrong: Dialogues.

In contrast with posts, which are for monologues, and comment sections, which are spaces for everyone to talk to everyone, a dialogue is a space for a few invited people to speak with each other.

I'm personally very excited about this as a way for people to produce lots of in-depth explanations of their world-models in public.

I think dialogues enable this in a way that feels easier — instead of writing an explanation for anyone who reads, you're communicating with the particular person you're talking with — and giving the readers a lot of rich nuance I normally only find when I overhear people talk in person.

In the rest of this post I'll explain the feature, and then encourage you to find a partner in the comments to try it out with.

What do dialogues look like?

Here is a screenshot of a dialogue with 3 users.

Behind the scenes, the editor interface is a bit different from other editors we've seen on the internet. It's inspired by collaborating in google docs, where you're all editing a document simultaneously, and you can see the other person's writing in-progress.

You can also submit a response with Cmd-Enter / Ctrl-Enter.

This also allows all participants to draft thoughtful replies simultaneously, and be able to see what the other person is planning to talk about next.

How do I create a dialogue?

First, hit the "New Dialogue" button in the menu in the top-right of the website.

This pops up a box where you are invited to give your dialogue a title and invite some people.

Then you'll be taken to the editor page!

Now you can start writing. The other dialogue participants will receive a notification and the dialogue will appear in their drafts. (Only invite people to dialogues with their consent!)

What are some already published dialogues?

Here are some links to published dialogues that have been using this feature.

Thomas Kwa's MIRI research experience (by Thomas Kwa, peterbarnett, Vivek Hebbar, Jeremy Gillen, jacobjacob & Raemon)
Navigating an ecosystem that might or might not be bad for the world (by habryka & kave)
Joseph Bloom on choosing AI Alignment over bio, what many aspiring researchers get wrong, and more (interview) (by Ruby & Joseph Bloom)
What is the optimal frontier for due diligence? (by RobertM & Ruby)
Feedback-loops, Deliberate Practice, and Tran

...
Don't Dismiss Simple Alignment Approaches
by Chris_Leong — 2023-10-07T00:35:26.789Z

I remember that when I first started in alignment there was this belief that in order to make alignment progress, you needed to be a genius. With the rise of interpretability, this expectation has moderated somewhat, but I still think it has influence and often leads people to overcomplicate.

Many recent alignment breakthroughs have been found by keeping it simple[1]:

Lie Detection:
• Contrast-consistent search (better known as discovering latent knowledge[2]) reads the truth out of a model without looking at the outputs, simply by using a linear probe with only two elements in its activation function[3].
• Pacchiardi, Owain and others use a logistic regression classifier[4] to predict whether a mis al was truthful from its answers to further questions. They were able to demonstrate this using three different kinds of questions and the results even generalised across models. This technique seems like it could be useful in practise, even if it doesn't scale all the way to superintelligence.
Superposition: Dictionary learning makes substantial progress on superposition by simply using a one hidden layer multi-layer perceptron as a sparse auto-encoder with L2 reconstruction loss and a L1 loss on hidden layer activations. Anthropic actually chose to avoid more sophisticated methods, as they were worried they might recover features that the model doesn't actually utilise.
Model-steering: Activation Engineering: Activation addition allows us to steer models by simply performing a forward pass for the concept you want to activate, saving the activations of a particular layer as a vector and adding them to future forward pass. See also Turntrout's prior work on steering a mouse by subtracting a cheese vector[5] and inference-time intervention[6].[7]
World model location: Neel Nanda found a linear representation in Othello-GPT. This one was a bit trickier: a previous paper had failed to recover it using linear probes. However, Neel managed to do this by searching for a representation of "my color" and by applying a bunch of other tweaks. See also Wes Gurnee's work with Max Tegmark, which used linear probes to find linear representations of latitude and longitude; in addition to one representing time[8].
Scalable Interpretability: OpenAI used a language model in order to label neurons based on the activations for some relevant examples using few-shot prompting...
Linking Alt Accounts
by jefftk — 2023-10-06T17:00:09.802Z

Sometimes people have something they want to say without it being
traceable back to their main identity, and on internet forums it's
common for people to use multiple accounts ("alts") for this. As
machine learning software gets better, however, it becomes
increasingly practical to link a person's accounts.

A few months ago someone ran a simple stylometry tool across Hacker
News comments, and identified many alts, including one for the
site's founder. To further demonstrate that this isn't just an
academic consideration, I recently did the same for LessWrong and the
EA Forum. I'm not going to share the code or the probabilities it
generated, and I've only looked at the output enough to be pretty sure
it's working. Trained on half of the data and tested on the other
half it was consistently able to link accounts, however, and it was
also able to identify at least one non-public alt account I already
knew about.

This is an example of a general problem with privacy: even if
something seems careful enough now, you can't trust the future to
keep things private.

(If you do want somewhat more protection now, however, I think best
practice is running your alt comments through an LLM to change the
style.)

Comment via: facebook, mastodon
Super-Exponential versus Exponential Growth in Compute Price-Performance
by moridinamael — 2023-10-06T16:23:56.714Z

In recent months and years I have seen sober analyses of compute price-performance suggesting that the price-performance in computing (that is, the amount of calculations per second that you can buy for a dollar) has a doubling time of something like 2-3 years. I do not think these figures are good predictors of future expectations, and I wish to explain why.

Over the years I have often returned to Kurzweil's[1] plot of price-performance in the 20th century. I occasionally update the plot on my own and marvel that the trend has persisted essentially unabated since it was published, illustrating a continuous and consistent trend from 1900 through 2023. For your reference and for the sake of clarity I have taken the original plot and added one recent point, the AMD RX 7600 GPU, which boasts 21.4 TFLOP/s (single-precision) at a price point of $269.99 as of this week. Take my word for it that the plots between 1995 and 2023 remain essentially on-trend.

This plot[2] has no "doubling time" because it is super-exponential, i.e. there is an exponent inside the exponent, and the effective doubling time gets shorter over time. I have not found any published reference to how the white dashed band is calculated, but my own best fit line is:

C(FLOP/s$1000)=10−7⋅exp(6.43⋅exp(0.01602⋅t))

where C is price-performance of compute in FLOP/s per $1000 and t in this case is years-since-1900. The instantaneous doubling time for this trend as of today would be about 0.93 years, less than half of even the most Pollyannaish of the recent forecasts. And the instantaneous doubling time obviously gets shorter each year.

The discrepancy between this <1 year doubling time and the >2 year doubling time observed in more recent publications is explained by the fact that trends calculated on the basis of narrow, recent time-frames will only capture one paradigm, e.g. the "GPU paradigm", which, like all individual paradigms in technology, exhibits S-curve behavior, starting out slow, accelerating, and then flattening. I also note that serious authors tend to present doubling-time figures that lean in the pessimistic direction.

Of course, it is entirely possible that 2023 marks the end of the validity of the above super-exponential trend. Perhaps, for some reason, no new computing paradigm arises to put us back on the white dashed band.

I feel it is important to consider that pr...
A personal explanation of ELK concept and task. 
by Zeyu Qin — 2023-10-06T03:55:44.895Z

Here, I tend to give my personal explanation of ELK concept and task based on the terms that have already emerged (Still in progress.).
The original statement of ELK: Given input samples set X (we are interested in), targeted model M, and task (goal) G, could we obtain a reporter or direct translator to honestly tell us the internal mechanism of M on G given X?
Translating into the existing terms: obtaining faithful model distillation on specific task or property.
faithful == honest, reporter == surrogate model (distilled or not)
The Long-Term Future Fund is looking for a full-time fund chair
by Linch — 2023-10-05T22:18:53.720Z
Link: https://forum.effectivealtruism.org/posts/fuQjsQPx8DpJma2TE/the-long-term-future-fund-is-looking-for-a-full-time-fund

EDIT 2023/11/21

We have finished evaluating the first batch of our applications.

As we have not yet finished the hiring process, people are still welcome to apply to us. We are still very likely to skim future applications, however we have no firm commitments to do so.

Candidates are welcome to message me either here or in other channels to flag their late applications with us.

The Long-Term Future Fund is looking for a full-time fund chair. You can apply

here

.

Summary

The Long-Term Future Fund is seeking a new full-time fund chair to lead strategy, fundraising, management, and grant evaluation.
The chair will articulate a vision for the fund, coordinate stakeholders, improve processes, represent the fund, and make final decisions.
The role offers a competitive salary with location flexibility, starting with a 3-month trial period.
The fund aims to distribute $5M-$15M (last year we distributed over $10M) annually to reduce existential risk, especially from AI. We expect the LTFF chair to make substantive contributions to our strategy and fundraising efforts.

Applications

are open now through October 21st.

The form might take up to an hour, including screening questions

Why is LTFF looking for a fund chair?

The Long-Term Future Fund (LTFF) has had a significant impact on the long-termist and AI safety funding ecosystems:

We are one of the few significant sources of funding for longtermist or AI safety funding, allowing (some) worldview diversification away from Open Phil.
In 2022, LTFF was operationally able to distribute 12 million dollars to over 250 small projects. This entails clearing nontrivial logistical hurdles in following nonprofit law across multiple countries, consistent operational capacity, and a careful eye towards

downside risk mitigation

.
We have the largest “always open” application form, where anybody can apply, connecting funding with a wide range of excellent grantees without pre-existing networks.
We are one of the primary donation options for new donors in longtermism, with a relatively well-known brand and accessibility.
We believe we are able to fund a wide range of excellent small longtermist projects, and increase grant capacity in the ecosystem overall.
We have contributed to improvements in the epistemics and transparency around the longtermist funding ecosystem, e.g. by writing very detailed and frequently viewed p

...
Provably Safe AI
by PeterMcCluskey — 2023-10-05T22:18:26.013Z
Link: https://bayesianinvestor.com/blog/index.php/2023/10/05/provably-safe-ai/

I've been hearing vague claims that automated theorem provers are able
to, or will soon be able to prove things about complex software such as
AIs.

Max Tegmark and Steve Omohundro have now published a paper, Provably
Safe Systems : The Only Path To Controllable
AGI, which convinces me that this is
a plausible strategy to help with AI safety.

Ambitions

The basic steps:

Write trustworthy AIs that are capable of searching for proofs and
verifying them.

Specify safety properties that we want all AIs to obey.

Train increasingly powerful Deep Learning systems.

Use Mechanistic Interpretability tools to translate knowledge from
deep learning systems into more traditional software that's more
transparent than neural nets.

Use the AIs from the first step to prove that the results have those
safety properties.

Require that any hardware that's sufficiently powerful to run
advanced AI be provably constrained to run only provably safe
software.

Progress in automated theorem proving has been impressive. It's
tantalizingly close to what we need to prove interesting constraints on
a large class of systems.

What Systems Can We Hope to Prove Properties about?

The paper convinced me that automated proof search and verification are
making important progress. My intuition still says that leading AIs are
too complex to prove anything about. But I don't have a strong argument
to back up that intuition. The topic is important enough that we ought
to be pursuing some AI safety strategies that have a significant risk of
being impossible. Throwing lots of compute at proving things could
produce unintuitive results.

My main reaction after figuring out the basics of the paper's proposals
was to decide that their strategy made it impossible to train new AIs on
high end GPUs.

Tegmark and Omohundro admit that powerful neural networks seem too messy
to prove much about. Yet they also say that neural networks are a key
step in creating better AIs:

First, note that the seemingly magical power of neural networks comes
not from their ability to execute, but from their ability to
learn. Once training is complete and it is time for execution, a
neural network is merely one particular massively parallel
computational architecture -- and there are many others based on
traditional software that can be similarly efficient.

But neural networks routinely crush the competition when it comes to
learning.

...
Stampy's AI Safety Info soft launch
by steven0461 — 2023-10-05T22:13:04.632Z

Stampy’s AI Safety Info

is a project to create an interactive FAQ about existential risk from AI, started by

Rob Miles

. Our goal is to build a single resource aimed at informing all audiences, whether that means giving them a basic introduction to the concepts, addressing their objections, or onboarding them into research or other useful projects. We currently have 280 answers live on the site, and hundreds more as drafts.

After running two ‘

Distillation

Fellowships

’, in which a small team of paid editors spent three months working to improve and expand the material, we think the site is ready for a soft launch. We’re making this post to invite the collective attention of LessWrong and the EA Forum, hoping that your feedback will help us prepare for a full launch that will use

Rob’s YouTube channel

to reach a large audience.

What we’d like to know

In roughly descending order of priority:

Where are our answers factually or logically wrong, especially in non-obvious ways?
Where are we leaving out key information from the answers?
What parts are hard to understand?
Where can we make the content more engaging?
Where have we made oversights?
What questions should we add?

We’re particularly interested in suggestions from experts on questions and answers related to their area of specialization – please let us know[1] if you’d be interested in having a call where you advise us on our coverage of your domain.

How to leave feedback

Click the edit button in the corner of any answer on

aisafety.info

to go to the corresponding Google doc:

Leave comments and suggestions on the doc.[2] We’ll process these to improve the answers.
To leave general feedback about the site as a whole, you can use

this form

, or comment on this post.

To discuss answers in more depth, or get involved with further volunteer writing and editing, you can join

Rob Miles’s Discord

or look at the

‘Get Involved’ guide on Coda

.

Front end

When exploring the site, you may notice that the front end has room for improvement. We welcome feedback on our

planned redesign

. AIsafety.info is built by volunteer developers – we’re hoping to get a prototype for this redesign working, but if someone reading this is willing to

step up and take the lead on that project

, we’ll achieve this goal faster. There’s also a more in-dept...
Impacts of AI on the housing markets
by PottedRosePetal — 2023-10-05T21:24:03.045Z

Introduction

First of all, some disclaimers.

This is all my opinion. I did not research this topic a lot, especially since I feel like it would always be inadequate and I simply do not have the time to do it properly. It is a very complex topic. It is no financial advice, it is no advice to do anything at all.

The reason I talk about this is that I feel like this is not talked about at all, and it seems like a very obscure field. Initially I wanted to ask about the general opinion on this matter, but the more I thought about it, the less I saw many alternatives the the scenario below for housing. But this is my world view, which almost certainly is lacking, and in the spirit of the forum name I want to correct it.

Unrelated to the post topic, this is my first post here and while I did read HPMOR and started with the sequences and lurked for some time, I still might not get it all right. I would love some feedback if you notice some parts where I argue badly.

I am no expert in well... anything. I have some insights into AI development, I have little insights into the housing market. My personal view based on some sources such as this or other leaks, which results in very aggressive timelines compared to some stuff I have read here before (for example this).

I think we will get public AGI (OpenAIs definition of it) in under 5 years with a very high probability, with a high probability in under 2 years. Similarly I think we will get embodiment AGI in 5 to 10 years.

While this might be up to debate, its not the point of this scenario, which should also be independent of how long those timelines are - albeit with longer timelines introducing more potential disruptions. I will also assume that all AIs in this scenario are successfully aligned and work to do good in general.

About the housing market I think it is artificially inflated to keep prices high due to many people profiting from it. The reasons are complex, I do not claim to know all them. I am unsure about how this will develop, which is the biggest uncertainty in this whole scenario. The market could completely collapse "2008 style" in a few months for all I know. I will try to keep my scenario independent of this, but I dont think I will manage to do so completely.

The Destination

So, first of all, what will the housing market look like in the far future. This...
Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
by Zac Hatfield-Dodds — 2023-10-05T21:01:39.767Z
Link: https://transformer-circuits.pub/2023/monosemantic-features/

Text of post based on our blog post as a linkpost for the full paper which is considerably longer and more detailed.

Neural networks are trained on data, not programmed to follow rules. We understand the math of the trained network exactly – each neuron in a neural network performs simple arithmetic – but we don't understand why those mathematical operations result in the behaviors we see. This makes it hard to diagnose failure modes, hard to know how to fix them, and hard to certify that a model is truly safe.

Luckily for those of us trying to understand artificial neural networks, we can simultaneously record the activation of every neuron in the network, intervene by silencing or stimulating them, and test the network's response to any possible input.

Unfortunately, it turns out that the individual neurons do not have consistent relationships to network behavior. For example, a single neuron in a small language model is active in many unrelated contexts, including: academic citations, English dialogue, HTTP requests, and Korean text. In a classic vision model, a single neuron responds to faces of cats and fronts of cars. The activation of one neuron can mean different things in different contexts.

In our latest paper, Towards Monosemanticity: Decomposing Language Models With Dictionary Learning, we outline evidence that there are better units of analysis than individual neurons, and we have built machinery that lets us find these units in small transformer models. These units, called features, correspond to patterns (linear combinations) of neuron activations. This provides a path to breaking down complex neural networks into parts we can understand, and builds on previous efforts to interpret high-dimensional systems in neuroscience, machine learning, and statistics.

In a transformer language model, we decompose a layer with 512 neurons into more than 4000 features which separately represent things like DNA sequences, legal language, HTTP requests, Hebrew text, nutrition statements, and much, much more. Most of these model properties are invisible when looking at the activations of individual neurons in isolation.

To validate that the features we find are significantly more interpretable than the model's neurons, we have a blinded human evaluator score their interpretability. The features (red) have much higher scores than the neurons (teal).

We additionally ta...