trentmkelly/LessWrong-43k huggingface dataset:
Thoughts on open source AI
by Sam Marks — 2023-11-03T15:35:42.067Z

Epistemic status: I only ~50% endorse this, which is below my typical bar for posting something. I’m more bullish on “these are arguments which should be in the water supply and discussed” than “these arguments are actually correct.” I’m not an expert in this, I’ve only thought about it for ~15 hours, and I didn’t run this post by any relevant experts before posting.

Thanks to Max Nadeau and Eric Neyman for helpful discussion.

Right now there's a significant amount of public debate about open source AI. People concerned about AI safety generally argue that open sourcing powerful AI systems is too dangerous to be allowed; the classic example here is "You shouldn't be allowed to open source an AI system which can

produce step-by-step instructions for engineering

novel pathogens." On the other hand, open source proponents argue that open source models haven't yet caused significant harm, and that trying to close access to AI will result in

concentration

of

power

in the hands of a few AI labs.

I think many AI safety-concerned folks who haven’t thought about this that much tend to vaguely think something like “open sourcing powerful AI systems seems dangerous and should probably be banned.” Taken literally, I think this plan is a bit naive: when we're colonizing Mars in 2100 with the help of our aligned superintelligence, will releasing the weights of GPT-5 really be a catastrophic risk?

I think a better plan looks something like "You can't open source a system until you've determined and disclosed the sorts of threat models your system will enable, and society has implemented measures to become robust to these threat models. Once any necessary measures have been implemented, you are free to open-source."

I'll go into more detail later, but as an intuition pump imagine that: the best open source model is always 2 years behind the best proprietary model (call it GPT-SoTA)[1]; GPT-SoTA is widely deployed throughout the economy and deployed to monitor for and prevent certain attack vectors, and the best open source model isn't smart enough to cause any significant harm without GPT-SoTA catching it. In this hypothetical world, so long as we can trust GPT-SoTA, we are safe from harms caused by open source models. In other words, so long as the best open source models lag sufficiently behind the best proprietary models and we’re smart about how we use our best pr...
Knowledge Base 6: Consensus theory of truth
by iwis — 2023-11-03T13:56:58.437Z

One of the theories defining the truth says that: “(…) truth is whatever is agreed upon, or in some versions, might come to be agreed upon, by some specified group. Such a group might include all human beings, or a subset thereof consisting of more than one person. (…) be agreed upon in an ideal speech situation.” [1]
“In an ideal speech situation, participants would be able to evaluate each other’s assertions solely on the basis of reason and evidence in an atmosphere completely free of any nonrational “coercive” influences, including both physical and psychological coercion. Furthermore, all participants would be motivated solely by the desire to obtain a rational consensus.” [2]

It seems that the method of evaluating correctness of information proposed in the post Knowledge Base 2: The structure and the method of building is close to this definition of truth.

[1] Wikipedia: Truth – Consensus theory
[2] Wikipedia: Ideal speech situation
Shouldn't we 'Just' Superimitate Low-Res Uploads?
by lukemarks — 2023-11-03T07:42:06.824Z

If you have three optimizers which are all very different (one is a neural network, the other was built by hand, and the final one materialized through a Boltzmann Brain-like process) but have identical preferences and an equivalent capacity for optimization, they probably end up doing similar things over a long enough time-scale.

I bring this up, because in discussions of uploading people seem to gravitate toward obtaining a digital encoding of the entire mind, as though the useful part of the upload contains the fact that it optimizes in a human-like way, when it seems like what we actually want is something that optimizes in any way so long as it is optimizing for the things we want, in all their complexity.

In Does Davidad's Uploading Moonshot Work?, @jacobjacob brings up the following:

What about the idea of "just" training a giant transformer to, instead of predicting next tokens in natural language, predicting neural activity?

@lisathiergart replies by saying that this probably violates the assumption that the product would be more aligned than the status quo. I raise the following objections:

The product you need from this training is not so complex as an approximation of the entire brain, 'only' its preferences. If you can extract a reward model from the transformer (which you might even be able to engineer to be superior using SOTA AI at the time of 'upload') and point an optimizer in that direction you can achieve a similar endpoint.
Training large neural networks on MEG data seems pretty good. Meta had lots of success with their work on decoding images from brain activity. The same goes for speech. This indicates to me that current neural networks are powerful enough to extract meaningful patterns from MEG data (the speech paper even had some success with EEG).
Deep learning does lots of surprising things when scaled sufficiently. If you asked most people what they thought about training a giant transformer on predicting the next token they probably wouldn't have forecasted GPT-4 by 2023. Maybe the same applies to predicting the next MEG reading?

This seems really easy to test. I thought about it for 30 minutes and came up with a few architectures based on Mistral-7B that I feel I could plausibly implement on short notice. I don't have any MEG hardware however, and it's expensive to obtain. From my research it doesn't look like there is sufficient

...
The other side of the tidal wave
by KatjaGrace — 2023-11-03T05:40:05.363Z

I guess there’s maybe a 10-20% chance of AI causing human extinction in the coming decades, but I feel more distressed about it than even that suggests—I think because in the case where it doesn’t cause human extinction, I find it hard to imagine life not going kind of off the rails. So many things I like about the world seem likely to be over or badly disrupted with superhuman AI (writing, explaining things to people, friendships where you can be of any use to one another, taking pride in skills, thinking, learning, figuring out how to achieve things, making things, easy tracking of what is and isn’t conscious), and I don’t trust that the replacements will be actually good, or good for us, or that anything will be reversible.

Even if we don’t die, it still feels like everything is coming to an end.
Does davidad's uploading moonshot work?
by Bird Concept — 2023-11-03T02:21:51.720Z

davidad has a 10-min talk out on a proposal about which he says: “the first time I’ve seen a concrete plan that might work to get human uploads before 2040, maybe even faster, given unlimited funding”.

I think the talk is a good watch, but the dialogue below is pretty readable even if you haven't seen it. I'm also putting some summary notes from the talk in the Appendix of this dialoge.

I think of the promise of the talk as follows. It might seem that to make the future go well, we have to either make general AI progress slower, or make alignment progress differentially faster. However, uploading seems to offer a third way: instead of making alignment researchers more productive, we "simply" run them faster. This seems similar to

OpenAI’s Superalignment

proposal of building an automated alignment scientist -- with the key exception that we might plausibly think a human-originated upload would have a better alignment guarantee than, say, an LLM.

I decided to organise a dialogue on this proposal because it strikes me as “huge if true”, yet when I’ve discussed this with some folks I've found that people generally don't seem that aware of it, and sometimes raise questions/confusions/objections that neither me nor the talk could answer.

I also invited Anders Sandberg, who co-authored the 2008 Whole Brain Emulation roadmap with Nick Bostrom, and co-organised the Foresight workshop on the topic where Davidad presented his plan, and Lisa Thiergart from MIRI, who also gave a talk at the workshop and has previously written about whole brain emulation here on LessWrong.

The dialogue ended up covering a lot of threads at the same time. I split them into six separate sections, that are fairly readable independently.

Barcoding as blocker (and synchrotrons)

davidad

One important thing is "what are the reasons that this uploading plan might fail, if people actually tried it." I think the main blocker is barcoding the transmembrane proteins. Let me first lay out the standard argument for why we "don't need" most of the material in the brain, such as DNA methylation, genetic regulatory networks, microtubules, etc.

Cognitive reaction times are fast.
That means individual neuron response times need to be even faster.
Too fast for chemical diffusion to go very far—only crossing the tiny synaptic gap distance.
So everything that isn't synaptic must be electrical.

...
One Day Sooner
by Screwtape — 2023-11-02T19:00:58.427Z

There is a particular skill I would like to share, which I wish I had learned when I was younger. I picked it up through working closely with a previous boss (a CTO who had founded a company and raised it up to hundreds of employees and multi-million dollar deals) but it wasn’t until I read The Story of VaccinateCA that I noticed it was a distinct skill and put into words how it worked. The Sazen for this skill is “One Day Sooner.” I would like to give warning before explaining further however: This skill can be hazardous to use. It is not the kind of thing “Rationalist Dark Art” describes because it does not involve deception, and I think it’s unlikely to damage much besides the user. It’s the kind of thing I’d be tempted to label a dark art however. Incautious use can make the user’s life unbalanced in ways that are mostly predictable from the phrase “actively horrible work/life balance.”

It works something like this: when you’re planning a project or giving a time estimate, you look at that time estimate and ask what it would take to do this one day sooner, and then you answer honestly and creatively.

The companion skill is one I call Never Drop A Ball.

What does it look like?

I used to work directly under the CTO of a medium sized software company. My team was frequently called upon to create software proofs of concept or sales demos. The timelines were sometimes what I will euphemistically call aggressive. Consider a hypothetical scene; it’s Thursday and you have just found out that a sales demo is on Tuesday which could use some custom development. Giving a quick estimate, you’d say this needs about a week of work and will be ready next Wednesday. What would it take to do this one day sooner?

Well, obviously you can work through the weekend. That gets you two more days. Given a couple of late evenings and getting enough total hours in is easy. That’s not the only thing though. There’s some resources from Marketing that would be good to have, you emailed them and they said they could meet with you on Monday. You want this faster though, so you walk over to their office and lean in, pointing out this is a direct assignment from the CTO so could we please have the meeting today instead. What else? Oh, there’s a bunch of specification writing and robust test writing you’d usually do. Some of that you still do, since it would be a disaster if you buil...
Propaganda or Science: A Look at Open Source AI and Bioterrorism Risk
by 1a3orn — 2023-11-02T18:20:29.569Z

0: TLDR

I examined all the biorisk-relevant citations from a policy paper arguing that we should ban powerful open source LLMs.

None of them provide good evidence for the paper's conclusion. The best of the set is evidence from statements from Anthropic -- which rest upon data that no one outside of Anthropic can even see, and on Anthropic's interpretation of that data. The rest of the evidence cited in this paper ultimately rests on a single extremely questionable "experiment" without a control group.

In all, citations in the paper provide an illusion of evidence ("look at all these citations") rather than actual evidence ("these experiments are how we know open source LLMs are dangerous and could contribute to biorisk").

A recent further paper on this topic (published after I had started writing this review) continues this pattern of being more advocacy than science.

Almost all the bad papers that I look at are funded by Open Philanthropy. If Open Philanthropy cares about truth, then they should stop burning the epistemic commons by funding "research" that is always going to give the same result no matter the state of the world.

1: Principles

What could constitute evidence that powerful open-source language models contribute or will contribute substantially to the creation of biological weapons, and thus that we should ban them?

That is, what kind of anticipations would we need to have about the world to make that a reasonable thing to think? What other beliefs are a necessary part of this belief making any sense at all?

Well, here are two pretty-obvious principles to start out with:

Principle of Substitution: We should have evidence of some kind that the LLMs can (or will) provide information that humans cannot also easily access through other means -- i.e., through the internet, textbooks, YouTube videos, and so on.

Blocker Principle: We should have evidence that the lack of information that LLMs can (or will) provide is in fact a significant blocker to the creation of bioweapons.

The first of these is pretty obvious. As example: There's no point in preventing a LLM from telling me how to make gunpowder, because I can find out how to do that from an encyclopedia, a textbook, or a novel like Blood Meridian. If you can substitute some other source of information for an LLM with only a little inconvenience, then an LLM does not contribute to the danger.

The ...
AI #36: In the Background
by Zvi — 2023-11-02T18:00:01.803Z

Wow, what a week. We had the Executive Order, which I read here so you don’t have to and then I have a tabulation of the reactions of others.

Simultaneously there was the UK AI Summit.

There was also robust related discussion around Responsible Scaling Policies, and the various filings companies did in advance of the Summit.

I touched on Anthropic’s RSP in particular in previous weeks, but I did not do a sufficiently close analysis and many others have offered more detailed thoughts as well, and the context has evolved.

So I am noting that I am not covering those important questions in the weekly roundup, and they will be covered by one or more later distinct posts. I also potentially owe an after action report from EA Global Boston, if I can find the time.

This post is instead about everything else.

Table of Contents

While top sections of this post are highlighted in bold, if you read one thing this week, I would read On the Executive Order.

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Which ones offer the most?

Language Models Don’t Offer Mundane Utility. Or so say many civilians.

GPT-4 Real This Time. Use all the ChatGPT features at once, coming soon.

Fun With Image Generation. I want my tank man.

Best Picture. Mission Impossible: Make Biden worry about AI.

Deepfaketown and Botpocalypse Soon. I’ve read (summaries of) your work.

They Took Our Jobs. Look how many are at risk, says yet another study.

Get Involved. UK Model Taskforce, Our World in Data and Open Phil, still.

OpenAI Frontier Risk and Preparedness Team. Looks good. They are hiring too.

Introducing. New AlphaFold, Time(Series)GPT, Phind as an LLM for code.

In Other AI News. Chinese overview of alignment, model weight theft methods.

Quiet Speculations. The branching paths of our possible futures.

The Quest for Sane Regulation. Tyler says do nothing, California explains why.

The Week in Audio. Shane Legg, Paul Christiano, Demis Hassabis.

Rhetorical Innovation. Make it as simple as possible, but no simpler.

Open Source AI is Unsafe and Nothing Can Fix This. 1918 flu edition.

Aligning a Smarter Than Human Intelligence is Difficult. AI hides its thoughts.

People Are Worried About AI Killing Everyone. Prove it is safe? What a concept.

Please Speak Directly Into This Microphone. The most extreme EO take of al

...
Doubt Certainty
by RationalDino — 2023-11-02T17:43:54.157Z

A feeling of certainly should usually be regarded as evidence of cognitive bias.

https://www.lesswrong.com/tag/absolute-certainty offers an extreme version of this principle. Absolute certainty requires a standard of evidence so high that we are more likely to have made a mistake than to truly be justified in claiming such certainty.

Less extreme forms of certainty do not necessarily require cognitive bias. For example I feel certain that the Sun will rise over my house in the morning. If challenged, I will admit that the odds are not 100%, and will give an estimate based mostly on the odds of my house not existing in the morning. Most people will accept that both my feeling and my estimate are rationally justifiable. Though maybe with quibbles about how the Sun does not truly rise.

But certainty can be tied to cognitive biases. Cognitive biases make it easy to think some things, and not their opposites. That leads to a feeling of certainty. Now suppose that we commit to the position that we're certain of. Now evidence against what we believe will cause cognitive dissonance - leading us to reject the evidence. Which means that certainty causes a cognitive bias which reinforces our certainty!

So when we encounter a feeling of certainty in ourselves or others, we should ask whether that certainty is because we have overwhelming evidence, or a cognitive bias. Bayesian reasoning tells us that the stronger the sense of certainty, and the more complex the chain of evidence to the conclusion, the more likely it is that we have encountered cognitive bias. Therefore, a feeling of certainty is evidence of cognitive bias.

To be clear, it always feels that your certainty is based on evidence. Which is why it is useful to realize that you should be extra suspicious of cognitive bias.

That's a nice sounding theory, but can I provide data suggesting that it really happens that way?

First, let's turn to Philip Tetlock's book Expert Political Judgement. Tetlock started with a group of bona fide experts. He asked them questions about their credentials and beliefs. And then asked them to make predictions about the future. Following Isaiah Berlin, he then divided them into foxes and hedgehogs with the hedgehogs having a big idea they were certain of, while foxes balanced many approaches.

On average, predictions were basically random. But the mistakes were far from equal. The hedgehogs ...
Saying the quiet part out loud: trading off x-risk for personal immortality
by disturbance — 2023-11-02T17:43:34.155Z

Statement: I want to deliberately balance the caution and the recklessness in developing AGI, such that it gets created in the last possible moment so that I and my close ones do not die.

This Statement confuses me. There are several observations I can make about it. There are also many questions I want to ask but have no idea how to answer. The goal of this post is to deconfuse myself, and to get feedback on the points that I raised (or failed to raise) below.

First observation: The Statement is directly relevant to LW interests.

It ties together the issues of immortality and AI risk, both of which are topics people here are interested in. There are countless threads, posts and discussions about high-level approaches to AI safety, both in the context of "is" (predictions) and "ought" (policy). At the same time, there is still a strong emphasis on the individual action, deliberating on which choices to make to improve the to marginal effects of living life in a certain way. The same is true for immortality. It has been discussed to death, both from the high-level and from the individual, how-do-I-sign-up-for-Alcor point of view. The Statement has been approached from the "is", but not from the "ought" perspective.
At the same time:

Second observation: No one talks about the Statement.

I have never met anyone who expressed this opinion, neither in-person nor online, even after being a part (although, somewhat on the periphery) of the rationalist community for several years. Not only that, I have not been able to find any post or comment thread on LW or SSC/ACX that discusses it, argues for or against it, or really gives it any attention whatsoever. I am confused by this since the Statement seems to be fairly straightforward.

One reason might be the:

Third observation: Believing in the Statement is low status, as it constitutes an almost-taboo opinion.

Not only no one is discussing it, but the few times when I expressed the Statement in person (at EA-infiltrated rationalists meetups), it was treated with suspicion or hostility. Although to be honest, I'm not sure how much this is me potentially misinterpreting the reactions. I got the impression that it is seen as sociopathic. Maybe it is?

Fourth observation: Believing in the Statement is incompatible with long-termism, and it runs counter to significantly valuing future civilisation in general.

Fifth o...
Mech Interp Challenge: November - Deciphering the Cumulative Sum Model
by CallumMcDougall — 2023-11-02T17:10:07.080Z

I'm writing this post to discuss solutions to the October challenge, and present the challenge for this November.

If you've not read the first post in this sequence, I'd recommend starting there - it outlines the purpose behind these challenges, and recommended prerequisite material.

November Problem

The problem for this month is interpreting a model which has been trained to classify the cumulative sum of a sequence.

The model is fed sequences of integers, and is trained to classify the cumulative sum at a given sequence position. There are 3 possible classifications:

0 (if the cumsum is negative),
1 (if the cumsum is zero),
2 (if the cumsum is positive).

For example, if the sequence is:

[0, +1, -3, +2, +1, +1]

Then the classifications would be:

[1, 2, 0, 1, 2, 2]

The model is not attention only. It has one attention layer with a single head, and one MLP layer. It does not have layernorm at the end of the model. It was trained with weight decay, and an Adam optimizer with linearly decaying learning rate.

I don't expect this problem to be as difficult as some of the others in this sequence, however the presence of MLPs does provide a different kind of challenge.

You can find more details on the Streamlit page. Feel free to reach out if you have any questions!

October Problem - Solutions

In the second half of the sequence, the attention heads perform the algorithm "attend back to (and copy) the first token which is larger than me". For example, in a sequence like:

[7, 5, 12, 3, SEP, 3, 5, 7, 12]

we would have the second 3 token attending back to the first 5 token (because it's the first one that's larger than itself), the second 5 attending back to 7, etc. The SEP token just attends to the smallest token.

Some more refinements to this basic idea:

The two attending heads split responsibilities across the vocabulary. Head 0.0 is the less important head; it deals with values in the range 28-37 (roughly). Head 0.1 deals with most other values.

In subsequences x < y < z where the three numbers are close together, x will often attend to z rather than to y. So why isn't this an adversarial example, i.e. why does the model still correctly predict y follows x?

Answer - the OV circuit shows that when we attend to source token s, we also boost things slightly less thn s, and suppress things slightly more than s.
So in the case of x < y < z, we have:

Attention to y

...