trentmkelly/LessWrong-43k huggingface dataset:
Progress Report 4: logit lens redux
by Nathan Helm-Burger — 2022-04-08T18:35:42.474Z

Prev: Progress Report 3

Next: Progress Report 5

Working on doing a side-by-side comparison of nostalgebraist's logit lens visualization and my neuron trace visualization derived from the ROME trace&edit project.

The one change made to nostalgebraist's code is to change the orientation. Instead of going bottom-to-top for layers, it now goes top-to-bottom for layers. I also changed the orientation from the ROME project to go top-to-bottom for layers and left-to-right (instead of top-to-bottom) for tokens. Seems like everyone has their own take on the best way to orient such visualizations, but I prefer my orientation so it's worth it to me to convert other's visualizations to my preferred orientation.

The induction heads described by the Transformer Circuits team show us at least one way in which information from earlier in the context is used later in the context. Let's take a look at the TC team's Harry Potter example. The word they particularly focus on is the unusual word 'Dursley' which is repeated multiple times in the context. If we choose a later instance of 'Dursley' to focus on, we should be able to see the importance of the earlier 'Dursley's in the answering of the current 'Dursley'.

Logit Lens

Interesting how there's that extra dark area in the logit-lens right below the instances of 'Dursley'. I think this means that it is 'extra surprising' in terms of what word might have been expected to come next in the sentence?

Neuron Importances

In this case, the neuron importances are pretty scattered so it's hard to make out a meaningful pattern from the plot. So here's the same information presented in a different way. In this plot, each pixel is instead colored according to the mean of the neuron importances for each layer.

Mean Layer Importances

So, I don't have a lot of interpretation for you yet. I'm just finding it satisfying to give update posts about my progress as I go along. If you want more explanation, feel free to ask in the comments!

Other ideas:

Possible practical application of neuron tracing/editing: preventing deceptive turns

We could use the neuron trace is to observe places in the model where information was stored during training, but not used in inference on a reasonably varied test set. We can simply delete those parts of the model, and then check that the model's performance on the validation set remains unchanged. ...
What would the creation of aligned AGI look like for us?
by Perhaps — 2022-04-08T18:05:24.902Z

I'm having a failure of imagination here, I assume an aligned AGI would upload human minds, preventing them from dying, but beyond that it's not clear to me what would happen. Would we just get wireheaded? Is there a CEV from human values that results in choice or life that has an analogue to a good life right now? Does anyone understand what would happen if you gave an AGI human values and told it to extrapolate from there?
Convincing All Capability Researchers
by Logan Riggs — 2022-04-08T17:40:25.488Z

This post is heavily based off this excellent comment (the author's name is not relevant)

Give the world's thousand most respected AI researchers $1M each to spend 3 months working on AI alignment, with an extra $100M if by the end they can propose a solution alignment researchers can't shoot down. I promise you that other than like 20 industry researchers who are paid silly amounts, every one of them would take the million. They probably won't make any progress, but from then on when others ask them whether they think alignment is a real unsolved problem, they will be way more likely to say yes. That only costs you a billion dollars! I literally think I could get someone reading this the money to do this (at least at an initially moderate scale) - all it needs is a competent person to step up.

Usually people object to paying people large sums of money to work on alignment because they don't expect them to produce any good work (mostly because it's very hard to specify alignment, see below). This is a feature, not a bug.

Being able to say "the 1000 smartest people working in AI couldn't make headway in Alignment in 3 months even when they were paid $1 million and a solution would be awarded $100 million" is very good for persuading existing researchers that this is a very hard problem.

Would this really go a long way to convincing those at major AI labs that alignment is hard? We could actually ask people working at these places if there were no progress made in those 3 months if it would change their minds.

Another problem is looooong timelines. They can agree it's hard, but their timelines may be too far to matter to work on it now. I could think of a couple counter-arguments that may work (but it would better to actually test them with real people)

1. To predict AGI (or just AI that has the capability of destroying the world), you could argue "It needs these 10 capabilities, and I predict it will get each capabilities at these different time periods". If someone actually did that and was historically correct, even within 5-10 years, I'd be impressed and trust their opinion. Have you successfully predicted AI capabilities of the past 10 years?

possibly mention most recent advances (Transformers, some RL, Neural Nets in 2010's)

2. If there's a 10% chance that it happens during you or your kids lifetime, then that's extremely important. For example, if I thoug

...
Language Model Tools for Alignment Research
by Logan Riggs — 2022-04-08T17:32:33.230Z

I do not speak for the rest of the people working on this project

[I think it's valuable to have clear, short intros on different research agendas & projects]

How does this reduce x-Risk?

AI will continue to become increasingly more powerful; we should leverage this to accelerate alignment research. Language model tasks will also follow this trend (if transformers don't lead to AGI, whatever's next will still be capable of language tasks so this argument doesn't rely on transformers scaling). If you believe that certain research agendas reduce x-risk, then clearly giving them better tools to do their work faster also reduces x-risk.

Differential Impact

Tools that can accelerate alignment research can probably be repurposed to accelerate capabilities research, so wouldn't developing these tools be net negative?

Yes. Especially if you gave them out to everybody or if you were a for-profit company with incentives to do so.

Only giving them out to alignment researchers is good. There's also lots of alignment-researcher-specific work to do, such as:

1. Collecting an alignment dataset
2. Understanding the workflows of alignment researchers
3. Making it incredibly easy for alignment researchers to use these tools
4. Keeping non-alignment specific datasets private

Though we could still increase capabilities by being the first to a capability and releasing that we succeeded. For example, OpenAI released their "inserting text" without telling people how they did it, but the people I work with, based off that information, figured out a way to do it too. The moral is that even just releasing that you succeeded is bits of information that those in the know can work backwards from.

Counterfactual Impact

Let's say a project like this never gets started, there are still huge economic incentives to make similar products and sell to large amounts of people. Elicit, Cohere, Jasper (previously Jarvis), OpenAI, DeepMind, and more in the years to come will create these products, so why shouldn't we just use their products since they're likely to beat us to it and do it better?

Good point. Beyond the points made in "differential impact," having infrastructure/people to quickly integrate the latest advances into alignment researcher's workflows is useful even in this scenario. This includes engineers, pre-existing code for interfaces, & a data-labeling pipeline.

Current Work

We've scraped L...
Takeaways From 3 Years Working In Machine Learning
by George3d6 — 2022-04-08T17:14:36.797Z
Link: https://www.epistem.ink/p/takeaways-from-3-years-working-in?s=w

Disclaimer: Feeling so-and-so about posting this on LW, but given how many people here work in ML or adjacent fields I might as well.

After 3 years working on automl at Mindsdb, I quit; I won’t be working on anything ml-related in a professional capacity, at least for a short while.

I am in the uncanny valley of finally understanding what I don’t know, and maybe even having a feel for what nobody knows.

I might as well write a summary of my takeaways, in hopes of them being useful to someone, or maybe just as a ritualistic gesture of moving on, a conceptual-space funeral.

Please don’t take this as an “expert summary”, there are dozens of thousands of people more suited than myself for doing that. Instead, think of it as a piece of outsider art, takeaways from someone that took an unusually deep dive into the zeitgeist without becoming part of “the community”.

i - What Is The Role Of Research?

The role of research into machine learning is, half a decade after first pondering it, still a mystery to me.

Most scientific sub-fields (the real ones) can claim they are a dual process of theory-building and data-gathering. Even if the standard model ends up being replaced, the data leading to its replacement will include the same set of experimental observations that lead to it being built. Even if the way we conceptualize the structure of DNA and the idea of genetics changes to better fit many-tissued eukaryotes, the observations these new concepts will have to fit shall remain unchanged.

In more theoretical areas, such as those revolving around the terms mathematics and computer science, the gains are at an almost purely-conceptual level, but they hold fast because the concepts are so foundational they seem unlikely to be replaced. Some alien species living inside deep space that we could barely understand as “life”, or some space-faring empire or languageless primate tribes that we could barely classify as “humans” might build impressive conceptual machinery without the alonzo-church thesis or even without concepts such as function, number, discreteness or continuity. But as long as those things hold true, a lot of old ideas in mathematics and computer science remain at most inefficiently phrased, but broadly valid.

But machine learning is at an odd intersection. Even if we grant it has as much theoretical rigor as, say, physics, it still lacks the timeless experimental ob...
[RETRACTED] It's time for EA leadership to pull the short-timelines fire alarm.
by Not Relevant — 2022-04-08T16:07:46.875Z

[EDIT 4/10/2022: This post was rash and ill-conceived, and did not have clearly defined goals nor meet the vaguely-defined ones. I apologize to everyone on here; you should probably update accordingly about my opinions in the future. In retrospect, I was trying to express an emotion of exasperation related to the recent news I later mention, which I do think has decreased timelines broadly across the ML world.

While I stand by my claims on roughly-human AGI probability, I no longer stand by my statement that "we should pull the fire-alarm". That is unlikely to lead to the calculated concerted effort we need to maximize our odds of successful coordination. Nor is it at all clear, given the timeline mechanism I described here, that AGI built in this way would be able to quickly FOOM, the primary point of concern for such a fire alarm.

I've left the rest of the post here as a record.

]

Based on the past week's worth of papers, it seems very possible (>30%) that we are now in the crunch-time section of a short-timelines world, and that we have 3-7 years until Moore's law and organizational prioritization put these systems at extremely dangerous levels of capability.[1]

The papers I'm thinking about:

https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html
https://twitter.com/andyzengtweets/status/1512089759497269251
https://openai.com/dall-e-2/
https://twitter.com/hausman_k/status/1511152160695730181

It seems altogether possible that it would not take long, given these advances and a moderate amount of iteration, to create an agentic system capable of long-term decision-making.

If you want to think of this as the public miscalibrated Bayesian-updating of one person, you should feel free to do that. If this was a conclusion you reached independently, though, I want to make sure we coordinate.

For those who haven't grappled with what actual advanced AI would mean, especially if many different organizations can achieve it:

No one knows how to build an AI system that accomplishes goals, that also isn't going to prevent you from turning it off. It's an unsolved research problem. Researchers have been trying for decades, but none of them think they've succeeded yet.
Unfortunately, for most conceivable goals you could give an AI system, the best way to achieve that goal (taken literally, which is the only thing computers know how to do) is

...
AIs should learn human preferences, not biases
by Stuart_Armstrong — 2022-04-08T13:45:06.910Z

A new paper by Rebecca Gorman and me, building on her ideas: The dangers in algorithms learning humans' values and irrationalities.

In essence, it is better for AIs at all power and alignment levels to learn human preferences (labelled as preferences) than to learn human biases (labelled as biases).

For an artificial intelligence (AI) to be aligned with human values (or human preferences), it must first learn those values. AI systems that are trained on human behavior, risk miscategorising human irrationalities as human values -- and then optimising for these irrationalities. Simply learning human values still carries risks: AI learning them will inevitably also gain information on human irrationalities and human behaviour/policy. Both of these can be dangerous: knowing human policy allows an AI to become generically more powerful (whether it is partially aligned or not aligned at all), while learning human irrationalities allows it to exploit humans without needing to provide value in return. This paper analyses the danger in developing artificial intelligence that learns about human irrationalities and human policy, and constructs a model recommendation system with various levels of information about human biases, human policy, and human values. It concludes that, whatever the power and knowledge of the AI, it is more dangerous for it to know human irrationalities than human values. Thus it is better for the AI to learn human values directly, rather than learning human biases and then deducing values from behaviour.
We Are Conjecture, A New Alignment Research Startup
by Connor Leahy — 2022-04-08T11:40:13.727Z

Conjecture is a new alignment startup founded by Connor Leahy, Sid Black and Gabriel Alfour, which aims to scale alignment research. We have VC backing from, among others, Nat Friedman, Daniel Gross, Patrick and John Collison, Arthur Breitman, Andrej Karpathy, and Sam Bankman-Fried. Our founders and early staff are mostly

EleutherAI

alumni and previously independent researchers like

Adam Shimi

. We are located in London.

Of the options we considered, we believe that being a for-profit company with products[1] on the market is the best one to reach our goals. This lets us scale investment quickly while maintaining as much freedom as possible to expand alignment research. The more investors we appeal to, the easier it is for us to select ones that support our mission (like our current investors), and the easier it is for us to guarantee security to alignment researchers looking to develop their ideas over the course of years. The founders also retain complete control of the company.

We're interested in your feedback, questions, comments, and concerns. We'll be hosting an AMA on the Alignment Forum this weekend, from Saturday 9th to Sunday 10th, and would love to hear from you all there. (We'll also be responding to the comments thread here!)

Our Research Agenda

We aim to conduct both conceptual and applied research that addresses the (

prosaic

) alignment problem. On the experimental side, this means leveraging our hands-on experience from EleutherAI to train and study state-of-the-art models without pushing the capabilities frontier. On the conceptual side, most of our work will tackle the general idea and problems of alignment like deception, inner alignment, value learning, and amplification, with a slant towards language models and

backchaining to local search

.

Our research agenda is still actively evolving, but some of the initial directions are:

New frames for reasoning about large language models:

What: Propose and expand on a frame of GPT-like models as simulators of various coherent text-processes called simulacra, as opposed to goal-directed agents (upcoming sequence to be published on the AF, see

this blogpost

for preliminary thoughts).
Why: Both an alternative perspective on alignment that highlights different questions, and a high-level model to study how large language models will scale and how they will influence AGI developm

...
Different perspectives on concept extrapolation
by Stuart_Armstrong — 2022-04-08T10:42:30.029Z

At the recent EAGx Oxford meetup, I ended up talking with a lot of people (18 people, back to back, on Sunday - for some reason, that day is a bit of a blur). Naturally, many of the conversations turned to value extrapolation/concept extrapolation, the main current focus of our Aligned AI startup. I explained the idea I explained multiple times and in multiple different ways. Different presentations were useful for people from different backgrounds.

So I've collected the different presentations in this post. Hopefully this will allow people to find the explanation that provides the greatest clarity for them. I think many will also find it interesting to read some of the other presentations: from our perspective, these are just different facets of the same phenomenon[1].

For those worried about AI existential risk

An superintelligence trained on videos of happy humans may well tile the universe with videos of happy humans - that is a standard alignment failure mode. But "make humans happy" is also a reward function compatible with the data.

So let D0 be the training data of videos of happy humans, R1 the correct "make humans happy" reward function, and R2 the degenerate reward function "make videos of happy humans"[2].

We'd want the AI to deduce R1 from D0. But even just generating R1 as a candidate is a good success. The AI could then get feedback as to whether R1 or R2 is correct, or maximise a conservative mix of R1 and R2 (e.g. R=log(R1)+log(R2)). Maximising that conservative mix will result in a lot of videos of happy humans - but also a lot of happy humans.

For philosophers

Can you define what a human being is? Could you make a definition that works, in all circumstances and in all universe, no matter how bizarre or alien the world becomes?

A full definition has eluded philosophers ever since humans were categorised as "featherless bipeds with broad flat nails".

Concept extrapolation has another way of generating this definition. We would point at all living humans in the world and say "these are humans[3]."

Then we would instruct the AI: "please extrapolate the concept of 'human' from this data". As long as the AI is capable of doing that extrapolation better than we could ourselves, this would give us an extrapolation of the concept "human" to new circumstances without needing to write out a full definition.

For ML engineers into image classification

Paper ...
Is there a possibility that the upcoming scaling of data in language models causes A.G.I.?
by ArtMi — 2022-04-08T06:56:50.146Z

In the past days we have seen DALL-E 2 and Google's Pathways Language Model achievements, and in the next months we will see the outcome of the recently discovered more optimal use of compute: https://www.lesswrong.com/posts/midXmMb2Xg37F2Kgn/new-scaling-laws-for-large-language-model

EY believes that A.G.I. could be achieved by:
1) Ability to train with less data.
2) Ability to improve itself.

I believe A.G.I. could be close if language models improve on their abilities to code, particularly at recursion.
Is there the possibility or do you believe the path to A.G.I. is orthogonal to the current ML pathway? Or somewhere in between? And if so, where?(This is the question that causes A.G.I.)
Could you please share any "AI to align A.G.I." resources do you have please?. The obviously "duh that's not possible" answer that i found is that for AI to do successful AI Safety Align Research it would probably already be too late. Maybe because it could also be used to research A.G.I.?
Good Heart Week Is Over!
by Ben Pace — 2022-04-08T06:43:46.754Z

Final leaderboard for the goodest of hearts

Good Heart Week has ended!

This week has seen a lot of content. I've had a great time reading posts and comments from new writers and returning writers, as well as polished-old-drafts from present writers :)

I'll write a retrospective in a week or two, but right now here's some information about what's happening and what to do next.

End Time: Good Heart week ended at midnight tonight (Thursday), Pacific Time. No tokens will be earned after that time.
Please Submit Financial Info: If you have 25+ Good Heart Tokens and you want to receive the money, please add either your PayPal address or ETH address or the name of a charity of your choice at lesswrong.com/payments/account. (It says "PayPal Info" but just add the other info in that field.)
Submission Deadline: You have a week to fill out the financial info. Please submit your financial info by EOD Thursday 14th (Pacific Time).
Security Concerns: This is hopefully wildly redundant, but please don't enter any passwords or secret keys or other private information when submitting financial info. I cannot assure you that this site is secure enough to keep such secrets. If you have further security (or anonymity) concerns about financial info feel free to PM me and we can try to figure something out.
Payout Date: One user on the leaderboard said to me it would make their life easier if the payment for this came after tax day on April 18th, so I'm tentatively planning to do payout the Friday after that. It also gives me time to do basic checks of the voting and such.
Retrospective: After the payouts I'll post a final update/retrospect on the week.

Let me know if you have any requests or questions at this time! I am also very interested in hearing your thoughts on and experiences from the whole week.
The Rationalist-Etcetera Diaspora: A SPREADSHEET!!
by Amelia Bedelia — 2022-04-08T05:43:43.760Z

For ye sweet springborn children too young to recall — know that LW is but one small isle of a grand and august archipelago!

Immediately after our community was born, we schismed a whole bunch. And now there are rat colonies (evolved and culturally drifted and not necessarily even rats now??) in all sorts of nooks and crannies, made for people who prefer them alongside or over our birthplace!

Here you can see Scott Alexander's map from an age lost to time (2014). While outdated to the precipice of uselessness, it is a beautiful expression of our community's tendency to sprawl everywhere. And it may give you a sense of the scale here!

What I hope to do is maintain an updated catalog of all these vaguely-rationalisty communities.

This is a spreadsheet including all the ones I know. I ask that others help to complete it by linking to active and public groups that share unique memeplexes with Less Wrong. If you're not sure whether a group meets these criteria but suspect it might, please include it and your peers can decide whether it belongs on this list.

I expect the sheet to eventually only take suggestions, but it is free to edit at the moment! Please feel free to edit in any prosocial way; 'tis only a draft.

I hope to find some new isles and to share some with you that you may not have known!

(h/t Jacob Falkovich for inspiration stemming from his post on ingroup)