trentmkelly/LessWrong-43k huggingface dataset:
Consider Applying to the Future Fellowship at MIT
by jefftk — 2022-10-25T15:40:03.839Z

Update 2022-11-11: given the collapse
of FTX and the resignation
of the FTX Future Fund team this fellowship does not look like it's happening.

Are you interested in doing a PhD or Masters on something altruistic?
The MIT Media Lab has a

new
fellowship funded by the

FTX
Future Fund for "ambitiously benevolent projects that could not be
accomplished anywhere else". They give some examples:

creating works that expand our moral circle,

designing to cultivate wisdom,

developing new technologies to promote or safeguard wellbeing, or

directly improving the lives of current beings in ways
anticipated to benefit future generations

If you're interested you would apply for admission to the MIT Media
Lab for Fall 2023 (details) and
also to the fellowship (details),
deadline 2022-12-01. If accepted to both, the fellowship would cover
your tuition and health insurance, pay you a stipend (~$45k/y),
and cover $25k/y in additional research costs.

While the application requires you to say which research groups you're
interested in joining, what it doesn't make clear is that this is,
unusually for the US, a direct-admission system: each faculty member
chooses individually who they would like to admit. This means
it's definitely worth reaching out specifically to someone whose group
you're interested in joining (though they may not respond—they
get tons of email) and also thinking carefully about how your research
interests intersect.

Unlike many programs, you can work on a very wide range
of things: art, science, engineering, law, governance, economics, etc,
and research that would be hard to do in a more traditional graduate
program is especially welcome. If you have questions about what the
Media Lab is like I'm happy to try to answer, though I've only been
here a few months.

(Disclosure: I work
at the NAO which is part
of Kevin Esvelt's Sculpting Evolution
group at the Media Lab, and Kevin asked if I might want to write
something about this opportunity.)
Beyond Kolmogorov and Shannon
by Alexander Gietelink Oldenziel — 2022-10-25T15:13:56.484Z

This post is the first in a sequence that will describe James Crutchfield's Computational Mechanics framework. We feel this is one of the most theoretically sound and promising approaches towards understanding Transformers in particular and interpretability more generally. As a heads up: Crutchfield's framework will take many posts to fully go through, but even if you don't make it all the way through there are still many deep insights we hope you will pick up along the way.

EDIT: since there was some confusion about this in the comments: These initial posts are supposed to be an introductionary and won't get into the actually novel aspects of Crutchfield's framework yet. It's also not a dunk on existing information- theoretic measures - rather an ode!

To better understand the capability and limitations of large language models it is crucial to understand the inherent structure and uncertainty ('entropy') of language data. It is natural to quantify this structure with complexity measures. We can then compare the performance of transformers to the theoretically optimal limits achieved by minimal circuits.[1] This will be key to interpreting transformers.

​The two most well-known complexity measures are the Shannon entropy and the Kolmogorov complexity. We will describe why these measures are not sufficient to understand the inherent structure of language. This will serve as a motivation for more sophisticated complexity measures that better probe the intrinsic structure of language data. We will describe these new complexity measures in subsequent posts. Later in this sequence we will discuss some directions for transformer interpretability work.

Compression is the path to understanding

Imagine you are an agent coming across some natural system. You stick an appendage into the system, effectively measuring its states. You measure for a million timepoints and get mysterious data that looks like this:

...00110100100100110110110100110100100100110110110100100110110100...

You want to gain an understanding of how this system generates this data, so that you can predict its output, so you can take advantage of the system to your own ends, and because gaining understanding is an intrinsic joy. In reality the data was generated in the following way: output 0, then 1, then you flip a fair coin, and then repeat. Is there some kind of framework or algorit...
What does it take to defend the world against out-of-control AGIs?
by Steven Byrnes — 2022-10-25T14:47:41.970Z

Intended audience: People very familiar with AGI safety / alignment discourse. Lots of jargon, lots of unspoken & unjustified background assumptions.

Confidence level: What I currently believe and why. I mainly work on technical alignment and don’t consider myself an expert on AGI deployment / governance issues. Interested in feedback and pushback. Pretty please with cherries on top do not make important real-world decisions based on this post.

Tl;dr:

Almost all of my posts are about technical aspects of the alignment problem. This post is instead assuming for the sake of argument that some group will manage to sculpt an AGI’s motivations such that it’s either under control / corrigible / docile, and/or has prosocial motivations, or is safe for some other reason. But this post is also assuming that it’s possible to mess up and make an AGI that seeks power and escapes control. (This post is focused on out-of-control AGI accidents; I’ll generally ignore bad actor / misuse and other problems.) The point of this post is to think through whether and how under-control “good” AGIs can defend the world against omnicidal out-of-control “bad” AGIs (including by preventing them from coming into existence in the first place), or if not, what can be done about that problem.

This is a topic of ongoing debate in the community:

An example pessimist would be Eliezer Yudkowsky, who thinks that we’re basically doomed unless one of the first groups with AGI performs a so-called

“pivotal act”

(more on which in Section 3.5.1) that aggressively prevents any other groups on Earth from making misaligned AGIs.
An example (relative) optimist would be Paul Christiano, who argues in

The strategy-stealing assumption

that, if a big tech company with a giant compute cluster trains a friendly aligned powerful AGI in year Y, we probably have little cause for global concern if it happens that, in year Y+2, some small group in an office park somewhere messes up and makes a misaligned power-seeking AGI, because whatever power- or resource-grabbing strategies that the latter can come up with and execute, the former probably would have come up with and executed those same strategies already—or even better strategies.

This post explains why I put myself in the pessimistic camp on this issue. I think Paul’s “strategy-stealing assumption” is a very bad assumption, and I will argue more g...
Refine: what helped me write more?
by Alexander Gietelink Oldenziel — 2022-10-25T14:44:14.813Z

(previously: All the posts I will never write Refine Blogpost Day #3: The shortforms I did write )

Things that helped my writing

Key levers were (1) writing shortforms instead of posts (2) writing drafts on LW and letting them marinate for a while instead of writing everything in one go (3) writing with other people - especially Adam Shai (4) lowering my expectations (5)

Writing Quickly while maintaining Epistemic Rigor

. (6) starting with a draft of something I wrote on Twitter or ranted to somebody about in person (7) having more personal connections with people that seemed genuinely excited to read what I wanted to write (big difference to being at the Conjecture office in London, surrounded by all kinds of people interested in AI alignment and back at home where I’m the only one). (8) paying somebody to streamline some of the tasks I have less comparative advantage in. (9) 'somebody is wrong on the internet' writing

What didn’t work

(1) An idea that sounded good in theory but turned out to not work so well was using OtterAI (speech-to-text software) to transcript rants. The transcript are too messy to serve as a useful draft.
(2) Writing with other people often failed when our writing goals & tastes weren't sufficiently aligned.

Shortforms I wrote this month

Eight reasons to care about Kelly betting: (1) Kelly betting is asymptotically dominant, (2) Evolution selects for Kelly Betting, (3) Selection Theorems and Formal Darwinism, (4) Ergodicity Economics, (5) Kelly betting and Entropy, (6) Relevance of Information, (7) Bayesian Updating, (8) Blackjack

Multi-Step Fidelity causes Rapid Capability Gain Many examples of Rapid Capability Gain can be explained by a sudden jump in fidelity of a multi-step error-prone process. As the single step error rate is lowered there is a sudden transition from a low fidelity to a high fidelity regime.

Math research as Game Design In highschool math you follow a recipe to win the game. In proof-based math you have to come up with your own winning strategy. Math research is about creating new games by coming up with simple yet elegant rules that make the game fun to play.

Trapdoor Functions and Prime Insights A common way to think about hard puzzles is that they require an insight or several insights. An insight is analogous to a private key to a trapdoor function. There are cases wh...
Logical Decision Theories: Our final failsafe?
by Noosphere89 — 2022-10-25T12:51:23.799Z
Link: https://www.lesswrong.com/posts/rP66bz34crvDudzcJ/decision-theory-does-not-imply-that-we-get-to-have-nice

Nate Soares has stated that logical decision theories don't actually work to get us nice things. I disagree with this view, and explain why, even if the AI is unaligned, we still can get good things. (Note: this assumes either a solution to ELK or a scenario where we don't need to solve it for arbitrary networks, and narrow elicitation is enough.)

Cruxes for my view here:

I'm less into hard takeoffs than he is, so I'm not super-worried about intelligences recursively-self improving up to the level where humans can't understand it all (My probability of FOOM is more like 2-3% for the first AGI.) Similarly, I don't think escaping the AI box is nearly as easy as MIRI thinks.

I think ELK or narrow elicitation will probably get us the logical correlations necessary to prevent the outcomes of "fooling the human into a deal" from working.

I view singleton rule as at best unstable, and not likely to occur.

I think the alignment problem is underdetermined, that is our present state has multiple futures from which we can choose.

Acausal trade in infinite multiverses does weird enough things such that raising the probability of success is reasonable.

Now onto that point of acausal trade:

Fixed points and acausal trade in infinite multiverses

In an infinite multiverse like ours is likely to be (the universe is probably isotropic, homogeneous and flat, which implies infinity), probability is messed with.

This means that things change. One of the ways that things change is the bargaining power of humans vs UFAI, and there is an infinite amount of both (assuming alignment doesn't have probability zero.)

Let's say that humans acausally trade across the Tegmark Level IV multiverse. Nate's simply wrong here about there being not enough resources to trade, since they're infinite between humans and their allies and UFAI and it's allies.

Even in a good old level Level I multiverse, infinite numbers of humans and UFAIs exist.

What if they cooperate acausally between themselves? Well there's an infinite amount of humans vs an infinite amount of UFAIs, and this is an infinity vs infinity scenario, which leads to a perfect tie of 1/2 of the universe a priori.

Similar reasoning applies to Everett branches, and it's why exponentially decaying measure and probability still ends up as infinite measure, or probability of 1 respectively.

Where did Nate and Eliezer go wrong?

Answer:...
What will the scaled up GATO look like? (Updated with questions)
by Amal  — 2022-10-25T12:44:39.184Z

Demis Hassabis mentioned a few months ago that Deepmind is in the middle of scaling up its generalist agent GATO. Based on this, I would except it to come out either by the end of the year or in the beginning of the next year. The original model had just 1 billion parameters and was able to play atari games, operate a robotic arm and be used as a (relativel small) language model, so scaling it up to several hundred billion(trillions?) parameters seems to have a great potential in capability progress.

There were similar discussions about what would the most impressive thing that GPT-4 would be capable of, and similarly what it won't be able to achieve. However, since GPT-4 is likely going to be limited to text, it is probable that scaled up GATO would exhibit an entirely new/different set of capabilities.

So let's discuss the following questions: What will the scaled up GATO model/data/training look like? What will it be capable of and what would be the most impressive capabilities? On the other hand, what are the things it won't achieve?

Update:

To get more engagement with the post I provide some more specific questions to make predictions on (Thanks Daniel Kokotajlo for the idea):

How many parameters will the model have? (Current Gato has size was 1 billion)

How large will the context window be?

Will we see some more addvanced algorithmic improvements? E.g. new type of long-term memory, actual RL and similar

What type of data would it be trained on?

Will we observe transfer learning? e.g. seeing improvements in language model after training on audio

Will GATO see bigger improvements from Chain-of-thought style prompting compared to a LLM of similar size?

Will it be able to play new Atari games without being trained on them?

Feel free to suggest more questions and I will add them.
Mechanism Design for AI Safety - Reading Group Curriculum
by Rubi J. Hudson — 2022-10-25T03:54:20.777Z

The Mechanism Design for AI Safety (MDAIS) reading group, announced here, is currently in it's eighth of twelve weeks. I'm very excited by the quality of discussions we've had so far, and for the potential of future work from members of this group. If you're interested in working at the intersection of mechanism design and AI safety, please send me a message so that I can keep you in mind for future opportunities.

Edit: we have completed this initial list and are now meeting on a monthly basis. You can sign up to attend the meetings here.

A number of people have reached out to ask me for the reading list we're using. Until now, I've had to tell them that it was still being developed, but at long last it has been finalized. This post is to communicate the list publicly for anyone curious about what we've been discussing, or who would like to follow along themselves. It goes week by week listing the papers covered, the topics of discussion, and any notes I have. After the first two weeks, the order of the papers covered is largely inconsequential.

Reading List

Updated as of October 25th, 2024

Week 1

Papers:

The Principal-Agent Alignment Problem in Artificial Intelligence by Dylan Hadfield-Menell
Incomplete Contracting and AI Alignment by Dylan Hadfield-Menell and Gillian Hadfield

Discussion: Introductions, formalization of the alignment problem, inverse reinforcement learning and cooperative inverse reinforcement learning

Notes: The Principal-Agent Alignment Problem in Artificial Intelligence is extremely long, essentially multiple papers concatenated, so discussing it in the first week gave people more prep time to read it. Incomplete Contracting and AI Alignment is much shorter and less formal but did not add much, in hindsight I would not had included it.

Week 2

Paper: Risks from Learned Optimization in Advanced Machine Learning Systems by Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant

Discussion: Inner vs. outer alignment, what applications mechanism design has for each "step" in alignment

Week 3

Paper: Decision Scoring Rules (Extended Version) by Caspar Oesterheld and Vincent Conitzer

Discussion: Oracle AI, making predictions safely

Week 4

Paper: Discovering Agents by Zachary Kenton, Ramana Kumar, Sebastian Farquhar, Jonathan Richens, Matt MacDermott, and Tom Everitt

Discussion: Defining agents, using causal influence...
Furry Rationalists & Effective Anthropomorphism both exist
by agentydragon — 2022-10-25T03:37:57.213Z

(cross-posted on EA Forum and my website)

Hi!

I’m Rai and I’m a furry (specifically, dragon). The last couple years, I’ve been running a Furry Rationalists Telegram group. It looks like we exist, and not everyone who should know we exist does yet, so I wanted to just write this to advertise that this furry+EA/rationality corner exists, and if you’re furry-adjacent & rationality-adjacent and nice, you’re invited to join us :)

Here’s the invite link for the Furry Rationalists group: https://adgn.link/furry-rationalists-telegram

There’s ~50 of us and we’re chill - we have self-improvement, science, and cute animal GIFs. If you’d like a preview, here’s the

guidelines + meta doc

. We’re 18+, but we’re not adult-oriented - we’re 18+ just so that we can talk about adult stuff if it does come up. If you happen to be <18 and wanna join, let me know, we might update this.

If you’re reading this a while later, and the link expired, contact me (via some method on

my website, agentydragon.com

), or look us up on

https://www.furry-telegram-groups.net/

, a search for “rationality” should find us.

There’s also a smaller Effective Anthropomorphism Discord server, run by

bird

: https://adgn.link/effective-anthropomorphism-discord

Come say hi, and feel free to share if you know anyone who’d be interested!
EA & LW Forums Weekly Summary (17 - 23 Oct 22')
by Zoe Williams — 2022-10-25T02:57:43.696Z

Supported by Rethink Priorities

This is part of a weekly series - you can see the full collection here. The first post includes some details on purpose and methodology.

If you'd like to receive these summaries via email, you can subscribe here.

Podcast version: prefer your summaries in podcast form? A big thanks to Coleman Snell for producing these! Subscribe on your favorite podcast app by searching for 'Effective Altruism Forum Podcast'.

EA Forum

Philosophy and Methodologies

Effective Altruism's Implicit Epistemology

by Violet Hour

Longtermist philosophy is pretty reasonable (future people matter, there might be a lot, and we can make a difference to them). However many outside EA find the priorities that have arisen from these (eg. AI safety & bio risk) to be weird. The author argues this is due to EA’s unusual epistemic culture, and uses this post to highlight these norms and how they influence our decision-making.

In particular, EAs tend to be comfortable with speculative reasoning, put numbers on things (even when they’re highly unsure), use those numbers as inputs to decision-making, but are still skeptical if all that leads to anything too speculative and fanatical. The author suggests being explicit about these norms because that allows better outside criticism, or if we’re really onto something, allows others to benefit from it.

The Relative Importance of the Severity and Duration of Pain

by William McAuliffe, Adam_Shriver

Pains vary in their severity and duration. This report reviews the research and philosophy on how to trade off these two dimensions, which can impact cause prioritization decisions.
Some viewpoints explored include that badness scales non-linearly with severity of pain, or that long-duration pain can only outweigh a high severity pain if it meets the bar of preventing pleasure or making more moments bad than good. Utilitarian views simply multiplying (severity X duration) are also presented. It’s also possible these trade-offs vary between individuals - one study found most participants make decisions as if adding severity and duration to get badness, but a minority multiply them.
Ethical constraints, severity being more salient in retrospect, imagination failures and other factors make research and experimentation in this area difficult. The authors are planning to gather scientists and philosophers at a workshop to develop new meth...
Dance Weekends: Tests not Masks
by jefftk — 2022-10-25T02:10:04.171Z

I'm really excited that contra dance weekends are starting up again,
and I just

got back
from playing with

Amy and

Audrey for the first time in

three years.
I'm also seeing a lot of weekends struggling with registration,
however, and choosing between going ahead with a small crowd or

cancelling.

Talking to people, in person and online, the big issue seems to be
masking. Dancing in a mask is less fun and more tiring, and many
people would rather stay home than dance for many hours masked.
Weekends that don't require masks are not really a "control group" for
ones that do, being different in a lot of ways, but that they are
seeing strong attendance is evidence in this direction.

Most weekends that require masks don't require testing: I think it
should be the other way around. Consider requiring daily testing,
on arrival at the hall, instead of mandatory masking:

Rapid tests, like masks, are no longer scarce: you can buy as
many of them as you want without worrying that there won't be enough
for others. As long as you
buy them in advance you don't need to worry that unavailability will
keep you from being able to go forward with the event.

At $3.75/test
shipped, the cost is low compared to what people are willing to pay to
attend a dance weekend. This is a different tradeoff from evening
dances, where requiring a test would require a more price-sensitive
crowd to move from ~$10
to ~$14 (though possibly still worth it there!).

Another way the tradeoff is different from evening dances is
that a weekend is much longer, so you're getting more hours of dancing
per test.

While testing isn't 100% protective, neither is masking. I'm
about ambivalent between being exposed to an N95-masked infected
person vs a testing-negative infected person. And I'd rather be
exposed to a testing-negative infected person than one wearing a
surgical or, especially, cloth mask.

This isn't to say that you can't require both masks and testing if you
think that level of precaution is warranted, but I think if you're
going to have one of them be optional it makes much more sense for it
to be masking.

I also think organizations that stop requiring masks should consider
subsidizing elastomeric P100
respirators. These are very effective at protecting the wearer,
likely more than having everyone in the room wear a high-filtration
mask, and are gen...