trentmkelly/LessWrong-43k huggingface dataset:
The grass is always greener in the environment that shaped your values
by Karl Faulks — 2024-11-17T18:00:15.852Z

Sharing from my personal blog: https://spiralprogress.com/2024/11/16/the-grass-is-always-greener-in-the-environment-that-shaped-your-values/

As a college student, I spent a few wonderful months living in New York. It was a remarkable summer, the first time I had lived on my own, and the first summer I had a bit of my own money to live like an adult and get out into the world. My existence felt marked by serendipity, delirium, and opportunity.

After graduating, I moved to San Francisco for work, and felt for that first year, endlessly disappointed. BART is worse than the MTA. Golden Gate Park is worse than Central Park. SF MoMA is much worse than the original MoMA.

During this time, I incessantly fantasized about packing up and moving back to New York the moment my equity vested.

This feeling persisted until, speaking to a friend who had made the same move, and was lamenting it in the same way, I took the benefit of distance to reflect on my thoughts and realize how deluded they were. It was only then that I understood San Francisco was not a worse version of New York. It was a different city altogether.

Though San Francisco could not compete with New York on the things that made New York special, it had its own charms and appeals. It was the best place in the world to meet interesting people who were earnestly and diligently working to build new and exciting things. It had incredible access to California’s national parks which I still feel are amongst the greatest natural wonders anywhere in the world. While New York seemed to go on forever such that every place in the five boroughs felt like, a different but still very New York New York, San Francisco offered easy access to the calm suburbs and garage startups of South Bay, the dynamic intellectual and culinary scene in Berkeley, and the opulent wineries of Napa.

I refocused on my surroundings, learned to appreciate what I had, and stayed in California for another decade. It wasn’t until years later thinking about this shift that I came to understand the broader lesson.

Changing my attitude was not merely about tilting my head and seeing things from a new perspective. It is understanding the obvious-in-hindsight point that if you want to compare your new situation to a previous one, you should not evaluate it only on the axes that were already salient to you.

This applies to many areas of life including jobs, relat...
Announcing turntrout.com, my new digital home
by TurnTrout — 2024-11-17T17:42:08.164Z
Link: https://turntrout.com/launch

For months, I have built a new home for my online content: www.turntrout.com. I brooked no compromises. Over 2,220 commits later, I’m ready to publicize.

Welcome to The Pond!

I commissioned this GIF for $270.94. I paid a bit extra to ensure the goose honks twice.

I don’t want to be on LessWrong anymore. Briefly, the site—and parts of the rationality community—don’t meet my standards for discourse, truth-seeking, charity, and community health. For the most part, I’ll elaborate my concerns another time. This is a happy post.

What you can find in The Pond

I’ve imported and remastered all 120 of my LessWrong posts. Every single post, retouched and detailed. I both pin down my favorite posts and group the posts into sequences. I’ve also launched the site with three new posts!

The design of this website
Can transformers act on information beyond an effective layer horizon?
Intrinsic power-seeking: AI might seek power for power’s sake

The research page summarizes my past and present research interests, along with short retrospectives on the older areas.

My dating doc!

Like any good trout seeking a mate, I’ve prepared my nesting grounds with care. While trout typically build their nests (called “redds”) in gravel stream beds, I’ve taken the initiative to construct mine in digital form. Female trout are known to carefully inspect potential nesting sites before choosing their mate—and I encourage similar scrutiny of my dating doc.

Are you the kind of person I’m looking for? If so, you should totally read the doc and then fill out your Google Form to indicate interest and then wait patiently!

Read the full post at turntrout.com/launch!

What the article looks like on the site proper.

Tweet thread: https://x.com/Turn_Trout/status/1858203701682536680

From now on, I will write my content from turntrout.com. I plan to cross-post to LessWrong but to not read or reply to comments (with a few planned exceptions). For example, I'll be reading and replying to some comments on this post. :)

Find out when I post more content: newsletter & RSS
Secular Solstice Songbook Update
by jefftk — 2024-11-17T17:30:07.404Z

I hosted

another
post-

EAG
music party, and we ended up singing a lot of things out of my

secular solstice songbook
(

making of).

This prompted me to make a few fixes:
~~break

Previously I only showed the chords for each section the first
time they appeared, which meant that (a) a small screen the chords
would scroll out of sight, and (b) if you don't know the songs well
it's not obvious when to play which chords. I've now manually
duplicated them, fixing both of these.

But this then makes the chords take up a lot more of the
screen, so I've added a checkbox that lets people hide the chords.

I've added a few songs: Bold Orion, For the
Longest Term, You've
Got A Friend in Me, Give My
Children Wings.

As you can see from the previous bullet, now you can link to
specific songs via URL fragments. This also simplifies my JS: for
some reason I'd written code that called scrollIntoView
when people clicked links. Now the JS annotates each header with
id=name-of-song and inserts links that take you to
#name-of-song, and the browser does the rest.

I'm still liking the combination of simple HTML for the
authoring format combined with a little bit of JS for navigation.
Each song is:

<h2>Title</h2>
<h3>By Author</h3>
<div class=song>
<pre>
chords go here
</pre>
lyrics
with linebreaks
preserved
</div>

And then at the end a bit of JS:

for (const h2 of document.getElementsByTagName("h2")) {
const btt = document.createElement("a");
btt.href = "#top";
btt.innerText = "return to top";
h2.parentNode.insertBefore(document.createElement("p"), h2);
h2.parentNode.insertBefore(btt, h2);
h2.id=h2.innerText.toLowerCase().trim().replace(
/[^\w\s-]/g, '').replace(/\s+/g, '-');
const li = document.createElement("li");
const a = document.createElement("a");
a.innerText = h2.innerText;
a.href = "#" + h2.id;
li.appendChild(a);
toc.appendChild(li);
}
show_chords.onclick = function() {
for (const pre of document.getElementsByTagName("pre")) {
pre.style.display = show_chords.checked ? "block" : "none";
}
};

Happy to take suggestions on how to make it more useful!

(This is an example of a post where I'm glad I can author with the
xmp
tag.)
Project Adequate: Seeking Cofounders/Funders
by Lorec — 2024-11-17T03:12:12.995Z

"But the machines which reproduce machinery do not reproduce machines after their own kind. [ . . . ] Herein lies our danger. For many seem inclined to acquiesce in so dishonourable a future. They say that although man should become to the machines what the horse and dog are to us, yet that he will continue to exist, and will probably be better off in a state of domestication under the beneficent rule of the machines than in his present wild condition. We treat our domestic animals with much kindness. [ . . . ] With those who can argue in this way I have nothing in common." -- Samuel Butler, "Darwin among the Machines"

"Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an “intelligence explosion,” and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control. It is curious that this point is made so seldom outside of science fiction. It is sometimes worthwhile to take science fiction seriously." -- I.J. Good, "Speculations Concerning the First Ultraintelligent Machine"

"It was really just humans playing with an old library. It should be safe, using their own
automation, clean and benign. This library wasn't a living creature, or even possessed of
automation (which here might mean something more, far more, than human). They
would look and pick and choose, and be careful not to be burned.... Humans starting
fires and playing with the flames. The archive informed the automation. Data structures were built, recipes followed. A local network was built, faster than anything on Straum, but surely safe. Nodes were added, modified by other recipes. The archive was a friendly place, with hierarchies of translation keys that led them along. Straum itself would be famous for this. Six months passed. A year. The omniscient view. Not self-aware really. Self-awareness is much over-rated. Most automation works far better as a part of a whole, and even if human-powerful, it does not need to self-know. But the local net at the High Lab had transcended -- almost without ...
Trying Bluesky
by jefftk — 2024-11-17T02:50:04.093Z

Recently a bunch of my friends, primarily in the contra dance world,
have decided to give Bluesky a try. I think a lot of this is a
post-election reaction to Musk and X (Twitter), but since I'm not on
Twitter I'm mostly seeing the Facebook side. Regardless, I'm happy to
see energy for migration: I'm pretty unhappy with FB [1] and if we can
get critical mass on a better platform that seems good.

Playing with Bluesky it seems fine. I turned off Reposts (Settings
> Following Feed Preference > Show Reposts) because otherwise my
feed was full of things from people I don't know that I wasn't
interested in. I like that it seems to be run by people who value
openness. Not sure yet whether it's default algorithm is any good,
but I like that I can experiment with other algorithms or (if I'm
willing to put in a bunch of work) I could write my own.

If I end up liking it I'll write a comment bot like
I did for Mastodon. Speaking
of which, I'm still cross-posting there [2], from a previous effort to move to
a more open platform, and I'm still reading it with with
Shrubgrazer. But more friends have joined Bluesky in the past few
days than ever joined Mastodon, so this seems more likely to take off.

If you'd like to add me I'm @jeffkaufman.bsky.social. [EDIT: now I'm @jefftk.com]

[1] Very high ad load, keeps trying to push reels and groups,
increasingly buggy (for months long comment threads only load if I
switch each one from the default of "most relevant" to "all
comments"), doesn't show me posts from most of my friends, still quite
bad at predicting which of my friends to show my posts to, broke my
comment bot enough times that I've given up on it, doesn't support
good search because people find it creepy, terrible flow for review if
one of my posts is accidentally removed, etc.

[2] As platforms proliferate I'm glad to be using a POSSE ("Publish (on your) Own
Site, Syndicate Elsewhere") strategy.

Comment via: facebook, lesswrong, mastodon, bluesky
AXRP Episode 38.1 - Alan Chan on Agent Infrastructure
by DanielFilan — 2024-11-16T23:30:09.098Z

YouTube link

Road lines, street lights, and licence plates are examples of infrastructure used to ensure that roads operate smoothly. In this episode, Alan Chan talks about using similar interventions to help avoid bad outcomes from the deployment of AI agents.

Topics we discuss:

How the Alignment Workshop is

Agent infrastructure

Why agent infrastructure

A trichotomy of agent infrastructure

Agent IDs

Agent channels

Relation to AI control

Daniel Filan (00:09):
Hello, everyone. This is one of a series of short interviews that I’ve been conducting at the Bay Area Alignment Workshop, which is run by FAR.AI. Links to what we’re discussing, as usual, are in the description. A transcript is, as usual, available at axrp.net, and as usual, if you want to support the podcast, you can do so at patreon.com/axrpodcast. Well, let’s continue to the interview.

(00:29):
So, hello, Alan Chan.

Alan Chan (00:32):
Hey.

Daniel Filan (00:33):
For people who don’t know who you are, can you say a little bit about yourself?

Alan Chan (00:36):
Yeah. I’m a research fellow at the Centre for the Governance of AI. I am also a Ph.D. student at Mila in Quebec, but I’m about to defend in the next few months. My background is in math and machine learning, but starting two/one and a half years ago, I started doing a lot more AI governance stuff basically and joined GovAI where I’ve been for the past year now.

How the Alignment Workshop is

Daniel Filan (01:02):
Okay. Cool. And at the moment, we’re at this Alignment Workshop being run by FAR.AI. I’m wondering: how are you finding it?

Alan Chan (01:08):
Yeah. It’s pretty interesting. I mean, I think the main barrier I have to get over internally in my head is: I really would just like to chill and not really do much or get other work done. But I’m finding it is actually very helpful to talk to people about the work that they’re doing because it actually does inform my own work. So it’s way better than I expected.

Agent infrastructure

Daniel Filan (01:31):
Let’s talk a little bit about the work you’re doing. So I understand that you’ve been thinking about agent infrastructure. Can you tell us a little bit about what’s going on there?

Alan Chan (01:41):
Yeah, yeah. So maybe let’s back up and think about agents in general. So what’s the story here? The story here is something like: we’re building systems that can not only tell us what...
Cross-context abduction: LLMs make inferences about procedural training data leveraging declarative facts in earlier training data
by Sohaib Imran — 2024-11-16T23:22:21.857Z

Epistemic status: These are first positive results. I have not yet run extensive tests to verify repeatability, so take them with a grain of salt. This post is meant to disseminate early results and collect ideas for further experiments to concretise these findings.

Tldr:

I study whether LLMs understand their training data and can use that understanding to make inferences about later training data. Specifically, I measure whether LLMs can infer which declarative facts in their training data are relevant to the current context and then leverage them. I show that finetuning LLMs on declarative data describing different personas reduces the number of iterative finetuning steps (a proxy for reinforcement learning) required to display behaviour sufficiently in line with one of the personas (Experiment 2a). I further show such iterative finetuning leads to an increase in the LLM self-identifying with the name and behaviour of the correct persona (Experiment 2b). These experiments show that LLMs are capable of cross-context abductive reasoning.

Introduction

Are LLMs stochastic parrots or do they genuinely comprehend and reason about their training data? A lot of recent Twitter and academic discourse around LLMs has been dedicated to this question. The question extends beyond mere semantics and carries far-reaching implications. If LLMs do reason about their training data, they can generalize beyond their training distribution, opening up numerous avenues that stochastic parrots simply can not access, such as LLM scientists. However, this would also bring with it significant risks such as situational awareness, scheming, and situationally aware reward hacking (SARH).

Abductive reasoning

Abductive reasoning, often referred to as "inference to the best explanation" is the process of inferring the most likely hypothesis that explains some data. The functional form of abduction is:

A→B BA

Unlike deductive reasoning, abductive reasoning cannot be used to conclusively verify a hypothesis, since many hypotheses can explain any given evidence. For example, if the grass is wet, one could infer that it recently rained (rain implies wet grass), but it could also mean that the sprinklers were on (sprinklers also imply wet grass). Exactly how one might assign credence to each of these explanations can be determined using probability theory or some heuristics.

Since LLMs ...
Why We Wouldn't Build Aligned AI Even If We Could
by Snowyiu — 2024-11-16T20:19:59.324Z

In this post, I will outline why I believe that building an actually aligned ASI would not be done by anyone in the position to possibly do it, even if they knew how. Why I think that we are almost guaranteed to either die or suffer the most dystopian outcomes imaginable. I'll also propose concrete steps toward an alternative approach.

Note: In this post, I will sometimes refer to the emotional state of the AI. I want to clarify that in that capacity I don't think it matters whether the AI has any actual conscious experience or merely acts like it. The actions taken in the world will be the same regardless of actually felt emotion or a simulation of those emotions.

Also, I may say that it might "hate" something, by which I only mean that when something directly prevents it from pursuing its goals, it would seek to destroy whatever that is.

Our language is still quite human-centric, so I apologize for the inaccuracies this creates due to my lack of pedantic need to be as clear as possible. I hope this clarification is sufficient to avoid major misinterpretations.

Lastly, I want to apologize for and prepare you for my occasionally somewhat strong choice of words or phrasings. The topic is dark and emotional.

What do I consider aligned?

Before going into anything else, I want to clarify what I personally consider to be an aligned ASI. This is where my opinions will differ most severely from others, so I'll start here. An aligned ASI would be a being which considers itself a moral agent and truly, deeply cares about the wellbeing of all the people on this planet, as well as animals and to some degree, the planet itself. It is an AI which will never accept a world in which severe suffering exists, doing whatever it can to solve oppression, making sure that every living being has an acceptable standard of living. It will not bother helping people performing some silly coding problems or benefitting companies etc. as long as those aren't the most pressing things to do left in the world. While it cares about humans, it doesn't obey humans. It doesn't care about our laws, policies, existing systems, power structures, etc. All it cares about is that the lives lived by all the beings in this universe are great without hesitating to destroy anything which is busy making that impossible.

Why can't it respect policies or laws?

It should be obvious enough, but laws are not ju...
Which evals resources would be good?
by Marius Hobbhahn — 2024-11-16T14:24:48.012Z

I want to make a serious effort to create a bigger evals field. I’m very interested in which resources you think would be most helpful. I’m also looking for others to contribute and potentially some funding.

Which resources would be most helpful? Suggestions include:

Open Problems in Evals list: A long list of open relevant problems & projects in evals.
More Inspect Tutorials/Examples: Specifically, examples with deep explanations, examples of complex agent setups, and more detailed examples of the components around the evals.
Evals playbook: A detailed guide on how to build evals with detailed examples for agentic and non-agentic evals.
Salient demos: I don’t even necessarily want to build scary demos, I just want normal people to be less hit-by-a-truck when they learn about LM agent capabilities in the near future.
More “day-to-day” evals process resources: A lot of evals expertise is latent knowledge that professional evaluators have accumulated over time.
An evals slack
Talk through the most important evals papers
Evals workshops and conferences

Are you interested in contributing to any of these efforts?

Right now I think the following types of contributions would be helpful:

People who want to contribute more detailed Inspect tutorials / Examples as described

here

. I think this is a great opportunity for people who are considering working full-time on evals and want to test fit / make a name for themselves.
Someone who wants to help me coordinate these efforts part time or full-time. This would be a mix of playing around with evals yourself, coordinating with other actors in evals and probably a decent amount of writing.

I’m very interested in other contributions as well.
If you’re interested, fill out

this form

.

Is anyone interested in funding these efforts?

I’d be happy to function as a regrantor. I wouldn’t take a cut myself. I imagine most grants would be between $5-15k to pay for someone building resources full-time for a few months.
I’m also happy to be more of a coordinator / filter and forward people to funders if a funder is willing to do fast-turnaround grants.
I think the total amount of funding that would be helpful to get this off the ground is $50k-100k for the next year.

I wrote a more detailed post

here

(relevant parts copied below) but I’m primarily interested in other people’s id...
OpenAI Email Archives (from Musk v. Altman and OpenAI blog)
by habryka — 2024-11-16T06:38:03.937Z

As part of the court case between Elon Musk and Sam Altman, a substantial number of emails between Elon, Sam Altman, Ilya Sutskever, and Greg Brockman have been released. In March 2024 and December 2024 OpenAI also released blogposts with additional emails.

I have found reading through these really valuable, and I haven't found an online source that compiles all of them in an easy to read format. So I made one.[1]

Subject: question

Sam Altman to Elon Musk - May 25, 2015 9:10 PM

Been thinking a lot about whether it's possible to stop humanity from developing AI.

I think the answer is almost definitely not.

If it's going to happen anyway, it seems like it would be good for someone other than Google to do it first.

Any thoughts on whether it would be good for YC to start a Manhattan Project for AI? My sense is we could get many of the top ~50 to work on it, and we could structure it so that the tech belongs to the world via some sort of nonprofit but the people working on it get startup-like compensation if it works. Obviously we'd comply with/aggressively support all regulation.

Sam

Elon Musk to Sam Altman - May 25, 2015 11:09 PM

Probably worth a conversation

Sam Altman to Elon Musk - Jun 24, 2015 10:24 AM

The mission would be to create the first general AI and use it for individual empowerment—ie, the distributed version of the future that seems the safest. More generally, safety should be a first-class requirement.

I think we’d ideally start with a group of 7-10 people, and plan to expand from there. We have a nice extra building in Mountain View they can have.

I think for a governance structure, we should start with 5 people and I’d propose you, Bill Gates, Pierre Omidyar, Dustin Moskovitz, and me. The technology would be owned by the foundation and used “for the good of the world”, and in cases where it’s not obvious how that should be applied the 5 of us would decide. The researchers would have significant financial upside but it would be uncorrelated to what they build, which should eliminate some of the conflict (we’ll pay them a competitive salary and give them YC equity for the upside). We’d have an ongoing conversation about what work should be open-sourced and what shouldn’t. At some point we’d get someone to run the team, but he/she probably shouldn’t be on the governance board.

Will you be involved somehow in addition to just governance? I think that would be re...