trentmkelly/LessWrong-43k huggingface dataset:
Concrete examples of doing agentic things?
by Jacob G-W — 2024-01-12T15:59:52.154Z

I've been thinking about agency recently. I want to be more agentic!

Sometime last year, I read Cultivating And Destroying Agency and Seven ways to become unstoppably agentic. I recently saw How to be More Agentic and it kind of spurred me to write this post (which is actually a form of agency). I credit these posts with expanding what I think is possible. What I found really helpful was the concrete examples of the things they did (you can just buy stuff online, you can cold email people, try to get as many rejections as possible, sign up to take a test w/o taking a class).

These examples felt really helpful because they expanded the domain of things I thought were possible. I want to expand my agency Overton window.

My model is that there are two parts to expanding agency:

Find out or think of things that you can do that you had not thought of before (because they are uncommon / not what society tells us to do / require some staring into the abyss)
Actually do them, get good results, and have a positive feedback loop

If you have examples of things that you have done are just totally orthogonal to what we think of normal, that were ultimately helpful (or not), please put them here. This will hopefully help with step 1. Actually Doing Things and getting a positive feedback loop requires effort (but it hopefully is fun and rewarding).

If what you did ended really badly, please still put it, but just also put that it ended badly. I can reverse advice. If it ended well, say that also! The more examples the better. My main goal is to expand my possibility hull (pictured above). If you are unsure whether the thing you are saying is already known, please say it anyways.
Land Reclamation is in the 9th Circle of Stagnation Hell
by Maxwell Tabarrok — 2024-01-12T13:36:27.159Z
Link: https://maximumprogress.substack.com/p/land-reclamation-is-in-the-9th-circle

Land reclamation is a process where swamps, wetlands, or coastal waters are drained and filled to create more dry land.

Despite being complex and technologically intensive, land reclamation is quite old and was common in the past. The

reclamation of the Dutch lowland swamps

since the 13th century is well-known. Perhaps less well known is that almost every major American city had major land reclamation projects in the 19th and 20th centuries.

Boston changed the most with well over half of the modern downtown being underwater during the American Revolution, but it’s not unique.

New York, San Francisco, Seattle, Chicago, Newark, Philadelphia, Baltimore, Washington, and Miami have all had several major land reclamation projects.

Today, land prices in these cities are higher than ever, dredging ships are bigger, construction equipment is more powerful, landfills and foundations are more stable, and rising sea levels provide even more reason to expand shorelines, but none of these cities have added any land in 50 years or more.

Land reclamation is a technologically feasible, positive-sum way to build our way out of a housing crisis and to protect our most important cities from flooding, but it’s never coming back.

The 9th Circle of Stagnation Hell

Land reclamation is simultaneously harried by every single one of the anti-progress demons who guard Stagnation Hell. Let’s take a trip to see what it’s like.

The first circle of Stagnation Hell is environmental review. The guardian demon, NEPA-candezzar, has locked congestion pricing and transmission lines in the corner and is giving them a thousand paper cuts an hour for not making their reports long enough. Land reclamation suffers from environmental review in the same way as all other major infrastructure projects, or it would if anyone even tried to get one approved. Reclamation clearly has environmental effects so a full Environmental Impact Statement would be required, adding 3-15 years to the project timeline.

There’s also NEPA-candezzar’s three headed dog: wetland conservation, which, while less common, is extra vicious. Lots of land reclamation happens by draining marshes and wetlands. NEPA reviews are arduous but ultimately standardless i.e they don’t set a maximum level of environmental damage, they just require that all possible options are considered. Wetland conservation is more straightforward: wetlands are federa...
What good is G-factor if you're dumped in the woods? A field report from a camp counselor.
by Hastings — 2024-01-12T13:17:23.829Z

I had a surprising experience with a 10 year old child "Carl" a few years back. He had all the stereotypical signals of a gifted kid that can be drilled into anyone by a dedicated parent- 1500 chess elo, constantly pestered me about the research I did during the semester, used big words, etc. This was pretty common at the camp. However, he just felt different to talk to- felt sharp. He made a serious but failed effort to acquire my linear algebra knowledge in the week and a half he was there.
Anyways, we were out in the woods, a relatively new environment for him. Within an hour of arriving, he saw other kids fishing, and decided he wanted to fish too. Instead of discussing this desire with anyone or acquiring a rod, he crouched down at the edge of the pond and just watched the fishes. He noticed one with only one eye, approached it from the side with no vision, grabbed it, and proudly presented it to the counselor in charge of fishing.
Until this incident I was basically sceptical that you could dump some Artemis-Fowl-figure into a new environment and watch them big-brain their way into solving arbitrary problems. Now I'm not sure.

His out-of the box problem solving rapidly shifted from winning camper-fish conflicts to winning camper-camper conflicts, and he became uncontrollable. I almost won by breaking down the claim "You have to do what I say" into "You want to stay at camp, here's the conditions where that happens, map it out- you can see that you're close to the limit of rules broken where you still get what you want." This bought two more days of control. Unfortunately, he seems to have interpreted this new system as "win untracably," and then was traced trying to poison another camper by exploiting their allergy. He's one of two campers out of several thousand I worked with that we had to send home early for behavior issues.

In the end, he was much less happy than the other campers I've had, but I also think he's one of the few that could survive "Hatchet" or "Call of the Wild" style- despite comparative lack of experience.
Addendum: he harassed and kept catching the poor half-blind fish for the duration of the stay, likely because he got so much positive attention the first time he caught it.
A Chinese Room Containing a Stack of Stochastic Parrots
by RogerDearnaley — 2024-01-12T06:29:50.788Z

Most people are familiar with John Searle's Chinese Room argument ("Minds, Brains, and Programs" (1980)):

Suppose that I'm locked in a room and given a large batch of Chinese writing. Suppose furthermore (as is indeed the case) that I know no Chinese, either written or spoken, and that I'm not even confident that I could recognize Chinese writing as Chinese writing distinct from, say, Japanese writing or meaningless squiggles. To me, Chinese writing is just so many meaningless squiggles.

Now suppose further that after this first batch of Chinese writing I am given a second batch of Chinese script together with a set of rules for correlating the second batch with the first batch. The rules are in English, and I understand these rules as well as any other native speaker of English. They enable me to correlate one set of formal symbols with another set of formal symbols, and all that 'formal' means here is that I can identify the symbols entirely by their shapes. Now suppose also that I am given a third batch of Chinese symbols together with some instructions, again in English, that enable me to correlate elements of this third batch with the first two batches, and these rules instruct me how to give back certain Chinese symbols with certain sorts of shapes in response to certain sorts of shapes given me in the third batch. Unknown to me, the people who are giving me all of these symbols call the first batch "a script," they call the second batch a "story. ' and they call the third batch "questions." Furthermore, they call the symbols I give them back in response to the third batch "answers to the questions." and the set of rules in English that they gave me, they call "the program."

Now just to complicate the story a little, imagine that these people also give me stories in English, which I understand, and they then ask me questions in English about these stories, and I give them back answers in English. Suppose also that after a while I get so good at following the instructions for manipulating the Chinese symbols and the programmers get so good at writing the programs that from the external point of view that is, from the point of view of somebody outside the room in which I am locked -- my answers to the questions are absolutely indistinguishable from those of native Chinese speakers. Nobody just looking at my answers can tell that I don't speak a word of Chin

...
Decent plan prize announcement (1 paragraph, $1k)
by lemonhope — 2024-01-12T06:27:44.495Z

Edit Jan 20: Winner & highlights

Say I'm about to do a real big training run on playing video games, predicting text, predicting physics, writing code that works, etc etc. Say I've got a real good neural net architecture and a whole lot of flops. Say I'm a company and I'm gonna use this thing for AI lawyers and coders etc for a profit. Say I'm mildly concerned it somehow kills me and am willing to throw a few $ to prevent that.

So what should I do? How should I train and deploy the model?

Comment below or answer at this link if you don't want to be plagiarized.

Prize goes to best answer. (I judge obviously.)

The shorter the answer the better.

Deadline is Wednesday January 17 anywhere on Earth but answering immediately is better/easier.

You may accept your prize as 50 pounds of quarters if you prefer.

Clarification jan 12: say I've got 1000x the gpt4 flops and that my architecture is to transformers as convolutions are to simple MLPs in vision (ie a lot better)

Clarification 2: an answer like "here's how to get strong evidence of danger so you know when to stop training" is valid but "here's how to wipe out the danger" is much better.

3: Example answer for nuclear generators: "Spring-load your control rods so they are inserted when power goes out. Build giant walls around reactor so if steam explodes then uranium doesn't go everywhere. Actually, use something low pressure instead if possible, like molten salt or boiling water. Lift the whole thing off the ground to avoid flood risk."

4: This is hypothetical. I am not actually doing this. I'm not a billionaire.

5: "Hire someone" and "contract it out " and "develop expertise" etc obviously do not count as answers.
introduction to solid oxide electrolytes
by bhauth — 2024-01-12T05:35:49.878Z
Link: https://www.bhauth.com/blog/materials/oxide%20electrolyte.html

physics

Batteries, fuel cells, and electrolyzers all involve:

an electrolyte which allows passage of some ions with a single charge state (such as Li+ or Na+)

material with variable charge state on both sides of the electrolyte (such as Li metal and Fe+3 in LFP batteries)

electrodes which transfer charge to/from material

Batteries typically use liquid electrolyte and variable-charge-state material that's insoluble in it.

A solid oxide electrolyte (hereafter SOE) is an electrolyte which is solid and conducts oxide ions, which I'd denote as [O]-2 if they weren't specially named. Oxide has high charge density, so it has strong interactions, so the activation energy for it shifting is high, so conducting it requires high temperatures.

Solid oxide fuel cells (SOFCs) using SOEs have been manufactured.

doping

The main current SOE types are doped ceria (eg GDC) and doped zirconia. When an oxide crystal with +4 cations has some replaced with +3 cations, some of the oxygen positions in the crystal are vacant. At high temperatures, other oxygen atoms can move into those vacancies.

As the amount of dopant increases, conductivity increases, reaches a maximum, and decreases. Why does conductivity start decreasing before the crystal structure is destabilized? That's because dopants tend to form clusters at high concentrations, typically with 4 or 6 dopant atoms. Those clusters stabilize some oxygen vacancies, increasing the activation energy for oxygen moving into them.

The most important property of dopants besides their charge state is their ionic radius. Mismatch between dopant size and the crystal lattice causes strain, which often destabilizes the normal state of the crystal relative to the intermediate state of an oxygen moving into a vacancy, reducing activation energy.

Strain is relieved at crystal surfaces, so dopants have a tendency to move towards the surface, and that tendency is proportional to the strain they cause. As the crystal surfaces thus have a higher dopant concentration, dopant clusters first become a significant problem near the crystal surface.

By using 2 dopants with different ionic radius, generally using more of the one that causes less strain, it's possible to somewhat mitigate cluster formation and thus slightly increase peak conductivity. So, there are many papers on co-doped ceria.

composites

People noticed that the conductivity of doped ceria c...
Apply to the 2024 PIBBSS Summer Research Fellowship
by Nora_Ammann — 2024-01-12T04:06:58.328Z

TLDR: We're hosting a 3-month, fully-funded fellowship to do AI safety research drawing on inspiration from fields like evolutionary biology, neuroscience, dynamical systems theory, and more. Past fellows have been mentored by John Wentworth, Davidad, Abram Demski, Jan Kulveit and others, and gone on to work at places like Anthropic, Apart research, or as full-time PIBBSS research affiliates.
​Apply here: https://www.pibbss.ai/fellowship (deadline Feb 4, 2024)

''Principles of Intelligent Behavior in Biological and Social Systems' (

PIBBSS

) is a research initiative focused on supporting AI safety research by making a specific epistemic bet: that we can understand key aspects of the alignment problem by drawing on parallels between intelligent behaviour in natural and artificial systems.

Over the last years we've financially supported around 40 researchers for 3-month full-time fellowships, and are currently hosting 5 affiliates for a 6-month program, while seeking the funding to support even longer roles. We also organise research retreats, speaker series, and maintain an active alumni network.

We're now excited to announce the 2024 round of our fellowship series!

The fellowship

Our Fellowship brings together researchers from fields studying complex and intelligent behavior in natural and social systems, such as evolutionary biology, neuroscience, dynamical systems theory, economic/political/legal theory, and more.

Over the course of 3-months, you will work on a project at the intersection of your own field and AI safety, under the mentorship of experienced AI alignment researchers. In past years, mentors included John Wentworth, Abram Demski, Davidad, Jan Kulveit - and we also have a handful of new mentors join us every year.

In addition, you'd get to attend in-person research retreats with the rest of the cohort (past programs have taken place in Prague, Oxford and San Francisco), and choose to join our regular speaker events where we host scholars who work in areas adjacent to our epistemic bet, like Michael Levin, Alan Love, and Steve Byrnes and a co-organised an event with Karl Friston.

The program is centrally aimed at Ph.D. or Postdoctoral researchers. However, we encourage interested individuals with substantial prior research experience in their field of expertise...
A Benchmark for Decision Theories
by StrivingForLegibility — 2024-01-11T18:54:40.300Z

One of the cottage industries of our community is the construction of new decision theories. I think our community is also in agreement about how to compare decision theories: the rational way to make decisions is the one that best enables us accomplish our goals.

In this post I'll describe a benchmark that frames our choice of decision theory as a strategic one, where outcomes depend on the decision theories chosen by many agents that are all trying to accomplish their own goals. Our decision theory should lead to good results if adopted universally, but also be unexploitable by agents that choose different decision theories.

Meta-Games

There is a particular type of game that I think will serve well as a benchmark for decision theories. I'll call them Meta-Games, which have Meta-Players. Each Meta-Player chooses a decision theory DT∈D, and then a random game G∈G is chosen to be played. Each player P∈P(G) is then randomly assigned to a decision theory chosen by a Meta-Player. The outcome of the game is then passed back to the Meta-Players as feedback for how well they did.

The number of Meta-Players doesn't need to match the number of Players in any particular game. Unless otherwise specified, we can imagine that each Meta-Player writes their decision theory on a piece of paper, and that each Player is assigned a decision theory by drawing pieces of paper, replacing after each draw. (Decision theories are IID, according to the distribution of decision theories chosen by the Meta-Players.)

When there is only one Meta-Player, we are studying "what decision theories lead to the best results if universalized?" That is, if every player in every game acted according to the same decision theory. "Leading to good results if universalized" seems like an important quality for any decision theory we'd actually want to widely advocate that people use.

One big reason to study this sort of Meta-Game is that we can bring in all of the conceptual machinery from game theory and start applying them to decision theories. Does one decision theory dominate another on a certain class of games? Can a new decision theory be better adapted to an environment and become more and more represented in the population?

Superrationality

In general, I think we're going to be most interested in symmetric Meta-Games; that is, where swapping the roles of two Meta-Players doesn'...
An even deeper atheism
by Joe Carlsmith — 2024-01-11T17:28:31.843Z

(Cross-posted from my website. Podcast version here, or search for "Joe Carlsmith Audio" on your podcast app.

This essay is part of a series I'm calling "Otherness and control in
the age of AGI." I'm hoping that individual essays can be read fairly
well on their own, but see
here
for brief summaries of the essays that have been released thus far.

Minor spoilers for Game of Thrones.)

In my last essay, I discussed Robin Hanson's critique of the AI risk
discourse – and in particular, the accusation that this discourse
"others" the AIs, and seeks too much control over the values that steer
the future. I find some aspects of Hanson's critique uncompelling and
implausible, but I do think he's pointing at a real discomfort. In fact,
I think that when we bring certain other Yudkowskian vibes into view – and in particular, vibes related to the "fragility of value," "extremal
Goodhart," and "the tails come apart" – this discomfort should deepen
yet further. In this essay I explain why.

The fragility of value

Engaging with Yudkowsky's work, I think it's easy to take away something
like the following broad lesson: "extreme optimization for a
slightly-wrong utility function tends to lead to valueless/horrible
places."

Thus, in justifying his claim that "any Future not shaped by a goal
system with detailed reliable inheritance from human morals and
metamorals, will contain almost nothing of worth," Yudkowsky argues that
value is
"fragile."

There is more than one dimension of human value, where if just that
one thing is lost, the Future becomes null. A single blow
and all value shatters. Not every single blow will
shatter all value - but more than one possible "single blow" will
do so.

For example, he suggests: suppose you get rid of boredom, and so spend
eternity "replaying a single highly optimized experience, over and over
and over again." Or suppose you get rid of "contact with
reality,"
and so put people into experience machines. Or suppose you get rid of
consciousness, and so make a future of non-sentient flourishing.

Now, as Katja Grace points
out,
these are all pretty specific sorts of "slightly different."[1] But
at times, at
least, Yudkowsky seems to suggest that the point generalizes to many
directions of subtle permutation: "if you have a 1000-byte exact
specification of worthwhile happiness, and you begin to mutate it,...
Motivating Alignment of LLM-Powered Agents: Easy for AGI, Hard for ASI?
by RogerDearnaley — 2024-01-11T12:56:29.672Z

TL;DR: If we manage to develop an AGI or even ASI agent in the near-term, then it is likely to be powered by, or at very least include, an LLM (probably supplemented with additional cognitive scaffolding, long-term memory, and logic). LLMs have a lot of specific alignment challenges, strengths, and alignment-related properties that have not yet been discussed very much on the Alignment Forum/Less Wrong (until recently most of the discussion has been about abstract agents with few assumptions about the underlying technology, apart from Reinforcement Learning). For example, since LLMs are trained to simulate humans, human psychology and evolutionary psychology usefully apply to them. They are normally trained partly on fiction, thus features of fiction can often also apply to them. They are also obviously very fluent with human languages, ontologies, and knowledge. I explore some of the consequences of these features, both for predicting likely alignment problems and for possible alignment strategies, first for AGIs and then for ASIs. I examine alignment as a partially psychological problem, of how to induce an LLM to reliably simulate a very atypical and specific variant of human-like motivations and mentality.

LLMs Are Simulators for Humans

Simulator Theory points out that Large Language Models (LLMs) are trained as base models using Stochastic Gradient Descent (SGD) to simulate human-derived token-generation processes on the Internet and in other content. This includes tokens from both autobiographical and non-autobiographical works by solo authors, multiple-author works with careful editing such as scientific papers or news articles, and even things like the output of wikipedia communities. It also includes the dialog and actions of fictional characters in fiction written by human authors. The sole reason why LLMs produce agentic behavior is that they have been trained to simulate human agentic behavior: if you instead, as DeepMind did, train them on weather data, they will instead simulate nonagentic weather patterns. So the agents simulated by LLMs are not mesa-optimizers of the SGD token-prediction loss: they are instead misgeneralizing adaption-executing mesaoptimizers of evolution just like the humans they were trained from. Thus the standard analysis of Outer and Inner Alignment doesn't apply to them at all, unless and until you start applying fine-tun...