trentmkelly/LessWrong-43k huggingface dataset:
Links for May
by Kaj_Sotala — 2024-06-01T10:20:02.005Z

In English

What would happen if a superintelligent AI was aligned with your values?

The details here are a little too much in the “superintelligence is magic that can achieve anything” direction to my taste (I don’t think that anything will just be instantly teleported into safety, superintelligent AI or not), but I don’t doubt that the same results could be achieved via more mundane means. And it’s nice to have some more uplifting visions of the future.

The Choose Your Own Adventure Book or Ghost Ship Model of Will

I first put the core idea into words when someone I met at a workshop said she often had trouble being on time for things. She would notice that it was time to go a meeting soon but that she still had three minutes, so she could, keep reading her novel. And then, ten minutes later she’d actually stop and end up late, which she didn’t like. To this, I said something like “Ah. Apparently, your choice points don’t happen very often when you’re reading a novel. So, if you want to avoid being late, maybe you should seize choice points shortly before you need to leave, because you probably won’t get another one in time.”

The metaphor here is that your agency has a structure somewhat more like a choose your own adventure book than the completely free “I can do whatever I want whenever I want” which we often see it as. Of course, the chose your own adventure book metaphor is too constraining; it offers too few choices compared to one’s experience of the real world. Nonetheless, it captures the sporadicalness of the choice points.

Patri Friedman on political views

In a comment someone implied that one should have “a stable set of political views that are interrelated and coherent to some degree.”

I think I might disagree with this, and thought y’all might find it interesting.

So my claim is that to the degree that political views are describing mechanisms and outcomes in the real world, the real world is so complex that I’m actually not sure that an accurate description would be “interrelated and coherent” to a significant degree.

In fact I’ll hypothesize that most of the time when people choose view B partly based on how related and coherent it is with view A, they are making a worse choice (for building an accurate model of reality) than if they chose view B solely based on how it seems to empirically fit the world.

Coherence is beautiful and appealing, our mind l

...
What do coherence arguments actually prove about agentic behavior?
by sunwillrise — 2024-06-01T09:37:28.451Z

(edit: discussions in the comments section have led me to realize there have been several conversations on LessWrong related to this topic that I did not mention in my original question post.

Since ensuring their visibility is important, I am listing them here: Rohin Shah has explained how consequentialist agents optimizing for universe-histories rather than world-states can display any external behavior whatsoever, Steven Byrnes has explored corrigibility in the framework of consequentialism by arguing poweful agents will optimize for future world-states at least to some extent, Said Achmiz has explained what incomplete preferences look like (1, 2, 3), EJT has formally defined preferential gaps and argued incomplete preferences can be an alignment strategy, John Wentworth has analyzed incomplete preferences through the lens of subagents but has then argued that incomplete preferences imply the existence of dominated strategies, and Sami Petersen has argued Wentworth was wrong by showing how incomplete preferences need not be vulnerable.)

In his first discussion with Richard Ngo during the 2021 MIRI Conversations, Eliezer retrospected and lamented:

In the end, a lot of what people got out of all that writing I did, was not the deep object-level principles I was trying to point to - they did not really get Bayesianism as thermodynamics, say, they did not become able to see Bayesian structures any time somebody sees a thing and changes their belief. What they got instead was something much more meta and general, a vague spirit of how to reason and argue, because that was what they'd spent a lot of time being exposed to over and over and over again in lots of blog posts.

Maybe there's no way to make somebody understand why corrigibility is "unnatural" except to repeatedly walk them through the task of trying to invent an agent structure that lets you press the shutdown button (without it trying to force you to press the shutdown button), and showing them how each of their attempts fails; and then also walking them through why Stuart Russell's attempt at moral uncertainty produces the problem of fully updated (non-)deference; and hope they can start to see the informal general pattern of why corrigibility is in general contrary to the structure of things that are good at optimization.

Except that to do the exercises at all, you need them to work within an expec

...
AI Safety: A Climb To Armageddon?
by kmenou — 2024-06-01T06:02:14.968Z
Link: https://arxiv.org/abs/2405.19832

by Herman Cappelen, Josh Dever and John Hawthorne

Abstract: This paper presents an argument that certain AI safety measures, rather than mitigating existential risk, may instead exacerbate it. Under certain key assumptions - the inevitability of AI failure, the expected correlation between an AI system's power at the point of failure and the severity of the resulting harm, and the tendency of safety measures to enable AI systems to become more powerful before failing - safety efforts have negative expected utility. The paper examines three response strategies: Optimism, Mitigation, and Holism. Each faces challenges stemming from intrinsic features of the AI safety landscape that we term Bottlenecking, the Perfection Barrier, and Equilibrium Fluctuation. The surprising robustness of the argument forces a re-examination of core assumptions around AI safety and points to several avenues for further research.
When does external behaviour imply interal structure?
by Tyler Tracy — 2024-05-31T16:41:42.727Z

I've been working on an AI safety camp project where we try to describe agent structure. This post defines some key concepts and conveys my reasoning about this topic so far. It is mostly conceptual. The first section discusses what structure is and what it means for an object's behavior to imply structure. The second part shows ways of thinking about program structure. Then, we discuss different types of agent structures.

Basic structure

Consider the following sequence of numbers

1, 11, 21, 1211, 111221, 312211, 13112221, 1113213211, 31131211131221

Even if you don't know the function that generates these numbers, you can still find some patterns in the numbers if you look long enough. For example, you can see that the numbers always get bigger and end with a 1. This doesn't fully describe the sequence, but it describes some structure in the numbers. (if you are curious, this is the look and say sequence)

Let's dive into the structure by doing a simple conceptual analysis. Structure is a property of entities. The structure of an entity is a logical or mathematical object that partially describes that entity. An entity may have many different types of structures.

Examples of things with structure

the sequence 01010101 has a repetitive structure

The US interstate system has a graph structure

Monopoly has a game-theoretic structure

Structures are built on top of each other. For example, rings have a group structure, which has a set structure.

Determining structure

Let's say we have some object O and want to show that it has some structure. O might have some already known structure, like being a Turing machine or a Python program.

To try to determine O 's structure, we will test its behavior by asking it questions and examining the responses. Take a set as an example. We can ask, "Is this element in the set?" or "How many elements are in the set?" When we ask enough questions, we can conclude that the set has a certain structure, like "only contains even numbers."

I like to call this "carving" out structure instead of building it up. We start with the prior that this set is a random one from the set of all sets, and then by asking questions, we narrow down the possible objects this set could be until we conclude it is a particular one or a particular set of ...
We might be dropping the ball on Autonomous Replication and Adaptation.
by Charbel-Raphaël — 2024-05-31T13:49:11.327Z

Here is a little Q&A

Can you explain your position quickly?

I think autonomous replication and adaptation in the wild is under-discussed as an AI threat model. And this makes me sad because this is one of the main reasons I'm worried. I think one of AI Safety people's main proposals should first focus on creating a nonproliferation treaty. Without this treaty, I think we are screwed. The more I think about it, the more I think we are approaching a point of no return. It seems to me that open source is a severe threat and that nobody is really on the ball. Before those powerful AIs can self-replicate and adapt, AI development will be very positive overall and difficult to stop, but it's too late after AI is able to adapt and evolve autonomously because Natural selection favors AI over humans.

What is ARA?

Autonomous Replication and Adaptation. Let’s recap this quickly. Today, generative AI functions as a tool: you ask a question and the tool answers. Question, answer. It's simple. However, we are heading towards a new era of AI, one with autonomous AI. Instead of asking a question, you give it a goal, and the AI performs a series of actions to achieve that goal, which is much more powerful. Libraries like AutoGPT or ChatGPT, when they navigate the internet, already show what these agents might look like.

Agency is much more powerful and dangerous than AI tools. Thus conceived, AI would be able to replicate autonomously, copying itself from one computer to another, like a particularly intelligent virus. To replicate on a new computer, it must navigate the internet, create a new account on AWS, pay for the virtual machine, install the new weights on this machine, and start the replication process.

According to METR, the organization that audited OpenAI, a dozen tasks indicate ARA capabilities. GPT-4 plus basic scaffolding was capable of performing a few of these tasks, though not robustly. This was over a year ago, with primitive scaffolding, no dedicated training for agency, and no reinforcement learning. Multimodal AIs can now successfully pass CAPTCHAs. ARA is probably coming.

It could be very sudden. One of the main variables for self-replication is whether the AI can pay for cloud GPUs. Let’s say a GPU costs $1 per hour. The question is whether the AI can generate $1 per hour autonomously continuously. Then, you have something like an exponential process. I t...
Tax Cuts and Innovation
by Maxwell Tabarrok — 2024-05-31T12:58:31.495Z
Link: https://www.maximum-progress.com/p/tax-cuts-and-innovation

The classic turn in metascience goes like this: Policy discussions about R&D usually focus on how much we should spend: should we

double the NIH budget

or should we stop

subsidizing university nonsense

? The meta-scientist notices that some types of science funding are

tens if not hundreds

of times more effective than others. If we can shift spending into more efficient mechanisms we could 10x R&D output without spending a dime. Instead of focusing on how much money we should spend on science, we should focus on how to spend money on science.

But what if the most important metascience policy wasn’t about spending at all?

Taxation and Innovation in the 20th Century

is a 2018 paper by Ufuk Akcigit, John Grigsby, Tom Nicholas, and Stefanie Stantcheva that provides some answers. They collect and clean new datasets on patenting, corporate and individual incomes, and state-level tax rates that extend back to the early 20th century. The headline result: taxes are a huge drag on innovation. A one percent increase in the marginal tax rate for 90th percentile income earners decreases the number of patents and inventors by 2%. The corporate tax rate is even more important, with a one percent increase causing 2.8% fewer patents and 2.3% fewer inventors.

Methods

Before considering what these effects mean for metascience policy, let’s make sure we can believe them.

The authors have a table of data where each row lists the state, the year, and the number of patents, the tax rates, and lots of other variables in that state-year.

The authors analyze this data with three different methods:

The first method is the simplest. They just regress the patent output of a state on it’s tax rate 3 years prior, with state and year fixed effects and some other basic controls. Fixed effects are flexible ways to control for differences across states or across time that would bias the regression and mask the causal effect of increasing taxes on patenting.

For example, California has high levels of innovation and high taxes. A simple regression would use this data point to support a positive relationship between innovation and taxes. But is this relationship really because high taxes cause more innovation? Or is it some characteristic of California that both draws innovation and allows for high taxes like the nice weather, for example.

Including a “California fixed effect” makes the regression...
The Gemini 1.5 Report
by Zvi — 2024-05-31T12:20:03.098Z

This post goes over the extensive report Google put out on Gemini 1.5.

There are no important surprises. Both Gemini Pro 1.5 and Gemini Flash are ‘highly capable multimodal models incorporating a novel mixture-of-experts architecture’ and various other improvements. They are solid models with solid performance. It can be useful and interesting to go over the details of their strengths and weaknesses.

The biggest thing to know is that Google improves its models incrementally and silently over time, so if you have not used Gemini in months, you might be underestimating what it can do.

I’m hitting send and then jumping on a plane to Berkeley. Perhaps I will see you there over the weekend. That means that if there are mistakes here, I will be slower to respond and correct them than usual, so consider checking the comments section.

Practical Questions First

The practical bottom line remains the same. Gemini Pro 1.5 is an excellent 4-level model. Its big advantage is its long context window, and it is good at explanations and has integrations with some Google services that I find useful. If you want a straightforward, clean, practical, ‘just the facts’ output and that stays in the ‘no fun zone’ then Gemini could be for you. I recommend experimenting to find out when you do and don’t prefer it versus GPT-4o and Claude Opus, and will continue to use a mix of all three and keep an eye on changes.

How is the improvement process going?

Imsys.org: Big news – Gemini 1.5 Flash, Pro and Advanced results are out!

– Gemini 1.5 Pro/Advanced at #2, closing in on GPT-4o

– Gemini 1.5 Flash at #9, outperforming Llama-3-70b and nearly reaching GPT-4-0125 (!)

Pro is significantly stronger than its April version. Flash’s cost, capabilities, and unmatched context length make it a market game-changer!

More excitingly, in Chinese, Gemini 1.5 Pro & Advanced are now the best #1 model in the world. Flash becomes even stronger!

We also see new Gemini family remains top in our new “Hard Prompts” category, which features more challenging, problem-solving user queries.

Here is the overall leaderboard:

Oriol Vinyals (VP of Research, DeepMind): Today we have published our updated Gemini 1.5 Model Technical Report. As Jeff Dean highlights [in the full report this post analyzes], we have made significant progress in Gemini 1.5 Pro across all key benchmarks; TL;DR: 1.5 Pro >

...
Less Anti-Dakka
by Mateusz Bagiński — 2024-05-31T09:07:10.450Z

It is written in More Dakka:

If something is a good idea, you need a reason to not try doing more of it.

Taken at face value, it implies the following:

If something is a bad idea, you need a reason to not try doing less of it.

Labels/concepts, such as More Dakka, Inadequate Equilibria, etc point to a puzzling phenomenon. When more of X gives better results (consistently, ~proportionally to the dose of X, etc), people surprisingly often stop adding/doing more of X long before they hit the point at which the costs of more X start to outweigh the marginal benefits of more X.[1]

We should be just as puzzled by the dual phenomenon. When less of X gives better results (consistently, ~[inversely proportionally] to the dose of X, etc), people typically stop decreasing X long before they hit the point at which the costs of removing X (e.g. because you need some amount of X to survive/live comfortably/whatever) start outweighing the marginal benefits of there being less of X.

Examples:

If you feel better, healthier, and/or have better biomarkers, when you decrease the amount of X in your diet, maybe you would benefit from cutting it down to zero.
If you feel slightly more free whenever you eliminate some unnecessary clutter, maybe you would benefit from removing all the clutter.
If you have consistently benefited from simplifying your code, maybe you would benefit from simplifying it as much as possible.

More globally, maybe you would benefit from adopting the "don't add complexity before you need it" approach.

You noticed you're trying to do too many things per day/week/month/year. You realize you don't care that much about most of those things, so you eliminate some of them. Your daily/weekly/monthly/yearly to-do list has only 10 items now. Your goals are clearer than before. You feel more free than before. Maybe you would benefit from cutting your list down to 5.
You've noticed more freedom/ease/flow when you simplified your note-taking system. Maybe you would benefit from simplifying it even more.
Some of your flashcards are badly formulated or contain information that is neither important nor interesting to you. Sometimes you get annoyed at one or two during your daily review, so you remove them. Maybe you would benefit from going on a purge and deleting all flashcards that you suspect might be net-negative. Maybe you should be way more selective in what i

...
Web-surfing tips for strange times
by eukaryote — 2024-05-31T07:10:25.805Z
Link: https://eukaryotewritesblog.substack.com/p/web-surfing-tips-for-strange-times

[This post is more opinion-heavy and aimlessly self-promoting than feels appropriate for Lesswrong. I wrote it for my site, Eukaryote Writes Blog, to show off that I now have a substack. But it had all these other observations about the state of the internet and advice woven in, and THOSE seemed more at home on Lesswrong, and I'm a busy woman with a lot of pictures of fish to review, so I'm just going to copy it over as posted without laboriously extricating the self-advertisement. Sorry if it's weird that it's there!]

Eukaryote Writes Blog is now syndicating to Substack. I have no plans for paygating content at the time, and new and old posts will continue to be available at EukaryoteWritesBlog.com. Call this an experiment and a reaching-out. If you’re reading this on Substack, hi! Thanks for joining me.

I really don’t like paygating. I feel like if I write something, hypothetically it is of benefit to someone somewhere out there, and why should I deny them the joys of reading it?

But like, I get it. You gotta eat and pay rent. I think I have a really starry-eyed view of what the internet sometimes is and what it still truly could be of a collaborative free information utopia.

But here’s the thing, a lot of people use Substack and I also like the thing where it really facilitates supporting writers with money. I have a lot of beef with aspects of the corporate world, some of it probably not particularly justified but some of it extremely justified, and mostly it comes down to who gets money for what. I really like an environment where people are volunteering to pay writers for things they like reading. Maybe Substack is the route to that free information web utopia. Also, I have to eat, and pay rent. So I figure I’ll give this a go.

Still, this decision made me realize I have some complicated feelings about the modern internet.

Hey, the internet is getting weird these days

(h/t Bing’s Copilot for the images)

Generative AI

Okay, so there’s generative AI, first of all. It’s lousy on Facebook and as text in websites and in image search results. It’s the next iteration of algorithmic horror and it’s only going to get weirder from here on out.

I was doing pretty well on not seeing generic AI-generated images in regular search results for a while, but now they’re cropping up, and sneaking (unmarked) onto extremely AI-averse platforms like Tumblr. It used to be that you cou...
There Should Be More Alignment-Driven Startups
by Vaniver — 2024-05-31T02:05:06.799Z

Many thanks to Brandon Goldman, David Langer, Samuel Härgestam, Eric Ho, Diogo de Lucena, and Marc Carauleanu, for their support and feedback throughout.

Most alignment researchers we sampled in our recent survey think we are currently not on track to succeed with alignment–meaning that humanity may well be on track to lose control of our future.

In order to improve our chances of surviving and thriving, we should apply our most powerful coordination methods towards solving the alignment problem. We think that startups are an underappreciated part of humanity’s toolkit, and having more AI-safety-focused startups would increase the probability of solving alignment.

That said, we also appreciate that AI safety is highly complicated by nature[1] and therefore calls for a more nuanced approach than simple pro-startup boosterism. In the rest of this post, we’ll flesh out what we mean in more detail, hopefully address major objections, and then conclude with some pro-startup boosterism.

Expand the alignment ecosystem with startups

We applaud and appreciate current efforts to align AI. We could and should have many more. Founding more startups will develop human and organizational capital and unlock access to financial capital not currently available to alignment efforts.

“The much-maligned capitalism is actually probably the greatest incentive alignment success in human history” – Insights from Modern Principles of Economics

The alignment ecosystem is limited on entrepreneurial thinking and behavior. The few entrepreneurs among us commiserate over this whenever we can.
We predict that many interested in alignment seem to do more to increase P(win) if they start thinking of themselves as problem-solvers specializing in a particular sub-problem first, deploying whatever approaches are appropriate in order to solve the smaller problem. Note this doesn’t preclude scaling ambitiously and solving bigger problems later on.[2]

Running a company that is targeting a particular niche of the giant problem seems like one of the best ways to go about this transition, unlocking a wealth of best practices that could be copied. For example, we’ve seen people in this space raise too little, too late, resulting in spending unnecessary time in the fundraising stage instead of doing work that advances alignment. We think this is oft...