trentmkelly/LessWrong-43k huggingface dataset:
What and how much makes a difference?
by Marius Adrian Nicoară — 2024-06-05T10:30:53.493Z

This election year, we'll have the whole shebang in Romania:

June 9: election of local officials and representatives to the European Parliament

September 15 and 29: first and second round of presidential elections

December 8: elections for the Romanian Parliament

Quite a workout for the civic muscles.

But I am reminded of a story told by Voltaire[*]. A band of actors was traveling through villages to explain the difference between despotism and democracy. They performed each system of governance as follows:

for despotism, the actors sat in a single file, facing the back of the person in front of them. The despot sat at the back of the line, with no one behind him. In front of him were people with roles of descending rank. The peasant sat at the front of the line, with all the others at his back. On a whim, the despot kicked the noble in front of him, who passed the kick down the ranks, until it reached the poor peasant, who had no option but to endure

for democracy, the actors sat in a circle, each person facing another's back. Now the despot had the peasant behind him. When the despot kicked the person in front of them, the kick would circle back to them and the despot quickly caught on that there were consequences for his whims.

One takeaway: only trust people who are punished for misbehaving. Another takeaway: eternal vigilance is the price of freedom.

Related to the last takeaway, I suspect that one reason why some people are reluctant to be civically active is that they want to avoid it taking over their lives. For such people, who would like to dip into the pond without sinking, I would recommend some advice:

meaningful developments don't happen at the speed of notifications, so it's OK to have a sparse news diet

a news story might be important, just not important to you. If it's not something that you can easily act upon, let it slide

if someone does something to make you angry, ask yourself: How easy is it for me to ignore this? Rage bait can be quite tempting

find a way to disconnect. Cultivate your garden, literally and metaphorically. Rest well and when the time comes for you to step up, you'll be able to bring the best version of yourself

The Romanian philosopher Andrei Pleșu has encouraged voting many times, even suggesting a trial with mandatory voting. In a different context, he confessed to leaving the tap water running while shav...
Announcing ILIAD —  Theoretical AI Alignment Conference
by Nora_Ammann — 2024-06-05T09:37:39.546Z

We are pleased to announce ILIAD — a 5-day conference bringing together 100+ researchers to build strong scientific foundations for AI alignment.

***

Apply

to attend by June 30!***

When: Aug 28 - Sep 3, 2024
Where: @Lighthaven (Berkeley, US)
What: A mix of topic-specific tracks, and unconference style programming, 100+ attendees. Topics will include Singular Learning Theory, Agent Foundations, Causal Incentives, Computational Mechanics and more to be announced.
Who: Currently confirmed speakers include: Daniel Murfet, Jesse Hoogland, Adam Shai, Lucius Bushnaq, Tom Everitt, Paul Riechers, Scott Garrabrant, John Wentworth, Vanessa Kosoy, Fernando Rosas and James Crutchfield.
Costs: Tickets are free. Financial support is available on a needs basis.

See our website here. For any questions, email

iliadconference@gmail.com

About ILIAD

ILIAD is a 100+ person conference about alignment with a mathematical focus. The theme is ecumenical, yet the goal is nothing less than finding the True Names of AI alignment.

Participants may be interested in all tracks, only one or two or none at all. The unconference format will mean participants have maximum freedom to direct their own time and energy.

Program and Unconference Format

ILIAD will feature an unconference format - meaning that participants can propose and lead their own sessions. We believe that this is the best way to release the latent creative energies in everyone attending.

That said, freedom can be scary! If taking charge of your own learning sounds terrifying, rest assured there will be plenty of organized sessions as well. We will also run the topic-specific workshop tracks such as:

Computational Mechanics is a framework for understanding complex systems by focusing on their intrinsic computation and information processing capabilities. Pioneered by J. Crutchfield, it has recently found its way into AI safety. This workshop is led by Paul Riechers.
Singular learning theory, developed by S. Watanabe, is the modern theory of Bayesian learning. SLT studies the loss landscape of neural networks, using ideas from statistical mechanics, Bayesian statistics and algebraic geometry. The track lead is Jesse Hoogland.
Agent Foundations uses tools from theoretical economics, decision theory, Bayesian epistemology, logic, game theory and more to deeply understand agents: how they reason, cooperate

...
Second-Order Rationality, System Rationality, and a feature suggestion for LessWrong
by Mati_Roy — 2024-06-05T07:20:10.178Z

Second-order rationality

Definition

By “second-order rationality” (and intelligence) I mean the study of rationally reasoning about other people’s rationality (and intelligence) in order to inform us about the world.

This is opposed to evaluating a proposition at face value, using the first-order evidence supporting it.

Second-order rationality is about updating your beliefs just from understanding the distribution of different beliefs or belief histories, possibly by grouping them across populations with different characteristics, without referring to any first-order evidence related to the nature of the belief.

I think it’s an area worth exploring more.

What’s your probability that this is a useful area to study? You can use your own operationalization. For this exercise to work, you should record your prediction before continuing to read this article. This will get used as an example of system rationality at the end of this post.

Elicit Prediction (forecast.elicit.org/binary/questions/584LZjIuE)

Edit to add: Use this link.

Note: I think I'm following the instructions from Embedded Interactive Predictions on LessWrong—I don't know what the above forecasting widget doesn't seem to work; do you know? (Edit to add: See comment)

How to use this second-order knowledge?

I’m guessing many of you are already intuitively anxious about the Dangers of deference and trying to psychoanalyze people into knowing reality. Me too! That’s okay–we can use our first-order rationality to review the value of second-order rationality, theoretically and experimentally.

Ultimately, I think second-order rationality can help with finding the hypotheses which are most valuable to verify at the object-level but cannot allow us to defer all the way.

And we already defer to a very large extent, at least in terms of deciding what's worth learning (anyone here avoided all human interactions since birth in order to rederive everything by themselves?[1])–this is just meant to study that practice.

Factors that correlate with beliefs

Other beliefs

Obliviously, beliefs will correlate with each other, often rationally so. For example, if you believe the pavement is wet, you’re more likely to believe it recently rained.

Values

For example, studying how values and beliefs correlate might help us correct for the just-world fallacy. Presumably, for rational agents, beliefs and values ...
Former OpenAI Superalignment Researcher: Superintelligence by 2030
by Julian Bradshaw — 2024-06-05T03:35:19.251Z
Link: https://situational-awareness.ai/

The AGI race has begun. We are building machines that can think and reason. By 2025/26, these machines will outpace many college graduates. By the end of the decade, they will be smarter than you or I; we will have superintelligence, in the true sense of the word.

In the link provided, Leopold Aschenbrenner explains why he believes AGI is likely to arrive within the decade, with superintelligence following soon after. He does so in some detail; the website is well-organized, but the raw pdf is over 150 pages.

Leopold is a former member of OpenAI's Superalignment team; he was fired in April for allegedly leaking company secrets. However, he contests that portrayal of events in a recent interview with Dwarkesh Patel, saying he leaked nothing of significance and was fired for other reasons.[1]

However, I am somewhat confused by the new business venture Leopold is now promoting, an "AGI Hedge Fund" aimed at generating strong returns based on his predictions of imminent AGI. In the Dwarkesh Patel interview, it sounds like his intention is to make sure financial resources are available to back AI alignment and any other moves necessary to help Humanity navigate a turbulent future. However, the discussion in the podcast mostly focuses on whether such a fund would truly generate useful financial returns.

If you read this post, Leopold[2], could you please clarify your intentions in founding this fund?

^

Specifically he brings up a memo he sent to the old OpenAI board claiming OpenAI wasn't taking security seriously enough. He was also one of very few OpenAI employees not to sign the letter asking for Sam Altman's reinstatement last November, and of course, the entire OpenAI superaligment team has collapsed for various reasons as well.

^

Leopold does have a LessWrong account, but hasn't linked his new website here after some time. I hope he doesn't mind me posting in his stead.
On “first critical tries” in AI alignment
by Joe Carlsmith — 2024-06-05T00:19:02.814Z

People sometimes say that AI alignment is scary partly (or perhaps: centrally) because you have to get it right on the “first critical try,” and can’t learn from failures.[1] What does this mean? Is it true? Does there need to be a “first critical try” in the relevant sense? I’ve sometimes felt confused about this, so I wrote up a few thoughts to clarify.

I start with a few miscellaneous conceptual points. I then focus in on a notion of “first critical try” tied to the first point (if there is one) when AIs get a “decisive strategic advantage” (DSA) over humanity – that is, roughly, the ability to kill/disempower all humans if they try.[2] I further distinguish between four different types of DSA:

Unilateral DSA: Some AI agent could take over if it tried, even without the cooperation of other AI agents (see footnote for more on how I'm individuating AI agents).[3]
Coordination DSA: If some set of AI agents coordinated to try to take over, they would succeed; and they could coordinate in this way if they tried.
Short-term correlation DSA: If some set of AI agents all sought power in problematic ways within a relatively short period of time, even without coordinating, then ~all humans would be disempowered.
Long-term correlation DSA: If some set of AI agents all sought power in problematic ways within a relatively long period of time, even without coordinating, then ~all humans would be disempowered.

I also offer some takes on our prospects for just not ever having “first critical tries” from each type of DSA (via routes other than just not building superhuman AI systems at all). In some cases, just not having a “first critical try” in the relevant sense seems to me both plausible and worth working towards. In particular, I think we should try to make it the case that no single AI system is ever in a position to kill all humans and take over the world. In other cases, I think avoiding “first critical tries,” while still deploying superhuman AI agents throughout the economy, is more difficult (though the difficulty of avoiding failure is another story).

Here’s a chart summarizing my takes in more detail.

Type of DSA	Definition	Prospects for avoiding AIs ever getting this type of DSA – e.g., not having a “first critical try” for such a situation.	What’s required for it to lead to doom
Unilateral DSA	Some AI agent could take over if it tried, even without

...
Takeoff speeds presentation at Anthropic
by Tom Davidson — 2024-06-04T22:46:35.448Z

This is a lightly edited transcript of a presentation about the risks of a fast takeoff that I (Tom Davidson) gave at Anthropic in September 2023. See also the video recording, or the slides.

None of the content necessarily reflects the views of Anthropic or anyone who works there.

Summary:

Software progress – improvements in pre-training algorithms, data quality, prompting strategies, tooling, scaffolding, and all other sources of AI progress other than compute – has been a major driver of AI progress in recent years. I guess it’s driven about half of total progress in the last 5 years.
When we have “AGI” (=AI that could fully automate AI R&D), the pace of software progress might increase dramatically (e.g. by a factor of ten).
Bottlenecks might prevent this – e.g. diminishing returns to finding software innovations, retraining new AI models from scratch, or computationally expensive experiments for finding better algorithms. But no bottleneck is decisive, and there’s a real possibility that there is a period of dramatically faster capabilities progress despite all of the bottlenecks.
This period of accelerated progress might happen just when new extremely dangerous capabilities are emerging and previously-effective alignment techniques stop working.
A period of accelerated progress like this could significantly exacerbate risks from misuse, societal disruption, concentration of power, and loss of control.
To reduce these risks, labs should monitor for early warning signs of AI accelerating AI progress. In particular they can: track the pace of software progress to see if it's accelerating; run evals of whether AI systems can autonomously complete challenging AI R&D tasks; and measure the productivity gains to employees who use AI systems in their work via surveys and RCTs.
Labs should implement protective measures by the time these warning signs occur, including external oversight and info security.

Intro

Hi everyone, really great to be here. My name’s Tom Davidson. I work at Open Philanthropy as a Senior Research Analyst and a lot of my work over the last couple of years has been around AI take-off speeds and the possibility that AI systems themselves could accelerate AI capabilities progress. In this talk I’m going to talk a little bit about that research, and then also about some steps that I think labs could take to reduce the risks cau...
A Reflection on Richard Hamming's "You and Your Research": Striving for Greatness
by aysajan — 2024-06-04T20:07:01.422Z
Link: https://www.aysajaneziz.com/a-reflection-on-richard-hammings-you-and-your-research-striving-for-greatness/

Introduction: A Commitment to Greatness

Before I dive into my reflections on Richard Hamming's insightful talk, "You and Your Research", I want to start by posing three critical questions to myself:

What are the important problems in my field?
What important problems am I working on?
If what I am doing is not important, then why am I working on them?

These questions are not meant to be answered once and then forgotten. They are meant to be a constant guiding light, a daily reminder of the higher purpose that drives my work. To that end, I am making a commitment to myself to recite the following two sentences out loud every single day:

Yes, I would like to do first-class work.
Yes, I would like to do something significant.

By keeping these aspirations front and center in my mind, I hope to cultivate the focus, the courage, and the resilience necessary to tackle the most important challenges in my field and to make a real and lasting contribution.

In this post, I want to reflect on an article that I have read multiple times and that never fails to inspire me: Richard Hamming's "You and Your Research". Despite focusing on how to do first-class, outstanding research in science, Hamming's wisdom offers invaluable lessons that can be applied to all aspects of life. His words constantly provoke me to examine how I allocate my time and effort.

This post is as much a summary of Hamming's key insights as it is a letter to myself - a reminder of the mindset and habits I must cultivate if I wish to do great, impactful work. I am writing this because, if I'm being honest with myself, I am struggling to do even mediocre research, let alone groundbreaking work. But I firmly believe that engaging with Hamming's ideas can help me break through my current limitations. I don't know if I will succeed in doing great research, but I owe it to myself to try. At the very least, I want to experience what it feels like to strive for excellence with every fiber of my being, regardless of the outcome.

Hamming structures his essay around two key questions: (1) What makes someone do great research when others do not? and (2) Why do people with tremendous promise often fail to live up to their potential? In pondering these questions through the lens of my own experiences, I hope to extract actionable lessons that can guide me on my own quest for greatness.

Part One: The Makings of a Great Researcher

L...
A Semiotic Critique of the Orthogonality Thesis
by Nicolas Villarreal — 2024-06-04T18:52:58.642Z

“Intelligence and final goals are orthogonal axes along which possible agents can freely vary. In other words, more or less any level of intelligence could in principle be combined with more or less any final goal.”

That quote above is the orthogonality thesis, as defined by Nick Bostrom, the AI safety researcher. His short paper on the topic would end up becoming somewhat paradigmatic for the field and greatly informs arguments that AI presents existential risk to humankind. A more general formulation of Hume’s famous guillotine, which says that what should be and what is are totally unrelated, Bostrom takes things one step further. He states that, if we imagine a graph with one axis for intelligence and another axis which stands for final goals, we could plot every possible computational agent and find no necessary connection between the two variables, sometimes suggesting there would be no correlation at all. He defined intelligence succinctly, as the capacity to make effective plans, to do “instrumental reasoning” which leads agents to accomplish their goals. I believe that placing this problem into the framework of semiotic theory, the theory of signs and symbols, can elucidate a number of Bostrom’s errors in reasoning.

What Bostrom doesn't define very well in the paper, and which is effectively the Achilles heel of his argument, is the problem of the “final goal”. Indeed, he appears to conflate very often the idea of “motivation” and “final goal”. Some examples:

“Intelligence and motivation can in this sense be thought of as a pair of orthogonal axes on a graph whose points represent intelligent agents of different paired specifications.”

“This would be the case if such a system could be constructed in a way that would make it motivated to pursue any given final goal.”

This understanding of motivations and final goals is actually misleading in an important way. A goal is, fundamentally, an idea. As the final step in a plan, you can write it out as a symbolic representation of the “world state” you are trying to achieve, although it could represent other things as well. In a planning computer agent, this will probably terminate in a bunch of 1s and 0s stored in its memory.

In order for this symbolic representation to be meaningful, it must be comparable and distinct from other symbolic representations. World state A in the agent's plan could be co...
Ideas for Next-Generation Writing Platforms, using LLMs
by ozziegooen — 2024-06-04T18:40:24.636Z

See some earlier discussion on Facebook, here.

I've been doing more writing recently and have been heavily leaning on LLMs to help. This has been useful, but it seems clear that the fundamental LLM abilities are far outpacing the frontends and tools that use them.

Here's a quick list of ideas that I'd like to see in new writing platforms. I came up with these quickly, so I'm sure there are many other great ideas.

Predictive Clustering: Whenever your writing is predictable (for example, when responding to something or after the first few sentences of a new post), an LLM could vaguely predict the points you might make. It could cluster these points, allowing you to point and click on the relevant cluster. For instance, in a political piece, you might first click, "I [Agree | Disagree | Raise Interesting Other Point | Joke]." You then select "Raise interesting point," and it presents you with 5-20 points you might want to raise, along with a text box to add your own. Once you add your point, you can choose a length.
Facial Feedback: A video camera could monitor your facial expressions. If you're grimacing or frowning, it would automatically realize something is wrong and suggest ways to improve the text. This feature would be optional, of course.
Parallel Writing: Text writing should be done in parallel. For long posts, simple orchestration could split up the post into parts and have different LLM calls do each part.
Incremental Edits: Making edits to long texts, like, "Make it all 20% more polite," shouldn't require rewriting the entire thing from scratch. We really need Diffs.
Multiple Drafts: There should be several attempts at each piece of writing for you to choose between. Maybe start with a few widely different styles, then narrow that down.
Attribute Sliders: Similar to the buttons idea, there could be custom sliders for attributes based on the text in question. For instance, when writing a fiction scene, the LLM might be uncertain how smart you want one of the characters to be, so it brings up a slider for "how smart is character X in this scene?". As you move the slider, the text changes.
Sidebar Suggestions: In any long document, there should be a sidebar with a long list of potential changes. The exact interface could vary, but there are many options.
Research Agents: We could use AI agents that gradually research and explain content. This should have two inf

...