trentmkelly/LessWrong-43k huggingface dataset:
My techno-optimism [By Vitalik Buterin]
by habryka — 2023-11-27T23:53:35.859Z
Link: https://www.lesswrong.com/out?url=https%3A%2F%2Fvitalik.ca%2Fgeneral%2F2023%2F11%2F27%2Ftechno_optimism.html

Vitalik wrote a post trying to make the case for his own take on techno-optimism summarizing it as an ideology he calls "d/acc". I resonate with a lot of it, though also have conflicting feelings about trying to create social movements and ideologies like this.

Below some quotes and the table of contents.

Last month, Marc Andreessen published his "techno-optimist manifesto", arguing for a renewed enthusiasm about technology, and for markets and capitalism as a means of building that technology and propelling humanity toward a much brighter future. The manifesto unambiguously rejects what it describes as an ideology of stagnation, that fears advancements and prioritizes preserving the world as it exists today. This manifesto has received a lot of attention, including response articles from Noah Smith, Robin Hanson, Joshua Gans (more positive), and Dave Karpf, Luca Ropek, Ezra Klein (more negative) and many others. Not connected to this manifesto, but along similar themes, are James Pethokoukis's "The Conservative Futurist" and Palladium's "It's Time To Build for Good". This month, we saw a similar debate enacted through the OpenAI dispute, which involved many discussions centering around the dangers of superintelligent AI and the possibility that OpenAI is moving too fast.

My own feelings about techno-optimism are warm, but nuanced. I believe in a future that is vastly brighter than the present thanks to radically transformative technology, and I believe in humans and humanity. I reject the mentality that the best we should try to do is to keep the world roughly the same as today but with less greed and more public healthcare. However, I think that not just magnitude but also direction matters. There are certain types of technology that much more reliably make the world better than other types of technology. There are certain types of technlogy that could, if developed, mitigate the negative impacts of other types of technology. The world over-indexes on some directions of tech development, and under-indexes on others. We need active human intention to choose the directions that we want, as the formula of "maximize profit" will not arrive at them automatically.

In this post, I will talk about what techno-optimism means to me. This includes the broader worldview that motivates my work on certain types of blockchain and cryptography applications and social technolo

...
Could Germany have won World War I with high probability given the benefit of hindsight?
by Roko — 2023-11-27T22:52:42.066Z

Related: Could World War I have been prevented given the benefit of hindsight?

Consider the following scenario: you are sent back in time some number of years before the beginning of WWI, say 1904 for concreteness, with the goal of changing Imperial German strategy before and during the war such that Germany wins with very high probability [meaning that throughout many small deviations of the timeline, including small variations in the enemy strategic response function, at least 95% of them lead to an Austro-German victory over at least France and Russia, taking lots of new territory and establishing a German empire on the world stage. We will define winning as either total destruction of the enemy military or a surrender by the Russia/France side of the war where Germany gets to dictate terms.]

This includes winning similar wars that happen slightly earlier or slightly later - you are aiming for a durable strategy for Germany and Austro-Hungary that makes them win over the opposing European powers.

To prevent silly strategies, minor details of the world you are sent to will be changed, so for example dates and names may differ very slightly.

Furthermore, you carry some evidence of extraordinary status that is credible to the leaders of Austro-Hungary and Germany. They will grant you an audience and they have an unusually large amount of patience to listen to you, but they will not blindly trust you. You need to convince them based on reasoning that they will accept. You may lie to them or tell them the truth. I will also stipulate that if you manage to convince the leadership of a plan, they will actually follow it with an unusually high degree of fanaticism and coherence.

You don't have detailed designs for post-1914 technology, but you have an ordinary informed layman's understanding of actual history. So you can tell them that nuclear bombs will be invented, but not exactly how, and they may or may not believe you.

Please answer with your strategy and why you think it would work (including in the face of likely reactions from the major players)!
Could World War I have been prevented given the benefit of hindsight?
by Roko — 2023-11-27T22:39:15.866Z

Consider the following scenario: you are sent back in time some number of years before the beginning of WWI, with the goal of preventing the war. This includes preventing similar wars that happen slightly earlier or slightly later - you are aiming for a peaceful, stable coexistence between the European powers. Unsophisticated strategies like warning The Archduke against going to Sarajevo may simply result in another similar casus belli happening.

To prevent silly strategies, minor details of the world you are sent to will be changed, so dates and names may differ very slightly, but the overall WWI path will unfold almost exactly as it did in our timeline.

Furthermore, you carry some evidence of extraordinary status that is credible to the leaders of WWI nations. They will grant you an audience, but they will not blindly trust you. You may lie to them or tell them the truth.

You don't have detailed designs for post-1914 technology, but you have an ordinary informed layman's understanding of actual history. So you can tell them that nuclear bombs will be invented, but not exactly how, and they may or may not believe you.

Please answer with your strategy for saving Europe from World War I, and why you think it would work (including in the face of likely reactions from the major players)!
AISC 2024 - Project Summaries
by NickyP — 2023-11-27T22:32:23.555Z

Apply to AI Safety Camp 2024 by 1st December 2023. All mistakes here are my own.

Below are some summaries for each project proposal, listed in order of how they appear on the website. These are edited by me, and most have not yet been reviewed by the project leads. I think having a list like this makes it easier for people to navigate all the different projects, and the original post/website did not have one, so I made this.

If a project catches your interest, click on the title to read more about it.

Note

that the summarisation here is lossy. The desired skills as here may be misrepresented, and if you are interested, you should check the original project for more details. In particular, many of the "desired skills" are often listed such that having only a few would be helpful, but this isn't consistent.

List of AISC Projects

To not build uncontrollable AI

1. Towards realistic ODDs for foundation model based AI offerings

Project Lead: Igor Krawczuk

Goal: Current methods for alignment applied to language models is akin to "blacklisting" behaviours that are bad.

Operational Design Domain

(OOD) is instead, akin to more exact "whitelisting" design principles, and now allowing deviations from this. The project wants to build a proof of concept, and show that this is hopefully feasible, economical and effective.

Team (Looking for 4-6 people):

"Spec Researcher": Draft the spec for guidelines, and publish a request for comments. Should have experience in safety settings
"Mining Researcher": Look for use cases, and draft the "slicing" of OOD.
"User Access Researcher": Write drafts on feasibility of KYC and user access levels.
"Lit Review Researcher(s)": Reading recent relevant literature on high-assurance methods for ML.
"Proof of Concept Researcher": build a proof of concept. Should have knowledge of OpenAI and interfacing with/architecting APIs.

2. Luddite Pro: information for the refined luddite

Project Lead: Brian Penny

Goal: Develop a news website filled with stories, information, and resources related to the development of artificial intelligence in society. Cover specific stories related to the industry and of widespread interest (e.g:

Adobe’s Firefly payouts

,

start of the Midjourney

,

proliferation of undress and deepfake apps

). Provide

valuable resources

(e.g:

list of experts

on AI,

book lists

, and pre-made letters/comments to

USCO

and&...
"Epistemic range of motion" and LessWrong moderation
by habryka — 2023-11-27T21:58:40.834Z

(Context for the reader: Gabriel reached out to me a bit more than a year ago to ask me to delete a few comments on this post by Jacob Hilton, who was working at OpenAI at the time. I referenced this in my recent dialogue with Olivia, where I quoted an email I sent to Eliezer about having some concerns about Conjecture partially on the basis of that interaction. We ended up scheduling a dialogue to talk about that and related stuff.)

habryka

You were interested in a dialogue, probably somewhat downstream of my conversation with Olivia and also some of the recent advocacy work you've been doing.

Gabriel Alfour

Yup.

Two things I'd like to discuss:

I was surprised by you (on a recent call) stating that you found LessWrong to be a good place for the Lying is Cowardice not Strategy post.
I think you misunderstand my culture. Especially around civility, and honesty.

habryka

Yeah, I am interested in both of the two things. I don't have a ton of context on the second one, so am curious about hearing a bit more.

Gabriel's principles for moderating spaces

Gabriel Alfour

About the second one:

I think people should be free to be honest in their private spaces.
I think people should be free to create their own spaces, enact their vision, and to the extent you participate in the space, you should help them.
If you invite someone to your place, you ought to not do things that would have caused them not to come if they knew ahead of time.

So, about my post and the OAI thing:

By 3, I feel ok writing my post on my blog. I feel ok with people dissing OAI on their blogs, and on their posts if you are ok with it (I take you as proxy for "person with vision for LW")
I feel much less ok about ppl dissing OAI on their own blog posts on LW. I assume that if they knew ahead of time, they would have been much less likely to participate.
I would have felt completely ok if you told me "I don't think your post has the tone required for LW, I want less adversariality / less bluntness / more charitability / more ingroupness"

How surprising are these to you?

Gabriel Alfour

Meta-comment: Would have been great to know that the thing with OAI shocked you enough to send a message to Eliezer about it.

Would have been much better from my point of view to talk about it publicly, and even have a dialogue/debate like this if you were already opened to it.

If you were already open to it, I should have offered. (

...
Apply to the Conceptual Boundaries Workshop for AI Safety
by Chris Lakin — 2023-11-27T21:04:59.037Z

Do you have experience with formal computer security, Active Inference, Embedded Agency, biological gap junctions, or other frameworks that distinguish agents from their environment? Apply to the Conceptual Boundaries Workshop for AI safety. February in Austin TX.

Website, more details, and application

Apply by December 22

For identifying, discussing, and strategizing about promising AI safety research directions pertaining to the boundaries that causally distance agents from their environment.

What are agent boundaries?

A few examples:

A bacterium uses its membrane to protect its internal processes from external influences.
A nation maintains its sovereignty by defending its borders.
A human protects their mental integrity by selectively filtering the information that comes in and out of their mind.

…a natural abstraction for safety?

Agent boundaries seem to be a natural abstraction representing the safety and autonomy of agents.

A bacterium survives only if its membrane is preserved.
A nation maintains its sovereignty only if its borders aren’t invaded.
A human mind maintains mental integrity only if it can hold off informational manipulation.

Maybe the safety of agents could be largely formalized as the preservation of their membranes.

These boundaries can then be formalized via Markov blankets.

Boundaries are also cool because they show a way to respect agents without needing to talk about their preferences or utility functions. Andrew Critch has said the following about this idea:

my goal is to treat boundaries as more fundamental than preferences, rather than as merely a feature of them. In other words, I think boundaries are probably better able to carve reality at the joints than either preferences or utility functions, for the purpose of creating a good working relationship between humanity and AI technology (

«Boundaries» Sequence, Part 3b

)

For instance, respecting the boundary of a bacterium would probably mean “preserving or not disrupting its membrane” (as opposed to knowing its preferences and satisfying them).

Protecting agents and infrastructure

By formalizing and preserving the important boundaries in the world, we could be in a better position to protect humanity from AI threats.

For example, critical computing infrastructure could be secured by creating strong boundaries around them. This can be enforced by cryptography and formal methods...
There is no IQ for AI
by Gabriel Alfour — 2023-11-27T18:21:26.196Z
Link: https://cognition.cafe/p/there-is-no-iq-for-ai

Most disagreement about AI Safety strategy and regulation stems from our inability to forecast how dangerous future systems will be. This inability means that even the best minds are operating on a vibe when discussing AI, AGI, SuperIntelligence, Godlike-AI and similar endgame scenarios. The trouble is that vibes are hard to operationalize and pin down. We don’t have good processes for systematically debating vibes.

Here, I’ll do my best and try to dissect one such vibe: the implicit belief in the existence of predictable intelligence thresholds that AI will reach.

This implicit belief is at the core of many disagreements, so much so that it leads to massively conflicting views in the wild. For example:

Yoshua Bengio writes an FAQ about

Catastrophic Risks from Superhuman AI

and Geoffrey Hinton

left Google to warn about these risks

. Meanwhile, the other Godfather of AI, Yann Lecunn, states that those concerns are overblown because we are “

nowhere near Cat-level and Dog-level AI

”. This is crazy! In a sane world we should anticipate technical experts to agree on technical matters, not to have completely opposite views predicated on vague notions of the IQ level of models.

People spend a lot of time arguing over

AI Takeoff speeds

which are difficult to operationalize. Many of these arguments are based on a notion of the general power level of models, rather than considering discrete AI capabilities. Given that the general power level of models is a vibe rather than a concrete fact of reality, it means disagreements revolving around them can’t be resolved.

AGI means

100 different things

, from

talking virtual assistants

in HER to OpenAI talking about “

capturing the light cone of all future value in the universe

”. The range of possibilities that are seriously considered implies “vibes-based” models, rather than something concrete enough to encourage convergent views.

Recent efforts to mimic Biosafety Levels in AI with a typology define

the highest risks of AI as “speculative

”. The fact that “speculative” doesn’t outright say “maximally dangerous” or “existentially dangerous” points also to “vibes-based” models. The whole point of Biosafety Levels is to define containment procedures for dangerous research. The most dangerous level should be the most serious and concrete one - the risks so obvious that we should work hard to prevent them from coming into existe...
Two concepts of an “episode” (Section 2.2.1 of “Scheming AIs”)
by Joe Carlsmith — 2023-11-27T18:01:29.153Z

(This is Section 2.2.1 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?”. There’s also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I’m hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section here, or search "Joe Carlsmith Audio" on your podcast app.)

Beyond-episode goals

Schemers are pursuing goals that extend beyond the time horizon of the
episode. But what is an episode?

Two concepts of an "episode"

Let's distinguish between two concepts of an episode.

The incentivized episode

The first, which I'll call the "incentivized episode," is the concept
that I've been using thus far and will continue to use in what follows.
Thus, consider a model acting at a time t1. Here, the rough idea is to
define the episode as the temporal unit after t1 that training
actively punishes the model for not optimizing – i.e., the unit of
time such that we can know by definition that training is not directly
pressuring the model to care about consequences beyond that time.

For example, if training started on January 1st of 2023 and completed on
July 1st of 2023, then the maximum length of the incentivized episode
for this training would be six months – at no point could the model have
been punished by training for failing to optimize over a
longer-than-six-month time horizon, because no gradients have been
applied to the model's policy that were (causally) sensitive to the
longer-than-six-month consequences of its actions. But the incentivized
episode for this training process could in principle be shorter than six
months as well. (More below.)

Now, importantly, even if training only directly pressures a model to
optimize over some limited period of time, it can still in fact create
a model that optimizes over some much longer time period – that's what
makes schemers, in my sense, a possibility. Thus, for example, if you're
training a model to get as many gold coins as possible within a ten
minute window, it could still, in principle, learn the goal "maximize
gold coins over all time" – and this goal might perform quite well (even
absent training gaming), or survive despite not performing all that well
(for example, because of the "slack" that training allows).

Indeed, to the exten...
[Linkpost] George Mack's Razors
by trevor — 2023-11-27T17:53:45.065Z
Link: https://twitter.com/george__mack/status/1728696014754676952

I don't use Twitter/X, I only saw this because it was on https://twitter.com/ESYudkowsky which I check every day (an example of a secure way to mitigate brain exposure to news feed algorithms).

The galaxy-brained word combinations here are at the standard of optimization I hold very highly (e.g. using galaxy-brained combinations of words to maximize the ratio of value to wordcount).

If someone were to, for example, start a human intelligence amplification coordination takeoff by getting the best of both worlds between the long, intuitive CFAR handbook and the short, efficient hammertime sequence, this is the level of writing skill that they would have to be playing at:

The most useful razors and rules I've found:

1. Bragging Razor - If someone brags about their success or happiness, assume it’s half what they claim

If someone downplays their success or happiness, assume it’s double what they claim

2. High Agency Razor - If unsure who to work with, pick the person that has the best chances of breaking you out of a 3rd world prison.

3. The Early-Late Razor - If it's a talking point on Reddit, you might be early. If it's a talking point on LinkedIn, you're definitely late.

4. Luck Razor - If stuck with 2 equal options, pick the one that feels like it will produce the most luck later down the line.

I used this razor to go for drinks with a stranger rather than watch Netflix. In hindsight, it was the highest ROI decision I've ever made.

5. Buffett's Law - "The value of every business is 100% subject to government interest rates" - Warren Buffett

6. The 6-Figure Razor - If someone brags about "6 figures" -- assume it's closer to $100K than $900K.

7. Parent Rule - Break down the investments your parents made in you: Time, Love, Energy, and Money.

If they are still alive, aim to hit a positive ROI (or at least break even.)

8. Instagram Razor - When you see a photo of an influencer looking attractive on Instagram -- assume there are 99 worse variations of that photo you haven't seen.

They just picked the best one.

9. Narcissism Razor - If worried about people's opinions, remember they are too busy worrying about other people's opinions of them. 99% of the time you're an extra in someone else's movie

10. Everyday Razor - If you go from doing a task weekly to daily, you achieve 7 years of output i

...
On possible cross-fertilization between AI and neuroscience [Creativity] 
by Bill Benzon — 2023-11-27T16:50:26.531Z

Cross-posted from New Savanna.

MIT Center for Minds, Brains, and Machines (CBMM), a panel discussion: CBMM10 - A Symposium on Intelligence: Brains, Minds, and Machines.

On which critical problems should Neuroscience, Cognitive Science, and Computer Science focus now? Do we need to understand fundamental principles of learning -- in the sense of theoretical understanding like in physics -- and apply this understanding to real natural and artificial systems? Similar questions concern neuroscience and human intelligence from the society, industry and science point of view.

Panel Chair: T. Poggio
Panelists: D. Hassabis, G. Hinton, P. Perona, D. Siegel, I. Sutskever

Quick Comments

1.) I’m a bit annoyed that Hassabis is giving neuroscience credit for the idea of episodic memory. As far as I know, the term was coined by a cognitive psychologist named Endel Tulving in the early 1970s, who stood it in opposition to semantic memory. That distinction was all over the place in the cognitive sciences in the 1970s and its second nature to me. When ChatGPT places a number of events in order to make a story, that’s episodic memory.

2.) Rather than theory, I like to think of what I call speculative engineering. I coined the phrase in the preface to my book about music (Beethoven’s Anvil), where I said:

Engineering is about design and construction: How does the nervous system design and construct music? It is speculative because it must be. The purpose of speculation is to clarify thought. If the speculation itself is clear and well-founded, it will achieve its end even when it is wrong, and many of my speculations must surely be wrong. If I then ask you to consider them, not knowing how to separate the prescient speculations from the mistaken ones, it is because I am confident that we have the means to sort these matters out empirically. My aim is to produce ideas interesting, significant, and clear enough to justify the hard work of investigation, both through empirical studies and through computer simulation.

3.) On Chomsky (Hinton & Hassabis): Yes, Chomsky is fundamentally wrong about language. Language is primarily a tool for conveying meaning from one person to another and only derivatively a tool for thinking. And he’s wrong that LLMs can learn any language and therefore they are useless for the scientific study of language. Another problem with Chomsky’s thinking is that h...