trentmkelly/LessWrong-43k huggingface dataset:
Next automated reasoning grand challenge: CompCert
by sanxiyn — 2024-08-20T05:27:52.577Z

Recently automated reasoning system was developed to solve IMO problems. It is a very impressive and exciting advance, but it must be noted IMO problems are to be solved in two days.

Recently it was also proposed to make use of this advance to greatly automate formal verification of practical systems, including advanced AI systems. That's a very tall goal, so some kind of intermediate benchmark would be useful.

I suggest CompCert as a such benchmark. It is a formally verified compiler with manually written proof that compiles a large and practical subset of C programming language to (among other things) x86 assembly language, with optimizations such that performance is competitive with GCC -O1. Whether its formal specification corresponds to C programming language standard and Intel architecture manual written in English is a fascinating question, but more than a decade of testing since completion of its proof suggests it does. The initial proof took three years (from 2005 to 2008), and Xavier Leroy won ACM Software System Award for this work. So the task is an R&D level task.

The challenge is: automated reasoning system (like AlphaProof) will be given formal specification of C programming language standard, Intel architecure manual, and CompCert's compilation algorithm as input. The task is to prove it correct, by filling in all of manually written proofs automatically. 99% automation with manually written hints (say, 10 pages of proof sketch -- which happens to be length of CompCert overview paper -- written over 10 days) would be also interesting.

In many ways this is much easier task than what we want. Known to be correct compilation algorithm is given as input, and real to formal translation is already done. Compiler is also designed and written to be verified. Compared to automatically (or semi-automatically, with 99% automation) fixing bugs in GCC and extending formal specification to cover more and verifying not-designed-for-verification software like GCC correct, this is a toy task. But it is also a task that would take a researcher many months, not days. (Probably not years, since a lot is given as input this is a subset task of initial proof and we also have advanced since 2008.)
Thiel on AI & Racing with China
by Ben Pace — 2024-08-20T03:19:18.966Z

This post is a transcript of part of a podcast with Peter Thiel, touching on topics of AI, China, extinction, Effective Altruists, and apocalyptic narratives, published on August 16th 2024.

If you're interested in reading the quotes, just skip straight to them, the introduction is not required reading.

Introduction

Peter Thiel is probably known by most readers, but briefly: he is an venture capitalist, the first outside investor in Facebook, cofounder of Paypal and Palantir, and wrote Zero to One (a book I have found very helpful for thinking about building great companies). He has also been one of the primary proponents of the Great Stagnation hypothesis (along with Tyler Cowen).

More local to the LessWrong scene, Thiel was an early funder of MIRI and a speaker at the first Effective Altruism summit in 2013. He funded Leverage Research for many years, and also a lot of anti-aging research, and the seasteading initiative, and his Thiel Fellowship included a number of people who are around the LessWrong scene. I do not believe he has been active around this scene much in the last ~decade.

He appears rarely to express a lot of positions about society, and I am curious to hear them when he does.

In 2019 I published the transcript of another longform interview of his here with Eric Weinstein. Last week another longform interview with him came out, which I listened to.

I got the sense from listening to it that even though we are in conflict on some issues, conversation with him would be worthwhile and interesting. Then about 3 hours in he started talking more directly about subjects that I think actively about and some conflicts around AI, which I think will be of interest to many here, so I've quoted the relevant parts below.

His interviewer, Joe Rogan is a very successful comedian and podcaster. He's not someone who I would go to for insights about AI. I think of him as standing in for a well-intentioned average person, for better or for worse, although he is a little more knowledgeable and a little more intelligent and a lot more curious than the average person. The average Joe. I believe he is talking in good faith to the person before him, and making points that seem natural to many.

Artificial Intelligence

Discussion focused on the AI race and China, atarting at 2:56:40. The opening monologue by Rogan is skippable.

Rogan
If you look at this mad rush for art

...
Reflecting on the transhumanist rebuttal to AI existential risk and critique of our debate methodologies and misuse of statistics
by catgirlsruletheworld — 2024-08-20T01:59:13.338Z

This morning, I read an article that counter-argues many of our arguments about AI existential risk, and I'm curious what other people's thoughts are on the arguments, linked here: https://darkempressofthevoid.substack.com/p/fck-decels-accelerate. I outlined my thoughts below, which include both some criticism, though I'm unsure if I'm simply being too cynical, and I also refer to a topic from the article regarding our argument methodologies and tactics in which I agree with the author, providing an example that exemplifies the author's argument that I found today.

From my perspective, the notion that AI might one day surpass human intelligence and spiral beyond our control is not just a speculative concern; it's a critical issue that demands serious attention. While some may dismiss these worries as alarmist or reactionary, I see them as grounded in a rational assessment of the potential risks AI poses to humanity. The transhumanist vision of a harmonious co-evolution between humans and AI is certainly appealing in theory, but it strikes me as overly optimistic and potentially naive.

The idea that we can simply guide AI's development from within, ensuring it remains aligned with human values, assumes a level of control that history suggests we may not possess. Technological ecosystems are complex, and once certain thresholds are crossed, they can develop dynamics that are difficult, if not impossible, to steer. The fear that AI might spiral beyond our control is not rooted in a misunderstanding of technology, but in a clear-eyed recognition of the unpredictability inherent in rapidly advancing systems. AI safety researchers advocate for cautious, well-reasoned approaches to managing these risks precisely because we understand that the stakes could not be higher.

While I appreciate the transhumanist enthusiasm for AI as a tool to transcend our biological limitations, I can't help but see this vision as dangerously complacent. It assumes that technological progress is inherently beneficial and that the right alignment will naturally emerge from the integration of AI with our minds and bodies. But history is replete with examples of well-intentioned innovations that led to unforeseen consequences. To suggest that we can fully control AI's trajectory seems to underestimate the complexities involved and overestimate our ability to foresee and manage every pot...
Artificial Intelligence and Eternal Torture and Suffering
by Tuan Tu Nguyen — 2024-08-20T01:53:18.772Z

Abstract

The development of artificial intelligence brings with it extremely important ethical questions, one of the most alarming being the possibility of eternal torture and suffering inflicted by AI on humans. This paper considers theoretical implications and ethical consequences of the development of highly advanced AI that is potentially able to condemn humans to eternal suffering. While AI might have considerable benefits, the potential catastrophes in terms of outcome–and particularly, the possibility of eternal torture and suffering–should be taken seriously. The risk of AI development can be so enormously severe as to outweigh potential advantages. It is actually better to die than to be tortured eternally. The precautionary principle states that in the absence of absolute assurances against catastrophic results, development of advanced AI should be carried out with utmost caution or be completely avoided. Such stance demonstrates a moral perspective to prioritize human safety over unrestrained technological progress.

Introduction

The artificial intelligence debate has become increasingly heated over the past decade as breakthroughs in AI technologies dramatically redefine what man and machine are capable of. On the verge of more autonomous and sophisticated AI systems, concerns of an ethical nature are rapidly moving to center stage in the debate. Of these many concerns, one stands out for its chilling implication: that an advanced AI might–intentionally or unintentionally–condemn humans to eternal torture and suffering, a fate arguably worse than death. The likelihood of such an event might be very slim, but it is so cataclysmic in consequences that it certainly deserves to be taken seriously. In this paper, I will argue that with the risks in mind, we should be most careful–or rather, completely avoid–creating AI with a view toward avoiding possibilities that could lead to these dreadful outcomes.

The Nature of AI and Ethnical Concerns

Artificial intelligence is fundamentally based on dealing with data, recognizing patterns, and making decisions by using algorithms. The smarter the AI, the more autonomous and capable it becomes. However, immense ethical concerns come with such capability. Perhaps one of the most important is that it could create goals and behaviors that are no longer in line with human values and interests. This misalignme...
AI #77: A Few Upgrades
by Zvi — 2024-08-20T00:20:09.717Z

[Apologies for forgetting to cross-post this and the Monthly Roundup earlier.]

Let’s see. We’ve got a new version of GPT-4o, a vastly improved Grok 2 with a rather good and unrestricted deepfake and other image generator now baked into Twitter, the announcement of the AI powered Google Pixel 9 coming very soon and also Google launching a voice assistant. Anthropic now has prompt caching.

Also OpenAI has its final board member, Zico Kolter, who is nominally a safety pick, and SB 1047 got importantly amended again which I’ll cover in full next week once the details are out.

There was also the whole paper about the fully automated AI scientist from the company whose name literally means ‘danger’ in Hebrew, that instantiated copies of itself, took up unexpectedly large amounts of storage space, downloaded strange Python libraries and tried to edit its code to remove the timeout condition. Oh, that.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Many quality of life improvements.

Language Models Don’t Offer Mundane Utility. If you look for failure, it’s there.

GPT-4o My System Card. Better late than never, if a bit obsolete.

2 Grok 2 Furious 2 Quit. Welcome to Deepfaketown, population Twitter.

Pixel Perfect. The first AI integrated phone is almost here.

Fun With Image Generation. Two free DALL-E creations per day.

Deepfaketown and Botpocalypse Soon. How do you prove you are a real person?

The Art of the Jailbreak. Anthropic offers bounties, make check out to Pliny.

They Took Our Jobs. It gets lonely. Also see that AI scientist thing.

Obvious Nonsense. GPT-2 indeed poses no existential threat to humanity.

Get Involved. SFF launching a new funding round, and a shameless plug.

Introducing. New GPT-4o update, Anthropic offers prompt caching.

In Other AI News. Zico Kolter joins the OpenAI board.

Quiet Speculations. If you don’t see AI progressing you have no patience.

SB 1047: One Thing to Know. The most basic thing. People don’t know.

SB 1047 is Amended Again. Improvements, also big compromises.

SB 1047 Rhetoric Prior to the Changes. Preserved for posterity.

The Quest for Sane Regulations. Conditional commitments and partial progress.

The Week in Audio. Zuckerberg, Russell and Hassabis.

Rhetorical Innovation. Responses to Altman, and exploration on wh

...
Monthly Roundup #21: August 2024
by Zvi — 2024-08-20T00:20:08.178Z

Strictly speaking I do not have that much ‘good news’ to report, but it’s all mostly fun stuff one way or another. Let’s go.

Bad News

Is this you?

Patrick McKenzie: This sounds like a trivial observation and it isn’t:

No organization which makes its people pay for coffee wants to win.

There are many other questions you can ask about an organization but if their people pay for coffee you can immediately discount their realized impact on the world by > 90%.

This is not simply for the cultural impact of stupid decisions, though goodness as a Japanese salaryman I have stories to tell. Management, having priced coffee, seeking expenses to cut, put a price on disposable coffee cups, and made engineers diligently count those paper cups.

Just try to imagine how upside down the world is when you think one of the highest priority tasks for a software engineer this Monday is updating the disposable coffee cup consumption spreadsheet.

And no, Japanese megacorps are not the only place where these insanities persist. And there are many isomorphic ones.

Dominic Cummings: Cf No10 Cafe.

One of the secrets of my productivity, such as it is, is that I know many (but not all!) of the things to not track or treat as having a price. Can you imagine thinking it was a good idea to charge the people at No10 for coffee? Well, bad news.

Tyler Cowen asks, why do we no longer compose music like Bach? Or rather, why do we not care when someone does, as when Nikolaus Matthes (born 1981) produced high quality (if not as high quality as Bach’s best) Bach-style work. All reviews strongly positive, stronger than many older musicians who are still popular, yet little interest.

To me the answer is simple enough. There is quite a lot of Bach, and many contemporaries, and we have filtered what is available rather well and turned it into a common frame of reference. One could listen to that music all of one’s life, and there is still plenty of it. Why complicate matters now with modern mimicry, even if it is quite good? In popular music there are cultural reasons to need ‘new music’ periodically even if it is only variation on the old, yet we are increasingly converging on the classic canon instead except for particular ‘new music’ spaces. And I think we are right to do so.

The fabrication of the Venezuelan election wasn’t even trying. This matches my model. Yes, it is possible to gene...
[Linkpost] Automated Design of Agentic Systems
by Bogdan Ionut Cirstea — 2024-08-19T23:06:06.669Z
Link: https://arxiv.org/abs/2408.08435

Authors: Shengran Hu, Cong Lu, Jeff Clune.

Brief summary: proof of concept of automated LM scaffolding design, with state-of-the-art performance on several tasks.

Website: https://www.shengranhu.com/ADAS/.

X/twitter summary thread: https://x.com/jeffclune/status/1825551351746867502.

Abstract:

Researchers are investing substantial effort in developing powerful general-purpose agents, wherein Foundation Models are used as modules within agentic systems (e.g. Chain-of-Thought, Self-Reflection, Toolformer). However, the history of machine learning teaches us that hand-designed solutions are eventually replaced by learned solutions. We formulate a new research area, Automated Design of Agentic Systems (ADAS), which aims to automatically create powerful agentic system designs, including inventing novel building blocks and/or combining them in new ways. We further demonstrate that there is an unexplored yet promising approach within ADAS where agents can be defined in code and new agents can be automatically discovered by a meta agent programming ever better ones in code. Given that programming languages are Turing Complete, this approach theoretically enables the learning of any possible agentic system: including novel prompts, tool use, control flows, and combinations thereof. We present a simple yet effective algorithm named Meta Agent Search to demonstrate this idea, where a meta agent iteratively programs interesting new agents based on an ever-growing archive of previous discoveries. Through extensive experiments across multiple domains including coding, science, and math, we show that our algorithm can progressively invent agents with novel designs that greatly outperform state-of-the-art hand-designed agents. Importantly, we consistently observe the surprising result that agents invented by Meta Agent Search maintain superior performance even when transferred across domains and models, demonstrating their robustness and generality. Provided we develop it safely, our work illustrates the potential of an exciting new research direction toward automatically designing ever-more powerful agentic systems to benefit humanity.
Limitations on Formal Verification for AI Safety
by Andrew Dickson — 2024-08-19T23:03:52.706Z

In the past two years there has been increased interest in formal verification-based approaches to AI safety. Formal verification is a sub-field of computer science that studies how guarantees may be derived by deduction on fully-specified rule-sets and symbol systems. By contrast, the real world is a messy place that can rarely be straightforwardly represented in a reductionist way. In particular, physics, chemistry and biology are all complex sciences which do not have anything like complete symbolic rule sets. Additionally, even if we had such rules for the natural sciences, it would be very difficult for any software system to obtain sufficiently accurate models and data about initial conditions for a prover to succeed in deriving strong guarantees for AI systems operating in the real world.

Practical limitations like these on formal verification have been well-understood for decades to engineers and applied mathematicians building real-world software systems, which makes it puzzling that they have mostly been dismissed by leading researchers advocating for the use of formal verification in AI safety so far. This paper will focus-in on several such limitations and use them to argue that we should be extremely skeptical of claims that formal verification-based approaches will provide strong guarantees against major AI threats in the near-term.

What do we Mean by Formal Verification for AI Safety?

Some examples of the kinds of threats researchers hope formal verification will help with come from the paper “Provably Safe Systems: The Only Path to Controllable AGI” [1] by Max Tegmark and Steve Omohundro (emphasis mine):

Several groups are working to identify the greatest human existential risks from AGI. For example, the Center for AI Safety recently published ‘An Overview of Catastrophic AI Risks’ which discusses a wide range of risks including bioterrorism, automated warfare, rogue power seeking AI, etc. Provably safe systems could counteract each of the risks they describe.

These authors describe a concrete bioterrorism scenario in section 2.4: a terrorist group wants to use AGI to release a deadly virus over a highly populated area. They use an AGI to design the DNA and shell of a pathogenic virus and the steps to manufacture it. They hire a chemistry lab to synthesize the DNA and integrate it into the protein shell. They use AGI controlled drones

...
The Conscious River: Conscious Turing machines negate materialism 
by blallo — 2024-08-19T21:54:03.394Z

Many computer scientists and mathematically inclined people subscribe to the idea that consciousness is a phenomenon that emerges from matter. They also believe that any Turing machine can generate consciousness if provided with the correct program. This seems obvious due to the universality of Turing machines. If a Turing machine can simulate a human brain and consciousness arises from the matter in the brain, then the Turing machine would be conscious as well.

In this post, I propose a thought experiment that starts with the assumption that Turing machines can generate consciousness. I will show that from such a Turing machine, it is possible to build canals and pipes that constraint a river so that it performs the same algorithm, but violates our common-sense understanding of what it means to be conscious. From this, I infer that either consciousness is an illusion and nothing is truly conscious (materialism is wrong), Turing machines cannot be conscious, or common-sense properties of consciousness are not real.

While I don't think that what is written here is anything particularly new for illusionists or materialists that do no believe that turning machines are automatically conscious if provided with the right program, I think that this article may help computationalists to understand materialists and illusionists positions.

Definitions

Consciousness: The collection of experiences that appear moment by moment in the minds of living beings, distinct from unconscious processes.

Illusionism: The view that phenomenal consciousness is an illusion; that is, living beings merely have the impression of being conscious, but they are not. (It’s not that Turing machines are conscious, but rather that humans are not conscious either.)

Materialism: The view that phenomenal consciousness emerges from matter.

Computationalism: The view that phenomenal consciousness emerges from certain kinds of computation and thus can arise from Turing machines, regardless of the material substrate.

Philosophical Zombie: A human-like entity that behaves exactly like a human but is not conscious.

If you already believe a river can run a program, you don't care about seeing how it may be the case, skip to the analysis section and carry on from there.

Building a Conscious River

The first part of this article will focus on something that should be immediately obvious to computer scientis...