trentmkelly/LessWrong-43k huggingface dataset:
The price is right
by EJT — 2023-10-16T16:34:38.023Z
Link: https://openairopensea.substack.com/p/the-price-is-right

[Saying an old thing in a new way]

The United States Department of Transportation will pay $11.8 million to save a life. You know what that means? It means that if you come to the United States Department of Transportation with a plan (barriers around the Grand Canyon, wider lanes on the expressway, no left-turns on Sundays, etc. etc. etc.), the United States Department of Transportation will take your plan – snatch the blueprints right out of your hand – and go away and calculate two numbers. The first is the cost: the cold hard cash required to make your plan a reality. The second is the expected number of Americans saved by your plan. They’ll think of every number of Americans that your plan could save and couple it up with the chance that your plan in fact saves that number of Americans. They’ll have each of these pairs be fruitful and multiply and then they’ll sum. I exaggerate and simplify, but only slightly.

Now the United States Department of Transportation – or the US Department of Transportation, as I’m sometimes bold enough to call them – has your two numbers: the cost (measured in dollars) and the benefit (measured in Americans). You’re summoned back into the head honcho’s office and she punches these numbers into the ceremonial calculator: dollars divided by American lives. If the answer is $11.8 million per life or less, she shakes your hand and writes a cheque. If the answer is more than $11.8 million per life, she says ‘Take a hike. Never talk to me again.’ I exaggerate and simplify, but only slightly.

Now. I am an American citizen; I care only about the two inevitables. Do you know what I mean by ‘the inevitables’?

Dash? Elastigirl?

No, no, no. The inevitables: death and taxes. I am an American citizen; I care only about death and taxes. I exaggerate and simplify, but only slightly.

Now every time the head honcho at the US DoT signs a cheque, my taxes go up. That’s b...
Dating in 2023 sucks. Why isn't AI helping?
by Andreas Chrysopoulos — 2023-10-16T12:31:04.346Z

So here's what I'm thinking.

I would argue that main bottleneck for people having an abundance of good potential partners is usually not their character or personality. Which sure can be improved, but it's not the main obstacle.

I believe the main issue is the bad matchmaking process. Meeting through friends, or chatting someone at the bar/club/restaurant, or at ones hobbies is just inefficient. How many new potential partners is the average person gonna meet in a month? 10? 50? 100? And most people are even too afraid to even start a conversation with a stranger.

We live in cities of millions... how many opportunities are being missed for amazing relationships?

So here is the question: How hard is it for an AI to be trained to match people? Is there anything happening in this area currently? And if not what are the main obstacles? If we managed to train it to respond with a certain amount of common sense (GPT4), even if it's 50% as accurate as a human at matchmaking, given the profiles of potential candidates, it would be huge massively more successful than Tinder.

NOTE: I'm assuming that the AI would be provided a wealth of information about the candidates: there could be a long questionnaire to be filled, an interview with the AI itself, access to socials, pictures and videos, etc.

UPDATE: While writing this I'm starting to realise the immense processing power required for something like this, and that we're not exactly there yet, but a simpler version could be made. Something where you just answer different questions about what you're looking for, what you want to do in your life, etc. Sure it would depend on trust, but incorrect profiles could be flagged and deleted.

And still, I think in a few years the AI will be there technology wise, to be able to conduct and interview and gauge the personality and characteristics of a person.

So again, what do you guys think? How hard is it for an AI to be trained to match people? Is there anything happening in this area currently? And if not what are the main obstacles?
Knowledge Base 4: General applications
by iwis — 2023-10-16T12:26:42.580Z

This post presents some possible applications of the general knowledge base that can be built using information management methods described in the post Knowledge Base 2: The structure and the method of building. Applications of the knowledge base constrained to products available on the market have been presented in the post Knowledge Base 3: Shopping advisor and other uses of knowledge base about products.

Similarly to the knowledge base about products, we can build a knowledge base on the following topics:

information about shops – e.g. opening hours, available products
Thanks to this information, the system will be able to answer the question: Where nearby can I buy windscreen wipers proper for my car? (exclude offers that are not optimal for me and show the most important details of optimal solutions if there are several such solutions, according to the criteria specified in my profile)
information about services: hotels, restaurants, repairmen, taxis, rooms for short-term renting
Examples of tasks: I want to rent a single room for several days that meets the criteria set in my profile, or I want to go to some place by bus, taxi, bike or using carpooling (use criteria set in my profile to propose a proper transport)
information about individual people: who they are, what they can do, what they have, what they want, …
Examples of tasks: Find people near me willing to start jogging, or Find a person near me able to fix a problem with the blinkers in my car (show estimated repair time, sort offers by price)
information about things
Example task: Check if someone has already reported the failure of traffic lights at an intersection. If not, report it; if yes, when is the repair scheduled?

The database could also contain information about many other objects/items/terms defined by users using the first, second, and third type of information (see figure 3 in the post Knowledge Base 2: The structure and the method of building).

The knowledge base could also store variable data – for example GPS locations of buses, taxis, and private cars (carpooling). Normally, all previous information values would be stored in the knowledge base but it can be disabled when needed for performance reasons.

A knowledge base user should be able to subscribe to properties of objects/items/terms. The user will be notified if the subscribed property of an object is set to a certain value. Thanks...
UNGA General Debate speeches on AI
by Odd anon — 2023-10-16T06:36:38.866Z

At the UN General Assembly's 2023 General Debate on September 19-26, many leaders spoke about artificial intelligence, and some specifically about existential risk and what to do about it. Attitudes ranged from advocating serious global action (Israel, the UK) to being completely dismissive (Malta). The following is a transcript of the parts of those speeches dealing with AI, sorted chronologically. Some particularly relevant parts bolded by me.

These speeches can be useful for getting a rough sense of various nations' stances on the issue, and indicate surprisingly strong support from the world's governments for taking AI risk seriously.

Day 1

United Nations

Delivered by UN Secretary-General António Guterres (link)

We must also face up to the looming threats posed to human rights by new technologies. Generative artificial intelligence holds much promise – but it may also lead us across a Rubicon and into more danger than we can control.

When I mentioned Artificial Intelligence in my General Assembly speech in 2017, only two other leaders even uttered the term. Now AI is on everyone's lips – a subject of both awe, and fear. Even some of those who developed generative AI are calling for greater regulation.

But many of the dangers of digital technology are not looming. They are here. The digital divide is inflaming inequalities. Hate speech, disinformation and conspiracy theories on social media platforms are spread and amplified by AI, undermining democracy and fueling violence and conflict in real life. Online surveillance and data harvesting are enabling human rights abuses on a mass scale. And technology companies and governments are far from finding solutions.

Excellencies,

We must move fast and mend things. New technologies require new and innovative forms of governance – with input from experts building this technology and from those monitoring its abuses. And we urgently need a Global Digital Compact -- between governments, regional organizations, the private sector and civil society -- to mitigate the risks of digital technologies, and identify ways to harness their benefits for the good of humanity.

Some have called for consideration of a new global entity on AI that could provide a source of information and expertise for Member States.

There are many different models -- inspired by such examples as the International Atomic Energy Agency, the Internationa...
AI Alignment [Incremental Progress Units] this week (10/08/23)
by Logan Zoellner — 2023-10-16T01:46:56.193Z
Link: https://midwitalignment.substack.com/p/ai-alignment-breakthroughs-this-week-d97

[edit: took out naming controversy stuff, as it was distracting from the point of the blog]
I am introducing a new rating system for each alignment breakthrough. The rating system will go from 1 star⭐ to 5 stars ⭐⭐⭐⭐⭐.

A 1 star ⭐ “breakthrough” represents incremental progress. This means, that while technically achieving a new milestone, this breakthrough was the result of known techniques and could have been easily predicted in advance by an expert in the field. An example of something I’ve posted in the past that should be considered 1 star ⭐ is Wustchen V3. An example of a hypothetical 1 star ⭐ breakthrough would be if RLHF on GPT-5 was found to work better than RLHF on GPT-4.

A 5 star ⭐⭐⭐⭐⭐ “breakthrough” represents a discovery that solves a significant unsolved problem that was considered to be a major obstacle on at least one AI Alignment path. An example of a 5 star ⭐⭐⭐⭐⭐ breakthrough that I’ve posted in the past would be neuron superposition. An example of a hypothetical 5 star ⭐⭐⭐⭐⭐ breakthrough would be if someone were to develop a system that could translate a human-language description of a math problem into a formal mathematical proof.

Now, without any further ado…

AI Alignment Breakthroughs this Week

This week there were breakthroughs in the areas of:

AI Evaluation

AI Agents

Mechanistic Interpretability

Explainable AI

Simulation

Making AI Do what we want

AI Art

AI Evaluation

PCA-Eval

What is it: a new benchmark for multi-modal decision making

What is new: evaluate multimodal models (like GPT-4V) by their ability to make decisions in different domains

What is it good for: Benchmarking is key for many AI safety strategies such as the Pause and RSPs

Rating: ⭐⭐

AI Agents

Adapting LLM Agents Through Communication

What is it: Improved AI agents

What is new: By fine-tuning the LLM, the agents can perform better

What is it good for: Factored Congnition, Bueracracy of AIs

Rating: ⭐

Mechanistic Interpretability

Research on infinite-width neural networks

What is it: research showing the behavior of infinite (large number of) parameter LLMs

What’s new: a specific map showing when NNs will under/overtrain

What is it good for: determining the stability of neural networks as they scale up

Rating:⭐⭐⭐

Reverse-engineering LLM components

What is it: research to understand LLM components

What’s new: discovery of “copy supress” heads that prevent the LLM from repeating...
Goodhart's Law in Reinforcement Learning
by jacek — 2023-10-16T00:54:11.669Z

Produced As Part Of The OxAI Safety Labs program, mentored by Joar Skalse.

TL;DR

This is a blog post introducing our new paper, "Goodhart's Law in Reinforcement Learning" (to appear at ICLR 2024). We study Goodhart's law in RL empirically, provide a geometric explanation for why it occurs, and use these insights to derive two methods for provably avoiding Goodharting.

Here, we only include the geometric explanation (which can also be found in the Appendix A) and an intuitive description of the early-stopping algorithm. For the rest, see the linked paper.

Introduction

Suppose we want to optimise for some outcome, but we can only measure an imperfect proxy that is correlated, to a greater or lesser extent, with that outcome. A prototypical example is that of school: we would like children to learn the material, but the school can only observe the imperfect proxy consisting of their grades. When there is no pressure exerted on students to do well on tests, the results will probably reflect their true level of understanding quite well. But this changes once we start putting pressure on students to do get good grades. Then, the correlation usually breaks: students learn "for the test", cheat, or use other strategies which make them get better marks without necessarily increasing their true understanding.

That "any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes" (or "when a measure becomes a target, it ceases to be a good measure") is called the Goodhart's law. Graphically, plotting the true reward and the proxy reward obtained along increasing optimisation pressure, we might therefore expect to see something like the following graph:

In machine learning, this phenomenon has been studied before. We have a long list lot of examples where agents discover surprising ways of getting reward without increasing their true performance. These examples, however, have not necessarily been quantified over increasing optimisation pressure: we only know that agents discovered those "reward hacks" at some point in their training. Even in cases where the effect was attributed specifically to the increasing optimisation pressure, we lacked a true, mechanistic understanding for why it happens.

Our paper fills this gap. The core contribution is an explanation for Goodhart's law in reinforcement learning in terms of the geometry of ...
My AI Predictions 2023 - 2026
by HunterJay — 2023-10-16T00:50:52.968Z

Epistemic status: My, mostly intuitive, guesses - with only a few days dwelling on it, and no serious research beyond what I already knew.
I work in the startup sphere, in field robotics, and I am about to have an opportunity to majorly shift what I am working on. To work out what projects might make sense on a multi-year time frame, I wrote up what I thought might happen in AI in the next couple of years as specifically as I could.
I found the exercise surprisingly useful. It turned a whole bunch of vague "X will get better over time" to actionable "X will be practical in around Y years". I don't think my guesses will end up actually being very accurate, but having something solid forced me to actually think about the future and make my gibberish-internal-intuitions into more-consistent-guesses. I was really surprised at how much it helped, actually.
So, here's that list of predictions. I'm sharing it here more as a "here's how you can do something similar" than as a "here's a well researched report on the future trends." (which it definitely is not). I didn't go to the trouble of putting my %'s on the guesses "X in year Y", but it's about 40%-70% for any given guess.
________________________________
[Written 6th October 2023]
Rest of 2023:

Small improvements to LLMs

Google releases something competitive to ChatGPT.
Anthropic and OpenAI slightly improve GPT-4 and Claude2
Meta or another group releases better open source models, up to around GPT-3.5 level.

Small improvements to Image Generation

Dalle3 gets small improvements.
Google or Meta releases something similar to Dalle3, but not as good.

Slight improvements to AI generated videos.

Basic hooking up of Dalle3 to video generation with tagged on software, not really good consumer stuff yet. Works in an interesting way, like Dalle1, but not useful for much yet.

Further experiments hooking LLMs up to robotics/cars, but nothing commercial released.
Small improvements in training efficiency and data usage, particularly obviously in smaller models becoming more capable than older, larger ones.

2024

GPT-5 or equivalent is released.

It’s as big a jump on GPT-4 as GPT-4 was on GPT-3.5.
Can do pretty much any task when guided by a person, but still gets things wrong sometimes.
Multimodal inputs, browsing, and agents based on it are all significantly better.

Agents can do basic tasks on computers -- like filling in forms, wor

...
Taxonomy of AI-risk counterarguments
by Odd anon — 2023-10-16T00:12:51.021Z

Partly inspired by The Crux List, the following is a non-comprehensive taxonomy of positions which imply that we should not be worried about existential risk from artificial superintelligence.

Each position individually is supposed to be a refutation of AI X-risk concerns as a whole. These are mostly structured as specific points of departure from the regular AI X-risk position, taking the other areas as a given. This may result in skipping over positions which have multiple complex dependencies.

Some positions are given made-up labels, including each of the top-level categories: "Fizzlers", "How-skeptics", "Why-skeptics", "Solvabilists", and "Anthropociders".

(Disclaimer: I am not an expert on the topic. Apologies for any mistakes or major omissions.)

Taxonomy

"Fizzlers": Artificial superintelligence is not happening.

AI surpassing human intelligence is fundamentally impossible (or at least practically impossible).

True intelligence can only be achieved in biological systems, or at least in systems completely different from computers.

Biological intelligences rely on special quantum effects, which computers cannot replicate.
Dualism: The mental and physical are fundamentally distinct, and non-mental physical constructions cannot create mental processes.
Intelligence results from complex, dynamic systems of a kind which cannot be modeled mathematically by computers.

Mysterianists: A particular key element of human thinking, such as creativity, common sense, consciousness, or conceptualization, is so beyond our ability to understand that we will not be able to create an AI that can achieve it. Without this element, superintelligence is impossible.
Intelligence isn't a coherent or meaningful concept. Capability gains do not generalize.
There is a fundamental ceiling on intelligence, and it is around where humans are.

"When-skeptics": ASI is very, very far away.

Moore's Law is stopping, scaling will hit fundamental limits, training data is running out and can't be easily supplemented, algorithmic improvements will level off, and/or other costs will skyrocket as AI gets better.
Existing methods will peak in capabilities, and future development will continue down an entirely different path, greatly delaying progress.
Biological anchors point to ASI taking a very long time.
In general, either in large engineering projects or AI in particular, progress tends to be more difficult tha

...
Thoughts on Hardware limits to Prevent AGI?
by jrincayc — 2023-10-15T23:45:38.316Z

Summary: An individual Commodore 64 is almost certainly safe, Top 10 super computers could almost certainly run a superpowerful AGI, but where is the safe line, and how would we get to the safe side?

I started thinking about this topic when I realized that we can safely use uranium because we have a field of nuclear criticality safety[1] but we have no field of computer foom safety (or Artificial General Intelligence takeoff safety).[2] For example, if we had such a field we might be able to have a function AGIT(architecture, time, flops, memory) → Bool to tell us if a computer could take off into an AGI or not with that amount of resources. Making this a total function (giving a value for all of its domain) might not be possible, but even a partial function could be useful. Note that by computer foom safety my worry is that an AI project will result in a powerful AGI that is neither controllable nor ethical and either results in a world substantially worse than humans would create on our own or results in humanity dying. The three main failure modes I worry about are 1, the AGI's utility function does not care about preventing sentient being deaths, 2, the AGI uses up substantial portions of the resources in the universe, and 3, the AGI does not get consent before "helping" sentient beings.[3] Note that an alternative way to preventing AGI by restricting hardware (an engineering control) is restricting AI programs from running on computers (an administrative control).

Alien Computer Instructions

A possible scenario where we would actually want to know what computers were provably safe is the following science fiction introduction:

Astronomers sighted a incoming interstellar object as it enters the solar system. Humans manage to send out a probe to fly by it, and discover it is artificial. The delta v required to catch up and then match velocities is horrendous, but humans manage to put together a second robot probe to intercept it. The probe intercepts the interstellar object and we discover that the object had been subject to both a strong electromagnetic pulse that fried any electronics, and a thermal shock (laser possibly) that also damaged the interstellar object. In examining the inside, the probe discoverers glass etched with pulses,[4] which after some creative engineering and improvising, the probe manages to read the...
Training a RL Model with Continuous State & Action Space in a Real-World Scenario
by Alexander Ries — 2023-10-15T22:59:51.898Z

Hello everyone,

I'm a Data Science student diving into an exciting thesis topic: using reinforcement learning to stabilize boats in rough seas by adjusting a keel's angle. But I am a bit concerned about the high complexity of the problem and the given situation:

Action Space: Continuous, representing the keel's angle adjustments.

State Space: Continuous, capturing the dynamic behavior of the sea, including waves.

Training Environment: Currently, the company only has a real-world water tank setup to simulate the sea conditions. There's no computer simulation available.

Given this setup, I have a couple of concerns:

Is it possible to train an RL model effectively in such a complex real-world scenario without first having a computer simulation? And if yes, what would be your initial steps in doing so?

Are there possibilities to reduce the problem's complexity while training exclusively in the real-world water tank simulation? (i.e. transforming the action space into a discrete action space?)

Any insights or advice would be greatly appreciated!