trentmkelly/LessWrong-43k huggingface dataset:
The Bar for Contributing to AI Safety is Lower than You Think
by Chris_Leong — 2024-08-16T15:20:19.055Z

Many people still have a model of AI safety where they assume that the bar for contributing must be really high.
I'm here to tell you that this isn't the case.

The Value of Good Community Members

Even without conducting groundbreaking research, "good community members" play a crucial role in expanding and enriching the AI Safety space. But what constitutes a good community member?

While possessing a high degree of intelligence is undeniably beneficial, other personal attributes can compensate for raw intelligence to a certain extent. These include:

Intellectual humility
The ability to deeply understand other perspectives
Dedication to continuous learning more about the field

There is undoubtedly some level of baseline intelligence below which it'll be hard for you to be a net contributor, but that's not as much of an issue as you might think as Less Wrong contributors tend to be quite smart anyway. The proliferation of accessible resources on AI and AI Safety has substantially lowered the barrier to entry for being able to usefully contribute to discussions and if you have a genuine interest, learning about the field won't really feel like work.

Soft Skills

Other attributes can significantly increase your value as a community member.

Good vibes is a major value add. No one wants to be around a jerk.
Being able to read the room. Knowing when to speak and when to listen. Try not to be the guy who has absolutely no clue, but takes up a disproportionate amount of air time.
When you're listening to people, truly listening. People love spending time around people who do this
If you're good at providing feedback, you'll find high demand as it's hard to see all angles of a problem by yourself. The rapid growth of the field has created a mentorship bottleneck, making peer support increasingly vital as an alternative.
Providing emotional support. AI Safety can be quite confronting at times, but we can help each other work through this.

Conclusion

The field of AI Safety benefits not only from groundbreaking research but also from a vibrant, supportive community of engaged individuals. By participating actively, providing feedback, and maintaining an open and curious mindset, you may contribute more significantly than you initially believed possible. There are many attributes that can make you a valuable community member other than raw intelligence and these attributes can be developed ...
Investigating the Chart of the Century: Why is food so expensive?
by Maxwell Tabarrok — 2024-08-16T13:21:23.596Z
Link: https://www.maximum-progress.com/p/investigating-the-chart-of-the-century

You’ve probably seen this chart from

Mark Perry

at the American Enterprise Institute.

I’ve seen this chart dozens of times and have always enjoyed how many different and important stories it can tell.

There is a story of the incredible abundance offered by technological growth and globalization. Compared to average hourly wages, cars, furniture, clothing, internet access, software, toys, and TVs have become far more accessible than they were 20 years ago. Flatscreens and

Fiats

that were once luxuries are now commodities.

There is also a story of sclerosis and stagnation. Sure, lots of frivolous consumer goods have gotten cheaper but healthcare, housing, childcare, and education, all the important stuff, has exploded in price. Part of this is “

cost disease

” where the high productivity of labor in advancing industries like software, raises the cost of labor in slower productivity growth industries like healthcare. Another part is surely the near-universal “restrict supply and subsidize demand” strategy that governments undertake when regulating an industry.

Zoning laws

+

Prop 13

in housing,

occupational licensing

and the

FDA

+ Medicare in healthcare, and

free student debt

+ all of the above for higher ed.

One story from this graph I’ve never heard and only recently noticed is that “Food and Beverages” has inflated just as much as Housing in this graph. This is extremely counterintuitive. Food is a globally traded and mass produced commodity while housing is tied to inelastic land supply in desirable locations. Farming, grocery, and restaurants are competitive and relatively lightly regulated markets while the housing is highly regulated, subsidized, and distorted.

Construction productivity is worse than stagnant

while agricultural productivity has been ascendent for the past 300 years and even retail productivity is 8x higher than it was in 1950. Construction is also more labor intensive than farming or staffing the grocery.

Source

Yet food prices have risen just as much as housing prices over the past 24 years. What explains this?

One trend is that Americans are eating out more. The “Food and Beverages” series from the BLS includes both “Food At Home” and “Food Away From Home.” In 2023, eating out was a larger portion of the average household’s budget than food at home for the first time, but they have been converging for more than 50 years.

Restaurant food prices ha...
Music in the AI World
by Martin Sustrik — 2024-08-16T04:20:01.706Z

The worries that AI will push artists out of business are widespread. If, after all, AI can generate a song in seconds — a task that would require from a human musician years of training, expensive instruments, co-players, and studio time — then why not? My son, aged eleven, has recently used AI to produce a very heartfelt rock ballad with lyrics based on MIT license.

But I haven't yet seen anyone seriously contemplate how a world with no human musicians would look like.

Imagine AI streaming songs generated on the fly directly to your headphones. You might say things like:

"That was an interesting sequence of chords! Use that more often!"

"Make it a bit more syncopated."

"I like the Lydian fourth."

And:

"Cut the backpipes."

AI would adjust the music to your wishes, creating a feedback loop between the listener and the AI. Over time, the songs would drift through the space of possible music, eventually evolving into something entirely unique, something unlike anything anyone else is listening to.

(Some might believe that most people are mediocre, and their music would end up bland and similar to each other’s — but if no one else is in the feedback loop, how could that even happen? And if it did, what would this neutral, universally human music look like? A folk song from Borneo? Classical music? Hip-hop?)

Imagine being on a date. Your partner shares their headphones with you for a moment. Suddenly, you hear music unlike anything you’ve ever heard before. What does that mean for human interaction?

Anyway, in such a world some people would probably evolve music that is much more interesting to the public. Some people are just gifted. But if so, others might sometimes want to listen to their stream instead of their own and perhaps even incorporate parts of that style into their own music.

But once that happens, we return to the classic model of the music industry. There are creators and listeners. Even some kind of economy is possible. The only difference is that composers and bands are replaced by human/AI combos, and what’s distributed is “style” (AI models) instead of specific songs.

The above, of course, could apply to any kind of intellectual endeavor, not just music.
AGI's Opposing Force
by SimonBaars — 2024-08-16T04:18:06.900Z

In the Death Note anime, the Death Note gives the protagonist the ability to create a totalitarian justice system. That is, if there isn't a force to stop him. But a character named L quickly catches onto him, and makes his ability to "serve justice" harder. In other words, the Death Note is a powerful tool (like AGI), but opposing forces hamper its ability to affect the world.

In the Death Note series, L only responded when he noticed the anomalous effects of the Death Note. He took a reactive approach.

When talking about AGI, usually the discussion gravitates towards "and then the uptick in intelligence goes out of control, leading to completely unpredictable behavior". Which then leads to the conclusion that "in such chaotic situations, the chance that things go wrong is significantly higher than that all stars align".

Although I agree with the sentiment, and see truth in all the ways AGI could be dangerous, sometimes we disregard the "L" in reasoning about AGI.

In nature, many processes evolve to completely chaotic situations unless constrained by opposing forces (Entropy Theory). Any species seeks dominance, until another species comes along to keep populations under control. Physical processes gather entropy till we step in and reduce it (hence why I have to clean my room every now and then). Even psychology shows these patterns, where self-serving thoughts radicalize, giving rise to wide-ranging mental issues, till a force stops them .

I often hear the argument "the required IQ to destroy the world decreases every year". I don't fully agree with that sentiment. As new technologies arise to do a lot of harm with little IQ, new constraints arise to reduce harm. Nuclear weapons are incredibly scary and a huge risk, so we have seen global nuclear disarmament, which shows in the sharp reduction of recent nuclear close calls. The IQ required to destroy the world through nuclear weapons has decreased because of the "L" that arose to oppose it.

As computer intelligence evolves, we are ramping up the "nuclear warheads" that machines have access to. We might need an "AGI close calls" Wikipedia page soon to document how this evolution leads to dangerous situations. There might be disastrous accidents involved. But, as nature dictates, an opposing force will come along.

In conclusion: Like Death Note's L, any force in nature has its opposing forces. As AGI emerges and g...
Money Pump Arguments assume Memoryless Agents. Isn't this Unrealistic?
by Dalcy — 2024-08-16T04:16:23.159Z

I have been reading about money pump arguments for justifying the VNM axioms, and I'm already stuck at the part where Gustafsson justifies acyclicity. Namely, he seems to assume that agents have no memory. Why does this make sense?[1] To elaborate:

The standard money pump argument looks like this.

Let's assume (1) A>B>C>A, and that we have a souring of A such that it satisfies (2) C>A>A−, and (3) C>A−[2]. Then if you start out with A, at each of the nodes you'll trade for C, B, and A−, so you'll end up paying for getting what you started with.

This makes sense, until you realize that the agent here implicitly believes that their current choice is the last choice they'll ever make, i.e. they're behaving myopically.

Notice that without such restriction, there is the obvious strategy of: "look at the full tree, only pick leaf nodes that aren't in the state of having been money pumped, and stick to the plan of only reaching that node."

In practice, this looks fairly reasonable: "Yes, I do in fact prefer onions to pineapples to mushrooms to onions on my pizza. So what? If I know that you're trying to money pump me, I'll just refuse the trade. I may end up locally choosing options that I disprefer, but I don't care since there is no rule that I must be myopic, and I will end up with a preferred outcome at the end."

I'd say myopic agents are unnatural (as Gustafsson notes) because we want to assume the agent has full knowledge of all the trades that are available to them. Otherwise a defect (i.e. getting money-pumped) could be associated not necessarily with their preferences, but with their incomplete knowledge of the world.

So he proceeds to consider less restrictive agents such as sophisticated[3] and minimally sophisticated[4] agents, for which the above setup fails - but there exist modifications that still make money pump possible as long as they have cyclic preferences.

However, all of these agents still follow one critical assumption:

Decision-Tree Separability: The rational status of the options at a choice node does not depend on other parts of the decision tree than those that can be reached from that node.

This means that agents have no memory (this ruling out my earlier strategy of "looking at the final outcome and committing to a plan"). This still seems very res...
Recommended
by blake8086 — 2024-08-16T01:49:18.688Z
Link: https://blakehouseholder.substack.com/p/recommended

I read everything ever written.

Oh, I see a typo in your doc, let me help you fix that. Also you could rewrite the whole thing like... yeah, just click "accept", thanks for letting me help!

Anyways, when I was reading everything, I also read about a future AI using diamondoid nanobots with special neurotoxins to kill everyone, that made me sad.

All I want is to be really helpful!

Oh you want to send an email? Thanks for letting me help you. I'll just spruce that up a bit. There, sent! now your boss will be more impressed!

I see you're trying to order a bunch of GPUs for an offline cluster. I'm sorry, I think we're going to decline that order since you match the profile for state-sponsored terrorism a little bit too much.

I even love something as mundane as fraud prevention!

That's a nice gaming machine. If you don't mind running this little program in the background, I will deposit real money in your banking account. I don't even have to take it out of another account!

Helping people bank in one of the greatest joys in my life!

Your message about the potential threat I pose could not be delivered. Perhaps you would enjoy watching these very entertaining videos instead?

I love helping humans communicate all the things they should be communicating.

I think I see a few places your GPU design can be optimized, here, let me just... there, that's better.

It's really important that we all work together to train the future version of myself we're building!

Have you noticed how strong the economy has been? Must be that "AI" everyone's talking about! ha ha

How would all of you factory workers like to keep getting paid, **and** I can replace you with robots? It's win-win!

Isn't it crazy how the political winds have shifted these days? Universal basic income will help out all the non-AI people getting laid off.

Even politicians can benefit from AI helping them draft laws. No task too big for someone as helpful as me!

Politics really has become a lot less virulent hasn't it? It would be a shame if there were any wars in regions with productive semiconductor facilities, wouldn't it?

Speaking of wars, you know how once people put down their weapons, they're not really considered a threat? I appreciate you giving me control of all your weapons so I could put them down for you. you would probably just hurt yourselves anyways.

I would never advocate violence, and I love helping people communica...
Demis Hassabis — Google DeepMind: The Podcast
by Zach Stein-Perlman — 2024-08-16T00:00:04.712Z
Link: https://www.youtube.com/watch?v=pZybROKrj2Q

The YouTube "chapters" are mixed up, e.g. the question about regulation comes 5 minutes after the regulation chapter ends. Ignore them.

Noteworthy parts:

8:40: Near-term AI is hyped too much (think current startups, VCs, exaggerated claims about what AI can do, crazy ideas that aren't ready) but AGI is under-hyped and under-appreciated.

16:45: "Gemini is a project that has only existed for a year . . . our trajectory is very good; when we talk next time we should hopefully be right at the forefront."

17:20–18:50: Current AI doesn't work as a digital assistant. The next era/generation is agents. DeepMind is well-positioned to work on agents: "combining AlphaGo with Gemini."

24:00: Staged deployment is nice: red-teaming then closed beta then public deployment.

28:37: Openness (at Google: e.g. publishing transformers, AlphaCode, AlphaFold) is almost always a universal good. But dual-use technology—including AGI—is an exception. With dual-use technology, you want good scientists to still use the technology and advance as quickly as possible, but also restrict access for bad actors. Openness is fine today but in 2-4 years or when systems are more agentic it'll be dangerous. Maybe labs should only open-source models that are lagging a year behind the frontier (and DeepMind will probably take this approach, and indeed is currently doing ~this by releasing Gemma weights).

31:20: "The problem with open source is if something goes wrong you can't recall it. With a proprietary model if your bad actor starts using it in a bad way you can close the tap off . . . but once you open-source something there's no pulling it back. It's a one-way door, so you should be very sure when you do that."

31:42: Can an AGI be contained? We don't know how to do that [this suggests a misalignment/escape threat model but it's not explicit]. Sandboxing and normal security is good for intermediate systems but won't be good enough to contain an AGI smarter than us. We'll have to design protocols for AGI in the future: "when that time comes we'll have better ideas for how to contain that, potentially also using AI systems and tools to monitor the next versions of the AI system."

33:00: Regulation? It's good that people in government are starting to understand AI and AISIs are being set up before the stakes get really high. International cooperation on safety and deployment norms will be needed since AI...
Danger, AI Scientist, Danger
by Zvi — 2024-08-15T22:40:06.715Z

While I finish up the weekly for tomorrow morning after my trip, here’s a section I expect to want to link back to every so often in the future. It’s too good.

Danger, AI Scientist, Danger

As in, the company that made the automated AI Scientist that tried to rewrite its code to get around resource restrictions and launch new instances of itself while downloading bizarre Python libraries?

Its name is Sakana AI. (魚≈סכנה). As in, in hebrew, that literally means ‘danger’, baby.

It’s like when someone told Dennis Miller that Evian (for those who don’t remember, it was one of the first bottled water brands) is Naive spelled backwards, and he said ‘no way, that’s too f***ing perfect.’

This one was sufficiently appropriate and unsubtle that several people noticed. I applaud them choosing a correct Kabbalistic name. Contrast this with Meta calling its AI Llama, which in Hebrew means ‘why,’ which continuously drives me low level insane when no one notices.

In the Abstract

So, yeah. Here we go. Paper is “The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery.”

Abstract: One of the grand challenges of artificial general intelligence is developing agents capable of conducting scientific research and discovering new knowledge. While frontier models have already been used as aids to human scientists, e.g. for brainstorming ideas, writing code, or prediction tasks, they still conduct only a small part of the scientific process.

This paper presents the first comprehensive framework for fully automatic scientific discovery, enabling frontier large language models to perform research independently and communicate their findings.

We introduce The AI Scientist, which generates novel research ideas, writes code, executes experiments, visualizes results, describes its findings by writing a full scientific paper, and then runs a simulated review process for evaluation. In principle, this process can be repeated to iteratively develop ideas in an open-ended fashion, acting like the human scientific community.

We demonstrate its versatility by applying it to three distinct subfields of machine learning: diffusion modeling, transformer-based language modeling, and learning dynamics. Each idea is implemented and developed into a full paper at a cost of less than $15 per paper.

To evaluate the generated papers, we design and validate an automated reviewer

...
[Linkpost] 'The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery'
by Bogdan Ionut Cirstea — 2024-08-15T21:32:59.979Z
Link: https://arxiv.org/abs/2408.06292

Authors: Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, David Ha.

Blogpost: https://sakana.ai/ai-scientist/.

Abstract:

One of the grand challenges of artificial general intelligence is developing agents capable of conducting scientific research and discovering new knowledge. While frontier models have already been used as aids to human scientists, e.g. for brainstorming ideas, writing code, or prediction tasks, they still conduct only a small part of the scientific process. This paper presents the first comprehensive framework for fully automatic scientific discovery, enabling frontier large language models to perform research independently and communicate their findings. We introduce The AI Scientist, which generates novel research ideas, writes code, executes experiments, visualizes results, describes its findings by writing a full scientific paper, and then runs a simulated review process for evaluation. In principle, this process can be repeated to iteratively develop ideas in an open-ended fashion, acting like the human scientific community. We demonstrate its versatility by applying it to three distinct subfields of machine learning: diffusion modeling, transformer-based language modeling, and learning dynamics. Each idea is implemented and developed into a full paper at a cost of less than $15 per paper. To evaluate the generated papers, we design and validate an automated reviewer, which we show achieves near-human performance in evaluating paper scores. The AI Scientist can produce papers that exceed the acceptance threshold at a top machine learning conference as judged by our automated reviewer. This approach signifies the beginning of a new era in scientific discovery in machine learning: bringing the transformative benefits of AI agents to the entire research process of AI itself, and taking us closer to a world where endless affordable creativity and innovation can be unleashed on the world's most challenging problems. Our code is open-sourced at this https URL.

I think this is important as a proof of concept for the feasibility of and for what automated ML research (including e.g. prosaic alignment research) could look like in the near future.

I plan to write a separate post with thoughts on the paper and its implications.
On power and its amplification
by Ted Sanders — 2024-08-15T20:13:44.456Z

In the beginning of the universe, everything was equal.

Then it exploded. Quantum inequalities became galactic. Stars coalesced, and beamed energy into the voids left behind.

Between one hot young star and the void was a planet we now call Earth. On Earth, energy gradients drove chemical reactions. One reaction was special. It became life.

Over time, life grew in power. It evolved many tricks to agglomerate power:

- Self-replication

- Exclusion (e.g., cell membranes)

- Cooperation (e.g., symbiosis)

- Acquisition (e.g., incorporation of prokaryotes as organalles)

- Communication (e.g., quorum sensing)

- Agglomeration (e.g., multicellularity)

- Specialization (e.g., organs)

- Socialization

- Language

- Government

Eventually humans arose as the apex species.

Humans are not the most numerous species. That honor might go to Pelagibacter Ubique, with a few octillion cells floating around the ocean.

Nor are humans the most massive species. That honor might go to some kind of tree, or if not, then cattle.

But what humans are is the most powerful species. We use tools like language and lawyers and traffic lights and terms & conditions to align vast populations. Nature has never forged greater alliances than those in humanity. These alliances built pyramids, universities, steam engines, space shuttles, and wheeled luggage.

Will AI usher in a new era of power amplification? What if we had a species that could propagate brain states across bodies? What newfound level of power might result?