trentmkelly/LessWrong-43k huggingface dataset:
Some thoughts on George Hotz vs Eliezer Yudkowsky
by TristanTrim — 2023-08-15T23:33:18.377Z

Just watched this debate with George Hotz and Eliezer Yudkowsky. Here's a few of my thoughts.

I think GH's claim that timelines are important was right, but he didn't engage with EY's point that we can't predict the timeline, and instead GH pulled out an appeal to ignorance, claiming that because we don't know the timeline, therefore it is many decades off.

I found the discussion of FOOM was glossed over too quickly, I think EY was wrong about it not containing any disagreement. They instead both spent a bunch of time arguing about superhuman agent's strategies even though they both agreed that human level intelligence cannot predict superhuman intelligence. Or, rather, Eliezer claimed we can make statements about superhuman actions of the type "the superhuman chess robot will make moves that cause it to win the chess game." A framing which seemed to drop out during heated speculation of superhuman AI behaviors.

It seemed to me like a lot of GH's argument rested on the assumption that AI's will be like humans, and will therefore continue the economy and be human friendly, which seemed really unsubstantiated. It would be interesting to hear more about why he expects that, and if he's read any of the theory on why we shouldn't expect that. Though, I believe a lot of that theory was from before much of RL and mesa optimizer theory and so needs revisiting.

I wasn't impressed at all by the notion that AI won't be able to solve the prisoners dilemma,

First because I don't think we have superhuman knowledge of what things actually are unsolvable,
And also because I don't think negotiation skill is capped by the prisoners dilemma being impossible (AI could still negotiate better than humans even if they can't form crypto stable pacts),
And finally because even if they don't cooperate with each other that still doesn't say anything about them being human friendly.

One of GH's points that I don't think got enough attention was that the orthogonality thesis applies to Humans as well, although he seemed to be unaware of shard theory, which I believe is a result of mesa optimizer theory with some influence from neuroscience (probably not enough influence). But even so, that's not relevant to the discussion of AI x-risk, the context in which GH was using it.

Overall, GH did not say anything that changed my view of the AI x-risk situation, he seemed to say "because it's possi...
Understanding the Information Flow inside Large Language Models
by Felix Hofstätter — 2023-08-15T21:13:27.595Z

This is the write-up for our (@cozyfractal and mine) capstone project during ARENA's 2023 summer iteration. Our project explored a novel approach for interpreting language models, focusing on understanding their internal flow of information. While the practical implementation was completed in just one week and lacks formal rigor, we believe it offers some interesting insights and holds promise as a foundation for future research in this area. The accompanying repository with code examples and more experiments can be found here.

We want to thank Alexandre Variengien whose original idea served as the inspiration for this work, and who provided extensive feedback as well as thought-provoking discussions. Additionally, we want to express our thanks to the organizers of ARENA and our fellow participants for fostering an environment that encouraged productive collaboration and learning.

Introduction

Imagine a transformer tackling a lengthy prompt, like a code snippet. Within this prompt, there are subsets spanning multiple tokens, forming semantic units like function and variable definitions. But how does the transformer handle this distributed information while calculating subsequent token probabilities? Do specific attention heads copy information from all tokens in a function and aggregate it at the end? Or does aggregation occur earlier, as has been suggested by recent work on factual recall in large language models (LLMs). Perhaps this involves tokens near the end of a semantic unit, or seemingly unrelated tokens like punctuation signs acting as "intermediate variables". Another interesting question is to what degree the distribution of information over tokens is redundant, allowing the network to make correct predictions even after removing some individual tokens, like a bracket from a function definition.

These dynamics - involving intermediate variables, aggregation, and distributed information - are examples of what we term the "information flow" through an LLM. Our project's objective was to develop a tool that sheds light on this intricate process. By comprehending the inner workings of LLMs as they process information, especially during impressive feats like chain-of-thought reasoning, we hope to gain insights into long-range context processing. Understanding how information is moved over large token-distances is crucial for AI capabilities like long...
Any research in "probe-tuning" of LLMs?
by Roman Leventov — 2023-08-15T21:01:32.838Z

Is there any research in "probe-tuning" of LLMs, i.e., tuning LLM's parameter weights such that a specific probe (classifier) is more reliably detecting certain markers throughout the context, such as grammatical errors, aggression, manipulation, certain political bias, etc.?

This is different from classical fine-tuning and RLHF. As well as classical fine-tuning, probe-tuning is a supervised ML method: it is based on human-annotated texts (contexts). However, probe-tuning should be more effective than classical fine-tuning for detecting many occurrences of a certain marker throughout the context. Probe-tuning doesn't train on LLM's own "original rollouts" at all, only on LLM's activations during the context pass through the LLM.

I imagine than before doing actual probe-tuning, first we should determine which probe in the LLM is most aligned to the training data (annotations) already, so that probe-tuning likely just attenuates some vaguely existing concept within the LLM.
Can AI Transform the Electorate into a Citizen’s Assembly 
by RoscoHunter — 2023-08-15T17:52:55.458Z

Motivation: Modern democratic institutions are detached from those they wish to serve [1]. In small societies, democracy can easily be direct, with all the members of a community gathering to address important issues. As civilizations get larger, mass participation and deliberation become irreconcilable, not least because a parliament can't handle a million-strong crowd. As such, managing large societies demands a concentrated effort from a select group. This relieves ordinary citizens of the burdens and complexities of governance, enabling them to lead their daily lives unencumbered. Yet, this decline in public engagement invites concerns about the legitimacy of those in power.

Lately, this sense of institutional distrust has been exposed and enflamed by AI-algorithms optimised solely to capture and maintain our focus. Such algorithms often learn exploit the most reactive aspects of our psyche including moral outrage and identity threat [2]. In this sense, AI has fuelled political polarisation and the retreat of democratic norms, prompting Harari to assert that "Technology Favors Tyranny" [3]. However, AI may yet play a crucial role in mending and extending democratic society [4]. The very algorithms that fracture and misinform the public can be re-incentivised to guide and engage the electorate in digital citizen’s assemblies. To see this, we must first consider the way that a citizen’s assembly traditionally works.

What’s a Citizens Assembly: A citizen's assembly consists of a small group that engages in deliberation to offer expert-advised recommendations on specific issues. Following group discussion, the recommendations are condensed into an issue paper, which is presented to parliament. Parliamentary representatives consider the issue paper and leverage their expertise to ultimately decide the outcome. Giving people the chance to experiment with policy in a structured environment aids their understanding of the laws that govern them, improves government transparency, and promotes feelings of democratic self-efficacy [5]. Moreover, the compromise required for a randomly selected group of individuals to reach a consensus provides an intuitive antidote to political polarisation and calcification.

How AI Can Augment Assembly: Having explored the conventional workings of a citizen's assembly, we will now examine how AI can guide ...
Ten Thousand Years of Solitude
by agp — 2023-08-15T17:45:34.556Z
Link: https://www.discovermagazine.com/the-sciences/ten-thousand-years-of-solitude

This is a linkpost for the article "Ten Thousand Years of Solitude", written by Jared Diamond for Discover Magazine in 1993, four years before he published Guns, Germs and Steel. That book focused on Diamond's theory that the geography of Eurasia, particularly its large size and common climate, allowed civilizations there to dominate the rest of the world because it was easy to share plants, animals, technologies and ideas. This article, however, examines the opposite extreme.

Diamond looks at the intense isolation of the tribes on Tasmania - an island the size of Ireland. After waters rose, Tasmania was cut off from mainland Australia. As the people there did not have boats, they were completely isolated, and did not have any contact - or awareness - of the outside world for ten thousand years.

How might a civilization develop, all on its own, for such an incredible period of time?

If you ask any anthropologist to summarize in one phrase what was most distinctive about the Tasmanians, the answer will surely be the most primitive people still alive in recent centuries.

The "entire material corpus" of Tasmania only amounted to two dozen items in total - and did not include mounted stone stools, bone tools, or any clothing at all. Despite average low temperatures in winter of 41 degrees Fahrenheit, the Tasmanians were completely naked. In addition to the poor quality of tools in Tasmania, they also refused to eat fish, which were plentiful in the waters around the island. The material culture and wellbeing of the Tasmanians was significantly worse off than that of the Australians.

Australian products absent in Tasmania included the spear-thrower, a hand-held device to increase a spear’s throwing distance and propulsive force; ground or polished stone tools; mounted stone tools, such as hatchets or adzes with a handle; bone tools, such as needles and awls; fire-making equipment, such as a fire drill; and nets, traps, or hooks to catch fish, birds, or mammals. Without mounted stone tools, Tasmanians couldn’t fell a big tree, hollow out a canoe, or carve a wooden bowl. Without bone tools, they couldn’t sew warm clothes or watertight bark canoes.

The poverty of the Tasmanians was shocking to the first European explorers. They did not understand how the Tasmanians could have reached the island without boats, and they didn't understand why the Tasmanians had astonis...
AISN #19: US-China Competition on AI Chips, Measuring Language Agent Developments, Economic Analysis of Language Model Propaganda, and White House AI Cyber Challenge
by Dan H — 2023-08-15T16:10:17.594Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-19

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required

Subscribe here to receive future versions.

US-China Competition on AI Chips

Modern AI systems are trained on advanced computer chips which are designed and fabricated by only a handful of companies in the world. The US and China have been competing for access to these chips for years. Last October, the Biden administration partnered with international allies to severely limit China’s access to leading AI chips.

Recently, there have been several interesting developments on AI chips. China has made several efforts to preserve their chip access, including smuggling, buying chips that are just under the legal limit of performance, and investing in their domestic chip industry. Meanwhile, the United States has struggled to build American chip manufacturing capacity, and has taken further steps to prevent Americans from investing in Chinese technology.

Nvidia slips under the export controls. The US only banned the export to China of chips which exceed a designated performance threshold. Nvidia, the American chip designer, responded to this rule by designing a chip that operates just below the performance threshold.

In response, Chinese firms including Alibaba, Baidu, ByteDance, and Tencent ordered $5 billion worth of Nvidia’s new chip. Unsurprisingly, Nvidia’s competitor AMD is also considering building a similar chip to sell to Chinese firms.

Smuggling chips into China. There is a busy market for smuggled AI chips in China. Vendors report purchasing the chips secondhand from American purchasers, or directly from Nvidia through companies headquartered outside of China. These chips regularly sell at twice their manufacturer suggested retail price, and are typically only available in small quantities. One option for preventing chips from making it to the black market would be for the United States to track the legal owner of each chip and occasionally verify that chips are still in the hands of their legal owners.

“The world’s most complex semiconductor tool, an ASML Twinscan NXE. It costs ~$200M, is the size of a bus, and contains hundreds of thousands of components. Just shipping it requires 40 freight containers, 20 trucks, and 3 cargo planes. As of late 2022, only ~150 have ever been manufactured. ASML is backlogged for yea

...
My checklist for publishing a blog post
by Steven Byrnes — 2023-08-15T15:04:56.219Z

Introduction

Checklists are good. I don’t use checklists much for my job though. (My to-do list is stylistically a kanban, not a checklist—details here & here.)

But I have one exception: My checklist for publishing blog posts (an activity that I’ve been doing with some regularity—you’re reading my 114th blog post just on this forum!)

I am sharing that checklist here, not because it’s particularly good, nor because I’m recommending that other people use it (obviously it’s tailored to my idiosyncratic needs), but because I’m interested in sharing ideas and getting feedback!

Related things on this forum include a 2012 essay-publishing checklist by gwern (edit: a commenter found a better / updated link to Gwern’s checklist here), and Justis’s writing advice list which is not directly a checklist but could be made into one (and indeed I copied a few items from it). Please comment with other references and suggestions! How would your own checklist differ from mine?

A couple more bits of commentary before we begin:

Checklist workflow:

Good news is that pretty much every productivity-related app (e.g. logseq, roam, obsidian, emacs-org-mode, trello, etc.) has a very nice workflow for checklists—where you make a reusable checklist template, and then insert a fresh (non-checked-off) copy into the appropriate context, and then check off the items one-by-one. If you don’t know the details, google it.

“Consider doing X” items:

You’ll notice that many of these checklist items are of the form “Consider doing X”. Often what that means for me in practice is:

I get to the checklist item “Consider doing X”;
I consider doing X, and decide not to;
I happily check it off.

That’s fine! It’s not always a good use of time to make a blog post higher-quality. The one you’re reading right now is a great example: I am writing this post very quickly, and I stand by that decision.

OK, that’s enough commentary! The rest of the post is the checklist itself.

The actual checklist! (2023-08-15 2024-01-30 version)

Copyediting items

Check for unexplained or unnecessary jargon & acronyms.
Check for jargon & acronyms that are defined in one part of the post and then used in a distant part of the post without repeating the definition.
Check for unnecessarily obscure words and cultural references (for non-native English speakers)
Check for vague “this”
Check for over-hedging
Consider checking that all the hy

...
The Dunbar Playbook: A CRM system for your friends
by Severin T. Seehrich — 2023-08-15T08:44:29.435Z
Link: https://amoretlicentia.substack.com/p/the-dunbar-playbook

Thanks to Dakota Quackenbush from

Authentic Bay Area

for an earlier version of this tool.

So far, the main motivation for my work as a community builder and authentic relating facilitator was meeting my own need for connection.

I think that was a mistake.

First, it is difficult to harvest the fruits of community while I’m the one responsible for creating and holding the space. Second, this motivation leads to botched incentives that end up serving neither the cause nor me. After all, the subset of broke EAs and hippies I enjoy spending my time with the most are not in too dire need of my services, nor particularly capable of helping me pay my rent.

In other words, I’ve finally given up on trying to poop where I eat. Instead of building a product for my in-group, I now try to anchor my life in my tribe, and use the energy I get there to build products that serve the outside world and pay my rent.

Wish me luck.

Because my life happens all over the globe and making new friends is more intuitive for me than sustaining long-term relationships, I want to be a bit strategic about building a tribe that keeps me energized.

That’s where the Dunbar Playbook comes into play.

Some theory: Dunbar’s Number

The Dunbar Playbook is named after Dunbar’s Number, the number of people one can maintain personal relationships with.

In his earlier research, anthropologist and evolutionary psychologist Robin Dunbar concluded that that’s about 150 people. Later, he found that there are actually

different circles of friendship

. Apparently, people have a handful of very close friends, a couple more best friends, and vastly more loose friends and acquaintances. (Who would have thought.)

The Atlantic cites the following layers, with each layer being ~3x the size of the preceding ones:

1.5 people: Intimates
5: Close friends
15: Best friends
50: Good friends
150: Friends
500: Acquaintances
1500: Known names
5000: Known faces

Of course, these are all just rough approximations. Introverts will invest more energy into fewer people, extraverts less into more. There are probably also cultural differences or something.

Introducing the Dunbar Playbook

However big or small these circles are for you: It will probably not hurt to be a explicit about who is part of which one. Fine-grained categories make it easier to track where your priorities lie.

The process for creating a Dunbar Playbook is simple: Make a list of people...
Optical Illusions are Out of Distribution Errors
by vitaliya — 2023-08-15T02:23:06.108Z

Our visual black box is trained on what we see in the real world. We don't process raw sensory data - we keep a more abstracted model in our heads. This is a blessing and a curse - it may either be more useful to have a sparser cognitive inventory, and work on a higher level, or to ignore preconceived notions and try to pick up details about What Actually Is. Marcus Hutter would have us believe this compression is itself identical to intelligence - certainly true for the representation once it hits your neurons. But as a preprocessing step? It's hard to be confident. Is that summarisation by our visual cortex itself a form of intelligence?

Ultimately, we reduce a three-dimensional world into a two-dimensional grid of retinal activations; this in turn gets reduced to an additional layer of summarisation, the idea of the "object". We start to make predictions about the movement of "objects", and these are usually pretty good, to the point where our seeming persistence of vision is actually a convenient illusion to disguise that our vision system is interrupt-based. And so when we look at a scene, rather than even being represented as a computer does an image, as any kind of array of pixels, there is in actuality a further degree of summarisation occurring; we see objects in relation to each other in a highly abstract way. Any impressions you might have about an object having a certain location, or appearance, only properly arise as a consequence of intentional focus, and your visual cortex decides it's worth decompressing further.

This is in part why blind spots (literal, not figurative) can go unidentified for so long: persistence of vision can cover them up. You're not really seeing the neuronal firings, it's a level lower than "sight". It probably isn't possible to do so consciously - we only have access to the output of the neural network, and its high-level features; these more abstracted receptive fields are all we ever really see.

This is what makes optical illusions work. Your visual cortex is pretrained to convert an interrupt stream from your eyeballs into summarised representations. And in doing so it makes certain assumptions - ones which, once correctly characterised, can be subverted with a particular adversarial stimulus. Each optical illusion shows its own failure of the visual cortex to properly represent "true reality" to us. The underlying thin...
A short calculation about a Twitter poll
by Ege Erdil — 2023-08-14T19:48:53.018Z

Recently, the following poll was posted on Twitter:

Everyone responding to this poll chooses between a blue pill or red pill.

if > 50% of ppl choose blue pill, everyone lives

if not, red pills live and blue pills die

Which do you choose?

If we linearize the problem of how much you value the lives of other poll respondents compared to your own, there's actually a way to quantify whether you should vote red or blue depending on your beliefs about how the rest of the population will vote.

Suppose that we normalize the value of choosing the red pill at zero, and let N be the total number of people responding to the poll excluding you. Then, choosing the blue pill only makes a difference in the following two cases:

If the number of people who vote blue excluding you, hereafter denoted B, is exactly ⌈N/2⌉, then your vote saves the lives of ⌈N/2⌉ people.

If B<⌈N/2⌉, then you die and get nothing in return.

If you value the lives of other poll respondents at a constant multiple ν of your own life, then you should pick blue if

P(B=⌈N/2⌉)⋅(ν⌈N/2⌉)>P(B<⌈N/2⌉)

and you should pick red if the inequality goes in the other direction.

A rough heuristic here is that if the distribution of B is unimodal with a maximum to the right of ⌈N/2⌉, you expect

⌈N/2⌉P(B=⌈N/2⌉)≥P(B<⌈N/2⌉)

so if ν=1 (meaning you value the lives of other poll respondents equally to your own) and you expect a blue majority in the precise sense defined above, you should always take the blue pill yourself. The exact cutoff on ν for you to take the blue pill is

ν≥P(B<⌈N/2⌉)P(B=⌈N/2⌉)⌈N/2⌉

A general heuristic is that if we think a red victory is a distinct possibility, so that P(B<⌈N/2⌉)=O(1), and the distribution doesn't decay too sharply around N/2, in general, we'll have P(B=⌈N/2⌉)=O(1/N) so that the cutoff ends up being ν≥O(1). In other words, you pick blue in realistic circumstances if you value other people's lives similarly to your own: ν can't be too far away from 1 if picking blue is to make sense as an individually altruistic decision, ignoring social pressures to pick blue et cetera.

Approximating B/N by a beta distribution for large N gives the following rough results:

Probability of a red victory
Expected share of blue votes
Cutoff value of 1/ν, i.e. minimum value of your life in units of others' lives to choose red

10%
55%
22.4

10%
65%
7.1

10%
75%
3.7

20%

...