~www_lesswrong_com | Bookmarks (706)

Published on March 15, 2025 3:24 PM GMTBy A. NobodyIntroductionThe emergence of Artificial General Intelligence (AGI) presents not just the well-theorized dangers of human extinction but also an often-overlooked inevitability: AGI-on-AGI warfare. This essay explores the hypothesis that the first signs of superintelligent AGI engaging in conflict will not be visible battles or disruptions but the sudden and unexplained failure of highly advanced AI...

Published on March 15, 2025 6:08 AM GMTBy A. NobodyIntroductionThroughout history, the ultra-wealthy have insulated themselves from catastrophe. Whether it’s natural disasters, economic collapse, or even nuclear war, billionaires believe that their resources—private bunkers, fortified islands, and elite security forces—will allow them to survive when the rest of the world falls apart. In most cases, they are right. However, an artificial general intelligence (AGI)...

Published on March 15, 2025 12:30 PM GMTAbstractThe emerging field of “AI safety” has attracted public attention and large infusions of capital to support its implied promise: the ability to deploy advanced artificial intelligence (AI) while reducing its gravest risks. Ideas from effective altruism, longtermism, and the study of existential risk are foundational to this new field. In this paper, we contend that overlapping...

Published on March 14, 2025 9:18 PM GMTAI4Science has the potential to surpass current frontier models (text, video/image, and sound) by several magnitudes. While some may arrive at similar conclusions through empirical evidence, we derive this insight from our "Deep Manifold" and provide a theoretical foundation to support it. The reasoning is straightforward: for the first time in history, an AI model can integrate geometric...

Published on March 14, 2025 11:07 PM GMTI’ve become so reliant on a GPS that using maps to direct myself feels like a foreign concept. Google Maps, Waze, whatever, if it's outside of my neighbourhood, I’m punching in the address before I head out. Sometimes I notice the GPS taking slower routes or sending me the wrong way as I get out of a...

Published on March 14, 2025 11:20 PM GMTIn September last year I posted an ad for a fellowship. This current post is the retrospective on how that went. Or, more accurately, it's the in-progress report of how it's going. The de-facto length of the fellowship was 3 months, but it went well enough that I'm extending it by another 4 months.There's not a particular...

Published on March 14, 2025 10:51 PM GMTAmerican democracy currently operates far below its theoretical ideal. An ideal democracy precisely captures and represents the nuanced collective desires of its constituents, synthesizing diverse individual preferences into coherent, actionable policy.Today's system offers no direct path for citizens to express individual priorities. Instead, voters select candidates whose platforms only approximately match their views, guess at which governmental...

Published on March 14, 2025 10:29 PM GMTThanks to everyone who took the Unofficial 2024 LessWrong Survey. For the results, check out the data below. The Data0. PopulationThere were two hundred and seventy nine respondents over thirty three days. Previous surveys have been run over the last decade and a half.2009: 1662011: 10902012: 11952013: 16362014: 15032016: 30832017: "About 300"2020: 612022: 1862023: 5582024: 279That’s an annoying...

Published on March 14, 2025 6:38 PM GMTRapid AI progress is the greatest driver of existential risk in the world today. But — if handled correctly — it could also empower humanity to face these challenges.Executive summary1. Some AI applications will be powerful tools for navigating existential risksThree clusters of applications are especially promising:Epistemic applications to help us anticipate and plan for emerging challengese.g. high-quality AI...

Published on March 14, 2025 3:00 PM GMT(Audio version here (read by the author), or search for "Joe Carlsmith Audio" on your podcast app. This is the fourth essay in a series that I’m calling “How do we solve the alignment problem?”. I’m hoping that the individual essays can be read fairly well on their own, but see this introduction for a summary of the essays...

Published on March 14, 2025 12:30 PM GMTDan Hendrycks, Eric Schmidt and Alexandr Wang released an extensive paper titled Superintelligence Strategy. There is also an op-ed in Time that summarizes. The major AI labs expect superintelligence to arrive soon. They might be wrong about that, but at minimum we need to take the possibility seriously. At a minimum, the possibility of imminent superintelligence will...

Published on March 14, 2025 9:48 AM GMTEpistemic status: somewhat rushed out in advance of the deadline tomorrow (Saturday the 15th) for the call for public comment on US AI policy. I think the issue is complex and deserves careful consideration.SummaryMethods and motivations for governments to control AGI and limit its proliferation seem to be underexploredCurrent laws appear adequate to take control of AGI...

Published on March 14, 2025 8:27 AM GMTA short science fiction story illustrating that if we fail to solve alignment, humanity risks losing not only 8 billion lives. He opened his eyes. The room was plain white."You remember enough?" asked the familiar voice."Enough," he replied. He stood. Everything balanced.He walked carefully down a bright hallway. At the end, a window showed leaves falling from a...

Published on March 14, 2025 4:52 AM GMTThis is a cross-post  - as some plots are meant to be viewed larger than LW will render them (and on a dark background) it is recommended this post be read via the original site.Thanks to Zach Furman for discussion and ideas and to Daniel Murfet, Dmitry Vaintrob, and Jesse Hoogland for feedback on a draft of...

Published on March 14, 2025 2:10 AM GMT While in a more remote area bike lights let you see where you're going, in a city there's usually enough light around for that. But you still need lights, so other people see you. [1] Unfortunately, lots of people end up biking without lights: it's easy to forget them, have them break, or end up out...

Published on March 13, 2025 8:36 PM GMTWhen communicating about existential risks from AI misalignment, is it more important to focus on policymakers/experts/other influential decisionmakers or to try to get the public at large to care about this issue?[1] I lean towards it being overall more important to communicate to policymakers/experts rather than the public. However, it may be valuable for certain individuals/groups to focus...

Published on March 13, 2025 7:18 PM GMTWe study alignment audits—systematic investigations into whether an AI is pursuing hidden objectives—by training a model with a hidden misaligned objective and asking teams of blinded researchers to investigate it.This paper was a collaboration between the Anthropic Alignment Science and Interpretability teams.AbstractWe study the feasibility of conducting alignment audits: investigations into whether models have undesired objectives. As...

Published on March 13, 2025 6:29 PM GMTWe (SaferAI) propose a risk management framework which we think should improve substantially upon existing Frontier Safety Frameworks if followed. It introduces and borrows a range of practice and concepts from other areas of risk management to introduce conceptual clarity and generalize some early intuitions that the field of AI safety independently came up with.To maintain readability...

Published on March 13, 2025 6:17 PM GMTWhy do adults pursue long-term and complex goals? People don’t come into the world wanting to amass followers on Instagram, get a Ph.D., or become rich, yet these and a myriad of other goals are what seem to motivate most people. I have been interested in how these complex and long-term goals develop, and I think it...

Published on March 13, 2025 6:16 PM GMTThis post is a distillation of a recent work in AI-assisted human coordination from Google DeepMind. The paper has received some press attention, and anecdotally, it has become the de-facto example that people bring up of AI used to improve group discussions. Since this work represents a particular perspective/bet on how advanced AI could help improve human coordination, the following explainer...

Published on March 13, 2025 10:24 AM GMTI scrutinise the so-called "reversal curse", wherein LLMs seem not consier inverse relationships between conceptual nodes.I show that, far from being a proof of a lack of logical skills, it is a normal artefact of saliency, known in humans as associative recall asymmetry, and propose a conceptual-network model of the causes which works independently of substrate.Discuss

Published on March 13, 2025 2:40 PM GMTThe most hyped event of the week, by far, was the Manus Marketing Madness. Manus wasn’t entirely hype, but there was very little there there in that Claude wrapper. Whereas here in America, OpenAI dropped an entire suite of tools for making AI agents, and previewed a new internal model making advances in creative writing. Also they...

Published on March 13, 2025 12:58 PM GMTIf there is an international project to build artificial general intelligence (“AGI”), how should it be designed? Existing scholarship has looked to historical models for inspiration, often suggesting the Manhattan Project or CERN as the closest analogues. But AGI is a fundamentally general-purpose technology, and is likely to be used primarily for commercial purposes rather than military...