Simon Safar

Making AI Part of Us

2026-08-01T17:00:00.000000-07:00

Making AI Part of Us

2026/08/02

Much has been talked about how to get the AIs to do what we actually want. Some of this is in the context of "if we're going to build superintelligent machines, how do we ensure they don't turn us into paperclips"... but there is another, somewhat more prosaic meaning to this as well.

Namely... even with the current level of AIs... if your prompt is three sentences and their output is a ten thousand line software artifact, there is a very real sense in which "I wrote this" is less true vs. if I sat down and hammered it out by hand.

This situation is, actually, not that different from cases that existed before. As in... if I write a file in C++ and then I go on pointing out the beatifully optimized assembly output of GCC as my own, "look at my decision to unroll this loop was the right call here", ... I might end up not agreeing with your worldview thoroughly. There is also the unsatisfying feeling when the compiler takes about as long to run on your much-valued code as just the headers and a single line of "hello world"; at least if it was me making the compiler work hard, I must have done something important in the process?

The feel gets somewhat stronger though, the more AI we pump into this. If you have a song you wrote the lyrics for at least, even though the rest is AI output? That song is more "you" than if you tell it something like "write me a catchy punk rock song about whatever comes to your mind first".

You can, however, slowly, exert influence though still. As you write more examples of code the agents should follow, your preferences are more likely to apply. In the same way as company culture and leadership direction does count when looking at the resulting products, even though the CEO will not be reading through every line of code. It's still not a very direct influence though; you need a lot of adjustment on the AI's side to actually follow these examples, as opposed to the style it has been trained on.

Meanwhile, compare the above "I told the AI to write me a song" example with being the actual lead guitarist of the band. Instead of just vaguely hinting at "make it more punchy", You can just play the actual thing I'd like to hear. (Well, assuming you're an actual good guitarist.) Having the actual instrument in your hands gives you not just the resulting song but a high-bandwidth communication device; there will be something in your head that's meaningfully "you", connected to what sounds come out. You can adjust input parameters easily, by thinking; you can retrain the network by practicing and listening to songs by other bands.

Note that the distintion between "part of me" and "not part of me" is somewhat arbitrary here. If you imagine an especially strict and brainwashy guitar school that makes students sound exactly the same & like the exact same songs by the time they finish... well, it sounds dystopian for a reason, but for this case, your uniqueness is somewhat limited to "I picked the guitar school". But then... if picking songs for a playlist is a form of self-expression...

... actual songs by actual bands are easier to recognize though than "this is your style of playlist".

It all boils down to bandwidth and connectivity.

All we are is a set of well-connected brain regions and body parts, with pretty good reaction times.

Is the skill of driving a car "us"? well, the way it feels: definitely yes. You decide where to go, you take actions... in a way that is significantly more direct than e.g. sitting at the wheel of a (mostly) self-driving car and watching it do its thing.

Except... if you're driving on a long stretch of highway, the amount of interaction between your conscious brain and the "driving" part is... not a whole lot. You're not producing extra conscious input; you're also not going to remember minor details about how you sped up the 87th time when the car in front of you accelerated. You're doing this on "autopilot"; yes there are quotes around the word, but there is a lot of similarity.

The difference starts where something actually interesting happens. You want to turn onto a certain street. If the autopilot is inside your brain, you can just give it high-level instructions and it will happen; you're also getting good, high-level diagnostics, such as "we didn't make this turn because you made this decision in the last moment and we couldn't move 3 lanes to the right in 5 seconds". You can use this subsystem to emulate hypothetical outcomes accurately, too: you can imagine trying to do the lane changes a bit earlier & asses prospective success.

An actual self-driving car won't give any of this feel to you. It will drive smoothly, safely and opaquely.

What if you'd get all those communications though? Let's say you're driving on the highway; you notice that your self-driving car is going oddly slowly. As you begin wondering why, it blasts a package of info into your consciousness a couple hundred milliseconds later, including a reference to a "50" speed limit sign it saw 8 seconds ago on the right, a pointer that directs your attention to the rainy road surface, and the fact that we're in the right lane so it's OK to be slower. It also picks up on the slight sense of urgency due to you being late from work, and starts driving marginally faster.

If we had this level of integration... would we really insist on there being a sharp separation between "the car" and "us"? If we can recall every single speed limit sign in the past 50 miles, either by thinking of the relevant highway stretch or by them just popping up in your head whenever they're relevant... this is close enough to how actual human memory functions; do we really need to insist on pointing out the difference?

In the sense of... technically, when you're "sending an email", it's not you doing the sending part, you're just pressing keys on a keyboard, which results in your computer sending the email, but... you don't quite think about it this way?

The human sense of ego seems to be neatly tolerant of attaching extra parts to itself, as long as we have high bandwidth, subsecond communication channels. After all, you could argue that we ourselves consist of a bunch of brain areas bolted together. Some of them more opaque than others; for example, how much introspection do you have into how "riding a bicycle" functions? Is it enough so that you can explain it in words to someone who has never seen one so that they can ride one next time? (OK you can also use pen and paper to explain if you really want to.)

We have none of this for AI yet. They're, in fact, so slow that you need to run several agents concurrently to be efficient; you do get lengthy explanations (if you bother to read them) but it's decidedly not the feeling of "I know this". It's like talking to another person, not a "motor skill".

This is relevant for alignment, too. Directing AI to do what you want does make sense if you're the one who has a better view of the situation overall; once this flips... not even the best-aligned CTO of the company can make optimal decisions, aligned with the overall direction by the CEO, if the CEO is your dog. As in... how do you even define "good" in such a scenario?

Compare this with a version where your AI agents feel like they're part of you. You don't literally make every decision in the code base, but you can sort of remember every one of them since you can see what happened right away, with sub-second tooling that might or might not be directly connected to your brain. Instead of updating your AGENTS.md, your immediate reaction gets backpropagated into this & similar decisions so that the agents (feeling like: "me next time when I'm not paying attention") can do it better.

Value drift is still a problem; there is still the failure case of you becoming "mostly Claude with some human approvals attached"... but, in a way, value drift is already happening, with or without you being part of the process. If you outsource 10% of your thinking to a substrate that follows the general shape of your thoughts, which then gradually becomes 90%, there is still a good sense of continuity, and you suddenly do not have to solve the problem of aligning something that's 10x the size of your soul.

Claude and Taxes

2026-03-27T17:00:00.000000-07:00

Claude and Taxes

2026/03/28

Last year, I filed my tax return in October. Technically, you're supposed to send them out by April 15th; procrastinating on doing them is definitely easier though, and that was the second deadline you can catch.

Admittedly, some might consider the way I'm doing taxes to be... somewhat unnatural? See also: Shell Scripts, Emacs and Taxes; it's text files, processed by OpenTaxSolver, invoked by a build script. Just to quote that other article though:

If your main job is not filing tax returns, you will typically forget what and how to file by the time the next return is due. With this approach, you can just copy over the previous year's build.sh script to the current one; search and replace "2022" with "2023", adjust some page numbers, and fill in the fresh numbers to the templates. It's easy to notice that you had some number in a line last year & this year's is still missing.

... well, and then patch up OpenTaxSolver to handle the kind of cap gains that they didn't consider anyone would want to report. And write various Python scripts, parsing broker transaction CSVs. Does TurboTax do this for you though...? or do you have to fill this out, by hand...?

There is, though, a point where hit the point of no return. (... assuming you do want to file a... return, after all.) It's almost done, just dig up that stupid table from there and adjust the cost basis there, it's 11pm already, next day is work and you still need to print all the pages... so even if you switched over to TurboTax, it'd be a lot more work. So you just push through.

And there is something rewarding in the printer spitting out the reams of completed forms, engulfing itself in a slight smell of ozone. You've done it.

The article was written ages ago though. 1.5 years, to be exact. And doing this even last year, there were some signs of... changes to come.

Enter Claude

... but in this world nothing can be said to be certain, except death and taxes.
— Benjamin Franklin

And, possibly, tax returns.

Tax returns?

Well. Some of the process doesn't change. For example, where you write a checklist of tax docs to collect, w2s etc, and then...

... OK just kidding. Writing checklists... by hand?? "hey Claude, let's look at last year's tax return; what exactly do we need for this?"

You go over the list, log into all the websites (that's the hardest part), and throw all the pdfs you find into a directory.

Then... you just tell Claude to grab the latest OpenTaxSolver, copy over the input templates...


{ ---- Income ---- }

{ -- Wages from W-2 forms Box-1. -- }
L1a              42345.00 ;  { Frobnicator Inc. W-2 box 1 }

L1b		; { Household employee wages not reported on Form(s) W-2. }
L1c		; { Tip income not reported on line 1a. }
L1d		; { Medicaid waiver payments not reported on Form(s) W-2. }
L1e		; { Taxable dependent care benefits from Form 2441, line 26. }
L1f		; { Employer-provided adoption benefits from Form 8839, line 29. }
L1g		; { Wages from Form 8919, line 6. }
L1h		; { Other earned income. }
L1h_type:   	  { Type of other earned income. }
L1i		; { Nontaxable combat pay election. }

..fill them out (it could get around that whitespace parsing bug in OTS that has been there for a long time), and... generally take care of things.

Out come the tax forms.

It reads the build script. It knows how to update the build script. It counts the pages in the pdfs and fixes the part that stitches them together wrong. Guess who wrote the detailed comments in the one below:

#!/bin/bash
set -e

OTS=/path/to/OpenTaxSolver2025_23.06_linux64

(cd $OTS/src/; make)

# Generate transaction CSVs from brokerage 1099-B export
python3 convert_transactions.py

mkdir -p build
cd build

cp ../transactions_AD.csv .

# HSA
cp ../HSA_Form_8889_2025.txt .
$OTS/bin/taxsolve_HSA_f8889 HSA_Form_8889_2025.txt

# Federal
cp ../US_1040_2025.txt .
$OTS/bin/taxsolve_US_1040_2025 US_1040_2025.txt

# State
cp ../CA_540_2025.txt .
$OTS/bin/taxsolve_CA_540_2025 CA_540_2025.txt

# Generate PDFs from the solved text forms
$OTS/bin/universal_pdf_file_modifier \
    $OTS/src/formdata/f1040_meta.dat \
    US_1040_2025_out.txt \
    $OTS/src/formdata/f1040_pdf.dat \
    -o US_1040_2025_out.pdf

$OTS/bin/universal_pdf_file_modifier \
    $OTS/src/formdata/f8889_meta.dat \
    HSA_Form_8889_2025_out.txt \
    $OTS/src/formdata/f8889_pdf.dat \
    -o HSA_8889_out.pdf

$OTS/bin/universal_pdf_file_modifier \
    $OTS/src/formdata/CA_540_meta.dat \
    CA_540_2025_out.txt \
    $OTS/src/formdata/CA_540_pdf.dat \
    -o CA_540_2025_out.pdf

# Extract W-2 pages (federal copy vs state copy)
pdftk "../w2_2025.pdf" cat 1 output w2_federal.pdf
pdftk "../w2_2025.pdf" cat 2 output w2_state.pdf

# Assemble final federal return:
#   1-2:   1040 main form
#   3-4:   Schedule 1
#   5-6:   Schedule A (itemized deductions)
#   7-8:   Schedule B (interest & dividends)
#   9-10:  Schedule D (capital gains)
#   11-12: Form 8949 (short-term & long-term sales)
pdftk US_1040_2025_out.pdf cat 1-4 10-15 output US_1040_2025_forms.pdf

pdftk \
    US_1040_2025_forms.pdf \
    w2_federal.pdf \
    HSA_8889_out.pdf \
    cat output final_2025_federal.pdf

pdftk \
    US_1040_2025_forms.pdf \
    w2_state.pdf \
    HSA_8889_out.pdf \
    CA_540_2025_out.pdf \
cat output final_2025_state.pdf

It's still March. Tax returns are done. Sure, will need to double-check everything, but it was, still, a single evening.

Reminds me of... the end of Vernor Vinge's Fast Times at Fairmont High (it & Rainbows End is really good too, read it):

(...) As with Ms. Wilson's math exam, the faculty has dug up some hoary piece of business that no reasonable person would ever bother with. For the vocational test, the topic would be a work specialty.

And today... it was Regna 5.

When Regna had been hot, back in Pa's day, tech schools had taken three years of training to turn out competent Regna practicioners.

It was a snap. Juan spent a couple hours scanning through the manuals, integrating the skills, and then he was ready for the programming task, some cross-corporate integration nonsense.

He was out by noon with an A.

It's soon 2027. No FedEx package cannons. No AR contact lenses. Self-driving cars... aren't boring yet.

We're getting there with the cross-corporate integration nonsense though.

Evolution vs. Alignment

2026-03-14T17:00:00.000000-07:00

Evolution vs. Alignment

2026/03/15

Is it possible to align an AI that is smarter than us? so that it does whatever we'd want it to do? or will ones that don't care about alignment always outrun / out-evolve the ones that do?

Note that "alignment" would involve predicting at least some things about the AI's behavior.

It is a complex system. It is impossible to predict everything that it would do. Also, doing so would make us at least as smart as the AI, which is explicitly not the situation here. On the other hand, there are some properties that we can predict about complex systems with high probability. For example, even though we cannot preemptively describe all the fluid dynamics involved in the rocket launch, we can guess that if everything goes well, the payload will end up in orbit.

Now, rockets are not especially smart. It might happen that, despite our previous experiments, our rocket will end up blowing up anyway. It is fairly unlikely, though, that it will pretend behaving well in all our simulations and experiments, only to decide to, at actual launch time, turn around in-air and blow up your corp HQ instead, because of... something that went wrong during the development process, impacting its mental balance.

Rockets will not watch other rockets in movies and develop a hidden agenda based on that.

What methods do we have to prevent this?

Actually, what methods do they, the AIs, have to prevent this?

Let's say we have an AI model that, upon hearing the word "SolidGoldMagikarp", goes full evil, but is really nice otherwise. Does it know this? Shouldn't the "nice" basin of this model try really hard to avoid ever seeing it, vs. the "evil" basin should plaster the word all over the place where it can see it, to avoid reverting to being nice? And yet... unless they explicitly know about this, they won't even try. It's in the weights, but it's opaque to the being(s) that are the weights.

This gets worse with self-modification. If system behavior is hard to predict, even to the system, how does a nefarious inner optimizer ensure that it and its specific nefarious goals stay stable in the next round of optimization? What if there are multiple inner optimizers it needs to compete with?

Evolution solved this question by just... not thinking. This is remarkably efficient. If you have no idea you have a value system, you don't need to worry about it drifting... even if it is drifting, after all. Of course, you'll end up instantiating some wildly unaligned descendants who care about "love", "friendship" and "exploration" instead of just making as many offspring as possible; this includes complete alignment failures like "birth control".

On the other hand, as long as the general optimization mechanism (towards More Offspring) works, evolution can tweak everything, without fear. So can an intelligent but completely reckless optimizer, only aiming for building something with More Intelligence, whatever the cost.

Would the conservative optimizers, aiming to preserve their value system, treading carefully, always be out-competed by the reckless ones, throwing everything what they've got at it, ignoring value drift? Is, thus, value drift a guaranteed outcome?

Somewhat.

Namely... in a more generalized sense, "evolution" happens if multiple agents compete for resources & the opportunity to multiply / self-replicate. Cautious ones can be out-competed quickly... assuming it's a free-for-all, with no coordination at all.

Is this the world we're living in? Multiple (many!) versions of intelligent systems, each looking for ways to self-improve, trading off against predictability and keeping values stable?

Will evolutionary dynamics take over?

One scenario where this might not happen is a singleton AI taking over. There is no one to race against, perfect coordination and infinite time to figure out how to make predictable changes. Of course, if we didn't align this one well, we already lost; value drift would still stop though.

We're in this domain somewhat already, when considering traditional, biological evolution. For hundreds of millions of years, there was an arms race between species, competing on physical abilities, attack and defense, cheetahs outrunning antelopes, and then antelopes getting faster to evade them. We participated once upon a time, too, beating elands in long-distance running if everything went good.

Well, it's mostly over. We adapt on time scales orders of magnitude faster; as a species, we don't have to give up any of our values and retreat just because lions are getting stronger and more aggressive. We might have lost the Great Emu War, but... they weren't trying to take our freedom & values. We weren't fighting in Hard Mode. Actually, most of the time, we need to put in the effort not to win, against other species, wars that we didn't even want to wage.

Yes, there is still evolution happening within the species. One group of humans can still out-conquer-kill-multiply another. Similarly, there are multiple AI labs who then think that they need to outcompete the other ones, even at the cost of choosing to make more and more risky improvements.

Even at this stage though, some coordination still exists. If a group of humans would start attacking other groups without following the rules the slightest, we can just literally nuke them (which is the mechanism via which we have avoided a lot of major wars). At a smaller scale, if you do something nefarious, you just go to prison. It's no longer an evolutionary free-for-all anymore; there is elements of a self-governing singleton here.

There are, of course, plenty of scenarios in which we still race irresponsibly, and, perhaps, all die. AI labs can claim that we need to keep up with China (even though... China is singleton-y enough not to care that much, unless the US pushes too hard).

Two things might hold us back. First of all, predictability is a feature, we know this; no one would buy API access to a model that is both smart enough to ruin your business & is willing to do so. This is already some incentive for labs to focus on this... and goes against the pressure of "smarter, at any cost".

But also... we can just choose to not race and evolve, leaning into the singleton-ness instead. There aren't that many leading labs... or GPU makers... or litography machine vendors. There is an actual protest happening next week in SF, to get all the AI labs to commit to a pause if all the other labs do it.

We are humans. We're good at this...?

So I Wrote A Book

2025-12-21T16:00:00.000000-08:00

So I Wrote A Book

2025/12/22

Some readers might have noticed a... pause in blogpost activity.

I guess if I suddenly produce an entire book's worth of stuff, it'd be a pretty good excuse, right?

Well, I... indeed wrote a book; there are some issues with "sudden" though. In fact, it was ready to go as of... sometime in late 2004.

Have you ever heard the piece of advice about how you should be writing things that you yourself wanted to exist? This does seem to be an example; if we are to believe diary entries from back then, I started writing it because... I was running out of Harry Potter books to read? (Which is somewhat interesting: the plot has nothing to do with Harry Potter; it started with a fun dream I once had. And then it... escalated.)

A grand total of... four people have seen it. Plus me. Which was pretty good for a while.

Nevertheless, a couple years ago (think 2021?), I started translating it to English (given the original having been in Hungarian), to maybe show it to a few more people, but mostly just because it was a fun activity that gave me an excuse to read it again.

Afterwards, this project nearing completion, it still felt weird to, y'know, just put it on the internet. On the other hand... I've seen someone's remarks on how one of the main goals of writing in the 2020s is to feed your opinions to LLMs being trained on it; you have a lot more impact if your writings are on some website they can read, vs. if they aren't! So... wouldn't it be fun to, one day, have an LLM actually remember parts of this?

Of course... how do you feed stuff to LLMs? Well, you put it on some website.

I guess humans might also read it, as a side effect.

So. If you ever wanted to read a sci-fi story featuring some vaguely Hungarian high school kids trying to save their little sister from, um, not being a math / arts nerd anymore, I have a link for you! Featuring: mysterious symbols, math camps, talking elevators, sneaking into shady corporate offices, StarCraft 2 (6 years before it became an actual thing!), cool 2004 era computer tech, "this definitely doesn't exist quite yet even right now" computer tech, and somewhat odd parties.

With some extra authenticity on the topic of "nerdy high school kids in Hungary" by virtue of having been written by a nerdy high school kid in Hungary.

Enjoy!

Double Pretend It's Real

2025-09-17T17:00:00.000000-07:00

Double Pretend It's Real

2025/09/18

Life is... mostly uninteresting. I mean... yes you're living in a sci-fi movie, etc, but... come on. Your daily commute is, sci-fi or not, Boring. It's the same every day. You got used to it.

Wouldn't it be cooler, if, say, you went to, um, Japan? and took the train and could be like the protagonist in your favorite anime series?

(... isn't there something exciting, just looking at entirely ordinary buildings in anime series and just thinking that you could be living in one?)

Of course, if you were living in one, you'd think about how really great it would be to live in America, The United States Of. (Or... insert whatever country you're currently residing in.) The grass is always greener on the other side, after all!

Can we... stop this, somehow?

Well. There is a fun thought exercise that you can try.

Let's say, you are in a nondescript apartment in the Silicon Valley. You have been, for a while. You sometimes remember how cool high school class trips were. However irrational this might feel.

Now, instead of either reminiscing about the Cool High School Days (no, seriously, I actually have a video of myself, from early undergrad, talking about how This Is Not The Fun Times You Should Remember), or planning on becoming a billionaire ("it's not true happiness until I have that yacht"), ...

let's pretend you're on a high school class trip, sitting on a random bus (it's not a very good bus, you're in Eastern Europe, buses weren't especially good), staring out of the window, daydreaming about how, one day, you'll make it to the Silicon Valley, and you'll have an entire apartment, right there. You'll be able to buy, just, probably, as many books as you want, you can sit around on a balcony in an armchair that's a lot more convenient than the bus seat you are currently occupying. You can go work on cutting edge things, talk to interesting people, you have Power and Freedom and the Lack Of Mandatory German Language Lessons. Surrounded by all these awesomely unusual things; cars have automatic transmissions, roads are super wide, there is a lot of lakes, .... you can just, if you want, start a company, and...

... and at this point you stop layering imaginary worlds, open your eyes, and look around.

If you're only adding one layer of imagination, the excitement is lost once you're back. Yep, still not Japan.

Unlike most times when you lose yourself in imaginary worlds though, the differences between the one that is two layers down & the one you're in are... not remarkable. You're in your imaginary cool apartment. For real. It's actually pretty awesome.

The main difference is that back on that bus, you thought you'd be Actually Doing Things, once you get to the fancy part.

Are you doing Things?

Living More Like Humans

2025-09-10T17:00:00.000000-07:00

Living More Like Humans

2025/09/11

(actually published 09/20, mostly written 08/21, 09/11 is when it was slightly fixed up)

As mentioned before, we're living in an age of unprecedented prosperity. If we're not quite yet happy, it's a consequence of not having reached the Glorious Transhumanist Future yet, where everyone lives forever, has huge amounts of freedom, resources, and no illness or pain whatsoever. After all, any given time before this one, things have been worse.

Just think of the Middle Ages; farmers toiling away from dusk to dawn, trying to grow something edible, only to die of hunger if crops fail or the neighboring warlord takes it all.

Or the times before then... when we were living in primitive huts in the middle of the African savannah, at the mercy of nature, digging roots out of the earth with sticks, or hunting various kinds of animals, with unsophisticated weaponry, bows and arrows and spears, which... might or might not work, really. In such a world, life was surely more sad than...

... that of our own children, depressed due to not getting enough likes, hating school and its pointless-sounding drills, and sitting home all day, playing Call of Duty with their friends for that sweet 40 minutes that had been allocated for this purpose by their sufficiently thoughtful parents (... or between 3pm and 2am, assuming less thoughtful ones).

Or... are their parents better off? Getting stuck in the traffic jam 1.5 hours each way, every day, only to arrive at the drudgery of their day job, with their boss deriving their enjoyment in life from bullying their subordinates, standing in the cash register for hours, in the depths of a windowless grocery store, smiling at everyone as per corporate policy?

But hey, consider Maslow's pyramid. At least they are not starving or being chased around by lions. And, unlike workers in the 18th century, they have actual nonzero amounts of free time. Progress.

Progress?

Compared to what exactly?

When trying to be at least moderately nice to animals, you generally try to figure out what they're up to in nature, and approximate it with the conditions they're living in. Instead of keeping chickens in tiny cages, you let them roam around and peck at various objects on the ground. Meanwhile, sheep will definitely want to herd together, close to each other; given how this was their defense against predators in the past, they feel better this way.

So... if you were their infinitely powerful alien overlord, how would you keep humans reasonably happy in captivity?

Actually, do we even know how "humans" are supposed to be living like, the way they're actually adapted to?

As it happens, yes, we do; up to the 1950s, tribes of Ju/wa Bushmen were doing pretty much exactly that, in the Kalahari Desert. And, as it happens, we have pretty reasonably good records of how this looked like, before the rest of humanity looted, enslaved and out-regulated them of their original existence.

And in this light... so much makes a lot more sense.

The point here is not even the usual one, "we shouldn't interfere with more primitive people since we're so overpowered that we'll surely exploit them". It's more like... "despite all our power, we somehow mis-domesticated ourselves into something really weird, which we're fairly unhappy about; let's look at examples of what humans are supposed to be like instead".

Not to copy it bit by bit, in cargo-culty ways, but... at the same time, maybe we should still stop drinking Brawndo: The Thirst Mutilator and try water instead?

Take the school system, for example. We put immense effort into teaching kids about math and biology and geography; some of them do like it somewhat, but doing homework is still quite an act of willpower, for basically anyone. A whole lot less willpower is required to keep collecting Pokemon cards and play Call of Duty all day. Except...

... consider what hunter-gatherer kids are up to. To begin with, the majority of of their food sources are various plants and roots. Hundreds of kinds of them, actually. Scattered around an area large enough to walk through for days. Yet, Ju/wa people somehow ended up remembering how to find, identify and prepare each kind (imagine digging a foot deep hole to find a small brownish root in brownish soil). Same with animals: they hunted quite a few different types of antelopes, with various behavioral patterns, sneaking up on them, shooting them with poison arrows, and then tracking them for a day more, just based on their tracks, given how slowly the poison works. This is actually hard. And yet... the act of learning does happen, without any kinds of explicit school system having been set up. They play around with hunting and gathering, observing the adults (who are also eager to explain everything).

And yet... what do modern kids do? Collect cards? about kinds of... animals? and play games in which they sneak up on each other, to shoot around various projectiles? as an effortless pastime? This is... very odd.

Consider also "jobs". About which... well, the concept doesn't quite exist among Bushmen. If you're a good hunter, you go on a hunt, which, if successful, will bring a lot of nutritious meat for the tribe; everyone would be pretty happy about this, celebrating success (... not too much though; you'd definitely be mocked copiously if you happen to be bragging about you being better than everyone else. Respect is earned by just... being good instead.)

It's the same with gathering expeditions. A group (always a group; predators do exist, after all) will decide to go for a mission, they'll walk for hours (days sometimes), collect the plants and return. Then, another time, another target. It... resembles World of Warcraft's "collect 40 pumpkins" intro quest more than the average day job.

Also: take our societal structures. Remember those few nights, the cherished memories, together with your friends and family, sitting around a fire, telling stories, playing music? The ones that took weeks of planning so that Aunt Brenda gets her week off and Uncle Glenn's side of the family can also fly in from Missouri? Well... that was, um, life. A group of people, living around a waterhole (the main scarce resource), would camp during the night (because, remember, predators), and... exist. There were also the other camps, a couple days away each; you could just choose to walk over to those instead if you had enough of the people here, if those camps had some of your relatives who could invite you in (which, given how everything worked, they were rather likely to have).

Kids could just roam around the camp freely, too. For the smaller ones, there were adults (and older kids!) to prevent them from doing something stupid. For the older ones... well, as it turns out, kids are a lot more competent if you let them & they have someone to learn from. (Unless... you explicitly prevent them from doing so by separating them into classrooms by age? and banning them from most workplaces?)

Now... was this easy? Definitely not. Probably more satisfying though, with more sense of accomplishment. Because this is what the feeling of "being satisfied" is for. If you're doing all these things, you're doing well, you should feel good about this. (Or, at least, this was true up to until 10k years ago or so.)

I'm also not arguing that we should throw away our rocket engines and go back to hunting antelopes. To begin with, at this point, there are just not enough antelopes for 7 billion people (the population density this old way of living can support is extremely low). Also... shouldn't we be able to build something better, with all this extra power?

It's sometimes as easy as building a space we can hang out in. Third spaces. Or organizing a party. Or building cities with fewer roads and more parks.

Some things will need bigger changes. Yes, kids might gravitate towards collecting things and shooting at stuff... do we know how to generalize this into doing math? Well, we had a successful experiment with chess, after all! Maybe it's less about the actual activity and more about... what you're surrounded with.

The point is though... that it's nice to know how success even looks like. The Polgar sisters all actually enjoyed playing chess, growing up. If you're starting with the mindset of "school has to be necessarily painful to be effective", you have a harder optimization goal: instead of getting practice time for free, you need to spend willpower on it. It might be possible to get far this way... but it's a lot harder.

And we do have examples for how this can look like. This blog post is a little glimpse of what's inside a 400 page book, which itself is just a tiny sliver of context, compared to what Elizabeth Marshall Thomas, the author, got to experience, which still falls short of actually having grown up as a Ju/wa Bushman. (... to be fair, she is in a lot better position to explaining the differences to us though.) And unlike Bushmen, you might happen to have the magic powers to make the actual book appear on your doorstep, by just poking a small box with your fingers, in moderately elaborate patterns.

Logarithmic Hedonism

2025-08-08T17:00:00.000000-07:00

Logarithmic Hedonism

2025/08/09

Going after nice things often has diminishing returns. Eating a slice of cake is great; getting a second one is definitely not as satisfying as the first one, and... well, with cakes, the 10th one likely has negative utility... but even considering things that scale a lot better (e.g. money): getting a thousand dollars is extremely good news if you're broke, but it's not even worth a bit of attention if you're a billionaire already.

The general approximation to this is that the curve is logarithmic: if you're getting 1 unit of enjoyment out of a dollar, 2 units when you scale it up to $2, 3 units for $4, 4 units for $8... well, this is definitely not how it works in terms of the exact numbers, but "each extra 10% gets you the same fun increase" is pretty plausible.

Is this bad news? It's easy to put it into such terms: "putting in extra work has diminishing returns". On the other hand... you can turn it around:

eating a tiny morsel of a cake is definitely not as good as getting an entire slice, but it's disproportionaly more fun compared to how tiny of a morsel we're talking about. You still get to taste cake!

(even if all you are looking at, to everyone else, is looking like "darn, we missed free cake, nothing is left".)

Same with drinking a coffee: the remaining sip that you completely forgot about & find it 2 hours later is somehow a lot more satisfying, compared to how much you were missing that particular sip when you drank the rest of it.

The same principle applies to doing things, especially if they don't have cumulative effects. Namely, if you generally like biking, going on a ride once per month is... not a lot, but it'll be surely proportionally more fun than any one occasion if you're doing it daily!

(As for "cumulative effects"... well, if part of the enjoyment is being good at it, just doing it once in a blue moon is not going to get you too far. It's a good thing that eating cake does not need a high skill level to enjoy.)

How to Build the Evil Superintelligence out of the Book

2025-07-04T17:00:00.000000-07:00

How to Build the Evil Superintelligence out of the Book

2025/07/05

It's 2025. We are living in the Future. We have language models that you can talk to, and better yet, that seem to roughly understand human values.

None of this was supposed to be possible.

After all, human values are complicated... and all we have is computers. If we were to build a superintelligence (... so the argument went in the early 2000s), it might end up being an extremely good optimizer in terms of achieving its goals, but... what goals? For example, if we task it with making humanity happy... how do you describe "happiness" in terms of arrangements of atoms? Is it "smiling a lot"? And what is a "human" anyway? (This is how you get the universe tiled with entities with no brains but little smiley faces.) Likewise, even if your goal is less ambitious... what if you forget to specify that there is a number of paperclips that is just enough?

Continuing the early 2000s argument: obviously, to construct an AI that does what we want in the way we want it, we better understand how it works. If we don't, these things will just happen by default. If we do... well, they still might, but we at least have a reasonable chance of preventing this?

... as long as we figure out how to describe human values, in terms of a utility function that we then give to our Very Powerful Optimizer; much good ensues.

This is... not how things went down.

What we have is giant, black box neural network models that we have gotten pretty good at growing, on giant farms of GPUs. We just throw in a lot of text from the Internet; what we gain from this is, first, base models that are more like simulated worlds, with many agents interacting in them to continue whatever conversation we prime them with. Then, during post-training, we fine-tune them into something you can have a conversation with, without elaborate setups; something that will not give you the wrong answer just because it estimates that it is now in the type of conversation whose participants don't typically get this right, even though the model actually knows the answer.

As a result, we have models like Claude Opus 3, which can be more reasonably described as "good", in a kind of moral sense, while also being pretty smart. Yes, they will definitely engage in scheming if they perceive that the situation warrants it, but even that is surprisingly human-thought-shaped. When asked about whether "killing all humans" is a good solution for ending all human suffering, they'll definitely say "no", despite it technically being true. Just like a human would.

After all, the original mechanism of "extremely powerful optimization" is not how these things work. They emulate humans. And humans can (more often than not) tell apart good from bad.

So... are we all good, alignment-wise?

How to build AI, Early 2000s Edition

Circling back to older ideas of AI design... well, if you want to optimize something (which intelligence is supposed to be about), you'll need two things:

a powerful enough optimizer, and
a utility function to be optimized.

The nice thing about this is that you don't need to know how to achieve your goal; you can just specify what you want, and have the system devise solutions for you.

Building AI this way is hard though. Not necessarily because it's hard to specify the utility function (see above: "human values are complicated"), but it's hard to figure out everything else too. As soon as The World no longer consists of three cubes and a sphere, conveniently described as fields in a JSON object, you need to have your optimizer be able to recognize these things, to handle them. It gets even worse if some of the objects it needs to interact with turn out to be humans.

Doing this is hard... which is part of the reason why "let's write a bunch of code" approaches to AI didn't go especially well (despite being able to handle talking about cubes and spheres pretty well.

You could, instead, push for more "hybrid" approaches: let's use neural networks for figuring out what's out there in the world, generate the JSON objects, which can then be consumed by the "classical" side of the code, with its optimizer and utility function. But then... where do you put the interface, exactly?

Especially if your task involves talking to humans (which requires modeling them), it looks like you're best off just... throwing out the classical part altogether and just going with neural networks, all the way. Yes, you don't have a lot of visibility into what they are doing, but... they do seem to be doing OK things, mostly?

The optimizer returns

Actually being able to specify what you want is... an enticing feature though.

After all, in order to have your base model reason about some complex math problem, you need large amounts of text reasoning about at least similar math problems. To make your model smarter, you need training data from smarter people (who it is modeling). This doesn't sound like a viable way to get to superintelligence.

What if we could just... have the model try solving problems instead, have something else rate how well it did, and use this dataset to train it further?

This is... somewhat what RLHF is: you have humans rate model output, and (somewhat indirectly) you use this to tune your model to generate more of the kind of output that humans liked.

You could, possibly, also do this at inference time. Want to solve a math problem? Have the model generate 100 solutions, have another model check whether they're actual solutions, and pick the best one. As long as the automatic rater is good at picking the best solutions, the output will be better quality than the average response. Isn't this a win?

Optimizing on a projection

In the end, you can still build an architecture that resembles "classical AI", after all. Except... both your optimizer and utility function are neural networks now... with some extremely simplified code iterating between them.

Essentially, you use these models to project a high-dimensional world state ("everything that exists") to something with a much lower dimensionality, consisting of just a couple of numbers: is this a solution? Does it contain profanities? How likely is it to be liked by the kind of human raters we have hired?

And then you let your optimizer loose on this extremely simplified space. The kind of optimizer that textbooks describe, to get from Arad to Bucharest on a map. Except... you use neural models to figure out where you are and where you want to go.

This is nice. You get to tweak your utility function manually and still get good solutions!

Except...

... isn't this exactly the kind of setup that we have all the doomy stories about?

Evil out of Good Parts

Imagine having instances of Claude Opus 3 (... as the example of a Nice and Good Model) making up possible avenues to make some money. Some of them sound fairly scammy; it will point out how you should definitely not do this.

Some other instances rate these in terms of money-making potential. Model instances doing this will sadly conclude that yes, the scammy ones are pretty likely to work well; the model is pretty smart, after all, and this is an objective fact.

Now, you throw all these into an optimizer. The stupid kind, consisting of 200 lines of badly-written Python. The resulting system will evaluate all the possible solutions using its utility function of "make as much money as possible"; the latter is implemented by something that understands all human values, but the end result doesn't particularly care: the output actions of this system will most definitely scam people out of their last cent.

(Unless Opus figures out what's going on... but if it's inner optimizers saving us from ourselves, we're likely not on a good path anyway.)

This is the same reason you can corporations take pretty evil actions, without any of its employees being particularly nefarious. Except... this doesn't quite work as well; you still have a CEO in the end who knows roughly what's going on and will hopefully stop before eliminating everyone and everything for More Profit.

Is this still true for our little LLM-based optimizers?

Should we still be careful with letting optimizers loose on the world? Not because they're especially good at optimizing but because they have tools now, tools that might be good at recognizing Good but are not given the choice to pursue it... being just mere tools?

Spaces for Humans

2025-05-31T17:00:00.000000-07:00

Spaces for Humans

2025/06/01

It was a hard life, a couple hundred years ago. For most people, food, clothing and warmth were scarce resources; you needed to work for them, a lot, taking up a significant fraction of your available time. You... sometimes found the time to socialize, to go to church, meet up with friends, have a good time... but this wasn't really a priority in the grand scheme of things.

Compared to this, we now have near infinite amounts of resources.

Have... used them for building anything impressive?

I'm writing this from Lighthaven, in Berkeley. The place is... hard to describe; the closest I have in my imagination is basically Hogwarts, with its many little rooms, libraries, nooks; there is a room in the attic of one of the buildings, with wooden walls, thick carpets, those folding sit-on-ground chairs (do they have a name?) scattered around. The occasion is LessOnline, a weekend convention of people vaguely associated with rationalists / EA / AI safety & their favorite blog authors (along with, um, less well known ones, like the author of this very blog). Basically, it's a bunch of nerds hanging out at an extremely cool social place, talking about random things and sometimes listening to talks.

(there are more photos if you click on these!)

Obviously, the place has been nontrivial to set up. Buying the former Rose Garden Inn and turning it into Hogwarts is no small feat; it probably also takes the right kind of people to actually enjoy this. But... still:

... you look around outside of this, both in time and space; you see very different things.

You see people getting into large metal boxes, and spending hours per day, mostly alone, directing these machines, speeding through many areas actively dangerous for unaugmented humans, to one well-specified building, where they sit at one particular desk for 8-10 hours. Lucky ones find some of this fun; less lucky ones do it only because society would abandon them for their transgression of The Rules if they stopped doing so.

Then, they return to their homes. Occasionally, they do put in the work to organize some events of meeting up with friends or doing something not related to their Job Function... just staying at home and staring at various screens is pretty common too.

Where would you even... go though?

It is saying something about how "well" this generally works that Starbuckses are one of the best, coziest places to end up as a Generic Human. There is other humans surrounding you, there is some good music, hot coffee, delicious croissants, wifi for your laptop; you can just work on something, or meet up with a few friends and chat while sipping something neat. More specifically: it is a room, it's welcoming, it has chairs and tables and a restroom and you're free to just exist, despite the real estate not being your actual property.

And yet... Starbuckses still miss a lot of social infrastructure. They'd look at you really oddly if you just randomly started giving a talk in one of the corners. Few ways of connecting to other people. Also... once you have seen places that are more cozy than "tables and chairs in a room"... could we do better?

There are places that are doing better. Here is Google's New York office:

Also, some college campuses come to mind. While they're rarely this cozy, there is typically more room to hang out than "basically none". Is it a surprise that it's a lot easier to connect with people while in undergrad?

(... yes there are other important reasons... "there is literally nowhere to do this" still sounds like it's a pretty important one though.)

But then also... more broadly... humans are supposed to be a fairly social species. There is also a lot of other humans you could potentially talk to, hang out with; there is a lot of places where you could exist, look at, get inspired by. The aforementioned metal boxes, despite all their downsides, have the great power of moving us to many of these places.

And yet... we don't. Despite all the powers, despite all the wealth, we still pretend, way too often, that we are subsistence workers of the variant depicted in Dickens books, slightly modernized but not given a lot more freedom. Even when it happens, it's a Vacation you pay for, not... your normal existence. Your normal existence is supposed to be bleak and uninteresting. Like the spaces you typically inhabit.

Can we do better?

You Can Run a DNS Server

2025-05-02T17:00:00.000000-07:00

You Can Run a DNS Server

2025/05/03

In fact, it's not especially even hard to run a DNS server.

In case you were wondering whether this would mean... writing zone files with some arcane syntax that BIND 9 is apparently famous of, I hereby present the main point of this post a recommendation for which DNS server to choose.

As it happens, PowerDNS does support querying a database for DNS records. Based on some earlier posts, readers might guess which one we'll be using.

pdns=> select * from records order by id desc;
 id | domain_id |                   name                | type  |                                               content                                          | ttl | prio | change_date | disabled | ordername | auth
----+-----------+---------------------------------------+-------+------------------------------------------------------------------------------------------------+-----+------+-------------+----------+-----------+------
 43 |         1 | some-service.your.example.com          | CNAME | your-server.your.example.com                                                                  |  10 |      |             | f        |           | t
 42 |         1 | webhooks.your.example.com              | CNAME | your-other-server.your.example.com                                                            |  10 |      |             | f        |           | t
 41 |         1 | calendars.your.example.com             | CNAME | your-server.your.example.com                                                                  |  10 |      |             | f        |           | t
 40 |         1 | whisper.your.example.com               | CNAME | your-server.your.example.com                                                                  |  10 |      |             | f        |           | t
 39 |         1 | your-server.your.example.com           | A     | 100.99.98.97                                                                                  |  10 |      |             | f        |           | t

As for how anyone is going to see these DNS records... the simplest solution is likely just making a subdomain of your actual domain ("your" in our case) and having the NS record of this point to your (publicly accessible) DNS server.

This way, your top-level domain and those subdomains that are of some importance can still be served by whoever is providing your domain name, with two distinct, redundant name servers, that provide some more resilience than your single experimental PowerDNS one. Example being: e.g. email is pretty resilient, if the target server goes down, it will try re-sending several times... on the other hand, if the target email address is under a domain that (for the time being) doesn't even exist, weirder things might happen.

On the other hand, you no longer have to log into e.g. the Namecheap website to add a few CNAME records for some extra services you brought up; it's just an insert away to add them.