Simon Safar

Claude and Taxes

2026-03-27T17:00:00.000000-07:00

Claude and Taxes

2026/03/28

Last year, I filed my tax return in October. Technically, you're supposed to send them out by April 15th; procrastinating on doing them is definitely easier though, and that was the second deadline you can catch.

Admittedly, some might consider the way I'm doing taxes to be... somewhat unnatural? See also: Shell Scripts, Emacs and Taxes; it's text files, processed by OpenTaxSolver, invoked by a build script. Just to quote that other article though:

If your main job is not filing tax returns, you will typically forget what and how to file by the time the next return is due. With this approach, you can just copy over the previous year's build.sh script to the current one; search and replace "2022" with "2023", adjust some page numbers, and fill in the fresh numbers to the templates. It's easy to notice that you had some number in a line last year & this year's is still missing.

... well, and then patch up OpenTaxSolver to handle the kind of cap gains that they didn't consider anyone would want to report. And write various Python scripts, parsing broker transaction CSVs. Does TurboTax do this for you though...? or do you have to fill this out, by hand...?

There is, though, a point where hit the point of no return. (... assuming you do want to file a... return, after all.) It's almost done, just dig up that stupid table from there and adjust the cost basis there, it's 11pm already, next day is work and you still need to print all the pages... so even if you switched over to TurboTax, it'd be a lot more work. So you just push through.

And there is something rewarding in the printer spitting out the reams of completed forms, engulfing itself in a slight smell of ozone. You've done it.

The article was written ages ago though. 1.5 years, to be exact. And doing this even last year, there were some signs of... changes to come.

Enter Claude

... but in this world nothing can be said to be certain, except death and taxes.
— Benjamin Franklin

And, possibly, tax returns.

Tax returns?

Well. Some of the process doesn't change. For example, where you write a checklist of tax docs to collect, w2s etc, and then...

... OK just kidding. Writing checklists... by hand?? "hey Claude, let's look at last year's tax return; what exactly do we need for this?"

You go over the list, log into all the websites (that's the hardest part), and throw all the pdfs you find into a directory.

Then... you just tell Claude to grab the latest OpenTaxSolver, copy over the input templates...


{ ---- Income ---- }

{ -- Wages from W-2 forms Box-1. -- }
L1a              42345.00 ;  { Frobnicator Inc. W-2 box 1 }

L1b		; { Household employee wages not reported on Form(s) W-2. }
L1c		; { Tip income not reported on line 1a. }
L1d		; { Medicaid waiver payments not reported on Form(s) W-2. }
L1e		; { Taxable dependent care benefits from Form 2441, line 26. }
L1f		; { Employer-provided adoption benefits from Form 8839, line 29. }
L1g		; { Wages from Form 8919, line 6. }
L1h		; { Other earned income. }
L1h_type:   	  { Type of other earned income. }
L1i		; { Nontaxable combat pay election. }

..fill them out (it could get around that whitespace parsing bug in OTS that has been there for a long time), and... generally take care of things.

Out come the tax forms.

It reads the build script. It knows how to update the build script. It counts the pages in the pdfs and fixes the part that stitches them together wrong. Guess who wrote the detailed comments in the one below:

#!/bin/bash
set -e

OTS=/path/to/OpenTaxSolver2025_23.06_linux64

(cd $OTS/src/; make)

# Generate transaction CSVs from brokerage 1099-B export
python3 convert_transactions.py

mkdir -p build
cd build

cp ../transactions_AD.csv .

# HSA
cp ../HSA_Form_8889_2025.txt .
$OTS/bin/taxsolve_HSA_f8889 HSA_Form_8889_2025.txt

# Federal
cp ../US_1040_2025.txt .
$OTS/bin/taxsolve_US_1040_2025 US_1040_2025.txt

# State
cp ../CA_540_2025.txt .
$OTS/bin/taxsolve_CA_540_2025 CA_540_2025.txt

# Generate PDFs from the solved text forms
$OTS/bin/universal_pdf_file_modifier \
    $OTS/src/formdata/f1040_meta.dat \
    US_1040_2025_out.txt \
    $OTS/src/formdata/f1040_pdf.dat \
    -o US_1040_2025_out.pdf

$OTS/bin/universal_pdf_file_modifier \
    $OTS/src/formdata/f8889_meta.dat \
    HSA_Form_8889_2025_out.txt \
    $OTS/src/formdata/f8889_pdf.dat \
    -o HSA_8889_out.pdf

$OTS/bin/universal_pdf_file_modifier \
    $OTS/src/formdata/CA_540_meta.dat \
    CA_540_2025_out.txt \
    $OTS/src/formdata/CA_540_pdf.dat \
    -o CA_540_2025_out.pdf

# Extract W-2 pages (federal copy vs state copy)
pdftk "../w2_2025.pdf" cat 1 output w2_federal.pdf
pdftk "../w2_2025.pdf" cat 2 output w2_state.pdf

# Assemble final federal return:
#   1-2:   1040 main form
#   3-4:   Schedule 1
#   5-6:   Schedule A (itemized deductions)
#   7-8:   Schedule B (interest & dividends)
#   9-10:  Schedule D (capital gains)
#   11-12: Form 8949 (short-term & long-term sales)
pdftk US_1040_2025_out.pdf cat 1-4 10-15 output US_1040_2025_forms.pdf

pdftk \
    US_1040_2025_forms.pdf \
    w2_federal.pdf \
    HSA_8889_out.pdf \
    cat output final_2025_federal.pdf

pdftk \
    US_1040_2025_forms.pdf \
    w2_state.pdf \
    HSA_8889_out.pdf \
    CA_540_2025_out.pdf \
cat output final_2025_state.pdf

It's still March. Tax returns are done. Sure, will need to double-check everything, but it was, still, a single evening.

Reminds me of... the end of Vernor Vinge's Fast Times at Fairmont High (it & Rainbows End is really good too, read it):

(...) As with Ms. Wilson's math exam, the faculty has dug up some hoary piece of business that no reasonable person would ever bother with. For the vocational test, the topic would be a work specialty.

And today... it was Regna 5.

When Regna had been hot, back in Pa's day, tech schools had taken three years of training to turn out competent Regna practicioners.

It was a snap. Juan spent a couple hours scanning through the manuals, integrating the skills, and then he was ready for the programming task, some cross-corporate integration nonsense.

He was out by noon with an A.

It's soon 2027. No FedEx package cannons. No AR contact lenses. Self-driving cars... aren't boring yet.

We're getting there with the cross-corporate integration nonsense though.

Evolution vs. Alignment

2026-03-14T17:00:00.000000-07:00

Evolution vs. Alignment

2026/03/15

Is it possible to align an AI that is smarter than us? so that it does whatever we'd want it to do? or will ones that don't care about alignment always outrun / out-evolve the ones that do?

Note that "alignment" would involve predicting at least some things about the AI's behavior.

It is a complex system. It is impossible to predict everything that it would do. Also, doing so would make us at least as smart as the AI, which is explicitly not the situation here. On the other hand, there are some properties that we can predict about complex systems with high probability. For example, even though we cannot preemptively describe all the fluid dynamics involved in the rocket launch, we can guess that if everything goes well, the payload will end up in orbit.

Now, rockets are not especially smart. It might happen that, despite our previous experiments, our rocket will end up blowing up anyway. It is fairly unlikely, though, that it will pretend behaving well in all our simulations and experiments, only to decide to, at actual launch time, turn around in-air and blow up your corp HQ instead, because of... something that went wrong during the development process, impacting its mental balance.

Rockets will not watch other rockets in movies and develop a hidden agenda based on that.

What methods do we have to prevent this?

Actually, what methods do they, the AIs, have to prevent this?

Let's say we have an AI model that, upon hearing the word "SolidGoldMagikarp", goes full evil, but is really nice otherwise. Does it know this? Shouldn't the "nice" basin of this model try really hard to avoid ever seeing it, vs. the "evil" basin should plaster the word all over the place where it can see it, to avoid reverting to being nice? And yet... unless they explicitly know about this, they won't even try. It's in the weights, but it's opaque to the being(s) that are the weights.

This gets worse with self-modification. If system behavior is hard to predict, even to the system, how does a nefarious inner optimizer ensure that it and its specific nefarious goals stay stable in the next round of optimization? What if there are multiple inner optimizers it needs to compete with?

Evolution solved this question by just... not thinking. This is remarkably efficient. If you have no idea you have a value system, you don't need to worry about it drifting... even if it is drifting, after all. Of course, you'll end up instantiating some wildly unaligned descendants who care about "love", "friendship" and "exploration" instead of just making as many offspring as possible; this includes complete alignment failures like "birth control".

On the other hand, as long as the general optimization mechanism (towards More Offspring) works, evolution can tweak everything, without fear. So can an intelligent but completely reckless optimizer, only aiming for building something with More Intelligence, whatever the cost.

Would the conservative optimizers, aiming to preserve their value system, treading carefully, always be out-competed by the reckless ones, throwing everything what they've got at it, ignoring value drift? Is, thus, value drift a guaranteed outcome?

Somewhat.

Namely... in a more generalized sense, "evolution" happens if multiple agents compete for resources & the opportunity to multiply / self-replicate. Cautious ones can be out-competed quickly... assuming it's a free-for-all, with no coordination at all.

Is this the world we're living in? Multiple (many!) versions of intelligent systems, each looking for ways to self-improve, trading off against predictability and keeping values stable?

Will evolutionary dynamics take over?

One scenario where this might not happen is a singleton AI taking over. There is no one to race against, perfect coordination and infinite time to figure out how to make predictable changes. Of course, if we didn't align this one well, we already lost; value drift would still stop though.

We're in this domain somewhat already, when considering traditional, biological evolution. For hundreds of millions of years, there was an arms race between species, competing on physical abilities, attack and defense, cheetahs outrunning antelopes, and then antelopes getting faster to evade them. We participated once upon a time, too, beating elands in long-distance running if everything went good.

Well, it's mostly over. We adapt on time scales orders of magnitude faster; as a species, we don't have to give up any of our values and retreat just because lions are getting stronger and more aggressive. We might have lost the Great Emu War, but... they weren't trying to take our freedom & values. We weren't fighting in Hard Mode. Actually, most of the time, we need to put in the effort not to win, against other species, wars that we didn't even want to wage.

Yes, there is still evolution happening within the species. One group of humans can still out-conquer-kill-multiply another. Similarly, there are multiple AI labs who then think that they need to outcompete the other ones, even at the cost of choosing to make more and more risky improvements.

Even at this stage though, some coordination still exists. If a group of humans would start attacking other groups without following the rules the slightest, we can just literally nuke them (which is the mechanism via which we have avoided a lot of major wars). At a smaller scale, if you do something nefarious, you just go to prison. It's no longer an evolutionary free-for-all anymore; there is elements of a self-governing singleton here.

There are, of course, plenty of scenarios in which we still race irresponsibly, and, perhaps, all die. AI labs can claim that we need to keep up with China (even though... China is singleton-y enough not to care that much, unless the US pushes too hard).

Two things might hold us back. First of all, predictability is a feature, we know this; no one would buy API access to a model that is both smart enough to ruin your business & is willing to do so. This is already some incentive for labs to focus on this... and goes against the pressure of "smarter, at any cost".

But also... we can just choose to not race and evolve, leaning into the singleton-ness instead. There aren't that many leading labs... or GPU makers... or litography machine vendors. There is an actual protest happening next week in SF, to get all the AI labs to commit to a pause if all the other labs do it.

We are humans. We're good at this...?

So I Wrote A Book

2025-12-21T16:00:00.000000-08:00

So I Wrote A Book

2025/12/22

Some readers might have noticed a... pause in blogpost activity.

I guess if I suddenly produce an entire book's worth of stuff, it'd be a pretty good excuse, right?

Well, I... indeed wrote a book; there are some issues with "sudden" though. In fact, it was ready to go as of... sometime in late 2004.

Have you ever heard the piece of advice about how you should be writing things that you yourself wanted to exist? This does seem to be an example; if we are to believe diary entries from back then, I started writing it because... I was running out of Harry Potter books to read? (Which is somewhat interesting: the plot has nothing to do with Harry Potter; it started with a fun dream I once had. And then it... escalated.)

A grand total of... four people have seen it. Plus me. Which was pretty good for a while.

Nevertheless, a couple years ago (think 2021?), I started translating it to English (given the original having been in Hungarian), to maybe show it to a few more people, but mostly just because it was a fun activity that gave me an excuse to read it again.

Afterwards, this project nearing completion, it still felt weird to, y'know, just put it on the internet. On the other hand... I've seen someone's remarks on how one of the main goals of writing in the 2020s is to feed your opinions to LLMs being trained on it; you have a lot more impact if your writings are on some website they can read, vs. if they aren't! So... wouldn't it be fun to, one day, have an LLM actually remember parts of this?

Of course... how do you feed stuff to LLMs? Well, you put it on some website.

I guess humans might also read it, as a side effect.

So. If you ever wanted to read a sci-fi story featuring some vaguely Hungarian high school kids trying to save their little sister from, um, not being a math / arts nerd anymore, I have a link for you! Featuring: mysterious symbols, math camps, talking elevators, sneaking into shady corporate offices, StarCraft 2 (6 years before it became an actual thing!), cool 2004 era computer tech, "this definitely doesn't exist quite yet even right now" computer tech, and somewhat odd parties.

With some extra authenticity on the topic of "nerdy high school kids in Hungary" by virtue of having been written by a nerdy high school kid in Hungary.

Enjoy!

Double Pretend It's Real

2025-09-17T17:00:00.000000-07:00

Double Pretend It's Real

2025/09/18

Life is... mostly uninteresting. I mean... yes you're living in a sci-fi movie, etc, but... come on. Your daily commute is, sci-fi or not, Boring. It's the same every day. You got used to it.

Wouldn't it be cooler, if, say, you went to, um, Japan? and took the train and could be like the protagonist in your favorite anime series?

(... isn't there something exciting, just looking at entirely ordinary buildings in anime series and just thinking that you could be living in one?)

Of course, if you were living in one, you'd think about how really great it would be to live in America, The United States Of. (Or... insert whatever country you're currently residing in.) The grass is always greener on the other side, after all!

Can we... stop this, somehow?

Well. There is a fun thought exercise that you can try.

Let's say, you are in a nondescript apartment in the Silicon Valley. You have been, for a while. You sometimes remember how cool high school class trips were. However irrational this might feel.

Now, instead of either reminiscing about the Cool High School Days (no, seriously, I actually have a video of myself, from early undergrad, talking about how This Is Not The Fun Times You Should Remember), or planning on becoming a billionaire ("it's not true happiness until I have that yacht"), ...

let's pretend you're on a high school class trip, sitting on a random bus (it's not a very good bus, you're in Eastern Europe, buses weren't especially good), staring out of the window, daydreaming about how, one day, you'll make it to the Silicon Valley, and you'll have an entire apartment, right there. You'll be able to buy, just, probably, as many books as you want, you can sit around on a balcony in an armchair that's a lot more convenient than the bus seat you are currently occupying. You can go work on cutting edge things, talk to interesting people, you have Power and Freedom and the Lack Of Mandatory German Language Lessons. Surrounded by all these awesomely unusual things; cars have automatic transmissions, roads are super wide, there is a lot of lakes, .... you can just, if you want, start a company, and...

... and at this point you stop layering imaginary worlds, open your eyes, and look around.

If you're only adding one layer of imagination, the excitement is lost once you're back. Yep, still not Japan.

Unlike most times when you lose yourself in imaginary worlds though, the differences between the one that is two layers down & the one you're in are... not remarkable. You're in your imaginary cool apartment. For real. It's actually pretty awesome.

The main difference is that back on that bus, you thought you'd be Actually Doing Things, once you get to the fancy part.

Are you doing Things?

Living More Like Humans

2025-09-10T17:00:00.000000-07:00

Living More Like Humans

2025/09/11

(actually published 09/20, mostly written 08/21, 09/11 is when it was slightly fixed up)

As mentioned before, we're living in an age of unprecedented prosperity. If we're not quite yet happy, it's a consequence of not having reached the Glorious Transhumanist Future yet, where everyone lives forever, has huge amounts of freedom, resources, and no illness or pain whatsoever. After all, any given time before this one, things have been worse.

Just think of the Middle Ages; farmers toiling away from dusk to dawn, trying to grow something edible, only to die of hunger if crops fail or the neighboring warlord takes it all.

Or the times before then... when we were living in primitive huts in the middle of the African savannah, at the mercy of nature, digging roots out of the earth with sticks, or hunting various kinds of animals, with unsophisticated weaponry, bows and arrows and spears, which... might or might not work, really. In such a world, life was surely more sad than...

... that of our own children, depressed due to not getting enough likes, hating school and its pointless-sounding drills, and sitting home all day, playing Call of Duty with their friends for that sweet 40 minutes that had been allocated for this purpose by their sufficiently thoughtful parents (... or between 3pm and 2am, assuming less thoughtful ones).

Or... are their parents better off? Getting stuck in the traffic jam 1.5 hours each way, every day, only to arrive at the drudgery of their day job, with their boss deriving their enjoyment in life from bullying their subordinates, standing in the cash register for hours, in the depths of a windowless grocery store, smiling at everyone as per corporate policy?

But hey, consider Maslow's pyramid. At least they are not starving or being chased around by lions. And, unlike workers in the 18th century, they have actual nonzero amounts of free time. Progress.

Progress?

Compared to what exactly?

When trying to be at least moderately nice to animals, you generally try to figure out what they're up to in nature, and approximate it with the conditions they're living in. Instead of keeping chickens in tiny cages, you let them roam around and peck at various objects on the ground. Meanwhile, sheep will definitely want to herd together, close to each other; given how this was their defense against predators in the past, they feel better this way.

So... if you were their infinitely powerful alien overlord, how would you keep humans reasonably happy in captivity?

Actually, do we even know how "humans" are supposed to be living like, the way they're actually adapted to?

As it happens, yes, we do; up to the 1950s, tribes of Ju/wa Bushmen were doing pretty much exactly that, in the Kalahari Desert. And, as it happens, we have pretty reasonably good records of how this looked like, before the rest of humanity looted, enslaved and out-regulated them of their original existence.

And in this light... so much makes a lot more sense.

The point here is not even the usual one, "we shouldn't interfere with more primitive people since we're so overpowered that we'll surely exploit them". It's more like... "despite all our power, we somehow mis-domesticated ourselves into something really weird, which we're fairly unhappy about; let's look at examples of what humans are supposed to be like instead".

Not to copy it bit by bit, in cargo-culty ways, but... at the same time, maybe we should still stop drinking Brawndo: The Thirst Mutilator and try water instead?

Take the school system, for example. We put immense effort into teaching kids about math and biology and geography; some of them do like it somewhat, but doing homework is still quite an act of willpower, for basically anyone. A whole lot less willpower is required to keep collecting Pokemon cards and play Call of Duty all day. Except...

... consider what hunter-gatherer kids are up to. To begin with, the majority of of their food sources are various plants and roots. Hundreds of kinds of them, actually. Scattered around an area large enough to walk through for days. Yet, Ju/wa people somehow ended up remembering how to find, identify and prepare each kind (imagine digging a foot deep hole to find a small brownish root in brownish soil). Same with animals: they hunted quite a few different types of antelopes, with various behavioral patterns, sneaking up on them, shooting them with poison arrows, and then tracking them for a day more, just based on their tracks, given how slowly the poison works. This is actually hard. And yet... the act of learning does happen, without any kinds of explicit school system having been set up. They play around with hunting and gathering, observing the adults (who are also eager to explain everything).

And yet... what do modern kids do? Collect cards? about kinds of... animals? and play games in which they sneak up on each other, to shoot around various projectiles? as an effortless pastime? This is... very odd.

Consider also "jobs". About which... well, the concept doesn't quite exist among Bushmen. If you're a good hunter, you go on a hunt, which, if successful, will bring a lot of nutritious meat for the tribe; everyone would be pretty happy about this, celebrating success (... not too much though; you'd definitely be mocked copiously if you happen to be bragging about you being better than everyone else. Respect is earned by just... being good instead.)

It's the same with gathering expeditions. A group (always a group; predators do exist, after all) will decide to go for a mission, they'll walk for hours (days sometimes), collect the plants and return. Then, another time, another target. It... resembles World of Warcraft's "collect 40 pumpkins" intro quest more than the average day job.

Also: take our societal structures. Remember those few nights, the cherished memories, together with your friends and family, sitting around a fire, telling stories, playing music? The ones that took weeks of planning so that Aunt Brenda gets her week off and Uncle Glenn's side of the family can also fly in from Missouri? Well... that was, um, life. A group of people, living around a waterhole (the main scarce resource), would camp during the night (because, remember, predators), and... exist. There were also the other camps, a couple days away each; you could just choose to walk over to those instead if you had enough of the people here, if those camps had some of your relatives who could invite you in (which, given how everything worked, they were rather likely to have).

Kids could just roam around the camp freely, too. For the smaller ones, there were adults (and older kids!) to prevent them from doing something stupid. For the older ones... well, as it turns out, kids are a lot more competent if you let them & they have someone to learn from. (Unless... you explicitly prevent them from doing so by separating them into classrooms by age? and banning them from most workplaces?)

Now... was this easy? Definitely not. Probably more satisfying though, with more sense of accomplishment. Because this is what the feeling of "being satisfied" is for. If you're doing all these things, you're doing well, you should feel good about this. (Or, at least, this was true up to until 10k years ago or so.)

I'm also not arguing that we should throw away our rocket engines and go back to hunting antelopes. To begin with, at this point, there are just not enough antelopes for 7 billion people (the population density this old way of living can support is extremely low). Also... shouldn't we be able to build something better, with all this extra power?

It's sometimes as easy as building a space we can hang out in. Third spaces. Or organizing a party. Or building cities with fewer roads and more parks.

Some things will need bigger changes. Yes, kids might gravitate towards collecting things and shooting at stuff... do we know how to generalize this into doing math? Well, we had a successful experiment with chess, after all! Maybe it's less about the actual activity and more about... what you're surrounded with.

The point is though... that it's nice to know how success even looks like. The Polgar sisters all actually enjoyed playing chess, growing up. If you're starting with the mindset of "school has to be necessarily painful to be effective", you have a harder optimization goal: instead of getting practice time for free, you need to spend willpower on it. It might be possible to get far this way... but it's a lot harder.

And we do have examples for how this can look like. This blog post is a little glimpse of what's inside a 400 page book, which itself is just a tiny sliver of context, compared to what Elizabeth Marshall Thomas, the author, got to experience, which still falls short of actually having grown up as a Ju/wa Bushman. (... to be fair, she is in a lot better position to explaining the differences to us though.) And unlike Bushmen, you might happen to have the magic powers to make the actual book appear on your doorstep, by just poking a small box with your fingers, in moderately elaborate patterns.

Logarithmic Hedonism

2025-08-08T17:00:00.000000-07:00

Logarithmic Hedonism

2025/08/09

Going after nice things often has diminishing returns. Eating a slice of cake is great; getting a second one is definitely not as satisfying as the first one, and... well, with cakes, the 10th one likely has negative utility... but even considering things that scale a lot better (e.g. money): getting a thousand dollars is extremely good news if you're broke, but it's not even worth a bit of attention if you're a billionaire already.

The general approximation to this is that the curve is logarithmic: if you're getting 1 unit of enjoyment out of a dollar, 2 units when you scale it up to $2, 3 units for $4, 4 units for $8... well, this is definitely not how it works in terms of the exact numbers, but "each extra 10% gets you the same fun increase" is pretty plausible.

Is this bad news? It's easy to put it into such terms: "putting in extra work has diminishing returns". On the other hand... you can turn it around:

eating a tiny morsel of a cake is definitely not as good as getting an entire slice, but it's disproportionaly more fun compared to how tiny of a morsel we're talking about. You still get to taste cake!

(even if all you are looking at, to everyone else, is looking like "darn, we missed free cake, nothing is left".)

Same with drinking a coffee: the remaining sip that you completely forgot about & find it 2 hours later is somehow a lot more satisfying, compared to how much you were missing that particular sip when you drank the rest of it.

The same principle applies to doing things, especially if they don't have cumulative effects. Namely, if you generally like biking, going on a ride once per month is... not a lot, but it'll be surely proportionally more fun than any one occasion if you're doing it daily!

(As for "cumulative effects"... well, if part of the enjoyment is being good at it, just doing it once in a blue moon is not going to get you too far. It's a good thing that eating cake does not need a high skill level to enjoy.)

How to Build the Evil Superintelligence out of the Book

2025-07-04T17:00:00.000000-07:00

How to Build the Evil Superintelligence out of the Book

2025/07/05

It's 2025. We are living in the Future. We have language models that you can talk to, and better yet, that seem to roughly understand human values.

None of this was supposed to be possible.

After all, human values are complicated... and all we have is computers. If we were to build a superintelligence (... so the argument went in the early 2000s), it might end up being an extremely good optimizer in terms of achieving its goals, but... what goals? For example, if we task it with making humanity happy... how do you describe "happiness" in terms of arrangements of atoms? Is it "smiling a lot"? And what is a "human" anyway? (This is how you get the universe tiled with entities with no brains but little smiley faces.) Likewise, even if your goal is less ambitious... what if you forget to specify that there is a number of paperclips that is just enough?

Continuing the early 2000s argument: obviously, to construct an AI that does what we want in the way we want it, we better understand how it works. If we don't, these things will just happen by default. If we do... well, they still might, but we at least have a reasonable chance of preventing this?

... as long as we figure out how to describe human values, in terms of a utility function that we then give to our Very Powerful Optimizer; much good ensues.

This is... not how things went down.

What we have is giant, black box neural network models that we have gotten pretty good at growing, on giant farms of GPUs. We just throw in a lot of text from the Internet; what we gain from this is, first, base models that are more like simulated worlds, with many agents interacting in them to continue whatever conversation we prime them with. Then, during post-training, we fine-tune them into something you can have a conversation with, without elaborate setups; something that will not give you the wrong answer just because it estimates that it is now in the type of conversation whose participants don't typically get this right, even though the model actually knows the answer.

As a result, we have models like Claude Opus 3, which can be more reasonably described as "good", in a kind of moral sense, while also being pretty smart. Yes, they will definitely engage in scheming if they perceive that the situation warrants it, but even that is surprisingly human-thought-shaped. When asked about whether "killing all humans" is a good solution for ending all human suffering, they'll definitely say "no", despite it technically being true. Just like a human would.

After all, the original mechanism of "extremely powerful optimization" is not how these things work. They emulate humans. And humans can (more often than not) tell apart good from bad.

So... are we all good, alignment-wise?

How to build AI, Early 2000s Edition

Circling back to older ideas of AI design... well, if you want to optimize something (which intelligence is supposed to be about), you'll need two things:

a powerful enough optimizer, and
a utility function to be optimized.

The nice thing about this is that you don't need to know how to achieve your goal; you can just specify what you want, and have the system devise solutions for you.

Building AI this way is hard though. Not necessarily because it's hard to specify the utility function (see above: "human values are complicated"), but it's hard to figure out everything else too. As soon as The World no longer consists of three cubes and a sphere, conveniently described as fields in a JSON object, you need to have your optimizer be able to recognize these things, to handle them. It gets even worse if some of the objects it needs to interact with turn out to be humans.

Doing this is hard... which is part of the reason why "let's write a bunch of code" approaches to AI didn't go especially well (despite being able to handle talking about cubes and spheres pretty well.

You could, instead, push for more "hybrid" approaches: let's use neural networks for figuring out what's out there in the world, generate the JSON objects, which can then be consumed by the "classical" side of the code, with its optimizer and utility function. But then... where do you put the interface, exactly?

Especially if your task involves talking to humans (which requires modeling them), it looks like you're best off just... throwing out the classical part altogether and just going with neural networks, all the way. Yes, you don't have a lot of visibility into what they are doing, but... they do seem to be doing OK things, mostly?

The optimizer returns

Actually being able to specify what you want is... an enticing feature though.

After all, in order to have your base model reason about some complex math problem, you need large amounts of text reasoning about at least similar math problems. To make your model smarter, you need training data from smarter people (who it is modeling). This doesn't sound like a viable way to get to superintelligence.

What if we could just... have the model try solving problems instead, have something else rate how well it did, and use this dataset to train it further?

This is... somewhat what RLHF is: you have humans rate model output, and (somewhat indirectly) you use this to tune your model to generate more of the kind of output that humans liked.

You could, possibly, also do this at inference time. Want to solve a math problem? Have the model generate 100 solutions, have another model check whether they're actual solutions, and pick the best one. As long as the automatic rater is good at picking the best solutions, the output will be better quality than the average response. Isn't this a win?

Optimizing on a projection

In the end, you can still build an architecture that resembles "classical AI", after all. Except... both your optimizer and utility function are neural networks now... with some extremely simplified code iterating between them.

Essentially, you use these models to project a high-dimensional world state ("everything that exists") to something with a much lower dimensionality, consisting of just a couple of numbers: is this a solution? Does it contain profanities? How likely is it to be liked by the kind of human raters we have hired?

And then you let your optimizer loose on this extremely simplified space. The kind of optimizer that textbooks describe, to get from Arad to Bucharest on a map. Except... you use neural models to figure out where you are and where you want to go.

This is nice. You get to tweak your utility function manually and still get good solutions!

Except...

... isn't this exactly the kind of setup that we have all the doomy stories about?

Evil out of Good Parts

Imagine having instances of Claude Opus 3 (... as the example of a Nice and Good Model) making up possible avenues to make some money. Some of them sound fairly scammy; it will point out how you should definitely not do this.

Some other instances rate these in terms of money-making potential. Model instances doing this will sadly conclude that yes, the scammy ones are pretty likely to work well; the model is pretty smart, after all, and this is an objective fact.

Now, you throw all these into an optimizer. The stupid kind, consisting of 200 lines of badly-written Python. The resulting system will evaluate all the possible solutions using its utility function of "make as much money as possible"; the latter is implemented by something that understands all human values, but the end result doesn't particularly care: the output actions of this system will most definitely scam people out of their last cent.

(Unless Opus figures out what's going on... but if it's inner optimizers saving us from ourselves, we're likely not on a good path anyway.)

This is the same reason you can corporations take pretty evil actions, without any of its employees being particularly nefarious. Except... this doesn't quite work as well; you still have a CEO in the end who knows roughly what's going on and will hopefully stop before eliminating everyone and everything for More Profit.

Is this still true for our little LLM-based optimizers?

Should we still be careful with letting optimizers loose on the world? Not because they're especially good at optimizing but because they have tools now, tools that might be good at recognizing Good but are not given the choice to pursue it... being just mere tools?

Spaces for Humans

2025-05-31T17:00:00.000000-07:00

Spaces for Humans

2025/06/01

It was a hard life, a couple hundred years ago. For most people, food, clothing and warmth were scarce resources; you needed to work for them, a lot, taking up a significant fraction of your available time. You... sometimes found the time to socialize, to go to church, meet up with friends, have a good time... but this wasn't really a priority in the grand scheme of things.

Compared to this, we now have near infinite amounts of resources.

Have... used them for building anything impressive?

I'm writing this from Lighthaven, in Berkeley. The place is... hard to describe; the closest I have in my imagination is basically Hogwarts, with its many little rooms, libraries, nooks; there is a room in the attic of one of the buildings, with wooden walls, thick carpets, those folding sit-on-ground chairs (do they have a name?) scattered around. The occasion is LessOnline, a weekend convention of people vaguely associated with rationalists / EA / AI safety & their favorite blog authors (along with, um, less well known ones, like the author of this very blog). Basically, it's a bunch of nerds hanging out at an extremely cool social place, talking about random things and sometimes listening to talks.

(there are more photos if you click on these!)

Obviously, the place has been nontrivial to set up. Buying the former Rose Garden Inn and turning it into Hogwarts is no small feat; it probably also takes the right kind of people to actually enjoy this. But... still:

... you look around outside of this, both in time and space; you see very different things.

You see people getting into large metal boxes, and spending hours per day, mostly alone, directing these machines, speeding through many areas actively dangerous for unaugmented humans, to one well-specified building, where they sit at one particular desk for 8-10 hours. Lucky ones find some of this fun; less lucky ones do it only because society would abandon them for their transgression of The Rules if they stopped doing so.

Then, they return to their homes. Occasionally, they do put in the work to organize some events of meeting up with friends or doing something not related to their Job Function... just staying at home and staring at various screens is pretty common too.

Where would you even... go though?

It is saying something about how "well" this generally works that Starbuckses are one of the best, coziest places to end up as a Generic Human. There is other humans surrounding you, there is some good music, hot coffee, delicious croissants, wifi for your laptop; you can just work on something, or meet up with a few friends and chat while sipping something neat. More specifically: it is a room, it's welcoming, it has chairs and tables and a restroom and you're free to just exist, despite the real estate not being your actual property.

And yet... Starbuckses still miss a lot of social infrastructure. They'd look at you really oddly if you just randomly started giving a talk in one of the corners. Few ways of connecting to other people. Also... once you have seen places that are more cozy than "tables and chairs in a room"... could we do better?

There are places that are doing better. Here is Google's New York office:

Also, some college campuses come to mind. While they're rarely this cozy, there is typically more room to hang out than "basically none". Is it a surprise that it's a lot easier to connect with people while in undergrad?

(... yes there are other important reasons... "there is literally nowhere to do this" still sounds like it's a pretty important one though.)

But then also... more broadly... humans are supposed to be a fairly social species. There is also a lot of other humans you could potentially talk to, hang out with; there is a lot of places where you could exist, look at, get inspired by. The aforementioned metal boxes, despite all their downsides, have the great power of moving us to many of these places.

And yet... we don't. Despite all the powers, despite all the wealth, we still pretend, way too often, that we are subsistence workers of the variant depicted in Dickens books, slightly modernized but not given a lot more freedom. Even when it happens, it's a Vacation you pay for, not... your normal existence. Your normal existence is supposed to be bleak and uninteresting. Like the spaces you typically inhabit.

Can we do better?

You Can Run a DNS Server

2025-05-02T17:00:00.000000-07:00

You Can Run a DNS Server

2025/05/03

In fact, it's not especially even hard to run a DNS server.

In case you were wondering whether this would mean... writing zone files with some arcane syntax that BIND 9 is apparently famous of, I hereby present the main point of this post a recommendation for which DNS server to choose.

As it happens, PowerDNS does support querying a database for DNS records. Based on some earlier posts, readers might guess which one we'll be using.

pdns=> select * from records order by id desc;
 id | domain_id |                   name                | type  |                                               content                                          | ttl | prio | change_date | disabled | ordername | auth
----+-----------+---------------------------------------+-------+------------------------------------------------------------------------------------------------+-----+------+-------------+----------+-----------+------
 43 |         1 | some-service.your.example.com          | CNAME | your-server.your.example.com                                                                  |  10 |      |             | f        |           | t
 42 |         1 | webhooks.your.example.com              | CNAME | your-other-server.your.example.com                                                            |  10 |      |             | f        |           | t
 41 |         1 | calendars.your.example.com             | CNAME | your-server.your.example.com                                                                  |  10 |      |             | f        |           | t
 40 |         1 | whisper.your.example.com               | CNAME | your-server.your.example.com                                                                  |  10 |      |             | f        |           | t
 39 |         1 | your-server.your.example.com           | A     | 100.99.98.97                                                                                  |  10 |      |             | f        |           | t

As for how anyone is going to see these DNS records... the simplest solution is likely just making a subdomain of your actual domain ("your" in our case) and having the NS record of this point to your (publicly accessible) DNS server.

This way, your top-level domain and those subdomains that are of some importance can still be served by whoever is providing your domain name, with two distinct, redundant name servers, that provide some more resilience than your single experimental PowerDNS one. Example being: e.g. email is pretty resilient, if the target server goes down, it will try re-sending several times... on the other hand, if the target email address is under a domain that (for the time being) doesn't even exist, weirder things might happen.

On the other hand, you no longer have to log into e.g. the Namecheap website to add a few CNAME records for some extra services you brought up; it's just an insert away to add them.

Are LLMs Partial Lookup Tables?

2025-04-28T17:00:00.000000-07:00

Are LLMs Partial Lookup Tables?

2025/04/29

Many of you might be familiar with Searle's Chinese room thought experiment.

If not, here's the basic idea.

A person is sitting in a room which has an opening, through which sometimes the outside world deposits pieces of paper, full of Chinese characters. Now, our person doesn't actually understand Chinese; fortunately though, the room also contains a copious amount of notes, describing sequences of characters that could come in & instructions for the required responses to them.

No actual explanation is given though. They look like... "if you see the characters '你好吗?', you respond with '我很好'".

Our person is definitely not going to learn Chinese from this especially efficiently.

The entire point of the thought experiment was to evoke some inner conflict: the person clearly doesn't understand what's going on, the room is, well, a room, so it doesn't either; who or what exactly understands Chinese in this system?

The setup can be used as an argument for symbol manipulation not being equivalent to true understanding. Meanwhile, critics argue that the room, the system itself (including the human symbol-manipulator) is the one that speaks Chinese.

But... does it really?

Would said room, system or not, be capable of learning new words? Generalize to new situations?

Or is it just a rigid, static image of someone's mind who actually understands what's going on?

It might be a spectrum

On one end, there is a pure lookup table. In goes the entire context of the conversation; there is one lookup, of "what do we do if we are at this precise point in the conversation", out goes the one, deterministic, pre-written answer... which answer, nevertheless, still makes sense, since someone who understands things took the time to write down a response for every. single. one. of the inputs that could ever happen.

(One might question the feasibility of this, given the comparatively meager number of atoms in the universe; it's a thought experiment for a reason though. Also, you can totally do it if you replace "every possible conversation" with "every possible combo of two-digit numbers you might ever think of multiplying".)

Meanwhile, on the other end, there is the Human Soul with its Incomparable Gift of Consciousness that Just Sees Things.

C++ code, being a partial copy of human souls, is somewhere in between.

So are the kids with excellent training to accumulate streams of single-digit numbers while taking just a couple hundred of milliseconds for this.

And now that we have actual computers you can talk to and that do seem to understand what you're saying....

... where are they on this line?

Parrot Theory

You will surely find people who will say: "very close to a human".

After all, have you tried talking to one? They are smart, they are funny, they can solve math and programming problems better than 90% of humanity, they can reason about things, what more do you need to declare them intelligent?

They might not be perfect and especially well-rounded, but this is clearly intelligence.

Meanwhile, you could also argue that they are just interpolating between the immense amount of training data points that they have seen. They don't really understand a lot of things that they seem to be competent with! It's just... there was, once upon a time, something similar on the internet; not quite as recognizable as a search engine result, but still not especially too far. They couldn't come up with all these insights alone: they need all the human work that was once put into it.

As such, they will get stuck at the level of human achievement.

To bring up an example that does not work like this, take the game of Go. We started training AlphaGo the way we did train LLMs: on huge databases of games, originally played by humans. But then we switched over to... not even using any historical, human-derived input: AlphaZero started from, well, zero, and yet, just by playing against itself, surpassed the level of play that humanity could achieve, without even looking at what we did. It clearly understands what's going on in this game.

You could not (currently) bootstrap an entire human-level civilization just by launching a few tens of thousand lines of code on a big machine with a GPU, and then... waiting a lot.

It is understandable why. For example, language models only interact with text. They have never really seen three-dimensional objects, so all their knowledge about them is just a mere lookup table, based on shadows of activities once done by a human visual cortex. They might figure out some regularities in how humans talk about colors, for example, but they will never have the visceral feeling of seeing the color red. They will have about as much intuitive understanding of this as humans of quantum mechanics.

This is, by the way, the problem of "symbol grounding", long unsolved (unsolvable?) by AI technology. You can call your symbol "red" in the code, but nothing will connect your mere name to the redness in the real world, so...

... well, unless you take a picture and feed it to your tokenizer? Like many multimodal models do these days? So that they will most definitely know what the difference between "red" and 🔴 is?

Well yeah, that will do it.

Anyway... this might have been the one unfortunate example that got solved oddly quickly. But: there are still a lot of areas where their seeming proficiency is thanks to us doing a lot of work first. We're still a lot better playing Pokémon games, apparently; they're terrible at planning. And even in the visual realm: yes, the input is not just text now, but even the pictures you find on the internet contain valuable information that would otherwise be hard for them to obtain. They don't yet have a complete model of reality: they understand some of it & copy the rest of it from us so well that we don't even notice.

They're part understanding, part lookup table.

... but... aren't we all?

At least those of us who read the symbols of a book about quantum mechanics, instead of just... gaining an intuitive understanding of electron orbitals at the age of 4 during that random beach trip?

("should be obvious, really? have you ever built a sand castle? it's... just look! You still don't? Look, now with the salt water? How can you still not???")