The Unreasonable Effectiveness of Thinking in an Empty Room

You can't really learn playing tennis from a book.

You try hitting the ball. You fail. You try hitting it again, slightly differently. Maybe you hit it once, and you learn from it. You repeat process, a lot. Eventually, you get pretty good at it.

Not even really good tennis players can write books though that explain every detail of how to make your arms move. You still need to be exposed to the actual real world with tennis balls and gravity and air resistance and actual opponents in it.

Of course, in the grand scale of things, you do learn pretty quickly. This is a consequence of a larger scale process, evolution, trying to and succeeding at figuring out how to make beings that can adapt to a very wide range of circumstances. The outcome includes not only opposable thumbs, but also a brain structure that can pick up new kinds of movements just by trying it a couple of times.

For both of these cases, the pattern is the same. Have something (humans!) interact with their environment, get feedback from the environment, and adjust some parameters (in our case, either DNA or neuronal connections), to make the outcome better.

While a lot of physical activities are like this, a wide range of intellectual ones would also count. Writing nice code, for example, is something that many books have been written about. And yet, it is hard to get good at it without experiencing first hand working on a larger system and dealing with the barely intelligible, messy horrors that other people, including yourself two weeks ago, have written.

Not being transferable is not the same thing though as only being learnable from the outside world. For example, hundreds of years ago, there might have been books and explanations about which plants are edible and which plants are not. People might even have been able to generalize somewhat. (For example, plants with a certain shape of leaf might be suspected being really bad for you, even if you haven't seen this particular kind.) On the other hand, you can't really make up a thought experiment to decide whether a brand new kind of plant is dangerous or not.

But we are capable of doing something even more impressive than all this.

Imagine solving a math problem. You're sitting in a corner with a pencil and notebook in hand. You sometimes scribble things on the paper; you're otherwise quiet.

The way this works is not an external feedback loop, in which

• you write out a possible solution
• you walk up to the teacher & have them grade it
• go back & immediately start writing up another solution that's maybe mildly better than the previous one, even if it's still wrong

(Some people do actually do this. They are not the ones who are especially good at math though.)

Instead, you're pretty sure that if you think you solved the problem, you solved the problem. You can double-check it in your head, using your knowledge. No external sources of feedback needed.

Better yet, the next time you solve a math problem similar to this, you will have experience gained from solving this one. This will allow you to be better, faster, and more effective at solving these new ones.

It's not just math. Engineers designing bridges will generally not build the very first one that they come up with, to see what happens. They keep reading books, solving practice problems, trying out and evaluating various solutions on paper. By the time they actually build a real bridge, it will be a pretty good one.

Of course, it's not an entirely pure distinction between learning from external and internal feedback. You need some of the former to have accurate enough models of the world to further iterate and improve on. (This is why a lot of medieval science didn't work. Too much thinking about principles and not enough looking at actual things to check whether they actually represent something or not.) Meanwhile, we are likely using a lot of internal feedback, even if the entire processing question looks like learning with a closed loop; this might be the reason why LLMs, which don't do this, need a whole ton of training data, way more than humans.

As an extreme example for what happens if this works well, take AlphaZero, which has beaten humanity's collective knowledge of Go, just by playing against itself, not using training data from humans at all. (Apart from, of course, the handwritten model of the game board & rules... which is still a pretty important component.)

Overall though, it's somewhat surprising that this works so well. After all, building a system that can do well in an environment is very different from having a system that is capable of modeling this well enough so that you can train on it. And indeed, there are a lot of fields where the recommendation is "just go practice and use your intuition".

But whenever you can do this, it's extremely powerful and lets you iterate a lot faster. (This is a key component of how we went from wooden sticks to computers.)