Lisp as a System Service

2022/01/17

... it keeps running & you can connect to it remotely!

This is post no. 68 for Kev Quirk's #100DaysToOffload challenge. The point is to write many things, not to write good ones. Please adjust quality expectations accordingly :)

Common Lisp (along with other Lisps) has a really neat development paradigm: you start up a Lisp image once and you keep incrementally updating it, adding code function by function.

This is, by the way, really powerful. You never have to restart anything. Or recompile anything. Things are just... open.

(The appropriate comparison is... a UNIX system where you can just go and edit a shell script / recompile a binary / modify a config file, vs. an embedded system where you need to wait 3 hours & reflash every time you change anything.)

In practice, to do this, you start Emacs, you run Slime, which launches an actual Lisp (e.g. SBCL) as a subprocess, and connects to it via a TCP port.

But then... this is where our neat analogy breaks down. You can SSH into a remote UNIX system and modify things! Would you able to SSH into... a running Lisp image?

Remote Lisp

... well, actually, yes, you can.

Namely, you can just call

(swank:create-server :port 4005  :dont-close t :coding-system "utf-8-unix")

from your SBCL process, which you can start independently from Emacs. Then, in Emacs, you can use slime-connect to... well, um, connect to a local TCP port, that you port-forwarded via SSH from the remote system.

The documentation also states:

there is a way to connect without an ssh tunnel, but it has the side-effect of giving the entire world access to your lisp image, so we’re not going to talk about it

... and then proceeds with not talking about it.

Actually, we're not going to talk about it either, since... even without this, the entire process is fairly hacky already. You:

launch SBCL in a terminal ssh'd into a remote system. You type stuff into it.
you then ssh yet another time, with a port forward
... and only then can you connect

not to even mention how exposing a Slime TCP port even on localhost is already a security hole, allowing anyone on the system to do stuff in your name. (With a port forward, that's both the local and the remote system.)

(... did I mention already that I think TCP ports are handled in a stupid way on UNIX systems? with no access rights whatsoever? Previously, previously.)

Also, if your client system disconnects, your SBCL process is now dead, which... kinda defeats part of the point. (Imagine your UNIX server dying if you drop the SSH connection.)

Solutions

As it happens, UNIX systems have way better sockets than localhost-only TCP ones: UNIX domain sockets!

Too bad Slime doesn't support them.

Or... it didn't. I ended up submitting a pull request that adds UNIX domain socket support, both on the server and the client side. So there is that.

(Thusly, disclaimer: none of the following will work with stock Slime. yet. It's surprisingly easy to compile for yourself though; it's basically just a git clone, followed with inserting the patched version into the search path.)

Then:

(swank:create-server-unix "/home/your_user_name/sbcl-local.socket" :dont-close t)

This creates a server that you can connect to, multiple times. It's also not accessible to anyone else (... OK I didn't actually check default permissions there, but it really shouldn't be hard to make it so.)

Then, you can port-forward with:

~$ ssh -L /home/your-user-name/sbcl.socket:/home/your-user-name/sbcl.socket your-server.net

and connect with slime-connect-unix in Emacs (... also new in the patch). No more security holes!

... except it still dies once your SBCL SSH session does.

The "Lisp Server" part

You can actually run your SBCL process as a system service.

I'm using s6 init on my main server, so doing it involves adding a service run file (e.g. at /service/sbcl/run) containing something along the lines of

#!/usr/local/bin/execlineb -P

/usr/local/bin/s6-setuidgid simon  # run as me
/usr/local/bin/fdmove -c 2 1 # redirect stderr to stdout
/usr/bin/sbcl --load /home/simon/local/slime/start-swank-unix.lisp

start-swank-unix.lisp is just starting the UNIX domain socket server:

(load (merge-pathnames "swank-loader.lisp" *load-truename*))

(swank-loader:init
 :delete nil         ; delete any existing SWANK packages
 :reload nil         ; reload SWANK, even if the SWANK package already exists
 :load-contribs nil) ; load all contribs

(trace swank::setup-server)

(swank:create-server-unix "/home/simon/sbcl-local.socket" :dont-close t)

(it's a direct descendant of start-swank.lisp, included with Slime.)

You can even add a log file with it, if you add a /service/sbcl/log/run:

#!/usr/local/bin/execlineb -P

/usr/local/bin/s6-log T /var/log/sbcl

(s6 is neat, you should check it out. I'm kinda hoping that That Init System that I Shall Not Name can do this too somehow.)

This way, you have a Lisp image as a system service you can connect to! Unlike earlier "I started it in a SSH window" hacks, you can launch a web server into this, and generally assume that it will stay up for a while.

Admittedly, you still need to use SSH to forward sockets, but it's being done in a significantly less hacky way, too.

... comments welcome, either in email or on the (eventual) Mastodon post on Fosstodon.