Fixing Samba Again

Apparently, file sharing between Linux and Windows is a thing that should be working but somehow is especially prone to breaking. Given how there is Kerberos involved, this is not especially surprising (yes I have debugged this before). I did decide to write a post about it though, just in case someone needs this particular solution... but also to demonstrate how you can dig into supposedly hard to fix things.

The setup

Did you know you can have Windows clients use Kerberos & log into Samba shares and SSH servers, even if you're not running a full Active Directory domain controller?

This is exactly what I'm doing. On my Linux server, there is a Samba server sharing the main hard drive with everyone, protected by Kerberos auth. You log into Windows with a user like EXAMPLE.COM\simon, and you magically get access to the shares without adding more passwords. Until... you don't.

How it started

Well, it was an upgrade of said file server from Chimaera to Daedalus (... these are Devuan versions roughly corresponding to Debian Bullseye & Bookworm, except without systemd. We... don't particularly like systemd.) Everything went remarkably smoothly... until the point when I tried to connect to it using my Windows machine.

I don't even particularly remember what the actual error message was... Windows is not especially good at providing useful ones. So... out of the toolkit comes tcpdump! For simplicity, let's ssh into the server & see what it sends to us. (...11.22 is the Windows box; it's all Tailscale by the way.)

sudo tcpdump -i tailscale0 host 100.82.11.22 and not port 22 -w smb_log.bin

We try opening the network share while this is running, then look at the results in wireshark:

Well. This... is at least something you can search for. It, however, is not something that supplies usable results immediately.

Logs

Let's look at Samba logs! (It, after all, was the one that decided that issued this message.) Conveniently, I already had verbosity dialed up all the way since the previous time I was trying to do something like this.

[2024/02/19 20:34:02.699432,  4] ../../source3/smbd/sec_ctx.c:206(push_sec_ctx)
  push_sec_ctx(0, 0) : sec_ctx_stack_ndx = 1
[2024/02/19 20:34:02.699453,  4] ../../source3/smbd/uid.c:566(push_conn_ctx)
  push_conn_ctx(0) : conn_ctx_stack_ndx = 0
[2024/02/19 20:34:02.699471,  4] ../../source3/smbd/sec_ctx.c:317(set_sec_ctx_internal)
  setting sec ctx (0, 0) - sec_ctx_stack_ndx = 1
[2024/02/19 20:34:02.699489,  5] ../../libcli/security/security_token.c:47(security_token_debug)
  Security token: (NULL)
[2024/02/19 20:34:02.699506,  5] ../../source3/auth/token_util.c:873(debug_unix_user_token)
  UNIX token of user 0
  Primary group is 0 and contains 0 supplementary groups
[2024/02/19 20:34:02.699590,  5] ../../auth/gensec/gensec_start.c:844(gensec_start_mech)
  Starting GENSEC submechanism gse_krb5
[2024/02/19 20:34:02.700590,  4] ../../source3/smbd/sec_ctx.c:443(pop_sec_ctx)
  pop_sec_ctx (0, 0) - sec_ctx_stack_ndx = 0
[2024/02/19 20:34:02.700648,  4] ../../source3/smbd/sec_ctx.c:206(push_sec_ctx)
  push_sec_ctx(0, 0) : sec_ctx_stack_ndx = 1
[2024/02/19 20:34:02.700669,  4] ../../source3/smbd/uid.c:566(push_conn_ctx)
  push_conn_ctx(0) : conn_ctx_stack_ndx = 0
[2024/02/19 20:34:02.700687,  4] ../../source3/smbd/sec_ctx.c:317(set_sec_ctx_internal)
  setting sec ctx (0, 0) - sec_ctx_stack_ndx = 1
[2024/02/19 20:34:02.700705,  5] ../../libcli/security/security_token.c:47(security_token_debug)
  Security token: (NULL)
[2024/02/19 20:34:02.700723,  5] ../../source3/auth/token_util.c:873(debug_unix_user_token)
  UNIX token of user 0
  Primary group is 0 and contains 0 supplementary groups
[2024/02/19 20:34:02.700762,  4] ../../source3/smbd/sec_ctx.c:443(pop_sec_ctx)
  pop_sec_ctx (0, 0) - sec_ctx_stack_ndx = 0
[2024/02/19 20:34:02.700810,  1] ../../source3/auth/auth_generic.c:211(auth3_generate_session_info_pac)
  auth3_generate_session_info_pac: Unexpected PAC for [simon@LTN.SIMONSAFAR.COM] in standalone mode - NT_STATUS_BAD_TOKEN_TYPE
          

Well OK, we could have also gotten BAD_TOKEN_TYPE from here instead. On the other hand... "Unexpected PAC in standalone mode"?? Isn't the client exactly the same as before? And what the hell is a PAC anyway?

OK so time to... actually look at the sources! Just check out Samba's Git repo, and look for the error. Or... since we have seen the logs anyway, we can just look up the file on GitHub directly:

static NTSTATUS auth3_generate_session_info_pac(struct auth4_context *auth_ctx,
						TALLOC_CTX *mem_ctx,
						struct smb_krb5_context *smb_krb5_context,
						DATA_BLOB *pac_blob,
						const char *princ_name,
						const struct tsocket_address *remote_address,
						uint32_t session_info_flags,
            struct auth_session_info **session_info)
// ...
        if (server_role != ROLE_STANDALONE) {
// ...

        }
    /* This is the standalone legacy code path */

  	if (pac_blob != NULL) {
		/*
		 * In standalone mode we don't expect a PAC!
		 * we only support MIT realms
		 */
		status = NT_STATUS_BAD_TOKEN_TYPE;
		DBG_WARNING("Unexpected PAC for [%s] in standalone mode - %s\n",
			    princ_name, nt_errstr(status));
		if (!NT_STATUS_IS_OK(status)) {
			goto done;
		}
	}
          

We are standalone mode (not a server in a domain, despite all the Kerberos), so... um... why?

Nevertheless... it does look like it's a Kerberos thing.

Kerberos

So how does this work anyway?

Microsoft's Active Directory is... (at least as of 2008) basically a combo of Kerberos and LDAP, with some extensions. LDAP is a directory service, storing info about users, computers, services, etc; it is being used to decide e.g. whether user X is supposed to connect to service Y. Meanwhile, Kerberos is used for authentication.

The Kerberos server (the "Key Distribution Center") is the one that authenticates users (e.g. by a password), and then gives them tickets. Said tickets are binary blobs that clients can then hand to services (e.g. our file server) to prove who they are, without handing out passwords to them directly. The same works with SSH, too; instead of giving the SSH server a password, clients can give them their ticket, basically waving around a document saying: "whoever is carrying this is simon@EXAMPLE.COM, for the purposes of connecting to ssh-server.example.com -- sincerely, the KDC". There is some fancy crypto included too so that this is secure. (More info here if anyone is interested.)

On PACs

So... Samba is in the process of examining this very blob. So... what is a PAC? As per ChatGPT:

A PAC, or Privileged Attribute Certificate, in the context of Kerberos, is a Microsoft extension to the Kerberos protocol. It is used in Windows environments to carry authorization information, including user group memberships and user rights.

Well but then can MIT Kerberos (the server we're running) give us one?

ChatGPT: MIT Kerberos itself does not inherently generate or include Privileged Attribute Certificates (PACs) as part of its ticket-granting service because PACs are a Microsoft extension to the Kerberos protocol, primarily used within Active Directory environments.

Makes sense. Kerberos is mostly just authentication; it doesn't really inherently know who has access to what; that's a Microsoft extension. Except... given the lack of other options... maybe the story is worth double-checking? In the docs of the MIT Kerberos server?

Well... um...

Wouldn't be drastically surprised if this was our upgrade there.

simon@the-file-server ~/l/samba (master) [100]> apt info krb5-kdc
Package: krb5-kdc
Version: 1.20.1-2+deb12u1
Priority: optional
Section: net
          

(... "more than likely"?)

So all that we need to do is adding

disable_pac = true
          

to our domain config in /etc/krb5kdc/kdc.conf... and then log out & in again (... since we do want to get a new ticket that doesn't have a PAC). Surprise: everything is working again!!!