Simon Safar

Fixing Samba Again

2024-02-18T16:00:00.000000-08:00

Fixing Samba Again

2024/02/19

Apparently, file sharing between Linux and Windows is a thing that should be working but somehow is especially prone to breaking. Given how there is Kerberos involved, this is not especially surprising (yes I have debugged this before). I did decide to write a post about it though, just in case someone needs this particular solution... but also to demonstrate how you can dig into supposedly hard to fix things.

How it started

Well, it was an upgrade of my file server from Chimaera to Dadelaus (... these are Devuan versions roughly corresponding to Debian Bullseye & Bookworm, except without systemd. We... don't particularly like systemd.) Everything went remarkably smoothly... until the point when I tried to connect to it using my Windows machine.

I don't even particularly remember what the actual error message was... Windows is not especially good at providing useful ones. So... out of the toolkit comes tcpdump!

Python REPLs anywhere

2024-02-14T16:00:00.000000-08:00

Python REPLs anywhere

2024/02/15

(Status: might need more testing; this might have been written overly too quickly.)

Programming languages that have an interactive prompt (REPL: a "read-eval-print loop") are nice because you can try out your code interactively, without having to stop & recompile everything; you can just type in your functions one by one and see whether they work.

Except... no one really wants to type in functions one by one. This is why Jupyter notebooks exist: they let you push this story a little bit further; you can at least edit your functions in cells, so no re-typing / up-arrow-pressing needed. But then, once you have a decent amount of code, you'd want to write an Actual Python Program. And how do you test those with REPLs?

PDB

Well, round one: PDB the debugger exists. You just

import pdb

at the beginning of the code, and then you can drop in a breakpoint with

pdb.set_trace()

It's fairly easy to do & it's more convenient to look at program state with it vs. a lot of print statements.

A less known feature of PDB is that you can get a full interactive shell prompt with the "i" command. While the (pdb) one is also OK for printing things, this one is less picky about what it evaluates. Thus... you now have a full REPL at the spot in your code where you actually need one, not just outside.

code.interact()

In case you want to drop into an interactive shell right away... you can use the actual module Python uses for interactive shells!

Try

import code

# ... and then in your code

code.interact()

This will give you the nice interactive prompt... except it's literally just that, without any local variables you have. Pretty useless, but it's a good starting point.

You can improve it by

code.interact(local=locals())

which gives you local variables... not the symbols in your module though. To fix that, you can just also add them:

code.interact(local={**locals(), **sys.modules[__name__].__dict__})

This way you can test out functions too.

... comments welcome, either in email or on the (eventual) Mastodon post on Fosstodon.

Logging Into Things

2024-01-29T16:00:00.000000-08:00

Logging Into Things

2024/01/30

(... in which we're returning to the practice of plentiful but not overly high quality posts.)

There is that alternate universe where everyone knows what public key cryptography is. Where you can identify yourself without remembering 23 different passwords that you need to type into various custom text boxes on websites. Where phone numbers aren't a reasonable way of identifying people.

(please don't insist on finding deep meaning in this image.)

Then, there is our current world, where... well, if you want to prove you're a unique user, you're using one of 3-4 big authentication providers... or you go back to the telcos, and have them demonstrate that you have a phone number. It... kind of makes sense, from a spam filtering perspective, but it's still not too great.

Wouldn't it be cool if we had Actual Infrastructure that handed out physical devices to people, which they could then use to identify themselves?

... actually, could it turn out that we do have such infrastructure already?

SIM cards exist.

If there is a sort of opposite to a cypherpunk utopia with holding your own keys, it's probably looking a lot like telcos. There is an impressive cultural difference between them and freedom-minded internet people: in telco-land, things are directed by central authorities, the fancy application logic is in the network itself (vs. the smart endpoint model of the internet), and of course, you're going to be charged for this by the minute. However, despite them resisting the idea of becoming dumb pipes, this eventually happened, so we're now in the world of...

... using either Google or Facebook as authentication providers.

Who then require you to have a phone number.

But then there is something else here. If you think back of the concept of having a physical device, carefully handed out to everyone, to prove their identity in a cryptographic way...

... this is actually what SIM cards are.

Of course, in a true telco fashion, we can't actually use them for crypto; we have to go through the circuitous route of phone numbers & text messages. I do find it remarkably ironic though, that...

... phones do VoIP?

(disclaimer: don't take me up too much on the technical details)

Once upon a time, mobile phones did voice as an Actual Thing. If you wanted to call someone, you'd get a digital channel, end to end, allocated by the network, which you could push bits through at a given bitrate. You add an audio codec and you gloss over some details; you get a voice call!

Eventually, people wanted mobile internet. The kind where you aren't charged by the minute. So... with 3G, we had circuit switched voice calls still (with pre-allocated pipes), plus packet networking. Very different. See, that's why we, the telco, charge you by the minute for voice calls!

By the time we got to 4G / LTE though, things have changed. The amount of data traffic exceeded voice by a lot & got up to pretty good quality / latency levels too; why even add dedicated infrastructure for voice calls? Especially if... we can switch back to 3G whenever someone calls you. (... this was seriously happening at some point.) Or... since 3G networks are being phased out already too... here is the idea: just do voice calls over the LTE packet network!

(Like everyone else started doing on the internet somewhat earlier.)

This is VoLTE, a.k.a. Voice over LTE. It sounds like Advanced Telco Magic (of which there is still a decent amount; it's not like we can reuse open standards in a simple way that doesn't give us opportunity to add our own patents). On the other hand: what's underneath is really just SIP calls, behind a VPN.

(Mostly. Yet again glossing over some details.)

Given how all this works, the existence of "wifi calling" is not even that surprising. The basic scheme is the same: you have a packet network connection already, through which you run a VPN (IPSec actually) connection. The trick is that you use the SIM card as the root of the crypto that lets you log into the VPN! From that on, the network already knows who you are; the rest is just plain VoIP calls.

The point is... we do have a neat, distributed authentication mechanism that almost everyone has an instance of, distributed between multiple companies, which can (could!) be used to log into things & sign stuff. It's doable.

Why are we then still relying on things like... credit card numbers?

Bookmarks

2024-01-21T16:00:00.000000-08:00

Bookmarks

2024/01/22

Computer science has some problems that have been solved already in better ways... and despite this, we keep solving them in ways that are worse than those. Mostly because the better solutions would involve controlling code / infrastructure / etc. that inventors of worse solutions do not have control over.

Browser bookmarks are one of these things.

The typical bookmark UI involves a tree of bookmarks that you can open. You can also search for them from the browser title bar. Optimally, they can be organized in a hierarchy. They... really don't do a lot. Better browsers do have a reasonable way of managing them, too.

They should be files.

As in... yes, we could reimplement a hierarchical file browser, but... on at least some operating systems... we already have something that closely resembles a hierarchical file browser.

(... it's the hierarchical file browser.)

As it turns out, most operating systems even have an "internet shortcut" file type, which, when double-clicked, will open the page it points to. They are really easy to create, too: you just grab the little icon left of the URL bar & drop it into a file browser / desktop / etc.

(The above directory was generated by just dropping some of the bookmarks into a random directory.)

Isn't this great?

Nevertheless, you might also have noticed that, despite the seeming similarities, the directory of URL files is significantly uglier than the in-browser variant. The UI is also... somewhat less convenient.

So why exactly...?

Here. They are now prettier. With very little extra effort.

As it turns out, you can pick icons for them! It's pretty efficient, too: all they store is a path & an icon ID, so you can either use the system-provided ones (to generate a quaint 90s feel) or supply your own.

Note that there is no such functionality in the Firefox bookmark manager. (Likely because no one thought you'd need such functionality.) The icon you get there is that of the actual website.

Which is... actually likely a better default than a Firefox icon. It's actually so good that no one bothered implementing an icon picker.

This is not a big loss. But then... you could have gotten one for free?

Bookmark sync

Browsers can sync bookmarks. With... themselves. As long as you sign in with a Google / Firefox / etc. account. Before this was a thing, you would (you probably still do?) have extensions explicitly for syncing bookmarks.

But... if they're files...

... just throw them into Dropbox? Or rsync them? Or throw them onto a network drive?

Dynamic bookmarks

Ever wanted a list of your top 10 pending pull requests available from your browser?

URL files are super simple:

[InternetShortcut]
URL=https://simonsafar.com/2024/not_using_terminals/

... even if you add an icon:

[InternetShortcut]
URL=https://simonsafar.com/2024/not_using_terminals/
IconIndex=22
HotKey=0
IDList=
IconFile=C:\Windows\System32\shell32.dll

... so if browsers actually used these instead of their own database, you could super easily generate a list of them.

Automated processing

You press Ctrl-D to bookmark something. It creates a file.

Some background process notices the file & archives the full page for you. Or makes an mp3 out of the video. Or downloads the paper pdf, sends it to your tablet, extracts all references and calls GPT-4 to make some Anki cards out of it for you to review.

The conclusion being: reusing existing abstractions is nice. And although turning everything into files is not necessarily always a good idea, we could possibly turn more things into files before we run out of the good ones.

On Not Using Terminals

2024-01-16T16:00:00.000000-08:00

On Not Using Terminals

2024/01/17

The typical hierarchy of computer users involves two levels: those who use UNIX terminals a lot, and... everyone else, who should but doesn't.

Said hierarchy is present in many minds of users of UNIX / Linux machines; actually, with PowerShell, even Windows users are slowly getting convinced that the True Way of using a computer involves opening one (or many) terminal windows, typing long commands (extra points for pipelines), and staring at the results as they scroll forth. This, due to its simplicity, is clearly superior to any other way in which one could use a computer... and this especially includes clicking on things.

With this, I present today's Mostly Weird Fact:

I don't actually use terminals that much. (Despite working a whole lot with Linux computers.)

In this post (and some of the followups?), I'll show how this might actually be nice, even.

Is this yet another weird Emacs post?

This statement is technically true. For the record, if you do want terminals, Emacs is a mostly excellent replacement for tmux or screen; you can start an instance on your server & use emacsclient to reconnect to your set of terminal buffers.

This is, however, not what I mean. Terminals within Emacs are still terminals, with a prompt, and an infinitely long piece of printer paper above, with all the things your previous commands printed.

There are other ways though.

Just not running commands

I'm wondering how much of UNIX terminal usage is a combination of cd and ls. They are excellent tools; I'm pretty sure there are also neat enhanced versions (e.g. a cd with a stack of past directories).

As it happens, Emacs has a mode for "editing" directories; it's called dired. If you open a directory instead of a file, you get something like this:

It's just like the output of ls. But then it's much better! It's just like another Emacs buffer with an open file; you can do whatever you can in those: search for text, move around, copy parts, etc. It's also much faster than exploring directories with plain ls:

Up a directory: just hit "^"
Open a subdir: hit "Enter" on it
"w" copies the name of the file, even without having to select it (... I'm literally using this right now to copy the image file name into the HTML file)
"C-u 0 w" copies the full path of a file
You can also just edit (as in "type into the middle of") file names (and do search-replace on them) if you hit "C-x C-q" (commit with "C-c C-c")

If you just want to peek into a subdir, you can have it dump the contents into the same buffer:

But... most importantly, for our Quest of Avoiding Terminals: you can run commands on them!

But actually still running commands

You might have thought that we'll go all the way & will avoid running UNIX commands altogether. Well... we're definitely not going that far. Being able to run commands is actually nice; it's just we don't have to do it in a terminal (a.k.a. infinite piece of printer paper) all the time.

As it happens, it's super easy to run commands in Emacs: "M-!" ("M" is Alt, basically) gets you a prompt:
... hit Enter & you get the results:

... also showcasing the fact that it's non-interactive: it was probably trying to ask for a new one, but we closed its input on it.

This is nice if you just want to run something quick, maybe look at the results, and have them disappear once you do anything else. (If you really want to, you can still dig them up from the "*Messages*" buffer.)

You can also run these commands async, either by lauching them as such by "M-&" or just by putting an & at the end of the command. Here is an example, with preparing a new command while a previous round of it is still running:

The difference is that the synchronous ones will block everything & just display their results ephemerally; for the async ones, the output gets written into a buffer, in which you can search, copy, etc, since they are just like any other buffer / open file in Emacs. (... there might be a recurring theme here.)

Finally, you can use dired (the directory editor) to pick files & run commands on them. You can just go to a file and hit "!"... or you can select (mark) multiple (press "m") and hit "!" after, to get an entire list of files:

After typing a command, results are shown just like in the above case.

Halfway conclusions

OK so we can... navigate around & kinda run commands? In a weird way? Without a terminal? And sometimes it's quicker?

Which is nice for some things. Quick & simple commands, like checking the date? Sure. Converting some random images to jpegs while looking at a directory listing? Possibly. But then we have our ssh sessions and long-running compilations and countless other cases, when one-offs are just inconvenient.

So... coming up in the followup post: compilation mode, some things about ssh and some other things about shell scripts!

Voice recognition with Emacs

2024-01-09T16:00:00.000000-08:00

Voice recognition with Emacs

2024/01/10

... because Emacs is the best for quick hacks to integrate with basically anything. Originally from 2023/09/03, edited a bit now because... we shall start posting again!

Enter Whisper

Being able to just talk to computers and have them understand what you're saying is something that I've been thinking about for a long time. There used to be systems that did work somewhat, but they were Windows-only, expensive, and hard to integrate with anything.

At some point though, rather abruptly, OpenAI released Whisper, a basically open-source voice recognition model that happens to be remarkably good. Good to the point of... you stop thinking about how good it is since it never really presents an issue. You can run it locally if you have a reasonably beefy machine, or, alternatively, they also provide an online service API that is dirt cheap if you are not planning on talking to it all day.

On the other hand, the current user interface is not especially great. They don't even have a good command line tool that you could just throw an mp3 file at, let alone something that that you could use interactively to save yourself from a bunch of typing.

Wouldn't it be great though to have a nice text editor we could integrate it into? One that is relatively easy to extend?

The Emacs way

What we would want is a button which, when pressed, would start recording your voice; when you press it again, it could inject it into the current buffer. (In case you were wondering: buffer is Emacs lingo for basically an open file.)

So what is the most hacky, simplistic way you can imagine making this happen? Let's just use using ffmpeg to record some sound in the background when the button is pressed. When it's pressed again, we just interrupt ffmpeg and pick up the file it has produced.


(defun ssafar--launch-voice-ffmpeg ()
  (unless ssafar-openai-token
    (error "No API token available"))
  (let* ((proc (start-process-shell-command
                "voice ffmpeg" "*ffmpeg*"
                "ffmpeg -y -t 60 -f pulse -i default /home/simon/voice_temp.mp3")))
    (setq ssafar-voice-ffmpeg-process proc)
    (message "recording...")
    (set-process-sentinel proc 'ssafar--voice-ffmpeg-sentinel)))

We put up a 60 second upper limit for how long it can run. This is typically enough for dictating a couple of sentences, but prevents it from accidentally recording 3 hours and then sending it out to OpenAI for processing.

The above code launches the process in the background; it also saves the reference to the running process to "ssafar-voice-ffmpeg-process"... which is just a global variable, essentially.

As for what the actual button does...


(defun ssafar-launch-or-commit-voice ()
  (interactive)
  (if (not ssafar-voice-ffmpeg-process)
      (ssafar--launch-voice-ffmpeg)
    (progn
      (interrupt-process ssafar-voice-ffmpeg-process)
      ;; technically, it might not be dead yet, but... eventually.
      )))

We just check whether our global variable for FFmpeg process is set or not. If this hasn't been set yet, we launch a new process. Meanwhile, if it exists already, we interrupt it & clean it up.

Background processes in Emacs can have so-called sentinels, which are functions that get called when anything changes in the processes state, for example it dies, finishes, and so on. Here's ours.


(defun ssafar--voice-ffmpeg-sentinel (proc event)
  ;; reset the variable
  (setq ssafar-voice-ffmpeg-process nil)
  (cond
   ;; interrupt is ok... why 255 is there is a question, but... oh well
   ((memq (process-exit-status proc) '(2 255))
    (message "finished; calling whisper...")
    (ssafar-voice-call-whisper)) ;; calling the actual service if everything went well with ffmpeg
   (t (message "... something went wrong with the process"))))

As for the service call:


(defun ssafar-openai-transcription (audio-path)
  "Send an audio file to OpenAI for transcription.
AUDIO-PATH should be the full path to the audio file."
  (unless ssafar-openai-token
    (error "No API token available"))

  (let ((command (format "curl -s https://api.openai.com/v1/audio/transcriptions \
-H \"Authorization: Bearer %s\" \
-H \"Content-Type: multipart/form-data\" \
-F file=\"@%s\" \
-F model=\"whisper-1\""
                         ssafar-openai-token
                         audio-path)))
    (json-read-from-string
     (let ((default-directory "~")) ;; so that it doesn't try doing stuff with tramp
       (shell-command-to-string command)
       ))))

Given how Whisper wants a multi-part post request, we don't really bother using the built-in Emacs version. Just call out to curl directly.

The actual code that calls this gives it the mp3 file that was recorded by ffmpeg, and calls a callback once done.


(defun ssafar-voice-call-whisper ()
  "Sends the current output file to whisper"
  (setq ssafar-last-voice-output
        (ssafar-openai-transcription
         "/home/simon/voice_temp.mp3"))
  (let ((the-text  (cdr (assoc 'text ssafar-last-voice-output))))
    (if ssafar-current-voice-callback
        (funcall ssafar-current-voice-callback the-text)
      (message the-text))))

This callback was set up when we first launched the enter process and it depends on whether we are in Emacs proper or some other X window. (As it happens, Emacs is also my window manager this time, but it would probably also work without, as long as we get some global hotkeys routed to it.) This way, if we're in an actual Emacs buffer, we inject the text; otherwise, we emulate keypresses that e.g. a browser can catch, so that all this works even if what we're looking at is another, non-Emacs app.


(defun ssafar--voice-callback-dwim (text)
  "Do something reasonable depending on the mode we're in"
  (cond
   ((derived-mode-p 'exwm-mode)
    (start-process-shell-command
     "xdotool" nil nil
     (format "xdotool type --clearmodifiers %s"
             (shell-quote-argument text))))
   (t (insert text)))
  (message "(whisper done)"))

As a final touch, you can make all this happen while you hit a particular key.


(global-set-key (kbd "") 'ssafar-launch-or-commit-voice)

Conclusion

Is this the optimal way of doing this? Probably not. On the other hand, the total amount of code required to do this is really not a lot, and compared to just a plain shell script, this actually gives some feedback to you about which state the process is in. For example, you do get the little "recording..." notification message while it's running; I did find this fairly useful instead of trying to figure out whether I have pressed the button an odd or an even number of times.

There are plenty of opportunities to improve on this: for example, currently they cannot handle additional recorded speech while processing the previous chunk. Also, ffmpeg does have a habit of splitting off the last second of audio when interrupted, apparently. (Might be just my fault of me interrupting it in a stupid way though.)

Nevertheless, as it turns out, having a voice recognition engine tightly integrated into your text editor is immensely useful. For example, this article was written mainly using this tool, making it way faster to finish. (As it turns out, most humans are way better at talking than typing, even if they are pretty good at typing.)

... comments welcome, either in email or on the (eventual) Mastodon post on Fosstodon.

Debian Packages for Everything!

2023-02-24T16:00:00.000000-08:00

Debian Packages for Everything!

02/25/2023

... in which /usr/local is no more. Also featuring large amounts of laziness with minor surprises.

The problem

You clone a git repo from GitHub. (... yes I know we're promoting Microsoft products here; we're not trying to solve All The Problems right now.) You compile it. And then... make install.

It then dumps everything into /usr/local. Which... kinda works... unless you want to remove it, figure out where it came from, install it on other machines, etc.

Ideally, you'd make a Debian package for it. You imagine making a Debian package for it. And... then... you realize that it's quite some work, so you just... don't.

checkinstall

Have you ever heard of checkinstall? It's a tool that does... almost what we'd want: it has you run make install, watches what happens, and packs the results into a Debian package.

It's not a lot harder than doing make install (or whatever command it takes to install) itself.

Of course, you still have to give it a package name, description, etc. on the command line:

~/some_git_repo/ $ checkinstall --pkgname some_git_repo \
      --pkgsource https://github.com/someone/some_git_repo \
      --pkgarch=amd64 \
      make install

This is still some work, so...

The shell script

... let's write a shell script that fetches metadata from GitHub & fills them out!

#!/bin/bash

# Get the URI of the local Git repository
URI=$(git remote get-url origin)

# Extract the owner and repository name from the URI
OWNER=$(echo $URI | cut -d "/" -f 4)
REPO=$(echo $URI | cut -d "/" -f 5 | cut -d "." -f 1)

# Retrieve information about the repository from the GitHub API
API_RESULT=$(curl https://api.github.com/repos/$OWNER/$REPO)

# Extract the name of the repository
NAME=$(echo $API_RESULT | jq -r '.name')

# Extract the description of the repository
echo $API_RESULT | jq -r '.description' >description-pak

# Extract the latest release tag of the repository
TAG=$(curl https://api.github.com/repos/$OWNER/$REPO/releases/latest | jq -r '.tag_name')

echo "Tag is: $TAG"

# Build the checkinstall command
echo checkinstall \
  --pkgname=$NAME \
  --pkgversion=$TAG \
  --pkgrelease=1 \
  --pkggroup=default \
  --pkgsource=$URI \
  --pkgarch=all \
  --maintainer=$OWNER \
  --install=no \
  --showinstall=yes \
  $@

We use the version hash as a "version" so that even if we come across the package later, we'll be able to tell where it came from.

Obviously, it's not a true, well-working Debian package; it doesn't have any dependencies added, it assumes that it works on any architecture, etc. Definitely not publishable right away... but... it's better than just dumping stuff in /usr/local!

... but... a shell script?

Here is a Python script doing the same thing! With some minor fixes (e.g. for missing tags):

#!/usr/bin/env python3

import subprocess
import json
import sys

# Get the URI of the local Git repository
uri = subprocess.check_output(["git", "remote", "get-url", "origin"]).decode("utf-8").strip()

# Extract the owner and repository name from the URI
owner, repo = uri.split("/")[-2:]
repo = repo.split(".")[0]

# Retrieve information about the repository from the GitHub API
api_result = json.loads(subprocess.check_output(["curl", "https://api.github.com/repos/{}/{}".format(owner, repo)]).decode("utf-8"))

# Extract the name of the repository
name = api_result["name"]

# Extract the description of the repository
with open("description-pak", "w") as f:
    f.write(api_result["description"])

# Extract the latest release tag of the repository
try:
    tag = json.loads(subprocess.check_output(["curl", "https://api.github.com/repos/{}/{}/releases/latest".format(owner, repo)]).decode("utf-8"))["tag_name"]
except:
    tag = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode("utf-8").strip()[:10]

print("Tag is: {}".format(tag))

# Build the checkinstall command
print("echo checkinstall \\\n  --pkgname={} \\\n  --pkgversion={} \\\n  --pkgrelease=1 \\\n  --pkggroup=default \\\n  --pkgsource={} \\\n  --pkgarch=all \\\n  --maintainer={} \\\n  --install=no \\\n  --showinstall=yes \\\n  {}".format(name, tag, uri, owner, " ".join(sys.argv[1:])))

... comments welcome, either in email or on the (eventual) Mastodon post on Fosstodon.

OK so why both?

As in... if we have a shell script... why rewrite it in Python? Also... isn't this code... a little bit... ugly?

Well, actually: the fun part about all this code is who it was written by. It was... ChatGPT, mostly. And then OpenAI's code model turned it into Python.

In fact, checkinstall itself was recommended by ChatGPT! And it also knows how to use it. Along with GitHub APIs. I just asked:

how do I get the author, name, description etc. from a github repo?

followed by

write a bash script that, based on the results of the above API, fills out a call to checkinstall (e.g. --pkgname, --pkgversion etc.)

It did mess up quoting $DESCRIPTION first, and made up some arguments to checkinstall that don't actually exist, but... it knows how to use it in general, and the resulting shell script is an excellent starting point (it's pretty much what you see above).

They're really nice tools. Also, in case you weren't worried about where this is going... maybe it's time to start doing so.

Talking to your Android phone from anywhere

2022-07-16T17:00:00.000000-07:00

Talking to your Android phone from anywhere

2022/07/17

... how to "ssh" into your Android phone from a server far away?

(tl;dr: combine adb tcpip with Tailscale!)

Why?

Ever wished your Android phone was an Actual Computer? (... well, it is, it's just hiding it well.) Like... if only you could, say, write a cron job that checks where it is? Or just remote-desktop into it? While your processes aren't randomly killed because Android doesn't like them?

... ADB?

I'll assume you know what ADB is. If not... you can look it up: it's the standard way of debugging Android phones for developers. You can run commands in a shell, pull files, start apps, send keyboard events... many things that Android works very hard to prevent arbitrary apps from doing. You don't even need root for most of it: you can just enable developer mode.

Except... there is a catch. It needs a USB cable between your computer & the phone. Which is fine if you just want to do Actual Development, but... it's slightly inconvenient if you just want to rsync a few photos off it. It's, at least inconvenient enough to make you stop doing whatever you want to do regularly. It's also impossible to automate.

Alternatives

There are apps that launch an SSH server on the phone. The problem is that... they're apps. Thus, they're a lot less powerful than ADB. (... also, I'm not entirely sure that they'd be allowed to keep running with an open port all the time).

No USB!

You can get around this though. Actually, Android supports remote debugging, via a TCP port, which has the same capabilities as doing it over USB!

The catch is... you need to first enable this with, well, a USB cable. It's not hard though; once your phone is connected, you can just say

~ $ adb tcpip [portnumber]

... and then you can connect your computer to it with

~ $ adb connect [ip address]:[the port you picked above]

... which will give you the same access as if you had an actual USB cable connected. Then, once you're done, you can disconnect with

~ $ adb disconnect

You then only have to repeat this after a phone restart.

As expected though, you generally have to be on the same wifi network though; phone carriers are doing a good job at firewalling away open ports (in our case, it's probably for the better). So you can do stuff with your phone while you're at home, but...

... enter Tailscale!

As you might have noticed, I kinda like Tailscale. It's a VPN thing that feels like you had actual LAN cabling connecting all your devices, no matter where they are.

Which... is... pretty much we'd need for this.

And, as it turns out, the ADB TCP server on the phone does happen to listen on the Tailscale IP, too. Which... you can talk to any time. Even if you're sitting in a cafe with a laptop and your phone is still on LTE. Even if you're in a cafe and it's a server back home that's doing the connecting with a cron job.

Practicalities

Admittedly, I haven't tested this solution a lot yet... but regardless, here are some preliminary findings.

It always works. You're now in the same class as big tech companies' notification services: even if the phone is asleep, you can wake it up!

It also looks kinda good on power. Having adb tcpip on doesn't prevent the phone from sleeping. Apparently, connecting, even leaving an actual shell open, doesn't prevent the phone from sleeping?

My experiment for this was just to run a for loop in the shell with a sleep 0.3 in it; it becomes a lot more choppy once the screen is not on (indicating that it's asleep most of the time). On the other hand... this might prevent it from going into deeper sleep levels; the jury is still out on how bad this is for, say, an entire day.

... remote desktop?

Well, yes, I mentioned that in the beginning. So, if you're interested: scrcpy is a thing. It's basically remote desktop (normally) over USB, with compressed video streaming of frames... but then it's using ADB for all of it, so you can do this with the above setup, too!

(... it's also open source, and looks reasonably non-overcomplicated.)

... comments welcome, either in email or on the (eventual) Mastodon post on Fosstodon.

Fancifying the Command Line with JSON

2022-04-16T17:00:00.000000-07:00

Fancifying the Command Line with JSON

2022/04/17

... tl;dr: JSON fits in well with command line piping!

This is technically no longer part of Kev Quirk's #100DaysToOffload challenge. The point, nevertheless, is still to write many things, not to write good ones. Please adjust quality expectations accordingly :)

I recently came across this article on some new-ish command line tools on HN. Which did serve as a reminder for the existence of jq, the JSON query / processing / piping tool.

You can combine this with e.g. jc, which can turn outputs of various command line tools to json, and achieve something like this:

$ jc ls
[{"filename": "battleship.png"}, {"filename": "index.html"}, {"filename": "liblist.png"}]
$ jc ls -l | jq
[
  {
    "filename": "battleship.png",
    "flags": "-rw-r--r--",
    "links": 1,
    "owner": "simon",
    "group": "simon",
    "size": 20947,
    "date": "Jan 24 18:11"
  },
  {
    "filename": "index.html",
    "flags": "-rw-r--r--",
    "links": 1,
    "owner": "simon",
    "group": "simon",
    "size": 15608,
    "date": "Mar 28 21:46"
  },
  {
    "filename": "liblist.png",
    "flags": "-rw-r--r--",
    "links": 1,
    "owner": "simon",
    "group": "simon",
    "size": 16763,
    "date": "Feb 5 13:00"
  }
]
$ jc ls -l | jq '.[] | {.filename, .size}'
{
  "name": "battleship.png",
  "date": "Jan 24 18:11"
}
{
  "name": "index.html",
  "date": "Mar 28 21:46"
}
{
  "name": "liblist.png",
  "date": "Feb 5 13:00"
}

Apart from the little chaos with jq and having to expand the JSON array into individual elements it can act on (the ".[]" part), it's a lot easier to handle (and less prone to breaking) than the standard awk + sed + grep combo.

One might even argue that just by doing JSON you can get most of the benefits of Powershell (the "passing objects around instead of just plain text" part), without the downsides (... of your tools not actually just dumping text into pipes, so they're somewhat... harder to implement?). Not having used Powershell a lot though, there might be a lot of things I'm missing here.

Nevertheless... this is yet another instance where, instead of custom borderline-human-readable syntax, we use something more standard, obtaining a lot more flexibility. Just like with programming languages!

... comments welcome, either in email or on the (eventual) Mastodon post on Fosstodon.

Day 100

2022-03-29T17:00:00.000000-07:00

Day 100

2022/03/30

This is post no. 100 for Kev Quirk's #100DaysToOffload challenge. The point is to write many things, not to write good ones. Please adjust quality expectations accordingly :)

So it has come to this.

(... relevant xkcd.)

If I'm counting correctly (I'm... hoping I'm counting correctly?), this is the 100th article of the series that started very close to a year ago, with Sockets: part 1. And... as it turns out, having a hundred articles solves the problem of "my site is a pointless, empty ghosttown with posts with the word 'test' in their title" fairly reasonably well!

For some background on what this entire thing is... there is also the article written halfway through. But... of course, the 100th of all of them has to be yet another retrospective! So here it is.

Some conclusions

Beeminder works. At least for me. Being threatened with having to pay them $30 if you don't finish today's article, even though you're tired and it's late in the evening and you really don't want to, does wonders. Sometimes I did end up getting ahead, just for fun (... kinda the green dots?), but most of the time, it was "last minute" posts.

At the same time though... often, I really didn't end up having the time to do something Actually Interesting and write about it. Some of the posts that I did like a decent amount were about e.g. hacking some MIDI stuff with Wireshark... or talking to an SMTP server by hand and writing about it; doing this needs a lot of time than "let's write something in half an hour before falling asleep". Instead, there was a lot of "weird philosophy speculation about the nature of programming", which might not have lead anywhere interesting.

Nevertheless... consistency gets you fairly far. Sure, not all articles ended up being especially fancy, but whenever I had the chance / mood to do something fancy, there was an article, because I was writing one every 2-3-4 days, so the process of creating one has been worked out fairly well.

At the same time... at some point, I stopped posting every article on my Mastodon account, since... I didn't think many of them made it over the quality bar. Which is still a lot better than not writing them because of speculation on them not making it over the quality bar.

Was it worth it?

Hell yes.

Much has been written about how writing ideas down helps actually thinking about them. Even if no one ends up reading them. So getting in the habit of doing this is nice already (even though it's been a specific, mostly computer-y subset of ideas so far).

But then... my article on writing shell scripts in Lisp has been surely visited a lot of times... just by me, as a reference. Not sure whether there were additional visitors, but this was already worth it. (... update: apparently, it's getting visits from google.com, of all places!)

On the other hand, my Emacs OS Mobile article made it into the Linux on Mobile newsletter! How cool is that?

That article was, by the way, the most popular one, by number of visits. Runner-ups include Web Vitals (in which it turns out that this site isn't super fast, after all... we might have fixed it with a new front page though?), the one on SMTP, the Amazon Prime's evilness one, and one trying to make certificates slightly more intuitive. Also, stuff on copyright, tabs, Signal and Discord.

What's next?

Well... first of all, yet another round of thanks to Kev for making up this idea!

And... no, the current cadence of articles is definitely not going to continue. I'll need more time to actually make them interesting.

On the other hand... they're also not going to stop coming entirely. Having an Actual Website with Actual Articles is something that I now know how to do; although I won't have "100 days" to blame for low quality, I hopefully have learned already not to care a lot. I do agree with Joels 101th article on this: it's just getting started!

On the other hand... not having to keep them coming will hopefully let me fix a few things. In no particular order:

... can we make an ActivityPub feed of all the articles so that anyone from the Fediverse can follow them? and then boost good ones from my actual account?

Also... actually posting them somewhere might be a good idea sometimes, too.

Along with... maybe a redesign for the actual articles, not just the title page?

But most importantly: this idea of consistently producing a ton of questionable-quality stuff to make some cool things in the process seems generalizable to... a lot of other things, too! So that part might continue. Maybe it will be large amounts of weird synth / piano songs; maybe finally learning Blender reasonably well; maybe writing some Actual Software (yes I do have some Lisp-related plans there).

Finally: thank you for everyone who ended up reading any of the many, many articles... hope you found something fun / interesting somewhere!

... comments welcome, either in email or on the (eventual) Mastodon post on Fosstodon.