Novation Circuit: backing it up
... as in: the web still sucks. Unlike the command line. Yes, the story will involve USB packets, too.
I have a Novation Circuit; it's a standalone device to make electronic music. It can produce drums, plus two synth channels that you can drive with external MIDI, too! The image below is a link to a review if you're interested in it a bit more. The cool thing about it is that you don't need to stare at a screen and do computery things to play on it. Which is the very reason I have it in the first place: too much "computers" otherwise.
It comes with some builtin sounds... but you can swap them out. I was playing around a bit with it already though; I wanted to save all the "songs" I created to a computer, before overwriting everything. Which it is definitely supposed to be able to do.
They do have... two ways to do this. You can either use a web-based utility called Novation Components... or... you can get the standalone version. To download the latter, you actually need to register the thing with them, plus you need an account, which... I do have, but... why exactly does this company know exactly who I am and what I've bought, just to use a device I already paid for? Plus, as it looked to me, their Standalone Application is basically an Electron app, doing the exact same thing as the "web" version.
OK, let's do this! Connect the device, restart in bootloader mode, click on button on web thing... resulting in a progress bar progressing neatly, stopping at about 95%, and then:
(... for better searchability, this is what it says: "Something went wrong when we were getting things from Circuit. Please check your Circuit connection and try again". (of course, you can't even copy out the text without resorting to developer tools, because... reasons.))
This is... well, um, not overly helpful. Dev tools doesn't show any visible errors, either. So... time to give up, be sad, and just proceed with the overwrite?
Well, nope. Never give up, never surrender.
MIDI, SysEX and state dumps
The neat thing about MIDI is that it's a standard from the '80s. Back then, things were... generally a lot simpler. Namely, MIDI mostly consists of messages like "a note was hit with velocity X", "it was released", or "fancy instrument parameter Y was tweaked". This was what actually went over the wire, using a serial protocol (31.25 Kbaud, slightly slower than a 56k modem).
There were things, however, that didn't quite fit into these. Thus, a "system exclusive" (SysEx for short) class of messages were created: they're manufacturer specific and could contain basically any type of information. A typical form of usage would be to download or upload all the parameters to / from a MIDI synth: which oscillators, what frequency, which filters, etc (these were how actual instrument sounds were made). Likewise, you could upload sample packs with these. (I actually have a '80s Korg synth, too, which uses these kind of messages for configuration.) Generally, you would send a (manufacturer-specific) SysEx message at the synth, to which it would respond with another SysEx message (or a series of messages), containing a full state dump.
Fortunately for us, Novation didn't invent a fancy new USB protocol: they're just using MIDI for this, too. Better yet: they also didn't seem to have overcomplicated everything: picking up some javascript using Dev Tools (network tab!) and throwing that minified, single-line ugliness into an online beautifier tool yields something like...
Well, I have no idea what thing does and don't really want to spend too much time figuring it out, but... if I were to guess: they're sending a short-ish message to the Circuit, which responds with a longer dump, just like any other MIDI device would. So... what do they send and how do we catch the response?
Snooping on MIDI
Their web thing is using MIDI support in Chrome / Edge. Sounds good far. Except... how exactly do we debug this? It's not like Dev Tools has a MIDI console.
It also seems somewhat tricky to route this through some other MIDI devices, as Components seemed fairly happy about seeing an actual USB device named "Circuit" in there. (Yes, the Circuit is connected via USB-based MIDI. It has actual MIDI ports, too; not sure whether they'd work for this purpose.) I tried intervening with MIDI OX, to no avail.
So... what does one do to interrupt MIDI traffic that's being wrapped into USB right away? Actually... do we have issues with... looking at USB packets directly instead? How involved could those get?
Well... enter Wireshark!!!
USB and Wireshark
(... side note: all I wanted is to make some music while not looking at screens. I am now dissecting USB packets with Wireshark. What's wrong with my life???)
So... yes, you can look at USB packets with Wireshark!!! Which is especially fun because you get to look at packets from your keyboard and mouse while using them to navigate between packets! It's... also counterproductive somewhat, as scrolling up results in more packets that will move you downwards. Maybe stopping the capture is kinda useful here.
Aaanyway: here is what you get!
(... click to make it full screen.)
So... you can see multiple things here! For example, those host to 1.7.1 messages are probably coming from the actual mouse click on the "Backup Circuit" button (or... however you start this); they're "interrupt" transfers (small but fairly low latency). Meanwhile, 1.6.1 is probably the Circuit; that SYSEX message followed by URB_BULK out was probably the request itself, acknowledged by the USB device. Following these, the host keeps sending out URB_BULK in messages, to which the device keeps responding with actual chunks of... probably what we're actually interested in. (Remember how with USB, everything is initiated by the host? This is an excellent example: the device sending data is always preceded with the host asking for more.)
The interesting part is the actual message we send out that causes the device to reply with the dump. Our hope is that there is indeed nothing fancy going on: we send that out, the device responds, all this needs to happen only once, and the response happens to be identical to the contents of those .syx files the software is talking about. (.syx, SysEx... must be something there.)
Time for an actual dump!
So... we have this sequence, starting with F0 00 20 29 ... and ending with 00 00 F7. Which is... encouraging: SysEx messages are supposed to start with an F0 and end with an F7 (... as per many of the informative articles you can look up on them). So... what we want is to send this out to the device, and listen to what comes in, right?
There might be fancy GUI tools for this... but... in simplicity, it's really hard to compete with amidi:
AMIDI(1) General Commands Manual AMIDI(1)
NAME
amidi - read from and write to ALSA RawMIDI ports
SYNOPSIS
amidi [-p port] [-s file | -S data] [-r file] [-d] [-t seconds] [-a]
DESCRIPTION
amidi is a command-line utility which allows one to receive and send
SysEx (system exclusive) data from/to external MIDI devices. It can
also send any other MIDI commands.
amidi handles only files containing raw MIDI commands, without timing
information. amidi does not support Standard MIDI (.mid) files, but
aplaymidi(1) and arecordmidi(1) do.
(...)
-l, --list-devices
Prints a list of all hardware MIDI ports.
(...)
-S, --send-hex="..."
Sends the bytes specified as hexadecimal numbers to the MIDI
port.
(...)
-d, --dump
Prints data received from the MIDI port as hexadecimal bytes.
Active Sensing and Clock bytes (FEh, F8h) will not be shown, un‐
less the -a or -c options have been given.
This option is useful for debugging.
(...)
-r, --receive=filename
Writes data received from the MIDI port into the specified file.
The file will contain raw MIDI commands (such as in a .syx
file); to record a Standard MIDI (.mid) file, use arecord‐
midi(1).
So let's try dumping it out!
~$ amidi -l
Dir Device Name
IO hw:1,0,0 Circuit Bootloader MIDI 1
~$ amidi -p hw:1,0,0 -S "f0 00 20 29 00 78 00 00 00 02 0e 00 00 00 00 00 01 0a 0d 00 00 00 f7" \
-d
F0 00 20 29 00 77 00 00 00 02 0E 00 00 00 00 00 01 0A 0D 00 00 00 F7
F0 00 20 29 00 79 30 44 45 4D 4F 5C 3B 00 00 00 00 00 00 00 00 00 00 00 00 44 65 6D 6F 20 00 53 65 73 73 69 6F 6E 00 20 20 20 20 20 20 20 00 20 20 20 20 20 20 20 40 20 20 20 20 20 20 1E 00 32 00 00 00 00 00 00 00 00 00 00 00 00 07 00 00 00 F7
F0 00 20 29 00 79 00 00 03 00 00 00 03 00 00 00 01 01 00 00 0F 66 00 00 00 48 0D 00 00 45 00 0D 00 00 4D 0D 00 00 00 41 0D 00 00 00 00 00 00 00 00 00 00 00 00 60 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 F7
(...)
Here we go!!! We can now just save the entire thing into a file:
amidi -p hw:1,0,0 -S "f0 00 20 29 00 78 00 00 00 02 0e 00 00 00 00 00 01 0a 0d 00 00 00 f7" \
-r circuit_dump.syx
And the best part: this actually works: you can load these .syx files into Novation Components itself!
... or I think it worked; it displays a loading bar, which finishes neatly. You can't really edit these here, but... I'll just assume it's the right thing. It looked happy enough with it.
Conclusions
USB might feel like it's pure magic. If you have the right tools, it's really not. To the point where it might be easier to deal with than MIDI directly.
(I'm not overly sad though that I didn't have to generate those .syx files from Wireshark dumps.)
Meanwhile... we seem to continue with the process of turning everything into web apps; they're... not always the most reliable things ever. On the plus side: 80s era simplicity sometimes does help making things nicer!
... comments welcome, either in email or on the (eventual) Mastodon post on Fosstodon.