Split MIDI Files with Python

In my previous post I showed how to use Raspberry Pi to automatically record MIDI files from digital piano whenever you turn it on. However, if you sit down and play for an hour, and get one big MIDI file, it is not very useful. So what to do?

Pianoteq has a feature that splits your playing session based on breaks you take between playing notes. So I decided to mimick this feature with a simple Python script that:

  1. Keeps tab on keys and pedals pressed
  2. Whenever X seconds elapse without any keys or pedals down, a new MIDI file is started
  3. Additionally, if user presses and releases sustain pedal several times in the end, filename for that MIDI is altered to “highlight” that file

I first thought to learn enough of MIDI file format to do everything from scratch, but there’s quite a bit of small details to handle, so in the end I decided to use an external toolkit from Craig Stuart Sapp called midifile to do the heavy lifting. You should be able to just clone the Git repo and make it with Raspberry Pi:

pi@raspberrypi:~ $ git clone https://github.com/craigsapp/midifile
pi@raspberrypi:~ $ cd midifile
pi@raspberrypi:~/midifile $ make

You should now have two useful commands in ~/midifile/bin: toascii to read a MIDI file and dump an ASCII (text) version of it, and tobinary to do the reverse. With Python’s Popen and smart piping, we can read and write the binary MIDI files as they were in this text format. You can check out how the format looks like with some MIDI file you have:

~/midifile/bin/toascii somemidi.mid | less

Here’s a sample of a MIDI file recorded by arecordmidi (I added the note in square brackets):

"MThd"
4'6
2'0
2'1
2'384

;;; TRACK 0 ----------------------------------
"MTrk"
4'50044
v0      ff 51 v3 t120
v0      ff 58 v4 '4 '2 '24 '8
v7147   90 '58 '29
v88     b0 '64 '5
v1      b0 '64 '7

[LOTS OF MIDI EVENTS]

v0      ff 2f v0

There’s some header data in the beginning, and each MIDI event is comprised of a deltatime field starting with ‘v’, and then fairly standard MIDI events in straightforward syntax. I hardcoded my program to expect single track which starts with tempo and speed data (ff 51 and ff 58 lines) which I use to calculate how many deltatime units is one second. I copy this header part to start of every MIDI file, and append the final “ff 2f v0” (END TRACK) event to the end. Here’s the full code:
Continue reading Split MIDI Files with Python

Using Raspberry Pi as an automatic MIDI logger

During my summer holidays I got an interesting idea: Pianoteq has a very nice feature of “always on MIDI logging” that saves everything you play on your keyboard while Pianoteq was on. I’ve previously made some MIDI projects and had a great idea:

How about building a small device that records everything I play on my piano, and save it as MIDI files?

This would enable me to later grab a good performance, and eliminate the “recording anxiety” I get if I know I’m recording and should definitely not do any mistakes during the next 1000+ notes. Furthermore, even with easy MIDI recording to USB stick, it’s still several manual steps plugging the memory stick in, starting recording, stopping it, lugging it to a computer, etc.

My first idea was to use some WLAN-enabled embedded device, but MIDI IN would require optoisolators and some custom electronics, and more modern digital pianos often come with only USB MIDI, so it could easily become an exercise in communication protocols. Fast forward a couple of minutes to my next revelation:

Raspberry Pi Model 0 W already has USB and WLAN, and it’s small. Why not use that?

Turns out using a RaspPi as fully automated MIDI logger is really easy. Read on for instructions!

Update: Also check out my follow-up post to split the recorded MIDI files automatically!

Recording MIDI with Raspbian

Turns out recording MIDI from a USB MIDI enabled device is really easy. When I plug in my Kawai CS-11 (sorry for the unsolicited link, I love my CS11 :) to the Pi (or just turn it on when it’s plugged in), dmesg shows that the Pi automatically notices the new MIDI device:

[  587.887059] usb 1-1.5: new full-speed USB device number 4 using dwc_otg
[  588.022788] usb 1-1.5: New USB device found, idVendor=0f54, idProduct=0101
[  588.022800] usb 1-1.5: New USB device strings: Mfr=0, Product=2, SerialNumber=0
[  588.022807] usb 1-1.5: Product: USB-MIDI
[  588.074579] usbcore: registered new interface driver snd-usb-audio

Once the USB MIDI device is found, you can use arecordmidi -l to list available MIDI ports:

pi@raspberrypi:~ $ arecordmidi -l
 Port    Client name                      Port name
 14:0    Midi Through                     Midi Through Port-0
 20:0    USB-MIDI                         USB-MIDI MIDI 1

Continue reading Using Raspberry Pi as an automatic MIDI logger

USB Mouse with ATmega32U4 Pro Micro Clone and LUFA

I have spent a fair amount of time with 8-bit AVR microcontrollers and one of the cooler things has been the V-USB library which implements low-speed USB with clever (and very time-critical) bit-banging. The popularity of my USB tutorials is a testament to its usefulness, and I’ve gotten lots of mileage out of that.

There are, however, some limitations to software USB with such a low spec microcontroller. USB communication hogs up the MCU completely during USB communication, which means you lose dozens of microseconds in random (or in many cases 8 ms) intervals. This rules out things like software UART at reasonable speeds (which I discovered when trying to implement MIDI on Adafruit Trinket). And more powerful ATmega328-based dev boards like Pro Trinket start to get quite large.

ProMicro on a breadboard

Not so with this tiny beauty shown in the image. It’s a ATmega32U4 based board, where the U4 means it has hardware USB support. The form factor is extremely compact 12 pin header length, which leaves 5 rows free on the smallest prototyping breadboards. That means you can have a DIP8 component with a few resistors on the same breadboard (such as a 6N137 optocoupler which is nice for MIDI… ;).

And the best part is, that because the chip is flashed with same firmware used in Arduino Leonardo (and a largely matching pinout), you can use Arduino for programming, and avrdude supports it out of the box.

Actually, scratch the above statement. The best part is the price. The board is based on Sparkfun Pro Micro 16 MHz, but it’s actually a Chinese clone, which you can get for $4 via DealExtreme and from quite many places in AliExpress: Just search for ATmega32U4 and they will come up. This means you can just order five and solder them into whatever project you’ll make permanently. And unlike Arduino Micro (for which clones exist as well), this has the micro-USB port already in place.

Using Pro Micro without Arduino IDE

Now you can just follow SparkFun’s instructions on how to use that thing on Arduino (short version: select Leonardo as board type, and look up the schematic if you are unsure which pins are connected to LEDs, etc.). But if you’re like me and want to get to raw metal, avr-gcc and avrdude is the way to go. Here’s a simple blinky demo:
Continue reading USB Mouse with ATmega32U4 Pro Micro Clone and LUFA

Mounting / Fixing Raspbian SD Card from Raspberry Pi

Shut down my Pi today and thought to make a copy of files in its SD card. This is what mount /dev/sdf2 /mnt had to say:

mount: wrong fs type, bad option, bad superblock on /dev/sdf2,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

Great. After trying parted and fsck, it became apparent that for some reason, the root partition is marked as being 1 block longer than the physical card. Must be a bug with Raspbian partition expansion or something.

Thankfully, I found this gold nugget which suggested using resize2fs to fix it. Turns out I had to run e2fsck first (and say “y” a couple of times):

sudo e2fsck /dev/sdf2
sudo resize2fs /dev/sdf2
mount /dev/sdf2 /mnt

Voilá! Fully functioning filesystem again.

Fixing Android Browser SVG Scaling with max-height

Ever had that moment where you started coding X and instead spent four hours doing Y instead? I just had it, and it stemmed from a quite simple idea: let’s do an embedded SVG bar chart with React (nice library by the way)! Because I just started with Bootstrap which makes it really simple to scale your content to the device at hand, I also wanted the diagram to scale. Easy, right? WRONG!!!

<svg version="1.1" width="100%" viewBox="0 0 200 100">
here are some primitives
</svg>

The viewBox attribute defines the internal coordinates for the SVG, and the width attribute should ensure the SVG fills the container. And it does, at least on desktop. And mobile Chrome. But not Android default browser. I tried different settings of preserveAspectratio. I tried CSS tricks that should work but they wouldn’t. After some hours, I gave up and went to sleep. And this morning I found this nugget: WebKit long had a bug, where SVG element is resized to 100% height, which on mobile device was interpreted as screen height (not the parent element)!

So long story short, if I use my S5 in portrait mode, the 1080×1920 screen makes the above SVG element 1920 pixels high, with 1080×540 (2:1) diagram in the middle of huge white area. The bug is reported in 2012: https://bugs.webkit.org/show_bug.cgi?id=82489. This was supposedly already fixed, but seems the new version hasn’t trickled to my mobile OS yet. Thankfully, the fix is quite simple and does seem to work, just add max-height to the element:

<svg version="1.1" style="max-height: 100%" viewBox="0 0 200 100">
here are some primitives
</svg>

I didn’t even need the width=”100%” because that seems to be the default. Doesn’t hurt, though. So, just sharing this for everyone, maybe Google will index this and help others in need.

SQLAlchemy Results to JSON – the Easy Way

Another short web coding tidbit for today. I’ve been working on a Bottle.py+SQLAlchemy based backend for a personal project, and found very little information on how to convert a SQLAlchemy Core resultset into JSON.

Here is a glimpse into what was most commonly suggested. Tons of code. But wait, did you know you can cast a row into a dict? No? Well, it makes a world of difference:

import json

# somewhere here, accounts table is defined with SQLAlchemy syntax

def example():
    res = conn.execute(select([accounts]))

    # return all rows as a JSON array of objects
    return json.dumps([dict(r) for r in res])

Now, which do you prefer, the last line or the longer 17-lines in StackOverflow? There is a slight problem though, namely date and numeric(x,y) fields, as they will turn into datetime.date and decimal.Decimal types. Thankfully, Python’s JSON library can be provided with a function to handle classes that the library itself doesn’t handle:

import decimal, datetime

def alchemyencoder(obj):
    """JSON encoder function for SQLAlchemy special classes."""
    if isinstance(obj, datetime.date):
        return obj.isoformat()
    elif isinstance(obj, decimal.Decimal):
        return float(obj)

def example():
    res = conn.execute(select([accounts]))

    # use special handler for dates and decimals
    return json.dumps([dict(r) for r in res], default=alchemyencoder)

Simple X Audio Switch (Double-Y)

I’ve had a small problem over a year: I have two headphone amplifiers, a transistor-based O2 headphone amp, and a tube-based Little Dot MKII. I also have two sets of headsets. Because switching the cables all the time is somewhat tedious, I haven’t really used the other amp and headphone set.

There are several Y switches that you can buy, but most seemed either expensive or mostly tailored towards electric guitar enthusiasts. And I would still need to combine two Y switches to toggle between the two amps and headphones. As the construction is really simple, I decided to solder my own. Here’s a short tutorial on how to make one yourself. You’ll need:

  1. 4 audio connectors (I chose 3.5mm over RCA), two to each end
  2. 2 switches with at least 3 connectors (left, right, ground) – I chose two 4 way switches
  3. A box where you can drill holes for the connectors and switches
  4. Small length of wire, some solder and soldering iron

1. The Schematic

Audio X switch schematic

The design of the switch is extremely simple. In both ends, three wires come from each of the two connectors to a switch, and the switches are connected to each other. I chose not to combine any ground lines to avoid ground loops. You need also to select a switch which breaks the connection on one side before connecting the other so the amplifiers will not feed into each other.

2. Soldering the wires

Despite the simple construction, there is a lot of soldering involved. Each of the four connectors will have ground plus left and right channels, and all have the switch on the other end. That makes 24 solder points (4x3x2), and there’s six more between the switches for a total of 30 points. The 3.5mm connectors had small “L” and “R” markings to tell which channel is left and which is right. I used different colors of wires for each type to avoid mixing L, R and ground.

solder
Continue reading Simple X Audio Switch (Double-Y)

Changing template tags in Bottle.py SimpleTemplate

A web coding tip for a change: After a long coding hiatus, I decided to try my hand at recoding my web-based budget software with AngularJS on the client side and Bottle.py handling the backend. Superb and compact combination, by the way!

Bottle.py comes with a great minimalist templating engine called SimpleTemplate, which uses {{ var }} syntax for inline variables. This does not mix well with client side AngularJS which uses the exactly same delimiters. There is an easy way to change Angular’s syntax with $interpolateProvider, but guess what? Many AngularJS additions, like the datagrid component ui-grid (previously ng-grid) don’t respect this setting, and just plain break with custom delimiters. Not nice

Changing Bottle.py template tags

So, what if you want to change SimpleTemplate syntax? Turns out there is very little documentation for it, even if it’s a single line change. Just locate this line in bottle.py:

default_syntax = '<% %> % {{ }}'

…and change it to:

default_syntax = '<% %> % [[ ]]'

And that’s it! Actually, not quite, because bottle default error page template is hardcoded for curly braces. So locate ERROR_PAGE_TEMPLATE in the same file, and change every {{var}} to [[var]].

There is elegant code in bottle.py that seems to enable replacing the default syntax, but unfortunately the only tip for using it seemed somewhat complicated. So I opted for this simple hack. I’ll sure regret it when I next time update my bottle.py, but, well, that’s not in this month, right?

Giving Raspberry Pi Camera Nearsight with Reading Glasses

Raspberry Pi with Reading Glasses

Back again! The summer holidays gave me some time to write after a long hiatus, and this time it’s a Raspberry Pi related article. I’ve had the excellent opportunity to play around with a Raspberry Pi Camera Module for a few days. Or actually modules, as I got both the normal and NoIR without IR filter (more about that later) from Farnell / Element 14. They also stock an excellent selection of Pi accessories, so be sure to check those out, too.

But without further ado, let’s get onward. I’m still thinking up cool projects to do with the camera, so if you have nice ideas, please feel free to share them in the comments section!

Unboxing and First Impressions

Raspberry Pi IR camera and a normal one

The camera modules arrived in simple boxes, branded with element14 logo and URL. A nice additional touch was an included instruction sheet outlining the installation procedure, as well as a link to www.element14.com/picamera with further info.

Both the IR-filtered (the one showing normal visible light) and the NoIR (the one without the filter, and thus showing both normal light AND infrared) have exact same outward appearance. The installation was quite easy, but the flat cable offers less positioning and flexing freedom that your standard webcam – obviously the Pi camera is meant for more integrated installations.

Raspberry Pi camera electronics
Continue reading Giving Raspberry Pi Camera Nearsight with Reading Glasses

Tutorial: State Machines with C Callbacks

State machine

Many electronics projects involve the device transitioning from one state to another. On a high level, it could be that your project is initially in a state where it awaits input, and once it receives it, it goes to another state where it performs a series of actions, eventually returning back to the initial state. On a lower level, many communications protocols are often simple or increasingly complex state machines. In many cases, an elegantly implemented state machine can simplify your code and make it easier to manage.

There are several methods to implement state machines programmatically starting from simple if-conditions to state variables and switch structures. In this tutorial I’ll cover a slightly more advanced method of using callbacks or “function pointers” as they are implemented in C. This has some performance benefits, makes up for some clean code, and you’ll learn a bit on the way!

State Machines with Conditionals

First, to introduce the idea of a state machine, lets take a simple example: Blinking LEDs. Imagine you want the LED to turn on for a second, and then turn off. We could do it with a simple 2-state machine:

enum states { 
  LED_ON,
  LED_OFF
};

enum states state = LED_OFF;

while(1) {
  if(state == LED_OFF) {
    led_on();
    state = LED_ON;
  } else {
    led_off();
    state = LED_OFF;
  }
  sleep(1); // sleep for a second
}

If you can understand the code above, you have pretty much grasped the fundamentals of state machines. We have some processing specific to given state, and when we want to go to another state, we use a variable (in this example it’s called state) to do that. And by the way, if you haven’t encountered enum before, you should check out Enumerated type in Wikipedia. It’s a very handful tool to add to your C coding arsenal.
Continue reading Tutorial: State Machines with C Callbacks