Slim Devices Utilities

Not only is the hardware cool, but the software is Open Source. Which is very cool.

So in the spirit of Open Source, I've written some utility functions

If you try this, please send me feedback. Email to
pfarrell at pfarrell dot com
(just despam it the usual way)

Table of Contents

• MusicUtil the main, high volume utilty
• AmazonCovers a utility to retrieve the cover art from a set of keywords
• AmazonTest a utility to display the results of a query of Amazon's XML service
• Source code information
• Design philosophy
• XML and program setup
• bugfixes log
• a new, low functionality utlity in Perl to convert flac files to mp3, called flac2mp3.pl
• Another low functionality Perl utility, called mk3u.pl that builds a .m3u file
• YALFU (yet another low functionality Perl utility) that converts files from flac to wav, called, amazingly enough, flac2wav.pl

MusicUtil

Features:

• Automatically Retrieve three sizes of cover art automatically by reading the internal tags of your music files. (This is the "covers" command.)
• Can read tags in MP3 (V1, V2.2 and V2.3), OggVorbis and Flac files.
• Can read Unicode (UTF-8) tags in OggVorbis and Flac files.
• Recursively chases down your music directory structure.
• Can locate directories that contain duplicate files (more than one type of music files (mp3, WMA, flac, ogg, etc.)
• Can locate files that have tags with Accented or otherwise "strange" characters.
• Can search tree for music files that are too small to be likely to be good.
• Can search tree and output list of Albums and Artists found, suitable for printing as an inventory
• Identifies empty (one pixel) .gif files and does not use them.
• Can search tree for empty (one pixel) .gif/.jpg artwork files (too small to be likely to be good) and deleted them.
• Can verify that all the flac files are clean and decodable
• Free
• Open Source

My first utility functions and programs are designed as shell commands. There is no GUI. If you must have a GUI, these are not the tools for you. They work in the Windows cmd.exe shell, Mac OSX's shell and in Linux's bash shell. They should work in other shells, but I have not tested them there. Actually, I haven't tested them with Mac's OS-X, but others have and report no problems.

I have not yet put appropriate Copyright messages, I'll release it under a BSD-style license. I am not a fan of GPL.

The code is written in Java, so it should run on any server that can run the SlimServer, assuming you have a Java system installed. If you don't have a Java system, you can get one (free) from Sun. It does not have to run on the same computer as your slimserver, if you have your songs on a networked fileserver.

Right now, this package has no Installation Wizard. If you need one, please wait.

So if you want to try it. download the pfarrell.jar file (aka the http://www.pfarrell.com/music/slimserver/pfarrell-latest.jar) file. and put it in your java installation's jre/lib/ext directory. (If you have installed earlier versions, delete that jar file.)

To verify that the installation is working properly:

• Start up a shell
• enter
     java com.pfarrell.utils.music.MusicUtil

It should write out a message similar to this:

java com.pfarrell.utils.music.MusicUtils [keyword] [directory]
   where keyword is one of test | scan | covers | dups | strange
         	 | help | version | flactest
   directory := starting path for searching

 

Which tells you that there are many legal keywords, which you will be able to use shortly (see final setup). The commands are

• test
• scan
• covers
• dups
• findsmall
• noonepix
• strange
• unamerican
• list
• flactest
• help
• version

and that the program expects a directory to start searching. The program assumes that your music is stored in a directory tree structure, containing .mp3, .ogg. or .flac files. [the current version can not parse Microsoft WMA files].

The functions behind the keywords are:

test

Test your configuration setup. Prints out the contents of your musicutil.conf file. This is mostly a debugging tool. It is useful to see if the program reads the same intitialzation values that you think have set. Because different Operating Systems and different Java VM's read preferences from assorted and non-standard directory paths, this is most useful so you know what file the program thinks it is reading.

scan

Scan thru all the files in the directory tree, writing the identifying tag information (artist, song title, album title, etc.) to STDOUT (to the shell). This is mostly a debugging tool. It is useful to see whether your ID3/Ogg tags are correct, if they contain any funky characters, etc.

dups

Scan thru all the files in the directory tree, looking for directories that contain two or more file types. This is handy when you are building a library and may have old files. I wrote it so I could tell if I had duplicate songs in WMA or MP3 format when I converted everything to FLAC.

covers

This is the key. Scan thru all the files in the directory tree, reading the identifying tag information (artist, album title, etc.) from the ID3 or Ogg/Vorbis or Flac tags. This data is then sent to Amazon.com using their XML interface, to obtain the URL of the cover art. Then use the URL to retrieve the image and store it on your local hard disk. While it is talking to Amazon, it gets the official album title and artist title, and the ASIN number. These are written out as a small text file (asin.txt) in the music directory, so you can use the data for other purposes.

If the option "--safe" is specified, the program will not overwrite existing .jpg artwork files.

The search algorithm works directly with the Amazon.com keyword search. The program takes all the words found for the AlbumTitle, Artist and Performer tags. These tags are processed against a stoplist so that unselective words are removed. The whole remaining set of keywords is used to select the album. This works in the vast majority of cases, allowing automatic selection of an entire tree of albums to be processed. But sometimes this algorithm is not sufficiently selective. See the AmazonCovers program for alternative approaches.

flactest

findsmall

noonepix

help

Write out a verbose help message, explaining these commands .

version

Write out a verbose message with the version number of each major module. Mostly of interest to developers and during debuging.

strange
unamerican

On my system, I have a lot of files names, genres, and artist with full Latin-1 characters. They typically include accented characters from western Eurpoean languages. Thus Andres Segovia is tagged as Andrés Segovia. This is not literally Unicode, just not US ASCII. Anyway, the HTML display is ugly, and the faceplate display is ugly too.

And yes, I took enough high school French to know that accented characters are not strange. I've made "unamerican" a synonym for this command. Clearly this is in honor of the witch hunts of the 50s, 60s, and 70s.

list

Scan thru all the files in the directory tree, reading all directories. This assumes that the songs are kept in a directory tree structure, with artist/album as the names. Output the unique Artist and Album names to STDOUT. This can be saved to a file, included in a HTML or Word inventory list, or even fed to a DBMS package. The artist/album names are collected accross all scanned directories, so that if you have a structure such as genre/artist/album the artist/album values will be collected accross all genres. This is good if you want to collapse genres, bad otherwise.

AmazonCovers

Features:

• Standalone test routine (used by the MusicUtil program.)
• Automatically Retrieve three sizes of cover art automatically searching for keywords specified on the command line.
• skips any directory containing a file named asin.txt
• supports the --safe option
• supports the --c option for classical testing
• Free
• Open Source

This program, like MusicUtil, is a shell command.

The AlbumCovers program allows cover art retreival to be tested. When run by itself, the class will query Amazon and then retreive the specified cover art and place it on the working direcory. So the command:

  java com.pfarrell.utils.music.AmazonCovers Eric Clapton unplugged

pwd=B:\jsrc
B000002MFE
Unplugged
Eric Clapton
http://images.amazon.com/images/P/B000002MFE.01.LZZZZZZZ.jpg
http://images.amazon.com/images/P/B000002MFE.01.MZZZZZZZ.jpg
http://images.amazon.com/images/P/B000002MFE.01.THUMBZZZ.jpg

And it would create four files in the current working directory (the pwd line) of

• cover.jpg
• albumartsmall.jpg
• thumb.jpg
• asin.txt

So the command:

 
   java com.pfarrell.utils.music.AmazonCovers --safe Eric Clapton unplugged

This program is also useful to override the stoplist logic of the main MusicUtil program. Simply provide the keywords you want to use as command line arguments, and they will be used for the search. For example, the ablum "Now That's What I Call Music! 7" a compilation listed as by "Various Artists" in the Amazon database has the song "Survivor" by "Destiny's Child". Sometimes ripping software will list the performer as "Destiny's Child" while Amazon will only find it using "Various Artists". So the solution is to use AmazonCovers to manually specify the keywords you want.

 java com.pfarrell.utils.music.AmazonCovers now that call music 7

        
  pwd=S:\test
  B00005LOAH 
  Now That's What I Call Music! 7 
  Various Artists 
  http://images.amazon.com/images/P/B00005LOAH.01.LZZZZZZZ.jpg
  http://images.amazon.com/images/P/B00005LOAH.01.MZZZZZZZ.jpg
  http://images.amazon.com/images/P/B00005LOAH.01.THUMBZZZ.jpg 

and it would write out the three image files to the s:\test directory.

Similarly, if you have the ASIN number for an item, you can just enter it for the search. ("ASIN" stands for "Amazon.com Standard Item Number.) By definition, Amazon's ASIN numbers are unique. So entering

   java com.pfarrell.utils.music.AmazonCovers B000002MFE

will result in

  B000002MFE 
  Unplugged
  Eric Clapton 
  http://images.amazon.com/images/P/B000002MFE.01.LZZZZZZZ.jpg
  http://images.amazon.com/images/P/B000002MFE.01.MZZZZZZZ.jpg
  http://images.amazon.com/images/P/B000002MFE.01.THUMBZZZ.jpg
 

AmazonTest

Features:

• Tells you what Amazon's XML interface will return searching for keywords specified on the command line.
• Lets you manually select graphics from duplicates
• supports the --c option for classical testing
• Free
• Open Source

This program, like MusicUtil, is a shell command.

The AmazonTest program displays the results of querying the Amazon XML interface. It executes a query against Amazon.com and displays the results to the standard output. The query only asks for, and get back, the first page of data from Amazon. If you execute a non-specific query, you can expect to get many pages of results. This program only displays the first page (usually ten results).

Executing the program with no parameters:

    java com.pfarrell.utils.music.AmazonTest 

will result in output similar to

   usage: java com.pfarrell.utils.music.AmazonTest <keys>
     where: 
      keys are the keywords to search for
       (separated by blanks)

A more interesting case puts some keywords in, and sees what Amazon can do.

  java com.pfarrell.utils.music.AmazonTest Eric Clapton unplugged

   B000002MFE
   Unplugged
   Eric Clapton
   http://images.amazon.com/images/P/B000002MFE.01.LZZZZZZZ.jpg
   http://images.amazon.com/images/P/B000002MFE.01.MZZZZZZZ.jpg
   http://images.amazon.com/images/P/B000002MFE.01.THUMBZZZ.jpg


   B00005Y7KA 
   Unplugged/Clapton Chronicles: The Best of Eric Clapton
   Eric Clapton
   http://images.amazon.com/images/P/B00005Y7KA.01.LZZZZZZZ.jpg
   http://images.amazon.com/images/P/B00005Y7KA.01.MZZZZZZZ.jpg
   http://images.amazon.com/images/P/B00005Y7KA.01.THUMBZZZ.jpg


   B00000ILLS 
   Unplugged
   Eric Clapton
   http://images.amazon.com/images/P/B00000ILLS.01.LZZZZZZZ.jpg 
   http://images.amazon.com/images/P/B00000ILLS.01.MZZZZZZZ.jpg
   http://images.amazon.com/images/P/B00000ILLS.01.THUMBZZZ.jpg

Source, JavaDocs, etc.

The distribution jar file contains just the executable (really jar) data. It contains the binary class files, ready to use. I've pulled out the source code and Javadocs, just to shrink the size of the downloads for non-developers.

Drop me a line if you want the source code.

If you change the sources, please send the changes back to me. If lots of folks want to change this, then I'll find a public CVS to store it all.

Configuration File

There is a global file that contains all sorts of configuration parameters used by the program. Parameters are stored in a standard Java "properties" file format, which can be edited with any editor.

In a Windows system, preference file is stored in the logged in user's C:\Documents and Settings\ directory. So if you are logged into your workstation as joeblow, the path to your file would be

      C:\Documents and Settings\joeblow\musicutil.conf

On a Unix/Linux system, the files are usually in your "~" directory.

[Some Mac user could help me with where they live in OS-X]

Configuration for Amazon's XML interface and proxy connections

Amazon

Amazon.com has a lot of wonderful data about books and music. This data is accessable through their Web Services connection. Go buy something from Amazon.

So we give them a plug in case anyone in the world doesn't know about them.

The key to using their XML interface is that they want to know how is using it and how they can sell more books and music and stuff because they have the interface. So, before you can use it, you have to register. Registering is free, but you have to do it. And you need two kinds of registration numbers to use their data, an AssociateId (or AffiliateId) and a SubscriptionId. If you do not have an assigned AssociateId, you can use "webservices-20". For more informations, check with the Amazon Associates Website.

These numbers must be placed in your musicutil.conf file. The lines would look something like:

    AmazonAffiliateId=joesmoecom-20
    AmazonSubscriptionId=X21Q1RJ5PGN5GY

Proxies

The MusicUtil program is a simple browser, as one, it is occasionally handy to debug transactions using a debug proxy such as WidgetBuilder's DevProxy. All the development of this tool was built with DevProxy. If you want to turn on proxy support, you need the following line in your musicutil.conf file.

      ProxyWanted=true

     ProxyWanted=true
     ProxyHost=localhost
     ProxyPort=8000

In general, you do not have to do this, it is not needed for standard firewall proxies, only for debugging proxies. Of course, if you want to seriously configure the proxy settings, you are welcome to change the source code. That is what Open Source is all about.

FlacTestCommand

The 'flactest' program executes a copy of the flac program against the files in the directory tree. There is no portable way to specify the file, especially since it may or may not be on the path. So, you specify the command to execute with the FlacTestComnmand parameter. For example, on my Windows 2000 box, I use

  FlacTestCommand="c:\\program files\\flac\\flac.exe" -t
 

The quotes are needed because of the spaces in the file directory name, and the two reverse slashes allow proper quoting for the file delimeter. Other operating systems will require their own special voodoo.

DebugLog

There are four configuration parameters that contol the printing of verbose debugging log messages. These default to off, but can be handy when trying to figure out what is going wrong. The two parameters are:

1. LogFileSpec
2. LogLevel
3. DebugVerbosity
4. DebugProgressCounter

The first, LogFileSpec, specified the complete path to the log file name that will be written to, and the second is the standard Java Logging "level" value. See http://java.sun.com/j2se/1.4.2/docs/api/java/util/logging/class-use/Level.html The third controls additional debugging output, controlled as a boolean. The fourth, DebugProgressCounter, controls how often progress reports are made.

Example values would looks like:

     LogFileSpec=d:\\tmp\\music.log 
     LogLevel=ALL 
     DebugVerbosity=true
     DebugProgressCounter=10

The log levels range from ALL through SEVERE to FINE down to NONE.

Note: the normal Java logger will output some, but not all, log messages to the console (STDOUT). For finer levels of logging, look at the generated file.

Near trivial Flac to MP3 utility

New, low functionality utlity in Perl to convert flac files to mp3, called flac2mp3.pl

This is not rocket science, but since flac comes up as a vastly better compression scheme than mp3, and since some folks like mp3s to feed personal players, burn compliation CDs, etc. it is handy to convert from flac to mp3.

If you want to do just one, and then create the MP3s on the fly, it is easy, just run flac and lame together. The appropriate command is:

flac -dcs in.flac | lame --silent --tt "TrackName" --resample 44100 - out.mp3

Which is pretty simple, but doesn't do wild cards, and I never can remember the proper switches, etc. So I wrote a perl script to drive this command. Get flac2mp3.pl It handles wildcards and can send output to a specified directory. Its only about 10 lines of Perl, outside the documentation and argument parsing. No rocket science, but handy.

This assumes that you have perl installed and working. It comes on all known Linux/Unix platforms. For Windows, if you need it, get it from ActiveState.com just follow the download links. Free, and all that.

My utility does what I wanted, so the defaults are reasonable for me. You get the source, you can change them. There are switches to handle most things. And as with the Java MusicUtil code, it is designed for use as shell commands, so GUI users will not love it.

Default features:

• Accepts single files or wildcards including wildcarded directories
• accepts output file specificiation
• allows specification of "q" quality switch, defaults to "2" which is "pretty good and kinda slow"
• passes whatever else you want to 'lame' directly
• uses the flac file name as the track title for the ID3 tags

Switches

• help Print out handly help message
• ? Print out handly help message
• output specify a directory to write output to
• quality sets the "q=" switch to lame, 0 is best quality and slow, 9 is worst quality but fast encoding
• lame accepts any and all switches that are passed directly to lame

MusicUtil design points, features, known limitations, etc.

Right now, we have not sucessfully tested it with "Classical" music. The cover retrieval code works well with pop, rock, jazz and other classes that Amazon calls "Popular music." But they have a separate XML search for classical. It is easy to have the software use the internal genre field to decide whether to look in Amazon's "Popular" or "Classical" libraries. The problem is that I can't figure out how to get a selective query using the classical mode. I do not recommend using the "covers" option for classical music.

The scanner looks for a "asin.txt" file in the scanned directory. If it finds one, it skips that directory. So if you want it to find an updated or replacement cover, delete the asin.txt file.

When Amazon returns many possible matching records, we arbitrarily take the first one. This may be a bad idea. One idea would be to store all the record's title, artist and ASIN in the asin.txt file. It would be easy to do, would it be useful?

We do not process Microsoft WMA files.

The internal MP3 file ID3 tag parser does not currently implement Unicode text encoding. If the ID3 analyzer finds a ID3 Version 2 tag block, it will ignore any ID3V1 data.

Hints

I found that the thumbnails look better if I go to the server Settings -> advanced settings -> and change the thumbnail size to 50, which is the size of the Amazon thumbnails. That way the browser doesn't try to rescale them.

If you are having problems, i.e. your browser wants to display or execute the jar file rather than downloading it, right click on the link. Then select "save" or "save link to disk"

For some albums, Amazon returns a one pixel '.gif' file for the cover art .jpg file. The program properly changes the name to "cover.gif" but the image is pretty useless. So we delete the file and make a note in the asin.txt file.

Stoplist

Before keywords are sent to the Amazon engine, we process them through a stoplist of words that occur too frequently to help selectivity (e.g. "the", "disk", "disc", "various", "artists", and the digits "0" thru "9"). These words help prevent false exclusions, such as multi-disk sets listed as "Live (1 of 2)" or "Various Artists". Of course, this stoplist also makes it hard to find some albums, so see AmazonCovers documentation examples.

Known bugs

Dolf Dijkstra has found a serious bug when using FLAC files created with EAC that include ID3 tags. Never occured to me that you'd use ID3 tags when you can use nice Ogg tags, but that is what EAC lets you do, and my code does not process it correctly. It doesn't even process it at all. I can reproduce it, but just found it and it looks nasty.

Technical information.

Does this use XML? Yes, the communication with amazon.com is made using XML over HTTP.

Does it use SOAP or SAX? Nope. It probably should, but I could not figure out how to make either SOAP or SAX work. I figured that easy to install and configure software is better than the flexibility that the package solutions bring.

Do this use an existing ID3, FLAC, GIF or Ogg tag parsing library? No. I programmed the parsers from the RFC and RFC equivalent documents. This was done, again, in the interest of having a self contained package. Plus the parsers are small and pretty easy to implement, so the "code reuse" argument was not that strong.

Why don't you use a HTTP client library, such as http://jakarta.apache.org/commons/httpclient instead of writing your own HTTP client? Three reasons that I can think of, not that it is clear that any of them, or even all three together, is a "good enough" reason. First, because I wrote my own http client for another use and had the code handy and knew how it worked. Second, because I didn't know that the Jakarta httpclient code existed at the time. And third, because I want a trival installation process since I don't have an automated installation utility. Many very nice libraries require other libraries, and are a pain to install without something like YUM for RPMs. I know of no such tool for Java .jar files. So see the answer above for SOAP or SAX. I tried it, and it worked, but didn't seem to add any way cool functionality.

Testing and stand alone routines

The package is documented using standard JavaDoc. The generated HTML files are included in the pfarrell-latest.jar file.

The basic structure is that MusicUtils is the main program, it drives the rest when you want automated processing. Depending on the command given to MusicUtils, it will instantiate an appropriate processor object, which is recursively envoked by the DirWalk object.

Sometimes you want to run just the processing function once. Most of the processors contain a main() routine just for this testing. Execute the class without arguments and see the proper usage instructions.

The most frequently used test programs are AmazonCovers and AmazonTest. The AmazonCovers program displays the current working directory, as this directory is written to with the three cover files and the ASIN.TXT file.

After you run one of the above programs, type out the ASIN.TXT file. Look at it.

Bugfixes and releasess:

Properly ignore .txt, .jpg, .gif, and .png files in the song directory, as they are unlikely to contain good ID3 or Ogg tags.

Feb 06: Convert one pixel .GIF files to the proper extension, even though we asked for a .JPG file.

Feb 20: properly parse critical fields from ID3 Version 2 format tags.
Remove extra printout (left over from debugging).

Feb 21: properly parse critical fields from ID3 Version 2 format tags in both "revision 2" and "revision 3" data..

Feb 24: skip directory (and slow fetch of cover art) if there is a file called asin.txt in the directory. Fix handling of very short ID3V2 data blocks

Feb 28: Fix Ogg/Vorbis parser to read Ogg files properly.

Mar 3: Add logic to detect one-pixel gif files returned from Amazon, if so, delete the appropriate file, and write into asin.txt. Cleanly handle MP3 files with no ID3 tags at all.

Mar 4: add "noonepix" command (no one pixel file) that will do a search and destroy mission on one pixel files. Many of these may have been created by earlier versions of this utility.

Mar 7: add "--safe" option to MusicUtil and AmazonCovers programs, which will not overwrite existing .jpg files.

Mar 13: fix parser so "--safe" option doesn't cause MusicUtil to die ugly and painful deaths.

Mar 14: Add "findsmall" command, which locates music files that are too small to be proper. Fix scan and dup functions that were broken in some intermediate version. Fix read past end of file bug in FetchURL. mostly visible on Mac OSx.

Mar 20: Add defensive code to XML parser, trying to reduce complaints when Amazon is slow.

Mar 21: add "--c" switch to AmazonTest and AmazonCovers. Add list command.

April 11: minor bug fixes.

April 19: explicitly ignore GEOB block. Try to handle frame sized specied by RealAudio properly, altho they violate the ID3 standard specs.

May 26: teach Ogg/Flac parser how to read and understand UTF-8 encoded Unicode. We now properly pass the extended characters to Amazon. Note, this does not fix the problem with the SlimServer and SqueezeBox not handling non-US-ASCII in tags for display.

June 1: put in structure to allow testing of Jakarta HTTP client code.

July 19: remove Jakara HTTP client code. Add "flactest" command that will test all flac files in directory tree with the specified external command. Add debugging log printouts. Documented in configuration section.

July 20: added DebugProgressCounter so that periodic dumps can be made.

Sept 21: Change to match new Amazon aws-beta service. Beta quality release.

Sept 27: fixed defaulting on missing LogLevel parameter in musicutil.conf

Sept 28: don't require : "LogFileSpec" and related field in musicutil.conf file. Default rationally. Still todo, fix Lots o data problem.

Oct. 17: try to be nice about case in musicutil.conf file. We can only do so much...

Oct 31: finally implement code to use new Amazon webservices model. Change parser to use a real XML parser. Added appropriate XML package files to distribution. This is still pre-release code

Nov 6: first fixes of user-found bugs in new code.

• add two system printouts to test command, current user dir and java home
• removed dependency on javax.servlet.http.*
  

Nov 7: don't assume that /var exists and is writable. Use system temp directory.

Nov 13: fix null pointer exception when Amazon didn't find any records. Fix Null pointer when amazon didn't have all three image sizes. Write dump file to /tmp/musicusilyyyymmdd file rather than naming it only for the date. Note: if you don't have a tmp or temp definied, it may try to write to your root.

Nov 15, 2004: fix still more null pointer problems. Thanks SB...

July 30, 2005: Cleanup documentation, revive code from CVS.

Credits for debugging:

Thanks to the many people who have helped debug and improve this code. Special thanks goes out to Bradley Feldman.

Links

• Tips on ripping and managing a collection of audio files are in my tips page.
• Hydrogen Audio.org http://www.hydrogenaudio.org/
• Richard's software squeezebox, http://softsqueeze.sourceforge.net which lets you play on any PC, mac or Linux
• The SqueezeBox comes with a small remote control. It works OK to control the device. But I've got way too many remotes to want to add another. Now I'm learning about hacking remotes with JP1.
• Nightly latest and greatest: http://www.slimdevices.com/downloads/nightly/latest

Links

• Pat's general music pages
• The unofficial SlimDevicies Audiophile users SlimDevices SqueezeBox Audiophile FAQ aka How to kill your CD player and learn to love it
• Tips on ripping and managing a collection of audio files are in my tips page.
• Hydrogen Audio.org http://www.hydrogenaudio.org/
• Richard's software squeezebox, http://softsqueeze.sourceforge.net which lets you play on any PC, mac or Linux
• The SqueezeBox comes with a small remote control. It works OK to control the device. But I've got way too many remotes to want to add another. Now I'm learning about hacking remotes with JP1.
• Nightly latest and greatest: http://www.slimdevices.com/downloads/nightly/latest