Shazam for your computer

Many people use Shazam to identify songs they here in their local cafe or walking down the street, and its a great app for discovering new music, but I have also seen many people asking for Shazam for their desktop computer, now why would you want that you can hardly carry it around with you ?

Identifying Music You Already Have

Users understand that Shazam can be used to identify music on their iPhone and want a solution for all those badly labelled tracks on their computer but this is not what Shazam is intended for. Shazam is now available for your Mac and Windows 8, but Shazam requires you to physically play a part of each song in order to identify it, so is not a very good solution if you have more than a handful of songs to identify.

A Free alternative

Shazam actually uses acoustic fingerprinting to identify songs but forces you to play them via a microphone, SongKong also uses audio fingerprinting but this is done automatically without you having to play the song, fingerprinting a song only takes a few seconds.

SongKong can be used to identify as many songs as you like for free, simply look in the Song Changes section of the report to see what the song was matched to.

To keep things simple the screenshot shows the result of just matching a single previously unknown song but SongKong can be used safely on thousands of songs.

SongKong only requires purchase if you want it to add the matching song information to the file itself.

Rinse My Music Replacement

A few weeks ago I blogged about TuneUp Media going under and my suspicion that it was related to Gracenote being sold. I have not been able to confirm this but guess what, another Mac tagger Rinse My Music powered by Gracenote has gone under as well, coincidence ?

In this case the company itself has not gone under but Rinse My Music is no longer available and no explanation has yet been forthcoming. Existing customers will find the product ceases to work in a couple of months

So I would like to try to help Rinse My Music customers in the same way as TuneUp Media customers by offering SongKong for a 30% discount until the end of March. Simply make the full price purchase in the normal way then email paultaylor@jthink.net with some proof of your Rinse My music product code and I'll refund 30% of the price.

SongKong does everything that Rinse My Music does and more. But it doesn't rely on closed databases like Gracenote instead it uses MusicBrainz and Discogs. I'm an Indie developer so when you deal with Jthink you are dealing with me direct not having to go through layers of anonymous support.
 
There is a free trial available so you can comprehensively try before you buy.

Fixing files direct from iTunes

Originally SongKong was very much folder focused, you select a folder and SongKong would fix all the files and sub-folders of that folder. Customers generally went to two extremes, either picking their top level music folder and fixing everything or just picking an album folder and fixing one album at a time.

Now both of these methods are great but we have added a bit more flexibility

Firstly, you can now select multiple folders, if you have five folders under an artist folder now you can just fix two, three or four of the folders in one go instead of matching one at a time or the whole lot. And if the folders are located in different locations SongKong works out the basefolder/sub folder split for each combination as this screenshot shows

Secondly, instead of just fixing complete folders you can now fix files within folders. There are at least three  important uses of this

Fixing songs direct from iTunes

iTunes does not show folders only files, this was a problem but now files can be dragged directly from iTunes into SongKong.

Fixing results of a search

Now you can do a search in Finder or Windows Explorer and drag the list of matching files direct into SongKong.

Fixing Songs direct from Windows Explorer

Now all supported music formats can be fixed by just right clicking in Windows and selecting Fix Song in SongKong from the context menu

TuneUp Media ceases trading and Gracenote is sold by Sony

Without any notice TuneUp Media has ceased trading leaving their customers high and dry. Knowing that Tuneup had received venture capital investment my first thought was that the investors had pulled the plug on them because they were not making enough profit. However, it was announced a couple of days later that Gracenote had been sold to Tribune Media Services, more details here. Tuneup used the Gracenote service for its song matching so I suspect that under new ownership Gracenote has decided to withdraw that service.

I would like to try to help Tune Up Media customers by offering SongKong for a 30% discount until the end of March (previously February but now extended to March). Simply make the full price purchase in the normal way then email paultaylor@jthink.net with some proof of your Tuneup media license and I'll refund 30% of the price. There is a free trial available so you can comprehensively try before you buy.

Now I would like to reassure you that the problem that have you have encountered with Tuneup Media will not occur with Jthink software because of the following reasons:

1. Jthink is wholly owned by myself Paul Taylor, so there is no pressure exterted on me by external investors. My priority is to provide high quality, useful software at a fair price. JThink software is available with free upgrades and support, so you only pay once. Because I am the designer and developer there are no messy compromises in the design and I am able to be highly responsive to the specific needs of customers, and adapt products to a customers specific requirements.

Jthink has been in business since 2006, my comprehensive tagger Jaikoz is still actively developed and improved but I think the simplicity of SongKong is a better solution for Tuneup customers.

2. SongKong uses the open source MusicBrainz service rather than the closed Gracenote service. The complete database and web service is freely available and I have my own complete copy. Although I currently use the web service hosted by MusicBrainz I could provide my own Musicbrainz server if necessary so even if MusicBrainz was to die a death I could continue the service. MusicBrainz has its own protections in place to make it highly unlikely this could ever happen.

Some more important requirements of an automatic music tag editor

Some more important requirements of a rules based music tag editor

Since this post I realised I'd missed some other important requirements for a fully automatic music tagger

1. There needs to be an audit trail
If you are trusting some of the most important data on your computer then you really want to know what it is getting up to.
 
2. There needs to be a way to roll back changes
However good automated matching is there should always be the possibility to undo the changes
it makes.

3. There needs to be a way to remove duplicates songs
As your song collection gets organised you'll probably find you have duplicates, and finding and removing duplicates is something that automated taggers can potentially be very good at.

4. A way of continuously monitoring and modifying your music collection.
Once you are happy with your automated setup wouldn't it be nice to not even have to think about it anymore.

and this is how SongKong meets these requirements

There needs to be an audit trail

SongKong creates a comprehensive report with details of exactly what has been matched to the Musicbrainz and Discogs databases and also exactly what changes have been made to your files.

There needs to be a way to roll back changes

Every time SongKong makes changes to a file the changes are stored in a database, and because it is in a database those changes are not lost when you close SongKong or restart your computer. If at a later date you decide that you do not like the changes that SongKong has made you can use Undo Changes to change the files back to how they were before changes made by SongKong. This undo facility works even if the files have been moved or renamed.
 
There needs to be a way to remove duplicates songs

SongKong lets you find duplicates, and when a duplicate is found decide the criteria for which if the duplicates to be deleted. But what is a duplicate, luckily because SongKong stores ids when matching songs it can accurately determine when a song really is a duplicate. SongKong lets you choose any combination of Acoustic Id, Song Id and Album Id to let you decide what is a duplicate. For example if the same song appears on two different albums  then you may consider these as different songs or you may consider them the same song.

A way of continuously monitoring and modifying your music collection 

Once you have SongKong configured to your liking a good way for working with new music is simply to setup a new folder that you dump new music into and have SongKong detect this and do its thing, this is easy done using the Watch Folder option.

Rules Based Music Tagging

With conventional music tagging there is lots of manual editing or semi-automatic tagging but SongKong is a rules based tagger. The basic idea is that rules define how you want your music collection to be organized, then the rules can be applied to your whole music collection without any manual editing required giving a totally consistent music collection with the minimum effort on your part.  You can change the rules and then reapply to the whole collection to maintain a consistent music collection.

There are three main aspects to incorporating rules based tagging, lets summarize the perfect system:

1. It would be possible to correctly identify every file in your music
2. Once a song is identified the database it has been matched to would contain every desired attribute of the song that you are interested.
3. It would be possible to define rules to extract and apply the data in any way required.

And this is SongKongs implementation:

1. SongKong generates Acoustic fingerprints for each song and this can be looked up in the Acoustid database to identify the song, working in a similar way to Shazam. Acoustid currently contains fingerprints for 16 million songs allowing the majority of your songs to be identified, but Acoustid  can only identify the song not necessarily the album. However by combining this with comparing existing meta-data in your songs we can match to the album as well in the MusicBrainz database. MusicBrainz provides high quality detailed data onmore than 1 million albums.

You can influence the importance of metadata such as by specifying a preference of matching albums form certain countries of a preferred format such as CD or Vinyl.
 
So MusicBrainz/Acoustid provides good coverage for most music collections. But we also use the Discogs database as an alternative source.

Even so, we cannot guarantee 100% matching, but testing shows that in the majority of cases approximately 90% of a collection can be matched to MusicBrainz or Discogs.

It has to be said both of these databases do have better coverage for Pop/Rock/Electronic western music than Classical and World Music. But there are many projects ongoing to fill these gaps, for example the Music Technology Group at the Universistat Pompea Fabra is working with MusicBrainz to add Indian Raag Music.

The good news is that MusicBrainz open approach is fast becoming the de-facto standard database of music information.

2.  Some Music databases contain only the most basic information such as artist, album, title and year of release. But the MusicBrainz database consists of an incredibly rich semantic model, and allows new relationships to be defined between entities.

When SongKong matches to a song in MusicBrainz it is guaranteed to find at least 19 fields (such as artist, album ecetera) and often an additional 35 fields. This includes high quality artwork, usually at a resolution of at least 600 x 600 pixels.

These fields also includes some MusicBrainz Ids, this means that  your songs are compatible with other MusicBrainz enabled applications. It also means you can always look up the original source of the data at any time in the future.

3. SongKong is intended to be easy to use, but there is no one right way to organize your data. Everybody has different requirements based on their personal preferences and how and where they are going to play their music.

Here are a few scenarios handled by SongKong:

You are a DJ only interested in individual songs and has no interest in the album information. SongKong lets you specify exactly how files are named and stored, you can use any piece of metadata for deriving your filename ,and can use powerful Javascript expressions to manipulate the names.

Your music is already stored in iTunes. SongKong can work with iTunes automatically, updating the information in iTunes for songs that are already under iTunes control and adding songs that are not iTunes control to iTunes. 

You like to process songs in a pipeline moving them from unmatched to matched location. SongKong allows you to move files as they are matched to a new location, it also allows files that it failed to match to moved as well.

I hope this post helps you understand the basics of rule based tagging. Jaikoz also incorporates rule  based tagging , but also provides manual editing and semi-automated tagging.
 

Zap App Nap

Last night I ran SongKong overnight on a large collection of songs on my MacbookPro and I was surprised to find it hadn't finished, in fact it had got less through less than 10% of the songs !

Investigation of the logs files showed that SongKong had stopped doing anything 15 minutes after leaving it, and only restarted when i touch the keyboard in the morning.

Now in Preferences:Energy Saver I have the Mac configured to power off the screen after 15 minutes yet this shouldn't preventing processing, and it certainly used to work.

Turns out the problem is a new feature in OSX Mavericks called App Nap. Intending to save power it can slow down or halt applications it doesnt think are being actively used, and this seems to be the case once the Mac has gone into screen saver mode.

Thankfully it is easy enough to disable AppNap for any application.

Simply use the Get Info window for a particular application. Locate the application for which you’d like to prevent the App Nap feature from monitoring, right-click (Command-click) on its icon, and select “Get Info.” You can also highlight the app in Finder and press Command-I. The Get Info window will appear and you’ll notice a new option in the General section: “Prevent App Nap.” Check this box and your selected app will run at full power in the background.

Unfortunately this setting doesn't seem to be preserved when you reboot your Mac.

In the next version of SongKong and Jaikoz I expect to have a permanent solution.

The Components of the Filename

There is sometimes some confusion about how SongKong views filenames so I'm going to try and explain with an example. The starting point of SongKong is to select a folder to process - we shall call this the Root Folder. Then SongKong tries to guess what part of the folder path contains metadata such as Artist name or Album name and which part does not contain metadata. It then splits the folder path into Base Folder and Sub Folder with the Sub Folder containing the part matching the metadata and the Base Folder not.

Then as files within the root folder are loaded into SongKong they will all have the same Root Folder. They will all have the same starting Sub Folder but it will have any additional subfolders appended to it. The last part of the path is termed the Filename.

The root/sub folder divide is important because if your songs are renamed from metadata only the sub folder part is renamed from your metadata in contrast if your songs are moved only the root folder is changed.

Now an example:

Root Folder:/Music/Blur
Base Folder:/Music
Sub Folder   :Blur

Files within /Music/Blur such as

/Music/Blur/ParkLife/Girl & Boys.mp3

would be split as:


Base Folder:/Music
Sub Folder  :Blur/ParkLife
Filename      :Girl & Boys.mp3

Hope that clarifies things.

Moving Matched and Unmatched songs in SongKong in 1.17

As every song goes through the SongKongs pipeline the last stage is always the Save Song stage. At this point SongKong looks to see if the song has been matched to Musicbrainz or Discogs and if it has and the Rename files based on Metadata match is enabled then the Filename including the Sub Folder portion is then renamed according to the metadata and the mask, but if not matched it is not renamed.

Additionally you can move matched files to a new folder so instead of being rooted at the existing Base folder they are moved to the Move Folder if set on the File Moving tab.  Unlike previous versions of SongKong files can be moved, even if the Rename files based on Metadata match option is not enabled.

To reiterate the Move Folder changes the Base Folder, whereas Rename Files changes the SubFolder and Filename.
 
Using a Move Folder can be a very useful way of working when sorting out a large collection of music, a common way of managing your songs is to move songs from an unsorted folder to a fixed folder as they are identified.

New in SongKong 1.17 is the Move Unmatched Folder option allowing you to move files that are not matched. This is useful because it allows you to differentiate between a song that SongKong has tried to match and failed to match, and a song that SongKong has not yet attempted to match. It is especially useful when processing a huge collection that you might decide to fix in stages.

Although of course SongKong imposes no limits on how many songs you can match !

From Music Players to Automated Tagging

A new version of SongKong has been released today with improvements in all areas, and I thought it  would be worth giving a historical background of music tagging and its development as I see it.

First we had the Musicplayer just for playing CD's on your computer. Then as MP3 became popular and users started ripping their CD's as files on their computer music players such as Winamp became popular. Originally no meta data could be stored in MP3s at all but it quickly became apparent that as users could store files on their computer they need to store information about the files, and just using the file name was not sufficient.

So ID3v11 was added, but this was very limiting and so a new system ID3v2 was developed that was much more powerful. Music playing software added functionality to allow this information to be added but music players tend to work one file at a time, and this wasn't a very efficient way to add meta data, new software was required the Tagger !

I don't know what the first tagger was but there were plenty of mp3 taggers with very similar names that worked in a very similar way, they allow you to modify multiple fields and were just that bit more efficient than stand alone Music players.

Around this time the iPod and iTunes was born, this massively increased the amount of digital music people were storing on their computers.

We also had new formats such as wma, ogg vorbis and mp4, so now we needed taggers that could edit multiple formats in a way that hid the differences.

As people got serious about listening to digitised music lossless formats  became more important such as Apple Lossless and Flac.

As peoples collections got larger modifying all their metadata by hand became unsustainable, but help was on hand in the guise of online Music databases. The first one was freecdb that could tag your music if you had the original CD by looking up a digital fingerprint but this wasn't much good for songs that you already had on your computer.

But then new databases such as Discogs and MusicBrainz were created and these allowed Automated Taggers that could lookup information for songs and get meta data for those songs without needing the original CD if they just had some meta data.

Then with audio fingerprinting it was possible to lookup songs without any meta data at all as long as the song is already in the database.

Most taggers were cautious in making full use of this and provided options from online databases but user made the final decision about the best match but some such as Jaikoz made the choice for the user, although the user could adjust the criteria used to pick the best match.

But Jaikoz still provides manual tagging together with automated tagging, it provides the user with the tools to do what they want, but with so much choice it can be difficult to be consistent, and time consuming.

Now as the online databases have got larger and more accurate, users music collections have got larger, and users time has got ever more limited we see the emergence of the Fully Automated Tagger or Rules Based Tagger.

The idea is that rules define how you want your music collection to be organized, then the rules can be applied to your whole music collection without any manual editing required giving a totally consistent music collection. You can change the rule and then reapply to the whole collection to maintain a consistent music collection. This the premise of SongKong and in my next post I'll explain how SongKong attempts to reach that wholly grail of a totally consistent music collection we are not quite there yet, but getting closer.

Using Jaikoz to modify the Release Time field on iTunes

iTunes has a Release Date field but you cannot actually edit the field in iTunes, at least not for music files,there is also a Year field and this is editable but sometimes it would be useful to use both.

In Jaikoz the Year field is clearly displayed in the default configuration but for mp3s the Release Date field is also available. By using View/Show ID3 and selecting the ID3 Edit tab you can see the Release Time (TDRL) field and if you enter a value in here and save changes the change is reflected in iTunes Release Date field.

Behind the scenes Release Time use the ID3v24 frame TDRL, whereas Year uses TDRC.

New MusicBrainz Server release, time to update

Twice a year on a set date MusicBrainz do a server release that includes changes to their schema database. Changes to the database can be disruptive for MusicBrainz customers who keep their own copy of the database whicc is why these changes are limited to two releases a year.

MusicBrainz did such a release on October 14th, the major change is that the concept of places have been added so that venues and recording studios can be added to the database. For example once the data is added you could see what albums have recorded at a particular studio. Maybe some of this information will be used by taggers at some point, but until a music player exists that can use this information I don't see much point.

If you are using the MusicBrainz VM with Jaikoz or SongKong I would advise a little patience, within the week there should be a new VM updated to the latest database and search, this is easier than trying to update the server yourself.

The new MusicBrainz server temporarily broke tagging with SongKong and Jaikoz, it exposed a bug in both applications and I quickly released new versions of SongKong (1.11.0) and Jaikoz (5.7.0) to resolve this issue. The change has temporarily been reverted to allow the earlier versions of SongKong and Jaikoz to function but will be added back in 1 months time so please get on and update your versions to the latest version.



 

New Song Changes Report in Songkong 1.9

With SongKong 1.9 we completely overhauled the song changes report, here is an example of such a report Song Changes Report

The main aims were to increase its responsiveness for big reports, and avoiding information overload by breaking down the information into smaller segments. To achieve this pages are split down into smaller parts and Javascript is liberally sprinkled over.

But the report is still quite complex so this post we explain the different sections in a bit more detail.

Summary

 

This summarizes how many songs have been loaded and checked, how long it took and how many and what percentage have been matched to MusicBrainz, Discogs and if we could find artwork for them.

One thing to consider when matching is whether to rematch songs that have already been matched or whether to ignore songs that are already matched. By default SongKong rematches all your songs but if you disable Basic:Match all songs, even if already matched then all songs that already have a Musicbrainz Id are ignored and not considered in the following sections.

MusicBrainz Summary

 

MusicBrainz is the primary data source used by SongKong.

This section shows what MusicBrainz releases your songs have been matched to. In the left handside we group the matches by the first letter of the album artist, if the first letter is not a letter they are put into the * grouping. Expanding on one of the letter groupings, lists the artists starting with that letter in alphabetical order. Selecting an artist from the list shows how your songs have been matched to their releases on the right hand side of the report.

By default only details of the first matching releases are shown, you can open close details by clicking on the small arrow in the release header information. The header tells you the name of the release together with its MusicBrainz Id, Record Label and Catalog No if known. It all shows you how many of the total songs on the release have found corresponding matches in your collection.

The details provide a hyperlink to open the MusicBrainz page for the release. Below this we list all the songs that were matched together with the song artist, Disc No and Track No.

If you have multiple copies of a song then the same release may be listed more than once, we always match the most complete match first.

Note this section looks at the details of your songs and their mapping to MusicBrainz, if the were already matched to MusicBrainz but not matched during this run of Fix Songs they will still show in this section.

Discogs Summary

This works in the same way as MusicBrainz. Songs can be matched to Discogs either by a direct search to Discogs or by a relationship existing from a MusicBrainz Release to a Discogs release. But Discogs is a secondary data source and Discogs matching and searching can be disabled, if both options are disabled in the Match tab then this section will not be shown.

Song Changes

 

This section shows what changes have been made to your files and is a listed as a file hierarchy. Each leaf folder can be selected and then the changes for each file in the folder are shown on the right handside of the report.

It is important to realize there is not a tight correspondence between songs matched in MusicBrainz/Discogs and Song Changes for the following reasons.

1. If the song was already matched to MusicBrainz and then rematched again to the same release they may not be any change required because the song already contains the fields matched.
2. Most songs can have Acoustids created, these are stored in the files. So even if no match was made to MusicBrainz or Discogs the song may still require saving
3. If the fiename mask has changed then the file will probably need to be renamed even if no metadata added to the files themselves.
4. If Fix Songs is canceled before its has completed songs may have been matched to MusicBrainz but interrupted before the changes could actually be written to file.

The list of changes made to a file is split into

1. Fields Added
2. Fields Modified
3. Fields Deleted

Songs Not Matched

Selecting this list all the songs that are not matched to either MusicBrainz or Discogs in the right hand side using a file hierarchy view.

Songs Not Matched to MuscBrainz

Selecting this list all the songs that are not matched to MusicBrainz in the right hand side using a file hierarchy view.

Songs Not Matched to Discogs

Selecting this list all the songs that are not matched to Discogs in the right hand side using a file hierarchy view.

SongKong 1.8 with new performance status bar and improved Virtual Machine performance

Whilst SongKong works quicker when using a MusicBrainz virtual machine I confess it was not working quite as quick as I hoped, and I didn't know where the problem lay so I added some performance statistics to the new SongKong taskbar in SongKong 1.8.

MusicBrainz Delay

The MusicBrainz Delay records the additional delays added to SongKong to prevent more than one query per second being sent to MusicBrainz inline with its terms of service. When using a virtual machine the delay should have been zero, but this was not the case. This uncovered an issue that rate limiting was only disabled when doing MusicBrainz lookups, but not for searches, fixing this problem boosted performance

MusicBrainz / AcoustId / Discogs Queue

SongKong uses a number of online databases for matching songs, not just MusicBrainz. All these databases have terms and conditions that restrict usage to one query per second so if SongKong is ready to send queries more quickly than that we have to queue the tasks. This status bar shows the queue for MusicBrainz, AcoustID and Discogs databases and is updated every thirty seconds. The most important queue is the MusicBrainz queue and this can be eliminated completely by using the local MusicBrainz server.

Memory Usage

When SongKong is started the maximum memory it can ever use is specified, this is the second value displayed and SongKong can never use more memory than this regardless of how much memory your computer actually has. Whereas the first value is how much of that memory SongKong is actually using. Note that SongKong uses a garbage collector to release memory that is no longer required and this is called periodically, if SongKong is nowhere near max memory usage it tries to recover memory less often than it could, and so the current memory usage may be higher than SongKong is actually using.

On Windows you can modify the maximum memory allocated by editing SongKong.ini (32-bit) and SongKong64.ini (64-bit)

On OSX you can modify the maximum memory allocated by editing /Applications/SongKong.app/Info.plist

On Linux you can modify the maximum memory allocated by editing songkong.sh

Other Fixes in SongKong 1.8

If you had the Update Genres option enabled it could cause Fix Songs to cancel prematurely, this is noe resolved.

If you removed all countries from Preferred Countries so that it was an empty list this will prevent Fix Songs from running

Faster SongKong matching with the new Musicbrainz VM

To speed up matching it is possible to configure your own local MusicBrainz servers instead of using the official MusicBrainz servers and use this with SongKong. This avoids the requirement to only send one query per second to the MusicBrainz server so speeds up the matching that can be achieved by SongKong,

Additionally this resolves an issue when trying to use multiple copies of SongKong at the same time in the same home network. Usually with a home network external services such as MusicBrainz see all your computers via the same ipaddress, and therefore the rate limiter is applied to all computers as if they were one computer, this effectively prevents you using multiple copies of SongKong on a home network in parallel. But if the server being used does not impose rate limiting then there is no longer a problem.

But the problem has always been the difficulty of setting up a MusicBrainz server in the first place !

Now at long last MusicBrainz have now released a new Musicbrainz VM containing both Musicbrainz Server AND Search Server I've been trying it out and it works great !

The latest official MusicBrainz instructions are available from here. but in this blog we document the instructions so that they are more relevant to using the Virtual Machine with SongKong as follows. Warning some of the steps can take a few hours to run but the procedure is very simple.

1. Ensure you have a BitTorrent client such as Vuze installed, this is required to download the Virtual Machine, and is a much more reliable way to download large files.
2. Download the 9GB MuscbrainzVM via your BitTorrent client
3. Windows/Linux user can download and install from VmPlayer this is free and more reliable then VirtualBox
4. Mac users can use Vm Fusion, although unfortunately this is not free, or try the less reliable VirtualBox
5. The remainder of these instructions assume you are using VMPlayer, but the steps for VMFusion and VirtualBox are much the same.
6. Start VMplayer and choose Import Virtual Machine from the File menu. Select the downloaded file.
7. Select Options and increase Virtual Memory to at least 2GB if your machine is powerful enough
8. Select Play Virtual Machine
9. Log in on the console using the username vm and password musicbrainz
10. Update the database with the latest data added since the VM was created using bin/replicate now this may take a while depending on how old the VM is.
11. Build search indexes, this will take a few hours bin/reindex
12. Get the IP address of your virtual machine.using command:ifconfig | grep eth0 -A 1
13. You can check it is working in browser by pointing to port 5000 of the ipaddress found previously e.g. http://10.1.1.104:5000
14. Configure SongKong, set Server:MusicBrainzServer and Server:MusicBrainz Search Server to the same address e.g. http://10.1.1.104:5000
15. Now SongKong will use the local MusicBrainz server rather than the offical MusicBrainz server. This should be faster and more reliable.

What is the point of the iTunes Copy files to iTunes Media folder when adding to library ?

SongKong 1.7 now has support for updating for iTunes on OSX, so it has feature parity with SongKong on Windows, and playing with this got me thinking about the point of certain iTunes options.

Copy files to iTunes Media folder when adding to library

This option is enabled by default and if this option is enabled whenever adding songs to the iTunes library it will make a copy of every file and add it to iTunes media folder. So you end up with two copies of every file, one in its original location and one stored in the iTunes media folder.

Why would you want two versions of each file ?

 

Managing multiple copies of files is an unnecessary headache,  maybe there is some concern that iTunes will modify the file in some negative way but it seems unlikely.

If you have this option enabled subsequent modifications to the original file within SongKong will have no effect on the file used by iTunes because iTunes will now be using the copied file in the media folder UNLESS you rename or move the file then iTunes will see it as a different file and add it again !

Keep iTunes Media folder organized

Also enabled by default, this renames files based on their metadata once they are added to iTunes if the copy files options is enabled.  But this only renames the copy of the file, it doesn't help you with the original file if it is misnamed. And more importantly you have no say on how your file is renamed, iTunes renaming mask is rather simple and probably not what you really want to use.
 If you are using SongKong flexible rename mechanism why would you use iTunes limited filename and you probably don't want to iTunes to rename them differently as they are copied to the iTunes media folder anyway.

And if you are using SongKong to modify files already within the iTunes media folder you certainly dont want iTunes moving and renaming files as they are getting matched ?

Although you can use both options with SongKong I would recommend disabling both.

Better track matching when submitting new releases to Musicbrainz

For some time Jaikoz has allowed you to submit new releases to Musicbrainz, the selected songs in Jaikoz are used to open the MusicBrainz Release Editor with all the details filled in, so that you just need to check the details and then submit the form.

But in MusicBrainz recordings can be linked to multiple releases so Jaikoz tries to identify when a recording already exists instead of creating a new recording. Doing this makes Musicbrainz more useful so that in an ideal world every recording listed in MusicBrainz is actually a unique recording, and we can see all the releases that the recording is based on.

Jaikoz did this by comparing track titles and durations but sometimes recordings were incorrectly linked when in fact they were different versions of same song.

A more accurate solution

Now with Jaikoz 5.5.0 we have replaced this by matching using Acoustids, only Musicbrainz recordings with the same acoustId as your track are potential candidates, and only then do we do further checks on title and duration to ensure a good match.

Solving a bug in iTunes that prevents artwork being displayed

Occasionally I've seen artwork that displays okay in most applications failing to display in iTunes or the Finder within OSX, recently I had a nice example of this issue and looked into it further.

How do mp3s store text ?

Now mp3s store data in fields called frames, and many such frames allow the text encoding to be chosen. The default is ISO-8859-1 character set but this isn't very good for storing non-latin text so you can also store the text using UTF-16, however the downside of this is that two bytes are used instead of one for each character so it is usual to use this unless you need to. Mp3s use the APIC frame to store artwork, and this includes the text encoding field

Why do you need a text encoding field for storing artwork ?

The APIC frame has a description field allowing to describe the artwork, this is rarely used but you can use it. In Jaikoz if you click on the edit artwork dialog you can add a description or change the type of the artwork.

What has this got to do with iTunes ?

Even if the text encoding field is set to UTF-16 the artwork will not display unless the description text is actually encoded as ISO-8859-1, this is a bug in Tunes

How does this manifest itself in Jaikoz ?

If you change the default text encoding to UTF-16 then Jaikoz will use this for every frame in the file whether you need it or not causing a problem for your artwork

So what can I do?

In Jaikoz 5.5.0 we use text encoding of ISO-8859-1 for APIC frames regardless of the value set for the text encoding, no other frames are affected. The only downside to this is that if you want to add a description for your image, and you add one that cannot be encoded in ISO8859-1 the description will not be saved.

How do I fix existing files ?

Just open them in Jaikoz 5.5.0 and use the File/Force Save option, this rewrites all the fields in the file even if they haven't changed so they are rewritten with the current preferences.

Updates iTunes from SongKong 1.6 for Windows

Despite its shortcomings iTunes is a massively popular music player, not least of all because its the official way to move music from your computer to your iPod. Windows users of SongKong can now get SongKong to update iTunes with any changes to your music library made by SongKong, this includes the following situations:

When the metadata of songs already in iTunes are modified by SongKong.
When songs already in iTunes are renamed or moved by SongKong.
When SongKong matches a file not in iTunes , the file is automatically added to iTunes.
When duplicates are deleted by SongKong, the deleted files are removed from iTunes.

Jaikoz has had this support for some time, but because of various bugs in iTunes it does not always work 100%. So for SongKong I rewrite it from scratch and have created a more robust solution, SongKong completely ignores the iTunes.xml file because this can get out of date or corrupt and instead talks exclusively to iTunes itself, this is much more reliable.

Why Windows only ?

ITunes integration is one of the few area of SongKong that we have to use different technologies for different platforms. For Windows we use COM, but for OSX we will use Applescript, it made sense to get it fully working in one technology first and then move to the other, support for OSX will come in SongKong 1.7.

New Save Artwork to Filesystem option in SongKong 1.5

Pleased to announce the release of SongKong 1.5, this includes the ability to the save artwork to the filesystem.

Fix Songs:Artwork Options:Save Artwork to Filesystem

Artwork is automatically always embedded in your music files, this is the most portable solution because you just need to copy your music files from your computer to another computer or iPod and your artwork will still be available. However it may be useful to save your artwork to the filesystem as separate image files as well for various reasons.

1. On Windows it looks for a file called folder.jpg or folder.png in a folder and if found uses it as an image for the folder itself.
2. Saving your images as separate files allows you to manipulate them in paint and photo editing applications.
3. Some Music players require artwork to be stored separate to the music files themselves to work properly.

Now folder is the default name of the images created by SongKong for compatability with the Windows but you can change this by altering Fix Songs:Artwork Options:Saved Artwork filename to anything you want. What you cannot do is specify a file type such as jpg or png. Images are output as the same format as the embedded artwork, these are typically the well supported jpg or png formats.

Better AcoustID matching in Jaikoz 5.4.0

How AcoustID works

When a song is analyzed and the fingerprint is submitted to AcoustID the AcoustID server returns an existing AcoustID if the fingerprint has been submitted before, or if the new fingerprint is so similar to an existing fingerprint - alternatively it allocates a new AcoustID for this fingerprint. When songs are matched to Musicbrainz by Metadata you can submit the fingerprint/MusicBrainz Id pairing to AcoustID and it will link the AcoustIDs to the MusicBrainz recording.

A new Discovery

But I discovered something new about AcoustID this month, a fingerprint can sometimes be associated with multiple AcoustIDs ! But only one AcoustID is a good match , the others are AcoustIDs that better match different but similar fingerprints. When a fingerprint/AcoustID pairing is submitted to AcoustID it is only assigned to the best primary AcoustID.

Fixes in Jaikoz 5.4.0

When Jaikoz was looking up the AcoustID for a fingerprint linked to multiple AcoustID it was actually keeping the worst matching AcoustID rather than the best one, this was usually one that was not linked to a MusicBrainz recording and hence not very useful. This has now been fixed in the new release of Jaikoz and will improve the AcoustID matching that Jaikoz does.

Fixing your AcoustIDs

This means that using earlier versions of Jaikoz you may have songs in your collection with the correct fingerprint but the incorrect AcoustID. Whilst this is not a major issue because the AcoustIDs are only a mechanism used to get matches yo may want to fix as follows:

1. Ensure the AcoustID column is visible in Jaikoz
2. Right click on the header and select Empty Column
3. Run Retrieve AcoustIDs
4. Save any files that have changed

Better artwork matching in Jaikoz 5.3.0

Artwork has been improving steadily in Jaikoz over the last year, especially since the Cover Art Archive was introduced but with hew version of Jaikoz things have got even better.

First some terminology, within MusicBrainz the concept of an Album is represented by a Release Group, then different versions of that release group are known as Releases. A release is the thing that we can actually buy and often varies slightly between countries. So a release group often contains different releases because it is released in different countries with slightly different tracks. Discogs also uses releases and has similar concept of release groups, known as Masters

Until this release Jaikoz checked the matching release for artwork on both the Cover Art Archive, Amazon and Discogs but just for the matching release. But now if we find no match we then check all the releases within the same release group and Discogs masters, artwork is usually the same for all releases within release group but often MusicBrainz does not contain artwork links for every single release in a release group so this change gives a big improvement.

I ran a test over 1000 songs , and matching artwork went up from 782 to 960 !

Full list of changes in this release can be found here

Introducing functions in rename masks

An important aspect of tagging your songs is renaming your songs so that they match the metadata within the song. Although most music players depend only on the metadata within the song itself, we all have different preference of how to name our files and this helps us to keep them organized. Some applications require the filename to be in a particular format.

Jaikoz used to have quite a simple language for the rename mask that allowed most metadata to be used in the filename but only provided simple if statements. Then this was replaced with support for full Javascript syntax, this allows you to do pretty much anything but creating the correct mask could be long winded and error prone.

Consider this simple mask

 (album.length>0 ? album + '-':'') + (trackno.length>0 ? trackno + '-':'') 
 + (title.length>0 ? title:'')

Although its fairly clear what it is doing. its very fiddly and easy to make a mistake when editing. Unfortunately as a number of you have commented the Javascript is not as easy to understand as the old syntax, I traded this for the power the Javascript notation gives us.

However recently discovered there is nothing to stop you defining functions within your filename mask so the above can be replaced by

 function ifnotempty(value,sep){
     return value != undefined && value.length > 0 ? value + sep : '';
 }
 ifnotempty(album,'-')+ ifnotempty(trackno,'-') + ifnotempty(title,'') 

As you can now see the complexity is put into the function once, then the resultant mask can be much more concise.

In Jaikoz 3.6 we have made use of functions to provide better default masks. Of course if you are upgrading Jaikoz it will keep your existing mask so you will not see them so I have posted them below.

Folder Rename Mask

 function ifnotempty(value,sep){
     return value.length > 0 ? value + sep : '';
 }
 
 function ifnotempty2(value1,value2,sep){
     return value1.length > 0 ? value1 + sep :value2.length > 0 ? value2 + sep:'' ;
 }
 
 ifnotempty2(albumartist,artist,folderseparator) + ifnotempty(album,folderseparator)
 + (disctotal>1 ? discno  + folderseperator : '') 

Filename Rename Mask

function pad(number, length) {
      var str = '' + number; 
      while (str.length < length) {  
          str = '0' + str;
      }
      return str;
  }
 
 function ifnotempty(value,sep){
     return value.length > 0 ? value + sep : '';
 }
 
 function ifnotempty2(value1,value2,sep){
     return value1.length > 0 ? value1 + sep :value2.length > 0 ? value2 + sep:'' ;
 }
 
 ifnotempty2(albumartist,artist,' - ') + ifnotempty(album,' - ') + 
 ifnotempty(pad(trackno,2),' - ') + title

In future versions of Jaikoz I'm going to add the ability to create and store custom javascript functions with a name to make this even easier, but hopefully in the meantime the use of functions should make creating masks simpler.

Why you should consider the database your tag editor is using to fix your songs

When you use a tag editor that automatically fixes your metadata you should really have a think about where that data comes from.

Photo by Paul Downey
Openness

There are a number of databases containing music information but some of these such are closed, this means you cannot actually browse the data independently never mind correct errors in the database. Often applications that use these type of database keep the identity of their source database hidden.

Accuracy

Sometimes the data is not very accurate. this may be because a database is not being used at all instead the tagger may just trying to hack information from websites and other sources

Linking

An online database can be used to correct your song metadata but for you to take adavantage of a myriad of other interesting applications you need to store a unique id that accurately identifies your song so you dont have to fuzzy match it again.

Metadata and Acoustic Matching

Matching songs by metadata is good but matching by the actual audio can be even better, but you need both: matching by Acoustic Id might identify the correct song but if the song exists on multiple releases it cannot help select the correct release.

The Jaikoz Solution
Openness

The primary database used by Jaikoz is MusicBrainz, MusicBrainz is a completely open database run by the non profit Metabrainz Foundation that can be edited by anybody with a free account. Jaikoz also uses as secondary source Discogs another online database that allows the data to be browsed and edited and freely downloaded.

Accuracy

MusicBrainz has comprehensive guidelines on the editing process and voting system to ensure the quality of edits and to protect the database from poor edits. Discogs also has guidelines and only allows adding of releases that you own, both systems expect the editor to provide another source for their edits.

Linking

MusicBrainz provides unique identifiers for songs, releases and artists (plus some others) and these identifiers are increasing being used by other sites such as Last.fm and the BBC,in fact Musicbrainz Artist Ids are fast becoming an industry standard. Discogs does not have such a sophisticaed system but when a release is matched to Discogs we store the the Discogs Release Url in your songs metadata.

Metadata and Acoustic Matching

Jaikoz uses AcoustId in addition to metadata matching to ensure the best possible match. Acoustid is yet another open system that you can contribute songs to and you can also flag when an acoustic id seems to be assigned to an incorrect song.

Overall at Jthink we are committed to using the best sources available.

Why Google gains market share

When I released Jaikoz five years ago (!) Paypal was the sensible way to go for processing payment. A few years later and some potential customers did not want to use Paypal so I added Google Checkout as well, but I didn't have time to fully automate the Google process.

Yesterday I decided to remedy this, the API and documentation for Google Checkout is way better than Paypal. Both are comprehensive and easy to use and I got things working within the day. I went back to the Paypal website to see what they are offering now, there was now a Java API but it achieves the dubious honour of being both limited and totally confusing.

Google also takes a smaller cut than Paypal so if you are buying a Jaikoz license please use the Google method.

So my point is integrating with Google is easier than other solutions, and this is one reason why I think Android will overtake iPhone within the year

Thanks to Brandon Schauer for the Photo above

Jaikoz is Global

Photo by Börkur Sigurbjörnsson Jaikoz had sales in 76 different countries last year:

Andorra,
Argentina,
Australia,
Austria,
Bahrain,
Belgium,
Bermuda,
Bolivia,
Brazil,
Bulgaria,
Canada,
Cayman Islands,
Chile,
China,
Colombia,
Costa Rica,
Croatia,
Cyprus,
Czech Republic,
Denmark,
Dominican Republic,
Ecuador,
Estonia,
Finland,
France,
Germany,
Greece,
Hong Kong,
Hungary,
India,
Indonesia,
Ireland,
Israel,
Italy,
Jamaica,
Japan,
Kuwait,
Latvia,
Lebanon,
Lithuania,
Luxembourg,
Malaysia,
Maldives,
Mexico,
Mozambique,
Namibia,
Netherlands,
Netherlands Antilles,
New Zealand,
Norway,
Paraguay,
Philippines,
Poland,
Portugal,
Romania,
Russia,
Russian Federation,
Saudi Arabia,
Singapore,
Slovenia,
South Africa,
South Korea,
Spain,
Sweden,
Switzerland,
Taiwan,
Thailand,
Trinidad and Tobago,
Turkey,
Ukraine,
United Arab Emirates,
United Kingdom,
United States,
Uruguay,
Viet Nam,
Yugoslavia,

AcoustId Fingerprinter

A new fingerprint database tool is being developed by long time MusicBrainz contributor
Lukáš Lalinský a little like AmpliFIND that Jaikoz may use in the future but currently its database of fingerprints needs to grow and for that they need your help. Please try out submitting fingerprints of your collection. its very simple to use. and can be downloaded from http://acoustid.org/fingerprinter

Thanks to Karen Cox for the photo

Jaikoz 4.0 NGS Beta now available

The Beta release is now available http://www.jthink.net/jaikoz/jsp/beta/start.jsp configured to run against the Musicbrains NGS Test Database. This test database is configured just like the production database and has already undergone alot of testing so it should be very stable. But this has meant a delay in it becoming available and the full release of Musicbrainz is less than two weeks away, so if you want any more changes to Musicbrainz best to get in there quick.

One of the exciting new features of Jaikoz is the ability to be able to add new releases to Musicbrainz. But remember this is only a test database so only add new releases for the purpose of testing, they will be lost when Musicbrainz goes live.

Whats going to be in Jaikoz 4.0 ?

With 3.9.0 out the door its time to think about the next release. I'm hopeful that the long delayed Musicbrainz NGS will be released soon, and if that is the case Jaikoz will be updated to use the much improved webservice to give better and faster results. Musicbrainz NGS is a massive Musicbrainz release so I think it fitting that Jaikoz gets a major release number to reflect this.

Also Musicbrainz NGS provides an way to semi-automatically add new releases. So within Jaikoz you'll be able to select a group of songs and select 'Add to Musicbrainz' . this will open up a Musicbrainz webpage with everything filled in, so you just have to check the info,add in anything extra you know and okay it. This is another way to really help improve the Musicbrainz Community Database and continue to provide a via alternative to closed databases such as Gracenotes.

Itunes is still the way the majority of users play their music so I'm going to make some further improvements to the iTunes integration to better handle when iTunes is set to copy and re-organise your music files. Windows users have not been been able to update iTunes using Java 64bit because of a problem with the Java COM library I use but I have found an alternative so I will rewrite the Windows iTunes support to use this instead.

Jaikoz in the dock or toolbar doesn't react to having files dropped on it, but this is now possible to do.

Manual Tag from Musicbrainz is due an overhaul and I'm expecting to replace this with a more album orientated one.

There has been a long running debate on shortcut keys. My feeling is that I'm going to simplify shortcut keys so that they always only work on the selected items, whereas currently there is one combination to apply to the selected item and another to apply to all visible in the window. Because the resulting shortcut keys will require less keys, and there will be less shortcuts to remember I think this is an improvement. It will be easy to work on all songs in the edit view by use of Cmd-A/Cntl-A or newly added Select All option. I'm also going to look again at customizable shortcut keys and toolbars.

When matching from AmpliFIND we only take the Acoustic Id, trouble is that if that Acoustic Id is not known to Musicbrainz and your song has no other metadata we cant get a match. AmpliFIND actaully has other basic data such as artist and song title so we should use this if our song doesn't contain the info, this might be enough to do a metadata match against Musicbrainz.

So that is the thinking at the moment

Songbird buggy and they dont seem to care

In Jaikoz 3.8.3 a user was complaining that Jaikoz couldn't open their mp4s. Turns out the problem was that they had been edited in Songbird, and Songbird save track no differently to every other mp4 player I ve tried.

I added a workaround to Jaikoz and raised a bug with the Songbird team.

A little time passed, they linked it to another issue which wasn't the same problem and then they just closed it. I complained and reopened it because nothing had been fixed, and now they have downgraded it to a suggestion for a future release. Because Apple have not written a nice neat specification, it is acceptable for the Songbird developers to ddclare no bug. I really hate it when developers bend over backwards to try and show there not at fault rather then trying to resolve the issue.

Songbird 22966

New fields added to Jaudiotagger

Jaikoz uses my open source Jaudiotagger library for reading and writing tags, so if I want to add a new field to Jaikoz it has to be added to jaudiotagger first.

Ive just added a bucket load of new fields that should make their way into Jaikoz.

CUSTOM1,
CUSTOM2,
CUSTOM3,
CUSTOM4,
CUSTOM5,
FBPM,
MUSICBRAINZ_RELEASE_GROUP_ID,
MUSICBRAINZ_WORK_ID,
OCCASION,
ORIGINAL_ALBUM,
ORIGINAL_ARTIST,
ORIGINAL_LYRICIST,
ORIGINAL_YEAR,
QUALITY,
SCRIPT,
TAGS,
TEMPO,
RATING,
PRODUCER,
MIXER,
DJMIXER,


If there is some field you really want that isn't currently in Jaikoz let me know.

iTunes can now export m3u playlists !

Smoother integration with iTunes is always something I endeavour to achieve, but iTune's doesn't make it easy. One longstanding problem was that iTunes playlists could only be exported in iTune's proprietry format which could not be read by Jaikoz.

I was looking at plugging this gap today, but whilst playing with iTunes 10 I noticed that when you save a playlist that there is now an option to save the playlist in m3u format



So the problem is solved because the contents of a m3u playlist can be opened in Jaikoz either using File/Open Playlist or by a simple drag and drop.

Now it would be really great if you could just drag a playlist directly from iTunes to Jaikoz, but that is not possible at the moment because of a bug in iTunes drag and drop support.

Matching to a different song version

Progress on the new version of Jaikoz with release orientated matching is progress well but I came across an interesting problem today.

As well as release matching, Jaikoz has the 'Prefer Do not match to Single Artist Compilations', for those of you who want songs linked to the original release, even if you don't have the original release. I was testing this with songs from Madonnas Immaculate Collection and I was always failing to match songs such as Holiday to the original album Madonna. Turns out it was because the track length differs by a whopping two minutes, 6.11 seconds for the original versus 4.06 and Jaikoz rejects song lengths that vary by more than 15 seconds (10 seconds in current release)


So the songs are clearly different version, and as a general rule we wouldn't want to match them, but I wonder if some of you actually would want to match to this earlier version regardless that you have the later version ?

The best duplicate song deleter around ?

Most of you try out Jaikoz for getting more metadata for your songs, but one of its less advertised features is duplicate song deletion.

The way Jaikoz find duplicates is very powerful because it uses the sonic matching of AmpliFIND and the expert metadata matching power of Musicbrainz. It is format independent so for example it could recognise that a song encoded as Flac and the same song encoded as Mp3 are the same song.


Once you have matched all your songs you have three choices for identifying duplicates.

1. Matching songs with the same Musicbrainz Id
These songs have been matched to the same release, so are very likely to be a duplicate

2. Matching songs with the same AmpliFIND Id
These songs should be sonically the same, but you may not want to delete them if you want to maintain the same song on two different releases. For example if the same song is in your collection on an original album and a compilation, you may want to keep both.

3. Matching songs with e same Musicbrainz Id AND AmpliFIND Id
These songs have been matched to the same release AND are sonically the same so are almost definitely a duplicate

This choice is made at Preferences/Local Correct/Delete Duplicates, and you can then run Delete Duplicates from the Action menu or as part of the Autocorrecter.

You can also just show potential duplicates and then make the final decision yourself using the filters in the Filter Menu.

The Preferences also let you specify whether to actually delete, move or link to the duplicates

Better matching in Jaikoz 3.4.2

Automatic matching is difficult but I've made a number of changes to improve the matching in Jaikoz in the latest release.

Jaikoz searches for possible matches in Musicbrainz then rescores them taking additional information into account to find the best match, it does this because an original Musicbrainz score only takes into account the search terms when scoring but we need to consider more values. For example we do not specify a duration in a search because some songs do not even have a duration within Musicbrainz so would never be returned by a search, but having got some potential results we want to give a higher score to those with a duration that matches the original song. Musicbrainz uses Lucene for searching with its own custom analyzer for deciding which songs are returned by a search and this latest release of Jaikoz uses the exact same analyzer to ensure scoring is compatible. This is one advantage of working on both Musicbrainz and Jaikoz !

When searching for a track we now consider more variations of the name because songs entered into Musicbrainz are normalized , for example We Have Explosive (Pt. 5) should be entered into Musicbrainz as We Have Explosive, Part 5 but they might not have been. This normalization is detailed in the Style Guidelines and In Jaikoz we now check for the title as it appears in your metadata and also as a normalized version as far as possible.

We also make workarounds for common errors in entering data. For example Musicbrainz Issue #5538 shows that users usually enter song titles as 'No. 1' , but in a large minority of cases enter 'No.1' , Jaikoz workrounds this issue.

Cluster Albums finds albums by artists with the same name but a different Release Id and tries to move the songs so that they are all on the same Release Id, note this is different to what 'Cluster' means in Musicbrainz Picard and perhaps I should have called it something different. Previously it did this by matching title against title for each Release Id being used, and picked the Release Id which had the most matches but now this has been improved. Firstly we use fuzzy matching on the title allowing for normalization as explained earlier. Secondly if all but a couple of tracks are successfully matched to one Release Id we allow matches on Acoustic Id and song length to shoehorn the remaining tracks into a potential release. This is really useful when the same song exists on two albums but is radically renamed between the two.

Musicbrainz Developments

I went to the yearly Musicbrainz summit in Nurenberg a couple of weeks ago, a good time was had by all , and the old town is certainly a lovely place

We discussed future developments, and their plenty of good news for Jaikoz users.

Musicbrainz were very happy with the work I'd done rewriting the existing search system and I am the main developer of the search for the new NGS release being developed so I'm able to understand how search works in Musicbrainz and improve it to suit Jaikoz better. This should ensure you get better results from Jaikoz then any other other tagger.

NGS has lots of new features including recognising the same recording on different albums, and handling multiple disc release better - this will help with getting better matching.

Work is going to be done on creating a genre system from the existing folksonomy cloud so this will help with a field that is currently quite poorly served by Musicbrainz. Knowing the attention to detail of Musicbrainz editors I'm hopeful that in time Musicbrainz many create the definitive genre list.

Historically Musicbrainz have been very cautious about having APIs to allow data to be added in to Musicbrainz. Sensibly they do not want it to end up a mess like freecddb but they have warmed to the idea now and I think this is a neccessary move to keep on top of the exposion in music being recorded. So in the New Year Im going to have a think about taking advantage of this loosening up of data entry.

And Musicbrainz have just hired their first fulltime developer Kuno Woudt , a well established Musicbrainz editor and developer so this should speed up Musicbrainz development.

New version of Jaudiotagger

I've just released the new version of Jaudiotagger, the tagging library used by Jaikoz and quite a few other applications. I've been happily removing code from the project and we now have a crisper, simpler API without any loss of functionality, and for good measure the repository has been converted from CVS to MVN.

Next question is should we move to Project Kenai

Better Searching in Jaikoz

I've been working hard with Musicbrainz over the last couple of months rewriting the Search Server to give better results, and I'm pleased to say its been a success and has now been released !

This will improve the results you get with any version of Jaikoz.

Fulfilling The Roadmap

Jaikoz 3.2.2 out now fixing the remaining problems in 3.2.0. 3.2 goes someway towards the roadmap with a simpler interface and simpler matching options. But I didn't manage the last option , more automated tests !

As a result there was a few issues with 3.2.0, so lesson learnt I do need more automated tests and I always should do a beta for these major releases !

I've also been busy rewriting the Musicbrainz Search Server, if all goes as plans this update should make it way onto Musicbrainz very soon.

Jaikoz Future Roadmap

Okay here are the priorities for the next few months.

1. Fix memory management so memory consumption isn't tied by number of songs loaded by use of database. This should allow you to load your complete collection if you wish AND subsequent restarts of Jaikoz should be able to use the cached data instead of reading from the file if it hasn't changed which will speed the initial file loading greatly.

2. Simplify Matching for better results. Jaikoz has many options to change how the matching works, but they are difficult to understand and some only make sense for individual files. Im going to drop some of these options and replace with ones that make more sense such as

Match the album that best matches existing metadata OR always prefer original albums even if better match with compilation

.

3. Improve results by improving Musicbrainz, Im going to work on the Musicbrainz Search server , fixing issues and improving performance.

4. Simplify Interface, new users find it difficult to understand the seperate tag and analyse tasks and the user interface. Im going to either simplify it further or provide a new Simple Mode, with the existing interface becoming Advanced Mode. Thinking along the lines of how Azereus retrofitted a new default interface when it becamwe Vuze.

5. Implement more automated tests for the GUI, I have many for the reading and writing to files but not for the Interface itself. I need to do this to prevent the occasional regression cropping up as it has in the past.

Jaikoz 2.9.2 available

Fixed a couple of problems including a regression in 2.9.0 wherby Jaikoz was incorrectly matching tracks with no album name with albums in the local cache causing a slow down for those of use with large caches.

Performance testing Musicbrainz Search Index

Musicbrainz have been looking at improving the performance of their search server. When search queries are sent to Musicbrainz they do not access the database, instead they access a Search Index that is built from the database and contains the information to be searched and allows full test search and other features not usually available from a database.

Currently Lucene is used, but is accessed using Python with as pylucene. In an attempt to boost performance a simplified version of the search code was developed using pure Java available from here, but this still didn't seem to be giving the required performance enhancements.

Ive tried some tuning mechanisms and code changes to see if I can make some improvements or at least get some benchmark figures.

The tests were performed on a MacBook Pro 2.66Ghz Core 2 Duo with 4GB of 1067 DDR3 RAM using OSX 10.5, So it is a good lab top but doesn't compare so well with a desktop of server. (I used a Macbook because it was 64bit so I could use 64 bit Java to address more than 2GB of memory whereas my Window PC is not, and I have no native linux desktop).

Summary of results:
Track OR query:Single Threaded test with insufficient memory, index on hard drive :1.91 query /sec
Track OR query:Single Threaded test with enough memory, index on hard drive :24.66 track query /sec
Track OR query:Best Multi Threaded test :43.96 queries /sec
Track query:Best Multi Threaded test :59.47 queries /sec
Release query:Threaded test :252.75 queries /sec

I created a test set of 10,011 track titles and 10,011 releases from the database, and I rebuilt the indexes with StopWords Filter removed to solve the stop words bug. I then created a test program to fire requests to the jetty (servlet searcher) from multiple threads, but I was getting inconsistent results and with Jetty not being the thing that was being tested I removed this from the equation.

I restarted by creating a test program that loaded pairs of track/release records into a queue, and then creating a number of threads to read the next pair of the queue and then send these directly to a Search class, this would perform the search and find the best hits and then return, the test results are for this setup. I found myself running a few tests, then making some more code changes to see if it made any difference, this is what I found:


Index Directory Location:
The macbook has a SSD drive, and external hard drive, and of course the index could also be loaded in memory. I expected loading the index into memory would give vastly superior performance but when I tried loading the Track Index (about 2.5 GB) the performance was terrible, I concluded that it must be swapping memory because of the memory required by the OS itself. I did some with the smaller Release Index as well, here the RAM Directory performed as well as the others but not any better. The SSD performed much better than the hard drive when I ran tests with insufficient memory, but when enough memory was allocated there was little difference between the two.

Memory allocated to JVM: With the default (of 64MB) it performed very badly on the hard drive, but when I increased it to 2GB there was a big improvement, but further adjustments (up and down) didn't make much difference. So it seems you need a reasonable but not ridiculous memory for decent results.

Code Improvements: Posted a few questions on the lucene mailing list and was given some optimizations for how the index is opened
new IndexSearcher(IndexReader.open(new NIODSDirectory(new File indexDir + "track_Index"),null),true)));
and on iterating through the results.
Opening using the index using this new method doubled the number of queries that could be processed by reducing contention on the index searcher, I haven't yet tried the iterating query improvements.

No of Threads:I ran tests using just a single thread at first then increased the number of threads to find the optimum throughput. With the code improvement in place I found you needed at least 30 threads for the best results , but additional threads didn't give further improvements. If the tests were performed on a Quad CPU or better I expect more threads would give more gains.

Query type:I tried an OR and a simple query against the track index
type=track&query=track:"trackname" OR release:"releasename"
type=track&query=track:"trackname"
Of course the OR query was slower , but not exponentially so - it was about 30% slower.

I then tried querying the release index, because it is much smaller the results were much better, with a 400% improvement in speed.

The full results are available here

and the amended zip of the code can be found here.

Unable to find 'is this it' album bug solved - almost

There is a longstanding bug in Musicbrainz that makes it difficult to find songs that contain a number of common stop words such as the,is,that,a . This is because these stop words have been removed from the search index so do not count towards a match. Album such as 'is this it?' by the Strokes have a real problem because it ONLY contains stop words.

I suggested a fix some time ago which didnt get acted on. Ive now implemented the fix successfully on a pure Java development server results at http://www.jthink.net/jaikoz/scratch/isthatitsearch.jpg. I need to reimplement in the existing code base, then hopefully this will prove the fix and get it rolled out.

Jaikoz 2.9 released

The Export feature has been added, and I think this could be very useful for some of you power users. Remember you can use it for:
backing up your metadata
editing of metadata within a spreadsheet
moving metadata from one file to another
sharing your song list with other applications or users

I wait for feedback on what you think of it and what uses you make of it.

Ive also been trawling through the bugs list trying to solve some long standing bugs that may have been forgotten. Im happy with results and aim to have Jaikoz essentially bug free within a couple of releases, of course there is still the ever increasing enhancements list....

Export Songs to Spreadsheet

Working on an Export feature which is simple and effective. It will allow you to export the details of your loaded Songs to a comma seperated file. You can then use this file as an archive of your metadata AND you can also open and edit values within a proper spreadsheet application and then import the changes back into Jaikoz.

So its give you tag backup and mega editing capabilities in one go, and you can also use the file created to share your songs list with friends or to create playlists.


There are a few decisions to be made yet though.
1. The export feature only works on the editable fields common to all formats so fields not supported by Jaikoz or only supported in the ID3 tabs view are not exported.
2. Artwork is not exported, its not appropriate to store large binary fields in this sort of file but I know artwork is very important to people so maybe it can be shoehorned into this feature somehow, or should I just have a seperate 'Export Artwork' that could export artwork either on a folder by folder by basis so its kept with the files or all lumped into a single folder.
3. The export only supports single instances of fields, so for example it would only export one genre per song.
4. Because not desirable to load all songs into Jaikoz in one go if you select a file that already exists Jaikoz should append the new entries, but would need to overwrite an entry if it already existed for the same file.
5. The first column of the created file would be the full filename, so that Import can work by matching the filename with a file open in Jaikoz and then update accordingly.
6. You might have two versions of a song , a flac version and a mp3 version and want to import metadata from the mp3 version to the flac file you would just have to edit the filename in the csv file.
7. If when exporting you are are replacing existing songs, should they be replaced in the same place, or afterwards. Would it be better to always sort the file alphabetically.
8. The data is encoded using utf8, this fully supports Unicode so all characters can be encoded and also it is economical with memory - only one byte is used for ascii chars. The only problem is that it is only the default encoding on Linux, so might not be the default choice when the csv file is open with some applications. For example Open Office on Windows Vista assumes that the encoding is windows-1252 , you have to tell it to use utf8.
9. In the future would also like to alow export to an xml format but xml not terribly useful for editing, this would also provide a solution for (3).
10. Could also create native spreadsheet formats such as .xls or .ods which is slightly more user friendly, but I dont think the extra effort involved is worth it at the moment.

Bugs, Enhancements or Testing

Jaikoz 2.8.4 now out with a few enhancements and a host of bugs fixes, some introduced in earlier versions, and there lies the rub. I've been concentrating on bug fixes recently and small enhancements rather than new features but the regressions came about due to not enough testing.

It is very difficult to get the balance right, I have automated tests that cover the reading and writing of metadata but not automated tests for the Jaikoz GUI itself.

So do I spend my time writing more tests, fixing problems or adding new features ?

I think the correct answer is to continue with all three, and ensure I do beta releases of all major releases, but has anyone else got any other views.

Opening Playlist Contents

In Jaikoz 2.8.2 I added support for opening Winamp Playlists, the most widely accepted playlist format. But I knew that what many of you really wanted is to drag and drop iTunes playlists, well you can - nearly.

The difficulties with iTune playlists are twofold. Firstly there is no playlist file so you have to communicate with iTunes instead of just reading the file. Secondly talking to iTunes is very different on Windows than on OSX , so the work effort doubles.

I now have dragging playlists from iTunes working beautifully on Windows and this will be available in Jaikoz 2.8.4 out today or tomorrow.

But OSX does not allow the playlist to be dragged to anywhere except iTunes itself (please anyone correct me if Im wrong) so it is impossible to even initiate a playlist drop. Instead I'll be creating some scripts for the iTunes Script Menu to allow you to send playlists to iTunes.

Get George Lamb off 6 Music

Nothing to do with Jaikoz, but I need to vent my spleen about him. Most days I work on Jaikoz from home alone, and the radio provides my company. I like Indie Music and BBC 6 Music was setup to provide a alternative Popular Music Radio Station with the emphasis on the Music. My favourite show was Gideon Coe between 10 a.m and 1.p.m and then suddently he was moved to a late night slot and George Lamb took his place. Geoge lamb may be okay on Radio 1 but doesn't not suit 6 Music.

Now you might be reading this and thinking good I hate Indie Music, but thats not the point. If you listened to a Dance Music station and the new DJ insisted on just talking about football and heavy metal I'm sure you wouldnt be happy.

George Lamb acts like a complete idiot and seems to be surrounded by a posse of morons. Their jokes are idiotic and extremely repetitive, they play hardly any music at all and most of the music they do play does not fit within remit of the station. They are constantly rude to musicians who have come into play live on the show. They have added in non music related features such as (non music) book and tv reviews. They are frequently on very dodgy grounds on the way they talk to their radio listeners, and they are dull.

He is a poor mans Chris Moyles or Russell Brand . After the Russell Brand /Andrew_Sachs debacle and the resignation of the 6 Music boss I really though Lamb would follow soon after but he is still there. So if you have listened to this show and feel like I do please consider joining Get George Lamb Off Music! and adding your name to the Get Lamb Out Petition