Better AcoustID matching in Jaikoz 5.4.0

How AcoustID works

When a song is analyzed and the fingerprint is submitted to AcoustID the AcoustID server returns an existing AcoustID if the fingerprint has been submitted before, or if the new fingerprint is so similar to an existing fingerprint - alternatively it allocates a new AcoustID for this fingerprint. When songs are matched to Musicbrainz by Metadata you can submit the fingerprint/MusicBrainz Id pairing to AcoustID and it will link the AcoustIDs to the MusicBrainz recording.

A new Discovery

But I discovered something new about AcoustID this month, a fingerprint can sometimes be associated with multiple AcoustIDs ! But only one AcoustID is a good match , the others are AcoustIDs that better match different but similar fingerprints. When a fingerprint/AcoustID pairing is submitted to AcoustID it is only assigned to the best primary AcoustID.

Fixes in Jaikoz 5.4.0

When Jaikoz was looking up the AcoustID for a fingerprint linked to multiple AcoustID it was actually keeping the worst matching AcoustID rather than the best one, this was usually one that was not linked to a MusicBrainz recording and hence not very useful. This has now been fixed in the new release of Jaikoz and will improve the AcoustID matching that Jaikoz does.

Fixing your AcoustIDs

This means that using earlier versions of Jaikoz you may have songs in your collection with the correct fingerprint but the incorrect AcoustID. Whilst this is not a major issue because the AcoustIDs are only a mechanism used to get matches yo may want to fix as follows:

1. Ensure the AcoustID column is visible in Jaikoz
2. Right click on the header and select Empty Column
3. Run Retrieve AcoustIDs
4. Save any files that have changed

Better artwork matching in Jaikoz 5.3.0

Artwork has been improving steadily in Jaikoz over the last year, especially since the Cover Art Archive was introduced but with hew version of Jaikoz things have got even better.

First some terminology, within MusicBrainz the concept of an Album is represented by a Release Group, then different versions of that release group are known as Releases. A release is the thing that we can actually buy and often varies slightly between countries. So a release group often contains different releases because it is released in different countries with slightly different tracks. Discogs also uses releases and has similar concept of release groups, known as Masters

Until this release Jaikoz checked the matching release for artwork on both the Cover Art Archive, Amazon and Discogs but just for the matching release. But now if we find no match we then check all the releases within the same release group and Discogs masters, artwork is usually the same for all releases within release group but often MusicBrainz does not contain artwork links for every single release in a release group so this change gives a big improvement.

I ran a test over 1000 songs , and matching artwork went up from 782 to 960 !

Full list of changes in this release can be found here

Introducing functions in rename masks

An important aspect of tagging your songs is renaming your songs so that they match the metadata within the song. Although most music players depend only on the metadata within the song itself, we all have different preference of how to name our files and this helps us to keep them organized. Some applications require the filename to be in a particular format.

Jaikoz used to have quite a simple language for the rename mask that allowed most metadata to be used in the filename but only provided simple if statements. Then this was replaced with support for full Javascript syntax, this allows you to do pretty much anything but creating the correct mask could be long winded and error prone.

Consider this simple mask

 (album.length>0 ? album + '-':'') + (trackno.length>0 ? trackno + '-':'') 
 + (title.length>0 ? title:'')

Although its fairly clear what it is doing. its very fiddly and easy to make a mistake when editing. Unfortunately as a number of you have commented the Javascript is not as easy to understand as the old syntax, I traded this for the power the Javascript notation gives us.

However recently discovered there is nothing to stop you defining functions within your filename mask so the above can be replaced by

 function ifnotempty(value,sep){
     return value != undefined && value.length > 0 ? value + sep : '';
 }
 ifnotempty(album,'-')+ ifnotempty(trackno,'-') + ifnotempty(title,'') 

As you can now see the complexity is put into the function once, then the resultant mask can be much more concise.

In Jaikoz 3.6 we have made use of functions to provide better default masks. Of course if you are upgrading Jaikoz it will keep your existing mask so you will not see them so I have posted them below.

Folder Rename Mask

 function ifnotempty(value,sep){
     return value.length > 0 ? value + sep : '';
 }
 
 function ifnotempty2(value1,value2,sep){
     return value1.length > 0 ? value1 + sep :value2.length > 0 ? value2 + sep:'' ;
 }
 
 ifnotempty2(albumartist,artist,folderseparator) + ifnotempty(album,folderseparator)
 + (disctotal>1 ? discno  + folderseperator : '') 

Filename Rename Mask

function pad(number, length) {
      var str = '' + number; 
      while (str.length < length) {  
          str = '0' + str;
      }
      return str;
  }
 
 function ifnotempty(value,sep){
     return value.length > 0 ? value + sep : '';
 }
 
 function ifnotempty2(value1,value2,sep){
     return value1.length > 0 ? value1 + sep :value2.length > 0 ? value2 + sep:'' ;
 }
 
 ifnotempty2(albumartist,artist,' - ') + ifnotempty(album,' - ') + 
 ifnotempty(pad(trackno,2),' - ') + title

In future versions of Jaikoz I'm going to add the ability to create and store custom javascript functions with a name to make this even easier, but hopefully in the meantime the use of functions should make creating masks simpler.

Why you should consider the database your tag editor is using to fix your songs

When you use a tag editor that automatically fixes your metadata you should really have a think about where that data comes from.

Photo by Paul Downey
Openness

There are a number of databases containing music information but some of these such are closed, this means you cannot actually browse the data independently never mind correct errors in the database. Often applications that use these type of database keep the identity of their source database hidden.

Accuracy

Sometimes the data is not very accurate. this may be because a database is not being used at all instead the tagger may just trying to hack information from websites and other sources

Linking

An online database can be used to correct your song metadata but for you to take adavantage of a myriad of other interesting applications you need to store a unique id that accurately identifies your song so you dont have to fuzzy match it again.

Metadata and Acoustic Matching

Matching songs by metadata is good but matching by the actual audio can be even better, but you need both: matching by Acoustic Id might identify the correct song but if the song exists on multiple releases it cannot help select the correct release.

The Jaikoz Solution
Openness

The primary database used by Jaikoz is MusicBrainz, MusicBrainz is a completely open database run by the non profit Metabrainz Foundation that can be edited by anybody with a free account. Jaikoz also uses as secondary source Discogs another online database that allows the data to be browsed and edited and freely downloaded.

Accuracy

MusicBrainz has comprehensive guidelines on the editing process and voting system to ensure the quality of edits and to protect the database from poor edits. Discogs also has guidelines and only allows adding of releases that you own, both systems expect the editor to provide another source for their edits.

Linking

MusicBrainz provides unique identifiers for songs, releases and artists (plus some others) and these identifiers are increasing being used by other sites such as Last.fm and the BBC,in fact Musicbrainz Artist Ids are fast becoming an industry standard. Discogs does not have such a sophisticaed system but when a release is matched to Discogs we store the the Discogs Release Url in your songs metadata.

Metadata and Acoustic Matching

Jaikoz uses AcoustId in addition to metadata matching to ensure the best possible match. Acoustid is yet another open system that you can contribute songs to and you can also flag when an acoustic id seems to be assigned to an incorrect song.

Overall at Jthink we are committed to using the best sources available.

Why Google gains market share

When I released Jaikoz five years ago (!) Paypal was the sensible way to go for processing payment. A few years later and some potential customers did not want to use Paypal so I added Google Checkout as well, but I didn't have time to fully automate the Google process.

Yesterday I decided to remedy this, the API and documentation for Google Checkout is way better than Paypal. Both are comprehensive and easy to use and I got things working within the day. I went back to the Paypal website to see what they are offering now, there was now a Java API but it achieves the dubious honour of being both limited and totally confusing.

Google also takes a smaller cut than Paypal so if you are buying a Jaikoz license please use the Google method.

So my point is integrating with Google is easier than other solutions, and this is one reason why I think Android will overtake iPhone within the year

Thanks to Brandon Schauer for the Photo above

Jaikoz is Global

Photo by Börkur Sigurbjörnsson Jaikoz had sales in 76 different countries last year:

Andorra,
Argentina,
Australia,
Austria,
Bahrain,
Belgium,
Bermuda,
Bolivia,
Brazil,
Bulgaria,
Canada,
Cayman Islands,
Chile,
China,
Colombia,
Costa Rica,
Croatia,
Cyprus,
Czech Republic,
Denmark,
Dominican Republic,
Ecuador,
Estonia,
Finland,
France,
Germany,
Greece,
Hong Kong,
Hungary,
India,
Indonesia,
Ireland,
Israel,
Italy,
Jamaica,
Japan,
Kuwait,
Latvia,
Lebanon,
Lithuania,
Luxembourg,
Malaysia,
Maldives,
Mexico,
Mozambique,
Namibia,
Netherlands,
Netherlands Antilles,
New Zealand,
Norway,
Paraguay,
Philippines,
Poland,
Portugal,
Romania,
Russia,
Russian Federation,
Saudi Arabia,
Singapore,
Slovenia,
South Africa,
South Korea,
Spain,
Sweden,
Switzerland,
Taiwan,
Thailand,
Trinidad and Tobago,
Turkey,
Ukraine,
United Arab Emirates,
United Kingdom,
United States,
Uruguay,
Viet Nam,
Yugoslavia,

AcoustId Fingerprinter

A new fingerprint database tool is being developed by long time MusicBrainz contributor
Lukáš Lalinský a little like AmpliFIND that Jaikoz may use in the future but currently its database of fingerprints needs to grow and for that they need your help. Please try out submitting fingerprints of your collection. its very simple to use. and can be downloaded from http://acoustid.org/fingerprinter

Thanks to Karen Cox for the photo

Jaikoz 4.0 NGS Beta now available

The Beta release is now available http://www.jthink.net/jaikoz/jsp/beta/start.jsp configured to run against the Musicbrains NGS Test Database. This test database is configured just like the production database and has already undergone alot of testing so it should be very stable. But this has meant a delay in it becoming available and the full release of Musicbrainz is less than two weeks away, so if you want any more changes to Musicbrainz best to get in there quick.

One of the exciting new features of Jaikoz is the ability to be able to add new releases to Musicbrainz. But remember this is only a test database so only add new releases for the purpose of testing, they will be lost when Musicbrainz goes live.

Whats going to be in Jaikoz 4.0 ?

With 3.9.0 out the door its time to think about the next release. I'm hopeful that the long delayed Musicbrainz NGS will be released soon, and if that is the case Jaikoz will be updated to use the much improved webservice to give better and faster results. Musicbrainz NGS is a massive Musicbrainz release so I think it fitting that Jaikoz gets a major release number to reflect this.

Also Musicbrainz NGS provides an way to semi-automatically add new releases. So within Jaikoz you'll be able to select a group of songs and select 'Add to Musicbrainz' . this will open up a Musicbrainz webpage with everything filled in, so you just have to check the info,add in anything extra you know and okay it. This is another way to really help improve the Musicbrainz Community Database and continue to provide a via alternative to closed databases such as Gracenotes.

Itunes is still the way the majority of users play their music so I'm going to make some further improvements to the iTunes integration to better handle when iTunes is set to copy and re-organise your music files. Windows users have not been been able to update iTunes using Java 64bit because of a problem with the Java COM library I use but I have found an alternative so I will rewrite the Windows iTunes support to use this instead.

Jaikoz in the dock or toolbar doesn't react to having files dropped on it, but this is now possible to do.

Manual Tag from Musicbrainz is due an overhaul and I'm expecting to replace this with a more album orientated one.

There has been a long running debate on shortcut keys. My feeling is that I'm going to simplify shortcut keys so that they always only work on the selected items, whereas currently there is one combination to apply to the selected item and another to apply to all visible in the window. Because the resulting shortcut keys will require less keys, and there will be less shortcuts to remember I think this is an improvement. It will be easy to work on all songs in the edit view by use of Cmd-A/Cntl-A or newly added Select All option. I'm also going to look again at customizable shortcut keys and toolbars.

When matching from AmpliFIND we only take the Acoustic Id, trouble is that if that Acoustic Id is not known to Musicbrainz and your song has no other metadata we cant get a match. AmpliFIND actaully has other basic data such as artist and song title so we should use this if our song doesn't contain the info, this might be enough to do a metadata match against Musicbrainz.

So that is the thinking at the moment

Songbird buggy and they dont seem to care

In Jaikoz 3.8.3 a user was complaining that Jaikoz couldn't open their mp4s. Turns out the problem was that they had been edited in Songbird, and Songbird save track no differently to every other mp4 player I ve tried.

I added a workaround to Jaikoz and raised a bug with the Songbird team.

A little time passed, they linked it to another issue which wasn't the same problem and then they just closed it. I complained and reopened it because nothing had been fixed, and now they have downgraded it to a suggestion for a future release. Because Apple have not written a nice neat specification, it is acceptable for the Songbird developers to ddclare no bug. I really hate it when developers bend over backwards to try and show there not at fault rather then trying to resolve the issue.

Songbird 22966

New fields added to Jaudiotagger

Jaikoz uses my open source Jaudiotagger library for reading and writing tags, so if I want to add a new field to Jaikoz it has to be added to jaudiotagger first.

Ive just added a bucket load of new fields that should make their way into Jaikoz.

CUSTOM1,
CUSTOM2,
CUSTOM3,
CUSTOM4,
CUSTOM5,
FBPM,
MUSICBRAINZ_RELEASE_GROUP_ID,
MUSICBRAINZ_WORK_ID,
OCCASION,
ORIGINAL_ALBUM,
ORIGINAL_ARTIST,
ORIGINAL_LYRICIST,
ORIGINAL_YEAR,
QUALITY,
SCRIPT,
TAGS,
TEMPO,
RATING,
PRODUCER,
MIXER,
DJMIXER,


If there is some field you really want that isn't currently in Jaikoz let me know.

iTunes can now export m3u playlists !

Smoother integration with iTunes is always something I endeavour to achieve, but iTune's doesn't make it easy. One longstanding problem was that iTunes playlists could only be exported in iTune's proprietry format which could not be read by Jaikoz.

I was looking at plugging this gap today, but whilst playing with iTunes 10 I noticed that when you save a playlist that there is now an option to save the playlist in m3u format



So the problem is solved because the contents of a m3u playlist can be opened in Jaikoz either using File/Open Playlist or by a simple drag and drop.

Now it would be really great if you could just drag a playlist directly from iTunes to Jaikoz, but that is not possible at the moment because of a bug in iTunes drag and drop support.

Matching to a different song version

Progress on the new version of Jaikoz with release orientated matching is progress well but I came across an interesting problem today.

As well as release matching, Jaikoz has the 'Prefer Do not match to Single Artist Compilations', for those of you who want songs linked to the original release, even if you don't have the original release. I was testing this with songs from Madonnas Immaculate Collection and I was always failing to match songs such as Holiday to the original album Madonna. Turns out it was because the track length differs by a whopping two minutes, 6.11 seconds for the original versus 4.06 and Jaikoz rejects song lengths that vary by more than 15 seconds (10 seconds in current release)


So the songs are clearly different version, and as a general rule we wouldn't want to match them, but I wonder if some of you actually would want to match to this earlier version regardless that you have the later version ?

The best duplicate song deleter around ?

Most of you try out Jaikoz for getting more metadata for your songs, but one of its less advertised features is duplicate song deletion.

The way Jaikoz find duplicates is very powerful because it uses the sonic matching of AmpliFIND and the expert metadata matching power of Musicbrainz. It is format independent so for example it could recognise that a song encoded as Flac and the same song encoded as Mp3 are the same song.


Once you have matched all your songs you have three choices for identifying duplicates.

1. Matching songs with the same Musicbrainz Id
These songs have been matched to the same release, so are very likely to be a duplicate

2. Matching songs with the same AmpliFIND Id
These songs should be sonically the same, but you may not want to delete them if you want to maintain the same song on two different releases. For example if the same song is in your collection on an original album and a compilation, you may want to keep both.

3. Matching songs with e same Musicbrainz Id AND AmpliFIND Id
These songs have been matched to the same release AND are sonically the same so are almost definitely a duplicate

This choice is made at Preferences/Local Correct/Delete Duplicates, and you can then run Delete Duplicates from the Action menu or as part of the Autocorrecter.

You can also just show potential duplicates and then make the final decision yourself using the filters in the Filter Menu.

The Preferences also let you specify whether to actually delete, move or link to the duplicates

Better matching in Jaikoz 3.4.2

Automatic matching is difficult but I've made a number of changes to improve the matching in Jaikoz in the latest release.

Jaikoz searches for possible matches in Musicbrainz then rescores them taking additional information into account to find the best match, it does this because an original Musicbrainz score only takes into account the search terms when scoring but we need to consider more values. For example we do not specify a duration in a search because some songs do not even have a duration within Musicbrainz so would never be returned by a search, but having got some potential results we want to give a higher score to those with a duration that matches the original song. Musicbrainz uses Lucene for searching with its own custom analyzer for deciding which songs are returned by a search and this latest release of Jaikoz uses the exact same analyzer to ensure scoring is compatible. This is one advantage of working on both Musicbrainz and Jaikoz !

When searching for a track we now consider more variations of the name because songs entered into Musicbrainz are normalized , for example We Have Explosive (Pt. 5) should be entered into Musicbrainz as We Have Explosive, Part 5 but they might not have been. This normalization is detailed in the Style Guidelines and In Jaikoz we now check for the title as it appears in your metadata and also as a normalized version as far as possible.

We also make workarounds for common errors in entering data. For example Musicbrainz Issue #5538 shows that users usually enter song titles as 'No. 1' , but in a large minority of cases enter 'No.1' , Jaikoz workrounds this issue.

Cluster Albums finds albums by artists with the same name but a different Release Id and tries to move the songs so that they are all on the same Release Id, note this is different to what 'Cluster' means in Musicbrainz Picard and perhaps I should have called it something different. Previously it did this by matching title against title for each Release Id being used, and picked the Release Id which had the most matches but now this has been improved. Firstly we use fuzzy matching on the title allowing for normalization as explained earlier. Secondly if all but a couple of tracks are successfully matched to one Release Id we allow matches on Acoustic Id and song length to shoehorn the remaining tracks into a potential release. This is really useful when the same song exists on two albums but is radically renamed between the two.

Musicbrainz Developments

I went to the yearly Musicbrainz summit in Nurenberg a couple of weeks ago, a good time was had by all , and the old town is certainly a lovely place

We discussed future developments, and their plenty of good news for Jaikoz users.

Musicbrainz were very happy with the work I'd done rewriting the existing search system and I am the main developer of the search for the new NGS release being developed so I'm able to understand how search works in Musicbrainz and improve it to suit Jaikoz better. This should ensure you get better results from Jaikoz then any other other tagger.

NGS has lots of new features including recognising the same recording on different albums, and handling multiple disc release better - this will help with getting better matching.

Work is going to be done on creating a genre system from the existing folksonomy cloud so this will help with a field that is currently quite poorly served by Musicbrainz. Knowing the attention to detail of Musicbrainz editors I'm hopeful that in time Musicbrainz many create the definitive genre list.

Historically Musicbrainz have been very cautious about having APIs to allow data to be added in to Musicbrainz. Sensibly they do not want it to end up a mess like freecddb but they have warmed to the idea now and I think this is a neccessary move to keep on top of the exposion in music being recorded. So in the New Year Im going to have a think about taking advantage of this loosening up of data entry.

And Musicbrainz have just hired their first fulltime developer Kuno Woudt , a well established Musicbrainz editor and developer so this should speed up Musicbrainz development.

New version of Jaudiotagger

I've just released the new version of Jaudiotagger, the tagging library used by Jaikoz and quite a few other applications. I've been happily removing code from the project and we now have a crisper, simpler API without any loss of functionality, and for good measure the repository has been converted from CVS to MVN.

Next question is should we move to Project Kenai

Better Searching in Jaikoz

I've been working hard with Musicbrainz over the last couple of months rewriting the Search Server to give better results, and I'm pleased to say its been a success and has now been released !

This will improve the results you get with any version of Jaikoz.

Fulfilling The Roadmap

Jaikoz 3.2.2 out now fixing the remaining problems in 3.2.0. 3.2 goes someway towards the roadmap with a simpler interface and simpler matching options. But I didn't manage the last option , more automated tests !

As a result there was a few issues with 3.2.0, so lesson learnt I do need more automated tests and I always should do a beta for these major releases !

I've also been busy rewriting the Musicbrainz Search Server, if all goes as plans this update should make it way onto Musicbrainz very soon.

Jaikoz Future Roadmap

Okay here are the priorities for the next few months.

1. Fix memory management so memory consumption isn't tied by number of songs loaded by use of database. This should allow you to load your complete collection if you wish AND subsequent restarts of Jaikoz should be able to use the cached data instead of reading from the file if it hasn't changed which will speed the initial file loading greatly.

2. Simplify Matching for better results. Jaikoz has many options to change how the matching works, but they are difficult to understand and some only make sense for individual files. Im going to drop some of these options and replace with ones that make more sense such as

Match the album that best matches existing metadata OR always prefer original albums even if better match with compilation

.

3. Improve results by improving Musicbrainz, Im going to work on the Musicbrainz Search server , fixing issues and improving performance.

4. Simplify Interface, new users find it difficult to understand the seperate tag and analyse tasks and the user interface. Im going to either simplify it further or provide a new Simple Mode, with the existing interface becoming Advanced Mode. Thinking along the lines of how Azereus retrofitted a new default interface when it becamwe Vuze.

5. Implement more automated tests for the GUI, I have many for the reading and writing to files but not for the Interface itself. I need to do this to prevent the occasional regression cropping up as it has in the past.

Jaikoz 2.9.2 available

Fixed a couple of problems including a regression in 2.9.0 wherby Jaikoz was incorrectly matching tracks with no album name with albums in the local cache causing a slow down for those of use with large caches.

Performance testing Musicbrainz Search Index

Musicbrainz have been looking at improving the performance of their search server. When search queries are sent to Musicbrainz they do not access the database, instead they access a Search Index that is built from the database and contains the information to be searched and allows full test search and other features not usually available from a database.

Currently Lucene is used, but is accessed using Python with as pylucene. In an attempt to boost performance a simplified version of the search code was developed using pure Java available from here, but this still didn't seem to be giving the required performance enhancements.

Ive tried some tuning mechanisms and code changes to see if I can make some improvements or at least get some benchmark figures.

The tests were performed on a MacBook Pro 2.66Ghz Core 2 Duo with 4GB of 1067 DDR3 RAM using OSX 10.5, So it is a good lab top but doesn't compare so well with a desktop of server. (I used a Macbook because it was 64bit so I could use 64 bit Java to address more than 2GB of memory whereas my Window PC is not, and I have no native linux desktop).

Summary of results:
Track OR query:Single Threaded test with insufficient memory, index on hard drive :1.91 query /sec
Track OR query:Single Threaded test with enough memory, index on hard drive :24.66 track query /sec
Track OR query:Best Multi Threaded test :43.96 queries /sec
Track query:Best Multi Threaded test :59.47 queries /sec
Release query:Threaded test :252.75 queries /sec

I created a test set of 10,011 track titles and 10,011 releases from the database, and I rebuilt the indexes with StopWords Filter removed to solve the stop words bug. I then created a test program to fire requests to the jetty (servlet searcher) from multiple threads, but I was getting inconsistent results and with Jetty not being the thing that was being tested I removed this from the equation.

I restarted by creating a test program that loaded pairs of track/release records into a queue, and then creating a number of threads to read the next pair of the queue and then send these directly to a Search class, this would perform the search and find the best hits and then return, the test results are for this setup. I found myself running a few tests, then making some more code changes to see if it made any difference, this is what I found:


Index Directory Location:
The macbook has a SSD drive, and external hard drive, and of course the index could also be loaded in memory. I expected loading the index into memory would give vastly superior performance but when I tried loading the Track Index (about 2.5 GB) the performance was terrible, I concluded that it must be swapping memory because of the memory required by the OS itself. I did some with the smaller Release Index as well, here the RAM Directory performed as well as the others but not any better. The SSD performed much better than the hard drive when I ran tests with insufficient memory, but when enough memory was allocated there was little difference between the two.

Memory allocated to JVM: With the default (of 64MB) it performed very badly on the hard drive, but when I increased it to 2GB there was a big improvement, but further adjustments (up and down) didn't make much difference. So it seems you need a reasonable but not ridiculous memory for decent results.

Code Improvements: Posted a few questions on the lucene mailing list and was given some optimizations for how the index is opened
new IndexSearcher(IndexReader.open(new NIODSDirectory(new File indexDir + "track_Index"),null),true)));
and on iterating through the results.
Opening using the index using this new method doubled the number of queries that could be processed by reducing contention on the index searcher, I haven't yet tried the iterating query improvements.

No of Threads:I ran tests using just a single thread at first then increased the number of threads to find the optimum throughput. With the code improvement in place I found you needed at least 30 threads for the best results , but additional threads didn't give further improvements. If the tests were performed on a Quad CPU or better I expect more threads would give more gains.

Query type:I tried an OR and a simple query against the track index
type=track&query=track:"trackname" OR release:"releasename"
type=track&query=track:"trackname"
Of course the OR query was slower , but not exponentially so - it was about 30% slower.

I then tried querying the release index, because it is much smaller the results were much better, with a 400% improvement in speed.

The full results are available here

and the amended zip of the code can be found here.

Unable to find 'is this it' album bug solved - almost

There is a longstanding bug in Musicbrainz that makes it difficult to find songs that contain a number of common stop words such as the,is,that,a . This is because these stop words have been removed from the search index so do not count towards a match. Album such as 'is this it?' by the Strokes have a real problem because it ONLY contains stop words.

I suggested a fix some time ago which didnt get acted on. Ive now implemented the fix successfully on a pure Java development server results at http://www.jthink.net/jaikoz/scratch/isthatitsearch.jpg. I need to reimplement in the existing code base, then hopefully this will prove the fix and get it rolled out.

Jaikoz 2.9 released

The Export feature has been added, and I think this could be very useful for some of you power users. Remember you can use it for:
backing up your metadata
editing of metadata within a spreadsheet
moving metadata from one file to another
sharing your song list with other applications or users

I wait for feedback on what you think of it and what uses you make of it.

Ive also been trawling through the bugs list trying to solve some long standing bugs that may have been forgotten. Im happy with results and aim to have Jaikoz essentially bug free within a couple of releases, of course there is still the ever increasing enhancements list....

Export Songs to Spreadsheet

Working on an Export feature which is simple and effective. It will allow you to export the details of your loaded Songs to a comma seperated file. You can then use this file as an archive of your metadata AND you can also open and edit values within a proper spreadsheet application and then import the changes back into Jaikoz.

So its give you tag backup and mega editing capabilities in one go, and you can also use the file created to share your songs list with friends or to create playlists.


There are a few decisions to be made yet though.
1. The export feature only works on the editable fields common to all formats so fields not supported by Jaikoz or only supported in the ID3 tabs view are not exported.
2. Artwork is not exported, its not appropriate to store large binary fields in this sort of file but I know artwork is very important to people so maybe it can be shoehorned into this feature somehow, or should I just have a seperate 'Export Artwork' that could export artwork either on a folder by folder by basis so its kept with the files or all lumped into a single folder.
3. The export only supports single instances of fields, so for example it would only export one genre per song.
4. Because not desirable to load all songs into Jaikoz in one go if you select a file that already exists Jaikoz should append the new entries, but would need to overwrite an entry if it already existed for the same file.
5. The first column of the created file would be the full filename, so that Import can work by matching the filename with a file open in Jaikoz and then update accordingly.
6. You might have two versions of a song , a flac version and a mp3 version and want to import metadata from the mp3 version to the flac file you would just have to edit the filename in the csv file.
7. If when exporting you are are replacing existing songs, should they be replaced in the same place, or afterwards. Would it be better to always sort the file alphabetically.
8. The data is encoded using utf8, this fully supports Unicode so all characters can be encoded and also it is economical with memory - only one byte is used for ascii chars. The only problem is that it is only the default encoding on Linux, so might not be the default choice when the csv file is open with some applications. For example Open Office on Windows Vista assumes that the encoding is windows-1252 , you have to tell it to use utf8.
9. In the future would also like to alow export to an xml format but xml not terribly useful for editing, this would also provide a solution for (3).
10. Could also create native spreadsheet formats such as .xls or .ods which is slightly more user friendly, but I dont think the extra effort involved is worth it at the moment.

Bugs, Enhancements or Testing

Jaikoz 2.8.4 now out with a few enhancements and a host of bugs fixes, some introduced in earlier versions, and there lies the rub. I've been concentrating on bug fixes recently and small enhancements rather than new features but the regressions came about due to not enough testing.

It is very difficult to get the balance right, I have automated tests that cover the reading and writing of metadata but not automated tests for the Jaikoz GUI itself.

So do I spend my time writing more tests, fixing problems or adding new features ?

I think the correct answer is to continue with all three, and ensure I do beta releases of all major releases, but has anyone else got any other views.

Opening Playlist Contents

In Jaikoz 2.8.2 I added support for opening Winamp Playlists, the most widely accepted playlist format. But I knew that what many of you really wanted is to drag and drop iTunes playlists, well you can - nearly.

The difficulties with iTune playlists are twofold. Firstly there is no playlist file so you have to communicate with iTunes instead of just reading the file. Secondly talking to iTunes is very different on Windows than on OSX , so the work effort doubles.

I now have dragging playlists from iTunes working beautifully on Windows and this will be available in Jaikoz 2.8.4 out today or tomorrow.

But OSX does not allow the playlist to be dragged to anywhere except iTunes itself (please anyone correct me if Im wrong) so it is impossible to even initiate a playlist drop. Instead I'll be creating some scripts for the iTunes Script Menu to allow you to send playlists to iTunes.

Get George Lamb off 6 Music

Nothing to do with Jaikoz, but I need to vent my spleen about him. Most days I work on Jaikoz from home alone, and the radio provides my company. I like Indie Music and BBC 6 Music was setup to provide a alternative Popular Music Radio Station with the emphasis on the Music. My favourite show was Gideon Coe between 10 a.m and 1.p.m and then suddently he was moved to a late night slot and George Lamb took his place. Geoge lamb may be okay on Radio 1 but doesn't not suit 6 Music.

Now you might be reading this and thinking good I hate Indie Music, but thats not the point. If you listened to a Dance Music station and the new DJ insisted on just talking about football and heavy metal I'm sure you wouldnt be happy.

George Lamb acts like a complete idiot and seems to be surrounded by a posse of morons. Their jokes are idiotic and extremely repetitive, they play hardly any music at all and most of the music they do play does not fit within remit of the station. They are constantly rude to musicians who have come into play live on the show. They have added in non music related features such as (non music) book and tv reviews. They are frequently on very dodgy grounds on the way they talk to their radio listeners, and they are dull.

He is a poor mans Chris Moyles or Russell Brand . After the Russell Brand /Andrew_Sachs debacle and the resignation of the 6 Music boss I really though Lamb would follow soon after but he is still there. So if you have listened to this show and feel like I do please consider joining Get George Lamb Off Music! and adding your name to the Get Lamb Out Petition

Try out the new Musicbrainz Collection Feature

Musicbrainz is now testing a host of new features including the ability to create a list of all your releases on the Musicbrainz server. This is a really good time to add your two penneth because server releases don't happen that often.

In jaikoz 2.6.0 I added the ability to upload your collection to Musicbrainz, so now Musicbrainz have made this available on their Test server you can try it out.

This is what you need to do in Jaikoz
Open Preferences
On 'Musicbrainz/Match tab' change the value of Musicbrainz Server to http://test.musicbrainz.org
On 'Musicbrainz/Submissions to Musicbrainz tab' change the value of your password to mb
Select Action/Remote Correct/Submit Collections to Musicbrainz to submit all releases with a Musicbrainz release Id to Musicbrainz.

Then to see your results in Musicbrainz login to http://test.musicbrainz.org and click on My collection

More details about this feature at http://wiki.musicbrainz.org/MusicCollectionIntroduction

There are a couple of bugettes at the moment. Jaikoz might display error 400 when submitting releases to Musicbrainz if any of the releases are not in the test server
but the other releases are still submitted successfully, go to the website to check.

If you have a problem with your login on the test server you need to remember to use the password mb not your usual password and you remove any cookies for test.musicbrainz.org and musicbrainz.org

Jaikoz changing its spots, becoming more Leopard Like.

One drawback of Java on the Mac platform is that it isn't easy to get applications to look really Mac like. Since Jaikoz started I've been using Quaqua to get round some of the issues, but OSX Leopard has made some radical changes to the UI which cannot yet be replicated in Quaqua. Luckily a new library is being developed rapidy called MacWidgets that goes a long way to getting that native look. Take a look at this screenshot of what Ive done so far.

Discogs

Ive been spending some time playing with Discogs whilst its not nearly as accurate or democratic as Musicbrainz it is still a very good resource, and contains a large number of releases not in Musicbrainz plus plenty of additional cross-checking information for releases that are in Musicbrainz. It is particulary good for electronic music and independent music from the vinyl era.

In the forthcoming release of Jaikoz I've added an Update tags from Discogs feature that works similar to the Update tags from Musicbrainz for any track that contains a Discogs Release url.
From Discogs we can currently get about ten different pieces of data including higher quality artwork.

And Because Discogs works only at release level you can copy/paste a single release url to all tracks in a release and Jaikoz will extract the track info from the release and will update them accordingly, so you can do this even for tracks without a Musicbrainz match. There will also be a Manual Tag from Discogs option added in a later release.

This got me thinking and in a later release Im also going to introduce an Update Tags from Musicbrainz Release Id so that if Jaikoz cant find the right release itself you just need to find the release id manually and paste it into all the tracks. This will be useful for those difficult to find releases, currently you have to enter the individual track id and for each track in the release which is a pain.

I'm also hoping that those of you who are active Musicbrainz contributors will be able to use the discogs information to improve the Musicbrainz database.

Jaikoz 2.5 released

Thanks to everyone for your feedback during the beta testing. Jaikoz 2.5 is now available, its better in many ways - enjoy.

2.5.0 Beta out now

Jaikoz v2.5.0 beta is out now please give it a go. One of the most exciting developments is support for fixing genres from Musicbainz folksonomy tags, and adding your own genres back to Musicbrainz. Unlike the majority of Musicbrainz data Folksonomy Tags are subjective rather than factual so it will be interesting to see how the folksonomy tag cloud develops.

Jaikoz is the first application to support submitting genres back to Musicbrainz. Before this release Musicbrainz Users had to enter tags using the website or write their own scripts to do this.

Testing of the New Musicbrainz Server

Ive been testing the new search server being developed by Musicbrainz. Currently Musicbrainz uses Lucene but the new server will use Xapian . Whilst Im not convinced that Xapian is intrinically better than Lucene it has helped the musicbrainz developers to solve some problems, and Ive been happy to help.

Jaikoz has embedded database

The new version is going to use the Apache derby database. This is a super small relational database that is 100% Java. So there should be no os specific problems , and because it can be embedded in your program there is no db install or services to install.

I am currently using it to store release and artist lookups from Musicbrainz so that a release/artist has to only be checked once. This already occurs in the current version of Jaikoz but everything is held in memory so memory becomes a problem over time, and the results are not kept between restarts of Jaikoz.

Im also taking advantage of this locally stored release data so that I dont actually have to a lookup for every track if I can find a good match by looking at already downloaded and used releases. Consider the usual case of all tracks in a release being looked up if the metadata is good enough the new system will only require
one track, artist and release lookup for the first track, all the other tracks can be matched using the downloaded release info , whereas with the current system a track query is done for every track.

So Jaikoz should be faster and use less memory.

In subsequent versions I want to use the database to hold track metadata instead of it being held in memory, this will then allow Jaikoz to be used on super large collectiuons without runing out of memory. But I'll never make the database the sole reposiitory of any info, so it will never be neccessary to preserve the database in order to access your metadata - it is only a tool to improve memory usage and performance.

PlugIns

I'm thinking about adding support for writing Plugins. Would this be of interest to anyone and if so would you be happy to write it in Java , or would you prefer a scripting language like Python ?

Scripting

I have been improving the Rename Filename/SubFolder Masks to allow a bit more control over how your files are named from your tags

i.e $if(%bestartist%,%bestartist%-)$if(%album%,%album%-)$if(%trackno%,%trackno%-)%title%
would could give you

The Cribs-The New Fellas-01-Hey Scenesters!.mp3
or
The New Fellas-01-Hey Scenesters!.mp3

depending on your metadata.

This syntax comes from http://wiki.hydrogenaudio.org/index.php?title=Foobar2000:Titleformat_Reference#.24if.28cond.2Cthen.2Celse.29

but for this release Jaikoz will not support much more than the simple if statement. However in later releases I plan to support more functions using a suitable templating/scripting language.

Jaikoz roundtrip processing of genres

I now have Jaikoz retrieving tags as genres from Musicbrainz, but not only that it can submit genres back to Musicbrainz. This means you can easily help build the Musicbrainz folksonomy, and you can use Musicbrainz to backup genres for your tracks. With a little bit of fiddling around it to easily tag your tracks with non genre information - such as the 'owned' tag.

The retrieve genres algorithm is as follows:
Get all tags for track
If any exist strip out any one we dont want using configurable blacklist
Favour tags that match Winamps genre list
If there are multiple matches pick the most popular tag one
If there are no tags that match Winamps genre list pick the most popular
If there are no matching tags repeat at release level.
If there are no matching tags repeat at artist level.

Only one genre is picked at the moment because most media players and audio formats only support a single genre so adding multiple genres doesnt seem to help much, but if you disagree let me know.

2.4.1 out now

New release today, originally I needed to do a release to fix a couple of issue with Jaikoz on Leopard, But I ended up doing a bit more

For brand new new installations (doesnt apply to upgrades) users will not see as many columns loaded by default, and on OSX they fields won't be colour coded. This is an attempt to make it a bit easier for users to not be confused by Jaikoz on first viewing. The Autocorrecters default configuaration now does remote corrects (such as Correct tags from Musicbrainz) as well as the local corrects.

Windows users will now see that their foreign language metadata can be viewed correctly in Windows Explorer.

Some options such as the View Artwork Fullsize and View Online were previously only available in the contextual (popup) menus, these have now been added to the top level menu as well.

The keyboard shortcuts have been standardised, and shortcuts moved to the most popular tasks.

You can now drag and drop a field onto another field.

More details at
http://www.jthink.net/jaikoz/jsp/news/build1046.jsp

Fixed the pesky IndexOutOfBoundsException Error

There has been one bug that some of you encounter the IndexOutOfBoundsException error. This has been very difficult to track down, I thought I had solved the problem in the last release but it was still happening occasionally. Anyway, I'm now confident this has been completely solved ready for the 2.4.1 release out soon

Musicbrainz and the BBC

You may know that the BBC uses Musicbrainz on the Music part of the website. Seeing as the BBC is probably the most repected broadcaster on the planet this is a great vote of confidence in the accuracy of the Musicbrainz data, good news for Jaikoz !

But I didn't realise until the summit last week that BBC employees add data into the database as well

http://wiki.musicbrainz.org/BBCTeamsEditingMusicBrainz

and it seems like their involvement is going to grow and grow, good news for everyone.

Musicbrainz Packages

It was great to meet some of the people behind Musicbrainz - a really nice bunch of guys. I have a much better understanding of what they are trying to achieve.

On a more technical level they have big plans for the database, the most important for Jaikoz is the concept of release packages. At the moment album are often released slighty differently in different countries - and sometimes there are multiple releases in the same country, each occurence of this generates a new release in Musicbrainz. But these releases are not tied together and there is no way to accurately distinguish between two versions of the same release and two completely different releases. As a user I normally want to see only one occurence of each release, and I want to see the one for the country I live in, Musicbrainz is going to introduce the concept of a Package to tie all the variation of one release together.

Musicbrainz summit

I'll be going to the Musicbrainz Summit in London this weekend, so should be interesting to see what they are going to be doing this year and see what it means for Jaikoz.

Fix for Leopard

A few of you have had an intermittent problem with Jaikoz v2.4.0 starting on Leopard, the error seen is shown as Unexpected Problem: n >= n where n can be any number and was linked to what additional devices you had attached to your Mac. This was due to a problem with a GUI library used by Jaikoz that has now been fixed. You can get a patch from the fixes page.

New native dialog support in Leopard

As well as the Java 1.6 release on Leopard the default release for Intel and PPC adds support for some native OSX features that are not available on the Tiger release . In Jaikoz 2.4 I made use of this to display a search box that looks like a native Search field


whereas on Tiger it looks like a standard OSX entry field



In the next release I hope to make much more use of these features, whilst maintaining Tiger compatability.

Apple release Java 1.6 on Leopard Intel

Apple has just released JavaUpdate1 - Java 1.6 for OSX Leopard (Intel). I tried it out with Jaikoz and it seems to work ok but I havent tested it extensively. If you install it using SoftwareUpdate Jaikoz will not use it by default but you can try Jaikoz with this if you wish as follows

• In Finder right click Jaikoz and select Show Package Contents
• Select Contents
• Right click Info.plist and select Open With Text Editor
  
• Find the line 1.5+ and change it to 1.6
• Save Changes
• Start Jaikoz in the console windows you should be able to confirm if it is using 1.6

Using 1.6 does not add any new features new Jaikoz, but it should increase its responsiveness.

Why have I created a Jaikoz Blog ?

Whilst the forum is a useful way of informing current users about new developments it has a few drawbacks.

• It doesnt discuss what I'm doing between releases, it is focused on new releases and the issues between them.
• It isnt so visible to new users of the site
  
• It isnt easy to browse chronologically
  

I'd like to keep you uptodate on what Im working on, and other things of interest, please feel free to comment.