Jaikoz Future Roadmap

Okay here are the priorities for the next few months.

1. Fix memory management so memory consumption isn't tied by number of songs loaded by use of database. This should allow you to load your complete collection if you wish AND subsequent restarts of Jaikoz should be able to use the cached data instead of reading from the file if it hasn't changed which will speed the initial file loading greatly.

2. Simplify Matching for better results. Jaikoz has many options to change how the matching works, but they are difficult to understand and some only make sense for individual files. Im going to drop some of these options and replace with ones that make more sense such as

Match the album that best matches existing metadata OR always prefer original albums even if better match with compilation

.

3. Improve results by improving Musicbrainz, Im going to work on the Musicbrainz Search server , fixing issues and improving performance.

4. Simplify Interface, new users find it difficult to understand the seperate tag and analyse tasks and the user interface. Im going to either simplify it further or provide a new Simple Mode, with the existing interface becoming Advanced Mode. Thinking along the lines of how Azereus retrofitted a new default interface when it becamwe Vuze.

5. Implement more automated tests for the GUI, I have many for the reading and writing to files but not for the Interface itself. I need to do this to prevent the occasional regression cropping up as it has in the past.

Jaikoz 2.9.2 available

Fixed a couple of problems including a regression in 2.9.0 wherby Jaikoz was incorrectly matching tracks with no album name with albums in the local cache causing a slow down for those of use with large caches.

Performance testing Musicbrainz Search Index

Musicbrainz have been looking at improving the performance of their search server. When search queries are sent to Musicbrainz they do not access the database, instead they access a Search Index that is built from the database and contains the information to be searched and allows full test search and other features not usually available from a database.

Currently Lucene is used, but is accessed using Python with as pylucene. In an attempt to boost performance a simplified version of the search code was developed using pure Java available from here, but this still didn't seem to be giving the required performance enhancements.

Ive tried some tuning mechanisms and code changes to see if I can make some improvements or at least get some benchmark figures.

The tests were performed on a MacBook Pro 2.66Ghz Core 2 Duo with 4GB of 1067 DDR3 RAM using OSX 10.5, So it is a good lab top but doesn't compare so well with a desktop of server. (I used a Macbook because it was 64bit so I could use 64 bit Java to address more than 2GB of memory whereas my Window PC is not, and I have no native linux desktop).

Summary of results:
Track OR query:Single Threaded test with insufficient memory, index on hard drive :1.91 query /sec
Track OR query:Single Threaded test with enough memory, index on hard drive :24.66 track query /sec
Track OR query:Best Multi Threaded test :43.96 queries /sec
Track query:Best Multi Threaded test :59.47 queries /sec
Release query:Threaded test :252.75 queries /sec

I created a test set of 10,011 track titles and 10,011 releases from the database, and I rebuilt the indexes with StopWords Filter removed to solve the stop words bug. I then created a test program to fire requests to the jetty (servlet searcher) from multiple threads, but I was getting inconsistent results and with Jetty not being the thing that was being tested I removed this from the equation.

I restarted by creating a test program that loaded pairs of track/release records into a queue, and then creating a number of threads to read the next pair of the queue and then send these directly to a Search class, this would perform the search and find the best hits and then return, the test results are for this setup. I found myself running a few tests, then making some more code changes to see if it made any difference, this is what I found:


Index Directory Location:
The macbook has a SSD drive, and external hard drive, and of course the index could also be loaded in memory. I expected loading the index into memory would give vastly superior performance but when I tried loading the Track Index (about 2.5 GB) the performance was terrible, I concluded that it must be swapping memory because of the memory required by the OS itself. I did some with the smaller Release Index as well, here the RAM Directory performed as well as the others but not any better. The SSD performed much better than the hard drive when I ran tests with insufficient memory, but when enough memory was allocated there was little difference between the two.

Memory allocated to JVM: With the default (of 64MB) it performed very badly on the hard drive, but when I increased it to 2GB there was a big improvement, but further adjustments (up and down) didn't make much difference. So it seems you need a reasonable but not ridiculous memory for decent results.

Code Improvements: Posted a few questions on the lucene mailing list and was given some optimizations for how the index is opened
new IndexSearcher(IndexReader.open(new NIODSDirectory(new File indexDir + "track_Index"),null),true)));
and on iterating through the results.
Opening using the index using this new method doubled the number of queries that could be processed by reducing contention on the index searcher, I haven't yet tried the iterating query improvements.

No of Threads:I ran tests using just a single thread at first then increased the number of threads to find the optimum throughput. With the code improvement in place I found you needed at least 30 threads for the best results , but additional threads didn't give further improvements. If the tests were performed on a Quad CPU or better I expect more threads would give more gains.

Query type:I tried an OR and a simple query against the track index
type=track&query=track:"trackname" OR release:"releasename"
type=track&query=track:"trackname"
Of course the OR query was slower , but not exponentially so - it was about 30% slower.

I then tried querying the release index, because it is much smaller the results were much better, with a 400% improvement in speed.

The full results are available here

and the amended zip of the code can be found here.

Unable to find 'is this it' album bug solved - almost

There is a longstanding bug in Musicbrainz that makes it difficult to find songs that contain a number of common stop words such as the,is,that,a . This is because these stop words have been removed from the search index so do not count towards a match. Album such as 'is this it?' by the Strokes have a real problem because it ONLY contains stop words.

I suggested a fix some time ago which didnt get acted on. Ive now implemented the fix successfully on a pure Java development server results at http://www.jthink.net/jaikoz/scratch/isthatitsearch.jpg. I need to reimplement in the existing code base, then hopefully this will prove the fix and get it rolled out.

Jaikoz 2.9 released

The Export feature has been added, and I think this could be very useful for some of you power users. Remember you can use it for:
backing up your metadata
editing of metadata within a spreadsheet
moving metadata from one file to another
sharing your song list with other applications or users

I wait for feedback on what you think of it and what uses you make of it.

Ive also been trawling through the bugs list trying to solve some long standing bugs that may have been forgotten. Im happy with results and aim to have Jaikoz essentially bug free within a couple of releases, of course there is still the ever increasing enhancements list....

Export Songs to Spreadsheet

Working on an Export feature which is simple and effective. It will allow you to export the details of your loaded Songs to a comma seperated file. You can then use this file as an archive of your metadata AND you can also open and edit values within a proper spreadsheet application and then import the changes back into Jaikoz.

So its give you tag backup and mega editing capabilities in one go, and you can also use the file created to share your songs list with friends or to create playlists.


There are a few decisions to be made yet though.
1. The export feature only works on the editable fields common to all formats so fields not supported by Jaikoz or only supported in the ID3 tabs view are not exported.
2. Artwork is not exported, its not appropriate to store large binary fields in this sort of file but I know artwork is very important to people so maybe it can be shoehorned into this feature somehow, or should I just have a seperate 'Export Artwork' that could export artwork either on a folder by folder by basis so its kept with the files or all lumped into a single folder.
3. The export only supports single instances of fields, so for example it would only export one genre per song.
4. Because not desirable to load all songs into Jaikoz in one go if you select a file that already exists Jaikoz should append the new entries, but would need to overwrite an entry if it already existed for the same file.
5. The first column of the created file would be the full filename, so that Import can work by matching the filename with a file open in Jaikoz and then update accordingly.
6. You might have two versions of a song , a flac version and a mp3 version and want to import metadata from the mp3 version to the flac file you would just have to edit the filename in the csv file.
7. If when exporting you are are replacing existing songs, should they be replaced in the same place, or afterwards. Would it be better to always sort the file alphabetically.
8. The data is encoded using utf8, this fully supports Unicode so all characters can be encoded and also it is economical with memory - only one byte is used for ascii chars. The only problem is that it is only the default encoding on Linux, so might not be the default choice when the csv file is open with some applications. For example Open Office on Windows Vista assumes that the encoding is windows-1252 , you have to tell it to use utf8.
9. In the future would also like to alow export to an xml format but xml not terribly useful for editing, this would also provide a solution for (3).
10. Could also create native spreadsheet formats such as .xls or .ods which is slightly more user friendly, but I dont think the extra effort involved is worth it at the moment.

Bugs, Enhancements or Testing

Jaikoz 2.8.4 now out with a few enhancements and a host of bugs fixes, some introduced in earlier versions, and there lies the rub. I've been concentrating on bug fixes recently and small enhancements rather than new features but the regressions came about due to not enough testing.

It is very difficult to get the balance right, I have automated tests that cover the reading and writing of metadata but not automated tests for the Jaikoz GUI itself.

So do I spend my time writing more tests, fixing problems or adding new features ?

I think the correct answer is to continue with all three, and ensure I do beta releases of all major releases, but has anyone else got any other views.

Opening Playlist Contents

In Jaikoz 2.8.2 I added support for opening Winamp Playlists, the most widely accepted playlist format. But I knew that what many of you really wanted is to drag and drop iTunes playlists, well you can - nearly.

The difficulties with iTune playlists are twofold. Firstly there is no playlist file so you have to communicate with iTunes instead of just reading the file. Secondly talking to iTunes is very different on Windows than on OSX , so the work effort doubles.

I now have dragging playlists from iTunes working beautifully on Windows and this will be available in Jaikoz 2.8.4 out today or tomorrow.

But OSX does not allow the playlist to be dragged to anywhere except iTunes itself (please anyone correct me if Im wrong) so it is impossible to even initiate a playlist drop. Instead I'll be creating some scripts for the iTunes Script Menu to allow you to send playlists to iTunes.

Get George Lamb off 6 Music

Nothing to do with Jaikoz, but I need to vent my spleen about him. Most days I work on Jaikoz from home alone, and the radio provides my company. I like Indie Music and BBC 6 Music was setup to provide a alternative Popular Music Radio Station with the emphasis on the Music. My favourite show was Gideon Coe between 10 a.m and 1.p.m and then suddently he was moved to a late night slot and George Lamb took his place. Geoge lamb may be okay on Radio 1 but doesn't not suit 6 Music.

Now you might be reading this and thinking good I hate Indie Music, but thats not the point. If you listened to a Dance Music station and the new DJ insisted on just talking about football and heavy metal I'm sure you wouldnt be happy.

George Lamb acts like a complete idiot and seems to be surrounded by a posse of morons. Their jokes are idiotic and extremely repetitive, they play hardly any music at all and most of the music they do play does not fit within remit of the station. They are constantly rude to musicians who have come into play live on the show. They have added in non music related features such as (non music) book and tv reviews. They are frequently on very dodgy grounds on the way they talk to their radio listeners, and they are dull.

He is a poor mans Chris Moyles or Russell Brand . After the Russell Brand /Andrew_Sachs debacle and the resignation of the 6 Music boss I really though Lamb would follow soon after but he is still there. So if you have listened to this show and feel like I do please consider joining Get George Lamb Off Music! and adding your name to the Get Lamb Out Petition

Try out the new Musicbrainz Collection Feature

Musicbrainz is now testing a host of new features including the ability to create a list of all your releases on the Musicbrainz server. This is a really good time to add your two penneth because server releases don't happen that often.

In jaikoz 2.6.0 I added the ability to upload your collection to Musicbrainz, so now Musicbrainz have made this available on their Test server you can try it out.

This is what you need to do in Jaikoz
Open Preferences
On 'Musicbrainz/Match tab' change the value of Musicbrainz Server to http://test.musicbrainz.org
On 'Musicbrainz/Submissions to Musicbrainz tab' change the value of your password to mb
Select Action/Remote Correct/Submit Collections to Musicbrainz to submit all releases with a Musicbrainz release Id to Musicbrainz.

Then to see your results in Musicbrainz login to http://test.musicbrainz.org and click on My collection

More details about this feature at http://wiki.musicbrainz.org/MusicCollectionIntroduction

There are a couple of bugettes at the moment. Jaikoz might display error 400 when submitting releases to Musicbrainz if any of the releases are not in the test server
but the other releases are still submitted successfully, go to the website to check.

If you have a problem with your login on the test server you need to remember to use the password mb not your usual password and you remove any cookies for test.musicbrainz.org and musicbrainz.org

Jaikoz changing its spots, becoming more Leopard Like.

One drawback of Java on the Mac platform is that it isn't easy to get applications to look really Mac like. Since Jaikoz started I've been using Quaqua to get round some of the issues, but OSX Leopard has made some radical changes to the UI which cannot yet be replicated in Quaqua. Luckily a new library is being developed rapidy called MacWidgets that goes a long way to getting that native look. Take a look at this screenshot of what Ive done so far.

Discogs

Ive been spending some time playing with Discogs whilst its not nearly as accurate or democratic as Musicbrainz it is still a very good resource, and contains a large number of releases not in Musicbrainz plus plenty of additional cross-checking information for releases that are in Musicbrainz. It is particulary good for electronic music and independent music from the vinyl era.

In the forthcoming release of Jaikoz I've added an Update tags from Discogs feature that works similar to the Update tags from Musicbrainz for any track that contains a Discogs Release url.
From Discogs we can currently get about ten different pieces of data including higher quality artwork.

And Because Discogs works only at release level you can copy/paste a single release url to all tracks in a release and Jaikoz will extract the track info from the release and will update them accordingly, so you can do this even for tracks without a Musicbrainz match. There will also be a Manual Tag from Discogs option added in a later release.

This got me thinking and in a later release Im also going to introduce an Update Tags from Musicbrainz Release Id so that if Jaikoz cant find the right release itself you just need to find the release id manually and paste it into all the tracks. This will be useful for those difficult to find releases, currently you have to enter the individual track id and for each track in the release which is a pain.

I'm also hoping that those of you who are active Musicbrainz contributors will be able to use the discogs information to improve the Musicbrainz database.

Jaikoz 2.5 released

Thanks to everyone for your feedback during the beta testing. Jaikoz 2.5 is now available, its better in many ways - enjoy.

2.5.0 Beta out now

Jaikoz v2.5.0 beta is out now please give it a go. One of the most exciting developments is support for fixing genres from Musicbainz folksonomy tags, and adding your own genres back to Musicbrainz. Unlike the majority of Musicbrainz data Folksonomy Tags are subjective rather than factual so it will be interesting to see how the folksonomy tag cloud develops.

Jaikoz is the first application to support submitting genres back to Musicbrainz. Before this release Musicbrainz Users had to enter tags using the website or write their own scripts to do this.

Testing of the New Musicbrainz Server

Ive been testing the new search server being developed by Musicbrainz. Currently Musicbrainz uses Lucene but the new server will use Xapian . Whilst Im not convinced that Xapian is intrinically better than Lucene it has helped the musicbrainz developers to solve some problems, and Ive been happy to help.

Jaikoz has embedded database

The new version is going to use the Apache derby database. This is a super small relational database that is 100% Java. So there should be no os specific problems , and because it can be embedded in your program there is no db install or services to install.

I am currently using it to store release and artist lookups from Musicbrainz so that a release/artist has to only be checked once. This already occurs in the current version of Jaikoz but everything is held in memory so memory becomes a problem over time, and the results are not kept between restarts of Jaikoz.

Im also taking advantage of this locally stored release data so that I dont actually have to a lookup for every track if I can find a good match by looking at already downloaded and used releases. Consider the usual case of all tracks in a release being looked up if the metadata is good enough the new system will only require
one track, artist and release lookup for the first track, all the other tracks can be matched using the downloaded release info , whereas with the current system a track query is done for every track.

So Jaikoz should be faster and use less memory.

In subsequent versions I want to use the database to hold track metadata instead of it being held in memory, this will then allow Jaikoz to be used on super large collectiuons without runing out of memory. But I'll never make the database the sole reposiitory of any info, so it will never be neccessary to preserve the database in order to access your metadata - it is only a tool to improve memory usage and performance.

PlugIns

I'm thinking about adding support for writing Plugins. Would this be of interest to anyone and if so would you be happy to write it in Java , or would you prefer a scripting language like Python ?

Scripting

I have been improving the Rename Filename/SubFolder Masks to allow a bit more control over how your files are named from your tags

i.e $if(%bestartist%,%bestartist%-)$if(%album%,%album%-)$if(%trackno%,%trackno%-)%title%
would could give you

The Cribs-The New Fellas-01-Hey Scenesters!.mp3
or
The New Fellas-01-Hey Scenesters!.mp3

depending on your metadata.

This syntax comes from http://wiki.hydrogenaudio.org/index.php?title=Foobar2000:Titleformat_Reference#.24if.28cond.2Cthen.2Celse.29

but for this release Jaikoz will not support much more than the simple if statement. However in later releases I plan to support more functions using a suitable templating/scripting language.

Jaikoz roundtrip processing of genres

I now have Jaikoz retrieving tags as genres from Musicbrainz, but not only that it can submit genres back to Musicbrainz. This means you can easily help build the Musicbrainz folksonomy, and you can use Musicbrainz to backup genres for your tracks. With a little bit of fiddling around it to easily tag your tracks with non genre information - such as the 'owned' tag.

The retrieve genres algorithm is as follows:
Get all tags for track
If any exist strip out any one we dont want using configurable blacklist
Favour tags that match Winamps genre list
If there are multiple matches pick the most popular tag one
If there are no tags that match Winamps genre list pick the most popular
If there are no matching tags repeat at release level.
If there are no matching tags repeat at artist level.

Only one genre is picked at the moment because most media players and audio formats only support a single genre so adding multiple genres doesnt seem to help much, but if you disagree let me know.

2.4.1 out now

New release today, originally I needed to do a release to fix a couple of issue with Jaikoz on Leopard, But I ended up doing a bit more

For brand new new installations (doesnt apply to upgrades) users will not see as many columns loaded by default, and on OSX they fields won't be colour coded. This is an attempt to make it a bit easier for users to not be confused by Jaikoz on first viewing. The Autocorrecters default configuaration now does remote corrects (such as Correct tags from Musicbrainz) as well as the local corrects.

Windows users will now see that their foreign language metadata can be viewed correctly in Windows Explorer.

Some options such as the View Artwork Fullsize and View Online were previously only available in the contextual (popup) menus, these have now been added to the top level menu as well.

The keyboard shortcuts have been standardised, and shortcuts moved to the most popular tasks.

You can now drag and drop a field onto another field.

More details at
http://www.jthink.net/jaikoz/jsp/news/build1046.jsp

Fixed the pesky IndexOutOfBoundsException Error

There has been one bug that some of you encounter the IndexOutOfBoundsException error. This has been very difficult to track down, I thought I had solved the problem in the last release but it was still happening occasionally. Anyway, I'm now confident this has been completely solved ready for the 2.4.1 release out soon

Musicbrainz and the BBC

You may know that the BBC uses Musicbrainz on the Music part of the website. Seeing as the BBC is probably the most repected broadcaster on the planet this is a great vote of confidence in the accuracy of the Musicbrainz data, good news for Jaikoz !

But I didn't realise until the summit last week that BBC employees add data into the database as well

http://wiki.musicbrainz.org/BBCTeamsEditingMusicBrainz

and it seems like their involvement is going to grow and grow, good news for everyone.

Musicbrainz Packages

It was great to meet some of the people behind Musicbrainz - a really nice bunch of guys. I have a much better understanding of what they are trying to achieve.

On a more technical level they have big plans for the database, the most important for Jaikoz is the concept of release packages. At the moment album are often released slighty differently in different countries - and sometimes there are multiple releases in the same country, each occurence of this generates a new release in Musicbrainz. But these releases are not tied together and there is no way to accurately distinguish between two versions of the same release and two completely different releases. As a user I normally want to see only one occurence of each release, and I want to see the one for the country I live in, Musicbrainz is going to introduce the concept of a Package to tie all the variation of one release together.

Musicbrainz summit

I'll be going to the Musicbrainz Summit in London this weekend, so should be interesting to see what they are going to be doing this year and see what it means for Jaikoz.

Fix for Leopard

A few of you have had an intermittent problem with Jaikoz v2.4.0 starting on Leopard, the error seen is shown as Unexpected Problem: n >= n where n can be any number and was linked to what additional devices you had attached to your Mac. This was due to a problem with a GUI library used by Jaikoz that has now been fixed. You can get a patch from the fixes page.

New native dialog support in Leopard

As well as the Java 1.6 release on Leopard the default release for Intel and PPC adds support for some native OSX features that are not available on the Tiger release . In Jaikoz 2.4 I made use of this to display a search box that looks like a native Search field


whereas on Tiger it looks like a standard OSX entry field



In the next release I hope to make much more use of these features, whilst maintaining Tiger compatability.

Apple release Java 1.6 on Leopard Intel

Apple has just released JavaUpdate1 - Java 1.6 for OSX Leopard (Intel). I tried it out with Jaikoz and it seems to work ok but I havent tested it extensively. If you install it using SoftwareUpdate Jaikoz will not use it by default but you can try Jaikoz with this if you wish as follows

• In Finder right click Jaikoz and select Show Package Contents
• Select Contents
• Right click Info.plist and select Open With Text Editor
  
• Find the line 1.5+ and change it to 1.6
• Save Changes
• Start Jaikoz in the console windows you should be able to confirm if it is using 1.6

Using 1.6 does not add any new features new Jaikoz, but it should increase its responsiveness.

Why have I created a Jaikoz Blog ?

Whilst the forum is a useful way of informing current users about new developments it has a few drawbacks.

• It doesnt discuss what I'm doing between releases, it is focused on new releases and the issues between them.
• It isnt so visible to new users of the site
  
• It isnt easy to browse chronologically
  

I'd like to keep you uptodate on what Im working on, and other things of interest, please feel free to comment.