Part 4 - Three Things to Understand about Fix Songs Results
Part 6 - What databases can be used for Song Identifcation
SongKong Tutorial: Part 5 - Fix Songs, Review Match Options
So you have run Fix Songs in Preview mode, but maybe it is not quite how you want it. This is the time to take a look at the options and fine tune, lets discuss some of the key options you may want to modify that affect song identification.
All these options are on the Match tab
Match: All Existing folders represent a single album
SongKong generally treats folders as an album, this means it will only allow a match to an album if all songs in the folder match to the same album. But it also tries to recognise the occasional folder full of essentially random songs and treat them differently, allowing songs in such a folder to be matched to different albums or not matched at all;.
But if you know that every folder really does represent an album (or disc of a multi-disc album) then you can enable this option to force SongKong to treat all folders in this way.
Match: Preferred Release Countries
Often Albums often have different versions for different territories and countries,
but there may be no physical difference between the track listing to help SongKong decide the release to choose.
For example the Italian and German version of a particular release may have different catalogue
numbers or be released by a different record label. But this is
information is very unlikely to be in your existing metadata so SongKong
cannot use that to decide the release to pick.
But you can select a list of preferred countries that your albums are from and selecting those countries will give releases from those countries a boost making it more likely that they will be selected.
We have a new release, named Dry after the seminal first album by P J Harvey released back in 1992, the only artist to ever have won the Mercury Music prize more than once.
Synology Release
This is an exciting release as it is the first to support the Synology Disk Station range of Nas servers ( as long as they have an Intel processor). The better news is we have tested with the most basic model the DS218+ and the results are great, almost the same as our main development PC !
Full instructions available here
If you are using Qnap instead, watch this space we should have a Qnap release very soon.
Naim Metadata Matching
Customers who have previously used a Naim server to rip their music to Wav have been able to transfer the Naim Wav metadata with our special Melco license.
Sometimes the Naim metadata is for a CD with a different number of tracks then the customer has. We used to just ignore these but we can now extract the album metadata and sometimes the track metadata.
We have also made improvements to parsing metadata edited using the Naim App (this is stored in the UserEdits.xml file).
We also now log to the Errors and Warnings section of the report any files that we had difficulties with.
Report Creation
After completing the task SongKong creates a detailed report, sometimes this could take a while. We have made some massive improvements to the way reports are created so now they are much, much faster.
Linux users, SongKong for Java 8
SongKong is now distributed with Java 10. But as a Linux user you can choose to install SongKong without the distributed version of Java. The songkong.sh is intended to work with Java 9 or Java 10 but you can still run SongKong with Java 8 if you wish, you just need to use songkong8.sh or songkongremote8.sh
Full list of improvements and fixes can be found here.
Part 3: Fix Songs - Use Preview Mode
Part 5: Fix Songs - Review Match Options
SongKong Tutorial Part 4: Three Key Things to understand about Fix Songs Results
The three most common misunderstandings about the results shown in a Fix Songs Report are as follows:
Metadata is not the Same as Filenames
SongKong fixes the metadata in your music files, by default it does not modify the actual filename so after matching your filenames may not necessarily match the metadata if previously they were badly named.
SongKong can very easily rename filenames based on metadata but it doesn't by default because this can cause problems for other applications and general confusion for the customer.
For example playlists are usually created as a list of filenames so if the name of a file changes this can break a playlist file.
Sometimes an artist changes their name (e.g Prince) so if SongKong uses a different version of the name than you were using previously this can be confusing
SongKong can easily rename files using the Rename files based on metadata in the Basic tab, but in the early stages we recommend against it.
Song Only versus Song matching
SongKong group songs firstly based on the folder structure and if that doesn't give results any existing metadata. If SongKong is unable to find an album match for all songs in a grouping then it does not match any of them to an album. This is important because the last thing you want SongKong to do is break up your albumsjust because it could identify some of the songs.
But if SongKong can correctly identify the song (as it usually can with AcoustIds) but not the album then we can add Song Only fields such as Title but do not modify fields such as Album to ensure we improve the metadata without possibly breaking up albums that SongKong simply could not find (perhaps because missing from MusicBrainz/Discogs databases) .
Genres are not changed by Default
Genres are subjective, what one person may simply describe as Rock, another may describe more specifically as Thrash Metal.
So although we can use genres and styles from Discogs we don't by default. You can easily remedy this by enabling Update Genres on the Basic tab. This will add genres but not replace existing ones.
If you want to replace existing genres then you need to make some adjustments on the Genres tab, we will discuss that later on.
Part 2: Status Report Customization
Part 4: Three key Things to Understand about Fix Songs Results
Everyone has different reasons to try SongKong, but it very likely involves Fix Songs, the automated song matching task.
Preview Mode
Without a license Fix Songs runs in Preview mode, this means you can see exactly how successful SongKong was in song identification but no files are actually modified.
After installing a license Preview Mode defaults to switched off.
Run Preview Mode on your Music Collection
If you did not run Fix Songs in Preview Mode before buying a license we highly recommend you re-enable preview and then run it now on your whole collection with the default options, or at least a sizeable section of it.
Summary
The report generated gives a very useful summary of the percentage of matches to different databases.
Matched to MusicBrainz
But it also allows you to see exactly what albums your songs have been matched to:
Song Changes
And what data has been added for songs:
Metadata Spreadsheet
Spreadsheet can be easily used to browse changes, new fields are added as Yellow, modified fields are added as Green:
It also gives the chance to find things you don't like about the changes SongKong would make. For example you may use certain metadata fields in a non-standard way and would not wish SongKong to modify these fields.
Part 1: Status Report
Part 3: Fix Songs, Use Preview Mode
Customizing Fields
By default the bar-chart only shows bars for the most common metadata. But SongKong can show a bar for any of the metadata it supports, and it supports over 130 metadata fields.
It is worthwhile reviewing the list of available fields by selecting the Update button and adding any that are you are interested in, this generally makes more sense then adding all fields.
Artwork Size Fields
The Status Report also shows a count of how many songs contain artwork, and also how many songs have artwork larger than 500 pixels. This gives a better idea of the percentage of songs that have artwork of an acceptable quality.
You may be interested in knowing how many songs have artwork at a higher (or lower) resolution then this. You can easily modify the Show artwork at least this size option to do this.
Now when you run the Status Report it will use the revised figures. If we increase the value of Show artwork at least this size to 1000 the status report shows a count of Artwork of 100%, but only 32% for Artwork with width of 1000px or greater.
We are pleased to announce SongKong 5.10, mainly a bug hunting release.
Release Names
Firstly a bit of fun, starting with this release we have given each release a name. Apple used to use wild cats and now uses places in California for macOS. Google uses something sweet for each version of Android
SongKong uses the name of a favourite album, but it must be a one word album name. This restriction has prevented adding any albums by some of our favorite bands (such as Radiohead) but one word names work better for release names.
The first release name is Loveless by My Bloody Valentine, if you have never heard it, its well worth a listen.
Difficult Wav Files
ID3 Chunk
The Wav file format uses chunks for different types of file information. It uses a DATA chunk to actually store the audio, and a fmt chunk for basic meta information about the audio. It can then store metadata within an INFO chunk or a id3 chunk, or both, or none. There are many other chunks that can be added to a Wav and the chunks can be in any order.
So reading and writing changes to Wavs is complex and can go wrong. I had one such example recently the customers Wav files contained ID3 chunks storing the metadata, the content of the chunk was correct but the chunk type of ID3 was incorrect it should be in lowercase, i.e id3.
So when SongKong read the file it ignored the existing ID3 chunk and wrote a new id3 chunk. Everything appeared fine the changes could be seen viewed on their computer, except the modifications made by SongKong were not showing up in the UPnP control app.
The UPnP server was Twonky, Twonky have only recently added supported for reading id3 data from Wav files. But unfortunately they read both id3 and ID3 chunks and because the ID3 chunk came first they used that one containing the old information.
We have now improved SongKong so it recognises an ID3 chunk as a ID3 chunk, but it saves the chunk type as id3 so that the file is now correct.
Padding
We had a problem with some other Wav files. After the audio data the file had some null padding, a series of zeroed bytes and this prevented later chunks being read.
Null padding can be allowed in Wav files but it should be within a PAD chunk. In this case it was not within a chunk, but we have now improved SongKong to recognise this invalid padding and remove it from the file on save so that the file is now correct.
Bug Fixes
This release contains a number of bug fixes and improvements, all listed on the News page
Part 2: Customizing Status Report
This is the start of a SongKong tutorial aimed at new users. SongKong is deceptively powerful, in fact some customers say it is perhaps too powerful !
So it is sensible to understand how it works before diving in and setting it to work on your music collection.
Status Report
Before you start making changes with SongKong we recommend you run the Status Report task over your complete music library, the first icon on the toolbar. This does two things:
- It creates a bar-chart giving you evidence of exactly what metadata your files currently contain
- Its creates a spreadsheet showing the location of your music files and all the metadata they contain.
and it doesn't take very long to run.
Completeness Bar-Chart
The bar-chart gives a nice easy to understand view showing the percentage of your songs that have a particular metadata field, i.e on the screenshot below you can see 98% of songs have an Album field, 93% have an Artist field, but 0% have a Sort Artist field.
Spreadsheet
The spreadsheet is in the standard well supported xlsx format. So now you have a record of your music library that exists totally independently of SongKong. This can be used as an archive of your music collection at that point in time and a way to catalogue your library.
It is multi-tabbed and shows all your existing metadata
We provide ourselves at JThink on providing the very best customer service. Every suggestion is considered and every problem investigated, SongKong is being continually improved. Over the years we have implemented more systems to help with this, so thought it was time to provide an update.
Types of Problem
Support problems can be split into three broad categories
- General help on how to do something
- Suggestion for improvement or new feature
- Particular problem encountered, SongKong not working as expected or giving errors
User Forum
The first two cases are best resolved on the Jthink User Forum
we try to respond within 24 hours 7 days a week, but by posting here you often get help from other SongKong Users. The SongKong Questions sub forum is usually the best forum to use. Facebook and Twitter are not monitored as closely so are not such a good choice as the forum.
Create Support Files
If your issue is in the third category the first thing to do is run Create Support Files. This is available from main menu in Remote Mode and the Help menu in the desktop mode.
First you need to enter a summary of the problem then this will upload SongKong log files and the last five SongKong Reports to the JThink server and then notify JThink that the support files have been uploaded.
The logs files will show more details then the Errors and Warning section of the report. If your problem is how songs were matched/renamed or moved it is much easier for JThink to resolve your problem if we can see your reports.
Email
If you need to send additional screenshots then email can used, but this should only be done after sending support files
Request for Additional Information
We may ask for additional information to try and resolve the problem. If the files are small you can use email, if they are larger such as music files then we recommend you use a file hosting service such as Dropbox (free for upto 2GB of data)
Remote Login
Occasionally we may decide that remote access to your computer would be the best approach. In this case you would need to install the free Team Viewer then by Email we can agree on a suitable time to remote connect, this usually would take no longer than 30 minutes.
Existing Issues and improvements
Whilst SongKong is not an open source project we try and be as open as we possibly can. As part of this you can view all the outstanding bugs and improvements at our JIRA tracker. Green issues are improvements/new features and Red issues are bugs, you'll be pleased to know most of the issues are Green.
SongKong 5.9:Improved Song Identification and Performance
We are pleased to announce SongKong 5.9 with improved song identification and performance.
Improvement to Albunack MusicBrainz Disc Ids
In SongKong 5.7 we introduced Albunack Disc Ids, this provides a way to identify complete albums without requiring acoustic fingerprinting, both speeding up song identification and increasing the number of matches that can be found because it was able to find albums without known acoustic fingerprints or existing metadata.
We have now improved this algorithm to get more matches. Testing against a 1200 album test set with SongKong 5.7 could identify 501 albums just using Albunack Disc Ids, with SongKong 5.9 that figure goes up to 570 albums, an improvement of 13 percent. (After subsequent matching with AcoustIds and metadata we can identify 1108 albums.
Better Performance
We have made a number of further performance improvements. Many of these improvements reduce cpu usage so have most impact on slower/single cpu computers. Such computers should see a 30% increase in performance.
- SongKongs internal database has been optimized to reduce the number of calls made to the database
- Scoring algorithm has been optimized
- AcousticBrainz data can now be looked up per album instead of having to do individual lookups for each song
Support Message now can be sent with Support Files
Now when you send us your support files you can send a support message at the same time. This makes the reason for sending the support files clear and avoids you having to send a separate email message.
Help can now be hidden
In Remote mode help text was displayed on screen making it easy to see what all the options do. But this does take up screen space so we have now added the option to toggle Help on/off with the Show Help/Hide Help button
Report Creation has own Progress Bar
Previously no progress bar was shown for the Report Creation that occurs after the task has completed. However the report creation itself can take some time if you have been working with a large collection. So now Report Creation has its own progress bar that increases as the report is created making it clearer when the task is going to complete. The Report Creation progress bar uses a 1-100 percentage scale.
User Log is now more useful
The user log, songkong_user0-0.log has now been improved. Previously user log only contained errors and a summary after a task had completed. We now add a summary of song groups matched as they are matched.
10/09/2018 18.18.12:BST:SEVERE: G1164:Matched to MB/AcoustId:Ave Maria:9 of 12 to Ave Maria:cf1d4e8f-ae69-46cb-9c5c-c941951ded58 by Vyatcheslav Kagan-Paley
10/09/2018 18.18.12:BST:SEVERE: G1164:Linked to Discogs:Ave Maria:9 of 12 to Ave Maria:7135466 by Vyatcheslav Kagan-Paley
10/09/2018 18.18.13:BST:SEVERE: G8883:Matched to MB/AlbunackAcoustId:Picture This:12 of 12 to Picture This:423f245e-1c75-4948-9f48-cc4c5c8159f4 by Wet Wet Wet
10/09/2018 18.18.13:BST:SEVERE: G8883:Linked to Discogs:Picture This:12 of 12 to Picture This:6859962 by Wet Wet Wet
Each line is preceded by date, and then a Group Id of the form Gnnnn. The Group Id is simply a way for us to identify a group of songs (usually a folder). Then we have a message explaining how matched to MusicBrainz or Discogs
Whilst the user log is primarily a debugging aid for use by JThink you may find this additional information useful.
Many Bugs Fixed
We have been hunting down some of the more obscure bugs in a sustained effort to try and get bugs down to zero.In this release there are over twenty bug fixes
The full list of improvements and fixes can be found on the News Page
Automatic adding of ISRCs to Jaikoz and SongKong
We are pleased to announce that the Albunack database now supports ISRC's (International Standard Recording Code). ISRC enables recordings to be uniquely and permanently identified.
We already had support for ISRC in Jaikoz and SongKong but the data was not in the database. Now it has been added you'll see your existing versions of Jaikoz and SongKong can now add the ISRC for many of your songs.
Of 19 million MusicBrainz recordings in Albunack we have ISRC's for 8 million, that is coverage of 42%
SongKong 5.8 Released Today
We uncovered a handful of issues with the SongKong 5.7 release, these have all been fixed with this release.
One of these issues was that Edit Metadata failed to work in regular Desktop mode. This was due to the move from Java 8 to Java 10 in the last release, this exposed a bug that wasn't picked up because I had retested Edit Metadata before the move to Java 10.
We also at last fixed an annoying issue whereby SongKong would sometimes report it was cancelled because of an Unreported Error. Because the error was unreported it was very difficult to track down what the underlying problem was, but eventually we got there and it has now been resolved.
Full list of fixes can be found at http://www.jthink.net/songkong/news.jsp
We are please to announce the latest version of SongKong today. There are many improvements in this release but we have concentrated on improved performance and further improvements to the Web Remote User Interface.
Albunack Disc Ids
We have added a new way of identifying your songs. If you already have folders representing a whole album (or disc of a multidisc album) and the filename/metadata can correctly identify the order of the tracks then we create an AlbunackDiscId based upon the order of the tracks and the length of each track, we can then compare that with albums in the MusicBrainz database.
This is similar to the freedb lookup but has three important difference:
1. You don't need the CD, it may never even have been available as a CD. The Freedb algorithm is based on CD disc offsets, AlbunackDiscId is based on track lengths. Both calculate track length to the nearest second.
2. We are looking for matches in Albunack database not Freedb. so once a match is found SongKong can add high quality MusicBrainz metadata not poor quality freedb data.
3. It is more accurate, freedb can return matches for CD's where total album length and number of tracks is the same but individual track lengths are different. Albunack Disc Ids only match if all the track lengths match as well.
This method allows us to find matches that we could not match before, since there are many releases in MusicBrainz that are not yet linked to AcoustIds.
In this first release we only use against MusicBrainz releases since these have more accurate track length data. But it also improves performance significantly as we no longer need to generate Acoustid fingerprints for all songs. However we do sometimes generate and check some acoustids for the matched release as an extra verification check.
Acoustic Fingerprinting
Previously we always fingerprinted every song and then did a lookup to find the AcoustId of each song. This only has to be done once for any song however the fingerprint generated is very cpu intensive and on slow machines can contribute significantly to total tagging time. Even on fast machines if the files are on the network then fingerprinting will be significantly slower. So now instead of automatically fingerprinting songs at the start they are only fingerprinted as required during the Match to MusicBrainz stage.
For this reason the Acoustid Fingerprinting option on the Match tab has been modified. No longer does it default to true and mean that it must be enabled for fingerprinting to be performed. It now defaults to false and should only be set to true if you want to force SongKong to create fingerprint and get Acoustids even when a successful match has already been achieved.
Other Performance Improvements
We spent some time evaluating SongKong performance using a profiler and found some unexpected bottlenecks that we have fixed.
Although we store AcousticBrainz data in Albunack we only have some of it because unfortunately they have not released a database dump for some time. This if we don't find AcousticBrainz data in Albunack we then lookup in AcousticBrainz.
But a lookup is required per track and if you rerun FixSongs on previously matched tracks that do not have AcousticBrainz data it will recheck. So now we are also now caching AcousticBrainz lookups so we dont retry failed lookups. If you really want to retry in case AcousticBrainz data has been added for the tracks then just run Empty Database, (ensuring you do not need to revert any changes first)
With the introduction of Albunack Discs Ids and all the other performance improvements you should typically see that SongKong performs 3 times faster then the previous release.
Remote User Interface
We have made further improvements to the web browser user interface.
You can now add, edit and delete rename masks and javascript functions to use in these masks, just like you can in the desktop interface.
We have moved tasks and links from the footer to a fixed navbar so that they are always visible even on vertically scrolling pages.
And the footer now just contains links to the SongKong site and social media accounts, these will all open in a new tab.
The Edit metadata task tables now automatically size for content.
Hi Res Screen support
SongKong has now been updated to Java 10, and this now provides full support for hi density resolution displays. This means screens whereby the ratio between pixels and resolution is not 1:1, the additional pixels are used to provide a sharper display.
There are many more improvements and fixes in this release and you can find the full list at http://www.jthink.net/songkong/news.jsp
