Saturday, October 31, 2009

Another reason Apple's XML is pointless

A few posts ago I talked about how to read one of Apple's Plist XML files. Sure it works, but I have another reason to despise it. Not only is it tough to read in your own parser, not only is it tough to find a third party parser that will read them, it now sucks the memory of your computer too. Using the Apache parsers I found before I was able to read the directory iTunes stored music to, but it wasn't until I tried v.5 on other computers that I was getting memory heap errors, or basically the virtual machine was running out of memory. It took a few hours to pin the error down to reading the XML file. I watched the amount of memory that was used by iTunesDSM before and after reading the XML file. The memory used would be 50mb or so and then balloon to 300ish mb. Ridiculous. Of course that's not to say that other XML files wouldn't produce the same amount of memory requirements, the XML file itself is somewhere around 12 megabytes. But first impressions last, and I will always try to avoid Property List XML files.

I did fix it though. I strictly read the file for what it is now, instead of using a parser. I read the lines coming from the xml file and scan for the string that contains the music folder references. Not only is it much, much, much faster, it takes a lot less memory.

I also made a few changes to the FileWriter class. Instead of the CleanDirectoryTask coming up with files that don't already exist, the FileWriter does. I also made a few changes to the dialog that appears when copying files. As before a simple progress dialog appears when copying files takes longer than a set amount time. It's a neat way of doing it...and it's in the ProgressMonitorInputStream class.

Friday, October 30, 2009

What I worked on today

After figuring out specifically when iTunes makes duplicates at least as far as I can tell. I was able to remove the feature that would temporarily move a new track from the itunes directory to a temp folder and then re-add the file to iTunes when iTunes was set to copy music to the folder. iTunes seems to keep from creating duplicates this way, so it's not needed anymore. I've also removed the requisite options.

Exclusions are still in though, this helps keep iTunesDSM from re-adding files outside of the library and thus preventing duplicates since one of the ways duplicates occur is by programmatically adding new tracks.

Also, I've been trying to keep Mac functionality and one area where this was a problem was in automatically locating the iTunes XML file. Well using System.getProperty("user.home") I am now able to automatically locate the xml file without user intervention on Mac. If the user has installed iTunes elsewhere and iTunesDSM can't find the xml file, then it asks the user. Another mac problem that seemed to come up but turned out to be a non issue, is if the options file was open in mac, a Concurrency Exception was thrown by my ReadOptionsTask class. Turns out you can't have the file open in Mac, but Windows it's ok. I have a feeling Windows is using a temporary file there.

Next, when iTunesDSM opens the first thing it does is look for the xml file, so sometimes, it looks like it's taking a while to locate it and the gui would drag. I've now added an indeterminate progress bar to let the user know it's finding it.

I've added a couple things to the blog. Mainly some personal information about me. I also added a features list on the side, I hadn't realized that was missing but yet so vital. It's about to get a lot bigger too!

Have a good friday everyone.

When does iTunes make duplicates???

Ok I've done some tests and here's when it happens.
  1. Tracks are added to the Automatically Add to iTunes folder and the files already exist.
  2. You have reinstalled iTunes and you set the copy to itunes folder option on, and then add music from the same folder that iTunes is set to store music. Every file is duplicated.
  3. Adding songs programmatically. If you are adding tracks from outside the iTunes library and iTunes already has those songs in the folder (and the copy option is on) duplicates will be created.

Thursday, October 29, 2009

Couple more updates

It's kind of weird to figure out exactly when iTunes will create duplicates. So I've added an exclusions list. Basically, it's just a text file that contains the paths of files that have been added to iTunes through iTunesDSM. The usual SwingWorker classes have been added too. I'm done for today.

v.5 Updates and Screenshots

A few updates to talk about today...

I have been working on seperating out the logic. As noted before, the SwingWorker Tasks I had were bloated because I kept adding on top of them, especially the task that operates on iTunes. Now the itunes task has been split up into an AddTracksTask class and a RemoveTracksTask class. These two tasks now handle communication to and from iTunes as well as a smaller ITunesWin class that does the simple things like getting and creating the playlists.

Also I've done some work on the options panel. Like the other day it will load all of the options from a previous instance of the program. It didn't remember the extensions that were selected from the previous time, but now it will. Basically every option that is listed in the options panel is now reloaded from previous executions. I've also simplified the act of getting the extensions from the options object, it now directly returns the extensions array and not the Extensions object which you then get the extensions array from.

Also, in the options class is the isReady() method that checks to make sure all the options are set that would be required for other tasks to run. For example, checking for duplicates before setting a directory will now throw up a dialog telling the user to set options. This makes it much easier when I'm about to start a SwingWorker task as all the settings are checked in one place rather than before each execution.

I realized I haven't posted any screen shots in a while so I added a few to this posting.


Approval approved

I got my topic idea grade back, it looks like I get to keep developing iTunesDSM! I plan on recording the next update's youtube video today. And mayyyybbbeeee posting the next version here soon. It's already been termed version .5.

Wednesday, October 28, 2009

The official topic idea I submitted today

Posted here for consistency...

I’ve been referring to my program as iTunes DSM (Duplicate Song Manager) and have been keeping up with the blog that acts as the design diary. In the past few weeks, I have added the ability to add songs to iTunes in a careful manner to prevent iTunes from duplicating files. What I have in mind for this semester is to move beyond the proof of concept phase and start fully testing the program as well as documenting it. I would like to use the techniques we learned in the SE Testing class as well as learn JUnit to gain more confidence in the program functioning correctly. I also need to design more classes to distribute the logic. One of the Java techniques I have learned is implementing SwingWorker classes that act as background tasks for Java Gui’s but the side effect has been a lot of code in just three classes. Testing the program will help me divide up the logic more thoroughly.

Coincidentally, quite a few people have found my program useful, including my family so I hope to move it to a more robust state.

iTunes duplication and version .5

I've spent a considerable amount of time in an attempt to prevent iTunes from creating duplicates when you add songs that were in the iTunes music directory but not in the library. iTunes used to create duplicates in this situation, but now it seems it's much smarter. In testing, I have found that regardless of the option in iTunes that copies tracks to the directory, iTunes will not make duplicates. So this means, if the option is on and files are already in the iTunes directory, they will not be duplicated when added. It seems iTunes is slowly whittling away the errors that would create these duplicates.

It also seems that iTunes is much more careful about creating duplicates in the auto add folder. When version 9 first came out, it was more difficult to add duplicates, but you could still force it if you added songs to the automatically add to itunes folder. Now itunes will still create duplicates but not add them to the library. So you might not notice duplicates are there, but it is still much less likely you'll accidently duplicate tracks.

So, I'm trying decide on whether to keep the option that fixes the old copy files bug. I know for the most part everyone is pretty much using the newest iTunes. In a sense, you are forced too.

Anyway, I'm still working. I'll probably record a new video here in the next few days for the next version.


Ok it seems, in the off chance there is still a possibility to duplicate songs that are added. Here's what happens. Suppose and album is already in the itunes directory but not in iTunes. Suppose now I add that same album outside the itunes directory on accident programmatically or by hand. iTunes will create duplicates and reference the duplicated files. Bad.

Again it's an off chance of something happening, but it means my hard work on keeping iTunes from doing this is a good thing.

Tuesday, October 27, 2009

How I have to handle options and jpanels

So I've been messing around with how to handle user options. It's amazing how much you learn from just doing. I have an Options object that I initialized in my top level object, the main class. I first tried to pass this object to all the objects created underneath i.e. my Panels. That didn't work because apparently JPanel constructors have to take zero arguments. You could tell that some kind of error occurred when Netbeans couldn't render a live view in the Design gui builder tab. So next I tried creating zero argument panels that right after have their options objects were initialized, I set the options object using a Setter method. This sometimes worked and sometimes didn't. It was really frustrating when sometimes Netbeans was able to render a preview and other times it couldn't. Netbeans is subtle about errors that appear by showing a little red circle with a white line icon at the bottom right of the screen. It took me a while to notice this since it would disappear quickly.

So how do you handle user options that must be synchronized between different objects when you can't pass the object in the constructor? I'm not sure how other programs do it. I'm guessing this isn't just a Java thing. But what I worked on today was a completely different way of handling options between objects. I basically, write the options to a text file and then read the file over and over again.

This presented a different problem though. It is now necessary to read the Plist xml file that iTunes maintains so that iTunesDSM and iTunes do not create duplicates. Reading the file takes a while. Back to my old friend SwingWorker. It's actually pretty easy to use SwingWorker objects. Later I'll post some examples. These have the distinct advantage that they don't freeze a GUI when longer tasks are executed. Basically the task is run in the background and the GUI still interacts with the user.

Now what basically happens is, as soon as an option is selected a SwingWorker object is instantiated to write the new options to the file in the background. When options are accessed later another SwingWorker object reads the file and returns the option object.

Tomorrow I need to read the Options object and set the Options Panel to the settings read in the file. That way, you set the options and they stay. Also, tomorrow the official topic idea is due. For formalization sake, I'll post the topic idea on here as well. The topic is supposed to be about what the final Capstone project will be and I want to use this project as my Capstone idea. If I am able to, then I'll push ahead with formal testing of iTunesDSM.

Sunday, October 25, 2009

Options and Settings

Making headway. I'm realizing how much of an update this is really going to be. I now have an options object that will keep track of user options and eventually I want to output these options to a file so that they don't have to be reset all the time, but for now they will reset when the program is re-run. I'm also consolidating the File types to filter options and placing that on the settings page as well. I haven't really decided how that will happen though. Also I'm noticing how large the SwingWorker task for ITunes functionality is gettings. Unfortunately, I'm not really sure how to fix that. It's not really good coding practice but I don't see how to make it any better as of yet. More to come this week.

In another note, I noticed I have a follower of my blog! That's cool. Hope everyone has a good rest of the weekend.


Saturday, October 24, 2009

Watched folder adding now keeps iTunes from creating duplicates

Short update today. Yesterday, I got iTunesDSM to correctly identify media files that were already present in the music folder and now keeps iTunes from creating duplicate files from those. I had to use the previous post's solution on how to read XML property list files to learn the location that iTunes stores music. The basic trick is to temporarily move the media file to an outside folder and then add the file to iTunes from that location. This is dependent on whether iTunes copies files to the media folder or not. There is still work to do as the number of options that are available are mounting. I have been debating on creating an Options object that just maintains all options the user would use. If I do then a new JPanel needs to be created that will list all the available options. I guess I got my work cut out for me.


Friday, October 23, 2009

Read a property list XML file in Java

Plist for short, a property list XML file is used by the Mac Operating System to store information about a whole host of applications for Mac. It's even used by iTunes on Windows as a secondary interface for information about all the tracks and playlists in iTunes. I have been searching forever on how to read these files without having to design my own Java classes as Plist files are simply pointless. My research report was on XML and Plist xml files simply destroy the XML ideal in my opinion. Regardless, someone at Cupertino thought they were a good idea. I finally figured out how to read them using Apache Commons java packages.

I jotted this down in my notes for iTunes DSM but you basically need to download these packages:
  1. Apache Commons Configuration - Plist XML files
  3. Provides Exceptions for Apache
  5. Apache Logging
  7. Apache Predicate
  9. Apache Base64
The notes are just for me to know exactly what classes are required by the URL's listed there will take you to the web pages to download the packages. I usually just download the Jar files as it is easier later to simply replace one file should an update be needed. Once these packages are imported I use this code to get the location iTunes is storing music in.

import org.apache.commons.configuration.ConfigurationException;
import org.apache.commons.configuration.plist.XMLPropertyListConfiguration;

public class ITunesXML {

private static String SPACE_CHAR = "%20";
private static String LOCATION_PREFIX = "file://localhost";

private XMLPropertyListConfiguration config;

public ITunesXML(String location){
config = new XMLPropertyListConfiguration(location);
catch(ConfigurationException e){
System.out.println("Exception ITunesXML Constructor");


public String getITunesDirectory(){
String loc = config.getString("Music Folder");
assert loc != null;
assert !loc.equals("");
loc = loc.replaceAll(SPACE_CHAR, " ");
loc = loc.replaceAll(LOCATION_PREFIX, "");
File file = new File(loc);
assert file.exists();
catch(Exception e){
System.out.println("Error in getITunesDirectory");
return loc;

public static void main(String[] args){
ITunesXML xml = new ITunesXML("C:\\location\\to\\iTunes Music Library.xml");



So here's a simple example on how to read the Mac OSX property list xml files in Java using Apache Commons packages! Now on to code the rest of the bug fixes.


For new users....

I realized this morning I have a lot of postings about things that don't work. These are all references to the next version (.2). The version you can download now (.1) is still good to go. It will still remove duplicates from the iTunes folder and clean iTunes of orphaned files. You can download version .1 on the right side of the blog.

This blog is used to fulfill a requirement for my Software Engineering masters project and so this is acting as a diary of the design/build process.

On another note, I woke up this morning with a solution to the previous posting about duplicates being created by the Add Tracks process if iTunes is set to copy songs to the folder. As of now, the solution depends on asking the user if the Copy option is selected in iTunes and if it is, then iTunesDSM will make a temporary copy of the track to an outside directory and then import that file into iTunes simultaneously deleting the file that was already in the iTunes folder. Basically, copy file, delete file, import file, iTunes copies file. If iTunes does not copy files to the music folder, then the file will remain in place.

I will try and code this solution this afternoon.

Thursday, October 22, 2009

Found a bug today

I've been trumping up the ability to add songs to itunes that are in the directory but not in the itunes library, for example, the file exists in c:\music where itunes stores the music, but the file has not been added to itunes yet. The bug occurs when the "Copy files to iTunes Media Folder when adding to library" option is turned on. Here's what happens....
  1. Add songs to library using iTunesDSM...itunes adds the song to the library and adds the file to the organized folder in the library root folder
  2. If the file was already there, it is duplicated, i.e. a number is added to the end of the file.
  3. iTunes references the duplicated file in the library...not the original.
  4. Tab over to the remove duplicates tab in iTunesDSM and the program dutifully finds all the recently created duplicates and the user backs them up, deletes them, and removes orphaned files in iTunes.
  5. All of the songs that were added in step 1-3 are removed since they are orphaned and we are now back to where we started.
How to fix this in the short term is to simply turn off the "Copy files...." option in iTunes while the adding process is progress assuming the files you want to add were already properly organized by another user.

If they are not properly organized before hand, the files will remain in place and itunes will reference that song outside of the usual itunes library.

This will be a tricky fix and will delay the next release.

In other aspects of the program...I have created a new icon for the program (at the top). It's basic but it gets rid of the java cup icon. Also the backup function will ask the user should the duplicate file overwrite an existing file. It's unlikely to happen but it needs to be there.


Wednesday, October 21, 2009

Still Working...

Still working. I keep messing around with different ideas I get. iTunes DSM is mostly ready for a next release but I like to tinker. You can still download version .1, but .2 will definitely have manual folder watching.

I have been thinking about how to run folder watching in the back ground, but the more I add, the more restricted the program becomes. For instance, the folder watching now only works in Windows as do most of the other features, which are disabled at start up on a Mac.

Some other more official blog notes...
Even though I wrote the MediaFiles class to find the files under the music directory, I can't use the class as SwingWorker classes allow you to cancel a task but that cancel boolean can not be passed to the MediaFiles class. It's a challenge I'm sure I'll figure out, but for now, I'm stuck with some repeated code.

I'm also having trouble rendering JCheckBox's in my SongTable class. It seems that a default JTable cell will render a JCheckBox when a Boolean object is passed into a table row, but if you want to change how the cell's look with a DefaultTableCellRenderer class the JCheckBoxes don't seem to work right. Another challenge, this one though is not really necessary as you can still select that might go to the chopping block for now.

Lastly, the Java default icon is lame. I've been thinking about a new icon for the DSM, but I'm not an artist. I spent a few minutes messing around with an icon idea and I've drawn a few sketches but that's about it. Anyone want to design an icon for me???

I'm also getting comments on my youtube video. I appreciate those - it lets me know that people are getting use out of the program!


Saturday, October 17, 2009

JTable work today

Worked on the JTable that display track information. It's much prettier! I'm also cleaning up my code. It's all working still, but I think next week I'm going to go back to making it robust.

EDIT: I forgot to mention...I also turned on table sorting, so a user can sort the table by clicking on the table headers. A nice feature.

Thursday, October 15, 2009

Improved iTunes DSM Folder Watching

Big update today! I'm still working on the portion of the program that adds tracks from other directories to the iTunes library. You can see in the screenshot the new tabbed interface. It may not be my favorite way of doing things but its a simple way of it.

The new folder watch mechanism does what you think it would. iTunes DSM will now scan a directory for files/songs that are not in the iTunes library and add them to iTunes. It first creates a playlist, adds the song to the playlist, which are then added to iTunes.

The playlist must be created as that is the only way to programmatically add songs to iTunes, but once the process is completed you can delete the playlist.

I'm still working out the kinks and the code is really ugly but soon, I'll post a new version. Weekend is coming up! Hope everyone has a good one!

Saturday, October 10, 2009

iTunes DSM Watched folder ...esque

Today, I've been experimenting still on a watched folder implementation for the iTunes DSM. At my house, sharing the single music folder over the network, it can be a pain adding songs when one user has bought a new track and it doesn't show up on other's iTunes. The implementation so far seems to work, of course it's all command line based at the moment. Look forward to a usable gui implementation in the near future though.

I also seperated out some of the logic in the program. Searching for media files was once the task of the TracksTask class, but has since moved to it's own MediaFiles class. This has many obvious advantages, one thing that took a while to fully understand is that the MediaFiles class searches for files recursively as before, but the class creates an ArrayList of File arrays. So basically, there is one array with a bunch of arrays in it. The arrays inside the array list are essentially the "folders" that were found within the root music directory and those folders have the filtered files in them. Doing it this way allowed me to maintain the duplicate searching method the same as it operates on a "folder" of files as well.

Anyway, I registed for Google analytics the other day and I can see the traffic my blog is getting. It's kind of neat to watch the traffic increase and hopefully people are getting a use out of my program! Hope everyone has a good weekend.


Wednesday, October 7, 2009

What's in the future?

I haven't had much time to work on iTunes DSM this week. I've had exams and projects due, but every now and then I'll open up iTunes DSM and work on it. I've been cleaning up code, and adding documentation. I've also been exploring what I want to add next. I've got a couple of ideas.
  1. It would be handy to check the folder for files that have not been added to the iTunes library yet, so maybe some kind of folder watching mechanism. The folder watching mechanism will require some addition code but I do not think it will be difficult to implement.
  2. The options a user is able to choose from is increasing exponentially every time I add components so a better way to handle all the multitudes of operations is going to be needed.
  3. The GUI looks lame.
  4. The default JFileChooser object used on Mac operating systems also looks lame.
I have plenty of options to choose from and like most projects, it is usually new features that are added first before GUI updates are done. I'm almost trying to make this project my capstone project rather than my Instant Messaging application I was developing. It's been a while since I have worked on that and I'm simply not interested in it. The goal was to create something anyone could setup but it's just not viable with all the other instant messaging applications out there.

Saturday, October 3, 2009

First release available for download

Figured out how to host my program through my school's servers today. Here's how to use iTunes DSM.

  1. Be sure java is installed on your computer and you have JRE 1.6. Mac users you must set this property as it seems OSX uses 1.5 by default. Search in Finder for Java Preferences.
  2. Download and unzip the application from here DSM
  3. Run MusicApp.jar by double clicking it.
  4. Open your iTunes music directory.
  5. Click Find Duplicates
For a demo check out the YouTube video here

DISCLAIMER: It goes without saying, I am not responsible for any damages to your music library. In my informal testing the program appears to function correctly, but as always make a backup of your data before using iTunes Duplicate Song Manager.

Friday, October 2, 2009

iTunes DSM Video Uploaded

The demo video has been uploaded to YouTube. Here's the link...