In Development Land Again: DupliFinder, Duplicate Image Removal

Interesting enough, the more stressed I become the more I desire to program. And not just random programming, but programming to solve a problem (namely one of me :)). What I’ve been working on over the past couple days is a nice Windows utility to work with and manage duplicate files. Right now it works on the head of the file (64KB) or entire file for comparison, but future versions (such as the one I’m working on now) are going to include audio stream matching (to compare audio regardless of the format or compression) and video frame matching along with a host of other options. Here’s a screen shot of it in action:

DupliFinder Duplicate File Finder

DupliFinder Duplicate File Finder

As you can see the UI is almost complete, allowing you to select a root folder and you can scan current and sub directories optionally. The only problem with it at the moment is it’s single-threaded, so when it is given a large folder to process it feels as if the program has frozen, but don’t worry it’s crunching away in the background.

The feature I’m adding right now is a preview/compare pane for easy verification that the file is the same, as well as giving the user to option to delete the “duplicate” or “original” in case of file confusion. This will be ready to go for the v1.0 release of the utility.

If anyone is interested, I can package up a preview version that has some quirks and the single-threaded problem. I had looked all over for a program that would allow you to delete duplicate images and other files and it’s ridiculous the amount of money those scam-like software companies try and charge for such simple (but effective) software! Anyways I’m contributing back to the community by releasing this under GPLv2. On top of handling photos (for which I’ve seen some software cost $30US!) it will handle all sorts of stream and data related information.

For any programmers out there reading this, it was programmed using Visual C# 2008 on a Windows 7 development box, so all sorts of shiny Windows 7 features may start peeking through soon :).

If you have any comments, please feel free to leave them below, and if you would like to play with the source or the compiled version of this app, just let me know in the comments and I will fire off some more information and a package. That’s all for now!

Theory In Object Detection

I have been playing with edge detection for a few weeks now, and slowly I’m gathering a solid knowledge base related to how to do object detection. Primarily I have been playing with facial recognition, but my end goal is to be able to track hands, for a virtual table PC, so that when I build my surface computer the adaption of the algorithms is easy. The goal is to use edge detection and to determine the shape of the enclosed object.

The current algorithm that I am building is based on some assumptions:

  • The person is feature complete (two eyes, two eyebrows, one nose, one mouth)
  • The room is relatively well lit (haven’t incorporated luminosity correction for shape detection, need to work on performance)
  • There is a notable edge difference between these features (enough for the algorithm to detect edges)

Assuming these things, the edge detection is mounted in a matrix and through various reduction methods, it reduces the amount of noise (isolated, or only sharing an edge with one cell). From these shapes we can determine facial structure based on clusters of a predefined shape.

While this is fun for face tracking purposes, it strays from the principle concept that is trying to be achieved with virtual touch computing. Facial features are relatively easy to track as they are static in context of each other, so predictable patterns even in rotation are possible. Fingers on the other hand are a more dynamic object to track, as spacing, position count are likely to change frequently. For example someone who wants to drag an object around may only hold out one finger and drag the object where they want to move it, while a rotation might use all five fingers.

On a surface such as a touch PC, this is relatively easy through the user of IR LEDs using the aforementioned FTIR (frustrated total internal reflection) methods of light detection, or through the idea of using a set of IR lasers close to the surface with planers to diffuse the beam over the table, but virtually there is lots of noise in the image. The algorithm that I am currently working on tries to filter out the noise through the following methods:

  • Identify the individual in the image (or what is visible of them) and filter out the background image to 0’s in the matrix
  • Process the general form outline, find where on the matrix the general shape of the hands are, and once this is processed identify the digits
  • Filter out everything except the digits and return the points which are visible

This sounds easier than it looks, however there are many limitations and issues when it comes to distinguish-ability. Another problem on top of this is light conditions, multiple people in the image and detectability of their shape. This is the area that I am working on now for performance reasons. Regardless, the skills I obtain developing this algorithm will make surface digit detection trivial in nature.

If anyone has any suggestions on how to detect full human motion based on image capture and processing, please let me know. I am always looking for a good algorithm 😉

Edge Detection Using C#

Whoa, it has been way too long since I have been on here, but I am coming back with some interesting new projects. I have spent many months working at the Pythian Group as a Perl software developer. Fantastic experience in my opinion, but anyways, I have been experimenting in my free time with using C# for various projects. I want to use it primarily because it is an easy and efficient language, but secondarily to help improve my skills.

I have been playing with my Neural Network software, and with the idea of starting a company. Basically I got my software to a state where it can parse data from text based sources into the neural network and firing a query would highlight related items down the neural path. I am going to continue playing with this idea as I think it has potential in specialized searching and data recollection niches.

The new toy project I am working on is Edge detection and cleaning in C#. Now I know there are like a billion projects out there that play with these ideas, but I would like to be able to play with it myself. I have assembled a working demo of this project, though the source is not quite ready to release yet for everyone’s eyes. Currently it uses a simple color edge detection algorithm, as that sort of algorithm will plug in nicely with my image signature detection algorithm that I was developing a few months back. Basically all it can do now is edge detection in real time, which is a feat for me, I have been away from the specialized programming for several months now, but am slowly getting back into it.

Edge Detection Algorithm at Work

Edge Detection Algorithm at Work

This is a sample shot from an early version of my algorithm. It detects edge colour differences in a given webcam image and currently overlays the original bitmap. While this traditionally works quite well, it tends to produce a lot of noise and wide edges. For this reason, I’m working on a newer algorithm that slices the image into zones, which then calculate the noise ratio independently and adjust the thresholds for the edge detection appropriately. Currently the changes are looking promising but need to undergo a large amount of changes before I will be able to rely on it for all my data processing.

The next step is to detect “object” edges, which are complete logical elements in images (house for example, window, etc) and process them into sub-object “textures” to be stored and signed using my other algorithms. This will allow the camera to begin to categorize and classify items in the real world, for what will be unveiled as my larger scale project in future posts. Hopefully sometime soon I will get a chance to clean the source and upload an example of my edge detection algorithms using the zone sampling method.

I am now finishing up my co-op term, meaning that I will have a lot more time now to play with programming, so expect more posts in the coming weeks and months!

Updates and Development Abound

I’m posting a few notes of projects that I’m working on, and design decisions that have come to mind in the past couple weeks. First off, exam season is finally over, meaning I’m back open to free thought! Well, I was before, but assignments and studying tend to encourage a one-track mind. Anyways, I have started developping a couple neat applications, both of them mostly for personal use but who knows. The source will come up this week, though not in its completed iteration.

Project One: The RegEx String Generator

I’ve been toying with this for a while, mostly for content organization, testing and such. As a result I’ve started to develop a DLL that allows its users to give a regex string as an input and either ask for iterated values (such as iterating through all possible strings) or to generate random values. I think that something like this would allow for much easier application testing as you would only need to include the library in your project and run test patterns on it. So far it has literals support, integer (both range and digit iteratation), and soon will support character ranges (and traditional arguments such as ‘w’ in regex). Overall this is still a young project, but with promising results.

Project Two: Surface PC

Now that it’s summer and I will start earning some money again I can start to develop my surface pc. I have sourced some suppliers for the components, and aside from working out the projection setup (have to amplify the projection distance somehow) I have figured out most details. I’ve decided to forgo FTIR for my first table iteration, and will try using corner mounted linearly planed lasers. This apparently has a much brighter result for the camera and will also allow a narrow band filter to be applied to the lens, allowing for daytime operation (a limitation of most general design FTIR tables). Also, I will start to write some multi-touch applications in the near future, though the issue remains that I will not be able to test them until I am sure that the table is fully functional. The project budget is looking near $850, which isn’t too bad, considering that includes a projector and basically every component (aside from the PC itself, though I am looking at a high def silent PC with a budget of around $300, so this might be a silent, nearly instant-on touch PC when I’m done with it). More posts and details and a detailed walkthrough to come as I design and build the table!

All in all, this summer looks promising for side projects, and given that I have my weekends free this year (unlike previous years) I will finally be able to once again make some progress on development. Look back here mid June time for the start of the Surface PC saga.