The five year project: .NET FileSystemWatcher clone for Python


Posted:   |  More posts about python software

In the time I've been using Python, no project has started and stopped, and started and stopped again, more than my goal of writing a file system monitor. Sure, it's a small and simple project in the grand scheme of things that could be accomplished over that time, but I like to finish what I start. The idea originally came from my father, also a Python user, suggesting something to work on, likely to help me learn but it'd also help him out. Years ago he wanted a multi-tabbed text editor with tail -f functionality. I think I was reading through a wxPython book at the time and figured, "sure, I can learn this and make that tool." Started it up, had the shell of a simple GUI written, then came time to get the file system updates. I probably got distracted by something, got hooked on something else, then totally forgot about the whole thing. For whatever reason, this happened every few months. About three years ago I tried to rejuvenate the whole thing and found Tim Golden's great "How do I..." page (pretty sure Dad sent this to me before). He has an example, three of them to be precise, covering exactly what I wanted to do: watch a directory for changes using Mark Hammond's pywin32. Awesome. I got something coded up pretty quickly and took the library in a different direction, using it at work to write a Windows service that would monitor our servers and look for crash dumps and email the team. It was super simple and paid off big time, but I kinda just whipped it together and it was poorly designed. Fast forward to a few months ago. I was bored and looking for something fun to work on -- ah, that file system watcher I've been half-assing for years. I thought to myself, "now that I actually know wtf I'm doing, I should do that, and I'm sure my Dad would get a kick out of it." Somewhere in the middle of all of this I was writing C# and used the System.IO.FileSystemWatcher API, which was really nice. I've always wanted the same functionality in Python and liked what they had, so it would be cool to do what they did. A few blogs around the web claimed the Win32 ReadDirectoryChangesW API was behind the scenes of FileSystemWatcher. True or not, it made sense and I was familiar with that from the Tim Golden examples and my watcher service. I've been writing and reading a lot of C code lately so I started hacking. After reading up on a few things, I came up with a much better C equivalent of what I had in that Windows service. It's multi-threaded, uses IO Completion Ports, and seemed to work pretty well. Pass in a directory and a callable, call the start method, then you'll get callbacks for creating files, renaming files, etc. Sweet, we're on the way. After fiddling around with that a bit, I figured it was good enough to build on. I started writing some tests and had simple things like the following working. [code lang="python"] >>> import watcher >>> import os >>> callback = lambda action, path: print(action, path) >>> w = watcher.Watcher(os.getcwd(), callback) >>> w.flags = watcher.FILE_NOTIFY_CHANGE_FILE_NAME >>> w.start() # Then I opened up vim and created a file called "hurf.durf" 1 .hurf.durf.swp 1 hurf.durf 2 .hurf.durf.swp [/code] That was cool and all, but I want to be able to follow one specific file, or files that match a certain pattern. I also want to be able to set callbacks for specific actions. Hmm, FileSystemWatcher can do that. Maybe I'll just build out a clone and see how it works. One of the first things I wanted to figure out was how to emulate the callback attaching and detaching like on Changed events. I needed a container that supplies += and -=, which is none of them. Easy enough, just inherit from one and provide the __iadd__ and __isub__ operators. Before you get outraged: I know that's "unpythonic", but I'm going for a clone here. Filling in the rest was pretty easy. There's a bunch of properties in FileSystemWatcher that map to the attributes and methods of the underlying Watcher. For example, FileSystemWatcher.NotifyFilter sets Watcher.flags, which is an OR'ed group of NotifyFilters, which are constants exposed by watcher from Win32. The weirdest part of the whole thing is that starting and stopping FileSystemWatcher is done by setting EnableRaisingEvents to True or False. It's not a method called start or stop like in the underlying Watcher (or anything else that needs to start and stop). It felt wrong perpetuating this weirdness, and again I know it's "unpythonic", but I'm going for a clone here. As for translating Watcher callbacks into FileSystemWatcher callbacks that work with all of the fancy filtering, it's just a simple queue, a regex, and a big if/elif block. Watcher calls its callback which puts the action and relative path into the queue. FileSystemWatcher pulls it out, sees if it matches the filter, then we figure out from the action which callback to call. If it's a rename, do a special dance, but otherwise create an update object, fill in the details, then start calling back to the user. [code lang="python"] >>> from FileSystemWatcher import FileSystemWatcher, NotifyFilters >>> import os >>> callback = lambda event: print(event.ChangeType, event.Name) >>> fsw = FileSystemWatcher(os.getcwd()) >>> fsw.Created += callback >>> fsw.NotifyFilter = NotifyFilters.FileName >>> fsw.EnableRaisingEvents = True >>> # Opened up Explorer and right clicked to create a new file 1 New Text Document.txt [/code] There you have it. It took 235 lines of pure Python for FileSystemWatcher and 466 lines of C for watcher for this five year project to be completed. If any future employers are reading this, I'm capable of writing more than 140 lines of code per year to complete a five year project, I swear. The project is now on PyPI under the name watcher, complete with a few binary installers. It's 3.x only because 2.x is dead, but I'll do a backport if people are interested (email me: first name at python.org). The project is up on bitbucket: https://bitbucket.org/briancurtin/watcher. It's not really complete but it works pretty well for most usages. I know of a bunch of bugs that I'll eventually fix, but feel free to report more or even fix some of them. Thanks for the idea, Dad.

Contents © 2014 Brian Curtin