Sunday, May 15, 2011

Overview of the JDK7 watch service

Time' s running and I found myself with too few things to talk about this week, but to become a craftsman requires discipline and a real will to share things with others even small things. And of course, I still expect others to share things with me :). I made the promise a few weeks ago to talk about a real time saver and take this opportunity today, even if the subject is quite small.

Me and one of this blog's subscribers (hi ze1mlp), have been working for a long time together on the implementation of a military tactical editor (ok they must shoot me now =D). The application had to interface to external systems in order to exchange legacy data. One of the processes required the lowest protocol of communication that can be imagined: the exchange of files deposited under a shared directory. The legacy external system could only push files, non communication or service connection was possible.

We decided to create a watch dog. Easy said, easy done... but with all the trivial impediments raised by file I/O manipulation specifically when your JVM is running under windows (or Linux by the way). And we found ourselves with double checking for file deletions because of the file descriptor pains , tuning for periods of polling losing a lot of CPU, creating all sorts of small classes in addition to the set of classes to deal with the watch dog implementation.

The JDK7 elegantly solves the watch dog problem with a new service facility offered by the FileSystem god object (no anti patterns here :) ).

As usual I repoened my beloved IntelliJ IDE and started coding a new learning test for this facility.

Helped with the following helper methods:

@Before
    public void setup() {
        fileSystem = FileSystems.getDefault();
    }

    private FileSystem fileSystem() {
        return fileSystem;
    }

    private FileSystemProvider provider() {
        return fileSystem().provider();
    }


    private Path directory() {
        return fileSystem().getPath("C:", "temp");
    }



I came to the following:

@Test
    public void test() throws IOException, InterruptedException {
//step1
        final WatchService service = 
                     monitorCreationIn(directory(), withWatchService());
//step2
        final String location = newFileIn(directory());
//step3
        final WatchKey modification = service.take();
        assertEquals(directory(), modification.watchable());
        for (final WatchEvent event : modification.pollEvents()) {
            if (OVERFLOW.equals(event.kind())) continue;
            assertThat(event.count(), is(equalTo(1)));
            assertEquals(ENTRY_CREATE, event.kind());

            assertThat(
                  ((Path) event.context()).endsWith(location), 
                   is(true)
            );
        }
        modification.reset();
    }

In step 1, I first claim my intent to watch the content of the temporary directory (obviously the directory() method) using a watch service.

The watch service is created as is:

private WatchService withWatchService() throws IOException {
        return fileSystem().newWatchService();
    }


Magic ! :) Everything is set or nearly set. I also have to register the directory path as a Watchable object of the new service for this one to check it.

private WatchService monitorCreationIn(
          final Path path, final WatchService withService
    ) throws IOException {
        path.register(withService, ENTRY_CREATE);
        return withService;
    }

I only want to observe creation so be it. I registered the ENTRY_CREATE StandardWatchEventKind implementation of WatchEvent.Kind... as the kind of event to be observed.

Then in step 2 I do create my file as a path abstraction into the temporay directory location. This was a nice kata to create a file with the new I/O. I proceeded this way:

private String newFileIn(final Path directory) throws IOException {
        String location = format("newFile{0}.dat", currentTimeMillis());
        final Path newPath = directory.resolve(location);
        assertThat(exists(newPath), is(not(true)));
        create(newPath);
        return location;
    }

    private void create(final Path newPath) throws IOException {
        provider().newByteChannel(newPath, withCreationOptions()).close();
    }

This new API rocks !!! (notice I did close the byte channel)

On the time I registered my watchable path instance, a watch key has been registered, This key is driven by a finite state machine counting the following states:

  • ready (to receive file system events)
  • signaled (file system events received)
  • invalid (self explanatory) 
Once created, the key is in ready state and it will switch into signaled states once events will have been attached to it.
There is one key per watchable (very close to the selector key when working with non blocking events in NIO.1).

So From step 3, in practice I would start a time loop, polling key after key to check my watchables life cycle. I created one single file so I catch the next one and assert that the watch key watchable matches my observed directory:

final WatchKey modification = service.take();
      assertEquals(directory(), modification.watchable());

So far, so good, I can now poll for the events bound to my watchable, through the key:

for (final WatchEvent event : modification.pollEvents()) {

but I will take care not to handle lost or discarded events :

if (OVERFLOW.equals(event.kind())) continue;

Then, I check that this is not a repeated event, that the expected kind is creation and that I find again the name of the file I have created:

assertThat(event.count(), is(equalTo(1)));
     assertEquals(ENTRY_CREATE, event.kind());
     assertThat(((Path) event.context()).endsWith(location), is(true));

Of course, if you want your watchable key to be ready again to be polled for upcoming events, you must reset it so it will switch again to the ready state.

Done. Test green.

A few line of code reproducing the problem we solved four years ago with a bit more of code, of pain and of tests. Moreover there is this similarity with the familiar readiness check mechanism in NIO.1 implementation for non blocking sockets. Of course, the whole stuff is event driven and seems to lay onto lower level OS mechanisms allowing for a kind of reactor pattern implementation (no mystery).

Thanks for following these explorations smaller than usual. Be seeing you !! =D

4 comments:

Martijn Verburg said...

Interesting stuff! We're covering the watch service in the book and I've also found it very useful.

I noticed that the APIs had changed a little from when I was working through some sample code. I'm curious, which OpenJDK or binary build did you use?

If you do find any issues, please don't forget to let Alan Bateman and the nio team know. They're really looking out for bug reports at the moment.

Globulon said...

Thank you a lot for your feedback. I have been playing with b140 build and installed the b142 yesterday. Runs nicely.
The IDE is IntelliJ 10.5 and the project management tool is Maven 3.0
I have subscribed to the MEAP of the book a few weeks ago. We really needed such a work for the new incoming jdk7 (8,...). Thank you again

enl8enmentnow said...

I'm curious as to why they didn't use the Observer pattern like we are used to when handling events in Java. Why did they choose to have a pollEvents() method rather then taking a listener class?

Globulon said...

Don't have a single idea. Maybe, this was the most portable solution...

Post a Comment