To download, run "git clone http://www.sbg.bio.ic.ac.uk/~sm2708/software/git/perl-filequeue.git/".
FileQueue.pm -- Simple file queue for multiple processes.
my $queue = IPC::FileQueue->new(filename => "queue"); my $queue = IPC::FileQueue->new(filename => "queue", line_size => 10);
IPC::FileQueue object represents a file on disk that contains one queue item per line. The file is locked when being read from or written to, and so it is safe for multiple processes to use one queue simultaneously.
This module is intended as a simple queue for use with in-house compute clusters, where it is desirable to have many processes processing separate items. If the length of time required to process each item is not known, then simply splitting the items into predfined chunks for each process may be inefficient. This module allows processes to grab items as needed.
This queue is fairly fast, as it make use of the append filemode and the "truncate" in perlfunc function, meaning that the file does not need to read into memory or even iterated through. Tests indicate that this module is significantly faster than both Tie::File and naive methods.
IPC::FileQueue accepts two parameters to its constructor:
Name of the file in which items will be stored.
line_size(Default = 10)
Approximate number of characters in a line. The queue will work even if this is wildly inaccurate, but it may be slower than necessary.
Push an item onto the end of the queue. This will modify the file and immediately release the lock. If the file is busy, this function blocks until it is free. If the file cannot be locked for some non-standard reason, this function croaks.
Pop an item from the end of the queue. As with push, this immediately modifies the file and releases the lock. If the file is busy, it blocks and if a lock cannot be obtained, it croaks.