Skip to content

Monthly Archives: June 2008

Why FileInputStream is slow.

22-Jun-08

On my first crack at a Java-based Wide Finder implementation, got a surprisingly slow result for using java.io.FileInputStream to read a file. Dug into the sources to track down the cause. (Is there a better public-reference to the OpenJDK code than here?) First, the FileInputStream Java class read() call turns into a native call to […]

Wide Finder 2 – over to the dark side

18-Jun-08

My first solution for Tim Bray’s Wide Finder 2 problem was aimed at ease of re-use before performance. Simple Perl scripts (run in a large number of worker processes) did the bulk of the processing. Got my times, posted the results, and was going to leave it at that. The damn problem got stuck in […]

Wrapping up Wide Finder 2

13-Jun-08

The main point of the Tim Bray’s Wide Finder 2 exercise was to solve a common problem (processing web server logs files) in a way that took full advantage of a large number of CPUs. As Sun is shipping machines with many cores per chip, and as Tim works for Sun – he has good […]

Wider finder – final result

12-Jun-08

[wrapup - added later] From running my implementation of Tim Bray’s Wide Finder 2 on the Sun test box. bannister@wfind01$ time ./feed-workers -n 30 -r `which perl` -s scripts/reduce.pl logs/O.all | time scripts/combine.pl > _x30_reduce_combine Fri Jun 13 04:03:32 2008 Scanning: logs/O.all Done with: logs/O.all Worker #21875 ended with status: 0 Worker #21874 ended with […]

Wide finder – combine for top N

12-Jun-08

[wrapup - added later] As the last exercise showed, doing the full combine of all the reduced data was just too fat to get acceptable performance. This iteration is to re-write combine.pl to limit the amount of data retained to not much more than the final “top N” result. The notion here is to only […]

property.import task for Ant

11-Jun-08

I have Ant build scripts that can be run on either Windows or Linux. There are a few properties that need to be set differently, depending on the platform is use. Been using this little snippet for a while now. import.js Array.prototype.map = function(f) { var a = []; for (var i=0; i<this.length; ++i) { […]

Wide finder – combine and sort

10-Jun-08

[wrapup - added later] Took another crack at the combine process for my implementation of wide finder. My theme is to try and come up with the something easily re-used and re-purposed, rather than the fastest possible specialized solution. The first round used the generic Perl sort operation, with an inline (two key) sort function. […]

Wide finder – first round

08-Jun-08

[wrapup - added later] Took my feed-workers tool from my example of solving the prior wide-finder exercise, and updated the scripts to match(?) the new strawman wide-finder benchmark. Updated the scripts over the weekend (yes, I’m sufficiently warped to find this sort of exercise amusing). Got onto the Sun 32-CPU box on Tuesday, and did […]