david kerr » php http://www.davidmkerr.com Weapons designer. Innovator, inventor, world changer Thu, 20 May 2010 14:30:12 +0000 en hourly 1 http://wordpress.org/?v=3.0 Parsing large files with pgfouine in linux http://www.davidmkerr.com/databases/parsing-large-files-with-pgfouine-in-linux/ http://www.davidmkerr.com/databases/parsing-large-files-with-pgfouine-in-linux/#comments Fri, 08 Jan 2010 00:01:45 +0000 Dave http://www.davidmkerr.com/?p=206 pgfouine is a nice logfile analyzer for PostgreSQL written in php.

I’m doing a trace on a very long running ETL process and the logfile generated is ~11GB.

I’m running up against a 2GB barrier in php for fopen(). If you’ve got a 64bit machine and can recompile php with -D_FILE_OFFSET_BITS=64 then you’re good to go. But in my case, I can’t do either.

The error i’d get is:

PHP Fatal error:  File  is not readable. in /var/lib/pgsql/pgfouine-1.1/include/GenericLogReader.class.php on line 85

So for Plan B I had to remember back to the days when 64 bit wasn’t even an option (back in my day, we had 8 bits and we liked ‘em!)

I used a named pipe since pgfouine expects a file and doesn’t seem to be able to read from stdin.

mknod /tmp/pg2 p
cat /var/log/postgres > /tmp/pg2 | ./pgfouine.php -file /tmp/pg2 > bla.html

Once that kicked off I stopped getting that error and pgfouine was able to process the file.

]]> http://www.davidmkerr.com/databases/parsing-large-files-with-pgfouine-in-linux/feed/ 0