Sunday, May 13, 2012

Parsing Large files with Pgfouine

ppgfouine is a nice logfile analyzer for PostgreSQL written in php.

When doing a trace on a very long running ETL process the logfile generated was ~11GB.

This left me running up against a 2GB barrier in php for fopen().

If you've got a 64bit machine and can recompile php with -D_FILE_OFFSET_BITS=64 then you're good to go. But in my case, I wasn't able to do either.

 The error i'd get is: PHP Fatal error: File is not readable. in /var/lib/pgsql/pgfouine-1.1/include/GenericLogReader.class.php on line 85

The simple solution was to use a named pipe since pgfouine expects a file and doesn't seem to be able to read from stdin.

mknod /tmp/pg2 p
cat /var/log/postgres > /tmp/pg2 | ./pgfouine.php -file /tmp/pg2 > bla.html
Another variation on this, for those of us using UTC and find that our postgres log is rotated mid-day is to do something like:
(gzcat /var/log/ && cat /var/log/postgres) > /tmp/pg2 | ./pgfouine.php -file /tmp/pgt > bla.html


  1. Thanks for this. I was hitting the exact same issue with a 4GB file and could not determine what exactly was wrong.

    I greatly appreciate you posting this!

  2. No problem! actually new version of pgfouine allow for reading from stdin
    pgfouine.php --help
    - read the log from stdin instead of -file

    so you could probably avoid the named pipe!


Note: Only a member of this blog may post a comment.

Web Statistics