Sunday, May 13, 2012

Parsing Large files with Pgfouine

ppgfouine is a nice logfile analyzer for PostgreSQL written in php.

When doing a trace on a very long running ETL process the logfile generated was ~11GB.

This left me running up against a 2GB barrier in php for fopen().

If you've got a 64bit machine and can recompile php with -D_FILE_OFFSET_BITS=64 then you're good to go. But in my case, I wasn't able to do either.

 The error i'd get is: PHP Fatal error: File is not readable. in /var/lib/pgsql/pgfouine-1.1/include/GenericLogReader.class.php on line 85

The simple solution was to use a named pipe since pgfouine expects a file and doesn't seem to be able to read from stdin.

mknod /tmp/pg2 p
cat /var/log/postgres > /tmp/pg2 | ./pgfouine.php -file /tmp/pg2 > bla.html
Another variation on this, for those of us using UTC and find that our postgres log is rotated mid-day is to do something like:
(gzcat /var/log/postgres.xyz.gz && cat /var/log/postgres) > /tmp/pg2 | ./pgfouine.php -file /tmp/pgt > bla.html

2 comments:

  1. Thanks for this. I was hitting the exact same issue with a 4GB file and could not determine what exactly was wrong.

    I greatly appreciate you posting this!

    ReplyDelete
  2. No problem! actually new version of pgfouine allow for reading from stdin
    pgfouine.php --help
    [snip]
    - read the log from stdin instead of -file
    [snip]

    so you could probably avoid the named pipe!

    ReplyDelete

Note: Only a member of this blog may post a comment.

Web Statistics