david kerr » Operating Systems

Parsing large files with pgfouine in linux

Dave — Fri, 08 Jan 2010 00:01:45 +0000

pgfouine is a nice logfile analyzer for PostgreSQL written in php.

I’m doing a trace on a very long running ETL process and the logfile generated is ~11GB.

I’m running up against a 2GB barrier in php for fopen(). If you’ve got a 64bit machine and can recompile php with -D_FILE_OFFSET_BITS=64 then you’re good to go. But in my case, I can’t do either.

The error i’d get is:

PHP Fatal error:  File  is not readable. in /var/lib/pgsql/pgfouine-1.1/include/GenericLogReader.class.php on line 85

So for Plan B I had to remember back to the days when 64 bit wasn’t even an option (back in my day, we had 8 bits and we liked ‘em!)

I used a named pipe since pgfouine expects a file and doesn’t seem to be able to read from stdin.

mknod /tmp/pg2 p
cat /var/log/postgres > /tmp/pg2 | ./pgfouine.php -file /tmp/pg2 > bla.html

Once that kicked off I stopped getting that error and pgfouine was able to process the file.

Enable “Desktop Favorites” Widget to Work with Windows 7

Dave — Wed, 04 Nov 2009 02:45:58 +0000

Desktop Favorites is one of my all time favourite widgets. But it stores it’s links as .lnk files which caused windows 7 to complain every time you open one.

To fix go here: http://support.microsoft.com/?kbid=315933 to enable My Computer Security Zone

Then go to Control Panel->Network and Internet->Internet Options

You’ll see My Computer under the Security Tab now.

Go to Custom level for the My Computer zone and choose “Launching application and unsafe files (not secure)” set to Enable.

Yes, this makes your system less secure, so don’t do it.

Fix For ATI Radeon HD 5770 “Unsupported Hardware” on Linux

Dave — Tue, 03 Nov 2009 23:30:37 +0000

Got one of these badboys and loaded up linux and was greeted with a big “Unsupported Hardware” watermark.

A little googling and i found this link:

http://www.phoronix.com/forums/showthread.php?t=19875

The script is below ( i take no responsibly for you messing your system up )

#!/bin/sh
DRIVER=/usr/lib/xorg/modules/drivers/fglrx_drv.so
for x in $(objdump -d $DRIVER|awk '/call/&&/EnableLogo/{print "\\x"$2"\\x"$3"\\x"$4"\\x"$5"\\x"$6}'); do
 sed -i "s/$x/\x90\x90\x90\x90\x90/g" $DRIVER
done

11/23/09 – *Edit* This fix is no longer required in the 9.11 drivers.

Perl script to allocate shared memory (and annoy sysadmins)

Dave — Mon, 02 Nov 2009 01:54:00 +0000

I once ran into an instance where the Unix admins didn’t believe me that i was running out of shared memory despite the errors, I was showing them. I wrote this perl script to allocate chunks of shared memory until it failed to prove to them that yes, regardless of what you have the global ulimits set to my user’s limits were lower.

#!/usr/bin/perl -w

use IPC::SysV qw(IPC_PRIVATE IPC_RMID S_IRWXU);

#$size = 314580992;
$size = 50000000;
$id = shmget(IPC_PRIVATE, $size, S_IRWXU) || die "$!";
sleep 10;
shmctl($id, IPC_RMID, 0)

HowTo Find info about Shared memory allocation

Dave — Mon, 02 Nov 2009 01:48:05 +0000

Sometimes you may see core dumps, etc related to lack of shared memory.
To check to see how much is currently being used you can use the below command:

kerrd@vexbert<10> ipcs -bm
IPC status from  as of Thu Oct 30 10:42:06 PDT 2008
T         ID      KEY        MODE        OWNER    GROUP      SEGSZ
Shared Memory:
m 1493172237   0x32d14fb8 --rw-r-----   oracle      dba 2097160192
m 1979711547   0x5e000b64 --rw-------     root     root        512
m  721420366   0xe867eba0 --rw-r-----   oracle      dba 2147491840
m         25   0x67e97344 --rw-r-----   oracle      dba  419438592
m         21   0xc4986724 --rw-rw----   oracle      dba  629153792
m         19   0x71e930c0 --rw-rw----   oracle      dba 1073758208
m          7   0x4154520e --rw-rw-rw-     root   tivoli    1048576
m          6   0x4154520a --rw-rw-rw-   tivoli   tivoli    1048576
m          5   0x4154520d --rw-rw-rw-   tivoli   tivoli    1048576
m          4   0x4154520c --rw-rw-rw-   tivoli   tivoli    1048576
m          3   0x4154520f --rw-rw-rw-   tivoli   tivoli    1048576
m          2   0x4154520b --rw-rw-rw-   tivoli   tivoli     524288
m          1   0x41545200 --rw-rw-rw-   tivoli   tivoli    1048576
m          0   0          --rw-------     root     root         64

The Oracle/DBA segments are related to Oracle database instances.
The SEGSZ is limited by the Unix kernel.