Brainfish Eat Fishbrain

How does WP (3) resolve (pretty) urls

2010-10-12T09:41:00.000-07:00

When I was checking a pretty custom WP site, I couldn't easily figure out how the URLs flow through the system, so I decided to write down what / how I found out how it works.

index.php -> wp-blog-header.php -> wp-load.php -> wp-includes/functions.php -> wp() -> (wp-includes/classes.php/WP)wp->main() -> wp->parse_request()

... is the basic flow used to decode the urls.

The first point you need to look at is:

// Fetch the rewrite rules.
$rewrite = $wp_rewrite->wp_rewrite_rules();

In my case this ($rewrite) contained a ton of rules which I couldn't really match with any code or rules in the database. Where are these rules coming from? The following comment to the function would indicate that it's cached:

* WP_Rewrite::rewrite_rules()} is that this method stores the rewrite rules
* in the 'rewrite_rules' option and retrieves it. This prevents having to

I wanted to make sure. To find this we have to dive into wp_rewrite_rules() which can be found in wp-includes/rewrite.php;

function wp_rewrite_rules() {
$this->rules = get_option('rewrite_rules');

The above $this->rules now already contained the 'ghost rules' so apparently they are stored in the database somehwere. To find out where, you just debug get_option() in functions.php. This brought me into:

$alloptions = wp_load_alloptions();

And options are retrieved, obviously, from:

if ( !$alloptions_db = $wpdb->get_results( "SELECT option_name, option_value FROM $wpdb->options WHERE autoload = 'yes'" ) )
$alloptions_db = $wpdb->get_results( "SELECT option_name, option_value FROM $wpdb->options" );

So basically we found out that this is a kind of cache indeed. To flush that cache, you go to the wp-admin->Settings->Permalinks and click Save Changes. After that it's flushed.

After flushing there were still a few of the mysterious rules in the rewrite list. Those were due to;

register_taxonomy( 'review', 'post', array( 'label' => __('Review') ) );

When a custom taxonomy was added, you will see rules like this:

[review/([^/]+)/feed/(feed|rdf|rss|rss2|atom)/?$] => index.php?review=$matches[1]&feed=$matches[2]
[review/([^/]+)/(feed|rdf|rss|rss2|atom)/?$] => index.php?review=$matches[1]&feed=$matches[2]
[review/([^/]+)/page/?([0-9]{1,})/?$] => index.php?review=$matches[1]&paged=$matches[2]
[review/([^/]+)/?$] => index.php?review=$matches[1]

Now that 'mystery' was solved, on with the rest of the flow:

In the parse_request() method you can find;

foreach ( (array) $rewrite as $match => $query) {

This is the loop which has all rewrite rules which WP uses to translate from pretty urls to arguments the WP application can use to get the post(s) and other data needed and mesh that with the templates.

if ( preg_match("#^$match#", $request_match, $matches) ||
preg_match("#^$match#", urldecode($request_match), $matches) ) {
// Got a match.
$this->matched_rule = $match;

Now WP found a match, so we can check what the result is:

die("$match => $query");

An example of a matching rule is:

review/([^/]+)/([^/]+)(/[0-9]+)?/?$ => index.php?$matches[1]&name=$matches[2]&page=$matches[3]

This rule is translated based on the URL and then it can be matched, this is done with this code:

$query = addslashes(WP_MatchesMapRegex::apply($query, $matches));

and the result is, for instance;

review/([^/]+)/([^/]+)(/[0-9]+)?/?$ => ms-windows-7&name=windows-7-is-better-but-still-sucks&page=

With the following method this is translated to a php array:

parse_str($query, $perma_query_vars);

This results into something like this:

Array ( [ms-windows-7] => [name] => windows-7-is-better-but-still-sucks [page] => )

With this information, it should be clear (and at least be easy to follow) how WP resolves it's URLs. With a debugger this is all easy enough to find, but it's still easy to see it all on a page.

Migrating a Wordpress installation from your localhost to a production server (using cpanel)

2010-05-22T06:39:00.000-07:00

Say you have locally WP installed under WAMP or MAMP or whatever under;

http//localhost:8888/somesite

and you want to run it, with all content and plugins and what have not under:

http://www.somesite.com

The steps to do this:

- create somesite.com in your WHM system (http://yourhost.com/whm) or make sure your site exists
- go to your local phpmyadmin, usually; http://localhost:8888/ and then clicking on phpmyadmin
- click on the database you are using
- click on export
- make sure SQL is selecting on the left
- click on go
- open another tab in your browser and open http://www.somesite.com/cpanel
- login
- create a database and a user for that database and remember what you you used there
- now open phpmyadmin in your cpanel
- click on the database you just created
- click on SQL
- copy/paste the sql from your localhost tab into this new db and click go
- now connect (with ftp) to your somesite.com server and copy over everything in your root directory from your localhost (for instance; htdocs/somesite/*)
- edit, remotely, your public_html/wp-config.php; change the values DB_NAME, DB_USER, DB_PASS to the values you used to create your db in cpanel
- now, basically, your site should work; simply try http://www.somesite.com
- if you get a database error, please check your remote wp-config.php to see what went wrong
- you probably now get some error; not found http://somesite.com:8888 or something like that; this is because the php options are wrong
- to fix this, just open SSH to your remote server and run;

cd /home/somesite
mysqldump YOURDBNAME wp_options > tmp.sql
sed -i 's/localhost:8888\/somesite/www.somesite.com/ig' tmp.sql
mysql YOURDBNAME < tmp.sql

Now your site will work.

Have fun.

Cpanel: Monitoring :2082 logins

2009-10-26T03:07:00.000-07:00

Simple script to send you all new logins of the day. Seeing something strange would trigger further research.

#!/usr/bin/perl

chdir('/root');

$d = `date +%m/%d/%Y`;
chomp($d);

@logins = `cat /usr/local/cpanel/logs/access_log|grep $d|awk '{print \$1 " " \$3}'|sort|uniq`;

$x = "";

foreach(@logins) {
 chomp; 
 /(.*?)\ (.*)/;
 next if $2 eq "-";
 $z = `whois $1|grep addres|tail -n 1`;
 chomp($z);
 $x.="$z $1 $2\n";
}

if (not -f "./latestscan") {
 `touch ./latestscan`;
}

$y = `cat ./latestscan`;

exit if $y eq $x;

open(F, ">latestscan");
print F $x; 
close F;

 $sendmail = "/usr/sbin/sendmail -t";
 open(SENDMAIL, "|$sendmail") or die "Cannot open $sendmail: $!"; 
 print SENDMAIL "Reply-to: root\@myserver.org\n"; 
 print SENDMAIL "Subject: Login report hostf1\n"; 
 print SENDMAIL "To: alerts\@yourserver.com\n"; 
 print SENDMAIL "Content-type: text/plain\n\n"; 
 print SENDMAIL $x; 
 close(SENDMAIL);

Watching series; I don't want to touch my computer

2009-10-25T00:58:00.000-07:00

When I watch series I don't want to touch my computer and I watch all episodes in one fell swoop; if I do something like mplayer *.mpg it crashes, so that doesn't work. This does, run like;


./play "*.mpg"


./play "*.avi"

The code;


#!/usr/bin/perl

$a = "";
foreach(@ARGV) {
 $a.=" " if $a; 
 $a.=$_;
}

while(1){foreach(glob("$a")){`mplayer -fs \"$_\"`}}

Cpanel security: scanning for usage or upload of c99 shell script (or other scripts)

2009-10-24T05:08:00.000-07:00

Sometimes users upload stuff to your server or use scripts you don't want used. To detect them fast, I wrote this script.


#!/usr/bin/perl
    
use Digest::Perl::MD5 'md5_hex';

chdir('/root/');
`touch ./scanned` if not -f "./scanned";

%h = ();
open(F, "scanned");
while() {
 chomp;
 $h{$_} = 1;
}
close F;

@x = `cd /etc/httpd/domlogs/; grep c99me *`;
open(F, ">>scanned");
$s = "";
foreach(@x) {
 chomp;
 $m = md5_hex($_);
 next if $h{$m};
 print F "$m\n";
 $s.=$_."\n";
}
close F;

if ($s) {
 $sendmail = "/usr/sbin/sendmail -t";
 open(SENDMAIL, "|$sendmail") or die "Cannot open $sendmail: $!"; 
 print SENDMAIL "Reply-to: root\@myserver.org\n"; 
 print SENDMAIL "Subject: Found some illegal stuff on server\n"; 
 print SENDMAIL "To: alerts\@somewhere.com\n"; 
 print SENDMAIL "Content-type: text/plain\n\n"; 
 print SENDMAIL $s; 
 close(SENDMAIL); 
}

Cpanel security: cron update all Wordpress installations on your server

2009-10-24T04:21:00.000-07:00

Two simple scripts. Use with caution and at your own risk. Might eat your machine and piss off all your users.


!/usr/bin/perl

`rm -fR wordpress`;
`wget http://wordpress.org/latest.zip`;
`unzip latest.zip`;

@all = `ls -la /home/|awk '{print \$3}'|grep -v root`;

foreach(@all) {
 chomp;
 next if /^$/;
 `./updatewp $_`;
}


#!/usr/bin/perl

$host = `hostname`; 
chomp($host);

$u = $ARGV[0]; 

exit if !$u; # user as arg

exit if not -f "/home/$u/public_html/wp-config.php"; # not wp install

# you shouldn't actually have the readme.html, but if it's there it's a bit faster
$v1 = `cat /home/$u/public_html/readme.html|grep Version > dev/null`;
$v1 =~ /Version\ (\d+\.\d+\.\d+)/;
$v1 = $1; 

$v2 = `cat wordpress/readme.html|grep Version`;
$v2 =~ /Version\ (\d+\.\d+\.\d+)/;
$v2 = $1;

exit if $v1 eq $v2; # already updated

`cp -a wordpress /home/$u/wp_int`;

`cp -rpf /home/$u/public_html/wp-config.php /home/$u/wp_int`;
`cp -rpf /home/$u/public_html/wp-content/* /home/$u/wp_int/wp-content/`;
`cp -rpf /home/$u/public_html/.htaccess /home/$u/wp_int/`;

`chown $u.$u /home/$u/wp_int`;

`cp -a /home/$u/public_html /home/$u/wpback\`date +%d%m%y\``;

`cp -rpf /home/$u/wp_int/* /home/$u/public_html/`;

`rm -fR /home/$u/wp_int`;

$x = `lynx -dump http://$host/~$u/wp-admin/upgrade.php`;

if ($x =~ /Database Upgrade Required/isgm) {
 `lynx -dump http://$host/~$u/wp-admin/upgrade.php?step=1\&backto=`;
}

print "Updated $u\n";

Mac OS X Firefox 3.5 intervals to 100% CPU about every 30 seconds

2009-08-03T07:37:00.000-07:00

Yesterday suddenly my Firefox started to show the busy mouse pointer about every 30 seconds. Very annoying as it was completely unusable during about 5 seconds, and then 30 seconds later again etc.

I removed my Profile in ~/Library and checked if it was FF or something in my profile. Ofcourse it was something in my profile.

Checking my files I saw one file that looked 'odd' ; places.sqlite was rather huge (I do browser a lot), however sqlite can handle quite big databases. Anyway; I ran a

rm -f places*

and restarted FF. No more 100%. Fixed.

What is going on I don't know; it looks like FF is running some kind of really bad query on that sqlite every +/- 30 s which is messing up the whole thing.

Checking/repairing all MySQL tables

2009-06-30T01:18:00.000-07:00

Caution: on busy servers this will make a lot of load. Use only when you suspect/know tables are broken.


#!/usr/bin/perl

$mysql = "/var/lib/mysql";

@res = `cd $mysql; find .|grep MYD`;

foreach(@res) {
 chomp;
 /\.\/(.*?)\/(.*?).MYD/;
 $tab = $1.".".$2;
 print `mysql --execute='repair table $tab'`; 
}

A simple, non bloated script for Amazon S3 backups (Linux, easy portable) - II - recurse to subdirs

2009-02-06T03:44:00.001-08:00

To follow this, please read this post first.

We needed to backup our images from http://www.picturepush.com to S3 for our premium members so we were searching for a simple script to do that one time. As in my previous post; there is no such thing. Bloated, uninstallable shit is the only thing there is.

So I changed the code a bit and added;


foreach($_SERVER["argv"] as $d) {
 recurse_copy($d, $d);
}
exit;

function recurse_copy($d, $org) {
 foreach(glob("$d/*") as $d1) {
  if ($d1=='..' || $d1=='.') continue;
  if (is_dir($d1)) {
   recurse_copy($d1, $org); 
  } else {
   // put on s3
   global $s3;
   global $bucket;
   $s3->putObjectFile($d1, $bucket, $d1);
   echo "Storing: ".$d1."\n";
  }
 }
}

below the last;


array_shift($_SERVER["argv"]);

A simple, non bloated script for Amazon S3 backups (Linux, easily portable)

2009-01-03T09:01:00.000-08:00

While searching for scripts to backup files from Linux -> S3 (servers) I was surprised how difficult it was to find any nice, trim ones. There are huge java jars filled with crap to do it ofcourse;


dbserv01:~# ls -lah js3tream.jar 
-rw-r--r-- 1 root root 3.2M 2007-12-19 15:07 js3tream.jar

Come on. 3.2M... What does that include? Windows Vista?

Sure js3tream is actually nice software. Portable and it works. It's incredibly, mindbogglingly slow (but ofcourse, Java is not slow these days as many people tell me...) on my quad Xeon servers. But yeah, it has the features you would want. Except encryption of the files I upload; you arrange that in a different way please. Come to think of it; other features are missing as well.

There is S3Sync in Ruby which is actually nice and working well, but still too much hassle to get going as simple script. But no complaints about the speed or size of that.

Then there is some rsync thing in Python (I forgot the name and the HELLLLLLLLLLLLLLLLLL I went to to install it will not make me remember it any time soon).

So, as almost all things in life, if you want it done right, you just have to do it yourself. I don't find it pretty at all, but it weighs in, including comments, at 36 kbs, which is, by far the smallest possible script I could find for the purpose I needed.

I'm not counting rar or gpg as they are not included in the other ones either, but when counting them it wouldn't go over 1 mb.

Download http://undesigned.org.za/2007/10/22/amazon-s3-php-class

And install:

- rar (apt-get install rar)
- gpg (apt-get install gnupg)
- php (apt-get install php5-cli php5-curl)

Then edit the Amazon S3.php; put on top of it;


#!/usr/bin/php


if (sizeof($_SERVER["argv"])<4) {
 echo "Usage: ./s3backup bucket ident dir dir dir\n";
 exit;
}

$bucket = $_SERVER["argv"][1];
$ident = $_SERVER["argv"][2];

define('PHPARTIALS_FILE_AWS_S3_ACCESSKEY', '');
define('PHPARTIALS_FILE_AWS_S3_SECRETKEY', '');
define('PHPARTIALS_FILE_RAR_ENCRYPT', '');
define('PHPARTIALS_FILE_GPG_ENCRYPT', '');

$s3 = new S3(PHPARTIALS_FILE_AWS_S3_ACCESSKEY, PHPARTIALS_FILE_AWS_S3_SECRETKEY);

#print_r($s3->getBucket($bucket));exit;

$l = $s3->listBuckets(true);

if (!in_array($bucket, $l)) {
 $s3->putBucket($bucket, S3::ACL_PRIVATE);
}

$fn = $ident."-".strftime('%d%m%y%H%M').".rar";

array_shift($_SERVER["argv"]);
array_shift($_SERVER["argv"]);
array_shift($_SERVER["argv"]);

$f = implode(' ', $_SERVER["argv"]);

$cmd = "rar a -hp".PHPARTIALS_FILE_RAR_ENCRYPT." $fn $f";

echo `$cmd`;

$fn1=$fn.".enc";

$cmd = "gpg --batch --yes --trust-model always --encrypt --recipient '".PHPARTIALS_FILE_GPG_ENCRYPT."' -o $fn1 $fn";

echo `$cmd`;

$s3->putObjectFile($fn, $bucket, baseName($fn1));

print "Backup created!\nSize: ".filesize($fn1)." bytes\n";

unlink($fn);
unlink($fn1);

exit;
?>

Rename the S3.php to something like s3backup and chmod 700 s3backup to make it executable.

You put your 2 keys for Amazon S3 in the Amazon defines, a password for rarring in the RAR password define and the name of the user your public key is for on that system to GPG encrypt the rar.

To add a public key for GPG, just put your GPG Public key in a file, say /root/mypub.key and run;

gpg -a --import /root/mypub.key

And all will be fine.

Filthy Rotten Scriptkiddies - Blackmailing Siteowners

2008-11-24T19:55:00.000-08:00

A friend of mine owns a site which has very little traffic and sells a niche product. Because he gets so little traffic, alarm bells immediately sound when suddenly there is a spike.

If there would be a spike, he would be rich probably, but the chances of that happening are very slim.

Anyway, this night, the guy called me up at 3:30 saying he received an email from some hotmail address saying something like ;

I understand how much revinue you bring in each month, I can bring that down to 0 if I wish. Think I'm kidding? Think this is a joke?

And then his demands.

So, me thinking this was indeed a joke, asking why he )(@#%()#@% woke me. But it wasn't a joke. The site was down. Very much so for 1.5 hour already. Costing my friend money.

A bit groggy from sleeping I could not imagine this being anything else than some lame DOS attack; one or a few computers bonking away on :80.

Unfortunately that was not the case at all.

This was a quite (for this kind of lame threat / blackmail) heavy DDOS. After a few minutes I already collected (and blocked) over 200 unique ips (from different classes mostly).

In this blog I have shown more than ones some ideas for catching and blocking DDOS attacks from within Linux and this one was rather a simple one and could be simply blocked using;

List of Linux tricks

Few notes here; Apache (or whatever you might use) queues all those income connections and leaves them connected even though they are blocked. Because of this,
during the attack, I run from the cron; */10 * * * * service httpd restart. Making Apache immediately kill of those bad connections, but finish off the real ones with a real response. At least you'll have service for most people using this method.

Ofcourse it required a bunch of tweaking as the attacker changed his strategy quite often to make it more difficult.

I don't get people who do this and I certainly don't understand how they can mount such a huge attack with so many different IPs.

Edit: attack has been going on for 9 hours now... site doing fine.

MacOSX / Bootcamp / Installing Windows XP without SPs / there is not enough disk space on $ntservicepackuninstall$ to install service pack bla

2008-11-13T15:11:00.000-08:00

Sometimes you have to do things you don't want. Today I needed to install something under Windows to convert some fileformat (media file format) for which I cannot find Linux or Mac converters. Not even Mplayer can play it. Only some weird proprietary Windows app.

So basically; I needed Windows. And I hate Windows. And not without passion; I really hate it. But ok, enough about that.

I have a Mac Mini standing on my desk gathering dust (oh did I mention I hate Mac OS X too?), so I thought ; I can install on that.

Not that easy. Requires an XP disk with SP2. And ofcourse I didn't have that. As I really don't want to pay for stuff I already have, I took an XP version from 2001 (so without any SPs) and tried to find a way to install that on the Mac Mini. Legal as I don't even have the computer anymore on which is belonged, but I have the 'authentic Windows crap sticker' still.

The installation from within Bootcamp went 'smooth', although Windows doesn't want to shutdown or restart, you have to force it down by holding the shutdown button.

And almost none of the software/drivers Apple provides work without SP2. Even worse (well... not much worse); Mac OS X doesn't boot anymore.

Anyway; I got the SP2 download and burnt it on a CD under Linux and tried to install it under the non-SP version on the Mac Mini. Unfortunately, it started yelping about some missing 4 mb's of space....... 4 mbs. Ofcourse I have more than 20 gb free so that could not be right.

Quite annoyingly, the Mac community is not of the technical details; when searching for these kind of problems, people keep repeating in forums that I insulted my Mac by installing Windows OR that I did not read and MUST HAVE SP2 Windows install CD OR that I should just reinstall everything. All very helpful. Not.

So I was faced with two problems;

- no more Mac OS X
- Windows couldn't install the SP so I couldn't install any drivers etc

The former was not that important for me as I don't like Mac OS X that much and certainly never use it if I can avoid it.

But for the latter I started this whole adventure, so I couldn't give that up.

I read all messages about this problem and the solution turned out to be very (very) simple;

Windows-R (Start->Run)
regedit

Add a String

key = BootDir
value = C:\

To

[HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Setup]

And rerun the SP2 installer.

Why didn't they just SAY so?

Anyway; if/when I have Mac OS X back I'll report it here and people can use nice old CD's of XP they find in car boot sales, instead of paying money to that company we all love.

Edit: Well, that was ofcourse kind of simple;

- remove the read only flag from c:\boot.ini
- edit boot.ini
- put at the end c:\CHAIN0="Mac OS X"
- save
- put this file in you c:\
- reboot

Not Boot Camp, but working fine.

Creating databases in cpanel remotely

2008-10-04T11:40:00.000-07:00

Lately I was wondering how one would go about creating databases in cpanel without having to log on. Do it from PHP or something and create a somewhat 'nicer' interface to that 'nice' cpanel stuff.

It proved to be trivial to do anything with the system; that is in face kind of nice and has been a relief for me as I thought it would be (much) more difficult.

Here is a code snippet how to do this easily. It return PHPartialsResult, but you can replace that with 'true' or something.


  function remote_account_create_db($host, $user, $pass, $dbname) {
   $conn = "http://$user:$pass@$host:2082/frontend/x3/sql/addb.html?db=$dbname";
   file_get_contents($conn); 
   return PHPartialsResult::ok();
  }

  function remote_account_create_dbuser($host, $user, $pass, $_user, $_pass) {
   $conn = "http://$user:$pass@$host:2082/frontend/x3/sql/adduser.html?user=$_user&pass=$pass&pass2=$pass";
   file_get_contents($conn); 
   return PHPartialsResult::ok(); 
  }

  function remote_account_create_dbuserconnection($host, $user, $pass, $_user, $_db) {
   $conn = "http://$user:$pass@$host:2082/frontend/x3/sql/addusertodb.html?ALL=ALL&user=$_user&db=$_db";
   file_get_contents($conn); 
   return PHPartialsResult::ok(); 
  }

Using tcpproxy for http forwarding? Add X-Forwarded-For!

2008-04-25T15:05:00.000-07:00

TCPProxy is one of the most beautiful tools I know of; it was nicely programmed, but above all; it has no bugs. None. We are routing millions of pageviews a day of hundreds of thousands of unique users through it and it has never let me down. It is fast, uses almost no CPU and is very very robust (I miss FTP support, but further it is totally perfect).

This unlike Apache (proxy pass) or Squid or the many other, less known, proxies I tried. Apache is especially crappy; proxy pass is riddled with bugs; under load it simply is unusable. And most of them are feature-heavy; too feature heavy; they are a pain to set up, but trivial things like not binding to 0.0.0.0 etc or simple config files they have not.

We have the necessity to route web traffic via different machines in different places and ofcourse I wanted to use tcpproxy for the task, but it has no x-forwarded-for and so we were pissing our users (with forums, blogs etc) off. So I decided to hack tcpproxy for this. And it works. Serving the millions of pages through it. With x-forwarded-for.

This was tested on 1.1.9;

In tcpproxy.c, search for these lines;


int proxy_request(config_t *x, int cfd)
{
int     sfd, rc, bytes;

and change to:


int     sfd, rc, bytes, first=1, i;
char    *hp, *hp1;
char    tmp[4096];

then search for;


  else if (write(sfd, buffer, bytes) != bytes) {
                                syslog(LOG_NOTICE, "client write() error");

and change that to;


else {
if (first && (
 (buffer[0] == 'G' && buffer[1] == 'E' && buffer[2] == 'T') ||
 (buffer[0] == 'P' && buffer[1] == 'O' && buffer[2] == 'S' && buffer[3] == 'T') ||
 (buffer[0] == 'P' && buffer[1] == 'U' && buffer[2] == 'T')
 )) {

 hp = &buffer;
 *(hp+bytes)='\0';
 hp = strstr(buffer, "\r\n\r\n");
 if (hp) {
  hp1 = &buffer;
  i = abs(hp1-hp);
  *hp = '\0';
  memcpy(tmp, buffer, bytes+1);
  strupr(buffer);
  if (!strstr(buffer, "X-FORWARDED-FOR")) {
   sprintf(buffer, "%s\r\nX-Forwarded-For: %s\r", tmp, x->client_ip);
   hp = &tmp;
   hp += i + 1;
   i = bytes - i - 1;
   bytes = i + strlen(buffer);
   hp1 = &buffer;
   hp1 += strlen(buffer);
   memcpy(hp1, hp, i);
  }
 }
}
first = 0;

if (write(sfd, buffer, bytes) != bytes) {
                                syslog(LOG_NOTICE, "client write() error");
 
           break;
                                }

}

Compile it and it'll add X-Forwarded-For.

Debootstrap must be the most brilliant thing ever conceived

2008-04-09T04:41:00.000-07:00

Are you lazy?
Are you tired of installing the same shit across Linux servers?
Are you sick of compiling the latest & greatest bag of mplayer/ffmpeg/lame on all your systems?
Are you becoming suicidal when someone asks you to send you a 'working out of the box version'?

You don't need to be :) Install once, run anywhere (on Linux ofcourse, but what else would you run anyway huh);

Install debootstrap;

apt-get install debootstrap

I personally find it nicest to have everything in a file (so you can easily copy one file and leave it at that);

First make a loopdevice (3 gb file);

dd if=/dev/zero of=/myETCH bs=1M count=3000

Set up kernel loopdevice if you didn't;

insmod loop.o

Set up the loop device;

losetup /dev/loop0 /myETCH

Format the loopdevice with ext3;

mkfs.ext3 /dev/loop0
mount /dev/loop0 /myETCH

then remove everything again;

umount /dev/loop0
losetup -d /dev/loop0

Your loopdevice is now ready for use!

mount it with;

mount -t ext3 /myEtch /etchinstall -o loop=/dev/loop0

If you do not want a loopback device, simply run;

mkdir /etchinstall

After this you can proceed with installing Debian etch;

debootstrap etch /etchinstall

Wait for it to be done.

Then mount proc;

mount -t proc proc /etchinstall/proc

then you can chroot it;

chroot /etchinstall

And there you are; your own nice, contained version of Debian Etch.

Install whatever you want into it. Provide your friends with the loop file /myEtch when you want to have 100% the environment as you!

Heavy encrypting files on your linux box

2008-03-31T15:50:00.000-07:00

After some nasty run-in with reality I decided to day to take better care of my passwords. By encrypting them with GnuPG, I thought I would stand a much better chance when my laptop gets stolen. I am lazy so I already have my HD encrypted, but, for the super paranoid, as I am becoming more and more, I now also have my passes seperate encrypted in a vault file with 4096 gpg encryption.

How? Very simple; first generate a pub/priv key pair;

gpg --gen-key

then add two files to your path;

enc;


#!/usr/bin/perl

$file = $ARGV[0];
$file1 = $ARGV[1]; 

if (!$file or !$file1) {
 print "Usage: ./enc input output\n";
 exit;
}

print `gpg --encrypt --recipient 'Your Name' -o $file1 $file`;

dec;


#!/usr/bin/perl

$file = $ARGV[0]; 

if (!$file) {
 print "Usage: dec input\n"; 
 exit;
}

print `gpg --decrypt $file`;

Then make sure you have zero writing tool to delete the original, to be encrypted, file and you're all done :)

Mysql (or other) dump in PHP

2008-03-09T21:09:00.000-07:00

I a lot of sourcode I see the calling of mysqldump to dump tables. Doing so needs all kind of OS dependent path settings and so on, so I thought I share a simple function for dumping a table from MySQL in PHP:


  function _mysqlDump($table, $toFile) {
   $f = fopen($toFile, "w");
   $qry = "select * from $table";
   $q = mysql_query($qry);
   if (!$q) {
    echo "Could not dump $table\n";
    return;
   }
   while($r = mysql_fetch_assoc($q)) {
    $i1 = ""; $i2 = "";
    foreach($r as $k=>$v) {
     if ($i1) {$i1.=","; $i2.=",";}
     $i1.=$k;
     $v = mysql_real_escape_string($v);
     $i2.="'$v'";
    }
    fputs($f, "insert into $table ($i1) values ($i2);\n");
   }
   fclose($f);
  }

The advantage of this is that it creates total inserts which is handy when you want to recreate the database with the content intact.

Inserting the content back into the database is trivial ofcourse:


   $f = fopen($toFile, "r");
   while($s=fgets($f, 128000)) {
    if (trim($s)) mysql_query($s);
   }
   fclose($f);

Commandline Rapidshare.com downloader

2008-03-08T21:28:00.000-08:00

Sometimes you would want to download files from rapidshare via the commandline because the webinterface sucks. Did I say sometimes? You *never* want to use the webinterface :)

You have to have a premium account to use it.


#!/usr/bin/perl

$user = $ARGV[0];
$pass = $ARGV[1];
$file = $ARGV[2];

`mkdir ~/.cookies; wget --save-cookies ~/.cookies/rapidshare --post-data "login=$user&password=$pass" -O - https://ssl.rapidshare.com/cgi-bin/premiumzone.cgi > /dev/null`;

open(F, $file);
while() {
 chomp;
 next if /^$/;
 print `wget -c --load-cookies ~/.cookies/rapidshare $_`;
}

Automatic destruction of phishing sites

2008-03-08T00:50:00.000-08:00

The Americans are getting worse with their phishing 'resolutions'; a lot of provider down your server and others implement draconian measures like giving you less than 1 hour to remove the site before they shutdown your equipment. Because we are a small company, we cannot sit behind email 24/7. So we had to be smart instead of hiring more people.

How does it work? Make an 'abuse' gmail account, have your mailsystem forward all abuse mails to it. Then run the below program in CRON */5. It'll basically remove all scamming sites from existence after a complaint.

Ofcourse you need to tweak it for your purpose/host.


#!/usr/bin/php

$mbox = imap_open ("{pop.gmail.com:995/pop3/ssl/novalidate-cert}INBOX", 
        "xxyyzzaabbccdd@gmail.com", "yourpass");

$msgs=imap_headers($mbox);
foreach ($msgs as $index=>$header) {
 $content = imap_body($mbox, $index+1);  
 $header = imap_fetchheader($mbox, $index+1);
 $msg = $content;
 $content = $header.$content;

 $res  = array();

 preg_match("/Subject: (.*)/", $content, $res);
 $res = $res[1];
 if (strpos($res, "Abuse")!==false) {
  $res1=array();
  preg_match_all("/(.*.yourdomain.com)/", $content, $res1);

  foreach($res1 as $r) {
   foreach($r as $r1) {
    $i = strpos($r1, ".yourdomain.com")-1; 
    $user= "";
    for(;$i>0;$i--) {
     if (in_array($r1[$i], array(' ', '/', '\n', '\r'))) break;
     $user=$r1[$i].$user;
    }  
    kill_user($user);

    mail("abuse@yourhoster.com", "Re: $res", 
        "Dear,\nThis account has been removed.\n\nThank you,\nA Friend\n\n\n$msg", 
        "From: afriend@yourdomain.com\n");
   }
  }

 }

}
imap_expunge($mbox);
imap_close($mbox);
?>

postfix, cannot connect to mysql server localhost

2008-03-03T13:49:00.000-08:00

When trying to run Postfix with virtual maps in MySQL, you can get the following error:


connect to mysql server localhost: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' postfix

This error is because postfix is running in chroot /var/spool/postfix. So it cannot find the unix socket for MySQL.

If you have the option to run mysql on 127.0.0.1 or on another IP, simply go to;

/etc/postfix

run;

grep localhost *

and replace all localhost to 127.0.0.1 or the IP in all files starting with mysql-virtual.

If you insist on running via unix sockets, you can simply fix it by running;

mkdir /var/spool/postfix/var
mount --bind /var /var/spool/postfix/var

Ater this, all will work fine.

Stupid DDOS attacks - a simple script and common sense

2008-02-29T11:24:00.000-08:00

Today we had some fun; a DDOS attack from some kind of botnet. We can hundreds (thousands) of IPs sending something to Apache like;


$MyNick Aando|$Lock EXTENDEDPROT

The new Linux kernel iptables can DROP/REJECT based on strings, but I was not going to recompile and install an experimental kernel for this 'little problem', risking more downtime than need-be.

So I used sniffing to detect problems:


tcpdump -n -vvv -i eth0 -A|grep -B 1 \$MyNick|grep \.80: > /tmp/ip.log

(grepping for port .80: because it is attacking my webserver)

Then a killer perl script to block the offending IPs:


#!/usr/bin/perl

%ips=();
open(F, "tail -f /tmp/ip.log|");
while(){
 chomp;
 /.*\)\ (\d+?\.\d+?\.\d+?\.\d+?)\..*?\>/;
 $ip = $1;
 next if $ip=~/127.0.0/;
 if (!$ips{$ip}) {
  $ips{$ip} = 1;
  `iptables -A INPUT -p tcp  --source $ip --dport 80 -j REJECT`;
  print "$ip blocked\n";
 }
}
close F;

This'll do it; server freed up and we could serve again nicely :) No need for recompiling at all!

Apache logs: simple log analyzer in Perl - II

2008-02-28T02:07:00.000-08:00

I added top 30 and bandwidth / minute;


#!/usr/bin/perl

# get handler to the logfile
open(LOG_FILE, "tail -f /var/log/httpd/access_log.users|");

# start the main loop
%users = ();
$cp=0;
$ct=time();
while() {
 chomp;
 
 $cp = time() + 5*60 if $cp==0;
 if (-f "/tmp/killtraffic") {
  %users=();
  `rm -f /tmp/killtraffic`;
 }

 my $t1 = time(); # timestamp for checkpointing

 my ($dom, $ip, $date, $req, $code, $ref, $client, $in, $out) 
                = /(.*?)\ (.*?)\ .*?\[(.*?)\]\ \"(.*?)\"\ (.*?)\ .*?\"(.*?)\"\ \"(.*?)\"\ (.*?)\ (.*)/;

 my ($method, $uri, $proto) = ($req =~ /(.*?)\ (.*?)\ (.*)/);

 my $get = 0;
 if ($uri =~ /(.*?)\?(.*)$/) {
  $uri = $1;
  $get = $2;
 }

 my $url = $dom;

 if (!$users{$url}) {
  $users{$url} = $in + $out;
 } else {
  $users{$url} = $users{$url} + $in + $out;
 }

 $cpc = time();
 if ( $cpc > $cp ) {
  $cp=0; 
  my $s="";
  $m = (time()-$ct) / 60;
  $count = 0;
  foreach(keys %users) {
   $s.=($users{$_}/1024)." kb - ".($users{$_}/1024/$m)." kb/min - ".$_."\n";
   $count++; 
   last if $count>30;
  }
  `echo "Count started $m minutes ago\n" > /tmp/traffic.log`;
  `echo "$s"|sort -n -r >> /tmp/traffic.log`;
 }
}


# never get here:
close LOG_FILE;

Apache logs: simple log analyzer in Perl

2008-02-28T01:38:00.000-08:00

Get Apache to log your bandwidth by putting in httpd.conf/apache2.conf;

LogFormat "%V %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %I %O" combined_io
CustomLog /var/log/httpd/access_log.users combined_io

And run the following perl script;


#!/usr/bin/perl

# get handler to the logfile
open(LOG_FILE, "tail -f /var/log/httpd/access_log.users|");

# start the main loop
%users = ();
$cp=0;
while() {
 chomp;
 
 $cp = time() + 5*60 if $cp==0; # set to refresh the traffic.log file every 5 min
 
 my $t1 = time(); # timestamp for checkpointing

 my ($dom, $ip, $date, $req, $code, $ref, $client, $in, $out) 
                = /(.*?)\ (.*?)\ .*?\[(.*?)\]\ \"(.*?)\"\ (.*?)\ .*?\"(.*?)\"\ \"(.*?)\"\ (.*?)\ (.*)/;

 my ($method, $uri, $proto) = ($req =~ /(.*?)\ (.*?)\ (.*)/);

 my $get = 0;
 if ($uri =~ /(.*?)\?(.*)$/) {
  $uri = $1;
  $get = $2;
 }

 my $url = $dom;

 if (!$users{$url}) {
  $users{$url} = $in + $out;
 } else {
  $users{$url} = $users{$url} + $in + $out;
 }

 $cpc = time();
 if ( $cpc > $cp ) {
  $cp=0; 
  my $s = "";
  foreach(keys %users) {
   $s.=($users{$_}/1024)." kb - ".$_."\n";
  }
  `echo "$s"|sort -n -r > /tmp/traffic.log`;
 }
}


# never get here:
close LOG_FILE;

By viewing /tmp/traffic.log, you'll see what/who is using most traffic.

A better killall and such

2008-02-24T23:09:00.000-08:00

Didn't you sometimes wish you had a 'superkill'; a kill that kills all processes with a certain text in it. Killall is supposed to do that, but actually never does, as you have to type the complete command which is a pain. So I usually type:

ps auxw|grep -i apache|awk '{print $2}'|xargs kill -9

in this case killing Apache (1 and 2).

To make life easier, as a regular part of my Debian and Ubuntu installs, I now have the command 'sk' (superkill) in /usr/sbin/sk;

#!/bin/sh

ps uaxw|grep -i $1|grep -v $0|grep -v grep|awk '{print $2}'|xargs kill -9

allowing me to run;

sk apache

and gone it is.

Now that I am writing this anyway; here is another missing command :)

find /some/directory -maxdepth 1 -type f -mtime +2 -exec rm {} \;

Into a file /usr/sbin/ro;

#!/bin/sh

find $1 -maxdepth 1 -type f -mtime +$2 -exec rm {} \;

Talking to SMTP / Mail via telnet to diagnoze mail problems on a server

2008-01-07T07:30:00.000-08:00

Mailserver diagnoses can be tricky: I personally find the easiest way to do it still using telnet in combination with the server logs to find out what exactly is wrong.

It is very easy to do:

telnet thehost.com 25

helo: myhost.com
mail from: me@myhost.com
rcpt to: someone@thehost.com
data

... type some data ...
.

quit

That is all; it will help to exactly determine where it goes wrong.

The case we had today;

- postfix didn't send mail ; it gave connection refused
- thoughts; firewall, network, dns etc.
- we tried the above and it worked fine, meaning postfix was the one refusing, not the network/firewall etc
- we fixed the postfix config and all was fine

PHP Posting to ASP.NET (ASPX) pages / sites

2008-01-06T08:30:00.000-08:00

If you want to post anything (for instance a login/password), you need to take the viewstate in order to be able to post anything to any aspx page, so request the page for the first time, in the same session:


  $ch = curl_init();

  curl_setopt($ch, CURLOPT_URL, $login_url);
  curl_setopt($ch, CURLOPT_HEADER, 1);
  curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
  curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
  curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookie.txt');
  curl_setopt($ch, CURLOPT_POST, 0);
  curl_setopt($ch, CURLOPT_POSTFIELDS, array());
  $data = curl_exec($ch);

Then you collect the viewstate;


  $vars = array("login"=>$user,
                "password"=>$pass);

  $i = strpos($data, "__VIEWSTATE");
  $i = strpos($data, "value=", $i);
  $i += 7;
  $j = strpos($data, "\"", $i+1);
  $viewstate = substr($data, $i, $j-$i);
  $vars['__VIEWSTATE'] = $viewstate;

After that you can safely login to the site;


  curl_setopt($ch, CURLOPT_URL, $login_url);
  curl_setopt($ch, CURLOPT_HEADER, 1);
  curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
  curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
  curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookie.txt');
  curl_setopt($ch, CURLOPT_POST, 1);
  curl_setopt($ch, CURLOPT_POSTFIELDS, $vars);
  $data = curl_exec($ch)

Analyzing MySQL usage; finding and fixing bottlenecks

2007-12-24T11:11:00.000-08:00

MyTop and Mtop are nice tools to analyze MySQL load and queries, however long running systems require another system, so I present a simple script to analyze longer running multi user databases for abusers and bottlenecks. The code is very easy to alter to retrieve other stats.
Otherwise you can always mail me to fix it for what you might need.


#!/usr/bin/perl

while(1) {
 # load all previous data, if any
 %users = (); 
 if (-f "./sql.log") {
  open(F, "./sql.log"); 
  while() {
   chomp; 
   # user no_queries tot_length
   /(.*?)\ (.*?)\ (.*)/;
   $users{$1} = "$2 $3";
  }
  close F;
 }

 @p = `mysql -u yyy --password=xxx --execute='show full processlist'`;
 for($i=3;$i<scalar(@p)-1;$i++) {
  $r = $p[$i]; 
  chomp($r); 
  @cols = split(/\t/, $r);

  next if $cols[4] eq "Sleep";

  $q = ""; 
  for($j=7;$j<scalar(@cols);$j++) {
   $q .= $cols[$j]." ";
  }
  next if $q=~/show full processlist/;
  next if $q eq "NULL";

  $u = $cols[1];  
  $l = $cols[5];

  $res = "";
  if (!$users{$u}) {
   $res = "1 $l";
  } else {
   $res = $users{$u}; 
   $res =~ /(.*?)\ (.*)/; 
   $l += $2;
   $t = $1 + 1;
   $res = "$t $l";
  }
  $users{$u} = $res;
 }
 
 open(F, ">./sql.log");
 foreach(keys(%users)) {
  $res = $users{$_};
  print F "$_ $res\n";
 }
 close F;
}

Automatically kill all processes that do not belong on your system

2007-12-21T05:08:00.000-08:00

A client of mine asked me for a program which would destroy 'illegal' processes. After a brief search I found the existing apps too limited or not configurable, so I threw one together that learns itself which processes are and are not allowed.

Simply run it like;

./process.pl learn

first; just do all stuff on your computer/server that is normal, so the system can learn.

After that press ctrl-c and rerun as:

./process.pl check

That's it!


#!/usr/bin/perl

$cmd = $ARGV[0]; 

if (!$cmd) {
 $cmd = "check"; 
}

%allowproc = (); 
if ($cmd eq "check") {
 open(F, "procs.log"); 
 while() {
  chomp;
  $allowproc{$_} = 1; 
 }
 close F; 
}

if ($cmd eq "learn") {
 open(F, ">procs.log"); 
}

while (1) {
 @procs = `ps auxwww`;
 foreach(@procs) {
  chomp; 
  /.*?\s+(.*?)\s+.*?\s+.*?\s+.*?\s+.*?\s+.*?\s+.*?\s+.*?\s+.*?\s+(.*)/;
  next if /^$/;
  next if /defunct/;
  next if /process.pl/;
  if (!$allowproc{$2}) {
   if ($cmd eq "learn") {
    $allowproc{$2} = 1;
    print F $2."\n";
   } else {
    if (!$allowproc{$2}) {
     `kill -9 $1`;
    }
   }
  }
 }
 sleep 1;
}

How to make sendmail send from another IP FROM address

2007-12-19T04:44:00.000-08:00

put, in /etc/mail/sendmail.cf;

O ClientPortOptions=Family=inet, Addr=YOURIP

where YOURIP is the address you want to use to send from.

Handy if one of your IPs is blacklisted or if it doesn't have PTR record (but others on the server have).

Removing the Linux ext filelimit for scripts that do not circumvent that limit (like Mihalism multi host image hosting script)

2007-12-18T08:26:00.000-08:00

With a few changes this will work on any script / system ofcourse, but it was done for Mihalism.

First you need to put all files in directories, go to the 'files' directory and run;

ls -la|awk '{print $8}'|perl -e 'while(<>){chomp; @a=split(//, $_);$d="";for($i=0;$i<5;$i++){$d.=@a[$i]."/"; print `mkdir $d`}; print `mv $_ $d`;}'

Then the code needs a bit of fixing;

open the file

source/global_functions.php

and add a function;

function calc_directory($file){
$dir = "";
for($i=0;$i<5;$i++) $dir .= $file[$i]."/";
return $dir;
}

then you need to fix all files that are calling the $path and $file to reflect the changed;

in the main directory of the script, run;

grep -R \$path.\$file *|awk '{print $1}'|uniq|perl -e 'while(<>){chomp; /^(.*):$/; $f=$1;$s=`cat $f`; $s=~s/\$path.\$filename/\$path.calc_directory$\$filename$/isgm; $s=~s/\$path.\$file/\$path.calc_directory$\$file$/isgm; open(F, ">$f"); print F $s; close F;}'

and then test; I had no time to test yet, but the idea should be clear enough.

Simple stuff to use and remember; one liner grep and replace and when the time on your server is not ok

2007-11-15T01:29:00.001-08:00

Grep and replace over all files in the current dit and the dirs below that;


perl -e '@f=`find . -type f`;foreach(@f){chomp;$s=`cat $_`;$s=~s/php/jsp/isgm;open(F,">$_");print F $s; close F;}';

When the time on the server is not ok, even after running ntpdate, probably your timezone is wrong. Run:

tzconfig

and set your timezone even when it looks ok already.

Preventing bugs in PHP : stripslashes

2007-11-14T07:06:00.000-08:00

To prevent code injection, most PHP code has magic_quotes_gpc = On. For more than one reason this bad and introduces a lot of (difficult to debug) problems in large code bases. So it is better to switch that off from the get go.

If you do not have permission to switch that off (like on a lot of free hosting services), you can call the following code before doing anything else.


if(get_magic_quotes_gpc()) {
 foreach($_REQUEST as $k=>$v) {
  $_REQUEST[$k] = stripslashes($v);
 }
 foreach($_POST as $k=>$v) {
  $_POST[$k] = stripslashes($v);
 }
 foreach($_GET as $k=>$v) {
  $_GET[$k] = stripslashes($v);
 }
}

Ofcourse do not forget to escape stuff that goes into your database!

Checking what is eating your memory.... Part III

2007-11-08T01:21:00.000-08:00

Finally I found out what it was AND it was / is (using this script) a major flaw in the (gz) output handler; a user on the server uses in his code;

ob_start("ob_gzhandler", 9);

something();

ob_flush();

If something() generates a certain amounts of data, the ob_gzhandler will get stuck in a loop, eating all memory.

In the user his particular case, because his database was not working, some pages always have this 'magic size' (probably some factor of 9 in this case).

I tried this on the latest stable of PHP 5 and the bug is still there; you can crash any php hosting machine on the net with this :(

As to prevent this kind of thing from happening, I put the following code on the machines;


#!/usr/bin/perl

$maxmem=10;

while(1) {
  @r=`ps auxwhww|awk '{print \$4,\$2,\$11,\$12,\$13,\$15}'|grep httpd-users|grep -v grep`;
  foreach(@r) {
   chomp;
   /(.*?)\ (.*?)\ (.*)/;
   if ($1>$maxmem) {
    print "$_ killed\n";
    `kill -9 $2`;
   }
  sleep 1;
}

Update: Only PHP5.x is affected, not lower (tested); didn't try higher. I have code to reproduce it every run.

Enterprise workflow................

2007-11-07T08:10:00.000-08:00

Today the server of an important client went down. Was not reachable. Not that interesting as, with the enormous amounts of servers we are managing these days, that happens more often.

What was nice about it is the process surrounding this issue. The server was the mail relay server. So the client could no longer send mail from his website to the outside world, as this mail goes via the relay server.

Unfortunately the admin was, for a brief moment not reachable to use remote powerboot to fix it, so I uttered the, in hindsight, dangerous words;

this is a mail relay, why don't we just set another relay for the moment?

Apparently the 10 minutes (even less...) required to change the relay in Postfix, were translated into the company (that hired me) workflow to:

1. ask John to create an issue in Jira
2. assign the issue to the person in charge of the admin work load distribution, Frank
3. Frank will check the roster of the admins and assign one; Paul
4. Paul will estimate the time needed to change the relay (a whopping 8 hours!) and assigns the issue back to Frank
5. Frank fixes the estimate by Paul by setting the estimate to estimate*PI and assigns back to John
6. John assigns the issue to sales
7. Sales talks to the client and asks for estimate*PI*120%*$100 ~ $3000
8. After approval the issue is assigned back to John
9. John assigns to Frank
10. Frank assigns to someone other than Paul (as that is how it goes)

During step 2 the admin came back and rebooted the server.

Unfortunately the above is not strange; any person working at a bigger company knows that this is actually not too bad compared to even bigger companies. In fact; Frank or Paul in steps 2,3,4 should have sent the issue back to John for lack of a clear RFP. And the estimate is actually quite low, considering someone has to write an RFP, FO and TO for this problem. And no, I am not kidding, as I see it happen every day.

Enterprises are weird entities that spend massive amounts of $ on nothing at all. Wondering if it is possible to run one without all that crud... It is a disease and I am waiting for 10 years already for IT to kick in and make at least the almost enterprises or middle sized companies a lot leaner than this. Hasn't happen though. People still buy ERP systems for millions that are unusable, they still hire 'enterprise Java' guys for $200/hour and consultants for $2500/day to tell them how to work Websphere and so on. A sick, sick world.

(the names have been changed to protect the innocent)

Checking what is eating your memory.... Part II

2007-11-06T01:14:00.000-08:00

The experiment of yesterday was showing me only apache2 doing this; I was hoping the strace would show me the path to the offending code (as I am almost 100% sure it is PHP doing it). Instead it showed me an entire file full of mmap2 which are taking away memory. No paths as the strace only kicks in when the system is already dying, so too late.

So I had to make some code to check this further:


#!/usr/bin/perl

$maxmem=10;

`mkdir memstrace`;
`rm memstrace/*`;

%detected=();

open(F, ">mem.log");
while(1) {
  @r=`ps auxwhww|awk '{print \$4,\$2,\$11,\$12,\$13,\$15}'|grep httpd-users|grep -v grep`;
  foreach(@r) {
   chomp;
   /(.*?)\ (.*?)\ (.*)/;
   if ($1>$maxmem and !$detected{$2}) {
    print F "$_\n";
    $detected{$2} = 1;
   }
   next if -f "memstrace/$2";
   if (!fork()) {
    `strace -f -o memstrace/$2 -p $2`;
    exit;
   }
  }
  sleep 1;
}
close F;

Which should store all Apache syscalls including the offending and then log which one is offending when it becomes that.

Checking what is eating your memory....

2007-11-05T09:03:00.000-08:00

We have several servers running > 50.000 sites. On those servers there is always 2 gb of memory free. Except sometimes when memory suddenly disappears in seconds and the system starts swapping, load jumps to > 400 in seconds after that. This happens once every few hours for a few days now.

When watching top, we see that memory of Apache2 grows to > 50%. So what site is doing that out of 50.000+? I made a script for checking that;


#!/usr/bin/perl

$maxmem=10;

`mkdir memstrace`;
open(F, ">mem.log");
while(1) {
  @r=`ps auxwhww|awk '{print \$4,\$2,\$11,\$12,\$13,\$15}'|sort -n|tail -n 5`;
  foreach(@r) {
   chomp;
   /(.*?)\ (.*?)\ (.*)/;
   if ($1 > $maxmem) {
    next if -f "memstrace/$2";
    print F "$_\n";
    if(!fork()) {
     `strace -f -o memstrace/$2 -p $2`
     exit;
    };
   }
  }
  sleep 1;
}
close F;

Perl FP : getting abusers from my mail logger one-liner

2007-10-29T06:47:00.000-07:00

To catch abusers of my free hosting service, I altered the PHP C source to write mail sending scripts (we only allow mail() to send mail; no fsockopen etc) to a log file. This file looks like this:

/www/user/htdocs/sendmail.php
/www/user/htdocs/sendmail.php
/www/user/htdocs/sendmail.php
/www/user/htdocs/sendmail.php
/www/user/htdocs/sendmail.php
/www/user/htdocs/sendmail.php
/www/user/htdocs/phpBB3/something.php
etc

Now I wanted to find people that are sending more than X mails and get a warning about them. For this I wrote the following code;

map { /(.*?)\ (.*)$/; if($1>$cm) {print "Abusing: $2\n";} } `cat $mf|sort|uniq -c|awk '{print \$1 " " \$2}'|sort -g`;

where $mf is the log file and $cm the X which is the maximum number of mails a user is allowed to send.

I run this in a cron and I actually block users with more than 400 mails. Works like a charm.

Linux disk stuff - what brand drive do you have

2007-09-28T02:56:00.000-07:00

I always forget commands I don't use daily; for instance, to get at least some idea about why, at one provider, more drives were breaking than at another provider, I needed the brand of the drivers. Ofcourse I cannot reach the servers physically, only via SSH. Searching Google didn't give me much on this, but after a long search in forums, I found it;

apt-get install smartmontools

box46:/disk1# smartctl --all /dev/sdc1
smartctl version 5.36 [i686-pc-linux-gnu] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

Device: ATA WDC WD2500YS-01S Version: 20.0
Serial number: WD-WCANY1884382
Device type: disk
Local Time is: Fri Sep 28 04:59:32 2007 CDT

And yes, Western Digital drives seem to break (much) more than Samsung and Hitachi for the same work + load.

register_globals Off is not the end of the world

2007-09-19T05:48:00.000-07:00

We are getting continues remarks from our hosting users that 'they cannot work with register_globals Off'.

Register_globals is a way to pass GET & POST variables to a PHP script as normal PHP variables, for instance;

test.php:
echo $test;
?>

test.php?test=hello

would result in

hello

when register_globals is On.

A lot of (older) apps are built on this system as 'security' was not a problem before :)

Thing is, this is not a security problem per-se, but it invites security issues; if you are not very careful you will introduce them. How? For instance (I actually found this in an open source app... I changed it here to match the same principle in a few lines);

test1.php:
if (!$user) echo "please enter your username";
else include('test2.php');
?>

test2.php:
if ($user)
mysql_query("select * from users where user = '$user'");
?>

A malicious user could now do the following;

test2.php?user='; select * from users;'

or something like that to get all users.

So the preferred way is to make it a bit more difficult for developers so they have to actually think about what they are doing and what variables they are passing; the better way is;

test.php:
echo $_GET['test'];
?>

and

test1.php:
$user = $_GET['user'];
if (!$user) echo "please enter your username";
else include('test2.php');
?>

test2.php:
if ($user)
mysql_query("select * from users where user = '$user'");
?>

executing test2.php as;

test2.php?user='; select * from users;'

would result in nothing as $user will be empty in test2.php.

Anyway; programmers using 'the old way' suck. But to keep the users of your back, you can easily help them by; a) educating them about the dangers b) telling them to rewrite the code. But most cannot or will not. And for them, you can give them this code and pray all will be fine;

foreach($_REQUEST as $k=>$v) $$k=$v;
?>

or;

foreach($_REQUEST as $k=>$v) $$k=$v;
foreach($_COOKIE as $k=>$v) $$k=$v;
?>

Putting this on top of all files (better is one generic header or include file, but even a lot of programs don't have that....).

It will result in the same idea as register_globals On :)

New Site!

2007-09-18T15:52:00.001-07:00

Niceee....

Combine MySQL databases ignoring keys

2007-09-18T15:43:00.000-07:00

I would like to do something like this;

on server1; mysqldump mysql > db1.sql
on server2; mysqldump mysql > db2.sql

on server3 (new server);

mysql --force mysql < db1.sql
mysql --force mysql < db2.sql

which should give me db1.sql users root etc but it should combination of the users from db1 + db2, so for instance;

db1.sql => users{root,a,b,c}
db2.sql => users{root,a,d,e}

on server3 I want to see;

users{root,a,b,c,d,e}

Ofcourse that doesn't work, as you get duplicate key messages on all double users.

So ofcourse I wrote something to fix that;

I first did;

mysql mysql < db1.sql
mysql mysqlold < db2.sql

PHP:


<?
 mysql_connect("localhost", "", ""); 
 mysql_select_db("mysqlold");
 
 $q = mysql_query("select * from mysqlold.user");
 while ($r = mysql_fetch_row($q)) {
  $q1 = mysql_query("select * from mysql.user where User='".$r[1]."'");
  if (mysql_num_rows($q1)) {
   continue;
  }
  $s="";
  foreach($r as $r1) {
   if ($s) $s.=",";
   $s.="'".$r1."'";
  }
  $qry = "insert into mysql.user values ($s)";
  mysql_query($qry);
 }
?>

Ofcourse, writing everything yourself was again much faster than searching the solution in the MySQL manual or on Google.

Online voting ... about anything !

2006-04-05T08:12:00.000-07:00

There are a lot of URL voting applications on the web which have gained a very large audience, like www.digg.com and www.reddit.com. These systems are really nice, but the problem is that they are not meant to make you vote on anything else but URLs.

Because of this, we created a generic voting application. Simply made to vote on anything. Favorite restaurant; put it in. Favorite movie; let' s have it. Favorite ... Whatever.

A very nice and effective application this week for the movinglabs; votelists.com !

And... Another great tool, the best so far... Sharepoint, eat your heart out (really, you should)

2006-03-30T03:30:00.000-08:00

Our latest and greatest tool is including great and cool Web 2.0 AJAX technology and is a replacement for all that sad and boring Excel-with-list sending over the mail. How often do you get an Excel sheet in your mail with no calculations, but only an enumeration of some sort ?

I get this daily; even yesterday one of my colleagues sent me an Excel with;

index servername URL action comment

He sent it to all people in the company (100) and asks if the responsible person for that site can please put some action in it and, if needed, some comment of what to do with this site/url.

...

A bit shocked I mailed back if we don't have Sharepoint for this and he replies; it is too much hassle to put it into SP and our SP will get a mess because of it.

...

So our new invention has it's merits. And we are using it already for everything.

Without further blabla, I proudly present; flexlists.com.

Also some other things we did; www.movinglabs.com.

And I added very secure CLIENTSIDE encryption for yourdraft.com.

Making phpBB a bit more secure

2006-03-27T18:50:00.000-08:00

I actually like phpBB. Why? Because it is mature and usually working perfectly for setting up a nice (looking) forum in a really short time. phpBB has a lot of 'security' problems. Usually these problems are not really problems that are in the forum software itself, but through it's massive use, the forum, if not propertly secured, is attacked by a lot of automated software on the internet.
When I set up a forum for the first time, I usually allow anonymous posting. This is one of those 'security' things; anonymous posting makes it very easy for a robot to run over the internet, and post crap like viagra and casino links.

To fix this, phpBB has an option to make only registered users post to a forum. This, however, is not working ok either, because the robots can register themselves and then post crap.

Luckily the system has a way to 'secure' the registration process by adding a captcha to the forum registration form. This is, however, in the case of phpBB, not safe.

So what can you do? Just change the captcha algorithm by your own! And when it is hacked, just replace it again! At least the automated bots on the internet don't stand a chance this way.

So here is a little howto for phpBB 2.x.

Edit file includes/usercp_confirm.php ; go to the line before;

if (@extension_loaded('zlib'))

(line 67 in my version)

and put

exit;

there.

Before that, you will put your code.

Now download the following captcha code, for instance:

http://www.ejeliot.com/pages/2

and put it in your phpBB main directory.

You are almost done now.

Add the following above the 'exit' in includes/usercp_confirm.php;

require('php-captcha.inc.php');
$aFonts = array('/usr/share/fonts/truetype/ttf-bitstream-vera/Vera.ttf');
$oPhpCaptcha = new PhpCaptcha($aFonts, 202, 43);
$oPhpCaptcha->Create('', $code);

Change this line;

$aFonts = array('/usr/share/fonts/truetype/ttf-bitstream-vera/Vera.ttf');

to a font (or more; it is an array ;) which actually exists on your system.

Now this isn't working still, because

$oPhpCaptcha->Create('', $code);

does not exist; you need to edit the php-captcha code. Open the file php-captcha.inc.php and find the function 'GenerateCode'; change it as follows;

function GenerateCode($generate=0) {
// reset code
$this->sCode = '';

if (!$generate) {
// leave original code here!
} else $this->sCode = $generate;

// save code in session variable
if ($this->bCaseInsensitive) {
$_SESSION[CAPTCHA_SESSION_ID] = strtoupper($this->sCode);
// etc; put original code here

Now find the first Create constructor (the second is of the sound captcha's!) and find the line;

$this->GenerateCode();

change this to:

$this->GenerateCode($generate);

Now the code will be working fine.

If you need to debug, go to the registration page of your phpBB site;

/profile.php?mode=register&agreed=true

open the HTML source and search for an img src attribute which contains;

/profile.php?mode=confirm&id=

open that in the browser and you'll see why something is not working.

The next tiny tool!

2006-03-21T08:03:00.000-08:00

As promised...

Not even a week further, the next tool has been released; yourdraft.com.

The tools fills a small gap in my webtool repository; a simple and no-nonsense site for putting drafts of documents and files without having to concern myself with irritating account information and so forth.

The editing is quite smooth and simple; you just start typing and all information will be stored for later viewing. No passwords etc to remember. It is not really secure, but it was not meant to be; you can work together on a document or show someone something for a short period and then remove it again.

Number three of our tools will be a bit more elaborate; it will fill a gap which, to date, was not filled by anything worth-while.

Creating MySQL databases securely

2006-03-17T14:44:00.000-08:00

When building sites I often need to set-up a MySQL database which has all proper settings so I can start working right away knowing that all is fine. Here is a little script which I use to do this;

#!/bin/sh

if [ ! -n "$1" ]; then
echo "Invoke as ./createdb.sh DATABASE_NAME"
exit -1
fi

DBNAME="$1"

mysqladmin --user=root --password=SOMEPASS create $DBNAME

PASS=`perl -e 'for($i=0;$i<6;$i++){print rand()*26%26};'`
mysql --user=root --password=SOMEPASS -e "grant all privileges on $DBNAME.* to $DBNAME@'localhost' identified by '$PASS';"
mysql --user=root --password=SOMEPASS -e "flush privileges;"

echo $PASS

Software writing in the fast lane

2006-03-15T23:11:00.000-08:00

I am often searching the internet for handy programs and tools. Unfortunately, the never have anything I need or I cannot find it (which means, they did not market it properly ofcourse ;).

Usually I do some brainstorming with a friend about this kind of thing and then we think about building this tool ourself if we don't like the existing ones or if we cannot find any. Our conversation usually go a bit like this;

me; would it not be nice to have X
he; yes that would be excellent, it would have features a,b,c
me; and ofcourse d,e,f,g,h
he; and it would certainly need i,j,k,l,m
me; right! that's the least it should have

After about 30 minutes in this kind of talk, we both conclude that we need a rapid web application development environment and that X is just a subset of that environment.

Needless to say; we don't have time, resources or even energy to start with the production of an RWADE...

We decided, after years of those kind of pointless sessions, to do the following; we will think of something which can be built, from concept to live application on the web, in less than 24 hours.
This forces it to be makeable and it cannot be an RWADE ;) We decided to release such a 'nice little tool' every week.

The first we have is forlater.net. A very simple tool (take a look) which was built in about 4 hours from concept to the site you see online.

Hiding affiliate urls

2006-02-11T07:06:00.000-08:00

Let's say you are a webmaster into a few affiliate programs. You will have URLs which will look like;

http://www.somesite.com/?id=xyz

or

http://www.somesite.com/affiliate/xyz

or something like that.

The best way of getting visitors to not go directly to the site is making them believe (and the searchengines for that matter), that you are not an affiliate but that it is your site.

You do this by registering a domain and then using mod_rewrite in Apache:

Let's say we registered www.someothersite.com and you want to rewrite, without users be able to see, this url to http://www.somesite.com/?id=xyz, you should set, in Apache;

ServerName www.someothersite.com
RewriteEngine On
RewriteRule (.*) http://www.somesite.com/$1?id=xyz [P]

No-one will be able to see the id in the URL without using frames, iframes etc.

Apache + Webdav + LDAP + Debian

2006-02-01T03:48:00.000-08:00

Getting those to work is not that difficult, but a LOT of site keep nagging about using:

LDAP_Server

You will get something like:

Invalid command 'LDAP_Server'

Prerequisites are:

- Apache 2 under Debian
- Installed and working (open)LDAP

Something like; apt-get install apache2

Symbolic link the dav_load in mods-available/ to mods-enabled/. Do the same with auth_ldap.load and dav_fs.conf and dav_fs.load.

Then add the following to apache2.conf:

Listen 666
<VirtualHost *:666>
DAVLockDB /var/lib/dav/lockdb

Alias /share /home/share

<Location "/share">
Options Indexes FollowSymLinks
AllowOverride None
order allow,deny
allow from all

DAV On
AuthType Basic

AuthLDAPBindDN cn=admin,dc=mydomain,dc=com
AuthLDAPBindPassword SOMEPASS
AuthLDAPURL ldap://127.0.0.1/ou=People,dc=mydomain,dc=com?uid?sub?(objectClass=*)

AuthName "Repository Access"
Require valid-user
</Location>
</VirtualHost>

/home/share

will be shared now under;

http://somedomain/share

You can use SSL by putting:

SSLEngine On

in the <Location> directive.

I had a problem that I also use Samba and SCP for access on the same system, meaning that there could be a potential sharing access problem; everything must be written with group user and then made 770 directory/file mask.
To fix this, put in /etc/init.d/apache2

umask 0007

somewhere at the start and restart Apache. It will work..

Installing LDAP on Debian

2006-01-31T12:43:00.000-08:00

Installing LDAP is not that simple. Some basic LDAP knowledge is needed, but for setting up quickly do;

apt-get install slapd ldap-utils migrationtools

Answer all questions; make sure you enter for the first to screens (DN and org) the same name, for instance domain.com. You don't have to do this, but if you don't know anything about LDAP this is advisable, because you'll get errors like;

ldap_bind: Invalid credentials

Now you can add your /etc/passwd users by migrating them with:

cd /usr/share/migrationtools
./migrate_passwd.pl /etc/passwd /tmp/passwd.ldif

When importing these like:

ldapadd -x -v -D cn=admin,dc=domain,dc=com -w YOURPASSWORD < /tmp/passwd.ldif

You'll probably get some error like:

parent does not exist

Because your passwd.ldif contains:

ou: Groups

If you want to know why, you should look at an organizational schema of LDAP, but if you don't care, just make an ldif file like:

dn: ou=People,dc=domain,dc=com
ou: People
objectClass: organizationalUnit

And add it;

ldapadd -x -v -D cn=admin,dc=domain,dc=com -w YOURPASSWORD < /tmp/ou.ldif

Now add the passwd.diff

ldapadd -x -v -D cn=admin,dc=domain,dc=com -w YOURPASSWORD < /tmp/passwd.ldif

and check if all went ok;

/usr/bin/ldapsearch -x -p 389 -h localhost -w YOURPASSWD -D 'cn=admin,dc=componence,dc=com' objectClass=*

It should show all you added and set.

Waiting for all threads to end in Java

2006-01-28T06:46:00.000-08:00

Looking in Google for some easy out-of-the-box way for waiting for all threads to end and then continue the main() (or something) method, I found that no simple way was indicated anywhere. Use thread.join(). Yeah sure, only problem is that I have multiple threads and I want all of them to run and time the running time.

For example: I have 4 threads:

Thread t1,t2,t3,t4;

which I run from a main() method in a Java class. The main method looks like this:

public void main(String[] args) {
... some stuff before

// first run
t1.start();
t2.start();
t3. start();
t4.start();

// second run
t1.start();
t2.start();
t3. start();
t4.start();
}

Now I want to not start the second run before the first is over. A way to do this, is by running:

public void main(String[] args) {
... some stuff before

// first run
t1.start();
t2.start();
t3. start();
t4.start();

// wait till all of these are done
t1.join();
t2.join();
t3. join();
t4.join();
... etc

For this trivial case, this is ok, but in the real program I am writing, a lot of threads are started, for instance from classes.methods all over the place, making keeping references to them for joining quite a challenge.

Anyway; Google did not indicate to me a straight, simple and generic way of doing it, so I did the following:

class ThreadCheck extends Thread {
List l = new ArrayList();

public void queueThread(Thread t) {
l.add(t);
}

public void run() {
for (int i=0;i<l.size();i++) {
try {
l.get(i).join();
} catch (Exception e) {
e.printStackTrace();
}
}
}

Now you can add your Threads to the object;

tc=ThreadCheck();
t1.start();
tc.queueThread(t1);
t2.start();
tc.queueThread(t2);
etc

I usually pass tc to the the Thread when creating it, so it can queue itself after it is started; this makes it even easier, because you don't need any references to the threads after creation.

Then you can start the tc;

tc.start();

and make sure all ends only when all threads are done:

tc.join();

To time the execution:

long st= System.currentTimeMillis();
itc.start(); tc.join();
long et= System.currentTimeMillis()-st;
System.out.println("Stuff ran for "+et+" milliseconds");

The above method works very well, but if anyone knows a better method, please mail me!

Catching Post Redirects with HttpClient

2006-01-03T16:46:00.000-08:00

For some very good reasons, RFC 2616 and HttpClient do not allow automatic POST redirects. A good reason, but irritating it is nontheless.
I had to get around it today for a bunch of cases which are clear and very common; the case where a POST continues into a GET. For instance; you entered a login/password and are redirected to another page when it is right/wrong without the need for the POST info to be posted to that page also.

The following code has been written for a simple proxy server, so everything is passed to the response;

First of all, you need to know if there is a redirect going on;

method is a PostMethod, request is the HttpServletRequest, response is the HttpServletResponse;

if (request.getMethod().equals("POST") && method.getResponseHeader("location")!=null) {

Then you don't need the current method anymore, because you are redirecting;

method.releaseConnection();

Get the location to redirect to;

String location = method.getResponseHeader("location").getValue();

And make a new GetMethod:

method = new GetMethod(dst+'/'+location);

dst is the destination domain which you know ofcourse, because you cannot use HttpClient without it.

Run the method;

statusCode = client.executeMethod(method);

Make sure the response knows that this is another URL: you can experiment with this to show any URL in the browser of the client:

response.setStatus(HttpStatus.SC_MOVED_TEMPORARILY);
response.setHeader("Location", location);

This works fine for the cases I tried like logging in and filling forms.

Windows XP and Samba connections

2005-12-31T02:26:00.000-08:00

When working for a long time with XP without resetting (yep, that's error number one right there), my laptop, when taken from location A to location B and back to A seems to be unable to connect to some Samba(SMB) resources protected with a password. Windows reports that my credentials (username/password) are incorrect while they are not.

I haven't been able to find out what causes this, but I have found how to work around this without resetting. It was quite trivial actually, but when talking to other people they would simply comment; 'just reboot, all will be fine'. This is not ok with me, because I use hibernate on my laptop so I don't have to wait to reboot Windows, which is still very very slow. Usually after a few days of hibernations, Windows starts to break; it doesn't work properly anymore; windows don't open/close, networking problems like described here today etc. I usually first kill explorer.exe with taskmgr and then restart it, if that doesn't work I log off and on (still takes a lot less than rebooting completely). This usually does the trick, except for the samba problem...

To see if you have the problem; just type a resource into explorer, for instance;

\\10.0.0.13\store

in my case.

It will be stuck for a while and then tell you you don't have enough rights on the resource and to contact your admin.

Open a cmd prompt (left-windows-r + cmd + enter) and type:

net use

You will see the connection you are trying to open in the list, frustratingly enough probably with a status 'OK' (no it isn't you cannot open it you piece of ()*%#@%**)#@).

Then type:

net use \\10.0.0.13\store /delete

And then try to open the connection again. It'll work now.

Moving servers without changing of IPs. No hassle migrations by proxy.

2005-12-30T06:34:00.000-08:00

Ever had to move a bunch of servers (in our case about 30) to another location in a hurry? Every wondered if it would be easy to give your clients the same ip address before and after the migration, at least for a little while until they have their DNS's updated?

Well, it is easy in fact. How? By using a neat little program called tcpproxy. This is actually all you need. You simply wip up a config file;

/etc/tcpproxy.conf

port 22
# old ip
interfaces 192.168.0.1
# new ip
server 192.168.1.1

and run ./tcpproxy

Two servers are moved in need of port 22? Simple, just do:

port 22
# old ip
interfaces 192.168.0.1
# new ip
server 192.168.1.1
# old ip
interfaces 192.168.0.2
# new ip
server 192.168.1.2

etc.

After entering a bunch of servers like this (up to thirty in our case, with every servers having bunches of open ports), you get a bit tired of this format. So let's get our trusty Perl working this for us.

Personally I like configuration formats which are clear, simple and fast. So I wipped up the following format:

servername:oldip:newip:ports

for example:

killer:34:138:22,80,81,8080

and another one:

killer:34:138:22,80,81,8080
killer2:35:139:22,80,81,143,993

etc.

As you might notice (I hope); the ip address is not complete; in our case, we move from one network to another in which the first 3 numbers in the ip address are the same. For instance:

Before IP: 192.168.0.*
After IP: 192.168.1.*

The script I made to read in the configuration file should have some configuration settings:

# settings
$oldr = "192.168.0"; # old ip address range
$newr = "192.168.1"; # new ip address range
$eth = "eth0"; # ethernet card to make aliases for

An observant reader might notice that I added the ethernet card; this is because we want to use one machine for the forwarding, not as many machines as there were before :) Also we would want the settings for this forwarding machine to be auto-generated. For the above case, we need to have:

ifconfig eth0:0 192.168.0.1 up
ifconfig eth0:1 192.168.0.2 up

To make tcpproxy to be able to listen to incoming requests on those addresses.

In case there already are alias interfaces on $eth, we need to figure out what they are, so we can decide where to start counting new aliases:

# calc interface start
$count = `ifconfig|grep $eth|tail -n 1|awk '{print \$1}'|grep :|sed -e s/$eth\://`;
if ($count eq "") {
$count = 0;
} else {
$count++;
}
$first = $count;

Next step is to generate all the settings for the /etc/tcpproxy.conf file and to generate a set and unset interface file for you to run to get the computer in the proper mode to listen to all incoming requests:

open(F, $f);
open(SET, "> ./set$eth.sh");
open(UNSET, "> ./unset$eth.sh");
print SET "#!/bin/sh\n";
print UNSET "#!/bin/sh\n";
while() {
chomp;
next if /^$/;
($umln,$ipo,$ipn,$pts) = split(/:/);
print SET "# $umln\nifconfig $eth:$count $oldr.$ipo up\n";
print UNSET "# $umln\nifconfig $eth:$count down\n" if $count == $first;
$count++;
foreach(split(/,/,$pts)) {
$port{$_}.="# $umln\n";
$port{$_}.=" interface $oldr.$ipo\n";
$port{$_}.=" server $newr.$ipn\n";
}
}
close F;
close SET;
close UNSET;

We cannot write the tcpproxy.conf as we go along, because it is only allowed to have one

port X

per set of interfaces and servers. In the above code we collected all ports with their interfaces settings, contained in $port.

We only have to put these into a nice string which is, in fact, the /etc/tcpproxy.conf file; we leave the saving of it to the reader. For fun it is nicely sorted by port :)

foreach(sort {$a <=> $b} keys %port) {
$tot .= "port $_\n";
$tot .= $port{$_};
$tot .= "\n";
}

Have fun!

Ok, so maybe you should store everything in gmail...

2005-10-18T13:18:00.000-07:00

I lately have been contemplating putting all my mails and some other content into Gmail. Why? Because Google is ofcourse a great searchengine, I don't give a ... about them 'invading my privacy' (running some statistical crap over mails to show some non-intrusive ads) and I find the AJAX client great.

So... I decided to take the step and send all my mail to both my Cyrus IMAP server and Gmail. I migrated all my IMAP mail (hundreds of thousands of mails) to Gmail using;

find . -type f -maxdepth 1 -name "*?." -printf "%T@ %h/%f\\n" |sort -n |colrm 1 11 |xargs -i bash -c "echo {} ; sendmail < {} myaddress@gmail.com ; sleep 2"

(which I found on some webpage which I forgot)

It 'migrates' one folder at a time ; you need access to the linux system on which the IMAP server (must be Cyrus!) resides.

I will put my experiences here soon.

Ripping DVDs with Linux

2005-10-02T01:10:00.000-07:00

Under Windows I never could rip DVDs in a working fashion; I have, under my XP Prof installation, about 40 tools installed which all promise easy ripping, but none of them work. You can rip the VOBs, but then they have wrong language, wrong subtitles and ofcourse they are 4+ gb big. When trying to make 'ok VOB rips' into AVI's or MPG's I tried integrated tools or non-integrated tools; both had the same effect; it didn't work. After hours of 'transcoding' there would be a file 'bla.avi', which would be, in reality complete crap.

Under Linux, I hoped, things might be better. At first, this appeared not the be the case ; I have been, for hours, muttering around with dvdrip which wouldn't compile (well, GTK-Perl wouldn't). This is probably because of that bastard x86_64 architecture of mine.

Anyway; it seemed that mplayer has native support of ripping dvds :) And ofcourse, mplayer rocks!

So, how-to very easily rip dvds with mplayer:

- get mplayer source (from http://mplayerhq.hu)
- try ./configure
- if you have gcc 4 installed, configure will whine about that it cannot continue; you can switch off the check for gcc; don't, because with gcc 4, mplayer won't compile; instead, install compat-gcc-32 and run ./configure --cc=gcc32
- after compilation, there are two binaries; mplayer and mencoder
- to check out what you want to rip with which language and which subtitles (if any), use;

mplayer dvd://NUMBER -alang LANG -slang LANG

For instance (for english and track 1);

mplayer dvd://1 -alang en -slang en

You will see something like this;

There are 22 titles on this DVD.
There are 37 chapters in this DVD title.
There are 1 angles in this DVD title.
DVD successfully opened.
Selected DVD audio channel: 128 language: en
Selected DVD subtitle channel: 0 language: en

- repeat the previous step until you have exactly what you want to rip
- to start ripping, use the following command:

./mencoder dvd://4 -alang en -slang en -ovc lavc -oac lavc -o test.avi

You can set the aspect ratio, I always use;

./mencoder dvd://4 -aspect 16:9 -alang en -slang en -ovc lavc -oac lavc -o test.avi

If you don't include it, your movie might appear 'squared' and when playing fullscreen, this is not that nice :)

- the above won't rip the subtitles though; you need to use the following to get the actual subtitles;

./mencoder dvd://4 -aspect 16:9-alang en -slang en -ovc lavc -oac lavc -vobsubout subtitles -vobsuboutindex 0 -sid 0 -o test.avi

where the vobsuboutindex and sid numbers are the number you saw when playing the movie with mplayer here;

Selected DVD subtitle channel: 0 language: en

- when finished ripping, you can play the movie with:

mplayer test.avi -vobsub subtitles

That's all !

Installing FC4 on an Acer Ferrari 3400

2005-09-19T12:55:00.000-07:00

Installing FC4 went even better than installing FC3; you also need to start with

linux nofb

to make sure you are seeing anything during startup. The rest is straightforward; you need
to select a 1400x1050 Generic LCD screen to get the best resolution going. Further everything
will go fine.

Except the wireless ofcourse.

To get it running, you need ndiswrapper with the 64 bit Windows driver for the Broadcom card.

Search for: 64-bit_Broadcom_54g_Drivers.zip on google to find the right drivers and download the ndiswrapper from the sourceforge.net site.

Unpack the ndiswrapper package and build it:

make
make install

Unpack the Broadcom drivers, and use

ndiswrapper -i netbc564.inf

Then you can load the ndiswrapper into the driver:

modprobe ndiswrapper

Now you can lookup what there is in your current wireless lan:

iwlist wlan0 scan

It should show a list (provided you are somewhere in reach of a WIFI lan)

You can connect to a specific access point, by doing;

iwconfig wlan0 essid "AccessPointName"

(You actually see the ESSID name when running the iwlist scan command)

You don't HAVE to do this; you can let the driver pick an accesspoint itself.

If you have WPA installed, you need wpa_supplicant; get it and install it.

A simple config I am using looks like this:

network={
ssid="AccessPointName"
proto=WPA
key_mgmt=WPA-PSK
pairwise=TKIP
group=TKIP
psk="ThisIsYourPassword"
priority=2
}

This should be in your

/etc/wpa_supplicant.conf

and then switch the WPA encryption on with:

wpa_supplicant -iwlan0 -c/etc/wpa_supplicant.conf -D ndiswrapper -B

If you have a DHCP server, then you can get the IP address like this:

dhclient wlan0

Or you can do it by hand with something like this (depending on your network);

ifconfig wlan0 192.168.1.194
route add default gw 192.168.1.1

Now everything is working!

Playing media files (like MP3s) is not standard in the applications like Gnome Music Player or
XMMS under FC4. You need some adjustments for that; do the following:

rpm -ivh http://rpm.livna.org/fedora/4/i386/RPMS.lvn/livna-release-4-0.lvn.5.4.noarch.rpm

Now you can install all cool stuff; to get MP3s working, do this:

yum install libmad
yum install libid3tag
yum install gstreamer-plugins-mp3

If you get any errors when you try to play, run

gstreamer-properties

and try some different configurations until it works. Then the music player should also work.

To burn CD/DVD I use k3b; use

yum install k3b

to get it installed.

Enjoy!

Windows 2003 and MySQL

2005-07-20T14:54:00.000-07:00

A short post here. Had to install and run MySQL on Windows 2003. Didn't work.
Just used the standard install for 4.0.25; it wouldn't start, not from a service and not from winmysqladmin.exe. Everything seemed ok though, but nothing in the logs and the Windows event viewer kept saying it 'couldn't find the file'. Which was rubbish.
Tried 3.23.58 which didn't work either. I just wanted MySQL to tell me what was wrong, but it said nothing, it just died.

Finally I got it to work using mysqld -T ; I don't know what that is, but it works. Nice experience with Windows again :) Thank god for apt-get install mysql-server!

Dragging and dropping controls in RealBasic

2005-07-05T14:34:00.000-07:00

Someone recently asked me to figure out how to make an application to drag & drop controls on a canvas in RealBasic (RealBasic 2005 to be exact). I thought that about a billion people would have done this by now and would have put their stuff on the web. This turned out not to be true.

The way to do it is actually trivial in any event-driven language, but in RealBasic it seems even easier :)

I will show the dragging and dropping of a pushbutton; you can figure out the rest for yourself. Do not pay attention to the quality of the code; I was only supposed to figure out how to do it, not win any prices!

Make a Window and put a PushButton on it. Now add some properties:

obj as Object
x as Integer
y as Integer
move as Boolean

Put in the MouseDown of the PushButton:

obj = me
return true

and in the MouseUp of the PushButton:

obj = nil
return true

Now put in the MouseMove of the Window:

dim ctl as RectControl

if obj<>nil then
ctl = RectControl(obj)
end if

if (not move and obj <> nil) then
dx = x - ctl.Left
dy = y - ctl.Top
move = true
MouseCursor = System.Cursors.ArrowAllDirections
end if

if (move and obj = nil) then
move = false
MouseCursor = System.Cursors.StandardPointer
end if

if (move and x >0 and y>0 and x < me.width and y < me.Height) then
ctl.Left = x - dx
ctl.Top = y-dy
me.refresh
end if

And that's it :)

Scaling and growing...

2005-06-10T04:27:00.000-07:00

After my clustering experiments which you read about in my last post we were capable of handling more than a 1000 concurrent visitors. We will see if the current growth can be managed by this clustering on the 4 systems currently running the site. It is clear to me now that the bottleneck was actually the MySQL, not the web part. Switching off the second webserver makes no difference, while switching off the second MySQL will completely stop the site from responding. So I removed the one webserver and the site is running on 3 webservers now.

The MySql installation has been tweaked for absolute performance with all the tricks in the books, but it cannot do more than it it doing now on that hardware. I need to figure out a way to see, in the next concurrent userbarrier, which part of the system needs improvement. I will think about it; if anyone has an idea; let's have it (put a comment please).

My next projects are;

- make a seperate site for chat and use that instead of the current chat (dunno which chat yet; have to find a nice one still)
- make a seperate site for the forum and use that instead of the current forum (probably phpbb)
- making a MySQL/PHP cache for the database queries on the current site
- fixing the master-slave replication so it won't fail if the master fails

More on this subject... Later.

Simple loadbalancing with PHP & MySQL

2005-06-08T06:29:00.000-07:00

I am running different sites on a lot of different hardware configurations. When a site becomes too slow you have different options:

buy a new,faster server
buy more servers
fix the software (optimize; better/more cache)

Usually we host sites built on Java/J2EE platform; these sites are made for medium size to (very) big companies and are all built to last and the be scalable. Clustering/failover/loadbalancing are options which are built into these systems by default. The main products my employer sells support clustering out-of-the-box.

Now enter my hobby projects, all of them in PHP and written in a few weekends and nights. One of them got a bit out of hand. When there were 75-100 people concurrent, my single CPU, cheapo hardware took 20 secs to render an average page. The entire site was not cached, it was loaded with features (the most feature-rich dating platform in the world, I dare to say) and written badly.

This site is running on very cheap (< Euro 300) hardware (Celeron 2,4 with 512 mb mem and 2 * 40 gb IDE hds); these boxes are homemade with the cheapest material money can buy. It is hardly possible to do it cheaper than that. Why did I choose this? Because it is actually a free (as in beer) site. So I don't want to spend a lot of money on hardware/hosting/anything.

I first fixed the code to be slightly more efficient and started, at the same time, to build in cache. You must know; datingsite have caching 'challenges', because users are continuesly doing searches and changing stuff while being logged-on. So I had to make cache-per-person for most of the pages. After these changes the site could hold out, on the same hardware, to 400-500 concurrent users.

After a while, the amount of concurrent users per day began to go over 500 and after a while, even over 600. The pages, at night, took 20 sec again to load :(

The site was completely not prepared for loadbalancing, in any way. So the next step I took was seperating the webpart and the database part and put them on different servers. You have to know that the webpart, with all user photos weighs about 4 gigabytes currently (with more than 30.000 members) and the database is 3 gigabytes. I seperated it on 2 servers with the same homemade crappy hardware described above.
This trick was great; the site could handle up to 700-800 users with ease suddenly.

But ofcourse, it was still growing and the 800 concurrent level was made 4 weeks ago. More users made the site slower and slower and slower and.......

The problem seemed to be high traffic on both the webfrontend and the database. The webfrontend could easily be fixed; I would add another box (or even more) and DNS them to be called www1, www2, www3 etc. Then I would have the index on the main domain throw a user to one of those boxes randomly:

$server[]="www1";
$server[]="www2";
$server[]="www3";
$r = rand(0,sizeof($server));
header("Location: ".$server[$r]);

This is the real lame-brain loadbalancing; better loadbalancing would be, ofcourse, using a hardware loadbalancer or a BIND DNS round-robin loadbalancer. But for hardware I don't have the money and for software I would need the domain to be managed by my DNS and it isn't and the provider I have it at doesn't know what loadbalancing is. So I will move, but for now, this works fine.

This helped a bit, but ofcourse overloaded the MySQL database servicing all these frontends...

Balancing MySQL is a bit harder. It is actually much harder. You need to be capable of replicating the MySQL to different servers and this is difficult; MySQL only support master-slave replication which is quite lame; you cannot write to the slaves. Also, you need to put on Logs for this replication which take I/O power and fill up your harddisk space fast.

So I decided to do this differently; I added another box (running 4 boxes now) and rewrote the software by changing all references to mysql_query to _mysql_query. I tested using:

function _mysql_query($qry) {
return mysql_query($qry);
}

When this worked, I filled in this method:

$master_db = make_mysql_connection(MASTER_DB);
$slave_db[] = make_mysql_connection(SLAVE_DB1);

function __mysql_query($qry) {
global $master_db;
global $slave_db;
$cmd = strtolower(substr($qry, 0, 6));
$conn = $master_db;
if ($cmd == "select") {
$choice = rand(0, 10);
if ($choice > 1 && $slave_db[0]) {
$conn = $slave_db[0];
}
} else {
for ($i=0;$i<sizeof($slave_db);$i++) {
if ($slave_db[$i]) {
mysql_query($qry, $slave_db[$i]);
}
}
}
return mysql_query($qry, $conn);
}

This way it will do all writes to all slaves and the master, and it will alternate reads between the master and the slave to some given random %.
Flaws: I will fix this function to round robin between X slaves and I will fix it to fail a slave (after which you have to re-make it with a DB dump from the master. For this last problem I will also figure something out I hope.

The current system is capable of packing a much larger punch and it is actually quite scalable, cheap and very easy to make and setup, even in environments you don't have control over the MySQL and Apache installs.

Aaaargghhh Session mixing in PHP

2005-06-07T01:07:00.000-07:00

I am webmaster of a free dating site for Dutch people. After passing the 30000 members, we started to experience problems with the session handling. People started to receive the same session ids. Ofcourse this is not very nice for a dating site :)
After searching a bit on Google, I found people with similar problems. They 'fixed' it by adding entropy file = /dev/urandom and entropy length = 64. Problem was, I already had this set for my site... So that didn't work.
To fix it, I simply created the following code;

function _session_start() {
$ipad = $_SERVER["REMOTE_ADDR"];
$ipad = str_replace(".", "c", $ipad)."c";
if (!session_id()) {
session_start();
}
if (substr(session_id(), 0, strlen($ipad))!=$ipad) {
session_destroy();
$ipad .= gen_rand(35);
session_id($ipad);
session_start();
}
}

were gen_rand generates 35 random letters and numbers. Replace all your session_start() with _session_start() and all will be fine.

On a datingsite this works fine; no big companies behind one IP address. You can use more eleborate schemes for creating this session id. For instance, adding a second random cookie on the client will actually make the chance of duplicate ids very very small.

Crossplaform document document indexing with Lucene and OpenOffice

2005-05-18T10:48:00.000-07:00

When you have ever worked with Lucene you know that, to index your documents, you have to convert your documents to text. This text is used by Lucene to create the index for the document.
For most formats there are convertors which are either written in Java or native. The problem is that you need to write indexing classes for a lot of different documents; you have to use a lot of different libraries and you have have to make sure your deployment machine contains those libraries...

...or you just install OpenOffice (http://www.openoffice.org) !

OpenOffice is capable of opening most productivity tool formats and even more formats are in the making as we speak. Opening Word, Excel, Powerpoint, Access, dBase, WordPerfect, all StarOffice and OpenOffice formats, Lotus 1-2-3, HTML, XML and much more are no problem for the latest version of OpenOffice.

Now only to get a document converted for indexing with Lucene. This is extremely trivial. I assume you have installed OpenOffice 1.9+ Beta.

Open Writer
Goto Tools -> Macros -> Organize Macros
Click on the Libraries tab
Select 'My Dialogs & Dialogs'
Click on New
Type 'MyLibraries'
Click on 'Modules'
Click on 'MyLibraries'
Click on New
Type 'Conversion'
Select 'Conversion'
Click on Edit
Copy/paste the following code into the editor:

Sub ConvertWordToTxt( cFile )
cURL = ConvertToURL( cFile )

' Open the document.
' Just blindly assume that the document is of a type that OOo will
' correctly recognize and open -- without specifying an import filter.
oDoc = StarDesktop.loadComponentFromURL( cURL, "_blank", 0, Array(_
MakePropertyValue( "Hidden", True ),_
) )

cFile = Left( cFile, Len( cFile ) - 4 ) + ".txt"
cURL = ConvertToURL( cFile )

' Save the document using a filter.
oDoc.storeToURL( cURL, Array(_
MakePropertyValue( "FilterName", "Text" ),_
)

oDoc.close( True )
End Sub
Function MakePropertyValue( Optional cName As String, Optional uValue ) As com.sun.star.beans.PropertyValue
Dim oPropertyValue As New com.sun.star.beans.PropertyValue
If Not IsMissing( cName ) Then
oPropertyValue.Name = cName
EndIf
If Not IsMissing( uValue ) Then
oPropertyValue.Value = uValue
EndIf
MakePropertyValue() = oPropertyValue
End Function

Now hit CRTL-s and close Writer.

You can now run this code like this:

"c:\program files\OpenOffice.org 1.9.79\program\soffice" -invisible "macro:///MyLibraries.Conversion.ConvertWordToPDF(DOCUMENTNAME)"

Substitute DOCUMENTNAME with your document and your document will be converted to .txt.

I will explore using this within Lucene a bit later!

Making your own storage server with cheap hardware

2005-05-17T04:42:00.000-07:00

Yesterday I started working on a cheap storage server. I had the following requirements:

I needed a versioning file system (so every document I store is versioned)
I needed access via Webdav, also for this versioning
It should have easy install
I wanted to use cheap hardware (my 400 mhz system I have gathering dust in my cupboard should do nicely)
I wanted to user friendly tools to support the system (under both Windows and Linux)
It had to be secure (SSL enabled on all levels)
It must be searchable, whatever document I throw in (doc, xls, odt, wpd, txt, xml, html, mp3 etc)

After a bit of searching, it became clear I would use Apache 2 and Subversion under Linux to achieve my goal.

The steps I took to install it all where;

Install Fedora Core 3, server install with all installation options marked off and SELinux off (!)
Get the network verified and working

Import the GPG key:

rpm --import http://www.fedora.us/FEDORA-GPG-KEY

Install Apache: yum install httpd
Install mod_ssl: yum install mod_ssl
Install Subversion: yum install subversion
Install mod_dav_svn: yum install mod_dav_svn

Now your FC installation is ready to go; it already fixed a key for you so the SSL installation is ready.

Make the repository directory:

mkdir /home/svn

chown -R apache.apache /home/svn

Make the directory for your security:

mkdir /home/secvsn

chgrp -R apache /home/secsvn
chmod -R 750 /home/secsvn

You have to change the configuration of Subversion under Apache; it is stored here;

/etc/httpd/conf.d/subversion.conf

Add the following:

<Location /svn>
DAV svn
SVNPath /home/svn
SVNAutoversioning on
AuthType Basic
AuthName "My SVN Repository"
AuthUserFile /home/secsvn/passwd
AuthzSVNAccessFile /home/secsvn/access
Require valid-user
</Location>

Make the password file:

htpasswd -cm /home/secvwn/passwd john

and type a password.

Then make the access file:

cat > /home/secsvn/access
[/]
john = rw

Meaning that john can read and write the entire repository so far. You can add more users if you want ofcourse; read the SVN book to see the syntax.

Start Apache:

/etc/init.d/httpd start

To make sure it starts at the start of the OS:

chkconfig httpd on

Now it should work; make sure you have the firewall open for Apache port 80, if you do not know or have this yet, type;

system-config-securitylevel

and set Apache to the list of allowed services.

From another computer, try;

http://your.ip/svn/

This should ask your for a name and password; this is the aforementioned and created user 'john'.

If you enter the user and it does not work, look in the /var/log/httpd/error.log file for clues. If there is something wrong with permissions, you problably mounted the repository on another drive and have SELinux security on. To fix this, either turn SELinux off or give the repository the right credentials (Google this).

[TODO: howto switch on SSL for this service]

How I made the search I will put somewhere here later, because it appeared my network card was broken and network switched off automatically after 30 minutes of use on the system.

Executing files with Windows native handler

2005-05-16T00:21:00.000-07:00

As I am currently building a cross platform file-manager for our document management tool in Java, I am in need for a way to have files executed by the Windows (and other OSes, but for most clients; Windows) shell as they would be executed when clicking on them in the Windows file explorer.

Doing this in Java is quite easy and a lot of people already figured that out, you simply use the rundll32 command to execute the following:

rundll32 url.dll,FileProtocolHandler URL

where URL can be something on internet or on your local drive. Say you want to 'run' a .txt file on your c: drive:

rundll32 url.dll,FileProtocolHandler file:///mytextfile.txt

opens, on my system with notepad.

The problem with this method is, that if there is no handler, nothing will happen...

After playing around a bit with the registry I found the following;

reg query HKCR\.ext

gives you information about the extension and what Windows will do with it.

In general, as I have discovered (correct me if I am wrong please!), if is in the result, the file can and will be executed by Windows. You can even use it to extract the file type. Example:

C:\>reg query HKCR\.dot

! REG.EXE VERSION 3.0

HKEY_CLASSES_ROOT\.dot
<NO NAME> REG_SZ OpenOffice.org.dot

HKEY_CLASSES_ROOT\.dot\PersistentHandler

So the code for running a file with the default associated application under Windows would be:

String file = "/test.html";
String ext = file.substring(file.lastIndexOf('.'));

Process p = Runtime.getRuntime().exec("reg query HKCR\\" + ext);
InputStream in = p.getInputStream();
p.waitFor();
byte[] buffer = new byte[in.available()];
in.read(buffer);
String result = new String(buffer);
in.close();

if (result.indexOf("<NO NAME>") > -1) {
p = Runtime.getRuntime().exec("rundll32 url.dll,FileProtocolHandler file://"+file);
} else {
System.err.println("No handler available for "+file);
}

There are probably ways to do this faster/easier, but currently we are not in need of this. However, if you know ways to do it, please comment on this post.

Installing Linux on an Acer Ferrari 3400

2005-05-15T15:00:00.000-07:00

I recently got to pick new notebook at my work to program Java on; because I like gadgets and because I have been wanting to test out those nice 64 bit processors for a long time already, I pick the Acer Ferrari. The Ferrari logo is a bit of shame (not enough money to buy the car, but still have a Ferrari is kinda sad). The red colour is nice though, and the system is thin and not too heavy.

Windows 64 bit is too new and not included. And why would I want to use Windows anyway?
Lazy as I am, I did not check if any Linux would run on it. I just assumed it would. After a bit of websearching I decided to try some distributions: Ubuntu 64 bit, Knoppix 64 bit and, last but not least Fedora Core 3 64 bit.

FC3 was the only Linux which installed nicely and detected almost everything. Be sure to start the install with "linux nofb", otherwise you'll see a very black screen during the install. The install will go flawless.

To use the wireless lan which is integrated under a 64 bit system, you'll need the 64 bit Windows driver for this Broadcom chipset. You can pick up the driver here; http://ubuntuforums.org/attachment.php?attachmentid=186
Then you should install ndiswrapper (http://ndiswrapper.sourceforge.net). Getting it to work is trivial;

tar xvzf ndiswrapper.tgz
cd ndiswrapper
make
make install
ndiswrapper -i netbc564.inf
modprobe ndiswrapper
iwlist wlan0 scan

now you should see a list of detected networks; if not, something is wrong.

If your computer hangs after the modprobe command, you probably have more than 1 gb of memory and not the latest version of ndiswrapper. I fixed the ndiswrapper together with the guys maintaining the wrapper (long live Open Source!); it took 1 day to figure out what it was and fix the bug (https://sourceforge.net/tracker/?func=detail&atid=604450&aid=1169978&group_id=93482).

After the wireless station is detected, you should make the network work;

ifconfig wlan0 some_ip up
route add default gw some_gw up

When using WEP or other encryption, some more steps are required (Google for iwlist, iwconfig etc).

I work happily with a 64 bit Linux system now, but I still have one problem; the battery overview is incorrect. I use FC3 with KDE and the default notebook app which shows battery status, but the status is actually wrong. If anyone has a fix, please comment on this post!