Checking/repairing all MySQL tables

Caution: on busy servers this will make a lot of load. Use only when you suspect/know tables are broken.

#!/usr/bin/perl

$mysql = "/var/lib/mysql";

@res = `cd $mysql; find .|grep MYD`;

foreach(@res) {
 chomp;
 /\.\/(.*?)\/(.*?).MYD/;
 $tab = $1.".".$2;
 print `mysql --execute='repair table $tab'`; 
}

A simple, non bloated script for Amazon S3 backups (Linux, easy portable) - II - recurse to subdirs

To follow this, please read this post first.

We needed to backup our images from http://www.picturepush.com to S3 for our premium members so we were searching for a simple script to do that one time. As in my previous post; there is no such thing. Bloated, uninstallable shit is the only thing there is.

So I changed the code a bit and added;

foreach($_SERVER["argv"] as $d) {
 recurse_copy($d, $d);
}
exit;

function recurse_copy($d, $org) {
 foreach(glob("$d/*") as $d1) {
  if ($d1=='..' || $d1=='.') continue;
  if (is_dir($d1)) {
   recurse_copy($d1, $org); 
  } else {
   // put on s3
   global $s3;
   global $bucket;
   $s3->putObjectFile($d1, $bucket, $d1);
   echo "Storing: ".$d1."\n";
  }
 }
}

below the last;

array_shift($_SERVER["argv"]);

A simple, non bloated script for Amazon S3 backups (Linux, easily portable)

While searching for scripts to backup files from Linux -> S3 (servers) I was surprised how difficult it was to find any nice, trim ones. There are huge java jars filled with crap to do it ofcourse;

dbserv01:~# ls -lah js3tream.jar 
-rw-r--r-- 1 root root 3.2M 2007-12-19 15:07 js3tream.jar

Come on. 3.2M... What does that include? Windows Vista?

Sure js3tream is actually nice software. Portable and it works. It's incredibly, mindbogglingly slow (but ofcourse, Java is not slow these days as many people tell me...) on my quad Xeon servers. But yeah, it has the features you would want. Except encryption of the files I upload; you arrange that in a different way please. Come to think of it; other features are missing as well.

There is S3Sync in Ruby which is actually nice and working well, but still too much hassle to get going as simple script. But no complaints about the speed or size of that.

Then there is some rsync thing in Python (I forgot the name and the HELLLLLLLLLLLLLLLLLL I went to to install it will not make me remember it any time soon).

So, as almost all things in life, if you want it done right, you just have to do it yourself. I don't find it pretty at all, but it weighs in, including comments, at 36 kbs, which is, by far the smallest possible script I could find for the purpose I needed.

I'm not counting rar or gpg as they are not included in the other ones either, but when counting them it wouldn't go over 1 mb.

Download http://undesigned.org.za/2007/10/22/amazon-s3-php-class

And install:

- rar (apt-get install rar)
- gpg (apt-get install gnupg)
- php (apt-get install php5-cli php5-curl)

Then edit the Amazon S3.php; put on top of it;

#!/usr/bin/php


if (sizeof($_SERVER["argv"])<4) {
 echo "Usage: ./s3backup bucket ident dir dir dir\n";
 exit;
}

$bucket = $_SERVER["argv"][1];
$ident = $_SERVER["argv"][2];

define('PHPARTIALS_FILE_AWS_S3_ACCESSKEY', '');
define('PHPARTIALS_FILE_AWS_S3_SECRETKEY', '');
define('PHPARTIALS_FILE_RAR_ENCRYPT', '');
define('PHPARTIALS_FILE_GPG_ENCRYPT', '');

$s3 = new S3(PHPARTIALS_FILE_AWS_S3_ACCESSKEY, PHPARTIALS_FILE_AWS_S3_SECRETKEY);

#print_r($s3->getBucket($bucket));exit;

$l = $s3->listBuckets(true);

if (!in_array($bucket, $l)) {
 $s3->putBucket($bucket, S3::ACL_PRIVATE);
}

$fn = $ident."-".strftime('%d%m%y%H%M').".rar";

array_shift($_SERVER["argv"]);
array_shift($_SERVER["argv"]);
array_shift($_SERVER["argv"]);

$f = implode(' ', $_SERVER["argv"]);

$cmd = "rar a -hp".PHPARTIALS_FILE_RAR_ENCRYPT." $fn $f";

echo `$cmd`;

$fn1=$fn.".enc";

$cmd = "gpg --batch --yes --trust-model always --encrypt --recipient '".PHPARTIALS_FILE_GPG_ENCRYPT."' -o $fn1 $fn";

echo `$cmd`;

$s3->putObjectFile($fn, $bucket, baseName($fn1));

print "Backup created!\nSize: ".filesize($fn1)." bytes\n";

unlink($fn);
unlink($fn1);

exit;
?>

Rename the S3.php to something like s3backup and chmod 700 s3backup to make it executable.

You put your 2 keys for Amazon S3 in the Amazon defines, a password for rarring in the RAR password define and the name of the user your public key is for on that system to GPG encrypt the rar.

To add a public key for GPG, just put your GPG Public key in a file, say /root/mypub.key and run;

gpg -a --import /root/mypub.key

And all will be fine.

Filthy Rotten Scriptkiddies - Blackmailing Siteowners

A friend of mine owns a site which has very little traffic and sells a niche product. Because he gets so little traffic, alarm bells immediately sound when suddenly there is a spike.

If there would be a spike, he would be rich probably, but the chances of that happening are very slim.

Anyway, this night, the guy called me up at 3:30 saying he received an email from some hotmail address saying something like ;

I understand how much revinue you bring in each month, I can bring that down to 0 if I wish. Think I'm kidding? Think this is a joke?

And then his demands.

So, me thinking this was indeed a joke, asking why he )(@#%()#@% woke me. But it wasn't a joke. The site was down. Very much so for 1.5 hour already. Costing my friend money.

A bit groggy from sleeping I could not imagine this being anything else than some lame DOS attack; one or a few computers bonking away on :80.

Unfortunately that was not the case at all.

This was a quite (for this kind of lame threat / blackmail) heavy DDOS. After a few minutes I already collected (and blocked) over 200 unique ips (from different classes mostly).

In this blog I have shown more than ones some ideas for catching and blocking DDOS attacks from within Linux and this one was rather a simple one and could be simply blocked using;

List of Linux tricks

Few notes here; Apache (or whatever you might use) queues all those income connections and leaves them connected even though they are blocked. Because of this,
during the attack, I run from the cron; */10 * * * * service httpd restart. Making Apache immediately kill of those bad connections, but finish off the real ones with a real response. At least you'll have service for most people using this method.

Ofcourse it required a bunch of tweaking as the attacker changed his strategy quite often to make it more difficult.

I don't get people who do this and I certainly don't understand how they can mount such a huge attack with so many different IPs.

Edit: attack has been going on for 9 hours now... site doing fine.

MacOSX / Bootcamp / Installing Windows XP without SPs / there is not enough disk space on $ntservicepackuninstall$ to install service pack bla

Sometimes you have to do things you don't want. Today I needed to install something under Windows to convert some fileformat (media file format) for which I cannot find Linux or Mac converters. Not even Mplayer can play it. Only some weird proprietary Windows app.

So basically; I needed Windows. And I hate Windows. And not without passion; I really hate it. But ok, enough about that.

I have a Mac Mini standing on my desk gathering dust (oh did I mention I hate Mac OS X too?), so I thought ; I can install on that.

Not that easy. Requires an XP disk with SP2. And ofcourse I didn't have that. As I really don't want to pay for stuff I already have, I took an XP version from 2001 (so without any SPs) and tried to find a way to install that on the Mac Mini. Legal as I don't even have the computer anymore on which is belonged, but I have the 'authentic Windows crap sticker' still.

The installation from within Bootcamp went 'smooth', although Windows doesn't want to shutdown or restart, you have to force it down by holding the shutdown button.

And almost none of the software/drivers Apple provides work without SP2. Even worse (well... not much worse); Mac OS X doesn't boot anymore.

Anyway; I got the SP2 download and burnt it on a CD under Linux and tried to install it under the non-SP version on the Mac Mini. Unfortunately, it started yelping about some missing 4 mb's of space....... 4 mbs. Ofcourse I have more than 20 gb free so that could not be right.

Quite annoyingly, the Mac community is not of the technical details; when searching for these kind of problems, people keep repeating in forums that I insulted my Mac by installing Windows OR that I did not read and MUST HAVE SP2 Windows install CD OR that I should just reinstall everything. All very helpful. Not.

So I was faced with two problems;

- no more Mac OS X
- Windows couldn't install the SP so I couldn't install any drivers etc

The former was not that important for me as I don't like Mac OS X that much and certainly never use it if I can avoid it.

But for the latter I started this whole adventure, so I couldn't give that up.

I read all messages about this problem and the solution turned out to be very (very) simple;

Windows-R (Start->Run)
regedit

Add a String

key = BootDir
value = C:\

To

[HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Setup]

And rerun the SP2 installer.

Why didn't they just SAY so?

Anyway; if/when I have Mac OS X back I'll report it here and people can use nice old CD's of XP they find in car boot sales, instead of paying money to that company we all love.

Edit: Well, that was ofcourse kind of simple;

- remove the read only flag from c:\boot.ini
- edit boot.ini
- put at the end c:\CHAIN0="Mac OS X"
- save
- put this file in you c:\
- reboot

Not Boot Camp, but working fine.

Creating databases in cpanel remotely

Lately I was wondering how one would go about creating databases in cpanel without having to log on. Do it from PHP or something and create a somewhat 'nicer' interface to that 'nice' cpanel stuff.

It proved to be trivial to do anything with the system; that is in face kind of nice and has been a relief for me as I thought it would be (much) more difficult.

Here is a code snippet how to do this easily. It return PHPartialsResult, but you can replace that with 'true' or something.

  function remote_account_create_db($host, $user, $pass, $dbname) {
   $conn = "http://$user:$pass@$host:2082/frontend/x3/sql/addb.html?db=$dbname";
   file_get_contents($conn); 
   return PHPartialsResult::ok();
  }

  function remote_account_create_dbuser($host, $user, $pass, $_user, $_pass) {
   $conn = "http://$user:$pass@$host:2082/frontend/x3/sql/adduser.html?user=$_user&pass=$pass&pass2=$pass";
   file_get_contents($conn); 
   return PHPartialsResult::ok(); 
  }

  function remote_account_create_dbuserconnection($host, $user, $pass, $_user, $_db) {
   $conn = "http://$user:$pass@$host:2082/frontend/x3/sql/addusertodb.html?ALL=ALL&user=$_user&db=$_db";
   file_get_contents($conn); 
   return PHPartialsResult::ok(); 
  }

Using tcpproxy for http forwarding? Add X-Forwarded-For!

TCPProxy is one of the most beautiful tools I know of; it was nicely programmed, but above all; it has no bugs. None. We are routing millions of pageviews a day of hundreds of thousands of unique users through it and it has never let me down. It is fast, uses almost no CPU and is very very robust (I miss FTP support, but further it is totally perfect).

This unlike Apache (proxy pass) or Squid or the many other, less known, proxies I tried. Apache is especially crappy; proxy pass is riddled with bugs; under load it simply is unusable. And most of them are feature-heavy; too feature heavy; they are a pain to set up, but trivial things like not binding to 0.0.0.0 etc or simple config files they have not.

We have the necessity to route web traffic via different machines in different places and ofcourse I wanted to use tcpproxy for the task, but it has no x-forwarded-for and so we were pissing our users (with forums, blogs etc) off. So I decided to hack tcpproxy for this. And it works. Serving the millions of pages through it. With x-forwarded-for.

This was tested on 1.1.9;

In tcpproxy.c, search for these lines;

int proxy_request(config_t *x, int cfd)
{
int     sfd, rc, bytes;

and change to:

int     sfd, rc, bytes, first=1, i;
char    *hp, *hp1;
char    tmp[4096];

then search for;

  else if (write(sfd, buffer, bytes) != bytes) {
                                syslog(LOG_NOTICE, "client write() error");

and change that to;

else {
if (first && (
 (buffer[0] == 'G' && buffer[1] == 'E' && buffer[2] == 'T') ||
 (buffer[0] == 'P' && buffer[1] == 'O' && buffer[2] == 'S' && buffer[3] == 'T') ||
 (buffer[0] == 'P' && buffer[1] == 'U' && buffer[2] == 'T')
 )) {

 hp = &buffer;
 *(hp+bytes)='\0';
 hp = strstr(buffer, "\r\n\r\n");
 if (hp) {
  hp1 = &buffer;
  i = abs(hp1-hp);
  *hp = '\0';
  memcpy(tmp, buffer, bytes+1);
  strupr(buffer);
  if (!strstr(buffer, "X-FORWARDED-FOR")) {
   sprintf(buffer, "%s\r\nX-Forwarded-For: %s\r", tmp, x->client_ip);
   hp = &tmp;
   hp += i + 1;
   i = bytes - i - 1;
   bytes = i + strlen(buffer);
   hp1 = &buffer;
   hp1 += strlen(buffer);
   memcpy(hp1, hp, i);
  }
 }
}
first = 0;

if (write(sfd, buffer, bytes) != bytes) {
                                syslog(LOG_NOTICE, "client write() error");
 
           break;
                                }

}

Compile it and it'll add X-Forwarded-For.