And I always like simple. But of course, it’s anything but.

The problem soon arises that there’s er, a bit of a lack of documentation how to actually get a CSV file into a SQLite database. So, cutting a long story short, here’s how I did it:

1. Make sure that the source file is stripped of quotation marks. I found the quickest way to do this on a larger file was to use the tiny freeware DOS utility Binsub with the following syntax:
   
binsub my.csv /034

   … where 034 is the decimal ASCII code for a double quote.
2. Use the utility dgSQLite to create a new database, and subsequently import your CSV into a table of your choice. This utility does most of the grunt work for you, and I think requires TCL/Tk for Windows to run. I had previously tried and failed to get things moving with the slightly more user-friendly SQLite Browser, however that only worked with SQLite 2.x files. (And somehow along the way I decided I was using 3.x.)

Lastly, to see if it worked, you can try the following snippet in your PHP5 installation:


$db = new SQLiteDatabase("path/to/your/database.sqlite");
$result = $db->query('SELECT * FROM your_table_name ');
while ($result->valid()) {
$result = $result->current();
print_r($result);
$result->next();
}


Note: This script has been updated somewhat now, but the basic story here remains the same.

Anyway, this got me thinking, and cutting a long story short it occurred to me that I already had a great blacklist of spam domains supplied by Jay Allen‘s MT-Blacklist, and it shouldn’t be so hard to use this list as a basis to remove the referer spam from the Shortstat database – and so, here’s the results of about 13 minutes of investigation:


include_once("configuration.php");
include_once("functions.php");
if ($shortstat) {
SI_pconnect();
$urlpatterns = mysql_query("SELECT ext_bl_item_text FROM mt_ext_bl_item");
while ($row = mysql_fetch_array($urlpatterns, MYSQL_NUM)) {
$query = "DELETE FROM si_shortstat WHERE domain LIKE \"%$row[0]\"";
@mysql_query($query);
}
}


I’ve called this “_despam.php” and installed in the Shortstat installation directory it will use your existing database connection settings. Note – the script makes the assumption that MT-Blacklist has been set up to use the same MySQL database that Shortstat uses, but that said, I imagine that’s most setups.

It certainly could do with a few more features (actually reporting back what it’s done/doing would be a start), but the basic functionality is there and lovely shiny clean reports are the result.

Update: Tony at juju.org has taken things a step further with a Perl script that ‘de-spams’ your server log files using MT-Blacklist too.