Preamble

Let’s get some terminology and other stuff out of the way.

Spam: unsolicited email.

Bayesian: reaching a conclusion using some uncertain elements.  The wikipedia entry explains it.

Bayesian classification systems: A strict way of describing programs like SpamBayes or SpamAssassin.

Title or title header: This is the part of the email that is displayed as the title of the email.

Body: This is the part of the email where most of the text of the email is kept.

Why I chose to work on this already-solved problem. As an email server admin, the problem isn’t ’solved.’    Bayesian systems are very good. (with some limitations) Focusing on the limitations stands the best chance of making spam filtering much better.  The notion of “standing on the shoulder of giants” applies.

What are you doing differently? First, I’m using only analyzing the title of the email.  Introducing body analysis is recreating the work of Bayesian filters.  Second, I’m using a less well known algorithm and  applying it to the spam problem.  Lastly, titles with variations on words can be reliably detected. ‘V1agra’ and ‘Viagr@’ and ‘Viagra’ are detected as similar.  The resulting corpus isn’t particularly large.

Reductionist dismisal of the work:

• This is a naive method, and therefore it’s non-special.  I’d argue it’s a moot point. It’s less than 100% naive for a couple of reasons, all of which can easily be shouted down by someone with a higher socio-economic rank and sufficient buzz-word use.
• I’m unqualified.   My socio-economic standing does not exceed the title researcher, programmer or scholar.  Nor is the effort mathematically spectacular.   Another person with sufficient socio-economic standing will probably implement it and garner far more attention.
• This article isn’t sufficiently filled with cryptic vocabulary, therefore the process isn’t special. It’s unfortunate that vocabulary is used to establish social rank and therefore exclude. I chose to minimize the buzz-wordiness. The next person pursuing this method won’t advance the concepts and use far more cryptic language. They will however ill garner more attention using specialized vocabulary.

The Proposal: Detect Spam Analyzing the Email Title

Why choose to analyze only the title?

1. Bayesian systems already analyze the email body.
2. Bayesian systems tend to fail to detect spam with brief titles and similarly brief bodies.
3. I want to be able to scale the solution well beyond Bayesian systems.  As some email admins already know, Bayesian email filters are extremely resource intensive when you  are hosting a large number of email accounts.

The Algorithm: Rabin-Karp

Why Rabin-Karp?

1. Because it seems to be suited to the job of  finding strings while exhibiting the potential to scale way beyond Bayesian classification systems.
2. It is already used as a plagarism detector.
3. Other string searching algorithms have problems when applied to spam detection.    Most string searching algo’s are built on the assumption of discreet words.  A spammer can break the filter by confounding the definition of a ‘word.’

Build a Corpus

Step 1: I grabbed 17 spam titles and stuck them in a text file. When building the corpus, I keep all characters between A and Z and force everything into lower case.  The program then loops through possible window values and generates hash values.  I send the results into a SQLite database.  There are *much* faster ways to store the data, but this worked for me in the research phase.

Some notes about the window used to compute the hashes.  The window range I used was a minimum of 3 and a maximum of 10 characters.  What one would do with the variability is an open question.  If one uses too big a window to detect spam, then the filter will fail to detect spam.  If one uses too small a window, this may be too resource intensive.  I would think that randomly assigning the window would make it harder to break the filter.  Maybe it’s a moot point and a small window should be the standard.

Test the Classification System

To test the accuracy of classifying emails, I do a simple SQL lookup on windowed (see wikipedia reference) hashes of a title string to see if the hash exists in the database.  If it does, it gets 3 points.  The process for each email title goes something like this:

• If the sum of points squared is less than 120 then the program keeps checking the title string for more hashes.
• If the current sum of points squared is greater than 120, then I assume it is spam and move to the next email.   The high score results because the title is very close to what’s in the corpus.  I don’t have to search the whole subject string if the score breaks 120 sooner than the end of the string.

Test Results

title	Score	Corpus Info
oysterperpetual cosmographdaytona	256	in corpus
Fwd: Cookie Booth this Sun?	0	Not spam, not in corpus
Casino’s_Best PlAyer is Welcome!!	64	Gold Best Casino : Usa Player welcome!!!!
I’m tired of viagra ads	144	Not in corpus, a title from an exasperated friend
sale visit our website today and buy replica items cheaper	9	Not in corpus
@derall and V1codin online	64	_Percocet__Adderall_Cialis _Viagra_Ritalin!!!

Next Step

I need a bigger body of email titles to test and add to the corpus.   If you would like to provide some, then please contact me.  The title header just needs to be in a flat file, one per line, no opening/closing quotes needed.

Another step to sanitize the subject data is to strip “the”, most prepositions, and “of the”, “in the”, “to the”, “on the”, “for the”, “and the”, “that the”, “at the”, “to be” and “in a” out of the subject string.  I don’t know what effect this would have, but it is all noise. ]]> http://www.michaelpapet.com/blog/title-only-spam-detection-research-1/feed 0 http://www.michaelpapet.com/blog/awstats-and-iis-debian-set-up http://www.michaelpapet.com/blog/awstats-and-iis-debian-set-up#comments Fri, 29 Jan 2010 05:28:14 +0000 White Goodman http://www.michaelpapet.com/blog/?p=207 This following should take the reader through a complete awstats setup from start to finish.  The reader should be cognizant of the possible impacts of some of the commands casually mentioned below.  This was done on Debian’s Lenny distro with apache2.  The reader should be very familiar with setting up an Apache2 instance.

1. Apt-get awstats.
2. Configure Apache2:  Use   /usr/share/doc/awstats/examples/apache.conf  as your template for whatever you end up with in /etc/apache2/sites-available/awstats.  Apache configs are subject to personal preferences too much to be more specific than this.
3. Link /etc/apache2/sites-available/awstats to /etc/apache2/sites/enabled/awstats. ‘ln -s /etc/apache2/sites-available/awstats /etc/apache2/sites-enabled/awstats’
4. Restart apache2 ‘/etc/init.d/apache2 restart’

IIS Log Set Up Minutae

Before you can make awstats useful, your logs from your IIS instance have to meet some minimum requirements.  I don’t remember the IIS logging defaults, but given Microsoft’s hostility to Free software, it’s likely a plain-vanilla IIS log will not work with awstats.

What’s especially miserable about IIS is setting the ‘host header value’ on Win2003.  Somehow this value ended up being null for me and any attempts to change it end up with a 404 error.  I have a hard time maintaining the value of Microsoft products with nasty bugs like this….

The following fields should be used in the log: Client IP Address, Service Name, URI Stem, URI Query, Protocol Status, Bytes Sent, User Agent, Referer.  Of those, Bytes Sent is required for awstats to work!!! There are other fields that might be useful in special situations like %logname, but that’s for you to poke through the default awstats.conf file and figure out based on your needs.

Now that your web server is ready, set up awstats for IIS logs.  My logs come from a Windows 2003 machine.

Awstats.conf  Set Up

1. Check a log.  You need to extract the hostname (s-sitename) from the log.  You will need this name and it will be referred to as $s-sitename from now on.
2. Copy an awstats.conf file to awstats.$s-sitename.conf where $s-sitename.
3. Open the newly copied awstats.$s-sitename.conf file.
4. Set the path of where the log files reside.
5. Make your log string.  It will probably be a custom incantation.
6. Set LogFile.  If your IIS logs are rotated, then you *might* be able to use logresolvemerge.pl to feed awstats logs.  logresolvemerge.pl does the work of feeding logs to awstats in the required time-sensitive manner.  Otherwise, you may need to work out feeding awstats logs oldest data first.
7. Change the SiteDomain value to whatever $s-sitename is.
8. Add HostAliases as needed.
9. Save the file and exit.
10. Change directory to your web root for your awstats page.
11. Run “/usr/lib/cgi-bin/awstats.pl -config=$s-sitename  -showcorrupted” Where $s-sitename is whatever your hostname may be in the log file.  This incantation will tell you if there are problems with the logs while processing .  IIS log incantations are fiddley, so there will probably be several tries before your log string (step 5) is okay and awstats works as expected.
12. Run “/usr/lib/cgi-bin/awstats.pl  -config=$s-sitename  -output > $your/path/to/web/root/index.html”  Where $your/path/to/web/root is the same as your apache config web root.  You’ll have basic information now. You should not need the -static option so frequently mentioned elsewhere if your Apache2 config is set up like the example given.  Optionally, you can set LogFile=”/some/custom/path” to feed awstats in a customized manner.

For the rest of the reports, I use the awstats_updateall.pl script located in /usr/share/doc/awstats/examples.  I recall it needs a copy of awstats.pl in the same directory for it to work.

Essential documentation for awstats is at their Sourceforge site.

Leave questions as comments.

Another reason why the telco’s government-granted Monopoly deserves to be modified to allow for more competition.

Picking up the phone does not qualify as service.

1. The server is off most of the time.  We just don’t use it very often and it doesn’t make sense to keep a dual-core PC running for no reason.
2. Manually update the tv schedule on my MythTV box. The attached script checks the date on a file and compares it to the current day.

mythtv manual schedule update

Attached script needs zap2xml to write the xml file(s).  Be sure to check your tuner id’s in the database and a couple of other adjustments will probably be required to have the script work for you.  But the fiddly bits about dates and MythTV commands are there.

It should be set up to run on login for your mythtv-backend session, and then perhaps as a cron job.

Questions? Leave them in the comments.

mythtv manual schedule update

Good news! I found the problem. It had nothing to do with Popfile per se, but my setup. Imagine that!

Looking through the word lists, there were tons of misspelled words in my Ham bucket.  Seeing another nail that needed a hammer, I wrote a script that moves misspelled words into the spam bucket.

If it helps anyone else, I’ve attached the script. here

This script requires an ODBC DSN to the SQLite database.  (That’s a mouthful!)  It will definitely work in Linux.  Other platforms?  Dunno.

Feedback is welcome in the comments.

On a side note, popfile in Debian’s repos is ancient.  Get it from popfile’s site. (version 1.1.1 or later!) Also worth noting is popfile’s xmlrpc interface.  Nice!

Before we begin, some good news: ANY PC will run Debian nicely. All the way from 500mhz CPU’s and 1.1GHZ machines all the way to recent quad-core machines. The key is to have about 1GB or more of RAM available. An old 500mhz desktops make great home file/print/scanner/backups/web servers. If you wish to run MythTV and record HD content, you will need at least a dual-core CPU.

Read on for the how-to!

One Step Back:

Debian doesn’t really do a great job at making it obvious which disk image you need. The vast majority of you need an image with i386 in the name. You want this one because you are changing from Microsoft Windows to Linux. The next group would be the users converting an older Mac with a PowerPC CPU. You want the file with PPC in the name.

Being a KDE user, Debian builds a single-cd installer that is just KDE. This makes things super-easy! Just look for the file with the CPU type and KDE in the name. For the vast majority of you that would be something like, debian-testing-i386-kde-CD.iso

There is more than one way to get onto the Debian Testing distro. I’m going to concentrate on downloading a Testing image and install.

Two Steps Forward:

1. Go to the weekly builds page. PowerPC user’s go here.
2a. If your CPU has a speed claim greater than 1 GHZ, then scroll all the way down and grab the debian-testing-CPU-TYPE-kde-CD-1.iso file.
2b. For older PC’s (~1 GHZ or slower), grab the other image at the bottom of the page, debian-testing-CPU-TYPE-xfce+lxde-CD-1.iso. XFCE4 as your window manager will keep things as fast as possible and it has tons of functionality.  At this time, LXDE is lighter still, but buggier than XFCE4.  So, choose XFCE4 from the session choices when logging in for the first time.

3. Wait while the image downloads.
4. After downloading, burn the CD image to a CD.
5. Boot the PC with the Debian Installer!

There are lots of how-to’s about installing Debian from this point forward, so use the Internet to get any other questions answered.

Please note: The script is dual licensed.

-If you are using it for your non-revenue producing Drupal site, then the script is distributed under the terms of the GNU GPL v3.0.

-If you are using it in a revenue-producing site, then please leave your email address in the comments to arrange Paypal payment of $20.  If you can’t come up with this script on your own then $20 is a bargain. I’m available for modifications to make it work better for your environment on a small-time contract basis.

This script is provided AS IS.  There are NO warranty or usefulness claims.

The only thing you need to do is to change some variables on the top of the script and set up a cron job to run it.

There are some consequences to using this script for which you need to be aware.

1. Pay attention to permissions/privileges on the path for dumping the database.
2. It makes database backups named YEARMONTHDAY.dump to the path of your choice. Delete old backups with cron. Test the backups.
3. This script shuts down your site temporarily when it re-indexes. MySQL has to have full lock on the database. The script then turns your site back on the moment it is done.
4. There’s some cruft-cleaning that goes on with this script.  I’m not sure how well it works for others, but it works for me.  You were warned!

Assuming you have read and understand the hazards of what was just written here is the script: drupal-db-maint.txt

Questions, comments can be submitted as comments.

1. Add Sid repo’s to your sources.list
2 create a /etc/apt/preferences file and pin testing at a higher priority than Sid.
3. Apt-get -t sid install kde-minimal
4. logout of your display manager. You should have the option to restart the display system (X).

After restarting X, you should be good to go.

To enable all of the desktop effects and eye candy:
You need to modify your xorg.conf to enable compositing. This is not rocket science, but depends entirely on your hardware, reading the man page (ex. man nv) and finding some examples on the Internet. All of which are readily available.

First, enable compositing by opening System Settings->Desktop.
You should see the enable/disable compositing button there. Enable it!
If the desktop complains, then go to the Advanced tab and click on the ‘Disable functionality test’ option.
You can Enable all Desktop Effects too.
Switch to the Advanced tab.
Compositing type: OpenGL

You should be good to go!

This might work.

1. Go to the xmlsoft.org website and get libxml$lastest_version$.tar.gz.
2. Extract it into c:\strawberry-perl If you are very knowledgeable, this isn’t necessary, but it makes the paths that follow work.
3. Open a console and cd into c:\strawberry-perl\libxml$yourVersion$\win32.
4. cscript configure.js compiler=mingw incdir=C:\strawberry-perl\mingw\include libdir=C:\strawberry-perl\mingw\lib iconv=no
include=C:\strawberry-perl\mingw\include lib=C:\strawberry-perl\mingw\lib prefix=C:\strawberry-perl\mingw\
5. enter c:\strawberry-perl\mingw\bin\mingw32-make.exe on the command line
6. I had make fail. Dowwwh! The fail had something to do with the declaration of uint32_t. I found it on line 29, commented that line out and added this just below it. typedef unsigned __int32
7. c:\strawberry-perl\mingw\bin\mingw32-make.exe install on the command line.
8. Now, if I’m lucky, XML::LibXML will work!

Typical form mail garbage that has nothing to do with the problem from Barracuda Networks. My replies are in bold. Thank you for contacting Barracuda Networks regarding your issue. Your issue is important to us. We have assigned a confirmation number: **** to this case.

We apologize for any inconvenience that this may have caused you. Since this is is your first request for this IP, the reputation of this IP address will be temporarily upgraded from “poor” for 48 hours or until we complete our investigation. When our investigation is complete, you will receive a decision via email. It may take up to 1 hour for the changes in the Barracuda Reputation System to propagate to all the Barracuda Spam Firewalls in the world.

I would be shocked to find out the investigation resources are hopelessly under-resourced. Just shocked!

There are a number of reasons your IP address may have been listed as “poor”, including:

1. The email server at this IP address contains a virus and has been sending out spam Nope
2. The email server at this IP address may be configured incorrectly Nope
3. The PC at this IP address may be infected with a virus or botnet software program Nope
4. An individual in the organization at this IP address may have a PC infected with a virus or botnet program Nope
5. This IP address may be a dynamic IP address which was previously utilized by a known spammer Nope
6. The marketing department of a company at this IP address may be sending out bulk emails that do not comply with the CAN-SPAM Act Nope
7. This IP address may have a insecure wireless network attached to it which could allow unknown users to use it’s network connection to send out bulk email Nope
8. In some rare cases, your recipients’ Barracuda Spam Firewall may be misconfigured Rare cases eh? Well, Nope.

If you do not think any of the above apply, please also contact the person who manages this IP address, as they may be better able to investigate this issue. That would be me. I’ll open a trouble-ticket to myself. Nope.