Michael Papet

Ice3.3 broke my apache/php5 config and I didn’t know it.

Perhaps the most elegant explanation of the dynamic forces in American Political culture.  The Ratchet Effect

I would hope, over time, that people with moderate political views (most Democrats and Republicans) come to the realization that the Ratchet Effect  is the source of their unhappiness with government.

The varied interests that make up the political Right applies as much leverage as possible to society through legislation and other political/legal activity.  The Rachet Effect  predicts that there will be more extreme candidates and legal challenges coming from the Republican party only because a compromise (the crux of politics)  is a ‘win.’

Just like the Democrats, the Republican voters (whomever still bothers to go) get to pick from some ugly choices and get no representation for their vote.

The next step is more difficult regardless of your political affiliation.  What kind of social/political organization breaks the cycle described by the Ratchet Effect?

Feinstein’s record on votes (there were a few) regarding the “Safe, Accountable, Fair, and Efficient Banking Act of 2010″  is startling.

• No on limiting leverage.
• No on the FDIC ‘consumer financial protection’ department.  This was weak, but better than nothing.

I didn’t dig around long enough to find the other parts of the bill voted on, but the whole thing is dead and Feinstein gets some credit.

There was another critical vote yesterday that might have set the stage for a GAO audit of the Federal Reserve.  She voted No on that too.

It’s no longer hard to wonder why you’ve got such dismally low trust in the U.S. Government.  Few Representatives are legislating the interests of the majority of Americans.

Barbara Boxer did The Right Thing and has voted for all of the items mentioned above.

I’ve been doing a bit of work on an adjunct to Bayesian spam filtering systems.  The basic idea is to  do some follow-on analysis of email messages Bayesian systems categorize as uncertains. I implement a hashing scheme, build a corpus, then check mail Bayesian filters don’t catch against the corpus.

Preamble

Let’s get some terminology and other stuff out of the way.

Spam: unsolicited email.

Bayesian: reaching a conclusion using some uncertain elements.  The wikipedia entry explains it.

Bayesian classification systems: A strict way of describing programs like SpamBayes or SpamAssassin.

Title or title header: This is the part of the email that is displayed as the title of the email.

Body: This is the part of the email where most of the text of the email is kept.

Why I chose to work on this already-solved problem. As an email server admin, the problem isn’t ’solved.’    Bayesian systems are very good. (with some limitations) Focusing on the limitations stands the best chance of making spam filtering much better.  The notion of “standing on the shoulder of giants” applies.

What are you doing differently? First, I’m using only analyzing the title of the email.  Introducing body analysis is recreating the work of Bayesian filters.  Second, I’m using a less well known algorithm and  applying it to the spam problem.  Lastly, titles with variations on words can be reliably detected. ‘V1agra’ and ‘Viagr@’ and ‘Viagra’ are detected as similar.  The resulting corpus isn’t particularly large.

Reductionist dismisal of the work:

• This is a naive method, and therefore it’s non-special.  I’d argue it’s a moot point. It’s less than 100% naive for a couple of reasons, all of which can easily be shouted down by someone with a higher socio-economic rank and sufficient buzz-word use.
• I’m unqualified.   My socio-economic standing does not exceed the title researcher, programmer or scholar.  Nor is the effort mathematically spectacular.   Another person with sufficient socio-economic standing will probably implement it and garner far more attention.
• This article isn’t sufficiently filled with cryptic vocabulary, therefore the process isn’t special. It’s unfortunate that vocabulary is used to establish social rank and therefore exclude. I chose to minimize the buzz-wordiness. The next person pursuing this method won’t advance the concepts and use far more cryptic language. They will however ill garner more attention using specialized vocabulary.

The Proposal: Detect Spam Analyzing the Email Title

Why choose to analyze only the title?

1. Bayesian systems already analyze the email body.
2. Bayesian systems tend to fail to detect spam with brief titles and similarly brief bodies.
3. I want to be able to scale the solution well beyond Bayesian systems.  As some email admins already know, Bayesian email filters are extremely resource intensive when you  are hosting a large number of email accounts.

The Algorithm: Rabin-Karp

Why Rabin-Karp?

1. Because it seems to be suited to the job of  finding strings while exhibiting the potential to scale way beyond Bayesian classification systems.
2. It is already used as a plagarism detector.
3. Other string searching algorithms have problems when applied to spam detection.    Most string searching algo’s are built on the assumption of discreet words.  A spammer can break the filter by confounding the definition of a ‘word.’

Build a Corpus

Step 1: I grabbed 17 spam titles and stuck them in a text file. When building the corpus, I keep all characters between A and Z and force everything into lower case.  The program then loops through possible window values and generates hash values.  I send the results into a SQLite database.  There are *much* faster ways to store the data, but this worked for me in the research phase.

Some notes about the window used to compute the hashes.  The window range I used was a minimum of 3 and a maximum of 10 characters.  What one would do with the variability is an open question.  If one uses too big a window to detect spam, then the filter will fail to detect spam.  If one uses too small a window, this may be too resource intensive.  I would think that randomly assigning the window would make it harder to break the filter.  Maybe it’s a moot point and a small window should be the standard.

Test the Classification System

To test the accuracy of classifying emails, I do a simple SQL lookup on windowed (see wikipedia reference) hashes of a title string to see if the hash exists in the database.  If it does, it gets 3 points.  The process for each email title goes something like this:

• If the sum of points squared is less than 120 then the program keeps checking the title string for more hashes.
• If the current sum of points squared is greater than 120, then I assume it is spam and move to the next email.   The high score results because the title is very close to what’s in the corpus.  I don’t have to search the whole subject string if the score breaks 120 sooner than the end of the string.

Test Results

Next Step

I need a bigger body of email titles to test and add to the corpus.   If you would like to provide some, then please contact me.  The title header just needs to be in a flat file, one per line, no opening/closing quotes needed.

Another step to sanitize the subject data is to strip “the”, most prepositions, and “of the”, “in the”, “to the”, “on the”, “for the”, “and the”, “that the”, “at the”, “to be” and “in a” out of the subject string.  I don’t know what effect this would have, but it is all noise.

It looks like the Republican party is fracturing.  A bunch of those people got together and are making every effort to differentiate themselves from the spendthrift GWB #43 era ‘conservatives’.  I’m excited to see some more ‘cat herding’ in Conservative politics. In this way, they are becoming very much like the Democrats.

They have a new strategically elegant but tactically unworkable document they are calling the “Mount Vernon Statement.”

I’ll ignore all of the messy historical baggage just like the average “Morning in America” conservative and say that I think much of what they write works for many Americans.  I’m easily categorized as a Liberal and yet I agree with many of the principles in the rhetoric.  That is, until they turn the rhetoric into tactics and policy.

Trojan Horse Part 1

• It would be nice if they actually meant they want encourage Religious Freedom.  Tactically, their version of Freedom is called Christianity and the rest of you are going to Hell. (See “nature’s God”)
• It would be nice if ‘ordered liberty’ was meant to encourage more social and political freedom.  Tactically, they want Government to alter the lives of the people they don’t agree with.
• Government is the new Self.  It’s clear they want to alter the lives of the people who do not live according to their rules.  See “moral self-government”
• The military and Conservative pet projects win big too.  They get big budgets to spend on projects wrapped in the perfectly circular argument they will be ‘responsible.’

The bomb drop is buried near the end.  “economic reforms grounded in market solutions.”

Trojan Horse, Part 2

Let’s review some facts regarding capitalism and how the vaunted ‘market solutions’ accelerate economic disparity and lead to a weaker economy.  This ‘market solutions’ newspeak is not exclusive to the Republicans.  The Dems are using the same kind of  ‘market driven’ newspeak and they are as much to blame for the last 20 years of economic disparity.

Question: What is the rough definition of  an economic market?

Answer: A place potential Buyers and Sellers meet to possibly exchange goods and services.

Examples of markets are a car dealership, the grocery store, ebay.com.

Question: Are markets all-inclusive?

Answer: Never.  There may be lots of willing suppliers for eggs at greater than $25/dozen, but there are very few buyers.  This perfectly satisfies the political argument of ‘market solutions’ and more specifically meets the definition of a market.  Mission Accomplished!!! (GWB ironic reference)

Examples of inclusive markets are Ferrari automobiles, private American Medical Insurance policies, your grocery store with the homeless person in back digging through the trash.   Each market example has customers and the rest just stand idle.

There are at least two consequences of  ‘market solutions’ groupthink.

1. Increase economic disparity.  10+ years of unfettered capitalism in the U.S. has shown (again) that it ends up making more people poorer and few people richer.  Diminished economic activity follows.  Fewer people have disposable income as a result of increasing economic disparity.
2. Political and social instability.  People are falling out of the economy (> 10% unemployment, closer to 17%.) and probably won’t have the same kind of wages when they eventually return to work.  This creates a great deal of social and political instability.

The economic consequences of the Mount Vernon Statement are clear, it will  increase the ranks of the working poor, decrease social and political stability all of which decreases economic activity.

For those wondering why I don’t pound the Democrats as mercilessly, the Democratic Party has always been equivalent to herding cats.

In the U.S.,  people get very uncomfortable discussing social class.  Any discussions of social class locality (“I’m in a lower social class than Mike Huckabee)  is on the good graces blacklist as a thoughtcrime.  One of the reasons it’s a thoughtcrime is because some of the circle of people that inform our political leaders use it as a weapon to increase income and political inequality.

If the unwashed masses are discouraged from discussing social class relativity/locality then the economic leaders have more freedom to create greater social and economic divisions.  There’s a perverse logic at work that somehow makes a few very rich, a tiny unstable middle class, and the vast majority with the least resources a Good Thing.  The longer discussions of social class are kept out of the scrum of daily life, the more time there is for increasing political and social inequality.

In America 2010, everyone is Middle Class.  Howard Stringer, Jamie Dimon, Frank Atlee, they are all Middle Class.  Only they are more Middle Class (and right) than the rest of the primitive populists.  The actions of these Middle Class Warriors  since the 1980’s has resulted in increased political and economic inequality.  Some obvious artifacts of this is the declining distribution of wealth.  Another artifact is the stagnant wage growth for the bottom 90%.

Another example, in this post.  Towards the end of the piece on banking regulation, the author labels the ideological assault as Serfdom circa 2010. What’s nice about the article is that someone else at least recognizes that income inequality is an explicit goal .   But then it tries to add some heat to the discussion by abusing the term Serfdom.  I don’t agree with the use of the vague word Serfdom.   Otherwise, the author describes the mechanics of failure at re-regulating the current form of Banking very elegantly.

Why does a country need a large middle class?  Greater political and social stability on which to build a vibrant economy.   The simple fact of the matter is creating a vibrant middle class require redistributing wealth. Uh oh.  Another American thoughtcrime.  I’m actively encouraging redistributing wealth.  Your average deregulator demagogue will now chime in with some faulty logic wrapped in Economics lingo “Redistributing income destroys wealth!”  or the hilarious “deregulated markets are more efficient!”   The general consequences of believing some version of the previous two arguments is greater economic and political disparity.

To specifically refute the wealth destroying argument,  empirical evidence like the banking interests mentioned in the article above repeatedly shows wealth being destroyed any number of ways.  I’d also argue most mature segments of the American economy are Oligopolies continuously destroying wealth anyway.

To specifically refute the deregulated marketeer, the theoretical model is elegant, but humans just don’t work like that and no amount of deregulation gets to the mythic perfectly competitive environment.   Empircal evidence repeatedly shows that unregulated market suppliers continuously destroy wealth by restraining competition.  Some empirical evidence of market deregulation are the repeal of Glass-Steagal Act and the consequent bank bailouts, California’s deregulation of electricity suppliers that lead to artificial scarcity and exorbitant pricing.

Maybe the end-game is class immobility? Maybe that’s the point behind the destruction of New Deal goals?  The next time an industry is deregulated, ( Gramm–Leach–Bliley Act) it’s another strike against the vast majority of Americans regardless of your political affiliation.

I have to give the people that did this two thumbs way up.  Does it mean something good or bad?  It is hard to tell.

It is very hard to tell.

One of the most visible symbols of America’s ruling class is the Bohemian Club.  Last year, they spent less than their usual $400 – $600 for single bottles (plural) of wine.  Hard times, right?

Now they are trying to run a Redwood timber harvest.  “Second-, third- and fourth-generation redwoods will be thinned…”   SFGate Leaving only the largest of the great trees.  This is a classic power monger remake of the physical world.  The mightiest few are meticulously cared for (themselves and the oldest growth trees) and fell anything less.  That probably means clearing a few old-growth trees to make the property better conform to their orderly world view too.

They wrapped the harvest in the previously politically correct and fail-safe  ‘fire suppression’ meme.   Now, if it were the case that they did just harvest the fast-growing pine-variety invaders, I’d have no problem with the harvest.  It’s the fact they documented clearing the younger redwoods.  This is an external validation of the epic scale of their self importance.  Who needs to be bothered with petty small trees that won’t be large enough to gaze in wonder at (like me and my fellow Bohemians) for at least 1000 years?

Is it the case that their business lives are so slow they have time for a little ‘grounds maintenance?’  Or maybe they can’t get their rapacious urges sufficiently validated by a hostile proletariat?  Like the typical maniacal messiah, do they simply blame the thankless populists that can’t possibly understand the enormous burdens of remaking the world in their view?

An inquiring populist would like to know…

An Olympic Doper has already been caught. Cross-country skiing is a great sport to watch too.  It’s on my watch list along with Skeleton, Luge, Bobsledding and Short Track Speed Skating.

There are probably others though…