The reason why American cyclists need UCI points is to get good position on a typical mountain bike race starting grid.  Some visible examples of where this matters is the Olympics and the annual World Championships require a good starting position to get into contention for a podium placing.

That’s the need, now USAC’s solution is to encourage promoters to use “UCI compliant courses.”  These courses aren’t very interesting to race for the weekend warrior classes, so there’s a big disparity in participation between well-attended races with non UCI compliant  courses.   If a promoter attempts to use a UCI compliant course, the turnout is weak after the first event because word gets around that the course is not interesting.  USAC has been beating this dead horse for over a decade now with no signs of changing.

What’s so boring about ‘UCI compliant’ courses?  The descents aren’t very long.  The repetitiveness of the course isn’t interesting at all. It looks like USAC likes their courses ‘lawyer safe.’  So, not terribly difficult.  Lapping riders isn’t really racing per se and the riders getting lapped don’t like it either.

To make matters worse, the way they structured Mountain Bike events that generate UCI points does not maximize a rider’s opportunity to collect UCI points for a World Cup/Olympic cross-country starting postion.  USAC needs to have some combination of XCO (cross country Olympic, 5-9km), XCP (cross country point to point, 25-60km), XCC (cross country circuit, less than 5km), XCT (time trial, 5-25km) events.

A more optimal UCI Points American series goes something like:

• Five races that generate UCI points. Ideally the Canadian UCI events do not conflict.
• One time trial event. This is where a crazy steep hill should be used to draw spectators.  Or, an up-and-down course.
• The time trial sorts out the cross country starting grid.
• One Cross Country event.  This would likely be a point-to-point in the U.S.  How about a giant loop that puts you within 10K of your start line?
• I’d argue for a restriction requiring riding the same bike for both events.

That’s a whole lot to pack into one day, so doing the events over two days is the likely scenario.

It makes justifying races to the locals much easier because the promoter can cite the tourism dollars generated.  The promoter also has a more compelling “Expo” area for vendors sales and marketing.  Food, bikes, massage all looks better when you know racers will be around for hours over a couple days!

It also gives the promoter and racer some choices.  Let’s say Saturday you have a Time Trial in the morning, then some short course preliminary races in the afternoon/evening.  Obligatory BBQ on Saturday night!  Sunday, the long cross country course event and the “finals” for the short course event.

Skinnier Tires:

33c is the maximum width for a tire.  For most of us, there’s no change because the calculation they use to get to 33c isn’t what’s printed on the side of the tire.  It looks like the average 35c meets the 33c limit.

Disc Brakes are Allowed:

This isn’t a big change for a competitive cyclocross racer. Disc brakes on a ‘cross bike aren’t a big advantage.

-They are heavy.  Even if they come up with a compact model, many generations away, they are really heavy.

-Braking power is relative.  ‘Cross bikes are not suspended mountain bikes.  A ‘cross racer’s speed is very limited by the lack of suspension, skinny tires, and intentionally smooth courses.   These limitations eliminate the need for huge braking power.  To put it another way, there isn’t a slope of head-sized rocks to traverse while flying down a steep section on a UCI/USAC cyclocross course.

I’m working against the bicycle industry on this one because 2012 ‘cross bikes will be made with disc brake capabilities and ‘cantilevers are dead!’ be shouted from the rooftops.

A Word of Caution

In almost every race/ride, you will be slower using a ‘cross bike with disc brakes.  They are a great deal heavier than cantilevers.  You have to compensate for that weight penalty by working much harder than your cantilevered riding partners.  There are no physics that make disc brakes better in most conditions. ‘Cross is still a Watts test on relatively smooth ground.

The greatest fear I have is the old problem of the front quick release skewer loosening with disc brakes.  Unless they mount the caliper on front, this could get ugly.  The other problem is the earliest designs that make it into production will have mechanical flaws that will only be obvious after mechanical failure. (aka crash)

Reality Check

Most of you reading this will ignore it and buy a ‘cross bike with disc brakes in 2012.  Some of you will spend lots of money just to have it for 2010/11.   What’s worse, many sponsored pros will be riding with disc brakes because their sponsor will require it.  You’ve been warned that there are no extra benefits to disc brakes on ‘cross bikes.

Like prior cycling subcultures, they’ll over-promise disc brakes as the new hotness that makes cycling better and therefore drive away more cycling consumers.

2010/11

I’m still looking forward to increased popularity for cyclocross in the U.S.  It’s a great sport and fun to watch too.

• No on limiting leverage.
• No on the FDIC ‘consumer financial protection’ department.  This was weak, but better than nothing.

I didn’t dig around long enough to find the other parts of the bill voted on, but the whole thing is dead and Feinstein gets some credit.

There was another critical vote yesterday that might have set the stage for a GAO audit of the Federal Reserve.  She voted No on that too.

It’s no longer hard to wonder why you’ve got such dismally low trust in the U.S. Government.  Few Representatives are legislating the interests of the majority of Americans.

Barbara Boxer did The Right Thing and has voted for all of the items mentioned above.

Preamble

Let’s get some terminology and other stuff out of the way.

Spam: unsolicited email.

Bayesian: reaching a conclusion using some uncertain elements.  The wikipedia entry explains it.

Bayesian classification systems: A strict way of describing programs like SpamBayes or SpamAssassin.

Title or title header: This is the part of the email that is displayed as the title of the email.

Body: This is the part of the email where most of the text of the email is kept.

Why I chose to work on this already-solved problem. As an email server admin, the problem isn’t ’solved.’    Bayesian systems are very good. (with some limitations) Focusing on the limitations stands the best chance of making spam filtering much better.  The notion of “standing on the shoulder of giants” applies.

What are you doing differently? First, I’m using only analyzing the title of the email.  Introducing body analysis is recreating the work of Bayesian filters.  Second, I’m using a less well known algorithm and  applying it to the spam problem.  Lastly, titles with variations on words can be reliably detected. ‘V1agra’ and ‘Viagr@’ and ‘Viagra’ are detected as similar.  The resulting corpus isn’t particularly large.

Reductionist dismisal of the work:

• This is a naive method, and therefore it’s non-special.  I’d argue it’s a moot point. It’s less than 100% naive for a couple of reasons, all of which can easily be shouted down by someone with a higher socio-economic rank and sufficient buzz-word use.
• I’m unqualified.   My socio-economic standing does not exceed the title researcher, programmer or scholar.  Nor is the effort mathematically spectacular.   Another person with sufficient socio-economic standing will probably implement it and garner far more attention.
• This article isn’t sufficiently filled with cryptic vocabulary, therefore the process isn’t special. It’s unfortunate that vocabulary is used to establish social rank and therefore exclude. I chose to minimize the buzz-wordiness. The next person pursuing this method won’t advance the concepts and use far more cryptic language. They will however ill garner more attention using specialized vocabulary.

The Proposal: Detect Spam Analyzing the Email Title

Why choose to analyze only the title?

1. Bayesian systems already analyze the email body.
2. Bayesian systems tend to fail to detect spam with brief titles and similarly brief bodies.
3. I want to be able to scale the solution well beyond Bayesian systems.  As some email admins already know, Bayesian email filters are extremely resource intensive when you  are hosting a large number of email accounts.

The Algorithm: Rabin-Karp

Why Rabin-Karp?

1. Because it seems to be suited to the job of  finding strings while exhibiting the potential to scale way beyond Bayesian classification systems.
2. It is already used as a plagarism detector.
3. Other string searching algorithms have problems when applied to spam detection.    Most string searching algo’s are built on the assumption of discreet words.  A spammer can break the filter by confounding the definition of a ‘word.’

Build a Corpus

Step 1: I grabbed 17 spam titles and stuck them in a text file. When building the corpus, I keep all characters between A and Z and force everything into lower case.  The program then loops through possible window values and generates hash values.  I send the results into a SQLite database.  There are *much* faster ways to store the data, but this worked for me in the research phase.

Some notes about the window used to compute the hashes.  The window range I used was a minimum of 3 and a maximum of 10 characters.  What one would do with the variability is an open question.  If one uses too big a window to detect spam, then the filter will fail to detect spam.  If one uses too small a window, this may be too resource intensive.  I would think that randomly assigning the window would make it harder to break the filter.  Maybe it’s a moot point and a small window should be the standard.

Test the Classification System

To test the accuracy of classifying emails, I do a simple SQL lookup on windowed (see wikipedia reference) hashes of a title string to see if the hash exists in the database.  If it does, it gets 3 points.  The process for each email title goes something like this:

• If the sum of points squared is less than 120 then the program keeps checking the title string for more hashes.
• If the current sum of points squared is greater than 120, then I assume it is spam and move to the next email.   The high score results because the title is very close to what’s in the corpus.  I don’t have to search the whole subject string if the score breaks 120 sooner than the end of the string.

Test Results

title	Score	Corpus Info
oysterperpetual cosmographdaytona	256	in corpus
Fwd: Cookie Booth this Sun?	0	Not spam, not in corpus
Casino’s_Best PlAyer is Welcome!!	64	Gold Best Casino : Usa Player welcome!!!!
I’m tired of viagra ads	144	Not in corpus, a title from an exasperated friend
sale visit our website today and buy replica items cheaper	9	Not in corpus
@derall and V1codin online	64	_Percocet__Adderall_Cialis _Viagra_Ritalin!!!

Next Step

I need a bigger body of email titles to test and add to the corpus.   If you would like to provide some, then please contact me.  The title header just needs to be in a flat file, one per line, no opening/closing quotes needed.

Another step to sanitize the subject data is to strip “the”, most prepositions, and “of the”, “in the”, “to the”, “on the”, “for the”, “and the”, “that the”, “at the”, “to be” and “in a” out of the subject string.  I don’t know what effect this would have, but it is all noise. ]]> http://www.michaelpapet.com/blog/title-only-spam-detection-research-1/feed 0 http://www.michaelpapet.com/blog/mount-vernon-statement-an-economic-trojan-horse http://www.michaelpapet.com/blog/mount-vernon-statement-an-economic-trojan-horse#comments Sat, 20 Feb 2010 06:57:25 +0000 White Goodman http://www.michaelpapet.com/blog/?p=247 It looks like the Republican party is fracturing.  A bunch of those people got together and are making every effort to differentiate themselves from the spendthrift GWB #43 era ‘conservatives’.  I’m excited to see some more ‘cat herding’ in Conservative politics. In this way, they are becoming very much like the Democrats.

They have a new strategically elegant but tactically unworkable document they are calling the “Mount Vernon Statement.”

I’ll ignore all of the messy historical baggage just like the average “Morning in America” conservative and say that I think much of what they write works for many Americans.  I’m easily categorized as a Liberal and yet I agree with many of the principles in the rhetoric.  That is, until they turn the rhetoric into tactics and policy.

Trojan Horse Part 1

• It would be nice if they actually meant they want encourage Religious Freedom.  Tactically, their version of Freedom is called Christianity and the rest of you are going to Hell. (See “nature’s God”)
• It would be nice if ‘ordered liberty’ was meant to encourage more social and political freedom.  Tactically, they want Government to alter the lives of the people they don’t agree with.
• Government is the new Self.  It’s clear they want to alter the lives of the people who do not live according to their rules.  See “moral self-government”
• The military and Conservative pet projects win big too.  They get big budgets to spend on projects wrapped in the perfectly circular argument they will be ‘responsible.’

The bomb drop is buried near the end.  “economic reforms grounded in market solutions.”

Trojan Horse, Part 2

Let’s review some facts regarding capitalism and how the vaunted ‘market solutions’ accelerate economic disparity and lead to a weaker economy.  This ‘market solutions’ newspeak is not exclusive to the Republicans.  The Dems are using the same kind of  ‘market driven’ newspeak and they are as much to blame for the last 20 years of economic disparity.

Question: What is the rough definition of  an economic market?

Answer: A place potential Buyers and Sellers meet to possibly exchange goods and services.

Examples of markets are a car dealership, the grocery store, ebay.com.

Question: Are markets all-inclusive?

Answer: Never.  There may be lots of willing suppliers for eggs at greater than $25/dozen, but there are very few buyers.  This perfectly satisfies the political argument of ‘market solutions’ and more specifically meets the definition of a market.  Mission Accomplished!!! (GWB ironic reference)

Examples of inclusive markets are Ferrari automobiles, private American Medical Insurance policies, your grocery store with the homeless person in back digging through the trash.   Each market example has customers and the rest just stand idle.

There are at least two consequences of  ‘market solutions’ groupthink.

1. Increase economic disparity.  10+ years of unfettered capitalism in the U.S. has shown (again) that it ends up making more people poorer and few people richer.  Diminished economic activity follows.  Fewer people have disposable income as a result of increasing economic disparity.
2. Political and social instability.  People are falling out of the economy (> 10% unemployment, closer to 17%.) and probably won’t have the same kind of wages when they eventually return to work.  This creates a great deal of social and political instability.

The economic consequences of the Mount Vernon Statement are clear, it will  increase the ranks of the working poor, decrease social and political stability all of which decreases economic activity.

For those wondering why I don’t pound the Democrats as mercilessly, the Democratic Party has always been equivalent to herding cats.

If the unwashed masses are discouraged from discussing social class relativity/locality then the economic leaders have more freedom to create greater social and economic divisions.  There’s a perverse logic at work that somehow makes a few very rich, a tiny unstable middle class, and the vast majority with the least resources a Good Thing.  The longer discussions of social class are kept out of the scrum of daily life, the more time there is for increasing political and social inequality.

In America 2010, everyone is Middle Class.  Howard Stringer, Jamie Dimon, Frank Atlee, they are all Middle Class.  Only they are more Middle Class (and right) than the rest of the primitive populists.  The actions of these Middle Class Warriors  since the 1980’s has resulted in increased political and economic inequality.  Some obvious artifacts of this is the declining distribution of wealth.  Another artifact is the stagnant wage growth for the bottom 90%.

Another example, in this post.  Towards the end of the piece on banking regulation, the author labels the ideological assault as Serfdom circa 2010. What’s nice about the article is that someone else at least recognizes that income inequality is an explicit goal .   But then it tries to add some heat to the discussion by abusing the term Serfdom.  I don’t agree with the use of the vague word Serfdom.   Otherwise, the author describes the mechanics of failure at re-regulating the current form of Banking very elegantly.

Why does a country need a large middle class?  Greater political and social stability on which to build a vibrant economy.   The simple fact of the matter is creating a vibrant middle class require redistributing wealth. Uh oh.  Another American thoughtcrime.  I’m actively encouraging redistributing wealth.  Your average deregulator demagogue will now chime in with some faulty logic wrapped in Economics lingo “Redistributing income destroys wealth!”  or the hilarious “deregulated markets are more efficient!”   The general consequences of believing some version of the previous two arguments is greater economic and political disparity.

To specifically refute the wealth destroying argument,  empirical evidence like the banking interests mentioned in the article above repeatedly shows wealth being destroyed any number of ways.  I’d also argue most mature segments of the American economy are Oligopolies continuously destroying wealth anyway.

To specifically refute the deregulated marketeer, the theoretical model is elegant, but humans just don’t work like that and no amount of deregulation gets to the mythic perfectly competitive environment.   Empircal evidence repeatedly shows that unregulated market suppliers continuously destroy wealth by restraining competition.  Some empirical evidence of market deregulation are the repeal of Glass-Steagal Act and the consequent bank bailouts, California’s deregulation of electricity suppliers that lead to artificial scarcity and exorbitant pricing.

Maybe the end-game is class immobility? Maybe that’s the point behind the destruction of New Deal goals?  The next time an industry is deregulated, ( Gramm–Leach–Bliley Act) it’s another strike against the vast majority of Americans regardless of your political affiliation.

It is very hard to tell.

Now they are trying to run a Redwood timber harvest.  “Second-, third- and fourth-generation redwoods will be thinned…”   SFGate Leaving only the largest of the great trees.  This is a classic power monger remake of the physical world.  The mightiest few are meticulously cared for (themselves and the oldest growth trees) and fell anything less.  That probably means clearing a few old-growth trees to make the property better conform to their orderly world view too.

They wrapped the harvest in the previously politically correct and fail-safe  ‘fire suppression’ meme.   Now, if it were the case that they did just harvest the fast-growing pine-variety invaders, I’d have no problem with the harvest.  It’s the fact they documented clearing the younger redwoods.  This is an external validation of the epic scale of their self importance.  Who needs to be bothered with petty small trees that won’t be large enough to gaze in wonder at (like me and my fellow Bohemians) for at least 1000 years?

Is it the case that their business lives are so slow they have time for a little ‘grounds maintenance?’  Or maybe they can’t get their rapacious urges sufficiently validated by a hostile proletariat?  Like the typical maniacal messiah, do they simply blame the thankless populists that can’t possibly understand the enormous burdens of remaking the world in their view?

An inquiring populist would like to know…

Someone inside the bowels of the NBC organization will immediately grasp the simple fact that Microsoft’s Silverlight is locking out ~20% of all web users because of a buggy Mac Silverlight runtime shipped with buggy  implementations for Firefox and Safari.  That simply drives viewers away.

I think the thing that bugs me the most is the pervasive culture of mediocrity that goes into decision to support a limited number of visitors to what is a huge investment/opportunity for NBC.  The attitude that alienating 20%  of your audience is perfectly okay is bewildering.  I’d think advertisers would feel differently about it, but my gut feeling is that mediocrity is just okay with them too.

The ‘Office Space’ moment that has already occurred was when the list of browser/OS combinations tested was severely limited due to ‘resource constraints.’   ‘ Resource constraint’ being the code words for,  “Do as little work as possible.”  and a workplace culture that aggressively penalizes risk takers.

The next ‘Office Space’ moment will be where Yahoo’s Olympic traffic beating NBC’s Olympic traffic is cast as a win for NBC, advertisers, Moms everywhere, and the U.S. of A.