When the website of a company does not appear as a result in a search for that company, you know there's got to be something seriously wrong with your Search Engine...

I wonder how many casual, non-expert, everyday users of the World Wide Web know that Google actively bans sites from appearing on the Google index. There are certain sites that will not appear as a result for a Google search, ever, no matter what you search for, even if you search specifically for the domain name of the site. I wonder how the casual, everyday users would feel about this if they knew.

We have a client's site that is, to the best we can figure, currently banned by Google ["currently" being way back in June 2003]. Search all you want, you will not get to our client from Google (except as an advertised link we had to pay to put in because of this -- think about the implications of that: They won't let our site appear, but they will take our money to make it appear). The best we can figure, our site got banned because of a Google policy to ban sites which use hidden text of any kind. (Mind you, we can't confirm this; Google is notoriously unresponsive to inquiries of this sort, and the policy is nowhere explicitly defined nor explained, only vaguely alluded to.) We use a "hidden" 1 pixel high frame by way of controlling some music our client wants to have play while the site is browsed. We use a "hidden" frame to accomplish this because we feel our site users should hear music, not be faced with the ugly mechanics necessary to make it function. By doing it this way, we can avoid having the music restart every time a new page is loaded, and we can give the user controls for the music which we can be sure will actually work, even if they browse off the page which spawned the music. We are in no way trying to manipulate our Google search result rankings. So imagine our surprise when quite by accident we happened to noticed that this site was suddenly no longer showing up in Google, even though it was before. We never received any notice of our site apparently being banned, other than our happening to notice.

Google, it seems, has moved beyond its original mission of providing a good, useful, and objective index of the World Wide Web as it exists, and is now trying to socially engineer the Web to what it feels it should be. The index no longer actually represents real sites that exist in the World Wide Web, and I wonder how the average user of Google would feel about this if they knew.

Why the drastic seeming response of actually banning a site from ever appearing as a Google search result? It seems Google has let itself get caught up in an arms race with a certain type of web developer who call what they do Search Engine Optimization: they figure out how Google comes to its results, and then try to skew things in their favor by manipulating those variables. Google doesn't like this, precisely because it skews the results, and eventually this kind of manipulation would result in the whole Google index becoming useless, much like an email account swamped under the deluge of unwanted commercial email -- spam. So Google has to do something to fight this kind of manipulation. The current level of escalation? If Google deems that you are trying to manipulate their index, instead of getting the better showing in search results you were looking for, you are now banned entirely and will never show up at all!

(Another thing they do is to keep secret what they do -- if you don't know what exactly they are doing, it is harder to counter it -- hence it being so hard to get any solid information if you do find yourself seemingly banned.)

The problem with this strong, punitive approach is two-fold, and immediately apparent, I would think, to the every day user trying to find what's out on the Web, but not so obvious to Google, caught up in the heat of the battle with these very specialized web masters. The first problem is that innocent parties might get caught in the cross-fire and find themselves banned unjustly. The second, and bigger, problem is that the everyday user is punished because the Google index no longer actively reflects real sites that exist. It is no longer the tool doing what it purports to do.

If I'm trying to find information about a certain subject or a product I can buy, I don't care if a site was trying to manipulate its placement on the Google index -- I want to be able to find that site as it relates to my search. If that site is now banned by Google, I will never find it, even if it was exactly what I was looking for. Clearly this is no good for anyone concerned. The average user suffers, and as more and more users discover their searches on Google to be no longer as useful as they once were, Google suffers as users move on and look for other, better tools.

The experts on Search Engine Optimization get together and exchange ideas on a forum at www.webmasterworld.com. This forum is even frequented by a purported Google employee, "GoogleGuy". (This back-channel communication seems to be the only source of information about Google policies.) In a discussion-thread about hidden text and what constitutes it (http://www.webmasterworld.com/forum3/13619.htm), the following was quoted from GoogleGuy (not about our site):

I'll be happy to give a very definite answer. Looking at the cache of your site, I think we did the right thing. On your root home page, I see a black background, with no white imagery anywhere in sight, and black text (Okay, text with 000001 as the color--is that supposed to fool people?) at the bottom that says "bodybuilding bodybuild body building weightlifting muscle supplements sports nutrition vitamins flex fitness bodybuilding." Sorry to get into specifics, but you did ask.

What part of that is not hidden text? You've taken it off your home page now, but do you think we can't check cached pages? So yes, I think you did have hidden text on your main page, and yes, I think you got caught. I would go over everything with a fine-toothed comb to make sure you've removed every bit of hidden text from your site. Within a month or so, if all of the hidden text is gone, you should should [sic] show back up in the index again.

So they seem quite satisfied that they caught an egregious user of hidden text, and feel quite justified that they banned his site from the index. But stop, step back, and think about it for a moment: how is the average user of the Web helped by this? If I, an average Google user, want to find information about body building and supplements, I will no longer be able to find this page at all, even though it sure does sound like it would have useful information for me!

Even worse, of course, are the innocents thrown out with this bath water. In this same discussion-thread there are at least two first person occurrences of sites dropped from the index despite no attempted manipulation. One was due to a mistake while coding, the other due to a novice user not realizing that Microsoft's FrontPage was being a little too friendly in its layout and not showing the user that she was actually building a site with dark text on a dark background. Both these people had their sites banned. Fortunately for them they 1) noticed, and 2) were aware of this elite little discussion board, and were able to piece together the reason for their site's banishment. How many thousands more are 1) not aware of their site's banishment, and 2) totally without the slightest clue as to why their site might have been banned in the first place!

Elsewhere in the discussion thread, GoogleGuy heartily endorses another poster's conclusion that Google does not wish to penalize anyone, and would rather that no one would ever try to manipulate their results so that they would never be forced to penalize anyone. What is disturbingly apparent throughout all this discussion is how ingrained is the assumption that Google has no choice but to dole out punishment. They don't want to, but fair is fair, evil-doers must be punished, there is a price to pay for freedom, etc., etc., etc.

It is easy to get sucked into the seeming logic and justice of it, but to do so is to overlook a very obvious and straight-forward alternative: if Google can determine that someone is trying to manipulate their results, how about they just ignore this attempted manipulation? Nothing will be gained by it. The site will receive the search placement it deserves, the manipulation attempts will be ignored. Thus the site is available as it should be, and where it should be, for the Google user. And since attempted manipulations will be ignored, inadvertent pseudo manipulations, which are in reality errors or things that should be ignored anyway (like in our apparently banned site), will not be punished, and will show up in the index exactly where they would have without the errors or useless bits. Everybody wins! --Though there is of course no satisfaction of sticking it to that jerk who tried to grab more than his share.

And that's where it stands right now. Does Google want to continue to maturely develop the best objective tool for finding things on the Web, or do they want the fleeting feeling of satisfaction in a vindictive narrow little squabble with a very small but to them odious group of web developers?

I wonder how the average user of Google would feel about this?