The decline of DMOZ and what it means

The Decline of Dmoz

In the Web world, ten years is a really long time for an institution to last. Google has not really been around that long.

DMOZ, AKA the Open Directory project,  is one of the oldest Web directories. It was founded in 1998 as Gnuhoo and bought out by different folks as it succeeded in the world.

At one time, directories were very important on the Web, because compilations of links were just about the only way to find anything. Yahoo! wasn't much good then, and  Google did not exist at all. A searchable directory built by users seemed like a great idea - and it was. By 2003, DMOZ Open Directory had some 3.5 million sites listed, and it was getting 3.5 million visits a month. But then DMOZ went into a decline, measured by visitors and accumulated site listings. In April 2009 it got under two million visitors for 4.5 million site (or is it page?) listings. according to Compete data. The number of listings has declined recently from 5 million. So, say Chris Crum, Dmoz is declining. While one should note that Compete evidently shows only US visitors, while dmoz.org has a world audience (and dmoz exists in several languages) the general judgment that DMOZ is getting less visitors has to be true. 

Dmoz's utility is not measured just by the number of visitors it gets. It is a directory that is used by Google and other search engines. Its RDF format data dump is used by about 300 other Web sites. So Dmoz indirectly is responsible for a lot of Web traffic. As a source of Authority it has an important purpose. It is the only all-purpose directory left on the Web. What happened to Dmoz is fairly obvious. Google's search algorithms got better and better, and so did those of the competition, and specialty search engines took over a lot of the work that might go to a directory like Dmoz. SERPs that just list the top sites are usually more convenient than trying to navigate through the Dmoz categories, which are often overlapping and arbitrary.  Google's News search cannot be duplicated by a hand compiled directory - not easily anyhow. But given the existence of search engines, Dmoz's primary function should be to list Web sites and pages and give them authority which search engines can rely on. That's an important function that cannot be measured in number of visitors to Dmoz. The Web continues to grow. The fact that the number of listings in Dmoz stopped growing (see Wikipedia article ) is proof enough that Dmoz is in decline.

Dmoz editorial policies and methods leave much to be desired, to say the least. A group of disgruntled folks have put up a   http://www.dmozsucks.org/  web site and there is not a shadow of a doubt that many of their complaints are justified. More about that below. But the decline in Dmoz visitors cannot be explained by that alone, since Dmoz probably sucked a lot worse in 2003.

My own experiences as Dmoz user, submitter and editor as well as those of the oppressed minions at dmozsucks could serve as guides for how to improve dmoz. It is doubtful if the improvement will bring many more visitors to dmoz. They aren't going to compete with Google. The improvements could keep DMOZ from becoming totally irrelevant as a source of authority and allow them to attract new editors. 

What you find when you search dmoz is a jumble of sites and pages that are not arranged according to quality. Bad sites or pages are very often duplicated in several different categories because they agree with the commercial interests or political bias of a particular editor. If an editor with a different political bias is put in charge of a category, you can often forget about getting your Web page or Web log listed, and the same is true of commercial listings. Despite supposed abuse procedures, there is really no appeal from editor decisions, and editors do not need to reply to you about their listings. In fact, they are not supposed to.  Supposedly, Dmoz.org always informs you of the status of a listing. In reality, they never do. The only way I ever found out that a submission was accepted in dmoz was by searching for it in dmoz.

A new dmoz editor quickly gets the feeling that they have entered a closed guild that does not want them there. Everyone is your supervisor as a new editor, and their seniority is determined by how long they have been dmoz editors. A physics professor can find themselves being lectured by a moron who runs a games directory, and explains exactly how a scientific topic ought to be ordered according to their view. Rules are invented by editors as they go along to justify arbitrary decisions. Once I was told that dmoz only accepts web sites. That was done in order to disqualify a listing the editor did not like. When I pointed out that there were numerous listings of individual pages, a different rule was invented. I was told that duplicate pages are not listed. When I pointed out a particularly vicious and ignorant "history" article about a controversial topic, listed from 3 different sources in about 8 different places, an excuse was found to defend those listings. Almost nothing I tried to list ever got listed, because the supervising editor always found something wrong with the description format, the description content or the Web site or page itself. It was obvious that these listings were no different in format or content from other listings, so the editor said that they had changed the formatting rules. 

Regrettably, volunteer directories are staffed by volunteers. Anyone who gets a bit of power, no matter how absurd, starts to use it, and the volunteers at dmoz.org often have more than a bit of power. And very often they volunteer in order to advance some political cause or some commercial interest. The Zeal directory was no better. There, people rated Web sites they submitted. Of course, they tended to give their own sites or those that served their interests good ratings.  Real altruists are few and far between. The moral of the story is that we shall have to pay for quality, and find a way to ensure that ratings and listings are unbiased, editors are courteous and not arbitrary etc. Google or another search engine could set up such a system, where reviewers are asked to rate top 10 or 20 pages for a given query, or asked for ratings of submitted pages, and paid for their opinions. The ratings could be utilized by search engines as well as by DMOZ, and the really bad sites could be excluded.  Google already does this with quality review of their own search results (see The Quality of Google Search Results)  

Google also now allows users to rearrange search results according to usefulness and relevance. It is not clear how many people take advantage of this or whether Google uses any of these data for making decisions about the order of results that are displayed. This system is open to abuse by mob opinion of course, and results of search queries that don't get a lot of action could be manipulated by a determined group of users. It also doesn't allow for submission and rapid advance of new, good Web pages. For popular topics, users must wait a long time before such pages will work their way up to the top 10 or 20 results (See The killer search engine bug).

Paid reviewers would produce a set of quality listings that might meet a market demand, but it might not happen any time soon. A compromise solution that could be instituted by whoever owns dmoz (Time Warner at present, I believe) would be to have a larger number of paid supervisors who do quality checks, including disguising themselves as people submitting Web pages or as novice editors, as well as checks of results, and who find and remove duplicate listings, spotlight glaringly biased political or commercial directories, and remove power-crazy and arbitrary editors. 

SEO Basics