The Life and
Near Death of DMOZ
By Jim Hedger

The casket was all but closed on the venerable Open Directory Project (ODP, or dmoz.org). A December 16 blog post by an ODP founder, Rich Skrenta, “DMOZ had 9 lives. Used up yet?” , suggested that the directory at DMOZ is now, like Marley’s ghost, deader than a doornail. DMOZ was down and, for over a month and a half, it looked like it was down for the count.

In reality, DMOZ is not dead though the rumours of its demise were not exactly exaggerated either. Because this six-week unscheduled outage followed several years of consumer dissatisfaction, lagging editorial energy, and layoffs at AOL, many made the logical assumption that the plug had been pulled.

While the website still functions as a searchable directory, its editing functions have only just been restored after six weeks of downtime. Since the last week in October, editors and submitters have been greeted by versions of a customized DMOZ 404 page. DMOZ was basically a dead directory referencing over 4 million websites spanning nearly 600,000 categories. Though editing has been restored, it is still not possible to submit new sites.

Even if webmasters could submit new sites, chances are they would not receive timely editorial attention. For the last few years, webmasters have complained about the now legendary backlog of sites awaiting review and inclusion. It can take months or even years for spelling mistakes to be corrected and an enormous number of the 590,000 categories that make up the directory do not even have editors. Though many webmasters consider the Open Directory useless because of that backlog, it still swings a big weight in the search sector.

The greatest success of the Open Directory Project stems from the free database access offered to any other search entity. The majority of search engines and directories use the ODP’s RDF-esque data-dump to help populate their databases. As every ODP listing is human edited, Google and other search engines have tended to treat ODP references as trustable sites. Carrying a PageRank of 8, links from the ODP continue to be considered Google-Gold by SEOs. Other search engines receiving results from the DMOZ directory include Ask, Yahoo and AOL. Clearly the ODP remains an important entity in the search space.

It has certainly earned its status as an important entity. The Open Directory Project has a long history that dates back to 1998. Since the day it went online as GnuHoo in June 1998 it has played a crucial, defining role in the evolution of the search sector and of the Internet.

Gnuhoo appeared on June 5 1998 in response to the rapid growth of the web. The number of new sites coming online in 1998 far exceeded the capacity of Yahoo’s editorial staff that was rumoured to number less than 200. Gnuhoo co-founders Richard Skrenta and Bob Truel believed they could create a better directory using an unlimited supply of volunteer editors than Yahoo could with their limited team of professional editors.

They were right. NewHoo grew faster than Yahoo did in the last half of 1998. Less than a year after it went online, the all-volunteer project had acquired 8,000 editors and over 430,000 websites. By then it had undergone two name changes and had been acquired by one of the largest emerging online entities.

Within days of being online, Gnuhoo had attracted enough attention to force a rapid succession of name changes. First the Free Software Foundation objected to the use of the term GNU after a Slashdot article misconnected the two projects. Gnuhoo was thus renamed Newhoo. A few days later, Yahoo raised issues about the use of the suffix “Hoo”. At the same time, Netscape Communications Corporation opened a dialogue with Skrenta about acquiring the upstart directory project as a major asset during their competitive phase with Microsoft.

Promising to respect the founders’ original intentions to keep the site a non-commercial entity, Netscape acquired the directory for $1 million in October 1998 and renamed it the Open Directory Project. ODP data was released freely under the Open Directory Licence. A month after Netscape bought ODP, America Online (AOL) purchased Netscape. AOL agreed to honour the Open Directory Licence, formalizing it in a Social Contract with the web community. This marks the real start of the ODP’s rise. By early spring 1999, most of the major search engines were pulling data from the ODP.

1998 and 1999 was a special time in the history of the Internet. Billions of dollars were invested as eager speculators and venture capitalists moved to cash in on the promise of instant riches. Start-up companies with no functional business plans became multi-million dollar concerns overnight. The first generation of instant online millionaires was spawned and talk of breaking the traditional business cycle was taken seriously. The bottom was about to fall out of what had become a stratospheric marketplace but at the time, very few saw the danger through the haze of the hype. When the sky fell, it fell hard. In a tangential way, the ODP was directly involved. Though it is technically a non-profit society, ownership of the ODP is considered a business asset.

The trigger event that led to the crash of 2000 was the most significant deal in the history of global publishing. In January 2000, less than a year after it had acquired Netscape and DMOZ, AOL purchased the Time Warner media empire for approximately $160 Billion in an all-stock deal. The excess of that deal, one in which an upstart tech firm absorbed the largest brick and mortar information and entertainment business in the world, made a number of analysts look at the silliness of it all. Within three months, the shares AOL used to buy Time Warner would be worth a fraction of their value when the deal was struck.

The Tech-crash of 2000 had a cascading effect across the web. Most, if not all, of those new businesses without business plans were quickly put out of business as the value of those firms had declined and no new sources of investment were forthcoming. Online properties supported by shareholders, such as Yahoo and AOL/Time Warner, were in sudden desperate trouble. 18-months of tech sector doldrums set in as the investment world started looking for a revenue source that could sustain the staggering costs of the sector.

A new search engine appeared on the scene around this time. It had a funny name and appeared to disregard the dominant portal or directory structure favoured by most search engines. Hidden behind its sparse front page and childish logo was a revolutionary way of producing what everyone agreed at the time were extraordinarily accurate search results. The age of Google began in late 2000. A year later, the power of viral marketing had propelled Google into the big leagues, making it a serious challenger to AltaVista, Lycos and Yahoo.

Google populated itself in part by using DMOZ data. In its earliest years, Google used DMOZ as its directory, displaying virtually mirrored results. Google’s unique method of judging page content by the number and value of incoming links made a listing at the Open Directory critically important for SEOs and webmasters. As Google’s popularity and reach grew, the value of a DMOZ link grew. Because ODP listings are human reviewed, Google has traditionally tended to trust them, thus producing stronger placements faster. Between 2001 and into 2005, Google was responsible for over 80% of all organic search listings either directly or through feeding competitors such as Yahoo and MSN.

When Google figured out how to make the paid-advertising system Overture was using make oodles of money, all hell broke lose again and we rapidly advanced to where we are today.

When Google became the most important search engine, search marketers began targeting the Open Directory with site submissions, often with several sites for the same company. As one ODP editor put it, “We never asked to be used by Google like this.” As the decade progressed, new methods of creating web documents (html editors, CMS, blogs, etc…) spurred another period of extraordinary growth that far surpassed the ability of DMOZ editors to keep up.

A classic dilemma existed. A link from DMOZ could mean the difference between weeks and months waiting for a good placement at Google. The ODP was never supposed to have such influence. The relationship Google’s algorithm created between itself, the Open Directory and webmasters wanting a DMOZ listing ended up threatening the open editorial policies originally envisioned. It was difficult to enlist new editors when many applicants were primarily motivated by the ability to insert their own sites.

Though it boasts almost 75,000 editors, it also contains over 590,000 directory categories and sub-categories. The Open Directory is enormous and continues to be driven by volunteers. One of two things happen; either its volunteer editors deal with an average of 8 categories each or some categories will have to go unedited. The latter tends to happen more often than the former and the public and search engines are left with a less than complete directory to draw from. Such has been the case for the past two or three years.

In their defence, the ODP editorial staff would suggest that the majority of sites they continue to see are junk advertisement pages designed for SEO or PPC purposes. Similar comments appear in any number of threads found at the Open Directory Resource Zone, a public chat forum designed to promote communication between editors and users.

With a massive backlog of unreviewed submissions and a huge demand from search marketers hungry for the rankings boost expected from a DMOZ listing, many felt the ODP was becoming an elite, secretive society. Editorial applicants reported their requests were going unanswered and allegations of corruption amongst rouge editors emerged. By end of 2005, the ODP appeared to be in total disarray with more sites in the review process than were actually in the directory. Throughout 2006, the ODP has become less and less relevant to the search marketing community until, towards the end of the year, it was gone.

Most of the directory appears to be functioning again though it is likely a version carped together using data from the last RDF file. When the server at AOL crashed, it took most of the current directory and all of its records with it. A number of meta editors have spent the past six weeks rebuilding the directory with the help of a few friendly AOL techs. The submit a site feature is, as of this time, not functioning.

Outwardly, the importance of the Open Directory was obvious but the greatest contributions to the Internet from the Open Directory team come from the people involved with the movement and the open-source philosophy that has descended from them.

When Netscape absorbed it, the Open Directory Project became part of an amazingly influential environment. Founded by legendary Marc Andreessen, Netscape was already part of the Open Source movement. Netscape founded the Mozilla Foundation in January 1998, nearly a year before it acquired DMOZ. The Mozilla Foundation introduced and marketed the Firefox browser.

The ODP was arguably the first successful long-term project that could fall under the general heading Web2.0. Its philosophy set the stage for the Wikipedia and other community based websites. Unlike other collaborative projects that predate it, the ODP was a truly grassroots endeavour. Participants didn’t need to be extraordinary technicians; they just had to be able to understand the editing techniques used by their community.

Though rumours of its death are obviously exaggerated, complaints about its demise are not. The ODP is a wonderful entity, but the power it inadvertently exerts is far greater than its ability to edit itself. Many have suggested the ODP should shut its door for good but perhaps this downtime has given its meta-editorial collective a chance to consider its role in the search community.

About The Author
Search marketing expert Jim Hedger is one of the most prolific writers in the search sector with articles appearing in numerous search related websites and newsletters, including SiteProNews, Search Engine Journal, ISEDB.com, and Search Engine Guide.

He is currently Senior Editor for the Jayde Online news sources SEO-News and SiteProNews. You can also find additional tips and news on webmaster and SEO topics by Jim at the SiteProNews blog.

Great history lesson. Let’s face it, DMOZ isn’t dead, but it is on artificial life support. Here’s what I mean:

The Web thrives on value. If you provide it, the Web community will see to your success. If you don’t, you’ll be abandoned. Other great sites have met that fate in the past.

Google owes its success to DMOZ. Since the search engine’s beginnings, it has relied upon DMOZ for important information. Google’s algorithms, from its earliest days until now, have relied heavily upon links and backlinks. DMOZ was an early source for Google in obtaining that information on the web sites it indexed. Today, Google has so improved upon its processes that it could effectively do what it does best with the assistance of DMOZ. Yet, it still relies on the antiquated system of DMOZ. One must wonder why.

The allegations of DMOZ editors being corrupt is just absurd. They’re volunteers, for crying out loud. They don’t get paid. They spend their own time, without compensation, between paid jobs, families, church picnics and their own social lives to do work that largely goes without thanks. Unfortunately, they are doing volunteer work in a world that rewards entrepreneurism.

Google entered the online world at a time when there was great hope. They got their big break at a time when that hope was shattered. In 2001, no one knew that five years later there would be successful Internet entrepreneurs. For all most of us knew, Internet moneymaking opportunities were urban legend. Google, however, changed that. In fact, one could say that Larry Page and Sergey Brin made e-commerce possible. They gave us a way to do it and, in the process, pave the way for search engine marketing.

To succeed online today, you’ve got to pay attention to Google. You don’t have to give DMOZ a second thought. I’ve talked to webmasters who have built successful online businesses who have never submitted a site to DMOZ. Most people who do submit to DMOZ do so because they are operating on information from 2001, when DMOZ made a much bigger difference. If you are serious about making money online today, you’d be wasting your time even messing around with DMOZ and it’s century-long process of getting your information wrong. Do even half the things right to get listed in any of the search engines, and you can be picked up by Google in three months. Why, then, worry about a free directory that contributes nothing?

The Internet has gone commercial. DMOZ is a charity. Give to it if you want to or feel you must. But realize that it won’t do any more for you than you can do for yourself.

Yahoo! adapted its processes to compete in today’s game. DMOZ hasn’t. And if the top thinkers at Google pulled their head out, they’d realize that they are the ones keeping DMOZ alive. If Google pulled the plug, the “largest human-edited directory” would die a quick death. Few money-minded webmasters would miss it.

An alternative would be for Google to buy DMOZ. But then, what would they gain from that? Nothing. Most of the information that DMOZ has that has any value, Google already has. What it doesn’t have it can get in 30 days. The next change the search engine giant needs to make to its ranking algorithms is the value it places on inbound links, particularly those from DMOZ. Instead of penalizing reciprocal links from like sites, it should discount anything coming from DMOZ. Then, DMOZ wouldn’t count, webmasters would stop submitting to it and within a short time there would be no more DMOZ. Would that really be all that bad? If you think so, build a monument with this inscription, “They built a giant; the giant destroyed them.”

ASM Development can help you with all your SEO & Web Design needs
SEO Blog

Recent Entries

• Why A Search Engine Googopoly Is Bad For The Internet
• Blogging For The Sake Of Blogging Is Not Blogging
• How To Create An Effective Social Marketing Tweet
• Internal Linking – Don’t Be Afraid Of The Long Tail
• Adding Your Scotland Based Local Business To AOL
• Don’t Be Too Quick To Dismiss Google Buzz
• Are Guest Posts A Bad Link Building Strategy?
• Can You Overdo SEO?
• Review Sites And SEO For Scotland Local Businesses
• What Effect Will HTML5 Have On Web Design And SEO

This entry was posted on Friday, December 22nd, 2006 at 6:52 am and is filed under MAIN. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.