The Open Directory Project: The Spirit of the Web

In June of 1998, a call went out across the Internet for help.

Sick and tired of Yahoo's old and dead links, Rich Skrenta, a California computer programmer, determined to create the web's most comprehensive directory. Poor Yahoo, with anywhere from 85 to 200 (depending upon whose statistics you believe) professional editors on staff, was simply unable to keep up with the growth of the Web.

Thus started the concept for GnuHoo, a directory similar to Yahoo, but edited by an unlimited number of volunteers.

On June 5, 1998, the site went live. By June 18, there were 200 editors, 27,000 sites, 2,000 categories. It was at this point that your humble NewsLetter editor joined the project.

There was no screening process. All an editor need to do was pick a category and begin editing. The idea was that if you had an interest in a topic, that topic would flourish with people who were passionate about the category at the helm.

There were no guidelines set forth, other than to go forth and multiply. It was unclear just how sites were to be described or categorized, and general anarchy reigned. It appeared in the beginning that most of the volunteer editors had joined simply to promote their own websites. There were behind-the-scenes struggles, with one editor putting in a site and another editor immediately removing the site. Unfortunately, there were no internal forums, and since there were no rules or guidelines, it was not clear to all editors that email would be a polite way to communicate.

The inspiration for the concept of the all-volunteer project as well as its name came from the GNU project, the well-established freeware developers' organization which was working to produce a free UNIXish operating system. GNU protested the name GnuHoo, and the name was changed to NewHoo.

By July 2, 1998, NewHoo boasted 400 editors, 31,000 sites and 3,900 categories. The long weekend produced still more advances, and by July 10, there were 1,200 editors, 40,000 sites. Yes, in just five weeks!

While it was taking Yahoo months to index sites, NewHoo was managing to review and list 99% of all sites submitted within 24 hours. Not that many people cared, other than the 1,200 editors and their close friends. Alas, NewHoo was still not a major directory.

The word must've gotten out big by July 14, because there were more than 1,600 editors! In a press release, Chris Tolles, one of the founders, was quoted as saying, "This won't be stable and static. I'm sure there will be pissing contests between editors and so forth. But the whole thing is self-governing. It will even itself out." And so it was. By now, those editors who had only signed on to promote their own sites had largely dropped out, with the exception of the rather large number who had discovered what it meant to be a part of such a project. The thrill of bringing back some of the spirit of the Internet.

NewHoo was being discussed in search engine forums across the Web, and people, myself included, were dragging their friends to NewHoo to become editors.

On November 17, 1998, word went out the Netscape had acquired NewHoo for their Netcenter Web Site. The very next day, there were 4,500 editors and 100,000 listed sites in 2,500 categories. This was, finally, the big time. Although we all knew that NewHoo had the best-edited directory and the most committed editors, it was time for the rest of the Net to find out about us. The exposure that Netscape would provide spurred everyone on, and challenges began to go out among editors.

There was another name change coming. This time, the name was to be the "Open Directory Project." It was also decided that ODP would be available to organizations and even individuals who wanted to use copies of the directory on their own sites.

Some time in early 1999, the chaos began to subside within the ODP. Threads began to appear in the ODP internal forums in the first couple of months of the new year, and a hierarchy of editors began taking shape in April. Real communication began to take place amongst the editors of all categories, and the directory was growing by leaps and bounds with solid, relevant listings.

April 16, 1999, it was announced that Lycos would use the ODP, which now had more than 8,000 editors, 430,000 sites and 65,000 categories. Naturally, HotBot went with its sister, Lycos.

Lycos said, "Unlike other Internet directories that rely on a small staff of paid editors, the Open Directory Project leverages the unique insights of a wide variety of Internet users to build the best directory on the planet."

Today, ODP boasts 597,222 sites, 11,500 editors, and 89,340 categories. The ODP database is used on more than 20 search engines and directories, including Netscape Open Directory, Netscape What's Related, Lycos, HotBot, Dogpile, Thunderstone, and Mars Society, and Linux.com Links.

Certainly now ODP is fast becoming the most looked at directory on the Web. And for that reason, it is featured in this month's issue.