Data Sharing or Scraping?

Jan 1, 2008 Michael Wurzer

Wired posted an article called Should Web Giants Let Startups Use The Information They Have About You that touches on many of the topics dear to the FBS Blog, especially the terms of use on which data should be shared. Here are some key quotes:

The Internet these days is supposed to be all about sharing. Thanks to a common commitment to open access and cooperation, the data mashups that have defined the Web2.0 phenomenon have exploded. . . .

But beneath all the kumbayas, there’s an awkward dance going on, an unregulated give-and-take of information for which the rules are still being worked out. And in many cases, some of the big guys that have been the source of that data are finding they can’t — or simply don’t want to — allow everyone to access their information, Web2.0 dogma be damned. The result: a generation of businesses that depend upon the continued good graces of a relatively small group of Internet powerhouses that philosophically agree information should be free — until suddenly it isn’t.

When data is put in a silo and application developers have to develop on top of that data repository, the power to change the rules for access is critical:

Such vulnerability to sudden data blackouts illustrates why some potential investors get nervous about funding scraping-dependent businesses. “Anybody who is a supplier to you has power over you,” says Allen Morgan, a venture capitalist at the Mayfield Fund who has invested in a raft of Web 2.0 companies, including Tagged, a teen social network and Slide, one of the most successful makers of Facebook applications. Morgan says that as those data providers help power more applications, they take on the role of operating systems — with a vested interest in consolidating their power. “Inevitably, they will feel compelled to compete with application developers in order to grow their business — and it’s an unfair fight.”

The article concludes wrongly, I think, in favor of the scrapers, when the reality is that what needs to occur is a three-way agreement among the data providers (in the case of MLSs, that’s the brokers), those wanting access to the data (third-party applications), and the MLSs or other entities aggregating the data (data repositories). All bring value to the equation and enabling interoperability and data exchange is key, but that requires mutual agreements on the terms on which such data exchange occurs.