innovation

Philip Larkin on modernism

From Robert Phillips’s interview of Philip Larkin in “The Art of Poetry No. 30” (The Paris Review: Summer 1982, No. 84):

It seems to me undeniable that up to this century literature used language in the way we all use it, painting represented what anyone with normal vision sees, and music was an affair of nice noises rather than nasty ones. The innovation of “modernism” in the arts consisted of doing the opposite. I don’t know why, I’m not a historian. You have to distinguish between things that seemed odd when they were new but are now quite familiar, such as Ibsen and Wagner, and things that seemed crazy when they were new and seem crazy now, like Finnegans Wake and Picasso.

Philip Larkin on modernism Read More »

The origin of the word “munge”, “hack”, & others

From Steven Levy’s Hackers: Heroes of the Computer Revolution (Penguin Books: 2001): 23:

The core members hung out at [MIT’s Tech Model Railroad Club in the late 1950s] for hours; constantly improving The System, arguing about what could be done next, developing a jargon of their own that seemed incomprehensible to outsiders who might chance on these teen-aged fanatics … When a piece of equipment wasn’t working, it was “losing”; when a piece of equipment was ruined, it was “munged” (Mash Until No Good); the two desks in the corner of the room were not called the office, but the “orifice”; one who insisted on studying for courses was a “tool”; garbage was called “cruft”; and a project undertaken or a product built not solely to fulfill some constructive goal, but with some wild pleasure taken in mere involvement, was called a “hack.”

This latter term may have been suggested by ancient MIT lingo– the word “hack” had long been used to describe the elaborate college pranks that MIT students would regularly devise, such as covering the dome that overlooked the campus with reflecting foil. But as the TMRC people used the word, there was serious respect implied. While someone might call a clever connection between relays a “mere hack,” it would be understood that, to qualify as a hack, the feat must be imbued with innovation, style, and technical virtuosity.

The origin of the word “munge”, “hack”, & others Read More »

How security experts defended against Conficker

From Jim Giles’ “The inside story of the Conficker worm” (New Scientist: 12 June 2009):

23 October 2008 … The dry, technical language of Microsoft’s October update did not indicate anything particularly untoward. A security flaw in a port that Windows-based PCs use to send and receive network signals, it said, might be used to create a “wormable exploit”. Worms are pieces of software that spread unseen between machines, mainly – but not exclusively – via the internet (see “Cell spam”). Once they have installed themselves, they do the bidding of whoever created them.

If every Windows user had downloaded the security patch Microsoft supplied, all would have been well. Not all home users regularly do so, however, and large companies often take weeks to install a patch. That provides windows of opportunity for criminals.

The new worm soon ran into a listening device, a “network telescope”, housed by the San Diego Supercomputing Center at the University of California. The telescope is a collection of millions of dummy internet addresses, all of which route to a single computer. It is a useful monitor of the online underground: because there is no reason for legitimate users to reach out to these addresses, mostly only suspicious software is likely to get in touch.

The telescope’s logs show the worm spreading in a flash flood. For most of 20 November, about 3000 infected computers attempted to infiltrate the telescope’s vulnerable ports every hour – only slightly above the background noise generated by older malicious code still at large. At 6 pm, the number began to rise. By 9 am the following day, it was 115,000 an hour. Conficker was already out of control.

That same day, the worm also appeared in “honeypots” – collections of computers connected to the internet and deliberately unprotected to attract criminal software for analysis. It was soon clear that this was an extremely sophisticated worm. After installing itself, for example, it placed its own patch over the vulnerable port so that other malicious code could not use it to sneak in. As Brandon Enright, a network security analyst at the University of California, San Diego, puts it, smart burglars close the window they enter by.

Conficker also had an ingenious way of communicating with its creators. Every day, the worm came up with 250 meaningless strings of letters and attached a top-level domain name – a .com, .net, .org, .info or .biz – to the end of each to create a series of internet addresses, or URLs. Then the worm contacted these URLs. The worm’s creators knew what each day’s URLs would be, so they could register any one of them as a website at any time and leave new instructions for the worm there.

It was a smart trick. The worm hunters would only ever spot the illicit address when the infected computers were making contact and the update was being downloaded – too late to do anything. For the next day’s set of instructions, the creators would have a different list of 250 to work with. The security community had no way of keeping up.

No way, that is, until Phil Porras got involved. He and his computer security team at SRI International in Menlo Park, California, began to tease apart the Conficker code. It was slow going: the worm was hidden within two shells of encryption that defeated the tools that Porras usually applied. By about a week before Christmas, however, his team and others – including the Russian security firm Kaspersky Labs, based in Moscow – had exposed the worm’s inner workings, and had found a list of all the URLs it would contact.

[Rick Wesson of Support Intelligence] has years of experience with the organisations that handle domain registration, and within days of getting Porras’s list he had set up a system to remove the tainted URLs, using his own money to buy them up.

It seemed like a major win, but the hackers were quick to bounce back: on 29 December, they started again from scratch by releasing an upgraded version of the worm that exploited the same security loophole.

This new worm had an impressive array of new tricks. Some were simple. As well as propagating via the internet, the worm hopped on to USB drives plugged into an infected computer. When those drives were later connected to a different machine, it hopped off again. The worm also blocked access to some security websites: when an infected user tried to go online and download the Microsoft patch against it, they got a “site not found” message.

Other innovations revealed the sophistication of Conficker’s creators. If the encryption used for the previous strain was tough, that of the new version seemed virtually bullet-proof. It was based on code little known outside academia that had been released just three months earlier by researchers at the Massachusetts Institute of Technology.

Indeed, worse was to come. On 15 March, Conficker presented the security experts with a new problem. It reached out to a URL called rmpezrx.org. It was on the list that Porras had produced, but – those involved decline to say why – it had not been blocked. One site was all that the hackers needed. A new version was waiting there to be downloaded by all the already infected computers, complete with another new box of tricks.

Now the cat-and-mouse game became clear. Conficker’s authors had discerned Porras and Wesson’s strategy and so from 1 April, the code of the new worm soon revealed, it would be able to start scanning for updates on 500 URLs selected at random from a list of 50,000 that were encoded in it. The range of suffixes would increase to 116 and include many country codes, such as .kz for Kazakhstan and .ie for Ireland. Each country-level suffix belongs to a different national authority, each of which sets its own registration procedures. Blocking the previous set of domains had been exhausting. It would soon become nigh-on impossible – even if the new version of the worm could be fully decrypted.

Luckily, Porras quickly repeated his feat and extracted the crucial list of URLs. Immediately, Wesson and others contacted the Internet Corporation for Assigned Names and Numbers (ICANN), an umbrella body that coordinates country suffixes.

From the second version onwards, Conficker had come with a much more efficient option: peer-to-peer (P2P) communication. This technology, widely used to trade pirated copies of software and films, allows software to reach out and exchange signals with copies of itself.

Six days after the 1 April deadline, Conficker’s authors let loose a new version of the worm via P2P. With no central release point to target, security experts had no means of stopping it spreading through the worm’s network. The URL scam seems to have been little more than a wonderful way to waste the anti-hackers’ time and resources. “They said: you’ll have to look at 50,000 domains. But they never intended to use them,” says Joe Stewart of SecureWorks in Atlanta, Georgia. “They used peer-to-peer instead. They misdirected us.”

The latest worm release had a few tweaks, such as blocking the action of software designed to scan for its presence. But piggybacking on it was something more significant: the worm’s first moneymaking schemes. These were a spam program called Waledac and a fake antivirus package named Spyware Protect 2009.

The same goes for fake software: when the accounts of a Russian company behind an antivirus scam became public last year, it appeared that one criminal had earned more than $145,000 from it in just 10 days.

How security experts defended against Conficker Read More »

Steve Jobs on mediocrity & market share

From Steven Levy’s “OK, Mac, Make a Wish: Apple’s ‘computer for the rest of us’ is, insanely, 20” (Newsweek: 2 February 2004):

If that’s so, then why is the Mac market share, even after Apple’s recent revival, sputtering at a measly 5 percent? Jobs has a theory about that, too. Once a company devises a great product, he says, it has a monopoly in that realm, and concentrates less on innovation than protecting its turf. “The Mac user interface was a 10-year monopoly,” says Jobs. “Who ended up running the company? Sales guys. At the critical juncture in the late ’80s, when they should have gone for market share, they went for profits. They made obscene profits for several years. And their products became mediocre. And then their monopoly ended with Windows 95. They behaved like a monopoly, and it came back to bite them, which always happens.”

Steve Jobs on mediocrity & market share Read More »

Why everyone wants a computer: socializing

From Paul Graham’s “Why TV Lost” (Paul Graham: March 2009):

The somewhat more surprising force was one specific type of innovation: social applications. The average teenage kid has a pretty much infinite capacity for talking to their friends. But they can’t physically be with them all the time. When I was in high school the solution was the telephone. Now it’s social networks, multiplayer games, and various messaging applications. The way you reach them all is through a computer. Which means every teenage kid (a) wants a computer with an Internet connection, (b) has an incentive to figure out how to use it, and (c) spends countless hours in front of it.

This was the most powerful force of all. This was what made everyone want computers. Nerds got computers because they liked them. Then gamers got them to play games on. But it was connecting to other people that got everyone else: that’s what made even grandmas and 14 year old girls want computers.

Why everyone wants a computer: socializing Read More »

An analysis of Google’s technology, 2005

From Stephen E. Arnold’s The Google Legacy: How Google’s Internet Search is Transforming Application Software (Infonortics: September 2005):

The figure Google’s Fusion: Hardware and Software Engineering shows that Google’s technology framework has two areas of activity. There is the software engineering effort that focuses on PageRank and other applications. Software engineering, as used here, means writing code and thinking about how computer systems operate in order to get work done quickly. Quickly means the sub one-second response times that Google is able to maintain despite its surging growth in usage, applications and data processing.

Google is hardware plus software

The other effort focuses on hardware. Google has refined server racks, cable placement, cooling devices, and data center layout. The payoff is lower operating costs and the ability to scale as demand for computing resources increases. With faster turnaround and the elimination of such troublesome jobs as backing up data, Google’s hardware innovations give it a competitive advantage few of its rivals can equal as of mid-2005.

How Google Is Different from MSN and Yahoo

Google’s technologyis simultaneously just like other online companies’ technology, and very different. A data center is usually a facility owned and operated by a third party where customers place their servers. The staff of the data center manage the power, air conditioning and routine maintenance. The customer specifies the computers and components. When a data center must expand, the staff of the facility may handle virtually all routine chores and may work with the customer’s engineers for certain more specialized tasks.

Before looking at some significant engineering differences between Google and two of its major competitors, review this list of characteristics for a Google data center.

1. Google data centers – now numbering about two dozen, although no one outside Google knows the exact number or their locations. They come online and automatically, under the direction of the Google File System, start getting work from other data centers. These facilities, sometimes filled with 10,000 or more Google computers, find one another and configure themselves with minimal human intervention.

2. The hardware in a Google data center can be bought at a local computer store. Google uses the same types of memory, disc drives, fans and power supplies as those in a standard desktop PC.

3. Each Google server comes in a standard case called a pizza box with one important change: the plugs and ports are at the front of the box to make access faster and easier.

4. Google racks are assembled for Google to hold servers on their front and back sides. This effectively allows a standard rack, normally holding 40 pizza box servers, to hold 80.

5. A Google data center can go from a stack of parts to online operation in as little as 72 hours, unlike more typical data centers that can require a week or even a month to get additional resources online.

6. Each server, rack and data center works in a way that is similar to what is called “plug and play.” Like a mouse plugged into the USB port on a laptop, Google’s network of data centers knows when more resources have been connected. These resources, for the most part, go into operation without human intervention.

Several of these factors are dependent on software. This overlap between the hardware and software competencies at Google, as previously noted, illustrates the symbiotic relationship between these two different engineering approaches. At Google, from its inception, Google software and Google hardware have been tightly coupled. Google is not a software company nor is it a hardware company. Google is, like IBM, a company that owes its existence to both hardware and software. Unlike IBM, Google has a business model that is advertiser supported. Technically, Google is conceptually closer to IBM (at one time a hardware and software company) than it is to Microsoft (primarily a software company) or Yahoo! (an integrator of multiple softwares).

Software and hardware engineering cannot be easily segregated at Google. At MSN and Yahoo hardware and software are more loosely-coupled. Two examples will illustrate these differences.

Microsoft – with some minor excursions into the Xbox game machine and peripherals – develops operating systems and traditional applications. Microsoft has multiple operating systems, and its engineers are hard at work on the company’s next-generation of operating systems.

Several observations are warranted:

1. Unlike Google, Microsoft does not focus on performance as an end in itself. As a result, Microsoft gets performance the way most computer users do. Microsoft buys or upgrades machines. Microsoft does not fiddle with its operating systems and their subfunctions to get that extra time slice or two out of the hardware.

2. Unlike Google, Microsoft has to support many operating systems and invest time and energy in making certain that important legacy applications such as Microsoft Office or SQLServer can run on these new operating systems. Microsoft has a boat anchor tied to its engineer’s ankles. The boat anchor is the need to ensure that legacy code works in Microsoft’s latest and greatest operating systems.

3. Unlike Google, Microsoft has no significant track record in designing and building hardware for distributed, massively parallelised computing. The mice and keyboards were a success. Microsoft has continued to lose money on the Xbox, and the sudden demise of Microsoft’s entry into the home network hardware market provides more evidence that Microsoft does not have a hardware competency equal to Google’s.

Yahoo! operates differently from both Google and Microsoft. Yahoo! is in mid-2005 a direct competitor to Google for advertising dollars. Yahoo! has grown through acquisitions. In search, for example, Yahoo acquired 3721.com to handle Chinese language search and retrieval. Yahoo bought Inktomi to provide Web search. Yahoo bought Stata Labs in order to provide users with search and retrieval of their Yahoo! mail. Yahoo! also owns AllTheWeb.com, a Web search site created by FAST Search & Transfer. Yahoo! owns the Overture search technology used by advertisers to locate key words to bid on. Yahoo! owns Alta Vista, the Web search system developed by Digital Equipment Corp. Yahoo! licenses InQuira search for customer support functions. Yahoo has a jumble of search technology; Google has one search technology.

Historically Yahoo has acquired technology companies and allowed each company to operate its technology in a silo. Integration of these different technologies is a time-consuming, expensive activity for Yahoo. Each of these software applications requires servers and systems particular to each technology. The result is that Yahoo has a mosaic of operating systems, hardware and systems. Yahoo!’s problem is different from Microsoft’s legacy boat-anchor problem. Yahoo! faces a Balkan-states problem.

There are many voices, many needs, and many opposing interests. Yahoo! must invest in management resources to keep the peace. Yahoo! does not have a core competency in hardware engineering for performance and consistency. Yahoo! may well have considerable competency in supporting a crazy-quilt of hardware and operating systems, however. Yahoo! is not a software engineering company. Its engineers make functions from disparate systems available via a portal.

The figure below provides an overview of the mid-2005 technical orientation of Google, Microsoft and Yahoo.

2005 focuses of Google, MSN, and Yahoo

The Technology Precepts

… five precepts thread through Google’s technical papers and presentations. The following snapshots are extreme simplifications of complex, yet extremely fundamental, aspects of the Googleplex.

Cheap Hardware and Smart Software

Google approaches the problem of reducing the costs of hardware, set up, burn-in and maintenance pragmatically. A large number of cheap devices using off-the-shelf commodity controllers, cables and memory reduces costs. But cheap hardware fails.

In order to minimize the “cost” of failure, Google conceived of smart software that would perform whatever tasks were needed when hardware devices fail. A single device or an entire rack of devices could crash, and the overall system would not fail. More important, when such a crash occurs, no full-time systems engineering team has to perform technical triage at 3 a.m.

The focus on low-cost, commodity hardware and smart software is part of the Google culture.

Logical Architecture

Google’s technical papers do not describe the architecture of the Googleplex as self-similar. Google’s technical papers provide tantalizing glimpses of an approach to online systems that makes a single server share features and functions of a cluster of servers, a complete data center, and a group of Google’s data centers.

The collections of servers running Google applications on the Google version of Linux is a supercomputer. The Googleplex can perform mundane computing chores like taking a user’s query and matching it to documents Google has indexed. Further more, the Googleplex can perform side calculations needed to embed ads in the results pages shown to user, execute parallelized, high-speed data transfers like computers running state-of-the-art storage devices, and handle necessary housekeeping chores for usage tracking and billing.

When Google needs to add processing capacity or additional storage, Google’s engineers plug in the needed resources. Due to self-similarity, the Googleplex can recognize, configure and use the new resource. Google has an almost unlimited flexibility with regard to scaling and accessing the capabilities of the Googleplex.

In Google’s self-similar architecture, the loss of an individual device is irrelevant. In fact, a rack or a data center can fail without data loss or taking the Googleplex down. The Google operating system ensures that each file is written three to six times to different storage devices. When a copy of that file is not available, the Googleplex consults a log for the location of the copies of the needed file. The application then uses that replica of the needed file and continues with the job’s processing.

Speed and Then More Speed

Google uses commodity pizza box servers organized in a cluster. A cluster is group of computers that are joined together to create a more robust system. Instead of using exotic servers with eight or more processors, Google generally uses servers that have two processors similar to those found in a typical home computer.

Through proprietary changes to Linux and other engineering innovations, Google is able to achieve supercomputer performance from components that are cheap and widely available.

… engineers familiar with Google believe that read rates may in some clusters approach 2,000 megabytes a second. When commodity hardware gets better, Google runs faster without paying a premium for that performance gain.

Another key notion of speed at Google concerns writing computer programs to deploy to Google users. Google has developed short cuts to programming. An example is Google’s creating a library of canned functions to make it easy for a programmer to optimize a program to run on the Googleplex computer. At Microsoft or Yahoo, a programmer must write some code or fiddle with code to get different pieces of a program to execute simultaneously using multiple processors. Not at Google. A programmer writes a program, uses a function from a Google bundle of canned routines, and lets the Googleplex handle the details. Google’s programmers are freed from much of the tedium associated with writing software for a distributed, parallel computer.

Eliminate or Reduce Certain System Expenses

Some lucky investors jumped on the Google bandwagon early. Nevertheless, Google was frugal, partly by necessity and partly by design. The focus on frugality influenced many hardware and software engineering decisions at the company.

Drawbacks of the Googleplex

The Laws of Physics: Heat and Power 101

In reality, no one knows. Google has a rapidly expanding number of data centers. The data center near Atlanta, Georgia, is one of the newest deployed. This state-of-the-art facility reflects what Google engineers have learned about heat and power issues in its other data centers. Within the last 12 months, Google has shifted from concentrating its servers at about a dozen data centers, each with 10,000 or more servers, to about 60 data centers, each with fewer machines. The change is a response to the heat and power issues associated with larger concentrations of Google servers.

The most failure prone components are:

  • Fans.
  • IDE drives which fail at the rate of one per 1,000 drives per day.
  • Power supplies which fail at a lower rate.

Leveraging the Googleplex

Google’s technology is one major challenge to Microsoft and Yahoo. So to conclude this cursory and vastly simplified look at Google technology, consider these items:

1. Google is fast anywhere in the world.

2. Google learns. When the heat and power problems at dense data centers surfaced, Google introduced cooling and power conservation innovations to its two dozen data centers.

3. Programmers want to work at Google. “Google has cachet,” said one recent University of Washington graduate.

4. Google’s operating and scaling costs are lower than most other firms offering similar businesses.

5. Google squeezes more work out of programmers and engineers by design.

6. Google does not break down, or at least it has not gone offline since 2000.

7. Google’s Googleplex can deliver desktop-server applications now.

8. Google’s applications install and update without burdening the user with gory details and messy crashes.

9. Google’s patents provide basic technology insight pertinent to Google’s core functionality.

An analysis of Google’s technology, 2005 Read More »

Lots of good info about the FBI’s far-reaching wiretapping of US phone systems

From Ryan Singel’s “Point, Click … Eavesdrop: How the FBI Wiretap Net Operates” (Wired News: 29 August 2007):

The FBI has quietly built a sophisticated, point-and-click surveillance system that performs instant wiretaps on almost any communications device, according to nearly a thousand pages of restricted documents newly released under the Freedom of Information Act.

The surveillance system, called DCSNet, for Digital Collection System Network, connects FBI wiretapping rooms to switches controlled by traditional land-line operators, internet-telephony providers and cellular companies. It is far more intricately woven into the nation’s telecom infrastructure than observers suspected.

It’s a “comprehensive wiretap system that intercepts wire-line phones, cellular phones, SMS and push-to-talk systems,” says Steven Bellovin, a Columbia University computer science professor and longtime surveillance expert.

DCSNet is a suite of software that collects, sifts and stores phone numbers, phone calls and text messages. The system directly connects FBI wiretapping outposts around the country to a far-reaching private communications network.

The $10 million DCS-3000 client, also known as Red Hook, handles pen-registers and trap-and-traces, a type of surveillance that collects signaling information — primarily the numbers dialed from a telephone — but no communications content. (Pen registers record outgoing calls; trap-and-traces record incoming calls.)

DCS-6000, known as Digital Storm, captures and collects the content of phone calls and text messages for full wiretap orders.

A third, classified system, called DCS-5000, is used for wiretaps targeting spies or terrorists.

What DCSNet Can Do

Together, the surveillance systems let FBI agents play back recordings even as they are being captured (like TiVo), create master wiretap files, send digital recordings to translators, track the rough location of targets in real time using cell-tower information, and even stream intercepts outward to mobile surveillance vans.

FBI wiretapping rooms in field offices and undercover locations around the country are connected through a private, encrypted backbone that is separated from the internet. Sprint runs it on the government’s behalf.

The network allows an FBI agent in New York, for example, to remotely set up a wiretap on a cell phone based in Sacramento, California, and immediately learn the phone’s location, then begin receiving conversations, text messages and voicemail pass codes in New York. With a few keystrokes, the agent can route the recordings to language specialists for translation.

The numbers dialed are automatically sent to FBI analysts trained to interpret phone-call patterns, and are transferred nightly, by external storage devices, to the bureau’s Telephone Application Database, where they’re subjected to a type of data mining called link analysis.

The numerical scope of DCSNet surveillance is still guarded. But we do know that as telecoms have become more wiretap-friendly, the number of criminal wiretaps alone has climbed from 1,150 in 1996 to 1,839 in 2006. That’s a 60 percent jump. And in 2005, 92 percent of those criminal wiretaps targeted cell phones, according to a report published last year.

These figures include both state and federal wiretaps, and do not include antiterrorism wiretaps, which dramatically expanded after 9/11. They also don’t count the DCS-3000’s collection of incoming and outgoing phone numbers dialed. Far more common than full-blown wiretaps, this level of surveillance requires only that investigators certify that the phone numbers are relevant to an investigation.

In the 1990s, the Justice Department began complaining to Congress that digital technology, cellular phones and features like call forwarding would make it difficult for investigators to continue to conduct wiretaps. Congress responded by passing the Communications Assistance for Law Enforcement Act, or CALEA, in 1994, mandating backdoors in U.S. telephone switches.

CALEA requires telecommunications companies to install only telephone-switching equipment that meets detailed wiretapping standards. Prior to CALEA, the FBI would get a court order for a wiretap and present it to a phone company, which would then create a physical tap of the phone system.

With new CALEA-compliant digital switches, the FBI now logs directly into the telecom’s network. Once a court order has been sent to a carrier and the carrier turns on the wiretap, the communications data on a surveillance target streams into the FBI’s computers in real time.

The released documents suggest that the FBI’s wiretapping engineers are struggling with peer-to-peer telephony provider Skype, which offers no central location to wiretap, and with innovations like caller-ID spoofing and phone-number portability.

Despite its ease of use, the new technology is proving more expensive than a traditional wiretap. Telecoms charge the government an average of $2,200 for a 30-day CALEA wiretap, while a traditional intercept costs only $250, according to the Justice Department inspector general. A federal wiretap order in 2006 cost taxpayers $67,000 on average, according to the most recent U.S. Court wiretap report.

What’s more, under CALEA, the government had to pay to make pre-1995 phone switches wiretap-friendly. The FBI has spent almost $500 million on that effort, but many traditional wire-line switches still aren’t compliant.

Processing all the phone calls sucked in by DCSNet is also costly. At the backend of the data collection, the conversations and phone numbers are transferred to the FBI’s Electronic Surveillance Data Management System, an Oracle SQL database that’s seen a 62 percent growth in wiretap volume over the last three years — and more than 3,000 percent growth in digital files like e-mail. Through 2007, the FBI has spent $39 million on the system, which indexes and analyzes data for agents, translators and intelligence analysts.

Lots of good info about the FBI’s far-reaching wiretapping of US phone systems Read More »

6 reasons why “content” has been devalued

From Jonathan Handel’s “Is Content Worthless?” (The Huffington Post: 11 April 2008):

Everyone focuses on piracy, but there are actually six related reasons for the devaluation of content. The first is supply and demand. Demand — the number of consumers and their available leisure time – is relatively constant, but supply — online content — has grown enormously in the last decade. Some of this is professional content set free from boundaries of time and space, now available worldwide, anytime, and usually at no cost (whether legally or not). Even more is user generated content (UGC) — websites, blogs, YouTube videos — created by non-professionals who don’t care whether they get paid, and who themselves pay little or nothing to create and distribute it.

The second is the loss of physical form. It just seems natural to value a physical thing more highly than something intangible. Physical objects have been with us since the beginning of time; distributable intangible content has not. Perhaps for that reason, we tend to focus on per-unit costs (zero for an intangible such as a movie download), while forgetting about fixed costs (such as the cost of making the movie in the first place). Also, and critically, if you steal something tangible, you deny it to the owner; a purloined DVD is no longer available to the merchant, for instance. But if you misappropriate an intangible, it’s still there for others to use. …

The third reason is that acquiring content is increasingly frictionless. It’s often easier, particularly for young people, to access content on the Internet than through traditional means. …

Fourth is that most new media business models are ad-supported rather than pay per view or subscription. If there’s no cost to the user, why should consumers see the content as valuable, and if some content is free, why not all of it? …

Fifth is market forces in the technology industry. Computers, web services, and consumer electronic devices are more valuable when more content is available. In turn, these products make content more usable by providing new distribution channels. Traditional media companies are slow to adopt these new technologies, for fear of cannibalizing revenue from existing channels and offending powerful distribution partners. In contrast, non-professionals, long denied access to distribution, rush to use the new technologies, as do pirates of professional content. As a result, technological innovation reduces the market share of paid professional content.

Finally, there’s culture. A generation of users has grown up indifferent or hostile to copyright, particularly in music, movies and software.

6 reasons why “content” has been devalued Read More »

Examples of tweaking old technologies to add social aspects

From Clay Shirky’s “Group as User: Flaming and the Design of Social Software” (Clay Shirky’s Writings About the Internet: 5 November 2004):

This possibility of adding novel social components to old tools presents an enormous opportunity. To take the most famous example, the Slashdot moderation system puts the ability to rate comments into the hands of the users themselves. The designers took the traditional bulletin board format — threaded posts, sorted by time — and added a quality filter. And instead of assuming that all users are alike, the Slashdot designers created a karma system, to allow them to discriminate in favor of users likely to rate comments in ways that would benefit the community. And, to police that system, they created a meta-moderation system, to solve the ‘Who will guard the guardians’ problem. …

Likewise, Craigslist took the mailing list, and added a handful of simple features with profound social effects. First, all of Craigslist is an enclosure, owned by Craig … Because he has a business incentive to make his list work, he and his staff remove posts if enough readers flag them as inappropriate. …

And, on the positive side, the addition of a “Nominate for ‘Best of Craigslist'” button in every email creates a social incentive for users to post amusing or engaging material. … The only reason you would nominate a post for ‘Best of’ is if you wanted other users to see it — if you were acting in a group context, in other words. …

Jonah Brucker-Cohen’s Bumplist stands out as an experiment in experimenting the social aspect of mailing lists. Bumplist, whose motto is “an email community for the determined”, is a mailing list for 6 people, which anyone can join. When the 7th user joins, the first is bumped and, if they want to be back on, must re-join, bumping the second user, ad infinitum. … However, it is a vivid illustration of the ways simple changes to well-understood software can produce radically different social effects.

You could easily imagine many such experiments. What would it take, for example, to design a mailing list that was flame-retardant? Once you stop regarding all users as isolated actors, a number of possibilities appear. You could institute induced lag, where, once a user contributed 5 posts in the space of an hour, a cumulative 10 minute delay would be added to each subsequent post. Every post would be delivered eventually, but it would retard the rapid-reply nature of flame wars, introducing a cooling off period for the most vociferous participants.

You could institute a kind of thread jail, where every post would include a ‘Worst of’ button, in the manner of Craigslist. Interminable, pointless threads (e.g. Which Operating System Is Objectively Best?) could be sent to thread jail if enough users voted them down. (Though users could obviously change subject headers and evade this restriction, the surprise, first noted by Julian Dibbell, is how often users respect negative communal judgment, even when they don’t respect the negative judgment of individuals. [ See Rape in Cyberspace — search for “aggressively antisocial vibes.”])

You could institute a ‘Get a room!’ feature, where any conversation that involved two users ping-ponging six or more posts (substitute other numbers to taste) would be automatically re-directed to a sub-list, limited to that pair. The material could still be archived, and so accessible to interested lurkers, but the conversation would continue without the attraction of an audience.

You could imagine a similar exercise, working on signal/noise ratios generally, and keying off the fact that there is always a most active poster on mailing lists, who posts much more often than even the second most active, and much much more often than the median poster. Oddly, the most active poster is often not even aware that they occupy this position (seeing ourselves as others see us is difficult in mediated spaces as well,) but making them aware of it often causes them to self-moderate. You can imagine flagging all posts by the most active poster, whoever that happened to be, or throttling the maximum number of posts by any user to some multiple of average posting tempo.

Examples of tweaking old technologies to add social aspects Read More »

Word of the day: creative destruction

From Wikipedia’s “Creative destruction” (13 July 2006):

Creative destruction, introduced by the economist Joseph Schumpeter, describes the process of industrial transformation that accompanies radical innovation. In Schumpeter’s vision of capitalism, innovative entry by entrepreneurs was the force that sustained long-term economic growth, even as it destroyed the value of established companies that enjoyed some degree of monopoly power. …

There are numerous types of innovation generating creative destruction in an industry:

New markets or products
New equipment
New sources of labor and raw materials
New methods of organization or management
New methods of inventory management
New methods of transportation
New methods of communication (e.g., the Internet)
New methods of advertising and marketing
New financial instruments
New ways to lobby politicians or new legal strategies

Word of the day: creative destruction Read More »

Prescription drug spending has vastly increased in 25 years

From Clifton Leaf’s “The Law of Unintended Consequences” (Fortune: 19 September 2005):

Whatever the answer, it’s clear who pays for it. You do. You pay in the form of vastly higher drug prices and health-care insurance. Americans spent $179 billion on prescription drugs in 2003. That’s up from … wait for it … $12 billion in 1980 [when the Bayh-Dole Act was passed]. That’s a 13% hike, year after year, for two decades. Of course, what you don’t pay as a patient you pay as a taxpayer. The U.S. government picks up the tab for one in three Americans by way of Medicare, Medicaid, the military, and other programs. According to the provisions of Bayh-Dole, the government gets a royalty-free use, forever, of its funded inventions. It has never tried to collect. You might say the taxpayers pay for the hat–and have it handed to them.

Prescription drug spending has vastly increased in 25 years Read More »

What patents on life has wrought

From Clifton Leaf’s “The Law of Unintended Consequences” (Fortune: 19 September 2005):

The Supreme Court’s decision in 1980 to allow for the patenting of living organisms opened the spigots to individual claims of ownership over everything from genes and protein receptors to biochemical pathways and processes. Soon, research scientists were swooping into patent offices around the world with “invention” disclosures that weren’t so much products or processes as they were simply knowledge–or research tools to further knowledge.

The problem is, once it became clear that individuals could own little parcels of biology or chemistry, the common domain of scientific exchange–that dynamic place where theories are introduced, then challenged, and ultimately improved–begins to shrink. What’s more, as the number of claims grows, so do the overlapping claims and legal challenges. …

In October 1990 a researcher named Mary-Claire King at the University of California at Berkeley told the world that there was a breast-cancer susceptibility gene–and that it was on chromosome 17. Several other groups, sifting through 30 million base pairs of nucleotides to find the precise location of the gene, helped narrow the search with each new discovery. Then, in the spring of 1994, a team led by Mark Skolnick at the University of Utah beat everyone to the punch–identifying a gene with 5,592 base pairs and codes for a protein that was nearly 1,900 amino acids long. Skolnick’s team rushed to file a patent application and was issued title to the discovery three years later.

By all accounts the science was a collective effort. The NIH had funded scores of investigative teams around the country and given nearly 1,200 separate research grants to learn everything there was to learn about the genetics of breast cancer.

The patent, however, is licensed to one company–Skolnick’s. Myriad Genetics, a company the researcher founded in 1991, now insists on doing all U.S. testing for the presence of unknown mutation in the two related genes, BRCA1 and BRCA2. Those who have a mutation in either gene have as high as an 86% chance of getting cancer, say experts. The cost for the complete two-gene analysis: $2,975.

Critics say that Myriad’s ultrarestrictive licensing of the technology–one funded not only by federal dollars but also aided by the prior discoveries of hundreds of other scientists–is keeping the price of the test artificially high. Skolnick, 59, claims that the price is justified by his company’s careful analysis of thousands of base pairs of DNA, each of which is prone to a mutation or deletion, and by its educational outreach programs.

What patents on life has wrought Read More »

1980 Bayh-Dole Act created the biotech industry … & turned universities into businesses

From Clifton Leaf’s “The Law of Unintended Consequences” (Fortune: 19 September 2005):

For a century or more, the white-hot core of American innovation has been basic science. And the foundation of basic science has been the fluid exchange of ideas at the nation’s research universities. It has always been a surprisingly simple equation: Let scientists do their thing and share their work–and industry picks up the spoils. Academics win awards, companies make products, Americans benefit from an ever-rising standard of living.

That equation still holds, with the conspicuous exception of medical research. In this one area, something alarming has been happening over the past 25 years: Universities have evolved from public trusts into something closer to venture capital firms. What used to be a scientific community of free and open debate now often seems like a litigious scrum of data-hoarding and suspicion. And what’s more, Americans are paying for it through the nose. …

From 1992 to September 2003, pharmaceutical companies tied up the federal courts with 494 patent suits. That’s more than the number filed in the computer hardware, aerospace, defense, and chemical industries combined. Those legal expenses are part of a giant, hidden “drug tax”–a tax that has to be paid by someone. And that someone, as you’ll see below, is you. You don’t get the tab all at once, of course. It shows up in higher drug costs, higher tuition bills, higher taxes–and tragically, fewer medical miracles.

So how did we get to this sorry place? It was one piece of federal legislation that you’ve probably never heard of–a 1980 tweak to the U.S. patent and trademark law known as the Bayh-Dole Act. That single law, named for its sponsors, Senators Birch Bayh and Bob Dole, in essence transferred the title of all discoveries made with the help of federal research grants to the universities and small businesses where they were made.

Prior to the law’s enactment, inventors could always petition the government for the patent rights to their own work, though the rules were different at each federal agency; some 20 different statutes governed patent policy. The law simplified the “technology transfer” process and, more important, changed the legal presumption about who ought to own and develop new ideas–private enterprise as opposed to Uncle Sam. The new provisions encouraged academic institutions to seek out the clever ideas hiding in the backs of their research cupboards and to pursue licenses with business. And it told them to share some of the take with the actual inventors.

On the face of it, Bayh-Dole makes sense. Indeed, supporters say the law helped create the $43-billion-a-year biotech industry and has brought valuable drugs to market that otherwise would never have seen the light of day. What’s more, say many scholars, the law has created megaclusters of entrepreneurial companies–each an engine for high-paying, high-skilled jobs–all across the land.

That all sounds wonderful. Except that Bayh-Dole’s impact wasn’t so much in the industry it helped create, but rather in its unintended consequence–a legal frenzy that’s diverting scientists from doing science. …

A 1979 audit of government-held patents showed that fewer than 5% of some 28,000 discoveries–all of them made with the help of taxpayer money–had been developed, because no company was willing to risk the capital to commercialize them without owning title. …

A dozen schools–notably MIT, Stanford, the University of California, Johns Hopkins, and the University of Wisconsin–already had campus offices to work out licensing arrangements with government agencies and industry. But within a few years Technology Licensing Offices (or TLOs) were sprouting up everywhere. In 1979, American universities received 264 patents. By 1991, when a new organization, the Association of University Technology Managers, began compiling data, North American institutions (including colleges, research institutes, and hospitals) had filed 1,584 new U.S. patent applications and negotiated 1,229 licenses with industry–netting $218 million in royalties. By 2003 such institutions had filed five times as many new patent applications; they’d done 4,516 licensing deals and raked in over $1.3 billion in income. And on top of all that, 374 brand-new companies had sprouted from the wells of university research. That meant jobs pouring back into the community …

The anecdotal reports, fun “discovery stories” in alumni magazines, and numbers from the yearly AUTM surveys suggested that the academic productivity marvel had spread far and wide. But that’s hardly the case. Roughly a third of the new discoveries and more than half of all university licensing income in 2003 derived from just ten schools–MIT, Stanford, the usual suspects. They are, for the most part, the institutions that were pursuing “technology transfer” long before Bayh-Dole. …

Court dockets are now clogged with university patent claims. In 2002, North American academic institutions spent over $200 million in litigation (though some of that was returned in judgments)–more than five times the amount spent in 1991. Stanford Law School professor emeritus John Barton notes, in a 2000 study published in Science, that the indicator that correlates most perfectly with the rise in university patents is the number of intellectual-property lawyers. (Universities also spent $142 million on lobbying over the past six years.) …

So what do universities do with all their cash? That depends. Apart from the general guidelines provided by Bayh-Dole, which indicate the proceeds must be used for “scientific research or education,” there are no instructions. “These are unrestricted dollars that they can use, and so they’re worth a lot more than other dollars,” says University of Michigan law professor Rebecca Eisenberg, who has written extensively about the legislation. The one thing no school seems to use the money for is tuition–which apparently has little to do with “scientific research or education.” Meanwhile, the cost of university tuition has soared at a rate more than twice as high as inflation from 1980 to 2005.

1980 Bayh-Dole Act created the biotech industry … & turned universities into businesses Read More »

Antitrust suits led to vertical integration & the IT revolution

From Barry C. Lynn’s “The Case for Breaking Up Wal-Mart” (Harper’s: 24 July 2006):

As the industrial scholar Alfred D. Chandler has noted, the vertically integrated firm — which dominated the American economy for most of the last century — was to a great degree the product of antitrust enforcement. When Theodore Roosevelt began to limit the ability of large companies to grow horizontally, many responded by buying outside suppliers and integrating their operations into vertical lines of production. Many also set up internal research labs to improve existing products and develop new ones. Antitrust law later played a huge role in launching the information revolution. During the Cold War, the Justice Department routinely used antitrust suits to force high-tech firms to share the technologies they had developed. Targeted firms like IBM, RCA, AT&T, and Xerox spilled many thousands of patents onto the market, where they were available to any American competitor for free.

Antitrust suits led to vertical integration & the IT revolution Read More »

AACS, next-gen encryption for DVDs

From Nate Anderson’s “Hacking Digital Rights Management” (Ars Technica: 18 July 2006):

AACS relies on the well-established AES (with 128-bit keys) to safeguard the disc data. Just like DVD players, HD DVD and Blu-ray drives will come with a set of Device Keys handed out to the manufacturers by AACS LA. Unlike the CSS encryption used in DVDs, though, AACS has a built-in method for revoking sets of keys that are cracked and made public. AACS-encrypted discs will feature a Media Key Block that all players need to access in order to get the key needed to decrypt the video files on the disc. The MKB can be updated by AACS LA to prevent certain sets of Device Keys from functioning with future titles – a feature that AACS dubs “revocation.” …

AACS also supports a new feature called the Image Constraint Token. When set, the ICT will force video output to be degraded over analog connections. ICT has so far gone unused, though this could change at any time. …

While AACS is used by both HD disc formats, the Blu-ray Disc Association (BDA) has added some features of its own to make the format “more secure” than HD DVD. The additions are BD+ and ROM Mark; though both are designed to thwart pirates, they work quite differently.

While the generic AACS spec includes key revocation, BD+ actually allows the BDA to update the entire encryption system once players have already shipped. Should encryption be cracked, new discs will include information that will alter the players’ decryption code. …

The other new technology, ROM Mark, affects the manufacturing of Blu-ray discs. All Blu-ray mastering equipment must be licensed by the BDA, and they will ensure that all of it carries ROM Mark technology. Whenever a legitimate disc is created, it is given a “unique and undetectable identifier.” It’s not undetectable to the player, though, and players can refuse to play discs without a ROM Mark. The BDA has the optimistic hope that this will keep industrial-scale piracy at bay. We’ll see.

AACS, next-gen encryption for DVDs Read More »

To combat phishing, change browser design philosophy

From Federico Biancuzzi’s “Phishing with Rachna Dhamija” (SecurityFocus: 19 June 2006):

We discovered that existing security cues are ineffective, for three reasons:

1. The indicators are ignored (23% of participants in our study did not look at the address bar, status bar, or any SSL indicators).

2. The indicators are misunderstood. For example, one regular Firefox user told me that he thought the yellow background in the address bar was an aesthetic design choice of the website designer (he didn’t realize that it was a security signal presented by the browser). Other users thought the SSL lock icon indicated whether a website could set cookies.

3. The security indicators are trivial to spoof. Many users can’t distinguish between an actual SSL indicator in the browser frame and a spoofed image of that indicator that appears in the content of a webpage. For example, if you display a popup window with no address bar, and then add an image of an address bar at the top with the correct URL and SSL indicators and an image of the status bar at the bottom with all the right indicators, most users will think it is legitimate. This attack fooled more than 80% of participants. …

Currently, I’m working on other techniques to prevent phishing in conjunction with security skins. For example, in a security usability class I taught this semester at Harvard, we conducted a usability study that shows that simply showing a user’s history information (for example, “you’ve been to this website many times” or “you’ve never submitted this form before”) can significantly increase a user’s ability to detect a spoofed website and reduce their vulnerability to phishing attacks. Another area I’ve been investigating are techniques to help users recover from errors and to identify when errors are real, or when they are simulated. Many attacks rely on users not being able to make this distinction.

You presented the project called Dynamic Security Skins (DSS) nearly one year ago. Do you think the main idea behind it is still valid after your tests?

Rachna Dhamija: I think that our usability study shows how easy it is to spoof security indicators, and how hard it is for users to distinguish legitimate security indicators from those that have been spoofed. Dynamic Security Skins is a proposal that starts from the assumption that any static security indicator can easily be copied by attacker. Instead, we propose that users create their own customized security indicators that are hard for an attacker to predict. Our usability study also shows that indicators placed in the periphery or outside of the user’s focus of attention (such as the SSL lock icon in the status bar) may be ignored entirely by some users. DSS places the security indicator (a secret image) at the point of password entry, so the user can not ignore it.

DSS adds a trusted window in the browser dedicated to username and password entry. The user chooses a photographic image (or is assigned a random image), which is overlaid across the window and text entry boxes. If the window displays the user’s personal image, it is safe for the user to enter his password. …

With security skins, we were trying to solve not user authentication, but the reverse problem – server authentication. I was looking for a way to convey to a user that his client and the server had successfully negotiated a protocol, that they have mutually authenticated each other and agreed on the same key. One way to do this would be to display a message like “Server X is authenticated”, or to display a binary indicator, like a closed or open lock. The problem is that any static indicator can be easily copied by an attacker. Instead, we allow the server and the user’s browser to each generate an abstract image. If the authentication is successful, the two images will match. This image can change with each authentication. If it is captured, it can’t be replayed by an attacker and it won’t reveal anything useful about the user’s password. …

Instead of blaming specific development techniques, I think we need to change our design philosophy. We should assume that every interface we develop will be spoofed. The only thing an attacker can’t simulate is an interface he can’t predict. This is the principle that DSS relies on. We should make it easy for users to personalize their interfaces. Look at how popular screensavers, ringtones, and application skins are – users clearly enjoy the ability to personalize their interfaces. We can take advantage of this fact to build spoof resistant interfaces.

To combat phishing, change browser design philosophy Read More »

Business, work, and good ideas

From Paul Graham’s “Why Smart People Have Bad Ideas” (April 2005):

This summer, as an experiment, some friends and I are giving seed funding to a bunch of new startups. It’s an experiment because we’re prepared to fund younger founders than most investors would. That’s why we’re doing it during the summer– so even college students can participate. …

The deadline has now passed, and we’re sifting through 227 applications. We expected to divide them into two categories, promising and unpromising. But we soon saw we needed a third: promising people with unpromising ideas. …

One of the most valuable things my father taught me is an old Yorkshire saying: where there’s muck, there’s brass. Meaning that unpleasant work pays. And more to the point here, vice versa. Work people like doesn’t pay well, for reasons of supply and demand. The most extreme case is developing programming languages, which doesn’t pay at all, because people like it so much they do it for free. …

So why were we afraid? We felt we were good at programming, but we lacked confidence in our ability to do a mysterious, undifferentiated thing we called “business.” In fact there is no such thing as “business.” There’s selling, promotion, figuring out what people want, deciding how much to charge, customer support, paying your bills, getting customers to pay you, getting incorporated, raising money, and so on. And the combination is not as hard as it seems, because some tasks (like raising money and getting incorporated) are an O(1) pain in the ass, whether you’re big or small, and others (like selling and promotion) depend more on energy and imagination than any kind of special training.

Business, work, and good ideas Read More »

Patenting is hurting scientific research & progress

From American Association for the Advancement of Science’s “The Effects of Patenting in the AAAS Scientific Community” [250 kb PDF] (2006):

Forty percent of respondents who had acquired patented technologies since January 2001 reported difficulties in obtaining those technologies. Industry bioscience respondents reported the most problems, with 76 percent reporting that their research had been affected by such difficulties. In contrast, only 35 percent of academic bioscience respondents reported difficulties that affected their research.

Of the 72 respondents who reported that their work had been affected by the technology acquisition process, 58 percent of those reported that their work was delayed. Fifty percent reported that they had to change their research, and 28 percent reported abandoning their research project as acquisition of the necessary technologies involved overly complex licensing negotiations.

Patenting is hurting scientific research & progress Read More »

Why big co’s are bad are creating new products

From Paul Graham’s “Hiring is Obsolete” (May 2005):

Buying startups also solves another problem afflicting big companies: they can’t do product development. Big companies are good at extracting the value from existing products, but bad at creating new ones.

Why? It’s worth studying this phenomenon in detail, because this is the raison d’etre of startups.

To start with, most big companies have some kind of turf to protect, and this tends to warp their development decisions. For example, Web-based applications are hot now, but within Microsoft there must be a lot of ambivalence about them, because the very idea of Web-based software threatens the desktop. So any Web-based application that Microsoft ends up with, will probably, like Hotmail, be something developed outside the company.

Another reason big companies are bad at developing new products is that the kind of people who do that tend not to have much power in big companies (unless they happen to be the CEO). Disruptive technologies are developed by disruptive people. And they either don’t work for the big company, or have been outmaneuvered by yes-men and have comparatively little influence.

Big companies also lose because they usually only build one of each thing. When you only have one Web browser, you can’t do anything really risky with it. If ten different startups design ten different Web browsers and you take the best, you’ll probably get something better.

The more general version of this problem is that there are too many new ideas for companies to explore them all. There might be 500 startups right now who think they’re making something Microsoft might buy. Even Microsoft probably couldn’t manage 500 development projects in-house.

Big companies also don’t pay people the right way. People developing a new product at a big company get paid roughly the same whether it succeeds or fails. People at a startup expect to get rich if the product succeeds, and get nothing if it fails. So naturally the people at the startup work a lot harder.

The mere bigness of big companies is an obstacle. In startups, developers are often forced to talk directly to users, whether they want to or not, because there is no one else to do sales and support. It’s painful doing sales, but you learn much more from trying to sell people something than reading what they said in focus groups.

And then of course, big companies are bad at product development because they’re bad at everything. Everything happens slower in big companies than small ones, and product development is something that has to happen fast, because you have to go through a lot of iterations to get something good.

Why big co’s are bad are creating new products Read More »

Jobs are unnecessary – just build something valuable

From Paul Graham’s “Hiring is Obsolete” (May 2005):

I think most undergrads don’t realize yet that the economic cage is open. A lot have been told by their parents that the route to success is to get a good job. This was true when their parents were in college, but it’s less true now. The route to success is to build something valuable, and you don’t have to be working for an existing company to do that. Indeed, you can often do it better if you’re not.

Jobs are unnecessary – just build something valuable Read More »