design

The psychology of waiting for your luggage at the airport

science / /

From Dan Ariely’s “Flying Frustrations” (21 November 2011):

Think about these two ways to get your luggage: With the original airport design, you walk ten minutes, but when you finally get to the carousel, your baggage gets there a minute after you (taking 11 minutes). In the other, you walk three minutes, but when you arrive you have to wait five minutes for your luggage (taking 8 minutes). The second scenario is faster, but people become more annoyed with the process because they have more idle time. As Sir Arthur Conan Doyle, Sr. noted, “I never remember feeling tired by work, though idleness exhausts me completely.”

The “good news” is that airports quickly reverted to their former (inefficient) system, and we now walk farther to our suitcases just to avoid the frustrations of idleness.

The psychology of waiting for your luggage at the airport Read More »

Steve Jobs, genius

art, business, history, technology / /

From Stephen Fry’s “Steve Jobs” (The New Adventures of Stephen Fry: 6 October 2011):

Henry Ford didn’t invent the motor car, Rockefeller didn’t discover how to crack crude oil into petrol, Disney didn’t invent animation, the Macdonald brothers didn’t invent the hamburger, Martin Luther King didn’t invent oratory, neither Jane Austen, Tolstoy nor Flaubert invented the novel and D. W. Griffith, the Warner Brothers, Irving Thalberg and Steven Spielberg didn’t invent film-making. Steve Jobs didn’t invent computers and he didn’t invent packet switching or the mouse. But he saw that there were no limits to the power that creative combinations of technology and design could accomplish.

I once heard George Melly, on a programme about Louis Armstrong, do that dangerous thing and give his own definition of a genius. “A genius,” he said, “is someone who enters a field and works in it and when they leave it, it is different. By that token, Satchmo was a genius.” I don’t think any reasonable person could deny that Steve Jobs, by that same token, was a genius too.

Steve Jobs, genius Read More »

Saul Bass changed how audiences view movie credits

art, history / /

From Christian Annyas’s “Saul Bass Title Sequences“:

“PROJECTIONISTS – PULL CURTAIN BEFORE TITLES”.

This is the text of a note that was stuck on the cans when the reels of film for “The Man With the Golden Arm” arrived at US movie theatres in 1955.

Until then the credits were referred to as ‘popcorn time.’ Audiences resented them and projectionists only pulled back the curtains to reveal the screen once they’d finished.
Saul Bass’ powerful title sequence for “The Man With the Golden Arm” changed the way directors and designers would treat the opening titles.

…

“For the average audience, the credits tell them there’s only three minutes left to eat popcorn. I take this ‘dead’ period and try to do more than simply get rid of names that filmgoers aren’t interested in. I aim to set up the audience for what’s coming; make them expectant.” — SAUL BASS

Saul Bass changed how audiences view movie credits Read More »

Hanoi’s last blacksmith

art, business, history, technology / /

From Seth Mydans’s “A Lone Blacksmith, Where Hammers Rang” (The New York Times: 25 November 2010):

HANOI, Vietnam — He is the last blacksmith on Blacksmith Street, dark with soot, his arms dappled with burns, sweating and hammering at his little roadside forge as a new world courses past him.

The son and grandson of blacksmiths, Nguyen Phuong Hung grew up when the street still rang with the sounds of the smithies, producing farm equipment, horseshoes and hand tools, before modern commerce and industrial production made them obsolete. “I still remember, when it was raining lightly, the streets were empty and that was all you could hear was the sounds of the hammers,” said Mr. Hung, 49.

…

The men who worked there left for lighter, better-paying work, and because the word was out that no modern woman would marry a blacksmith, Mr. Hung said. There may be other blacksmiths working in Vietnam, he said, but not here in the capital.

…

“Once I am gone the street will have no meaning anymore,” he said. “Blacksmith Street will be only a name.” That has been the fate of almost all the 36 narrow streets in Hanoi’s tree-shaded Ancient Quarter, each of them named for the guilds that once controlled them — Fan Street, China Bowl Street, Sweet Potato Street, Conical Hat Street.

There is nothing like this little corner of the urban past anywhere else in Vietnam. Only four of the streets have retained something of their original businesses, said Nguyen Vinh Phuc, a leading historian of Hanoi. There are still jewelry shops on Silver Street, sweets and pastries on Sugar Street, votive papers and toys on Votive Paper Street and pots and pans on Tin Street.

…

Traders have done business on this spot since the ninth century, Mr. Phuc said. The 36 guilds established themselves at the start of the 19th century.

…

Blacksmith Street got its name at the beginning of the 19th century, Mr. Phuc said, when French colonial administrators sent out a call for metal workers to help build the Long Bien bridge over the Red River. It was designed by the French architect Gustave Eiffel and became a target of American bombing raids during the Vietnam War.

Mr. Hung’s family has been here from the start, and like his father and grandfather he was called to help out around the forge when he was just a boy, as young as 6. But he rebelled and left for jobs as a driver and factory worker until, when he was 35, his father called him back. “My father told me this is the family trade and I’m the only one left to do it,” Mr. Hung said. “He said, ‘Just watch me work and you’ll learn what to do.’”

Mr. Hung discovered that he loved the work, and that it was his destiny to be a blacksmith. He remembered his father’s words: “When the iron glows red, you earn your money. That is your life.”

…

Mr. Hung has set up a little tea table on the sidewalk, refilling a thermos from a huge iron kettle that swings gently above the hot coals. A giant bamboo pipe leans against the table, and passersby are welcome to stop for a lungful of strong tobacco.

…

Mr. Hung hammers with the confidence of a master, bare-handed as he works because he says gloves would dull his touch. Wearing a pair of plastic sandals, he ignores the sparks that sting his feet and pepper his shirt with holes. Flames and smoke gush from the hot metal as he tempers it in a bucket of oil. By the end of the day, his arms and face are black with soot.

Hanoi’s last blacksmith Read More »

Unix: An Oral History

business, history, technology / /

From ‘s “Unix: An Oral History” (: ):

Multics

Gordon M. Brown

…

[Multics] was designed to include fault-free continuous operation capabilities, convenient remote terminal access and selective information sharing. One of the most important features of Multics was to follow the trend towards integrated multi-tasking and permit multiple programming environments and different human interfaces under one operating system.

…

Moreover, two key concepts had been picked up on in the development of Multics that would later serve to define Unix. These were that the less important features of the system introduced more complexity and, conversely, that the most important property of algorithms was simplicity. Ritchie explained this to Mahoney, articulating that:

The relationship of Multics to [the development of Unix] is actually interesting and fairly complicated. There were a lot of cultural things that were sort of taken over wholesale. And these include important things, [such as] the hierarchical file system and tree-structure file system – which incidentally did not get into the first version of Unix on the PDP-7. This is an example of why the whole thing is complicated. But any rate, things like the hierarchical file system, choices of simple things like the characters you use to edit lines as you’re typing, erasing characters were the same as those we had. I guess the most important fundamental thing is just the notion that the basic style of interaction with the machine, the fact that there was the notion of a command line, the notion was an explicit shell program. In fact the name shell came from Multics. A lot of extremely important things were completely internalized, and of course this is the way it is. A lot of that came through from Multics.

The Beginning

Michael Errecart and Cameron Jones

Files to Share

…

The Unix file system was based almost entirely on the file system for the failed Multics project. The idea was for file sharing to take place with explicitly separated file systems for each user, so that there would be no locking of file tables.

…

A major part of the answer to this question is that the file system had to be open. The needs of the group dictated that every user had access to every other user’s files, so the Unix system had to be extremely open. This openness is even seen in the password storage, which is not hidden at all but is encrypted. Any user can see all the encrypted passwords, but can only test one solution per second, which makes it extremely time consuming to try to break into the system.

…

The idea of standard input and output for devices eventually found its way into Unix as pipes. Pipes enabled users and programmers to send one function’s output into another function by simply placing a vertical line, a ‘|’ between the two functions. Piping is one of the most distinct features of Unix …

Language from B to C

… Thompson was intent on having Unix be portable, and the creation of a portable language was intrinsic to this. …

Finding a Machine

Darice Wong & Jake Knerr

… Thompson devoted a month apiece to the shell, editor, assembler, and other software tools. …

Use of Unix started in the patent office of Bell Labs, but by 1972 there were a number of non-research organizations at Bell Labs that were beginning to use Unix for software development. Morgan recalls the importance of text processing in the establishment of Unix. …

Building Unix

Jason Aughenbaugh, Jonathan Jessup, & Nicholas Spicher

…

The Origin of Pipes

The first edition of Thompson and Ritchie’s The Unix Programmer’s Manual was dated November 3, 1971; however, the idea of pipes is not mentioned until the Version 3 Unix manual, published in February 1973. …

…

Software Tools

…

grep was, in fact, one of the first programs that could be classified as a software tool. Thompson designed it at the request of McIlroy, as McIlroy explains:

One afternoon I asked Ken Thompson if he could lift the regular expression recognizer out of the editor and make a one-pass program to do it. He said yes. The next morning I found a note in my mail announcing a program named grep. It worked like a charm. When asked what that funny name meant, Ken said it was obvious. It stood for the editor command that it simulated, g/re/p (global regular expression print)….From that special-purpose beginning, grep soon became a household word. (Something I had to stop myself from writing in the first paragraph above shows how firmly naturalized the idea now is: ‘I used ed to grep out words from the dictionary.’) More than any other single program, grep focused the viewpoint that Kernighan and Plauger christened and formalized in Software Tools: make programs that do one thing and do it well, with as few preconceptions about input syntax as possible.

…

eqn and grep are illustrative of the Unix toolbox philosophy that McIlroy phrases as, “Write programs that do one thing and do it well. Write programs to work together. Write programs that handle text streams, because that is a universal interface.” This philosophy was enshrined in Kernighan and Plauger’s 1976 book, Software Tools, and reiterated in the “Foreword” to the issue of The Bell Systems Technical Journal that also introduced pipes.

…

Ethos

Robert Murray-Rust & Malika Seth

…

McIlroy says,

This is the Unix philosophy. Write programs that do one thing and do it well. Write programs to work together. Write programs that handle text streams because, that is a universal interface.

…

The dissemination of Unix, with a focus on what went on within Bell Labs

Steve Chen

…

In 1973, the first Unix applications were installed on a system involved in updating directory information and intercepting calls to numbers that had been changed. This was the first time Unix had been used in supporting an actual, ongoing operating business. Soon, Unix was being used to automate the operations systems at Bell Laboratories. It was automating the monitoring, involved in measurement, and helping to rout calls and ensure the quality of the calls.

…

There were numerous reasons for the friendliness the academic society, especially the academic Computer Science community, showed towards Unix. John Stoneback relates a few of these:

Unix came into many CS departments largely because it was the only powerful interactive system that could run on the sort of hardware (PDP-11s) that universities could afford in the mid ’70s. In addition, Unix itself was also very inexpensive. Since source code was provided, it was a system that could be shaped to the requirements of a particular installation. It was written in a language considerably more attractive than assembly, and it was small enough to be studied and understood by individuals. (John Stoneback, “The Collegiate Community,” Unix Review, October 1985, p. 27.)

…

The key features and characteristics of Unix that held it above other operating systems at the time were its software tools, its portability, its flexibility, and the fact that it was simple, compact, and efficient. The development of Unix in Bell Labs was carried on under a set of principles that the researchers had developed to guide their work. These principles included:

(i) Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new features.

(ii) Expect the output of every program to become the input to another, as yet unknown, program. Don’t clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don’t insist on interactive input.

(iii) Design and build software, even operating systems, to be tried early, ideally within weeks. Don’t hesitate to throw away the clumsy parts and rebuild them.

(iv) Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you’ve finished using them.”

(M.D. McIlroy, E.N. Pinson, and B.A. Tague “Unix Time-Sharing System Forward,” The Bell System Technical Journal, July-Aug 1088 vol 57, number 6 part 2. P. 1902)

Unix: An Oral History Read More »

William Burroughs on the necessary changes in biology

art, language & literature, science / /

From Conrad Knickerbocker’s interview of William S. Burroughs in “The Art of Fiction No. 36” (The Paris Review: Fall 1965, No. 35):

Science eventually will be forced to establish courts of biologic mediation, because life-forms are going to become more incompatible with the conditions of existence as man penetrates further into space. Mankind will have to undergo biologic alterations ultimately, if we are to survive at all. This will require biologic law to decide what changes to make. We will simply have to use our intelligence to plan mutations, rather than letting them occur at random. Because many such mutations—look at the saber-toothed tiger—are bound to be very poor engineering designs. The future, decidedly, yes. I think there are innumerable possibilities, literally innumerable. The hope lies in the development of nonbody experience and eventually getting away from the body itself, away from three-dimensional coordinates and concomitant animal reactions of fear and flight, which lead inevitably to tribal feuds and dissension.

William Burroughs on the necessary changes in biology Read More »

Why we get disoriented in malls

business, science, security / /

From Wikipedia’s “Gruen transfer” (28 September 2009):

In shopping mall design, the Gruen transfer refers to the moment when consumers respond to “scripted disorientation” cues in the environment. It is named for Austrian architect Victor Gruen (who disavowed such manipulative techniques) …

The Gruen transfer refers to the moment when a consumer enters a shopping mall, and, surrounded by an intentionally confusing layout, loses track of their original intentions. Spatial awareness of their surroundings play a key role, as does the surrounding sound and music. The effect of the transfer is marked by a slower walking pace and glazed eyes.

Why we get disoriented in malls Read More »

RFID security problems

business, politics, science, security, tech in changing society / /

Old British passport cover
Creative Commons License photo credit: sleepymyf

2005

From Brian Krebs’ “Leaving Las Vegas: So Long DefCon and Blackhat” (The Washington Post: 1 August 2005):

DefCon 13 also was notable for being the location where two new world records were set — both involved shooting certain electronic signals unprecedented distances. Los Angeles-based Flexilis set the world record for transmitting data to and from a “passive” radio frequency identification (RFID) card — covering a distance of more than 69 feet. (Active RFID — the kind being integrated into foreign passports, for example — differs from passive RFID in that it emits its own magnetic signal and can only be detected from a much shorter distance.)

…

The second record set this year at DefCon was pulled off by some teens from Cincinnati, who broke the world record they set last year by building a device capable of maintaining an unamplified, 11-megabit 802.11b wireless Internet connection over a distance of 125 miles (the network actually spanned from Utah into Nevada).

From Andrew Brandt’s “Black Hat, Lynn Settle with Cisco, ISS” (PC World: 29 July 2005):

Security researcher Kevin Mahaffey makes a final adjustment to a series of radio antennas; Mahaffey used the directional antennas in a demonstration during his presentation, “Long Range RFID and its Security Implications.” Mahaffey and two of his colleagues demonstrated how he could increase the “read range” of radio frequency identification (RF) tags from the typical four to six inches to approximately 50 feet. Mahaffey said the tags could be read at a longer distance, but he wanted to perform the demonstration in the room where he gave the presentation, and that was the greatest distance within the room that he could demonstrate. RFID tags such as the one Mahaffey tested will begin to appear in U.S. passports later this year or next year.

2006

From Joris Evers and Declan McCullagh’s “Researchers: E-passports pose security risk” (CNET: 5 August 2006):

At a pair of security conferences here, researchers demonstrated that passports equipped with radio frequency identification (RFID) tags can be cloned with a laptop equipped with a $200 RFID reader and a similarly inexpensive smart card writer. In addition, they suggested that RFID tags embedded in travel documents could identify U.S. passports from a distance, possibly letting terrorists use them as a trigger for explosives.

At the Black Hat conference, Lukas Grunwald, a researcher with DN-Systems in Hildesheim, Germany, demonstrated that he could copy data stored in an RFID tag from his passport and write the data to a smart card equipped with an RFID chip.

From Kim Zetter’s “Hackers Clone E-Passports” (Wired: 3 August 2006):

In a demonstration for Wired News, Grunwald placed his passport on top of an official passport-inspection RFID reader used for border control. He obtained the reader by ordering it from the maker — Walluf, Germany-based ACG Identification Technologies — but says someone could easily make their own for about $200 just by adding an antenna to a standard RFID reader.

He then launched a program that border patrol stations use to read the passports — called Golden Reader Tool and made by secunet Security Networks — and within four seconds, the data from the passport chip appeared on screen in the Golden Reader template.

Grunwald then prepared a sample blank passport page embedded with an RFID tag by placing it on the reader — which can also act as a writer — and burning in the ICAO layout, so that the basic structure of the chip matched that of an official passport.

As the final step, he used a program that he and a partner designed two years ago, called RFDump, to program the new chip with the copied information.

The result was a blank document that looks, to electronic passport readers, like the original passport.

Although he can clone the tag, Grunwald says it’s not possible, as far as he can tell, to change data on the chip, such as the name or birth date, without being detected. That’s because the passport uses cryptographic hashes to authenticate the data.

…

Grunwald’s technique requires a counterfeiter to have physical possession of the original passport for a time. A forger could not surreptitiously clone a passport in a traveler’s pocket or purse because of a built-in privacy feature called Basic Access Control that requires officials to unlock a passport’s RFID chip before reading it. The chip can only be unlocked with a unique key derived from the machine-readable data printed on the passport’s page.

To produce a clone, Grunwald has to program his copycat chip to answer to the key printed on the new passport. Alternatively, he can program the clone to dispense with Basic Access Control, which is an optional feature in the specification.

…

As planned, U.S. e-passports will contain a web of metal fiber embedded in the front cover of the documents to shield them from unauthorized readers. Though Basic Access Control would keep the chip from yielding useful information to attackers, it would still announce its presence to anyone with the right equipment. The government added the shielding after privacy activists expressed worries that a terrorist could simply point a reader at a crowd and identify foreign travelers.

In theory, with metal fibers in the front cover, nobody can sniff out the presence of an e-passport that’s closed. But [Kevin Mahaffey and John Hering of Flexilis] demonstrated in their video how even if a passport opens only half an inch — such as it might if placed in a purse or backpack — it can reveal itself to a reader at least two feet away.

…

In addition to cloning passport chips, Grunwald has been able to clone RFID ticket cards used by students at universities to buy cafeteria meals and add money to the balance on the cards.

He and his partners were also able to crash RFID-enabled alarm systems designed to sound when an intruder breaks a window or door to gain entry. Such systems require workers to pass an RFID card over a reader to turn the system on and off. Grunwald found that by manipulating data on the RFID chip he could crash the system, opening the way for a thief to break into the building through a window or door.

And they were able to clone and manipulate RFID tags used in hotel room key cards and corporate access cards and create a master key card to open every room in a hotel, office or other facility. He was able, for example, to clone Mifare, the most commonly used key-access system, designed by Philips Electronics. To create a master key he simply needed two or three key cards for different rooms to determine the structure of the cards. Of the 10 different types of RFID systems he examined that were being used in hotels, none used encryption.

Many of the card systems that did use encryption failed to change the default key that manufacturers program into the access card system before shipping, or they used sample keys that the manufacturer includes in instructions sent with the cards. Grunwald and his partners created a dictionary database of all the sample keys they found in such literature (much of which they found accidentally published on purchasers’ websites) to conduct what’s known as a dictionary attack. When attacking a new access card system, their RFDump program would search the list until it found the key that unlocked a card’s encryption.

“I was really surprised we were able to open about 75 percent of all the cards we collected,” he says.

2009

From Thomas Ricker’s “Video: Hacker war drives San Francisco cloning RFID passports” (Engadget: 2 February 2009):

Using a $250 Motorola RFID reader and antenna connected to his laptop, Chris recently drove around San Francisco reading RFID tags from passports, driver licenses, and other identity documents. In just 20 minutes, he found and cloned the passports of two very unaware US citizens.

RFID security problems Read More »

How security experts defended against Conficker

business, history, security, tech in changing society / /

From Jim Giles’ “The inside story of the Conficker worm” (New Scientist: 12 June 2009):

23 October 2008 … The dry, technical language of Microsoft’s October update did not indicate anything particularly untoward. A security flaw in a port that Windows-based PCs use to send and receive network signals, it said, might be used to create a “wormable exploit”. Worms are pieces of software that spread unseen between machines, mainly – but not exclusively – via the internet (see “Cell spam”). Once they have installed themselves, they do the bidding of whoever created them.

If every Windows user had downloaded the security patch Microsoft supplied, all would have been well. Not all home users regularly do so, however, and large companies often take weeks to install a patch. That provides windows of opportunity for criminals.

…

The new worm soon ran into a listening device, a “network telescope”, housed by the San Diego Supercomputing Center at the University of California. The telescope is a collection of millions of dummy internet addresses, all of which route to a single computer. It is a useful monitor of the online underground: because there is no reason for legitimate users to reach out to these addresses, mostly only suspicious software is likely to get in touch.

The telescope’s logs show the worm spreading in a flash flood. For most of 20 November, about 3000 infected computers attempted to infiltrate the telescope’s vulnerable ports every hour – only slightly above the background noise generated by older malicious code still at large. At 6 pm, the number began to rise. By 9 am the following day, it was 115,000 an hour. Conficker was already out of control.

That same day, the worm also appeared in “honeypots” – collections of computers connected to the internet and deliberately unprotected to attract criminal software for analysis. It was soon clear that this was an extremely sophisticated worm. After installing itself, for example, it placed its own patch over the vulnerable port so that other malicious code could not use it to sneak in. As Brandon Enright, a network security analyst at the University of California, San Diego, puts it, smart burglars close the window they enter by.

Conficker also had an ingenious way of communicating with its creators. Every day, the worm came up with 250 meaningless strings of letters and attached a top-level domain name – a .com, .net, .org, .info or .biz – to the end of each to create a series of internet addresses, or URLs. Then the worm contacted these URLs. The worm’s creators knew what each day’s URLs would be, so they could register any one of them as a website at any time and leave new instructions for the worm there.

It was a smart trick. The worm hunters would only ever spot the illicit address when the infected computers were making contact and the update was being downloaded – too late to do anything. For the next day’s set of instructions, the creators would have a different list of 250 to work with. The security community had no way of keeping up.

No way, that is, until Phil Porras got involved. He and his computer security team at SRI International in Menlo Park, California, began to tease apart the Conficker code. It was slow going: the worm was hidden within two shells of encryption that defeated the tools that Porras usually applied. By about a week before Christmas, however, his team and others – including the Russian security firm Kaspersky Labs, based in Moscow – had exposed the worm’s inner workings, and had found a list of all the URLs it would contact.

…

[Rick Wesson of Support Intelligence] has years of experience with the organisations that handle domain registration, and within days of getting Porras’s list he had set up a system to remove the tainted URLs, using his own money to buy them up.

It seemed like a major win, but the hackers were quick to bounce back: on 29 December, they started again from scratch by releasing an upgraded version of the worm that exploited the same security loophole.

This new worm had an impressive array of new tricks. Some were simple. As well as propagating via the internet, the worm hopped on to USB drives plugged into an infected computer. When those drives were later connected to a different machine, it hopped off again. The worm also blocked access to some security websites: when an infected user tried to go online and download the Microsoft patch against it, they got a “site not found” message.

Other innovations revealed the sophistication of Conficker’s creators. If the encryption used for the previous strain was tough, that of the new version seemed virtually bullet-proof. It was based on code little known outside academia that had been released just three months earlier by researchers at the Massachusetts Institute of Technology.

…

Indeed, worse was to come. On 15 March, Conficker presented the security experts with a new problem. It reached out to a URL called rmpezrx.org. It was on the list that Porras had produced, but – those involved decline to say why – it had not been blocked. One site was all that the hackers needed. A new version was waiting there to be downloaded by all the already infected computers, complete with another new box of tricks.

Now the cat-and-mouse game became clear. Conficker’s authors had discerned Porras and Wesson’s strategy and so from 1 April, the code of the new worm soon revealed, it would be able to start scanning for updates on 500 URLs selected at random from a list of 50,000 that were encoded in it. The range of suffixes would increase to 116 and include many country codes, such as .kz for Kazakhstan and .ie for Ireland. Each country-level suffix belongs to a different national authority, each of which sets its own registration procedures. Blocking the previous set of domains had been exhausting. It would soon become nigh-on impossible – even if the new version of the worm could be fully decrypted.

Luckily, Porras quickly repeated his feat and extracted the crucial list of URLs. Immediately, Wesson and others contacted the Internet Corporation for Assigned Names and Numbers (ICANN), an umbrella body that coordinates country suffixes.

…

From the second version onwards, Conficker had come with a much more efficient option: peer-to-peer (P2P) communication. This technology, widely used to trade pirated copies of software and films, allows software to reach out and exchange signals with copies of itself.

Six days after the 1 April deadline, Conficker’s authors let loose a new version of the worm via P2P. With no central release point to target, security experts had no means of stopping it spreading through the worm’s network. The URL scam seems to have been little more than a wonderful way to waste the anti-hackers’ time and resources. “They said: you’ll have to look at 50,000 domains. But they never intended to use them,” says Joe Stewart of SecureWorks in Atlanta, Georgia. “They used peer-to-peer instead. They misdirected us.”

The latest worm release had a few tweaks, such as blocking the action of software designed to scan for its presence. But piggybacking on it was something more significant: the worm’s first moneymaking schemes. These were a spam program called Waledac and a fake antivirus package named Spyware Protect 2009.

…

The same goes for fake software: when the accounts of a Russian company behind an antivirus scam became public last year, it appeared that one criminal had earned more than $145,000 from it in just 10 days.

How security experts defended against Conficker Read More »

Meeting expectations, no matter how silly, in design

art, business, technology / /

From Operator No. 9’s “That decorating touch” (Interactive Week: 24 April 2000): 100:

Intel AnyPoint Wireless:

Dan Sweeney, general manager of Intel’s Home Networking division, says that when the company showed consumer focus groups the AnyPoint Wireless home networking system …, people became very confused, because there wasn’t a visible antenna. The desktop version of the wireless adapter — about the size of a deck of cards — has an antenna hidden inside it. ‘They looked at it and said, “That’s not a radio!”‘ Sweeney says. So Intel’s industrial designers added a tiny little plastic tip on top of the unit that is supposed to resemble an antenna. It actually looks — and I’m sure this was not intended by the designers — kind of like the type of hat klansmen or maybe religious leaders — bishops? vicars? — wear. Then again, maybe I just need to get out more often.

Meeting expectations, no matter how silly, in design Read More »

The Uncanny Valley, art forgery, & love

art, business, history, on writing, science, security, tech in changing society, weird / /

Apply new wax to old wood
Creative Commons License photo credit: hans s

From Errol Morris’ “Bamboozling Ourselves (Part 2)” (The New York Times: 28 May 2009):

[Errol Morris:] The Uncanny Valley is a concept developed by the Japanese robot scientist Masahiro Mori. It concerns the design of humanoid robots. Mori’s theory is relatively simple. We tend to reject robots that look too much like people. Slight discrepancies and incongruities between what we look like and what they look like disturb us. The closer a robot resembles a human, the more critical we become, the more sensitive to slight discrepancies, variations, imperfections. However, if we go far enough away from the humanoid, then we much more readily accept the robot as being like us. This accounts for the success of so many movie robots — from R2-D2 to WALL-E. They act like humans but they don’t look like humans. There is a region of acceptability — the peaks around The Uncanny Valley, the zone of acceptability that includes completely human and sort of human but not too human. The existence of The Uncanny Valley also suggests that we are programmed by natural selection to scrutinize the behavior and appearance of others. Survival no doubt depends on such an innate ability.

EDWARD DOLNICK: [The art forger Van Meegeren] wants to avoid it. So his big challenge is he wants to paint a picture that other people are going to take as Vermeer, because Vermeer is a brand name, because Vermeer is going to bring him lots of money, if he can get away with it, but he can’t paint a Vermeer. He doesn’t have that skill. So how is he going to paint a picture that doesn’t look like a Vermeer, but that people are going to say, “Oh! It’s a Vermeer?” How’s he going to pull it off? It’s a tough challenge. Now here’s the point of The Uncanny Valley: as your imitation gets closer and closer to the real thing, people think, “Good, good, good!” — but then when it’s very close, when it’s within 1 percent or something, instead of focusing on the 99 percent that is done well, they focus on the 1 percent that you’re missing, and you’re in trouble. Big trouble.

…

Van Meegeren is trapped in the valley. If he tries for the close copy, an almost exact copy, he’s going to fall short. He’s going to look silly. So what he does instead is rely on the blanks in Vermeer’s career, because hardly anything is known about him; he’s like Shakespeare in that regard. He’ll take advantage of those blanks by inventing a whole new era in Vermeer’s career. No one knows what he was up to all this time. He’ll throw in some Vermeer touches, including a signature, so that people who look at it will be led to think, “Yes, this is a Vermeer.”

…

Van Meegeren was sometimes careful, other times astonishingly reckless. He could have passed certain tests. What was peculiar, and what was quite startling to me, is that it turned out that nobody ever did any scientific test on Van Meegeren, even the stuff that was available in his day, until after he confessed. And to this day, people hardly ever test pictures, even multi-million dollar ones. And I was so surprised by that that I kept asking, over and over again: why? Why would that be? Before you buy a house, you have someone go through it for termites and the rest. How could it be that when you’re going to lay out $10 million for a painting, you don’t test it beforehand? And the answer is that you don’t test it because, at the point of being about to buy it, you’re in love! You’ve found something. It’s going to be the high mark of your collection; it’s going to be the making of you as a collector. You finally found this great thing. It’s available, and you want it. You want it to be real. You don’t want to have someone let you down by telling you that the painting isn’t what you think it is. It’s like being newly in love. Everything is candlelight and wine. Nobody hires a private detective at that point. It’s only years down the road when things have gone wrong that you say, “What was I thinking? What’s going on here?” The collector and the forger are in cahoots. The forger wants the collector to snap it up, and the collector wants it to be real. You are on the same side. You think that it would be a game of chess or something, you against him. “Has he got the paint right?” “Has he got the canvas?” You’re going to make this checkmark and that checkmark to see if the painting measures up. But instead, both sides are rooting for this thing to be real. If it is real, then you’ve got a masterpiece. If it’s not real, then today is just like yesterday. You’re back where you started, still on the prowl.

The Uncanny Valley, art forgery, & love Read More »

The watchclock knows where your night watchman is

business, security / /

Detex Watchclock Station
Creative Commons License photo credit: 917press

From Christopher Fahey’s “Who Watches the Watchman?” (GraphPaper: 2 May 2009):

The Detex Newman watchclock was first introduced in 1927 and is still in wide use today.

&hellip What could you possibly do in 1900 to be absolutely sure a night watchman was making his full patrol?

An elegant solution, designed and patented in 1901 by the German engineer A.A. Newman, is called the “watchclock”. It’s an ingenious mechanical device, slung over the shoulder like a canteen and powered by a simple wind-up spring mechanism. It precisely tracks and records a night watchman’s position in both space and time for the duration of every evening. It also generates a detailed, permanent, and verifiable record of each night’s patrol.

What’s so interesting to me about the watchclock is that it’s an early example of interaction design used to explicitly control user behavior. The “user” of the watchclock device is obliged to behave in a strictly delimited fashion.

…

The key, literally, to the watchclock system is that the watchman is required to “clock in” at a series of perhaps a dozen or more checkpoints throughout the premises. Positioned at each checkpoint is a unique, coded key nestled in a little steel box and secured by a small chain. Each keybox is permanently and discreetly installed in strategically-placed nooks and crannies throughout the building, for example in a broom closet or behind a stairway.

The watchman makes his patrol. He visits every checkpoint and clicks each unique key into the watchclock. Within the device, the clockwork marks the exact time and key-location code to a paper disk or strip. If the watchman visits all checkpoints in order, they will have completed their required patrol route.

The watchman’s supervisor can subsequently unlock the device itself (the watchman himself cannot open the watchclock) and review the paper records to confirm if the watchman was or was not doing their job.

The watchclock knows where your night watchman is Read More »

Extreme male brains

business, education, science / /

From Joe Clark’s “The extreme Google brain” (Fawny: 26 April 2009):

… Susan Pinker’s The Sexual Paradox, which explains, using scientific findings, why large majorities of girls and women behave almost identically at different stages of their lives – while large minorities of boys and men show vast variability compared to each other and to male norms.

Some of these boys and men exhibit extreme-male-brain tendencies, including an ability to focus obsessively for long periods of time, often on inanimate objects or abstractions (hence male domination of engineering and high-end law). Paradoxically, other male brains in these exceptional cases may have an ability to experiment with many options for short periods each. Pejoratively diagnosed as attention-deficit disorder, Pinker provides evidence this latter ability is actually a strength for some entrepreneurs.

The male brain, extreme or not, is compatible with visual design. It allows you to learn every font in the Letraset catalogue and work from a grid. In fact, the male-brain capacity for years-long single-mindedness explains why the heads of large ad agencies and design houses are overwhelmingly male. (It isn’t a sexist conspiracy.)

In the computer industry, extreme male brains permit years of concentration on hardware and software design, while also iterating those designs seemingly ad infinitum. The extreme male brain is really the extreme Google brain. It’s somewhat of a misnomer, because such is actually the average brain inside the company, but I will use that as a neologism.

Google was founded by extreme-male-brain nerds and, by all outward appearances, seems to hire only that type of person, not all of them male.

Extreme male brains Read More »

How to increase donations on non-profit websites

business, language & literature / /

From Jakob Nielsen’s “Donation Usability: Increasing Online Giving to Non-Profits and Charities” (Alertbox: 30 March 2009):

We asked participants what information they want to see on non-profit websites before they decide whether to donate. Their answers fell into 4 broad categories, 2 of which were the most heavily requested:

  • The organization’s mission, goals, objectives, and work.
  • How it uses donations and contributions.

That is: What are you trying to achieve, and how will you spend my money?

Sadly, only 43% of the sites we studied answered the first question on their homepage. Further, only a ridiculously low 4% answered the second question on the homepage. Although organizations typically provided these answers somewhere within the site, users often had problems finding this crucial information.

…

In choosing between 2 charities, people referred to 5 categories of information. However, an organization’s mission, goals, objectives, and work was by far the most important. Indeed, it was 3.6 times as important as the runner-up issue, which was the organization’s presence in the user’s own community.

How to increase donations on non-profit websites Read More »

Google’s server farm revealed

business, tech in changing society, technology / /

From Nicholas Carr’s “Google lifts its skirts” (Rough Type: 2 April 2009):

I was particularly surprised to learn that Google rented all its data-center space until 2005, when it built its first center. That implies that The Dalles, Oregon, plant (shown in the photo above) was the company’s first official data smelter. Each of Google’s containers holds 1,160 servers, and the facility’s original server building had 45 containers, which means that it probably was running a total of around 52,000 servers. Since The Dalles plant has three server buildings, that means – and here I’m drawing a speculative conclusion – that it might be running around 150,000 servers altogether.

Here are some more details, from Rich Miller’s report:

The Google facility features a “container hanger” filled with 45 containers, with some housed on a second-story balcony. Each shipping container can hold up to 1,160 servers, and uses 250 kilowatts of power, giving the container a power density of more than 780 watts per square foot. Google’s design allows the containers to operate at a temperature of 81 degrees in the hot aisle. Those specs are seen in some advanced designs today, but were rare indeed in 2005 when the facility was built.

Google’s design focused on “power above, water below,” according to [Jimmy] Clidaras, and the racks are actually suspended from the ceiling of the container. The below-floor cooling is pumped into the hot aisle through a raised floor, passes through the racks and is returned via a plenum behind the racks. The cooling fans are variable speed and tightly managed, allowing the fans to run at the lowest speed required to cool the rack at that moment …

[Urs] Holzle said today that Google opted for containers from the start, beginning its prototype work in 2003. At the time, Google housed all of its servers in third-party data centers. “Once we saw that the commercial data center market was going to dry up, it was a natural step to ask whether we should build one,” said Holzle.

Google’s server farm revealed Read More »

Reasons Windows has a poor security architecture

security / /

From Daniel Eran Dilger’s “The Unavoidable Malware Myth: Why Apple Won’t Inherit Microsoft’s Malware Crown” (AppleInsider: 1 April 2008):

Thanks to its extensive use of battle-hardened Unix and open source software, Mac OS X also has always had security precautions in place that Windows lacked. It has also not shared the architectural weaknesses of Windows that have made that platform so easy to exploit and so difficult to clean up afterward, including:

  • the Windows Registry and the convoluted software installation mess related to it,
  • the Windows NT/2000/XP Interactive Services flaw opening up shatter attacks,
  • a wide open, legacy network architecture that left unnecessary, unsecured ports exposed by default,
  • poorly designed network sharing protocols that failed to account for adequate security measures,
  • poorly designed administrative messaging protocols that failed to account for adequate security,
  • poorly designed email clients that gave untrusted scripts access to spam one’s own contacts unwittingly,
  • an integrated web browser architecture that opened untrusted executables by design, and many others.

Reasons Windows has a poor security architecture Read More »

Defining social media, social software, & Web 2.0

business, history, social software, tech in changing society / /

From danah boyd’s “Social Media is Here to Stay… Now What?” at the Microsoft Research Tech Fest, Redmond, Washington (danah: 26 February 2009):

Social media is the latest buzzword in a long line of buzzwords. It is often used to describe the collection of software that enables individuals and communities to gather, communicate, share, and in some cases collaborate or play. In tech circles, social media has replaced the earlier fave “social software.” Academics still tend to prefer terms like “computer-mediated communication” or “computer-supported cooperative work” to describe the practices that emerge from these tools and the old skool academics might even categorize these tools as “groupwork” tools. Social media is driven by another buzzword: “user-generated content” or content that is contributed by participants rather than editors.

… These tools are part of a broader notion of “Web2.0.” Yet-another-buzzword, Web2.0 means different things to different people.

For the technology crowd, Web2.0 was about a shift in development and deployment. Rather than producing a product, testing it, and shipping it to be consumed by an audience that was disconnected from the developer, Web2.0 was about the perpetual beta. This concept makes all of us giggle, but what this means is that, for technologists, Web2.0 was about constantly iterating the technology as people interacted with it and learning from what they were doing. To make this happen, we saw the rise of technologies that supported real-time interactions, user-generated content, remixing and mashups, APIs and open-source software that allowed mass collaboration in the development cycle. …

For the business crowd, Web2.0 can be understood as hope. Web2.0 emerged out of the ashes of the fallen tech bubble and bust. Scars ran deep throughout Silicon Valley and venture capitalists and entrepreneurs wanted to party like it was 1999. Web2.0 brought energy to this forlorn crowd. At first they were skeptical, but slowly they bought in. As a result, we’ve seen a resurgence of startups, venture capitalists, and conferences. At this point, Web2.0 is sometimes referred to as Bubble2.0, but there’s something to say about “hope” even when the VCs start co-opting that term because they want four more years.

For users, Web2.0 was all about reorganizing web-based practices around Friends. For many users, direct communication tools like email and IM were used to communicate with one’s closest and dearest while online communities were tools for connecting with strangers around shared interests. Web2.0 reworked all of that by allowing users to connect in new ways. While many of the tools may have been designed to help people find others, what Web2.0 showed was that people really wanted a way to connect with those that they already knew in new ways. Even tools like MySpace and Facebook which are typically labeled social networkING sites were never really about networking for most users. They were about socializing inside of pre-existing networks.

Defining social media, social software, & Web 2.0 Read More »

Socioeconomic analysis of MySpace & Facebook

business, education, politics, social software, tech in changing society / /

From danah boyd’s “Viewing American class divisions through Facebook and MySpace” (danah boyd: 24 June 2007):

When MySpace launched in 2003, it was primarily used by 20/30-somethings (just like Friendster before it). The bands began populating the site by early 2004 and throughout 2004, the average age slowly declined. It wasn’t until late 2004 that teens really started appearing en masse on MySpace and 2005 was the year that MySpace became the “in thing” for teens.

Facebook launched in 2004 as a Harvard-only site. It slowly expanded to welcome people with .edu accounts from a variety of different universities. In mid-2005, Facebook opened its doors to high school students, but it wasn’t that easy to get an account because you needed to be invited. As a result, those who were in college tended to invite those high school students that they liked. Facebook was strongly framed as the “cool” thing that college students did.

…

In addition to the college framing, the press coverage of MySpace as dangerous and sketchy alienated “good” kids. Facebook seemed to provide an ideal alternative. Parents weren’t nearly as terrified of Facebook because it seemed “safe” thanks to the network-driven structure.

…

She argues that class divisions in the United States have more to do with lifestyle and social stratification than with income. In other words, all of my anti-capitalist college friends who work in cafes and read Engels are not working class just because they make $14K a year and have no benefits. Class divisions in the United States have more to do with social networks (the real ones, not FB/MS), social capital, cultural capital, and attitudes than income. Not surprisingly, other demographics typically discussed in class terms are also a part of this lifestyle division. Social networks are strongly connected to geography, race, and religion; these are also huge factors in lifestyle divisions and thus “class.”

…

The goodie two shoes, jocks, athletes, or other “good” kids are now going to Facebook. These kids tend to come from families who emphasize education and going to college. They are part of what we’d call hegemonic society. They are primarily white, but not exclusively. They are in honors classes, looking forward to the prom, and live in a world dictated by after school activities.

MySpace is still home for Latino/Hispanic teens, immigrant teens, “burnouts,” “alternative kids,” “art fags,” punks, emos, goths, gangstas, queer kids, and other kids who didn’t play into the dominant high school popularity paradigm. These are kids whose parents didn’t go to college, who are expected to get a job when they finish high school. These are the teens who plan to go into the military immediately after schools. Teens who are really into music or in a band are also on MySpace. MySpace has most of the kids who are socially ostracized at school because they are geeks, freaks, or queers.

In order to demarcate these two groups, let’s call the first group of teens “hegemonic teens” and the second group “subaltern teens.”

…

Most teens who exclusively use Facebook are familiar with and have an opinion about MySpace. These teens are very aware of MySpace and they often have a negative opinion about it. They see it as gaudy, immature, and “so middle school.” They prefer the “clean” look of Facebook, noting that it is more mature and that MySpace is “so lame.” What hegemonic teens call gaudy can also be labeled as “glitzy” or “bling” or “fly” (or what my generation would call “phat”) by subaltern teens. Terms like “bling” come out of hip-hop culture where showy, sparkly, brash visual displays are acceptable and valued. The look and feel of MySpace resonates far better with subaltern communities than it does with the upwardly mobile hegemonic teens. … That “clean” or “modern” look of Facebook is akin to West Elm or Pottery Barn or any poshy Scandinavian design house (that I admit I’m drawn to) while the more flashy look of MySpace resembles the Las Vegas imagery that attracts millions every year. I suspect that lifestyles have aesthetic values and that these are being reproduced on MySpace and Facebook.

I should note here that aesthetics do divide MySpace users. The look and feel that is acceptable amongst average Latino users is quite different from what you see the subculturally-identified outcasts using. Amongst the emo teens, there’s a push for simple black/white/grey backgrounds and simplistic layouts. While I’m using the term “subaltern teens” to lump together non-hegemonic teens, the lifestyle divisions amongst the subalterns are quite visible on MySpace through the aesthetic choices of the backgrounds. The aesthetics issue is also one of the forces that drives some longer-term users away from MySpace.

…

Teens from poorer backgrounds who are on MySpace are less likely to know people who go to universities. They are more likely to know people who are older than them, but most of their older friends, cousins, and co-workers are on MySpace. It’s the cool working class thing and it’s the dominant SNS at community colleges. These teens are more likely to be interested in activities like shows and clubs and they find out about them through MySpace. The subaltern teens who are better identified as “outsiders” in a hegemonic community tend to be very aware of Facebook. Their choice to use MySpace instead of Facebook is a rejection of the hegemonic values (and a lack of desire to hang out with the preps and jocks even online).

Class divisions in military use

A month ago, the military banned MySpace but not Facebook. This was a very interesting move because the division in the military reflects the division in high schools. Soldiers are on MySpace; officers are on Facebook. Facebook is extremely popular in the military, but it’s not the SNS of choice for 18-year old soldiers, a group that is primarily from poorer, less educated communities. They are using MySpace. The officers, many of whom have already received college training, are using Facebook. The military ban appears to replicate the class divisions that exist throughout the military. …

MySpace is the primary way that young soldiers communicate with their peers. When I first started tracking soldiers’ MySpace profiles, I had to take a long deep breath. Many of them were extremely pro-war, pro-guns, anti-Arab, anti-Muslim, pro-killing, and xenophobic as hell. Over the last year, I’ve watched more and more profiles emerge from soldiers who aren’t quite sure what they are doing in Iraq. I don’t have the data to confirm whether or not a significant shift has occurred but it was one of those observations that just made me think. And then the ban happened. I can’t help but wonder if part of the goal is to cut off communication between current soldiers and the group that the military hopes to recruit.

…

Thoughts and meta thoughts

…

People often ask me if I’m worried about teens today. The answer is yes, but it’s not because of social network sites. With the hegemonic teens, I’m very worried about the stress that they’re under, the lack of mobility and healthy opportunities for play and socialization, and the hyper-scheduling and surveillance. I’m worried about their unrealistic expectations for becoming rich and famous, their lack of work ethic after being pampered for so long, and the lack of opportunities that many of them have to even be economically stable let alone better off than their parents. I’m worried about how locking teens indoors coupled with a fast food/junk food advertising machine has resulted in a decrease in health levels across the board which will just get messy as they are increasingly unable to afford health insurance. When it comes to ostracized teens, I’m worried about the reasons why society has ostracized them and how they will react to ongoing criticism from hegemonic peers. I cringe every time I hear of another Columbine, another Virgina Tech, another site of horror when an outcast teen lashes back at the hegemonic values of society.

I worry about the lack of opportunities available to poor teens from uneducated backgrounds. I’m worried about how Wal-Mart Nation has destroyed many of the opportunities for meaningful working class labor as these youth enter the workforce. I’m worried about what a prolonged war will mean for them. I’m worried about how they’ve been told that to succeed, they must be a famous musician or sports player. I’m worried about how gangs provide the only meaningful sense of community that many of these teens will ever know.

Given the state of what I see in all sorts of neighborhoods, I’m amazed at how well teens are coping and I think that technology has a lot to do with that. Teens are using social network sites to build community and connect with their peers. They are creating publics for socialization. And through it, they are showcasing all of the good, bad, and ugly of today’s teen life.

…

In the 70s, Paul Willis analyzed British working class youth and he wrote a book called Learning to Labor: How Working Class Kids Get Working Class Jobs. He argued that working class teens will reject hegemonic values because it’s the only way to continue to be a part of the community that they live in. In other words, if you don’t know that you will succeed if you make a run at jumping class, don’t bother – you’ll lose all of your friends and community in the process. His analysis has such strong resonance in American society today. I just wish I knew how to fix it.

Socioeconomic analysis of MySpace & Facebook Read More »

ODF compared & constrasted with OOXML

business, technology / /

From Sam Hiser’s “Achieving Openness: A Closer Look at ODF and OOXML” (ONLamp.com: 14 June 2007):

An open, XML-based standard for displaying and storing data files (text documents, spreadsheets, and presentations) offers a new and promising approach to data storage and document exchange among office applications. A comparison of the two XML-based formats–OpenDocument Format (“ODF”) and Office Open XML (“OOXML”)–across widely accepted “openness” criteria has revealed substantial differences, including the following:

  • ODF is developed and maintained in an open, multi-vendor, multi-stakeholder process that protects against control by a single organization. OOXML is less open in its development and maintenance, despite being submitted to a formal standards body, because control of the standard ultimately rests with one organization.
  • ODF is the only openly available standard, published fully in a document that is freely available and easy to comprehend. This openness is reflected in the number of competing applications in which ODF is already implemented. Unlike ODF, OOXML’s complexity, extraordinary length, technical omissions, and single-vendor dependencies combine to make alternative implementation unattractive as well as legally and practically impossible.
  • ODF is the only format unencumbered by intellectual property rights (IPR) restrictions on its use in other software, as certified by the Software Freedom Law Center. Conversely, many elements designed into the OOXML formats but left undefined in the OOXML specification require behaviors upon document files that only Microsoft Office applications can provide. This makes data inaccessible and breaks work group productivity whenever alternative software is used.
  • ODF offers interoperability with ODF-compliant applications on most of the common operating system platforms. OOXML is designed to operate fully within the Microsoft environment only. Though it will work elegantly across the many products in the Microsoft catalog, OOXML ignores accepted standards and best practices regarding its use of XML.

Overall, a comparison of both formats reveals significant differences in their levels of openness. While ODF is revealed as sufficiently open across all four key criteria, OOXML shows relative weakness in each criteria and offers fundamental flaws that undermine its candidacy as a global standard.

ODF compared & constrasted with OOXML Read More »

An analysis of Google’s technology, 2005

business, history, science, security, tech in changing society, technology / /

From Stephen E. Arnold’s The Google Legacy: How Google’s Internet Search is Transforming Application Software (Infonortics: September 2005):

The figure Google’s Fusion: Hardware and Software Engineering shows that Google’s technology framework has two areas of activity. There is the software engineering effort that focuses on PageRank and other applications. Software engineering, as used here, means writing code and thinking about how computer systems operate in order to get work done quickly. Quickly means the sub one-second response times that Google is able to maintain despite its surging growth in usage, applications and data processing.

Google is hardware plus software

The other effort focuses on hardware. Google has refined server racks, cable placement, cooling devices, and data center layout. The payoff is lower operating costs and the ability to scale as demand for computing resources increases. With faster turnaround and the elimination of such troublesome jobs as backing up data, Google’s hardware innovations give it a competitive advantage few of its rivals can equal as of mid-2005.

…

How Google Is Different from MSN and Yahoo

Google’s technologyis simultaneously just like other online companies’ technology, and very different. A data center is usually a facility owned and operated by a third party where customers place their servers. The staff of the data center manage the power, air conditioning and routine maintenance. The customer specifies the computers and components. When a data center must expand, the staff of the facility may handle virtually all routine chores and may work with the customer’s engineers for certain more specialized tasks.

Before looking at some significant engineering differences between Google and two of its major competitors, review this list of characteristics for a Google data center.

1. Google data centers – now numbering about two dozen, although no one outside Google knows the exact number or their locations. They come online and automatically, under the direction of the Google File System, start getting work from other data centers. These facilities, sometimes filled with 10,000 or more Google computers, find one another and configure themselves with minimal human intervention.

2. The hardware in a Google data center can be bought at a local computer store. Google uses the same types of memory, disc drives, fans and power supplies as those in a standard desktop PC.

3. Each Google server comes in a standard case called a pizza box with one important change: the plugs and ports are at the front of the box to make access faster and easier.

4. Google racks are assembled for Google to hold servers on their front and back sides. This effectively allows a standard rack, normally holding 40 pizza box servers, to hold 80.

5. A Google data center can go from a stack of parts to online operation in as little as 72 hours, unlike more typical data centers that can require a week or even a month to get additional resources online.

6. Each server, rack and data center works in a way that is similar to what is called “plug and play.” Like a mouse plugged into the USB port on a laptop, Google’s network of data centers knows when more resources have been connected. These resources, for the most part, go into operation without human intervention.

Several of these factors are dependent on software. This overlap between the hardware and software competencies at Google, as previously noted, illustrates the symbiotic relationship between these two different engineering approaches. At Google, from its inception, Google software and Google hardware have been tightly coupled. Google is not a software company nor is it a hardware company. Google is, like IBM, a company that owes its existence to both hardware and software. Unlike IBM, Google has a business model that is advertiser supported. Technically, Google is conceptually closer to IBM (at one time a hardware and software company) than it is to Microsoft (primarily a software company) or Yahoo! (an integrator of multiple softwares).

Software and hardware engineering cannot be easily segregated at Google. At MSN and Yahoo hardware and software are more loosely-coupled. Two examples will illustrate these differences.

Microsoft – with some minor excursions into the Xbox game machine and peripherals – develops operating systems and traditional applications. Microsoft has multiple operating systems, and its engineers are hard at work on the company’s next-generation of operating systems.

…

Several observations are warranted:

1. Unlike Google, Microsoft does not focus on performance as an end in itself. As a result, Microsoft gets performance the way most computer users do. Microsoft buys or upgrades machines. Microsoft does not fiddle with its operating systems and their subfunctions to get that extra time slice or two out of the hardware.

2. Unlike Google, Microsoft has to support many operating systems and invest time and energy in making certain that important legacy applications such as Microsoft Office or SQLServer can run on these new operating systems. Microsoft has a boat anchor tied to its engineer’s ankles. The boat anchor is the need to ensure that legacy code works in Microsoft’s latest and greatest operating systems.

3. Unlike Google, Microsoft has no significant track record in designing and building hardware for distributed, massively parallelised computing. The mice and keyboards were a success. Microsoft has continued to lose money on the Xbox, and the sudden demise of Microsoft’s entry into the home network hardware market provides more evidence that Microsoft does not have a hardware competency equal to Google’s.

…

Yahoo! operates differently from both Google and Microsoft. Yahoo! is in mid-2005 a direct competitor to Google for advertising dollars. Yahoo! has grown through acquisitions. In search, for example, Yahoo acquired 3721.com to handle Chinese language search and retrieval. Yahoo bought Inktomi to provide Web search. Yahoo bought Stata Labs in order to provide users with search and retrieval of their Yahoo! mail. Yahoo! also owns AllTheWeb.com, a Web search site created by FAST Search & Transfer. Yahoo! owns the Overture search technology used by advertisers to locate key words to bid on. Yahoo! owns Alta Vista, the Web search system developed by Digital Equipment Corp. Yahoo! licenses InQuira search for customer support functions. Yahoo has a jumble of search technology; Google has one search technology.

Historically Yahoo has acquired technology companies and allowed each company to operate its technology in a silo. Integration of these different technologies is a time-consuming, expensive activity for Yahoo. Each of these software applications requires servers and systems particular to each technology. The result is that Yahoo has a mosaic of operating systems, hardware and systems. Yahoo!’s problem is different from Microsoft’s legacy boat-anchor problem. Yahoo! faces a Balkan-states problem.

There are many voices, many needs, and many opposing interests. Yahoo! must invest in management resources to keep the peace. Yahoo! does not have a core competency in hardware engineering for performance and consistency. Yahoo! may well have considerable competency in supporting a crazy-quilt of hardware and operating systems, however. Yahoo! is not a software engineering company. Its engineers make functions from disparate systems available via a portal.

…

The figure below provides an overview of the mid-2005 technical orientation of Google, Microsoft and Yahoo.

2005 focuses of Google, MSN, and Yahoo

The Technology Precepts

… five precepts thread through Google’s technical papers and presentations. The following snapshots are extreme simplifications of complex, yet extremely fundamental, aspects of the Googleplex.

Cheap Hardware and Smart Software

Google approaches the problem of reducing the costs of hardware, set up, burn-in and maintenance pragmatically. A large number of cheap devices using off-the-shelf commodity controllers, cables and memory reduces costs. But cheap hardware fails.

In order to minimize the “cost” of failure, Google conceived of smart software that would perform whatever tasks were needed when hardware devices fail. A single device or an entire rack of devices could crash, and the overall system would not fail. More important, when such a crash occurs, no full-time systems engineering team has to perform technical triage at 3 a.m.

The focus on low-cost, commodity hardware and smart software is part of the Google culture.

…

Logical Architecture

Google’s technical papers do not describe the architecture of the Googleplex as self-similar. Google’s technical papers provide tantalizing glimpses of an approach to online systems that makes a single server share features and functions of a cluster of servers, a complete data center, and a group of Google’s data centers.

…

The collections of servers running Google applications on the Google version of Linux is a supercomputer. The Googleplex can perform mundane computing chores like taking a user’s query and matching it to documents Google has indexed. Further more, the Googleplex can perform side calculations needed to embed ads in the results pages shown to user, execute parallelized, high-speed data transfers like computers running state-of-the-art storage devices, and handle necessary housekeeping chores for usage tracking and billing.

…

When Google needs to add processing capacity or additional storage, Google’s engineers plug in the needed resources. Due to self-similarity, the Googleplex can recognize, configure and use the new resource. Google has an almost unlimited flexibility with regard to scaling and accessing the capabilities of the Googleplex.

…

In Google’s self-similar architecture, the loss of an individual device is irrelevant. In fact, a rack or a data center can fail without data loss or taking the Googleplex down. The Google operating system ensures that each file is written three to six times to different storage devices. When a copy of that file is not available, the Googleplex consults a log for the location of the copies of the needed file. The application then uses that replica of the needed file and continues with the job’s processing.

…

Speed and Then More Speed

…

Google uses commodity pizza box servers organized in a cluster. A cluster is group of computers that are joined together to create a more robust system. Instead of using exotic servers with eight or more processors, Google generally uses servers that have two processors similar to those found in a typical home computer.

Through proprietary changes to Linux and other engineering innovations, Google is able to achieve supercomputer performance from components that are cheap and widely available.

…

… engineers familiar with Google believe that read rates may in some clusters approach 2,000 megabytes a second. When commodity hardware gets better, Google runs faster without paying a premium for that performance gain.

…

Another key notion of speed at Google concerns writing computer programs to deploy to Google users. Google has developed short cuts to programming. An example is Google’s creating a library of canned functions to make it easy for a programmer to optimize a program to run on the Googleplex computer. At Microsoft or Yahoo, a programmer must write some code or fiddle with code to get different pieces of a program to execute simultaneously using multiple processors. Not at Google. A programmer writes a program, uses a function from a Google bundle of canned routines, and lets the Googleplex handle the details. Google’s programmers are freed from much of the tedium associated with writing software for a distributed, parallel computer.

…

Eliminate or Reduce Certain System Expenses

Some lucky investors jumped on the Google bandwagon early. Nevertheless, Google was frugal, partly by necessity and partly by design. The focus on frugality influenced many hardware and software engineering decisions at the company.

…

Drawbacks of the Googleplex

…

The Laws of Physics: Heat and Power 101

…

In reality, no one knows. Google has a rapidly expanding number of data centers. The data center near Atlanta, Georgia, is one of the newest deployed. This state-of-the-art facility reflects what Google engineers have learned about heat and power issues in its other data centers. Within the last 12 months, Google has shifted from concentrating its servers at about a dozen data centers, each with 10,000 or more servers, to about 60 data centers, each with fewer machines. The change is a response to the heat and power issues associated with larger concentrations of Google servers.

The most failure prone components are:

  • Fans.
  • IDE drives which fail at the rate of one per 1,000 drives per day.
  • Power supplies which fail at a lower rate.

Leveraging the Googleplex

…

Google’s technology is one major challenge to Microsoft and Yahoo. So to conclude this cursory and vastly simplified look at Google technology, consider these items:

1. Google is fast anywhere in the world.

2. Google learns. When the heat and power problems at dense data centers surfaced, Google introduced cooling and power conservation innovations to its two dozen data centers.

3. Programmers want to work at Google. “Google has cachet,” said one recent University of Washington graduate.

4. Google’s operating and scaling costs are lower than most other firms offering similar businesses.

5. Google squeezes more work out of programmers and engineers by design.

6. Google does not break down, or at least it has not gone offline since 2000.

7. Google’s Googleplex can deliver desktop-server applications now.

8. Google’s applications install and update without burdening the user with gory details and messy crashes.

9. Google’s patents provide basic technology insight pertinent to Google’s core functionality.

An analysis of Google’s technology, 2005 Read More »