linux

Talk about Markdown to SLUUG this Wednesday

personal, technology / By / 7 November 2011

I’ll be giving a talk to the St. Louis UNIX Users Group next Wednesday night about Markdown, a tool I absolutely love.

You’re invited to come. Please do – I think you’ll definitely learn a lot.

Date: Wednesday, Nov. 9, 2011
Time: 6:30 – 9 pm
Where: 11885 Lackland Rd., St Louis, MO 63146
Map: http://g.co/maps/6gg9g
Directions: http://www.sluug.org/resources/meeting_info/map_graybar.shtml

Here’s the description:

John Gruber, the inventor of Markdown, describes it this way: “Markdown is a text-to-HTML conversion tool for web writers. Markdown allows you to write using an easy-to-read, easy-to-write plain text format, then convert it to structurally valid XHTML (or HTML). Thus, ‘Markdown’ is two things: (1) a plain text formatting syntax; and (2) a software tool, written in Perl, that converts the plain text formatting to HTML. … The overriding design goal for Markdown’s formatting syntax is to make it as readable as possible. The idea is that a Markdown-formatted document should be publishable as-is, as plain text, without looking like it’s been marked up with tags or formatting instructions.”

This talk by Scott Granneman & Bill Odom will cover the basics of Markdown’s syntax, key variants of Markdown, tools for composing Markdown (including vim, of course!), and ways you can easily transform a plain text file written in Markdown into HTML, JSON, TXT, LaTeX, man, MediaWiki, Textile, DocBook XML, ODT, EPUB, Slidy and S5 HTML and JavaScript slide shows, RTF, or even Word!

If you have any questions, please contact me. Hope to see you there!

A vote for CrossOver

business, tech help / By / 30 August 2010

Let me recommend Codeweavers’ CrossOver, a commercial implementation of WINE that works on Linux & Mac OS X. It’s reasonably priced, & it makes setting up & configuring both WINE and the programs that run inside WINE much easier. Plus, the company is made up of good people, & they’re very upfront on their site about what works with WINE, what mostly works, what kinda works, & what doesn’t work at all.

http://www.codeweavers.com/

Linux Phrasebook in Russian

business, language & literature, personal, tech help, technology / By / 29 November 2009

My book, Linux Phrasebook, which is still selling well & still just as useful today as when it came out in 2006 (& will be for another decade or two, given how consistent the Linux command line is), has been translated into Russian. You can find it at this Russian website, where I found out that it’s translated title is Linux Карманный справочник, which looks pretty cool. The other cool thing I found out from going to the page is how to display my name in Russian: Скотт Граннеман.

Australian police: don’t bank online with Windows

business, security, tech in changing society / By / 30 October 2009

From Munir Kotadia’s “NSW Police: Don’t use Windows for internet banking” (ITnews: 9 October 2009):

Consumers wanting to safely connect to their internet banking service should use Linux or the Apple iPhone, according to a detective inspector from the NSW Police, who was giving evidence on behalf of the NSW Government at the public hearing into Cybercrime today in Sydney.

Detective Inspector Bruce van der Graaf from the Computer Crime Investigation Unit told the hearing that he uses two rules to protect himself from cybercriminals when banking online.

The first rule, he said, was to never click on hyperlinks to the banking site and the second was to avoid Microsoft Windows.

“If you are using the internet for a commercial transaction, use a Linux boot up disk – such as Ubuntu or some of the other flavours. Puppylinux is a nice small distribution that boots up fairly quickly.

…

Van der Graaf also mentioned the iPhone, which he called “quite safe” for internet banking.

“Another option is the Apple iPhone. It is only capable of running one process at a time so there is really no danger from infection,” he said.

Apple’s role in technology

business, history, tech in changing society, technology / By / 11 August 2009
Image representing iPhone as depicted in Crunc...
Image via CrunchBase

From Doc Searls’s “The Most Personal Device” (Linux Journal: 1 March 2009):

My friend Keith Hopper made an interesting observation recently. He said one of Apple’s roles in the world is finding categories where progress is logjammed, and opening things up by coming out with a single solution that takes care of everything, from the bottom to the top. Apple did it with graphical computing, with .mp3 players, with on-line music sales and now with smartphones. In each case, it opens up whole new territories that can then be settled and expanded by other products, services and companies. Yes, it’s closed and controlling and the rest of it. But what matters is the new markets that open up.

Outline for an Unpublished Linux Textbook

on writing, personal, security / By / 9 October 2012

Back in 2004 or so, I was asked to write an outline for a college textbook that would be used in courses on Linux. I happily complied, producing the outline you can see on my website. The editor on the project loved the outline & showed it several professors to get their reactions, which were uniformly positive, with one prof reporting back that (& I’m paraphrasing here) “It was like this author read my mind, as this is exactly the book I’d like to use in my course!” Sadly, the book was never written, because the editor’s boss didn’t like the fact that I didn’t have a PhD in Computer Science. I thought that to be a silly reason then, & I think it’s a silly reason to reject the book now.

However, their loss is your gain. Here’s the outline for the book. Yes, it’s sadly outdated. Yes, it focuses quite a bit on SUSE, but that was what the publisher wanted. Yes, Linux has come a LONG way since I wrote this outline. But I still think it’s a damn good job, and you may find it interesting for historical reasons. So, enjoy!

Some facts about GPL 2 & GPL 3

business, law, tech in changing society / By / 25 April 2009

From Liz Laffan’s “GPLv2 vs GPLv3: The two seminal open source licenses, their roots, consequences and repercussions” (VisionMobile: September 2007):

From a licensing perspective, the vast majority (typically 60-70%) of all open source projects are licensed under the GNU Public License version 2 (GPLv2).

…

GPLv3 was published in July 2007, some 16 years following the creation of GPLv2. The purpose of this new license is to address some of the areas identified for improvement and clarification in GPLv2 – such as patent indemnity, internationalisation and remedies for inadvertent license infringement (rather than the previous immediate termination effect). The new GPLv3 license is nearly double the length of the GPLv2 …

…

GPLv3 differs to GPLv2 in several important ways. Firstly it provides more clarity on patent licenses and attempts to clarify what is meant by both a distribution and derivative works. Secondly it revokes the immediate termination of license clause in favour of licensee opportunities to ‘fix’ any violations within a given time-period. In addition there are explicit ‘Additional Terms’ which permits users to choose from a fixed set of alternative terms which can modify the standard GPLv3 terms. These are all welcome, positive moves which should benefit all users of the GPLv3 license.

Nonetheless there are three contentious aspects of GPLv3 that have provoked much discussion in the FOSS community and could deter adoption of GPLv3 by more circumspect users and organisations.

Open source & patents

business, law, politics, tech in changing society / By / 25 April 2009

From Liz Laffan’s “GPLv2 vs GPLv3: The two seminal open source licenses, their roots, consequences and repercussions” (VisionMobile: September 2007):

Cumulatively patents have been doubling practically every year since 1990. Patents are now probably the most contentious issue in software-related intellectual property rights.

…

However we should also be aware that software written from scratch is as likely to infringe patents as FOSS covered software – due mainly to the increasing proliferation of patents in all software technologies. Consequently the risk of patent infringement is largely comparable whether one chooses to write one’s own software or use software covered by the GPLv2; one will most likely have to self-indemnify against a potential patent infringement claim in both cases.

…

The F.U.D. (fear, uncertainty and doubt) that surrounds patents in FOSS has been further heightened by two announcements, both instigated by Microsoft. Firstly in November 2006 Microsoft and Novell1 entered into a cross- licensing patent agreement where Microsoft gave Novell assurances that it would not sue the company or its customers if they were to be found infringing Microsoft patents in the Novell Linux distribution. Secondly in May 2007 Microsoft2 restated (having alluded to the same in 2004) that FOSS violates 235 Microsoft patents. Unfortunately, the Redmond giant did not state which patents in particular were being infringed and nor have they initiated any actions against a user or distributor of Linux.

…

The FOSS community have reacted to these actions by co-opting the patent system and setting up the Patent Commons (http://www.patentcommons.org). This initiative, managed by the Linux Foundation, coordinates and manages a patent commons reference library, documenting information about patent- related pledges in support of Linux and FOSS that are provided by large software companies. Moreover, software giants such as IBM and Nokia have committed not to assert patents against the Linux kernel and other FOSS projects. In addition, the FSF have strengthened the patent clause of GPLv3…

My new book – Google Apps Deciphered – is out!

business, education, history, personal, social software, tech help, tech in changing society, technology / By / 5 February 2009

I’m really proud to announce that my 5th book is now out & available for purchase: Google Apps Deciphered: Compute in the Cloud to Streamline Your Desktop. My other books include:

(I’ve also contributed to two others: Ubuntu Hacks: Tips & Tools for Exploring, Using, and Tuning Linux and Microsoft Vista for IT Security Professionals.)

Google Apps Deciphered is a guide to setting up Google Apps, migrating to it, customizing it, and using it to improve productivity, communications, and collaboration. I walk you through each leading component of Google Apps individually, and then show my readers exactly how to make them work together for you on the Web or by integrating them with your favorite desktop apps. I provide practical insights on Google Apps programs for email, calendaring, contacts, wikis, word processing, spreadsheets, presentations, video, and even Google’s new web browser Chrome. My aim was to collect together and present tips and tricks I’ve gained by using and setting up Google Apps for clients, family, and friends.

Here’s the table of contents:

  • 1: Choosing an Edition of Google Apps
  • 2: Setting Up Google Apps
  • 3: Migrating Email to Google Apps
  • 4: Migrating Contacts to Google Apps
  • 5: Migrating Calendars to Google Apps
  • 6: Managing Google Apps Services
  • 7: Setting Up Gmail
  • 8: Things to Know About Using Gmail
  • 9: Integrating Gmail with Other Software and Services
  • 10: Integrating Google Contacts with Other Software and Services
  • 11: Setting Up Google Calendar
  • 12: Things to Know About Using Google Calendar
  • 13: Integrating Google Calendar with Other Software and Services
  • 14: Things to Know About Using Google Docs
  • 15: Integrating Google Docs with Other Software and Services
  • 16: Setting Up Google Sites
  • 17: Things to Know About Using Google Sites
  • 18: Things to Know About Using Google Talk
  • 19: Things to Know About Using Start Page
  • 20: Things to Know About Using Message Security and Recovery
  • 21: Things to Know About Using Google Video
  • Appendix A: Backing Up Google Apps
  • Appendix B: Dealing with Multiple Accounts
  • Appendix C: Google Chrome: A Browser Built for Cloud Computing

If you want to know more about Google Apps and how to use it, then I know you’ll enjoy and learn from Google Apps Deciphered. You can read about and buy the book at Amazon (http://www.amazon.com/Google-Apps-Deciphered-Compute-Streamline/dp/0137004702) for $26.39. If you have any questions or comments, don’t hesitate to contact me at scott at granneman dot com.

An analysis of Google’s technology, 2005

business, history, science, security, tech in changing society, technology / By / 28 November 2008

From Stephen E. Arnold’s The Google Legacy: How Google’s Internet Search is Transforming Application Software (Infonortics: September 2005):

The figure Google’s Fusion: Hardware and Software Engineering shows that Google’s technology framework has two areas of activity. There is the software engineering effort that focuses on PageRank and other applications. Software engineering, as used here, means writing code and thinking about how computer systems operate in order to get work done quickly. Quickly means the sub one-second response times that Google is able to maintain despite its surging growth in usage, applications and data processing.

Google is hardware plus software

The other effort focuses on hardware. Google has refined server racks, cable placement, cooling devices, and data center layout. The payoff is lower operating costs and the ability to scale as demand for computing resources increases. With faster turnaround and the elimination of such troublesome jobs as backing up data, Google’s hardware innovations give it a competitive advantage few of its rivals can equal as of mid-2005.

…

How Google Is Different from MSN and Yahoo

Google’s technologyis simultaneously just like other online companies’ technology, and very different. A data center is usually a facility owned and operated by a third party where customers place their servers. The staff of the data center manage the power, air conditioning and routine maintenance. The customer specifies the computers and components. When a data center must expand, the staff of the facility may handle virtually all routine chores and may work with the customer’s engineers for certain more specialized tasks.

Before looking at some significant engineering differences between Google and two of its major competitors, review this list of characteristics for a Google data center.

1. Google data centers – now numbering about two dozen, although no one outside Google knows the exact number or their locations. They come online and automatically, under the direction of the Google File System, start getting work from other data centers. These facilities, sometimes filled with 10,000 or more Google computers, find one another and configure themselves with minimal human intervention.

2. The hardware in a Google data center can be bought at a local computer store. Google uses the same types of memory, disc drives, fans and power supplies as those in a standard desktop PC.

3. Each Google server comes in a standard case called a pizza box with one important change: the plugs and ports are at the front of the box to make access faster and easier.

4. Google racks are assembled for Google to hold servers on their front and back sides. This effectively allows a standard rack, normally holding 40 pizza box servers, to hold 80.

5. A Google data center can go from a stack of parts to online operation in as little as 72 hours, unlike more typical data centers that can require a week or even a month to get additional resources online.

6. Each server, rack and data center works in a way that is similar to what is called “plug and play.” Like a mouse plugged into the USB port on a laptop, Google’s network of data centers knows when more resources have been connected. These resources, for the most part, go into operation without human intervention.

Several of these factors are dependent on software. This overlap between the hardware and software competencies at Google, as previously noted, illustrates the symbiotic relationship between these two different engineering approaches. At Google, from its inception, Google software and Google hardware have been tightly coupled. Google is not a software company nor is it a hardware company. Google is, like IBM, a company that owes its existence to both hardware and software. Unlike IBM, Google has a business model that is advertiser supported. Technically, Google is conceptually closer to IBM (at one time a hardware and software company) than it is to Microsoft (primarily a software company) or Yahoo! (an integrator of multiple softwares).

Software and hardware engineering cannot be easily segregated at Google. At MSN and Yahoo hardware and software are more loosely-coupled. Two examples will illustrate these differences.

Microsoft – with some minor excursions into the Xbox game machine and peripherals – develops operating systems and traditional applications. Microsoft has multiple operating systems, and its engineers are hard at work on the company’s next-generation of operating systems.

…

Several observations are warranted:

1. Unlike Google, Microsoft does not focus on performance as an end in itself. As a result, Microsoft gets performance the way most computer users do. Microsoft buys or upgrades machines. Microsoft does not fiddle with its operating systems and their subfunctions to get that extra time slice or two out of the hardware.

2. Unlike Google, Microsoft has to support many operating systems and invest time and energy in making certain that important legacy applications such as Microsoft Office or SQLServer can run on these new operating systems. Microsoft has a boat anchor tied to its engineer’s ankles. The boat anchor is the need to ensure that legacy code works in Microsoft’s latest and greatest operating systems.

3. Unlike Google, Microsoft has no significant track record in designing and building hardware for distributed, massively parallelised computing. The mice and keyboards were a success. Microsoft has continued to lose money on the Xbox, and the sudden demise of Microsoft’s entry into the home network hardware market provides more evidence that Microsoft does not have a hardware competency equal to Google’s.

…

Yahoo! operates differently from both Google and Microsoft. Yahoo! is in mid-2005 a direct competitor to Google for advertising dollars. Yahoo! has grown through acquisitions. In search, for example, Yahoo acquired 3721.com to handle Chinese language search and retrieval. Yahoo bought Inktomi to provide Web search. Yahoo bought Stata Labs in order to provide users with search and retrieval of their Yahoo! mail. Yahoo! also owns AllTheWeb.com, a Web search site created by FAST Search & Transfer. Yahoo! owns the Overture search technology used by advertisers to locate key words to bid on. Yahoo! owns Alta Vista, the Web search system developed by Digital Equipment Corp. Yahoo! licenses InQuira search for customer support functions. Yahoo has a jumble of search technology; Google has one search technology.

Historically Yahoo has acquired technology companies and allowed each company to operate its technology in a silo. Integration of these different technologies is a time-consuming, expensive activity for Yahoo. Each of these software applications requires servers and systems particular to each technology. The result is that Yahoo has a mosaic of operating systems, hardware and systems. Yahoo!’s problem is different from Microsoft’s legacy boat-anchor problem. Yahoo! faces a Balkan-states problem.

There are many voices, many needs, and many opposing interests. Yahoo! must invest in management resources to keep the peace. Yahoo! does not have a core competency in hardware engineering for performance and consistency. Yahoo! may well have considerable competency in supporting a crazy-quilt of hardware and operating systems, however. Yahoo! is not a software engineering company. Its engineers make functions from disparate systems available via a portal.

…

The figure below provides an overview of the mid-2005 technical orientation of Google, Microsoft and Yahoo.

2005 focuses of Google, MSN, and Yahoo

The Technology Precepts

… five precepts thread through Google’s technical papers and presentations. The following snapshots are extreme simplifications of complex, yet extremely fundamental, aspects of the Googleplex.

Cheap Hardware and Smart Software

Google approaches the problem of reducing the costs of hardware, set up, burn-in and maintenance pragmatically. A large number of cheap devices using off-the-shelf commodity controllers, cables and memory reduces costs. But cheap hardware fails.

In order to minimize the “cost” of failure, Google conceived of smart software that would perform whatever tasks were needed when hardware devices fail. A single device or an entire rack of devices could crash, and the overall system would not fail. More important, when such a crash occurs, no full-time systems engineering team has to perform technical triage at 3 a.m.

The focus on low-cost, commodity hardware and smart software is part of the Google culture.

…

Logical Architecture

Google’s technical papers do not describe the architecture of the Googleplex as self-similar. Google’s technical papers provide tantalizing glimpses of an approach to online systems that makes a single server share features and functions of a cluster of servers, a complete data center, and a group of Google’s data centers.

…

The collections of servers running Google applications on the Google version of Linux is a supercomputer. The Googleplex can perform mundane computing chores like taking a user’s query and matching it to documents Google has indexed. Further more, the Googleplex can perform side calculations needed to embed ads in the results pages shown to user, execute parallelized, high-speed data transfers like computers running state-of-the-art storage devices, and handle necessary housekeeping chores for usage tracking and billing.

…

When Google needs to add processing capacity or additional storage, Google’s engineers plug in the needed resources. Due to self-similarity, the Googleplex can recognize, configure and use the new resource. Google has an almost unlimited flexibility with regard to scaling and accessing the capabilities of the Googleplex.

…

In Google’s self-similar architecture, the loss of an individual device is irrelevant. In fact, a rack or a data center can fail without data loss or taking the Googleplex down. The Google operating system ensures that each file is written three to six times to different storage devices. When a copy of that file is not available, the Googleplex consults a log for the location of the copies of the needed file. The application then uses that replica of the needed file and continues with the job’s processing.

…

Speed and Then More Speed

…

Google uses commodity pizza box servers organized in a cluster. A cluster is group of computers that are joined together to create a more robust system. Instead of using exotic servers with eight or more processors, Google generally uses servers that have two processors similar to those found in a typical home computer.

Through proprietary changes to Linux and other engineering innovations, Google is able to achieve supercomputer performance from components that are cheap and widely available.

…

… engineers familiar with Google believe that read rates may in some clusters approach 2,000 megabytes a second. When commodity hardware gets better, Google runs faster without paying a premium for that performance gain.

…

Another key notion of speed at Google concerns writing computer programs to deploy to Google users. Google has developed short cuts to programming. An example is Google’s creating a library of canned functions to make it easy for a programmer to optimize a program to run on the Googleplex computer. At Microsoft or Yahoo, a programmer must write some code or fiddle with code to get different pieces of a program to execute simultaneously using multiple processors. Not at Google. A programmer writes a program, uses a function from a Google bundle of canned routines, and lets the Googleplex handle the details. Google’s programmers are freed from much of the tedium associated with writing software for a distributed, parallel computer.

…

Eliminate or Reduce Certain System Expenses

Some lucky investors jumped on the Google bandwagon early. Nevertheless, Google was frugal, partly by necessity and partly by design. The focus on frugality influenced many hardware and software engineering decisions at the company.

…

Drawbacks of the Googleplex

…

The Laws of Physics: Heat and Power 101

…

In reality, no one knows. Google has a rapidly expanding number of data centers. The data center near Atlanta, Georgia, is one of the newest deployed. This state-of-the-art facility reflects what Google engineers have learned about heat and power issues in its other data centers. Within the last 12 months, Google has shifted from concentrating its servers at about a dozen data centers, each with 10,000 or more servers, to about 60 data centers, each with fewer machines. The change is a response to the heat and power issues associated with larger concentrations of Google servers.

The most failure prone components are:

  • Fans.
  • IDE drives which fail at the rate of one per 1,000 drives per day.
  • Power supplies which fail at a lower rate.

Leveraging the Googleplex

…

Google’s technology is one major challenge to Microsoft and Yahoo. So to conclude this cursory and vastly simplified look at Google technology, consider these items:

1. Google is fast anywhere in the world.

2. Google learns. When the heat and power problems at dense data centers surfaced, Google introduced cooling and power conservation innovations to its two dozen data centers.

3. Programmers want to work at Google. “Google has cachet,” said one recent University of Washington graduate.

4. Google’s operating and scaling costs are lower than most other firms offering similar businesses.

5. Google squeezes more work out of programmers and engineers by design.

6. Google does not break down, or at least it has not gone offline since 2000.

7. Google’s Googleplex can deliver desktop-server applications now.

8. Google’s applications install and update without burdening the user with gory details and messy crashes.

9. Google’s patents provide basic technology insight pertinent to Google’s core functionality.

The X Window System defined

tech help, technology / By / 21 November 2008

From Ellen Siever’s “What Is the X Window System” (O’Reilly Media: 25 August 2005):

X was intentionally designed to provide the low-level mechanism for managing the graphics display, but not to have any control over what is displayed. This means that X has never been locked into a single way of doing things; instead, it has the flexibility to be used in many different ways. Both the simplest window manager and the most complex desktop environment can, and do, use the X Window System to manage the display.

…

When you run the X Window System, the X server manages the display, based on requests from the window manager. The window manager is an application that is itself an X client, with responsibility for managing the appearance and placement of the windows on the screen.

X itself has no role in determining the appearance of the screen, or what users are allowed to do with windows. That is the job of the window manager. For example, some window managers allow you to double-click in a window’s title bar and roll up the window into the title bar like rolling up a window shade (this is referred to as shading). Other window managers don’t have that feature. X doesn’t care; it’s a window manager concern. The X server’s job is to provide the low-level support so the window manager and other applications can shade or not, as they choose.

…

The X server manages the display hardware. The server captures input events from the user via keyboard or mouse (or other input device) and passes the information to a client application that has requested it. It also receives requests from the application to perform some graphical action. For example, if you use your mouse to move a window on the screen, the X server passes the information to the window manager, which responds by telling the server where to reposition the window, and the X server performs the action. If the client is a calculator, such as xcalc, it might request that digits be displayed into the window as the user clicks on buttons to enter a number.

…

The window manager controls the general operation of the window system; in particular, it controls the geometry and aesthetics of your X display. With the window manager you can change the size and position of windows on the display, reshuffle windows in a window stack, and so on.

A wireless router with 2 networks: 1 secure, 1 open

business, technology / By / 21 November 2008

From Bruce Schneier’s “My Open Wireless Network” (Crypto-Gram: 15 January 2008):

A company called Fon has an interesting approach to this problem. Fon wireless access points have two wireless networks: a secure one for you, and an open one for everyone else. You can configure your open network in either “Bill” or “Linus” mode: In the former, people pay you to use your network, and you have to pay to use any other Fon wireless network. In Linus mode, anyone can use your network, and you can use any other Fon wireless network for free. It’s a really clever idea.

Tim O’Reilly defines cloud computing

business, technology / By / 3 May 2009

From Tim O’Reilly’s “Web 2.0 and Cloud Computing” (O’Reilly Radar: 26 October 2008):

Since “cloud” seems to mean a lot of different things, let me start with some definitions of what I see as three very distinct types of cloud computing:

1. Utility computing. Amazon’s success in providing virtual machine instances, storage, and computation at pay-as-you-go utility pricing was the breakthrough in this category, and now everyone wants to play. Developers, not end-users, are the target of this kind of cloud computing.

This is the layer at which I don’t presently see any strong network effect benefits (yet). Other than a rise in Amazon’s commitment to the business, neither early adopter Smugmug nor any of its users get any benefit from the fact that thousands of other application developers have their work now hosted on AWS. If anything, they may be competing for the same resources.

That being said, to the extent that developers become committed to the platform, there is the possibility of the kind of developer ecosystem advantages that once accrued to Microsoft. More developers have the skills to build AWS applications, so more talent is available. But take note: Microsoft took charge of this developer ecosystem by building tools that both created a revenue stream for Microsoft and made developers more reliant on them. In addition, they built a deep — very deep — well of complex APIs that bound developers ever-tighter to their platform.

So far, most of the tools and higher level APIs for AWS are being developed by third-parties. In the offerings of companies like Heroku, Rightscale, and EngineYard (not based on AWS, but on their own hosting platform, while sharing the RoR approach to managing cloud infrastructure), we see the beginnings of one significant toolchain. And you can already see that many of these companies are building into their promise the idea of independence from any cloud infrastructure vendor.

In short, if Amazon intends to gain lock-in and true competitive advantage (other than the aforementioned advantage of being the low-cost provider), expect to see them roll out their own more advanced APIs and developer tools, or acquire promising startups building such tools. Alternatively, if current trends continue, I expect to see Amazon as a kind of foundation for a Linux-like aggregation of applications, tools and services not controlled by Amazon, rather than for a Microsoft Windows-like API and tools play. There will be many providers of commodity infrastructure, and a constellation of competing, but largely compatible, tools vendors. Given the momentum towards open source and cloud computing, this is a likely future.

2. Platform as a Service. One step up from pure utility computing are platforms like Google AppEngine and Salesforce’s force.com, which hide machine instances behind higher-level APIs. Porting an application from one of these platforms to another is more like porting from Mac to Windows than from one Linux distribution to another.

The key question at this level remains: are there advantages to developers in one of these platforms from other developers being on the same platform? force.com seems to me to have some ecosystem benefits, which means that the more developers are there, the better it is for both Salesforce and other application developers. I don’t see that with AppEngine. What’s more, many of the applications being deployed there seem trivial compared to the substantial applications being deployed on the Amazon and force.com platforms. One question is whether that’s because developers are afraid of Google, or because the APIs that Google has provided don’t give enough control and ownership for serious applications. I’d love your thoughts on this subject.

3. Cloud-based end-user applications. Any web application is a cloud application in the sense that it resides in the cloud. Google, Amazon, Facebook, twitter, flickr, and virtually every other Web 2.0 application is a cloud application in this sense. However, it seems to me that people use the term “cloud” more specifically in describing web applications that were formerly delivered locally on a PC, like spreadsheets, word processing, databases, and even email. Thus even though they may reside on the same server farm, people tend to think of gmail or Google docs and spreadsheets as “cloud applications” in a way that they don’t think of Google search or Google maps.

This common usage points up a meaningful difference: people tend to think differently about cloud applications when they host individual user data. The prospect of “my” data disappearing or being unavailable is far more alarming than, for example, the disappearance of a service that merely hosts an aggregated view of data that is available elsewhere (say Yahoo! search or Microsoft live maps.) And that, of course, points us squarely back into the center of the Web 2.0 proposition: that users add value to the application by their use of it. Take that away, and you’re a step back in the direction of commodity computing.

Ideally, the user’s data becomes more valuable because it is in the same space as other users’ data. This is why a listing on craigslist or ebay is more powerful than a listing on an individual blog, why a listing on amazon is more powerful than a listing on Joe’s bookstore, why a listing on the first results page of Google’s search engine, or an ad placed into the Google ad auction, is more valuable than similar placement on Microsoft or Yahoo!. This is also why every social network is competing to build its own social graph rather than relying on a shared social graph utility.

This top level of cloud computing definitely has network effects. If I had to place a bet, it would be that the application-level developer ecosystems eventually work their way back down the stack towards the infrastructure level, and the two meet in the middle. In fact, you can argue that that’s what force.com has already done, and thus represents the shape of things. It’s a platform I have a strong feeling I (and anyone else interested in the evolution of the cloud platform) ought to be paying more attention to.

How to run a command repeatedly

tech help / By / 2 October 2008

You can use the watch command, but it unfortunately isn’t available for Mac OS X. At least, from Apple. Sveinbjorn Thordarson (great name!) has a version of watch that you can download and compile on your OS X box. It’s available at http://www.sveinbjorn.org/watch_macosx.

Or, you can use this shell script:

while true ; do foo ; sleep 1 ; done

This will run foo every second until you press Ctrl-C to cancel the script.

I for one welcome our new OS overlords: Google Chrome

business, technology / By / 2 September 2008

As some of you may have heard, Google has announced its own web browser, Chrome. It’s releasing the Windows version today, with Mac & Linux versions to follow.

To educate people about the new browser & its goals, they release a 38 pg comic book drawn by the brilliant Scott McCloud. It’s a really good read, but it gets a bit technical at times. However, someone did a “Reader’s Digest” version, which you can read here:

http://technologizer.com/2008/09/01/google-chrome-comic-the-readers-digest-version

I highly encourage you to read it. This browser is doing some very interesting, smart things. And it’s open source, so other browsers can use its code & ideas.

If you want to read the full comic, you can do so here:

http://www.google.com/googlebooks/chrome/

BTW … I don’t think Chrome has the potential of becoming the next big browser; I think instead it has the potential to become the next big operating system. See http://www.techcrunch.com/2008/09/01/meet-chrome-googles-windows-killer/ for more on that.

Dropbox for Linux is coming soon

business, tech help, technology / By / 1 September 2008

According to this announcement, a Linux client for Dropbox should be coming out in a week or so:

http://forums.getdropbox.com/topic.php?id=2371&replies=1

I’ve been using Dropbox for several months, and it’s really, really great.

What is it? Watch this video:

http://www.getdropbox.com/screencast

It’s backup and auto-syncing done REALLY well. Best of all, you can sync between more than one computer, even if one is owned by someone else. So I could create a folder then share it with Robert. It shows up on his machine. If either of us changes files in the folder, those changes are auto-synced with each other.

Very nice.

So check it out when you get a chance. 2 GB are free. After that, you pay a small fee.

What actions change MAC times on a UNIX box?

tech help, technology / By / 20 April 2008

From Holt Sorenson’s “Incident Response Tools For Unix, Part Two: File-System Tools” (SecurityFocus: 17 October 2003):

Various commands change the MAC [modify, access, and change] times in different ways. The table below shows the effects that some common commands have on MAC times. These tables were created on Debian 3.0 using an ext2 file system contained in a flat file mounted on a loopback device. … Experimenting with your own system to verify the information in the tables below is encouraged. These tables can serve as a general guide, however.

How common commands change MACtimes for a directory (foo):
Action atime ctime mtime
creation (mkdir foo) X X X
directory move (mv foo bar) X X
file creation (touch foo/foo) X X
file creation (dd if=/dev/zero of=foo/foo count=1) X X
list directory (ls foo) X
change directory (cd foo)
file test (-f foo)
file move/rename (mv foo foo_mvd) X X
permissions change (chmod/chown <some_perm> foo) X
file copy (mv foo_mvd foo) X X
file edit (vim foo) X X
file edit (emacs foo) X X X
file edit (nvi/nano foo)
How common commands change MACtimes for a file (f1):
Action atime ctime mtime
creation (touch foo) X X X
creation (dd if=/dev/zero of=foo count=1) X X X
rename (mv foo bar)
permissions change (chmod <some_perm> foo) X
copy (cp foo bar) X
copy overwrite (cp bar foo) X X
append (cat >> foo) X X
overwrite (cat > foo) X X
truncate (cp /dev/null foo) X X
list file (ls foo)
edit (vim/emacs/xemacs/joe/jed foo) X X X
edit (ed/nvi/vi (sun)/vi (obsd)/nano/pico foo) X1 X1 X1
1 – all times changed, but atime is slightly older than mtime and ctime

The ls command can be used to show the modify, access or change times of files. The following table shows various ls commands that sort in reverse order by mtime, atime, or ctime. This causes ls to list the most recent times last.

displaying MACtimes using ls:
Linux (ls from GNU fileutils) OpenBSD Solaris
mtime ls -latr –full-time ls -latTr ls -latr
atime ls -laur –full-time ls -lauTr ls -laur
ctime ls -lacr –full-time ls -lacTr ls -lacr

Find out a hard drive’s UUID

1 Comment / tech help, technology / By / 26 July 2007

If you want to add a device like an external hard drive to your /etc/fstab file, it helps if you know the hard drive’s UUID. If you use K/Ubuntu, the following command will display the UUID, along with other useful information.

$ sudo vol_id /dev/sdo1
Password:
ID_FS_USAGE=filesystem
ID_FS_TYPE=ext3
ID_FS_VERSION=1.0
ID_FS_UUID=4857d4bb-5f6b-4f21-af62-830ebae92cff
ID_FS_LABEL=movies
ID_FS_LABEL_SAFE=movies

Ubuntu Edgy changes to fstab

tech help / By / 11 December 2006

I upgraded my Ubuntu Linux desktop today from Dapper to Edgy. It appears that in /etc/fstab, LABEL= no longer works, and you must now use UUID=.

http://ubuntuforums.org/showthread.php?t=278652

So my fstab now looks like this, for instance (these are all external drives):

UUID=a3d8a126-a7fc-4994-9675-748ed62c3109 /media/music           xfs      rw,user,noauto  0  0
UUID=e6e83a83-7487-4f22-a7ac-42cb100dfe24 /media/music-copy      reiserfs rw,user,noauto  0  0
UUID=99198c52-3f9e-4255-9326-7891a90223ac /media/temp            reiserfs rw,user,noauto  0  0
UUID=e0e73b81-f432-4b9e-918c-595fbfb1ac93 /media/data            ext3     rw,user,noauto  0  0
UUID=2296551a-1d7d-4aff-9aea-873121464c9a /media/data-copy       ext3     rw,user,noauto  0  0
UUID=e04e7b7a-b429-4a0f-a458-6af0c120bb9b /media/music-rock      xfs      rw,user,noauto  0  0
UUID=af39f5e1-1554-4dac-be5c-1f5028ee9503 /media/music-rock-copy xfs      rw,user,noauto  0  0

Edgy also converts any old fstab entries for /dev/hda1 and so on to the new UUID method as well.

For more on labels & uuid in fstab, see: http://ubuntuforums.org/showthread.php?t=283131