unix

Talk about Markdown to SLUUG this Wednesday

personal, technology / By / 7 November 2011

I’ll be giving a talk to the St. Louis UNIX Users Group next Wednesday night about Markdown, a tool I absolutely love.

You’re invited to come. Please do – I think you’ll definitely learn a lot.

Date: Wednesday, Nov. 9, 2011
Time: 6:30 – 9 pm
Where: 11885 Lackland Rd., St Louis, MO 63146
Map: http://g.co/maps/6gg9g
Directions: http://www.sluug.org/resources/meeting_info/map_graybar.shtml

Here’s the description:

John Gruber, the inventor of Markdown, describes it this way: “Markdown is a text-to-HTML conversion tool for web writers. Markdown allows you to write using an easy-to-read, easy-to-write plain text format, then convert it to structurally valid XHTML (or HTML). Thus, ‘Markdown’ is two things: (1) a plain text formatting syntax; and (2) a software tool, written in Perl, that converts the plain text formatting to HTML. … The overriding design goal for Markdown’s formatting syntax is to make it as readable as possible. The idea is that a Markdown-formatted document should be publishable as-is, as plain text, without looking like it’s been marked up with tags or formatting instructions.”

This talk by Scott Granneman & Bill Odom will cover the basics of Markdown’s syntax, key variants of Markdown, tools for composing Markdown (including vim, of course!), and ways you can easily transform a plain text file written in Markdown into HTML, JSON, TXT, LaTeX, man, MediaWiki, Textile, DocBook XML, ODT, EPUB, Slidy and S5 HTML and JavaScript slide shows, RTF, or even Word!

If you have any questions, please contact me. Hope to see you there!

Speaking at SLUUG: Amazing, Stupendous, Mind-Blowing Apps for iPad2

business, personal, technology / By / 5 June 2011

Jans Carton & I are delivering a talk at the St. Louis UNIX Users Group at 6:30 pm this Wednesday, 8 June 2011, titled “Amazing, Stupendous, Mind-Blowing Apps for iPad2”. We’ll be demoing iPad apps live for everyone. If you want to find out more about the iPad, or discover some awesome new iPad apps, then come hear Jans & I speak.

Here’s the complete description:

It’s becoming crystal clear that tablets and other mobile devices are leading the computing revolution into the next decade, and at the forefront (at least for now) is Apple’s iPad. Scott Granneman & Jans Carton will demonstrate iPad apps that they find particularly useful, cool, or amazing – sometimes all three at the same time! You’ll see apps for everything from video to games to reading to productivity, and much, much more. We guarantee you’ll see something that makes you think, and something else that makes you go “Wow!”.

The meeting will be held at Graybar Electric Co., Inc. at 11885 Lackland Road, 63146. Directions are at http://sluug.org/resources/meeting_info/map_graybar.shtml, & a Google Map can be found at http://goo.gl/maps/jnpX.

Unix: An Oral History

business, history, technology / By / 24 November 2010

From ‘s “Unix: An Oral History” (: ):

Multics

Gordon M. Brown

…

[Multics] was designed to include fault-free continuous operation capabilities, convenient remote terminal access and selective information sharing. One of the most important features of Multics was to follow the trend towards integrated multi-tasking and permit multiple programming environments and different human interfaces under one operating system.

…

Moreover, two key concepts had been picked up on in the development of Multics that would later serve to define Unix. These were that the less important features of the system introduced more complexity and, conversely, that the most important property of algorithms was simplicity. Ritchie explained this to Mahoney, articulating that:

The relationship of Multics to [the development of Unix] is actually interesting and fairly complicated. There were a lot of cultural things that were sort of taken over wholesale. And these include important things, [such as] the hierarchical file system and tree-structure file system – which incidentally did not get into the first version of Unix on the PDP-7. This is an example of why the whole thing is complicated. But any rate, things like the hierarchical file system, choices of simple things like the characters you use to edit lines as you’re typing, erasing characters were the same as those we had. I guess the most important fundamental thing is just the notion that the basic style of interaction with the machine, the fact that there was the notion of a command line, the notion was an explicit shell program. In fact the name shell came from Multics. A lot of extremely important things were completely internalized, and of course this is the way it is. A lot of that came through from Multics.

The Beginning

Michael Errecart and Cameron Jones

Files to Share

…

The Unix file system was based almost entirely on the file system for the failed Multics project. The idea was for file sharing to take place with explicitly separated file systems for each user, so that there would be no locking of file tables.

…

A major part of the answer to this question is that the file system had to be open. The needs of the group dictated that every user had access to every other user’s files, so the Unix system had to be extremely open. This openness is even seen in the password storage, which is not hidden at all but is encrypted. Any user can see all the encrypted passwords, but can only test one solution per second, which makes it extremely time consuming to try to break into the system.

…

The idea of standard input and output for devices eventually found its way into Unix as pipes. Pipes enabled users and programmers to send one function’s output into another function by simply placing a vertical line, a ‘|’ between the two functions. Piping is one of the most distinct features of Unix …

Language from B to C

… Thompson was intent on having Unix be portable, and the creation of a portable language was intrinsic to this. …

Finding a Machine

Darice Wong & Jake Knerr

… Thompson devoted a month apiece to the shell, editor, assembler, and other software tools. …

Use of Unix started in the patent office of Bell Labs, but by 1972 there were a number of non-research organizations at Bell Labs that were beginning to use Unix for software development. Morgan recalls the importance of text processing in the establishment of Unix. …

Building Unix

Jason Aughenbaugh, Jonathan Jessup, & Nicholas Spicher

…

The Origin of Pipes

The first edition of Thompson and Ritchie’s The Unix Programmer’s Manual was dated November 3, 1971; however, the idea of pipes is not mentioned until the Version 3 Unix manual, published in February 1973. …

…

Software Tools

…

grep was, in fact, one of the first programs that could be classified as a software tool. Thompson designed it at the request of McIlroy, as McIlroy explains:

One afternoon I asked Ken Thompson if he could lift the regular expression recognizer out of the editor and make a one-pass program to do it. He said yes. The next morning I found a note in my mail announcing a program named grep. It worked like a charm. When asked what that funny name meant, Ken said it was obvious. It stood for the editor command that it simulated, g/re/p (global regular expression print)….From that special-purpose beginning, grep soon became a household word. (Something I had to stop myself from writing in the first paragraph above shows how firmly naturalized the idea now is: ‘I used ed to grep out words from the dictionary.’) More than any other single program, grep focused the viewpoint that Kernighan and Plauger christened and formalized in Software Tools: make programs that do one thing and do it well, with as few preconceptions about input syntax as possible.

…

eqn and grep are illustrative of the Unix toolbox philosophy that McIlroy phrases as, “Write programs that do one thing and do it well. Write programs to work together. Write programs that handle text streams, because that is a universal interface.” This philosophy was enshrined in Kernighan and Plauger’s 1976 book, Software Tools, and reiterated in the “Foreword” to the issue of The Bell Systems Technical Journal that also introduced pipes.

…

Ethos

Robert Murray-Rust & Malika Seth

…

McIlroy says,

This is the Unix philosophy. Write programs that do one thing and do it well. Write programs to work together. Write programs that handle text streams because, that is a universal interface.

…

The dissemination of Unix, with a focus on what went on within Bell Labs

Steve Chen

…

In 1973, the first Unix applications were installed on a system involved in updating directory information and intercepting calls to numbers that had been changed. This was the first time Unix had been used in supporting an actual, ongoing operating business. Soon, Unix was being used to automate the operations systems at Bell Laboratories. It was automating the monitoring, involved in measurement, and helping to rout calls and ensure the quality of the calls.

…

There were numerous reasons for the friendliness the academic society, especially the academic Computer Science community, showed towards Unix. John Stoneback relates a few of these:

Unix came into many CS departments largely because it was the only powerful interactive system that could run on the sort of hardware (PDP-11s) that universities could afford in the mid ’70s. In addition, Unix itself was also very inexpensive. Since source code was provided, it was a system that could be shaped to the requirements of a particular installation. It was written in a language considerably more attractive than assembly, and it was small enough to be studied and understood by individuals. (John Stoneback, “The Collegiate Community,” Unix Review, October 1985, p. 27.)

…

The key features and characteristics of Unix that held it above other operating systems at the time were its software tools, its portability, its flexibility, and the fact that it was simple, compact, and efficient. The development of Unix in Bell Labs was carried on under a set of principles that the researchers had developed to guide their work. These principles included:

(i) Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new features.

(ii) Expect the output of every program to become the input to another, as yet unknown, program. Don’t clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don’t insist on interactive input.

(iii) Design and build software, even operating systems, to be tried early, ideally within weeks. Don’t hesitate to throw away the clumsy parts and rebuild them.

(iv) Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you’ve finished using them.”

(M.D. McIlroy, E.N. Pinson, and B.A. Tague “Unix Time-Sharing System Forward,” The Bell System Technical Journal, July-Aug 1088 vol 57, number 6 part 2. P. 1902)

Reasons Windows has a poor security architecture

security / By / 26 March 2009

From Daniel Eran Dilger’s “The Unavoidable Malware Myth: Why Apple Won’t Inherit Microsoft’s Malware Crown” (AppleInsider: 1 April 2008):

Thanks to its extensive use of battle-hardened Unix and open source software, Mac OS X also has always had security precautions in place that Windows lacked. It has also not shared the architectural weaknesses of Windows that have made that platform so easy to exploit and so difficult to clean up afterward, including:

  • the Windows Registry and the convoluted software installation mess related to it,
  • the Windows NT/2000/XP Interactive Services flaw opening up shatter attacks,
  • a wide open, legacy network architecture that left unnecessary, unsecured ports exposed by default,
  • poorly designed network sharing protocols that failed to account for adequate security measures,
  • poorly designed administrative messaging protocols that failed to account for adequate security,
  • poorly designed email clients that gave untrusted scripts access to spam one’s own contacts unwittingly,
  • an integrated web browser architecture that opened untrusted executables by design, and many others.

How to run a command repeatedly

tech help / By / 2 October 2008

You can use the watch command, but it unfortunately isn’t available for Mac OS X. At least, from Apple. Sveinbjorn Thordarson (great name!) has a version of watch that you can download and compile on your OS X box. It’s available at http://www.sveinbjorn.org/watch_macosx.

Or, you can use this shell script:

while true ; do foo ; sleep 1 ; done

This will run foo every second until you press Ctrl-C to cancel the script.

What actions change MAC times on a UNIX box?

tech help, technology / By / 20 April 2008

From Holt Sorenson’s “Incident Response Tools For Unix, Part Two: File-System Tools” (SecurityFocus: 17 October 2003):

Various commands change the MAC [modify, access, and change] times in different ways. The table below shows the effects that some common commands have on MAC times. These tables were created on Debian 3.0 using an ext2 file system contained in a flat file mounted on a loopback device. … Experimenting with your own system to verify the information in the tables below is encouraged. These tables can serve as a general guide, however.

How common commands change MACtimes for a directory (foo):
Action atime ctime mtime
creation (mkdir foo) X X X
directory move (mv foo bar) X X
file creation (touch foo/foo) X X
file creation (dd if=/dev/zero of=foo/foo count=1) X X
list directory (ls foo) X
change directory (cd foo)
file test (-f foo)
file move/rename (mv foo foo_mvd) X X
permissions change (chmod/chown <some_perm> foo) X
file copy (mv foo_mvd foo) X X
file edit (vim foo) X X
file edit (emacs foo) X X X
file edit (nvi/nano foo)
How common commands change MACtimes for a file (f1):
Action atime ctime mtime
creation (touch foo) X X X
creation (dd if=/dev/zero of=foo count=1) X X X
rename (mv foo bar)
permissions change (chmod <some_perm> foo) X
copy (cp foo bar) X
copy overwrite (cp bar foo) X X
append (cat >> foo) X X
overwrite (cat > foo) X X
truncate (cp /dev/null foo) X X
list file (ls foo)
edit (vim/emacs/xemacs/joe/jed foo) X X X
edit (ed/nvi/vi (sun)/vi (obsd)/nano/pico foo) X1 X1 X1
1 – all times changed, but atime is slightly older than mtime and ctime

The ls command can be used to show the modify, access or change times of files. The following table shows various ls commands that sort in reverse order by mtime, atime, or ctime. This causes ls to list the most recent times last.

displaying MACtimes using ls:
Linux (ls from GNU fileutils) OpenBSD Solaris
mtime ls -latr –full-time ls -latTr ls -latr
atime ls -laur –full-time ls -lauTr ls -laur
ctime ls -lacr –full-time ls -lacTr ls -lacr

Differences between Macintosh & Unix programmers

business, history, tech in changing society, technology / By / 30 July 2006

From Eric Steven Raymond’s “Problems in the Environment of Unix” (The Art of Unix Programming: 19 September 2003):

Macintosh programmers are all about the user experience. They’re architects and decorators. They design from the outside in, asking first “What kind of interaction do we want to support?” and then building the application logic behind it to meet the demands of the user-interface design. This leads to programs that are very pretty and infrastructure that is weak and rickety. In one notorious example, as late as Release 9 the MacOS memory manager sometimes required the user to manually deallocate memory by manually chucking out exited but still-resident programs. Unix people are viscerally revolted by this kind of mal-design; they don’t understand how Macintosh people could live with it.

By contrast, Unix people are all about infrastructure. We are plumbers and stonemasons. We design from the inside out, building mighty engines to solve abstractly defined problems (like “How do we get reliable packet-stream delivery from point A to point B over unreliable hardware and links?”). We then wrap thin and often profoundly ugly interfaces around the engines. The commands date(1), find(1), and ed(1) are notorious examples, but there are hundreds of others. Macintosh people are viscerally revolted by this kind of mal-design; they don’t understand how Unix people can live with it. …

In many ways this kind of parochialism has served us well. We are the keepers of the Internet and the World Wide Web. Our software and our traditions dominate serious computing, the applications where 24/7 reliability and minimal downtime is a must. We really are extremely good at building solid infrastructure; not perfect by any means, but there is no other software technical culture that has anywhere close to our track record, and it is one to be proud of. …

To non-technical end users, the software we build tends to be either bewildering and incomprehensible, or clumsy and condescending, or both at the same time. Even when we try to do the user-friendliness thing as earnestly as possible, we’re woefully inconsistent at it. Many of the attitudes and reflexes we’ve inherited from old-school Unix are just wrong for the job. Even when we want to listen to and help Aunt Tillie, we don’t know how — we project our categories and our concerns onto her and give her ‘solutions’ that she finds as daunting as her problems.

Cultural differences between Unix and Windows

business, history, technology / By / 5 July 2006

From Joel Spolsky’s “Biculturalism” (Joel on Software: 14 December 2003):

What are the cultural differences between Unix and Windows programmers? There are many details and subtleties, but for the most part it comes down to one thing: Unix culture values code which is useful to other programmers, while Windows culture values code which is useful to non-programmers.

This is, of course, a major simplification, but really, that’s the big difference: are we programming for programmers or end users? Everything else is commentary. …

Let’s look at a small example. The Unix programming culture holds in high esteem programs which can be called from the command line, which take arguments that control every aspect of their behavior, and the output of which can be captured as regularly-formatted, machine readable plain text. Such programs are valued because they can easily be incorporated into other programs or larger software systems by programmers. To take one miniscule example, there is a core value in the Unix culture, which Raymond calls “Silence is Golden,” that a program that has done exactly what you told it to do successfully should provide no output whatsoever. It doesn’t matter if you’ve just typed a 300 character command line to create a file system, or built and installed a complicated piece of software, or sent a manned rocket to the moon. If it succeeds, the accepted thing to do is simply output nothing. The user will infer from the next command prompt that everything must be OK.

This is an important value in Unix culture because you’re programming for other programmers. As Raymond puts it, “Programs that babble don’t tend to play well with other programs.” By contrast, in the Windows culture, you’re programming for Aunt Marge, and Aunt Marge might be justified in observing that a program that produces no output because it succeeded cannot be distinguished from a program that produced no output because it failed badly or a program that produced no output because it misinterpreted your request.

Similarly, the Unix culture appreciates programs that stay textual. They don’t like GUIs much, except as lipstick painted cleanly on top of textual programs, and they don’t like binary file formats. This is because a textual interface is easier to program against than, say, a GUI interface, which is almost impossible to program against unless some other provisions are made, like a built-in scripting language. Here again, we see that the Unix culture values creating code that is useful to other programmers, something which is rarely a goal in Windows programming.

Which is not to say that all Unix programs are designed solely for programmers. Far from it. But the culture values things that are useful to programmers, and this explains a thing or two about a thing or two. …

The Unix cultural value of visible source code makes it an easier environment to develop for. Any Windows developer will tell you about the time they spent four days tracking down a bug because, say, they thought that the memory size returned by LocalSize would be the same as the memory size they originally requested with LocalAlloc, or some similar bug they could have fixed in ten minutes if they could see the source code of the library. …

When Unix was created and when it formed its cultural values, there were no end users. Computers were expensive, CPU time was expensive, and learning about computers meant learning how to program. It’s no wonder that the culture which emerged valued things which are useful to other programmers. By contrast, Windows was created with one goal only: to sell as many copies as conceivable at a profit. …

For example, Unix has a value of separating policy from mechanism which, historically, came from the designers of X. This directly led to a schism in user interfaces; nobody has ever quite been able to agree on all the details of how the desktop UI should work, and they think this is OK, because their culture values this diversity, but for Aunt Marge it is very much not OK to have to use a different UI to cut and paste in one program than she uses in another.

Unix specs vs. Windows specs

business, history, tech help, technology / By / 16 June 2006

From Peter Seebach’s Standards and specs: Not by UNIX alone (IBM developerWorks: 8 March 2006):

In the past 20 years, developers for “the same” desktop platform (“whatever Microsoft ships”) have been told that the API to target is (in this order):

* DOS
* Win16
* OS/2
* Win32
* WinNT
* WinXP
* and most recently .NET.

Of course, that list is from last year, and now the “stable” target that you should be developing for, if you have an eye for the future, is Vista.

It hasn’t been quite as bad in the Macintosh world, where the number of major API changes has been limited: classic single-tasking Mac OS, classic multitasking Mac OS (System 7), Carbon (System 8/9 and preview of OS X), and Cocoa (OS X), but even there, the cost of migration has been significant. At least OS X finally offers a stable UNIX API for the back-end part of programs, allowing developers to ignore the API creep except in GUI code.

By contrast, twenty-year-old UNIX utilities still compile and run. A new desktop computing API will come and everyone will have to rewrite for it, but mountains will erode away before read() and write() stop working. This is the reason that all the hassle of formal UNIX standards has had so little effect on practical UNIX software development; the core API is simple, clean, and well-designed, and there is no need to change it significantly.

… UNIX users have been switching hardware platforms since the 1970s; it’s no big deal. …

Just as there are many varieties of UNIX, there are many UNIX standards:

* Probably the oldest standard that people still refer to is AT&T’s 1985 System V Interface Definition (SVID). This standard shows up, for instance, in man pages describing the standards compliance of functions that have been in the C library “forever.”
* Meanwhile, X/Open (now the Open Group) was developing “portability guides” with names like XPG2, XPG3, and so on. XPG1 was actually released in 1995. The XPG guides are largely subsumed into newer specs, but once again, are still referred to sometimes in documentation.
* The IEEE’s POSIX standard showed up in 1990 with updates in 1992 and 1993 and a second edition in 1996. It’s still a viable standard, although it has suffered from poor accessibility. POSIX specs have names like 1003.x; for instance, 1003.1 and 1003.2, which refer to different parts of the standard, or 1003.1-1988 and 1003.1-1990, which refer to two versions of the standard.
* The fairly ominous sounding “Spec 1170” (also known as “UNIX 98” or “Single Unix Specification”) is probably the most complete specification; it is produced by the Open Group, and is effectively a descendant of the XPG series. In practice, this is “the” UNIX standard these days, although it’s a little large; this has had an impact on conformance testing.
* The Linux Standards Base is not strictly a UNIX standard, but it’s a standardization effort relevant to a very large number of developers working with code designed to run “on UNIX.” …

You can look at OS specifications in two very different ways: one is from the point of view of a developer trying to port an application, and the other is from the point of view of the user trying to interact with the system.

UNIX conveniently blurs this distinction. The primary user interface is also one of the primary development environments; therefore, UNIX specifications often cover not only the C language API, but also the shell environment and many of the core utilities shell programmers rely on. …

From the perspective of a developer who’s seen many Unix-like systems, Linux is probably mostly sort of similar to System V. The heavy focus on GNU utilities gives a sort of surreal combination of Berkeley and System V features, but if you have to guess whether Linux does something the Berkeley way or the System V way, go with System V. This is especially true of system startup; nearly all Linux systems use the System V /etc/inittab and /etc/rc.d structure, or something very close to it. …

Unix vs Windows: NYC vs Celebration

security, tech in changing society, technology / By / 14 June 2006

From David HM Spector’s Unfinished Business Part 2: Closing the Circle (LinuxDevCenter: 7 July 2003):

The UNIX world is the result of natural evolution, not the outgrowth of a planned community. UNIX is a lot like New York City: dynamic, always reinventing itself, adapting to new needs and realities. Windows is a lot like Celebration, USA: static, a set piece of predictability, slow to provide new services and very resistant to change or difference of view or opinion.

The inventor of UNIX on its security … or lack thereof

history, security, technology / By / 13 June 2006

From Dennis M. Ritchie’s “On the Security of UNIX” (: ):

The first fact to face is that UNIX was not developed with security, in any realistic sense, in mind; this fact alone guarantees a vast number of holes. (Actually the same statement can be made with respect to most systems.) The area of security in which UNIX is theoretically weakest is in protecting against crashing or at least crippling the operation of the system. The problem here is not mainly in uncritical acceptance of bad parameters to system calls – there may be bugs in this area, but none are known – but rather in lack of checks for excessive consumption of resources. Most notably, there is no limit on the amount of disk storage used, either in total space allocated or in the number of files or directories. Here is a particularly ghastly shell sequence guaranteed to stop the system:

while : ; do
mkdir x
cd x
done

Either a panic will occur because all the i-nodes on the device are used up, or all the disk blocks will be consumed, thus preventing anyone from writing files on the device. …

The picture is considerably brighter in the area of protection of information from unauthorized perusal and destruction. Here the degree of security seems (almost) adequate theoretically, and the problems lie more in the necessity for care in the actual use of the system. …

It must be recognized that the mere notion of a super-user is a theoretical, and usually practical, blemish on any protection scheme. …

On the issue of password security, UNIX is probably better than most systems. Passwords are stored in an encrypted form which, in the absence of serious attention from specialists in the field, appears reasonably secure, provided its limitations are understood. … We have observed that users choose passwords that are easy to guess: they are short, or from a limited alphabet, or in a dictionary. Passwords should be at least six characters long and randomly chosen from an alphabet which includes digits and special characters.

Of course there also exist feasible non-cryptanalytic ways of finding out passwords. For example: write a program which types out login: on the typewriter and copies whatever is typed to a file of your own. Then invoke the command and go away until the victim arrives. …

The fact that users can mount their own disks and tapes as file systems can be another way of gaining super-user status. Once a disk pack is mounted, the system believes what is on it. Thus one can take a blank disk pack, put on it anything desired, and mount it. There are obvious and unfortunate consequences. For example: a mounted disk with garbage on it will crash the system; one of the files on the mounted disk can easily be a password-free version of su; other files can be unprotected entries for special files. The only easy fix for this problem is to forbid the use of mount to unprivileged users. A partial solution, not so restrictive, would be to have the mount command examine the special file for bad data, set-UID programs owned by others, and accessible special files, and balk at unprivileged invokers.

A brief history of backdoors

history, security, technology / By / 29 November 2005

From Network Magazine:

Ken Thompson, a designer of the Unix OS, explained his magic password, a password that once allowed him to log in as any user on any Unix system, during his award acceptance speech at the Association for Computing Machinery (ACM) meeting in 1984. Thompson had included a backdoor in the password checking function that gets included in the login program. The backdoor would get installed in new versions of the Unix system because the compiler had Trojan Horse code that propagated the backdoor code to new versions of the compiler. Thompson’s magic password is the best known, and most complex in distribution, backdoor code.