Category: Uncategorized

Friends have asked about the crash of Colgan 3407, stalled by a very junior captain and a moderately junior first officer. How could two experienced pilots have made the kind of mistake that befalls rusty Private pilots in a little four-seater? Evidence suggests that both pilots were fatigued and one was sick. The captain had failed quite a few check rides, but the first officer had 1600 hours before she joined Colgan and held a current flight instructor certificate. She probably built up most of those 1600 hours saving students from their mistakes. Sick and tired, evidence suggests that she could not save the captain from his mistakes.

Politicians have focussed on the low pay for regional airline pilots. I think that AIG, Wall Street, and our public school systems demonstrate that paying employees more does not necessarily generate higher performance. Working for $19,000 per year and living with mom doesn’t sound very glamorous, but there are plenty of people who want to do it. Paying Rebecca Shaw more would not have saved the airplane and its passengers.

What could have saved the airplane? Well rested pilots.

Most airlines have a seniority-based system for everything, including, critically, scheduling. A senior pilot at a major airline might be able to arrange his schedule so that he need only work 8 or 10 days per month. He will be able to choose his home base so that it is close to his actual house. The senior pilot will have a short commute and 20-22 days per month of rest.

The junior pilot, by contrast, gets the trips and the schedule that the senior pilots don’t want. The result may be 22 days month of 16-hour days (measured not by flight time but by hotel room to hotel room). A typical 16-hour day may include a 6-hour stop at an airport where the airline does not have a base and therefore there will be nowhere for the pilot to rest. He or she will be sitting near a gate, in uniform, reading a book, trying to shut out the noise of thousands of passengers walking by and hundreds of public address announcements.

Note that the least experienced pilots at an airline are getting the least rest. The most experienced crews are getting the most rest. I.e., the crews that really need to be sharp to do the job are the ones who are flying while tired.

The seniority system for pay and schedule increases commuting time. Suppose that an experienced Boeing 737 captain lives in New York and flies for an airline with a New York base. His wife gets transferred to Los Angeles and he follows her. His airline doesn’t have a base on the west coast. You’d think that he would quit and join an LA-based airline flying Boeing 737s, right? Doing so would cost him a 70 percent cut in pay and a 50 percent increase in hours worked. Having lost all of his seniority, he would start as the most junior first officer at his new employer. It might take him 15 years to work his way back up to captain. What will he do? He can fly free on any airline, so he’ll keep his job in New York and start every 4-day trip with a 6-hour flight from Los Angeles, possibly followed by a night in a “crash pad” (2BR apartment shared with 10 other pilots).

Does it have to be done this way? No. NetJets and many other corporate jet operators have the same schedule for all of their pilots. In the case of NetJets, it is 7 days on and 7 days off.

A few simple ideas for improving airline safety and giving the future Rebecca Shaws a chance to save the passengers:

• require that airlines come up with a scheduling system that gives an equal amount of work and rest to all pilots (the average amount of work done by pilots would not change, so this should not cost the airlines anything extra)
• require that airports served by commercial airlines build crew rest lounges that pilots from any airline can use for naps, etc. (airports collect a hefty tax on every passenger who goes through, so this should not break their budgets)
• come up with a procedure whereby pilots can move from airline to airline without having to start over at the bottom of the pay scale (there will no longer be a “schedule scale” so we don’t have to worry about that), in order to discourage long-distance commuting

[Note that the typical tiny airport in the U.S., which the average person would call an “airstrip”, and which may not have any full-time staff, will have a comfortable lounge in which visiting pilots can rest. There will be sofas. There may be recliner chairs. Some of these small airports for private planes even have small bedrooms for naps. How come the guy flying a four-seat prop plane has a better place to rest than the pilot of a 150-passenger jet?]

Full post, including comments

I’m giving a talk at Wordcamp 2009, May 30 in San Francisco. A portion of the talk is based on a new article that I’ve drafted, “How the Web and the Weblog have changed Writing”. For this article and talk I need examples of great 20-30 page essays that are published on the Web but not in print. I also need examples of 1-4 paragraph Weblog postings that show the power of the medium. I could illustrate the article exclusively with my owns writings, but upon a careful review of everything that I’ve written in the last 16 years it seems that I haven’t written anything great.

Suggestions would be welcome either via email to philg@mit.edu or in the comments section below.

Full post, including comments

A tale of two servers…

Server #1: I attended a presentation by a guy with a background in law and business, what we programmers commonly call “an idiot”. He had set out to build a site where people could type in the books, dvds, and video games that they owned and were willing to swap with others. Because he didn’t know anything about programming, he stupidly listened to Microsoft and did whatever they told him to do. He ended up with a SQL Server machine equipped with 64 GB of RAM and a handful of front-end machines running ASP.NET pages that he and few other mediocrities cobbled together as best they could. The site, swaptree.com, handles its current membership of more than 400,000 users (growing rapidly) without difficulty. A typical query looks like “In exchange for this copy of Harry Potter, what are the 400,000 other users willing to trade? And then for each of those things, what else might some third user (chosen from the 400,000) give in a three-way trade.” This kind of query is done many times a second and may return more than 20,000 rows.

Server #2: A non-technical friend hired an MIT-educated software engineer with 20 years of experience to build an innovative shopping site, presenting Amazon-style pages with thumbnails and product descriptions. Let’s call my friend’s site mitgenius.com. The programmer, being way smarter than the swaptree idiot, decided to use Ruby on Rails, the latest and greatest Web development tool. As only a fool would use obsolete systems such as SQL Server or Oracle, our brilliant programmer chose MySQL. What about hosting? A moron might have said “this is a simple site just crawling its way out of prototype stage; I’ll buy a server from Dell and park it in my basement with a Verizon FiOS line going out.” An MIT genius, though, would immediately recognize the lack of scalability and reliability inherent in this approach.

How do you get scale and reliability? Start by virtualizing everything. The database server should be a virtual “slice” of a physical machine, without direct access to memory or disk, the two resources that dumb old database administrators thought that a database management system needed. Ruby and Rails should run in some virtual “slices” too, restricted maybe to 500 MB or 800 MB of RAM. More users? Add some more slices! The cost for all of this hosting wizardry at an expert Ruby on Rails shop? $1100 per month.

For the last six months, my friend and his programmer have been trying to figure out why their site is so slow. It could take literally 5 minutes to load a user page. Updates to the database were proceeding at one every several seconds. Was the site heavily loaded? About one user every 10 minutes.

I began emailing the sysadmins of the slices. How big was the MySQL database? How big were the thumbnail images? It turned out that the database was about 2.5 GB and the thumbnails and other stuff on disk worked out to 10 GB. The servers were thrashing constantly and every database request went to disk. I asked “How could this ever have worked?” The database “slice” had only 5 GB of RAM. It was shared with a bunch of other sites, all of which were more popular than mitgenius.com. Presumably the database cache would be populated with pages from those other sites’ tables because they were more frequently accessed.

How could a “slice” with 800 MB of RAM run out of memory and start swapping when all it was trying to do was run an HTTP server and a scripting language interpreter? Only a dinosaur would use SQL as a query language. Much better to pull entire tables into Ruby, the most beautiful computer language ever designed, and filter down to the desired rows using Ruby and its “ActiveRecord” facility.

Not helping matters was the fact that the sysadmins found some public pages that went into MySQL 1500 times with 1500 separate queries (instead of one query returning 1500 rows).

In reviewing email traffic, I noticed much discussion of “mongrels” being restarted. I never did figure out what those were for.

As the MIT-trained software engineer had never produced any design documentation, I could not criticize his system design. However, I suggested naively that a site with 12.5 GB of data required to produce customer pages would need a server with at least 12.5 GB of RAM ($500 retail for the DIMMs?). In the event that different customers wanted to look at different categories of products, it would not be sufficient to have clever indices or optimized queries. Every time the server needed to go to disk it was going to be 100,000 times slower than pulling data from RAM.

My Caveman Oracle/Lisp programmer solution: 2U Dell server with 32 GB of RAM and two disk drives mirrored. No virtualization. MySQL and Ruby on Rails running as simultaneous processes in the same Linux. Configure the system with no swap file so that it will use all of its spare RAM as file system cache (we tore our hair out at photo.net trying to figure out why a Linux machine with 12 GB of RAM, bought specifically to serve JPEGs, would only use about 1/3rd for file system cache; it stumped all of the smartest sysadmins and the answer turned out to be removing the swap file). Park at local ISP and require that the programmer at least document enough of the system that the ISP’s sysadmin can install it. If usage grows to massive levels, add some front-end machines and migrate the Ruby on Rails processes to those.

What am I missing? To my inexperienced untrained-in-the-ways-of-Ruby mind, it would seem that enough RAM to hold the required data is more important than a “mongrel”. Can it be that simple to rescue my friend’s site?

[August 2009 update: The site has been running for a couple of months on its own cheapo Dell pizza box server with 16 GB of RAM. The performance problems that the Ruby on Rails experts had been chasing for months disappeared and the site is now responsive.]

[March 2010 update: My friend is now in discussions with some large companies interested in using his technology or service. Andrew Grumet and I published “Software Design Review” as a result of this experience.]

Full post, including comments

A friend in Bethesda, Maryland (Washington, D.C. suburb) needs to buy a car for her nanny. Mission: transport 2-year-old kid and sometimes big dog for a total of 7500 miles per year. Being fashionable is not important, though the car should be reasonably reliable and safe.

I thought that a 2008 Chevrolet Malibu from a rental car agency would be a good choice. Or maybe a Chrysler minivan with 50,000 miles on it.

Ideas? Where to get a good value used car in the DC/Maryland area?

Full post, including comments

Good article in today’s Boston Globe about the travails of being a Harvard undergraduate from a middle class family. The school doesn’t make sense academically for most undergrads; they’d be better off going to a college where the professors are paid to talk to undergraduates, rather than a research university where professors are paid to talk to post-docs and graduate students. The Ivy League line has always been “The professors will ignore you, but you’ll make a lot of connections with society’s future leaders.” It seems that doesn’t work for kids from average families. The future leaders don’t want to talk to them and they end up getting an inferior education to what they would have received at a school where they would have been an academic stand-out.

Full post, including comments

Ford was in the news today, selling 300 million new shares partly in order to pay medical expenses for union retirees (source). This will dilute existing shareholders who might well ask “How come the company needs to pay so much for retiree medical expenses?”

I think the answer is that a typical unionized auto worker would join a Detroit automaker at age 18. He would be eligible to retire 30 years later, at age 48, with a full pension including medical benefits for a wife and kids. For nearly 20 years, until this guy is eligible for Medicare, Ford will be paying his medical bills, his wife’s medical bills, and possibly his kids’ medical bills.

What about GM and Chrysler? The companies have already receive billions from taxpayers that they’ve used to meet retiree health care obligations. Under the terms of the latest Chrysler bailout, the union fund gets stock in Chrysler and then the government puts billions of taxpayer dollars into the post-Chapter 11 Chrysler. Essentially the U.S. taxpayer is paying these costs, though in a way that looks like investment rather than direct transfer. (Forbes magazine published a related article on the Chrysler bankruptcy today.)

Consider a 48-year-old woman who gets divorced and is forced to reenter the workforce. She goes to work at Walmart, the nation’s number one private employer, at a salary substantially less than an auto worker earned on the assembly line, while playing cards in a “job bank”, or collecting a pension. She won’t be able to retire until age 70. Every year that she works she will pay taxes to support a 48-year-old retired United Auto Worker who is enjoying his vacation house on a lake, water skiing behind his boat, etc.

This would not have surprised Mancur Olson, but it seems rather tough on America’s older workers.

Full post, including comments

Finished reading The Big Rich: The Rise and Fall of the Greatest Texas Oil Fortunes by Bryan Burrough. The first half of the book is the best, chronicling the discovery of different fields around Texas and Louisiana, the rise of some individuals, and the importance of oil to our efforts in World War II.

The second half is weaker, but it contains a full account of the Hunt brothers’ attempt to corner the world silver market in the late 1970s. Teamed with oil-rich Arabs, mostly Saudis, the Hunts ended up controlling 77 percent of the world’s privately held silver and, with the help of the Jimmy Carter-era hyperinflation, had driven the price up from $6 per ounce to $50 per ounce.

As government regulators cracked down on this speculation and the price of silver began to fall, banks who’d lent money to the Hunts were at risk. There were fears of a banking collapse and the Federal Reserve chairman, Paul Volcker, got involved to urge a rescue.

Intermittent chapters cover the pernicious influence of rich Texans on national politics. The author ridicules these guys for being “ultra right-wing” and mourns the fact that their money seeded a right-wing political movement. Burrough concentrates on the Texans’ fears of Communists and conspiracy. Burrough, who seems to long for a return to the days of FDR, does not describe what for modern readers would be an unimaginable degree of government control over private industry. The government decided how much oil a well owner could pump, at what price he could sell it, and where it could be sold. Shipping a petroleum product by rail? The government decided how much the railroad would charge for that. How about by truck? The government established rates (see Mancur Olson for how slowly the U.S. economy grew until the Ford Administration’s deregulation program took hold). It is not surprising that an industry hobbled by this much government regulation would attempt to lobby its way out, or at least get prices increased.

The book ends with a paragraph about how happy these old geezers might have been to see Ronbo, King Bush I, and then W. in the White House.

What about an account of how Texas morphed itself into the natural gas hub of the U.S.? Something about Enron and how those guys related to the old oil elite? Perhaps ending with Halliburton moving its headquarters to Dubai would have been more instructive for readers trying to understand how world power is shifting.

Conclusion: the book is worth reading for its family dramas and as a reminder that quickly built fortunes can evaporate just as quickly.

Full post, including comments

Robert Arnott has written an interesting article comparing stocks versus bonds as an investment: “Bonds: Why Bother?”

A few excerpts: “Starting any time we choose from 1979 through 2008, the investor in 20-year Treasuries (consistently rolling to the nearest 20-year bond and reinvesting income) beats the S&P 500 investor. In fact, from the end of February 1969 through February 2009, despite the grim bond collapse of the 1970s, our 20-year bond investors win by a nose.”

Arnott goes back to 1801 and notes that stocks do tend to return 2.5 percent per year more than bonds. Unfortunately this is of little comfort to a stock investor who buys in at a peak. The U.S. stock market has spent 173 out of 207 years below a previous peak. “The peak of 1802 was not convincingly exceeded until 1877, a startling 75 years later. … the drop from 1929–32 was so severe that share prices, expressed in real terms, briefly dipped below 1802 levels.” How about more recent history? “In real, inflation-adjusted terms, the 1965 peak for the S&P 500 was not exceeded until 1993, a span of 28 years.”

Arnott notes that an indexed investor suffers badly from a bubble in a particular stock or bond. If a stock goes up to a fantastic price level, e.g., Cisco in the dotcom boom, the index fund is forced to buy a lot of it.

“For the long-term investor, stock markets are supposed to give us steady gains, interrupted by periodic bear markets and occasional jolts like 1987 or 2008. The opposite—long periods of disappointment, interrupted by some wonderful gains—appears to be more accurate.”

Full post, including comments

Today I’m a judge for the MITX Technology Awards, held at Microsoft’s palatial new office in Kendall Square, Cambridge. Microsoft will be spending $100 million in remodeling and rent on this facility over the years, but apparently couldn’t find enough cash for a few $200 desktop machines to park in each conference room.

Each conference room does have a video projector and the building is covered by various 802.11 wireless networks. Anyone who wants to use the Web in a meeting is supposed to bring his or her own laptop and hook it up to the network and the projector. For our meeting, the ensuing harlequinade occupied two Microsoft employees and one guest for a full 15 minutes. When they were done we could see Web pages, but the entire screen would go blank every 30 seconds and stay dark for several seconds.

What will it take for corporate America to decide that a meeting room should have a hardwired Web browser, with some sort of screen viewable by all in the room?

Full post, including comments

I was just about to recycle the annual report that I received from Brookline Bancorp, a small savings and loan here in Massachusetts that is presumably typical of an old-school bank. Then I thought it might be interesting to see how the sleepy old way of doing business fared in 2008.

The bank had assets of $2.6 billion. Of that, more than $2 billion was money that it expected to get repaid on loans it had issued to borrowers. As long as the Massachusetts real estate market does not collapse, this money should continue to flow in. The bank allows only $28 million for expected loan losses, roughly 1% of the total owed to them. Loans that had been sold upstream and converted into cash had been invested almost exclusively in government-guaranteed securities (some of them Fannie Mae and the like, which were not explicitly guaranteed until recently).

Profit before taxes was $21 million. Roughly 40 percent of that profit was paid in taxes, leaving $12.8 million for investors, down from $21 million in 2006. (I.e., an investor in Brookline would give up roughly 50 percent of his or her profits to the government through the corporate tax and then through state and federal income tax on dividend payments.)

How did the employees do? They were paid $21 million in 2008, up from $19 million in 2006. The 74-year-old CEO was paid more than 10 percent of the total received by all 220 full-time employees. Richard P. Chapman, Jr. earned $2.24 million in 2008 (100X what a teller earns), $2.2M in 2007, and $1.7M in 2006. Was it necessary to pay the guy this much to prevent another bank from hiring him away? Presumably not, as his letter to shareholders indicated that he felt that he was too old to continue as CEO and would be retiring this year.

How does a guy at the end of a long career in local banking see our prospects? “All the risks in 2009 seem on the down side,” writes Mr. Chapman.

[Could the Board have gotten someone to manage 220 people for less than $2.24M? Consider Gary Kelly, the 52-year-old CEO of Southwest Airlines. He earned about $1M in 2008 (source) while managing an enterprise with $11 billion in revenue and more than 35,000 employees. Gary Kelly works in a much more challenging industry and is at much greater risk of having to work evenings or weekends, especially if there is a problem with one of Southwest’s more than 500 Boeing 737s.]

Full post, including comments