Category: Uncategorized

I’m giving a talk at Wordcamp 2009, May 30 in San Francisco. A portion of the talk is based on a new article that I’ve drafted, “How the Web and the Weblog have changed Writing”. For this article and talk I need examples of great 20-30 page essays that are published on the Web but not in print. I also need examples of 1-4 paragraph Weblog postings that show the power of the medium. I could illustrate the article exclusively with my owns writings, but upon a careful review of everything that I’ve written in the last 16 years it seems that I haven’t written anything great.

Suggestions would be welcome either via email to philg@mit.edu or in the comments section below.

Full post, including comments

A tale of two servers…

Server #1: I attended a presentation by a guy with a background in law and business, what we programmers commonly call “an idiot”. He had set out to build a site where people could type in the books, dvds, and video games that they owned and were willing to swap with others. Because he didn’t know anything about programming, he stupidly listened to Microsoft and did whatever they told him to do. He ended up with a SQL Server machine equipped with 64 GB of RAM and a handful of front-end machines running ASP.NET pages that he and few other mediocrities cobbled together as best they could. The site, swaptree.com, handles its current membership of more than 400,000 users (growing rapidly) without difficulty. A typical query looks like “In exchange for this copy of Harry Potter, what are the 400,000 other users willing to trade? And then for each of those things, what else might some third user (chosen from the 400,000) give in a three-way trade.” This kind of query is done many times a second and may return more than 20,000 rows.

Server #2: A non-technical friend hired an MIT-educated software engineer with 20 years of experience to build an innovative shopping site, presenting Amazon-style pages with thumbnails and product descriptions. Let’s call my friend’s site mitgenius.com. The programmer, being way smarter than the swaptree idiot, decided to use Ruby on Rails, the latest and greatest Web development tool. As only a fool would use obsolete systems such as SQL Server or Oracle, our brilliant programmer chose MySQL. What about hosting? A moron might have said “this is a simple site just crawling its way out of prototype stage; I’ll buy a server from Dell and park it in my basement with a Verizon FiOS line going out.” An MIT genius, though, would immediately recognize the lack of scalability and reliability inherent in this approach.

How do you get scale and reliability? Start by virtualizing everything. The database server should be a virtual “slice” of a physical machine, without direct access to memory or disk, the two resources that dumb old database administrators thought that a database management system needed. Ruby and Rails should run in some virtual “slices” too, restricted maybe to 500 MB or 800 MB of RAM. More users? Add some more slices! The cost for all of this hosting wizardry at an expert Ruby on Rails shop? $1100 per month.

For the last six months, my friend and his programmer have been trying to figure out why their site is so slow. It could take literally 5 minutes to load a user page. Updates to the database were proceeding at one every several seconds. Was the site heavily loaded? About one user every 10 minutes.

I began emailing the sysadmins of the slices. How big was the MySQL database? How big were the thumbnail images? It turned out that the database was about 2.5 GB and the thumbnails and other stuff on disk worked out to 10 GB. The servers were thrashing constantly and every database request went to disk. I asked “How could this ever have worked?” The database “slice” had only 5 GB of RAM. It was shared with a bunch of other sites, all of which were more popular than mitgenius.com. Presumably the database cache would be populated with pages from those other sites’ tables because they were more frequently accessed.

How could a “slice” with 800 MB of RAM run out of memory and start swapping when all it was trying to do was run an HTTP server and a scripting language interpreter? Only a dinosaur would use SQL as a query language. Much better to pull entire tables into Ruby, the most beautiful computer language ever designed, and filter down to the desired rows using Ruby and its “ActiveRecord” facility.

Not helping matters was the fact that the sysadmins found some public pages that went into MySQL 1500 times with 1500 separate queries (instead of one query returning 1500 rows).

In reviewing email traffic, I noticed much discussion of “mongrels” being restarted. I never did figure out what those were for.

As the MIT-trained software engineer had never produced any design documentation, I could not criticize his system design. However, I suggested naively that a site with 12.5 GB of data required to produce customer pages would need a server with at least 12.5 GB of RAM ($500 retail for the DIMMs?). In the event that different customers wanted to look at different categories of products, it would not be sufficient to have clever indices or optimized queries. Every time the server needed to go to disk it was going to be 100,000 times slower than pulling data from RAM.

My Caveman Oracle/Lisp programmer solution: 2U Dell server with 32 GB of RAM and two disk drives mirrored. No virtualization. MySQL and Ruby on Rails running as simultaneous processes in the same Linux. Configure the system with no swap file so that it will use all of its spare RAM as file system cache (we tore our hair out at photo.net trying to figure out why a Linux machine with 12 GB of RAM, bought specifically to serve JPEGs, would only use about 1/3rd for file system cache; it stumped all of the smartest sysadmins and the answer turned out to be removing the swap file). Park at local ISP and require that the programmer at least document enough of the system that the ISP’s sysadmin can install it. If usage grows to massive levels, add some front-end machines and migrate the Ruby on Rails processes to those.

What am I missing? To my inexperienced untrained-in-the-ways-of-Ruby mind, it would seem that enough RAM to hold the required data is more important than a “mongrel”. Can it be that simple to rescue my friend’s site?

[August 2009 update: The site has been running for a couple of months on its own cheapo Dell pizza box server with 16 GB of RAM. The performance problems that the Ruby on Rails experts had been chasing for months disappeared and the site is now responsive.]

[March 2010 update: My friend is now in discussions with some large companies interested in using his technology or service. Andrew Grumet and I published “Software Design Review” as a result of this experience.]

Full post, including comments

A friend in Bethesda, Maryland (Washington, D.C. suburb) needs to buy a car for her nanny. Mission: transport 2-year-old kid and sometimes big dog for a total of 7500 miles per year. Being fashionable is not important, though the car should be reasonably reliable and safe.

I thought that a 2008 Chevrolet Malibu from a rental car agency would be a good choice. Or maybe a Chrysler minivan with 50,000 miles on it.

Ideas? Where to get a good value used car in the DC/Maryland area?

Full post, including comments

Good article in today’s Boston Globe about the travails of being a Harvard undergraduate from a middle class family. The school doesn’t make sense academically for most undergrads; they’d be better off going to a college where the professors are paid to talk to undergraduates, rather than a research university where professors are paid to talk to post-docs and graduate students. The Ivy League line has always been “The professors will ignore you, but you’ll make a lot of connections with society’s future leaders.” It seems that doesn’t work for kids from average families. The future leaders don’t want to talk to them and they end up getting an inferior education to what they would have received at a school where they would have been an academic stand-out.

Full post, including comments

Ford was in the news today, selling 300 million new shares partly in order to pay medical expenses for union retirees (source). This will dilute existing shareholders who might well ask “How come the company needs to pay so much for retiree medical expenses?”

I think the answer is that a typical unionized auto worker would join a Detroit automaker at age 18. He would be eligible to retire 30 years later, at age 48, with a full pension including medical benefits for a wife and kids. For nearly 20 years, until this guy is eligible for Medicare, Ford will be paying his medical bills, his wife’s medical bills, and possibly his kids’ medical bills.

What about GM and Chrysler? The companies have already receive billions from taxpayers that they’ve used to meet retiree health care obligations. Under the terms of the latest Chrysler bailout, the union fund gets stock in Chrysler and then the government puts billions of taxpayer dollars into the post-Chapter 11 Chrysler. Essentially the U.S. taxpayer is paying these costs, though in a way that looks like investment rather than direct transfer. (Forbes magazine published a related article on the Chrysler bankruptcy today.)

Consider a 48-year-old woman who gets divorced and is forced to reenter the workforce. She goes to work at Walmart, the nation’s number one private employer, at a salary substantially less than an auto worker earned on the assembly line, while playing cards in a “job bank”, or collecting a pension. She won’t be able to retire until age 70. Every year that she works she will pay taxes to support a 48-year-old retired United Auto Worker who is enjoying his vacation house on a lake, water skiing behind his boat, etc.

This would not have surprised Mancur Olson, but it seems rather tough on America’s older workers.

Full post, including comments

Finished reading The Big Rich: The Rise and Fall of the Greatest Texas Oil Fortunes by Bryan Burrough. The first half of the book is the best, chronicling the discovery of different fields around Texas and Louisiana, the rise of some individuals, and the importance of oil to our efforts in World War II.

The second half is weaker, but it contains a full account of the Hunt brothers’ attempt to corner the world silver market in the late 1970s. Teamed with oil-rich Arabs, mostly Saudis, the Hunts ended up controlling 77 percent of the world’s privately held silver and, with the help of the Jimmy Carter-era hyperinflation, had driven the price up from $6 per ounce to $50 per ounce.

As government regulators cracked down on this speculation and the price of silver began to fall, banks who’d lent money to the Hunts were at risk. There were fears of a banking collapse and the Federal Reserve chairman, Paul Volcker, got involved to urge a rescue.

Intermittent chapters cover the pernicious influence of rich Texans on national politics. The author ridicules these guys for being “ultra right-wing” and mourns the fact that their money seeded a right-wing political movement. Burrough concentrates on the Texans’ fears of Communists and conspiracy. Burrough, who seems to long for a return to the days of FDR, does not describe what for modern readers would be an unimaginable degree of government control over private industry. The government decided how much oil a well owner could pump, at what price he could sell it, and where it could be sold. Shipping a petroleum product by rail? The government decided how much the railroad would charge for that. How about by truck? The government established rates (see Mancur Olson for how slowly the U.S. economy grew until the Ford Administration’s deregulation program took hold). It is not surprising that an industry hobbled by this much government regulation would attempt to lobby its way out, or at least get prices increased.

The book ends with a paragraph about how happy these old geezers might have been to see Ronbo, King Bush I, and then W. in the White House.

What about an account of how Texas morphed itself into the natural gas hub of the U.S.? Something about Enron and how those guys related to the old oil elite? Perhaps ending with Halliburton moving its headquarters to Dubai would have been more instructive for readers trying to understand how world power is shifting.

Conclusion: the book is worth reading for its family dramas and as a reminder that quickly built fortunes can evaporate just as quickly.

Full post, including comments

Robert Arnott has written an interesting article comparing stocks versus bonds as an investment: “Bonds: Why Bother?”

A few excerpts: “Starting any time we choose from 1979 through 2008, the investor in 20-year Treasuries (consistently rolling to the nearest 20-year bond and reinvesting income) beats the S&P 500 investor. In fact, from the end of February 1969 through February 2009, despite the grim bond collapse of the 1970s, our 20-year bond investors win by a nose.”

Arnott goes back to 1801 and notes that stocks do tend to return 2.5 percent per year more than bonds. Unfortunately this is of little comfort to a stock investor who buys in at a peak. The U.S. stock market has spent 173 out of 207 years below a previous peak. “The peak of 1802 was not convincingly exceeded until 1877, a startling 75 years later. … the drop from 1929–32 was so severe that share prices, expressed in real terms, briefly dipped below 1802 levels.” How about more recent history? “In real, inflation-adjusted terms, the 1965 peak for the S&P 500 was not exceeded until 1993, a span of 28 years.”

Arnott notes that an indexed investor suffers badly from a bubble in a particular stock or bond. If a stock goes up to a fantastic price level, e.g., Cisco in the dotcom boom, the index fund is forced to buy a lot of it.

“For the long-term investor, stock markets are supposed to give us steady gains, interrupted by periodic bear markets and occasional jolts like 1987 or 2008. The opposite—long periods of disappointment, interrupted by some wonderful gains—appears to be more accurate.”

Full post, including comments

Today I’m a judge for the MITX Technology Awards, held at Microsoft’s palatial new office in Kendall Square, Cambridge. Microsoft will be spending $100 million in remodeling and rent on this facility over the years, but apparently couldn’t find enough cash for a few $200 desktop machines to park in each conference room.

Each conference room does have a video projector and the building is covered by various 802.11 wireless networks. Anyone who wants to use the Web in a meeting is supposed to bring his or her own laptop and hook it up to the network and the projector. For our meeting, the ensuing harlequinade occupied two Microsoft employees and one guest for a full 15 minutes. When they were done we could see Web pages, but the entire screen would go blank every 30 seconds and stay dark for several seconds.

What will it take for corporate America to decide that a meeting room should have a hardwired Web browser, with some sort of screen viewable by all in the room?

Full post, including comments

I was just about to recycle the annual report that I received from Brookline Bancorp, a small savings and loan here in Massachusetts that is presumably typical of an old-school bank. Then I thought it might be interesting to see how the sleepy old way of doing business fared in 2008.

The bank had assets of $2.6 billion. Of that, more than $2 billion was money that it expected to get repaid on loans it had issued to borrowers. As long as the Massachusetts real estate market does not collapse, this money should continue to flow in. The bank allows only $28 million for expected loan losses, roughly 1% of the total owed to them. Loans that had been sold upstream and converted into cash had been invested almost exclusively in government-guaranteed securities (some of them Fannie Mae and the like, which were not explicitly guaranteed until recently).

Profit before taxes was $21 million. Roughly 40 percent of that profit was paid in taxes, leaving $12.8 million for investors, down from $21 million in 2006. (I.e., an investor in Brookline would give up roughly 50 percent of his or her profits to the government through the corporate tax and then through state and federal income tax on dividend payments.)

How did the employees do? They were paid $21 million in 2008, up from $19 million in 2006. The 74-year-old CEO was paid more than 10 percent of the total received by all 220 full-time employees. Richard P. Chapman, Jr. earned $2.24 million in 2008 (100X what a teller earns), $2.2M in 2007, and $1.7M in 2006. Was it necessary to pay the guy this much to prevent another bank from hiring him away? Presumably not, as his letter to shareholders indicated that he felt that he was too old to continue as CEO and would be retiring this year.

How does a guy at the end of a long career in local banking see our prospects? “All the risks in 2009 seem on the down side,” writes Mr. Chapman.

[Could the Board have gotten someone to manage 220 people for less than $2.24M? Consider Gary Kelly, the 52-year-old CEO of Southwest Airlines. He earned about $1M in 2008 (source) while managing an enterprise with $11 billion in revenue and more than 35,000 employees. Gary Kelly works in a much more challenging industry and is at much greater risk of having to work evenings or weekends, especially if there is a problem with one of Southwest’s more than 500 Boeing 737s.]

Full post, including comments

I flew with my first helicopter instrument student today (ink still wet on my Helicopter CFII). He drove all the way to Boston from Dayton, Ohio to fly with us because he hadn’t been happy with the instructors in his region. The guy was a reasonably good instrument pilot but somewhat befuddled with the big challenges of doing approaches. His instructors had taught him to start a timer at the final approach fix, even on an ILS in case the glide slope failed and they wanted to turn it into a localizer approach. He was supposed to use a timer to hold, and a timer for procedure turns, even though his previous trainer aircraft had a Garmin 430 GPS. They hadn’t taught him to use more than the most basic features of the Garmin.

I pointed out that the new FAA Practical Test Standards required an applicant to use a moving map if available. I noted that a professional crew of two airline pilots would not try to salvage an ILS into a localizer approach if the glide slope were to fail. Since they did not brief the LOC approach they would go missed, ask for delay vectors, brief the LOC approach and come back to do it. Why would a general aviation pilot by himself try to do something that an airline crew wouldn’t do?

What about timing? The Garmin shows you where to hold, how to enter the hold, where to start procedure turns, when you’ve reached the missed approach point. Why would you run a timer to second-guess the Garmin? If the Garmin fails, ask for vectors from ATC.

I told him that I had recently passed an ATP checkride where I timed nothing, telling the examiner that I was going to turn off my brain and rely on the Garmin. Why was he making his instrument checkride tougher than my ATP ride?

He is doing great and I think he’ll be ready for a checkride in another week or so. I said “As soon as you can do all of this with about 20 percent of your attention, you’re ready. You need to save the other 80 percent to watch for and handle the unexpected.”

Full post, including comments