A friend recently attended a demo of Microsoft’s latest operating system, Longhorn. This is the long-delayed replacement for Windows NT, which was introduced in the early 1990s and has been improved into Win2000 and WinXP. The main risk for a project like Longhorn is Second System Syndrome. A group of programmers is given an existing product and a list of 100 things that people have said they don’t like about the product. They boldly plan to build a Second (new) System from scratch that will solve all of the problems the First System solved plus have the 100 new nice-to-have features. Traditionally Second Systems are late and often, on balance, don’t solve any more problems than the First System. Longhorn was particularly at risk due to the fact that a larger challenge was attacked with very similar tools to those used to build Windows NT circa 1990. Programmers haven’t gotten any smarter since 1990; how can we expect better results than were obtained back then unless there is a shift to radically new tools?
Longhorn has in fact been running quite late (two years?) and Microsoft has been reducing the scope of its ambition. My friend was underwhelmed by the result. One of the big features of Longhorn is a more searchable file system, something that WinXP users can get right now by downloading Google Desktop Search. Longhorn will let you create a virtual folder that represents the results of a persistent search, e.g., you can have a folder with all files containing the word “Samoyed” and it will be updated quickly when a new matching document appears anywhere in the file system.
Does Longhorn have a versioned file system? No. I.e., you can’t ask the system to show you what a spreadsheet or document looked like two months ago. Probably the vast majority of user-created documents in a file system are there because of this lack of versioning in WinXP’s NTFS file system. You have “Whizco Contract”, “Whizco Contract pre-lawyers”, “Whizco Contract post-legal-review”, “Whizco Contract with comments from Whizco”, “Whizco Contract 20050510”, “Whizco Contract Final”, “Whizco Contract Final Signature Copy”, etc. If Longhorn had a versioned file system, as became available for commercial Unices in the early 1990s, there would only be perhaps 1/10th as many user-created documents on the typical system.
The hierarchical file system with its folders and subfolders was created for mainframe programmers in the early 1960s. Let’s call it a Multics-style file system to pay homage to this pioneering system built in the 1960s by MIT and Honeywell. Alan Cooper, one of the creators of Visual Basic, and a perennial consultant to Microsoft, has persuasively attacked the idea of exposing this hierarchy to end-users. It was built by programmers for programmers but somehow leaked out into consumer consciousness with the Apple Macintosh and the original DOS on the IBM PC. Do we need this? How many documents can one person create? Even a professional writer such as Stephen King hasn’t generated an overwhelming number of stories and novels.
Why not start with a versioned file system? Then we can get rid of the “save” and “save as” commands in word processors and spreadsheet programs and replace them with a “name this version” command. When you’re done working on a document you close it. If you want to go back a couple of months you ask for the version circa 20050301 or a version with a specific name. Given a versioned file system we provide access to documents via a chronology. If you’re looking for something related to your 1999 taxes, scroll back to April 1999 and it will probably be there. If that doesn’t work resort to a full-text search. If we need to organize a bit more let us aggregate documents in named folders, as in a mail-reading program, most of which don’t allow or don’t encourage subfolders.
There is nothing wrong with the hierarchical file system as a tool. Assuming you know a file’s name, it provides O[log N] access to a corpus of N documents. This makes a hierarchical file system great for computer programs and computer programmers but why should users have to see the innards?
Apple and the Linux folks aren’t doing any better in this area than Microsoft. For years I have been hoping that Google or Yahoo! will show the way with a replacement for Microsoft Office and the underlying desktop file system. The average household user of a personal computer doesn’t need anything with many more features that the Palm OS or Microsoft Outlook and probably has far fewer megabytes of documents than he or she has of archived email.
It may be that my imagination is constrainted because the heirchical file system is all I’ve ever known, but I think exposing some sort of a heirchical structure to the user (regardless of how the underlying implementation works) fits in with the way most people think.
And really, nothing in today’s system prevents users from not using the heirchical nature at all if they don’t want to. I’m sure we’ve all seen coworkers with damn near every document they’ve ever received or created sitting right on their desktop in a virtual garbage pile.
However, a versioning file system would be *extremely* nice( though companies like ClearCase would probably vigorously disagree). But that’s the programmer in me talking. Just as you think the heirchical file system is of no use to regular users, I think a versioning file system would not be appreciated by these same users. MS is probably spending their resources wisely on 3D accelerated widgets, which are sure to cause the typical user’s eyes to widen in awe. Tell a user about a versioning file system, and they’re more likely to glaze over in boredom. And this is why marketing/sales departments driving strategy is occasionally a good thing.
Sounds great, but hard to comprehend. I can figure out folders because I know what a physical folder is.
A versioning file system was a feature of RSX-11M since mid-70’s and of VAX VMS from the very beginning. It’s surprising it didn’t find its way into NT via Dave Cutler’s team.
A non-hierarchical filesystem would probably make system and program files less accessible to users. The windows way of installing programs with it’s DLL hell is already a mess, imagine not being able to get into the black box that holds them to fix it? Before we can have a replacement non-hierarchical filesystem, lots of other things needs changing and made more reliable. The need for backward compatibility makes this virtualy impossible for MS.
But it would have been much easier for Apple with it’s single self-contained program files and where installing, say, another video codec involves nothing more than dropping the file in one location. They should have done it in OS X.
Very soon, you should be able to mount a DAV-enabled Subversion repository as a Network share (you can in fact do that now). If you turn Autoversioning on it on, it does just what you describe: save every version of the document. Of course, it doesn’t have the very easy search in the history that you propose, but at least it’s a nice step towards what you’re describing.
The user does not need to see *all* the files, like on de Mac where applications are just objects to the user, under the hood they are folders with folders and files in them.
Currently a folder knows which files belong to it. The files don’t contain information about its folder(s). I would prefer if this was reversed: So that one could ‘tag’ a file with a ‘category’ and this category would then be a folder. Some behavior should then change: renaming, deletion and merging of folders for example.
One could imagine that one could use these tags also for file typing, ‘image’, ‘movie’, ‘jpg’, ‘pdf’, ‘executable’ instead of the very weird filename.ext hack currently being used. These should be hidden (interpreted) by the gui.
Lastly, if we use a gui, then there is no real reason to demand a filename for every file: give it a preview icon. We don’t name every pencel on our desk so why demand a name for every bit of data? And with search technology nothing will be lost. So maybe the filename could be seen more as optional comment? This works the other way as well: At other times you would want to be able to give to a few names to a file (in different contexts).
Versioning for source files as us geeks do is easy but I am just wondering how well you could diff something like an 8MB Power Point file with lots of images. I don’t think Office would make edits to the underlying file, rather it saves it as new every time and the position of almost all data in the file could differ quite a bit. Make 20 minor changes, and do you end up with 160MB in your repository, or can it be more efficient than that?
Another thing that can be changed in today’s computers : an “Application” is a technical term. I don’t see why people should worry about which application is taking care of which function in a computer. Every thing must be integrated in the same manner in a computer. MS is tending toward this kind of “total” integration with “Office Suite”, but they don’t go farther enough. They should provide an OS that add software organization constraints to software developers in order to achieve a “total” integration (this should also cover the software delivery process, and a business model…), instead, MS is putting effort in allowing software developers to develop in any ways they like. Result : you got an unstable OS and unstable software with no good integration.
@Bas: What about a separate diff for every file format? One can imagine that a text-diff should be different from an xml-diff. Two xml files could different in text and still generate the same xml-tree, maybe make the file system aware of the type of data? The same could then apply to graphics. Now, I do think that compression could become a problem. Maybe wait for faster and bigger hdd’s?
The directory structure presented to and created by the user does NOT yield the logarithmic access time. Rather this is achieved by storing additional information about the records in the so called Catalog File B*-tree of the HFS.
But a B*-Tree or any other search tree with access times of O(log n) could also be used as an index for a flat file system. Thus logarithmic access time is not an unique feature of the hierarchical file system.
What’s mutually exclusive about versioning and hierarchy? Admittedly, it’s been a very long time, but I (think I) remember subdirectories and automatic versioning on TENEX/TOPS-20 lo, those many years ago (circa 1978). Much the same capability was present (as an earlier comment noted) on VMS. Oh, and I have another (recovered?) memory that at one time Windows NT was supposed to support pluggable file systems (with MS providing FAT and NTFS) – which would have let some enterprising developers build versioning.
I’m surprised that nobody has pointed out that Windows DOES have a versioned file system. It’s called the Volume Shadow Copy Service. It’s just not well publicized.
Also, Longhorn hasn’t “arrived.”
Also, Longhorn doesn’t qualify as “second system syndrome” because there is no attempt to build a new system from scratch.
Ahhh, “versioning”. I have my own concept called Distributed Virtual Personal Computer (DVPC) that includes something I call “smart versioning”. The trick is not to name or number your versions, but just treat them as a chronological. Users intuitively understand time and the idea of “show me what this document looked like yesterday afternoon” is truly intuitive. In fact, organizing all of your data chronologically is easy to grasp (“Show me all the work I did over the past week” [so I can do my status report]).
My concept proposal was intended to virtualize data storage and eliminate user concerns about backup, but smart versioning seemed just as important.
See: http://basetechnology.com/dvpc.htm
It’s only an idea, a vision, a concept, and given the sad state of basic technology research, I don’t ever expect that it or anything similar to it will ever be implemented.
Finally, the user-oriented goal is not “versioning” per se, but simply to make a 100% commitment to preserving all work done by the user.
I also agree that by definition Longhorn could not be suffering from “second system syndrome” since MS has long since passed the “version 2” milestone. You could consider NT as a “second system”, but even that hurdle has long since been exceeded.
— Jack Krupansky
Interesting idea to embed the versioning process into the OS.
Here’s an historical “fact” (e.g. true to the best of my knowledge, but fundamentally impossible to prove scientifically)… in 1994 I found that just in IBM “big iron” mainframe space (at least some one can pretty much count them… 15,000 licenses) no more than 10% had PURCHASED an automated source versioning tool. (We’ll not discuss the chasm between purchase & proper install & organizational use).
Naturally I was appalled. If this was the best the “pros” could do after 40+ years of mainframe work, what was the situation out on “little boxes”… Unix, AS400, VAX, Windows, Mac….?
One Unix/academic friend tells me that EVERYONE on Unix uses CVS (or equivalent)… I harbor doubts.
Software configuration/version management is a less than loved activity… lots & lots of people using computers know absolutely zilch about the folder/file structure & could care less.
Matter of fact… anyone out there EVER had (or heard of any) serious training on how to organize folders/files.
Dawn has broken over marble head… it has occurred to me that folks over in library (… err… “special collections”) land have been dealing with such mundane filing/classification/indexing/cataloging issues for several millennia… how come computer folks seem to believe they’re wrestling with a new problem?
To me losing access to the heiarchical file system would be tantamount to loosing control.
Take a look at Neal Stephensons essay “In the beginning was the command line”:
http://www.spack.org/wiki/InTheBeginningWasTheCommandLine
Neal used to love his sleek MacOS interface until one day his current manuscript was GONE… No recovery, no browsing through old temp files or directories, just.. gone…
This concept was alien to me, since I have been attacking the file system with undelete tools, etc since the late ’80’s.
Why would anybody want to lose so much control? Well, welcome to the database, just the sort of thing to store each file as an object with adequate meta-data to facilitate storage and retrieval… Not to mention all of those groovy backup and recovery tools.. ooo and you can even support the whole thing on a a raw file system.
Sure, its a better tool, can be represnted on the front end in any manner that the end user can be trained to use, but please, please,please don’t try and lock me out from some low-level tool that will allow me to (with proper authentication) roll up my sleeves and toss out all the chrome-y front ends to get the data out of the back end structure.
Can you really expect MS to meet those kind od requirements? How about AS400?
Eh, I don’t get the lure of versioning, at least for the layperson, as a computer science disciple myself, I agree with the eggheads who’ve demonstrated and promoted the superiority over a hierarchical filesystem. Most users today have grokked the folders concept and are comfortable with it, and changing it would lead to an increase in annoy/irritant factor for them. Second, you say Apple (I assume you mean OS X) hasn’t addressed problem, but I think they’ve produced an elegant upgrade to the Unix filesystem standard where ~/Documents, ~/Music, ~/Movies, etc. all hang off of your home directory and stay outside the Applications and system repositories on the machine, in stark contrast to Windows implementations (though they are becoming more *nix-like also) where all the gunk goes into “Program Files”. And if you use your machine heavily for over 6 months, the advantage of a journalized filesystem (like many Unix and OS X offer) is a big benefit over running Win XP (don’t know if Longhorn is journalized or not, have eschewed the platform for the last 5 years in lieu of Linux/Mac OS X, unless required to do so by work, but now most dev shops are going Linux/Mac too, at least for web development)
And your characterization of a hierarchical filesystem being rooted in “mainframe programmers” is off the mark, if you consider that MVS and OS/360, which were the dominant industry hardware serving commercial interests, actually are more linear in nature than DoS/Unix – typically, most files of interest (that change frequently or semi-frequently) are lumped into a PDS (partitioned dataset) where thousands of 8 character filenames comingle and a scheme is typically utilized to indicate type, owner, etc
“Alan Cooper, one of the creators of Visual Basic, and a perennial consultant to Microsoft, has persuasively attacked the idea of exposing this [file] hierarchy to end-users. It was built by programmers for programmers but somehow leaked out into consumer consciousness with the Apple Macintosh and the original DOS on the IBM PC. Do we need this?”
I just ran across an interesting application called “Archy”, apparently developed by the late Jef Raskin, which doesn’t use the file metaphor. You don’t have to open files, save them, etc.; you just start typing.
http://rchi.raskincenter.org/aboutrchi/
It’s still in the prototype stage. Right now it looks a lot like a text editor.
Maybe the whole concept of files, folders, and applications is to hard for most people? They where developed more or less by programmers for programmers. Maybe it’s not the right solution for all? We’ve replaced the romain numeral system, which some could use very well, with something anyone could learn easy. Why not do the same with the filesystem when it gets in our way?
Maybe that a digital soup is more useable? It would at the very least require much less training.
ADVERTISING FEATURE
Sorry to spam, but I hope not too off-topic. I just released the new version of SdiDesk, which is the very preliminary beginning of a project which tries to wrap the hierarchical file system beneath a version controlled, wiki-like interface. Might interest people looking for ideas about alternatives.
http://www.nooranch.com/sdidesk/wiki/wiki.cgi
/ADVERTISING FEATURE
Hilarious — Joel is yet again right that there is a poorly-marketed (yet technically workable) Microsoft solution. See the following obscure document:
http://www.microsoft.com/technet/prodtechnol/windowsserver2003/library/TechRef/2b0d2457-b7d8-42c3-b6c9-59c145b7765f.mspx
If Apple had developed this, it would be branded TimeTraveler and would be the central feature of a $150 minimum upgrade code-named “Wildcat.” Steve Jobs would make an incredibly slick demo where he uses a slider control to rewind the development of an “Incredibles” scene from finished product to sketch. The TV ad would be set to “Bob Dylan’s Dream” and feature some kind of version control scenario to make you cry, like someone reverting his father’s eulogy to make it more sentimental.
Microsoft, on the other hand, decided to go with Volume Shadow Copy Service. Maybe by 3.0 it will be shortened to “ShadowCopy TM” ;->
Phil,
That’s a great idea. I can’t remember how many times CVS has saved my sorry ass. It would rock if something like that was applied to the file system as a whole.
I am on OS X 10.4, and Spotlight is very nice when you have a ton of stuff in your home folder. As new filters arrive it will be able to index most of the file types that we use (not that it matters, 99% of the files we generate at work are either ascii, MS Office formats or PDF, all of which are already indexed by Spotlight).
Sometimes I wish I could query my filesystem the same way I query a database server. Just learn ANSI SQL and you would be able to find whatever the hell is it you need!
Sure, if you define “better” as “increasing the artificial stratification between ‘users’ and ‘programmers’ by restricting the former to a dumbed-down and less-useful set of tools by which machines dictate work flow and idea organization to human beings rather than the other way around”, then yes…Linux and MacOS are “no better than Microsoft”. And thank God for that!
Now of course, Microsoft users are already accustomed to having their choices dictated from Redmond, and probably would agree with your attitude. I for one am very glad that, for those of us not likewise brainwashed, there’s Linux.
Most people are intuitively capable of understanding hierarchical models of organization. Adding versioning features is a nice idea, and would probably improve things for a lot of people. But no versioning filesystem I’ve ever seen has deprived the user of their ability to exploit hierarchical organization when the task the user is performing is suited to that. (Note also that it’s the USER who decides what’s suitable, rather than the software.)
And while we’re at it, what’s with this bogus distinction between “email archives” and “documents”? Email archives ARE documents. This is not even an “everything is a file” unix-ism…Microsoft apps treat them the same way.
Sorry, Phil…you’re a great writer and a great programmer, but your dreams for the future of the computer industry feel a lot like nightmares to me.
Hilarious, Ryan!
Ryan,
I can see why MS is not shouting about Shadow Copy from the treetops…
This concept is nothing new!
It has been possible to store ‘change vectors’ and thereby provide ‘point in time’ data recovery from SAN products for as long as I have worked with the technology (five years at least).
It seemed alien to me, because Oracle has offered ‘point in time’ recovery for as long as I can remember (15 years).
fwsiw, the model of database as a file system is (I think) already well established in the AS/400 line of products.
So, why would somebody NOT want to integrate a database into/as their operating system?performanceperformanceperformance
There is just a ton of overhead maintaining indexes, catalogs,etc that do not exist on current file systems. In fact there are more than a few Oracle systems running today that live on raw devices with (essentially) no file system.
If you want to see a real shiny, happy future with full version control, a nice user interface, some nifty development languages and (hopefully) an oracle back end then skip on over to documentum.com
They were recently acquired by emc…
MS…Innovation? pffft!
Glad you liked it JR. Gary, I hear you, but when has “old concept” EVER stopped Microsoft from shouting something from the treetops? I’m not gonna say they never innovate but their bread and butter is taking existing concepts — graphical operating system, stable server OS, Web browser — and trying to improve them for the masses, even when other people were there first. And, really, the same is true for Apple. I could install svn or cvs and put all my files in it and have what Philip is talking about, but that’s not the point. The point is it would be truly cool if someone like Microsoft or Apple made the technology easy. Maybe documentum.com is leading the way, maybe they’ll get acquired by Google, I don’t know. In any case, I suspect Microsoft is keeping quiet because 1> they just don’t do marketing as well as Apple, 2> maybe they don’t realize how cool ShadowCopy could be for the masses, 3> maybe they don’t think it’s ready. But I doubt 3> considering they’ll be touting many G1 feaurtes in Longhorn.
PS RE performance, Microsoft’s raison d’etre for at least the past ten years seems to be “Converting Moore’s Law directly into profit.” In other words, when you buy a zippy new 3 GHz processor with more RAM than a Bill Clinton-era hard drive, Microsoft’s job is to add enough features that you don’t much notice the speedup ;->
MS VSCS fails in this department as it isn’t easy to use by your mom. What you need is is very simple control panel where you can select which places you want versioned. (“My Documents” is default) The you should be able to just right click on any file and select “Versions>View>Some Older version”. A little try icon could pop up when it needs your attention, like if the repository is getting a bit full and some older version need to go.
The windows FS has enough hooks in it to not even need it integrated with the OS core; a third party developer could semi-easily do it.
Of course a completely integrated OS/Application suite would be nice; apps can use the OS API to make versioning accesible from within the app itself.
Longhorn is far more than 2 years late. The concept of a distributed, searchable file system was proposed for the Cairo project, which was NT 4.0. It was first announced at the professional developers conference in Anahim, 1994. So if Longhorn goes GA next year it will only be 12 years late.
As for the whole need for a versioning file system: the majority of corporate users don’t need it. The technical ones already have one, most likely – CVS, SourceSafe, ClearCase or whatever. The average Word user could probably do without folders entirely and throw everything into one directory. Heck, a lot of them do just that – sometimes because their corporate IT people make them save to a shared drive. It’s no answer.
No one is mentioning the fact that a hierarchical file system allows one to use the same filename in two different folders (directories). Otherwise you would have the burden of giving each file a unique name.
An earlier comment proposed making filenames optional, but that doesn’t seem like a solution to me. Sure, we don’t name every pencil on our desk, but that is because the pencils are interchangeable. People, cities, streets, pets, and yes, files, are not interchangeable and therefore get names.
@ Eric: The main problem with folders and files currently (which sometimes makes me go insane) is that most documents don’t fit this or that folder. They belong in more then one. Copy is not reasonable because then they go out of sync, sym-link are not a solution because they break when you move them to another disk…. I really just want tags: Being able to assign a few categories to a file and have this form a folder would be a much beter solution. You can always make it more or less back-ward by tweaking the copy and move behavior. Only the deletion of a folder could be tricky…
Doesn’t Tiger’s Spotlight feature cover some of the features of Longhorn’s canned search folders? I haven’t really had time to play with all of the features of Spotlight, but it is my understanding that one can create ‘loaders’ for various document types and Spotlight will index the meta-data. That way, you will be able to have a file in multiple ‘smart’ folders based on meta-data. Maybe there really is a use for the properties window in MS Apps.
Gideon: That would be just like how today’s image management applications work; using set association (cf. http://www.photools.com/imfs_0003.php)
How many documents can I create? Easily, by now, over 50K once I include all the references, presentations, emails, images, etc. It’s not that hard, honest.
Hierarchy, especially poly-hierarchy, is an interesting and useful form of metadata (think inheritance and specialization). It’s not for everyone, but it’s very useful. I’ve been disappointed how many search tools ignore the metadata of the containing directory (doing the descendant tree is hard, so I’ll excuse that omission).
“The average household user of a personal computer doesn’t need anything with many more features that the Palm OS or Microsoft Outlook and probably has far fewer megabytes of documents than he or she has of archived email.”
Depends on what you mean by “document”. I have less than 100 MB of “documents” if you mean word processing documents. But I have well over 6000 digital photos (and that’s a small number of photos for even an amateur photographer). I have nearly 15,000 songs in iTunes, occupying over 90 GB of hard drive space. I just had to buy a new hard drive because I filled up a 200 GB hard drive with captured DV (my music is on a separate hard drive, so the 200 GB is nearly all DV).
Cooking Channel
Racing Game
The blog is very useful.
http://www.semeistvoto.com