Things that I learned from Michael Stonebraker

by philg

Michael Stonebraker came to talk to our three-day lab course on SQL programming. He’s a great example of why the research university should not be shut down precipitously. Here’s what I learned from him about the current state of database management systems.

For transaction processing (“Small Data”), the standard RDBMS variants work pretty well as long as the database fits into RAM (1 TB is a reasonable size) and as long as you’re not trying to do more than about 1,000 updates per second.

For high-volume transaction processing you need a new architecture where everything is in RAM, which means that transactions happen fast enough that the system doesn’t need to try to be doing 100 of them simultaneously, in various stages of completion. How is the Durability part of the ACID test met if everything is in RAM? Stonebraker points out that you need to have real-time failover anyway so why not let the failover system give you the D in ACID? If that’s not good enough, put an uninterruptible power supply behind both servers. It turns out that customers don’t actually trust this so everyone takes a 5 percent performance hit and logs transaction requests to an SSD. (The log is what the DBMS was told to do, not a data log of blocks in their old and new states.) Stonebraker has drawers full of companies that he has started for every possible database management challenge and his personal solution in this area is VoltDB.

For traditional “business intelligence” or “data warehousing” queries, the column-oriented shared-nothing DBMSes such as Vertica (yet another Stonebraker-founded company, sold to HP) end up being 50X faster than row-oriented DBMSes (e.g., Oracle, MySQL). Why? The database is too big to fit into RAM and you’re usually interested only about 1/50th of the columns in any one query. Thus the system needs to fetch and scan only about 1/50th as much stuff from disk as would a row-oriented DBMS.

What customers want and doesn’t exist right now is a DBMS to handle Big Data and Big Analytics. This will be an “Array DBMS” and it will be good for machine learning, clustering, trend detection, etc. The Array DBMS will be good at handling the common “inner loops” of “Big Analytics” such as matrix multiply, QR decomposition, SVD decomposition, and linear regression. Somehow I have a feeling that this might be Stonebraker’s next company!

8 thoughts on “Things that I learned from Michael Stonebraker”

  1. Doesn’t this cry out for some sort of massively parallel processor architecture? I know that they have had some success in using graphic chips to solve (non-graphical) parallel problems because graphics chips are geared to providing solutions for every pixel on your screen all at once and it’s possible to “fool” the graphics chip into solving problems that don’t involve pixels – to the chip,it’s all data and it doesn’t care if it is a color and luminance value for a pixel that it is processing or a data point in some other array. But I assume that those chips are hardware limited by the size of their assumed arrays which would be equal to the maximum pixel count of the screen that they are supposed to drive but perhaps you can array the graphics chips themselves until you get a parallel space equal in size to your desired matrix. Having everything in memory at once still isn’t fast enough if you have to solve each cell sequentially.

  3. Actually big data analytics are also moving in-memory, e.g. with Spark supplanting Hadoop at many Machine Learning shops. It’s better to get results quickly on a sample of your data that fits in RAM and iterate quickly as you figure out better questions to ask, than waiting laboriously for long batch computation on 100% of your data, just as interactive and personal computing supplanted the batch processing mainframes of yesteryear.

  4. Izzie: Your solution makes a lot of sense, in an alternate universe where CPUs don’t have to wait the equivalent of a thousand instructions to fetch data from memory.

    Graphics processing is pretty unique in that it involves a lot of computation on data sets sized in megabytes, rather than giga or terabytes. Database queries, generally speaking, do not involve a lot of computation.

    But parallelism makes a lot of sense, as long as you parallelize the bottlenecks between CPU and memory, and between memory and disk. This usually involves distributing your data over multiple systems, which is also where you run into tricky problems with the C part of ACID.

    Unless HP’s R&D department pulls a rabit out of its hat, improvements in processor architecture are likely to be incremental and largely meaningless in the next decade.

  5. Parallelism makes a lot of sense for sorting and searching, particularly merge sort. It is easy to optimize queries for parallel processing and it is being done a lot on database application side, why not in DBMS itself? I believe MS SQL Server already supports multiple physical servers for one virtual server, as well as server cloud. Transaction logging makes huge sense for replication among servers dedicated to same data scheme by enabling fast data replication among servers by log shipping. Interesting to see details of new ‘column-oriented’ store and how it differs from in-memory views into large tables and joins. 5% of performance penalty seems small and could be reduced by implementing transaction log as an asynchronous persistent service executing on parallel dedicated server.

  7. A joke for the enterprise DBMS crowd …

    Q: what’s difference between a data architect and a terrorist?
    A: you can negotiate with a terrorist

Comments are closed.