The Future of the Database

New Groups of Users

Within the last few years, relational database management systems
(RDBMS), such as Oracle and Sybase, has seen a whole new class of
users. Up until recently, an RDBMS was primarily a tool for the big
corporations. They’d use it to manage their payroll, for production
management, for financial management, etc. In short, mission-critical
applications where data integrity and security is far more important
than flexibility or cost. Joe Nerd had no interest in using a database
for his own stuff. There’s no way that he could ever create enough
data all by himself, to warrant the huge cost and pain of installing
and maintaining Oracle.

Enter the web, and the picture changes. Now, Joe Nerd wants to
run a web site. Any interesting web site includes a way for
the users of the web site to collaborate. Joe Nerd may own and run it,
but the users of the web site share their comments, talk to each
other, and in general contribute to the contents of the web site. To
do that, you need to store your data somewhere and be able to retrieve
them dynamically, as a browser requests a web page.

Databases are extremely well suited to this type of
application. They take care of all the concurrency issues and
does transaction management. Since only the RDBMS touches the files,
the risk of corrupting the data files are low. And the query optimizer
will make your data retrieval life easy. In short, you can’t really
build an interesting web site without having a database sitting behind
it.

This changes the user base for databases. The corporate MIS department
developer gets company from Joe Nerd in the programmer user space. In
the end user space, the user of the final application, the corporate
staff gets company from Jane websurfer. As happened with the PC and
with the Internet, when a technology is exposed to a new group of
users, radical changes happen. But what changes?

Open Source Databases

The new users will be the type of people disposed for
contributing to an open source project. These people haven’t
really had any use for databases before, but they do now. Since open
source programmers primarily program for their own needs, they’ll
start hacking on relation database management systems. There is
already one RDBMS that’s open source from the ground up, namely <a
href=”http://www.postgresql.org/”>PostgreSQL. And <a
href=”http://www.interbase.com/”>InterBase, previously a
secret-source project, has had its source opened, although not under
GPL. The secret-source
database vendors, such as Oracle and Sybase, will begin to feel the
pressure from open source competition within the next few years.

New Demands

But the new types of users also has different demands than the
old ones did. The applications used in banks don’t change
much over time: Their primary concern is data integrity and security
over flexibility. The programmers working for banks are used to
old-fashioned development tools. The users of banking applications are
used to arcane interaction experiences. And banks are wealthy enough
that they don’t mind having to spend a lot of resources on maintaining
a database server. Needless to say, this is not the case for Joe Nerd
and his web application.

Flexible Data Models

On the web, the name of the game is change. A web application
is constantly evolving in response to user needs. Often
times, you even want the users of your web site to be able to define
and evolve their own data models, a’ la a web-based MS Access
application.

Current RDBMS software like Oracle and Sybase don’t support this
scenario very well. They are designed for static data
models. They’re very narrowminded as to what types of changes
you’re allowed to make. More importantly, changing the data model
isn’t transactional, so you can’t group two or three changes together
and make sure that either all or none of them are executed. This is
very critical, as there may be a programming error between the second
and the third change, so only the two first are executed. These first
changes have to somehow be undone, or we’ll have them lying useless
around forever.

With current RDBMS software, the developer can chose to bite the
bullet, do the data model changes very carefully, and be
prepared to manually untangle the mess that will ineviably arise from
time to time, when the changes break. Or he can chose to re-implement
part of what the RDBMS was supposed to do on top of the RDBMS, which
will be slower, less powerful and more work than having the RDBMS do
its job.

The market will demand flexible data models. The
RDBMS will have to acommodate it by making data definition more
flexible and, most importantly, transactional. We need it, if we are
to deliver high-quality applications to the end users, and it makes a
lot more sense to put it into the database layer, rather than to have
every application invent its own ad-hoc solution on top.

Modern Programming Environment

I’ve worked a lot with Oracle lately, and it’s just soooo
seventies. Oracle’s proprietary PL/SQL language lack modern
language features and doesn’t support compound data types. And
the development environment, i.e. getting your code into the RDBMS and
maintaining it once its in there, is very cumbersome.

Developers coming from a background in hacking Perl, Tcl, Java using
modern development tools will have a hard time getting used to
that. They won’t understand why they can’t write their code in
Perl, Tcl, Java or C, just because it runs inside the database
server. And indeed, with PostgreSQL, you can actually write
your stored procedures in Tcl. And there are ways to get Java to run
inside Oracle, though the process of getting it in there and
maintaining your code in the database is even more cumbersome than for
stored procedures written in PL/SQL.

Even more, with databases being used for the web, the data
we’re storing in the database is more and more strings.
Consequently, the RDBMS will have to get better at managing and
manipulating strings. Currently, Oracle has some strange limitations,
where you have to completely change your syntax if a string might
exceed 4000 characters. And string manipulation in PL/SQL is
horrible. This will have to change.

Supporting Explorative Use

The undo function of most desktop applications is one of the most
important functions—and one of the hardest to implement on top
of an RDBMS. The reason it is so important is that it
encourages exploration by the user. The user can
learn the application by trying, confident that he can undo any
changes that he regrets.

Databases already have a wonderful tool for this in the
transaction: You can make lots of changes to the database,
but until you decide that you’re done and commit the changes, you
still have the option of undoing everything by issuing a rollback
command. But you can only undo what you’ve done so far in this
transaction. The instant you’ve committed, you can’t change your mind
in any other way than by manually and carefully reversing your
actions.

I’d like to see a RDBMS that’ll let me pick an old (but not
too old) transaction and have that undone. This will
make it easier for applications to offer a general undo function to
users, and users will be happier. To implement an undo feature in a
database application today is very hard work.

The tricky thing here is the interaction among multiple
users. What if I post a question on a discussion forum, some
other user answers my question, and then I want to undo? Well,
basically, I can’t. That’s the hard fact of life. If you say something
out loud and someone else acts on it, it’s too late to take it
back. But if noone’s yet acted on what the user just did, we might as
well let her take it back.

I’m not familiar enough with the details of database theory to judge
whether it’s possible or not, but I’d guess that it would be safe to
undo an action if no later transaction has read the data the user just
touched and made other changes after having read it. The
RDBMS knows whether that’s the case or not.

Ease of Maintenance

Joe Nerd doesn’t have spare time or money for installing and
maintaing a database installation. The RDBMS has to take on
more of the job of maintaining itself and running smoothly, so the
developer can focus on adding the features that matters to the users
of his web site. I haven’t tried installing and maintaining too many
different RDBMS’, but I do know that Oracle is quite a beast to deal
with, and that both Interbase and PostgreSQL are much better.

The current trouble falls in two parts: Instaling it and making sure
it keeps running smoothly. Installation should be completely
straight-forward, and have reasonable defaults, so you don’t
even have to change any parameters before installing it. It can surely
be done.

Databases often get slower and slower as they get bigger. The RDBMS
should be able to automatically defragment as needed, grow as
needed, and in general do a reasonable job of running
smoothly without human intervention. I don’t have first-hand
experience with this, but others have told me that Interbase does an
excellent job here—proof that it can be done.

Conclusion

In summary, I believe that the twin effect of new users
starting to use databases for new things will have a profound
effect on the database market. Within the next three years, I expect
to see open source databases have a great impact, and I expect to see
them pioneering a number of innovations in RDBMS software.