Mnesia Schema Free Operation

Introduction

Most programmers are aware of the distinction between statically typed and dynamically typed programming languages, with respective advantages and disadvantages.  Analogously, databases can be schema driven or schema-free.  All relational database engines that I am familiar with are schema driven, while recent champions of the schema-free approach include CouchDB and SimpleDB.  A schema-free database can simplify issues around the evolution of the database-driven application, in exchange for more work from the application developer (and more opportunities to introduce defects).

Since the Erlang ethos embraces continuous application upgrades and operational simplicity, it is natural to ask whether Mnesia can be used in an essentially schema-free way.  The answer is yes.  Futhermore, relational database engines utilize the schema for validation, consistency, indexing, and query planning, but Mnesia's built-in capabilities in this regard are primitive that the cost of forgoing them is minimal (typically, the application programmer does these things manually).  This, plus the ability to store an arbitrary Erlang term in an Mnesia table, leads to a natural way to use Mnesia in a disciplined but essentially schema-free manner.

The Strategy

An approach I've taken successfully on large projects in the past is as follows:

• Mnesia tables require a schema, but the table is created with a trivial initial specification and never changed over the lifetime of the application.
• Records in the table consist of a key (Mnesia requirement) and an additional record which is coupled with an API for manipulation.
• As the application evolves, the implementation term changes for newly created database records, and the application interprets old implementation terms to the extent possible.

An example is best, so let's do one.  Our database driven application will utilize a table called notes to keep web-sticky notes.

-record (note, { id, impl }).

mnesia:create_table (note, [ { attributes, record_info (fields, note) } ]).

So far so good.  Initially our note will have an id and an owner.

-module (notes).
-export ([ make_note/2,
           get_id/1,
           get_owner/1,
           set_id/2,
           set_owner/2 ]).

-include ("notes.hrl").

-record (notes_impl, { owner }).

make_note (Id, Owner) ->
  #note { id = Id,
          impl = #notes_impl { owner = Owner }
  }.

get_id (#note { id = Id }) ->
  Id.

get_owner (#note { impl = #notes_impl { owner = Owner } }) ->
  Owner.

set_id (Note = #note{}, Id) ->
  Note#note { id = Id }.

set_owner (Note = #note { impl = #notes_impl {} }, Owner) ->
  Note#note { impl = #notes_impl { owner = Owner } }.

The declaration of #notes_impl{} is localized to the notes module, and by convention any access or mutation of a note is done exclusively via the functions in the notes module.

Well our application happily exists for a while persisting notes to the data store.  Finally we get a feature request that our notes contain some content in addition to an owner.  Now we are going to make the change without changing the schema, by changing the notes module.

-module (notes).
-export ([ make_note/3,
           get_id/1,
           get_owner/1,
           get_contents/1,
           set_id/2,  
           set_owner/2,
           set_contents/2 ]).

-include ("notes.hrl").

-record (notes_impl, { owner }). % deprecated
-record (notes_implv2, { owner, contents="" }).

make_note (Id, Owner, Contents) ->
  #note { id = Id,
          impl = #notes_implv2 { owner = Owner, contents = Contents }
  }.
   
get_id (#note { id = Id }) ->
  Id.

get_owner (#note { impl = #notes_implv2 { owner = Owner } }) ->
  Owner;
get_owner (Note) ->
  get_owner (convert_old (Note)).
 
get_contents (#note { impl = #notes_implv2 { contents = Contents } }) ->
  Contents;
get_contents (Note) ->
  get_contents (convert_old (Note)).
 
set_id (Note = #note{}, Id) ->
  Note#note { id = Id }.

set_owner (Note = #note { impl = Impl = #notes_implv2 {} }, Owner) ->
  Note#note { impl = Impl#notes_implv2 { owner = Owner } };
set_owner (Note, Owner) ->
  set_owner (convert_old (Note), Owner).

set_contents (Note = #note { impl = Impl = #notes_implv2 {} }, Contents) ->
  Note#note { impl = Impl#notes_implv2 { contents = Contents } };
set_contents (Note, Contents) ->
  set_contents (convert_old (Note), Contents).
                     
%%% Private          
                     
convert_old (Note = #note { impl = #notes_impl { owner = Owner } }) ->
  Note#note { impl = #notes_implv2 { owner = Owner } }.

The changes can be summarized thusly:

1. A new implementation record (#notes_implv2{}) containing the new field (contents) is introduced.
2. The constructor is modified to return the latest implementation type.
3. The accessors and mutators are modified to pattern match on the latest implementation type; and to otherwise attempt to convert an old implementation to the latest implementation and try again.
   • I like to centralize all the "backwards compatibility" logic into a single function like convert_old/1 for maintainability.

In practice the database will consist of a mixture of the latest and older implementation records, with individual records being migrated when mutated and persisted.  In this case there is a perfectly reasonable interpretation of old records so this is not a problem.  Sometimes this is not possible.  In such cases I would suggest doing an "unreasonable" interpretation of old records, which is designed to operate only transiently, while the database is incrementally transformed to the latest implementation record (e.g., using an expensive one-time operation which joins against or precomputes from another data source).

Pros and Cons

There's no free lunch.  Let's enumerate some of the disadvantages:

• We've exchanged record syntax (e.g., Note#note.contents) for function calls (e.g., notes:get_contents (Note)).  It's (probably irrelevantly) slower and more code maintenance, although fancy use of parse-transforms might be able to mitigate both of these problems.
  • The backwards compatibility mapping is an interesting source of defects and good testing strategies are required to cover all the cases.
• We can't use Mnesia's built-in indexing capabilities on the "hidden columns".  In my experience this is totally moot, since I always end up making my own indices (e.g., functional indices, ordered indices, composite column indices, etc).
• QLC usage is also frustrated.  Again in my experience this is totally moot since I tend to manually query plan my joins in Mnesia.
• Nice-to-haves like mnesia:dirty_update_counter/3 are unavailable.  In my opinion this is a real pain point, but a hybrid approach can admit this.

So why do it?  There is one big advantage:

• The application and the datastore are now loosely coupled, rather than strongly coupled.  Therefore, synchronization requirements between the two are greatly mitigated.

What this means in practice is, you can:

• Launch a software change to an entire cluster a few boxes at a time, without worry.
• Continue to operate your datastore with no delays or performance degradations during launch, and (if necessary) migrate the data asynchronously using a background process (e.g., consuming only spare capacity).

In my experience the advantages outweigh the disadvantages.  YMMV.

Final Note

It is possible to take a hybrid approach, with records that have some columns represented explicitly and part of the schema, and other columns represented implicitly via an implementation member as indicated here.  This can be a way to instrument an application for flexibility to future changes.