Mnesia Distributed Hello World

Table of Contents

• Introduction
• Hello World
• Problem Solved
• Dynamic Maintenance
  • Schema
  • Other Tables

Introduction

One of the simultaneously coolest and most challenging features of Mnesia is distributed operation.  Before even tackling the complexities of managing failure cases, some developers have trouble starting it up, leading to solicitations for help like the following:

Hi
I've recently started learning erlang/mnesia and need help setting up mnesia
on two nodes. Here's the series of steps (on same host):
$cd ~/test/foo
$erl -sname foo -setcookie chip_cookie

1. mnesia:create_schema([node()]).
2. mnesia:start().
3. rd(person, {name,age}).
4.
mnesia:create_table(person,[{attributes,record_info(fields,person)},{disc_copies,[node()]}]).
5. mnesia:dirty_write(#person{name="george", age=30}).

# in another window, on same host:
$cd ~/test/bar
$erl -sname bar -setcookie chip_cookie
1> mnesia:start().
#back on foo node:
net_adm:ping('foo@*hostname*').
*pong*
mnesia:add_table_copy(schema, 'bar@*hostname*', disc_copies).
*{aborted,{badarg,schema,disc_copies}} <--- WHY DOES THIS FAIL?*
mnesia:add_table_copy(schema, 'bar@*hostname*', ram_copies).
*{atomic,ok}*
mnesia:add_table_copy(person, 'bar@*hostname*', disc_copies).
*{atomic,ok}*
*# back on node bar*
mnesia:info().
*It doesn't show the 'person' table that was copied from foo. Why is that?*
How can I add more nodes to an existing mnesia database which was created
with only one node ie. mnesia:create_schema([node()]).
Thanks
--
Yellowfish Technologies Inc
http://www.yellowfish.biz
praveen.ray@yellowfish.biz

It happens alot in Erlang that the error messages only make sense if you know alot about Erlang already, as is the case above.

Hello World

First, let's do a distributed hello world, and then we'll be in a better position to understand what went wrong for the poster.  You can follow along by typing in these commands into two OS shells, which I identify as shell1 and shell2.

shell1% erl -name one -setcookie yum
shell2% erl -name two -setcookie yum

At this point you have two Erlang shells started.  For this next bit, everything we will type will be inside the Erlang shells, which I identify as one@ted-teds-computer.local and two@ted-teds-computer.local.  In what follows you should substitute ted-teds-computer.local with your hostname as identified by the Erlang shell.

(one@ted-teds-computer.local)1> mnesia:start ().
ok
(two@ted-teds-computer.local)1> mnesia:start ().
ok
(one@ted-teds-computer.local)2> mnesia:change_config (extra_db_nodes, [ 'two@ted-teds-computer.local' ]).
{ok,['two@ted-teds-computer.local']}
(one@ted-teds-computer.local)3> mnesia:system_info (running_db_nodes).
['two@ted-teds-computer.local', 'one@ted-teds-computer.local']

At this point you have a distributed Mnesia system running on the two nodes.  Let's dissect the steps.

1. Mnesia is started on both nodes via mnesia:start/0.
   • Since this is a fresh start of Mnesia on two brand new nodes, each node creates a ram-resident schema.  To make these nodes have persistent databases the schema must be converted to disc-resident.  Once converted to disc-resident, this will be remembered across node restarts.
2. On (either) one of the nodes Mnesia is told there is an additional node, via mnesia:change_config/2.  Behind the scenes, it tries to contact the Mnesia on the other node and merge the schemas.  Assuming the schemas are compatible, a unified schema containing all the nodes is created and installed on all the nodes.
   • Newly created ram-resident schemas are always considered consistent with any other schema, in order to facilitate reintroduction of stateless nodes.  Disc-resident schemas that were part of a distributed schema but were out of contact for a while, e.g. due to node failure, will also be made consistent if possible.  However, two persistent schemas created in isolation cannot be merged in this fashion; it's really intended for building up a distributed schema starting with a single node.

So far so good.  To persist the situation so that it survives restart we need to convert the schemas to disc-resident.

(one@ted-teds-computer.local)4> mnesia:change_table_copy_type(schema, node(), disc_copies).
{atomic,ok}
(two@ted-teds-computer.local)2> mnesia:change_table_copy_type(schema, node(), disc_copies).
{atomic,ok}

At this point (but not before) you should see directories on your disk where you started the Erlang nodes.

shell1% ls -l Mnesia.*
Mnesia.one@ted-teds-computer.local:
total 40
-rw-r--r--   1 pmineiro  admin   170 Jun 26 14:43 DECISION_TAB.LOG
-rw-r--r--   1 pmineiro  admin   195 Jun 26 14:43 LATEST.LOG
-rw-r--r--   1 pmineiro  admin  8824 Jun 26 14:43 schema.DAT

Mnesia.two@ted-teds-computer.local:
total 40
-rw-r--r--   1 pmineiro  admin   170 Jun 26 14:43 DECISION_TAB.LOG
-rw-r--r--   1 pmineiro  admin   195 Jun 26 14:43 LATEST.LOG
-rw-r--r--   1 pmineiro  admin  8824 Jun 26 14:43 schema.DAT

Only at this point, with a persistent on-disc database, can you succesfully create disc-based table copies.

(one@ted-teds-computer.local)5> mnesia:create_table (foo, [ { disc_copies, mnesia:system_info (running_db_nodes) } ]).
{atomic,ok}
(one@ted-teds-computer.local)6> mnesia:dirty_write ({ foo, hello, world }).
ok
(two@ted-teds-computer.local)3> mnesia:dirty_read (foo, hello).
[{foo,hello,world}]

The data was written on the first node and read on the second.  Cool!

Problem Solved

Now you have enough intuition that I can explain what was wrong with the original poster's sequence.

1. He creates a disc-resident schema on the node foo via mnesia:create_schema/1.
2. He starts mnesia on the node foo via mnesia:start/0.  Node foo is now running a disc-resident schema.
3. He creates a disc-resident table called person on node foo via mnesia:create_table/2, and writes a record via mnesia:dirty_write/1.
4. He creates a ram-resident schema on the node bar via mnesia:start/0.
   • So far so good, but now the trouble starts.
5. He tries to add a disc-resident schema on the node bar via mnesia:add_table_copy/3.  This fails because he already has a table copy of schema on the node bar which is ram-resident.  He would need to call mnesia:change_table_copy_type/3 instead to change the schema to disc-resident.
6. He tries to add a disc-resident copy of the person table on node bar via mnesia:add_table_copy/3, which would work, except that adding a disc-resident table copy to a node with a ram-resident schema is not allowed; first the schema must be made disc-resident.

For pedagogical fun, see if you can modify the sequence in the original message to cause the person table to be replicated disc-based on both nodes.

Dynamic Maintenance

Schema

In general if one wants to create or update a distributed Mnesia configuration, the sequence of operations is:

1. mnesia:start/0
   • If this is the first time starting mnesia for this node, a fresh ram-based schema will be created, which can be merged with any existing schema.  If this is not the first time starting mnesia for this node, any previously created disc-based schema will be loaded, otherwise a fresh ram-based schema will be created.
2. mnesia:change_config (extra_db_nodes, NodeList)
   • In theory this can be done at any time in order to add additional nodes to the mnesia distributed schema, assuming the additional nodes have compatible schemas.  In practice, the best way to ensure a compatible schema is to join the distributed configuration as a fresh ram-based schema which is guaranteed to merge with any existing schema.
3. mnesia:change_table_copy_type (schema, node (), disc_copies)
   • This will change the local mnesia schema type from ram-based to disc-based.

Schemafinder is a project on Google Code which automates these steps of maintaining a distributed mnesia configuration, working in conjunction with a nodefinder strategy to discover other Erlang nodes.  Schemafinder tries to avoid extra work by not calling mnesia:change_config/2 if the node list has not changed, and not calling mnesia:change_table_copy_type/3 if the schema is already disc-based.  Although these functions are idempotent, they acquire a schema transaction in order to run which can introduce latency when a cluster is under extreme load.  You can inspect the relevant bit of code for inspiration.

In the event you want to remove a node from the schema, you can use mnesia:del_table_copy/2 on the schema table.

Other Tables

For other tables, tables can be initialized with a node list using the ram_copies, disc_copies, and/or disc_only_copies options to mnesia:create_table/2.  In addition an existing table can have additional copies placed via mnesia:add_table_copy/3, and removed via mnesia:del_table_copy/2.  Note a node can only have one copy of a table, so if you want to change the type of a table at a node use mnesia:change_table_copy_type/3.  Continuing with the above example:

(one@ted-teds-computer.local)7> mnesia:create_table (bar, [ { ram_copies, [ node () ] } ]).
{atomic,ok}
(two@ted-teds-computer.local)4> mnesia:table_info (bar, active_replicas).
['one@ted-teds-computer.local']
(two@ted-teds-computer.local)5> mnesia:add_table_copy (bar, node (), disc_copies).
{atomic,ok}
(two@ted-teds-computer.local)6> mnesia:table_info (bar, active_replicas).
['two@ted-teds-computer.local',
 'one@ted-teds-computer.local']
(two@ted-teds-computer.local)7> mnesia:add_table_copy (bar, node (), ram_copies).
{aborted,{already_exists,bar,'two@ted-teds-computer.local'}}
(two@ted-teds-computer.local)8> mnesia:change_table_copy_type (bar, node (), ram_copies).
{atomic,ok}

The sequence is as follows:

1. Initially table bar is created with a single ram copy on node one.
2. A disc copy is added to node two.
   • Note there is no requirement that the copy type be the same across the nodes.
3. An attempt to add another copy of the table to node two fails.
4. The copy on node two is changed from disc copies to ram copies.