Using Innostore with Riak
Posted by dizzyd
Innostore is an Erlang application that provides an API for storing and retrieving key/value data using the InnoDB storage system. This storage system is the same one used by MySQL for reliable, transactional data storage. It's a proven, fast system and perfect for use with Riak if you have a large amount of data to store. Let's take a look at how you can use Innostore as a backend for Riak.
(Note: I assume that you have successfully built an instance of Riak for your platform. If you built Riak from source in ~/riak, then set $RIAK to ~/riak/rel/riak.")
We first get started by grabbing a stable release of Innostore. You'll need to download the source for a release from: http://bitbucket.org/basho/innostore/downloads/
Looking in the "Tags & snapshots" section, you should download the source for the highest available RELEASE_* tag. In my case, RELEASE_4 is the most recent release, so I'll grab the bz2 file associated with it:
http://bitbucket.org/basho/innostore/get/RELEASE_4.tar.bz2
Once I have the source code, it's time to unpack it and build:
$ tar -xjf innostore-RELEASE_4.tar.bz2
$ cd innostore
$ make
Depending on the speed of the machine you are building on, this may take a few minutes to complete. At the end, you should see a series of unit tests run, with the output ending:
=======================================================
All 7 tests passed.
100222 7:43:58 InnoDB: Shutdown completed; log sequence number 90283
Cover analysis: /Users/dizzyd/src/public/innostore/.eunit/index.html
Now that we have successfully built innostore, it's time to install it into the Riak distribution:
$ ./rebar install target=$RIAK/lib
If you look in the $RIAK/lib directory now, you should see the innostore-4 directory alongside a bunch of .ez files and other directories which compose the Riak release.
Now, we need to tell Riak to use the innostore driver as a backend. Make sure Riak is not running. Edit $RIAK/etc/app.config, setting the value for "storage_backend" as follows:
{storage_backend, innostore_riak},
In addition, append the configuration for the Innostore application after the SASL section:
{sasl, [ ....
]}, %% < -- make sure you add a comma here!!
{innostore, [
{data_home_dir, "data/innodb"}, %% Where data files go
{log_group_home_dir, "data/innodb"}, %% Where log files go
{buffer_pool_size, 2147483648} %% 2G in-memory buffer in bytes
]}
You may need to adjust the directories for your data_home_dir and log_group_home_dirs to match where you want the inno data and log files to be stored. If possible, make sure that the data and log dirs are on separate disks -- this can yield much better performance.
Once you've completed the changes to $RIAK/etc/app.config, you're ready to start Riak:
$ $RIAK/bin/riak console
As it starts up, you should see messages from Inno that end with something like:
100220 16:36:58 InnoDB: highest supported file format is Barracuda.
100220 16:36:58 Embedded InnoDB 1.0.3.5325 started; log sequence number 45764
That's it! You're ready to start using Riak for storing truly massive amounts of data.
1.44 am
Posted by dizzyd
It's 1.44 am. Woke up feeling weird; then my mind went running, afraid of what it might find.
I was diagnosed with follicular lymphoma three weeks ago now.
I'm blessed in a lot of ways. The cancer is slow moving, non aggressive -- or so it appears at this point. I might not even require treatment in the near future. Even if I do require treatment, survival rates have jumped from 60% to 90% in the past five years -- the treatment for this cancer is progressing quickly. My company, Basho, has been wonderful to me in terms of helping me sort out a variety of insurance issues and arranging access to very good doctors.
All of these things are probably the reason I've not had any trouble sleeping until tonight.
It's still scary though. Cancer -- just the word inspires fear when you first hear it. You are struck, relatively quickly, with the fragility and preciousness of life. You suddenly have a deep desire to grow old. The prospect of death is a powerful incentive to live.
I cried more the first few days and weeks than I ever have in my 32 years. I cried because I was scared. I cried because I was worried about my wife, our 2 year old and the new baby on the way. I cried because it felt unfair, unwarranted! I cried because I realized that there were some areas of my life that I had wasted -- and I wondered if I would have the chance to rectify them.
As I've gotten further into this process, emotions have settled out a bit. I realize now just how good I have it with this cancer. What I'm facing is absolutely nothing compared to other people I know with chronic medical conditions. It's a smudge on the screen; a minor distraction. There might be some tough times ahead, but my overall probability for immediate mortality is relatively stable and low.
That said, I'm determined to make the most of this challenge. If I must go through this valley, I'm going to extract every bit of growth from it that I can. I choose to grow, to push my boundaries in every dimension: physically, spiritually, mentally, emotionally. I choose to spend more time with my family and less time with wandering the mental spaces of coding. I choose to listen more and speak less. I choose to be grateful that all of these realizations have been granted to me at 32 instead of 64.
It's now 2.21 am. I think it was just the Chinese food from dinner that woke me up.