Setting up YCSB for low latency data store testing

Overview 
When confronted with a problem, my first instinct is to look around to see where that problem has been handled before. This is because  I believe that code is a liability, and I want to minimize risk by using code that has been vetted, tested, and put into production by others, and only add to it when necessary.

Right now I have several problems around storing and accessing lots of data in real time. I have several diverse use cases that span applications, but the one thing all of these use cases have in common is that there is no need for transactional integrity. There is also a need for scale beyond which a traditional RDBMS can provide.  The first (un) requirement and the second very urgent requirement are pushing me towards (open source) low-latency, NOSQL data stores.

The two kinds of NOSQL data stores I'm looking at are Document Stores and Key-Value stores. Here is a great post discussing the differences between the two.

Some of the questions I need answered to address projects in progress and planned:

1. What Document Stores and Key-Value Stores out there have heavy adoption rates, a corporate sponsor, other community support that indicate good performance and support? 
2. How much 'eventual consistency' can an application live with? If data doesn't need to be transactional, can it really be eventually consistent? 
3. Is there a Document Store that is fast enough to act as a Key-Value store, since it would be easier to manage one piece of software over two. 
4. How do different Key Value stores compare to one another? Anecdotal evidence is one thing, hard data that I can refer to makes me feel much better.
5. What happens when I shut a node down? How hard is it to restore?
6. What are the costs of maintenance of different stores? How hard are they to set up?

I wanted to have question 1and 2 narrow down the range somewhat,  evaluate that range against 3 and 4 to filter out slower candidates, leaving me with a smaller set to run by questions 5 and 6.

In order to answer 3 and 4 above, I need to compare and contrast both Doc and KV stores in an 'apples to apples' way to gauge performance.

I was psyching myself up to write a generic test framework, when someone pointed me to Brian Cooper  and YCSB, the Yahoo Cloud Serving Benchmark.  I had originally dismissed it as being out of date, but a quick perusal of the code on GitHub convinced me that updating it would not be that hard because it cleanly separates specific database calls from core functionality.

YCSB implements different database client abstraction layers, and provides good documentation on how to set them up: https://github.com/brianfrankcooper/YCSB/wiki/getting-started.

Not Quite Ready For Prime Time

Before I could fully use YCSB, I had to fix up a couple of things. There are patches submitted for some of these fixes in the root project, but they hadn't been accepted yet. It made more sense for me to fork a repo and make the changes I needed (and push them if they hadn't already been pushed up to the upstream origin repo): https://github.com/arunxarun/YCSB

Here are some of the fixes I added, they havent been integrated into the master repo yet:

1. The Cassandra7Client needed to be retrofitted to use ByteBuffers instead of byte[]s.
2. The MongoDbClient was throwing a ClassCastException in the insert() method because it was casting a double encoded in a string to an Integer. 
3. The MongoDbClient was not connecting to non localhost MongoDB instances because it wasn't appending the database name to the base database url.
4. There was no truncate functionality. For a Document Store like Mongo, this meant I had to manually truncate the db every time I wanted to reload data. I implemented the truncate method in the DB abstract class (and pushed it to the adaptor classes I used) so that I could do this via YCSB.

I'm going to continue adding functionality -- right now I'm in the middle of adding delete functionality because we want to benchmark that as well --  to my fork and pushing it up if I think it could be useful for other people.

Running A WorkLoad

In YCSB terminology, a workload is a defined set of operations on a database. Workloads are stored as flat files, and executed by specific classes that extend the abstract Workload class. I'm using the CoreWorkload class (the default) for now, and may extend later. CoreWorkload lets me set the proportion of Reads vs Writes vs Updates vs Deletes in a separate property file. There are default core workload files stored in the $YCSB_HOME/workloads directory. They break out like this:

• workloada = 50/50 read/update ratio
• workloadb = 95/5 read/update ratio
• workloadc = 100/0 read/update ratio
• workloadd = 95/5 read/update ratio
• workloade = 95/5 scan/insert ratio
• workloadf = 50/50 read/read-modify-write ratio

Because they are property files, workload files can be copied/tweaked as needed.  If needed, I can also override the CoreWorkLoad class to do something different, but I haven't had to do that yet, even though I've added new functionality.

I followed the section on Running a Workload, below are my notes in addition to those instructions. 

Building YCSB
Pretty self explanatory: there is an ant target for each db client you wish to compile with:

    ant dbcompile-[DB Client Name]

Just make sure your all of the jars your DB Client class needs are in the $YCSB_HOME/db/[Client DB]/lib directory. Note that sometimes, like in the case of Mongo, you may have to find those jars (slf4j, log4j) in other places.

Data Store Setup
There are some a generic setup steps for all Data Stores:

1. Create a [namespace/schema-like-element] called 'userspace'. For example, in Cassandra this would be a keyspace. In Mongo, a database, etc. 
2. Create a [table-like element] in 'userspace', called 'data'. Again, in Cassandra this would be a column family, in Mongo, a collection, in HBase a column family, in MySQL a table. 

DB Specific details are found on the usage page.

Running YCSB

When running YCSB, make sure you specify the jar files used for the DB Client. The first command you will run is the load command: 

java -cp $YCSB_INSTALL/db/[DB Client dir]/lib/*:$YCSB_INSTALL/build/ycsb.jar com.yahoo.ycsb.CommandLine -db [DB Client class] 


Once you are on the commandline make sure you can connect and see the namespace/keyspace you've created. With that sanity check done, it's time to run a workload. In order to do this I also need to load some data. This is done using the command line client from ycsb.jar:

java -cp $YCSB_INSTALL/db/[DB Client dir]/lib/*:$YCSB_INSTALL/build/ycsb.jar   com.yahoo.ycsb.Client -db [DB Client class]  -p [commandline props] -P [property files] -s -load  > out

Some explanation of the available commandline parameters, note that in the above I'm running with one thread, no target ops, and loading the database via the -load parameter.

• -threads n: execute using n threads (default: 1) - can also be specified as the "threadcount" property using -p
• -target n: attempt to do n operations per second (default: unlimited) - can also be specified as the "target" property using -p
• -load: run the loading phase of the workload
• -t: run the transactions phase of the workload (default)
• -db dbname: specify the name of the DB to use (default: com.yahoo.ycsb.BasicDB) - can also be specified as the "db" property using -p
• -P propertyfile: load properties from the given file. Multiple files can be specified, and will be processed in the order specified
• -p name=value: specify a property to be passed to the DB and workloads; multiple properties can be specified, and override any values in the propertyfile
• -s: show status during run (default: no status)
• -l label: use label for status (e.g. to label one experiment out of a whole batch)
• -truncate, my own special addition, to clean out data stores between runs. 

Some notes on the properties files I'm loading: the first one specifies the actual workload configuration. I'm using workloads/workloada, which looks like this:

recordcount=100000
operationcount=100000
workload=com.yahoo.ycsb.workloads.CoreWorkload

readallfields=true

readproportion=0.5
updateproportion=0.5
scanproportion=0
insertproportion=0

requestdistribution=zipfian

When I specify later properties files, they override the values set in the previous ones (commandline props override everything). I take advantage of this by creating other property files that override recordcount, insertioncount, and set db specific properties that are accessed in the DB Client classes.

The output of the run is the average, min, max, 95th and 99th percentile latency for each operation type (read, update, etc.), a count of the return codes for each operation, and a histogram of latencies for each operation.

The histogram shows the number of calls that were returned within the specified number of milliseconds. For example:

0 45553
1 2344
2 399
3 25
4 5
5 0

reads like this:

• 45553 calls returned in 0 ms
• 2344 calls returned in 1ms
• 399 calls returned in 2 ms
• ....

That's All For Now...


YCSB as it is provides a very solid foundation for me to do testing across candidate data stores.  While it is very well documented, the code could use some love. I intend to give it enough love to evaluate the data stores I want to (in my fork), and push that love upstream. In the future I may end up writing a DB Client for some of the commercial stores we need to evaluate, as well as fix things that bug me.