Waving Not Drowning

Schema On Read? Not so fast!

2011-11-02T07:05:00.000-07:00

I just got back from HadoopWorld. I have many thoughts on what I saw and heard there, but that is probably a separate post. I've been trying to write something for the last 3 months, and HadoopWorld gave me the clarity I needed to finish it.

There was this phrase I kept hearing in the halls and meeting rooms...."Schema on Read". Schema on Read, in contrast to Schema on Write, gives you the freedom to define your schema after you've stored your data. Schema on Read sounds like Freedom. And who wouldn't like Freedom from the Tyranny of Schemas?

If, by the way, that phrase rings a bell, it may be because of this:

All Downfall Meme kidding aside, Schema on Read sure seems nice. Because there is nothing in any of the current NoSQL storage engines that enforces a schema down to the column level, can we just not worry about schemas? What's the point?

Schemas -- good for the consumer, bad for the producer.

The point is that schemas are a guarantee to a consumer of the data. They guarantee that the data follows a specific format, which makes it easier to consume. That's great for the consumer. They get to write processes that can safely assume that the data has structure.

But...not without a cost, to someone. For the producer of the data, schemas can suck. Several reasons:

Because you never get them right the first time, and you're left doing schema versioning and writing upgrade scripts to compensate for your sins of omission -- basically you get nailed for not being omniscient when you designed the schema.
Because they don't do well with variability. If you have data with a high rate of variability, your schema is guaranteed to change every time you encounter values/types/ranges that you didn't expect.

The various parties pumping freedom from schemas via NoSQL technologies seem to have an additional implicit message -- that even though you don't have to lock down your data to a schema, you still get the benefits of having one -- the data is still usable. Specifically, if you don't define or partially define the data, you can still get value from it. Because you're storing it. Is that true?

Sure it is. Sort of. Take a file in HDFS. If the file isn't formatted in a specific manner, can it still be processed? As long as you can split it, you can process it. Will the processing code need to account for an undefined range of textual boundary conditions? Absolutely. That code will be guaranteed to break arbitrarily because the format of the data is arbitrary.

The same thing can happen with column families. Any code that processes a schema-free column family needs to be prepared to deal with any kind of column that is added into that column family. Again, the processing code needs to be incredibly flexible to deal with potentially unconstrained change. Document stores are another example where even though the data is parse-able, your processing code may need to achieve sentience a la Skynet in order to process it.

So, yes, you can get value from randomly formatted data, if you can write bulletproof, highly adaptable code. That will eventually take over the world and produce cyborgs with Austrian accents that travel back in time.

But what about those of us that process (semi) structured data? Web logs, for example. Or (XML/JSON) data feeds. Things that have some kind of structure, where the meaning and presence -- aka the semantics -- of fields may change but the separators between them don't. Do we really need freedom from the tyranny of something that guarantees structure when we are processing things that have a basic structure?

Yes. Even though format may be well defined, semantics can be quite variable. Fields may be optional, mileage may vary. Putting some kind of schematic constraint on all data invalidates one of the key bonuses of big data platforms -- we wouldn't be able to clean it because we wouldn't be able to load it if we had to adhere to some kind of well defined format. In the big data world, imposing a schema early on would not only suck, it would suck at scale.

However, the moment we have done some kind of data cleansing, and have data that is ready for consumption, it makes sense to bind that data to a schema. Note: by schema I'm talking about a definition of the data that is machine readable. JSON keys and values work quite well, regardless of the actual storage engine.

Because the moment you guarantee your data can adhere to a schema, you liberate your data consumers. You give them...freedom! Freedom from....the tyranny of undefined data!

But wait, that's not all...

What else comes along for free? How about a quality bar by which you can judge all data you ingest? If your data goes through a cleansing process, you could publish how much data didn't make it through the cleansing process. Consumers could opt out of data consumption if too much data was lost.

And when your data changes (because it will) your downstream customers can opt out because none of the data would pass validation. This fast failure mode is much preferred to the one in which you discover that your financial reports were wrong after a month because of an undetected format change. That isn't an urban myth, it actually happened to -- ahem -- a friend of mine, who was left to contemplate the phrase that 'pain is a very effective motivator' while scrambling to fix the issue :)

So what does this all mean?
While the costs of Schema on Write are (a) well known and (b) onerous, Schema on Read is not much better the moment you have to maintain processes that consume the data.

However, by leveraging the flexibility of Hadoop, Cassandra, HBase, Mongo, etc, and loading the data in without a schema, I can then rationalize (clean) the data and apply a schema at that point. This provides freedom to the data producer while they are discovering what the data actually looks like, and freedom to the data consumer because they know what to expect. It also lets me change over time in a controlled manner that my consumers can opt in or out of.

That's not Schema on Read or Schema on Write, it's more like Eventual Schema. And I think it's a rational compromise between the two.

Rolling out splittable lzo on CDH3

2011-04-18T20:42:00.000-07:00

Until splittable lzo, compression options in HDFS were limited. Gzip generated unsplittable output -- great for reducing allocated block usage, terrible for mapreduce efficiency. Bz2 generated splittable output, but took far too long to be effectively used in production.

When we wanted to start incorporating compression into our storage procedures, splittable lzo was the only rational option to ensure parallel processing of compressed files.

We had tried to use bz2 compression on files prior to ingestion, but it took much longer -- approximately 20x as long as gzip compression on the same file.

For a 1GB text file,

gzip -1 took ~ 25 seconds (actually, this is strange. I was expecting gzip to be slightly faster than lzo)
lzo -1 took ~ 9 seconds, indexing took another 4.
bzip2 -1 took ~ 3 minutes.

I set the max speed of each compression routine to provide a relative benchmark: in reality we would be running at a slower speed that increased compression.

Installing The Bits

The java and native source for splittable lzo can be found at https://github.com/kevinweil/hadoop-lzo. If you're using the Cloudera distro, you should use the https://github.com/toddlipcon/hadoop-lzo fork.

The cluster I was installing splittable lzo on was running Centos and walled off from the rest of the world. I found it easiest to generate RPMs on a box with the same architecture, then install those RPMs on all nodes in the cluster. I did this using the https://github.com/toddlipcon/hadoop-lzo-packager code, which takes the native and java components and installs them to the right locations. Note that since I was building on a Centos box, I ran

./run.sh --no-deb

to build RPMs only. There were two rpms, the standard one and the debug-info one. The naming convention appears to be YYYYmmDDHHMMSS.full.version.git_hash_of_hadoop_lzo_project.arch, to allow you to upgrade when either the packaging code or the original hadoop lzo code changes.

The RPMs installed the following java and native bits (note that the packager timestamps the jars):

rpm -ql cloudera-hadoop-lzo-20110414162014.0.4.10.0.g2bd0d5b-1.x86_64

/usr/lib/hadoop-0.20/lib/cloudera-hadoop-lzo-20110414162014.0.4.10.0.g2bd0d5b.jar
/usr/lib/hadoop-0.20/lib/native
/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64
/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/libgplcompression.a
/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/libgplcompression.la
/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/libgplcompression.so
/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/libgplcompression.so.0
/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/libgplcompression.so.0.0.0

rpm -ql cloudera-hadoop-lzo-debuginfo-20110414162014.0.4.10.0.g2bd0d5b-1.x86_64

/usr/lib/debug
/usr/lib/debug/usr
/usr/lib/debug/usr/lib
/usr/lib/debug/usr/lib/hadoop-0.20
/usr/lib/debug/usr/lib/hadoop-0.20/lib
/usr/lib/debug/usr/lib/hadoop-0.20/lib/native
/usr/lib/debug/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64
/usr/lib/debug/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/libgplcompression.so.0.0.0.debug
/usr/lib/debug/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/libgplcompression.so.0.debug
/usr/lib/debug/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/libgplcompression.so.debug

Hadoop Configuration Changes

After installing the bits via RPMs, There were a couple of changes necessary to get Hadoop to recognize the new codec.

In core-site.xml:

<property>
  <name>io.compression.codecs</name>
  <value>
    org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,
    com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec,
    org.apache.hadoop.io.compress.BZip2Codec
  </value>
 </property>
 <property>
   <name>io.compression.codec.lzo.class
   <value>com.hadoop.compression.lzo.LzoCodec</value>
 </property>

registers the codec in the codec factory.
In mapred-site.xml:

<property>
   <name>mapred.compress.map.output</name>
   <value>true</value>
 </property>
 <property>
   <name>mapred.map.output.compression.codec</name>
   <value>com.hadoop.compression.lzo.LzoCodec</value>
 </property>

sets intermediate output to be lzo compressed. After pushing configs out to all nodes in the cluster, I restarted the cluster. The next step was to verify that lzo was installed correctly.

Validation

There were some hiccups I ran into during validation -- all pilot error, but I wanted to put them all in one place for next time. My validation steps looked like this:

(1) create an lzo file that was greater than my block size.
(2) upload and index it.
(3) run a mapreduce using the default IdentityMapper
(4) verify that multiple mappers were run from the one lzo file.
(5) verify that the output was the same size and format as the input.

My first mistake: I lzo compressed a set of files. The splittable lzo code only works with a single file. This took me a while to figure out -- mostly due to tired brain. After I had catted the files together into a single file, then lzo'd that file, I was able to upload it to HDFS and index it:

hadoop jar /usr/lib/hadoop/lib/cloudera-hadoop-lzo-20110414162014.0.4.10.0.g2bd0d5b.jar com.hadoop.compression.lzo.LzoIndexer /tmp/out.lzo

This created an index file. From this great article on the Cloudera site: "Once the index file has been created, any LZO-based input format can split compressed data by first loading the index, and then nudging the default input splits forward to the next block boundaries."

Since I had an uploaded, indexed file at this point, I moved to step 3 and 4. Before I could make the IdentityMapper, I needed to get the LZO bits on my mac so that the IdentityMapper could run.

Detour: Getting the Bits on my Mac

I dev on a Mac, but run the cluster on Centos (I can already feel the wrath of Ted Dziuba coming down from on high). I found the instructions here to be adequate to get the changes I needed to make to the IdentityMapper code to compile.

Back to Validation

I ran an IdentityMapper on the original source (side note: in 0.20, to run IdentityMapper, just don't specify a mapper, the default Mapper class implements pass through mapping). I watched the cluster to make sure that the original file was split out across mappers. It wasnt. I was stumped -- I knew this was something simple, but couldn't see what it was.

After a gentle reminder from Cloudera Support (one of many in the last couple of days, actually:), I set my input format class to LzoTextInputFormat, which -- as the same article above mentions in the next sentence -- "splits compressed data by first loading the index, and then nudges the default input splits forward to the next block boundaries. With these nudged splits, each mapper gets an input split that is aligned to block boundaries, meaning it can more or less just wrap its InputStream in an LzopInputStream and be done." When I had used the default TextInputFormat, the mapreduce was working, but the input was being compressed and not split.

job.setInputFormatClass(LzoTextInputFormat.class);

Once I had observed splitting behavior from my indexed lzo file by confirming multiple map tasks, I made sure that output was recompressed as lzo by setting FileOutputFormat properties:

FileOutputFormat.setCompressOutput(job, true); 
FileOutputFormat.setOutputCompressorClass(job, LzopCodec.class) ;

This is different from instructions in Hadoop: The Definitive Guide, and I found it after some googling around. The instructions in the book -- setting properties in the Configuration objct -- did not work -- most likely because the book was written for an earlier version of Hadoop.

Once I had added those lines to my Tool subclass, I was able to get compressed output that matched my compressed input: the exact result I was looking for when validating using the IdentityMapper.

HDFS file size vs allocation

2011-04-11T22:15:00.000-07:00

Recently, I had to understand HDFS at a deeper level that had nothing to do with running mapreduce jobs or writing to the FileSystem API. Specifically, I had to understand the way that HDFS interacts with the underlying filesystem, and the difference between actual HDFS file size and the way HDFS calculates available storage when using quotas.

We recently discovered a bunch of files that were much smaller than our allocated block size -- on average they took up roughly 1/10th of an allocated block.

This was not the standard small file problem, where the namenode requires too much memory to track metadata for large (10s of millions) numbers of files at 150 bytes of metadata per file.

My immediate conclusion was that these small files were effectively taking up a block at a time, and that we were running out of space -- fast! -- because that was the behavior I thought I was seeing at the HDFS level -- I thought that storage was allocated a block at a time, and quotas were determined based on available blocks.

That last statement is partially correct. Storage is allocated a block -- actually a block * replication factor -- at a time. However quotas are determined based on available bytes. A space quota, according to the docs is "a hard limit on the number of bytes used by files in the tree rooted at that directory. Block allocations fail if the quota would not allow a full block to be written."

This is what that means: the only time files are measured in blocks is at block allocation time. The rest of the time, files are measured in bytes. The space quota is calculated against the number of bytes, not blocks, left in the cluster. That number of bytes is converted to the number of blocks (not bytes) that would be required to store a file when a user tries to upload a file. The key here is that space is calculated in blocks at allocation time, so no matter how small a file is, you will always need 1 block * replication factor available to put it in the cluster.

HDFS Operational Details

I spent some time asking, researching, and re-reading the book, and found that making analogies from a standard filesystem to understand HDFS helped me immensely -- to a point (more on that later).

In a standard filesystem, an inode contains file metadata, like permissions, ownership, last time changed, etc, in addition to a set of pointers that point to all blocks that comprise the file. Inodes are kept in a specific location in the filesystem are used to access files.

The inode and block equivalents in HDFS are distributed across the namenode and the datanode.

The namenode maintains file system metadata, which is analogous to the inodes in a standard FS. This metadata is stored in {dfs.name.dir}/current. Datanodes contain blocks of data, stored as block files in the underlying filesystem.

On the datanode, HDFS stores block data in files in the directory specified by dfs.data.dir, which defaults to {hadoop.tmp.dir}/dfs/data/current. HDFS may create subdirectories underneath that dir to balance out files across directories (many filesystems have a file-per-directory limit). The raw data per block is kept in two files, a blk_NNNN file, and a corresponding blk_NNNN_XXXX.meta file, which contains the block checksum, used in block integrity checks.

The block file and checksum file information is periodically sent to the namenode as a blockreport -- i.e. at HDFS startup (HDFS enters safemode while the namenode processes block reports from it's consituent datanodes). Note that each datanode has no idea which block files map to which actual files. It just tracks the blocks. This makes the namenode very critical to HDFS functionality.

To summarize: the metadata that inodes maintain in a standard FS is maintained in the HDFS namenode, and actual file data that is maintained in filesystem blocks in a standard FS is maintained in HDFS blocks on datanodes, which store that block data in block files, maintain checksums of the block data for integrity checking, and update the namenode with information about the blocks they manage.

FileSystem Analogies That Do and Don't Work

In a standard filesystem, disks have a minimum amount of data that they can read or write to, this is called a disk block. Unix disk blocks are 512 bytes. FileSystems also have minimum read/write filesystem blocks that are typically 1-2kb.

Files on a standard filesystem are typically much larger than a block in size. Since most files are not exactly X blocks in size, the 'remainder' of the file that does not fill up a block still takes up that much space on the system. In general (ReiserFS being one exception) the difference between the files real size and it's block size -- the slack -- cannot actually be used for any other file.

In HDFS, if a file is smaller than a block in size, it does not take up an entire HDFS block on disk. There is no concept of HDFS block 'slack space'. A small file takes up as many bytes as it would in a normal filesystem because it is stored as a block file in the normal filesystem. This is where the definition of HDFS block differs from a traditional file block, and this is where my mental model of HDFS as a filesystem failed me :)

While the file and block analogy is valid in HDFS, the size of the blocks makes the difference between file allocated size (always represented in blocks) and file actual size (always in bytes) much larger than it would be on a traditional file system. So you can't treat allocated vs actual size as equivalents, like you effectively can on a traditional filesystem where the block size to file size ratio is relatively tiny.

Small Files on the Datanodes

At allocation time, a small file will require a single block file per datanode. Note that the actual number of blocks required to store that file on the cluster depends on HDFS replication policy, which defaults to 3. So factoring in replication, a small (less than 1 block) file is replicated at three identical block files on separate nodes.

That block file is the same size as the small file -- large files would span several blocks and be split into block size files -- a large file that was 350MB big on a system with 128MB block size would be split into 3 blocks, the first two of 128MB, the last one of 94MB. Each of those would be replicated according to the replication policy of the cluster. The only files that don't take up space on the datanodes are zero byte files, which still take up space on the namenode.

Regardless of actual size, at allocation time, HDFS treats a small file as having a minimum size of one HDFS block per datanode when it is calculating available disk space. So, even if a file is really small, if there is less than a three blocks available on the cluster, the file cannot be stored on the system.

Space Quotas
HDFS only has less than the number of replicated blocks left than it needs to store a file when it is either running out of space, or, more commonly, if there is a space quota on the directory that the file is being copied to. Calculating storage cost in blocks allows HDFS safely store data to a known maximum size, no matter what the actual size of the file is. HDFS will only permit new block creation if there is enough disk space to create a block on N datanodes, where N is the replication factor.

This article shows how HDFS block size, combined with the replication factor, not file actual size, determines available space.

Conclusion

Is this really a problem? Sort of...it's a matter of efficiency. Space quotas are checked by the amount of remaining space on a datanode disk. If a block file takes up 12MB on a system that has 128MB block, there are effectively 114MB available to be added into the available bytes for the space quota -- for a replication factor of 3, that would be 342MB available, or 2.67 blocks. While you could argue that effectively .67 blocks of that space is wasted, 2 blocks of that space is still available for quota calculations. While 2.67 blocks is less than the minimum amount of space required to store a file of _any_ size in an HDFS with a replication factor of 3, if you were to have 2 small files of 12MB, you have 5.34 blocks available across the system -- effectively if you always mod 'leftover space' by replication factor, at most you are wasting replication factor # of blocks.

Granted that's not the most efficient use of disk, but it's not as if a small file takes up a 'virtual' block that gets factored in the next time a file is copied into the cluster.

The bigger problem with small files is the lack of efficiency that is encountered in mapreduce operations. Reducing the number of mappers being used and traversing blocks of data at a time is not possible with small files -- one mapper is spun up per file, and the overhead involved in copying the jar file to the task tracker node, starting up the JVM, etc, only makes sense if there is a substantial amount of data to process. You can't go wrong with large files -- they will split across blocks, which are processed more efficiently.

Setting up YCSB for low latency data store testing

2011-03-14T21:41:00.000-07:00

Overview
When confronted with a problem, my first instinct is to look around to see where that problem has been handled before. This is because I believe that code is a liability, and I want to minimize risk by using code that has been vetted, tested, and put into production by others, and only add to it when necessary.

Right now I have several problems around storing and accessing lots of data in real time. I have several diverse use cases that span applications, but the one thing all of these use cases have in common is that there is no need for transactional integrity. There is also a need for scale beyond which a traditional RDBMS can provide. The first (un) requirement and the second very urgent requirement are pushing me towards (open source) low-latency, NOSQL data stores.

The two kinds of NOSQL data stores I'm looking at are Document Stores and Key-Value stores. Here is a great post discussing the differences between the two.

Some of the questions I need answered to address projects in progress and planned:

What Document Stores and Key-Value Stores out there have heavy adoption rates, a corporate sponsor, other community support that indicate good performance and support?
How much 'eventual consistency' can an application live with? If data doesn't need to be transactional, can it really be eventually consistent?
Is there a Document Store that is fast enough to act as a Key-Value store, since it would be easier to manage one piece of software over two.
How do different Key Value stores compare to one another? Anecdotal evidence is one thing, hard data that I can refer to makes me feel much better.
What happens when I shut a node down? How hard is it to restore?
What are the costs of maintenance of different stores? How hard are they to set up?

I wanted to have question 1and 2 narrow down the range somewhat, evaluate that range against 3 and 4 to filter out slower candidates, leaving me with a smaller set to run by questions 5 and 6.

In order to answer 3 and 4 above, I need to compare and contrast both Doc and KV stores in an 'apples to apples' way to gauge performance.

I was psyching myself up to write a generic test framework, when someone pointed me to Brian Cooper and YCSB, the Yahoo Cloud Serving Benchmark. I had originally dismissed it as being out of date, but a quick perusal of the code on GitHub convinced me that updating it would not be that hard because it cleanly separates specific database calls from core functionality.

YCSB implements different database client abstraction layers, and provides good documentation on how to set them up: https://github.com/brianfrankcooper/YCSB/wiki/getting-started.

Not Quite Ready For Prime Time

Before I could fully use YCSB, I had to fix up a couple of things. There are patches submitted for some of these fixes in the root project, but they hadn't been accepted yet. It made more sense for me to fork a repo and make the changes I needed (and push them if they hadn't already been pushed up to the upstream origin repo): https://github.com/arunxarun/YCSB

Here are some of the fixes I added, they havent been integrated into the master repo yet:

The Cassandra7Client needed to be retrofitted to use ByteBuffers instead of byte[]s.
The MongoDbClient was throwing a ClassCastException in the insert() method because it was casting a double encoded in a string to an Integer.
The MongoDbClient was not connecting to non localhost MongoDB instances because it wasn't appending the database name to the base database url.
There was no truncate functionality. For a Document Store like Mongo, this meant I had to manually truncate the db every time I wanted to reload data. I implemented the truncate method in the DB abstract class (and pushed it to the adaptor classes I used) so that I could do this via YCSB.

I'm going to continue adding functionality -- right now I'm in the middle of adding delete functionality because we want to benchmark that as well -- to my fork and pushing it up if I think it could be useful for other people.

Running A WorkLoad

In YCSB terminology, a workload is a defined set of operations on a database. Workloads are stored as flat files, and executed by specific classes that extend the abstract Workload class. I'm using the CoreWorkload class (the default) for now, and may extend later. CoreWorkload lets me set the proportion of Reads vs Writes vs Updates vs Deletes in a separate property file. There are default core workload files stored in the $YCSB_HOME/workloads directory. They break out like this:

workloada = 50/50 read/update ratio
workloadb = 95/5 read/update ratio
workloadc = 100/0 read/update ratio
workloadd = 95/5 read/update ratio
workloade = 95/5 scan/insert ratio
workloadf = 50/50 read/read-modify-write ratio

Because they are property files, workload files can be copied/tweaked as needed. If needed, I can also override the CoreWorkLoad class to do something different, but I haven't had to do that yet, even though I've added new functionality.

I followed the section on Running a Workload, below are my notes in addition to those instructions.

Building YCSB
Pretty self explanatory: there is an ant target for each db client you wish to compile with:

ant dbcompile-[DB Client Name]

Just make sure your all of the jars your DB Client class needs are in the $YCSB_HOME/db/[Client DB]/lib directory. Note that sometimes, like in the case of Mongo, you may have to find those jars (slf4j, log4j) in other places.

Data Store Setup
There are some a generic setup steps for all Data Stores:

Create a [namespace/schema-like-element] called 'userspace'. For example, in Cassandra this would be a keyspace. In Mongo, a database, etc.
Create a [table-like element] in 'userspace', called 'data'. Again, in Cassandra this would be a column family, in Mongo, a collection, in HBase a column family, in MySQL a table.

DB Specific details are found on the usage page.

Running YCSB

When running YCSB, make sure you specify the jar files used for the DB Client. The first command you will run is the load command:

java -cp $YCSB_INSTALL/db/[DB Client dir]/lib/*:$YCSB_INSTALL/build/ycsb.jar com.yahoo.ycsb.CommandLine -db [DB Client class]

Once you are on the commandline make sure you can connect and see the namespace/keyspace you've created. With that sanity check done, it's time to run a workload. In order to do this I also need to load some data. This is done using the command line client from ycsb.jar:

java -cp $YCSB_INSTALL/db/[DB Client dir]/lib/*:$YCSB_INSTALL/build/ycsb.jar com.yahoo.ycsb.Client -db [DB Client class] -p [commandline props] -P [property files] -s -load > out

Some explanation of the available commandline parameters, note that in the above I'm running with one thread, no target ops, and loading the database via the -load parameter.

-threads n: execute using n threads (default: 1) - can also be specified as the "threadcount" property using -p
-target n: attempt to do n operations per second (default: unlimited) - can also be specified as the "target" property using -p
-load: run the loading phase of the workload
-t: run the transactions phase of the workload (default)
-db dbname: specify the name of the DB to use (default: com.yahoo.ycsb.BasicDB) - can also be specified as the "db" property using -p
-P propertyfile: load properties from the given file. Multiple files can be specified, and will be processed in the order specified
-p name=value: specify a property to be passed to the DB and workloads; multiple properties can be specified, and override any values in the propertyfile
-s: show status during run (default: no status)
-l label: use label for status (e.g. to label one experiment out of a whole batch)
-truncate, my own special addition, to clean out data stores between runs.

Some notes on the properties files I'm loading: the first one specifies the actual workload configuration. I'm using workloads/workloada, which looks like this:

recordcount=100000
operationcount=100000
workload=com.yahoo.ycsb.workloads.CoreWorkload

readallfields=true

readproportion=0.5
updateproportion=0.5
scanproportion=0
insertproportion=0

requestdistribution=zipfian

When I specify later properties files, they override the values set in the previous ones (commandline props override everything). I take advantage of this by creating other property files that override recordcount, insertioncount, and set db specific properties that are accessed in the DB Client classes.

The output of the run is the average, min, max, 95th and 99th percentile latency for each operation type (read, update, etc.), a count of the return codes for each operation, and a histogram of latencies for each operation.

The histogram shows the number of calls that were returned within the specified number of milliseconds. For example:

reads like this:

45553 calls returned in 0 ms
2344 calls returned in 1ms
399 calls returned in 2 ms
....

That's All For Now...

YCSB as it is provides a very solid foundation for me to do testing across candidate data stores. While it is very well documented, the code could use some love. I intend to give it enough love to evaluate the data stores I want to (in my fork), and push that love upstream. In the future I may end up writing a DB Client for some of the commercial stores we need to evaluate, as well as fix things that bug me.

Setting up CDH3 Hadoop on my new Macbook Pro

2011-01-05T21:46:00.000-08:00

A New Machine

I'm fortunate enough to have recently received a Macbook Pro, 2.8 GHz Intel dual core, with 8GB RAM. This is the third time I've turned a vanilla mac into a ninja coding machine, and following my design principle of "first time = coincidence, second time = annoying, third time = pattern", I've decided to write down the details for the next time.

Baseline
This section details the pre-hadoop installs I did.

Java
Previously I was running on Leopard, i.e. 10.4, and had to install soylatte to get the latest version of Java. In Snow Leopard, java jdk 1.6.0_22 is installed by default. That's good enough for me, for now.

Gcc, etc.
In order to get these on the box, I had to install XCode, making sure to check the 'linux dev tools' option.

MacPorts
I installed MacPorts in case I needed to upgrade any native libs or tools.

Eclipse
I downloaded the 64 bit Java EE version of Helios.

Tomcat
Tomcat is part of my daily fun, and these instructions to install tomcat6 where helpful. One thing to note is that in order to access the tomcat manager panel, you also need to specify

<role rolename="manager"/>

prior to defining

<user username="admin" password="password" roles="standard,manager,admin"/>

Also, I run tomcat standalone (no httpd), so the mod_jk install part didnt apply. Finally, I chose not to daemonize tomcat because this is a dev box, not a server, and the instructions for compiling and using jsvc for 64 bit sounded iffy at best.

Hadoop
I use the CDH distro. The install was amazingly easy, and their support rocks. Unfortunately, they don't have a dmg that drops Hadoop on the box configured and ready to run, so I need to build up my own psuedo mac node. This is what I want my mac to have (for starters):

distinct processes for namenode, job tracker node, and datanode/task tracker nodes.
formatted HDFS
Pig 0.8.0

I'm not going to try to auto start hadoop because (again) this is a dev box, and start-all.sh should handle bringing up the JVMs around namenode, job tracker, datanode/tasktracker.

I am installing CDH3, because I've been running it in psuedo-mode on my Ubuntu dev box for the last month and have had no issues with it. Also, I want to run Pig 0.8.0, and that version may have some assumptions about the version of Hadoop that it needs.

All of the CDH3 Tarballs can be found at http://archive.cloudera.com/cdh/3/, and damn, that's a lot of tarballs.

I downloaded hadoop 0.20.2+737, it's (currently) the latest version out there. Because this is my new dev box, I decided to forego the usual security motivated setup of the hadoop user. When this decision comes back to bite me, I'll be sure to update this post. In fact, for ease of permissions/etc, I decided to install under my home dir, under a CDH3 dir, so I could group all CDH3 related installs together. I symlinked the hadoop-0.20+737 dir to hadoop, and I'll update it if CDH3 updates their version of hadoop.

After untarring to the directory, all that was left was to make sure the ~/CDH3/hadoop/bin directory was in my .profile PATH settings.

Psuedo Mode Config
I'm going to set up Hadoop in psuedo distributed mode, just like I have on my Ubuntu box. Unlike Debian/Red Hat CDH distros, where this is an apt-get or yum command, I need to set up conf files on my own.

Fortunately the example-confs subdir of the Hadoop install has a conf.psuedo subdir. I needed to modify the following in core-site.xml:

<property>
     <name>hadoop.tmp.dir</name>
     <value>changed_to_a_valid_dir_I_own</value>
</property>

and the following in hdfs-site.xml:

<property>
     
     <name>dfs.name.dir</name>
     <value>changed_to_a_different_dir_I_own</value>
</property>

I also had to create masters and slaves files in the example-confs/conf.pseudo directory:

echo localhost > master
echo localhost > slave

finally, I symlinked the conf dir at the top level of the Hadoop install to example-configs/conf.pseudo after saving off the original conf:

mv ./conf install-conf
ln -sf ./example-confs/conf.pseudo conf

Pig
Installing Pig is as simple as downloading the tar, setting the path up, and going, sort of. The first time I ran pig, it tried to connect to the default install location of hadoop, /usr/lib/hadoop-0.20/. I made sure to set HADOOP_HOME to point to my install, and verified that the grunt shell connected to my configured HDFS (on port 8020).

More To Come
This psuedo node install was relatively painless. I'm going to continue to install Hadoop/HDFS based tools that may need more (HBase) or less (Hive) configuration, and update in successive posts.

Pig SPLITs, JOINs, and COGROUPs to manipulate multiple relations

2010-12-20T22:43:00.000-08:00

I've been playing around with Pig and UDFs for the last couple of weeks as we try to convert an application from using SQL to do ETL to using Pig for the same transforms.

In this particular application, we need to 'thread' logged messages together by fields that they can be joined on. Different messages represent different state around a single meta-state, kind of like a session, that unifies the different mesages. Messages can have a specific type, lets call those A,B,C,and D. The joining rules are:

A joins B on field y
B joins C on field y
D joins A on field x,y,z

Split

The first step prior to joining messages is to separate them into relations that only contain A,B,C, or D messages using the Pig SPLIT statement. SPLIT works like this:

SPLIT tuple INTO something IF condition, something else IF other condition.....);

basically SPLIT is a case statement, and I needed to write UDFs to implement the condition tests by comparing the input GMT against the specified day.

Writing UDFs for the SPLIT

In previous posts I've written eval UDFs. Those take input and transform it to something else. In this case I needed to implement filter UDFs. Filter UDFs return a boolean value based on their input.

I've found that the 'top down' approach works well when designing UDFs. By that I mean write the UDFs as they would be used in script:

SPLIT RAW_DATA INTO A IF isA(), B IF isB(), C IF isC(), D IF isD();

and then implement them. Because of the boolean nature of the UDFs I need to implement four different methods because I need to perform four tests in the SPLIT statement above. I'm basically going to implement the pattern:

public class IsA extends FilterFunc {

 @Override
 public Boolean exec(Tuple someTuple) throws IOException {
           return testForA(someTuple); 

 }
        
        protected Boolean testForA(Tuple someTuple) {
              ..... // determine if this is a type A, or not.
        }


}

So the SPLIT statement above works as advertised, partitioning the original raw data out by message type.

JOINing Relations

The next part of threading the messages together is to JOIN them along common fields. The JOIN statement groups relations by a single field:

JOINED_AB = JOIN A BY y, B BY y;

NOTE that this JOIN is an inner join, outer joins are a whole other beast. It simply aggregates all fields of B and C together.So the JOINED_AB relation looks like:

a::x,a::y,a::p,b::q,b::y,b::z

If you want to have an authoritative value of y for each tuple of JOINED_AB, you would need to explicitly generate it:

JOINED_AB = FOREACH JOINED_AB GENERATE a::y as y, .....;

In the case above, recall that

A joins B on field y
B joins C on field y
D joins A on field x,y,z

to knit these fields together, you would

JOINED_AB = JOIN A ON y, B on y;

JOINED_AB = FOREACH JOINED_AB GENERATE B::y as y,*;

JOINED_AB_C = JOIN JOINED_AB ON y, C on y;

At this point we want to join D to JOINED, but that needs to be done along a multiple column match. JOIN only handles single column matches. It's time to use COGROUP.

COGROUPing Relations

The first thing we need to do (for clarity) is to regenerate some of the fields in the JOINED relation:

JOINED = FOREACH JOINED generate A::x as x, A::y as y, A::z as z;

This allows us to COGROUP without having to dereference by sub-tuple:

ALL_DATA = COGROUP JOINED ON (x,y,z) D on (x,y,z);

This relation is actually comprised of all fields of A,B,C,and D, but because we joined A,B,and C into JOINED before joining it to D, the tuple structure looks like this:

ALL_DATA: (x,y.z), {JOINED_AB_C: { JOINED_AB::x,JOINED_AB::y,JOINED_AB::z,JOINED_AB::A::field1,

JOINED_AB::B::field2}, D: {x,y,z,..}}

In other words like a GROUP, that takes members of the same relation and binds tuples by similar fields ,creating a group and a bag that holds a list of matching tuples, COGROUP takes members of different relations, binds them by similar fields, and creates a bag that contains a single instance of both relations where those relations have common fields. In fact the COGROUP and GROUP operations are the same, it's just common practice to use COGROUP when grouping multiple relations, GROUP when grouping the same relation.

Writing a custom PIG Loader

2010-12-01T18:17:00.000-08:00

Foreward

I've been pretty happy using the default pig loader, which takes as input the delimiter of a CSV, and loads tuples into memory as specified:

A = LOAD '/csv/input/inputs*' USING PigStorage() AS (field1,field2,..fieldN)

However I'm in the middle of doing some transforms on a csv with ~ 226 fields. Yikes. For most of these transforms, we don't need all 226 fields, in fact we probably only need a reasonable subset, but which reasonable subset depends on what we are trying to do. Ideally I'd like to be able to extract the values I want into a tuple like this:

A = LOAD 'somefile' using CustomLoader(1,4,45,100...) as (timestamp:long, id:long, url:chararray...);

So, it's time to write a custom loader.

Setup

Hadoop
I first installed the CDH3 distro of hadoop -- specifically the psuedo mode configuration, which runs all hadoop core services, i.e. namenode, datanode, jobtracker and tasktracker on a single box. I then installed hadoop-pig. Cloudera makes this easy by leveraging apt-get and installing to the standard *nix hierarchical locations. The CDH3 version of Hadoop uses /etc/alternatives to allow for easy version switching, and logs reside in the usual /var/logs location.

sudo apt-get install hadoop-0.20-conf-pseudo

after starting core services as described in the link, I then installed CDH3 pig (version 0.7.0) via apt-get:

sudo apt-get install hadoop-pig

CDH3 Pig installs the pig shell script in /usr/bin, and provides libs in /usr/lib/pig. In order to run the pig shell, you need to set JAVA_HOME to /usr/lib/jvm/java-6-sun.

Mavenization

With the necessary services installed, I set up a maven project, mainly for brainless dependency management:

mvn archetype:create  -DarchetypeGroupId=org.apache.maven.archetypes

-DgroupId=org.arunxarun.data.prototypes  -DartifactId=pigloader

I then wanted to bring in the pig jars as dependencies. I found hadoop-0.20-core in the mvnrepository, but could not find pig.jar or pig-core.jar in any maven repository. So I installed the pig and pig-core jars to my local repository from the /usr/lib/pig directory where they had been put by the apt-get install. I did that after creating versionless symlinks to the real jars whose names contained version information:

mvn install:install-file -Dfile=/usr/lib/pig/pig-core.jar -DgroupId=org.apache.hadoop -DartifactId=hadoop-pig-core -Dversion=0.7.0 -Dpackaging=jar

mvn install:install-file -Dfile=/usr/lib/pig/pig.jar -DgroupId=org.apache.hadoop -DartifactId=hadoop-pig -Dversion=0.7.0 -Dpackaging=jar

Finally, I made sure that the dependencies were referenced in my pom file:

<dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-pig-core</artifactId>
      <version>0.7.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-pig</artifactId>
      <version>0.7.0</version>
    </dependency>

Implementation

As of 0.7.0, Pig loaders extend the LoadFunc abstract class.This means they need to override 4 methods:

getInputFormat() this method returns to the caller an instance of the InputFormat that the loader supports. The actual load process needs an instance to use at load time, and doesn't want to place any constraints on how that instance is created.
prepareToRead() is called prior to reading a split. It passes in the reader used during the reads of the split, as well as the actual split. The implementation of the loader usually keeps the reader, and may want to access the actual split if needed.
setLocation() Pig calls this to communicate the load location to the loader, which is responsible for passing that information to the underlying InputFormat object. This method can be called multiple times, so there should be no state associated with the method (unless that state gets reset when the method is called).
getNext() Pig calls this to get the next tuple from the loader once all setup has been done. If this method returns a NULL, Pig assumes that all information in the split passed via the prepareToRead() method has been processed.

Here is the current implementation: note that the constructor takes a var arg set of Strings, which is the only kind of argument that can be used with a Pig Loader. Also note from above that RecordReader is set in prepareToRead, but actually used in getNext().

public class CustomLoader extends LoadFunc {

 private static final String DELIM = "\t";
 private static final int DEFAULT_LIMIT = 226;
 private int limit = DEFAULT_LIMIT;
 private RecordReader reader;
 private List indexes;
 private TupleFactory tupleFactory;

 /**
  * Pig Loaders only take string parameters. The CTOR is really the only interaction
  * the user has with the Loader from the script.  
  * @param indexesAsStrings
  */
 public CustomLoader(String...indexesAsStrings) {
  this.indexes = new ArrayList();
  for(String indexAsString : indexesAsStrings) {
   indexes.add(new Integer(indexAsString));
  }
  
  tupleFactory = TupleFactory.getInstance();
 }
 
 
 @Override
 public InputFormat getInputFormat() throws IOException {
   return new TextInputFormat();

 }

 /**
  * the input in this case is a TSV, so split it, make sure that the requested indexes are valid, 
  */
 @Override
 public Tuple getNext() throws IOException {
  Tuple tuple = null;
  List values = new ArrayList();
  
  try {
   boolean notDone = reader.nextKeyValue();
   if (!notDone) {
       return null;
   }
   Text value = (Text) reader.getCurrentValue();
   
   if(value != null) {
    String parts[] = value.toString().split(DELIM);
    
    for(Integer index : indexes) {
     
     if(index > limit) {
      throw new IOException("index "+index+ "is out of bounds: max index = "+limit);
     } else {
      values.add(parts[index]);
     }
    }
    
    tuple = tupleFactory.newTuple(values);
   }
   
  } catch (InterruptedException e) {
   // add more information to the runtime exception condition. 
   int errCode = 6018;
            String errMsg = "Error while reading input";
            throw new ExecException(errMsg, errCode,
                    PigException.REMOTE_ENVIRONMENT, e);
  }

  return tuple;

 }

 @Override
 public void prepareToRead(RecordReader reader, PigSplit pigSplit)
   throws IOException {
  this.reader = reader; // note that for this Loader, we don't care about the PigSplit.
 }

 @Override
 public void setLocation(String location, Job job) throws IOException {
  FileInputFormat.setInputPaths(job, location); // the location is assumed to be comma separated paths. 

 }

}

Testing
Testing a Pig UDF requires two steps: basic unit testing and integration testing via a script. I'm including this section because it also shows how the loader is accessed via Pig Latin.

Unit Testing: Mocking the Reader
I've implemented a MockRecordReader that I can pass into my CustomLoader via prepareToRead(). The MockRecordReader will be accessed when getNext() is called. Note that I've only implemented the methods I need. This is by no means a fully functional RecordReader:

public class MockRecordReader extends RecordReader {

 private BufferedReader reader;
 private long key;
 private boolean linesLeft;

 
 /**
  * call this to load the file
  * @param fileLocation
  * @throws FileNotFoundException 
  */
 
 
 public MockRecordReader(String fileLocation) throws FileNotFoundException {
  reader  = new BufferedReader(new FileReader(fileLocation));
  key = 0;
  linesLeft = true;
  }

 @Override
  public void close() throws IOException {
   // TODO Auto-generated method stub
   
  }

 @Override
  public Long getCurrentKey() throws IOException, InterruptedException {
   return key;
  }

 @Override
  public Text getCurrentValue() throws IOException, InterruptedException {
  String line = reader.readLine();
  
  if(line != null) {
   key++;
  } else {
   linesLeft = false;
  }
   return new Text(line);
  }

 @Override
  public float getProgress() throws IOException, InterruptedException {
   // dont need this for unit testing
   return 0;
  }

 @Override
  public void initialize(InputSplit arg0, TaskAttemptContext arg1)
      throws IOException, InterruptedException {
   // not initializing anything during unit testing.
   
  }

 @Override
  public boolean nextKeyValue() throws IOException, InterruptedException {
   return linesLeft;
  }

}

Implementing Units using the MockRecordReader is super easy. Note that I load the MockRecordReader up with some fake data for testing purposes.

public class CustomLoaderTest {

 @Test
 public void testValidInput() throws Exception{
  MockRecordReader reader = new MockRecordReader("src/test/resources/valid1line_hit_data.tsv");
  
  CustomLoader custLoader = new CustomLoader("0","2","4");
  
  custLoader.prepareToRead(reader, null);
  
  Tuple tuple = custLoader.getNext();
  
  assertNotNull(tuple);
  
  String ts = (String)tuple.get(0);
  assertNotNull(ts);
  assertEquals(ts,"1130770920");
  
  String language = (String)tuple.get(1);
  assertEquals(language,"en-ca");
  String someCt = (String)tuple.get(2);
  assertEquals(someCt,"675");
  
  
 }
 
 @Test(expected=IOException.class)
 public void testInvalidInput() throws Exception {
  MockRecordReader reader = new MockRecordReader("src/test/resources/valid1line_hit_data.tsv");
  
  CustomLoader custLoader = new CustomLoader("300");
  
  custLoader.prepareToRead(reader, null);
  
  Tuple tuple = custLoader.getNext();
 }
  
}

Integration Testing
Now that I know I haven't generated any NPEs from basic usage (NOTE: there are plenty more tests that I could do around bad format), it's integration test time. Integration testing a loader via Pig Latin is pretty simple: load data, then dump it, and validate that it looks like it should. Right now this is manual, basically running the script below, but output could/should be automatically validated.

Note that in order to use the UDF I've written, I need to specifically register it as shown in the first line below.

register '../../../target/CustomLoader-1.0-SNAPSHOT.jar'

-- the loader is fully path specified, and args are passed in using single quotes.
-- the file being loaded exists at the specified location in HDFS

A = LOAD '/test/hit_data.tsv' USING com.foo.bar.CustomLoader('0','2','6','19') AS (zero:long,two:chararray,six:long,nineteen:chararray);

C =  GROUP A BY zero;

-- this forces pig to execute the query plan up to the DUMP, which means invoking the loader. 

DUMP C;

-- note that the same loader can be invoked with a different number of arguments, and
-- fields don't have to be cast
-- the file being loaded exists at the specified location in HDFS

B = LOAD '/test/hit_data.tsv' USING com.foo.bar.CustomLoader('100','200');

DUMP B;

Conclusion
Writing the code took about 10 minutes. Testing it took much longer. That seems to be the pattern for me when writing (simple) UDFs. What I've noticed about Pig scripts and UDFS is that in order to validate functionality throughout the script/UDF lifecycle you always need to validate the generated Tuples to feel confident that changes have been made without regression.

Other than the lack of automation around integration testing, the actual Loader works as advertised -- it might need to change to accommodate new requirements, but it will work just fine for prototypical work with multi column CSV files.

Pull Parsing with STAX

2010-11-17T20:18:00.000-08:00

Foreward
Due to some events beyond my control, I lost the source code for parsing my Garmin TCX data into CSV format. I had not gotten around to Git-ifying my source, and also had not backed it up via JungleDisk/Mozy, so the original Ruby code that used the LibXML SAX parser is gone.

That might not necessarily be a bad thing. In SAX parsing, events happen without any surrounding context. It is up to the programmer to supply the context, and doing that in a legible, maintainable way with the SAX event driven model is a challenge. When I had discovered issues with the parsing code, fixing those issues required a lot of time to determine the actual state at the time the bug occurred. SAX parsing code has a significant maintenance penalty.

DOM parsing is much more straightforward because you navigate the structure of the XML, and implicitly get the associated context. Unfortunately, it is prohibitively expensive because it requires that the entire document get loaded into memory.

STAX parsing is a reasonable compromise between the two extremes. It streams the file (i.e. only loading into memory what it needs), while allowing the developer to navigate the structure of the XML. In other words, you have context without memory overhead.

STAX can be used to read or write XML files -- in my application, that converted .TCX files to a CSV format, I was focused on reading XML, not writing it. For reading files, STAX provides two different APIs. The cursor based API uses XMLStreamReader. The iterator based API uses XMLEventReader. The key difference between the two is that the iterator based API treats events as first class objects and allows the user to peek ahead at the next element to be fetched. This supplies the user with more context than the iterator based API. That context comes with additional resource consumption, but still much less than loading an entire DOM into memory.

Onward
In my first STAX program, I wanted to see how far I could get with the cursor based API. Specifically, given it's better performance characteristics, would it provide enough context for me to write easily maintainable code?

A Moment...
A quick side note on why (again) I'm choosing to convert an XML stream to CSV. XML is great because DTDs and Schemas provide a way to validate document integrity when there are large numbers of optional elements. In the case of TCX, there are few optional elements -- the elements that exist always contain the same kinds of data. With an unchanging format, CSV makes more sense because it is a more compact representation of a stable data set.

Implementation Details
In my re-implementation of TCX->CSV parsing code, I needed to transform a set of nested parameters into two different CSV formats. In order to explain what I needed to do, I need to go into detail about what the Garmin TCX XML looks like, and what I wanted to extract from it.

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>
<TrainingCenterDatabase xmlns="http://www.garmin.com/xmlschemas/TrainingCenterDatabase/v2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.garmin.com/xmlschemas/TrainingCenterDatabase/v2 http://www.garmin.com/xmlschemas/TrainingCenterDatabasev2.xsd">

  <Activities>
    <Activity Sport="Biking">
      <Id>2010-11-04T22:02:59Z</Id>
      <Lap StartTime="2010-11-04T22:02:59Z">
        <TotalTimeSeconds>1798.6000000</TotalTimeSeconds>
        <DistanceMeters>15412.0742188</DistanceMeters>
        <MaximumSpeed>12.8037510</MaximumSpeed>
        <Calories>581</Calories>
        <AverageHeartRateBpm xsi:type="HeartRateInBeatsPerMinute_t">
          <Value>140</Value>
        </AverageHeartRateBpm>
        <MaximumHeartRateBpm xsi:type="HeartRateInBeatsPerMinute_t">
          <Value>158</Value>
        </MaximumHeartRateBpm>
        <Intensity>Active</Intensity>
        <Cadence>0</Cadence>
        <TriggerMethod>Manual</TriggerMethod>
        <Track>
          <Trackpoint>
            <Time>2010-11-04T22:03:00Z</Time>
            <Position>
              <LatitudeDegrees>47.5834731</LatitudeDegrees>
              <LongitudeDegrees>-122.2491668</LongitudeDegrees>
            </Position>
            <AltitudeMeters>17.4411621</AltitudeMeters>
            <DistanceMeters>26.8590393</DistanceMeters>
            <HeartRateBpm xsi:type="HeartRateInBeatsPerMinute_t">
              <Value>85</Value>
            </HeartRateBpm>
            <SensorState>Absent</SensorState>
          </Trackpoint>
          ...
        </Track>
     </Lap>
     ...
   </Activity>
     ...
 <TrainingCenterDatabase>

I want to extract two kinds of data out of the XML stream above:

Lap summary data. Lap summary data is good for high level comparisons of effort. The basic components of Lap summary data of the same general duration can be compared across laps.
Trackpoint data. Trackpoint data -- elevation, heart rate, lat/long can be analyzed/transformed across arbitrary intervals to measure input effort and output speed.

Lap Summary Data will look like this:
activity_id,lap_id,total_time,total_distance,max_speed,total_calories,average_heartrate,max_heartrate
Trackpoint detail data will look like this:

lap_id,trackpoint_id,time, latitude,longitude,altitude,distance,heartrate

Both files may end up being used when correlating track points to their parent laps and activities.

Initializing the SAX Parser
I've created a class TCXPullParser to parse Garmin TCX data. In the constructor I initialize the STAX parser:

XMLStreamReader parser;
         .....
         /**
  * ctor, with all file names to write to and read from.
  * 
  * @param lapSummaryWriter
  * @param trackDetailWriter
  * @param fileToParse
  * @throws IOException
  * @throws XMLStreamException
  */
 public TCXPullParser(CSVWriter lapSummaryWriter, CSVWriter trackDetailWriter,
     String fileToParse) throws IOException, XMLStreamException {
  
  FileInputStream fis = new FileInputStream(fileToParse);
  XMLInputFactory factory = XMLInputFactory.newInstance();
  parser = factory.createXMLStreamReader(fis);
                ...
 }

Initialization is simple, the parser is created from the XMLInputFactory, and takes the FileInputStream created on the input file name. From this point my primary access to the file is through the parser object. I use the parser object to advance the cursor (by calling next()), inspect the type of element (as a return value from parser.next()), and grab text (parser.getText()). These three methods, with some additional functionality I've added, give me enough context to actually top-down parse the XML.

Pull Parsing

One advantage of using pull parsing is that the code is in charge of when events are fired. This lets us do things like skip processing/move directly to an element that we are interested in by using the parser.next() method and checking the returned element type:

/**
 * skips parser to the start element of the specified element name, while
 * stopElementName has not been encountered.
 * 
 * @param parser
 * @param elementName
 * @param stopElementName
 *          if we get this far, we've gone too far.
 * @return true if element is found
 * @throws Exception
 */
protected boolean skipTo(XMLStreamReader parser, String elementName,
    String stopElementName) throws Exception {
 boolean found = false;
 int parseType = parser.getEventType();
 while (parser.hasNext()) {
     parseType = parser.next();
     if (parseType == XMLStreamReader.CHARACTERS) {
             continue;
     }
     String elName = parser.getLocalName();
        if (parseType == XMLStreamReader.START_ELEMENT) {
  if (elName.equals(elementName)) {
         found = true;
      break;
  } else if (elName.equals(stopElementName)) {
      // in the case where we are looking across parallel elements
      // or into a container element,
      // stop when we find the stop element
     found = false;
    break;
  }
         } else if (parseType == XMLStreamReader.END_ELEMENT
         && elName.equals(stopElementName)) {
  // in the case where we are looking within a container element, stop
  // when we reach the end of that container element
  found = false;
  break;
     }
 }
 return found;
}

I typically use skipTo() to move to the next instance of an element, before it's containing element end tag is reached. For example, when I'm parsing the contents of a Trackpoint tag:

<Trackpoint>
   <Time>2010-11-04T22:03:00Z</Time>
   <Position>
   <LatitudeDegrees>47.5834731</LatitudeDegrees>
   <LongitudeDegrees>-122.2491668</LongitudeDegrees>
   </Position>
   <AltitudeMeters>17.4411621</AltitudeMeters>
   <DistanceMeters>26.8590393</DistanceMeters>
   <HeartRateBpm xsi:type="HeartRateInBeatsPerMinute_t">
   <Value>85</Value>
   </HeartRateBpm>
   <SensorState>Absent</SensorState>
</Trackpoint>

This is the code to parse that data:

/**
 * parses a single trackpoint and writes an output line to the trackDetail CSV
 * writer.
 * 
 * @param parser
 * @throws Exception
 */
protected void parseTrackPoint(XMLStreamReader parser, String lapId,
    String trackPointId) throws Exception {
 trackDetailWriter.writeArg(lapId);
 trackDetailWriter.writeArg(trackPointId);
 skipTo(parser, TIME, TRACKPOINT);
 trackDetailWriter.writeArg(getTimeValue(parser, parser.next()));
 skipTo(parser, LAT, TRACKPOINT);
 trackDetailWriter.writeArg(getValue(parser, parser.next()));
 skipTo(parser, LONG, TRACKPOINT);
 trackDetailWriter.writeArg(getValue(parser, parser.next()));
 skipTo(parser, ALT, TRACKPOINT);
 trackDetailWriter.writeArg(getValue(parser, parser.next()));
 skipTo(parser, DIST, TRACKPOINT);
 trackDetailWriter.writeArg(getValue(parser, parser.next()));
 skipTo(parser, HEARTRATE, TRACKPOINT);
 trackDetailWriter.writeArg(getValue(parser, parser.next()));
 trackDetailWriter.flushArgs();
}

With skipTo in place, I needed to extract the data from the XML. The text inside of elements is CHARACTER data, and is accessed by calling parser.next() after hitting the enclosing tag START_ELEMENT. The CHARACTER data is accessed via XMLStreamParser.getText():

/**
 * extracts a double value from a character stream.
 * 
 * @param parser
 * @param parseType
 * @return the double, or -1 if the element is not CHARACTERS. will also
 *         thrown runtime exception if parsing fails.
 */
private double getValue(XMLStreamReader parser, int parseType) {
 if (parseType == XMLStreamConstants.CHARACTERS) {
  return Double.parseDouble(parser.getText());
 } else {
  return -1;
 }
}

The combination of skipTo and getValue() allows me to extract Double values from the XML. I'm using Double to validate the format of the value, even though I'm going to persist that value back as a string. When extracting timestamps, I extract the data to a long:

private long getTimeValue(XMLStreamReader parser, int parseType)
    throws ParseException {
 if (parseType == XMLStreamConstants.CHARACTERS) {
  String raw = parser.getText();
  String date = raw.substring(0, raw.indexOf('T'));
  String time = raw.substring(raw.indexOf('T') + 1, raw.indexOf('Z'));
  SimpleDateFormat sdf = new SimpleDateFormat("MM-dd-yyyy-HH:mm:ss");
  Date actual = sdf.parse(date + '-' + time);
  return actual.getTime();
 } else {
  return -1;
 }
}

Separating Reading XML from Writing CSV
In the code above there are calls to a trackDetailWriter object. I chose to separate the writing of the CSV from the reading of the XML in order to test the XML reading logic more easily. This simplified things a lot, it allowed me to pass in CSVWriter objects, it relieved the parsing code of having to manage/open/close destination files, and it allowed me to write test implementations of CSVWriter that stored the data in memory for me to check during unit tests.

public interface CSVWriter {

 /**
  * write a single arg
  * @param arg
  * @throws Exception
  */
 public void writeArg(Object arg) throws Exception;
 
 /**
  * flush all pending args as a single CSV line
  * @throws Exception 
  */
 public void flushArgs() throws Exception;
 
}

The default implementation (used at runtime) looks like this:

/**
 * 
 * @author Arun Jacob
 *
 * push comma separated values to a file. 
 */
public class DefaultCSVWriterImpl implements CSVWriter {

 private FileWriter writer;
 private StringBuffer buffer;
 
 public DefaultCSVWriterImpl(String fileName) throws IOException {
  writer = new FileWriter(fileName);
  buffer = new StringBuffer();
 }
  
 /**
  * close the file: REQUIRED for all file writers.
  * @throws Exception
  */
 public void close() throws Exception {
  writer.flush();
  writer.close();
 }

 @Override
 public void writeArg(Object arg) throws Exception {
  buffer.append(arg);
  buffer.append(",");
 }

 @Override
 public void flushArgs() throws Exception {
  writeToFile(buffer);
 }

 /**
  * flush the contents of the buffer to file
  * @param buffer
  * @throws IOException
  */
 private void writeToFile(StringBuffer buffer) throws IOException {
  if(buffer.charAt(buffer.length()-1) == ',') {
   // remove the last comma before writing. 
   writer.write(buffer.toString().substring(0,buffer.length()-1));
  } else {
   writer.write(buffer.toString());
  }
  
  resetBuffer(buffer);
 }

 /**
  * clear out the StringBuffer
  * @param buffer
  */
 private void resetBuffer(StringBuffer buffer) {
  if(buffer.length() > 0) {
   buffer.delete(0,buffer.length());
  }
 }

}

The test implementation (used to verify that values are being pulled from the XML correctly) looks like this:

public class TestTrackPointCSVWriter implements CSVWriter {
 
 static final String TRACKPOINTID = "TrackPointId";

 static final String LAPID = "LapId";

 static final String ACTIVITYID = "activityId";

 List<Object> args;
 Map<String,Object> argsMap;
 public TestTrackPointCSVWriter() {
  args = new ArrayList<Object>();
  argsMap = new HashMap<String,Object>();
 }
 

 @Override
 public void writeArg(Object arg) throws Exception {
  args.add(arg);
  
 }

 @Override
 public void flushArgs() throws Exception {
  argsMap.put(LAPID, args.get(0));
  argsMap.put(TRACKPOINTID, args.get(1));
  argsMap.put(TCXPullParser.TIME, args.get(2));
  argsMap.put(TCXPullParser.LAT,args.get(3));
  argsMap.put(TCXPullParser.LONG, args.get(4));
  argsMap.put(TCXPullParser.ALT, args.get(5));
  argsMap.put(TCXPullParser.DIST, args.get(6));
  argsMap.put(TCXPullParser.HEARTRATE, args.get(7));
  args.clear();
  
 }

 
 /**
  * validation method
  * @param key
  * @return
  */
 public Object get(String key) {
  return argsMap.get(key);
 }

}

Conclusion
I'm not sure what I was thinking when I wrote the original SAX parser for TCX data, other than I just like to write in Ruby. The additional context that I get by being able to pull tags instead of getting them pushed at me makes the code much easier to follow and therefore maintain.

Datamining my GPS/HRM Data, Step 2: Pig for ETL

2010-09-24T22:11:00.000-07:00

Overview
Now that I've extracted 3 years worth of GPS/HRM data into CSV format, I want to get some basic summary information. Specifically, I want

the total distance covered for running, biking and 'other' (usually skate skiing) per month.
average run and ride mileage per month
average running and cycling pace per month

Pretty no brainer stuff that I could probably write a quick 'n dirty ruby script to do for my paltry 40MB of data.

But...I want to use Pig. While this may seem like a candidate for the "Cutting Butter With A Chainsaw" award, I'm actually trying to show that Pig Scripting is pretty damn useful for ETL. The nice thing about Pig is that when I'm dealing with 40GB or TB of input data, the same script can be run across a multi node cluster without changes to the basic logic.

I'm also going to try and capture what I've learned from being in the trenches with Pig over the last couple of months. We use it at work to process TBs of log data, and while it definitely has it's warty aspects, I find that it is well suited for basic summary, grouping, and filtering operations.

Pig Setup

I downloaded and setup as described here. Pig assumes that you've got Hadoop installed, and specifically DFS started, as well. I start DFS from the bin directory of my hadoop install: start-dfs.sh.
I'm running Pig in psuedo-local mode (with my local box set up as a single node hadoop cluster):

pig -fpigfile.pig

I symlinked the pig shell script to /usr/bin/pig for ease of use. I have HADOOP_HOME and JAVA_HOME defined, the pig shell script uses both, and assumes that it is located in a bin directory in a standard pig install.

Pig Latin, UDFs, and Tuples

Pig scripts are written in Pig Latin. I'm going to cover a small subset of Pig Latin in this script, see full references part I and II for comprehensive overviews. Where Pig Latin falls short, Pig has User Defined Functions that allow loading, storing, and transforming data. UDFs are written in java and extend defined interfaces.

Pig Latin and UDFs manipulate Tuples to generate sets of Tuples. Tuples are groups of 1..N values, where a value can be an int, a float, a string, another tuple, or a bag (set) of values. In Pig, you see a lot of

X = {do something in pig on} Y;

where Y is an original set of Tuples and X is the transformed result. Transformations of Data are done as Map-Reduce jobs. The complexity of what you are transforming determines how many map-reduce jobs are run.

The Goods

Load the data.

I'm loading the detail files generated from the XML->csv transform I did last time. I loaded all detail files into pig using the default PigStorage() UDF. In order to use this UDF, I had to specify the format of all fields in each row:

A = LOAD '/csv/input/details*' USING PigStorage() AS (activityId:chararray, lapId:chararray, time:chararray, latitude:float, longitude:float, alt:float,dist:float,hr:int);

This gives me a tuple A that contains data as specified above. Note that this tuple represents all of the rows of loaded data. Also note that I used a wildcard to load all files in at the same time. PigStorage() also allows the user to separate locations by comma, which is really great when loading from multiple locations.

Note that the locations above are in HDFS. I uploaded all of my data into HDFS using the hadoop fs -put command.

Extract the month from the timestamp using a custom UDF.

Right now I've got the timestamp formatted as a character array. I'm going to need to extract month data and add it to the list of tuple fields in order to get monthly averages. This is where the simple functionality of the pig script is supplanted by a UDF.

UDFs, as mentioned before, are used to load, store, and transform/extract data. I'm going to write one that extracts the month as an integer between 1-12 from the time string. Note in the listing below I broke out the actual formatting into a separate function that I could unit test w/o having to create a Tuple instance.

package com.infovoracious.udfs;

import java.io.IOException;
import java.util.Calendar;
import org.apache.pig.EvalFunc;
import org.apache.pig.data.Tuple;

public class GetMonth extends EvalFunc package com.infovoracious.udfs;

import java.io.IOException;
import java.util.Calendar;
import org.apache.pig.EvalFunc;
import org.apache.pig.data.Tuple;

public class GetMonth extends EvalFunc (Integer) { 
// parens represent brackets, this is a templated class

private static final int TIME_INDEX = 2;

  @Override
  public Integer exec(Tuple input) throws IOException {
    if (input == null || input.size() == 0)
      return null;
    try {
      // looks like this: 2010-02-24T14:22:29Z
      String str = (String) input.get(TIME_INDEX);
      if(str != null) {
        return DateUtils.dateFromFormattedTime(str,Calendar.WEEK_OF_MONTH)
      } else { 
        return -1;
      }
      return DateUtils.dateFromFormattedTime(str, Calendar.MONTH) + 1;
    } catch (Exception e) {
      throw new IOException("Caught exception processing input row ", e);
    }
  }

}

  private static final int TIME_INDEX = 0;

  @Override
  public Integer exec(Tuple input) throws IOException {
    if (input == null || input.size() == 0)
      return null;
    try {
      // looks like this: 2010-02-24T14:22:29Z
      String str = (String) input.get(TIME_INDEX);

      return DateUtils.dateFromFormattedTime(str, Calendar.MONTH) + 1;
    } catch (Exception e) {
      throw new IOException("Caught exception processing input row ", e);
    }
  }

}

I also created a UDF to generate the week of the month as well. I'm storing both the month and the week of the month as additional columns.

Load the UDFs.

In order for my script to see the UDFs I have just written, I need to load the jar that the UDFs reside in, and alias the methods so that they can be called in the script:

This is done by using the REGISTER keyword at the top of the script to load the jar, and the DEFINE keywords to alias UDFs. Note that when I define the UDF, I'm specifying it's constructor. In the cases above, GetMonth and GetWeekOfMonth both have default (no parameter) constructors.

REGISTER foo.jar;
    DEFINE extract_month com.infovoracious.udfs.GetMonth();
    DEFINE extract_week com.infovoracious.udfs.GetWeekOfMonth();

I invoke the methods like this:

-- the load from last time
A = LOAD '/csv/input/details*' USING PigStorage() AS (activityId:chararray, lapId:chararray, time:chararray, latitude:float, longitude:float, alt:float,dist:float,hr:int);

-- adding columns to a tuple.
B = FOREACH A GENERATE *, extract_month(time) as month , extract_week(time) as week;

Sort by month and week of month.
Now that we have the columns, we can sort by them. Sorting by month, week of month, groups values by unique month,week of month tuples:

C = GROUP B BY (month,week);

This creates a tuple that looks like this:

{month, week}, array length 1..N of {activityId,lapId,time,latitude,longitude,alt,dist,hr}

actual values would look like this:

{1,1},[{..},{..},..]
{1,2},[{..}...]
...

Summarize data.
With columns now grouped, I can summarize data from the grouped tuples. I want to summarize mileage per week. I can do that as follows (showing the whole script for continuity)

-- the load from last time
A = LOAD '/csv/input/details*' USING PigStorage() AS (activityId:chararray, lapId:chararray, time:chararray, latitude:float, longitude:float, alt:float,dist:float,hr:int);

-- adding columns to a tuple.
B = FOREACH A GENERATE *, extract_month(time) as month , extract_week(time) as week;

C = GROUP B BY (month,week);

-- the tuple now looks like group(month,week) B(tuple contents), 
-- so I reference dist as a member of the B tuple.

D = FOREACH C GENERATE *, SUM(B.dist) as total_dist;

Flatten the grouping columns.
Using the FLATTEN keyword to remove tuple nesting has different effects depending on where you use it. If you use FLATTEN to flatten the grouped columns, it merely removes the bag from around the grouped columns:

{month, week}, [array of tuples]

becomes

month, week, [array of tuples].

Flattening out the array generates a new row for each array element:

1,2,[{a,b},{c,d}]

becomes

1,2 ,a,b

and

1,2,c,d

In this case I want to flatten out the grouping columns:

-- the load 
A = LOAD '/csv/input/details*' USING PigStorage() AS (activityId:chararray, lapId:chararray, time:chararray, latitude:float, longitude:float, alt:float,dist:float,hr:int);

-- adding columns to a tuple.
B = FOREACH A GENERATE *, extract_month(time) as month , extract_week(time) as week;

-- grouping
C = GROUP B BY (month,week);

-- summing (and discarding what I don't need)
D = FOREACH C GENERATE SUM(dist) as total_dist;

-- flattening:
-- note how I'm referring to the grouping columns by the 'group' special keyword.
-- note how total_dist is not scoped by an enclosing tuple b/c of the way it was generated above.
E = FOREACH D GENERATE FLATTEN(group),total_dist;

Store it.
Finally, I want to store the results of my work. This is done using the default PigStorage() store UDF:

STORE E into '$some_dir';

Note that I didn't need to specify 'using PigStorage()'. I also used a variable, which I pass into the pig script as a key-value pair, like this:

pig -p some_dir=foo...

And that's it! I've summed my mileage from each week per month over the last three years: in six lines of script!

Conclusion

Pig is a powerful tool to clean up and perform basic operations on data. When you know what you need to do, and have to do it on a lot of data, it works remarkably well. That said, there are a couple of things that start to hurt over time:

When you write a lot of Pig, you start wanting to re-use sections of script, i.e. make them functions. Except you cant. Yet. So you end up either cutting and pasting (bad), or saving variables off for later use (confusing when you try to use them later) instead of writing script based UDFs (preferred).
There is no conditional execution of an operation. I can't test a variable state and then execute one or another statements based on that variable. What this means in real world use is that I do all of my conditional testing in bash, prior to invoking the pig script, and build up variables based on the results of those conditional tests. Those variables then get accessed by operations.
The default pig serialization format is great for the data I was using above because it had no commas which are the default Pig serialization format delimiter. You can change the delimiter, but that begs the question of what happens when someone has injected your delimiter into the data you are trying to process.
Higher level workflow is not solved with Pig. We started out running pig via a set of cron jobs, which soon turned into an admin's worst nightmare. Besides just being hard to maintain, we were also not taking advantage of the actual sources and destinations of pig data to schedule the work. In other words, if Pig job A produced intermediate format B and then continued running, we had no way of starting Pig Job C that loaded format B until Pig job A completed. We are currently evaluating oozie at work to see if it can improve on cron+bash (a low bar, I know).

Warts and all, Pig sure beats writing map reduces for simple grouping/aggregation functionality. We still write map reduces when we are doing more complex operations, and our ETL job flows end up looking like a combination of pig, raw map-reduce, and HDFS access. But Pig really allows us to get away from a lot of boilerplate work (and maintenance!) for processing the data. Which gives us more time to analyze it.

Next
Of course, just getting summary statistics on my data hasn't really scratched the itch that started me off in the first place. While I now have an idea of my general fitness month to month, I still don't know enough about the amount of hard versus easy running/cycling I did, or how fast I was going when I was going hard or easy. Answering those questions will help me determine my fitness, which I previously defined as heart rate vs pace, accounting for terrain. In order to get a metric for that definition of fitness, I need to define of what 'hard' and 'easy' really are for me, and analyze how specific values of those workouts have changed for the better or worse over time.

Data Mining My GPS/HRM Data: Step 1, Formatting the Data

2010-09-07T22:45:00.000-07:00

I've been wanting to analyze the data from my Garmin 305 for a while now. I've been a casual runner/biker/hardcore data geek for a while now, and last year I started doing triathlons, which means even more data to analyze. While I've always been curious, I just haven't had a great 'need' to analyze it until now...

I'm switching from a primarily heart rate based training program to a 'pace based' training program. The former had me training within specific heart rate zones, the latter has me running at specific pace ranges. I'm a 'measurer', and I'm curious to see how effective (or not!) the pace based training program is. Fortunately I have one device that tracks both heart rate and speed, and I can analyze the effect that the pace based training has relative to the effect that the heart rate based training has on my overall fitness.

In this case I'm measuring fitness as a combination of heart rate vs terrain covered, i.e. hilly vs flat, vs pace. In order to measure my fitness up to now and going forward, I need to answer the following questions:

how much time did I spend in 'recovery' mode, where my heart rate was < 70% max
how much time did I spend in 'pain cave' mode, where my heart rate was > 85-90% max?
how much faster (or slower) did I get at the same heart rate over the last year?
for the new pace based program, how much time am I spending at the different paces, i.e. recovery pace, base pace, marathon pace, 1/2 marathon pace, 10k pace, 5k pace, 1 mile pace?
what is my average heart rate for those paces?
how much faster (slower) am I getting?

I'm hoping to answer these questions using several approaches and several technologies that I've been using at work, and others that I've been itching to try.

The first thing I needed to do prior to doing any analysis was to format the data into a format that I could easily operate on. The data is exported from the device into a format called tcx, which is a schema-validated XML.

I need the data in csv format, mainly because the tools I want to process the data with are all hadoop based, and while I've read that it is not only possible, but easy, to process XML with hadoop, hadoop works best with csv formats. XML is a nice format for nested data, and this is nested data, with the following structure

activities

activity

laps

lap -- contains summary averages from trackpoint data (see below)

trackpoints

trackpoint -- contains snapshot heart rate, altitude, distance, etc.

XML is especially good when there are optional attributes. CSV tends to suck with optional attributes, because nothing is optional. In this case none of the attributes are optional, so ultimately XML is overkill for storing this data.

I collapsed this structure into two csv lists: summary data and detail data, because I planned to act on summary and detail data separately.

The summary data contains lap summary data:

activity id, lap id, total time, total distance, max speed, max heart rate, average heart rate, calories, number of trackpoints.

The detail data contains trackpoint data:

lap id,time, latitude, longitude, altitude, distance, heart rate

To do the conversion, I used the ruby libxml Sax parser. In order to use the libxml sax parser, I needed to create a callback handler that implemented the methods I wanted to override.


class PostCallbacks
  include XML::SaxParser::Callbacks

  def on_start_element_ns(element, attributes, prefix, uri, namespaces)
  ...
  end


  def on_characters(chars) 
  ...
  end


  def on_end_element_ns (name, prefix, uri) 
  ...
  end
end

In the callback handler, I maintained state to track nested XML objects. Typically I would assign state in the on_start_element_ns() method, act on that state in the on_characters() method, and release state in the on_end_elemebt_ns() method. I would also flush my results to disk occasionally to avoid taking up an unreasonable amount of memory.

I had about 40 Meg of data from the previous 3 years, which was parsed into csv files in approximately 44 seconds. I'm more than happy with that performance right now, because this is essentially a one-off job to get the data.

Next Up: setting up a data processing pipeline using Pig.

Beware the Emerging God Object

2010-08-27T21:34:00.000-07:00

Most God Objects are pretty obvious. Bloated, side effect filled, their existence guaranteed by collective fear about refactoring them to a state where their previous functionality is not reproducible.

The God Object that derailed me for the last couple of days was not bloated, it's functions (for the most part) did not have side effects. It did a lot, but did not seem to be overwrought. In fact, I'm pretty sure it only had a minor God Complex until I started refactoring it!

As a software engineer, my goal is to produce the most effective, robust software with the least amount of effort. So, in an attempt to not repeat last week's thrash-fest, I'm going to catalog the warning signs, the resolution, and the corrected approach, because I'm pretty sure I'm going to be in this situation again.

The Task:
We need to make an existing protoype table driven. The prototype functionality configured and ran A/B tests. Its configuration data was currently hardcoded into an enum.

Warning Signs Before We Even Started:

The core logic of the A/B test configuration resided in a single object.
Despite having several distinct (logical) sub components, this object was responsible for serializing itself and those sub objects.
This object existed in both our configuration time and runtime environments, which have significant differences, the biggest one being that at configuration time, all objects are read/written to a relational database, and at runtime, they are read from a BDB store. Most conceptual objects in the system have config time and runtime implementations.
We could not get rid of this object because it was still being used in production. We would have to implement a phased approach to upgrading it.
I did not know the codebase well.
Our sprints are a week, so we felt compelled to fit in an end to end task into a week.

The first four warning signs are simply facts. The last two are things that I should pay more attention to. If I don't know the codebase well, and I'm trying to make significant improvements in a week, my chances of success go way down. I chose to ignore that reality.

Warning Signs Once We Started:

The table driven logic did not mesh well with the existing logic. The developer who had written the code had serialized all sub objects into a JSON string. The object was then serialized into a BDB file as a set of attributes and the JSON string, not an object graph. Trying to work with the new table data (an object graph) mean that:
Any change I made to the code had side effects. This is because I was trying to preserve the original signature of the object to maintain backward compatibility with the rest of the system. This became obvious when:
The original code, fairly straightforward, became conditional-heavy. Instead of acknowledging the additional complexity,
I was choosing to 'hack' more because I knew that we would have time to rework the code. I knew I was hacking because
The developer I was working with started to get more confused with every checkin I made.

Do Over!
When his brow started cramping up from being so furrowed, I realized that things were wrong. We grabbed a whiteboard and went to work, depicting the original workflow, how adding table driven configurations changed that workflow, and how to change the existing workflow as little as possible to reduce the amount of rework that we would have to do in the runtime portion of the code. Most of the good ideas came from our dev lead, who had been silently (and then not so silently) observing us trying to make sense of it all. I mention that because he was operating at a distance from a lot of the complexity, and his solution avoided most of it. I felt that he out of all of us was able to make the necessary leap out of the stew of constraints and details, and reframe the problem in a way that made it a much less complex problem. That's the genius of software. That's what I do, but only sometimes. That's why I'm writing this down.

The Problems and Their Solutions:

The old functionality contained both runtime behavior and configuration and behavior. We chose to separate the two, resulting in a POJO configuration data object used at configuration time, and the 'action' object that knew how to initialize given one of those data objects and contained runtime behavior.
More separation of concerns: The old way of storing data in a JSON string and the new table driven way of storing data did not belong together in the same object. Even though we separated the old object into a data object and an 'action' object, we chose to keep the old data separate from the new data, and access both through getters/setters, which insulated us from the underlying implementations.
Composition: because we didn't have enough time to rewrite the infrastructure that wrote the configuration object into a BDB, we needed the new data to be persisted into configuration object as part of the JSON string. Fortunately, once we separated out the data from the configuration object, we could compose the JSON string outside of the configuration object using the POJOs, then set it prior to serializing the configuration object to the BDB file.

Conclusions:
The warning signs before start played out into more significant problems before they were fully resolved. And, when I think back on the process, I realize I was uncomfortable with the changes I was making, but I chose to push on. Because time was relatively compressed, I found myself making 'poor choices', which ultimately backfired. Next time (these are notes to myself, the 'you' I'm taking to is 'me' :)

Acknowledge risks up front. Not a lot of time to get it done? That's a risk. Don't know the codebase? That's another risk (assuming the codebase is non trivial).
Examine the old code and determine if there are concerns that should be separated before anything else is done.
Determine the scope of those changes to see if they can be met in the desired time frame.
If not, break the project into smaller chunks.
If it feels easy it's right. If it feels hard, it's less right. Be more right.
Do not hack. When you find yourself hacking, stop. Back up and revisit initial assumptions. Step away from the problem.
Discuss ideas often, with others. If someone else can understand it easily, it can't be that bad. If, on the other hand, their brow starts to wrinkle, and your blood pressure starts to go up because there are not words in the English language to describe what you need to describe, it might just be worse than you think.

More Conclusions:

God objects never started out as God Objects. They started out as high level concepts -- "oh, I need something that does X", and mutate because of bolted on additional functionality.
Most developers don't set out to create a God Object. In this specific case I was trying to save effort by reducing the amount of change to the original code. However, the additional effort to explain and understand my 'bolt ons' was costing real time and effort.
The earlier you de-factor complexity out of a God Object, the better. Decomposing layered functionality into discrete objects makes it really easy for other people to recognize what those objects are for, and not overlay them with functionality that addresses other concerns.
Conversely, the longer you wait, the harder it is. God Objects tend to have code with lots of side-effects and implicit assumptions. Refactoring those successfully is hard, even with comprehensive unit tests.

Synchronization Redux

2010-04-21T21:48:00.000-07:00

In my last post, I discussed how I had refactored the synchronization around some data structures to reduce the need to lock those data structures except when necessary. My solution was cleaner in that the synchronization was localized, but the actual synchronization was still a little bit hairy in that it required a recheck inside the lock:

int newVersion = specialFooCache.getCacheVersion();

 // we only want one thread to refresh the cache, and we 
 // want the other ones to keep using the old data structures.

 if(newVersion != cacheVersion && inProgress == false) {
  synchronized(lock) {
   // first thread is in b/c cacheVersion has not been updated yet.

   // All other threads    evaluate to false.        
   int checkVersion = flexibleStrategyCache.getCacheVersion();
   if(checkVersion != cacheVersion) {

}
}
}

But I was (a) stuck and (b) had other code to refactor under the gun. So I didn't think about it much until today in the code review. I'm really glad I code reviewed with John, because he is a relentless simplifier whose lives by the 'less code' motto.

Right away I could tell that the whole recheck thing wasn't sitting well with him. The more I explained, the more concerned he looked, until he spotted a potential race condition between threads that might have piled up outside the synchronize when the someCache had incremented while it was being reloaded. These threads would immediately reload even when they didn't have to, which didn't affect the integrity of the in memory data structures, but was definitely a bad use of cpu. He also was insistent that there had to be an easier way, and under his direction this is that easier way:


 int newVersion = 0;
 // lock is shorter
 synchronized(lock) {
  newVersion = specialFooCache.getCacheVersion();
       
  // leave right away if in progress
  if(cacheVersion == newVersion || inProgress == true) {
   return;
 }
       
  inProgress = true;
 }
                
           
 try {
  // load from cache
 
 } catch (Exception e) {
 ...
 } finally {
  synchronized(lock) {
   // set the version. 
   cacheVersion = newVersion;
   // we are done here.
   inProgress = false;
           
  }
 }

A couple of things to note:
(1) This solution is definitely easier and more performant in addition to being correct. The lock is not held during the reload, and when it is released other threads will exit the function immediately if a cache rewrite is in progress.
(2) The key thing that I learned from watching John solve the problem is that he wasn't happy until it was dead simple. When I don't take this approach, or abandon it under pressure, there are some potentially costly ramifications. The original solution had a complexity smell around it that I chose to ignore. The problem with ignoring that smell is that it comes back to haunt after you've forgotten why you wrote it in the first place. The solution is, of course, to write code that is simple enough to re-understand months from now. Or simple enough for anyone not familiar with your code to understand.
(3) the lock at the bottom is not actually necessary since (a) the variables being set are volatile and (b) only one thread will get this far. However, since only one thread is getting this far, it is not a performance issue, and actually lends some clarity to the resetting of test conditions.
(4) I still feel like this is somewhat a work in progress...I'm not convinced that (3) is really required (even though we both signed off on it). But I do feel a lot better about the current algorithm. That said,
(5) I love code reviews. The knowledge transfer is huge. I'm always looking for ways to strip my code down, and the insight I get from a good reviewer (like John) raises my game tremendously.

Synchronization Requirements

2010-04-15T22:42:00.000-07:00

Recently I was lucky enough to pull the short straw and refactor the dreaded FooManager (name changed to protect the innocent). FooManager was a horrendously complex, overwrought class that suffered from at least two God Complexes, and my team had been playing hot potato with it for the last couple of months, since we learned we had inherited it.

My mission was to make FooManager understandable/maintainable without requiring heavy ingestion of psychotropic substances or becoming suicidal. After doing the usual de-Godification and Cleverectomy procedures, i.e. choosing decent abstractions, removing static methods, etc, I was left with a fundamental synchronization dilemma. It was far from the most annoying part of FooManager, but it was definitely the most interesting to refactor.

FooManager received updates for each specific Foo subclass from different caches. It registered for those updates using a callback interface. The cache notified FooManager whenever it changed. This seemed like an innocent enough pub/sub pattern. But it made managing the datastructures in FooManager -- the ones that held pre-computed indexes of Foos by name, Foos by id, Foos by other id, etc -- tricky.

All access to these data structures were done within read/write locks. I think the motivation behind using a read/write lock instead of a more generic synchronization mechanism was to optimize reads at the expense of writes. This made sense, given that writes were infrequent and driven by periodic cache changes, and reads did not need immediate access to new data. But it meant that any time you wanted to access the structures, you did so within the confines of a read/write lock. Even when all you wanted to do was read the data, you needed to acquire the read lock on it.

The event-callback and the read/write lock had a Design-Pattern-as-a-Hammer smell to it, and as I thought about it some more, I realized why.

(1) The data structures under read/write locks did not have to be immediately consistent with the cache, they had to be eventually consistent with the cache. That meant that a cache change did not require immediate and total synchronization.
(2) The read/write lock was overkill, but necessary because of the asynchronous nature of the cache refill.
(3) The loading performance of the cache was really minimal, since the cache data structures were in memory by the time the notification occurred.
(4) What we really wanted to do was only block when a thread was updating the data structures. We didn't want other threads re-updating the data structures. But they shouldn't be blocked from accessing the current data structures while the update was going on.

The eventual consistency and low reload overhead of the system made it possible to do away with the event callback interface and poll the cache for an update by checking a cache version number via a synchronized method. Heres how it worked:

Every request would check the cache version.
The first thread that got an updated version number would enter a lock and update a set of temporary data structures.
All other threads would block on the lock, enter once the original writer method had exited, make the version check, get the same version that they had entered with, and not attempt a reload.


synchronized protected int checkExpired(int currentVersion) throws Exception{

  int newVersion = specialFooCache.getCacheVersion();
  if(newVersion != currentVersion) {
   // update temp data structures.
   ...
   // assign temps to member data structures
   ...
  }
  return newVersion;

}

Prior to every request, I checked the cache version and reloaded if necessary:


 // cacheVersion is a member variable of FooManager
 cacheVersion = checkExpired(cacheVersion);
 // now access structure w/o locks.

This was, well, OK. It was definitely more simple, but now all reader threads were blocked while waiting for the first one to update the data structures.

Thinking back to the original conditions, I remembered that as long as the data structures were eventually updated, all non writing threads would be just as happy using the original structures in an unblocked manner. In other words, we only needed to make sure that one thread reloaded the original structures.

This required synchronization at a more granular level: I removed synchronization on the checkExpired() method, and inside of it, blocked the rewrite code using a conditional statement followed by a lock, and allowed all threads that had gotten past the conditional and stuck on the lock to exit once (a) the lock was released and (b) they realized that they didn't need to do the cache reload.


protected void checkExpired() {

 int newVersion = specialFooCache.getCacheVersion();

 // we only want one thread to refresh the cache, and we 
 // want the other ones to keep using the old data structures.

 if(newVersion != cacheVersion && inProgress == false) {
  synchronized(lock) {
   // first thread is in b/c cacheVersion has not been updated yet.

   // All other threads    evaluate to false.        
   int checkVersion = specialFooCache.getCacheVersion();
   if(checkVersion != cacheVersion) {   

    inProgress = true;
    // reload from cache
    .....
    // set data structures so that subsequent threads get
    // the new data
    ....
    // set the version. 
    cacheVersion = newVersion;
    // we are done here.
    inProgress = false;
   } // end version recheck
  } // end lock. 
 } // end version and inProgress check

Note that in order to prevent thread local caching and allow all threads to get immediate access to the data structures after they were reassigned, I declared the data structures that I was updating as volatile.

The simplification of the cache update process made it a whole lot easier to understand, I think that the conditions -- specifically the eventual consistency -- allowed us some latitude in when reader threads actually got the data. There isn't a specific by the numbers approach to solving synchronization issues, but it always helps to understand what you are synchronizing, and more importantly, what you don't have to synchronize.

Using Google Maps API v3 + JQuery.

2010-04-01T22:56:00.000-07:00

My side project to display and do some detailed data mining of my GPS exercise data, has been languishing for the better part of a year while my day job(s) have been taking most of my day and night time. Garmin has a pretty decent Mac desktop program that provides graphing, graph zooming, and stats by mile, but I'd rather have all of that functionality (and more!) via a web UI. I'd also like to integrate the results of the data mining into that UI in a useful way, i.e. mining relative effort over similar terrain to track actual fitness over time.

I decided that this project is going to be about having fun and nothing more, and as such decided to write the UI first, because all of my work these days is server side, Java, so 'fun' for me involves more dynamic languages, i.e. JavaScript and Ruby.

I wanted to display my gps data as a route on a map, which meant getting up to speed on the Google Maps API, and writing a quick dummy server that could dump out some route data for me.

I decided to use the latest Google Maps API v3 , and of course JQuery for the front end work, and mocked up a quick backend server using Sinatra. I can always redo that backend in something more robust once I want to actually deploy, but for now getting the data to the page is more important than how fast that data is retrieved, or machine resources consumed by serving that data.

Part 1: Displaying The Map

I needed to include the google maps api v3 js:

http://maps.google.com/maps/api/js?sensor=false
and the latest JQuery:
http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js

Then in the ready function, I created a map by pointing it to a div and passing in my options.


$(document).ready(function() {
    var myLatlng = new google.maps.LatLng(47.5615, -122.2168);
    var myOptions = {
          zoom: 12,
          center: myLatlng,
          mapTypeId: google.maps.MapTypeId.TERRAIN
    };
    var map = new google.maps.Map($("#map_canvas")[0], myOptions);

Note above that I'm assigning the map to a div with id = "map_canvas".

Part 2: Parsing the GPS data

Eventually I'd like to upload data directly from my device, but for now I'm going to skip that part and 'pretend' I've already done it. GPS data is exported in the TCX format, which is Garmin-proprietary, but is the easiest to use right now. My current desktop program has the ability to export 1 to N days worth of data into tcx.

At some point parsing tcx using a DOM based parser was going to start hurting, so I decided to use a SAX based parser from the start. My usual choice for quick n dirty XML/HTML parsing, hpricot, option was therefore not an option. I investigated nokogiri, but eventually settled on libxml, mostly because the rdoc on sax parsing was very clear, and it was much faster for sax parsing.

I mainly wanted to parse lat-long data out of the tcx file and dump the coordinates into another file in JSON format. Here is my 5 minute hacked together code:


class PostCallbacks
include XML::SaxParser::Callbacks

def initialize(write_file) 
  @state="unset"
  @write_file = File.open(write_file,"w")
  @buffer = "{\"data\" : ["

end

def on_start_element_ns(element, attributes, prefix, uri, namespaces)

  if element == 'LatitudeDegrees'
   @state = "in_lat"
  elsif element == 'LongitudeDegrees'
   @state = "in_long"
  end
end

def on_characters(chars) 
  if(@state=="in_lat") 
   @buffer += "{\"lat\": #{chars}"
  elsif(@state == "in_long")
   @buffer +=  ", \"long\": #{chars}},"
  end
end

def on_end_element_ns(element,prefix,uri)  

  @state="unset"
end

def on_end_document() 

  @buffer = @buffer.slice(0,@buffer.length-1)
  @buffer += ("]}")
  @write_file.puts(@buffer)
  @write_file.close()
end

end

parser = XML::SaxParser.file(ARGV[0])
parser.callbacks = PostCallbacks.new(ARGV[1])
parser.parse

Part 3: Serving The GPS Data Up

My goal here was to basically dump that generated file as a response to an AJAX request. Sinatra is perfect for delivering quick services like this. I use Sinatra's DSL to handle a GET request as follows:


require 'rubygems'
require 'sinatra'
require 'open-uri'
require 'json'

  get '/sample_path.json' do
   content_type :json
   File.open("../output/out.json") do | file | 
   file.gets
   end

  end

The ../output/out.json is where I parsed the lat-long data from the tcx file into.
I ended up spending a lot of time debugging my get method (http://localhost:7000/sample_path.json). In FF I was unable to get a response back from my GET request. I googled around and apparently there are some compatibility issues with firefox 3.6.2 and JQuery. I was however able get the code to work in Safari, and I'm considering downgrading to FF 3.5 because I haven't seen those kind of problems with that browser, and Firebug is an essential part of my debugging library.

Part 4: Drawing The Data On the Map
The JS code that made the request loads the results into google.maps.MVCArray, which it then uses to create a polyline superimposed on the map:


var url = "http://localhost:4567/sample_path.json";
$.ajax({
  type: "GET",
      url: url,
      beforeSend: function(x) {
        if(x && x.overrideMimeType) {
           x.overrideMimeType("application/json;charset=UTF-8");
       }
      },
      dataType: "json",
      success: function(data,success){

         var latLongArr = data['data'];
        var pathCoordinates = new google.maps.MVCArray();
        for(i = 0; i < latLongArr.length; i++) {

     // each coordinate is put into a LatLng.
     var latlng = new google.maps.LatLng(latLongArr[i]['lat'],
        latLongArr[i]['long']);
     pathCoordinates.insertAt(i,latlng);
   }
   // and this is where we actually draw it.
   var polyOptions = {
     path: pathCoordinates,
     strokeColor: '#ff0000',
     strokeOpacity: 1.0,
     strokeWeight: 1
   };

   poly = new google.maps.Polyline(polyOptions); poly.setMap(map);
   }
});

The Result

Not super impressive, but a good start!

Don't Drive by Dumb Dogmatic Data

2010-03-09T22:06:00.000-08:00

My posting has fallen off severely since taking up the new job, mostly because I have had nothing intelligent to say while I'm trying to ramp up on the technology as well as the business drivers behind the technical decisions being made. However, something did come up today that I want to remember, and I've heard more than once that if you don't write it down, you don't remember it.

I've been thinking a lot about numbers. In this brave new world, a lot of companies that are 'data driven'. Meaning they make their decisions based on the data that is around them, and if they can't measure it, they can't manage it.

That is a statement that sounds basically logical, assuming that
(1) what is being measured is clearly understood by everyone, and
(2) changes that occur to the measurement correlate well to overall system state.

In the engineering world, people don't have a lot of patience with numbers that are not explainable. So the mythical 'sales numbers' that drive entire sales teams off of cliffs every quarter are usually sneered at by engineers, who hold themselves up as the high priests and priestesses of logic.

For engineering treams, numbers like TPS, MTTF, etc, are not only easy to conceptualize, but changes in them are good indicators of system functionality. More importantly to engineering organizations, you don't have to be a software developer to understand what a decrease in TPS or MTTF means to the business.

So engineering management is always looking for other numbers that encapsulate system health. Again, this is a perfectly reasonable goal, because good metrics serve as a useful abstraction layer around the grimy bits of the sausage factory. However, I think that the quest for engineering is one more piece of evidence that shows that how rational starting points end up being ridiculous the moment logic is abandoned in favor of dogma.

We've all laughed ourselves silly at the old stories of measuring programmer productivity by lines of code written, but what are the programmers of tomorrow going to laugh at? My first candidate would be measuring the quality of unit tests by the unit test code coverage metric -- specifically what percentage of total lines of code are covered by unit tests.

These days unit test code coverage is easy to get. We get ours from a Cobertura plugin for Maven.
Code coverage is one of those measurements that initially sounds really good. If the test coverage decreases, that's bad, right? If it increases, well, good job to the developers!

Wait, not so fast. If line coverage is supposed to be an indicator of quality, that implies that just because a test causes code to be exercised, the test is good. But wait, I can write lots of tests with zero assertions. I've verified that in very specific cases there are no NPEs, but that's about it.

If I take unit test line coverage to represent the quantity of unit tests written without looking at the number of assertions being made per test, I'm only seeing part of the picture. If unit test coverage number changes indicate that testing is or is not being done on new code, there could be lots of false positives and negatives. For example, when I add a bunch of code in a finally block, and the function I'm adding that code to is in a unit test, my line coverage goes up without me actually writing any more tests. Conversely, if I'm adding that finally block to a function that is not covered, my line coverage goes down. In either case, do the corresponding line coverage increases and decreases actually mean anything about the quality of the tests written?

What are good indicators of unit test quality if line coverage is misleading? As someone who writes a lot of unit tests, I would venture that test quality has some correlation to assertion density, with some caveats. In other words, what and how much is being checked when a method is tested? Assuming that the tested method that returns a value, there is at a minimum one thing to check. If the value is a structure, there is more.

In any case, assertion density usually means that verification is being taken seriously, and also that any changes to the code have to pass all assertions - or the assertions need to be changed to match the new code. Either case requires explicit validation of the contract put in place by the assertions in the unit test. Note that assertion density is only valid when measuring direct output -- if a test is verifying data that is not a direct output of the method being tested, is it condoning code side effects? Assertion density needs to be normalized by the number of acceptable assertions, i.e. the number of things you can check in the return value, if there is a return value. The assertion density metric should score badly if data that is not explicitly related to the method output is being checked. But maybe that would be conflating the concerns of side effect free code and high quality unit testing.

Another metric in unit testing that correlates to good coverage is conditional branch coverage. If I can assume that every block of code may contain one or more possibly nested conditional statements, then I know that I've at least got decent coverage when a high percentage of conditional branches are covered. I dont think that branch coverage means a lot without assertion density checks, but it does mean a lot more than simple line coverage. Ironically, Cobertura provides branch coverage, but all of the QA managers I've worked with have gravitated towards line coverage as the more meaningful metric.

Ideally I would like to see a number based on assertion density and branch coverage. This number would behave well across a wide range of assertion and branch coverage input, sort of like the half your age plus seven dating metric. That would make it meaningful, and a good measurement to drive test quality 'up and to the right'.

Impedance Mismatch

2009-12-21T22:52:00.000-08:00

Post Mortem:

Long story short: every time I've had a project fail, one recurring theme has been bad technology choices. There have been others, like group dysfunctionality aka 'collective asshat fatigue', where the entire group stops functioning to avoid dealing with one or more aberrant personalities. Bad project scoping/definition also contribute to failure rates, but I don't think that there is much intersection between groundbreaking work and the prototypical well defined, well understood project. So it would seem that issues of scoping as well as team dynamics are (a) exacerbated by and therefore (b) secondary to the bad technology choices that put the project in jeopardy.

I thought I had gotten better at detecting when I was making a bad technology choice, but I recently made one. Fortunately we were able to turn things around, but it was hard. In the interest of not making this class of bad decision again -- because the definition of insanity is to do the same thing and expect different results -- I want to dissect what went wrong, what went right, and what I learned.

The Choice:

I recently started a new job. My first task was to jump in and assist on a prototype project by writing feed parsers that would parse millions of rows of comma separated values from feeds into various records in an SQL database. I was initially constrained to using Java. I took a standard object=row approach persisting single objects at a time.

The code was tdd, separation of concerns was good, all unit tests passed (coverage was good). Several hundred lines of code were required to parse, clean, validate, and insert data from various feeds into the database. I did not use an ORM, I used straight SQL via JDBC.

The Results:

The performance was horrendous. Inserting several million rows took hours, hours that we simply didn't have. The performance impacted the effectiveness of every down the line operation, and was jeopardizing the success of the overall project. Not a good way to start a new job :)

The Workaround:

We ended up ditching Java completely and rely on existing unix commandline tools to parse the files, insert into temp tables, and do bulk updates/inserts of rows from those temp tables into the main, 'canonical' tables. In other words, the 100s of lines of java parsing and insertion code that I wrote in a week or so (counting unit testing) and frantically reworked several times to try and speed up was replaced by something like this:


cat rawfile.csv | cut -d, -f1 | tr ":upper:" ":lower:" | sed -e"s/^m//g" | sort | uniq > psql -c "copy tablename from stdin using delimiters ','" | actual_queries.pl

This took a multi hour query down to 5 minutes. There was a bunch of pre-formatting prior to inserting into the database, and a perl script that ran afterwards, using DBI to copy/update from the temp table.

In general, the one rule that emerged was 'do as much processing before going to the db'. For example, determining set exclusion/intersection, which is something I would have definitely gone to code or SQL for, could be done via commandline via the comm utility:


comm -12 <(sort file1) <(sort file2) gives the intersection of file1 and file2.
comm -13 <(sort file1) <(sort file2) gives unique lines from file2
etc.

Conclusions:

Conclusion 1: Tests Still Required.

The good thing about piping a bunch of common unix tools together is that they have been around for a long, long time. Meaning you don't have to worry about the integrity of the data as much as you have to worry about using the tool options correctly. The bad thing about this approach is that the only kind of testing is integration testing, and it is easy to blow off when the initial solution works (or seems to).

After getting bitten when the queries worked but the data had integrity issues that manifested in the logic, we ended up writing a bunch of scripts that verified data integrity by making queries and inspecting result sets. We also leveraged the database, adding constraints that would allow the script to fail fast and alert us to schematic integrity issues, like duplicate rows.

Conclusion 2: It's Not the Databases Fault.

The database is a very convenient scapegoat, but the truth is that I spoon fed data into the database, and I could only move as fast as I could move my spoon (in Java). The better approach is to bulk feed data into the database, via bulk copies and bulk inserts/updates. Again, verification/validation scripts and constraints are required.

Conclusion 3: SQL Good, ORM Bad.

The truth is that we could have done this in Java, had we just used the same SQL we ended up using in the Workaround. My mistake when using Java was to put on my ORM blinders, which are great for when I want to pretend that there is some arbitrary data store underneath my code. This works until it doesnt, usually at 12AM the day of a release.

Multiple FAILS mean I'm done pretending the database is some fuzzy abstract data 'store', because I will use one when I want to want to mine data along arbitrary axes -- in other words, I'll use a database precisely to use SQL and not some mapping to it. SQL is a mature and extremely powerful way to ask open ended questions of a schema. If I don't want to ask open ended questions of my data, I shouldn't use a database. Because that's what they're built for. BTW I haven't used Hive or Pig yet, but these seem to be the QL solutions for much larger datasets than the one I was working with.

Conclusion 4: When in Doubt, Go Cheap, Go Fast.

However, just because we could have done it in Java doesn't mean we should have. Perl or Ruby or Python or Bash and the plethora of solid utilities available will now always be my first option when putting together a data input operation at this particular scale.

I think there will always be those opportunities that present themselves as vaguely defined chances to hit it big. Instead of taking lots of time up front to define the work involved at the expense of the actual opportunity, I'm going to move ahead with cheap and fast technologies that let me change path extremely quickly, because I'm sure I will need to at least once during the course of an ill defined project.

Conclusion 5: Keep the blinders off!

This entire experience was a huge reminder to me to be open minded about choosing the right tool for the job. This was an instance where I let the technology choice mandate my implementation decisions, instead of the other way around. Every time I do that, I get screwed. If, instead of putting my head down and running as fast as I could, I had initially asked questions about the duration of the project, the intention of the code, the performance constraints on data input, etc, I could have easily justified the use of Perl/unix tools/raw SQL, and saved a lot of late nights/coding angst.

Conclusion 6: It's The People, Stupid

One thing that overwhelmingly shone through even in the grimmest of moments was the quality and class of the people I was working with. They all stayed completely focused on the solution, and did not point any fingers even when to do so would have been more than understandable. Furthermore, they were able to keep their sense of humor intact. While I didn't necessarily enjoy making this big of a mistake at a new job, the level of teamwork, professionalism, and respect from my new co-workers was complete confirmation of my reasons for jumping ship from my old company.

Using Zookeeper to Implement a Redundant Persistence Service

2009-10-15T21:59:00.000-07:00

Preface

I've previously detailed how we use Zookeeper to generate unique sequenceIDs for items created by a pool of worker processes. Zookeeper is the ideal solution for this problem because of its strict order/version enforcing that allows completely non-blocking access to data, it's redundancy, and its easy to understand node and data paradigm.

Our current approach has multiple clients (one per worker process) requesting the latest sequence ID from a Zookeeper node. These sequence IDs are critical to the integrity of our data: they cannot be duplicated. Using Zookeeper version information, multiple clients request values and try to update those values with a specific version. This update will fail if someone else has updated the version in the meantime, so the client handles that failure, gets the latest value, increments it, and tries to update again. Again this is a non-blocking approach to updating a value and so there is no system level bottleneck.

Redundancy Required

In order to make our system truly redundant, we need to account for what happens if all zookeeper instances went offline by persisting the latest generated sequence IDs -- again we absolutely need to not duplicate IDs to maintain system integrity. When we persist sequence IDs, it is possible to restart all zookeeper instances and have them pick up where they left off. Note that we can reduce the amount of persisting needed by 'reserving' a future ID, persisting it, and only modifying it when the generated IDs actually get to that value. In other words, persist ID 100, and update that value to 200 when the system generates ID = 200. This maintains ID uniqueness across system restarts at the loss of 100 values, which is a decent tradeoff.

Persistence via Watches

The simplest implementation of a persistence service takes advantage of Zookeepers watch functionality, which lets a client register for notifications when a node goes away or its value changes. The client gets notified every time a watched value changes, and receives an Event object with the details of the change. In our case, the client is a Persistence Service, which retrieves the location of the updated data from the Event object, retrieves the data, and determines whether it needs to reserve a future block of numbers as described above. Note that Zookeeper Watches are defined to be one time triggers, so it is necessary to reset the watch if you want to keep receiving notifications about the data of a specific node, or the children of a specific node.

You watch data by registering a Watcher callback interface with Zookeeper. The Watcher interface implements the process() method, which handles the WatchedEvent parameter. The following process() method is determines when to persist the next reserved value.

public void process(WatchedEvent event) {
        Stat stat = new Stat();
        
        EventType type = event.getType();
        String node = event.getPath();
        if(type.equals(EventType.NodeDataChanged)) {
            
            try {
                
                byte inData[] = getData(sessionZooKeeper,event.getPath(),stat);
                currentSequenceID = SequenceHelper.longFromByteArray(inData);
                
                if(currentSequenceID % RESERVE_AMOUNT == 0) {
                    persistSequenceID(node,currentSequenceID+RESERVE_AMOUNT);
                }
                
            } catch (Exception e) {
                logger.error(e);
            }
            
        }
        else if(type.equals(EventType.NodeDeleted)) {
            logger.error("data node "+event.getPath()+" was deleted");
        }
        
        // every time you process a watch you need to re-register for the next one. 
        try {
            addWatch(sessionZooKeeper,this,node);
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
        
    }

In this case, I point Zookeeper to the Watcher object when I retrieve a ZooKeeper instance:

public ZooKeeper getZooKeeper() throws Exception {
        
        return new ZooKeeper(hostString,DEFAULT_TIMEOUT,watcher);
    }

Watchers, as is implied above, are bound to the lifecycle of the Zookeeper object they are passed into. Watchers can also be established when checking to see if a node exists, or when retrieving children of a node, because the client may want to be notified if a node is created or if children of a node are created.

Redundancy and Leader Election/Succession

Of course, a persistence service is useless if it is not redundant, especially if the integrity of our data requires us to persist reserved sequence IDs. We only want one process at a time persisting sequence IDs. If that process goes down, we want another process to step in immediately. In other words, we want a single leader to be selected from a group of 2 or more independent processes, and we want immediate succession if that leader were to go away.

In order to handle Leader Election and Leader Succession, the multiple persistence processes create and watch Sequential-Ephemeral nodes. Sequential-Ephemeral nodes have the following properties:

They automatically increment if there is a previous numbered node.
They go away if the ZooKeeper instance that created them goes away.

The persistence processes use these two properties when they start up:

They create a Sequential-Ephemeral node (note, this requires that they have a ZooKeeper instance open for as long as they are alive, so that the node sticks around).
They check to see if there is a lower numbered Sequential-Ephemeral node.
If not, they are the leader, and they register to watch for changes on the nodes used to track sequence IDs.
If there is a lower numbered sequential-ephemeral node, they register to watch that node. Specifically, they want to get notified if that node goes away.
They only want to watch the nearest lower numbered node. This avoids a 'swarm' that would happen if all lower nodes watched the single leader node. So succession is guaranteed to be orderly and only done by one node at a time.

public synchronized void startListening() throws Exception {
       
       // create a persistent node to store Sequential-Ephemerals under.
       if(checkExists(sessionZooKeeper, SEQUENCE_MAINTAINER_LEADER_NODE) == false) {
           createNode(sessionZooKeeper, SEQUENCE_MAINTAINER_LEADER_NODE, null,
              CreateMode.PERSISTENT);
       }
       
       // sequential nodes end in /n_

       String path = SEQUENCE_MAINTAINER_LEADER_NODE+PREFIX;
       createdNode= createNode(sessionZooKeeper, path, null,
           CreateMode.EPHEMERAL_SEQUENTIAL);
       
       // this method transforms the sequential node string to a number
       // for easy comparision
       int sequence = sequenceNum(createdNode);
       
       // only watch the sequence if we are the lowest node.
       if(doesLowerNodeExist(sequence) == false) {
           logger.debug("this node is the primary node");
           isPrimary = true;
           loadSequenceMaintainers();
       }
       else 
       {
           // this node is a backup, watch the next node for failure.
           isPrimary = false;
           
           watchedNode = getNextLowestNode(sequence);
           if(watchedNode != null) {
               logger.info("this node is not primary, it is watching "+watchedNode);
               boolean added = super.addWatch(sessionZooKeeper,this,watchedNode);
               if(added == false) {
                   throw new SequenceIDDoesntExist(watchedNode);
               }
           }
           else {
               throw new SequenceIDDoesntExist(watchedNode);
           }
       }
       
       isListening = true;
    }

In the code above, I've abstracted a lot of the details of interacting with Zookeeper under methods. Hopefully the method names make it clear what I'm doing. There is enough specific documentation about the ZooKeeper API, I'm more interested in showing the logic behind electing a leader.

The code above shows that only one process will be the leader. The other processes will be watching the next lowest node, waiting for it to fail. This watching, of course, is done via the Watcher::process() method:

public void process(WatchedEvent event) {
        
        EventType type = event.getType();
        String path = event.getPath();
        
        if((path != null && path.equals(watchedNode)) && (type != null &&
           type.equals(EventType.NodeDeleted))) {
            
            try {
                int watchSequence = this.sequenceNum(watchedNode);
                if(this.doesLowerNodeExist(watchSequence) == false) {
                    logger.debug("watcher of "+watchedNode+" is now primary");
                    isPrimary = true;
                    watchedNode = null;
                    // now you are the leader! the previous leader has gone away.
                    // note you are no longer watching anyone, so no need to
                    // re-register the watch.
                    loadSequenceMaintainers();
                }
                else {
                    logger.debug("watcher of "+watchedNode+" is not yet primary");
                    // there is lower node that is not the leader. 
                    // so watch it instead. 
                    watchedNode = getNextLowestNode(watchSequence);
                    boolean success = addWatch(sessionZooKeeper, this, watchedNode);
                    if(success == false) {
                        throw new SequenceIDDoesntExist(watchedNode));
                    }
                }
            } catch (Exception e) {
                // fail fast for now
                logger.error(e.getMessage());
                throw new RuntimeException(e);
            }
        }
         ......
    }

Note that if an intermediate process fails, the 'watching' process then gets the next available lowest node to watch and watches it. If all intermediate processes fail, the watching process becomes the leader.

Dynamic Node Detection

Once these long lived services are up and running, I don't want to have to restart them if I am adding another sequence node to be tracked. We do this all of the time, because we are running multiple instances of the data pipeline and tracking sequences across all of them.

This again can be handled by setting a watch, this time on the children of a top level node, and restricting sequence node creation to directly underneath that node. In other words, have a zookeeper node called /all-sequences and stick sequence-1...sequence-N underneath it. We set the watch on the node when we check to see if it has children:

children = zooKeeper.getChildren(sequenceParentNode, true);

This registers the class that created the zooKeeper instance as the Watcher. In the process() handler, we detect whether children have been deleted or added. Unfortunately, we can only detect if children underneath a node have changed, so it is up to us to determine which ones have been deleted and which ones have been added:

public void process(WatchedEvent event) {
        
        EventType type = event.getType();
        String path = event.getPath();
        ....
        if(path != null && path.startsWith(sequenceParentNode) && (type != null)) {
            // getting this notification implies that you are a primary b/c that 
            // is the only way to register for it. 
            try {               
                if(type.equals(EventType.NodeChildrenChanged)) {
                    // figure out missing and new nodes (expensive, but this 
                    // only happens when a node is manually added)
                    List newSequences = new ArrayList();
                    List missing = new ArrayList();
                    List children = this.getChildren(sessionZooKeeper,
                        sequenceParentNode,false);
                    for(String child : children) {
                        if(this.sequenceMaintainers.containsKey(child) == false) {
                            newSequences.add(child);
                        }
                    }
                    
                    for(String currentSequence : sequenceMaintainers.keySet()) {
                        if(children.contains(currentSequence) == false) {
                            missing.add(currentSequence);
                        }
                    }
                    
                    // add new sequences to watch list
                    for(String child : newSequences) {
                        String sequencePath = sequenceParentNode+"/"+child;
                        sequenceMaintainers.put(child,
                           new SequenceMaintainer(s3Accessor,
                             hosts,sequencePath,true));
                    }
                    
                    for(String child : missing) {
                        sequenceMaintainers.remove(child);
                    }
                    
                }
                
                boolean success = addWatch(sessionZooKeeper, this, 
                   sequenceParentNode);
                if(success == false) {
                    throw new SequenceIDDoesntExist(sequenceParentNode));
                }
            } catch (Exception e) {
                // fail fast for now
                e.printStackTrace();
                throw new RuntimeException(e);
            }
        }

Conclusion

I'm pretty amazed how much I was able to leverage two main points of functionality -- nodes and watches -- to build a persistent sequence watching service. Once again it seems that picking the right primitives is what makes ZooKeeper so practical for distributed synchronization.

Quick Webserver setup with Jersey and Jetty

2009-10-01T20:57:00.000-07:00

No More Hand Rolling

In our data pipeline, we have different components that we communicate with via web services. In the beginning, there were only three commands needed: pause, restart, and reload. So I wrote a quick Servlet, loaded up embedded Jetty, and called it good. The Servlet contained some method handling code to parse path strings:

String path = request.getPathInfo();
        
        if(path.equals(ROOT)) {
            response.setContentType("text/html");
            response.setStatus(HttpServletResponse.SC_OK);
            response.getWriter().println("name");
            response.getWriter().println("getInfo()");
        }
        else if(path.equals(STATS)) {
            response.setContentType("text/json");
            response.setStatus(HttpServletResponse.SC_OK);
            
            JSONObject responseObject = new JSONObject();
            responseObject.put("service", name);
            responseObject.put("status",status.toString().toLowerCase());
            responseObject.put("statistics", jobStatsAnalyzer.toJSON());
            response.getWriter().println(responseObject);
        }
        else if(path.startsWith(SERVICE_PREFIX)) {
            response.setContentType("text/html");
            
            responseCode =  
              processServiceRequest(path.substring(SERVICE_PREFIX.length(),
                path.length()),response);
            response.setStatus(responseCode);
            
        }
        else {
           ......
        }

And the processServiceRequest call is equally complex because it has to parse the next section of the path. Still, because there were only three methods and it took little time to code up, I felt fine about hand rolling the Servlet, even through there was a lot of boilerplate path parsing code.

One of our components now needs (much) more path handling. It is a validation component -- it collects transformation exceptions thrown by the view processing components. Those exceptions are dumped to an SQS queue, picked up by the validator, and dumped into a Lucene index that allows us to query for exceptions by various exception metadata. The validator needs to expose a more complex Rest Interface that allows a data curator to find exception (resources) by that various metadata (sub resources), i.e. by exception type. They can then fix the the root cause of the exceptions, and validate that the exceptions go away via a set of scripts that query the validator web service.

One option to extend the current web service functionality would be to subclass the custom Servlet, but that's a lot more boilerplate code and I know that we are probably going to need to extend another component in another way, which would mean more code. More code to debug, more code to maintain, more code to understand.

Jersey aka JAX-RS aka JSR 311 allows you to compose restful services using annotations. It is an alternative to hand rolling servlets that lets you declaratively bind REST methods (GET/PUT/POST/etc), paths, and handler functions. It handles serializing data from POJOs to XML/JSON and vice versa. I had been wanting to check it out for some time now, but simply didn't have a concrete need to do so.

Jersey And Jetty

I decided to stick with Jetty as my servlet container because launching an embedded instance was so brain dead. But I decided to use the Jersey servlet and see how hard it would be to re-implement my hand rolled servlet. The way to bind the Jersey Servlet to Jetty uses Jetty's ServletHolder class to instantiate the Jersey servlet and initialize it's annotation location as well as the location of the resources it is going to use to handle web requests. The code below shows how the Jetty ServletHolder is initalized with the Jersey ServletContainer (which actually implements the standard Servlet interface) and then bound to a context that allows the ServletContainer to handle all requests to the server.

sh = new ServletHolder(ServletContainer.class);
        
sh.setInitParameter("com.sun.jersey.config.property.resourceConfigClass", RESOURCE_CONFIG);
sh.setInitParameter("com.sun.jersey.config.property.packages", handlerPackageLocation);
        
server = new Server(port);
        
Context context = new Context(server, "/", Context.SESSIONS);
context.addServlet(sh, "/*");
server.start();

The com.sun.jersey.config.property.packages parameter points to the location of the Jersey annotated resources that the Jersey ServletContainer uses when handling requests. Those resources are simply POJOs (Plain Old Java Objects) marked up with Jersey annotations.

Jersey Resources

In order to parse a specific path, you create and object and use the @Path annotation. A method in the POJO is bound to that path by default. You can also parse subpaths by binding them to other methods via the @Path annotation. Here is an example:

@Path("/")
public class DefaultMasterDaemonService {

        
    private ServiceHandler serviceHandler;


    // this one handles root requests
    @GET
    @Produces("text/plain")
    public String getInformation() {
        return serviceHandler.getInfo();
    }
    
    // this one handles /stats requests
    @GET
    @Path("/stats")
    @Produces("application/json")
    public DaemonStatus getStatus() {
        return serviceHandler.getStatus();
        
    }
    .....
}

Basic Annotations

There are a couple of annotations above worth discussing in addition to the @Path annotation.
The HTTP method that is bound to the POJO method is specified via the @GET, @POST, @PUT, @DELETE, and @HEAD annotations.
The returned content Mime type is specified with the @Produces annotation. In the example above, a request to the root path returns some informational text, and a request to /stats returns JSON.

Returning JSON and XML

In order to return JSON/XML, you need to leverage JAXB annotations to make your data objects serializable to JSON/XML. Note: remember to always include a default constructor on your data objects. Otherwise you get exceptions trying to serialize those objects.

I also found that unless I did _not_ declare getters and setters, I would also get serialization errors. I had not seen this before, and therefore assume that it is something specific to Jersey Serialization.

Here is an example of a JAXB annotated object that I use to return Status:

@XmlRootElement()
public class DaemonStatus {
    // apparently methods to access these are added at serialization time??
    @XmlElement
    public String serviceName;
    @XmlElement
    public String status;
    @XmlElement
    public JobStatsData jobStatsData;
    
    // need this one!
    public DaemonStatus() {
        
    }
    
    public DaemonStatus(String serviceName,String status,JobStatsData jobStatsData) {
        this.serviceName = serviceName;
        this.status = status;
        this.jobStatsData = jobStatsData;
        
    }
}

So all I needed to do to get JSON/XML in my return type was to create a JAXB annotated object, and specify what I wanted the method to produce via the Jersey @Produces annotation. Less code = more fun!

Parameterizing Path Components

Our components have Pause/Restart/Reload functionality accessible via the http://host/services/{pause|restart|reload} path, using POST. Jersey lets me parameterize the last part of the path, which makes the syntax of the command explicit while allowing me to only code string matching for the parameterized part:

@POST
@Path("/service/{action}")
public void doAction(@PathParam("action") String action) throws Exception {
        
  if(action.equals(MasterDaemon.PAUSE)) {
    serviceHandler.pause();
  }
  else if(action.equals(MasterDaemon.RELOAD)) {
    serviceHandler.reload();
  }
  else if(action.equals(MasterDaemon.RESUME)) {
    serviceHandler.resume();
  }
  else {
    throw new Exception("No such action supported: "+action);
  }
}

I've delegated the real meat of the action to a serviceHandler component, but this kind of path handling is about as easy as it gets. Note that the action parameter is specified via the @PathParam annotation directly in the method argument list.

Conclusion

I only really scratched the surface of what Jersey can do. In my case I don't have to parse query parameters, but that is easily done by specifying a @QueryParam argument to the handler method in the same way I specified the @PathParam. From what I've been able to understand, you can only access query params as strings (but that's pretty reasonable).

I really liked how quickly I was able to toss out my hand coded servlet and trade up to the Jersey one. Other people on the team were able to wire up rich REST interfaces on several components almost immediately, which let all of us go back to focusing on real requirements.

I usually 'cast a jaundiced eye' towards anything that even has a hint of framework in it, but Jersey was super fast to learn and using it instead of hand coded servlets has already saved us a lot of time and finger strain.

Using ExecutorCompletionService to synchronize multithreaded workflow

2009-09-10T22:10:00.000-07:00

Today I ran into a problem where I needed to make sure that one multithreaded phase of processing had completely ended before starting another. Specifically, I was retrieving documents from S3 to load into a lucene index, and wanted to retry all document requests that had failed due to S3 flakiness, connectivity issues, i.e. standard distributed computing error conditions.

In other situations requiring synchronization between threads, I've used a CountDownLatch. This works really well when you know the exact number of threads that you need to synchronize. You initialize the latch with the number of threads that you are synchronizing. When they finish work they decrement the latch, when the latch count goes to 0 you continue processing.

This time was different because instead of synchronizing threads, I was trying to halt processing until all asynchronous submitted tasks had completed processing. I was queuing up several hundred thousand tasks into a thread pool, and did not know when that thread pool would be finished with the work, or how many threads would be running when the entire job completed, or even exactly how many tasks I had to run -- that number depended on the number of documents being fetched, which is always growing.

Fortunately my situation was not a unique one. I figured that the first place to look was the java concurrency library, and when I did some research, I found that ExecutorCompletionService was exactly what I needed.

ExecutorCompletionService works with a supplied Executor using Runnable/Callable tasks. I decided to use Callable tasks, as they allowed me to return and inspect a value, and throw exceptions. As those tasks complete, they are placed onto a queue that can be accessed via the poll() or take() methods. This approach simplified my life:

It allowed me to use tasks and task status to control logic flow instead of threads and thread status.
It freed me up from having to know how many threads or tasks I was working with.
It provided me with an access point to each task that I could use to analyze the results.
When the ExecutorCompletionService queue was empty, I knew it was time to then retry all failed results.

Point (1) above is the foundation for the points that follow. When I used Future and Callable to implement the work I needed to do, I was able to return the results I cared about and process them. Specifically it allowed the code that ran the original key fetch loop to not worry about tracking and storing exceptions, which made for much simpler looping logic.

ExecutorCompletionService is a great example of how picking the right primitives makes it easy to compose highly functional helper objects. In this case the primitives involved were a (thread pool) executor and a (linked blocking) queue. (side note: I don't mean to sound like such a concurrent lib fanboy, but I'm really happy I didn't have to write this code myself, and really happy that the choice of primitives that the authors used enabled creation of classes like the ExecutorCompletionService. This stuff used to take significant effort on my part, and significant bugs were usually introduced :)

Here is an example of the ExecutorCompletionService in action, the relevant bits bolded and italicized.

public void buildEntireIndex() throws Exception {

        boolean moreKeys = true;
        int submittedJobCt = 0;
        // tracking all failed keys for later retry
        Map failedKeys = new HashMap();
        
        // ErrorCapturingExecutor is subclassed from ThreadPoolExecutor. 
        //I override ThreadPoolExecutor.afterExecution() to queue and later analyze exceptions.
        LinkedBlockingQueue linkedBlockingQueue = new LinkedBlockingQueue();
        ErrorCapturingExecutor executor = new ErrorCapturingExecutor(threadCount, 
            threadCount, 0l, TimeUnit.SECONDS, linkedBlockingQueue);
        
        // CompletionService works with the supplied executor and queues up tasks as they finish.
        executorCompletionService = new ExecutorCompletionService(executor);
        
        
        String lastKey = null;
        
        while(moreKeys  == true) {    
            Set keys =  viewKeyRetriever.retrieveKeys(lastKey);
            
            if(keys.size() > 0) {
                String array[] = new String[keys.size()];
                keys.toArray(array); 
                lastKey = array[keys.size()-1];
                
                // I need to keep the number of waiting tasks bounded. 
                if(linkedBlockingQueue.size() > MAXQUEUESIZE) {
                    Thread.sleep(WAITTIME);
                }
                
                // this is where I actually submit tasks
                processKeys(keys);
                
                // I only know how many jobs I need to wait for when I've retrieved all keys.
                submittedJobCt++; 
                
            }
            else {
                moreKeys = false;
            }
            
        }

        
        // I use the ExecutorCompletionService queue to check on jobs as they complete. 
        for(int i = 0; i < submittedJobCt;i++) {
            
            Future finishedJob = executorCompletionService.take();
            
            // at this point, all I really care about is failures that I need to retry.
            BuildResults results = finishedJob.get();
            
            if(results.hasFailures()) {
                failedKeys.putAll(results.getFailedKeys());
            }
            
            indexBuilderMonitor.update(results);
        }
        
        // I can use failedKeys to retry processing on keys that have failed. 
        // Logic omitted for clarity.
        
        ...
        executor.shutdown();

    }

The ProcessKeys method is shown below: I broke it out because I needed to call it again when re-processing failed keys

private void processKeys(Set keys) {
        // the builder builds the index from the content retrieved using the passed in keys.
        final IndexBuilder builder = indexBuilderFactory.createBuilder(keys);
        
            
          executorCompletionService.submit(new Callable() {
          @Override  
          public BuildResults call() throws Exception {
              BuildResults buildResults = null; 
              try {
                  // buildResults contains all information I need to post process the task.
                  buildResults = builder.build();
              } catch (Exception e) {
                  throw e; // caught by ErrorCapturingExecutor
              }
            
              return buildResults;
          }

        
        });
        
    }

Using ThreadLocal to pass in dummy components

2009-09-09T11:54:00.000-07:00

Synchronization of data access across multiple threads is tricky. While Java's threading primitives are fairly easy to understand and use, there can be unintended performance consequences of making an object thread safe. Depending on how objects synchronize other objects, you can also end up with deadlocks that are no fun to debug. For example if object1 tries to lock object2 while object2 is trying to lock object1, you're in for a long night.

In general, anything that reduces synchronization of data across threads reduces the potential for unintended consequences. An alternative to making an object threadsafe is to make it thread-local. Thread local objects provide a separate copy of themselves for all threads. Each thread can only see it's local instance of that object, and is free to modify it at will, without needing to synchronize.

Thread-local variables used to have significant performance issues, and there have been bugs in previous to 1.6 versions. Also, it is possible to easily run out of memory with large numbers of threads using large thread-local objects. But assuming you go in with your eyes open, reducing synchronization across the threads in your application is good for performance and can significantly reduce complexity.

Another benefit of thread-local variables (as if simplification and performance gains aren't enough) is that they make it easy to swap in stub components at unit test time. Why would you do this instead of passing in the component? I ended up using thread-local variables for my components when I had to instantiate an object via the Class.forName() method, and didnt know/want to know about how to wire up dependent components. It's a trick I want to remember (so I'm writing it down :)

I implement the component as a thread-local variable is via an anonymous inner class:

ThreadLocal sequenceClientLocal = new ThreadLocal() {
        @Override
        protected SequenceClient initialValue() {
            SequenceClient sequenceClient =  null;
            try {
                sequenceClient = SequenceClientImpl.getInstance(hosts,hexIdNode);
            } catch (Exception e) {
                sequenceClient = null;
            }
            return sequenceClient;
        }
    };

In order to swap this default value out for a stub file, I add a setter to override it:

public void setSequenceClientLocal(ThreadLocal sequenceClientLocal) {
        this.sequenceClientLocal = sequenceClientLocal;
    }

At unit test time, I can stub in a dummy class by calling the setter:

public class TestCollapserAgent {

  @Before
  public void setUp() throws Exception {

    sequenceClient = new DummySequenceClientImpl(1);

    collapserAgent.setSequenceClientLocal(new ThreadLocal() {
            @Override
            protected SequenceClient initialValue() {
                return sequenceClient;
            }
        });
    ....
  }

  // unit tests follow....

}

The Ratio of Ceremony vs Essence, aka Framework Debt

2009-09-03T22:02:00.000-07:00

This is going to be the kind of post I had sworn off of: a lot of opinion, with some rambling mixed in. I apologize in advance for that, but occasionally I need to vent about things that deeply disturb me, and venting tends to be nonlinear.

I just spent the last week trying to work with a legacy system component that was implemented using the Spring framework. This component read data from a database into a Lucene index wrapped by Compass. At the time of implementation, the lead engineer was using JPA, to load database records into POJOs, which he then annotated so that they could be serialized via JAXB, which enabled Compass to read them in as Lucene Documents. Whew!

Because time was limited and the code was already in production, I decided to ignore my fundamental misgivings about frameworks and Java Acronyms, and make the minimal modifications to the existing source that would get it to take input from S3 instead of a database.

After a day of struggle, I had figured out what was going on, and was astounded by the amount of code required just to set up the relatively simple business logic. When I hit a 'schema not found' error trying to load the application.xml, I gave up, ripped out the business logic, and re-implemented the entire thing in a matter of hours. With a lot less code. I know that the original implementation of the Spring based code took a week or so to write.

The massive increase in efficiency is not because I'm a brilliant coder. I wish I was, but I've worked with brilliant coders and I'm not one of them. It's because the actual business logic was pretty minimal. The logic required to implement and maintain the Spring application required a lot of code that could only be described as Ceremonial, as opposed to Essential business logic. I first read about Ceremonial vs Essential code here, the night after I had exorcised Spring from the logic. The timing couldn't have been more appropriate.

What is Ceremonial code? It is code that has nothing to do with implementing a business requirement. In Spring, I define Ceremonial code as:

Configuration as code
Dependency Injection
(Pedantic use of) Interfaces

The three examples above are not terribly bad, in fact they come from decent intentions ("the road to hell..."). But put together they have an exponentially bad effect. They are, when added to a developer's blind belief in the goodness of all things Frameworky, the Four Horsemen of the (Framework) Apocalypse.

Configuration As Code

Separating configuration into a data file is inherently a good idea. You don't want to hardcode variables that you would then have to rebuild the application to change. I'm not sure how this basically sound idea warped into "hey, let's put EVERYTHING into configuration", but the biggest problem with this approach is that now part of the logic is in code, the other part is in a massive XML file. You need both to understand the control flow of the application, so you spend a lot of time toggling back and forth, saying "What class is being called? Oh, let me check in xml configuration. Oh, that's the class. Great. What was I doing?" Maybe some people see this kind of rapid mental stack management as interesting and novel brain training. I see it as wasting time, time that I could be spending either coding or testing a feature that someone is paying me to implement.

Dependency Injection

This too, starts off as a great idea. One of the big issues people (OK, I'm talking about myself, but using the word 'people' to get some more legitimacy) had with EJB 2.0 code was that it was really hard to test. You had to have the whole stack up and running, and validating the integrity of different components in the stack was so hard we just didnt do it.

Dependency Injection/Inversion of Control allows you to parameterize the components of an object, making that object really easy to test. Just expose components with getters and setters, and you can dummy them up and test that object in true isolation! Again, there is still nothing really flawed at this point.

The major flaw in Dependency Injection comes at implementation. Objects need all of their components in a known, initialized state, in order to function effectively. Dependency Injection as implemented in Spring is usually done in the configuration file. Objects that are created in the configuration file have all of their components set in their configuration.

It is very easy to miss setting a component in the configuration file. This means that the object will initialize in a bad state that becomes apparent when you try to use it. People use constructors because they can specify components as parameters to the constructor, which is an explicit way of saying "This component needs components X, Y, and Z to run".

Using a constructor provides a foolproof way to successfully initialize an object without having to test for initialization success. If the constructor returns, you're good. If not, you know that the object is not usable.

In order to be able to be configurable via Spring, objects must (a) have a default (no argument) public constructor and expose all of their required components via setters. There is no way to enforce that setup has been correct, so the developer has to spend time looking at the getters and setters of the object to determine what components they need to supply at configuration time. When I compare that effort to the effort of looking at the constructor parameters, it feels very inefficient.

Pedantic Use of Interfaces

The goal of the Java Interface is (a) separate functionality from initialization, and (b) provide a contract that a caller and callee can communicate across. This makes sense in the following two cases:

You have a complex object and you only want to expose a part of it to a caller. For example you have a parent class and you want to expose a callback interface to the child class.
You have multiple implementations of the same functionality and you don't want the caller to care about which object they are calling.

What I see all over Java-Land, and especially in Spring, is interfaces being used because someone got pedantic about separating functionality from initialization. I fail to see the use of an interface when used to abstract the implementation of all of the methods of a single class. You're writing two structures when one could do the job just fine. Actually, you end up writing three structures: the interface, the implementation, and a factory object, which is more ceremonial code. Even if you need the interface, you could still have the implementation object return an instance of itself cast to the interface via a static initialization method:

public class S3AccessorImpl implements S3Accessor {

    private static final int DEFAULT_SET_SIZE = 1000;
    private S3Service service;
    private Logger logger;

    public static S3Accessor getInstance(AWSCredentials creds) throws S3ServiceException {
        return new S3AccessorImpl(creds);
    }
    
    
    protected S3AccessorImpl(AWSCredentials creds) throws S3ServiceException {
        logger = Logger.getLogger(this.getClass());
        service = new RestS3Service(creds);

    }
    ...
}

In spite of my comments above, I am a fan of using interfaces as the boundaries between an components because it facilitates easier unit testing. But I'm not entirely sold on abstracting the creation of an object to a Factory that returns the interface that object implements -- not when the above method (a) hides creation from the caller and (b) doesn't require an extra class with a single 'createFoo' method.

Also, I don't understand always writing interfaces first, then implementation classes second. I tend to implement classes until I have a real need for an interface, i.e. during unit testing when I am going to submit a 'dummy' component in place of a real one.

Conclusion

My recent experience with Spring has reminded me of the existence of 'Framework Debt'. Framework Debt is the Technical Debt required to implement a solution with a given Framework. In short it is determined by the ratio of time spent writing and maintaining ceremonial code vs the amount of time spent writing and maintaining essential business code. The problem with most frameworks, Spring included, is that they do not distinguish between ceremonial and essential code, because to them, it's _all_ essential code. And, to work in that particular framework, ceremonial code is absolutely essential, and having to maintain and understand a bunch of logic that has nothing to do with application functionality seems inherently wrong to me.

I actually do like some frameworks I've run into. Rails is great because of it's 'convention over configuration', but that is another kind of technical debt. Fortunately it is pretty low in Rails, and as a result applications can be rapidly developed in Rails without losing maintainability. But even Rails feels too heavy for me at times. I do write apps that don't need the overhead of MVC. For these apps, Sinatra allows me to quickly get path routing out of the way and concentrate on the underlying code.

Zookeeper and Concurrency

2009-08-27T21:08:00.000-07:00

Recently I ran into a problem that seemed to require more of a solution than I was willing to implement. We are currently migrating an application from using a single worker, single database to having multiple workers running in parallel using s3 as the primary means of storage. (Side note: this migration is only possible because the application doesn't actually require any of the unplanned, interactive queries that only a database is good at. The choice of a database as the persistence mechanism for this application was not a popular one, and only grew less popular as more time was spent putting out database related fires than implementing new features). One of the legacy requirements of the system, which supports an in production website, was that the IDs for all new items had to be distinct integers.

Without this legacy requirement I would have put in some kind of GUID scheme and called it an (early) day. The existence of previous items with IDs that other services relied upon made a GUID generation scheme not possible. However the requirement of distinct integers requires coordination between the agents, who would need to make sure they are not creating the same ID for different objects.

My initial thought was to implement a simple service that provided an integer that it would auto increment with every request. The big problem with this approach is that the service would be a massive, non redundant bottleneck unless it too was replicated, and then it would be faced with the same problem that the original workers faced wrt keeping integers synchronized across different processes.

So I had a bad feeling about starting down that path, and was putting it off, when a colleague at work suggested that I check out Zookeeper. Zookeeper, was created by Yahoo research specifically to solve the kind of synchronization problems that I was having, in a highly performant, fault tolerant way. In other words, this was the service that I was trying not to write :)

Zookeeper at 10000 feet consists of multiple services that maintain a hierarchical namespace consisting of nodes that can have child nodes. Each node can have associated data, limited to under 1MB, meant to be used for coordination/synchronization.

Zookeeper is in the words of its creators, "needed to be general enough to address our coordination needs and simple enough to implement a correct high performance service. We found that we were able to achieve {this} by trading strong synchronization for strong ordering guarantees and a wait-free interface."

What that means is that node access is 'no wait', meaning when you ask for data you get it, but you do not have an exclusive lock on the data. This is quite different than the mutex based locking model that I'm used to, and at first I didn't see how I could use this to guarantee unique IDs to multiple agents creating multiple items without getting an exclusive lock on the data and making a modification.

What I didn't get (until another colleague walked me through some of his code) is that any and all changes to the data are taken, and versioned. When I request data, I get back an object corresponding to the version of that data. When I submit data, I can specify that the submit will only succeed if the data (and therefore it's version) hasn't been updated from the version that I have. So if I get node data that is an integer ID, increment it, and try to update the data it back, two things can happen (excluding connection loss, which must also be dealt with):

I can return successfully, meaning that my change was accepted because no one else had made changes since I had retrieved the data.
I can get a Bad Version exception, which means I need to get the data again, and try to re-increment the new value.

The code below shows the method that requests, recieves, and attempts to increment the data:


public String getNextId() throws Exception {
     ZooKeeper zk = null;
     String hexId = null;
  
     boolean keepGoing = true;
  
     while(keepGoing  == true) {
         try {
             Stat nodeStat = new Stat();
             Stat setDataStat  = null;
             zk = getZooKeeper();
             byte data[] = getDataWithRetries(zk,
                  sequenceName, 
                  nodeStat);
             ByteBuffer buf = ByteBuffer.wrap(data);
          
             long value = buf.getLong();
          
             value++;
          
             buf.rewind();
          
             buf.putLong(value);
          
          
             try {
                 setDataStat = setDataWithRetries(
                        zk,sequenceName,
                        buf.array(),nodeStat);
                 hexId = Long.toHexString(value);
                 break;
             }
             catch(KeeperException e) {
                 if(e.code().equals(Code.BADVERSION)) {
                     nodeStat = setDataStat;
                 }
                  
             }
         } finally {
          
             // always need to close out the session!
             zk.close();
         }
      
     }
  
  
     return hexId;
 }

I've wrapped calls to zookeeper with a getZookeeper() method, and pass the retrieved Zookeeper instance into two methods: getDataWithRetries(), and setDataWithRetries(). Both methods try to recover from connection losses as best they can.

The getDataWithRetries method takes a Zookeeper instance, the path to the node being accessed, and a Stat structure that will contain retrieved data version information. It returns the retrieved data in a byte array. Note how in this method I'm only going to recover from connection losses, because this is a read operation.


   protected byte[] getDataWithRetries(
          ZooKeeper zooKeeper,
          String path,
          Stat nodeStat) throws Exception{
    
    
      byte data[] = null;
    
      int i = 0;
    
      while(i < RETRY_COUNT)   {     
          try {                 
                  i++;
                data = zooKeeper.getData(path, 
                      false, 
                      nodeStat);
                break;
           } 
           catch(KeeperException e) {
                 if(e.code().equals(Code.CONNECTIONLOSS)) 
                 {
                     continue;
                 }
                 else if(e.code().equals(Code.NODEEXISTS)) 
                 {
                     break;
                 }
                 else {
                     throw e;
                 }
             }
         }
        if(i >= RETRY_COUNT) {
          throw new KeeperException.ConnectionLossException();
      }
    
      return data;
    
  }

Once I have the data, I increment it. Note that Zookeeper data is always kept as a byte array, so I convert it in order to increment it:


ByteBuffer buf = ByteBuffer.wrap(data);
            
long value = buf.getLong();
            
value++;
            
buf.rewind();
            
buf.putLong(value);

and then try to resubmit it back to Zookeeper. This is where things get interesting. If someone has modified the data before I could get back to it, I need to get the new value of the data and try again. In the setDataWithRetries() method below, I only handle connection exceptions, and blow out if there is a BADVERSION exception:



protected Stat setDataWithRetries(
            ZooKeeper zooKeeper,
            String path, 
            byte data[],
            Stat stat) throws Exception{
    
      int i = 0;
      Stat statFromSet = null;
    
      while(i < RETRY_COUNT)
            try {
                 i++;
                 statFromSet = zooKeeper.setData(path, 
                      data,
                      stat.getVersion());
                 break;
             }
             catch(KeeperException e) {
                 if(e.code().equals(Code.CONNECTIONLOSS)) 
                 {
                     continue;
                 }
                 else if(e.code().equals(Code.BADVERSION)) 
                 {
                     // differentiating for debug purposes
                     throw e;
                 }
                 else {
                     throw e;
                 }
             }
         }

      if(i > RETRY_COUNT) {
          throw new KeeperException.ConnectionLossException();
      }
    
      return statFromSet;
    
  }

The calling code of setDataWithRetries() handles the BADVERSION exception by getting the data again, and retrying the submit:


 try {
    setDataStat = setDataWithRetries(zk,
        sequenceName,
        buf.array(),
        nodeStat);
    hexId = Long.toHexString(value);
    break;
 }
 catch(KeeperException e) {
     if(!e.code().equals(Code.BADVERSION)) 
     {
           throw e;
     }
 }

So each agent tries to get an ID until they succeed, at which point they know they've got a unique one.

The thing I really like about the strong versioning and ordering approach, now that I understand it, is that it acknowledges concurrency and makes it easy to deal with. Locking, on the other hand, seems like an attempt to abstract away the concurrency by enforcing serialization, which works OK when you are managing machine or process local resources, but can have huge performance impacts when you are trying to synchronize access across multiple machines.

The next thing I'm considering using Zookeeper for is configuration changes. Right now I push configuration changes out to my worker nodes by hand and force a restart via their web service interface. I would like them to be able to reload themselves automatically when state changes. This is a step up from the simple code detailed in this post, it means I need to use Zookeepers notification capabilities to alert listening processes when the configuration changes.

Using java.util.concurrent.CountDownLatch to synchronize startup/shutdown

2009-08-11T10:54:00.001-07:00

I've been a big fan of the Java concurrency library since I stumbled upon it a while back. Before it came along, I was relegated to writing my own thread pools, schedulers, etc. Which meant, of course, that I was relegated to introducing lots of subtle and deviant bugs into code that had nothing to do with the actual product I was delivering.

The java.util.concurrent library freed me up to go ahead and focus on what I was really trying to deliver instead of re-inventing a hard to write wheel. Plus they're way smarter than me about concurrency. I highly recommend reading Java Concurrency In Practice, even if you dont code in Java, because the concurrency issues they discuss are universal, even if the solutions are in Java.

In the latest installment of 'how java.util.concurrent made me a happier, more productive developer', I was implementing a web service layer to control the run state of a set of worker threads. These workers needed to be started/stopped/paused/resumed/{insert favorite action here}. I don't want to continue processing on the calling thread (the web service start/stop/etc methods) until I am sure that the action requested by the caller has completed across all worker threads, which are running asynchronously.

My first thought was to write a pair of interfaces that allowed me to synchronize when an action was requested and when it was completed. In other words:


public interface Worker {
public void start(Master master);
public void stop(Master master);
public void pause(Master master);
..
};

public interface Master {
public void started(Worker worker);
public void stopped(Worker worker);
public void paused(Worker worker);
}

The problem with these interfaces and this design is that every time I needed to add an action to Worker, I needed to add a corresponding 'completed' message to Master. Also, the implementation of Master would need to track each worker against a worker pool, and scan that pool to see if an action was completed. Clearly way too much work to write, let alone understand 3 months later. Also, I knew that this was a pretty common problem, probably solved by the concurrency lib. So I cracked open the book....

java.util.concurrent.CountDownLatch is, in the words of the guys who wrote the library, "a synchronizer that can delay the progress of threads until it reaches it's terminal state". Hmm. Using the synchronizer frees me up from having to track the specific kind of state of N specific workers:

public interface Worker
 public enum RunState {
  STOPPING,
  STOPPED,
  STARTING,
  STARTED,
  PAUSING,
  PAUSED,
  RESTARTING
 };
 public boolean start(TaskCompleted listener) throws Exception
 public void stop(TaskCompleted listener) throws Exception;
 public void pause(TaskCompleted listener) throws Exception;
 public void restart(TaskCompleted listener) throws Exception;
 public void reload(TaskCompleted listener) throws Exception;
 public RunState getState() throws Exception;
}

public interface TaskCompleted {
public void completed();
}

In the code above,

(1) I no longer care about which action is completed, or which worker completed the action, which means

(2) I no longer am keeping state for X workers in order to return from the call.

(3) The TaskCompleted interface can be implemented as an anonymous class in the response.

The CountDownLatch is pretty simple: it blocks until it's internal count reaches zero:

private void pauseWorkers() throws InterruptedException {
   final CountDownLatch waitForPause = new CountDownLatch(workers.size());
   for(WorkerBase worker : workers) {
       worker.pause(new TaskCompleted() {
           public void completed() {
               waitForPause.countDown();
           }
       });
   }

   waitForPause.await();

   // and now we're paused.

}

In the code above, the CountDownLatch is initialized to the number of workers I have. I iterate through the list of workers and perform the 'pause' action on them. Then I wait for the latch to get counted down to 0 before proceeding. I am keeping no state on the workers, I only care when they've completed their requested action. I suppose that for I could replace the anonymous implementation with an actual (dirt simple) implememtation that takes the counter and decrements it.

Upgrading to Eclipse Galileo 3.5 from Ganymede 3.4 on Mac OSX

2009-07-16T16:11:00.000-07:00

These are my notes on what I had to do to upgrade to Eclipse Galileo.

Why?

My main motivation was have 1.6 be my default JDK. With Ganymede I had to set my default JAVA_HOME env var to point to 1.5, and point 1.6 dependent apps -- like my command line mvn builds -- to the (non default) 1.6 JDK. That's exactly the kind of thing I forget 5 minutes after I do it.

What?

Just in case I need to do this again: as far as I could tell, upgrading to a major version of Eclipse currently requires a full, clean install. Which means no associated plugins. So I'm writing down the plugins I need to install, where to get them, etc.

Base Install

I installed the 32 bit Cocoa version of Galileo from http://www.eclipse.org/downloads/download.php?file=/technology/epp/downloads/release/galileo/R/eclipse-jee-galileo-macosx-cocoa.tar.gz

The diff between Cocoa and Carbon and 32 vs 64 bit is explained in detail here.

The tar file unloads to an eclipse directory: make sure you move your old version out of this dir if that's where you have it!

Plugin Installs

these were in order (Maven required Subversion)

(1) Subversion Plugin: I followed these instructions to install subclipse.

(2) Maven Plugin: I installed from http://m2eclipse.sonatype.org/update.

Follow Up

(1) I needed to change my JAVA_HOME environment var to point to my 1.6 install (I use soylatte).

(2) I needed to upgrade my subversion client to > 1.4 otherwise I saw an 'unsble to launch default SVN client' when trying to browse my SVN repo. I downloaded the latest svn client, restarted eclipse, and all was well.

Running Zookeeper on the Mac with Soylatte and Eclipse 3.4

2009-06-30T15:20:00.000-07:00

I've been using Zookeeper to store a sequence number that a large number of processes can access and increment in a coordinated manner.

Zookeeper has a nice, simple interface, and exposes a set of primitives that easily allow me to implement guaranteed synchronized access to my magic sequence number. I'll post more later on the specific solution, but right now I want to detail some of the issues I've run into and the workarounds I've put in place.

I run on a Mac (OSX/Leopard), use Eclipse 3.4 for my Java development, and use soylatte for my JDK. I think a lot of other people run with this setup. I'm using Zookeeper 3.1.3.

My initial setup steps:

I downloaded Zookeeper, untarred it, and installed in /usr/local.
I created a symlink from zookeeper-3.1.1 to zookeeper
From that dir I ran sudo ./bin/zkServer start.

I immediately ran into a strange issue: I could connect to the zookeeper instance:

ZooKeeper zk = new ZooKeeper("127.0.0.1:2181",ZookeeperBase.DEFAULT_TIMEOUT,this);

but could not create a node on it:

zk.create("/HELLO", foo, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);

I kept getting timeouts. I've written my code to be 'timeout proof' because connection loss errors are to be expected under load in distributed environments, but I do kick out after 5 retries. Besides, I wouldn't expect to get the ConnectionLoss error when I am connecting to a localhost instance.

It turns out that the there have been soylatte nio issues with Zookeeper. I talked to Satish (he's on the email thread in the link, and we both work at Evri), and he said he had success using the latest version of 1.6 that mac 'officially' supports.

I switched to the latest Apple supported java 1.6 version: when I pointed my java binaries at 1.6, Zookeeper worked great, but Eclipse couldn't restart -- some more online research showed that this was another known issue.

So in the end: I

(1) created a java16 symlink to/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/bin/java

(2) used that symlink in zkServer.sh

(3) kept my $JAVA_HOME pointing to 1.5 by symlinking/System/Library/Frameworks/JavaVM.framework/Versions/1.5 to/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK

Seattle Rock n Roll Marathon Race Report

2009-06-27T13:27:00.000-07:00

Well, the big event has come and gone. And I'm happy to have crossed the finish line under my own power! It was a great first marathon experience. To summarize, after not being able to run very much in the last six weeks due to 'life happening' and some late breaking bronchitis, I decided to run the marathon with no particular time goal in mind. However, if I could come close to my original goal of sub 4 hours, that would be gravy :)

I ended up getting 4:10 (according to my watch, which I stopped during a short but necessary bathroom break). The race was going according to sub 4 pace and plan until mile 21, when massive calf cramps in both legs dramatically slowed me down. Cramping of any kind is frustrating, because you have to basically shut it down completely. I wasn't even breathing hard, but I simply could not move any faster. I was able to fight the cramps off for a while, but by mile 25 they were pretty constant, and spreading from my calves to my groin and quads. In the last half mile I was cramping with every step, but I was basically in a tunnel of people at that point, and they cheered me on through the cramps. While it wasn't really 'fun' at the time, I'll never forget struggling down the finishing stretch and the support the crowd gave me that got me through it.

The highlight of my race was seeing my friends Lori and Anthony at the 18.5 mile mark (when I was still feeling pretty good), with their 'Run Arun' sign. Lori was training for the marathon when she injured her knee. In her place I would have been pouting and eating bon-bons on the couch, but she and Anthony came out and cheered us all on -- making the killer sign, giving me pretzels, gatorade and hugs (which must have been pretty gross since I was drenched in sweat). Thanks guys, seeing you both after the long grind uphill was exactly what I needed!

For a first marathon, the Seattle Rock n Roll was perfect. The weather was great, the bands were great (especially for a guy that doesn't train with an iPod), the water stations were perfectly placed, and the course was challenging, with the biggest hill coming on at mile 15-18, running up the false flat of highway 99. For me, the toughest part was running past the turnoff towards the finish on mile 23, and knowing I still had 3 long, cramp filled miles to go. That was mental.

My cramping was probably due to the lack of recent training. I had trained for a specific pace, and kept that pace for the first 21 miles. But the time off caught up with me in the end, and my target pace was probably too fast for my current level of fitness. I guess I'll have to claim a 'moral' victory. And there will always be next year!

Streaming Data with a Worker/Agent based approach

2009-06-23T20:55:00.000-07:00

Where I was going....

In my last post I described how at work, we were investigating using Hadoop in a non batch setting. I mentioned that despite not using Hadoop's ability to collate keyed data from large data sets, we were still investigating Hadoop because of the built in robustness of the system:

Nodes are checked via 'heartbeat'
task status is centrally tracked
failed tasks are retried.
Work is pulled from the central JobTracker by TaskTrackers.

The basic pain points of maintaining highly available and robust functionality across a cluster of machines is taken care of, and was the primary motivator for us to try and stream data across a batch driven system.

However as we moved into implementation it became fairly obvious that we were pounding a square peg into a round hole. A lot has been written about how Hadoop and HDFS doesn't work particularly well with small files -- the recommended solutions usually involve concatenating those files into something bigger to reduce the number of seeks per map job. While these problems were understandable in a system optimized to process huge amounts of data in batch, waiting to batch up large files wasn't an option given the low latency requirement of our end users.

Especially disconcerting was the amount of work (and code) spent bundling queued work items into small files, and submitting those files as individual jobs. The standard worker model --having multiple processes with multiple threads per process running on multiple machines access SQS and process the data -- seemed so much simpler than creating artificial batches.

A Swift Change of Direction

The rewrite took a matter of hours, dropped out a lot of code, and was a minor change to the overall architecture, which uses SQS to transition between workflow states, and S3 to persist the results of data transformations. The move away from Hadoop was limited to intermediate worker processes -- we still use Hadoop to get the data into the system, because we are collating data across a set of keys when importing data. The latency went from somewhat indeterminate across mini batches to being the average time to process per thread. And the workers were easily subclassed from the Callable class -- developers could implement new workers by overriding a single method that took a string as input. When latency of the system went up, simply adding more machines running more processes would take care of the problem.

Distributed Availability and Retry Logic

Of course, that simplicity came with a price tag -- we lost the distributed bookkeeping that Hadoop provided. Specifically, we would have to implement:

thread and process failure detection
machine failure detection
retry logic

All of which is non trivial to implement. However, our need to stream instead of batch data meant that we would have ended up having to do the retry logic differently than Hadoop anyways. We need to catch and retry data failures at a work item level, not at an arbitrarily determined file split level.

Our retry logic is pretty simple, and uses S3 to persist workflow state per work item. We traverse a list of items in the queue, determine which ones have 'stalled out', and submit them to the appropriate queue as part of a retry. At the same time we clean up work items that have been fully processed, and get average processing time per workflow process. These three things are best done in an asynchronous manner, as -- you guessed it -- Hadoop jobs. They need to take advantage of Hadoop's collation functionality.

Our thread failure logic is also pretty simple. Because I'm starting up Callable tasks and making them run until I shut them down, I can check to see if any of them have finished prematurely by calling isDone() on the Futures returned when submitting them to the ExecutorService.

Process failure can be monitored (and logged) by a watchdog program. Repeated process failure in this case is symptomatic of an uncaught exception being thrown in one of the process threads.

Machine failure is also easily monitorable. I need to expose a simple service on each machine to detect process and thread failures, and if that process is not reachable, I can assume that the machine is offline.

These may be fairly limited and crude methods of getting a highly available system in place, but they feel like the right primitives to implement because while I don't know why the system is going to fail, each of these methods gives me a way to know how it is failing.

The Conclusion (so far)

The morals of the story at this point are:

frameworks can be extremely powerful if used for their strengths, and extremely limiting if used for their secondary benefits. When it feels like I'm pounding a square peg into a round hole, I probably am. I think this is called 'design smell', and now that I know what it smells like, I'll start backing up a lot sooner in an effort to find the right tool for the job.
It is always a good sign when a refactoring drops out lots of code.
Having to implement the availability and robustness of the system we are writing has actually made it easier to understand. Even though we are implementing functionality that we once got for free, at least we understand the limitations of the availability and robustness solutions we put in place.

Streaming Data with Hadoop and SQS -- a work in progress

2009-06-08T21:40:00.001-07:00

Hadoop is by design a batch oriented system. Take a bunch of data, run a series of maps and reduces with it across X machines, and come back when it's done. A Hadoop cluster has high throughput and high latency. In other words, it takes a while, but a lot of stuff gets done.

At work, I'm leading a team that is implementing a data processing pipeline. The primary use case that we need to address involves quickly handling changes in the entity data we care about -- people, places, and things that act or act upon other people, places and things. Specifically, this means:

detect that there has been a change
get the changed data into the system
apply a series of transforms to it for downline rendering and searching systems

This use case requires a streaming solution. Each piece of data needs to be transformed and pushed into production relatively quickly. Note that there is an acceptable latency. If, for instance, if Famous Actor X dies, our system needs to detect it and update the data within the hour. However, detecting/updating data within a day would be too slow to be useful to someone wanting to know what was up with Famous Actor X.

At first glance, Hadoop is not a good fit for this solution. It takes file based inputs and produces file based outputs, so any individual piece of data moving though the system is limited by the speed at which an entire input set can be run through a cluster.

However, Hadoop has the following features that make it ideal for distributing work.

The MapReduce abstraction is a very powerful one that can be applied to many different kinds of data transformation problems.
The primary Mapper And Reducer interfaces are simple enough to allow many different developers to ramp up on the system in minimal time.
HDFS allows the developer to not worry about where they put job data.
Cluster setup and maintenance, thanks to companies like Cloudera (who fixed the issues I was seeing with S3, thanks!), is taken care of.
The logic around distributing the work is completely partitioned away from the business logic that actually comprises the work.
Jobs that fail are re-attempted.
I'm sure there's more..

As our system scales, we can only assume that the number of concurrent inputs and therefore the system load will grow. If we were to take the lowest initial effort route and write our own multithreaded scheduler, we would have a much more straightforward solution.

However, as the workload grows to swamp a single machine, we would eventually end up having to deal with the headaches of distributed computing -- protocol, redundancy, failover/retry, synchronization, etc.

In fact, I was fully intending to write a quick scheduler app to stream data and throw it away when we reached a scale that required distribution, at which point I was going to use Hadoop. However I soon realized that I would be solving problems -- scheduling, data access, retry logic -- and those were just the initial non distributed issues -- that were already addressed by Hadoop.

Still, there is the high latency/inherent batchiness of Hadoop. In order to work around the high latency problem, we're trying to enable streaming between MapReduces via SQS. The input of the entity data into the system and the various transforms of that data can be treated as a series of MapReduce clusters, where the transformation is done during the map, and any necessary collation is done during the reduce. The Reduce phase of each MapReduce can send a notification for the piece of data it has to the next MapReduce cluster.

Of course, it is really inefficient to run MapReduce over a single piece of data at a time. So the listener on each SQS queue buckets incoming messages using a max messages/max time threshold to aggregate data. When a threshold is reached, the system then writes the collected messages to a file that it then starts a MapReduce job on. This is mini batching, and as long as it delivers the data within the specified system latency, it's all good.

What we are doing is definitely a work in progress. The reason why is that there are several 'dials' in this process that need to be tuned and tweaked based on the size of the input data.

number of messages/amount of time to wait. Of course, the larger the input files, the more 'efficient' the system. On the other hand, making those files larger may imply an increase in latency.
number of concurrent jobs -- there is a sweet spot here -- it (again) doesn't make sense to launch 100 concurrent jobs. We will need to pay attention to how long a job takes before deciding to adjust the number of concurrent jobs up or down.
number of transformations -- the bucketing required implies that every transform has a built in latency, that factors into the overall latency.
cluster size -- it makes no sense to run 20 nodes where 2 would suffice, but there will be times when 20 is necessary

Some of the dials may be adjustable on the fly -- sending messages to a cluster via an 'admin' queue allows us to change message size/max time/concurrent job numbers dyamically. Other dials may require a stop-reconfig-start of a cluster. One benefit of using SQS is that no messages are lost while we tweak these 'hard stop' dials.

We're not doing a classical MapReduce where we transform data and then collate it under a distinct key. We're doing simple transformations with the data. The reduce piece isn't really needed, because there is no collation to be done. Plenty of people use MapReduce this way, primarily because it allows them to easily decompose their logic into parallel processing units.

We are even going further by undermining one of the key strengths of MapReduce, the ability to operate on large chunks of data, and instead running lots of smaller jobs concurrently. The Hadoop framework makes this possible. I'm not sure how optimal this is, and expect that we will be tweaking the message size and concurrency dials for some time. One of the advantages that the underlying Hadoop framework offers us is flexibility, as evidenced by the kinds of dials we can tweak to get optimal system throughput and latency.

I'll keep updating as we learn more about how the system behaves, and what the 'right' settings of the dials are based on the amount of data being input. I don't know if this is necessarily the most correct or elegant way to build a pipeline, but I do know that it is a good sign the Hadoop framework is this flexible.

logrotate and scripts that just cant let go.

2009-06-01T11:58:00.000-07:00

I just found out that a logrotate job I had configured X months ago wasn't working when the lead came up to me and said 'what the $#!k is a 5GB file doing in my /var/log?' He was pissed because this was the second time logrotate had not been configured correctly (both times, my fault). The first time, we discovered logrotate.conf cannot have comments in it. This time, it looked like logrotate had run, but the script had kept the filehandle to the old log file open and was continuing to log to the rotated file.

One way to do this is send a HUP to the process in the postrotate script of the logrotate. This would mean I had to modify the script to trap the HUP signal, release the filehandle, get a handle to the new (same name) log file, and keep rolling on. I decided not to do this because it involved modifying the original script and I didn't have that much time to relearn things.

The second way I ended up doing this was to kill and restart the process during postrotate. Here is the config file in /etc/logrotate.d

/var/log/response_process.log {
daily
missingok
rotate 52
compress
delaycompress
notifempty
create 640 root adm
sharedscripts
postrotate
kill $(ps ax | grep process.rb | grep -v 'grep' | awk '{print $1}')
ruby process.rb -logfile:/var/log/response_process.log > /dev/null 2>&1
endscript
}

kind of hacky, but I didn't have to expend any mental effort making sure that the filehandle was truly closed in the script.

Running a Hadoop 0.20 Cluster using S3 as input/output

2009-05-13T14:39:00.001-07:00

I've been changing a database ETL application into a set of MapReduces up on EC2. I need s3 as my input and output for each MapReduce, and was excited to see that Hadoop had s3 filesystem support built in.

After stumbling through the ec2 scripts in 18.3, and finding a much easier go of it via the Cloudera scripts, I ran into a blocking issue (for me, anyway) with the version of Hadoop (based on 18.3) on the Cloudera AMI -- there were issues writing to S3 as the output.

I then started looking at 0.19.0, where the issue was fixed, but found another issue (again, s3 related, this time reading the input directory). I was able to reproduce this issue on my local box immediately, which saved some time.

This left me with 0.20.0, which claimed to have both issues fixed. I tested 0.20.0 on my local box with a small data set, and it passed. The next step was to build an AMI with Hadoop 0.20.0 on it, deploy that AMI to a reasonable sized cluster, and try to get through an entire run of my 133 million record input set, which was estimated to reduce to a 7.5 million record output set.

I decided to start by using the 0.20.0 src/contrib/ec2 scripts. The learning experience working with the original src/contrib/ec2 files in 18.3, and then working with the Cloudera scripts allowed me to move much faster this time.

In order to build an image using the scripts, you need to specify the following (in addition to the account access variables detailed here).

HADOOP_VERSION -- I set to 0.20.0

S3_BUCKET -- I used my own bucket to store the AMI.

INSTANCE_TYPE -- Amazon small and medium instances are 32 bit, Large and XLarge instances are 64 bit. Specifying INSTANCE_TYPE lets the shell load the correct base OS image.

JAVA_BINARY_URL -- the download link to the version of Java you want to use. Note this varies depending on the architecture (i386 or X86_64). For i386 I used:

http://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/en_US/-/USD/VerifyItem-Start/jdk-6u13-linux-i586.bin?BundledLineItemUUID=teRIBe.ohNsAAAEhz0pwgkAW&OrderID=yI1IBe.oPMYAAAEhuEpwgkAW&ProductID=RGtIBe.ou1AAAAEfpVYcydOO&FileName=/jdk-6u13-linux-i586.bin

Note that I then had to change my JAVA_VERSION to match the minor version specified: i.e. for the link above I had to set JAVA_VERSION to 1.6.0_13.

Now that all variables were configured, I ran

hadoop-ec2 create-image

to create the exact Hadoop image I needed.

With the image created I then used

hadoop-ec2 initialize-cluster mycluster 20

to create a 20 node cluster. I logged in, and the first thing I noticed was that JobTracker was not running on the master, and TaskTracker was not running on the slaves. Even though they were specified to start right after the NameNode and DataNode (respectively) in the shell file executed at AMI boot time:


if [ "$IS_MASTER" == "true" ]; then
# MASTER
...
# Hadoop
# only format on first boot
[ ! -e /mnt/hadoop/dfs ] && "$HADOOP_HOME"/bin/hadoop namenode -format

"$HADOOP_HOME"/bin/hadoop-daemon.sh start namenode
"$HADOOP_HOME"/bin/hadoop-daemon.sh start jobtracker
else
# SLAVE
...
# Hadoop

"$HADOOP_HOME"/bin/hadoop-daemon.sh start datanode
"$HADOOP_HOME"/bin/hadoop-daemon.sh start tasktracker

fi

So I ran the following command to get the (internal) names of the slave nodes (from my laptop):


ec2-describe-instances | grep -w 'infocloud' | grep -ve 'infocloud-cluster.*' | awk '{print $5}

In this line I grep for my security group (infocloud) and then excluded the non AMI lines that contained my cluster name (infocloud-cluster.*), and finally print the fifth element in the list. This gives me a list of (Amazon EC2) internal domain names, like this:


domU-12-31-39-02-B4-F3.compute-1.internal
domU-12-31-39-00-B5-12.compute-1.internal
domU-12-31-39-00-5D-E3.compute-1.internal
domU-12-31-39-00-56-46.compute-1.internal
domU-12-31-39-00-58-51.compute-1.internal
domU-12-31-39-00-A8-B6.compute-1.internal
domU-12-31-39-00-85-D1.compute-1.internal
domU-12-31-39-01-74-22.compute-1.internal
domU-12-31-39-00-E8-94.compute-1.internal
domU-12-31-39-00-C6-13.compute-1.internal
domU-12-31-39-00-DC-65.compute-1.internal
domU-12-31-39-00-4D-D3.compute-1.internal
domU-12-31-39-01-5C-B6.compute-1.internal
domU-12-31-39-00-B2-54.compute-1.internal
domU-12-31-39-00-66-06.compute-1.internal
domU-12-31-39-00-E5-B7.compute-1.internal
domU-12-31-39-00-68-06.compute-1.internal
domU-12-31-39-00-88-46.compute-1.internal
domU-12-31-39-00-7D-C8.compute-1.internal
domU-12-31-39-00-A1-08.compute-1.internal
domU-12-31-39-00-C2-15.compute-1.internal

Note that the first node in this list is the root node. I echoed this output into a file that I then pushed up to the master:

hadoop-ec2 push infocloud-cluster nodes.txt

and then wrote some ruby to parse it:


HADOOP_HOME="/usr/local/hadoop-0.20.0"

File.open("slaves.txt") do | file |
cmd = "ssh #{slave} #{HADOOP_HOME}/bin/hadoop-daemon.sh start"
while(slave = file.gets)

slave = slave.chomp
hostname = `hostname`
if(slave.starts_with(hostname))
  cmd += " jobtracker"
else
  cmd += " tasktracker"
end

`#{cmd}`

end

end

So I start the job tracker for the master node (where I run the job from), otherwise I start the task tracker. Note I shouldn't have to do this, and I'm still trying to figure out why the original command in the remote startup script didn't work.

Once job tracker and task trackers were started, the cluster was effectively up. I'm going to see if I can get the remote startup script to work as designed, because that final step is hacky.

Finally, once I had started up the cluster successfully, I noticed that there was only one node configured to do reduces. I remedied this by changing my generated hadoop-site.xml, which, btw, is flagged deprecated for the 0.20.0 version (it still works, but probably not for the next version). The hadoop-site.xml is generated in hadoop-ec2-init-remote.sh, here is what I modified:


<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/mnt/hadoop</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://$MASTER_HOST:50001</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>hdfs://$MASTER_HOST:50002</value>
</property>
<property>
<name>tasktracker.http.threads</name>
<value>80</value>
</property>
<property>
<name>mapred.reduce.parallel.copies</name>
<value>20</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>20</value>
</property>
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>3</value>
</property>
<property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>3</value>
</property>
<property>
<name>mapred.output.compress</name>
<value>true</value>
</property>
<property>
<name>mapred.output.compression.type</name>
<value>BLOCK</value>
</property>
<property>
<name>dfs.client.block.write.retries</name>
<value>3</value>
</property>
<property>
<name>fs.s3n.awsAccessKeyId</name>
<value>YOUR ACCESS KEY ID</value>
</property>
<property>
<name>fs.s3n.awsSecretAccessKey</name>
<value>YOUR SECRET KEY</value>
</property>
</configuration>

With these changes in place, I have started a hadoop job that has both s3 inputs and outputs.

My Guest Post on Cloudera

2009-05-11T15:15:00.000-07:00

I used to think that this blog had an audience of one -- me. Not even my mom reads this blog. However the guys from Cloudera stumbled upon a recent post I had written about getting Hadoop running in EC2, and pointed me to their EC2 setup scripts, which pick up where src/contrib/ec2 left off. They let me write up a guest post on configuring and running a MapReduce job using their scripts. Check it out!

A Reality Check

2009-05-11T11:43:00.000-07:00

From an email I sent last Friday:

Hi,

Last week, Leela had developed an abscess (bacterial infection) behind her throat wall under the base of the skull, where all the nerves and muscles attach to the spine. It inflamed her neck and shoulder muscles and they spasmed. Over three days it had progressed from what we thought was a slight neck sprain to the point where her head was locked down to one side, she was running a fever, and was in a massive amount of pain. Yesterday I took her to the doctor first thing in the AM and he sent me directly to the ER at Childrens Hospital.

After multiple tests, they figured out the abscess thing and did surgery to remove it around 7 last night. Because we and her doctor had originally thought it was a neck sprain and nothing more, the infection had time to spread and she needs to be hospitalized until it disappears from the surrounding tissue.

The good news is that -- unlike a lot of kids at Childrens -- the situation has a very high probability of a good resolution, and we were able to catch it before it occluded the carotid artery or her windpipe. Also, she's an extremely tough little girl, I thought so before this but after yesterday I'm convinced that she has the constitution of a Navy Seal.

Childrens Hospital is an extremely hard place for a parent to be, but the staff is amazing. We are so lucky to have them here in Seattle. I have been in a lot of emergency rooms and I know that it would have been much harder to get the kind of treatment and attention that Leela received at Childrens at any other hospital (holy run on sentences, batman). We're not sure how long she is going to need to be hospitalized, it depends on how quickly she can recover. Lopa spent the night last night, and we'll be trading off over the next couple of days.

Update: She is coming home today (Monday) after going into the ER on Friday. It's amazing how fast kids bounce back. She had a rough Saturday, she was in pain, constipated, and still fighting the infection. But on Sunday she was back to herself again, giggling, teasing me, and smiling.

Before this whole thing went down, I had a big 'todo' list for the week and weekend:

(1) finish up guest blog post for cloudera.

(2) assemble extracted data up to cloud via MapReduce.

(3) Go on 26 mile slow run

(4) mow lawn and attempt to fix waterfall.

(5) Play goalie for Pele's Nightmare on Friday night.

(6) Take Kiran to his lacrosse game.

(7) Celebrate Mothers day down at Seward park on bikes.

(8) refactor some old code using java concurrency lib goodies.

All of that got massively pre-empted, and as I sat in the ER looking at my little 5 year old daughter scream in pain, all of it really didn't matter anymore. The only thing that did matter was finding out what was wrong with her and fixing it. Fortunately the doctors were able to do just that, and through the weekend I was able to come up with a better todo list:

(1) be with my family and enjoy them.

(2) celebrate the little things in life.

(3) never take the people I love for granted.

(4) all that other stuff is gravy.

You would think that after watching my dad lose his battle with cancer 2 years ago that these priorities would be second nature, but that's not the way my mind works, I tend to 'glorify the mundane'. This past weekend was very hard, but at the same time it was a good reminder of what really matters.

Video on Cloud Computing Amazon EC2

2009-05-07T14:43:00.000-07:00

Again, more of a pointer than a blog post.

Cloud Computing on Amazon AWS EC2 from ray@sharethis.com on Vimeo.

Economy of Hadoop on AWS

2009-05-07T13:40:00.000-07:00

This is more of a note to myself, so I don't lose the video, and can fwd to others.

Big Data: On Cloud Computing and Hadoop from Roberto Monge on Vimeo.

Configuring a Hadoop cluster on EC2

2009-04-14T15:24:00.001-07:00

I've been ramping up on Amazon Elastic MapReduce, but now I need to process a 32GB file located in an Elastic Block Store, and there is no way I know of to get the AMIs that Amazon Elastic MapReduce starts up to mount an arbitrary EBS. So now it's time to roll my own Hadoop Cluster out on EC2.

I looked around for a while, and found this somewhat out of date tutorial by Tom White that pointed me to a set of ec2 helper scripts in the src/contrib subdir of the Hadoop installation.

Unfortunately, those scripts did not get me 'all the way there', but they were a start. I'm going to try to roll my changes into those ec2 helper scripts before I have to do set up another cluster:)

Setting Up a Multi Node Hadoop Cluster on EC2

Prior to setting up a Multi Node Hadoop Cluster, I set up a single node standalone installation. I recommend doing this because it allowed me to make sure my code worked, i.e. my jar file was valid, my Mapper and Reducer were working, etc.

In order to set up a multi node hadoop cluster, the standard Hadoop Cluster setup instructions applied to the EC2 environment meant that I would have to do the following

find an AMI with hadoop on it
bring up N+1 of those
make one the master and the rest the slaves
change the master config to account for the slaves
change the slave config to point to the master
allow Hadoop component port access between master and slave for namenode and datanode communication
start the system up.

The scripts at {hadoop src location}/src/contrib/ec2/bin use the ec2 API shells to do attempt to do all of the above. They fall short in a couple of key areas, and need to be extended. I'm going to detail the necessary steps I took to get a cluster fully operational so that I can extend those scripts in the future.

What the scripts do:

Find AMIs and starting up instances, N slave instances and 1 master instance.
Allow you to log into the master as well as push files out to it.
Generate a private/public key on the master, and push the public key out to the slaves to enable password-less ssh.
Push the master hadoop-site.xml out to all slaves.

What they do not do:

they do not configure the master conf/slaves file to contain the IPs of all slaves.
they do not set up security groups with overridden port values specified in the /etc/rc.local of the AMI I was using. Those values are catted to conf/hadoop-site.xml. To be honest, there is no way they could actually be aware of those values unless the scripts were synchronized to the image, which they weren't.

Both of these mean that true distributed startup doesn't happen. But the failure is 'silent', so unless you are looking at the logs on multiple machines, you don't know that things are failing.

Initial Script Setup Steps

Here are the steps I used to get working with the scripts. Note that the AMI the scripts point to by default has version 0.17 of Hadoop installed.

(1) I configured my EC2_PRIVATE_KEY and EC2_CERT env vars to point to the .pem files I generated for them.
(2) In {hadoop src location}/src/contrib/ec2/bin/hadoop-ec2-env.sh, I set the following env vars:

AWS_ACCOUNT_ID={acct number}
AWS_ACCESS_KEY_ID={key id}
AWS_SECRET_ACCESS_KEY={secret key}
KEY_NAME={name of KeyPair you want to use} NOTE: on the KeyPair, the hadoop-ec2 scripts assume that the generated private key for your keypair resides in the same directory you configured your EC2_PRIVATE_KEY in.

(3) {hadoop src location}/src/contrib/ec2/bin/hadoop-ec2 {name of cluster} {number of desired nodes} to start up a cluster with the AMI configured at the S3_BUCKET location specified in the conf file.

At this point, I thought the cluster was up and running, but when I tried to copy a large file to the cluster, I got this error:

org.apache.hadoop.ipc.RemoteException: java.io.IOException: File

/user/root/input could only be replicated to 0 nodes, instead of 1

at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)

at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)

at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)

at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)

I googled this and found out that it implied that my data nodes were failing (but I hadn't seen that!). I checked the masters and slaves files in the master machine conf file, and they only contained localhost, which meant that the master knew nothing about the slaves at startup.

I stopped hadoop, changed the conf/slaves config file to include the Amazon internal names of all slaves, and restarted. This time I could see the remote slave data nodes start up. So I tried the copy again, and got the same failure.

When I went out to a slave machine, I looked at the datanode log file in the log directory (on this AMI, configured at /mnt/hadoop/logs. I saw that the datanode service was trying to contact the master with no success.

This ended up being because of the security policy of EC2. In EC2, you need to explicitly configure which ports are accessible on each instance via EC2 Security Groups. In summary, the current scripts assumed defaults from hadoop-default.xml, and I had overridden some of those defaults in hadoop-site.xml.

Summary:

extended conf/slaves to include IP addresses of slave instances.
added 50001 and 50002 access to the master security group (meaning that slave nodes could talk to the master on those ports)
added 50010 access to the slave security group (same meaning for master to slaves)

At this point Hadoop was configured with 4 slave nodes and 1 master.

Attaching an EBS to the master and copying EBS data to HDFS

(1) My data source was located in a Elastic Block Store volume. These are mounted like so:

ec2-attach-volume {volume ID} -i {image ID} -d {device location on image, i.e. /dev/sda}

(2) In order to actually access the data, you mount that volume like you would mount a drive:

mount {name of dir to map to} {name of device to mount, i.e. /dev/sda}

Running the Job

(1) Once I mounted the volume, I needed to log into the master to start the job.

{hadoop src location}/src/contrib/ec2/bin/hadoop-ec2 {name of cluster} login

(2) Then I need to push the file to HDFS for processing.

hadoop fs -mkdir input

hadoop fs -copyFromLocal {location of articles.tsv} input

(3) From my local box, I push my jar out to the master:

{hadoop src location}/src/contrib/ec2/bin/hadoop-ec2 {name of cluster} push {name of my jar file}

(4) On the master box, I start the job:

hadoop jar {name of jar} input output (NOTE: job will fail immediately if output exists in hdfs)

Developing an Application on Amazon Elastic MapReduce

2009-04-08T13:11:00.000-07:00

Follow up from last week:

I've been interested in parallel processing for a while, but didn't have the time/opportunity to get into it until recently, when a group of us were asked to rewrite a data extraction system that was built around a single (relational) database, and suffering from severe i/o contention.

The actual mechanics of what we were trying to do with that system lent themselves to parallel processing, specifically there was a lot of data transformation and aggregation that we needed to do -- in other words map-reduce.

Right around the time that we decided to do the rewrite, Amazon came out with Elastic MapReduce. Prior to EMR, our options were to either set up a local Hadoop cluster, or assemble one out on EC2. My experience with Hadoop was zero, and my experience with the map/reduce algorithm was limited to working through the canonical Inverted Index example in my head. So I wanted to minimize the learning curve of setting up a cluster while still being able to validate our ideas. I don't know if we are eventually going to end up building our own AMIs for a custom Hadoop cluster, but not having to deal with those details at this point allows us to focus on the ideas instead of the infrastructure. Yay!

In my last post I explained the steps I took to run the basic streaming sample. Next up, I wanted to run my own map reduce job. In order to develop an EMR application utilizing a custom jar, I took the following steps:

(1) Set up a single node hadoop installation on my dev box.

Because Time Is Money on EC2, I wanted to get my code as right as I could before running it in the cloud. The Hadoop site provides a great step by step install sheet for Leopard.

(2) Went through the basic Hadoop Tutorial. This tutorial provides great explanations of each class the developer has to implement, as well as the helper classes. Whenever I got stuck, the answer was usually found in the tutorial.

(3) Installed the IBM MapReduce Tools Plugin for Eclipse. While I am not quite able to get it debugging a map/reduce job from my local Hadoop install, I can run my program in 'standalone' mode as a Java Application, as long as Hadoop is started on my system.

(4) After debugging in the IDE, I ran the job as a standard hadoop job:

hadoop jar {name of jar} {input dir} {output dir}

I chose to export the jar from eclipse (instead of writing a build.xml), as part of this I got to specify the main class and autogenerate the MANIFEST.MF.

The input directory and output directory needed to be specified as HDFS directories. The output directory could not exist.

NOTE on the application driver: I took a clue from example code (eventually) after running into unzip errors which a couple of posts said were due to bad configuration. I derived the driver class from Configuration, extended the Tool interface.

(5) After running successfully as a local Hadoop Job, I ran the job from the cloud,

(a) I copied the jar up to s3

(b) I copied the input files up to s3

I tried to run my job as it was configured for hadoop, with the input and output specified as s3n://{bucket-name} with no success.

In order to diagnose the problem, I enabled logging, which I should have done immediately. When configuring a job via the AWS Management console, select 'Advanced Options' in the 'configure EC2 instances' section. You need to specify the log destination using s3n://{bucket-name} format.

With logging enabled, I saw the following exception:

java.lang.IllegalArgumentException: Path must be absolute: s3n://java-emr-test

and it made no sense, even with googling. I ran the cloudburst sample, and noticed that they specified a full filename as the input parameter. When I did the same, my run succeeded. I'm not sure why I had to do this, and want to make sure I really need to do it, because it will mean an additional read step of intermediate output directories prior to subsequent map reduce steps.

Now that I've got this working through the UI, I'm going to look at the entangledstate ruby tool because it appears to allow programmatic configuration of multi step job flows.

How I got rolling in the cloud

2009-04-06T16:14:00.000-07:00

I recently jumped at the chance to research re-implementing a project here at work in the Amazon cloud. I've been curious about running EC2 instances for a while, and when AMZN announced Elastic MapReduce, their cloud implementation that removed the need to hand assemble Hadoop clusters, I really didn't have any excuses left.

There was a bit of FUD involved in actually getting into an actual cloud -- creating running instances, talking to various services, etc, in addition to trying a new approach to a system that was quickly approaching non functional.

This FUD was complicated by my head cold and the medication I was taking, but despite that fog (yes, I always blame it on the drugs) I was able to muddle through and get something going. My notes (aka a series of pointers to other peoples work):

(1) I needed to view some sample code at http://elasticmapreduce.s3.amazonaws.com/ , i.e. code that was not in my personal s3 store. I tried to build a couple of S3 browsers, and was about to embark on a yak shaving exercise due to a misconfigured ant build on my dev box when I decided to try s3curl instead. s3curl and irb loaded with hpricot allowed me to get an XML listing of keys in a bucket, then parse the returned XML and download the source code files I wanted to see specifically, the AWS Elastic MapReduce Freebase sample code. I'm 100% sure I could have done this via a UI, but really didn't want to get distracted trying to fix a secondary issue.

(2) For browsing and syncing my personal s3 store: I used the S3 Firefox Organizer plugin. Especially useful when inspecting the output of a map-reduce run.

(3) For configuring AMIs and binding EBS volumes of public instance data, I used ElasticFox, another FF plugin. The tutorial walks you through the details of how to generate a keypair, create an instance from an AMI, and bind to an EBS.

(4) The application I'm working on (for work) processes wikipedia and freebase, both of which can be painful and time consuming to get dumps of. Freebase has done the 'right thing' and posted public instances of the Freebase data store as well as a 'cleaned up' version of the Wikipedia data store that is suitable for a postgres database. Just having these volumes available removes at least 4 hours of setup and maintenance time from our process.

(5) As part of their announcement, Amazon posted a tutorial on how to use Elastic MapReduce using Freebase data. I found this great PDF that walked me through using the CLI to set up several different workflows using different mappers and reducers to find the most popular people in American Football. The mappers and reducers output data to S3 and SimpleDB, which was great for me to see since I didn't have a lot of familiarity with either.

That's it for now. I'm going to write more as I prototype key parts of the system and try to figure out the best way to implement.

Thoughts on GPS data analysis.

2009-03-27T06:25:00.000-07:00

Notes to self:

I'm collecting craploads of GPS data, delimited by 'event' (i.e. a run/ride/ski).
I want to correlate that data to compare metadata associated with that GPS data (heart rate/pace)
I want to do this in a non n^2 fashion.

Some thoughts:

Similar 'events' usually center around the same GPS range. What is the margin of error for that range?
If I can quickly get similar events grouped by center, than I can chunk them up by distance. Every event has a starting point, and even though GPS data will not be identical, distance would be a good way to divide events into similar subsections.
Once I get similar event subsections, I can start comparing (i.e. graphing/analyzing) HR/pace data.
Of course, all of this assumes that I've got a system that displays basic statistics for each event, and I'm not there yet.

On the Couch(DB): Part 1: setup on mac OSX Leopard

2009-03-16T20:48:00.000-07:00

Why CouchDB? I'm running into situations where a traditional database is giving me headaches:

(1) At work, I'm seeing people spend more and more time maintaining the database and tweaking the schema behind an ETL application used to normalize input data into a standard format as the data being persisted starts to get into the TB range.

(2) At home I want to write an app storing my GPS data from my runs and rides, and I have no idea what the metadata around that data is going to be.

The common denominator for both of these situations is that a traditional database schema, with very explicitly defined tables and relations, is a liability, either eventually, as in case 1, or initially, as in case 2. In either case I'm more likely than not to be wrong with any schema design choices that I make.

In both situations, a more flexible approach would be to treat each discrete set of data as a bag of attributes (key value pairs) with a couple of unique keys that I can search on. I could see creating a Lucene or a BDB with those keys and a pointer to a file containing the rest of the data. This would work pretty well, until it was time to go to production or scale, or both. When I go to production, I want the data to be highly available. When I scale, I want the data to be partitionable. To do either I would have to extend my naive hashing scheme.

CouchDB is a document based database -- in other words each document is an attribute bag identified by 1..N keys -- that is fault tolerant and distributable, with incremental replication. In other words it is the extension my naive hashing scheme needs to be a real boy instead of a puppet.

Enough hype and bloviation...I need to install couchdb on a macbook pro running leopard. I followed the instructions found here, and quickly discovered I had neglected to install XCode when I had upgraded to Leopard. Doh!

btw, if you ever do this, you'll find that macports has downloaded and staged components into /opt/local/var/macports/build, and may not be able to build them successfully after failure due to lack of compilers, etc. Removing the staged directories for the components that were downloaded is the best way to 'reset' macports. Trying to configure, build, and install macported components yourself is not :)

I did run into a minor bump: tcl was installed at 8.4.14_0 on my box, I needed to upgrade to 8.5.6_0 because sudo crapped out trying to build tk. I did this by hand:

sudo port install tcl --version 8.5

sudo port deactivate tcl @8.4.14_0

sudo port activate tcl @8.5.6_0 (using sudo port list | grep tcl to get exact version numbers)

OK, with XCode and dependencies installed, I then followed the rest of the instructions, which, beyond building and installing couchdb, show you how to create the required couchdb system acct, install it as a service, and launch it at startup. All Leopard specific, but all highly convenient.

I also found this one step install -- good for getting up and going, but I think in the long run I want to build and understand couchdb a little more than this approach lets me.

Next up: simple couchdb access.

Intervals -- why am I doing this again?

2009-03-15T13:30:00.000-07:00

Today I got bitch slapped by reality, or actually the weather, which is one variant of reality. I was planning to do a long run of at 16-18 miles, mixed in with 10 1 mile intervals. It was raining slightly, or 'misting' as we like to say around here (in Seattle we have many different terms for rain), nothing too hard, but it was 35 degrees, so I put on a shell instead of a tech tee, strapped my headlamp on (thanks to Daylight Savings, it is now pitch black at 6:00 AM), and headed out, full of ambition.

I had downloaded the days workout into my Garmin. 10x1 mile at 8:40. I felt a little slow going out, but chalked it up to the disoriented feeling I was getting running in the rain, in the dark. It is fairly surreal -- my headlamp lights up a small cone of streaking raindrops, and very little else, so I start to zone out (and slow down).

When I started the intervals, I knew something was wrong. My get up and go had got up and went. And it was raining harder. No more mist, more like psuedo hail. Then snow, big fat flakes that chilled my face to numb as they fell and stuck. Still, I soldiered on, trying to maintain a pace that seemed pretty achievable from the glow of my laptop that morning, but was proving to be a painful challenge on the hilly Mercer Island loop.

A younger and less intelligent me would have ground myself down into a little nub trying to do the intervals, but I'm older, and less inclined to hurt for no reason, so I quit. I turned around, changed my plans from 18 miles to 11, and packed it in. I ran a couple of intervals on the way back, but by then my legs were so frozen that it felt like I was running through (very cold) molasses.

Later, in the shower, I analyzed the workout. Heres the thing. I know I'm fairly motivated. I know that I'm working hard. My heart rate told me so. And that ended up being the key. Why am I trying to maintain some arbitrary pace up and down a hilly course? I should be running these intervals by heart rate -- by setting a target to stay above -- or below, depending on the goal of the interval -- I'll run at a specific effort, and the pace will be what it is. This feels a lot more real than trying to do intervals at X:30/mile 'because so and so said to', or because X:30 means I'll run a marathon in Y:45. Those are completely arbitrary, completely false goals, and I will crash and burn trying to make them work. On the other hand, I firmly believe that if I can run to a specific heart rate, I will see improvements in speed, because I will see improvements in efficiency.

Old Guy Training

2009-03-13T21:21:00.000-07:00

This whole marathon training thing has brought me face to face with my advancing years.
The last time I seriously trained for running was when I was 30, training for my first 1/2 Marathon. I had never run 13 miles before and as a result I prepared as well as I could, which meant that I went out, ran as fast as I could for as long as I could, took the next day off, and did it again.

This time around that approach really doesn't work. I just can't recover between efforts, even with a day off. Maybe it's the kids, maybe it's the job, maybe it's the additional 10 years. But after a long run or a speed workout I'm cooked for at least a day if not two. However, I'm running much longer than I ever used to, and at least as fast.

Which reminds me of my favorite 'old guy' story. Actually it's an 'old bull' story. There are two bulls, one young, one old, sitting at the top of a hill, overlooking a pasture, staring down at a bunch of attractive (to bulls, anyway) cows. The young bull says to the older one: "Hey Pops, lets run down the hill as fast as we can and fuck a cow!". The old bull turns, looks at the younger one and says "Hey kid, lets walk down and fuck em all." I wish I had been this intelligent about training earlier, when I actually had the ability to recover quickly from a hard workout. But youth is definitely wasted on the young, and I can't turn back the clock.

These days I'm running one long day, up to 1/2 to 3/4 is at marathon pace, and one fast day, consisting of mile splits at 1-2 minutes faster than marathon pace. Both runs include stretches of 'Gallow-walking', as recommended by Jeff Galloway. I usually walk until my pulse drops to <> 15 miles, the other one will be around 8-9 miles, and the easy runs all under 5.

I'm hoping that the speed work as well as the marathon pace runs on the longer days will effectively get me faster. I definitely feel like speed is the last thing to come around. The other stuff, like stretching, climbing, pullups and squats, really helps balance out the pounding from the running. As the days get longer I'll probably skip one of the runs and get in a medium length bike ride.

One benefit of being 40 is simply 10 more years of experience. In the last 10 years I have grown a lot as a person. I've become a father, I've lost my father to cancer, my entire perspective has shifted from 'what can I do for fun next?' to 'what can I do for my family?'. The additional responsibility gives me the perspective to not take training so seriously. At the same time I am able to fully commit to training when I am doing it, because my time is so limited. Any pretensions of actually being 'elite' have disappeared completely. I'm doing this because I like it, not because I'm good at it. I can honestly say that I enjoy running more than I ever did before, because it gives me 2-3+ hours of silence in which I can work things out.

Last week my long run was a relatively flat 18 miles -- in freezing hail and rain. I froze parts of my body I never want to freeze again. However the run itself was a great confidence booster -- mostly due to the weather. Training has been a constant adventure, pushing these distances that I've never done before has been a lot of fun.

The marathon course was published online yesterday, and it actually looks kind of hilly. So that, plus the fact that we're going to Disneyland next week, makes me want to 'burn one down' on Sunday, maybe a good 18 miles with hills. An easy run on Monday, climbing on Tuesday, and speedwork on Wednesday prior to leaving for CA will probably set me up just right to take it easy and recover while in the land of Mickey Mouse.

Migrating an RoR app from Ubuntu Feisty to CentOS 5.2 Part 4: Trying not to get impaled by Vlad

2009-03-12T11:32:00.000-07:00

This is a continuation from Part 3:

I've installed mod rails
I've built and migrated the database (and enabled script/console)
I've built and installed RRDTools.

In this part I get Vlad the Deployer working.

Local Machine Changes
(1) In my local /config/deploy.rb I changed the :domain value to point to my new box:

set :domain, "xen-5.evri.corp"

(2) I tried to run vlad:setup. My remote install commands, run via SSH, were not being found.

Server Machine Changes

The problem with ssh seems to that my user environment is not being established with a non interactive terminal. Even after I modified my /etc/profile or files in /etc/profile.d or ~/.bashrc , even though that is how I understood non interactive terminals get their env variables. The problem was that the default sshd_config does not allow user environment variables to be set via sshd. So in /etc/ssh/sshd_config, change

# PermitUserEnvironment no

to

PermitUserEnvironment yes

I reran vlad:setup

rake vlad:setup rake full_vlad (I've modified my rakefile as follows: task :full_vlad =>['vlad:update','vlad:migrate'...all other tasks I need to do ]
As a final step, I asked OPS to map the old server name "dashboard" to map to the new server.

Are We There Yet?
No.

I run a set of ruby cron jobs on the machine (TODO: migrate these to use DaemonSpawn??), I migrate cron settings over by

crontab -l > cron
scp cron to new machine
crontab cron

Now I run rake full_vlad, and the server deploys after a full source update and database migration.

Unfortunately, my graphs are not showing any data. wtf?

I call rrdtool via IO.popen to extract my data from rrd files. On the centos box, IO.popen is not returning a readable IO object. I cannot repro with script/console from the production box (i.e. that gives me valid IO).

Some sleep and investigation reveals that the environment variables for my user are not being set when Ruby spawns a process. This, again, is unique to centos. So I hardcode the path of rrdtool in my IO.popen call, and am now receiving data.

Migrating an RoR app from Ubuntu Feisty to CentOS 5.2 Part 3: Installing RRD

2009-03-12T10:53:00.000-07:00

Continued from Part 2:
I've installed RRD before. I followed these instructions, and took the following additional steps to install development headers for pango and libxml2 (missing from default install of centos)

sudo yum install libxml2-devel

sudo yum install pango-devel

In the instructions above, I downloaded rrdtool source. Next I build and install rrdtool
(1) ran make in rrdtool dir
(2) sudo make install to install in /usr/local/rrdtool-1-3.6
(3) modify the path in /etc/profile to include /usr/local/rrdtool-1-3.6/bin

Next up, the home stretch, Impaling myself with Vlad.

Migrating an RoR app from Ubuntu Feisty to CentOS 5.2 Part 2: Database Migration

2009-03-12T10:16:00.000-07:00

Part 2 in a series of 4

In Part 1, I got Phusion Passenger up and running. Now I needed to migrate my database schema across. I use Postgres 8.3, which was not installed on the centos box.

Installing Postgres
I need to make sure postgres is (a) installed and (b) configured.

(a) installing postgres: following steps outlined here.
(b) configuring my database and user: my database settings from database.yml are:

development:
adapter: postgresql
host: localhost
database: deploy_monitor_development
username: deploy_monitor
password: deploy_monitor

test:
adapter: postgresql
host: localhost
database: deploy_monitor_test
username: deploy_monitor
password: deploy_monitor

production:
adapter: postgresql
host: localhost
database: deploy_monitor
username: deploy_monitor
password: deploy_monitor

Because this is a production box, I only need to configure the production database, meaning I only need to add the production user deploy_monitor and the database deploy_monitor.

I followed the instructions from the postgresql site to add a user and create a database.
I also needed to install the following gems:

postgres (0.7.9.2008.01.28)
postgres-pr (0.5.1)

to enable Rails to connect to Postgres.

Console Access
A lot of times I need to log into the box and check something via script/console, i.e. to run an ActiveRecord query.
In order to run console, I need to build readline:

(1) Install readline and readline-devel

yum install readline
yum install readline-devel

(2) build/install ruby readline bindings

cd {ruby src}/ext/readline
ruby extconf.rb
make
sudo make install

Continued: Part 3, Installing RRD

Migrating an RoR app from Ubuntu Feisty to CentOS 5.2 Part 1: Setting up mod_rails

2009-03-03T13:54:00.000-08:00

Notes from migrating an app from an overloaded Ubuntu server box to a new, virgin centOS box. The app is a Rails app, with a postgres db, served up via mod_rails (passenger). The destination box has apache2 and Ruby 1.8.7 installed.

My app uses phusion passenger, which is fairly easy to install.

Steps:
(1) sudo gem install passenger
(2) I then needed to generate the apache mod for passenger.

sudo passenger-install-apache-2-module

This step told me the following:

* To install GNU C++ compiler: Please run yum install gcc-c++ as root. * To install OpenSSL support for Ruby: Please (re)install Ruby with OpenSSL support by downloading it from http://www.ruby-lang.org/. * To install Apache 2 development headers: Please run yum install httpd-devel as root.
Installing gcc-c++

sudo yum install gcc-c++, this was painless. Yum is installed by default on the version of centos I was migrating to (5.2)

Enabling Ruby with Open SSL

Unfortunately the company approved version of Ruby 1.8.7 did not include ext/openssl, so I needed to download, build, and install on my own.

(1) download source

wget ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.7-p72.tar.gz

(2) untar, cd to extracted dir, and run ./configure
(3) make

At this point, you usually sudo make install, but I didn't want to re-install ruby, I just wanted to add openssl to an existing installation.
(4) cd extracted dir/ext/openssl
(5) ruby extconf.rb
(6) make
(7) sudo make install

Onwards/Upwards
I re-ran sudo passenger-install-apache-2-module, which completed successfully. As instructed, I pasted the following into /etc/httpd/conf.d/passenger.conf, because all files in conf.d are loaded by httpd.conf:

LoadModule passenger_module /evri/ruby/lib/ruby/gems/1.8/gems/passenger-2.0.6/ext/apache2/mod_passenger.so PassengerRoot /evri/ruby/lib/ruby/gems/1.8/gems/passenger-2.0.6 PassengerRuby /evri/ruby/bin/ruby
I then put in the server directives to map my public directory as well as a specified VirtualHost to my app.

Options FollowSymLinks
AllowOverride none
Order allow,deny
Allow from all

ServerAdmin webmaster@localhost
ServerName xen-5
ServerAlias xen-5.local xen-5.evri.corp
DocumentRoot /var/www/rails/dashboard/current/public
ErrorLog /var/www/rails/dashboard/current/log/server.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel debug

Phusion Passenger aka mod_rails was configured.

Next: Database Migration

An Open Letter to the Creators of Flot

2009-02-14T21:23:00.000-08:00

Dear creator(s) of Flot, who I only know from this blurb on your google code page:

The development so far has mostly been done by Ole Laursen, sponsored by IOLA, a small Danish web-development house focusing on Django and jQuery.

I want to thank you for making such a kick ass graphing tool. First of all, you have cleanly separated data from code: I really think that isolating the actual graphing call to this method:

$.plot(...)

really allowed me to focus on what I should be focusing on when using a graphing tool: the data I wanted to present, instead of the mechanism I was going to use to present it.

Second, you have eliminated the need for me to write some kind of JS wrapper around Google Chart API. I cant tell you how much it means to me, a deeply lazy person, to not have to do work. Especially work that seemed peripheral to my main focus.

Third, thank you for basing your engine on JQuery. As a lazy person, I love the way I can do transformational operations on DOM elements by chaining function calls. I'm hardly a JS guru, but JQuery makes me feel like a badass mofo with very little effort on my part. Flot does the same.

Fourth, I really like the way you've made it possible to interact with the data, and zoom in. For better or for worse, I've decided to write my own monitoring system, and this is the kind of functionality that gave me migraines, because it is essential to what I'm trying to do, but very time consuming to get right.

I decided to write my own monitoring system because the current monitoring system we had in place made the silly assumption that software doesn't migrate. Actually this assumption isn't all that silly, we just happen to move services around from machine to machine, and any system that assumed that services were somewhat statically located (again, a valid assumption in most cases) was S.O.L in our world. Also, by being smart about when to send data across, a custom monitoring system ends up scaling much better than traditional SNMP based approaches.

Ahem. Anyways, as the sole implementor of this system, I've been fielding a lot of complaints from both the business team and the devs about the quality of the graphs, the lack of preciseness, the lack of interactivity. Being able to receive events on mouseovers and clicks lets me, as Peter Frampton would say, "come alive". It's always nice to be able to exceed expectations.

Finally, Ole and other makers of Flot, thanks for making me look like a UI wizard, even though I'm purely a server side guy. The fact that I was able to change out graphing engines in a couple of hours says a lot about how easy Flot is to use. You guys made a total pain in the ass operation a lot of fun. Flot is a great tool, and I look forward to using it more in my app.

Flot resources:

Marathon, Man!

2009-01-30T21:11:00.000-08:00

Two weeks ago I turned 40. Usually, I get depressed around my birthdays, but I'm 40, and my knees and back actually feel pretty decent. I'm running 12 miles on my long runs, and while I'm not moving too fast, I am still moving.

So, I'm thinking its January, I'm already doing 12 miles, and it feels decent. Maybe its time to step up and see if I can actually do a Marathon...like in May or June. What better way to celebrate turning 40?

I'm going to sleep on it. Actually I'm going to see how the next couple of runs go. I'm stretching to 13, 14 miles over the next couple of weeks, and if those feel decent, I'll start to work in some speed.

Bulk Resource Uploads via ActiveResource

2008-12-30T10:15:00.000-08:00

Background:
I recently had to reduce the across the wire trips for the monitoring app I had hastily thrown together because the amount of time spent making trips serializing and deserializing individual resources was beginning to affect monitoring performance. The Second Fallacy of Distributed Computing was beginning to rear it's ugly Putinesque head.

I knew that this was coming, but premature optimization has never worked out for me, so I went with the default ActiveResource approach -- everything is a resource, and a CRUD operation on a resource maps to the corresponding http 'verb' -- until smoke started pouring out of my servers.

My basic requirements:

Create a web service that can store data for hundreds of individual datapoints at 5 minute intervals.
Those datapoints can come and go.
The implementor of the statistics gathering code really doesn't need to know the by the wire details of how their data is getting to my web service.

Implied in these requirements is the need for efficiency:

I shouldn't have to perform individual CRUD ops on each statistic every 5 minutes.
I shouldn't have to make an over the wire request for data every time I want to read that data.

From those implications I arrived at the following distilled technical requirements:

I need to bulk upload statistics, and create/update them in one transaction in order to reduce the need for individual CRUD ops. At this point I'm going to choose to 'fail fast', aborting if a single create/update fails, so that I know if something is wrong.
I need to keep a client side cache of those statistics around, only updating them when they've changed (important aside: because this is a monitoring application, it is assumed that each statistic belongs to a single client, so there is no need for out of band updates).

The Juicy Bits
I'd love to go into a long digression about how I explored every which way to do this, but I'll summarize by saying that my final solution had the following advantages:

Uses the existing ActiveResource custom method infrastructure
No custom routes need to be defined
Complexity hidden from the user, restricted to client side upload_statistics call and server side POST handler method.
The priesthood of High REST will not need to crucify me at the side of the road.

ActiveResource extension:

I needed to extend default ActiveResource. By default, AR is not aware of data model relationships. For example, invoking the to_xml method on an AR class only shows it's attributes, even if you specify other classes to include, like this:

ARDerivedClass.to_xml(:include=>[childClass])

This limitation makes being smart about bulk updates pretty hard. I needed to introduce the notion of a client side cache, initialized and synchronized as needed.

My data model looks roughly like this:

Monitor=>has many=>Statistics

The default AR implementation of this looks like


class Statistic <>

I've extended as follows:

implemented an add_statistic method to Monitor that caches Statistic objects locally
Added an upload_statistics method to the Monitor that serializes the client local statistics and then sends them to the server.
modified the default POST handler for Statistic objects to handle bulk creates/updates.
initially loaded the statistics cache on the client side.
lazy synced the cache to the server side, updating on find and delete requests.

Client and Server code by Operation

I want to point out a couple of things in this code:

(1) Cache loading is done in Monitor.initialize(). That way it gets called whether the client is retrieving or creating a Monitor.

  def initialize(attributes = {}, logger = Logger.new(STDOUT))
if(@@logger == nil)
 @@logger = logger
end

@statistics = {}

if(attributes["statistics"] != nil)
 attributes["statistics"].each do | single_stat_attributes|
   @@logger.debug("loading #{single_stat_attributes["name"]}")
   @statistics[single_stat_attributes["name"]] = Statistic.new(single_stat_attributes)
 end

end
super(attributes)
end

This required the following modification on the Monitor controller (server) side:


def index

if(params[:name] == nil)
@monitor_instances = Monitor.find(:all)
else
@monitor_instances = Monitor.find_all_by_name(params[:name])
end

respond_to do |format|
format.html  #index.html.erb
format.xml  { render :xml => @monitors.to_xml(:include=>[:statistics]) }
           format.json  { render :json => @monitors.to_json(:include=>[:statistics])}
      end
end

I needed to make sure that returned monitor instances included child statistics in order to load the client side cache.
(2) get_statistic and delete_statistic synchronize with the server side.
(3) I've added a new upload_statistics method. I wanted to override save, but what I found at runtime is that the ActiveResource.save method calls update, which loads statistics as attributes. This wont work for us because some of those attributes may not exist on the server side, so an 'update' operation is invalid. In upload_statistics, a custom AR method posts the client side cache of statistics to the StatisticsController on the server side:

def upload_statistics


if(@statistics.length > 0)
data = @statistics.to_xml
self.post(:statistics,{:bulk=>"true"},data)
end

end

Note that the first parameter is the method name, the second is the param options, and the third is the actual post data (that contains the serialized client side map of the statistics. The actual path that this POST gets sent to is /monitor_instances/:id/statistics.xml

In the server, I do not have to add/create any new routes, but I do need to make sure that the default POST handler checks for the bulk parameter and handles accordingly.




# POST /statistics
# POST /statistics.xml
def create

if(params[:bulk] == nil)
# handle a single update
else
#handle a bulk update
end
end

Marshalling and Saving stats on the Client side.

In the StatisticsController,create handler, I need to unmarshall the xml into statistics. There are these instructions to extend ActiveRecord via the standard lib/extensions.rb mechanism, but they won't work for me because I'm serializing a hash, not an array of Statistic objects. So I need to deserialize and create/update objects by 'hand', which actually isn't that hard:




cmd = request.raw_post
monitor_instance = MonitorInstance.find(params[:monitor_instance_id])
logger.debug(cmd)
hash =  Hash.from_xml(cmd)

       hash["hash"].values.each do | options |
stat = Statistic.find(:first,
:conditions=>["monitor_instance_id = #{params[:monitor_instance_id]} and name = '#{options["name"]}'"])

if(stat == nil)
 #create a new Statistic object
else
 # update existing statistic object
end
end

respond_to do |format|
statistics = Statistic.find(:all,
:conditions=>["monitor_instance_id = #{params[:monitor_instance_id]}"])
format.xml  { render :xml => statistics.to_xml, :status => :created, :location => monitor_instance_path(@monitor_instance) }
end

In the code above, I deserialize the xml payload using Hash.from_xml, which creates a hash around the hash encoded in the xml data.

To get to the original hash of statistics options, I had to extract them from the encoded hash:

hash = Hash.from_xml(cmd)
hash["hash"].values.each do | options |
# create / update the stat that corresponds to options["name"] under the monitor
end

Summary
This took a lot longer than expected, because I ran into issues with trying to use standard methods, i.e. save, that I still don't understand. However, I know a lot more about AR and how to extend it to do more intelligent sub resource handling.

Best advertainment webisode ever.

2008-12-21T08:48:00.001-08:00

This one made me laugh so hard I pulled something.

http://bewareofthedoghouse.com/VideoPage.aspx

If this is the future of ads, I'm hooked!

Sinatra, my new favorite prototype playground

2008-12-18T20:04:00.000-08:00

About a week ago I was trying to get something ready for the first annual Evri Hack-a-thon, a concentrated 2 day affair where we focused on putting together cool apps with the new Evri API . The event was a blast, I for one rediscovered how fun writing code for code's sake really is.

I was implementing a 'music browser' mostly in javascript, and needed a proxy server to make calls out to those services that didn't have JSONP support.

slight digression here. JSONP is the coolest thing since sliced bread. I say that as someone who loves bread, even more so when it is sliced. The ability to retrieve data w/o a backend is so powerful I _almost_ understand why it's been seen as a Terrible, Horrible, No Good Hack. But not really, because it makes life as a developer so much easier.

I wanted to spend most of my time in the JavaScript, not futzing with the backend server. Because I've been mostly coding in Ruby for the last year, that ruled out rolling up a quick Java Servlet -- I didn't want to spend any time installing Tomcat/Jetty and associated jars, and having to remember how that world worked. I also didn't want to write a Rails app -- seemed ridiculous when I didn't have a data model.

I looked around at a couple of lightweight Ruby Frameworks, like Camping and Merb. Camping would have required me to down version to 1.8.5, and Merb overwhelmed me with the volume of configuration choices. In other words,my ideal proxy server had to be stone cold simple because I simply didn't have the time for anything else.

Enter Sinatra. Elegant, concise, and witty, just like it's namesake. Here is how you configure a path to /json/getjswidgets in Sinatra:

get "/json/getjswidgets" do
cb = params[:callback]
href = params[:href]
...
end

A couple of things to note in the example above:
(1) params are retrieved with the params hash, just like in Rails. So this method was actually called as:

/json.getjswidgets?callback={temp callback name}&href={some value}

(2) all paths are handled with the same 'get...do...end' syntax. It's that simple.

Another example:

get "/json/artists/:name/album" do
cb = params[:callback]
name = params[:name]
....
end

Note that the name parameter is embedded in the path, just like you do in the standard routes.rb file in rails.

Once you get past the path routing (which takes about as long as it does to read this sentence), Sinatra continues to be blissfully easy by allowing you to render the view via erb, builder, haml, and sass. You can render the view inline, or modularize it by putting the files in a view directory.

Helper methods are defined in a helpers block:

helpers do
def helper_method
...
end

...
end

Static assets are kept in a public directory -- again, Sinatra takes a "if it ain't broke..." approach that really minimizes the learning curve. Normally, I loathe the whole "But Ours Go To Eleven!" mindset that I see in frameworks because it means that I have to once again learn another unique set of concepts to get anything done. Sinatra does the exact opposite in leveraging a well known, well used, well understood set of conventions/concepts from Rails while stripping the concept of a framework down to that which is as simple as possible, but not simpler. Sinatra, you're my new BFF!

Converting to a Single Speed

2008-12-07T07:16:00.000-08:00

Why Single Speed? A combination of 'luck' and timing have led me to re-rig my commuter bike as a single speed. The 'luck' part was a pulley on my circa 1995 rear derailleur exploding. The timing part is the rise of single speeds in general. I've been noticing the rise in single speeds in the last couple of years as a bike commuter. They look so....simple and maintenance free!

I'm riding a 14 year old Kona Kula, once my singletrack steed, now my urban commuting stalwart. The key thing about converting a bike with vertical dropouts into a single speed is that you can't slide the rear wheel back and forth to get the perfect chain tension. You need a chain tensioner. A chain tensioner is like a derraileur-lite that pulls the chain taut. There are several brands out there, all of which make a simple, bullet proof device.

The other key thing about converting a standard bike into a singlespeed is what to do with your rear cluster. There are a number of freehub to singlespeed conversion kits out there that provide spacers and cogs to replace your freewheel.

I ended up choosing the Forte Singlespeed conversion kit, made by the Performance Bicycle house brand. This was the only brand I found out there that offered the freehub spacers and cogs, as well as the chain tensioner, for by far the cheapest price -- for $25 I got everything, including 3 cogs to experiment with. Compare that to the Surly solution, which was going to cost $50 for the chain tensioner, and $30 for the spacers, and $10+ for the cog.

It also had what I considered to be a key feature: it allowed me to adjust the horizontal placement of the tensioner. This was important because I had no idea where I would be placing the cog to line up with the chainring.

I also wanted to try using an original cog and chainring, since I had replaced them a year ago and they weren't completely beat down yet. The preferred way to go is to do a clean replacement, but that would require a new chain and front chainring, and I wasn't sure that I could find a replacement front chainring without a special order.

Installation was easy, and gave me a chance to clean my bike for the first time in 6 years!

Tools required:

a cassette removal tool and chainwhip for freewheel removal.
an allen wrench for the usual.
a crank puller to remove the inner chainring on the triple.
a chain tool to break and resize the chain.

Step 1: remove the chainrings. The only way to get the inner chainring out is to remove the crank from the bike. The optimal position for the new chainring is in the middle position of the triple crank. But the chainring I wanted was 44 tooth and too big to use in the middle position -- it rubbed the chainstay -- so I had to keep it in the outer position:

This is where the horizontal adjustablilty of the Forte chain tensioner became really useful. It let me slide the tensioner cog over to the outside with an allen wrench.

Step 2: remove the freewheel using the chainwhip and the freewheel tool.
Step 3: install the chain tensioner where the real derailleur used to be.
Step 4: position the singlespeed cog -- using spacers to fill up the freehub around that cog -- and the chain tensioner cog so that they are inline with the chainring. This is important. If you don't line things up, the chain will derail. In the picture below, note the spacers around the cog. Because I installed my chainring on the outermost position, I've had to position the cog at the outer end of the freehub (with only one spacer between it and the cassette lockring).

Step 5: whip out the chain tool resize the chain so that the chain tensioner is engaged (i.e. it has tension).

I used a 16 tooth cog from my old freewheel, and my existing chain. This may not work because that cog was designed to be 'shiftable', and the ramps on the cog body may derail the chain. However, I wanted to give this a try before buying a new chain and front chainring.

Basic Auth over HTTP using Ruby, Net::HTTP

2008-11-26T13:28:00.000-08:00

I'm writing this one down because it took way too long for me to stumble around it. The Net::HTTP class provides http transport level page access. Most of the time I use open-uri, which treats web pages like files, because that is, as the kids say, one hella fine way to roll.

Too bad it doesn't work with Basic Auth.

I've got a service at http://db-import that listens on 8080. It requires valid credentials. I want to get some type data from it and parse it with Hpricot. Normally I would do this like so:

doc = open("http://db-import:8080/rest/entityTypes.xml) { |f| Hpricot(f) }

however, the requirement of basic auth makes me do this:

ENTITY_TYPES_REQUEST = "http://db-import:8080/importapi/rest/entityTypes.xml"
.....
uri = URI.parse(ENTITY_TYPES_REQUEST)
Net::HTTP.start(uri.host,uri.port) do |http|
req = Net::HTTP::Get.new(uri.path)
req.basic_auth user,pass
response = http.request(req)
end

doc = Hpricot(response.body)

A couple of things to note (that got me):

I needed to specify the hostname w/o the transport. Instead of "http://db-import", specify "db-import". Yeah, that's kind of obvious after the fact :). I URI
HTTP.start only opens the connection, the user then makes all requests/process all responses within the connection block. So in the code above I first configure the request object with basic auth and then use it to make the request.

Not terribly hard, but I do tend to trip up on details and wanted to spare some pain the next time around.

I dont like running (data) naked

2008-11-24T21:27:00.000-08:00

Yesterday morning, at 6:45, I was on semi autopilot, stepping out the door for my morning run. I grabbed my trusty Garmin 305, walked out the door, and hit the on button. And waited. And tried again. I figured my gloves were a little too thick, so I took one off and then pressed again. And pressed harder. Nothing.
After almost 2 years of pretty much day in day out use, in rain, wind, and snow, through heat and cold, thick and thin, my little friend had left the building. Operating on pure reflex, I plugged it back into the recharging cradle and went back outside, bereft.

This was a truly sad moment for me. There I was, in the dark and cold, trying to get excited about going running without second by second updates on heart rate, pace, altitude, and distance covered. At that moment I realized that I was being ridiculous, even diva-like. I mean, wasn't running for running's sake not enough? Would I have even had this mental conversation 2 years ago?

Well....no. Not at 0-dark-45 in the morning. When I'd rather be in bed, warm and comfortable, dozing in and out of consciousness. Instead, I'm standing in a slight drizzle with my headlight strapped on, bundled up from head to toe in waterproof yet breathable and oh-so-reflective winter running gear. It would be different if, say, I was running on the beach at Kauai, wearing shorts and a t-shirt. I dont think I would need motivation coming from my wrist-top computer.

Then again, maybe I would. I mean the coolest thing about the GPS/HRM is that it tells a story, of where I've been and - literally - what I've done. It tells a story and then persists it, for later recall. When I upload my run to the computer, I get to see how ~~slow~~ fast I went, the hills on the route, the overall distance, and I get to remember how I felt at specific points in the run. And if I don't remember, my heart rate tells me. It's sort of like a data photo album, where the mix of lat/long, altitude, and heart rate combine to give me a snapshot of how I felt at every point in the run.

I took off on the run anyway, shamed by my dependence on data, determined to experience 'pure' running without instrumentation. And I actually did. I couldn't refer to my data feed, so I started to pay attention to my form, my breathing, my stride, my forward lean. I knew the mileage of the route I was running (6.23 to be exact), but didn't know exactly how far I had gone, or how far I had left. And although I knew that I was somewhere between 125 and 145 bpm, I had to pace myself by how I felt at that moment, not how my watch was telling me how I felt.

So, yeah, I enjoyed it, a little. And I was actually resigned to a month of 'naked' running while I sent my little buddy back to Garmin to be refurbed. It is, after all, the middle of winter, and I'm not training for anything in particular, more doing long runs to justify eating all those XMas sugar cookies.

I had just convinced myself that this whole zen running thing was good, really good. But when I went down to the garage to pack the HRM up so I could ship it back to Garmin for refurb I noticed that it was on, telling me that it was fully charged. Slowly, disbelieving, I turned it back on, and watched it search fruitlessly for a satellite connection. "Are you indoors?" it asked me. It seemed a little irritated. I turned it off, put it back in the cradle, and went back upstairs -- all of a sudden tomorrows early morning run is looking a lot more fun.

Notes from the (Javascript) Noob: conditionally enable console debugging

2008-11-18T11:26:00.001-08:00

Today I ran into a problem when the primary user of my monitoring app wanted to know why graphs werent rendering for him. I checked the site from my machine and all looked good. I checked the site from another devs machine, and again, everything was rendering. At this point I was confused.

I knew it had to be something in the javascript rendering, so I had the user install firebug. Instead of a JS error (or 10), the page loaded fine. Hmmm. I then wanted to see if the requests I was firing from the page to the Google Charts API were actually going through. We tabbed to the FB net tab, which was disabled. When I had him enable that, plus the console, the graphs rendered.

Doh! I was using console.log to check a value, and forgot that not everyone in the known universe runs with FB enabled. In order to continue to log, I've done this:

function log(str) {
var c = window.console; if (c) {
console.log(str);
}
}

I'm kind of surprised that (a) I wasn't getting a 'console object not defined' in the naive install of FB (which evaluated JS, but had console/net logging turned off), and that (b) if console was present as implied by (a), that logging would degrade gracefully. But the code above works, and I'll take that over sheer speculation.

Kiran and Leela and Pork n Beans

2008-11-09T21:26:00.000-08:00

This morning, hopped up on (whole wheat) pancakes and (lite) syrup, the kids and I rocked out to Weezer. In this age of Rock Band and Guitar Hero, it might seem lame to jam with tennis rackets, but we're old school. Kiran, Leela, consider yourselves blackmailed :)

I think I've got a Soccer Problem

2008-11-07T22:56:00.000-08:00

When I was seven, all I liked to do was read. Read read read. My mom was and is a very wise woman and decided that being a wimpy, nerdy bookworm was the fast track to many beatdowns, and signed me up for AYSO soccer.

I hated the first season, didn't really understand what the hell was going on, and wanted to quit. I'm not sure why I didn't, but by the end of the second season I really loved the game. I loved the smell of the field, the oranges at halftime, and the feeling of being part of something bigger than just me. I loved playing, touching the ball, and would dribble and shoot on an imaginary goal framed by trees for hours and hours after school.

Note that love doesn't imply ability. I'm not overly coordinated, and that, coupled with a serious vision problem (brought on by all that reading), and my reluctance to wear glasses on the field, washed me out of soccer by high school. I really missed playing and got back into it when I turned 30.

People that play soccer when they're older tend to fall into two camps. There are the ex college/high school studs/studettes, who have amazing touch and vision and ability. They know exactly where they are, where everyone else is, and what is going to happen next. Then there are the rest of us, hacks who occasionally get a good touch or light up a good run and feel that all too brief moment of being connected to the worlds most amazing game.

I'm a spaz, occasionally doing something nice, sometimes having great games, sometimes having terrible games, most of the time having randomly great and terrible moments in the same game. My only real gifts are speed and endurance, both of which are slowly disappearing as I get older. I can pass OK, and have decent field vision at times, but my first touch is more accidental than deliberate, I have no air game, and I have a pathetically wimpy shot.

I've been on the same team for about six years. It's a great group of men and women, most of whom are much better than I am, and very patient. One thing I've noticed over the years is that we've started to focus less on the actual games and more on the beers after the game. Its just as fun to give each other crap after the game as it is to play. Sometimes more fun.

Every season I swear it will be my last. In tonights game I was trying to move the ball across the field with a defender at my hip. I tried to reverse on him when all of a sudden I found myself flat on the ground with a really bad calf cramp. I made it clear to the ref that the defender had nothing to do with me ending up on the ground, and limped off the field to enjoy the rest of the game as a spectator. I don't know why my body chose that moment to betray me, but it was enough to end my night.

I'm not sure why I keep coming back. As mentioned above, my speed is no longer keeping me in the game. Seattles dirt fields play havoc on my knees and ankles. Pacific Northwest weather in the late fall/early spring is the opposite of warm and dry. Guys in their 20s are starting to burn by me, making me feel like a slow old man. And lately, more often than not, I find myself in the middle of the game with no anticipation, consistently a 1/2 second too late to the ball, and (even though I now see 20/15 thanks to lasik), completely tunnel visioned.

But there are those moments, really brief ones, where occasionally I get a glimpse of what it is like to really play the beautiful game. Tonight I got a pass, touched it to my inside, and moved the ball up the field. I could see everything, and it felt like I had all the time in the world. I drew a defender to me and flicked the ball to an open space right in front of my wing, who touched it once and lofted a beautiful high shot over the goalies outstretched arms. It was textbook, it was beautiful, and for that brief moment I was not a spaz, I was a player. It's an elusive high that keeps me coming back looking for more.

I'll take 400mg of ibu and walk off that leg cramp now. It hurts, but I think not playing would hurt worse. Maybe I'll quit next season.

Crontarded (sigh)

2008-11-03T11:15:00.000-08:00

Note to self: sometimes mistakes are painful. Sometimes they are funny. Sometimes they are both, and sometimes they are painful, but funny in retrospect. In any case, the best approach is to document it, so that it _never_happens_again. Here is an email I sent earlier today:

Subject: HI! I'm an idiot!

Hey Adam, you know when you came up to me and told me db-import was getting pegged every 6 hours? And Gil, you know when you were asking me one day around noon why you were handling requests every 1 minute?

well, in the crontab I was running a job like this:

* */6 * * *
which is really a great way of saying: every 6 hours, run this task every 1 minute. You see, I knew that, I just didn't _know_ that.

I've amended the offending crontab entry to :

00 */6 * * *

so that the job can run once and only once, every 6 hours, like God intended.

So, I'm sorry. You both can join the long list of people that (a) should punch me or (b) should get a free beer from me. The way I'm f*cking up today, that line will shortly be stretching around the block.

-- Arun

Those of you keeping score at home will know that the score now reads:
Compilers and OS's (not including windoze): 12,000
Arun: 0

Logrotate: a tale of two config locations

2008-10-30T17:12:00.000-07:00

I was trying to make sure my logfiles didnt grow disproportionately large by rotating them via logrotate.d.

Logrotate has two entry points:

/etc/logrotate.d -- this directory contains files that maintain the config settings of all logfiles you want to rotate.

/etc/logrotate.conf -- this file allows you to specify application specific log rotate settings as well.

I'm writing this down now because I had forgotten that I had already configured logrotate for one of my applications and modified the general config file logrotate.conf. When I tried to simulate log rotation by running with the -d parameter:

logrotate -d /etc/logrotate.conf

I recieved a 'duplicate entry' error, which led me to (re)discover the application config files in /etc/logrotate.d.

In general, I think it's a much better idea to do application level logrotate configuration in /etc/logrotate.d. It keeps files manageable and readable.

Here is a sample logrotate config file:

/var/www/rails/dashboard/current/log/*.log {
weekly # once a week
rotate 10 # keep 10 copies
copytruncate # keep original file handle (but truncate file)
delaycompress # delay compression until next rotation
compress # compress it
notifempty # do nothing if you don't need to
missingok # it's not a bad thing to not have a log file.
}

Generating a public key for those automated remote scripts

2008-10-30T15:49:00.000-07:00

I've been using vlad to deploy apps lately, which makes deployment a breeze. However, for a complicated deploy, I'm usually asked at least 10 times to re-enter my password.

I was originally too lazy to set up a public key, but after doing 10 deploys one day (another story), I reconsidered. After all, typing the same thing over and over again = violates all kinds of fairly logical assertions, like DRY for one.

Here is what I did to set up a public key on my deployment server.

(1) on my client box:
ssh-keygen -t dsa

ssh-keygen will ask for a passphrase. Dont enter one -- that kills the whole point of using a public key to automate ssh/scp actions!

ssh-keygen generates two files in ~/.ssh:
id_rsa -- your private key used in ssh authentication.
id_rsa.pub -- the public key you can spray out to machines you want to copy things to.

Then do the following:

scp ~/.ssh/id_rsa.pub machine:/~/.ssh/authorized_keys2

now your logins should be password (and pain!) free.

Creating Objects on the fly in Ruby

2008-10-21T15:43:00.000-07:00

I'm writing a file parser where I want to plug in different parsing modules depending on the kind of file I need to parse.

In order to do this without having to change code, I'm storing the configuration in a YAML file, like this:

home_page_total_clicks:

id: 2
match_string: "home_page"
measurement: "total_clicks"
clazz_name: "DummyParser"
date_comparision: 1

I'm specifying that for pages that are categorized as 'home_page_total_clicks', that I want to instantiate a class named "DummyParser". I'm thinking that in the future I could allow someone to specify an arbitrary parser to an abitrary file type.

The way Ruby allows you do instantiate classes from specified strings relys on the fact that all classes are constants, that you can retrieve:

clazz = Kernel.const_get(file_process_data.clazz_name)
processor = clazz.new(file_process_data)

and tada! a new processor -- note how I've assumed that a processor takes a file_process_data as an input. This will fail processors that don't have initialize methods that dont take file_process_data.