Logrotate: a tale of two config locations

I was trying to make sure my logfiles didnt grow disproportionately large by rotating them via logrotate.d.

Logrotate has two entry points:

/etc/logrotate.d -- this directory contains files that maintain the config settings of all logfiles you want to rotate.

/etc/logrotate.conf -- this file allows you to specify application specific log rotate settings as well.

I'm writing this down now because I had forgotten that I had already configured logrotate for one of my applications and modified the general config file logrotate.conf. When I tried to simulate log rotation by running with the -d parameter:

logrotate -d /etc/logrotate.conf

I recieved a 'duplicate entry' error, which led me to (re)discover the application config files in /etc/logrotate.d.

In general, I think it's a much better idea to do application level logrotate configuration in /etc/logrotate.d. It keeps files manageable and readable.

Here is a sample logrotate config file:

/var/www/rails/dashboard/current/log/*.log {
weekly # once a week
rotate 10 # keep 10 copies
copytruncate # keep original file handle (but truncate file)
delaycompress # delay compression until next rotation
compress # compress it
notifempty # do nothing if you don't need to
missingok # it's not a bad thing to not have a log file.
}

Generating a public key for those automated remote scripts

I've been using vlad to deploy apps lately, which makes deployment a breeze. However, for a complicated deploy, I'm usually asked at least 10 times to re-enter my password.

I was originally too lazy to set up a public key, but after doing 10 deploys one day (another story), I reconsidered. After all, typing the same thing over and over again = violates all kinds of fairly logical assertions, like DRY for one.

Here is what I did to set up a public key on my deployment server.

(1) on my client box:
ssh-keygen -t dsa

ssh-keygen will ask for a passphrase. Dont enter one -- that kills the whole point of using a public key to automate ssh/scp actions!

ssh-keygen generates two files in ~/.ssh:
id_rsa -- your private key used in ssh authentication.
id_rsa.pub -- the public key you can spray out to machines you want to copy things to.

Then do the following:

scp ~/.ssh/id_rsa.pub machine:/~/.ssh/authorized_keys2

now your logins should be password (and pain!) free.

Creating Objects on the fly in Ruby

I'm writing a file parser where I want to plug in different parsing modules depending on the kind of file I need to parse.

In order to do this without having to change code, I'm storing the configuration in a YAML file, like this:

home_page_total_clicks:

id: 2
match_string: "home_page"
measurement: "total_clicks"
clazz_name: "DummyParser"
date_comparision: 1

I'm specifying that for pages that are categorized as 'home_page_total_clicks', that I want to instantiate a class named "DummyParser". I'm thinking that in the future I could allow someone to specify an arbitrary parser to an abitrary file type.

The way Ruby allows you do instantiate classes from specified strings relys on the fact that all classes are constants, that you can retrieve:

clazz = Kernel.const_get(file_process_data.clazz_name)
processor = clazz.new(file_process_data)

and tada! a new processor -- note how I've assumed that a processor takes a file_process_data as an input. This will fail processors that don't have initialize methods that dont take file_process_data.

Swimming Breakthrough this AM

Image via WikipediaSome history: when I was twelve, my mom decided that the one way to 'drown proof' me and my sister was to get us on a swim team. My summers, and then my winters, became filled with laps and yards and intervals. Only one problem. I was a terrible swimmer.

I had the technique of a drowning man (hence the title of my blog), coupled with low body fat -- in the 6% range -- and the upper body of a Kenyan marathoner. Eventually, in high school, I started to dread swimming so much I would actually have nightmares about going to practice. I dropped out to spare myself any more sleepless nights, and proceeded to apply any lung capacity I had built up swimming to inhaling bong hits, which eventually led to more sleepless nights, but that's another story.

Fast forward to now, and I'm quickly closing in on 40. I've been an incredibly average bike racer, run some 1/2 marathons, and while my genetics don't point to world class anything, I do actually enjoy running and biking. So triathlons would seem like a natural next step, especially since soccer is becoming a beer and ibuprofen aided affair, and climbing takes too much time away from the kids right now.

But the thought of swimming, and the indelible imprints of suffering through thousands of yards very slowly and painfully, kept me focused on other things. Until now. I decided to actually take the time to learn how to swim, via Total Immersion. Another positive factor: my body fat has doubled, so I'm not quite the sinker I used to be.

Total Immersion teaches you how to swim better via a series of progressive drills. These drills start out very basic, i.e. you are floating on your back and kicking. They build up from there, but the keys are

• swimming "downhill" by keeping your head down instead of looking forward.
• swimming on your side, and pivoting from side to side.
• driving that pivot from your core
• pushing your chest down into the water because the air in your lungs will help you float.
  
• barely kicking

This was completely counter to the way I swam, which was with my head looking forward, using my arms to drag myself through the water until my shoulders hurt, kicking spasmodically to try and float as effortlessly as the much better swimmers around me.

After a month of working on these drills, I was swimming with much less effort than I ever had before, but I still felt that something was missing. I still felt that I was expending a lot of energy, that it was hard to breathe, and that I was struggling to swim downhill.

After reading and re-reading the drills section of the Total Immersion book, I decided to try using FistGloves. They are what they sound like: rubber gloves that force your hands closed. What that does is dramatically reduce the amount of surface area that you have to work with. The idea is that by reducing your hand area, you will be forced to concentrate on balance as well as stroke.

Again, this is counter to what I had been taught. To work on stroke, our coaches used to give us paddles and pull buoys. The paddles increased the surface area during the pull, increasing the workload on the shoulders. The buoys were used to let us concentrate on pulling. The result was supposed to be increased strength that resulted in increased speed, but I always felt fast until I took the paddles off, and then I felt slow. And heavy, especially since I had to put the buoy away.

This morning was my first go with the FistGloves. I put them on, then pushed off of the wall. I immediately slowed to a snails pace. The bottom of the pool didn't glide by at all, and I felt completely out of balance. I finished the first 25 yards and had to rest.

I pushed off again and adjusted my balance, keeping my head down, focusing on driving from my hips (there was no other way to generate force). To breathe, I couldn't lift my head because I couldn't generate enough force with my arms. I had to roll to the side -- the way that I was supposed to. I had to pick my elbows up high, and drive my fist into the water by my goggles, and continue driving forward with that fist as I pivoted through to float on my side -- again, just like the Total Immersion drills.

Most importantly, I could feel massive changes in the amount of water I was pushing back with my forearms, just by keeping my elbows raised. I noticed -- for the first time ever -- that I dropped my leading elbow when breathing, and that my corresponding 'push' on the water completely disappeared. Removing my hands from the equation made any and all deficiencies completely stand out.

Maybe 300 yards later, I almost felt like I was moving normally. So I took the gloves off and pushed off. WOW. It was like someone had strapped dinner plates on my hands. I was gliding. Effortlessly. Flip turns, which I normally loathe because they require me to hold my breath, were fun because I was flying into the wall. Bilateral breathing, which used to destroy my form, became easy once I rolled to my side instead of of lifting my head. I concentrated on keeping my elbows high both in and out of the water, and extending my leading hand out for as long as I could.

This is the biggest breakthrough I've had swimming, ever. It was definitely one of those flow experiences. I'm having fun, in the pool, which 25 years ago was more of a source of torture than anything else. Now I'm actually excited to do a triathlon because I'll enjoy all 3 legs. I can't wait to get back in the pool!

rake and vlad and mod_rails

I'm using Vlad to deploy my application, because it's so damn simple. Actually, I tried Capistrano, which also seemed pretty simple, until it just didn't work.

I'm not entirely sure why, and I didn't delve into the details, because a friend walked by and said "Capistrano? Really? All the Kool Kidz are using Vlad. Don't be a dork!". Actually, he said "life is too short to read documentation. Vlad only requires a couple of variables in your deploy.rb, and no capify. And it works."

The one thing I _do_ like about vlad is that it builds on top of rake, which means there is one less thing for me to keep track of.

Here are my notes about using Vlad with mod_rails:

(0) my deploy.rb contains the essentials:

set :application, "myapp"
set :domain, "machinename.foo.corp"
set :deploy_to, " /var/www/deploylocation"
set :repository, "http://svn/svn/proj_name/trunk/"
set :user, "arun"

(1) mod_rails detects restarts by scanning RAILS_ROOT/tmp/restart.txt. So a restart is super simple:
namespace :vlad do
remote_task :restart do
run "touch #{deploy_to}/current/tmp/restart.txt"
end
end

Note there is no vlad:restart, and when I had added this in, mod_rails support wasn't in mainline vlad.
(2) running remote tasks is insanely trivial:

remote_task :link_dirs do

run("ln -sf #{deploy_to}/data #{deploy_to}/current/public/data")

run("ln -sf #{deploy_to}/graphs #{deploy_to}/current/public/graphs")

run("ln -sf #{deploy_to}/current/doc #{deploy_to}/current/public/doc")
end

remote_task :chmod_cron do
run "chmod 4755 #{deploy_to}/current/sh/deploy_monitor.sh"
end

(3) sometimes, I've got to just run one of these. Usually, I've got to run all of them, in addition to the standard vlad update and migrate tasks. Rake specifies ordered dependencies like this:

task :full_vlad => ['vlad:update','vlad:migrate','vlad:link_dirs','vlad:chmod_cron','vlad:restart1']

Note the way that the vlad namespace is specified in the dependency array.

Google Charts API

I've been putting together a monitoring app using Rails as the app framework and RRD as the data storage / graphing engine. Early on, I was the one driving a lot of the requirements. Now that the app is in production, I'm getting a lot of reasonable requests that I didn't think of.

One of these is to display some specific data in a pie chart. For some of the statistics I was collecting, a pie chart made much more sense than the standard 1..n dataset in a time series RRD graph. When I first heard the requirement, my instinct was to use one of the well known ruby graphing engines, like Gruff or Scruffy. Scruffy, in particular, looks really useful. However I wanted to get a quick prototype out in front of my users to make sure they really wanted a pie graph, and for that the Google Chart API was the tool for the job. By reducing graph construction to an API call, Google makes it this easy:



basically, I've embedded the following call in an image tag:

http://chart.apis.google.com/chart?cht=chart_type&chd=t:data1,data2,..,dataN&chs=widthxheight&chl=label1|label2|..|labelN

While I may end up switching this out for a more full featured server side graphing engine -- the one at PullMonkey looks pretty sweet even if it does require flash -- the point is that I'm leveraging an API to get my app off the ground very quickly. Life is good when app construction is this fast.

Dealing with HTTP Timeouts in Ruby

The standard ruby open-uri library makes connecting to local and remote resources completely transparent, which is great most of the time, it lets me do things like this:

stats = Hpricot(open(interface_url))

and I can easily replace the url with a file at unit test time.

However, yesterday I was trying to consume some XML from a web service that is on a machine that is...pegged, like pegged with a load of 30, because it's a data mining box in the middle of a huge run. Also, producing the XML requires database queries, which exacerbates the load problem. So it took about 2 minutes to get back to me -- 1 minute more than the standard open-uri timeout.

Open-uri calls Net:HTTP to do it's HTTP protocol based file opens, and Net::HTTP is a lot more like a standard HTTP library in another language -- i.e. it lets you set timeouts. Unfortunately with flexibility comes some complexity, but it's nothing worth crying about:

@site_url = URI.parse(site_url)

http = Net::HTTP.new(@site_url.host, @site_url.port)
http.read_timeout=360 # timeout in seconds. Yeah, that's 6 minutes.
req = Net::HTTP::Get.new @site_url.path

res = http.start { |web|
web.request(req)
}

stats = Hpricot(res.body)

Of course, real production code would wrap this in a begin..rescue block and retry.

Moving from Wordpress

So I moved from wordpress. Mainly so I could take advantage of the Evri Widget. It's kind of sad to have worked on something so hard and not be able to take advantage of a widget that showcases what we do at Evri.

I've found blogging to be a great way to file away information that I have found useful in the past -- mostly technical stuff. I also use it to document my less embarrassing (the real embarrassing ones will just have to live on in my mind) experiences and aspirations. Lately life has been moving so fast that documenting it -- even if in snippets -- is much more useful than sitting around repeating history. Or, to use my favorite Joe Biden quote from last night: "Past is Prologue".