wshaffer

I'm working on a project where we're trying to get a bunch of data that has been kept on internal wiki pages into a database, so that it can be searchable, we can have automatic detection of duplicates, various other stuff.

Part of my contribution to this effort is to get the data off these wiki pages and into CSV files that can be imported into the database. It's a pretty trivial effort if you've got the Ruby gem Nokogiri (which parses HTML and XML files).

Well, it's sort of trivial. So far, about 20% of my time has been spent writing the part of the script that does the real work, and 80% has been spent dealing with oddities caused by unexpected white space, white space that Ruby does not recognize as white space by default ( ), and quirks of people's wiki markup.

My guess is that this is probably par for the course when web scraping.

Also, I wrote documentation for my homebrew hacky script that probably 2 other people besides me are ever gonna use, because that's how I roll.

When I'm done with this project, I'm considering switching from Ruby to Python. I like working in Ruby, but Python is quite literally what all the cool kids are using, since it seems to be the current language of choice for teaching children to program.

I don't remember how I did programming in the days before Google and Stack Overflow. I think I spent a lot of time flipping through books. Or possibly actually memorized language syntax, which seems like a horrible waste of neurons that might be more profitably put to other uses.

However, if you want to get good results out of Google, you've got to know how to search for the right thing. There is a thing that I often want to do in Ruby, where I've got a variable that holds a string, and I want to put the value of that variable into another string. And somehow I always end up searching for something like "ruby string substitution" and getting pages and pages of stuff on the sub method, which is nice but not what I want at all.

So this is a note to myself: the thing you want to do is called "variable interpolation" and it works a little something like this:

thingy = "variable interpolation"
puts ("When you want to put a variable into a string, that's called #{thingy}.")


After a bit of research, I found a much better way to do the file name manipulation I was talking about in my previous post.

Basically, it boils down to:

require "pathname"
input_file = Pathname.new(ARGV[0])
new_base = (input_file.basename(input_file.extname)).to_s + "_counts" + input_file.extname.to_s
output_file = input_file.dirname + Pathname.new(new_base)

I used Pathname instead of File because the documentation suggests that it's more robust at dealing with different file pathing conventions on different OSes.

I'm a little dubious about the dance I had to do there of converting path fragments to strings, concatenating them, and then converting back to a pathname, but trying to concatenate the path fragments directly kept giving me extra / in the path.

A couple of months ago, I started learning to program in Ruby. Largely because I somehow stumbled across Brian Marick's book, Everyday Scripting with Ruby, which poses the questions: "Do you spend too much of your working life copying and pasting or otherwise manually manipulating data? Wouldn't you like to be able to get a computer to do that for you?" To which I answered, "Yes!"

So, I've been slowly working my way through Marick's book, and playing around with ruby koans, and reading bits of why's poignant guide to ruby, which has got to be one of the most extraordinary pieces of technical writing ever committed.

And I've written some little toy scripts, but today I wrote my very first properly useful script. To give a bit of background: For FOGcon, we keep the code for our website backend in a source code repository called Github. We use Github, among other things, to track the bugs and issues that people report in using the database. And one of the things I periodically have to do is get information about issues out of Github and pass it on to people who don't have Github accounts and so don't have the ability to access our issues tracker directly. Up until today, I have been doing this by clicking around in Github and copying and pasting.

But today, I thought, "I should be able to script this." And lo and behold, with a bit of tinkering, I now have a script that logs into Github, downloads all the open issues in a given milestone, and prints out the issue number, title, and description for each one.

Interestingly, the breakdown of how I developed the script went something like this:
~30 minutes: Determining that there is a Github API, that a suitable Ruby wrapper for the API exists, and researching whether this wrapper can access the information I want.
~1 hour 30 minutes: Installing various necessary Ruby 'gems' (a.k.a libraries) and dinking around trying to resolve various dependency issues.
15 minutes: Writing and testing the script.

This was pretty much tinker toy programming, in that the Octokit Ruby library that I used did all the hard work. All I had to know was how to do some basic operations on an array, and how to print out the information. The entire script is 13 lines long.

So, yeah, this decision to learn a bit of Ruby scripting seems to be paying off.