David Welton: Generating Rails Fixtures From Large Datasets
I've been working with Tomaso Minelli doing some Rails work, and one thing we needed to do was set up some tests. The situation we are in is that we already have a large set of data to work with in our development environment, but getting that data into .yml files is no easy task. Since we have a fairly large number of interdependent tables, dumping one of them alone is pretty useless, unless you manage to get the other tables it links to, and so on and so forth. Dumping all the data would make testing quite slow, and really is too much to be useful. Here's the rough solution we arrived at:
It's written in a somewhat quick and dirty way, but it gets the job done. What it does:
It takes a single class - hopefully a well connected one - fetches a few of them, and goes from there, exploring all the other objects that the initial ones are connected to, and then dumping them in .yml files in the fixtures/ directory. It could certainly use more work:
@fixtures =
@seen =
# Do something with a fixture. In this case, we store it for later.
def handle_fixture(record)
@fixtures[record.class.table_name] = []
@fixtures[record.class.table_name] << record
end
# Get some sort of unique way of identifiying a record - class name and id.
def unique_record(record)
return record.class.to_s + " " + record.id.to_s
end
# Grab all the relations of a class and add them to the queue.
def add_fixture(record)
ur = unique_record(record)
if @seen[ur]
puts "LOOP!"
return
end
@seen[ur] = true
puts "Fixture for # record.class.name - ID # record.id "
handle_fixture(record)
record.class.reflections.each do k, r
begin
if r.macro == :has_many
r.macro == :has_and_belongs_to_many
related = record.send(r.name).send('find', :all, :limit => STARTING_LIMIT)
else
related = record.send(r.name)
end
related = [related] unless related.is_a?(Array)
related.each do related_record
if related_record
puts " New Record # related_record.class.name # related_record.id "
@todo << related_record
end
end
rescue => e
puts " ERROR: # e :\n# e.backtrace.inspect "
end
end
end
@todo = []
namespace :db do
namespace :fixtures do
desc 'Create YAML test fixtures starting from a single model using reflections'
task :modelbase => :environment do
STARTING_CLASS = Foobar
STARTING_FIND = :all
STARTING_LIMIT = 1
tmp = 0
totale = STARTING_CLASS.find(STARTING_FIND, :limit => STARTING_LIMIT).size
STARTING_CLASS.find(STARTING_FIND, :limit => STARTING_LIMIT).each do m
tmp += 1
puts "# tmp su # totale : # m.id "
@todo << m
end
while true
if @todo.length == 0
break
end
add_fixture(@todo.pop)
end
puts "\n\n# '='*80 \nScan Finished\n# '='*80 \n\n"
@fixtures.each do name, fixture
if fixture.size > 0
i = "000"
filename = "# RAILS_ROOT /test/fixtures/# name .yml"
puts "Create # filename for class # fixture.class.name "
FileUtils.mkdir(File.dirname(filename)) unless Dir[File.dirname(filename)].size > 0
File.open("# RAILS_ROOT /test/fixtures/# name .yml", 'w') do file
file.write fixture.inject( ) hash, record
hash["# record.class.table_name _# i.succ! "] = record.attributes
hash
.to_yaml
end
else
puts "Fixture for # name is EMPTY"
end
end
end
end
end
- Additional starter classes, for ones that aren't connected to others.
- Intelligent examination of data and data sets...
- For example you could check to see what classes aren't connected to the others and use them as starter classes.
- If you wanted to waste a lot of cycles, you could also try and aim for a representative set of examples. For an object that has_many something elses, you could try and find one with zero, one with one, and one with a lot, to cover various cases. That could quickly get out of hand for largish data sets though.
- I'm sure there are various other tweaks, knobs, and levers that could be added to work more efficiently. I'd love to see them!
(Part 1 of an ongoing series: shiny things in telepathy-glib)
When I first read 
debian-installer rc1 is going to be officially announced next week.
However, the netboot images are already available. I'm now looking for
people who are interested in testing these installer images on ARM and
MIPS based machines.
If you have a spare ARM or MIPS machine or want to reinstall your
machine for some reason (for example to move from arm to armel), please
see my messages on 
Bash is probably the most common command-line shell in the 
Change is upon us, lets unite and embrace it. Maybe we can make this country a better place and once again be a Role Model to the world of a young ( yes, 221 years young ) Democracy taking its place in the international community for the betterment of all as an Example Nation, not a unnecessarily forceful and arrogant one.
Change is here, and tonight I will sleep well for the first time in ages dreaming of this change.
I’ve flirted with it in the past and resisted it for a while, but you can now follow me on 
Apparently I'm supposed to be blogging once a day, every day, for November. I missed yesterday and doubt I'll have something to say every day, but let's try...
First, let me make one thing clear. This isn't constructive criticism. This is just criticism. It's directed at software that's so wrong-headed that there's no way to make it significantly better, and everyone involved would be much better spending their time doing something else instead of trying to fix any of what I'm about to describe. It's not worth it. Sit in a park or something instead. Meet new and interesting people. Take up a hobby that doesn't involve writing shell scripts for Linux. You'll be happier. I'll be happier. Everyone wins.
