Creating custom Minitest reporters

Minitest it’s having some momentum right now. I’ve been using it on some personal projects and, coming from RSpec, I really missed its awesome execution report. Minitest has some 3rd-party reporters, but they don’t get any close to RSpec’s awesomeness.

So I decided to dive into Minitest source code and learn how to create a custom Minitest reporter similar to RSpec’s reporter. Here’s the final result:

Note that it even gives you the command for running only the test that failed, pretty much like RSpec. Sweet! By the way, if you want to have this reporter too, just install minitest-utils.

It turns out creating custom reporters in Minitest is a really simple task, so I’ll guide you in this article, creating two different reporters.

Understanding Minitest’s plugin system

Minitest’s plugin system relies on file discovery. It will load all files present on your $LOAD_PATH within the minitest/*_plugin.rb pattern. Let’s say I’m creating a gem called minitest-utils. I can create a file at lib/minitest/utils_plugin.rb and Minitest will load it. This file can implement two methods on Minitest module, as you can see below.

module Minitest
  def self.plugin_utils_init(options)
    # ...
  end

  def self.plugin_utils_options(opts, options)
    # ...
  end
end

Your methods have to follow the Minitest.plugin_*_init and Minitest.plugin_*_options naming scheme. Let’s breakdown these methods’ responsibility.

First, if you intend to extend the CLI, you have to implement the Minitest.plugin_*_options method. It receives two parameters containing the OptionParser instance and a hash containing options like the output IO instance. One could extend the CLI including options for displaying colored test output.

module Minitest
  def self.plugin_utils_options(opts, options)
    opts.on('--color', 'Display colored output') do |color|
      options[:color] = color
    end
  end
end

Minitest’s default reporter accepts multiple reporters, implemented as the Minitest::CompositeReporter class. For custom reporters, and other extensions, you have to implement the Minitest.plugin_*_init method, adding your reporter to the composite reporter.

module Minitest
  def self.plugin_utils_init(options)
    Minitest.reporter << GrowlReporter.new(options[:io], options)
  end
end

Now that you know how the plugin system works, let’s create a custom reporter.

Creating a custom reporter

Here’s how Minitest’s default reporter looks like.

Not that useful. So, let’s make it shine with colors.

For this to work, we’ll need to clear out default reporters from the composite reporter. We can do it in the Minitest.plugin_utils_init method.

module Minitest
  def self.plugin_colored_reporter_init(options)
    Minitest.reporter.reporters.clear
    Minitest.reporter << ColoredReporter.new(options[:io], options)
  end
end

As you can see, we’re using a ColoredReporter class. Let’s define it.

class ColoredReporter < Minitest::StatisticsReporter
end

Now, you have to define the methods used for reporting. The first one is ColoredReporter#record, which is called after every test execution. This is the place we’ll output the realtime progress with all those dots and other failing indications.

class ColoredReporter < Minitest::StatisticsReporter
  def record(result)
    super

    result_code = result.result_code
    io.print color(result_code, RESULT_CODE_TO_COLOR[result_code])
  end
end

Note that we’re calling super; we have to do this so that Minitest::StatisticsReporter can compute the result status counting. Now, we have to implement the color method, which is pretty straight-forward.

class ColoredReporter < Minitest::StatisticsReporter
  RESULT_CODE_TO_COLOR = {
    'S' => :yellow,
    '.' => :green,
    'F' => :red,
    'E' => :red
  }

  COLOR_CODE = {
    red: 31,
    green: 32,
    yellow: 33,
    blue: 34,
    none: 0
  }

  def record(result)
    super

    result_code = result.result_code
    io.print color(result_code, RESULT_CODE_TO_COLOR[result_code])
  end

  def color(text, color = :none)
    code = COLOR_CODE[color]
    "\e[#{code}m#{text}\e[0m"
  end
end

Next, we have to implement the ColoredReporter#report method. This is where we show which tests failed, and can also display the statistics of the executed tests. In this case, we’ll basically call some methods.

class ColoredReporter < Minitest::StatisticsReporter
  # ...

  def report
    super

    io.puts "\n\n"
    io.puts statistics
    io.puts aggregated_results
    io.puts summary
  end
end

The ColoredReporter#statistics method will calculate the execution time, and some other info.

class ColoredReporter < Minitest::StatisticsReporter
  # ...

  def statistics
    "Finished in %.6fs, %.4f runs/s, %.4f assertions/s." %
      [total_time, count / total_time, assertions / total_time]
  end
end

The ColoredReporter#aggregated_results method will display the tests that failed or were skipped.

class ColoredReporter < Minitest::StatisticsReporter
  # ...

  def aggregated_results
    filtered_results = results.sort_by {|result| result.skipped? ? 1 : 0 }

    filtered_results.each_with_index.map { |result, i|
      color("\n%3d) %s" % [i+1, result], result.skipped? ? :yellow : :red)
    }.join + "\n"
  end
end

Finally, let’s display the test execution summary, with number of tests, assertions, failures, and more.

class ColoredReporter < Minitest::StatisticsReporter
  # ...

  def summary
    summary = "%d runs, %d assertions, %d failures, %d errors, %d skips" %
      [count, assertions, failures, errors, skips]

    color = :green
    color = :yellow if skips > 0
    color = :red if errors > 0 || failures > 0

    color(summary, color)
  end
end

And we’re done! Now we have more visibility about what’s happening in our tests, as you can see below.

One more example

What about displaying Growl notifications (or any other notification system you may have)? Using the test_notifier gem, this is quite simple.

class TestNotifierReporter < Minitest::StatisticsReporter
  def report
    super

    stats = TestNotifier::Stats.new(:minitest, {
      count: count,
      assertions: assertions,
      failures: failures,
      errors: errors
    })

    TestNotifier.notify(status: stats.status, message: stats.message)
  end
end

You can add it to the composite reporter just like before.

module Minitest
  def self.plugin_colored_reporter_init(options)
    Minitest.reporter.reporters.clear
    Minitest.reporter << ColoredReporter.new(options[:io], options)
    Minitest.reporter << TestNotifierReporter.new
  end
end

On Mac and Growl, this is what you’ll have:

Wrapping up

Minitest is really lightweight and extensible. I never really cared about the assertion syntax (expect vs should vs assert vs must/wont), but RSpec’s feedback made all the difference. It gives you the command for running failing tests and allows you to rapidly view what went wrong. Now that I have the same level of feedback, I’ll probably stick to it. Rails has so many custom test case classes (mailers, jobs, generators, and more) that I can just use it, instead of trying to figure out how to do it with RSpec.