Jekyll: Custom Liquid Tags

4th Dec 2009 | Tags: blog

The base install of Jekyll at the moment doesn’t let you run any arbitrary ruby code. This is so that they can use it for github pages and not need to worry about making a super-secure sandbox just to generate some HTML.

Unfortunately, that means we’re out of luck for creating custom liquid filters. The most annoying deficiency for me is tags. The way the default liquid map filter works isn’t friendly with @site.tags, so to generate my Tags page I had to do some really crazy stuff with capture:

<div id="articles">
  <table>
    
      <tr><th>blog</th>
          <th><a name="blog" class="anchor">&nbsp;</th></tr>
      
      <tr><th>coding</th>
          <th><a name="coding" class="anchor">&nbsp;</th></tr>
      
      <tr><th>git</th>
          <th><a name="git" class="anchor">&nbsp;</th></tr>
      
      <tr><th>ruby</th>
          <th><a name="ruby" class="anchor">&nbsp;</th></tr>
      
      <tr><th>rails</th>
          <th><a name="rails" class="anchor">&nbsp;</th></tr>
      
      <tr><th>programming</th>
          <th><a name="programming" class="anchor">&nbsp;</th></tr>
      
      <tr><th>sql</th>
          <th><a name="sql" class="anchor">&nbsp;</th></tr>
      
      <tr><th>hoptoad</th>
          <th><a name="hoptoad" class="anchor">&nbsp;</th></tr>
      
      <tr><th>merb</th>
          <th><a name="merb" class="anchor">&nbsp;</th></tr>
      
      <tr><th>linux</th>
          <th><a name="linux" class="anchor">&nbsp;</th></tr>
      
      <tr><th>osx</th>
          <th><a name="osx" class="anchor">&nbsp;</th></tr>
      
      <tr><th>clojure</th>
          <th><a name="clojure" class="anchor">&nbsp;</th></tr>
      
      <tr><th>sicp</th>
          <th><a name="sicp" class="anchor">&nbsp;</th></tr>
      
      <tr><th>specs</th>
          <th><a name="specs" class="anchor">&nbsp;</th></tr>
      
      <tr><th>datamapper</th>
          <th><a name="datamapper" class="anchor">&nbsp;</th></tr>
      
      <tr><th>snippet</th>
          <th><a name="snippet" class="anchor">&nbsp;</th></tr>
      
      <tr><th>holmes</th>
          <th><a name="holmes" class="anchor">&nbsp;</th></tr>
      
      <tr><th>seaside</th>
          <th><a name="seaside" class="anchor">&nbsp;</th></tr>
      
      <tr><th>camping</th>
          <th><a name="camping" class="anchor">&nbsp;</th></tr>
      
      <tr><th>Rails</th>
          <th><a name="Rails" class="anchor">&nbsp;</th></tr>
      
      <tr><th>code</th>
          <th><a name="code" class="anchor">&nbsp;</th></tr>
      
      <tr><th>education</th>
          <th><a name="education" class="anchor">&nbsp;</th></tr>
      
      <tr><th>tech</th>
          <th><a name="tech" class="anchor">&nbsp;</th></tr>
      
      <tr><th>commandline</th>
          <th><a name="commandline" class="anchor">&nbsp;</th></tr>
      
      <tr><th>gaming</th>
          <th><a name="gaming" class="anchor">&nbsp;</th></tr>
      
      <tr><th>management</th>
          <th><a name="management" class="anchor">&nbsp;</th></tr>
      
      <tr><th>web</th>
          <th><a name="web" class="anchor">&nbsp;</th></tr>
      
      <tr><th>Ruby</th>
          <th><a name="Ruby" class="anchor">&nbsp;</th></tr>
      
      <tr><th>boardgame</th>
          <th><a name="boardgame" class="anchor">&nbsp;</th></tr>
      
      <tr><th>wip</th>
          <th><a name="wip" class="anchor">&nbsp;</th></tr>
      
      <tr><th>notes</th>
          <th><a name="notes" class="anchor">&nbsp;</th></tr>
      
  </table>
</div>

Fortunately, it wasn’t to hard to make a fork, and in my fork I added a super simple code loading option. Now, I can add a quick extension in _lib/filters.rb like so:

module Jekyll
  module Filters
    def keys(input)
      input.keys
    end

    def tagged(input, tag)
      input.select{|post| post.tags.include? tag}
    end
  end
end

Now tags.html looks like this:

<div id="articles">
  <table>
    
      <tr><th>blog<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />

<title>Ruby HTTP Clients in 2025 - set_trace_func</title>

<!-- Meta -->
<meta name="description" content="Just a random coder writing about technology, web development, and various other geeky topics." />
<meta name="author" content="Jamie Macey">
<link rel="alternate" type="application/atom+xml" title="Atom Feed" href="/index.xml" />

<!-- Favicons -->
<!-- 
<link rel="apple-touch-icon" href="/docs/5.2/assets/img/favicons/apple-touch-icon.png" sizes="180x180">
<link rel="icon" href="/docs/5.2/assets/img/favicons/favicon-32x32.png" sizes="32x32" type="image/png">
<link rel="icon" href="/docs/5.2/assets/img/favicons/favicon-16x16.png" sizes="16x16" type="image/png">
<link rel="manifest" href="/docs/5.2/assets/img/favicons/manifest.json">
<link rel="mask-icon" href="/docs/5.2/assets/img/favicons/safari-pinned-tab.svg" color="#712cf9">
<link rel="icon" href="/docs/5.2/assets/img/favicons/favicon.ico">
-->

<!-- Style -->
<link rel="stylesheet" href="/_bridgetown/static/index.BESXTBS6.css" />

<!-- Script -->
<!-- <script src="/_bridgetown/static/index.FRS7NT6W.js" defer></script> -->


    <link type="application/atom+xml" rel="alternate" href="https://blog.tracefunc.com/index.xml" title="set_trace_func" />
  </head>
  <body class="post ">
    <header>
  <nav>
    <ul>
      <li><h1><a href="/">set_trace_func</a></h1></li>
      <!-- The following float right, and are defined right-to-left -->
      <li class="float-right"><a href="/notes/">Notes</a></li>
      <li class="float-right"><a href="/projects/">Projects</a></li>
      <li class="float-right"><a href="/tags/">Tags</a></li>
      <li class="float-right"><a href="/archive/">Archive</a></li>
    </ul>
  </nav>
</header>


    <main>
      <aside>


<div>
  <h4>About</h4>
  <picture class="float-end w-50 rounded-circle">
    <source srcset="/assets/memoji.webp" type="image/webp">
    <source srcset="/assets/memoji.jpeg" type="image/jpeg">
    <img src="/assets/memoji.png" alt="Avatar of Author" loading="lazy" class="w-100">
  </picture>

  <p>Jamie Macey is a senior software engineer with over 15 years experience
            in the Ruby and Rails ecosystems, largely on the back-end.</p>
  <p class="mb-0">Husband, father, gamer, and all-around geek. Ask about my latest
            3d print, or toy software project.</p>
</div>

<div>
  <h4>Elsewhere</h4>
  <ol class="list-unstyled">
    <li>
      <a href="https://github.com/jamie/"">GitHub</a>
    </li>
    <li style="display: none;">
      <a rel="me" href="https://ruby.social/@jamie_ca">Mastodon</a>
    </li>
    <li>
      <a href="/static/jamie-macey/">Resume</a>
    </li>
    <li>
      <a href="https://www.linkedin.com/in/jamiemacey/">Linkedin</a>
    </li>
    <li>
      <a href="mailto:jamie.blog@tracefunc.com">Contact</a>
    </li>
  </ol>
</div>
</aside>

      <section>
  <article>
  <h2><a href="/2025/02/27/ruby-http-clients-2025/">Ruby HTTP Clients in 2025</a></h2>
  <blockquote>
    27th Feb 2025
    
      | Tags:
      <a href="/tags/#blog">blog</a> <a href="/tags/#coding">coding</a> 
    
  </blockquote>

  
    <p>At the dayjob, we do payments, both credit card and bank transfers.
Over the years, I’ve done over a dozen integrations to support the banks our customers use, using a mix of HTTP and SFTP.</p>

<p>The latest one we had was working with a bank that used an HTTP API to submit transactions, but was extremely picky about how the request was made.
We needed to make an HTTP call with:</p>

<ul>
  <li>a custom Authorization header format (literally Basic auth but with a different name)</li>
  <li>a custom (nonstandard) header</li>
  <li>a multipart form data field with a required content-type</li>
  <li>a separate multipart file upload</li>
</ul>

<p>After massaging our request enough to get it working with Net::HTTP, I thought I’d give some other options a shot for comparison.</p>

<!-- EXCERPT -->

<p>There was <a href="https://honeyryderchuck.gitlab.io/2023/10/15/state-of-ruby-http-clients-use-httpx.html">a blog post by honeyryderchuck</a>
a bit over a year ago that did a big shootout, and I’m running through the subset of those that I’ve actually used:
Net::HTTP, <a href="">Faraday</a>, <a href="">HTTP</a>, and <a href="">HTTPX</a>.
(I’ve also used <a href="">HTTParty</a> in past years, but it’s not what I’d recommend for this much request complexity.)</p>

<p>First, some boilerplate - sample data, and a common class stub and usage example:</p>

<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
</pre></td><td class="rouge-code"><pre><span class="k">class</span> <span class="nc">HttpClient</span>
  <span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">auth</span><span class="p">)</span>
    <span class="vi">@host</span> <span class="o">=</span> <span class="s1">'example.com'</span>
    <span class="vi">@path</span> <span class="o">=</span> <span class="s1">'/api/path'</span>
    <span class="vi">@auth</span> <span class="o">=</span> <span class="n">auth</span>
  <span class="k">end</span>

  <span class="k">def</span> <span class="nf">passcode</span>
    <span class="no">Base64</span><span class="p">.</span><span class="nf">encode64</span><span class="p">(</span><span class="s2">"</span><span class="si">#{</span><span class="vi">@auth</span><span class="p">[</span><span class="ss">:user</span><span class="p">]</span><span class="si">}</span><span class="s2">:</span><span class="si">#{</span><span class="vi">@auth</span><span class="p">[</span><span class="ss">:pass</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span><span class="p">).</span><span class="nf">chomp</span>
  <span class="k">end</span>

  <span class="k">def</span> <span class="nf">post</span><span class="p">(</span><span class="n">criteria</span><span class="p">,</span> <span class="n">transactions_csv</span><span class="p">)</span>
    <span class="c1"># ...</span>
  <span class="k">end</span>
<span class="k">end</span>

<span class="n">auth</span> <span class="o">=</span> <span class="p">{</span><span class="ss">user: </span><span class="s1">'user'</span><span class="p">,</span> <span class="ss">pass: </span><span class="s1">'pass'</span><span class="p">}</span>
<span class="n">client</span> <span class="o">=</span> <span class="no">HttpClient</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">auth</span><span class="p">)</span>

<span class="n">criteria</span> <span class="o">=</span> <span class="p">{</span> <span class="ss">process_date: </span><span class="s2">"</span><span class="si">#{</span><span class="no">Date</span><span class="p">.</span><span class="nf">today</span><span class="p">.</span><span class="nf">strftime</span><span class="p">(</span><span class="s1">'%Y%m%d'</span><span class="p">)</span><span class="si">}</span><span class="s2">"</span> <span class="p">}</span>
<span class="n">transactions_csv</span> <span class="o">=</span> <span class="o">&lt;&lt;~</span><span class="no">CSV</span><span class="p">.</span><span class="nf">gsub</span><span class="p">(</span><span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"</span><span class="se">\r\n</span><span class="s2">"</span><span class="p">)</span><span class="sh">
  E,C,001,99001,09400313371,10000,TABC00001001,ACME Corp
  E,C,002,99002,09400313372,20000,TABC00001002,John Doe
  E,C,003,99003,09400313373,30000,TABC00001003,Jane Doe
</span><span class="no">CSV</span>
<span class="n">client</span><span class="p">.</span><span class="nf">post</span><span class="p">(</span><span class="n">criteria</span><span class="p">,</span> <span class="n">transactions_csv</span><span class="p">).</span><span class="nf">body</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>This sets up a class to hold the URL and authentication, and a generic helper for the Authorization header.
Then we set up our form data blob (to be rendered to JSON) and file data - note that Net::HTTP expects an actual File
object for the file upload (and Faraday wrapping Net::HTTP needs the same), but in this example we’d be generating it live from database records and so we expect to wrap it in a StringIO.</p>

<h3 id="nethttp">Net::HTTP</h3>

<p>Our baseline implementation is Net::HTTP.
To get multipart form support, we need the <code class="highlighter-rouge">multipart-post</code> gem, and <code class="highlighter-rouge">require 'net/http/post/multipart'</code>.</p>

<p>As part of our API experimentation we also tried <code class="highlighter-rouge">http-form_data</code>, which is part of the HTTP project, but that needed a bunch of boilerplate to get set up, but <code class="highlighter-rouge">multipart-post</code> winds up being fairly clean.
It’s still Net::HTTP though, which means suffering through its old-school object-oriented API.
The only really exciting thing to call out here is the magic <code class="highlighter-rouge">:parts</code> header that <code class="highlighter-rouge">multipart-post</code> supports to let us provide the content-type for the formdata part of the payload.</p>

<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
</pre></td><td class="rouge-code"><pre><span class="k">def</span> <span class="nf">post_net_http</span><span class="p">(</span><span class="n">criteria</span><span class="p">,</span> <span class="n">transactions_csv</span><span class="p">)</span>
  <span class="n">form_data</span> <span class="o">=</span> <span class="p">{</span>
    <span class="ss">criteria: </span><span class="n">criteria</span><span class="p">.</span><span class="nf">to_json</span><span class="p">,</span>
    <span class="ss">transactions: </span><span class="no">UploadIO</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span>
      <span class="no">StringIO</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">transactions_csv</span><span class="p">),</span>
      <span class="s1">'text/plain'</span><span class="p">,</span> <span class="s1">'transactions.csv'</span>
    <span class="p">)</span>
  <span class="p">}</span>

  <span class="n">headers</span> <span class="o">=</span> <span class="p">{</span>
    <span class="s1">'Authorization'</span> <span class="o">=&gt;</span> <span class="s2">"Passcode </span><span class="si">#{</span><span class="n">passcode</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span>
    <span class="s1">'filetype'</span>      <span class="o">=&gt;</span> <span class="s1">'STD'</span><span class="p">,</span>
    <span class="ss">:parts</span>          <span class="o">=&gt;</span> <span class="p">{</span>
      <span class="ss">criteria: </span><span class="p">{</span><span class="s1">'Content-Type'</span> <span class="o">=&gt;</span> <span class="s1">'application/json'</span><span class="p">},</span>
    <span class="p">}</span>
  <span class="p">}</span>

  <span class="n">request</span> <span class="o">=</span> <span class="no">Net</span><span class="o">::</span><span class="no">HTTP</span><span class="o">::</span><span class="no">Post</span><span class="o">::</span><span class="no">Multipart</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="vi">@path</span><span class="p">,</span> <span class="n">form_data</span><span class="p">,</span> <span class="n">headers</span><span class="p">)</span>

  <span class="n">http</span> <span class="o">=</span> <span class="no">Net</span><span class="o">::</span><span class="no">HTTP</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="vi">@host</span><span class="p">,</span> <span class="mi">443</span><span class="p">).</span><span class="nf">tap</span> <span class="k">do</span> <span class="o">|</span><span class="n">http</span><span class="o">|</span>
    <span class="n">http</span><span class="p">.</span><span class="nf">use_ssl</span> <span class="o">=</span> <span class="kp">true</span>
  <span class="k">end</span>
  <span class="n">http</span><span class="p">.</span><span class="nf">request</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
<span class="k">end</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<h3 id="faraday">Faraday</h3>

<p>As mentioned, Faraday is a wrapper around Net::HTTP and so winds up looking fairly similar.
It needs a separate <code class="highlighter-rouge">faraday-multipart</code> gem for multipart support, but doesn’t need special hand-holding to manage part content-types.
Also, actually setting up the connection object is much more readable.
My biggest gripe is headers being provided as an argument to <code class="highlighter-rouge">new</code>, and not being supported by <code class="highlighter-rouge">conn.request :headers</code> or something.</p>

<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
</pre></td><td class="rouge-code"><pre><span class="k">def</span> <span class="nf">post_faraday</span><span class="p">(</span><span class="n">criteria</span><span class="p">,</span> <span class="n">transactions_csv</span><span class="p">)</span>
  <span class="n">form_data</span> <span class="o">=</span> <span class="p">{</span>
    <span class="ss">criteria: </span><span class="no">Faraday</span><span class="o">::</span><span class="no">Multipart</span><span class="o">::</span><span class="no">ParamPart</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span>
      <span class="n">criteria</span><span class="p">.</span><span class="nf">to_json</span><span class="p">,</span>
      <span class="s1">'application/json'</span>
    <span class="p">),</span>
    <span class="ss">transactions: </span><span class="no">Faraday</span><span class="o">::</span><span class="no">Multipart</span><span class="o">::</span><span class="no">FilePart</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span>
      <span class="no">StringIO</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">transactions_csv</span><span class="p">),</span>
      <span class="s1">'text/plain'</span><span class="p">,</span>
      <span class="s1">'transactions.csv'</span>
    <span class="p">)</span>
  <span class="p">}</span>

  <span class="n">headers</span> <span class="o">=</span> <span class="p">{</span>
    <span class="s1">'filetype'</span> <span class="o">=&gt;</span> <span class="s1">'STD'</span>
  <span class="p">}</span>

  <span class="n">http</span> <span class="o">=</span> <span class="no">Faraday</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="s2">"https://</span><span class="si">#{</span><span class="vi">@host</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span> <span class="n">headers</span><span class="p">:)</span> <span class="k">do</span> <span class="o">|</span><span class="n">conn</span><span class="o">|</span>
    <span class="n">conn</span><span class="p">.</span><span class="nf">request</span> <span class="ss">:authorization</span><span class="p">,</span> <span class="s1">'Passcode'</span><span class="p">,</span> <span class="n">passcode</span>
    <span class="n">conn</span><span class="p">.</span><span class="nf">request</span> <span class="ss">:multipart</span>
  <span class="k">end</span>

  <span class="n">http</span><span class="p">.</span><span class="nf">post</span><span class="p">(</span><span class="vi">@path</span><span class="p">,</span> <span class="n">form_data</span><span class="p">)</span>
<span class="k">end</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<h3 id="http-the-gem">HTTP (the gem)</h3>

<p>For HTTP The Gem, despite being a separate gem at least the multipart form support is tagged as a dependency so it’s available by default.
It’s a slight step down from the others in that you need to juggle two different classes while assembling the form data, but otherwise it’s nearly identical to Faraday.
On the plus side, it’s smart enough to handle a String file part as well as an IO, and the HTTP gem chained API calls are always pleasant to work with.</p>

<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
</pre></td><td class="rouge-code"><pre><span class="k">def</span> <span class="nf">post_http</span><span class="p">(</span><span class="n">criteria</span><span class="p">,</span> <span class="n">transactions_csv</span><span class="p">)</span>
  <span class="n">form</span> <span class="o">=</span> <span class="no">HTTP</span><span class="o">::</span><span class="no">FormData</span><span class="o">::</span><span class="no">Multipart</span><span class="p">.</span><span class="nf">new</span><span class="p">({</span>
    <span class="ss">criteria: </span><span class="no">HTTP</span><span class="o">::</span><span class="no">FormData</span><span class="o">::</span><span class="no">Part</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span>
      <span class="n">criteria</span><span class="p">.</span><span class="nf">to_json</span><span class="p">,</span>
      <span class="ss">content_type: </span><span class="s1">'application/json'</span>
    <span class="p">),</span>
    <span class="ss">transactions: </span><span class="no">HTTP</span><span class="o">::</span><span class="no">FormData</span><span class="o">::</span><span class="no">Part</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span>
      <span class="n">transactions_csv</span><span class="p">,</span>
      <span class="ss">content_type: </span><span class="s1">'text/plain'</span><span class="p">,</span>
      <span class="ss">filename: </span><span class="s1">'transactions.csv'</span>
    <span class="p">)</span>
  <span class="p">})</span>

  <span class="n">http</span> <span class="o">=</span> <span class="no">HTTP</span>
    <span class="p">.</span><span class="nf">auth</span><span class="p">(</span><span class="s2">"Passcode </span><span class="si">#{</span><span class="n">passcode</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
    <span class="p">.</span><span class="nf">headers</span><span class="p">(</span><span class="s1">'filetype'</span> <span class="o">=&gt;</span> <span class="s1">'STD'</span><span class="p">)</span>

  <span class="n">http</span><span class="p">.</span><span class="nf">post</span><span class="p">(</span><span class="s2">"https://</span><span class="si">#{</span><span class="vi">@host</span><span class="si">}#{</span><span class="vi">@path</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span> <span class="n">form</span><span class="p">:)</span>
<span class="k">end</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<h3 id="httpx">HTTPX</h3>

<p>Finally, HTTPX: no separate gem for multipart, it’s supported out of the box.
No wrapper classes, just a nested hash.
Same readable chained API as HTTP, though the <code class="highlighter-rouge">with</code> method feels incongruous.</p>

<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
</pre></td><td class="rouge-code"><pre><span class="k">def</span> <span class="nf">post_httpx</span><span class="p">(</span><span class="n">criteria</span><span class="p">,</span> <span class="n">transactions_csv</span><span class="p">)</span>
  <span class="n">form</span> <span class="o">=</span> <span class="p">{</span>
    <span class="ss">criteria: </span><span class="p">{</span>
      <span class="ss">content_type: </span><span class="s1">'application/json'</span><span class="p">,</span>
      <span class="ss">body: </span><span class="n">criteria</span><span class="p">.</span><span class="nf">to_json</span>
    <span class="p">},</span>
    <span class="ss">transactions: </span><span class="p">{</span>
      <span class="ss">content_type: </span><span class="s1">'text/plain'</span><span class="p">,</span>
      <span class="ss">filename: </span><span class="s1">'transactions.csv'</span><span class="p">,</span>
      <span class="ss">body: </span><span class="n">transactions_csv</span>
    <span class="p">}</span>
  <span class="p">}</span>

  <span class="n">http</span> <span class="o">=</span> <span class="no">HTTPX</span>
    <span class="p">.</span><span class="nf">plugin</span><span class="p">(</span><span class="ss">:auth</span><span class="p">)</span>
    <span class="p">.</span><span class="nf">authorization</span><span class="p">(</span><span class="s2">"Passcode </span><span class="si">#{</span><span class="n">passcode</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
    <span class="p">.</span><span class="nf">with</span><span class="p">(</span><span class="ss">headers: </span><span class="p">{</span><span class="s1">'filetype'</span> <span class="o">=&gt;</span> <span class="s1">'STD'</span><span class="p">})</span>

  <span class="n">http</span><span class="p">.</span><span class="nf">post</span><span class="p">(</span><span class="s2">"https://</span><span class="si">#{</span><span class="vi">@host</span><span class="si">}#{</span><span class="vi">@path</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span> <span class="n">form</span><span class="p">:)</span>
<span class="k">end</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<h3 id="thoughts">Thoughts</h3>

<p><strong>Net::HTTP</strong> is a gnarly mess of an API, and depends on another gem so I can’t even really say “it’s available and built-in.” Do not like.</p>

<p><strong>Faraday</strong> is a bit verbose, but building the multipart section is highly orthogonal and easy to see what’s going on.</p>

<p><strong>HTTP</strong> is very similar to Faraday for multipart - simpler in that there’s only one Part class, but needs a Multipart wrapper at toplevel instead of just a Hash. I’m a huge fan of the rest of its API for actually making requests, though, and is my overall preference for overall usability and readability. (Also, it’s pretty straightforward to write a wrapper that builds HTTP FormData objects out of the same simple hash HTTPX wants.)</p>

<p><strong>HTTPX</strong> is a bit worse API than HTTP (<code class="highlighter-rouge">with(headers: ...)</code> being nonobvious) but gets bonus points for being pure Ruby. Lack of support by <code class="highlighter-rouge">vcr</code> for mocking tests (as of this writing) is a big bummer though - if not for that it’d get my top nod.</p>

  
</article>

</section>

<nav aria-label="Pagination">
  <ul>
    
    <li>
      <a href="/2020/04/22/git-archaeology-html/">
        &laquo; Git Archaeology
      </a>
    </li>
    
    
  </ul>
</nav>

    </main>

    <footer>
  <p>Styled with <a href="https://classless.de/">Classless.css</a>.</p>
</footer>


    <script data-goatcounter="https://tracefunc.goatcounter.com/count" async src="//gc.zgo.at/count.js"></script>
  </body>

</html>
<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />

<title>Migrating Disqus - set_trace_func</title>

<!-- Meta -->
<meta name="description" content="Just a random coder writing about technology, web development, and various other geeky topics." />
<meta name="author" content="Jamie Macey">
<link rel="alternate" type="application/atom+xml" title="Atom Feed" href="/index.xml" />

<!-- Favicons -->
<!-- 
<link rel="apple-touch-icon" href="/docs/5.2/assets/img/favicons/apple-touch-icon.png" sizes="180x180">
<link rel="icon" href="/docs/5.2/assets/img/favicons/favicon-32x32.png" sizes="32x32" type="image/png">
<link rel="icon" href="/docs/5.2/assets/img/favicons/favicon-16x16.png" sizes="16x16" type="image/png">
<link rel="manifest" href="/docs/5.2/assets/img/favicons/manifest.json">
<link rel="mask-icon" href="/docs/5.2/assets/img/favicons/safari-pinned-tab.svg" color="#712cf9">
<link rel="icon" href="/docs/5.2/assets/img/favicons/favicon.ico">
-->

<!-- Style -->
<link rel="stylesheet" href="/_bridgetown/static/index.BESXTBS6.css" />

<!-- Script -->
<!-- <script src="/_bridgetown/static/index.FRS7NT6W.js" defer></script> -->


    <link type="application/atom+xml" rel="alternate" href="https://blog.tracefunc.com/index.xml" title="set_trace_func" />
  </head>
  <body class="post ">
    <header>
  <nav>
    <ul>
      <li><h1><a href="/">set_trace_func</a></h1></li>
      <!-- The following float right, and are defined right-to-left -->
      <li class="float-right"><a href="/notes/">Notes</a></li>
      <li class="float-right"><a href="/projects/">Projects</a></li>
      <li class="float-right"><a href="/tags/">Tags</a></li>
      <li class="float-right"><a href="/archive/">Archive</a></li>
    </ul>
  </nav>
</header>


    <main>
      <aside>


<div>
  <h4>About</h4>
  <picture class="float-end w-50 rounded-circle">
    <source srcset="/assets/memoji.webp" type="image/webp">
    <source srcset="/assets/memoji.jpeg" type="image/jpeg">
    <img src="/assets/memoji.png" alt="Avatar of Author" loading="lazy" class="w-100">
  </picture>

  <p>Jamie Macey is a senior software engineer with over 15 years experience
            in the Ruby and Rails ecosystems, largely on the back-end.</p>
  <p class="mb-0">Husband, father, gamer, and all-around geek. Ask about my latest
            3d print, or toy software project.</p>
</div>

<div>
  <h4>Elsewhere</h4>
  <ol class="list-unstyled">
    <li>
      <a href="https://github.com/jamie/"">GitHub</a>
    </li>
    <li style="display: none;">
      <a rel="me" href="https://ruby.social/@jamie_ca">Mastodon</a>
    </li>
    <li>
      <a href="/static/jamie-macey/">Resume</a>
    </li>
    <li>
      <a href="https://www.linkedin.com/in/jamiemacey/">Linkedin</a>
    </li>
    <li>
      <a href="mailto:jamie.blog@tracefunc.com">Contact</a>
    </li>
  </ol>
</div>
</aside>

      <section>
  <article>
  <h2><a href="/2009/12/22/migrating-disqus-html/">Migrating Disqus</a></h2>
  <blockquote>
    22nd Dec 2009
    
      | Tags:
      <a href="/tags/#blog">blog</a> 
    
  </blockquote>

  
    <p>In changing this blog over to jekyll, my urls changed (there’s now a trailing slash). Easy enough to tell google about it, just set up redirects, but there’s no easy way to tell <a href="http://disqus.com/">Disqus</a> about it so my comments migrate over.</p>

<p>The good news is that it’s pretty straightforward using their API, the only bad news is that I can’t delete the new threads auto-generated for the new urls, so I’m just moving them out of the way.</p>

<p>I’m using the <a href="http://httparty.rubyforge.org">HTTParty</a> gem to wrap API access, like so:</p>

<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
</pre></td><td class="rouge-code"><pre><span class="nb">require</span> <span class="s1">'rubygems'</span>
<span class="nb">require</span> <span class="s1">'httparty'</span>
<span class="nb">require</span> <span class="s1">'json'</span>

<span class="k">class</span> <span class="nc">Disqus</span>
  <span class="kp">include</span> <span class="no">HTTParty</span>
  <span class="n">base_uri</span> <span class="s1">'disqus.com'</span>
  <span class="nb">format</span> <span class="ss">:json</span>

  <span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">version</span><span class="o">=</span><span class="s1">'1.1'</span><span class="p">)</span>
    <span class="vi">@key</span> <span class="o">=</span> <span class="n">key</span>
    <span class="vi">@version</span> <span class="o">=</span> <span class="n">version</span>
  <span class="k">end</span>

  <span class="k">def</span> <span class="nf">auth</span>
    <span class="p">{</span><span class="ss">:user_api_key</span> <span class="o">=&gt;</span> <span class="vi">@key</span><span class="p">,</span> <span class="ss">:api_version</span> <span class="o">=&gt;</span> <span class="vi">@version</span><span class="p">}</span>
  <span class="k">end</span>

  <span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="n">action</span><span class="p">,</span> <span class="n">opts</span><span class="o">=</span><span class="p">{})</span>
    <span class="n">result</span> <span class="o">=</span> <span class="nb">self</span><span class="p">.</span><span class="nf">class</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="s2">"/api/</span><span class="si">#{</span><span class="n">action</span><span class="si">}</span><span class="s2">/"</span><span class="p">,</span> <span class="ss">:query</span> <span class="o">=&gt;</span> <span class="n">opts</span><span class="p">.</span><span class="nf">merge</span><span class="p">(</span><span class="n">auth</span><span class="p">))</span>
    <span class="n">result</span><span class="p">[</span><span class="s2">"message"</span><span class="p">]</span>
  <span class="k">end</span>

  <span class="k">def</span> <span class="nf">post</span><span class="p">(</span><span class="n">action</span><span class="p">,</span> <span class="n">opts</span><span class="o">=</span><span class="p">{})</span>
    <span class="n">result</span> <span class="o">=</span> <span class="nb">self</span><span class="p">.</span><span class="nf">class</span><span class="p">.</span><span class="nf">post</span><span class="p">(</span><span class="s2">"/api/</span><span class="si">#{</span><span class="n">action</span><span class="si">}</span><span class="s2">/"</span><span class="p">,</span> <span class="ss">:body</span> <span class="o">=&gt;</span> <span class="n">opts</span><span class="p">.</span><span class="nf">merge</span><span class="p">(</span><span class="n">auth</span><span class="p">).</span><span class="nf">to_params</span><span class="p">)</span>
    <span class="nb">p</span> <span class="n">result</span>
    <span class="n">result</span><span class="p">[</span><span class="s2">"message"</span><span class="p">]</span>
  <span class="k">end</span>
<span class="k">end</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Do note that I’m adding trailing slashes to the api calls to avoid a redirect. Doesn’t matter for the GET, but the redirect on POST was causing issues.</p>

<p>With this in hand, I’m grabbing my forum, looping through the threads, and renaming any that have comments (a whopping 3 of them).</p>

<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
</pre></td><td class="rouge-code"><pre><span class="n">key</span> <span class="o">=</span> <span class="s2">"secret"</span> <span class="c1"># get yours at http://disqus.com/api/get_my_key/</span>
<span class="n">disqus</span> <span class="o">=</span> <span class="no">Disqus</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>

<span class="n">forum</span> <span class="o">=</span> <span class="n">disqus</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="ss">:get_forum_list</span><span class="p">).</span><span class="nf">first</span> <span class="c1"># I just have one</span>
<span class="n">forum_api_key</span> <span class="o">=</span> <span class="n">disqus</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="ss">:get_forum_api_key</span><span class="p">,</span> <span class="ss">:forum_id</span> <span class="o">=&gt;</span> <span class="n">forum</span><span class="p">[</span><span class="s2">"id"</span><span class="p">])</span>

<span class="n">start</span> <span class="o">=</span> <span class="mi">0</span> <span class="c1"># manual pagination, eww</span>
<span class="kp">loop</span> <span class="k">do</span>
  <span class="n">threads</span> <span class="o">=</span> <span class="n">disqus</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="ss">:get_thread_list</span><span class="p">,</span> <span class="ss">:forum_id</span> <span class="o">=&gt;</span> <span class="n">forum</span><span class="p">[</span><span class="s2">"id"</span><span class="p">],</span> <span class="ss">:start</span> <span class="o">=&gt;</span> <span class="n">start</span><span class="p">)</span>
  <span class="k">break</span> <span class="k">if</span> <span class="n">threads</span><span class="p">.</span><span class="nf">empty?</span>

  <span class="n">threads</span><span class="p">.</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">thread</span><span class="o">|</span>
    <span class="n">posts</span> <span class="o">=</span> <span class="n">disqus</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="ss">:get_thread_posts</span><span class="p">,</span> <span class="ss">:thread_id</span> <span class="o">=&gt;</span> <span class="n">thread</span><span class="p">[</span><span class="s2">"id"</span><span class="p">])</span>
    <span class="k">next</span> <span class="k">if</span> <span class="o">!</span><span class="n">posts</span><span class="p">.</span><span class="nf">empty?</span>
    
    <span class="n">target_url</span> <span class="o">=</span> <span class="n">thread</span><span class="p">[</span><span class="s2">"url"</span><span class="p">]</span><span class="o">+</span><span class="s2">"/"</span>

    <span class="c1"># There's another thread in the way...</span>
    <span class="k">if</span> <span class="n">other_thread</span> <span class="o">=</span> <span class="n">disqus</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span>
      <span class="ss">:get_thread_by_url</span><span class="p">,</span> 
      <span class="ss">:forum_api_key</span> <span class="o">=&gt;</span> <span class="n">forum_api_key</span><span class="p">,</span>
      <span class="ss">:url</span> <span class="o">=&gt;</span> <span class="n">target_url</span>
    <span class="p">)</span>
      <span class="c1"># free up the url we want to use</span>
      <span class="n">disqus</span><span class="p">.</span><span class="nf">post</span><span class="p">(</span>
        <span class="ss">:update_thread</span><span class="p">,</span>
        <span class="ss">:forum_api_key</span> <span class="o">=&gt;</span> <span class="n">forum_api_key</span><span class="p">,</span>
        <span class="ss">:thread_id</span> <span class="o">=&gt;</span> <span class="n">other_thread</span><span class="p">[</span><span class="s2">"id"</span><span class="p">],</span>
        <span class="ss">:url</span> <span class="o">=&gt;</span> <span class="n">target_url</span> <span class="o">+</span> <span class="s1">'old'</span>
      <span class="p">)</span>
    <span class="k">end</span>

    <span class="c1"># update thread url</span>
    <span class="n">disqus</span><span class="p">.</span><span class="nf">post</span><span class="p">(</span>
      <span class="ss">:update_thread</span><span class="p">,</span>
      <span class="ss">:forum_api_key</span> <span class="o">=&gt;</span> <span class="n">forum_api_key</span><span class="p">,</span>
      <span class="ss">:thread_id</span> <span class="o">=&gt;</span> <span class="n">thread</span><span class="p">[</span><span class="s2">"id"</span><span class="p">],</span>
      <span class="ss">:url</span> <span class="o">=&gt;</span> <span class="n">target_url</span>
    <span class="p">)</span>
  <span class="k">end</span>

  <span class="n">start</span> <span class="o">+=</span> <span class="mi">25</span>
<span class="k">end</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Et voilà, old comments are in the right place now.</p>

  
</article>

</section>

<nav aria-label="Pagination">
  <ul>
    
    <li>
      <a href="/2009/12/04/jekyll-custom-liquid-tags-html/">
        &laquo; Jekyll: Custom Liquid Tags
      </a>
    </li>
    
    
    <li class="float-right">
      <a href="/2011/01/19/isolating-rails-html/">
        Isolating Rails &raquo;
      </a>
    </li>
    
  </ul>
</nav>

    </main>

    <footer>
  <p>Styled with <a href="https://classless.de/">Classless.css</a>.</p>
</footer>


    <script data-goatcounter="https://tracefunc.goatcounter.com/count" async src="//gc.zgo.at/count.js"></script>
  </body>

</html>
The base install of [Jekyll][] at the moment doesn't let you run _any_ arbitrary ruby code. This is so that they can use it for github pages and not need to worry about making a super-secure sandbox just to generate some HTML.

[Jekyll]: http://github.com/mojombo/jekyll

Unfortunately, that means we're out of luck for creating custom liquid filters. The most annoying deficiency for me is tags. The way the default liquid `map` filter works isn't friendly with @site.tags, so to generate my [Tags][] page I had to do some really crazy stuff with `capture`:

[Tags]: /tags/

```html
<div id="articles">
  <table>
    {% for tag_ in @site.tags %}
      {% capture tag %}{{ tag_ | first }}{% endcapture %}
      <tr><th>{{ tag }}</th>
          <th><a name="{{ tag }}" class="anchor">&nbsp;</th></tr>
      {% for post in @site.posts %}
        {% if post.tags contains tag %}
          <tr><td>{{ post.date | date: '%b %e, %Y' }}</td>
              <td><a href="{{ post.url }}">{{ post.title }}</a></td></tr>
        {% endif %}
      {% endfor %}
    {% endfor %}
  </table>
</div>

Fortunately, it wasn’t to hard to make a fork, and in my fork I added a super simple code loading option. Now, I can add a quick extension in _lib/filters.rb like so:

module Jekyll
  module Filters
    def keys(input)
      input.keys
    end

    def tagged(input, tag)
      input.select{|post| post.tags.include? tag}
    end
  end
end

Now tags.html looks like this:

<div id="articles">
  <table>
    {% for tag in @site.tags|keys|sort %}
      <tr><th>{{ tag }}</th>
          <th><a name="{{ tag }}" class="anchor">&nbsp;</th></tr>
      {% for post in @site.posts|tagged:tag %}
        <tr><td>{{ post.date | date: '%b %e, %Y' }}</td>
            <td><a href="{{ post.url }}">{{ post.title }}</a></td></tr>
      {% endfor %}
    {% endfor %}
  </table>
</div>

There’s a bit of trickery there that liquid doesn’t document very well on lines 3 and 5 - in the second half of the for block you can chain filters on the collection you’re iterating over. The short format used is something along the lines of collection|filter:arg,arg,arg|filter...

Similarly, I had some ugly code in my regular archive page to group by year and put headings in:

{% for post in site.posts %}
  {% unless post.next %}
    <tr><th>{{ post.date | date: '%Y' }}</th><th>&nbsp;</th></tr>
  {% else %}
    {% capture year %}{{ post.date | date: '%Y' }}{% endcapture %}
    {% capture nyear %}{{ post.next.date | date: '%Y' }}{% endcapture %}
    {% if year != nyear %}
      <tr><th>{{ post.date | date: '%Y' }}</th><th>&nbsp;</th></tr>
    {% endif %}
  {% endunless %}

  ...
{% endfor %}

Now, a few extra liquid filters later, it looks like this:

{% for post in site.posts %}
  {% if post|last_of_year? %}
    <tr><th>{{ post.date | date: '%Y' }}</th><th>&nbsp;</th></tr>
  {% endif %}
  
  ...
{% endfor %}

If you want to get easy extensions in your own project, rather than maintaining Yet Another Jekyll Fork, please vote up my merge request on github.

As a side note, blogging about liquid is a pain. The least pain I’ve found so far is to use liquid to output the leading open brace for all tags. Looks like garbage in my text editor, but it gets the job done:

{{'{'}}{ post.title }}

Blogging about blogging about liquid (as above) I leave as an exercise to the reader. Well, it’s that time of the year again, but I thought I’d put up the New Years Resolutions a bit early, since there’s one that applies here.

I’m finding that I have a bunch of coding projects that I want to work on, but not enough time for them. One of my resolutions this year is to spend my Saturday mornings doing some coding instead of sleeping in, and then blog about something from the week after lunch. I don’t know how these posts will work, but maybe something useful will come of them. If not, it’ll serve as a good reminder of what I’ve been doing over the course of the year.

As a prelude to this, here’s a list of the projects I’m working on at the moment (or hoping to work on in the coming year).

“Competitive” Programming

Sometimes I wind up looking for something outside of the normal, day-to-day stuff I usually do, and there’s a few places I go looking at the moment.

I follow the rubyquiz threads on the ruby-talk mailing list, and find that I do about one a month. They’re usually just little apps that take an hour or two, that I can focus on clean code.

Second up is Code Golf. I’ve tackled a few problems with Ruby, and am really curious how some people are managing to squeeze the same functionality I have into half the code size, for some problems. For the uninitiated, golfing started to gain popularity in the Perl community, where people would try to create the smallest source file (measured in bytes) that could satisfy a problem - Code Golf is essentially the same thing, but it supports Ruby, PHP, and Python as well.

The last of the bunch is the Sphere Online Judge, or SPOJ. The guys that run SPOJ have collected several hundred programming problems from a number of programming competitions (the one I’ve seen the most is the regional ACM competitions) and have a system set up to accept submissions in over two dozen languages and run them against a validator to check that it’s been done correctly. Some of the problems have size restrictions (mostly to prevent hard-coding solutions) but the biggest problem I’ve come up against doing things in Ruby is execution time. I’ve got a few problems that I have a working solution for, but I still need to speed it up by about 10x to get under the time limit on their server. That said, I’ve found those ones to be quite rewarding to work on, as I’ve managed to get one in particular sped up by over 100x from the initial naive solution just by profiling the code and tweaking the algorithm. Sometimes though, Ruby just is too slow, and I just leave the solution on my hard drive.

Fluxx (AI)

I was gifted Fluxx for Christmas, and it’s an absurdly fun card game that’s quite easy to play. It should be fairly trivial to develop a framework for playing the game, and plug in some AI players that just select cards at random from their hand to play - eventually one of them will win, I’m sure. Once that’s done, I want to toy around with creating a (or many) custom AI to beat the snot out of the random players, and then improve on it - mostly just for kicks.

RCov-c1

I started work a while back on a modification to RCov, a fast ruby code-coverage tool. RCov itself is line coverage tool. Much more useful than line coverage is branch coverage, which Mauricio refers to as C1 on the RCov page. I came up with the idea that it shouldn’t be too hard to come up with a modification of Ruby2Ruby that takes the branches in the code, and unwraps all the conditionals. So instead of this:

    if (x and y)
      foo
    else
      bar
    end

I wanted it to output something like this:

    if x
      if y
        foo
      else
        bar
      end
    else
      bar
    end

If a line coverage tool were to be run on the output of this translation, it should be able to approximate a branch coverage tool fairly well. Alas, while I have a Ruby2Ruby subclass that performs this translation correctly for significantly convoluted nested conditionals (finally), something on my laptop no longer plays nice with RubyInline, so I can’t get it to run. If I get enough desire (or enough people start bugging me) I might try reinstalling Ubuntu to see if that helps (it started to break about the time I upgraded from Dapper to Edgy), or else I might try upgrading to 1.8.5 and see if that’s the problem.

rPlug

My day job is writing Rails, and I toy around with it in my off-time as well. I need more than the minimal support script/plugin gives me for maintaining my plugins, and while I think piston is great, the fact that it’s SVN only makes it a no-go for me. We have an svn repo that we work from at work, and I carry it around through an svk mirror on my laptop. I’ve started a potential replacement for piston that stores its meta-information in config/plugins.yml instead of svn properties. It currently can pull plugins from an SVN repository and handle the upgrade path assuming the rails app is in an SVK repository (ie, it does enough to work for me). I’m aiming to broaden that, though, which is why I need…

SourceControl

I’m a big fan of Mercurial, and the whole distributed-version-control scene (so darcs and monotone too), and I’m disappointed that so many utilities coming out for ruby/rails development are so svn-focused. I mean, we’ve got RSCM, right? So why was it news a few months back that Capistrano gained support for Mercurial because someone agreed to maintain that bit of code? Why aren’t they using RSCM?

For me, it’s because RSCM refused to run on my linux box (no joke, the gem I pulled down had a conditional that white-listed platforms, and ‘linux’ was not one of them), and from what I have been able to glean from it it’s a tool that still focuses on SCM tools the wrong way. To RSCM, the baseline is centralized version control (svn), and any distributed version control tools need to conform to that mindset. I want to see how things look going the other way - svn is like hg but without the ability to push/pull, not the other way around.

I’ll be investigating the RSCM api to see if I can keep it mostly compatable (or add an extra lib to require that can add the correct api points) but I can’t quite say I’ve written any code for this yet.

RubyTests

I’m wanting to contribute to the RubyTests project, and actually have some un-committed code out at the moment, but haven’t managed to find the time to spend on this lately. I do think that having a unified test suite is a step in the right direction, especially since we’re starting to see a proliferation of alternative implementations, and want to find time to work on this.

Shimmer

Shimmer is a rails-based web photo album app that I wanted to make some tweaks to and use to host some of my photos for family to browse and download. Just laziness that I haven’t gotten around to this lately.

Weight

One of my other New Years’ resolutions is to lose some weight - I’m aiming for 2 lbs a month. But, because I’m a geek I want to log my progress on the computer. I did an exercise routine that I managed in Excel for about 9 months, but wasn’t losing any weight from it so I kinda just stopped as life got busy in August this year. Also, excel was boring to look at. I want to hack together a Camping app that will log my daily weights (and possibly incorporate the exercise tracking as well - see the Hacker’s Diet on that) and provide a pretty graph to follow my progress with. Heck, I might even go for broke and not test it :P

Devdot

Last up is my uber-project. The one I’ve been wanting to spend a lot of time on for at least the last year, but never got around to it. I’m sure anyone who could possibly read this post knows Trac. Ah, Trac, you least-bad of a bunch of bad apps. Rails has a me-too project in Collaboa which, now that I look, seems to have picked up steam again as of the beginning of November. Sadly, it looks like just another Trac clone.

I want something different. I want something that can act more as a project front-end than a repository tack-on. I want automation. Magic, even. I’d like devdot to be smarter than the average tool. I want it to build a new source tarball when it sees a commit. I want it to pick up releases from tags in the repository. Make tarballs for them. Hell, even make gems and send them over to rubyforge. I want it to do rdoc generation and hosting, automatically. Pipe through test runs and coverage reports. And heck, while I’m at it I might as well add ticket and milestone management, and blow past Trac’s features in one swift blow. I even have dreams of a message-board that auto-fronts to a mailing list, or RSS feed - whatever floats your boat.

And from the very beginning, I want it to not be for “Subversion projects”. I want it to be multi-project from the get-go. I want to give it to why and find it a better replacement for his multitude of trac instances. Same for Ryan Davis’ stuff.

But, you know, all in good time. It’ll get there when it gets there. In the meanwhile, I’m in it for the fun :P

And So, It Begins

So having enumerated everything (wow, 9 projects - no wonder none of them get any time) I’ll be back on the 6th (if anyone’s listening) hopefully doing something useful. </th> <th><a name=”blog<!doctype html>

Ruby HTTP Clients in 2025 - set_trace_func

Ruby HTTP Clients in 2025

27th Feb 2025 | Tags: blog coding

At the dayjob, we do payments, both credit card and bank transfers. Over the years, I’ve done over a dozen integrations to support the banks our customers use, using a mix of HTTP and SFTP.

The latest one we had was working with a bank that used an HTTP API to submit transactions, but was extremely picky about how the request was made. We needed to make an HTTP call with:

a custom Authorization header format (literally Basic auth but with a different name)
a custom (nonstandard) header
a multipart form data field with a required content-type
a separate multipart file upload

After massaging our request enough to get it working with Net::HTTP, I thought I’d give some other options a shot for comparison.

There was a blog post by honeyryderchuck a bit over a year ago that did a big shootout, and I’m running through the subset of those that I’ve actually used: Net::HTTP, Faraday, HTTP, and HTTPX. (I’ve also used HTTParty in past years, but it’s not what I’d recommend for this much request complexity.)

First, some boilerplate - sample data, and a common class stub and usage example:

class HttpClient
  def initialize(auth)
    @host = 'example.com'
    @path = '/api/path'
    @auth = auth
  end

  def passcode
    Base64.encode64("#{@auth[:user]}:#{@auth[:pass]}").chomp
  end

  def post(criteria, transactions_csv)
    # ...
  end
end

auth = {user: 'user', pass: 'pass'}
client = HttpClient.new(auth)

criteria = { process_date: "#{Date.today.strftime('%Y%m%d')}" }
transactions_csv = <<~CSV.gsub("\n", "\r\n")
  E,C,001,99001,09400313371,10000,TABC00001001,ACME Corp
  E,C,002,99002,09400313372,20000,TABC00001002,John Doe
  E,C,003,99003,09400313373,30000,TABC00001003,Jane Doe
CSV
client.post(criteria, transactions_csv).body

This sets up a class to hold the URL and authentication, and a generic helper for the Authorization header. Then we set up our form data blob (to be rendered to JSON) and file data - note that Net::HTTP expects an actual File object for the file upload (and Faraday wrapping Net::HTTP needs the same), but in this example we’d be generating it live from database records and so we expect to wrap it in a StringIO.

Net::HTTP

Our baseline implementation is Net::HTTP. To get multipart form support, we need the multipart-post gem, and require 'net/http/post/multipart'.

As part of our API experimentation we also tried http-form_data, which is part of the HTTP project, but that needed a bunch of boilerplate to get set up, but multipart-post winds up being fairly clean. It’s still Net::HTTP though, which means suffering through its old-school object-oriented API. The only really exciting thing to call out here is the magic :parts header that multipart-post supports to let us provide the content-type for the formdata part of the payload.

def post_net_http(criteria, transactions_csv)
  form_data = {
    criteria: criteria.to_json,
    transactions: UploadIO.new(
      StringIO.new(transactions_csv),
      'text/plain', 'transactions.csv'
    )
  }

  headers = {
    'Authorization' => "Passcode #{passcode}",
    'filetype'      => 'STD',
    :parts          => {
      criteria: {'Content-Type' => 'application/json'},
    }
  }

  request = Net::HTTP::Post::Multipart.new(@path, form_data, headers)

  http = Net::HTTP.new(@host, 443).tap do |http|
    http.use_ssl = true
  end
  http.request(request)
end

Faraday

As mentioned, Faraday is a wrapper around Net::HTTP and so winds up looking fairly similar. It needs a separate faraday-multipart gem for multipart support, but doesn’t need special hand-holding to manage part content-types. Also, actually setting up the connection object is much more readable. My biggest gripe is headers being provided as an argument to new, and not being supported by conn.request :headers or something.

def post_faraday(criteria, transactions_csv)
  form_data = {
    criteria: Faraday::Multipart::ParamPart.new(
      criteria.to_json,
      'application/json'
    ),
    transactions: Faraday::Multipart::FilePart.new(
      StringIO.new(transactions_csv),
      'text/plain',
      'transactions.csv'
    )
  }

  headers = {
    'filetype' => 'STD'
  }

  http = Faraday.new("https://#{@host}", headers:) do |conn|
    conn.request :authorization, 'Passcode', passcode
    conn.request :multipart
  end

  http.post(@path, form_data)
end

HTTP (the gem)

For HTTP The Gem, despite being a separate gem at least the multipart form support is tagged as a dependency so it’s available by default. It’s a slight step down from the others in that you need to juggle two different classes while assembling the form data, but otherwise it’s nearly identical to Faraday. On the plus side, it’s smart enough to handle a String file part as well as an IO, and the HTTP gem chained API calls are always pleasant to work with.

def post_http(criteria, transactions_csv)
  form = HTTP::FormData::Multipart.new({
    criteria: HTTP::FormData::Part.new(
      criteria.to_json,
      content_type: 'application/json'
    ),
    transactions: HTTP::FormData::Part.new(
      transactions_csv,
      content_type: 'text/plain',
      filename: 'transactions.csv'
    )
  })

  http = HTTP
    .auth("Passcode #{passcode}")
    .headers('filetype' => 'STD')

  http.post("https://#{@host}#{@path}", form:)
end

HTTPX

Finally, HTTPX: no separate gem for multipart, it’s supported out of the box. No wrapper classes, just a nested hash. Same readable chained API as HTTP, though the with method feels incongruous.

def post_httpx(criteria, transactions_csv)
  form = {
    criteria: {
      content_type: 'application/json',
      body: criteria.to_json
    },
    transactions: {
      content_type: 'text/plain',
      filename: 'transactions.csv',
      body: transactions_csv
    }
  }

  http = HTTPX
    .plugin(:auth)
    .authorization("Passcode #{passcode}")
    .with(headers: {'filetype' => 'STD'})

  http.post("https://#{@host}#{@path}", form:)
end

Thoughts

Net::HTTP is a gnarly mess of an API, and depends on another gem so I can’t even really say “it’s available and built-in.” Do not like.

Faraday is a bit verbose, but building the multipart section is highly orthogonal and easy to see what’s going on.

HTTP is very similar to Faraday for multipart - simpler in that there’s only one Part class, but needs a Multipart wrapper at toplevel instead of just a Hash. I’m a huge fan of the rest of its API for actually making requests, though, and is my overall preference for overall usability and readability. (Also, it’s pretty straightforward to write a wrapper that builds HTTP FormData objects out of the same simple hash HTTPX wants.)

HTTPX is a bit worse API than HTTP (with(headers: ...) being nonobvious) but gets bonus points for being pure Ruby. Lack of support by vcr for mocking tests (as of this writing) is a big bummer though - if not for that it’d get my top nod.

<!doctype html>

Migrating Disqus - set_trace_func

Migrating Disqus

22nd Dec 2009 | Tags: blog

In changing this blog over to jekyll, my urls changed (there’s now a trailing slash). Easy enough to tell google about it, just set up redirects, but there’s no easy way to tell Disqus about it so my comments migrate over.

The good news is that it’s pretty straightforward using their API, the only bad news is that I can’t delete the new threads auto-generated for the new urls, so I’m just moving them out of the way.

I’m using the HTTParty gem to wrap API access, like so:

require 'rubygems'
require 'httparty'
require 'json'

class Disqus
  include HTTParty
  base_uri 'disqus.com'
  format :json

  def initialize(key, version='1.1')
    @key = key
    @version = version
  end

  def auth
    {:user_api_key => @key, :api_version => @version}
  end

  def get(action, opts={})
    result = self.class.get("/api/#{action}/", :query => opts.merge(auth))
    result["message"]
  end

  def post(action, opts={})
    result = self.class.post("/api/#{action}/", :body => opts.merge(auth).to_params)
    p result
    result["message"]
  end
end

Do note that I’m adding trailing slashes to the api calls to avoid a redirect. Doesn’t matter for the GET, but the redirect on POST was causing issues.

With this in hand, I’m grabbing my forum, looping through the threads, and renaming any that have comments (a whopping 3 of them).

key = "secret" # get yours at http://disqus.com/api/get_my_key/
disqus = Disqus.new(key)

forum = disqus.get(:get_forum_list).first # I just have one
forum_api_key = disqus.get(:get_forum_api_key, :forum_id => forum["id"])

start = 0 # manual pagination, eww
loop do
  threads = disqus.get(:get_thread_list, :forum_id => forum["id"], :start => start)
  break if threads.empty?

  threads.each do |thread|
    posts = disqus.get(:get_thread_posts, :thread_id => thread["id"])
    next if !posts.empty?
    
    target_url = thread["url"]+"/"

    # There's another thread in the way...
    if other_thread = disqus.get(
      :get_thread_by_url, 
      :forum_api_key => forum_api_key,
      :url => target_url
    )
      # free up the url we want to use
      disqus.post(
        :update_thread,
        :forum_api_key => forum_api_key,
        :thread_id => other_thread["id"],
        :url => target_url + 'old'
      )
    end

    # update thread url
    disqus.post(
      :update_thread,
      :forum_api_key => forum_api_key,
      :thread_id => thread["id"],
      :url => target_url
    )
  end

  start += 25
end

Et voilà, old comments are in the right place now.

<div id="articles">
  <table>
    {% for tag_ in @site.tags %}
      {% capture tag %}{{ tag_ | first }}{% endcapture %}
      <tr><th>{{ tag }}</th>
          <th><a name="{{ tag }}" class="anchor">&nbsp;</th></tr>
      {% for post in @site.posts %}
        {% if post.tags contains tag %}
          <tr><td>{{ post.date | date: '%b %e, %Y' }}</td>
              <td><a href="{{ post.url }}">{{ post.title }}</a></td></tr>
        {% endif %}
      {% endfor %}
    {% endfor %}
  </table>
</div>

Fortunately, it wasn’t to hard to make a fork, and in my fork I added a super simple code loading option. Now, I can add a quick extension in _lib/filters.rb like so:

module Jekyll
  module Filters
    def keys(input)
      input.keys
    end

    def tagged(input, tag)
      input.select{|post| post.tags.include? tag}
    end
  end
end

Now tags.html looks like this:

<div id="articles">
  <table>
    {% for tag in @site.tags|keys|sort %}
      <tr><th>{{ tag }}</th>
          <th><a name="{{ tag }}" class="anchor">&nbsp;</th></tr>
      {% for post in @site.posts|tagged:tag %}
        <tr><td>{{ post.date | date: '%b %e, %Y' }}</td>
            <td><a href="{{ post.url }}">{{ post.title }}</a></td></tr>
      {% endfor %}
    {% endfor %}
  </table>
</div>

Similarly, I had some ugly code in my regular archive page to group by year and put headings in:

{% for post in site.posts %}
  {% unless post.next %}
    <tr><th>{{ post.date | date: '%Y' }}</th><th>&nbsp;</th></tr>
  {% else %}
    {% capture year %}{{ post.date | date: '%Y' }}{% endcapture %}
    {% capture nyear %}{{ post.next.date | date: '%Y' }}{% endcapture %}
    {% if year != nyear %}
      <tr><th>{{ post.date | date: '%Y' }}</th><th>&nbsp;</th></tr>
    {% endif %}
  {% endunless %}

  ...
{% endfor %}

Now, a few extra liquid filters later, it looks like this:

{% for post in site.posts %}
  {% if post|last_of_year? %}
    <tr><th>{{ post.date | date: '%Y' }}</th><th>&nbsp;</th></tr>
  {% endif %}
  
  ...
{% endfor %}

If you want to get easy extensions in your own project, rather than maintaining Yet Another Jekyll Fork, please vote up my merge request on github.

{{'{'}}{ post.title }}

As a prelude to this, here’s a list of the projects I’m working on at the moment (or hoping to work on in the coming year).

“Competitive” Programming

Sometimes I wind up looking for something outside of the normal, day-to-day stuff I usually do, and there’s a few places I go looking at the moment.

I follow the rubyquiz threads on the ruby-talk mailing list, and find that I do about one a month. They’re usually just little apps that take an hour or two, that I can focus on clean code.

Fluxx (AI)

RCov-c1

    if (x and y)
      foo
    else
      bar
    end

I wanted it to output something like this:

    if x
      if y
        foo
      else
        bar
      end
    else
      bar
    end

rPlug

SourceControl

RubyTests

Shimmer

Weight

Devdot

But, you know, all in good time. It’ll get there when it gets there. In the meanwhile, I’m in it for the fun :P

And So, It Begins

So having enumerated everything (wow, 9 projects - no wonder none of them get any time) I’ll be back on the 6th (if anyone’s listening) hopefully doing something useful. “ class=”anchor”> </th></tr>

  <tr><th>coding<!doctype html>

Ruby HTTP Clients in 2025 - set_trace_func

Ruby HTTP Clients in 2025

27th Feb 2025 | Tags: blog coding

At the dayjob, we do payments, both credit card and bank transfers. Over the years, I’ve done over a dozen integrations to support the banks our customers use, using a mix of HTTP and SFTP.

The latest one we had was working with a bank that used an HTTP API to submit transactions, but was extremely picky about how the request was made. We needed to make an HTTP call with:

a custom Authorization header format (literally Basic auth but with a different name)
a custom (nonstandard) header
a multipart form data field with a required content-type
a separate multipart file upload

After massaging our request enough to get it working with Net::HTTP, I thought I’d give some other options a shot for comparison.

First, some boilerplate - sample data, and a common class stub and usage example:

class HttpClient
  def initialize(auth)
    @host = 'example.com'
    @path = '/api/path'
    @auth = auth
  end

  def passcode
    Base64.encode64("#{@auth[:user]}:#{@auth[:pass]}").chomp
  end

  def post(criteria, transactions_csv)
    # ...
  end
end

auth = {user: 'user', pass: 'pass'}
client = HttpClient.new(auth)

criteria = { process_date: "#{Date.today.strftime('%Y%m%d')}" }
transactions_csv = <<~CSV.gsub("\n", "\r\n")
  E,C,001,99001,09400313371,10000,TABC00001001,ACME Corp
  E,C,002,99002,09400313372,20000,TABC00001002,John Doe
  E,C,003,99003,09400313373,30000,TABC00001003,Jane Doe
CSV
client.post(criteria, transactions_csv).body

Net::HTTP

Our baseline implementation is Net::HTTP. To get multipart form support, we need the multipart-post gem, and require 'net/http/post/multipart'.

def post_net_http(criteria, transactions_csv)
  form_data = {
    criteria: criteria.to_json,
    transactions: UploadIO.new(
      StringIO.new(transactions_csv),
      'text/plain', 'transactions.csv'
    )
  }

  headers = {
    'Authorization' => "Passcode #{passcode}",
    'filetype'      => 'STD',
    :parts          => {
      criteria: {'Content-Type' => 'application/json'},
    }
  }

  request = Net::HTTP::Post::Multipart.new(@path, form_data, headers)

  http = Net::HTTP.new(@host, 443).tap do |http|
    http.use_ssl = true
  end
  http.request(request)
end

Faraday

def post_faraday(criteria, transactions_csv)
  form_data = {
    criteria: Faraday::Multipart::ParamPart.new(
      criteria.to_json,
      'application/json'
    ),
    transactions: Faraday::Multipart::FilePart.new(
      StringIO.new(transactions_csv),
      'text/plain',
      'transactions.csv'
    )
  }

  headers = {
    'filetype' => 'STD'
  }

  http = Faraday.new("https://#{@host}", headers:) do |conn|
    conn.request :authorization, 'Passcode', passcode
    conn.request :multipart
  end

  http.post(@path, form_data)
end

HTTP (the gem)

def post_http(criteria, transactions_csv)
  form = HTTP::FormData::Multipart.new({
    criteria: HTTP::FormData::Part.new(
      criteria.to_json,
      content_type: 'application/json'
    ),
    transactions: HTTP::FormData::Part.new(
      transactions_csv,
      content_type: 'text/plain',
      filename: 'transactions.csv'
    )
  })

  http = HTTP
    .auth("Passcode #{passcode}")
    .headers('filetype' => 'STD')

  http.post("https://#{@host}#{@path}", form:)
end

HTTPX

Finally, HTTPX: no separate gem for multipart, it’s supported out of the box. No wrapper classes, just a nested hash. Same readable chained API as HTTP, though the with method feels incongruous.

def post_httpx(criteria, transactions_csv)
  form = {
    criteria: {
      content_type: 'application/json',
      body: criteria.to_json
    },
    transactions: {
      content_type: 'text/plain',
      filename: 'transactions.csv',
      body: transactions_csv
    }
  }

  http = HTTPX
    .plugin(:auth)
    .authorization("Passcode #{passcode}")
    .with(headers: {'filetype' => 'STD'})

  http.post("https://#{@host}#{@path}", form:)
end

Thoughts

Net::HTTP is a gnarly mess of an API, and depends on another gem so I can’t even really say “it’s available and built-in.” Do not like.

Faraday is a bit verbose, but building the multipart section is highly orthogonal and easy to see what’s going on.

</th> <th><a name=”coding<!doctype html>

Ruby HTTP Clients in 2025 - set_trace_func

Ruby HTTP Clients in 2025

27th Feb 2025 | Tags: blog coding

At the dayjob, we do payments, both credit card and bank transfers. Over the years, I’ve done over a dozen integrations to support the banks our customers use, using a mix of HTTP and SFTP.

The latest one we had was working with a bank that used an HTTP API to submit transactions, but was extremely picky about how the request was made. We needed to make an HTTP call with:

a custom Authorization header format (literally Basic auth but with a different name)
a custom (nonstandard) header
a multipart form data field with a required content-type
a separate multipart file upload

After massaging our request enough to get it working with Net::HTTP, I thought I’d give some other options a shot for comparison.

First, some boilerplate - sample data, and a common class stub and usage example:

class HttpClient
  def initialize(auth)
    @host = 'example.com'
    @path = '/api/path'
    @auth = auth
  end

  def passcode
    Base64.encode64("#{@auth[:user]}:#{@auth[:pass]}").chomp
  end

  def post(criteria, transactions_csv)
    # ...
  end
end

auth = {user: 'user', pass: 'pass'}
client = HttpClient.new(auth)

criteria = { process_date: "#{Date.today.strftime('%Y%m%d')}" }
transactions_csv = <<~CSV.gsub("\n", "\r\n")
  E,C,001,99001,09400313371,10000,TABC00001001,ACME Corp
  E,C,002,99002,09400313372,20000,TABC00001002,John Doe
  E,C,003,99003,09400313373,30000,TABC00001003,Jane Doe
CSV
client.post(criteria, transactions_csv).body

Net::HTTP

Our baseline implementation is Net::HTTP. To get multipart form support, we need the multipart-post gem, and require 'net/http/post/multipart'.

def post_net_http(criteria, transactions_csv)
  form_data = {
    criteria: criteria.to_json,
    transactions: UploadIO.new(
      StringIO.new(transactions_csv),
      'text/plain', 'transactions.csv'
    )
  }

  headers = {
    'Authorization' => "Passcode #{passcode}",
    'filetype'      => 'STD',
    :parts          => {
      criteria: {'Content-Type' => 'application/json'},
    }
  }

  request = Net::HTTP::Post::Multipart.new(@path, form_data, headers)

  http = Net::HTTP.new(@host, 443).tap do |http|
    http.use_ssl = true
  end
  http.request(request)
end

Faraday

def post_faraday(criteria, transactions_csv)
  form_data = {
    criteria: Faraday::Multipart::ParamPart.new(
      criteria.to_json,
      'application/json'
    ),
    transactions: Faraday::Multipart::FilePart.new(
      StringIO.new(transactions_csv),
      'text/plain',
      'transactions.csv'
    )
  }

  headers = {
    'filetype' => 'STD'
  }

  http = Faraday.new("https://#{@host}", headers:) do |conn|
    conn.request :authorization, 'Passcode', passcode
    conn.request :multipart
  end

  http.post(@path, form_data)
end

HTTP (the gem)

def post_http(criteria, transactions_csv)
  form = HTTP::FormData::Multipart.new({
    criteria: HTTP::FormData::Part.new(
      criteria.to_json,
      content_type: 'application/json'
    ),
    transactions: HTTP::FormData::Part.new(
      transactions_csv,
      content_type: 'text/plain',
      filename: 'transactions.csv'
    )
  })

  http = HTTP
    .auth("Passcode #{passcode}")
    .headers('filetype' => 'STD')

  http.post("https://#{@host}#{@path}", form:)
end

HTTPX

Finally, HTTPX: no separate gem for multipart, it’s supported out of the box. No wrapper classes, just a nested hash. Same readable chained API as HTTP, though the with method feels incongruous.

def post_httpx(criteria, transactions_csv)
  form = {
    criteria: {
      content_type: 'application/json',
      body: criteria.to_json
    },
    transactions: {
      content_type: 'text/plain',
      filename: 'transactions.csv',
      body: transactions_csv
    }
  }

  http = HTTPX
    .plugin(:auth)
    .authorization("Passcode #{passcode}")
    .with(headers: {'filetype' => 'STD'})

  http.post("https://#{@host}#{@path}", form:)
end

Thoughts

Net::HTTP is a gnarly mess of an API, and depends on another gem so I can’t even really say “it’s available and built-in.” Do not like.

Faraday is a bit verbose, but building the multipart section is highly orthogonal and easy to see what’s going on.

” class=”anchor”> </th></tr>

  <tr><th>git<!doctype html>

Git Archaeology - set_trace_func

Git Archaeology

22nd Apr 2020 | Tags: git

Recently at work we passed a major milestone on our codebase, and I wanted to see if I could run some analysis over time on authorship and see how long some early contributors’ work stuck around in the product.

A bit of experimentation and random googling left me with these three scripts.

The first is dig.sh, which will accept the path to the git repository to analyze (because I’m making a few dozen files, and don’t want to dirty the primary repo), a date to process, and optionally a commit to analyze. If a specific commit is not provided, it’ll look up the first commit before the provided date.

How this looks in practice is something like ./dig.sh ../my_repo 2011-04-01. Because we run a monorepo now, and have merged a few external repos together, this sometimes didn’t pick up a proper mainline commit, I occasionally had to come back on a manual pass and touch it up: ./dig.sh ../my_repo 2011-04-01 4dbdd82.

The actual meat of the script is the last 3 lines, which gets a recursive directory listing as of the commit in question, filters to files that match a provided pattern, runs git blame across all of them, and counts the number of entries for each author.

#!/bin/bash

# ARGV: repo_path, date, commit

export DIR=`pwd`
cd $1

export DATE=$2

if [ -n "$3" ]; then
  export COMMIT=$3
else
  export COMMIT=$(git rev-list -1 --before="$DATE" master)
fi

export PATTERN='\t(app|lib)/.*\.(rb|js|erb|haml)$'

git ls-tree -r $COMMIT | egrep -o "$PATTERN" | while read f; do
  git blame -w -M -C -C --line-porcelain $COMMIT -- $f;
done | egrep -a '^author ' | sort | uniq -c > $DIR/$DATE.txt

Next up is dig_all.sh, which is just a barebones orchestration script. Like the above, you provide a repo path and branch, and it will sequentially run through history. Due to the growth in the repo over time, early years would take under 10 minutes/month to process, but months this year were running over 3 hours. Thus the start/end year arguments, so I could run a few scripts simultaneously - it’s impressively not disk-bound on my mac, I did some testing and could run 3 at once without significantly slowing the runtime.

#!/bin/bash

# ARGV: repo_path, HEAD_branch, start_year, end_year

for year in $(seq $3 $4); do
  for month in $(seq -w 1 12); do
    export DATE=$year-$month-01
    export COMMIT=$(cd $1 && git rev-list -1 --before="$DATE" $2)
    if [ -n "$COMMIT" ]; then
      echo $DATE $(uptime | cut -d' ' -f 1)
      ./dig.sh $1 $DATE $COMMIT
    fi
  done
done

Once we’ve got all the data, I wanted to make a bar chart race out of it. massage.rb to the rescue, collating the raw data, and then outputting a CSV.

#!/usr/bin/env ruby

require 'csv'
require 'pp'

data = {}
Dir['*.txt'].each do |file|
  File.readlines(file).each do |line|
    date, _ = file.split('.')
    count, name = line.strip.split(' author ')

    data[[name, date]] = count
  end
end

names = data.keys.map(&:first).uniq.sort
dates = data.keys.map(&:last).uniq.sort

out = []
out << [''] + dates.map{|d| Date.parse(d).strftime("%b %Y") }
names.each do |name|
  shortname = name.match(/^([^ ]+..)/)[1]
  out << [shortname + '.'] + dates.map{ |d| data[[name, d]] || "" }
end

puts out.map(&:to_csv)

Et voila (names anonymized to protect the guilty):

<!doctype html>

Git Rebase by Example - set_trace_func

Git Rebase by Example

20th Jun 2017 | Tags: git

Today at work, I determined that my current work in progress branch was going to want to be merged into production ahead of our next scheduled merge/deploy of master. To make it easier to merge into our production branch for a hot-fix deploy, I want to have the branch based off production instead of master.

So, I could do this manually:

$ git reset HEAD~ # move HEAD commit back to staging
$ git stash save # move uncommited changes to stash
# repeat until all my changes are in stash
$ git checkout production
$ git stash pop
$ git commit ...
# again, repeat until complete

However, this is a pain in the neck because it’s all manual, and it loses the commit metadata (timestamps, but also author info if you’re moving commits from multiple people). Also, because I prefer small merges to big merges, I’ve merged master into my branch a few times, and that means deciding to commit or wipe commits.

Instead, let’s git do all the work, and use git rebase to move my commits to their new home, and also filter out those merges.

Before getting started, we should backup my branch, by just doing a git checkout -b wip-rebase. Then, to confirm exactly what my changes were we can run git cherry -v origin/master. This will list all the commits on the current branch that are not on the specified branch.

Then, I want to start by getting rid of all my extra merges, and filter the branch to just these outbound changes. For this we’ll do an interactive rebase: git rebase -i acfad~, where acfad is the commit hash at the top of my change list (you can also just use a visual git browser or whatever to find the base commit for your changes). This will pop open an editor, and let me decide exactly how I want to handle each commit.

This is a powerful tool, and all its options are beyond the scope of this article, but it will let you do anything from editing a commit message, to combining or reordering commits, to executing shell commands between specific commits.

Here, we’re just dropping commits, so we can scroll down the editor here and any commit that wasn’t listed in our git cherry output we can change the pick to drop. Or, even easier, delete the line.

Now save and quit the editor, and git will do its job. We can verify with another git cherry -v origin/master and compare output now with earlier output. Note that our oldest commits are unchanged, but the commit hash for the newer ones is changed. This is because the hash of a commit depends on the hashes of its parents, and by deleting merged commits from our history we are of course unable to use them as parents anymore. Also, looking directly at the git log we can see that deleting all commits from one branch of a merge commit will also delete the merge itself. For our purposes, as long as the commit messages in both git cherry outputs are the same, then we’re doing fine.

Now that the busy work is behind us, we can get to the actual goal of moving the commits. The command for this will be git rebase --onto origin/production acfad~ – again we’re referring to the parent of the first commit in our branch.

This is the part that works exactly as I described above, git is “rewinding head to replay your work on top of it”. As with any merge (applying a commit in a changed context) you can expect to see some conflicts and can resolve them as usual. However when done, instead of commiting we’ll move on with git rebase --continue. If at any time things get confused and you make a mistake, git rebase --abort will roll you back to your branch as it was before rebasing.

Once you’re done with the conflicts, git will finalize the rebase behind the scenes, and leave you back at your current branch having updated the root parent, and touched all the commits on the way down. We can confirm this with a final git cherry -v origin/master.

Now is a good time to run specs (or let CI work that for you) and do whatever other testing you need to confirm everything functions as it should.

<!doctype html>

Joining Git Repositoires - set_trace_func

Joining Git Repositoires

15th May 2012 | Tags: git

At work, we provide an API for our app and maintain web-based documentation for said API. We originally had the documentation in a separate git repo, but as it makes much more sense to maintain the docs directly alongside the code it documents we wanted to merge the two repositories. This was done in two steps.

Moving files

First, we need to prep the docs repo such that the content is in a reasonable location, rather than the root directory. This is done fairly easily with git filter-branch (zsh):

% mkdir -p doc/api_docs
% git checkout -b for_transplant # work in a branch for safety
% for file in <files/dirs to move>;
    do git filter-branch --tree-filter \
      "test -e $file && mv $file doc/api_docs || echo skip" \
      HEAD;
    done

This goes through out commit history, and runs through each commit moving old content into a subdirectory. It’s slow in that it does a full history pass for each file, but I didn’t care to figure out how to move everything except the docs directory.

Merging the repos

First, for clarity, let’s make a new branch based off of the original commit in our destination repo:

% git log --oneline | tail -n 1
e7c9feb Initial commit
% git checkout e7c9feb
% git checokut -b doc_import

First, prepare the main repo for the incoming transplant by cloning a bare copy (git refuses to pull an external repo into a normal checkout).

% git clone --bare git@github.com:example/myapp.git myapp-bare

We can then pull in the commit objects from the other repository:

% cd myapp-bare
% git fetch -f ../api_doc_site for_transplant:api_docs

The -f tells git to ignore the different initial commits, and then we explicitly specify the remote and local branches to move commits to. You might need to resolve a merge conflict at this point, if for example both repositories committed a different .gitignore in their initial commit.

We can now go back to our regular repo and pull those branches in:

% cd ../myapp
% git remote add bare ../myapp-bare
% git pull bare api_docs

Finally, merging that branch into master gets things all up-to-date, and we can start unified work while maintaining the full original history of the documentation.

</th> <th><a name=”git<!doctype html>

Git Archaeology - set_trace_func

Git Archaeology

22nd Apr 2020 | Tags: git

A bit of experimentation and random googling left me with these three scripts.

#!/bin/bash

# ARGV: repo_path, date, commit

export DIR=`pwd`
cd $1

export DATE=$2

if [ -n "$3" ]; then
  export COMMIT=$3
else
  export COMMIT=$(git rev-list -1 --before="$DATE" master)
fi

export PATTERN='\t(app|lib)/.*\.(rb|js|erb|haml)$'

git ls-tree -r $COMMIT | egrep -o "$PATTERN" | while read f; do
  git blame -w -M -C -C --line-porcelain $COMMIT -- $f;
done | egrep -a '^author ' | sort | uniq -c > $DIR/$DATE.txt

#!/bin/bash

# ARGV: repo_path, HEAD_branch, start_year, end_year

for year in $(seq $3 $4); do
  for month in $(seq -w 1 12); do
    export DATE=$year-$month-01
    export COMMIT=$(cd $1 && git rev-list -1 --before="$DATE" $2)
    if [ -n "$COMMIT" ]; then
      echo $DATE $(uptime | cut -d' ' -f 1)
      ./dig.sh $1 $DATE $COMMIT
    fi
  done
done

Once we’ve got all the data, I wanted to make a bar chart race out of it. massage.rb to the rescue, collating the raw data, and then outputting a CSV.

#!/usr/bin/env ruby

require 'csv'
require 'pp'

data = {}
Dir['*.txt'].each do |file|
  File.readlines(file).each do |line|
    date, _ = file.split('.')
    count, name = line.strip.split(' author ')

    data[[name, date]] = count
  end
end

names = data.keys.map(&:first).uniq.sort
dates = data.keys.map(&:last).uniq.sort

out = []
out << [''] + dates.map{|d| Date.parse(d).strftime("%b %Y") }
names.each do |name|
  shortname = name.match(/^([^ ]+..)/)[1]
  out << [shortname + '.'] + dates.map{ |d| data[[name, d]] || "" }
end

puts out.map(&:to_csv)

Et voila (names anonymized to protect the guilty):

<!doctype html>

Git Rebase by Example - set_trace_func

Git Rebase by Example

20th Jun 2017 | Tags: git

So, I could do this manually:

$ git reset HEAD~ # move HEAD commit back to staging
$ git stash save # move uncommited changes to stash
# repeat until all my changes are in stash
$ git checkout production
$ git stash pop
$ git commit ...
# again, repeat until complete

Instead, let’s git do all the work, and use git rebase to move my commits to their new home, and also filter out those merges.

Now is a good time to run specs (or let CI work that for you) and do whatever other testing you need to confirm everything functions as it should.

<!doctype html>

Joining Git Repositoires - set_trace_func

Joining Git Repositoires

15th May 2012 | Tags: git

Moving files

First, we need to prep the docs repo such that the content is in a reasonable location, rather than the root directory. This is done fairly easily with git filter-branch (zsh):

% mkdir -p doc/api_docs
% git checkout -b for_transplant # work in a branch for safety
% for file in <files/dirs to move>;
    do git filter-branch --tree-filter \
      "test -e $file && mv $file doc/api_docs || echo skip" \
      HEAD;
    done

Merging the repos

First, for clarity, let’s make a new branch based off of the original commit in our destination repo:

% git log --oneline | tail -n 1
e7c9feb Initial commit
% git checkout e7c9feb
% git checokut -b doc_import

First, prepare the main repo for the incoming transplant by cloning a bare copy (git refuses to pull an external repo into a normal checkout).

% git clone --bare git@github.com:example/myapp.git myapp-bare

We can then pull in the commit objects from the other repository:

% cd myapp-bare
% git fetch -f ../api_doc_site for_transplant:api_docs

We can now go back to our regular repo and pull those branches in:

% cd ../myapp
% git remote add bare ../myapp-bare
% git pull bare api_docs

Finally, merging that branch into master gets things all up-to-date, and we can start unified work while maintaining the full original history of the documentation.

” class=”anchor”> </th></tr>

  <tr><th>ruby<!doctype html>

Building an Object Graph in Rails - set_trace_func

Building an Object Graph in Rails

4th Jun 2015 | Tags: ruby rails

I was needing to do some object cleanup in our rails app the other day, and purge some malformed objects, so I put together a quick script using some ActiveRecord reflection to walk the object chain.

# Squelch SQL logs, if you're running from rails console
ActiveRecord::Base.logger.level = 1

# Output formatters. Trailing ':' on both, plus the staggered indent
# of 4n and 4n+2 makes the output valid YAML, if automated analysis
# is called for.
def puts_node(node, indent)
  puts "    "*indent + node.class.name + "#" + node.id.to_s + ":"
end
def puts_assoc(assoc, indent)
  puts "    "*indent + "  " + assoc.to_s + ":"
end

def puts_tree(node, seen=[], indent=0)
  puts_node(node, indent)

  unless seen.include? node
    # seen maintains a list of nodes to avoid mutual recursion
    seen << node
    node.class.reflections.keys.each do |assoc|
      # To see all locations an object is referenced, get rid of the
      # "- seen" here. I only cared about which objects were present
      # anywhere in the tree, so this was fine.
      associated = Array(node.send(assoc)) - seen
      next if associated.empty?

      # Print an entry for the association, then recurse
      puts_assoc(assoc, indent)
      associated.each do |subnode|
        puts_tree(subnode, seen, indent+1)
      end
    end
  end

  # Also outputs a footer listing all seen objects once, take it or leave it
  if indent.zero?
    puts
    seen.each {|node| puts_node(node, indent)}
  end

  nil # return nil to avoid flooding terminal in rails console
end

# Usage: call with the root node for the object graph
puts_tree(User.find(42))

Worked like a charm, and made it easy to compare my bad object with other good ones. Just be careful of any global objects (a common shared subscription package, for instance, that has_many :users) that could lead to traversing your entire database, or logging associations that could overwhelm your output on older or heavily used objects. Subtracting a blacklist from reflection keys on line 18 would do the trick there.

<!doctype html>

No More Bundle Exec - set_trace_func

No More Bundle Exec

6th Sep 2012 | Tags: programming ruby

Update 2023: Just use direnv with layout ruby.

Bundler is pretty darn good. Installing all your gems globally sucks. bundle install --path does a great job of fixing that but it means you need to bundle exec any shell commands you want to run, which again sucks. There are lots of attempts to fix this, but they’re all fairly convoluted.

I’m a fan of simpler solutions wherever possible. I use zsh as my shell, which has a handler you can hook into if the command you’re trying to run is not found. It’s a simple matter to hook that into a custom shell script from your ~/.zshrc:

function command_not_found_handler() {
    ~/bin/command-not-found $*
}

I know bash supports this kind of handler (Ubuntu uses it to provide command helpers for not-yet-installed programs) but I don’t know the exact details. Alas, my favorite shell ever, fish, only provides the executable to its corresponding helper, so while it can suggest an alternate command, it can’t auto-correct it.

My script happens to be in Ruby, but it could just as easily be a standard shell script as all I’m doing is some file existence tests:

#!/usr/bin/env ruby

# ARGV is the entire command we wanted to run, but we
# really only care about the actual executable for fallbacks
command = ARGV.first

def run(cmd)
  $stderr.puts "Running #{cmd.inspect} instead"
  system(cmd)
end

case
when File.exist?("./.bundle/config") && File.exist?("./bin/#{command}")
  run("bundle exec #{ARGV.join(' ')}")

else
  exit 127
end

Now, as long as you’re being sure to bundle install --binstubs it should Just Work. And because it only functions if you’re in a directory that’s been bundled, you don’t run into the security risks that you would by trying to get ./bin added to your $PATH directly.

Lastly, the case statement instead of an if is a bit redundant in the simple case above, I’ve actually got a few more filters for things like isolate and git - don’t forget to quote anything that might need space literals:

# Paste git repo url to clone it
when command =~ /^git(@|:\/\/).*\.git$/
  run("git clone #{command.inspect}")

# paste compressed url to download+extract it
when command =~ /^(?:ftp|https?):\/\/.+\.t(?:ar\.)?gz$/
  run("curl #{command.inspect} | tar xzv")

when File.exist?("./tmp/isolate/ruby-1.8/bin/#{command}")
  run("rake isolate:sh['#{ARGV.join(' ')}']")

<!doctype html>

Parsing JSON in SQL - set_trace_func

Parsing JSON in SQL

19th Nov 2011 | Tags: ruby rails sql

The Problem: You have a database column with some data serialzed as JSON in it that you’d like to pull out into its own column to index it.

The Solution: Run a data migration to pull the value out. Table has 5 million rows and you don’t want to round trip all that data through ActiveRecord? Just parse the json directly with some SQL:

def json(key, field='params')
  key_json = "\"#{key}\":"

  # key start/end locations, including ""
  k_a = "LOCATE('#{key_json}', #{field})"
  k_z = "LOCATE('\"', #{field}, #{k_a}+1)" # this is terminating "

  # is there a space after colons?
  spad = "IF(LOCATE('\": ', #{field}), 1, 0)"

  # is value a string?
  val_string = "LOCATE(CONCAT('#{key_json}', IF(#{spad},' ',''), '\"'), #{field}, #{k_a})"
  qpad = "IF(#{val_string}, 1, 0)"

  # value start/end locations, excluding "" if present
  v_a = "(#{k_z}+1 + 1 + #{spad} + #{qpad})" # 1 for colon, spad for optional space, qpad for possible quote

  end_if_string = "LOCATE('\"', #{field}, #{v_a})"
  end_if_not_string = "IF(LOCATE(',', #{field}, #{v_a}), LOCATE(',', #{field}, #{v_a}), LOCATE('}', #{field}, #{v_a}))"

  v_z = "IF(#{val_string}, #{end_if_string}, #{end_if_not_string})"

  value_string = "SUBSTRING(#{field} FROM #{v_a} FOR (#{v_z} - #{v_a}))"
  "IF(#{k_a}, #{value_string}, NULL)"
end

up do
  execute "
    UPDATE model_table
    SET status = #{json('status')}
  "
end

The generated sql looks pretty gnarly but mysql ran through it stupidly fast. I shudder to think how long it’d take activerecord to load and update each record individually.

<!doctype html>

Isolating Rails - set_trace_func

Isolating Rails

19th Jan 2011 | Tags: rails ruby

Rails 3 is now very friendly with regards to dropping Bundler support, only loading it if it’s installed and a Gemfile exists. Since Isolate is so awesome, I thought I’d just drop a quick script in here to convert an existing Rails app to use Isolate instead of Bundler.

#!/usr/bin/env ruby

require 'fileutils'

File.open("Isolate", 'w') do |isolate|
  File.readlines("Gemfile").each do |line|
    next if line =~ /^\w*#/
    next if line =~ /^source/
    next if line =~ /^\w*$/

    line.sub!(/, :require.*(,|$)/, '\1')
    line.sub!(/^([ \t#]*)group/, '\1environment')
    
    if line =~ /:git/
      line = "# Don't use git, build it as a gem\n# " + line
    end

    isolate.puts line
  end
end

File.open("config/boot.rb", 'a') do |boot|
  boot.puts
  boot.puts("require 'isolate/now'")
end

FileUtils.rm('Gemfile')

This should convert an existing Gemfile to an Isolate file, remove the Gemfile (so that rails won’t try to load it), and update the app to load Isolate appropriately.

I’m basically only guessing that the group/environment setup is correct, so if anyone has any corrections to this let me know and I’ll update it.

The latest version of rubygems seems to be more gung-ho about trying to clean up old versions of gems. This is fine, except that OSX seems to be very protective of its bundled gem directory, refusing to uninstall gems located there even though I’m running the gem command via sudo.

As a result, while gem list tells me that I’ve got activerecord v 2.2.2 and 1.15.6 installed, gem cleanup fails miserably:

jamie@juliet ~> sudo gem cleanup
Cleaning up installed gems...
Attempting to uninstall activerecord-1.15.6
ERROR:  While executing gem ... (Gem::InstallError)
    Unknown gem activerecord = 1.15.6

I appreciate Leopard not wanting to break the bundled versions of gems, but having a functioning gem cleanup is helpful to me. Turns out it’s pretty easy to just prevent rubygems from checking the Ruby.framework gem path, just create (or edit) ~/.gemrc and include the following:

gempath:
  - /Library/Ruby/Gems/1.8
  - /Users/jamie/.gem/ruby/1.8

Async-observer is great. Fast, easy to use API, and it Just Works. The downside is that the backend, Beanstalkd, doesn’t support persistent messages in case the server crashes. I hear it’s on the roadmap, though.

However, there’s another messaging backend, RabbitMQ, that seems just as easy to get set up, and does support persistent messages. So, how to get these two bits of tech working together? Well, if you’re hosting your app on Thin (or another app server that runs in EventMachine), it’s pretty straightforward.

First, install the amqp ruby library to connect to rabbit, and then add a tiny bit of setup.

In config/environment.rb:

require 'mq'
class BeanstalkPoolImpersonator
  def initialize(opts={})
    @opts = opts
  end

  def connect
    connection = AMQP.connect(@opts)
    @channel = channel = MQ.new(connection)
  end

  def use(queue)
    @queue = MQ::Queue.new(@channel, queue)
  end

  def yput(obj, pri, delay, ttr)
    p [obj, pri, delay, ttr]
    @queue.publish(YAML.dump(obj))
  end

  def last_server
    :last_server_stub
  end

  def subscribe(*args, &blk)
    @queue.subscribe(*args, &blk)
  end
end

Then, instead of connecting via Beanstalk::Pool.new, do this:

AsyncObserver::Queue.queue = BeanstalkPoolImpersonator.new()

You can pass an options hash to the new call, providing user, pass, vhost, host, or port as necessary.

Then, in your workers, load up the async_observer worker class, and extend like so:

class RabbitWorker < AsyncObserver::Worker
  def run()
    EM.run do
      AsyncObserver::Queue.queue.connect
      AsyncObserver::Queue.queue.use('1.0')
      AsyncObserver::Queue.queue.subscribe do |headers, msg|
        job = OpenStruct.new(:ybody => YAML.load(msg), :body => msg, :stats => [])
        job.id = headers.properties[:delivery_tag]
        safe_dispatch(job)
      end
    end
  end
end

Create the new worker the same way you would for the AO::Worker, and you’re set:

    RabbitWorker.new(binding).run()

Note: I’m maintaining a merb port of async-observer on github.

class Array
  alias old_map map

  def map(method=nil, *args, &blk)
    return old_each(&blk) if block_given?
    old_map do |e|
      e.send(method, *args)
    end
  end
end

ary = %w(hello world)
puts ary.map(:reverse).join(" ")
puts ary.map(:*, 2).join(" ")
puts ary.map(:slice, 1, 3).join(" ")

It should be trivial to do this for the other common enumerable methods, the only places I see Symbol#to_proc used, anyway.

I posit that the answer is question 2.

Question #2: Why do I need to do this method hackery in Array instead of Enumerable? I can understand Array having its own definitions of the standard collection methods for performance reasons, but in a properly OO system, that should be transparent to me.

#!/usr/bin/env ruby col = ARGV.pop.to_i-1 while line = gets puts line.chomp.split(/\s+/)[col] end

For when you just want a list of filenames from version control, hg st | grep '?' | col 2

And because I can never remember the standard unix tool that does the same thing, and awk is awkward.

As an update to a previous article,

if $specs_timed.nil? && ENV.has_key?('SLOW')
  $specs_timed = true
  $timings = []

  Spec::Example::ExampleGroup.prepend_before do
    @start = Time.now
  end
  Spec::Example::ExampleGroup.append_after do
    elapsed = Time.now - @start
    if elapsed > ENV['SLOW'].to_f
      $timings << [elapsed, "#{self.class.description} #{description}"]
    end
  end

  at_exit do
    puts "\nSlow Specs:"
    $timings.sort{|a,b| a.first <=> b.first}.each do |time, name|
      puts " %7.4f #{name}" % time
    end
    puts "  None!" if $timings.empty?
  end
end

Then, simply run

rake SLOW=0.1

Two gotchas if you’re using Rails though: instead of hooking S::E::ExampleGroup, you’ll need to hook Spec::Rails::Example::RailsExampleGroup. Second, if you have any spec failures the timings don’t seem to get output, since spec/rails aborts execution after failing.

It seems that there were people making both audio and video recordings of talks at RejectConf this year.

I gave a short talk on the RCov hack I’ve been working on (mentioned previously) which seemed to go well. The quality of the video isn’t that great, but Geoff’s recording of just the audio is good. It should go along well with the slides (pdf) if anyone’s curious.

Pat Eyler’s April contest asks what changes could improve Ruby without losing the feel of the language. I’ve got four ideas. Even I would consider them quite radical, and don’t think they’re likely to occur, but if in the future Matz decided that sweeping changes were in order and that backwards compatability wasn’t an issue, they’d be on my wish list.

First, trim Ruby’s core. Go through the whole shooting match and pull out anything that isn’t likely to be used by at least 80% of the population, and move it into the standard library. Pay special attention to anything written in pure Ruby. CSV, Generator, PrettyPrint, RSS, RUNIT, Rinda, SOAP, and others could be pushed out of Ruby’s core.

Similarly, anything with redundancies probably should be moved out too - Ruby’s current core has getopts.rb, GetOptLong, OptionParser, Options, and there’s a handful of 3rd party libraries available to parse commandline options. Do these all really need to be in core? (Heck, do they all need to be in StdLib for that matter? getopts.rb has as its only comment that it is deprecated and to use another library instead.)

Cleaning up Ruby’s core would allow alternate implementations to focus more specifically on what is needed to get Ruby running, without the distractions of extra bundled libraries. It would also provide a smaller base of knowledge for those new to Ruby to learn before they can claim to “know” the language.

My second wish would be to improve the OO-ness of Ruby. Yes, we’re miles ahead of Python and its large number of global functions, but a split from some of the old Perlisms could possibly result in an improvement of maintainability. If a number of the magic global variables were scrapped (or at least replaced with the English module) someone who doesn’t muck about with them on a day-to-day basis can at least have a chance of figuring out what’s going on. I know that I still need to look up $$, $_, $’ and others when I see them, and I’ve been working in Ruby for about a year and a half now.

If everyone were forced to use English-style accessors, and actually go through the OO interface for information (ie, Regexp.last_match instead of magic globals) it should lead to more self-documenting code.

Third, I’d like to see what could be done to trim down the syntax a little. Do we really need ?A to provide us the same information as “A”[0]? I’m loathe to suggest %w() and friends for trimming as I find them useful fairly often, but that’s also something that could be looked at. Having the language specified with a parser generator means that alternative implementations can be almost guaranteed to be matching the spec with little effort. Making the core language simpler, with fewer gotchas, also improves the speed at which it can be learned.

Lastly, I’d like to see a step taken towards standardizing on some kind of Unicode, and updating the core language (strings, regexps, and all) to make it as seamless to use as possible. The Rails guys have got things started with multibyte strings in ActiveSupport, but I think it would need to be baked in to the language to be as comprehensive as possible.

All four of these changes would require a lot of work, and likely result in incompatabilities from current versions of Ruby (excepting possibly the last one). However, I think taking these as a base to work on streamlining the language, it could result in a Ruby that is easier to learn, easier to work with, and easier to duplicate or extend, while still retaining the core features that make it what it is.

Comments

Best entry so far in my opinion. I agree with all of your points, particularly the first two.

Peter Cooper, at 19:01, Apr 28 2007

Finally got the comment problem fixed, looks like an older version of Mephisto was doing weird things with the page cache. If anyone happens to notice that comments are back up, you can join in now :)

Jamie, at 19:13, Apr 30 2007

Rplug has a new release up, which should now be useful for the world at large, as it has gained support for projects in subversion.

The update process now preserves the .svn turds rather than breaking the working copy, which is possible now that SourceControl has taught svn (and svk) how the manifest command should be implemented (11 lines of ruby).

I should probably do a check after I’ve done the export and cull any now-empty directories from the plugin dir, but that’ll come in time, I’m sure.

RPlug and SourceControl now officially have Gems out. SourceControl is probably useless for anybody at the moment, but if you are working on a rails repository under SVK and want to manage SVN-backed plugins, RPlug should handle it just fine. Just gem install rplug -y. More compatability to come in the future.

[Updates below]

I’ve been having problems getting SourceControl deployed, turns out (unsurprisingly) to be user error - I’m new to this whole rubyforge/gem scene.

So, for the record, prior to releasing a gem using Hoe, one needs to get rubyforge configured. For me, this wound up being:

$ rubyforge setup
$ rubyforge config rplug
$ rubyforge config sourcecontrol

After all that, SourceControl is deploying just fine.

I’m presuming that the initial problem was that the gem (and internal file structure) is source_control, but due to limitations on rubyforge the project name is sourcecontrol - somewhere along the way that confusion stopped it from working.

Today, I went mucking around with the packages for it, removed the old one named ‘sourcecontrol’ and added ‘source_control’ - removing ~/.rubyforge/auto-config.yml and re-running the rubyforge setup/config picked up the new package id, and everything seems to run just fine now.

Well, it’s got the basic functionality it needs, so I’m about to put out a 0.1.0 gem for RPlug. It has a dependency on SourceControl, which I think only deserves a 0.0.5 release because it only does the bare minimum to support RPlug at the moment.

Both projects are entirely up in subversion if anyone wants to check them out, but they’re not quite ready for public consumption at the moment.

Example usage and output follows.

% rplug install exception_logger http://svn.techno-weenie.net/projects/plugins/exception_logger svn
Recorded exception_logger, run 'rplug update' to pull the latest revision

% rplug update
Working in project dir /home/jamie/dev/redvase
Updating exception_logger...
  upgrading to revision 2733
  updating local repository
  Done.
Updating mocha...
  Done.
Updating helper_test...
  Done.
Updating arts...
  Done.
Updating liquid...
  Done.

% rplug status
Working in project dir /home/jamie/dev/redvase
Managing the following plugins:
  arts, revision 70
  exception_logger, revision 2
  helper_test, revision 85
  liquid, revision 140
  mocha, revision 99
Not Managing the following plugins:
  test_timer

% rplug update -p exception_logger -r 2563
Working in project dir /home/jamie/dev/redvase
Updating exception_logger...
  upgrading to revision 2563
  updating local repository
  Done.

For those new to the blog, I’m currently reinventing a few wheels here - RPlug is a replacement for Piston that stores meta-info in config/plugins.yml rather than the version control system, and which does not tie itself directly to Subversion even when given a compatible system (like SVK). It does this by using SourceControl (itself intended as a replacement for RSCM) to handle the interface to the SCM system.

class BrokenError < StandardError
  def backtrace
    raise(StandardError.new)
  end
end

begin
  raise BrokenError.new
rescue e
  puts 'rescued'
end

Because of the exception in the backtrace generation, processing just dies. If you have an at_exit block, it will still be run, so I suppose I’m not really crashing the ruby interpreter, I suppose, but it comes close.

Found this one out migrating a rails app from 1.1.6 to 1.2. Instead of doing this:

render 'controller/action'

the deprecation warning suggests the following:

render :file => 'controller/action'

Unfortunately, this causes the error if you’re still trying to run in 1.1.6. A more complete fix is to make sure to add use_full_path to the render call to prevent an older TemplateError from horking, like so:

render :file => 'controller/action', :use_full_path => true

One of the big things I learned at University was that while “Recursion is a Wonderful Thing” (Thank you, Dr. Roelants), sometimes the performance can really hurt. Those times, it can pay to spend the effort turning that recursive function into a simple loop. Sure, it might not be as clean, or as elegant, or as natural to understand, but we’re looking at performance here, right?

Ryan Davis recently posted about using RubyInline to optimize a recursive factorial method. He ended with a caveat that sometimes you need to look at other things than just moving the code into C for speed. His idea was to cache the data as it goes along. There are times when that won’t help you in the log run (for example, generating a stats graph where caching as you draw helps, but the cached values will be stale the next time you need to do it) but changing it around to iterative can sometimes give you a further speedup.

def fib_iter(n)
  return 1 if n < 3  
  f = f1 = 1
  (2..n).each do
    f, f1 = (f+f1), f
  end
  f
end

The benchmarking speaks for itself. (Same parameters as Ryan’s benching, 10,000 runs doing fib(15)):

                      user     system      total        real
fib-ruby         21.180000   3.640000  24.820000 ( 24.989140)
fib-hash-reset    0.510000   0.070000   0.580000 (  0.609976)
fib-cache-reset   0.510000   0.050000   0.560000 (  0.570715)
fib-iter          0.160000   0.020000   0.180000 (  0.209565)
fib-hash          0.020000   0.000000   0.020000 (  0.034616)
fib-cached        0.020000   0.010000   0.030000 (  0.035222)

Benchmarks for fib-ruby and fib-cached come from Ryan’s post. fib-iter and fib-hash are mine.

The two “-reset” methods are indicative of times when global caching won’t help you, which is still a significant speedup over the uncached versions. (For fib(15), uncached will need ~610 method calls, compared to ~15) The iterative method is about 1/3 their speed, but when you can globally cache you can get huge gains - if I increased the number of runs in the benchmark, the discrepancy between fib-iter and fib-cached would increase even more.

So once again, it seems that there’s a different best solution for two different problems.

And the fib-hash benchmark? It’s not significantly faster than Ryan’s fib-cached method, but it bumps the fib logic from a method that uses a hash into the hash itself. It’s a neat trick I picked up a while ago, but probably too ugly to make significant use of unless your benchmarking tells you otherwise - it’s really hard to read at first glance:

def hashfib(n)
  return 1 if n <= 1
  h = Hash.new{|h,k| h[k] = h[k-1] + h[k-2] }
  h[1] = 1
  h[2] = 1
  h[n]
end

The cached version uses @@h instead of h, and

=s it.

I was doing a bit of data processing the other night. A little copying here, a bit of typing there, formatting into YAML, then loaded into a Ruby script. Loop through the hashes YAML loaded, and try to make some sense out of it.

I’m happy to say that I wound up doing the most comfortable thing for munging the data, and it turned out pretty well: OpenStruct. For those who don’t know about it (require ‘ostruct’), OpenStruct is exactly as the name says. It’s a struct, in that it just holds data, but it is open for extending after you’ve created it. One can almost treat it like a Hash, but with method calls instead of indexing. (In fact, this week’s RubyQuiz was converting YAML-loaded Hashes to OpenStructs)

What I was doing was looping through the Hashes, and creating OpenStructs on the fly to hold the data. At the same time, I was back-referring to previous OpenStructs and appending data to them. I didn’t think much of it until I thought to myself that I needed to do some calculations on the data, and the most logical spot for it was in one of my OpenStruct objects.

I was disappointed for a moment because I knew the methods didn’t fit in OpenStruct itself, when I realized that it was just time to refactor a bit - take the OpenStructs that were holding the data, promote them to instances of a concrete class, and fit the logic in there.

A quick class def, a handful of attr_accessors, rename the OpenStruct instantiation to my new class, and I was off again, none worse for the wear. Ahh, duck typing, I couldn’t have done it without you.

</th> <th><a name=”ruby<!doctype html>

Building an Object Graph in Rails - set_trace_func

Building an Object Graph in Rails

4th Jun 2015 | Tags: ruby rails

I was needing to do some object cleanup in our rails app the other day, and purge some malformed objects, so I put together a quick script using some ActiveRecord reflection to walk the object chain.

# Squelch SQL logs, if you're running from rails console
ActiveRecord::Base.logger.level = 1

# Output formatters. Trailing ':' on both, plus the staggered indent
# of 4n and 4n+2 makes the output valid YAML, if automated analysis
# is called for.
def puts_node(node, indent)
  puts "    "*indent + node.class.name + "#" + node.id.to_s + ":"
end
def puts_assoc(assoc, indent)
  puts "    "*indent + "  " + assoc.to_s + ":"
end

def puts_tree(node, seen=[], indent=0)
  puts_node(node, indent)

  unless seen.include? node
    # seen maintains a list of nodes to avoid mutual recursion
    seen << node
    node.class.reflections.keys.each do |assoc|
      # To see all locations an object is referenced, get rid of the
      # "- seen" here. I only cared about which objects were present
      # anywhere in the tree, so this was fine.
      associated = Array(node.send(assoc)) - seen
      next if associated.empty?

      # Print an entry for the association, then recurse
      puts_assoc(assoc, indent)
      associated.each do |subnode|
        puts_tree(subnode, seen, indent+1)
      end
    end
  end

  # Also outputs a footer listing all seen objects once, take it or leave it
  if indent.zero?
    puts
    seen.each {|node| puts_node(node, indent)}
  end

  nil # return nil to avoid flooding terminal in rails console
end

# Usage: call with the root node for the object graph
puts_tree(User.find(42))

<!doctype html>

No More Bundle Exec - set_trace_func

No More Bundle Exec

6th Sep 2012 | Tags: programming ruby

Update 2023: Just use direnv with layout ruby.

function command_not_found_handler() {
    ~/bin/command-not-found $*
}

My script happens to be in Ruby, but it could just as easily be a standard shell script as all I’m doing is some file existence tests:

#!/usr/bin/env ruby

# ARGV is the entire command we wanted to run, but we
# really only care about the actual executable for fallbacks
command = ARGV.first

def run(cmd)
  $stderr.puts "Running #{cmd.inspect} instead"
  system(cmd)
end

case
when File.exist?("./.bundle/config") && File.exist?("./bin/#{command}")
  run("bundle exec #{ARGV.join(' ')}")

else
  exit 127
end

# Paste git repo url to clone it
when command =~ /^git(@|:\/\/).*\.git$/
  run("git clone #{command.inspect}")

# paste compressed url to download+extract it
when command =~ /^(?:ftp|https?):\/\/.+\.t(?:ar\.)?gz$/
  run("curl #{command.inspect} | tar xzv")

when File.exist?("./tmp/isolate/ruby-1.8/bin/#{command}")
  run("rake isolate:sh['#{ARGV.join(' ')}']")

<!doctype html>

Parsing JSON in SQL - set_trace_func

Parsing JSON in SQL

19th Nov 2011 | Tags: ruby rails sql

The Problem: You have a database column with some data serialzed as JSON in it that you’d like to pull out into its own column to index it.

def json(key, field='params')
  key_json = "\"#{key}\":"

  # key start/end locations, including ""
  k_a = "LOCATE('#{key_json}', #{field})"
  k_z = "LOCATE('\"', #{field}, #{k_a}+1)" # this is terminating "

  # is there a space after colons?
  spad = "IF(LOCATE('\": ', #{field}), 1, 0)"

  # is value a string?
  val_string = "LOCATE(CONCAT('#{key_json}', IF(#{spad},' ',''), '\"'), #{field}, #{k_a})"
  qpad = "IF(#{val_string}, 1, 0)"

  # value start/end locations, excluding "" if present
  v_a = "(#{k_z}+1 + 1 + #{spad} + #{qpad})" # 1 for colon, spad for optional space, qpad for possible quote

  end_if_string = "LOCATE('\"', #{field}, #{v_a})"
  end_if_not_string = "IF(LOCATE(',', #{field}, #{v_a}), LOCATE(',', #{field}, #{v_a}), LOCATE('}', #{field}, #{v_a}))"

  v_z = "IF(#{val_string}, #{end_if_string}, #{end_if_not_string})"

  value_string = "SUBSTRING(#{field} FROM #{v_a} FOR (#{v_z} - #{v_a}))"
  "IF(#{k_a}, #{value_string}, NULL)"
end

up do
  execute "
    UPDATE model_table
    SET status = #{json('status')}
  "
end

The generated sql looks pretty gnarly but mysql ran through it stupidly fast. I shudder to think how long it’d take activerecord to load and update each record individually.

<!doctype html>

Isolating Rails - set_trace_func

Isolating Rails

19th Jan 2011 | Tags: rails ruby

#!/usr/bin/env ruby

require 'fileutils'

File.open("Isolate", 'w') do |isolate|
  File.readlines("Gemfile").each do |line|
    next if line =~ /^\w*#/
    next if line =~ /^source/
    next if line =~ /^\w*$/

    line.sub!(/, :require.*(,|$)/, '\1')
    line.sub!(/^([ \t#]*)group/, '\1environment')
    
    if line =~ /:git/
      line = "# Don't use git, build it as a gem\n# " + line
    end

    isolate.puts line
  end
end

File.open("config/boot.rb", 'a') do |boot|
  boot.puts
  boot.puts("require 'isolate/now'")
end

FileUtils.rm('Gemfile')

This should convert an existing Gemfile to an Isolate file, remove the Gemfile (so that rails won’t try to load it), and update the app to load Isolate appropriately.

I’m basically only guessing that the group/environment setup is correct, so if anyone has any corrections to this let me know and I’ll update it.

As a result, while gem list tells me that I’ve got activerecord v 2.2.2 and 1.15.6 installed, gem cleanup fails miserably:

jamie@juliet ~> sudo gem cleanup
Cleaning up installed gems...
Attempting to uninstall activerecord-1.15.6
ERROR:  While executing gem ... (Gem::InstallError)
    Unknown gem activerecord = 1.15.6

gempath:
  - /Library/Ruby/Gems/1.8
  - /Users/jamie/.gem/ruby/1.8

First, install the amqp ruby library to connect to rabbit, and then add a tiny bit of setup.

In config/environment.rb:

require 'mq'
class BeanstalkPoolImpersonator
  def initialize(opts={})
    @opts = opts
  end

  def connect
    connection = AMQP.connect(@opts)
    @channel = channel = MQ.new(connection)
  end

  def use(queue)
    @queue = MQ::Queue.new(@channel, queue)
  end

  def yput(obj, pri, delay, ttr)
    p [obj, pri, delay, ttr]
    @queue.publish(YAML.dump(obj))
  end

  def last_server
    :last_server_stub
  end

  def subscribe(*args, &blk)
    @queue.subscribe(*args, &blk)
  end
end

Then, instead of connecting via Beanstalk::Pool.new, do this:

AsyncObserver::Queue.queue = BeanstalkPoolImpersonator.new()

You can pass an options hash to the new call, providing user, pass, vhost, host, or port as necessary.

Then, in your workers, load up the async_observer worker class, and extend like so:

class RabbitWorker < AsyncObserver::Worker
  def run()
    EM.run do
      AsyncObserver::Queue.queue.connect
      AsyncObserver::Queue.queue.use('1.0')
      AsyncObserver::Queue.queue.subscribe do |headers, msg|
        job = OpenStruct.new(:ybody => YAML.load(msg), :body => msg, :stats => [])
        job.id = headers.properties[:delivery_tag]
        safe_dispatch(job)
      end
    end
  end
end

Create the new worker the same way you would for the AO::Worker, and you’re set:

    RabbitWorker.new(binding).run()

Note: I’m maintaining a merb port of async-observer on github.

class Array
  alias old_map map

  def map(method=nil, *args, &blk)
    return old_each(&blk) if block_given?
    old_map do |e|
      e.send(method, *args)
    end
  end
end

ary = %w(hello world)
puts ary.map(:reverse).join(" ")
puts ary.map(:*, 2).join(" ")
puts ary.map(:slice, 1, 3).join(" ")

It should be trivial to do this for the other common enumerable methods, the only places I see Symbol#to_proc used, anyway.

I posit that the answer is question 2.

#!/usr/bin/env ruby col = ARGV.pop.to_i-1 while line = gets puts line.chomp.split(/\s+/)[col] end

For when you just want a list of filenames from version control, hg st | grep '?' | col 2

And because I can never remember the standard unix tool that does the same thing, and awk is awkward.

As an update to a previous article,

if $specs_timed.nil? && ENV.has_key?('SLOW')
  $specs_timed = true
  $timings = []

  Spec::Example::ExampleGroup.prepend_before do
    @start = Time.now
  end
  Spec::Example::ExampleGroup.append_after do
    elapsed = Time.now - @start
    if elapsed > ENV['SLOW'].to_f
      $timings << [elapsed, "#{self.class.description} #{description}"]
    end
  end

  at_exit do
    puts "\nSlow Specs:"
    $timings.sort{|a,b| a.first <=> b.first}.each do |time, name|
      puts " %7.4f #{name}" % time
    end
    puts "  None!" if $timings.empty?
  end
end

Then, simply run

rake SLOW=0.1

It seems that there were people making both audio and video recordings of talks at RejectConf this year.

Comments

Best entry so far in my opinion. I agree with all of your points, particularly the first two.

Peter Cooper, at 19:01, Apr 28 2007

Finally got the comment problem fixed, looks like an older version of Mephisto was doing weird things with the page cache. If anyone happens to notice that comments are back up, you can join in now :)

Jamie, at 19:13, Apr 30 2007

Rplug has a new release up, which should now be useful for the world at large, as it has gained support for projects in subversion.

I should probably do a check after I’ve done the export and cull any now-empty directories from the plugin dir, but that’ll come in time, I’m sure.

[Updates below]

I’ve been having problems getting SourceControl deployed, turns out (unsurprisingly) to be user error - I’m new to this whole rubyforge/gem scene.

So, for the record, prior to releasing a gem using Hoe, one needs to get rubyforge configured. For me, this wound up being:

$ rubyforge setup
$ rubyforge config rplug
$ rubyforge config sourcecontrol

After all that, SourceControl is deploying just fine.

Both projects are entirely up in subversion if anyone wants to check them out, but they’re not quite ready for public consumption at the moment.

Example usage and output follows.

% rplug install exception_logger http://svn.techno-weenie.net/projects/plugins/exception_logger svn
Recorded exception_logger, run 'rplug update' to pull the latest revision

% rplug update
Working in project dir /home/jamie/dev/redvase
Updating exception_logger...
  upgrading to revision 2733
  updating local repository
  Done.
Updating mocha...
  Done.
Updating helper_test...
  Done.
Updating arts...
  Done.
Updating liquid...
  Done.

% rplug status
Working in project dir /home/jamie/dev/redvase
Managing the following plugins:
  arts, revision 70
  exception_logger, revision 2
  helper_test, revision 85
  liquid, revision 140
  mocha, revision 99
Not Managing the following plugins:
  test_timer

% rplug update -p exception_logger -r 2563
Working in project dir /home/jamie/dev/redvase
Updating exception_logger...
  upgrading to revision 2563
  updating local repository
  Done.

class BrokenError < StandardError
  def backtrace
    raise(StandardError.new)
  end
end

begin
  raise BrokenError.new
rescue e
  puts 'rescued'
end

Found this one out migrating a rails app from 1.1.6 to 1.2. Instead of doing this:

render 'controller/action'

the deprecation warning suggests the following:

render :file => 'controller/action'

render :file => 'controller/action', :use_full_path => true

def fib_iter(n)
  return 1 if n < 3  
  f = f1 = 1
  (2..n).each do
    f, f1 = (f+f1), f
  end
  f
end

The benchmarking speaks for itself. (Same parameters as Ryan’s benching, 10,000 runs doing fib(15)):

                      user     system      total        real
fib-ruby         21.180000   3.640000  24.820000 ( 24.989140)
fib-hash-reset    0.510000   0.070000   0.580000 (  0.609976)
fib-cache-reset   0.510000   0.050000   0.560000 (  0.570715)
fib-iter          0.160000   0.020000   0.180000 (  0.209565)
fib-hash          0.020000   0.000000   0.020000 (  0.034616)
fib-cached        0.020000   0.010000   0.030000 (  0.035222)

Benchmarks for fib-ruby and fib-cached come from Ryan’s post. fib-iter and fib-hash are mine.

So once again, it seems that there’s a different best solution for two different problems.

def hashfib(n)
  return 1 if n <= 1
  h = Hash.new{|h,k| h[k] = h[k-1] + h[k-2] }
  h[1] = 1
  h[2] = 1
  h[n]
end

The cached version uses @@h instead of h, and

=s it.

” class=”anchor”> </th></tr>

  <tr><th>rails<!doctype html>

Building an Object Graph in Rails - set_trace_func

Building an Object Graph in Rails

4th Jun 2015 | Tags: ruby rails

I was needing to do some object cleanup in our rails app the other day, and purge some malformed objects, so I put together a quick script using some ActiveRecord reflection to walk the object chain.

# Squelch SQL logs, if you're running from rails console
ActiveRecord::Base.logger.level = 1

# Output formatters. Trailing ':' on both, plus the staggered indent
# of 4n and 4n+2 makes the output valid YAML, if automated analysis
# is called for.
def puts_node(node, indent)
  puts "    "*indent + node.class.name + "#" + node.id.to_s + ":"
end
def puts_assoc(assoc, indent)
  puts "    "*indent + "  " + assoc.to_s + ":"
end

def puts_tree(node, seen=[], indent=0)
  puts_node(node, indent)

  unless seen.include? node
    # seen maintains a list of nodes to avoid mutual recursion
    seen << node
    node.class.reflections.keys.each do |assoc|
      # To see all locations an object is referenced, get rid of the
      # "- seen" here. I only cared about which objects were present
      # anywhere in the tree, so this was fine.
      associated = Array(node.send(assoc)) - seen
      next if associated.empty?

      # Print an entry for the association, then recurse
      puts_assoc(assoc, indent)
      associated.each do |subnode|
        puts_tree(subnode, seen, indent+1)
      end
    end
  end

  # Also outputs a footer listing all seen objects once, take it or leave it
  if indent.zero?
    puts
    seen.each {|node| puts_node(node, indent)}
  end

  nil # return nil to avoid flooding terminal in rails console
end

# Usage: call with the root node for the object graph
puts_tree(User.find(42))

<!doctype html>

Parsing JSON in SQL - set_trace_func

Parsing JSON in SQL

19th Nov 2011 | Tags: ruby rails sql

The Problem: You have a database column with some data serialzed as JSON in it that you’d like to pull out into its own column to index it.

def json(key, field='params')
  key_json = "\"#{key}\":"

  # key start/end locations, including ""
  k_a = "LOCATE('#{key_json}', #{field})"
  k_z = "LOCATE('\"', #{field}, #{k_a}+1)" # this is terminating "

  # is there a space after colons?
  spad = "IF(LOCATE('\": ', #{field}), 1, 0)"

  # is value a string?
  val_string = "LOCATE(CONCAT('#{key_json}', IF(#{spad},' ',''), '\"'), #{field}, #{k_a})"
  qpad = "IF(#{val_string}, 1, 0)"

  # value start/end locations, excluding "" if present
  v_a = "(#{k_z}+1 + 1 + #{spad} + #{qpad})" # 1 for colon, spad for optional space, qpad for possible quote

  end_if_string = "LOCATE('\"', #{field}, #{v_a})"
  end_if_not_string = "IF(LOCATE(',', #{field}, #{v_a}), LOCATE(',', #{field}, #{v_a}), LOCATE('}', #{field}, #{v_a}))"

  v_z = "IF(#{val_string}, #{end_if_string}, #{end_if_not_string})"

  value_string = "SUBSTRING(#{field} FROM #{v_a} FOR (#{v_z} - #{v_a}))"
  "IF(#{k_a}, #{value_string}, NULL)"
end

up do
  execute "
    UPDATE model_table
    SET status = #{json('status')}
  "
end

The generated sql looks pretty gnarly but mysql ran through it stupidly fast. I shudder to think how long it’d take activerecord to load and update each record individually.

<!doctype html>

Isolating Rails - set_trace_func

Isolating Rails

19th Jan 2011 | Tags: rails ruby

#!/usr/bin/env ruby

require 'fileutils'

File.open("Isolate", 'w') do |isolate|
  File.readlines("Gemfile").each do |line|
    next if line =~ /^\w*#/
    next if line =~ /^source/
    next if line =~ /^\w*$/

    line.sub!(/, :require.*(,|$)/, '\1')
    line.sub!(/^([ \t#]*)group/, '\1environment')
    
    if line =~ /:git/
      line = "# Don't use git, build it as a gem\n# " + line
    end

    isolate.puts line
  end
end

File.open("config/boot.rb", 'a') do |boot|
  boot.puts
  boot.puts("require 'isolate/now'")
end

FileUtils.rm('Gemfile')

This should convert an existing Gemfile to an Isolate file, remove the Gemfile (so that rails won’t try to load it), and update the app to load Isolate appropriately.

I’m basically only guessing that the group/environment setup is correct, so if anyone has any corrections to this let me know and I’ll update it.

Update: This code is stale, I’ve extracted a gem of it and posted on github.

First, install the amqp ruby library to connect to rabbit, and then add a tiny bit of setup.

In config/environment.rb:

require 'mq'
class BeanstalkPoolImpersonator
  def initialize(opts={})
    @opts = opts
  end

  def connect
    connection = AMQP.connect(@opts)
    @channel = channel = MQ.new(connection)
  end

  def use(queue)
    @queue = MQ::Queue.new(@channel, queue)
  end

  def yput(obj, pri, delay, ttr)
    p [obj, pri, delay, ttr]
    @queue.publish(YAML.dump(obj))
  end

  def last_server
    :last_server_stub
  end

  def subscribe(*args, &blk)
    @queue.subscribe(*args, &blk)
  end
end

Then, instead of connecting via Beanstalk::Pool.new, do this:

AsyncObserver::Queue.queue = BeanstalkPoolImpersonator.new()

You can pass an options hash to the new call, providing user, pass, vhost, host, or port as necessary.

Then, in your workers, load up the async_observer worker class, and extend like so:

class RabbitWorker < AsyncObserver::Worker
  def run()
    EM.run do
      AsyncObserver::Queue.queue.connect
      AsyncObserver::Queue.queue.use('1.0')
      AsyncObserver::Queue.queue.subscribe do |headers, msg|
        job = OpenStruct.new(:ybody => YAML.load(msg), :body => msg, :stats => [])
        job.id = headers.properties[:delivery_tag]
        safe_dispatch(job)
      end
    end
  end
end

Create the new worker the same way you would for the AO::Worker, and you’re set:

    RabbitWorker.new(binding).run()

Note: I’m maintaining a merb port of async-observer on github.

Note 2: This worker is somewhat fragile, if the RabbitMQ server goes down it will just hang forever waiting for more jobs. I’ll need to figure out a solution to that before we move this into production (and I wrap it up in a gem), but I thought I’d get this out and about now. We’re about to start up a new project at work, and we’ve decided to go with Merb (yay!) rather than Rails. Before we get started, though, we wanted to make sure that we’d be able to integrate well with the various plugins available for Rails.

The first two libraries we wanted to use were ultrasphinx, an interface to the Sphinx fulltext search engine, and async-observer, an abstraction library using the beanstalkd work queue library to delay actions to be processed at a later time, rather than while processing the page.

The good news is that both of the projects are available on GitHub, which means easy forks, and easy contributions back to the source if my changes are as good as I think they are.

So, today I’d like to talk about the basic changes you’ll need to make to a Rails plugin so that it will play nicely with Merb, and a few of the extra hooks that come into play. In the near future, I hope to provide a bit of a primer on adapting a plugin that uses ActiveRecord so that it will also work with Datamapper. Preview tip from that post, if you’re calling any methods provided by ActiveRecord, please make an adapter class/module to pass those methods through, as it makes ports like these much easier.

Basics

To get your plugin to get picked up properly, Rails requires an init.rb file in the plugin root. The equivalent for Merb is a file named after your plugin, in the /lib directory. Copying /init.rb to /lib/async_observer.rb worked for that one, Ultrasphinx is somewhat better behaved in that its init.rb just required ultrasphinx, so both Rails and Merb worked for me out of the box.

If you depend on anything in Merb, you’ll need to add to the docs that applications should add the plugin dependency inside a Merb::BootLoader.before_app_loads block - otherwise nothing in Merb is defined yet. This is of particular importance if you want to switch behaviour based on whether the Rails or Merb constants are defined. As Rails handles loading plugins itself, there’s no concern for keeping things special for Rails.

If you plan on dealing with the app directory structure or environment, an easy way to do it is:

if defined?(Rails)
  ROOT = RAILS_ROOT
  ENV = RAILS_ENV
elsif defined?(Merb)
  ROOT = Merb.root
  ENV = (Merb.env == 'rake' ? 'development' : Merb.env)
end

The additional rake environment transparently proxies to the development db connection, so if you just want to compare your plugin’s interpretation of ENV the above will make that cleaner.

Rake Tasks

Rails automatically loads any files matching tasks/*.rake in the plugin dir.

Merb needs to be told explicitly, relative to the lib directory. The canonical example from the docs is:

if defined?(Merb::Plugins)
  Merb::Plugins.add_rakefiles "merb_sequel" / "merbtasks"
end

Unfortunately, it seems that it wants specified a single file with a .rb extension, which is incompatible with the Rails Way. The easiest fix I’ve found is to add a file called tasks.rb under /lib, inside which you just manually require the individual rake files:

    load File.expand_path(File.join(File.dirname(__FILE__), '..', '..', 'tasks', 'merb_sequel.rake'))

Do note that the load is necessary, require doesn’t pick the file up properly.

Finally, if you have any tasks that depend on environment, the easiest way to get compatability with both frameworks is to add task :environment => :merb_env to your merbtasks.rb file.

Generators

I haven’t looked into generators in too much depth, but the API between Rails::Generator::Base and Merb::GeneratorBase seem different enough to warrant not reusing the generation script. If you conditionally define a generator based on the defined?ness of those two base classes, you should be able to reuse all your generation templates, and both Rails and Merb look in the same place for generators, so that should be the only adaptation necessary.

ORM Integration

Check back next time, as I write up my experience writing an ActiveRecord shim for the latest DataMapper. It’s vaguely ugly. I highly recommend if you’re working on a plugin now that interacts with models to abstract any access to the database into a module, and include it appropriately. This goes double if you’re using any AR magic ;)

As a follow-up to my previous post, here’s some gotchas to be aware of if you’re looking to support both ActiveRecord and DataMapper in a Merb (and/or Rails) plugin.

The Strategy

The best way I’ve found to handle multiple ORM support in your plugin is not to start monkeypatching around to make one ORM handle like another. I’ve done it, and can tell you that wrapping one ORM’s backend into another is ugly.

The better way is to localize the points where your plugin interacts with the data model, with an eye to swapping them out. For the above example, I would be better off taking the method that used the reflection method and putting it inside an ActiveRecord-specific module. Then, create a DataMapper-specific module that defines the same method, but instead relies on the DM backend to get at the association information. Finally, when the plugin was loaded I could just include one of the modules based on which ORM was loaded into the runtime.

Basic Translation

Now that we have a plan, we can start translating our extracted functions from AR bits to DM bits. There’s a bunch of fairly straightforward transformations we can make.

It’s unfortunate that there isn’t more unity between the two, as from a library-developer’s perspective it would make this sort of thing much easier, but the DM team is pretty vocal about wanting the best API they can get, and not worrying about being hobbled by how AR does things. I don’t particularly disagree.

ActiveRecord	DataMapper
.find(:all, ...)	.all(...)
.find(:first, ...)	.first(...)
.find(id)	.get(...)
.find_all_by_id(id)	.all(:id => ids)
.table_name	.storage_name
.primary_key	.key.first.name
.connection	repository.adapter

Raw SQL

If you’re running raw SQL queries, firstly, I’m sorry. Secondly, you want to run .query instead of .execute. Thirdly, if you care about getting the results of the query back, AR returns an array of arrays, DM returns an array of hashlike objects, so you want to map them for their values array. The hashlike object in question is order-preserving, so you’ll get things out in the right order. If you’re concerned, grab one of the result objects and verify that the keys array is in the correct order.

Hooks

ActiveRecord defines a few hook points, along the lines of before_create and after_save. DataMapper uses (a modified version of) the Extlib gem, allowing it to hook pretty much any method. The syntax is like before(:create) and after(:save). AR’s hooks pass in the object to work with, DM’s have the object available as self.

In before hooks, the AR hook chain stops if your method returns false, in DM you must throw :halt.

Things You Shouldn’t Be Doing Anyways

If you’re manually setting @attribute values in your AR code, you’ll need to use instance_variable_set for DataMapper. I recommend writing manual accessor methods to wrap it for abstraction.

If you’re wanting some arbitrary data structures back, I recommend using OpenStruct (require 'ostruct') to pass structured data back and forth. This was especially handy when I wanted some results from DM to look like AR, because I was just doing an adapter (bad me!) and the client code wanted to interact with the AR object. Just be aware that OpenStruct doesn’t quite clear out all its methods, so you might want to define some custom readers anyway. I had problems with type in particular, which is a deprecated alias of class - redefining the method to return @table[:type] fixed that up nicely.

This is why I hate Rails’ hackery to interact RESTfully from the browser, where it’s not exactly natively supported.

link_to 'Destroy', post_path(post), :method => :delete

Specifying the HTTP verb to use (and then passing it as _method in the query string) is superfluous. If I tell Rails to use restful routes, I should be able to leave it in Rails’ hands to treat DELETE /posts/42 and POST /posts/42/delete as the same controller/action. Shoehorning hacks to make the browser behave is the wrong integration point.

Less so is the redundancy that occurs with post_path(post).

I’ll expand on the solution I’m using for route helpers in Merb in my next post.

It seems that there were people making both audio and video recordings of talks at RejectConf this year.

I’m coming late to the controversy, I know. I was talking with a co-worker about Rails and Seaside the other day, and after describing the Seaside structure and philosophy compared to Rails I got to thinking that there’s really not as much overlap as some people think between the two.

Rails, at least since v1.2, has a focus on information. It says, I have a bunch of knowledge I’d like to share with the world. Working with routes makes accessing that information fairly uniform, and also allows for deep linking - a reference to that piece of information that won’t change. It recognizes that while it’s possible to provide access to this information with simple flat files, if you want to provide dynamic views, or frequently updating data, or even provide for display customizations, Rails has facilities for getting you most of the way there.

Seaside, on the other hand, has more of a focus on the application. It provides for a workflow, and pauses in that workflow every so often to display a web page to the user. It says, I want to let you get something done, here, go to it. It provides a framework that lets you write an application similar to a desktop application, but which uses a web browser for its UI and can provide a centralized storage system for the data it manipulates.

Just looking at these, it’s easy to see where one framework shines and the other would require more work to get there.

Anything working with a data-centric view or large-scale multi-user behaviour could run very well in Rails. The Blog example is ubiquitous, but also a forum, or news site, or many other applications involving user feedback and the option to deep-link to pages.

Sites with a more workflow-driven, single-user view would do well by Seaside. For example, I think doing an internet banking front-end in Seaside would be excellent. One user working through steps for a number of actions (think of paying a bill - usually 3-4 page loads in sequence), without the need to reference any specific page in the system. Users log in, and can essentially ignore the URL in the address bar for the duration of their visit.

While both kinds of applications can (and have) been done with the other framework, it seems silly to bolt on extraneous features (like meaningful URLs in Seaside, or managing serious page flow in Rails) when you could switch and play to the strengths of the framework. Given the somewhat orthogonal strengths of Seaside and Rails, I can only see the increased choice they bring as a good thing.

Rplug has a new release up, which should now be useful for the world at large, as it has gained support for projects in subversion.

I should probably do a check after I’ve done the export and cull any now-empty directories from the plugin dir, but that’ll come in time, I’m sure.

Both projects are entirely up in subversion if anyone wants to check them out, but they’re not quite ready for public consumption at the moment.

Example usage and output follows.

% rplug install exception_logger http://svn.techno-weenie.net/projects/plugins/exception_logger svn
Recorded exception_logger, run 'rplug update' to pull the latest revision

% rplug update
Working in project dir /home/jamie/dev/redvase
Updating exception_logger...
  upgrading to revision 2733
  updating local repository
  Done.
Updating mocha...
  Done.
Updating helper_test...
  Done.
Updating arts...
  Done.
Updating liquid...
  Done.

% rplug status
Working in project dir /home/jamie/dev/redvase
Managing the following plugins:
  arts, revision 70
  exception_logger, revision 2
  helper_test, revision 85
  liquid, revision 140
  mocha, revision 99
Not Managing the following plugins:
  test_timer

% rplug update -p exception_logger -r 2563
Working in project dir /home/jamie/dev/redvase
Updating exception_logger...
  upgrading to revision 2563
  updating local repository
  Done.

Rather than hook into Test::Unit::TestSuite, I’m hooking into TestCase, and providing a global report via an at_exit hook. Just add the following file to your lib folder, require it from test_helper, and most of the time it will just sit there, quietly doing nothing. Call it into action by setting the environment variable TEST_TIMER with a float, and it will output the elapsed time of any test taking longer than that.

Example run:

# TEST_TIMER=0.25 rake test:units TEST=test/unit/creative_test.rb
/usr/bin/rake:17:Warning: require_gem is obsolete.  Use gem instead.
(in /home/jamie/dev/redvase)
/usr/bin/ruby1.8 -Ilib:test "/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader.rb" "test/unit/creative_test.rb"
Loaded suite /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader
Started
......................................................................................
Finished in 10.116575 seconds.

86 tests, 164 assertions, 0 failures, 0 errors

Test Benchmark Results
  0.2927 CreativeTest#test_delayed_click_count_with_third_party_stats
  0.2982 CreativeTest#test_impression_count_for_date_range_with_third_party_stats_offset
  0.3240 CreativeTest#test_global_creative_stats_should_return_correct_default_values
  6.2505 CreativeTest#test_click_count

Source file, lib/test_timer.rb

if ENV.has_key? 'TEST_TIMER' and
   !Test::Unit::TestCase.method_defined? :untimed_run

  class Test::Unit::TestCase
    cattr_reader :benchmark_data
    @@benchmark_data = {}
    alias untimed_run run

    def run(result, &progress_block)
      start = Time.now
      untimed_run(result, &progress_block)
      finish = Time.now
      elapsed = finish - start
      if elapsed > ENV['TEST_TIMER'].to_f
        name =~ /(.*)\((.*)\)/
        @@benchmark_data["#{$2}##{$1}"] = elapsed
      end
    end
  end

  # at_exit hooks run in reverse order, so in order to run after
  # Test::Unit's hook, we need to nest at_exit calls.
  at_exit do
    at_exit do
      results = Test::Unit::TestCase.benchmark_data
      unless results.empty?
        puts "\nTest Benchmark Results"
        results.sort{|a,b| a.last <=> b.last }.each do |key,value|
          puts " %7.4f #{key}" % value
        end
      end
    end
  end
end

So I’ve been spending some time lately working on upgrading the existing codebase for some projects at work such that they’ll work in Rails 1.2 once it’s released. Sadly, the upgrade process is not without its rough edges, and after two days of poking at it (it being a 5800 LOC app with 8800 LOC of tests) I’m still not completely done - the test run does not pass cleanly.

However, I have managed to get rid of most of the niggly deprecation warnings, so the output of the rake run is down to 450k from a high of about 2.2mb. Fun times. The changes required to silence most of the warnings are…

ActiveRecord

find_all and find_first are deprecated. Use find(:all) and find(:first) instead.

If you have a has_many which is :dependent, make sure you’re specifying :destroy or :delete_all, rather than true. If you’ve got true you probably want to replace it with :destroy. Slower, but safer.

Routes

Just a short note here, :requirements regexps no longer accept anchors. We have something along these lines:

map.connect ':controller/:action/:foo', :requirements => { :foo => /^(bar|baz)$/ }

Such that that route only fires if the third url part is exactly bar or baz. To silence 1.2, just remove the ^ and $ anchors.

Controllers/Views

The instance variables @params, @session, @request, and @flash are deprecated in controllers and views, use the version without the @. Note, don’t try and change this globally in your tests, or all hell will break loose. Oddly, assigns(:flash) in a test seems to trigger the warning for accessing @flash.

If you want to have your link_to go by post instead of get, use :method => :post instead of :post => true.

(Update: This will not throw errors, but won’t work in 1.1.6, so wait until you’re actually running 1.2 to do this change.)

Rendering with a string (render ‘template’) is no longer allowed. The deprecation warning says to render :file => ‘template’ instead, but if you want your code to continue to work in Rails 1.1.6 you’ll need to add a :use_full_path => true to the call.

start_form_tag and end_form_tag are now deprecated. The suggested replacement is to pass a block to a form_tag call, but that does not work at all in Rails 1.1.6. My preferred fix is to use a bare form_tag to start the form, and a hard-coded to finish it. Same goes with remote_form_tag for those AJAXy forms. That shuts up all the deprecation warnings, and allows for a fairly simple multiline regexp to blockify them up in the future. I was looking at something like <%= ?(remote_)?form_tag([^%]) ?%>(.?) and replacing with _<% $1form_tag$2 do %>$3<% end %>

We’ve got a few places where we’re redirecting to a named route: redirect_to :login_url. This calls url_for, which is deprecated. I think the correct solution is to just drop the colon and redirect_to login_url, but this doesn’t work in 1.1.6 and I haven’t quite tested it yet.

Tests

Lastly, a change which I completely disagree with, assert_template_has and friends are now deprecated. Use assert(@response.has_session_object?(key)) instead, my ass. This changes removes a useful failure message like <:login> is not a template object and brings me back to the glory days of is not true. I know I’ll be rewriting those as custom assertions for my test_helper, thank you very much.

To Be Continued

Like I said, I’m only half done this migration, but when I get the rest of it sorted, I’ll be posting a follow-up right here. See you then. Well, I managed to get the weight-tracking app functional (graph and all) in about 220 lines, just tweaking the look now. It’s a single-script Camping app using Gruff for graphing, with SQLite for data storage. Not exactly the most efficient app (I’m cheating by using a lot of mostly-null records in the database) but it gets the job done. There were a few things that got me stuck for a bit that weren’t obviously mentioned in the camping docs, so I thought I’d put them down here.

If you’re planning on letting Camping handle migrations for you (class Weight::Models::CreateEntries < V 0.1) be sure to require ‘camping/db’, which is what defines the V method. The equivalent of Rails’ /params/ method for get and post variables is /input/ in Camping. Found that one by accident on a JRuby tutorial, of all things.

Not exactly a camping thing, but if you want a non 4:3 ratio gruff graph, send a string ‘1000x350’ or similar.

That being all that I can think of browsing over the source, I think I’m safe recommending camping for quick prototyping. The best part is that if you want to switch over to a full-fledged rails app, you can just copy/paste the models and migrations (Rails and Camping both use ActiveRecord) and if you want to use markaby for your rails views, you can copy them over as well. A little more work organizing the views and correcting urls and such, but mostly painless. Just don’t forget to do the testing ;)

Chris Abad wrote yesterday about his experience with dynamic attributes, and I thought I’d share mine.

I’m doing something similar to collect data POSTed to a form, but my data is slightly more structured than Chris’. I have a few fields that I expect to be populated most of the time, and the possiblity of arbitrary fields being set as well. My models look like this:

create_table "leads", :force => true do |t|
  t.column "email", :string, :default => "", :null => false
  t.column "firstname", :string
  t.column "lastname", :string
  t.column "ip", :string, :default => "", :null => false
end
create_table "lead_infos", :force => true do |t|
  t.column "lead_id", :integer, :default => 0, :null => false
  t.column "name", :string, :default => "", :null => false
  t.column "value", :string, :default => "", :null => false
end

Lead, of course, has_many :lead_infos. Thus, I can assume that most leads will have an email, first and last name, and an IP address. The name fields are optional, but common enough to warrant being in the main table (also makes for easier lookups and duplicate checking). Other things, like address, city, zip, etc. I want to hang on to if provided, so I store them as a LeadInfo.

I’m in the same boat as Chris though, as I want to provide uniform access to the data points in a Lead, as well as its LeadInfos, using the ‘name’ field as a key. Chris added an after_find hook that moved all the correct data in, but since I’m such a fan of metaprogramming, I decided that I would use method_missing like so:

class Lead < ActiveRecord::Base
  has_many :lead_infos, :dependent => true

  def method_missing(methodname, *args)
    begin
      super
    rescue NameError
      name = methodname.to_s.chomp('=')
      if (methodname.to_s =~ /=$/)
        LeadInfo.create(:name => name, :value => args.first, :lead_id => self.id)
        self
      else
        LeadInfo.find(:first, :conditions => ['name = ?', name]) or raise
        lead_infos.find_by_name(name).value rescue ''
      end
    end
  end

  ...
end

Walking through this, I first make sure to call super so that ActiveRecord’s method_missing gets run first. If it can’t find anything to do, then it is the Lead’s turn to try.

We start by stripping a trailing = if it exists to get the correct name to use. If the = existed, it’s being used as a setter, so we just create the new LeadInfo record based off the current Lead, and return self so that we can chain calls if we desire.

Otherwise, it’s a getter, so I first verify that there is at least one LeadInfo with the supplied name - if not, then something is very wrong and we want to re-raise the NameError. Elsewise we so a search for the given value and return it, or an empty string if it is not set (the empty string catch-all is specific for the things I’m doing with this bit of hackery, so might not be applicable to everybody).

The performance difference between my code and Chris’ is that I take 2 DB queries for each access to the LeadInfo data, but only when you ask for it. I’m not caching the result (which would take some strain away) because my use of it is solely one access each time I load the Lead from the DB, so caching wouldn’t help me. On the other hand, if I do a grand find of a bunch of leads (and in a few places in my code that’s quite a lot) I’m not getting hit with extra db hits that I’m not going to use most of the time.

There’s benefits to both what I’ve done and what Chris has done, so anyone reading these can take their pick :)

Comments

Nice write up. I was going to go the MethodMissing route if it weren’t for my requirement to have all the attributes insterted into the objects attributes hash. I think you’re write about the catch-all being specific to you. Most people would probably want to re-raise the error and deal with that appropriately elsewhere.

Chris Abad, at 18:45, Sep 13 2006 Rails’ routing framework is a pretty capable beast, but it does sometimes still need a bit of help doing more exotic things.

I ran across an example from someone a few months back (either on the mailing list or in a blog post, I can’t find any trace of it now) that was using multiple routes to match the same thing - a GUID that could belong to one of many different models. This was done with an overloaded Regexp subclass that pattern matched the GUID, and then looked it up to see if it existed for that model. I wanted to do something similar to that for a project I’m working on, and since I couldn’t find an example to copy off of, I went delving inside routing.rb on my own.

The long and short of it is that the following code seems to be a workable solution for me, while being generic enough (the only real custom line is the last one inside the module def) for anyone to incorporate into their code.

module ActionController::Routing
  module ConditionConstants; end

  class CustomCondition < Regexp
    def initialize(name, &match)
      super '^$' # pretend we're a regular empty string regexp
      @name, @match = name, match
    end
    def =~(other); @match.call(other); end
    def inspect;   @name;              end
  end

  def self.custom_condition(name, &block)
    ConditionConstants.const_set(name, CustomCondition.new(name, &block))
  end

  custom_condition('ProjectCondition'){|other| Project.find_by_url(other) }
end

The only other thing to do is include ActionController::Routing::ConditionConstants inside the block attached to draw so your connect calls can see the constants being defined - AC::Routing seems to have no problem seeing them, even though they’re inside a module of their own.

Anyone interested in the hows and whys, feel free to read on…

The goal then, is to come up with an object that can perform an arbitrary condition for use in routes. My usage is a simple lookup in one of my ActiveRecord models, but really you could do anything you wanted. So let’s take a look through the generation of a route.

First off, you create routes using the draw method of ActionController::Routing::Routes, which accepts a block detailing the routes you want to connect. Routes is actually not a class with a class method draw, but rather is an instance of AC::Routing::RouteSet. draw yields the RouteSet to the content block, so let’s take a look at it for a moment.

The RouteSet#connect method takes a bunch of arguments, and passes them directly into the constructor of AC::Routing::Route. Route accepts two parameters: a path, and an options hash. The path can be either a string (which is split on ‘/’) or an array. The options hash is populated with either defaults or conditions for the various parts of the path.

While you can explicitly define :defaults and :conditions with their own subhashes, but Route is smart enough to do some thinking for you: if the value for a given key is_a?(Regexp), it is treated as a condition, otherwise it is considered a default. Therefore, this is the first test that our custom condition must pass. Fortunately, it’s trivial to write an is_a? method on whatever class we end up writing that returns true for Regexp, so it’s not a big stumbling block.

Now, I have to admit I snuck a bit ahead of the game here - when Route was doing data massaging on the path, it is creating AC::Routing::Components to store each part of the path. There are actually four different subclasses of components, each created by the base Component class. The one we’re interested in is DynamicComponent, which is created when the path looks like a symbol. ControllerComponent matches on the explicit symbol :controller so that it can deal with modules, so we don’t need to worry about it.

This comes in later on in RouteSet#draw. After creating the various Routes, it calls methods named write_generation and write_recognition. write_generation sets up rules for turning a params hash into an actual URL. write_recognition does the other way, which is what we want.

write_recognition then, assembles the recognition rules for each of its Routes, which through a roundabout way calls the same on each Component. The DynamicComponent we were looking at then calls the class method Routing.test_condition with its condition. This leads us to our second constraint on our custom class - test_condition runs the condition through a case statement, putting classes on the whens. The when Regexp condition behaves the same as if Regexp === condition, which only evaluates true when condition is an instance of Regexp or a subclass.

This is significantly more difficult to fake than being able to redefine is_a?, and I really didn’t want to come up with a solution that required hacking into Regexp to get anything done. The next two lines after the when (that’s 38 and 39, for those playing at home with Rails 1.1.2) add two other wrinkles in the behaviour.

The first is that if the Regexp instance is not bookended by beginning-of-string and end-of-string matchers, a new instance is created that is wrapped so. However, this isn’t as bad as it looks at first. Since we’re not actually using the Regexp source for any pattern matching, we can satisfy this criteria by setting the pattern to /^$/, which matches an empty string.

The second is significantly trickier, in that when our matcher is inspected, it needs to output as a string the ruby code used to create it. This is fine for normal regular expressions, as /^$/.inspect does actually print out “/^$/”, but if we want our matcher to be highly dynamic, it effectively rules out using blocks or procs directly as we would expect. Additionally, the output of the inspect call is then directly sent an =~ call with the part of the path it is trying to match.

My first thought on this was to use classes, since I could create an instance that would pass the “is a subclass of Regexp” test, output the name of the class, and use a class method to do the actual matching (coming later), but that was looking much too ugly. Then I realized that the reason I was drawn to classes is that it is a constant that knows how to look itself up. If I were to teach a Regexp subclass the name I’m about to assign to it, it will know how to reference itself directly, and storing it in a constant means I should be able to get easy access to it by the time we get that deep into the code.

Thus began my CustomCondition class.

I first set up an empty module named ConditionConstants that I would use to house my constants.

Next up was the class - a subclass of Regexp to pass the is_a? and === tests, but with an overoaded initialize. This first set up the Regexp base to use the empty string regex listed above to prevent Routing from stepping over the constant. initialize also accepted the name of the constant it is about to be stored in, and a block of code to execute later on.

CustomCondition#inspect did the obvious, and output the name we were to be remembering.

CustomCondition#=~ simply called the stored block with the argument we are trying to match, allowing the creation of the object to completely drive its purpose.

Lastly, since all the work was being done in the ActionController::Routing module, I created a class method that would do the creation and assignment for me so that I was doing less direct repetition.

Once that was all in place, I simply called my helper method with the name of the constant, and a block encapsulating the behaviour. Done and done.

</th> <th><a name=”rails<!doctype html>

Building an Object Graph in Rails - set_trace_func

Building an Object Graph in Rails

4th Jun 2015 | Tags: ruby rails

I was needing to do some object cleanup in our rails app the other day, and purge some malformed objects, so I put together a quick script using some ActiveRecord reflection to walk the object chain.

# Squelch SQL logs, if you're running from rails console
ActiveRecord::Base.logger.level = 1

# Output formatters. Trailing ':' on both, plus the staggered indent
# of 4n and 4n+2 makes the output valid YAML, if automated analysis
# is called for.
def puts_node(node, indent)
  puts "    "*indent + node.class.name + "#" + node.id.to_s + ":"
end
def puts_assoc(assoc, indent)
  puts "    "*indent + "  " + assoc.to_s + ":"
end

def puts_tree(node, seen=[], indent=0)
  puts_node(node, indent)

  unless seen.include? node
    # seen maintains a list of nodes to avoid mutual recursion
    seen << node
    node.class.reflections.keys.each do |assoc|
      # To see all locations an object is referenced, get rid of the
      # "- seen" here. I only cared about which objects were present
      # anywhere in the tree, so this was fine.
      associated = Array(node.send(assoc)) - seen
      next if associated.empty?

      # Print an entry for the association, then recurse
      puts_assoc(assoc, indent)
      associated.each do |subnode|
        puts_tree(subnode, seen, indent+1)
      end
    end
  end

  # Also outputs a footer listing all seen objects once, take it or leave it
  if indent.zero?
    puts
    seen.each {|node| puts_node(node, indent)}
  end

  nil # return nil to avoid flooding terminal in rails console
end

# Usage: call with the root node for the object graph
puts_tree(User.find(42))

<!doctype html>

Parsing JSON in SQL - set_trace_func

Parsing JSON in SQL

19th Nov 2011 | Tags: ruby rails sql

The Problem: You have a database column with some data serialzed as JSON in it that you’d like to pull out into its own column to index it.

def json(key, field='params')
  key_json = "\"#{key}\":"

  # key start/end locations, including ""
  k_a = "LOCATE('#{key_json}', #{field})"
  k_z = "LOCATE('\"', #{field}, #{k_a}+1)" # this is terminating "

  # is there a space after colons?
  spad = "IF(LOCATE('\": ', #{field}), 1, 0)"

  # is value a string?
  val_string = "LOCATE(CONCAT('#{key_json}', IF(#{spad},' ',''), '\"'), #{field}, #{k_a})"
  qpad = "IF(#{val_string}, 1, 0)"

  # value start/end locations, excluding "" if present
  v_a = "(#{k_z}+1 + 1 + #{spad} + #{qpad})" # 1 for colon, spad for optional space, qpad for possible quote

  end_if_string = "LOCATE('\"', #{field}, #{v_a})"
  end_if_not_string = "IF(LOCATE(',', #{field}, #{v_a}), LOCATE(',', #{field}, #{v_a}), LOCATE('}', #{field}, #{v_a}))"

  v_z = "IF(#{val_string}, #{end_if_string}, #{end_if_not_string})"

  value_string = "SUBSTRING(#{field} FROM #{v_a} FOR (#{v_z} - #{v_a}))"
  "IF(#{k_a}, #{value_string}, NULL)"
end

up do
  execute "
    UPDATE model_table
    SET status = #{json('status')}
  "
end

The generated sql looks pretty gnarly but mysql ran through it stupidly fast. I shudder to think how long it’d take activerecord to load and update each record individually.

<!doctype html>

Isolating Rails - set_trace_func

Isolating Rails

19th Jan 2011 | Tags: rails ruby

#!/usr/bin/env ruby

require 'fileutils'

File.open("Isolate", 'w') do |isolate|
  File.readlines("Gemfile").each do |line|
    next if line =~ /^\w*#/
    next if line =~ /^source/
    next if line =~ /^\w*$/

    line.sub!(/, :require.*(,|$)/, '\1')
    line.sub!(/^([ \t#]*)group/, '\1environment')
    
    if line =~ /:git/
      line = "# Don't use git, build it as a gem\n# " + line
    end

    isolate.puts line
  end
end

File.open("config/boot.rb", 'a') do |boot|
  boot.puts
  boot.puts("require 'isolate/now'")
end

FileUtils.rm('Gemfile')

This should convert an existing Gemfile to an Isolate file, remove the Gemfile (so that rails won’t try to load it), and update the app to load Isolate appropriately.

I’m basically only guessing that the group/environment setup is correct, so if anyone has any corrections to this let me know and I’ll update it.

Update: This code is stale, I’ve extracted a gem of it and posted on github.

First, install the amqp ruby library to connect to rabbit, and then add a tiny bit of setup.

In config/environment.rb:

require 'mq'
class BeanstalkPoolImpersonator
  def initialize(opts={})
    @opts = opts
  end

  def connect
    connection = AMQP.connect(@opts)
    @channel = channel = MQ.new(connection)
  end

  def use(queue)
    @queue = MQ::Queue.new(@channel, queue)
  end

  def yput(obj, pri, delay, ttr)
    p [obj, pri, delay, ttr]
    @queue.publish(YAML.dump(obj))
  end

  def last_server
    :last_server_stub
  end

  def subscribe(*args, &blk)
    @queue.subscribe(*args, &blk)
  end
end

Then, instead of connecting via Beanstalk::Pool.new, do this:

AsyncObserver::Queue.queue = BeanstalkPoolImpersonator.new()

You can pass an options hash to the new call, providing user, pass, vhost, host, or port as necessary.

Then, in your workers, load up the async_observer worker class, and extend like so:

class RabbitWorker < AsyncObserver::Worker
  def run()
    EM.run do
      AsyncObserver::Queue.queue.connect
      AsyncObserver::Queue.queue.use('1.0')
      AsyncObserver::Queue.queue.subscribe do |headers, msg|
        job = OpenStruct.new(:ybody => YAML.load(msg), :body => msg, :stats => [])
        job.id = headers.properties[:delivery_tag]
        safe_dispatch(job)
      end
    end
  end
end

Create the new worker the same way you would for the AO::Worker, and you’re set:

    RabbitWorker.new(binding).run()

Note: I’m maintaining a merb port of async-observer on github.

The good news is that both of the projects are available on GitHub, which means easy forks, and easy contributions back to the source if my changes are as good as I think they are.

Basics

If you plan on dealing with the app directory structure or environment, an easy way to do it is:

if defined?(Rails)
  ROOT = RAILS_ROOT
  ENV = RAILS_ENV
elsif defined?(Merb)
  ROOT = Merb.root
  ENV = (Merb.env == 'rake' ? 'development' : Merb.env)
end

The additional rake environment transparently proxies to the development db connection, so if you just want to compare your plugin’s interpretation of ENV the above will make that cleaner.

Rake Tasks

Rails automatically loads any files matching tasks/*.rake in the plugin dir.

Merb needs to be told explicitly, relative to the lib directory. The canonical example from the docs is:

if defined?(Merb::Plugins)
  Merb::Plugins.add_rakefiles "merb_sequel" / "merbtasks"
end

    load File.expand_path(File.join(File.dirname(__FILE__), '..', '..', 'tasks', 'merb_sequel.rake'))

Do note that the load is necessary, require doesn’t pick the file up properly.

Finally, if you have any tasks that depend on environment, the easiest way to get compatability with both frameworks is to add task :environment => :merb_env to your merbtasks.rb file.

Generators

ORM Integration

As a follow-up to my previous post, here’s some gotchas to be aware of if you’re looking to support both ActiveRecord and DataMapper in a Merb (and/or Rails) plugin.

The Strategy

Basic Translation

Now that we have a plan, we can start translating our extracted functions from AR bits to DM bits. There’s a bunch of fairly straightforward transformations we can make.

ActiveRecord	DataMapper
.find(:all, ...)	.all(...)
.find(:first, ...)	.first(...)
.find(id)	.get(...)
.find_all_by_id(id)	.all(:id => ids)
.table_name	.storage_name
.primary_key	.key.first.name
.connection	repository.adapter

Raw SQL

Hooks

In before hooks, the AR hook chain stops if your method returns false, in DM you must throw :halt.

Things You Shouldn’t Be Doing Anyways

If you’re manually setting @attribute values in your AR code, you’ll need to use instance_variable_set for DataMapper. I recommend writing manual accessor methods to wrap it for abstraction.

This is why I hate Rails’ hackery to interact RESTfully from the browser, where it’s not exactly natively supported.

link_to 'Destroy', post_path(post), :method => :delete

Less so is the redundancy that occurs with post_path(post).

I’ll expand on the solution I’m using for route helpers in Merb in my next post.

It seems that there were people making both audio and video recordings of talks at RejectConf this year.

Just looking at these, it’s easy to see where one framework shines and the other would require more work to get there.

Rplug has a new release up, which should now be useful for the world at large, as it has gained support for projects in subversion.

I should probably do a check after I’ve done the export and cull any now-empty directories from the plugin dir, but that’ll come in time, I’m sure.

Both projects are entirely up in subversion if anyone wants to check them out, but they’re not quite ready for public consumption at the moment.

Example usage and output follows.

% rplug install exception_logger http://svn.techno-weenie.net/projects/plugins/exception_logger svn
Recorded exception_logger, run 'rplug update' to pull the latest revision

% rplug update
Working in project dir /home/jamie/dev/redvase
Updating exception_logger...
  upgrading to revision 2733
  updating local repository
  Done.
Updating mocha...
  Done.
Updating helper_test...
  Done.
Updating arts...
  Done.
Updating liquid...
  Done.

% rplug status
Working in project dir /home/jamie/dev/redvase
Managing the following plugins:
  arts, revision 70
  exception_logger, revision 2
  helper_test, revision 85
  liquid, revision 140
  mocha, revision 99
Not Managing the following plugins:
  test_timer

% rplug update -p exception_logger -r 2563
Working in project dir /home/jamie/dev/redvase
Updating exception_logger...
  upgrading to revision 2563
  updating local repository
  Done.

Example run:

# TEST_TIMER=0.25 rake test:units TEST=test/unit/creative_test.rb
/usr/bin/rake:17:Warning: require_gem is obsolete.  Use gem instead.
(in /home/jamie/dev/redvase)
/usr/bin/ruby1.8 -Ilib:test "/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader.rb" "test/unit/creative_test.rb"
Loaded suite /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader
Started
......................................................................................
Finished in 10.116575 seconds.

86 tests, 164 assertions, 0 failures, 0 errors

Test Benchmark Results
  0.2927 CreativeTest#test_delayed_click_count_with_third_party_stats
  0.2982 CreativeTest#test_impression_count_for_date_range_with_third_party_stats_offset
  0.3240 CreativeTest#test_global_creative_stats_should_return_correct_default_values
  6.2505 CreativeTest#test_click_count

Source file, lib/test_timer.rb

if ENV.has_key? 'TEST_TIMER' and
   !Test::Unit::TestCase.method_defined? :untimed_run

  class Test::Unit::TestCase
    cattr_reader :benchmark_data
    @@benchmark_data = {}
    alias untimed_run run

    def run(result, &progress_block)
      start = Time.now
      untimed_run(result, &progress_block)
      finish = Time.now
      elapsed = finish - start
      if elapsed > ENV['TEST_TIMER'].to_f
        name =~ /(.*)\((.*)\)/
        @@benchmark_data["#{$2}##{$1}"] = elapsed
      end
    end
  end

  # at_exit hooks run in reverse order, so in order to run after
  # Test::Unit's hook, we need to nest at_exit calls.
  at_exit do
    at_exit do
      results = Test::Unit::TestCase.benchmark_data
      unless results.empty?
        puts "\nTest Benchmark Results"
        results.sort{|a,b| a.last <=> b.last }.each do |key,value|
          puts " %7.4f #{key}" % value
        end
      end
    end
  end
end

ActiveRecord

find_all and find_first are deprecated. Use find(:all) and find(:first) instead.

Routes

Just a short note here, :requirements regexps no longer accept anchors. We have something along these lines:

map.connect ':controller/:action/:foo', :requirements => { :foo => /^(bar|baz)$/ }

Such that that route only fires if the third url part is exactly bar or baz. To silence 1.2, just remove the ^ and $ anchors.

Controllers/Views

If you want to have your link_to go by post instead of get, use :method => :post instead of :post => true.

(Update: This will not throw errors, but won’t work in 1.1.6, so wait until you’re actually running 1.2 to do this change.)

Tests

To Be Continued

Not exactly a camping thing, but if you want a non 4:3 ratio gruff graph, send a string ‘1000x350’ or similar.

Chris Abad wrote yesterday about his experience with dynamic attributes, and I thought I’d share mine.

create_table "leads", :force => true do |t|
  t.column "email", :string, :default => "", :null => false
  t.column "firstname", :string
  t.column "lastname", :string
  t.column "ip", :string, :default => "", :null => false
end
create_table "lead_infos", :force => true do |t|
  t.column "lead_id", :integer, :default => 0, :null => false
  t.column "name", :string, :default => "", :null => false
  t.column "value", :string, :default => "", :null => false
end

class Lead < ActiveRecord::Base
  has_many :lead_infos, :dependent => true

  def method_missing(methodname, *args)
    begin
      super
    rescue NameError
      name = methodname.to_s.chomp('=')
      if (methodname.to_s =~ /=$/)
        LeadInfo.create(:name => name, :value => args.first, :lead_id => self.id)
        self
      else
        LeadInfo.find(:first, :conditions => ['name = ?', name]) or raise
        lead_infos.find_by_name(name).value rescue ''
      end
    end
  end

  ...
end

Walking through this, I first make sure to call super so that ActiveRecord’s method_missing gets run first. If it can’t find anything to do, then it is the Lead’s turn to try.

There’s benefits to both what I’ve done and what Chris has done, so anyone reading these can take their pick :)

Comments

Chris Abad, at 18:45, Sep 13 2006 Rails’ routing framework is a pretty capable beast, but it does sometimes still need a bit of help doing more exotic things.

module ActionController::Routing
  module ConditionConstants; end

  class CustomCondition < Regexp
    def initialize(name, &match)
      super '^$' # pretend we're a regular empty string regexp
      @name, @match = name, match
    end
    def =~(other); @match.call(other); end
    def inspect;   @name;              end
  end

  def self.custom_condition(name, &block)
    ConditionConstants.const_set(name, CustomCondition.new(name, &block))
  end

  custom_condition('ProjectCondition'){|other| Project.find_by_url(other) }
end

Anyone interested in the hows and whys, feel free to read on…

Thus began my CustomCondition class.

I first set up an empty module named ConditionConstants that I would use to house my constants.

CustomCondition#inspect did the obvious, and output the name we were to be remembering.

CustomCondition#=~ simply called the stored block with the argument we are trying to match, allowing the creation of the object to completely drive its purpose.

Once that was all in place, I simply called my helper method with the name of the constant, and a block encapsulating the behaviour. Done and done.

” class=”anchor”> </th></tr>

  <tr><th>programming<!doctype html>

No More Bundle Exec - set_trace_func

No More Bundle Exec

6th Sep 2012 | Tags: programming ruby

Update 2023: Just use direnv with layout ruby.

function command_not_found_handler() {
    ~/bin/command-not-found $*
}

My script happens to be in Ruby, but it could just as easily be a standard shell script as all I’m doing is some file existence tests:

#!/usr/bin/env ruby

# ARGV is the entire command we wanted to run, but we
# really only care about the actual executable for fallbacks
command = ARGV.first

def run(cmd)
  $stderr.puts "Running #{cmd.inspect} instead"
  system(cmd)
end

case
when File.exist?("./.bundle/config") && File.exist?("./bin/#{command}")
  run("bundle exec #{ARGV.join(' ')}")

else
  exit 127
end

# Paste git repo url to clone it
when command =~ /^git(@|:\/\/).*\.git$/
  run("git clone #{command.inspect}")

# paste compressed url to download+extract it
when command =~ /^(?:ftp|https?):\/\/.+\.t(?:ar\.)?gz$/
  run("curl #{command.inspect} | tar xzv")

when File.exist?("./tmp/isolate/ruby-1.8/bin/#{command}")
  run("rake isolate:sh['#{ARGV.join(' ')}']")

Heroku kicks ass. Very nifty tech running the show, and a really simple interface for deploying - just push up your git repository.

I run this blog through webby, which translates some boring old markdown into html so that I can serve it as static files. I like that amount of snappiness. Heroku wants to run something ruby - rails, merb, or barebones rack.

Turns out, setting up rack to pass static files through is pretty easy, and since I think it just uses send_file behind the scenes, it should be all set up for Heroku to cache it in their Varnish layer, if I ever happen to get a traffic spike.

All you need is a config.ru file like so, and then be sure to include the generated output dir in your git repository:

# I don't have file extensions on entry permalinks
Rack::Mime::MIME_TYPES.merge!("" => "text/html")

class DefaultIndexFile
  def initialize(app, &block)
    @app = app
    yield self if block_given?
  end

  def call(env)
    env['PATH_INFO'] << 'index.html' if env['PATH_INFO'] =~ /\/$/
    @app.call(env)
  end
end

use DefaultIndexFile

run Rack::File.new('output')

A little math can be a dangerous thing. Here’s a quick benchmark for Fibonacci numbers (Y axis in seconds, 20,000 iterations):

Graph

The line on the left, marked A, is the typical naive recursive solution, based as directly as possible on the mathematical definition of the function:

def fib_r(n)
  return 1 if n <= 2
  fib_r(n-1) + fib_r(n-2)
end

Note the telltale exponential curve - this is what we call Very Bad. The line there stops at Fib(10).

A little jiggery lets us convert the recursive solution into an iterative one, line B:

def fib_i(n)
  x = y = 1
  (n-1).times do
    x, y = y, x+y
  end
  x
end

My personal favorite Fib function is (ab)using a hash’s default value function to act as a memoizer. Line C is the fastest of the bunch (constant time of ~0.012s) but does have some extra memory overhead.

def fib_h(n)
  @h ||= Hash.new{ |h,k|
    if k < 2
      h[k] = k
    else
      h[k] = h[k-1] + h[k-2]
    end
  }
  @h[n]
end

If we were in an embedded system or something, the memory overhead would probably be bad, but fear not! Math can save us!

Line D is also constant time, but about 0.037s. It’s a fun little bit of math mentioned in SICP, and turns Fibonacci numbers into a very simple (if unintuitive) bit of math:

SQRT5 = Math.sqrt(5.0)
PHI = (1 + SQRT5) / 2
PSI = (1 - SQRT5) / 2

def fib_m(n)
  ((PHI ** n - PSI ** n) / SQRT5).to_i
end

Daniel Manges did up instructions on storing explicit versions of gems in your rails app. If instead you’re using merb, you probably want to do that too, as it makes for much easier deploys.

Thankfully, it’s much less pain in merb:

gem install async-observer -i gems

The -i option tells rubygems to install the gems to that directory, and when you require the gem from inside merb, it will look in the local gems directory first, and find yours.

Optionally, you can skip generating docs by adding --no-rdoc --no-ri to the line, and depending on what you’re installing, you may want to --ignore-dependencies as well.

If the gem in question builds a binary extension, you may be out of luck if you try to deploy it to a different architecture.

The only problem I’ve found so far is that gem cleanup won’t accept a directory to clean, so you’ll need to manually remove individual gems when you upgrade:

gem uninstall -i async-observer

This helper is based on some other controller/view helpers I’ve been working on and planning on blogging soon, with a nod to the specs present in the merb-mailer library itself.

I’m still considering the idea of separate specs for UserMailer and its views, but I think the overhead is too much for mailers, compared to the benefits we get for regular controllers/views. I think this is a result of the way the send_mail helper functions.

# in a controller
send_mail UserMailer, :hello, {
  :from => "greeter@example.com",
  :to => @person.email,
  :subject => "Greetings"
}, {
  :name => @person.name
}

The controller spec can simply stub/mock the send_mail call as appropriate.

# spec/spec_helper.rb
Merb::Mailer.delivery_method = :test_send
def describe_mail(mailer, template, &block)
  describe "/#{mailer.to_s.downcase}/#{template}" do
    before :each do
      @mailer_class, @template = mailer, template
      @assigns = {}
    end

    def deliver(send_params={}, mail_params={})
      mail_params = {:from => "from@example.com", :to => "to@example.com", :subject => "Subject Line"}.merge(mail_params)
      @mailer_class.new(send_params).dispatch_and_deliver @template.to_sym, mail_params
      @mail = Merb::Mailer.deliveries.last
    end

    instance_eval &block
  end
end

# spec/mailers/user\_mailer\_spec.rb
require File.join(File.dirname(__FILE__),'..','spec_helper')

describe_mail UserMailer, :hello do
  it "should say hello" do
    deliver :name => "Jamie"
    @mail.text.should == "Hello Jamie"
  end
end

I’m a big fan of custom rspec describers, as above. The fact that before and after blocks are transparently inherited is a huge win over test/unit, where you’d need to explicitly call super.

# app/mailers/user_mailer.rb
class UserMailer < Merb::MailController
  def hello
    render_mail
  end  
end

# app/mailers/views/user_mailer/hello.text.erb
Hello <%= params[:name] %>

As an update to a previous article,

if $specs_timed.nil? && ENV.has_key?('SLOW')
  $specs_timed = true
  $timings = []

  Spec::Example::ExampleGroup.prepend_before do
    @start = Time.now
  end
  Spec::Example::ExampleGroup.append_after do
    elapsed = Time.now - @start
    if elapsed > ENV['SLOW'].to_f
      $timings << [elapsed, "#{self.class.description} #{description}"]
    end
  end

  at_exit do
    puts "\nSlow Specs:"
    $timings.sort{|a,b| a.first <=> b.first}.each do |time, name|
      puts " %7.4f #{name}" % time
    end
    puts "  None!" if $timings.empty?
  end
end

Then, simply run

rake SLOW=0.1

HGTV in Canada produces a tv show called Holmes on Homes that follows general contractor Mike Holmes as he visits failed renovations, provides commentary on the sorry situation of the work done, and then goes about fixing them.

I was recommended to the series by a friend of mine, who has suggested that the shows and situations very often have a correlation to the world of software development. After seeing the first four episodes, I decided he was right, that all the episodes I’ve seen have direct quotes that are applicable, and that I should start blogging them.

So, consider this the “front page” article on this, I’ll fill in the individual episode links as I get around to them. The list of episodes is just the ones I have available to watch (on DVD or from HGTV) at the moment.

Season One

Additional Grief
Soggy Sorority
Botched Basement
Attica! Attica / Crappy Capping
Flimsy Floor
Kitchen Catastrophe
Window Pain
Faulty Showers
Tiles and Tribulations
Site Unseen
Sweet Home Abandoned
Whole House Disaster

Season Two

Terrible Terrace
Drafty Ducting
Ramp Revamp
Flooded Foundation
Garage Grievance
Lamin-Ain’t
Roof Goof
Floor Fiasco
Doozy Jacuzzi
No Grout About It
Jacking the Box
Access Denied
Hell’s Kitchen
Holmes for the Holidays

Season Five

Holmes Inspection
Showing the Cracks
What a Mesh

Season Six

Due Date
Frozen Assets
Gone to Pot
Lack of Truss
Clean Slate
Completely Incomplete
Nashville Kitchen
Pasadena 911
Shaky Foundation
Stone Walled
Third Time Lucky

Season Seven

Hit the Deck
Rocky Reno
Paradise Island

Specials/Unaired

Lien on Me

It seems that there were people making both audio and video recordings of talks at RejectConf this year.

Just looking at these, it’s easy to see where one framework shines and the other would require more work to get there.

Rplug has a new release up, which should now be useful for the world at large, as it has gained support for projects in subversion.

I should probably do a check after I’ve done the export and cull any now-empty directories from the plugin dir, but that’ll come in time, I’m sure.

[Updates below]

I’ve been having problems getting SourceControl deployed, turns out (unsurprisingly) to be user error - I’m new to this whole rubyforge/gem scene.

So, for the record, prior to releasing a gem using Hoe, one needs to get rubyforge configured. For me, this wound up being:

$ rubyforge setup
$ rubyforge config rplug
$ rubyforge config sourcecontrol

After all that, SourceControl is deploying just fine.

Both projects are entirely up in subversion if anyone wants to check them out, but they’re not quite ready for public consumption at the moment.

Example usage and output follows.

% rplug install exception_logger http://svn.techno-weenie.net/projects/plugins/exception_logger svn
Recorded exception_logger, run 'rplug update' to pull the latest revision

% rplug update
Working in project dir /home/jamie/dev/redvase
Updating exception_logger...
  upgrading to revision 2733
  updating local repository
  Done.
Updating mocha...
  Done.
Updating helper_test...
  Done.
Updating arts...
  Done.
Updating liquid...
  Done.

% rplug status
Working in project dir /home/jamie/dev/redvase
Managing the following plugins:
  arts, revision 70
  exception_logger, revision 2
  helper_test, revision 85
  liquid, revision 140
  mocha, revision 99
Not Managing the following plugins:
  test_timer

% rplug update -p exception_logger -r 2563
Working in project dir /home/jamie/dev/redvase
Updating exception_logger...
  upgrading to revision 2563
  updating local repository
  Done.

Example run:

# TEST_TIMER=0.25 rake test:units TEST=test/unit/creative_test.rb
/usr/bin/rake:17:Warning: require_gem is obsolete.  Use gem instead.
(in /home/jamie/dev/redvase)
/usr/bin/ruby1.8 -Ilib:test "/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader.rb" "test/unit/creative_test.rb"
Loaded suite /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader
Started
......................................................................................
Finished in 10.116575 seconds.

86 tests, 164 assertions, 0 failures, 0 errors

Test Benchmark Results
  0.2927 CreativeTest#test_delayed_click_count_with_third_party_stats
  0.2982 CreativeTest#test_impression_count_for_date_range_with_third_party_stats_offset
  0.3240 CreativeTest#test_global_creative_stats_should_return_correct_default_values
  6.2505 CreativeTest#test_click_count

Source file, lib/test_timer.rb

if ENV.has_key? 'TEST_TIMER' and
   !Test::Unit::TestCase.method_defined? :untimed_run

  class Test::Unit::TestCase
    cattr_reader :benchmark_data
    @@benchmark_data = {}
    alias untimed_run run

    def run(result, &progress_block)
      start = Time.now
      untimed_run(result, &progress_block)
      finish = Time.now
      elapsed = finish - start
      if elapsed > ENV['TEST_TIMER'].to_f
        name =~ /(.*)\((.*)\)/
        @@benchmark_data["#{$2}##{$1}"] = elapsed
      end
    end
  end

  # at_exit hooks run in reverse order, so in order to run after
  # Test::Unit's hook, we need to nest at_exit calls.
  at_exit do
    at_exit do
      results = Test::Unit::TestCase.benchmark_data
      unless results.empty?
        puts "\nTest Benchmark Results"
        results.sort{|a,b| a.last <=> b.last }.each do |key,value|
          puts " %7.4f #{key}" % value
        end
      end
    end
  end
end

ActiveRecord

find_all and find_first are deprecated. Use find(:all) and find(:first) instead.

Routes

Just a short note here, :requirements regexps no longer accept anchors. We have something along these lines:

map.connect ':controller/:action/:foo', :requirements => { :foo => /^(bar|baz)$/ }

Such that that route only fires if the third url part is exactly bar or baz. To silence 1.2, just remove the ^ and $ anchors.

Controllers/Views

If you want to have your link_to go by post instead of get, use :method => :post instead of :post => true.

(Update: This will not throw errors, but won’t work in 1.1.6, so wait until you’re actually running 1.2 to do this change.)

Tests

To Be Continued

Like I said, I’m only half done this migration, but when I get the rest of it sorted, I’ll be posting a follow-up right here. See you then.

class BrokenError < StandardError
  def backtrace
    raise(StandardError.new)
  end
end

begin
  raise BrokenError.new
rescue e
  puts 'rescued'
end

Found this one out migrating a rails app from 1.1.6 to 1.2. Instead of doing this:

render 'controller/action'

the deprecation warning suggests the following:

render :file => 'controller/action'

render :file => 'controller/action', :use_full_path => true

Well, I managed to get the weight-tracking app functional (graph and all) in about 220 lines, just tweaking the look now. It’s a single-script Camping app using Gruff for graphing, with SQLite for data storage. Not exactly the most efficient app (I’m cheating by using a lot of mostly-null records in the database) but it gets the job done. There were a few things that got me stuck for a bit that weren’t obviously mentioned in the camping docs, so I thought I’d put them down here.

Not exactly a camping thing, but if you want a non 4:3 ratio gruff graph, send a string ‘1000x350’ or similar.

def fib_iter(n)
  return 1 if n < 3  
  f = f1 = 1
  (2..n).each do
    f, f1 = (f+f1), f
  end
  f
end

The benchmarking speaks for itself. (Same parameters as Ryan’s benching, 10,000 runs doing fib(15)):

                      user     system      total        real
fib-ruby         21.180000   3.640000  24.820000 ( 24.989140)
fib-hash-reset    0.510000   0.070000   0.580000 (  0.609976)
fib-cache-reset   0.510000   0.050000   0.560000 (  0.570715)
fib-iter          0.160000   0.020000   0.180000 (  0.209565)
fib-hash          0.020000   0.000000   0.020000 (  0.034616)
fib-cached        0.020000   0.010000   0.030000 (  0.035222)

Benchmarks for fib-ruby and fib-cached come from Ryan’s post. fib-iter and fib-hash are mine.

So once again, it seems that there’s a different best solution for two different problems.

def hashfib(n)
  return 1 if n <= 1
  h = Hash.new{|h,k| h[k] = h[k-1] + h[k-2] }
  h[1] = 1
  h[2] = 1
  h[n]
end

The cached version uses @@h instead of h, and

=s it.

Chris Abad wrote yesterday about his experience with dynamic attributes, and I thought I’d share mine.

create_table "leads", :force => true do |t|
  t.column "email", :string, :default => "", :null => false
  t.column "firstname", :string
  t.column "lastname", :string
  t.column "ip", :string, :default => "", :null => false
end
create_table "lead_infos", :force => true do |t|
  t.column "lead_id", :integer, :default => 0, :null => false
  t.column "name", :string, :default => "", :null => false
  t.column "value", :string, :default => "", :null => false
end

class Lead < ActiveRecord::Base
  has_many :lead_infos, :dependent => true

  def method_missing(methodname, *args)
    begin
      super
    rescue NameError
      name = methodname.to_s.chomp('=')
      if (methodname.to_s =~ /=$/)
        LeadInfo.create(:name => name, :value => args.first, :lead_id => self.id)
        self
      else
        LeadInfo.find(:first, :conditions => ['name = ?', name]) or raise
        lead_infos.find_by_name(name).value rescue ''
      end
    end
  end

  ...
end

Walking through this, I first make sure to call super so that ActiveRecord’s method_missing gets run first. If it can’t find anything to do, then it is the Lead’s turn to try.

There’s benefits to both what I’ve done and what Chris has done, so anyone reading these can take their pick :)

Comments

Chris Abad, at 18:45, Sep 13 2006 I was doing a bit of data processing the other night. A little copying here, a bit of typing there, formatting into YAML, then loaded into a Ruby script. Loop through the hashes YAML loaded, and try to make some sense out of it.

Rails’ routing framework is a pretty capable beast, but it does sometimes still need a bit of help doing more exotic things.

module ActionController::Routing
  module ConditionConstants; end

  class CustomCondition < Regexp
    def initialize(name, &match)
      super '^$' # pretend we're a regular empty string regexp
      @name, @match = name, match
    end
    def =~(other); @match.call(other); end
    def inspect;   @name;              end
  end

  def self.custom_condition(name, &block)
    ConditionConstants.const_set(name, CustomCondition.new(name, &block))
  end

  custom_condition('ProjectCondition'){|other| Project.find_by_url(other) }
end

Anyone interested in the hows and whys, feel free to read on…

Thus began my CustomCondition class.

I first set up an empty module named ConditionConstants that I would use to house my constants.

CustomCondition#inspect did the obvious, and output the name we were to be remembering.

CustomCondition#=~ simply called the stored block with the argument we are trying to match, allowing the creation of the object to completely drive its purpose.

Once that was all in place, I simply called my helper method with the name of the constant, and a block encapsulating the behaviour. Done and done.

Great thought that came to me in the shower this morning: Agile software development is like driving.

Your team is driving the project to its destination. It’s travelling down a multi-lane highway. Some team members like driving in the fast lane, others take it a little more deliberate and cautious and go in the slow lane. But they’re all heading to the same place.

The team members like to mix it up a little, swapping cars and who drives, so that everyone can get a feel for the entire fleet.

The important thing is that everyone is in constant communication, even while they’re driving. If we find that we need to change destinations, it’s not a problem for everyone to make the right exit - even the guy in the fast lane - because there’s always someone in each car with the map. This is why we work in pairs: driving with a map in front of you is a hell of a lot harder than having a co-pilot/navigator.

Out to lunch with the department today, bemoaning the state of Computer Science education. The general consensus is that the local universities seem to be teaching Java in an SWF setting - Structs With Functions. This leads to a seriously flawed notion of what OO programming is, and can seriously warp the still-forming mind of a new coder.

Because I am wont to do so (and have been doing so for that last two weeks or so, much to everyone’s chagrin), I took this a step further. Really, if you’re writing Java then what you have are methods, not functions. Even if you use them like that. So rather than have SWF, you have SWM, or Swim.

This is followed by much entertainment exploring the analogy - things start off easy, but as you do more of it it’s harder to keep afloat, there’s a possibility you might just wash out, etc. Then Jon comes up with the whopper:

“So that’s why they call it the waterfall method of software development”

</th> <th><a name=”programming<!doctype html>

No More Bundle Exec - set_trace_func

No More Bundle Exec

6th Sep 2012 | Tags: programming ruby

Update 2023: Just use direnv with layout ruby.

function command_not_found_handler() {
    ~/bin/command-not-found $*
}

My script happens to be in Ruby, but it could just as easily be a standard shell script as all I’m doing is some file existence tests:

#!/usr/bin/env ruby

# ARGV is the entire command we wanted to run, but we
# really only care about the actual executable for fallbacks
command = ARGV.first

def run(cmd)
  $stderr.puts "Running #{cmd.inspect} instead"
  system(cmd)
end

case
when File.exist?("./.bundle/config") && File.exist?("./bin/#{command}")
  run("bundle exec #{ARGV.join(' ')}")

else
  exit 127
end

# Paste git repo url to clone it
when command =~ /^git(@|:\/\/).*\.git$/
  run("git clone #{command.inspect}")

# paste compressed url to download+extract it
when command =~ /^(?:ftp|https?):\/\/.+\.t(?:ar\.)?gz$/
  run("curl #{command.inspect} | tar xzv")

when File.exist?("./tmp/isolate/ruby-1.8/bin/#{command}")
  run("rake isolate:sh['#{ARGV.join(' ')}']")

Heroku kicks ass. Very nifty tech running the show, and a really simple interface for deploying - just push up your git repository.

All you need is a config.ru file like so, and then be sure to include the generated output dir in your git repository:

# I don't have file extensions on entry permalinks
Rack::Mime::MIME_TYPES.merge!("" => "text/html")

class DefaultIndexFile
  def initialize(app, &block)
    @app = app
    yield self if block_given?
  end

  def call(env)
    env['PATH_INFO'] << 'index.html' if env['PATH_INFO'] =~ /\/$/
    @app.call(env)
  end
end

use DefaultIndexFile

run Rack::File.new('output')

A little math can be a dangerous thing. Here’s a quick benchmark for Fibonacci numbers (Y axis in seconds, 20,000 iterations):

Graph

The line on the left, marked A, is the typical naive recursive solution, based as directly as possible on the mathematical definition of the function:

def fib_r(n)
  return 1 if n <= 2
  fib_r(n-1) + fib_r(n-2)
end

Note the telltale exponential curve - this is what we call Very Bad. The line there stops at Fib(10).

A little jiggery lets us convert the recursive solution into an iterative one, line B:

def fib_i(n)
  x = y = 1
  (n-1).times do
    x, y = y, x+y
  end
  x
end

def fib_h(n)
  @h ||= Hash.new{ |h,k|
    if k < 2
      h[k] = k
    else
      h[k] = h[k-1] + h[k-2]
    end
  }
  @h[n]
end

If we were in an embedded system or something, the memory overhead would probably be bad, but fear not! Math can save us!

Line D is also constant time, but about 0.037s. It’s a fun little bit of math mentioned in SICP, and turns Fibonacci numbers into a very simple (if unintuitive) bit of math:

SQRT5 = Math.sqrt(5.0)
PHI = (1 + SQRT5) / 2
PSI = (1 - SQRT5) / 2

def fib_m(n)
  ((PHI ** n - PSI ** n) / SQRT5).to_i
end

Daniel Manges did up instructions on storing explicit versions of gems in your rails app. If instead you’re using merb, you probably want to do that too, as it makes for much easier deploys.

Thankfully, it’s much less pain in merb:

gem install async-observer -i gems

The -i option tells rubygems to install the gems to that directory, and when you require the gem from inside merb, it will look in the local gems directory first, and find yours.

Optionally, you can skip generating docs by adding --no-rdoc --no-ri to the line, and depending on what you’re installing, you may want to --ignore-dependencies as well.

If the gem in question builds a binary extension, you may be out of luck if you try to deploy it to a different architecture.

The only problem I’ve found so far is that gem cleanup won’t accept a directory to clean, so you’ll need to manually remove individual gems when you upgrade:

gem uninstall -i async-observer

This helper is based on some other controller/view helpers I’ve been working on and planning on blogging soon, with a nod to the specs present in the merb-mailer library itself.

# in a controller
send_mail UserMailer, :hello, {
  :from => "greeter@example.com",
  :to => @person.email,
  :subject => "Greetings"
}, {
  :name => @person.name
}

The controller spec can simply stub/mock the send_mail call as appropriate.

# spec/spec_helper.rb
Merb::Mailer.delivery_method = :test_send
def describe_mail(mailer, template, &block)
  describe "/#{mailer.to_s.downcase}/#{template}" do
    before :each do
      @mailer_class, @template = mailer, template
      @assigns = {}
    end

    def deliver(send_params={}, mail_params={})
      mail_params = {:from => "from@example.com", :to => "to@example.com", :subject => "Subject Line"}.merge(mail_params)
      @mailer_class.new(send_params).dispatch_and_deliver @template.to_sym, mail_params
      @mail = Merb::Mailer.deliveries.last
    end

    instance_eval &block
  end
end

# spec/mailers/user\_mailer\_spec.rb
require File.join(File.dirname(__FILE__),'..','spec_helper')

describe_mail UserMailer, :hello do
  it "should say hello" do
    deliver :name => "Jamie"
    @mail.text.should == "Hello Jamie"
  end
end

I’m a big fan of custom rspec describers, as above. The fact that before and after blocks are transparently inherited is a huge win over test/unit, where you’d need to explicitly call super.

# app/mailers/user_mailer.rb
class UserMailer < Merb::MailController
  def hello
    render_mail
  end  
end

# app/mailers/views/user_mailer/hello.text.erb
Hello <%= params[:name] %>

As an update to a previous article,

if $specs_timed.nil? && ENV.has_key?('SLOW')
  $specs_timed = true
  $timings = []

  Spec::Example::ExampleGroup.prepend_before do
    @start = Time.now
  end
  Spec::Example::ExampleGroup.append_after do
    elapsed = Time.now - @start
    if elapsed > ENV['SLOW'].to_f
      $timings << [elapsed, "#{self.class.description} #{description}"]
    end
  end

  at_exit do
    puts "\nSlow Specs:"
    $timings.sort{|a,b| a.first <=> b.first}.each do |time, name|
      puts " %7.4f #{name}" % time
    end
    puts "  None!" if $timings.empty?
  end
end

Then, simply run

rake SLOW=0.1

Season One

Additional Grief
Soggy Sorority
Botched Basement
Attica! Attica / Crappy Capping
Flimsy Floor
Kitchen Catastrophe
Window Pain
Faulty Showers
Tiles and Tribulations
Site Unseen
Sweet Home Abandoned
Whole House Disaster

Season Two

Terrible Terrace
Drafty Ducting
Ramp Revamp
Flooded Foundation
Garage Grievance
Lamin-Ain’t
Roof Goof
Floor Fiasco
Doozy Jacuzzi
No Grout About It
Jacking the Box
Access Denied
Hell’s Kitchen
Holmes for the Holidays

Season Five

Holmes Inspection
Showing the Cracks
What a Mesh

Season Six

Due Date
Frozen Assets
Gone to Pot
Lack of Truss
Clean Slate
Completely Incomplete
Nashville Kitchen
Pasadena 911
Shaky Foundation
Stone Walled
Third Time Lucky

Season Seven

Hit the Deck
Rocky Reno
Paradise Island

Specials/Unaired

Lien on Me

It seems that there were people making both audio and video recordings of talks at RejectConf this year.

Just looking at these, it’s easy to see where one framework shines and the other would require more work to get there.

Rplug has a new release up, which should now be useful for the world at large, as it has gained support for projects in subversion.

I should probably do a check after I’ve done the export and cull any now-empty directories from the plugin dir, but that’ll come in time, I’m sure.

[Updates below]

I’ve been having problems getting SourceControl deployed, turns out (unsurprisingly) to be user error - I’m new to this whole rubyforge/gem scene.

So, for the record, prior to releasing a gem using Hoe, one needs to get rubyforge configured. For me, this wound up being:

$ rubyforge setup
$ rubyforge config rplug
$ rubyforge config sourcecontrol

After all that, SourceControl is deploying just fine.

Both projects are entirely up in subversion if anyone wants to check them out, but they’re not quite ready for public consumption at the moment.

Example usage and output follows.

% rplug install exception_logger http://svn.techno-weenie.net/projects/plugins/exception_logger svn
Recorded exception_logger, run 'rplug update' to pull the latest revision

% rplug update
Working in project dir /home/jamie/dev/redvase
Updating exception_logger...
  upgrading to revision 2733
  updating local repository
  Done.
Updating mocha...
  Done.
Updating helper_test...
  Done.
Updating arts...
  Done.
Updating liquid...
  Done.

% rplug status
Working in project dir /home/jamie/dev/redvase
Managing the following plugins:
  arts, revision 70
  exception_logger, revision 2
  helper_test, revision 85
  liquid, revision 140
  mocha, revision 99
Not Managing the following plugins:
  test_timer

% rplug update -p exception_logger -r 2563
Working in project dir /home/jamie/dev/redvase
Updating exception_logger...
  upgrading to revision 2563
  updating local repository
  Done.

Example run:

# TEST_TIMER=0.25 rake test:units TEST=test/unit/creative_test.rb
/usr/bin/rake:17:Warning: require_gem is obsolete.  Use gem instead.
(in /home/jamie/dev/redvase)
/usr/bin/ruby1.8 -Ilib:test "/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader.rb" "test/unit/creative_test.rb"
Loaded suite /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader
Started
......................................................................................
Finished in 10.116575 seconds.

86 tests, 164 assertions, 0 failures, 0 errors

Test Benchmark Results
  0.2927 CreativeTest#test_delayed_click_count_with_third_party_stats
  0.2982 CreativeTest#test_impression_count_for_date_range_with_third_party_stats_offset
  0.3240 CreativeTest#test_global_creative_stats_should_return_correct_default_values
  6.2505 CreativeTest#test_click_count

Source file, lib/test_timer.rb

if ENV.has_key? 'TEST_TIMER' and
   !Test::Unit::TestCase.method_defined? :untimed_run

  class Test::Unit::TestCase
    cattr_reader :benchmark_data
    @@benchmark_data = {}
    alias untimed_run run

    def run(result, &progress_block)
      start = Time.now
      untimed_run(result, &progress_block)
      finish = Time.now
      elapsed = finish - start
      if elapsed > ENV['TEST_TIMER'].to_f
        name =~ /(.*)\((.*)\)/
        @@benchmark_data["#{$2}##{$1}"] = elapsed
      end
    end
  end

  # at_exit hooks run in reverse order, so in order to run after
  # Test::Unit's hook, we need to nest at_exit calls.
  at_exit do
    at_exit do
      results = Test::Unit::TestCase.benchmark_data
      unless results.empty?
        puts "\nTest Benchmark Results"
        results.sort{|a,b| a.last <=> b.last }.each do |key,value|
          puts " %7.4f #{key}" % value
        end
      end
    end
  end
end

ActiveRecord

find_all and find_first are deprecated. Use find(:all) and find(:first) instead.

Routes

Just a short note here, :requirements regexps no longer accept anchors. We have something along these lines:

map.connect ':controller/:action/:foo', :requirements => { :foo => /^(bar|baz)$/ }

Such that that route only fires if the third url part is exactly bar or baz. To silence 1.2, just remove the ^ and $ anchors.

Controllers/Views

If you want to have your link_to go by post instead of get, use :method => :post instead of :post => true.

(Update: This will not throw errors, but won’t work in 1.1.6, so wait until you’re actually running 1.2 to do this change.)

Tests

To Be Continued

Like I said, I’m only half done this migration, but when I get the rest of it sorted, I’ll be posting a follow-up right here. See you then.

class BrokenError < StandardError
  def backtrace
    raise(StandardError.new)
  end
end

begin
  raise BrokenError.new
rescue e
  puts 'rescued'
end

Found this one out migrating a rails app from 1.1.6 to 1.2. Instead of doing this:

render 'controller/action'

the deprecation warning suggests the following:

render :file => 'controller/action'

render :file => 'controller/action', :use_full_path => true

Not exactly a camping thing, but if you want a non 4:3 ratio gruff graph, send a string ‘1000x350’ or similar.

def fib_iter(n)
  return 1 if n < 3  
  f = f1 = 1
  (2..n).each do
    f, f1 = (f+f1), f
  end
  f
end

The benchmarking speaks for itself. (Same parameters as Ryan’s benching, 10,000 runs doing fib(15)):

                      user     system      total        real
fib-ruby         21.180000   3.640000  24.820000 ( 24.989140)
fib-hash-reset    0.510000   0.070000   0.580000 (  0.609976)
fib-cache-reset   0.510000   0.050000   0.560000 (  0.570715)
fib-iter          0.160000   0.020000   0.180000 (  0.209565)
fib-hash          0.020000   0.000000   0.020000 (  0.034616)
fib-cached        0.020000   0.010000   0.030000 (  0.035222)

Benchmarks for fib-ruby and fib-cached come from Ryan’s post. fib-iter and fib-hash are mine.

So once again, it seems that there’s a different best solution for two different problems.

def hashfib(n)
  return 1 if n <= 1
  h = Hash.new{|h,k| h[k] = h[k-1] + h[k-2] }
  h[1] = 1
  h[2] = 1
  h[n]
end

The cached version uses @@h instead of h, and

=s it.

Chris Abad wrote yesterday about his experience with dynamic attributes, and I thought I’d share mine.

create_table "leads", :force => true do |t|
  t.column "email", :string, :default => "", :null => false
  t.column "firstname", :string
  t.column "lastname", :string
  t.column "ip", :string, :default => "", :null => false
end
create_table "lead_infos", :force => true do |t|
  t.column "lead_id", :integer, :default => 0, :null => false
  t.column "name", :string, :default => "", :null => false
  t.column "value", :string, :default => "", :null => false
end

class Lead < ActiveRecord::Base
  has_many :lead_infos, :dependent => true

  def method_missing(methodname, *args)
    begin
      super
    rescue NameError
      name = methodname.to_s.chomp('=')
      if (methodname.to_s =~ /=$/)
        LeadInfo.create(:name => name, :value => args.first, :lead_id => self.id)
        self
      else
        LeadInfo.find(:first, :conditions => ['name = ?', name]) or raise
        lead_infos.find_by_name(name).value rescue ''
      end
    end
  end

  ...
end

Walking through this, I first make sure to call super so that ActiveRecord’s method_missing gets run first. If it can’t find anything to do, then it is the Lead’s turn to try.

There’s benefits to both what I’ve done and what Chris has done, so anyone reading these can take their pick :)

Comments

Chris Abad, at 18:45, Sep 13 2006 I was doing a bit of data processing the other night. A little copying here, a bit of typing there, formatting into YAML, then loaded into a Ruby script. Loop through the hashes YAML loaded, and try to make some sense out of it.

Rails’ routing framework is a pretty capable beast, but it does sometimes still need a bit of help doing more exotic things.

module ActionController::Routing
  module ConditionConstants; end

  class CustomCondition < Regexp
    def initialize(name, &match)
      super '^$' # pretend we're a regular empty string regexp
      @name, @match = name, match
    end
    def =~(other); @match.call(other); end
    def inspect;   @name;              end
  end

  def self.custom_condition(name, &block)
    ConditionConstants.const_set(name, CustomCondition.new(name, &block))
  end

  custom_condition('ProjectCondition'){|other| Project.find_by_url(other) }
end

Anyone interested in the hows and whys, feel free to read on…

Thus began my CustomCondition class.

I first set up an empty module named ConditionConstants that I would use to house my constants.

CustomCondition#inspect did the obvious, and output the name we were to be remembering.

CustomCondition#=~ simply called the stored block with the argument we are trying to match, allowing the creation of the object to completely drive its purpose.

Once that was all in place, I simply called my helper method with the name of the constant, and a block encapsulating the behaviour. Done and done.

Great thought that came to me in the shower this morning: Agile software development is like driving.

The team members like to mix it up a little, swapping cars and who drives, so that everyone can get a feel for the entire fleet.

“So that’s why they call it the waterfall method of software development”

” class=”anchor”> </th></tr>

  <tr><th>sql<!doctype html>

Parsing JSON in SQL - set_trace_func

Parsing JSON in SQL

19th Nov 2011 | Tags: ruby rails sql

The Problem: You have a database column with some data serialzed as JSON in it that you’d like to pull out into its own column to index it.

def json(key, field='params')
  key_json = "\"#{key}\":"

  # key start/end locations, including ""
  k_a = "LOCATE('#{key_json}', #{field})"
  k_z = "LOCATE('\"', #{field}, #{k_a}+1)" # this is terminating "

  # is there a space after colons?
  spad = "IF(LOCATE('\": ', #{field}), 1, 0)"

  # is value a string?
  val_string = "LOCATE(CONCAT('#{key_json}', IF(#{spad},' ',''), '\"'), #{field}, #{k_a})"
  qpad = "IF(#{val_string}, 1, 0)"

  # value start/end locations, excluding "" if present
  v_a = "(#{k_z}+1 + 1 + #{spad} + #{qpad})" # 1 for colon, spad for optional space, qpad for possible quote

  end_if_string = "LOCATE('\"', #{field}, #{v_a})"
  end_if_not_string = "IF(LOCATE(',', #{field}, #{v_a}), LOCATE(',', #{field}, #{v_a}), LOCATE('}', #{field}, #{v_a}))"

  v_z = "IF(#{val_string}, #{end_if_string}, #{end_if_not_string})"

  value_string = "SUBSTRING(#{field} FROM #{v_a} FOR (#{v_z} - #{v_a}))"
  "IF(#{k_a}, #{value_string}, NULL)"
end

up do
  execute "
    UPDATE model_table
    SET status = #{json('status')}
  "
end

The generated sql looks pretty gnarly but mysql ran through it stupidly fast. I shudder to think how long it’d take activerecord to load and update each record individually.

</th> <th><a name=”sql<!doctype html>

Parsing JSON in SQL - set_trace_func

Parsing JSON in SQL

19th Nov 2011 | Tags: ruby rails sql

The Problem: You have a database column with some data serialzed as JSON in it that you’d like to pull out into its own column to index it.

def json(key, field='params')
  key_json = "\"#{key}\":"

  # key start/end locations, including ""
  k_a = "LOCATE('#{key_json}', #{field})"
  k_z = "LOCATE('\"', #{field}, #{k_a}+1)" # this is terminating "

  # is there a space after colons?
  spad = "IF(LOCATE('\": ', #{field}), 1, 0)"

  # is value a string?
  val_string = "LOCATE(CONCAT('#{key_json}', IF(#{spad},' ',''), '\"'), #{field}, #{k_a})"
  qpad = "IF(#{val_string}, 1, 0)"

  # value start/end locations, excluding "" if present
  v_a = "(#{k_z}+1 + 1 + #{spad} + #{qpad})" # 1 for colon, spad for optional space, qpad for possible quote

  end_if_string = "LOCATE('\"', #{field}, #{v_a})"
  end_if_not_string = "IF(LOCATE(',', #{field}, #{v_a}), LOCATE(',', #{field}, #{v_a}), LOCATE('}', #{field}, #{v_a}))"

  v_z = "IF(#{val_string}, #{end_if_string}, #{end_if_not_string})"

  value_string = "SUBSTRING(#{field} FROM #{v_a} FOR (#{v_z} - #{v_a}))"
  "IF(#{k_a}, #{value_string}, NULL)"
end

up do
  execute "
    UPDATE model_table
    SET status = #{json('status')}
  "
end

The generated sql looks pretty gnarly but mysql ran through it stupidly fast. I shudder to think how long it’d take activerecord to load and update each record individually.

” class=”anchor”> </th></tr>

  <tr><th>hoptoad[Hoptoad](http://hoptoadapp.com) has been bugging us for a week or two now to upgrade to v2 of its API, so we did that this week for our project at work. Except we're running Merb, not Rails.

Previously, we’ve been using Atmos’ merb_hoptoad plugin, but it looks like it’s been abandoned now in favor of a rack handler, and we needed to hack it a bit to support running multiple sites (with different API keys) off our single codebase. I’m always happy throwing code away, though, so I thought I’d try using the official Hoptoad Notifier plugin and see how hard it is to get working. It wasn’t.

First, you probably want to make a gem of the plugin. I forked it a few days back, and just added a jeweller task to create the gem locally. Install in the local gems directory (or system-wide in production if you want to do it the hard way) and add it as a dependency.

Then, to actually use it, just add the following in the right places:

# init.rb, in Merb::Bootloader.after_app_loads

HoptoadNotifier.configure do |config|
  config.api_key = '...'
  config.environment_name = Merb.env
  config.project_root = Merb.root
  # See http://github.com/thoughtbot/hoptoad_notifier/ README.rdoc
  # for other config settings. You probably want to think about
  # params_filters and maybe ignore.
end

# application.rb, if you want available manually.
# If you just want it for completely unexpected errors you can stick it in
# exceptions.rb instead

def notify_hoptoad(error=nil)
  error ||= request.exceptions.first

  data = {
    :controller       => params[:controller],
    :action           => params[:action],
    :url              => "#{request.protocol}://#{request.host}#{request.uri}",
    # Looks like hoptoad is filtering these itself, we don't need to worry about it
    # other than configuring what needs to be filtered
    :parameters       => params.to_hash,
    :session_data     => session.to_hash,
    :cgi_data         => request.env.to_hash,
    :environment_vars => ENV.to_hash.merge(:RAILS_ENV => Merb.env)
  }

  HoptoadNotifier.notify(error, data)
end

# exceptions.rb, override default error handlers to submit to hoptoad

def internal_server_error
  notify_hoptoad
  render
end

def standard_error
  notify_hoptoad
  render
end    

# other spots you handle exceptions inline, just pass in the exception

begin
  ...
rescue => e
  notify_hoptoad(e)
end

It’s not exactly as magical as the Rails plugin’s auto-including, but it looks like it’s getting the job done for us. </th> <th><a name=”hoptoadHoptoad has been bugging us for a week or two now to upgrade to v2 of its API, so we did that this week for our project at work. Except we’re running Merb, not Rails.

Then, to actually use it, just add the following in the right places:

# init.rb, in Merb::Bootloader.after_app_loads

HoptoadNotifier.configure do |config|
  config.api_key = '...'
  config.environment_name = Merb.env
  config.project_root = Merb.root
  # See http://github.com/thoughtbot/hoptoad_notifier/ README.rdoc
  # for other config settings. You probably want to think about
  # params_filters and maybe ignore.
end

# application.rb, if you want available manually.
# If you just want it for completely unexpected errors you can stick it in
# exceptions.rb instead

def notify_hoptoad(error=nil)
  error ||= request.exceptions.first

  data = {
    :controller       => params[:controller],
    :action           => params[:action],
    :url              => "#{request.protocol}://#{request.host}#{request.uri}",
    # Looks like hoptoad is filtering these itself, we don't need to worry about it
    # other than configuring what needs to be filtered
    :parameters       => params.to_hash,
    :session_data     => session.to_hash,
    :cgi_data         => request.env.to_hash,
    :environment_vars => ENV.to_hash.merge(:RAILS_ENV => Merb.env)
  }

  HoptoadNotifier.notify(error, data)
end

# exceptions.rb, override default error handlers to submit to hoptoad

def internal_server_error
  notify_hoptoad
  render
end

def standard_error
  notify_hoptoad
  render
end    

# other spots you handle exceptions inline, just pass in the exception

begin
  ...
rescue => e
  notify_hoptoad(e)
end

It’s not exactly as magical as the Rails plugin’s auto-including, but it looks like it’s getting the job done for us. “ class=”anchor”> </th></tr>

  <tr><th>merb[Hoptoad](http://hoptoadapp.com) has been bugging us for a week or two now to upgrade to v2 of its API, so we did that this week for our project at work. Except we're running Merb, not Rails.

Then, to actually use it, just add the following in the right places:

# init.rb, in Merb::Bootloader.after_app_loads

HoptoadNotifier.configure do |config|
  config.api_key = '...'
  config.environment_name = Merb.env
  config.project_root = Merb.root
  # See http://github.com/thoughtbot/hoptoad_notifier/ README.rdoc
  # for other config settings. You probably want to think about
  # params_filters and maybe ignore.
end

# application.rb, if you want available manually.
# If you just want it for completely unexpected errors you can stick it in
# exceptions.rb instead

def notify_hoptoad(error=nil)
  error ||= request.exceptions.first

  data = {
    :controller       => params[:controller],
    :action           => params[:action],
    :url              => "#{request.protocol}://#{request.host}#{request.uri}",
    # Looks like hoptoad is filtering these itself, we don't need to worry about it
    # other than configuring what needs to be filtered
    :parameters       => params.to_hash,
    :session_data     => session.to_hash,
    :cgi_data         => request.env.to_hash,
    :environment_vars => ENV.to_hash.merge(:RAILS_ENV => Merb.env)
  }

  HoptoadNotifier.notify(error, data)
end

# exceptions.rb, override default error handlers to submit to hoptoad

def internal_server_error
  notify_hoptoad
  render
end

def standard_error
  notify_hoptoad
  render
end    

# other spots you handle exceptions inline, just pass in the exception

begin
  ...
rescue => e
  notify_hoptoad(e)
end

It’s not exactly as magical as the Rails plugin’s auto-including, but it looks like it’s getting the job done for us. Update: This code is stale, I’ve extracted a gem of it and posted on github.

First, install the amqp ruby library to connect to rabbit, and then add a tiny bit of setup.

In config/environment.rb:

require 'mq'
class BeanstalkPoolImpersonator
  def initialize(opts={})
    @opts = opts
  end

  def connect
    connection = AMQP.connect(@opts)
    @channel = channel = MQ.new(connection)
  end

  def use(queue)
    @queue = MQ::Queue.new(@channel, queue)
  end

  def yput(obj, pri, delay, ttr)
    p [obj, pri, delay, ttr]
    @queue.publish(YAML.dump(obj))
  end

  def last_server
    :last_server_stub
  end

  def subscribe(*args, &blk)
    @queue.subscribe(*args, &blk)
  end
end

Then, instead of connecting via Beanstalk::Pool.new, do this:

AsyncObserver::Queue.queue = BeanstalkPoolImpersonator.new()

You can pass an options hash to the new call, providing user, pass, vhost, host, or port as necessary.

Then, in your workers, load up the async_observer worker class, and extend like so:

class RabbitWorker < AsyncObserver::Worker
  def run()
    EM.run do
      AsyncObserver::Queue.queue.connect
      AsyncObserver::Queue.queue.use('1.0')
      AsyncObserver::Queue.queue.subscribe do |headers, msg|
        job = OpenStruct.new(:ybody => YAML.load(msg), :body => msg, :stats => [])
        job.id = headers.properties[:delivery_tag]
        safe_dispatch(job)
      end
    end
  end
end

Create the new worker the same way you would for the AO::Worker, and you’re set:

    RabbitWorker.new(binding).run()

Note: I’m maintaining a merb port of async-observer on github.

Create your spec in spec/views/layout/application.html.erb_spec.html, and add this class to it:

class Layout < Application
  layout nil
  def application; render; end
end

Then just test it as you would any other ordinary view.

We’re about to start up a new project at work, and we’ve decided to go with Merb (yay!) rather than Rails. Before we get started, though, we wanted to make sure that we’d be able to integrate well with the various plugins available for Rails.

The good news is that both of the projects are available on GitHub, which means easy forks, and easy contributions back to the source if my changes are as good as I think they are.

Basics

If you plan on dealing with the app directory structure or environment, an easy way to do it is:

if defined?(Rails)
  ROOT = RAILS_ROOT
  ENV = RAILS_ENV
elsif defined?(Merb)
  ROOT = Merb.root
  ENV = (Merb.env == 'rake' ? 'development' : Merb.env)
end

The additional rake environment transparently proxies to the development db connection, so if you just want to compare your plugin’s interpretation of ENV the above will make that cleaner.

Rake Tasks

Rails automatically loads any files matching tasks/*.rake in the plugin dir.

Merb needs to be told explicitly, relative to the lib directory. The canonical example from the docs is:

if defined?(Merb::Plugins)
  Merb::Plugins.add_rakefiles "merb_sequel" / "merbtasks"
end

    load File.expand_path(File.join(File.dirname(__FILE__), '..', '..', 'tasks', 'merb_sequel.rake'))

Do note that the load is necessary, require doesn’t pick the file up properly.

Finally, if you have any tasks that depend on environment, the easiest way to get compatability with both frameworks is to add task :environment => :merb_env to your merbtasks.rb file.

Generators

ORM Integration

As a follow-up to my previous post, here’s some gotchas to be aware of if you’re looking to support both ActiveRecord and DataMapper in a Merb (and/or Rails) plugin.

The Strategy

Basic Translation

Now that we have a plan, we can start translating our extracted functions from AR bits to DM bits. There’s a bunch of fairly straightforward transformations we can make.

ActiveRecord	DataMapper
.find(:all, ...)	.all(...)
.find(:first, ...)	.first(...)
.find(id)	.get(...)
.find_all_by_id(id)	.all(:id => ids)
.table_name	.storage_name
.primary_key	.key.first.name
.connection	repository.adapter

Raw SQL

Hooks

In before hooks, the AR hook chain stops if your method returns false, in DM you must throw :halt.

Things You Shouldn’t Be Doing Anyways

If you’re manually setting @attribute values in your AR code, you’ll need to use instance_variable_set for DataMapper. I recommend writing manual accessor methods to wrap it for abstraction.

Daniel Manges did up instructions on storing explicit versions of gems in your rails app. If instead you’re using merb, you probably want to do that too, as it makes for much easier deploys.

Thankfully, it’s much less pain in merb:

gem install async-observer -i gems

The -i option tells rubygems to install the gems to that directory, and when you require the gem from inside merb, it will look in the local gems directory first, and find yours.

Optionally, you can skip generating docs by adding --no-rdoc --no-ri to the line, and depending on what you’re installing, you may want to --ignore-dependencies as well.

If the gem in question builds a binary extension, you may be out of luck if you try to deploy it to a different architecture.

The only problem I’ve found so far is that gem cleanup won’t accept a directory to clean, so you’ll need to manually remove individual gems when you upgrade:

gem uninstall -i async-observer

This helper is based on some other controller/view helpers I’ve been working on and planning on blogging soon, with a nod to the specs present in the merb-mailer library itself.

# in a controller
send_mail UserMailer, :hello, {
  :from => "greeter@example.com",
  :to => @person.email,
  :subject => "Greetings"
}, {
  :name => @person.name
}

The controller spec can simply stub/mock the send_mail call as appropriate.

# spec/spec_helper.rb
Merb::Mailer.delivery_method = :test_send
def describe_mail(mailer, template, &block)
  describe "/#{mailer.to_s.downcase}/#{template}" do
    before :each do
      @mailer_class, @template = mailer, template
      @assigns = {}
    end

    def deliver(send_params={}, mail_params={})
      mail_params = {:from => "from@example.com", :to => "to@example.com", :subject => "Subject Line"}.merge(mail_params)
      @mailer_class.new(send_params).dispatch_and_deliver @template.to_sym, mail_params
      @mail = Merb::Mailer.deliveries.last
    end

    instance_eval &block
  end
end

# spec/mailers/user\_mailer\_spec.rb
require File.join(File.dirname(__FILE__),'..','spec_helper')

describe_mail UserMailer, :hello do
  it "should say hello" do
    deliver :name => "Jamie"
    @mail.text.should == "Hello Jamie"
  end
end

I’m a big fan of custom rspec describers, as above. The fact that before and after blocks are transparently inherited is a huge win over test/unit, where you’d need to explicitly call super.

# app/mailers/user_mailer.rb
class UserMailer < Merb::MailController
  def hello
    render_mail
  end  
end

# app/mailers/views/user_mailer/hello.text.erb
Hello <%= params[:name] %>

So, DataMapper 0.3 was behaving weirdly for me, and I thought I’d try upgrading to 0.9 to see how things are there. Overall I’m quite liking it, but there’s a few catches:

It’s incompatible with Vlad for deployment, haven’t looked into it but something in DM is making the ‘repository’ value unsettable, which Vlad uses to determine the checkout path.
Legacy connections beware, there doesn’t seem to be a current alternative for set_table_name at the moment.
:memory: is no longer a good name for your test database when using sqlite. Use a fully-qualified connection string like sqlite://:memory: instead.
DataMapperPersistence has been renamed DataMapperResource. Include it in models.
Validations are an add-on now, include DataMapper::Validate in your model (or even re-open DM:Resource and include it there).
Properties now take a class instead of a symbol (you can guess at all the main ones), and require the id to be specified like so: property :id, Fixnum, :serial => true
Associations are renamed, has_one is now one_to_one, has_many is one_to_many or many_to_many, etc. Haven’t delved in deep yet though, so I’m not sure how to define who gets the foreign key.
New migration code is just now getting in to dm-core, and auto_migrate! is a thing of the past. Sucks for those of us using sqlite in-memory test databases that need a fresh migration every time.

My installation Rakefile follows, just stuff it in an empty directory and it’ll do everything from there. Much thanks to Atmos for a good starting point and some setup help.

desc "Fetch and Install DM and Merb"
task :install_all do 
  config = CONFIG['dm']
  fetch config[:user], config[:repos]
  install config[:install]
  config = CONFIG['merb']
  fetch config[:user], config[:repos]
  install config[:install]
end

desc "Uninstall DM and Merb"
task :uninstall_all do
  uninstall CONFIG['dm'][:gems]  
  uninstall CONFIG['merb'][:gems]  
end

desc "Download latest sources for :project from git"
task :fetch, :project do |task, args|
  config = CONFIG[args[:project]]
  fetch config[:user], config[:repos]
end

desc "Install :project from git"
task :install, :project do |task, args|
  config = CONFIG[args[:project]]
  install config[:install]
end

desc "Uninstall :project"
task :uninstall, :project do |task, args|
  config = CONFIG[args[:project]]
  uninstall config[:gems]
end

def fetch(user, repos)
  base = File.expand_path(".")
  Dir.chdir base do
    repos.each do |repo|
      repo_dir = "#{base}/#{repo}"
      unless File.directory?(repo_dir)
        %x{git clone git://github.com/#{user}/#{repo}.git }
      end
      Dir.chdir(repo_dir) { %x{git pull} }
    end
  end
end

def install(modules)
  base = File.expand_path(".")
  modules.each do |lib|
    Dir.chdir("#{base}/#{lib}") do
      cmd = "sudo rake install 2>/dev/null |" +
            " grep -v '^\(in' |" +
            " grep -v '^[0-9] gem' |" +
            " grep -v '^[IUc\. ]'"
      puts %x{#{cmd}}
    end
  end
end

def uninstall(gems)
  gems.each do |name|
    puts %x{yes | sudo gem uninstall #{name} -aI}
  end
end

CONFIG = {
  'dm' => {
    :user => 'sam',
    :repos => %w(
      do
      dm-core
      dm-more),
    :install => %w(
      do/data_objects
      do/do_sqlite3
      do/do_mysql
      do/do_postgres
      dm-core
      dm-more/merb_datamapper
      dm-more/dm-migrations
      dm-more/dm-serializer
      dm-more/dm-validations
    ),
    :gems => %w(
      data_objects
      do_sqlite3
      do_mysql
      do_postgres
      dm-core
      merb_datamapper
      dm-migrations
      dm-serializer
      dm-validations
    )
  },
  'merb' => {
    :user => 'wycats',
    :repos => %w(
      merb-core
      merb-more
      merb-plugins
      merb-plugins/merb_param_protection),
    :install => %w(
      merb-core
      merb-more
      merb-plugins
    ),
    :gems => %w(
      merb
      merb-action-args
      merb-assets
      merb-builder
      merb-cache
      merb-core
      merb-gen
      merb-haml
      merb-mailer
      merb-more
      merb-parts
      merb_activerecord
      merb_datamapper
      merb_helpers
      merb_param_protection
      merb_rspec
      merb_sequel
      merb_stories
      merb_test_unit
    )
  }
}

</th> <th><a name=”merbHoptoad has been bugging us for a week or two now to upgrade to v2 of its API, so we did that this week for our project at work. Except we’re running Merb, not Rails.

Then, to actually use it, just add the following in the right places:

# init.rb, in Merb::Bootloader.after_app_loads

HoptoadNotifier.configure do |config|
  config.api_key = '...'
  config.environment_name = Merb.env
  config.project_root = Merb.root
  # See http://github.com/thoughtbot/hoptoad_notifier/ README.rdoc
  # for other config settings. You probably want to think about
  # params_filters and maybe ignore.
end

# application.rb, if you want available manually.
# If you just want it for completely unexpected errors you can stick it in
# exceptions.rb instead

def notify_hoptoad(error=nil)
  error ||= request.exceptions.first

  data = {
    :controller       => params[:controller],
    :action           => params[:action],
    :url              => "#{request.protocol}://#{request.host}#{request.uri}",
    # Looks like hoptoad is filtering these itself, we don't need to worry about it
    # other than configuring what needs to be filtered
    :parameters       => params.to_hash,
    :session_data     => session.to_hash,
    :cgi_data         => request.env.to_hash,
    :environment_vars => ENV.to_hash.merge(:RAILS_ENV => Merb.env)
  }

  HoptoadNotifier.notify(error, data)
end

# exceptions.rb, override default error handlers to submit to hoptoad

def internal_server_error
  notify_hoptoad
  render
end

def standard_error
  notify_hoptoad
  render
end    

# other spots you handle exceptions inline, just pass in the exception

begin
  ...
rescue => e
  notify_hoptoad(e)
end

First, install the amqp ruby library to connect to rabbit, and then add a tiny bit of setup.

In config/environment.rb:

require 'mq'
class BeanstalkPoolImpersonator
  def initialize(opts={})
    @opts = opts
  end

  def connect
    connection = AMQP.connect(@opts)
    @channel = channel = MQ.new(connection)
  end

  def use(queue)
    @queue = MQ::Queue.new(@channel, queue)
  end

  def yput(obj, pri, delay, ttr)
    p [obj, pri, delay, ttr]
    @queue.publish(YAML.dump(obj))
  end

  def last_server
    :last_server_stub
  end

  def subscribe(*args, &blk)
    @queue.subscribe(*args, &blk)
  end
end

Then, instead of connecting via Beanstalk::Pool.new, do this:

AsyncObserver::Queue.queue = BeanstalkPoolImpersonator.new()

You can pass an options hash to the new call, providing user, pass, vhost, host, or port as necessary.

Then, in your workers, load up the async_observer worker class, and extend like so:

class RabbitWorker < AsyncObserver::Worker
  def run()
    EM.run do
      AsyncObserver::Queue.queue.connect
      AsyncObserver::Queue.queue.use('1.0')
      AsyncObserver::Queue.queue.subscribe do |headers, msg|
        job = OpenStruct.new(:ybody => YAML.load(msg), :body => msg, :stats => [])
        job.id = headers.properties[:delivery_tag]
        safe_dispatch(job)
      end
    end
  end
end

Create the new worker the same way you would for the AO::Worker, and you’re set:

    RabbitWorker.new(binding).run()

Note: I’m maintaining a merb port of async-observer on github.

Create your spec in spec/views/layout/application.html.erb_spec.html, and add this class to it:

class Layout < Application
  layout nil
  def application; render; end
end

Then just test it as you would any other ordinary view.

The good news is that both of the projects are available on GitHub, which means easy forks, and easy contributions back to the source if my changes are as good as I think they are.

Basics

If you plan on dealing with the app directory structure or environment, an easy way to do it is:

if defined?(Rails)
  ROOT = RAILS_ROOT
  ENV = RAILS_ENV
elsif defined?(Merb)
  ROOT = Merb.root
  ENV = (Merb.env == 'rake' ? 'development' : Merb.env)
end

The additional rake environment transparently proxies to the development db connection, so if you just want to compare your plugin’s interpretation of ENV the above will make that cleaner.

Rake Tasks

Rails automatically loads any files matching tasks/*.rake in the plugin dir.

Merb needs to be told explicitly, relative to the lib directory. The canonical example from the docs is:

if defined?(Merb::Plugins)
  Merb::Plugins.add_rakefiles "merb_sequel" / "merbtasks"
end

    load File.expand_path(File.join(File.dirname(__FILE__), '..', '..', 'tasks', 'merb_sequel.rake'))

Do note that the load is necessary, require doesn’t pick the file up properly.

Finally, if you have any tasks that depend on environment, the easiest way to get compatability with both frameworks is to add task :environment => :merb_env to your merbtasks.rb file.

Generators

ORM Integration

As a follow-up to my previous post, here’s some gotchas to be aware of if you’re looking to support both ActiveRecord and DataMapper in a Merb (and/or Rails) plugin.

The Strategy

Basic Translation

Now that we have a plan, we can start translating our extracted functions from AR bits to DM bits. There’s a bunch of fairly straightforward transformations we can make.

ActiveRecord	DataMapper
.find(:all, ...)	.all(...)
.find(:first, ...)	.first(...)
.find(id)	.get(...)
.find_all_by_id(id)	.all(:id => ids)
.table_name	.storage_name
.primary_key	.key.first.name
.connection	repository.adapter

Raw SQL

Hooks

In before hooks, the AR hook chain stops if your method returns false, in DM you must throw :halt.

Things You Shouldn’t Be Doing Anyways

If you’re manually setting @attribute values in your AR code, you’ll need to use instance_variable_set for DataMapper. I recommend writing manual accessor methods to wrap it for abstraction.

Daniel Manges did up instructions on storing explicit versions of gems in your rails app. If instead you’re using merb, you probably want to do that too, as it makes for much easier deploys.

Thankfully, it’s much less pain in merb:

gem install async-observer -i gems

The -i option tells rubygems to install the gems to that directory, and when you require the gem from inside merb, it will look in the local gems directory first, and find yours.

Optionally, you can skip generating docs by adding --no-rdoc --no-ri to the line, and depending on what you’re installing, you may want to --ignore-dependencies as well.

If the gem in question builds a binary extension, you may be out of luck if you try to deploy it to a different architecture.

The only problem I’ve found so far is that gem cleanup won’t accept a directory to clean, so you’ll need to manually remove individual gems when you upgrade:

gem uninstall -i async-observer

This helper is based on some other controller/view helpers I’ve been working on and planning on blogging soon, with a nod to the specs present in the merb-mailer library itself.

# in a controller
send_mail UserMailer, :hello, {
  :from => "greeter@example.com",
  :to => @person.email,
  :subject => "Greetings"
}, {
  :name => @person.name
}

The controller spec can simply stub/mock the send_mail call as appropriate.

# spec/spec_helper.rb
Merb::Mailer.delivery_method = :test_send
def describe_mail(mailer, template, &block)
  describe "/#{mailer.to_s.downcase}/#{template}" do
    before :each do
      @mailer_class, @template = mailer, template
      @assigns = {}
    end

    def deliver(send_params={}, mail_params={})
      mail_params = {:from => "from@example.com", :to => "to@example.com", :subject => "Subject Line"}.merge(mail_params)
      @mailer_class.new(send_params).dispatch_and_deliver @template.to_sym, mail_params
      @mail = Merb::Mailer.deliveries.last
    end

    instance_eval &block
  end
end

# spec/mailers/user\_mailer\_spec.rb
require File.join(File.dirname(__FILE__),'..','spec_helper')

describe_mail UserMailer, :hello do
  it "should say hello" do
    deliver :name => "Jamie"
    @mail.text.should == "Hello Jamie"
  end
end

I’m a big fan of custom rspec describers, as above. The fact that before and after blocks are transparently inherited is a huge win over test/unit, where you’d need to explicitly call super.

# app/mailers/user_mailer.rb
class UserMailer < Merb::MailController
  def hello
    render_mail
  end  
end

# app/mailers/views/user_mailer/hello.text.erb
Hello <%= params[:name] %>

So, DataMapper 0.3 was behaving weirdly for me, and I thought I’d try upgrading to 0.9 to see how things are there. Overall I’m quite liking it, but there’s a few catches:

It’s incompatible with Vlad for deployment, haven’t looked into it but something in DM is making the ‘repository’ value unsettable, which Vlad uses to determine the checkout path.
Legacy connections beware, there doesn’t seem to be a current alternative for set_table_name at the moment.
:memory: is no longer a good name for your test database when using sqlite. Use a fully-qualified connection string like sqlite://:memory: instead.
DataMapperPersistence has been renamed DataMapperResource. Include it in models.
Validations are an add-on now, include DataMapper::Validate in your model (or even re-open DM:Resource and include it there).
Properties now take a class instead of a symbol (you can guess at all the main ones), and require the id to be specified like so: property :id, Fixnum, :serial => true
Associations are renamed, has_one is now one_to_one, has_many is one_to_many or many_to_many, etc. Haven’t delved in deep yet though, so I’m not sure how to define who gets the foreign key.
New migration code is just now getting in to dm-core, and auto_migrate! is a thing of the past. Sucks for those of us using sqlite in-memory test databases that need a fresh migration every time.

My installation Rakefile follows, just stuff it in an empty directory and it’ll do everything from there. Much thanks to Atmos for a good starting point and some setup help.

desc "Fetch and Install DM and Merb"
task :install_all do 
  config = CONFIG['dm']
  fetch config[:user], config[:repos]
  install config[:install]
  config = CONFIG['merb']
  fetch config[:user], config[:repos]
  install config[:install]
end

desc "Uninstall DM and Merb"
task :uninstall_all do
  uninstall CONFIG['dm'][:gems]  
  uninstall CONFIG['merb'][:gems]  
end

desc "Download latest sources for :project from git"
task :fetch, :project do |task, args|
  config = CONFIG[args[:project]]
  fetch config[:user], config[:repos]
end

desc "Install :project from git"
task :install, :project do |task, args|
  config = CONFIG[args[:project]]
  install config[:install]
end

desc "Uninstall :project"
task :uninstall, :project do |task, args|
  config = CONFIG[args[:project]]
  uninstall config[:gems]
end

def fetch(user, repos)
  base = File.expand_path(".")
  Dir.chdir base do
    repos.each do |repo|
      repo_dir = "#{base}/#{repo}"
      unless File.directory?(repo_dir)
        %x{git clone git://github.com/#{user}/#{repo}.git }
      end
      Dir.chdir(repo_dir) { %x{git pull} }
    end
  end
end

def install(modules)
  base = File.expand_path(".")
  modules.each do |lib|
    Dir.chdir("#{base}/#{lib}") do
      cmd = "sudo rake install 2>/dev/null |" +
            " grep -v '^\(in' |" +
            " grep -v '^[0-9] gem' |" +
            " grep -v '^[IUc\. ]'"
      puts %x{#{cmd}}
    end
  end
end

def uninstall(gems)
  gems.each do |name|
    puts %x{yes | sudo gem uninstall #{name} -aI}
  end
end

CONFIG = {
  'dm' => {
    :user => 'sam',
    :repos => %w(
      do
      dm-core
      dm-more),
    :install => %w(
      do/data_objects
      do/do_sqlite3
      do/do_mysql
      do/do_postgres
      dm-core
      dm-more/merb_datamapper
      dm-more/dm-migrations
      dm-more/dm-serializer
      dm-more/dm-validations
    ),
    :gems => %w(
      data_objects
      do_sqlite3
      do_mysql
      do_postgres
      dm-core
      merb_datamapper
      dm-migrations
      dm-serializer
      dm-validations
    )
  },
  'merb' => {
    :user => 'wycats',
    :repos => %w(
      merb-core
      merb-more
      merb-plugins
      merb-plugins/merb_param_protection),
    :install => %w(
      merb-core
      merb-more
      merb-plugins
    ),
    :gems => %w(
      merb
      merb-action-args
      merb-assets
      merb-builder
      merb-cache
      merb-core
      merb-gen
      merb-haml
      merb-mailer
      merb-more
      merb-parts
      merb_activerecord
      merb_datamapper
      merb_helpers
      merb_param_protection
      merb_rspec
      merb_sequel
      merb_stories
      merb_test_unit
    )
  }
}

” class=”anchor”> </th></tr>

  <tr><th>linuxI picked up a [new machine][] this week to turn into a home media server, and had a small hoop to jump through with the Broadcom [wifi card][] I picked up. Problem being, office and all my monitors were upstairs, wifi was downstairs, and it's difficult to install new drivers without net access.

But, Linux Mint popped up a “new hardware” dialog, saying it wanted to use b43-fwripper to set up the drivers. Good news, it gave me the url it failed to download from. Manually download the deb, open it up on my other machine using Archive Manager (rather than GDebi), and navigate through to the postinstall script, at data.tar.gz/./usr/share/b43-fwcutter/, and I find a reference to two other files.

After downloading those, copy all three files to the new box. Copy the .deb to /var/cache/apt/archives, and then apt-get install b43-fwripper. Ignore the failure, then go to the dir with the other two files, and manually run the 3 useful lines from the install script (don’t forget to add sudo to the last one):

b43-fwcutter -w /lib/firmware wl_apsta-3.130.20.0.o
tar xfvj broadcom-wl-4.150.10.5.tar.bz2
b43-fwcutter --unsupported -w /lib/firmware broadcom-wl-4.150.10.5/driver/wl_apsta_mimo.o

Reboot, and voila - local wifi networks show up in the tray applet. I presume these steps will work for an Ubuntu install as well, since Mint is based on Ubuntu, but I haven’t tried it as yet - in fact, the required .deb is on the Ubuntu 9.04 livecd. I also don’t know if these files will work for other Broadcom wireless adapters, but if your modern linux system can do a hardware detect and tell you what driver it’s trying to pull down, that should be a good starting point. I’m running Ubuntu 6.10 on my laptop, and a few weeks back the sound decided that it would just stop working. I finally got fed up enough with it to do a reinstall. I thought I’d try out the new beta for Feisty, but the installer repeatedly froze on me at 63% on the install, so I gave up and put 6.10 back on. I try to keep tabs on what software I install as extras to ease reinstalls, and tweaked my reinstall scripts from last time to the final forms shown below.

After all this, I did manage to get sound up and running, and it even properly mixes now between Quod Libet, Gaim, and Firefox/flash. So, total win. Also I got to remove some cruft from my install, and I should be able to get up and running quick when I upgrade to 7.04. No in-place upgrade this time.

This first script takes a custom sources.list (just edited to enable universe/multiverse from all sources, and add a few extras for jedit and some others) and copies it into place, and then installs a new kernel, shell, network manager, and graphics driver. It sets up the networking for WPA (which I use at home). Lastly, it installs a special driver for my video hardware, with an init-script to let my LCD run at native resolution. Fun times I had figuring that one out.

#!/bin/sh

#sudo cp /etc/apt/sources.list /etc/apt/sources.list.old
#sudo cp sources.list /etc/apt/sources.list
#sudo apt-get update

# Environment
sudo apt-get install \
  build-essential zsh zsh-doc deborphan \
  linux-image-686 linux-restricted-modules-686 \
  nvidia-glx network-manager network-manager-gnome yakuake

# Get WPA networking set up
sudo echo auto lo > /etc/network/interfaces
sudo echo iface lo inet loopback >> /etc/network/interfaces
sudo echo ENABLED=0 > /etc/default/wpasupplicant
sudo /etc/init.d/dbus restart

# Fix screen res
cd 855resolution
make clean
make
sudo cp 855resolution /usr/sbin/855resolution
sudo cp 855res /etc/init.d/855res
sudo chmod 755 /etc/init.d/855res
sudo update-rc.d 855res defaults 19
cd ..

# Set shell
echo Enter your password to set zsh as your default shell
chsh -s /usr/bin/zsh

echo "\n\n\n\n\n\n"
echo Please reboot your system now to load the new kernel
echo and get X running at the correct resolution.
echo

After rebooting for a real X (and most of my working environment set up), I run the second script, which is essentially just one big batch-install. Version control, database, Java, Ruby, RubyGems (manually), a bunch of a/v codecs for gstreamer, and then a handful of actual applications.

#!/bin/sh

# Development
sudo apt-get install \
  subversion svk darcs mercurial \
  mysql-client mysql-server sqlite3 sqlite3-doc sun-java5-jre \
    sun-java5-plugin ia32-sun-java5-plugin sun-java5-fonts \
    ttf-sazanami-gothic ttf-sazanami-mincho \
  ruby1.8 ruby1.8-dev rdoc1.8 ri1.8 irb1.8 libyaml-ruby \
  libzlib-ruby libmysql-ruby librmagick-ruby libgd-ruby1.8

# Rubygems
wget http://rubyforge.org/frs/download.php/5207/rubygems-0.8.11.tgz
tar xzvf rubygems-0.8.11.tgz
cd rubygems-0.8.11
sudo ruby setup.rb
cd ..
rm -rf rubygems-0.8.11
rm rubygems-0.8.11.tgz
sudo gem update --system
sudo gem install rails camping nitro

# Multimedia/Entertainment
#  lame libdvdcss2
sudo apt-get install \
  alsa-oss vorbis-tools gstreamer0.10-ffmpeg \
  gstreamer0.10-plugins-bad gstreamer0.10-plugins-bad-multiverse \
  gstreamer0.10-plugins-ugly gstreamer0.10-plugins-ugly-multiverse \
  flashplayer-mozilla unrar unace p7zip \
  msttcorefonts gsfonts-x11 xfonts-intl-european
sudo fc-cache -f -v # reload font cache

# Apps
sudo apt-get install \
  quodlibet quodlibet-plugins quodlibet-ext \
  gnugo quarry wine jedit gnucash gnucash-docs

echo "\n\n\n\n\n\n"
echo Please edit /etc/firefox/firefoxrc and change \"none\" to \"aoss\"
echo

Comments

Wow, way more complicated than my installation process: http://dev.technomancy.us/phil/wiki/UbuntuInstallation

The most annoying part for me is grabbing all those firefox extensions since it’s not easy to script. I’ve got fiesty herd 3 on my laptop, and it’s working fine–better suspend than I got with edgy. But it’s always hit-or-miss with these prereleases.

Phil Hagelberg, at 10:51, Feb 20 2007

</th> <th><a name=”linuxI picked up a new machine this week to turn into a home media server, and had a small hoop to jump through with the Broadcom wifi card I picked up. Problem being, office and all my monitors were upstairs, wifi was downstairs, and it’s difficult to install new drivers without net access.

b43-fwcutter -w /lib/firmware wl_apsta-3.130.20.0.o
tar xfvj broadcom-wl-4.150.10.5.tar.bz2
b43-fwcutter --unsupported -w /lib/firmware broadcom-wl-4.150.10.5/driver/wl_apsta_mimo.o

#!/bin/sh

#sudo cp /etc/apt/sources.list /etc/apt/sources.list.old
#sudo cp sources.list /etc/apt/sources.list
#sudo apt-get update

# Environment
sudo apt-get install \
  build-essential zsh zsh-doc deborphan \
  linux-image-686 linux-restricted-modules-686 \
  nvidia-glx network-manager network-manager-gnome yakuake

# Get WPA networking set up
sudo echo auto lo > /etc/network/interfaces
sudo echo iface lo inet loopback >> /etc/network/interfaces
sudo echo ENABLED=0 > /etc/default/wpasupplicant
sudo /etc/init.d/dbus restart

# Fix screen res
cd 855resolution
make clean
make
sudo cp 855resolution /usr/sbin/855resolution
sudo cp 855res /etc/init.d/855res
sudo chmod 755 /etc/init.d/855res
sudo update-rc.d 855res defaults 19
cd ..

# Set shell
echo Enter your password to set zsh as your default shell
chsh -s /usr/bin/zsh

echo "\n\n\n\n\n\n"
echo Please reboot your system now to load the new kernel
echo and get X running at the correct resolution.
echo

#!/bin/sh

# Development
sudo apt-get install \
  subversion svk darcs mercurial \
  mysql-client mysql-server sqlite3 sqlite3-doc sun-java5-jre \
    sun-java5-plugin ia32-sun-java5-plugin sun-java5-fonts \
    ttf-sazanami-gothic ttf-sazanami-mincho \
  ruby1.8 ruby1.8-dev rdoc1.8 ri1.8 irb1.8 libyaml-ruby \
  libzlib-ruby libmysql-ruby librmagick-ruby libgd-ruby1.8

# Rubygems
wget http://rubyforge.org/frs/download.php/5207/rubygems-0.8.11.tgz
tar xzvf rubygems-0.8.11.tgz
cd rubygems-0.8.11
sudo ruby setup.rb
cd ..
rm -rf rubygems-0.8.11
rm rubygems-0.8.11.tgz
sudo gem update --system
sudo gem install rails camping nitro

# Multimedia/Entertainment
#  lame libdvdcss2
sudo apt-get install \
  alsa-oss vorbis-tools gstreamer0.10-ffmpeg \
  gstreamer0.10-plugins-bad gstreamer0.10-plugins-bad-multiverse \
  gstreamer0.10-plugins-ugly gstreamer0.10-plugins-ugly-multiverse \
  flashplayer-mozilla unrar unace p7zip \
  msttcorefonts gsfonts-x11 xfonts-intl-european
sudo fc-cache -f -v # reload font cache

# Apps
sudo apt-get install \
  quodlibet quodlibet-plugins quodlibet-ext \
  gnugo quarry wine jedit gnucash gnucash-docs

echo "\n\n\n\n\n\n"
echo Please edit /etc/firefox/firefoxrc and change \"none\" to \"aoss\"
echo

Comments

Wow, way more complicated than my installation process: http://dev.technomancy.us/phil/wiki/UbuntuInstallation

Phil Hagelberg, at 10:51, Feb 20 2007

” class=”anchor”> </th></tr>

  <tr><th>osxI've been trying to figure out a minimal way of getting the geoip_city gem installed on my mac, and having [seen][] a few [ideas][] on google, I'm gonna consolidate into a minimal solution.  I originally wanted to work with the macports version of libgeoip, but had absolutely no luck with that, so here's something a bit more manual:

# we need to be root for the ARCHFLAGS later to stick
sudo su

mkdir -p /usr/local/src
cd /usr/local/src

# Download and install latest geoip
curl -O http://geolite.maxmind.com/download/geoip/api/c/GeoIP-1.4.5.tar.gz
tar -xzf GeoIP-1.4.5.tar.gz
cd GeoIP-1.4.5
./configure
make
make check
make install

# install the gem
export ARCHFLAGS='-arch i386'
gem install geoip_city

The main sticking point for me was that setting ARCHFLAGS doesn’t continue through to a sudo gem install, so we need to do the whole thing as root.

But, now that we’ve done that, we can grab the GeoLiteCity database and point our app at it:

curl -O http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
gunzip GeoLiteCity.dat.gz

As a result, while gem list tells me that I’ve got activerecord v 2.2.2 and 1.15.6 installed, gem cleanup fails miserably:

jamie@juliet ~> sudo gem cleanup
Cleaning up installed gems...
Attempting to uninstall activerecord-1.15.6
ERROR:  While executing gem ... (Gem::InstallError)
    Unknown gem activerecord = 1.15.6

gempath:
  - /Library/Ruby/Gems/1.8
  - /Users/jamie/.gem/ruby/1.8

Of course, you’ll need to use your own username on the last line there, and you might want to double-check the existing GEM PATH values from gem env output so you don’t accidentally clobber anything else. Also, if you’ve got a previously-generated .gemrc, the gempath key needs to be a string, not a symbol, or else it won’t get picked up. Whee.</th> <th><a name=”osxI’ve been trying to figure out a minimal way of getting the geoip_city gem installed on my mac, and having seen a few ideas on google, I’m gonna consolidate into a minimal solution. I originally wanted to work with the macports version of libgeoip, but had absolutely no luck with that, so here’s something a bit more manual:

# we need to be root for the ARCHFLAGS later to stick
sudo su

mkdir -p /usr/local/src
cd /usr/local/src

# Download and install latest geoip
curl -O http://geolite.maxmind.com/download/geoip/api/c/GeoIP-1.4.5.tar.gz
tar -xzf GeoIP-1.4.5.tar.gz
cd GeoIP-1.4.5
./configure
make
make check
make install

# install the gem
export ARCHFLAGS='-arch i386'
gem install geoip_city

The main sticking point for me was that setting ARCHFLAGS doesn’t continue through to a sudo gem install, so we need to do the whole thing as root.

But, now that we’ve done that, we can grab the GeoLiteCity database and point our app at it:

curl -O http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
gunzip GeoLiteCity.dat.gz

As a result, while gem list tells me that I’ve got activerecord v 2.2.2 and 1.15.6 installed, gem cleanup fails miserably:

jamie@juliet ~> sudo gem cleanup
Cleaning up installed gems...
Attempting to uninstall activerecord-1.15.6
ERROR:  While executing gem ... (Gem::InstallError)
    Unknown gem activerecord = 1.15.6

gempath:
  - /Library/Ruby/Gems/1.8
  - /Users/jamie/.gem/ruby/1.8

  <tr><th>clojureSo, some [Hacker News][] users recently [decided][] that it'd be a good idea to [work through][] the [SICP][] together and make a kind of book club out of the deal over irc.

When I heard of it, I thought it’d be a good excuse to work through the text myself (since it was just sitting on my bookshelf), and also stretch my brain a bit more teaching myself Clojure. I’m almost done the reading and exercises for section 1.1, and I’m already finding it a valuable exercise.

The third exercise asks us to “define a procedure that takes three numbers as arguments and returns the sum of the squares of the two larger numbers.” At either extremity are two solutions. The straightforward solution a new programmer can generate based solely on the first 20 pages of the text looks something like this:

(defn sum-largest-squares [a b c]
  (cond (and (< a b) (< a c)) (+ (* b b) (* c c))
        (and (< b a) (< b c)) (+ (* a a) (* c c))
        (and (< c a) (< c b)) (+ (* a a) (* b b))))

Some tests to prove correctness, returning 2^2 + 3^2, or 13, are:

(sum-largest-squares 1 2 3)
(sum-largest-squares 2 3 1)
(sum-largest-squares 3 1 2)

An interesting (and enlightening) exercise is to take our naive function, and just start randomly refactoring, and see where it takes us. One step at a time, making sure we are always passing our handful of test cases. The goal is to decompose and abstract the function into more manageable pieces, with less repetition. To start:

(defn sum-largest-squares [a b c]
  (cond (and (< a b) (< a c)) (+ (square b) (square c))
        (and (< b a) (< b c)) (+ (square a) (square c))
        (and (< c a) (< c b)) (+ (square a) (square b))))

The square function is trivially pulling out (* x x), and conveniently is already present in Clojure.

Second step is to pull out the three conditional clauses into sum-of-squares:

(defn sum-of-squares [a b]
  (+ (square a) (square b)))

(defn sum-largest-squares [a b c]
  (cond (and (< a b) (< a c)) (sum-of-squares b c)
        (and (< b a) (< b c)) (sum-of-squares a c)
        (and (< c a) (< c b)) (sum-of-squares a b)))

Next, we can probably abstract the conditionals themselves, since what we’re looking for is the smallest of three elements (min is also provided by clojure):

(defn sum-largest-squares [a b c]
  (cond (= a (min a b c)) (sum-of-squares b c)
        (= b (min a b c)) (sum-of-squares a c)
        (= c (min a b c)) (sum-of-squares a b)))

Now, we’ve got a much more parallel structure here, so we should be able to collapse the conditional, and process the values first.

I’ll stop here, but there are a few different ways to get started with the next step - most involve transitioning our three arguments into a list at some point. I’ve found it instructive to play around with the different ways of decomposing the problem (including a few side steps along the way to the route I’ve followed above). I’m thinking that as I continue with the SICP exercies, I’ll try and make a habit of not stopping with the first solution that springs out of my text editor, but to try and explore the structure of the code through refactoring.

Also, while I never managed to figure out refactoring steps that would produce it, I think the most straightforward functional solution (rather than the semi-iterative solution we started with above) winds up being this - it has a certain elegance to it, no?

(defn sum-largest-squares [a b c]
  (apply + (map square (rest (sort (list a b c))))))

</th> <th><a name=”clojureSo, some Hacker News users recently decided that it’d be a good idea to work through the SICP together and make a kind of book club out of the deal over irc.

(defn sum-largest-squares [a b c]
  (cond (and (< a b) (< a c)) (+ (* b b) (* c c))
        (and (< b a) (< b c)) (+ (* a a) (* c c))
        (and (< c a) (< c b)) (+ (* a a) (* b b))))

Some tests to prove correctness, returning 2^2 + 3^2, or 13, are:

(sum-largest-squares 1 2 3)
(sum-largest-squares 2 3 1)
(sum-largest-squares 3 1 2)

(defn sum-largest-squares [a b c]
  (cond (and (< a b) (< a c)) (+ (square b) (square c))
        (and (< b a) (< b c)) (+ (square a) (square c))
        (and (< c a) (< c b)) (+ (square a) (square b))))

The square function is trivially pulling out (* x x), and conveniently is already present in Clojure.

Second step is to pull out the three conditional clauses into sum-of-squares:

(defn sum-of-squares [a b]
  (+ (square a) (square b)))

(defn sum-largest-squares [a b c]
  (cond (and (< a b) (< a c)) (sum-of-squares b c)
        (and (< b a) (< b c)) (sum-of-squares a c)
        (and (< c a) (< c b)) (sum-of-squares a b)))

Next, we can probably abstract the conditionals themselves, since what we’re looking for is the smallest of three elements (min is also provided by clojure):

(defn sum-largest-squares [a b c]
  (cond (= a (min a b c)) (sum-of-squares b c)
        (= b (min a b c)) (sum-of-squares a c)
        (= c (min a b c)) (sum-of-squares a b)))

Now, we’ve got a much more parallel structure here, so we should be able to collapse the conditional, and process the values first.

(defn sum-largest-squares [a b c]
  (apply + (map square (rest (sort (list a b c))))))

” class=”anchor”> </th></tr>

  <tr><th>sicpSo, some [Hacker News][] users recently [decided][] that it'd be a good idea to [work through][] the [SICP][] together and make a kind of book club out of the deal over irc.

(defn sum-largest-squares [a b c]
  (cond (and (< a b) (< a c)) (+ (* b b) (* c c))
        (and (< b a) (< b c)) (+ (* a a) (* c c))
        (and (< c a) (< c b)) (+ (* a a) (* b b))))

Some tests to prove correctness, returning 2^2 + 3^2, or 13, are:

(sum-largest-squares 1 2 3)
(sum-largest-squares 2 3 1)
(sum-largest-squares 3 1 2)

(defn sum-largest-squares [a b c]
  (cond (and (< a b) (< a c)) (+ (square b) (square c))
        (and (< b a) (< b c)) (+ (square a) (square c))
        (and (< c a) (< c b)) (+ (square a) (square b))))

The square function is trivially pulling out (* x x), and conveniently is already present in Clojure.

Second step is to pull out the three conditional clauses into sum-of-squares:

(defn sum-of-squares [a b]
  (+ (square a) (square b)))

(defn sum-largest-squares [a b c]
  (cond (and (< a b) (< a c)) (sum-of-squares b c)
        (and (< b a) (< b c)) (sum-of-squares a c)
        (and (< c a) (< c b)) (sum-of-squares a b)))

Next, we can probably abstract the conditionals themselves, since what we’re looking for is the smallest of three elements (min is also provided by clojure):

(defn sum-largest-squares [a b c]
  (cond (= a (min a b c)) (sum-of-squares b c)
        (= b (min a b c)) (sum-of-squares a c)
        (= c (min a b c)) (sum-of-squares a b)))

Now, we’ve got a much more parallel structure here, so we should be able to collapse the conditional, and process the values first.

(defn sum-largest-squares [a b c]
  (apply + (map square (rest (sort (list a b c))))))

</th> <th><a name=”sicpSo, some Hacker News users recently decided that it’d be a good idea to work through the SICP together and make a kind of book club out of the deal over irc.

(defn sum-largest-squares [a b c]
  (cond (and (< a b) (< a c)) (+ (* b b) (* c c))
        (and (< b a) (< b c)) (+ (* a a) (* c c))
        (and (< c a) (< c b)) (+ (* a a) (* b b))))

Some tests to prove correctness, returning 2^2 + 3^2, or 13, are:

(sum-largest-squares 1 2 3)
(sum-largest-squares 2 3 1)
(sum-largest-squares 3 1 2)

(defn sum-largest-squares [a b c]
  (cond (and (< a b) (< a c)) (+ (square b) (square c))
        (and (< b a) (< b c)) (+ (square a) (square c))
        (and (< c a) (< c b)) (+ (square a) (square b))))

The square function is trivially pulling out (* x x), and conveniently is already present in Clojure.

Second step is to pull out the three conditional clauses into sum-of-squares:

(defn sum-of-squares [a b]
  (+ (square a) (square b)))

(defn sum-largest-squares [a b c]
  (cond (and (< a b) (< a c)) (sum-of-squares b c)
        (and (< b a) (< b c)) (sum-of-squares a c)
        (and (< c a) (< c b)) (sum-of-squares a b)))

Next, we can probably abstract the conditionals themselves, since what we’re looking for is the smallest of three elements (min is also provided by clojure):

(defn sum-largest-squares [a b c]
  (cond (= a (min a b c)) (sum-of-squares b c)
        (= b (min a b c)) (sum-of-squares a c)
        (= c (min a b c)) (sum-of-squares a b)))

Now, we’ve got a much more parallel structure here, so we should be able to collapse the conditional, and process the values first.

(defn sum-largest-squares [a b c]
  (apply + (map square (rest (sort (list a b c))))))

” class=”anchor”> </th></tr>

  <tr><th>specsJust a quick little snippet for those trying to write specs for their layout.

Create your spec in spec/views/layout/application.html.erb_spec.html, and add this class to it:

class Layout < Application
  layout nil
  def application; render; end
end

Then just test it as you would any other ordinary view.

This helper is based on some other controller/view helpers I’ve been working on and planning on blogging soon, with a nod to the specs present in the merb-mailer library itself.

# in a controller
send_mail UserMailer, :hello, {
  :from => "greeter@example.com",
  :to => @person.email,
  :subject => "Greetings"
}, {
  :name => @person.name
}

The controller spec can simply stub/mock the send_mail call as appropriate.

# spec/spec_helper.rb
Merb::Mailer.delivery_method = :test_send
def describe_mail(mailer, template, &block)
  describe "/#{mailer.to_s.downcase}/#{template}" do
    before :each do
      @mailer_class, @template = mailer, template
      @assigns = {}
    end

    def deliver(send_params={}, mail_params={})
      mail_params = {:from => "from@example.com", :to => "to@example.com", :subject => "Subject Line"}.merge(mail_params)
      @mailer_class.new(send_params).dispatch_and_deliver @template.to_sym, mail_params
      @mail = Merb::Mailer.deliveries.last
    end

    instance_eval &block
  end
end

# spec/mailers/user\_mailer\_spec.rb
require File.join(File.dirname(__FILE__),'..','spec_helper')

describe_mail UserMailer, :hello do
  it "should say hello" do
    deliver :name => "Jamie"
    @mail.text.should == "Hello Jamie"
  end
end

I’m a big fan of custom rspec describers, as above. The fact that before and after blocks are transparently inherited is a huge win over test/unit, where you’d need to explicitly call super.

# app/mailers/user_mailer.rb
class UserMailer < Merb::MailController
  def hello
    render_mail
  end  
end

# app/mailers/views/user_mailer/hello.text.erb
Hello <%= params[:name] %>

</th> <th><a name=”specsJust a quick little snippet for those trying to write specs for their layout.

Create your spec in spec/views/layout/application.html.erb_spec.html, and add this class to it:

class Layout < Application
  layout nil
  def application; render; end
end

Then just test it as you would any other ordinary view.

This helper is based on some other controller/view helpers I’ve been working on and planning on blogging soon, with a nod to the specs present in the merb-mailer library itself.

# in a controller
send_mail UserMailer, :hello, {
  :from => "greeter@example.com",
  :to => @person.email,
  :subject => "Greetings"
}, {
  :name => @person.name
}

The controller spec can simply stub/mock the send_mail call as appropriate.

# spec/spec_helper.rb
Merb::Mailer.delivery_method = :test_send
def describe_mail(mailer, template, &block)
  describe "/#{mailer.to_s.downcase}/#{template}" do
    before :each do
      @mailer_class, @template = mailer, template
      @assigns = {}
    end

    def deliver(send_params={}, mail_params={})
      mail_params = {:from => "from@example.com", :to => "to@example.com", :subject => "Subject Line"}.merge(mail_params)
      @mailer_class.new(send_params).dispatch_and_deliver @template.to_sym, mail_params
      @mail = Merb::Mailer.deliveries.last
    end

    instance_eval &block
  end
end

# spec/mailers/user\_mailer\_spec.rb
require File.join(File.dirname(__FILE__),'..','spec_helper')

describe_mail UserMailer, :hello do
  it "should say hello" do
    deliver :name => "Jamie"
    @mail.text.should == "Hello Jamie"
  end
end

I’m a big fan of custom rspec describers, as above. The fact that before and after blocks are transparently inherited is a huge win over test/unit, where you’d need to explicitly call super.

# app/mailers/user_mailer.rb
class UserMailer < Merb::MailController
  def hello
    render_mail
  end  
end

# app/mailers/views/user_mailer/hello.text.erb
Hello <%= params[:name] %>

” class=”anchor”> </th></tr>

  <tr><th>datamapperAs a follow-up to my [previous post][], here's some gotchas to be aware of if you're looking to support both ActiveRecord and DataMapper in a Merb (and/or Rails) plugin.

The Strategy

Basic Translation

Now that we have a plan, we can start translating our extracted functions from AR bits to DM bits. There’s a bunch of fairly straightforward transformations we can make.

ActiveRecord	DataMapper
.find(:all, ...)	.all(...)
.find(:first, ...)	.first(...)
.find(id)	.get(...)
.find_all_by_id(id)	.all(:id => ids)
.table_name	.storage_name
.primary_key	.key.first.name
.connection	repository.adapter

Raw SQL

Hooks

In before hooks, the AR hook chain stops if your method returns false, in DM you must throw :halt.

Things You Shouldn’t Be Doing Anyways

If you’re manually setting @attribute values in your AR code, you’ll need to use instance_variable_set for DataMapper. I recommend writing manual accessor methods to wrap it for abstraction.

So, DataMapper 0.3 was behaving weirdly for me, and I thought I’d try upgrading to 0.9 to see how things are there. Overall I’m quite liking it, but there’s a few catches:

It’s incompatible with Vlad for deployment, haven’t looked into it but something in DM is making the ‘repository’ value unsettable, which Vlad uses to determine the checkout path.
Legacy connections beware, there doesn’t seem to be a current alternative for set_table_name at the moment.
:memory: is no longer a good name for your test database when using sqlite. Use a fully-qualified connection string like sqlite://:memory: instead.
DataMapperPersistence has been renamed DataMapperResource. Include it in models.
Validations are an add-on now, include DataMapper::Validate in your model (or even re-open DM:Resource and include it there).
Properties now take a class instead of a symbol (you can guess at all the main ones), and require the id to be specified like so: property :id, Fixnum, :serial => true
Associations are renamed, has_one is now one_to_one, has_many is one_to_many or many_to_many, etc. Haven’t delved in deep yet though, so I’m not sure how to define who gets the foreign key.
New migration code is just now getting in to dm-core, and auto_migrate! is a thing of the past. Sucks for those of us using sqlite in-memory test databases that need a fresh migration every time.

My installation Rakefile follows, just stuff it in an empty directory and it’ll do everything from there. Much thanks to Atmos for a good starting point and some setup help.

desc "Fetch and Install DM and Merb"
task :install_all do 
  config = CONFIG['dm']
  fetch config[:user], config[:repos]
  install config[:install]
  config = CONFIG['merb']
  fetch config[:user], config[:repos]
  install config[:install]
end

desc "Uninstall DM and Merb"
task :uninstall_all do
  uninstall CONFIG['dm'][:gems]  
  uninstall CONFIG['merb'][:gems]  
end

desc "Download latest sources for :project from git"
task :fetch, :project do |task, args|
  config = CONFIG[args[:project]]
  fetch config[:user], config[:repos]
end

desc "Install :project from git"
task :install, :project do |task, args|
  config = CONFIG[args[:project]]
  install config[:install]
end

desc "Uninstall :project"
task :uninstall, :project do |task, args|
  config = CONFIG[args[:project]]
  uninstall config[:gems]
end

def fetch(user, repos)
  base = File.expand_path(".")
  Dir.chdir base do
    repos.each do |repo|
      repo_dir = "#{base}/#{repo}"
      unless File.directory?(repo_dir)
        %x{git clone git://github.com/#{user}/#{repo}.git }
      end
      Dir.chdir(repo_dir) { %x{git pull} }
    end
  end
end

def install(modules)
  base = File.expand_path(".")
  modules.each do |lib|
    Dir.chdir("#{base}/#{lib}") do
      cmd = "sudo rake install 2>/dev/null |" +
            " grep -v '^\(in' |" +
            " grep -v '^[0-9] gem' |" +
            " grep -v '^[IUc\. ]'"
      puts %x{#{cmd}}
    end
  end
end

def uninstall(gems)
  gems.each do |name|
    puts %x{yes | sudo gem uninstall #{name} -aI}
  end
end

CONFIG = {
  'dm' => {
    :user => 'sam',
    :repos => %w(
      do
      dm-core
      dm-more),
    :install => %w(
      do/data_objects
      do/do_sqlite3
      do/do_mysql
      do/do_postgres
      dm-core
      dm-more/merb_datamapper
      dm-more/dm-migrations
      dm-more/dm-serializer
      dm-more/dm-validations
    ),
    :gems => %w(
      data_objects
      do_sqlite3
      do_mysql
      do_postgres
      dm-core
      merb_datamapper
      dm-migrations
      dm-serializer
      dm-validations
    )
  },
  'merb' => {
    :user => 'wycats',
    :repos => %w(
      merb-core
      merb-more
      merb-plugins
      merb-plugins/merb_param_protection),
    :install => %w(
      merb-core
      merb-more
      merb-plugins
    ),
    :gems => %w(
      merb
      merb-action-args
      merb-assets
      merb-builder
      merb-cache
      merb-core
      merb-gen
      merb-haml
      merb-mailer
      merb-more
      merb-parts
      merb_activerecord
      merb_datamapper
      merb_helpers
      merb_param_protection
      merb_rspec
      merb_sequel
      merb_stories
      merb_test_unit
    )
  }
}

</th> <th><a name=”datamapperAs a follow-up to my previous post, here’s some gotchas to be aware of if you’re looking to support both ActiveRecord and DataMapper in a Merb (and/or Rails) plugin.

The Strategy

Basic Translation

Now that we have a plan, we can start translating our extracted functions from AR bits to DM bits. There’s a bunch of fairly straightforward transformations we can make.

ActiveRecord	DataMapper
.find(:all, ...)	.all(...)
.find(:first, ...)	.first(...)
.find(id)	.get(...)
.find_all_by_id(id)	.all(:id => ids)
.table_name	.storage_name
.primary_key	.key.first.name
.connection	repository.adapter

Raw SQL

Hooks

In before hooks, the AR hook chain stops if your method returns false, in DM you must throw :halt.

Things You Shouldn’t Be Doing Anyways

If you’re manually setting @attribute values in your AR code, you’ll need to use instance_variable_set for DataMapper. I recommend writing manual accessor methods to wrap it for abstraction.

So, DataMapper 0.3 was behaving weirdly for me, and I thought I’d try upgrading to 0.9 to see how things are there. Overall I’m quite liking it, but there’s a few catches:

It’s incompatible with Vlad for deployment, haven’t looked into it but something in DM is making the ‘repository’ value unsettable, which Vlad uses to determine the checkout path.
Legacy connections beware, there doesn’t seem to be a current alternative for set_table_name at the moment.
:memory: is no longer a good name for your test database when using sqlite. Use a fully-qualified connection string like sqlite://:memory: instead.
DataMapperPersistence has been renamed DataMapperResource. Include it in models.
Validations are an add-on now, include DataMapper::Validate in your model (or even re-open DM:Resource and include it there).
Properties now take a class instead of a symbol (you can guess at all the main ones), and require the id to be specified like so: property :id, Fixnum, :serial => true
Associations are renamed, has_one is now one_to_one, has_many is one_to_many or many_to_many, etc. Haven’t delved in deep yet though, so I’m not sure how to define who gets the foreign key.
New migration code is just now getting in to dm-core, and auto_migrate! is a thing of the past. Sucks for those of us using sqlite in-memory test databases that need a fresh migration every time.

My installation Rakefile follows, just stuff it in an empty directory and it’ll do everything from there. Much thanks to Atmos for a good starting point and some setup help.

desc "Fetch and Install DM and Merb"
task :install_all do 
  config = CONFIG['dm']
  fetch config[:user], config[:repos]
  install config[:install]
  config = CONFIG['merb']
  fetch config[:user], config[:repos]
  install config[:install]
end

desc "Uninstall DM and Merb"
task :uninstall_all do
  uninstall CONFIG['dm'][:gems]  
  uninstall CONFIG['merb'][:gems]  
end

desc "Download latest sources for :project from git"
task :fetch, :project do |task, args|
  config = CONFIG[args[:project]]
  fetch config[:user], config[:repos]
end

desc "Install :project from git"
task :install, :project do |task, args|
  config = CONFIG[args[:project]]
  install config[:install]
end

desc "Uninstall :project"
task :uninstall, :project do |task, args|
  config = CONFIG[args[:project]]
  uninstall config[:gems]
end

def fetch(user, repos)
  base = File.expand_path(".")
  Dir.chdir base do
    repos.each do |repo|
      repo_dir = "#{base}/#{repo}"
      unless File.directory?(repo_dir)
        %x{git clone git://github.com/#{user}/#{repo}.git }
      end
      Dir.chdir(repo_dir) { %x{git pull} }
    end
  end
end

def install(modules)
  base = File.expand_path(".")
  modules.each do |lib|
    Dir.chdir("#{base}/#{lib}") do
      cmd = "sudo rake install 2>/dev/null |" +
            " grep -v '^\(in' |" +
            " grep -v '^[0-9] gem' |" +
            " grep -v '^[IUc\. ]'"
      puts %x{#{cmd}}
    end
  end
end

def uninstall(gems)
  gems.each do |name|
    puts %x{yes | sudo gem uninstall #{name} -aI}
  end
end

CONFIG = {
  'dm' => {
    :user => 'sam',
    :repos => %w(
      do
      dm-core
      dm-more),
    :install => %w(
      do/data_objects
      do/do_sqlite3
      do/do_mysql
      do/do_postgres
      dm-core
      dm-more/merb_datamapper
      dm-more/dm-migrations
      dm-more/dm-serializer
      dm-more/dm-validations
    ),
    :gems => %w(
      data_objects
      do_sqlite3
      do_mysql
      do_postgres
      dm-core
      merb_datamapper
      dm-migrations
      dm-serializer
      dm-validations
    )
  },
  'merb' => {
    :user => 'wycats',
    :repos => %w(
      merb-core
      merb-more
      merb-plugins
      merb-plugins/merb_param_protection),
    :install => %w(
      merb-core
      merb-more
      merb-plugins
    ),
    :gems => %w(
      merb
      merb-action-args
      merb-assets
      merb-builder
      merb-cache
      merb-core
      merb-gen
      merb-haml
      merb-mailer
      merb-more
      merb-parts
      merb_activerecord
      merb_datamapper
      merb_helpers
      merb_param_protection
      merb_rspec
      merb_sequel
      merb_stories
      merb_test_unit
    )
  }
}

” class=”anchor”> </th></tr>

  <tr><th>snippet#!/usr/bin/env ruby
col = ARGV.pop.to_i-1
while line = gets
  puts line.chomp.split(/\s+/)[col]
end

For when you just want a list of filenames from version control, hg st | grep '?' | col 2

And because I can never remember the standard unix tool that does the same thing, and awk is awkward.

</th> <th><a name=”snippet#!/usr/bin/env ruby col = ARGV.pop.to_i-1 while line = gets puts line.chomp.split(/\s+/)[col] end

For when you just want a list of filenames from version control, hg st | grep '?' | col 2

And because I can never remember the standard unix tool that does the same thing, and awk is awkward.

” class=”anchor”> </th></tr>

  <tr><th>holmesHGTV in Canada produces a tv show called [Holmes on Homes][] that follows general contractor Mike Holmes as he visits failed renovations, provides commentary on the sorry situation of the work done, and then goes about fixing them.

Season One

Additional Grief
Soggy Sorority
Botched Basement
Attica! Attica / Crappy Capping
Flimsy Floor
Kitchen Catastrophe
Window Pain
Faulty Showers
Tiles and Tribulations
Site Unseen
Sweet Home Abandoned
Whole House Disaster

Season Two

Terrible Terrace
Drafty Ducting
Ramp Revamp
Flooded Foundation
Garage Grievance
Lamin-Ain’t
Roof Goof
Floor Fiasco
Doozy Jacuzzi
No Grout About It
Jacking the Box
Access Denied
Hell’s Kitchen
Holmes for the Holidays

Season Five

Holmes Inspection
Showing the Cracks
What a Mesh

Season Six

Due Date
Frozen Assets
Gone to Pot
Lack of Truss
Clean Slate
Completely Incomplete
Nashville Kitchen
Pasadena 911
Shaky Foundation
Stone Walled
Third Time Lucky

Season Seven

Hit the Deck
Rocky Reno
Paradise Island

Specials/Unaired

Lien on Me

Part of a series.

In additional grief, a couple adds a 10’x10’ addition onto their kitchen, which includes a door to outside with a small deck and stairs down to ground level.

You have the client specifying in detail we would like the switches here, we would like certain things this way, and of course you have the contractor going not a problem, not a problem. But I’m told each and every day they came home it was done differently than they had talked about.

Communication’s number one here. There was no proper communication. It did not happen the way the contractor said it would happen. Make sure your contractor complies with what you ask for. If he doesn’t, stop the job.

The biggest step here, and one that I’m having to work on, is put things in writing. A vague “yeah, yeah” in a phone call or an in-person chat is easily and quickly forgotten, but if you take notes and fire off an email summary after the discussion it serves as a permanent record. As long as it’s available to both parties, it can then be used during development and afterward to confirm that features are being done as requested.

If you can get away with it, make it very clear up front that you’ll be using Software X (be it Trac, Mingle, or otherwise) as the be-all-end-all of development plan, and nothing gets done unless it’s in there. Then just make sure you populate the system after any discussions.

This was underpriced in the beginning, as soon as I was told the price of 26,000 for the addition plus the kitchen I knew right off the bat that it was definitely too cheap, so they had to cut corners.

Well, obviously they thought they did a great job, they actually charged I think it was about $10,000 more than the original price for all the little extras they got them on…. This job was actually too cheap to start with.

Price is an often-taboo point of discussion, and it’s easy for either side to feel that they’re not getting an appropriate price for the work performed. The best thing I can suggest on this front is many small milestones. From a contractor’s perspective, I want to go no more than two weeks without being paid, but it’s hard for the client to justify that unless they have working software available to them.

This is something that I think the Agile crowd picked up on right - perform work in small iterations, with a focus being on working software. Avoid incremental development (first Users, then Profiles, then Friends) and instead iteratively improve on the functional base (first basic login information with a friendship association, then add more user information, then add user profiles, then add friendship information to the profiles). As you reach milestones in the development, the client releases funds.

If you’re doing a long-term project with weekly billing, rather than a fixed-scope project and milestone billing, it’s the client’s responsibility to make sure that each week they’re seeing useful, functional improvements to the software. In either case, if at any time the client isn’t happy with the work, it’s their responsibility to stop development as soon as possible.

This is guys that think they know what they’re doing [when they don’t], and it shows.

As you can see, as we’re pulling it down we’re finding problem after problem. What we have to do, to put this back together, to make it right, is not a simple project.

I just want to tear this down, and do it again.

Being brought in on a contract to extend a project, or even finish one up, very often this last thought is the feeling that resonates. But as long as the foundation is solid, a total rewrite is usually wasted effort. Better to run through with a fine-toothed comb and identify problem points, and limit the wholesale rework to just those areas.

No outside contractors came in, they done the electrical, they done the plumbing, they done everything.

Not everybody can do everything. You’ve heard the saying, jack of all trades, master of none.

We brought in quite a few professionals on this job. You’re gonna hire a contractor, I think you really want to make sure he’s gonna bring in the right people. Licensed carpenter, licensed electrician, that’s what I want to see on my place - qualified people.

We are as good as the people around us. This is a big lesson learned here, make sure they bring in the right people.

Web development is funny in that sometimes you get individual contractors claiming to be able to handle everything from the back-end database design to coming up with a front-end look and feel for a site, and then behind the scenes go to other people to cover the areas they’re not as strong in. It’s done to save face, as though there’s an admission of inadequacy when someone admits they’re a great coder but have no sense of visual style whatsoever.

I much rather dealing with clients who at the beginning ask if we have a designer we can go to, or if they should look for someone else to cover that aspect of the job. It’s a perspective that shows that they appreciate finding experts in our individual fields to provide them with the best quality work we can.

</th> <th><a name=”holmesHGTV in Canada produces a tv show called Holmes on Homes that follows general contractor Mike Holmes as he visits failed renovations, provides commentary on the sorry situation of the work done, and then goes about fixing them.

Season One

Additional Grief
Soggy Sorority
Botched Basement
Attica! Attica / Crappy Capping
Flimsy Floor
Kitchen Catastrophe
Window Pain
Faulty Showers
Tiles and Tribulations
Site Unseen
Sweet Home Abandoned
Whole House Disaster

Season Two

Terrible Terrace
Drafty Ducting
Ramp Revamp
Flooded Foundation
Garage Grievance
Lamin-Ain’t
Roof Goof
Floor Fiasco
Doozy Jacuzzi
No Grout About It
Jacking the Box
Access Denied
Hell’s Kitchen
Holmes for the Holidays

Season Five

Holmes Inspection
Showing the Cracks
What a Mesh

Season Six

Due Date
Frozen Assets
Gone to Pot
Lack of Truss
Clean Slate
Completely Incomplete
Nashville Kitchen
Pasadena 911
Shaky Foundation
Stone Walled
Third Time Lucky

Season Seven

Hit the Deck
Rocky Reno
Paradise Island

Specials/Unaired

Lien on Me

Part of a series.

In additional grief, a couple adds a 10’x10’ addition onto their kitchen, which includes a door to outside with a small deck and stairs down to ground level.

You have the client specifying in detail we would like the switches here, we would like certain things this way, and of course you have the contractor going not a problem, not a problem. But I’m told each and every day they came home it was done differently than they had talked about.

Communication’s number one here. There was no proper communication. It did not happen the way the contractor said it would happen. Make sure your contractor complies with what you ask for. If he doesn’t, stop the job.

This was underpriced in the beginning, as soon as I was told the price of 26,000 for the addition plus the kitchen I knew right off the bat that it was definitely too cheap, so they had to cut corners.

Well, obviously they thought they did a great job, they actually charged I think it was about $10,000 more than the original price for all the little extras they got them on…. This job was actually too cheap to start with.

This is guys that think they know what they’re doing [when they don’t], and it shows.

As you can see, as we’re pulling it down we’re finding problem after problem. What we have to do, to put this back together, to make it right, is not a simple project.

I just want to tear this down, and do it again.

No outside contractors came in, they done the electrical, they done the plumbing, they done everything.

Not everybody can do everything. You’ve heard the saying, jack of all trades, master of none.

We brought in quite a few professionals on this job. You’re gonna hire a contractor, I think you really want to make sure he’s gonna bring in the right people. Licensed carpenter, licensed electrician, that’s what I want to see on my place - qualified people.

We are as good as the people around us. This is a big lesson learned here, make sure they bring in the right people.

” class=”anchor”> </th></tr>

  <tr><th>seasideI'm coming late to the [controversy][], I know. I was talking with a co-worker about Rails and Seaside the other day, and after describing the Seaside structure and philosophy compared to Rails I got to thinking that there's really not as much overlap as some people think between the two.

Just looking at these, it’s easy to see where one framework shines and the other would require more work to get there.

</th> <th><a name=”seasideI’m coming late to the controversy, I know. I was talking with a co-worker about Rails and Seaside the other day, and after describing the Seaside structure and philosophy compared to Rails I got to thinking that there’s really not as much overlap as some people think between the two.

Just looking at these, it’s easy to see where one framework shines and the other would require more work to get there.

” class=”anchor”> </th></tr>

  <tr><th>campingWell, I managed to get the weight-tracking app functional (graph and all) in about 220 lines, just tweaking the look now. It's a single-script Camping app using Gruff for graphing, with SQLite for data storage. Not exactly the most efficient app (I'm cheating by using a lot of mostly-null records in the database) but it gets the job done. There were a few things that got me stuck for a bit that weren't obviously mentioned in the camping docs, so I thought I'd put them down here.

Not exactly a camping thing, but if you want a non 4:3 ratio gruff graph, send a string ‘1000x350’ or similar.

</th> <th><a name=”campingWell, I managed to get the weight-tracking app functional (graph and all) in about 220 lines, just tweaking the look now. It’s a single-script Camping app using Gruff for graphing, with SQLite for data storage. Not exactly the most efficient app (I’m cheating by using a lot of mostly-null records in the database) but it gets the job done. There were a few things that got me stuck for a bit that weren’t obviously mentioned in the camping docs, so I thought I’d put them down here.

Not exactly a camping thing, but if you want a non 4:3 ratio gruff graph, send a string ‘1000x350’ or similar.

” class=”anchor”> </th></tr>

  <tr><th>Rails# Ruby Testing

General

vcr to stub out remote calls and allow unit tests with network calls

Rspec

rspec-mocks for test doubles/mocks
rspec-tracer to help diagnose flaky specs

Test-Unit

rr for test doubles/mocks

Rails System Testing

Blog post arguing for CDP, Ferrum, and Cuprite which provides a pure Ruby Capybara driver talking to Chrome.
Running system tests with Playwright leads to more reliable tests than Selenium as it’s more aware of JS runtime execution, no need to wait arbitrary times for the browser to “settle”.

Rails Job Queues

GoodJob is a multithreaded, postgres-backed, ActiveJob-compatible backend. Blog post

Rails Deferred Content

render_async is a pretty straightforward setup that lets you defer rendering of various data blocks until after the initial page load. Sounds very similar to what HEY! is doing, allows you to set up your primary pages as static (thus cache-friendly), and defer per-account personalization until later.

Turbo Frames is the aforementioned HEY! solution, now open-source. See also this writeup on a practical application of the library.

Quality Gems

TODO: Make a blog post out of this.

A shout-out to some gems/tools that have a well-deserved place in my Rails toolbox.

Pagy (ActiveRecord pagination, focused on performance)
Pry (debugger)
Rbspy (sampling profiler) [Rust]
rack-mini-profiler, flamegraph, stackprof, memory_profiler (performance analysis tooling) - (reference custom instrumentation)
rubocop / rufo
Hamlit
Fivemat (rspec/cucumber formatter, progress but one line per file)
Cells
d3js (chart-buildling library) [JS]
sidekiq
Bullet (detects N+1 queries) - Compare to newcomer prosopite
Insomnia (HTTP api tool)
HTTP.rb (compare Typhoeus, others as a blog post?)

And some TODOs that I want to try out.

api_struct for writing API clients, as a step up from my usual bit of raw httprb and JSON->hash data bags on the response.

Evil Martians

Posted an article about their Gemfile of dreams with a summary of the things they use frequently and why. (Note: this got an update April 2026)

ActiveRecord

For better organization, try to be consistent about macro methods (there’s no general standard ordering, just pick one and be consistent in your project).

ActiveJob

Style guide

Admin Panels

Mission Control is probably the default
Modern Queue as an alternative# ActionMailer

Some advice on naming - RecipientMailer#resource_change_intention.

</th> <th><a name=”Rails# Ruby Testing

General

vcr to stub out remote calls and allow unit tests with network calls

Rspec

rspec-mocks for test doubles/mocks
rspec-tracer to help diagnose flaky specs

Test-Unit

rr for test doubles/mocks

Rails System Testing

Blog post arguing for CDP, Ferrum, and Cuprite which provides a pure Ruby Capybara driver talking to Chrome.
Running system tests with Playwright leads to more reliable tests than Selenium as it’s more aware of JS runtime execution, no need to wait arbitrary times for the browser to “settle”.

Rails Job Queues

GoodJob is a multithreaded, postgres-backed, ActiveJob-compatible backend. Blog post

Rails Deferred Content

Turbo Frames is the aforementioned HEY! solution, now open-source. See also this writeup on a practical application of the library.

Quality Gems

TODO: Make a blog post out of this.

A shout-out to some gems/tools that have a well-deserved place in my Rails toolbox.

Pagy (ActiveRecord pagination, focused on performance)
Pry (debugger)
Rbspy (sampling profiler) [Rust]
rack-mini-profiler, flamegraph, stackprof, memory_profiler (performance analysis tooling) - (reference custom instrumentation)
rubocop / rufo
Hamlit
Fivemat (rspec/cucumber formatter, progress but one line per file)
Cells
d3js (chart-buildling library) [JS]
sidekiq
Bullet (detects N+1 queries) - Compare to newcomer prosopite
Insomnia (HTTP api tool)
HTTP.rb (compare Typhoeus, others as a blog post?)

And some TODOs that I want to try out.

api_struct for writing API clients, as a step up from my usual bit of raw httprb and JSON->hash data bags on the response.

Evil Martians

Posted an article about their Gemfile of dreams with a summary of the things they use frequently and why. (Note: this got an update April 2026)

ActiveRecord

For better organization, try to be consistent about macro methods (there’s no general standard ordering, just pick one and be consistent in your project).

ActiveJob

Style guide

Admin Panels

Mission Control is probably the default
Modern Queue as an alternative# ActionMailer

Some advice on naming - RecipientMailer#resource_change_intention.

” class=”anchor”> </th></tr>

  <tr><th>code# What Comes After MVC

Via a Railsconf talk, What Comes After MVC by Peter Harkins.

Advice: most of these extractions can be applied partially, but the farther we go the better our code looks.

Goal: Split code based on two axes, mutability & side effects.

Immutable means: when we call methods on it with the same arguments, we get the same results.

No side effect means: when we call methods, no other objects change.

Value - immutable, no side effects

Mostly a constructor + query/converstion methods.
Also useful to have comparisons (==, <=>) and typecasts (to_s, to_str, to_a, inspect, etc). Delegate if that makes sense. Use dry-equalizer or similar if that helps.
Should be able to freeze after initialize without breaking anything. See also adamantium for an improved auto-freeze.

Extract values from ActiveRecord models to make them easier to reason about, and group similar behaviours. Don’t have your values call or return ActiveRecord objects - they’re implicitly mutable.

Consider overriding getter/setter methods for an attribute to auto-promote primitives (strings, ints, timestamps, etc) to value objects. Rails 5 attributes API helps here, but is a bit wordy.

Testing:

no let
no stub
no factory
no mocks
assert on results

Entity - mutable, no side effects

Job is to have an identity, and wrap up values.
Often has very little code, because it doesn’t do much (on account of no side effects).
Overall similar to an AR model, but without any side effects.

Extracting Entities from ActiveRecord:

Find identity (probably primary key).
Extract Values
Drive out side effects to Adapters & Shells.

Controvertial Opinion: ActiveRecord models shouldn’t call their or other models queries (scopes, find, where) or lifecycle methods (create, save, reload) - eg. any methods with side effects.

Testing:

few lets for Values
maybe factories
maybe stub entites, but not Values
assert on results
assert on object state

Adapter - immutable, side effects

(named after Hexagonal pattern)

Wraps interaction with the external world (includes your own database!)
Usually a pretty thin wrapper.

Testing:

few lets for Values
often stubs
asserts on mocks for outgoing queries/commands
asserts on results are probbaly not worth much

Shell - mutable, side effects

Sequence of transformations, imperative code. Sometimes can get away with functional composition of individual steps. (Elixir’s |> embodies this concept very succinctly.)
Rails Controller actions can be compared to a Shell.
General shape: talk to adapters, coordinate Values and Entities to do work.
Harder to reason about, try to keep small.

Testing:

fixtures with real-world data
might need factories to create enough Entities
expect on results, state, and mocks
Integration: one happy path to ensure objects glue together properly, and regression tests as necessary for confidence
if you have one integrated test, can stub Adapters later

Other - mutable, side effects

This is typical Rails code.

Closing Notes

Immutable objects cannot call mutable objects, effect-free code cannot call code with side effects. Thus:

Values may only depend on other values.
Entities might collaborate with other entities, and also encapsulate values.
Adapters may use values, but are unlikely to depend on other adapters.
Shells (and other, legacy code) can continue to do whatever it wants.

The benefit of extracting Values, Entities, and Adapters out of the regular ball of code is so that we can have smaller pieces of code that are (a) easy to reason about, and (b) easy to test.

Empirical truth: once your tests start using ActiveRecord, they slow down immensely.

Stimulus JS

Main Site

Simple how-to tutorial using Stimulus to handle loading an HTML fragment from the server (Rails).

BetterStimulus has a collection of guidelines and best-practices.

example of testing stimulus with jest

Ruby

Ruby is a dynamic programming language.

It’s got some good libraries, like Rails and Middleman

Ruby Upgrade Notes

Ruby 2.7

There’s a big thread talking about the Ruby 2.7/3.0 upgrade path, includes some good ideas for managing the warning storm related to kwargs (see especially Florent’s post, warning.rb to squelch prod/dev log output, and Deprecation Toolkit to keep a to-do list both look handy).

Webpacker

Migrating from Sprockets to Webpacker

Ruby Gems

Just a general collection of gems I’ve known and loved (or want to try out).

Comparisons

Coditsu will produce a diff between versions of a gem. Great if you want to see what’s changed, or need to do serious auditing of dependencies - run a bundle update, and then hit this for all the changes in Gemfile.lock
RailsDiff will produce a diff between the output of rails new, handy for making sure you’re picking up new default configs as you roll the update train.

Code Coverage

Debride finds uncalled methods
Rcov measures coverage while running tests
Coverband measures coverage in production

Commandline

Github has a CLI

Database

Bullet detects N+1 queries

Debuggers

Byebug is my go-to for tracing, to step into function calls, skip to next line, and continue execution.
Pry is the other really popular one.
Jard looks interesting, provides a multi-window debugger environment, with a GUI stack trace, local variable dumps, threads, etc.
Ruby is also getting a native debugger in 3.1 with more interesting breakpoints/callbacks that might prove useful for IDE integration.

Worker Queues

All these queues support ActiveJob in Rails

Que leans heavily on Postgres-specific features for queue management, keeps it in the DB
Resque is Redis-backed, includes a management site
Sidekiq also Redis-backed, includes management site, and includes a Pro paid offering with some additional enterprisey features

Rails

Dual-boot rails upgrade strategy that I’ve seen mentioned a few times.

Synvert helps automatically convert syntax for some rails updates, and a few other gems.

Architecture

Some good notes on code organization from Jason Swett, the rails testing guy.

Admin Interface

ActiveAdmin has a lot of mindshare, but I use it extensively and hate it with a passion. Between the obnoxious DSL, the shitty html-in-ruby that is Arbre, and the amazingly bad performance hit reloading a large admin site from scratch on any code change in development, cannot recommend.

Administrate I’ve tried in a smaller app, it has a bit of magic API for working off of common templates (so for defining fields for the index/show pages, etc), but easily allows you to generate whatever view you need so you can override it with simple erb.

Avo is new option, and is paid for extended features. Runs Hotwire on the backend, but might be worth looking at. Has a useful default design for filters/actions that doesn’t take up a lot of space from the main interface screens.

Rails Time Testing

Old code would use the TimeCop gem, but Rails 5.2 has native time helpers - migration example.

Rails App Skeletons

Starter apps for Rails with a bunch of common decisions already made.

There’s an Airtable with a larger list, and last-updated dates.

Instant Rails from Jason Swett

Devise
RSpec
Local docker

Suspenders from Thoughtbot

Since long time
Heroku-ready!
Bourbon/Sass -> Tailwind
DJ -> Que
CircleCI

Rails Composer

app scaffold :/

Zen Rails

Rails 6.1, Ruby 3.0
dev: rubocop, brakeman, byebug…
test: rspec, capybara w/ ChromeDriver, simplecov
Devise
Pundit

Wheel

Heroku-ready
ActiveAdmin
Sidekiq
CircleCI
Bootstrap -> Tailwind

Jumpstart Rails $150/450

Since June 2019
Tailwind
Stripe/Braintree
Admin
Sidekiq

Bullet Train Free, Pro

Modern Javascript Frameworks

React
VueJS
Svelte single-file components, plain-looking javascript (with just local vars, compiler does the magic)

Javascript In Rails

Without going full-on with a Rails API backend, and pure React/VueJS frontend, there’s still some options to add interactivity and responsiveness to the frontend when moving on from jQuery.

Stimulus JS (web) is a BaseCamp extraction, doesn’t do any rendering, just exists to provide some really straightforward interaction. Plays well with asset pipeline & turbolinks.
- If you want to supercharge it, check out StimulusReflex or Motion for high-performance reactive components, powered by Stimulus and Websockets.
react-rails also works with asset pipeline/webpacker, and gives you a straightforward way to render react components from your views. Works best with smaller components.
Alpine JS directly bills itself as a minimal alternative to Vue/React, can be written inline or extracted to helper methods. Can be readily used as a raw javascript include (via CDN or asset pipeline) to be rails-friendly without bringing in NPM/webpack unnecessarily.
Inertia is “the glue that connects” server-side and client-side frameworks.

Books

Modern Front-End Development for Rails by Noel Rappin - webpack, typescript, Hotwire, Turbo, Stimulus, React, dealing with state, testing.

Computer Science

Teach Yourself CS is a pointer to some good reference material on a few major pillars of computer science, that you might not have had exposure to as a self-taught programmer. Computer architecture to data structures to networking to compilers, and more.

Coding Style

My preferred coding styles, for spinning up new projects:

Ruby

Use Standard, which is built on top of RuboCop.

See this post for a good Standard+Rubocop project config, so you can just use Rubocop tooling in your editor.

See also this post, safe-auto-correct is now a default!

A full (manual) configuration set for rails+rspec, noteworthy for the Layout/ClassStructure setting.

Javascript

Use Standard JS, which is built on top of eslint.

Git notes:

Create a .git-blame-ignore-revs file with one commit hash per line (comments with #) to get [Github to ignore those commits] when doing git blame and related analysis. Run git config blame.ignoreRevsFile .git-blame-ignore-revs to get your local git to do likewise.

Now whenever you do a few automated cleanups (one type per commit yeah?) just add that commit hash to the file, and enjoy the lack of noise in your git history.

Code Profiling

Snippet to make a useful encapsulation in Rails apps.

config/initializers/rubyprof.rb:

class Prof
  def self.callstack(filename='callstack.html')
    result = nil
    profile = RubyProf.profile do
      result = yield
    end
    report = RubyProf::CallStackPrinter.new(profile)
    File.open(filename, 'w') do |file|
      report.print(file)
    end
    result
  end
end

CSS in Javascript

Styled JSX - gives reasonable style tags like Vue, scoped to just the current component. Unlike styled-components, you don’t need to define specifically-wrapped elements.
Styled-componentsstyled-components is probably the community forerunner, 26k stars
Alternately, write less CSS with Tailwind. See a good use of components in docs.

</th> <th><a name=”code# What Comes After MVC

Via a Railsconf talk, What Comes After MVC by Peter Harkins.

Advice: most of these extractions can be applied partially, but the farther we go the better our code looks.

Goal: Split code based on two axes, mutability & side effects.

Immutable means: when we call methods on it with the same arguments, we get the same results.

No side effect means: when we call methods, no other objects change.

Value - immutable, no side effects

Mostly a constructor + query/converstion methods.
Also useful to have comparisons (==, <=>) and typecasts (to_s, to_str, to_a, inspect, etc). Delegate if that makes sense. Use dry-equalizer or similar if that helps.
Should be able to freeze after initialize without breaking anything. See also adamantium for an improved auto-freeze.

Extract values from ActiveRecord models to make them easier to reason about, and group similar behaviours. Don’t have your values call or return ActiveRecord objects - they’re implicitly mutable.

Consider overriding getter/setter methods for an attribute to auto-promote primitives (strings, ints, timestamps, etc) to value objects. Rails 5 attributes API helps here, but is a bit wordy.

Testing:

no let
no stub
no factory
no mocks
assert on results

Entity - mutable, no side effects

Job is to have an identity, and wrap up values.
Often has very little code, because it doesn’t do much (on account of no side effects).
Overall similar to an AR model, but without any side effects.

Extracting Entities from ActiveRecord:

Find identity (probably primary key).
Extract Values
Drive out side effects to Adapters & Shells.

Controvertial Opinion: ActiveRecord models shouldn’t call their or other models queries (scopes, find, where) or lifecycle methods (create, save, reload) - eg. any methods with side effects.

Testing:

few lets for Values
maybe factories
maybe stub entites, but not Values
assert on results
assert on object state

Adapter - immutable, side effects

(named after Hexagonal pattern)

Wraps interaction with the external world (includes your own database!)
Usually a pretty thin wrapper.

Testing:

few lets for Values
often stubs
asserts on mocks for outgoing queries/commands
asserts on results are probbaly not worth much

Shell - mutable, side effects

Sequence of transformations, imperative code. Sometimes can get away with functional composition of individual steps. (Elixir’s |> embodies this concept very succinctly.)
Rails Controller actions can be compared to a Shell.
General shape: talk to adapters, coordinate Values and Entities to do work.
Harder to reason about, try to keep small.

Testing:

fixtures with real-world data
might need factories to create enough Entities
expect on results, state, and mocks
Integration: one happy path to ensure objects glue together properly, and regression tests as necessary for confidence
if you have one integrated test, can stub Adapters later

Other - mutable, side effects

This is typical Rails code.

Closing Notes

Immutable objects cannot call mutable objects, effect-free code cannot call code with side effects. Thus:

Values may only depend on other values.
Entities might collaborate with other entities, and also encapsulate values.
Adapters may use values, but are unlikely to depend on other adapters.
Shells (and other, legacy code) can continue to do whatever it wants.

The benefit of extracting Values, Entities, and Adapters out of the regular ball of code is so that we can have smaller pieces of code that are (a) easy to reason about, and (b) easy to test.

Empirical truth: once your tests start using ActiveRecord, they slow down immensely.

Stimulus JS

Main Site

Simple how-to tutorial using Stimulus to handle loading an HTML fragment from the server (Rails).

BetterStimulus has a collection of guidelines and best-practices.

example of testing stimulus with jest

Ruby

Ruby is a dynamic programming language.

It’s got some good libraries, like Rails and Middleman

Ruby Upgrade Notes

Ruby 2.7

Webpacker

Migrating from Sprockets to Webpacker

Ruby Gems

Just a general collection of gems I’ve known and loved (or want to try out).

Comparisons

Coditsu will produce a diff between versions of a gem. Great if you want to see what’s changed, or need to do serious auditing of dependencies - run a bundle update, and then hit this for all the changes in Gemfile.lock
RailsDiff will produce a diff between the output of rails new, handy for making sure you’re picking up new default configs as you roll the update train.

Code Coverage

Debride finds uncalled methods
Rcov measures coverage while running tests
Coverband measures coverage in production

Commandline

Github has a CLI

Database

Bullet detects N+1 queries

Debuggers

Byebug is my go-to for tracing, to step into function calls, skip to next line, and continue execution.
Pry is the other really popular one.
Jard looks interesting, provides a multi-window debugger environment, with a GUI stack trace, local variable dumps, threads, etc.
Ruby is also getting a native debugger in 3.1 with more interesting breakpoints/callbacks that might prove useful for IDE integration.

Worker Queues

All these queues support ActiveJob in Rails

Que leans heavily on Postgres-specific features for queue management, keeps it in the DB
Resque is Redis-backed, includes a management site
Sidekiq also Redis-backed, includes management site, and includes a Pro paid offering with some additional enterprisey features

Rails

Dual-boot rails upgrade strategy that I’ve seen mentioned a few times.

Synvert helps automatically convert syntax for some rails updates, and a few other gems.

Architecture

Some good notes on code organization from Jason Swett, the rails testing guy.

Admin Interface

Rails Time Testing

Old code would use the TimeCop gem, but Rails 5.2 has native time helpers - migration example.

Rails App Skeletons

Starter apps for Rails with a bunch of common decisions already made.

There’s an Airtable with a larger list, and last-updated dates.

Instant Rails from Jason Swett

Devise
RSpec
Local docker

Suspenders from Thoughtbot

Since long time
Heroku-ready!
Bourbon/Sass -> Tailwind
DJ -> Que
CircleCI

Rails Composer

app scaffold :/

Zen Rails

Rails 6.1, Ruby 3.0
dev: rubocop, brakeman, byebug…
test: rspec, capybara w/ ChromeDriver, simplecov
Devise
Pundit

Wheel

Heroku-ready
ActiveAdmin
Sidekiq
CircleCI
Bootstrap -> Tailwind

Jumpstart Rails $150/450

Since June 2019
Tailwind
Stripe/Braintree
Admin
Sidekiq

Bullet Train Free, Pro

Modern Javascript Frameworks

React
VueJS
Svelte single-file components, plain-looking javascript (with just local vars, compiler does the magic)

Javascript In Rails

Without going full-on with a Rails API backend, and pure React/VueJS frontend, there’s still some options to add interactivity and responsiveness to the frontend when moving on from jQuery.

Stimulus JS (web) is a BaseCamp extraction, doesn’t do any rendering, just exists to provide some really straightforward interaction. Plays well with asset pipeline & turbolinks.
- If you want to supercharge it, check out StimulusReflex or Motion for high-performance reactive components, powered by Stimulus and Websockets.
react-rails also works with asset pipeline/webpacker, and gives you a straightforward way to render react components from your views. Works best with smaller components.
Alpine JS directly bills itself as a minimal alternative to Vue/React, can be written inline or extracted to helper methods. Can be readily used as a raw javascript include (via CDN or asset pipeline) to be rails-friendly without bringing in NPM/webpack unnecessarily.
Inertia is “the glue that connects” server-side and client-side frameworks.

Books

Modern Front-End Development for Rails by Noel Rappin - webpack, typescript, Hotwire, Turbo, Stimulus, React, dealing with state, testing.

Computer Science

Coding Style

My preferred coding styles, for spinning up new projects:

Ruby

Use Standard, which is built on top of RuboCop.

See this post for a good Standard+Rubocop project config, so you can just use Rubocop tooling in your editor.

See also this post, safe-auto-correct is now a default!

A full (manual) configuration set for rails+rspec, noteworthy for the Layout/ClassStructure setting.

Javascript

Use Standard JS, which is built on top of eslint.

Git notes:

Now whenever you do a few automated cleanups (one type per commit yeah?) just add that commit hash to the file, and enjoy the lack of noise in your git history.

Code Profiling

Snippet to make a useful encapsulation in Rails apps.

config/initializers/rubyprof.rb:

class Prof
  def self.callstack(filename='callstack.html')
    result = nil
    profile = RubyProf.profile do
      result = yield
    end
    report = RubyProf::CallStackPrinter.new(profile)
    File.open(filename, 'w') do |file|
      report.print(file)
    end
    result
  end
end

CSS in Javascript

Styled JSX - gives reasonable style tags like Vue, scoped to just the current component. Unlike styled-components, you don’t need to define specifically-wrapped elements.
Styled-componentsstyled-components is probably the community forerunner, 26k stars
Alternately, write less CSS with Tailwind. See a good use of components in docs.

” class=”anchor”> </th></tr>

  <tr><th>education# Technical Books

Considering reading/re-reading some of these, I think they came from a Reddit or HN discussion. Probably worth summary blog posts.

Pragmatic programmer
Clean code
Code complete
Refactoring
Head first design patterns
Mythical man month
Clean coder
Working effectively with legacy code
Design patterns
Cracking the coding interview
Soft skills
Don’t make me think
Code - petzold
Introduction to algorithms
People ware
Programming pearls
Patterns of enterprise app architecture
SICP
Art of computer programming
Domain driven design

Computer Science

Considering reading/re-reading some of these, I think they came from a Reddit or HN discussion. Probably worth summary blog posts.

Pragmatic programmer
Clean code
Code complete
Refactoring
Head first design patterns
Mythical man month
Clean coder
Working effectively with legacy code
Design patterns
Cracking the coding interview
Soft skills
Don’t make me think
Code - petzold
Introduction to algorithms
People ware
Programming pearls
Patterns of enterprise app architecture
SICP
Art of computer programming
Domain driven design

Computer Science

  <tr><th>tech# Text Chat

“Gold Standard” seems to be Slack
Open-source, self-hosted alternative is Mattermost. Extremely similar UX to Slack.
Gamer-marketed Discord ticks most of the same boxes, but is designed for more open communities, and lacks video calls (but does voice calls just fine, and will handle screenshare for a limited number of attendees).
Open-source “slack-with-threads” is Zulip, wants to use a smaller number of channels with consistent team membership, and then create ad-hoc named threads (with their own history) to spin off discussions so that those conversations can be followed easier.
Static Site Generators

I currently use Middleman.

I’m keeping an eye on Bridgetown which is still a Ruby base for static content (it started as a fork of Jekyll), but is modern-javascript-aware. There’s a pipeline ready to go if you want to integrate React/VueJS/Stimulus JS components or CSS frameworks like Bulma/Tailwind that benefit from a build/purge step.

Remote Work

Sneek $10/user, gives a webcam-shot screen throughout the day, click to video call, supports group chat, screen shares. Good for simulating office presence.
Hello free, ad-hoc video calls, “brandable” URI

HTTP Benchmarking

wrk is a “more modern” ab for hitting individual endpoints
hey is a more minimal alternative, I like its response time histograms. The response status code counts to let you confirm auth/request is correct (200) but also let you see at what point your webserver starts rejecting traffic due to load is a super useful feature.
Locust can define a more real-world usage pattern in python, with multiple concurrent users

Graphviz

An amazing quick reference is https://ncona.com/2020/06/create-diagrams-with-code-using-graphviz/

Docker Compose

Rebuild cluster with updated images:

docker compose pull    # fetches new images
docker compose down    # quits servers, removes from docker
docker compose up -d   # brings cluster back up, uses new images
docker image prune -f  # removes now-unused images

Docker Desktop on MacOS is super obnoxious. Disk access is slower than on Linux (and I think has to round trip through RAM), it randomly drops network shares (sometimes even local disk mounts). I don’t recommend for anything long-running like a home media center. Far better doing so with Linux as a host OS.

Desktop Backups

I use Time Machine for backups on the local network, and Arq for offsite.

For both, I need to exclude a few extra locations that are either transient, or easy to replicate.

/Applications, ~/Applications - Application binaries
/Library MacOS system settings
/System MacOS operating system
/private/var System cache, swap files
~/Downloads Transient storage
~/Library/Application Support/Steam Games
~/Library/Arq Metadata about backups, will re-download from cloud storage if I need to restore backups from Arq
~/Movies Transient storage when travelling
~/VirtualBox VMs Disk images, should manually back up anything in there
~/.asdf Binaries
~/.homebrew, /usr/local Binaries
~/code Programming projects, if it’s not in git it doesn’t exist (Github counts as offsite backup) - excluding the whole thing means I don’t need to deal with direnv/logs/node_modules/vendor/cache/whatever else individually, and some of those get out of hand pretty quickly.

Arq 6

The new Arq 6 has a different backup selector. By default it backs up /Applications, /Users, and /Library/Application Support, and has an extensive exclusion list (including Cache, Downloads, …).

For my own preferences, /Library/Application Support is only a few hundred MB so I kept it in, and added the following excluded items:

Applications (covers both paths)
VirtualBox VMs
.direnv
.asdf
Movies
Library/Application Support/Steam
com.docker.docker (where it keeps its containers)
node_modules for Javascript
vendor/cache and vendor/gems for Ruby

Arq 7

Now has an available default backup option of non-system files, being /Users with these exclusions:

.DocumentRevisions-V100
.MobileBackups
.MobileBackups.trash
.Spotlight-V100
.TemporaryItems
.Trash
.Trashes
.dbfseventsd
.dropbox
.dropbox.cache
.fseventsd
.hotfiles.btree
.vol
Backups.backupdb
Cache
Caches
Library/Metadata/CoreSpotlight
Downloads
DerivedData
node_modules
Logs
iTunes/iTunes Media/Downloads
iTunes/iTunes Media/Podcasts
iTunes/Album Artwork
iTunes/Previous iTunes Libraries
Library/Application Support/CrashReporter
Library/Application Support/Dropbox
Library/Application Support/Google
Library/Application Support/MobileSync/Backup
Library/Containers/com.apple.mail/Data/Library/Mail Downloads
Library/Containers/com.apple.mail/Data/DataVaults
Library/Developer
Library/Google/GoogleSoftwareUpdate
Library/Metadata/CoreSpotlight
Library/Mobile Documents
Library/Mirrors
Library/PubSub/Database
Library/PubSub/Downloads
Library/PubSub/Feeds
Library/Safari/Favicon Cache
Library/Safari/Icons.db
Library/Safari/Touch Icons Cache
Library/Safari/WebpageIcons.db
Library/Safari/HistoryIndex.sk
MailData/AvailableFeeds
MailData/BackingStoreUpdateJournal
MailData/Envelope Index
MailData/Envelope Index-journal
MailData/Envelope Index-shm
MailData/Envelope Index-wal

I have added:

Skip items excluded by Time Machine rules
Applications
VirtualBox VMs
.direnv
.asdf
Movies
Library/Application Support/Steam
com.docker.docker (where it keeps its containers)
vendor/cache and vendor/gems for Ruby

TODO: I can probably prune some more space here (Library/Group Containers looks like it might be iCloud related and I can redownload, Library/Caches/Homebrew is good to clear out, and Library/Arq just seems too self-referential to be worth backing up?)

</th> <th><a name=”tech# Text Chat

“Gold Standard” seems to be Slack
Open-source, self-hosted alternative is Mattermost. Extremely similar UX to Slack.
Gamer-marketed Discord ticks most of the same boxes, but is designed for more open communities, and lacks video calls (but does voice calls just fine, and will handle screenshare for a limited number of attendees).
Open-source “slack-with-threads” is Zulip, wants to use a smaller number of channels with consistent team membership, and then create ad-hoc named threads (with their own history) to spin off discussions so that those conversations can be followed easier.
Static Site Generators

I currently use Middleman.

Remote Work

Sneek $10/user, gives a webcam-shot screen throughout the day, click to video call, supports group chat, screen shares. Good for simulating office presence.
Hello free, ad-hoc video calls, “brandable” URI

HTTP Benchmarking

wrk is a “more modern” ab for hitting individual endpoints
hey is a more minimal alternative, I like its response time histograms. The response status code counts to let you confirm auth/request is correct (200) but also let you see at what point your webserver starts rejecting traffic due to load is a super useful feature.
Locust can define a more real-world usage pattern in python, with multiple concurrent users

Graphviz

An amazing quick reference is https://ncona.com/2020/06/create-diagrams-with-code-using-graphviz/

Docker Compose

Rebuild cluster with updated images:

docker compose pull    # fetches new images
docker compose down    # quits servers, removes from docker
docker compose up -d   # brings cluster back up, uses new images
docker image prune -f  # removes now-unused images

Desktop Backups

I use Time Machine for backups on the local network, and Arq for offsite.

For both, I need to exclude a few extra locations that are either transient, or easy to replicate.

/Applications, ~/Applications - Application binaries
/Library MacOS system settings
/System MacOS operating system
/private/var System cache, swap files
~/Downloads Transient storage
~/Library/Application Support/Steam Games
~/Library/Arq Metadata about backups, will re-download from cloud storage if I need to restore backups from Arq
~/Movies Transient storage when travelling
~/VirtualBox VMs Disk images, should manually back up anything in there
~/.asdf Binaries
~/.homebrew, /usr/local Binaries
~/code Programming projects, if it’s not in git it doesn’t exist (Github counts as offsite backup) - excluding the whole thing means I don’t need to deal with direnv/logs/node_modules/vendor/cache/whatever else individually, and some of those get out of hand pretty quickly.

Arq 6

For my own preferences, /Library/Application Support is only a few hundred MB so I kept it in, and added the following excluded items:

Applications (covers both paths)
VirtualBox VMs
.direnv
.asdf
Movies
Library/Application Support/Steam
com.docker.docker (where it keeps its containers)
node_modules for Javascript
vendor/cache and vendor/gems for Ruby

Arq 7

Now has an available default backup option of non-system files, being /Users with these exclusions:

.DocumentRevisions-V100
.MobileBackups
.MobileBackups.trash
.Spotlight-V100
.TemporaryItems
.Trash
.Trashes
.dbfseventsd
.dropbox
.dropbox.cache
.fseventsd
.hotfiles.btree
.vol
Backups.backupdb
Cache
Caches
Library/Metadata/CoreSpotlight
Downloads
DerivedData
node_modules
Logs
iTunes/iTunes Media/Downloads
iTunes/iTunes Media/Podcasts
iTunes/Album Artwork
iTunes/Previous iTunes Libraries
Library/Application Support/CrashReporter
Library/Application Support/Dropbox
Library/Application Support/Google
Library/Application Support/MobileSync/Backup
Library/Containers/com.apple.mail/Data/Library/Mail Downloads
Library/Containers/com.apple.mail/Data/DataVaults
Library/Developer
Library/Google/GoogleSoftwareUpdate
Library/Metadata/CoreSpotlight
Library/Mobile Documents
Library/Mirrors
Library/PubSub/Database
Library/PubSub/Downloads
Library/PubSub/Feeds
Library/Safari/Favicon Cache
Library/Safari/Icons.db
Library/Safari/Touch Icons Cache
Library/Safari/WebpageIcons.db
Library/Safari/HistoryIndex.sk
MailData/AvailableFeeds
MailData/BackingStoreUpdateJournal
MailData/Envelope Index
MailData/Envelope Index-journal
MailData/Envelope Index-shm
MailData/Envelope Index-wal

I have added:

Skip items excluded by Time Machine rules
Applications
VirtualBox VMs
.direnv
.asdf
Movies
Library/Application Support/Steam
com.docker.docker (where it keeps its containers)
vendor/cache and vendor/gems for Ruby

” class=”anchor”> </th></tr>

  <tr><th>commandline# fzf

fzf is an incremental search tool for the terminal.

Here’s a good summary of setup and usage. Just out of the gate, adding it to zsh and knowing that you can type and hit ctrl+r to match is good enough to get rolling.

fzf-tab replaces the zsh default tab completion to just naively run through fzf, it’s also great.

Local Development Tooling

Just a summary of the tools I’m using to manage my local development environment.

2020

yadm, asdf, direnv.

2025

chezmoi is a dotfiles manager that helps me keep my overall shell environment in sync across machines. It’s a minor improvement over yadm, templates are slightly better for mac/linux and personal/work alternates, and I have a preference for file copies over symlinks - if I just have a one-off config change on one machine, I can still do a config sync without needing to juggle git merges.
mise is a manager for version managers, as well as a shell environment manager. I primarily work in Ruby, but dabble all over, and it makes a lot of sense to replace rvm/rbenv, nvm, pyenv, etc with a single tool that has a consistent interface - and mise is an improvement over asdf in that it’s zero-config out of the box and smart enough to not need to manually juggle plugin setup. mise install in a Rails project will just do the right thing, setting up the ruby & node versions the repo is already tagged with. It also handles environment variables, and while I like the simplicity of layout ruby coming from direnv, it’s easy enough to work with a mise preset script to bootstrap project dirs.

To investigate

devenv builds atop Nix to set up 100% reproducible local environments/dependencies.

Git

Tips and Tricks

Alias to list recently changed branches

Additional Tooling

delta as an alternate diff viewer
p4merge as my default mergetool
GitUI is a fast interactive client for the terminal

</th> <th><a name=”commandline# fzf

fzf is an incremental search tool for the terminal.

Here’s a good summary of setup and usage. Just out of the gate, adding it to zsh and knowing that you can type and hit ctrl+r to match is good enough to get rolling.

fzf-tab replaces the zsh default tab completion to just naively run through fzf, it’s also great.

Local Development Tooling

Just a summary of the tools I’m using to manage my local development environment.

2020

yadm, asdf, direnv.

2025

chezmoi is a dotfiles manager that helps me keep my overall shell environment in sync across machines. It’s a minor improvement over yadm, templates are slightly better for mac/linux and personal/work alternates, and I have a preference for file copies over symlinks - if I just have a one-off config change on one machine, I can still do a config sync without needing to juggle git merges.
mise is a manager for version managers, as well as a shell environment manager. I primarily work in Ruby, but dabble all over, and it makes a lot of sense to replace rvm/rbenv, nvm, pyenv, etc with a single tool that has a consistent interface - and mise is an improvement over asdf in that it’s zero-config out of the box and smart enough to not need to manually juggle plugin setup. mise install in a Rails project will just do the right thing, setting up the ruby & node versions the repo is already tagged with. It also handles environment variables, and while I like the simplicity of layout ruby coming from direnv, it’s easy enough to work with a mise preset script to bootstrap project dirs.

To investigate

devenv builds atop Nix to set up 100% reproducible local environments/dependencies.

Git

Tips and Tricks

Alias to list recently changed branches

Additional Tooling

delta as an alternate diff viewer
p4merge as my default mergetool
GitUI is a fast interactive client for the terminal

” class=”anchor”> </th></tr>

  <tr><th>gaming# Root Organizer

Organizing some 3d printed trays for Root to fit current expansions in the main box.

TODO: Use this OpenSCAD library!!!

Measurements

Core box internals are 278x215mm, ~66mm depth. With the Otter card offer off on the long side of the box (sadly won’t fit across a short edge, I’m totally considering trimming it a few mm), I can fit two rows of 95mm trays.

As to depth, faction boards/rules 20mm, single map 17mm, and 25mm-deep trays will actually fit everything inside.

Components

Considering going all-in on multi-box set. With 90mm long trays at full 66mm depth, can slot in a deck of cards horizontally, or make faction boxes with a handful of cards + all their meeples and tokens. Base game box would fit 24 trays (3x8) at 25mm spacing. Actually would have a similar vibe to just stuffing it full of deck boxes, depending on if 25x90x66 is sufficient volume for everything.

Base Game
- Marquise de Cat
- Eyrie (6 cards)
- Woodland Alliance
- Vagabond (15 quest cards, multiple character cards, with Vagabond Pack)
Riverfolk
- Lizard Cult
- Riverfolk Company (card holder)
Underworld
- Underground Duchy
- Corvid Conspiracy
Marauder
- Lord of the Hundreds (8 cards, 1 die)
- Keepers in Iron
Homeland
- Twilight Council
- Lilypad Diaspora
- Knaves of Deepwood (overlap w/ Vagabond?)
Other
- 4 dice, 4 ruins, 12 clearing items
- Base shared deck
- Exiles & Partisans shared deck
- Squires & Disciples shared deck (from Homeland)
- Landmarks
- Hirelings (Base + Marauder, Riverfolk, Underworld, Homeland hirelings)
- Resin Clearings?

First box then would have 12 factions (assuming Knaves and Vagabonds share a tray), 3 decks, landmarks, a few hirelings (meeple swap for faction hirelings, so we just need to stock up the neutral hirelings), and misc components (dice etc). That’s maybe 18-20 trays worth, could afford to spend extra space on the resin clearings.

Second box would then be for maps (3 now with Homeland expansion!) and faction boards? Is that an argument in favor of the cardstock faction boards to save space?

I think the riverfolk card block and tunnels can fit alongside the map boards, which would be convenient for space.

Tray Sizing

Baseline 95mm long, 25mm deep, varying width as follows:
- Cats - 60mm *
- Birds - 40mm
- Mice - 35mm
- Vagabond - 40mm, better if bigger + has more dividers, also needs 8.5mm of cards
- Lizard - 55mm
- Otter - 35mm
- Crow - 35mm
- Mole - 55mm (or 70mm incl. cards, 27mm height) (5.5mm of cards)
- Resin Clearing Markers - 90mm max
- Exile Deck + Eyrie + Faction Overviews - 70mm
- Dice
- Shop Items
- Tower/Raft
- Tunnels - 102mm, awkward.
- Alt. Exile Deck - 70mm
Layout (275mm width x 2 rows of 95mm):
- 275mm: Exile Deck, Cats, Birds, Mice, Otter, Crow
- 270mm: Vagabond+Cards, Lizard, Mole, Clearing
  - Maybe could bump clearing another 5mm and squeeze in dice/shop around them? Also possible to flex on height? Still leaves tower/raft unaccounted for, and tunnels with nowhere to live.
  - Tunnels sideways need 30.2mm (so 31mm total height on trays) which would free up a lot of space restrictions. 35mm height would I think let us get clearing markers vertically. Yellow/red fit a 55mm tray but want a 45/50mm subdivision. Orange with a 35/60mm subdivision leaves plenty of room for dice/shop/tower/raft. That only bumps up overall width from current resin measure +20mm. Tunnels still an awkward item for inside a tray.

Round Two

Baseline 95mm long, 35mm deep, 2x 275mm rows

Overall 3x25, 3x35, 3x45 for factions+resin, 105, 70, 20? for remaining components. Still need to account for vagabond cards.

Row 1, 270mm
- Exile Deck + Eyrie + Faction Overviews - 70mm, could print 75mm and include shop items?
- Cats - 45mm
- Mole - 45mm
- Birds - 35mm
- Mice - 25mm
- Otter - 25mm
- Crow - 25mm
Row 2, 255mm
- Resin Clearing Markers - 45mm (!!!)
- Lizard - 35mm
- Vagabond - 25mm, but 35mm with a divider is better - 40mm for 9x figs, nesting trays for 12 ruin/16 start/16 influence (or maybe look into actual use, and one tray for 1 Vaga, with a lower tray for a 2 Vaga game?). Divider only needs to go part way, and then nesting tray can sit atop it.
- Alt. Exile Deck - ~~70mm~~ 105mm TODO: Test this
  - Tunnels on top
  - Extra Dice in extra 35mm
- Spare 20mm? TODO: Test this
  - Dice
  - Shop Items
  - Tower/Raft

Round Three - Marauders and Hirelings

Need to account for two new factions, plus mercenaries. At this point worth considering deeper trays across two boxes. One box with factions (individual trays, plus all faction boards on top), speculating 25-26mm for faction boards gives 40mm tray depth (up from 35 in round two) and not needing to account for clearings/decks/dice. Maybe swing hireling pieces, or trade extra height for clockwork player boards? Then the second box with both map boards, clearings, decks, dice, landmarks, etc. Keeping in mind that expansion boxes are not as deep as the main box. Push comes to shove maybe keep hirelings (and clockwork?) in the hireling box :)

WIP TODOS

Print alternate trays at 35mm height, re-measure factions? Also maybe consider switching from two rows @ 95mm to three rows at 65mm wide?
- 10 total factions, warrior counts at 2x25, 3x20, 3x15, 2x10 - that’s x155 total, box has capacity for 5.5mm per. Currently aiming to scale 3mm per, so cats would have 65mm by 75mm available. Slightly larger compared to Round 2’s 45x95 (area 4875 vs 4275) and taller to boot.
  - 25 Cats/Lizards/(Duchy given lots of tokens)
  - 20 Birds/Rats
  - 15 Badgers/Crows/Otters
  - 10 Alliance/Vagabond
- Also worth considering that Eyrie, Vagabond, Duchy, and Hundreds all have some cards - those plus Otter’s card stand would be a nice-to-have to fit in the base box.
- Finally, also consider the draft cards and mercenary mats - I don’t think it’s actually worth managing the space for distinct mercenary meeples (just repurpose the faction meeples), but unique mercenaries like the moose should be included. Maybe embiggen the Vagabond tray?
Another idea: Splitting the box into a 3x3 grid makes each grid large enough to hold cards. This allows those that include cards (birds/rats/duchy/vagabond) to have their own cards in their tray, meeples and tokens on top; another tray for the actual deck + drafting cards, and a half-size tray for dice & global items
- 10 factions + deck + dice/items implies splitting 3 fullsize trays to half size, might be easier to run 5x full size and 6x two-thirds size, 65x60.
- And actually, looks like making one tray half-wide but double-tall (landscape) would be a good side for closed tunnels on the mountain map, while also holding other common components like dice, default items, etc.
Figure out where subdivisions make sense inside trays, to separate the warriors/tokens.
Measure up what impact we’d have putting the tunnel tokens alongside the edge of the box, beside the Otter’s card holder, if we need to shrink tray lengths.

From various thinking above, best design currently looks like splitting the box area 3x3 to be individually big enough for the common card deck + drafting/setup cards, and factions that require cards to function, subdivided to appropriate sizes.

That would be Eyrie (leaders), Vagabond (vagabonds and quests), the Duchy (nobles), and Lord of Hundreds (moods), all of which have enough meeples and tokens to justify the large bin space.

That leaves the other 4 sections of primary subdivision to hold the remaining 6 factions, and common components (dice, landmarks, etc).

HOWEVER, there’s no way we’ll fit all the faction boards + maps in a single box anyway, so maybe it’s better to just size factions as-needed, make sure we’ve got room in the core box for all the minis (incl. hirelings), and then leave an expansion box to manage the two maps and all the common components (hireling boards, resin clearing markers, both decks, etc).

Component Measurements

Global

4 Dice
12 Items (sq)

Cards: 63x88mm

Lake/Mountain Maps

1 Ferry
1 Tower
6 Closed Paths

Buildings/Tokens 10.6/5mm thick All factions: 1 VP token (18.3mm sq)

Marquise de Cat

25 Warriors (9mm, 16.7mm, 22mm)
27 Tokens: 18 buildings, 9 wood

Eyrie

20 Warriors (9mm, 18.1mm, 22mm)
7 Buildings
4 cards

Alliance

10 Warriors (9mm, 19.5mm, 19mm)
13 Tokens: 3 bases, 10 outrage

Vagabond

9 Meeples: 9 Vagabond Pawns (var.)
17 S items (sq)
8 R items (sq)
12 (14) Relationships (sq)
4 Ruins
Many cards

Lizard Cult

25 Warriors (8.3mm, 17.5mm, 19.5mm)
16 Buildings

Riverfolk Company

15 Warriors (7.9mm, 16mm, 20mm)
9 Tokens
3 gems

Corvids

15 Warriors (9.2mm, 16.8mm, 19.2mm)
8 Tokens

The Duchy

29 Meeples: 20 warriors, 9 crowns
10 Tokens: 6 buildings, 3 tokens, 1 burrow (large)
9 cards

Keepers in Iron

15 Meeples: 15 warriors
21 Tokens: 12 relic, 3 caravan, 5 waystation, 1 mission

Lord of the Hundreds

21 Meeples: 20 warriors, 1 warlord
11 Tokens: 6 storage, 5 mob
8 cards
1 dice

2026 Rework

https://counterslayer.com/

Box interior: 134mm by 215mm, 52mm depth
Warriors: 9.2mm thick, 26mm square
Tokens: 2.1mm thick, 20.1mm square
Dice: 16mm cube
Hirelings
- 1 die (1 extra)
- 12 tokens
- 3 unlock markers 36x55mm
Cats: 12
Frogs: 10+5
Ducks: 9+4
Moles: 8+3
Badgers: 6+6
Bats: 8
Rats 6 + one die
Crows: 5
Musicians: 5
Birds: 5
Alliance: 4
Lizards: 4
Porcupines: 4
Bear: 1+3
Otters 33 tall 40 wide
Moose 32 tall, 29.5 wide
Landmarks 51mm wide, stack 9 edge on
Roll20 Character Tokens

Use tokenstamp to turn character art into decent character icons, with transparency etc.

For consistency in my current campaign:

Border: row 5, first
Fade: row 2, first
Border Tint: RGB 45,145,175
BG Color: RGB 95,125,135

Queendomino

Custom Insert

Box interior: 256x256mm (approx)

Dominos: (48 king, 48 queen, 12 giant + 17 quest)

79.9mm long
40.3mm short
(8) 25.4mm deep

Dispenser then must handle 60 tiles (base + giants), 192mm internal height, plus 10mm to reach underneath. Secondary dispenser for 48 tiles (second base), 155mm + 10mm.

Castle Tiles (9)

40.2x40.2
(4) 13mm deep

Buildings (32 queen)

40.1x40.1mm
(8) 17.6mm deep

Towers (15 queen)

10.1mm front-back
15mm base, 12.7mm top (can alternate for slightly tighter packing)
24.8mm high

Gloomhaven

Fan Resources

There’s an extended deck of battle goals for more variety.

Expansion

Forgotten Circles got a lot of changes for the Diviner class in a 2nd printing. </th> <th><a name=”gaming# Root Organizer

Organizing some 3d printed trays for Root to fit current expansions in the main box.

TODO: Use this OpenSCAD library!!!

Measurements

As to depth, faction boards/rules 20mm, single map 17mm, and 25mm-deep trays will actually fit everything inside.

Components

Base Game
- Marquise de Cat
- Eyrie (6 cards)
- Woodland Alliance
- Vagabond (15 quest cards, multiple character cards, with Vagabond Pack)
Riverfolk
- Lizard Cult
- Riverfolk Company (card holder)
Underworld
- Underground Duchy
- Corvid Conspiracy
Marauder
- Lord of the Hundreds (8 cards, 1 die)
- Keepers in Iron
Homeland
- Twilight Council
- Lilypad Diaspora
- Knaves of Deepwood (overlap w/ Vagabond?)
Other
- 4 dice, 4 ruins, 12 clearing items
- Base shared deck
- Exiles & Partisans shared deck
- Squires & Disciples shared deck (from Homeland)
- Landmarks
- Hirelings (Base + Marauder, Riverfolk, Underworld, Homeland hirelings)
- Resin Clearings?

Second box would then be for maps (3 now with Homeland expansion!) and faction boards? Is that an argument in favor of the cardstock faction boards to save space?

I think the riverfolk card block and tunnels can fit alongside the map boards, which would be convenient for space.

Tray Sizing

Baseline 95mm long, 25mm deep, varying width as follows:
- Cats - 60mm *
- Birds - 40mm
- Mice - 35mm
- Vagabond - 40mm, better if bigger + has more dividers, also needs 8.5mm of cards
- Lizard - 55mm
- Otter - 35mm
- Crow - 35mm
- Mole - 55mm (or 70mm incl. cards, 27mm height) (5.5mm of cards)
- Resin Clearing Markers - 90mm max
- Exile Deck + Eyrie + Faction Overviews - 70mm
- Dice
- Shop Items
- Tower/Raft
- Tunnels - 102mm, awkward.
- Alt. Exile Deck - 70mm
Layout (275mm width x 2 rows of 95mm):
- 275mm: Exile Deck, Cats, Birds, Mice, Otter, Crow
- 270mm: Vagabond+Cards, Lizard, Mole, Clearing
  - Maybe could bump clearing another 5mm and squeeze in dice/shop around them? Also possible to flex on height? Still leaves tower/raft unaccounted for, and tunnels with nowhere to live.
  - Tunnels sideways need 30.2mm (so 31mm total height on trays) which would free up a lot of space restrictions. 35mm height would I think let us get clearing markers vertically. Yellow/red fit a 55mm tray but want a 45/50mm subdivision. Orange with a 35/60mm subdivision leaves plenty of room for dice/shop/tower/raft. That only bumps up overall width from current resin measure +20mm. Tunnels still an awkward item for inside a tray.

Round Two

Baseline 95mm long, 35mm deep, 2x 275mm rows

Overall 3x25, 3x35, 3x45 for factions+resin, 105, 70, 20? for remaining components. Still need to account for vagabond cards.

Row 1, 270mm
- Exile Deck + Eyrie + Faction Overviews - 70mm, could print 75mm and include shop items?
- Cats - 45mm
- Mole - 45mm
- Birds - 35mm
- Mice - 25mm
- Otter - 25mm
- Crow - 25mm
Row 2, 255mm
- Resin Clearing Markers - 45mm (!!!)
- Lizard - 35mm
- Vagabond - 25mm, but 35mm with a divider is better - 40mm for 9x figs, nesting trays for 12 ruin/16 start/16 influence (or maybe look into actual use, and one tray for 1 Vaga, with a lower tray for a 2 Vaga game?). Divider only needs to go part way, and then nesting tray can sit atop it.
- Alt. Exile Deck - ~~70mm~~ 105mm TODO: Test this
  - Tunnels on top
  - Extra Dice in extra 35mm
- Spare 20mm? TODO: Test this
  - Dice
  - Shop Items
  - Tower/Raft

Round Three - Marauders and Hirelings

WIP TODOS

Print alternate trays at 35mm height, re-measure factions? Also maybe consider switching from two rows @ 95mm to three rows at 65mm wide?
- 10 total factions, warrior counts at 2x25, 3x20, 3x15, 2x10 - that’s x155 total, box has capacity for 5.5mm per. Currently aiming to scale 3mm per, so cats would have 65mm by 75mm available. Slightly larger compared to Round 2’s 45x95 (area 4875 vs 4275) and taller to boot.
  - 25 Cats/Lizards/(Duchy given lots of tokens)
  - 20 Birds/Rats
  - 15 Badgers/Crows/Otters
  - 10 Alliance/Vagabond
- Also worth considering that Eyrie, Vagabond, Duchy, and Hundreds all have some cards - those plus Otter’s card stand would be a nice-to-have to fit in the base box.
- Finally, also consider the draft cards and mercenary mats - I don’t think it’s actually worth managing the space for distinct mercenary meeples (just repurpose the faction meeples), but unique mercenaries like the moose should be included. Maybe embiggen the Vagabond tray?
Another idea: Splitting the box into a 3x3 grid makes each grid large enough to hold cards. This allows those that include cards (birds/rats/duchy/vagabond) to have their own cards in their tray, meeples and tokens on top; another tray for the actual deck + drafting cards, and a half-size tray for dice & global items
- 10 factions + deck + dice/items implies splitting 3 fullsize trays to half size, might be easier to run 5x full size and 6x two-thirds size, 65x60.
- And actually, looks like making one tray half-wide but double-tall (landscape) would be a good side for closed tunnels on the mountain map, while also holding other common components like dice, default items, etc.
Figure out where subdivisions make sense inside trays, to separate the warriors/tokens.
Measure up what impact we’d have putting the tunnel tokens alongside the edge of the box, beside the Otter’s card holder, if we need to shrink tray lengths.

That would be Eyrie (leaders), Vagabond (vagabonds and quests), the Duchy (nobles), and Lord of Hundreds (moods), all of which have enough meeples and tokens to justify the large bin space.

That leaves the other 4 sections of primary subdivision to hold the remaining 6 factions, and common components (dice, landmarks, etc).

Component Measurements

Global

4 Dice
12 Items (sq)

Cards: 63x88mm

Lake/Mountain Maps

1 Ferry
1 Tower
6 Closed Paths

Buildings/Tokens 10.6/5mm thick All factions: 1 VP token (18.3mm sq)

Marquise de Cat

25 Warriors (9mm, 16.7mm, 22mm)
27 Tokens: 18 buildings, 9 wood

Eyrie

20 Warriors (9mm, 18.1mm, 22mm)
7 Buildings
4 cards

Alliance

10 Warriors (9mm, 19.5mm, 19mm)
13 Tokens: 3 bases, 10 outrage

Vagabond

9 Meeples: 9 Vagabond Pawns (var.)
17 S items (sq)
8 R items (sq)
12 (14) Relationships (sq)
4 Ruins
Many cards

Lizard Cult

25 Warriors (8.3mm, 17.5mm, 19.5mm)
16 Buildings

Riverfolk Company

15 Warriors (7.9mm, 16mm, 20mm)
9 Tokens
3 gems

Corvids

15 Warriors (9.2mm, 16.8mm, 19.2mm)
8 Tokens

The Duchy

29 Meeples: 20 warriors, 9 crowns
10 Tokens: 6 buildings, 3 tokens, 1 burrow (large)
9 cards

Keepers in Iron

15 Meeples: 15 warriors
21 Tokens: 12 relic, 3 caravan, 5 waystation, 1 mission

Lord of the Hundreds

21 Meeples: 20 warriors, 1 warlord
11 Tokens: 6 storage, 5 mob
8 cards
1 dice

2026 Rework

https://counterslayer.com/

Box interior: 134mm by 215mm, 52mm depth
Warriors: 9.2mm thick, 26mm square
Tokens: 2.1mm thick, 20.1mm square
Dice: 16mm cube
Hirelings
- 1 die (1 extra)
- 12 tokens
- 3 unlock markers 36x55mm
Cats: 12
Frogs: 10+5
Ducks: 9+4
Moles: 8+3
Badgers: 6+6
Bats: 8
Rats 6 + one die
Crows: 5
Musicians: 5
Birds: 5
Alliance: 4
Lizards: 4
Porcupines: 4
Bear: 1+3
Otters 33 tall 40 wide
Moose 32 tall, 29.5 wide
Landmarks 51mm wide, stack 9 edge on
Roll20 Character Tokens

Use tokenstamp to turn character art into decent character icons, with transparency etc.

For consistency in my current campaign:

Border: row 5, first
Fade: row 2, first
Border Tint: RGB 45,145,175
BG Color: RGB 95,125,135

Queendomino

Custom Insert

Box interior: 256x256mm (approx)

Dominos: (48 king, 48 queen, 12 giant + 17 quest)

79.9mm long
40.3mm short
(8) 25.4mm deep

Dispenser then must handle 60 tiles (base + giants), 192mm internal height, plus 10mm to reach underneath. Secondary dispenser for 48 tiles (second base), 155mm + 10mm.

Castle Tiles (9)

40.2x40.2
(4) 13mm deep

Buildings (32 queen)

40.1x40.1mm
(8) 17.6mm deep

Towers (15 queen)

10.1mm front-back
15mm base, 12.7mm top (can alternate for slightly tighter packing)
24.8mm high

Gloomhaven

Fan Resources

There’s an extended deck of battle goals for more variety.

Expansion

Forgotten Circles got a lot of changes for the Diviner class in a 2nd printing. “ class=”anchor”> </th></tr>

  <tr><th>management# Job Levels

Buffer is exceptionally open about their salaries (the whole company has salaries posted publicly). They’ve shared blog posts about their 2019 change to their compensation formula and also a nice post about career paths and what that looks like between makers & Managers.

</th> <th><a name=”management# Job Levels

” class=”anchor”> </th></tr>

  <tr><th>web# Stimulus JS

Main Site

Simple how-to tutorial using Stimulus to handle loading an HTML fragment from the server (Rails).

BetterStimulus has a collection of guidelines and best-practices.

example of testing stimulus with jest

Modern Javascript Frameworks

React
VueJS
Svelte single-file components, plain-looking javascript (with just local vars, compiler does the magic)

Middleman

How-to article on setting up Middleman with Tailwind CSS, so that you can @apply styles from Tailwind against auto-rendered content (like markdown->html). I’m doing exactly that here! </th> <th><a name=”web# Stimulus JS

Main Site

Simple how-to tutorial using Stimulus to handle loading an HTML fragment from the server (Rails).

BetterStimulus has a collection of guidelines and best-practices.

example of testing stimulus with jest

Modern Javascript Frameworks

React
VueJS
Svelte single-file components, plain-looking javascript (with just local vars, compiler does the magic)

Middleman

  <tr><th>Ruby# Tracking Unused Code

Ruby tools for tracking unused code:

debride does some static analysis
rcov builds reports of what lines get executed arbitrarily (typically would be run on your test suite)
coverband runs rcov in production to show what code is actually being used by users, rather than just tests.

Ruby Testing

General

vcr to stub out remote calls and allow unit tests with network calls

Rspec

rspec-mocks for test doubles/mocks
rspec-tracer to help diagnose flaky specs

Test-Unit

rr for test doubles/mocks

Quality Gems

TODO: Make a blog post out of this.

A shout-out to some gems/tools that have a well-deserved place in my Rails toolbox.

Pagy (ActiveRecord pagination, focused on performance)
Pry (debugger)
Rbspy (sampling profiler) [Rust]
rack-mini-profiler, flamegraph, stackprof, memory_profiler (performance analysis tooling) - (reference custom instrumentation)
rubocop / rufo
Hamlit
Fivemat (rspec/cucumber formatter, progress but one line per file)
Cells
d3js (chart-buildling library) [JS]
sidekiq
Bullet (detects N+1 queries) - Compare to newcomer prosopite
Insomnia (HTTP api tool)
HTTP.rb (compare Typhoeus, others as a blog post?)

And some TODOs that I want to try out.

api_struct for writing API clients, as a step up from my usual bit of raw httprb and JSON->hash data bags on the response.

Evil Martians

Posted an article about their Gemfile of dreams with a summary of the things they use frequently and why. (Note: this got an update April 2026)

</th> <th><a name=”Ruby# Tracking Unused Code

Ruby tools for tracking unused code:

debride does some static analysis
rcov builds reports of what lines get executed arbitrarily (typically would be run on your test suite)
coverband runs rcov in production to show what code is actually being used by users, rather than just tests.

Ruby Testing

General

vcr to stub out remote calls and allow unit tests with network calls

Rspec

rspec-mocks for test doubles/mocks
rspec-tracer to help diagnose flaky specs

Test-Unit

rr for test doubles/mocks

Quality Gems

TODO: Make a blog post out of this.

A shout-out to some gems/tools that have a well-deserved place in my Rails toolbox.

Pagy (ActiveRecord pagination, focused on performance)
Pry (debugger)
Rbspy (sampling profiler) [Rust]
rack-mini-profiler, flamegraph, stackprof, memory_profiler (performance analysis tooling) - (reference custom instrumentation)
rubocop / rufo
Hamlit
Fivemat (rspec/cucumber formatter, progress but one line per file)
Cells
d3js (chart-buildling library) [JS]
sidekiq
Bullet (detects N+1 queries) - Compare to newcomer prosopite
Insomnia (HTTP api tool)
HTTP.rb (compare Typhoeus, others as a blog post?)

And some TODOs that I want to try out.

api_struct for writing API clients, as a step up from my usual bit of raw httprb and JSON->hash data bags on the response.

Evil Martians

Posted an article about their Gemfile of dreams with a summary of the things they use frequently and why. (Note: this got an update April 2026)

” class=”anchor”> </th></tr>

  <tr><th>boardgame# Tiny Epic Dinosaurs

Box dimensions: 108mm x 167mm, approx 35mm deep Clearance for manual: 1.8mm

Cards: 41x63.5mm

A. Research 11.2mm
B. Public contracts 9.5mm
C. Private contracts 2.7mm
D. Rivals 2.7mm
B+C combined 11.8mm
A+D combined 13.7mm
B+C+D 14.5mm
!D Rivals stored underneath insert, with setup tokens

Panels: 88x126.3mm

Base game 4.9mm
Base+Lab 6.1mm

Lab + Tokens underneath: 5.5mm

Token wells:

Insert w/ 6 wells for gameplay (15mm, plus 7mm topper for panels):
- 15ea Steg, Brach, Velo, Allo
- 17 unique
- 26 barriers
Underlay w/ wells for setup (10mm):
- 4 players: 5 ranchers, 3 tokens
- Turn markers

15mm die

Height budget

35mm Box
- 2mm manual
- 33mm insert
  - 21mm upper tray
    - 6.5mm panels
    - 14mm token supply wells
    - 0.5mm base
  - 11.5mm lower tray setup wells
  - 0.5mm base
- Card slot surround, no higher than 20mm
- ?Slot for D6 between card decks

Spirit Island

core/earth box: 280x280x75mm branch box: 160x280x50

Island Boards: bounding box: 220x280, complex; 15mm deep

Spirit Boards: 230x153x (37); 53mm Invader Board: base: 147x161x8mm branch: 161x210x5mm

Adversary/Scenario:

153x101x9mm

Cards: Mini: Invader: 44x68x6mm (4mm earth reminders) Normal: 89x64mm Base: 47mm Branch, Promo: 53mm Earth: 59mm

Player Components:

13 presence
6 reminder
3x $3, 5x $1
4 fear
1 beast, 1 disease
1 blight, 6 dahan, 1 city, 1 town
2 invaders (first turn) 90x45x22mm tray is sufficient other ‘default’ tray sizes:
- 68x90
- 69x82
- 69x90

But also don’t forget:

additional fear, beast, disease, wilds, unrest, badlands, scenario
money
elements
adversary reminders
gameplay pool of dahan, blight, invaders, towns, cities

Root Organizer

Organizing some 3d printed trays for Root to fit current expansions in the main box.

TODO: Use this OpenSCAD library!!!

Measurements

As to depth, faction boards/rules 20mm, single map 17mm, and 25mm-deep trays will actually fit everything inside.

Components

Base Game
- Marquise de Cat
- Eyrie (6 cards)
- Woodland Alliance
- Vagabond (15 quest cards, multiple character cards, with Vagabond Pack)
Riverfolk
- Lizard Cult
- Riverfolk Company (card holder)
Underworld
- Underground Duchy
- Corvid Conspiracy
Marauder
- Lord of the Hundreds (8 cards, 1 die)
- Keepers in Iron
Homeland
- Twilight Council
- Lilypad Diaspora
- Knaves of Deepwood (overlap w/ Vagabond?)
Other
- 4 dice, 4 ruins, 12 clearing items
- Base shared deck
- Exiles & Partisans shared deck
- Squires & Disciples shared deck (from Homeland)
- Landmarks
- Hirelings (Base + Marauder, Riverfolk, Underworld, Homeland hirelings)
- Resin Clearings?

Second box would then be for maps (3 now with Homeland expansion!) and faction boards? Is that an argument in favor of the cardstock faction boards to save space?

I think the riverfolk card block and tunnels can fit alongside the map boards, which would be convenient for space.

Tray Sizing

Baseline 95mm long, 25mm deep, varying width as follows:
- Cats - 60mm *
- Birds - 40mm
- Mice - 35mm
- Vagabond - 40mm, better if bigger + has more dividers, also needs 8.5mm of cards
- Lizard - 55mm
- Otter - 35mm
- Crow - 35mm
- Mole - 55mm (or 70mm incl. cards, 27mm height) (5.5mm of cards)
- Resin Clearing Markers - 90mm max
- Exile Deck + Eyrie + Faction Overviews - 70mm
- Dice
- Shop Items
- Tower/Raft
- Tunnels - 102mm, awkward.
- Alt. Exile Deck - 70mm
Layout (275mm width x 2 rows of 95mm):
- 275mm: Exile Deck, Cats, Birds, Mice, Otter, Crow
- 270mm: Vagabond+Cards, Lizard, Mole, Clearing
  - Maybe could bump clearing another 5mm and squeeze in dice/shop around them? Also possible to flex on height? Still leaves tower/raft unaccounted for, and tunnels with nowhere to live.
  - Tunnels sideways need 30.2mm (so 31mm total height on trays) which would free up a lot of space restrictions. 35mm height would I think let us get clearing markers vertically. Yellow/red fit a 55mm tray but want a 45/50mm subdivision. Orange with a 35/60mm subdivision leaves plenty of room for dice/shop/tower/raft. That only bumps up overall width from current resin measure +20mm. Tunnels still an awkward item for inside a tray.

Round Two

Baseline 95mm long, 35mm deep, 2x 275mm rows

Overall 3x25, 3x35, 3x45 for factions+resin, 105, 70, 20? for remaining components. Still need to account for vagabond cards.

Row 1, 270mm
- Exile Deck + Eyrie + Faction Overviews - 70mm, could print 75mm and include shop items?
- Cats - 45mm
- Mole - 45mm
- Birds - 35mm
- Mice - 25mm
- Otter - 25mm
- Crow - 25mm
Row 2, 255mm
- Resin Clearing Markers - 45mm (!!!)
- Lizard - 35mm
- Vagabond - 25mm, but 35mm with a divider is better - 40mm for 9x figs, nesting trays for 12 ruin/16 start/16 influence (or maybe look into actual use, and one tray for 1 Vaga, with a lower tray for a 2 Vaga game?). Divider only needs to go part way, and then nesting tray can sit atop it.
- Alt. Exile Deck - ~~70mm~~ 105mm TODO: Test this
  - Tunnels on top
  - Extra Dice in extra 35mm
- Spare 20mm? TODO: Test this
  - Dice
  - Shop Items
  - Tower/Raft

Round Three - Marauders and Hirelings

WIP TODOS

Print alternate trays at 35mm height, re-measure factions? Also maybe consider switching from two rows @ 95mm to three rows at 65mm wide?
- 10 total factions, warrior counts at 2x25, 3x20, 3x15, 2x10 - that’s x155 total, box has capacity for 5.5mm per. Currently aiming to scale 3mm per, so cats would have 65mm by 75mm available. Slightly larger compared to Round 2’s 45x95 (area 4875 vs 4275) and taller to boot.
  - 25 Cats/Lizards/(Duchy given lots of tokens)
  - 20 Birds/Rats
  - 15 Badgers/Crows/Otters
  - 10 Alliance/Vagabond
- Also worth considering that Eyrie, Vagabond, Duchy, and Hundreds all have some cards - those plus Otter’s card stand would be a nice-to-have to fit in the base box.
- Finally, also consider the draft cards and mercenary mats - I don’t think it’s actually worth managing the space for distinct mercenary meeples (just repurpose the faction meeples), but unique mercenaries like the moose should be included. Maybe embiggen the Vagabond tray?
Another idea: Splitting the box into a 3x3 grid makes each grid large enough to hold cards. This allows those that include cards (birds/rats/duchy/vagabond) to have their own cards in their tray, meeples and tokens on top; another tray for the actual deck + drafting cards, and a half-size tray for dice & global items
- 10 factions + deck + dice/items implies splitting 3 fullsize trays to half size, might be easier to run 5x full size and 6x two-thirds size, 65x60.
- And actually, looks like making one tray half-wide but double-tall (landscape) would be a good side for closed tunnels on the mountain map, while also holding other common components like dice, default items, etc.
Figure out where subdivisions make sense inside trays, to separate the warriors/tokens.
Measure up what impact we’d have putting the tunnel tokens alongside the edge of the box, beside the Otter’s card holder, if we need to shrink tray lengths.

That would be Eyrie (leaders), Vagabond (vagabonds and quests), the Duchy (nobles), and Lord of Hundreds (moods), all of which have enough meeples and tokens to justify the large bin space.

That leaves the other 4 sections of primary subdivision to hold the remaining 6 factions, and common components (dice, landmarks, etc).

Component Measurements

Global

4 Dice
12 Items (sq)

Cards: 63x88mm

Lake/Mountain Maps

1 Ferry
1 Tower
6 Closed Paths

Buildings/Tokens 10.6/5mm thick All factions: 1 VP token (18.3mm sq)

Marquise de Cat

25 Warriors (9mm, 16.7mm, 22mm)
27 Tokens: 18 buildings, 9 wood

Eyrie

20 Warriors (9mm, 18.1mm, 22mm)
7 Buildings
4 cards

Alliance

10 Warriors (9mm, 19.5mm, 19mm)
13 Tokens: 3 bases, 10 outrage

Vagabond

9 Meeples: 9 Vagabond Pawns (var.)
17 S items (sq)
8 R items (sq)
12 (14) Relationships (sq)
4 Ruins
Many cards

Lizard Cult

25 Warriors (8.3mm, 17.5mm, 19.5mm)
16 Buildings

Riverfolk Company

15 Warriors (7.9mm, 16mm, 20mm)
9 Tokens
3 gems

Corvids

15 Warriors (9.2mm, 16.8mm, 19.2mm)
8 Tokens

The Duchy

29 Meeples: 20 warriors, 9 crowns
10 Tokens: 6 buildings, 3 tokens, 1 burrow (large)
9 cards

Keepers in Iron

15 Meeples: 15 warriors
21 Tokens: 12 relic, 3 caravan, 5 waystation, 1 mission

Lord of the Hundreds

21 Meeples: 20 warriors, 1 warlord
11 Tokens: 6 storage, 5 mob
8 cards
1 dice

2026 Rework

https://counterslayer.com/

Box interior: 134mm by 215mm, 52mm depth
Warriors: 9.2mm thick, 26mm square
Tokens: 2.1mm thick, 20.1mm square
Dice: 16mm cube
Hirelings
- 1 die (1 extra)
- 12 tokens
- 3 unlock markers 36x55mm
Cats: 12
Frogs: 10+5
Ducks: 9+4
Moles: 8+3
Badgers: 6+6
Bats: 8
Rats 6 + one die
Crows: 5
Musicians: 5
Birds: 5
Alliance: 4
Lizards: 4
Porcupines: 4
Bear: 1+3
Otters 33 tall 40 wide
Moose 32 tall, 29.5 wide
Landmarks 51mm wide, stack 9 edge on

</th> <th><a name=”boardgame# Tiny Epic Dinosaurs

Box dimensions: 108mm x 167mm, approx 35mm deep Clearance for manual: 1.8mm

Cards: 41x63.5mm

A. Research 11.2mm
B. Public contracts 9.5mm
C. Private contracts 2.7mm
D. Rivals 2.7mm
B+C combined 11.8mm
A+D combined 13.7mm
B+C+D 14.5mm
!D Rivals stored underneath insert, with setup tokens

Panels: 88x126.3mm

Base game 4.9mm
Base+Lab 6.1mm

Lab + Tokens underneath: 5.5mm

Token wells:

Insert w/ 6 wells for gameplay (15mm, plus 7mm topper for panels):
- 15ea Steg, Brach, Velo, Allo
- 17 unique
- 26 barriers
Underlay w/ wells for setup (10mm):
- 4 players: 5 ranchers, 3 tokens
- Turn markers

15mm die

Height budget

35mm Box
- 2mm manual
- 33mm insert
  - 21mm upper tray
    - 6.5mm panels
    - 14mm token supply wells
    - 0.5mm base
  - 11.5mm lower tray setup wells
  - 0.5mm base
- Card slot surround, no higher than 20mm
- ?Slot for D6 between card decks

Spirit Island

core/earth box: 280x280x75mm branch box: 160x280x50

Island Boards: bounding box: 220x280, complex; 15mm deep

Spirit Boards: 230x153x (37); 53mm Invader Board: base: 147x161x8mm branch: 161x210x5mm

Adversary/Scenario:

153x101x9mm

Cards: Mini: Invader: 44x68x6mm (4mm earth reminders) Normal: 89x64mm Base: 47mm Branch, Promo: 53mm Earth: 59mm

Player Components:

13 presence
6 reminder
3x $3, 5x $1
4 fear
1 beast, 1 disease
1 blight, 6 dahan, 1 city, 1 town
2 invaders (first turn) 90x45x22mm tray is sufficient other ‘default’ tray sizes:
- 68x90
- 69x82
- 69x90

But also don’t forget:

additional fear, beast, disease, wilds, unrest, badlands, scenario
money
elements
adversary reminders
gameplay pool of dahan, blight, invaders, towns, cities

Root Organizer

Organizing some 3d printed trays for Root to fit current expansions in the main box.

TODO: Use this OpenSCAD library!!!

Measurements

As to depth, faction boards/rules 20mm, single map 17mm, and 25mm-deep trays will actually fit everything inside.

Components

Base Game
- Marquise de Cat
- Eyrie (6 cards)
- Woodland Alliance
- Vagabond (15 quest cards, multiple character cards, with Vagabond Pack)
Riverfolk
- Lizard Cult
- Riverfolk Company (card holder)
Underworld
- Underground Duchy
- Corvid Conspiracy
Marauder
- Lord of the Hundreds (8 cards, 1 die)
- Keepers in Iron
Homeland
- Twilight Council
- Lilypad Diaspora
- Knaves of Deepwood (overlap w/ Vagabond?)
Other
- 4 dice, 4 ruins, 12 clearing items
- Base shared deck
- Exiles & Partisans shared deck
- Squires & Disciples shared deck (from Homeland)
- Landmarks
- Hirelings (Base + Marauder, Riverfolk, Underworld, Homeland hirelings)
- Resin Clearings?

Second box would then be for maps (3 now with Homeland expansion!) and faction boards? Is that an argument in favor of the cardstock faction boards to save space?

I think the riverfolk card block and tunnels can fit alongside the map boards, which would be convenient for space.

Tray Sizing

Baseline 95mm long, 25mm deep, varying width as follows:
- Cats - 60mm *
- Birds - 40mm
- Mice - 35mm
- Vagabond - 40mm, better if bigger + has more dividers, also needs 8.5mm of cards
- Lizard - 55mm
- Otter - 35mm
- Crow - 35mm
- Mole - 55mm (or 70mm incl. cards, 27mm height) (5.5mm of cards)
- Resin Clearing Markers - 90mm max
- Exile Deck + Eyrie + Faction Overviews - 70mm
- Dice
- Shop Items
- Tower/Raft
- Tunnels - 102mm, awkward.
- Alt. Exile Deck - 70mm
Layout (275mm width x 2 rows of 95mm):
- 275mm: Exile Deck, Cats, Birds, Mice, Otter, Crow
- 270mm: Vagabond+Cards, Lizard, Mole, Clearing
  - Maybe could bump clearing another 5mm and squeeze in dice/shop around them? Also possible to flex on height? Still leaves tower/raft unaccounted for, and tunnels with nowhere to live.
  - Tunnels sideways need 30.2mm (so 31mm total height on trays) which would free up a lot of space restrictions. 35mm height would I think let us get clearing markers vertically. Yellow/red fit a 55mm tray but want a 45/50mm subdivision. Orange with a 35/60mm subdivision leaves plenty of room for dice/shop/tower/raft. That only bumps up overall width from current resin measure +20mm. Tunnels still an awkward item for inside a tray.

Round Two

Baseline 95mm long, 35mm deep, 2x 275mm rows

Overall 3x25, 3x35, 3x45 for factions+resin, 105, 70, 20? for remaining components. Still need to account for vagabond cards.

Row 1, 270mm
- Exile Deck + Eyrie + Faction Overviews - 70mm, could print 75mm and include shop items?
- Cats - 45mm
- Mole - 45mm
- Birds - 35mm
- Mice - 25mm
- Otter - 25mm
- Crow - 25mm
Row 2, 255mm
- Resin Clearing Markers - 45mm (!!!)
- Lizard - 35mm
- Vagabond - 25mm, but 35mm with a divider is better - 40mm for 9x figs, nesting trays for 12 ruin/16 start/16 influence (or maybe look into actual use, and one tray for 1 Vaga, with a lower tray for a 2 Vaga game?). Divider only needs to go part way, and then nesting tray can sit atop it.
- Alt. Exile Deck - ~~70mm~~ 105mm TODO: Test this
  - Tunnels on top
  - Extra Dice in extra 35mm
- Spare 20mm? TODO: Test this
  - Dice
  - Shop Items
  - Tower/Raft

Round Three - Marauders and Hirelings

WIP TODOS

Print alternate trays at 35mm height, re-measure factions? Also maybe consider switching from two rows @ 95mm to three rows at 65mm wide?
- 10 total factions, warrior counts at 2x25, 3x20, 3x15, 2x10 - that’s x155 total, box has capacity for 5.5mm per. Currently aiming to scale 3mm per, so cats would have 65mm by 75mm available. Slightly larger compared to Round 2’s 45x95 (area 4875 vs 4275) and taller to boot.
  - 25 Cats/Lizards/(Duchy given lots of tokens)
  - 20 Birds/Rats
  - 15 Badgers/Crows/Otters
  - 10 Alliance/Vagabond
- Also worth considering that Eyrie, Vagabond, Duchy, and Hundreds all have some cards - those plus Otter’s card stand would be a nice-to-have to fit in the base box.
- Finally, also consider the draft cards and mercenary mats - I don’t think it’s actually worth managing the space for distinct mercenary meeples (just repurpose the faction meeples), but unique mercenaries like the moose should be included. Maybe embiggen the Vagabond tray?
Another idea: Splitting the box into a 3x3 grid makes each grid large enough to hold cards. This allows those that include cards (birds/rats/duchy/vagabond) to have their own cards in their tray, meeples and tokens on top; another tray for the actual deck + drafting cards, and a half-size tray for dice & global items
- 10 factions + deck + dice/items implies splitting 3 fullsize trays to half size, might be easier to run 5x full size and 6x two-thirds size, 65x60.
- And actually, looks like making one tray half-wide but double-tall (landscape) would be a good side for closed tunnels on the mountain map, while also holding other common components like dice, default items, etc.
Figure out where subdivisions make sense inside trays, to separate the warriors/tokens.
Measure up what impact we’d have putting the tunnel tokens alongside the edge of the box, beside the Otter’s card holder, if we need to shrink tray lengths.

That would be Eyrie (leaders), Vagabond (vagabonds and quests), the Duchy (nobles), and Lord of Hundreds (moods), all of which have enough meeples and tokens to justify the large bin space.

That leaves the other 4 sections of primary subdivision to hold the remaining 6 factions, and common components (dice, landmarks, etc).

Component Measurements

Global

4 Dice
12 Items (sq)

Cards: 63x88mm

Lake/Mountain Maps

1 Ferry
1 Tower
6 Closed Paths

Buildings/Tokens 10.6/5mm thick All factions: 1 VP token (18.3mm sq)

Marquise de Cat

25 Warriors (9mm, 16.7mm, 22mm)
27 Tokens: 18 buildings, 9 wood

Eyrie

20 Warriors (9mm, 18.1mm, 22mm)
7 Buildings
4 cards

Alliance

10 Warriors (9mm, 19.5mm, 19mm)
13 Tokens: 3 bases, 10 outrage

Vagabond

9 Meeples: 9 Vagabond Pawns (var.)
17 S items (sq)
8 R items (sq)
12 (14) Relationships (sq)
4 Ruins
Many cards

Lizard Cult

25 Warriors (8.3mm, 17.5mm, 19.5mm)
16 Buildings

Riverfolk Company

15 Warriors (7.9mm, 16mm, 20mm)
9 Tokens
3 gems

Corvids

15 Warriors (9.2mm, 16.8mm, 19.2mm)
8 Tokens

The Duchy

29 Meeples: 20 warriors, 9 crowns
10 Tokens: 6 buildings, 3 tokens, 1 burrow (large)
9 cards

Keepers in Iron

15 Meeples: 15 warriors
21 Tokens: 12 relic, 3 caravan, 5 waystation, 1 mission

Lord of the Hundreds

21 Meeples: 20 warriors, 1 warlord
11 Tokens: 6 storage, 5 mob
8 cards
1 dice

2026 Rework

https://counterslayer.com/

Box interior: 134mm by 215mm, 52mm depth
Warriors: 9.2mm thick, 26mm square
Tokens: 2.1mm thick, 20.1mm square
Dice: 16mm cube
Hirelings
- 1 die (1 extra)
- 12 tokens
- 3 unlock markers 36x55mm
Cats: 12
Frogs: 10+5
Ducks: 9+4
Moles: 8+3
Badgers: 6+6
Bats: 8
Rats 6 + one die
Crows: 5
Musicians: 5
Birds: 5
Alliance: 4
Lizards: 4
Porcupines: 4
Bear: 1+3
Otters 33 tall 40 wide
Moose 32 tall, 29.5 wide

Landmarks 51mm wide, stack 9 edge on “ class=”anchor”> </th></tr>

<tr><th>wip# Root Organizer

Organizing some 3d printed trays for Root to fit current expansions in the main box.

TODO: Use this OpenSCAD library!!!

Measurements

As to depth, faction boards/rules 20mm, single map 17mm, and 25mm-deep trays will actually fit everything inside.

Components

Base Game
- Marquise de Cat
- Eyrie (6 cards)
- Woodland Alliance
- Vagabond (15 quest cards, multiple character cards, with Vagabond Pack)
Riverfolk
- Lizard Cult
- Riverfolk Company (card holder)
Underworld
- Underground Duchy
- Corvid Conspiracy
Marauder
- Lord of the Hundreds (8 cards, 1 die)
- Keepers in Iron
Homeland
- Twilight Council
- Lilypad Diaspora
- Knaves of Deepwood (overlap w/ Vagabond?)
Other
- 4 dice, 4 ruins, 12 clearing items
- Base shared deck
- Exiles & Partisans shared deck
- Squires & Disciples shared deck (from Homeland)
- Landmarks
- Hirelings (Base + Marauder, Riverfolk, Underworld, Homeland hirelings)
- Resin Clearings?

Second box would then be for maps (3 now with Homeland expansion!) and faction boards? Is that an argument in favor of the cardstock faction boards to save space?

I think the riverfolk card block and tunnels can fit alongside the map boards, which would be convenient for space.

Tray Sizing

Baseline 95mm long, 25mm deep, varying width as follows:
- Cats - 60mm *
- Birds - 40mm
- Mice - 35mm
- Vagabond - 40mm, better if bigger + has more dividers, also needs 8.5mm of cards
- Lizard - 55mm
- Otter - 35mm
- Crow - 35mm
- Mole - 55mm (or 70mm incl. cards, 27mm height) (5.5mm of cards)
- Resin Clearing Markers - 90mm max
- Exile Deck + Eyrie + Faction Overviews - 70mm
- Dice
- Shop Items
- Tower/Raft
- Tunnels - 102mm, awkward.
- Alt. Exile Deck - 70mm
Layout (275mm width x 2 rows of 95mm):
- 275mm: Exile Deck, Cats, Birds, Mice, Otter, Crow
- 270mm: Vagabond+Cards, Lizard, Mole, Clearing
  - Maybe could bump clearing another 5mm and squeeze in dice/shop around them? Also possible to flex on height? Still leaves tower/raft unaccounted for, and tunnels with nowhere to live.
  - Tunnels sideways need 30.2mm (so 31mm total height on trays) which would free up a lot of space restrictions. 35mm height would I think let us get clearing markers vertically. Yellow/red fit a 55mm tray but want a 45/50mm subdivision. Orange with a 35/60mm subdivision leaves plenty of room for dice/shop/tower/raft. That only bumps up overall width from current resin measure +20mm. Tunnels still an awkward item for inside a tray.

Round Two

Baseline 95mm long, 35mm deep, 2x 275mm rows

Overall 3x25, 3x35, 3x45 for factions+resin, 105, 70, 20? for remaining components. Still need to account for vagabond cards.

Row 1, 270mm
- Exile Deck + Eyrie + Faction Overviews - 70mm, could print 75mm and include shop items?
- Cats - 45mm
- Mole - 45mm
- Birds - 35mm
- Mice - 25mm
- Otter - 25mm
- Crow - 25mm
Row 2, 255mm
- Resin Clearing Markers - 45mm (!!!)
- Lizard - 35mm
- Vagabond - 25mm, but 35mm with a divider is better - 40mm for 9x figs, nesting trays for 12 ruin/16 start/16 influence (or maybe look into actual use, and one tray for 1 Vaga, with a lower tray for a 2 Vaga game?). Divider only needs to go part way, and then nesting tray can sit atop it.
- Alt. Exile Deck - ~~70mm~~ 105mm TODO: Test this
  - Tunnels on top
  - Extra Dice in extra 35mm
- Spare 20mm? TODO: Test this
  - Dice
  - Shop Items
  - Tower/Raft

Round Three - Marauders and Hirelings

WIP TODOS

Print alternate trays at 35mm height, re-measure factions? Also maybe consider switching from two rows @ 95mm to three rows at 65mm wide?
- 10 total factions, warrior counts at 2x25, 3x20, 3x15, 2x10 - that’s x155 total, box has capacity for 5.5mm per. Currently aiming to scale 3mm per, so cats would have 65mm by 75mm available. Slightly larger compared to Round 2’s 45x95 (area 4875 vs 4275) and taller to boot.
  - 25 Cats/Lizards/(Duchy given lots of tokens)
  - 20 Birds/Rats
  - 15 Badgers/Crows/Otters
  - 10 Alliance/Vagabond
- Also worth considering that Eyrie, Vagabond, Duchy, and Hundreds all have some cards - those plus Otter’s card stand would be a nice-to-have to fit in the base box.
- Finally, also consider the draft cards and mercenary mats - I don’t think it’s actually worth managing the space for distinct mercenary meeples (just repurpose the faction meeples), but unique mercenaries like the moose should be included. Maybe embiggen the Vagabond tray?
Another idea: Splitting the box into a 3x3 grid makes each grid large enough to hold cards. This allows those that include cards (birds/rats/duchy/vagabond) to have their own cards in their tray, meeples and tokens on top; another tray for the actual deck + drafting cards, and a half-size tray for dice & global items
- 10 factions + deck + dice/items implies splitting 3 fullsize trays to half size, might be easier to run 5x full size and 6x two-thirds size, 65x60.
- And actually, looks like making one tray half-wide but double-tall (landscape) would be a good side for closed tunnels on the mountain map, while also holding other common components like dice, default items, etc.
Figure out where subdivisions make sense inside trays, to separate the warriors/tokens.
Measure up what impact we’d have putting the tunnel tokens alongside the edge of the box, beside the Otter’s card holder, if we need to shrink tray lengths.

That would be Eyrie (leaders), Vagabond (vagabonds and quests), the Duchy (nobles), and Lord of Hundreds (moods), all of which have enough meeples and tokens to justify the large bin space.

That leaves the other 4 sections of primary subdivision to hold the remaining 6 factions, and common components (dice, landmarks, etc).

Component Measurements

Global

4 Dice
12 Items (sq)

Cards: 63x88mm

Lake/Mountain Maps

1 Ferry
1 Tower
6 Closed Paths

Buildings/Tokens 10.6/5mm thick All factions: 1 VP token (18.3mm sq)

Marquise de Cat

25 Warriors (9mm, 16.7mm, 22mm)
27 Tokens: 18 buildings, 9 wood

Eyrie

20 Warriors (9mm, 18.1mm, 22mm)
7 Buildings
4 cards

Alliance

10 Warriors (9mm, 19.5mm, 19mm)
13 Tokens: 3 bases, 10 outrage

Vagabond

9 Meeples: 9 Vagabond Pawns (var.)
17 S items (sq)
8 R items (sq)
12 (14) Relationships (sq)
4 Ruins
Many cards

Lizard Cult

25 Warriors (8.3mm, 17.5mm, 19.5mm)
16 Buildings

Riverfolk Company

15 Warriors (7.9mm, 16mm, 20mm)
9 Tokens
3 gems

Corvids

15 Warriors (9.2mm, 16.8mm, 19.2mm)
8 Tokens

The Duchy

29 Meeples: 20 warriors, 9 crowns
10 Tokens: 6 buildings, 3 tokens, 1 burrow (large)
9 cards

Keepers in Iron

15 Meeples: 15 warriors
21 Tokens: 12 relic, 3 caravan, 5 waystation, 1 mission

Lord of the Hundreds

21 Meeples: 20 warriors, 1 warlord
11 Tokens: 6 storage, 5 mob
8 cards
1 dice

2026 Rework

https://counterslayer.com/

Box interior: 134mm by 215mm, 52mm depth
Warriors: 9.2mm thick, 26mm square
Tokens: 2.1mm thick, 20.1mm square
Dice: 16mm cube
Hirelings
- 1 die (1 extra)
- 12 tokens
- 3 unlock markers 36x55mm
Cats: 12
Frogs: 10+5
Ducks: 9+4
Moles: 8+3
Badgers: 6+6
Bats: 8
Rats 6 + one die
Crows: 5
Musicians: 5
Birds: 5
Alliance: 4
Lizards: 4
Porcupines: 4
Bear: 1+3
Otters 33 tall 40 wide
Moose 32 tall, 29.5 wide
Landmarks 51mm wide, stack 9 edge on

</th> <th><a name=”wip# Root Organizer

Organizing some 3d printed trays for Root to fit current expansions in the main box.

TODO: Use this OpenSCAD library!!!

Measurements

As to depth, faction boards/rules 20mm, single map 17mm, and 25mm-deep trays will actually fit everything inside.

Components

Base Game
- Marquise de Cat
- Eyrie (6 cards)
- Woodland Alliance
- Vagabond (15 quest cards, multiple character cards, with Vagabond Pack)
Riverfolk
- Lizard Cult
- Riverfolk Company (card holder)
Underworld
- Underground Duchy
- Corvid Conspiracy
Marauder
- Lord of the Hundreds (8 cards, 1 die)
- Keepers in Iron
Homeland
- Twilight Council
- Lilypad Diaspora
- Knaves of Deepwood (overlap w/ Vagabond?)
Other
- 4 dice, 4 ruins, 12 clearing items
- Base shared deck
- Exiles & Partisans shared deck
- Squires & Disciples shared deck (from Homeland)
- Landmarks
- Hirelings (Base + Marauder, Riverfolk, Underworld, Homeland hirelings)
- Resin Clearings?

Second box would then be for maps (3 now with Homeland expansion!) and faction boards? Is that an argument in favor of the cardstock faction boards to save space?

I think the riverfolk card block and tunnels can fit alongside the map boards, which would be convenient for space.

Tray Sizing

Baseline 95mm long, 25mm deep, varying width as follows:
- Cats - 60mm *
- Birds - 40mm
- Mice - 35mm
- Vagabond - 40mm, better if bigger + has more dividers, also needs 8.5mm of cards
- Lizard - 55mm
- Otter - 35mm
- Crow - 35mm
- Mole - 55mm (or 70mm incl. cards, 27mm height) (5.5mm of cards)
- Resin Clearing Markers - 90mm max
- Exile Deck + Eyrie + Faction Overviews - 70mm
- Dice
- Shop Items
- Tower/Raft
- Tunnels - 102mm, awkward.
- Alt. Exile Deck - 70mm
Layout (275mm width x 2 rows of 95mm):
- 275mm: Exile Deck, Cats, Birds, Mice, Otter, Crow
- 270mm: Vagabond+Cards, Lizard, Mole, Clearing
  - Maybe could bump clearing another 5mm and squeeze in dice/shop around them? Also possible to flex on height? Still leaves tower/raft unaccounted for, and tunnels with nowhere to live.
  - Tunnels sideways need 30.2mm (so 31mm total height on trays) which would free up a lot of space restrictions. 35mm height would I think let us get clearing markers vertically. Yellow/red fit a 55mm tray but want a 45/50mm subdivision. Orange with a 35/60mm subdivision leaves plenty of room for dice/shop/tower/raft. That only bumps up overall width from current resin measure +20mm. Tunnels still an awkward item for inside a tray.

Round Two

Baseline 95mm long, 35mm deep, 2x 275mm rows

Overall 3x25, 3x35, 3x45 for factions+resin, 105, 70, 20? for remaining components. Still need to account for vagabond cards.

Row 1, 270mm
- Exile Deck + Eyrie + Faction Overviews - 70mm, could print 75mm and include shop items?
- Cats - 45mm
- Mole - 45mm
- Birds - 35mm
- Mice - 25mm
- Otter - 25mm
- Crow - 25mm
Row 2, 255mm
- Resin Clearing Markers - 45mm (!!!)
- Lizard - 35mm
- Vagabond - 25mm, but 35mm with a divider is better - 40mm for 9x figs, nesting trays for 12 ruin/16 start/16 influence (or maybe look into actual use, and one tray for 1 Vaga, with a lower tray for a 2 Vaga game?). Divider only needs to go part way, and then nesting tray can sit atop it.
- Alt. Exile Deck - ~~70mm~~ 105mm TODO: Test this
  - Tunnels on top
  - Extra Dice in extra 35mm
- Spare 20mm? TODO: Test this
  - Dice
  - Shop Items
  - Tower/Raft

Round Three - Marauders and Hirelings

WIP TODOS

Print alternate trays at 35mm height, re-measure factions? Also maybe consider switching from two rows @ 95mm to three rows at 65mm wide?
- 10 total factions, warrior counts at 2x25, 3x20, 3x15, 2x10 - that’s x155 total, box has capacity for 5.5mm per. Currently aiming to scale 3mm per, so cats would have 65mm by 75mm available. Slightly larger compared to Round 2’s 45x95 (area 4875 vs 4275) and taller to boot.
  - 25 Cats/Lizards/(Duchy given lots of tokens)
  - 20 Birds/Rats
  - 15 Badgers/Crows/Otters
  - 10 Alliance/Vagabond
- Also worth considering that Eyrie, Vagabond, Duchy, and Hundreds all have some cards - those plus Otter’s card stand would be a nice-to-have to fit in the base box.
- Finally, also consider the draft cards and mercenary mats - I don’t think it’s actually worth managing the space for distinct mercenary meeples (just repurpose the faction meeples), but unique mercenaries like the moose should be included. Maybe embiggen the Vagabond tray?
Another idea: Splitting the box into a 3x3 grid makes each grid large enough to hold cards. This allows those that include cards (birds/rats/duchy/vagabond) to have their own cards in their tray, meeples and tokens on top; another tray for the actual deck + drafting cards, and a half-size tray for dice & global items
- 10 factions + deck + dice/items implies splitting 3 fullsize trays to half size, might be easier to run 5x full size and 6x two-thirds size, 65x60.
- And actually, looks like making one tray half-wide but double-tall (landscape) would be a good side for closed tunnels on the mountain map, while also holding other common components like dice, default items, etc.
Figure out where subdivisions make sense inside trays, to separate the warriors/tokens.
Measure up what impact we’d have putting the tunnel tokens alongside the edge of the box, beside the Otter’s card holder, if we need to shrink tray lengths.

That would be Eyrie (leaders), Vagabond (vagabonds and quests), the Duchy (nobles), and Lord of Hundreds (moods), all of which have enough meeples and tokens to justify the large bin space.

That leaves the other 4 sections of primary subdivision to hold the remaining 6 factions, and common components (dice, landmarks, etc).

Component Measurements

Global

4 Dice
12 Items (sq)

Cards: 63x88mm

Lake/Mountain Maps

1 Ferry
1 Tower
6 Closed Paths

Buildings/Tokens 10.6/5mm thick All factions: 1 VP token (18.3mm sq)

Marquise de Cat

25 Warriors (9mm, 16.7mm, 22mm)
27 Tokens: 18 buildings, 9 wood

Eyrie

20 Warriors (9mm, 18.1mm, 22mm)
7 Buildings
4 cards

Alliance

10 Warriors (9mm, 19.5mm, 19mm)
13 Tokens: 3 bases, 10 outrage

Vagabond

9 Meeples: 9 Vagabond Pawns (var.)
17 S items (sq)
8 R items (sq)
12 (14) Relationships (sq)
4 Ruins
Many cards

Lizard Cult

25 Warriors (8.3mm, 17.5mm, 19.5mm)
16 Buildings

Riverfolk Company

15 Warriors (7.9mm, 16mm, 20mm)
9 Tokens
3 gems

Corvids

15 Warriors (9.2mm, 16.8mm, 19.2mm)
8 Tokens

The Duchy

29 Meeples: 20 warriors, 9 crowns
10 Tokens: 6 buildings, 3 tokens, 1 burrow (large)
9 cards

Keepers in Iron

15 Meeples: 15 warriors
21 Tokens: 12 relic, 3 caravan, 5 waystation, 1 mission

Lord of the Hundreds

21 Meeples: 20 warriors, 1 warlord
11 Tokens: 6 storage, 5 mob
8 cards
1 dice

2026 Rework

https://counterslayer.com/

Box interior: 134mm by 215mm, 52mm depth
Warriors: 9.2mm thick, 26mm square
Tokens: 2.1mm thick, 20.1mm square
Dice: 16mm cube
Hirelings
- 1 die (1 extra)
- 12 tokens
- 3 unlock markers 36x55mm
Cats: 12
Frogs: 10+5
Ducks: 9+4
Moles: 8+3
Badgers: 6+6
Bats: 8
Rats 6 + one die
Crows: 5
Musicians: 5
Birds: 5
Alliance: 4
Lizards: 4
Porcupines: 4
Bear: 1+3
Otters 33 tall 40 wide
Moose 32 tall, 29.5 wide

Landmarks 51mm wide, stack 9 edge on “ class=”anchor”> </th></tr>

<tr><th>notes# Zettelkasten

Zettelkasten is a method of taking notes.

Zettlr is a note-taking app that supports Zettelkasten (but also general notes, research papers, books, etc with PDF export). </th> <th><a name=”notes# Zettelkasten

Zettelkasten is a method of taking notes.

Zettlr is a note-taking app that supports Zettelkasten (but also general notes, research papers, books, etc with PDF export). “ class=”anchor”> </th></tr>

</table> </div>

There's a bit of trickery there that liquid doesn't document very well on lines 3 and 5 - in the second half of the for block you can chain filters on the collection you're iterating over. The short format used is something along the lines of `collection|filter:arg,arg,arg|filter...`

Similarly, I had some ugly code in my regular [archive][] page to group by year and put headings in:

[archive]: /archive/

```html

Now, a few extra liquid filters later, it looks like this:

If you want to get easy extensions in your own project, rather than maintaining Yet Another Jekyll Fork, please vote up my merge request on github.

{{ post.title }}

Blogging about blogging about liquid (as above) I leave as an exercise to the reader.