Watermarking image attachments with Active Storage can be a little tricky so here’s a quick write up of what I did to get it working. 

The TL;DR
Use a variant and pass a draw command to mogrify. Like this.

The non-TL;DR

Behind the scenes Active Storage 5.2  is using minimagick for image transformations aka variants.  I am specifically calling out 5.2 because from the looks of the code on Github Rails 6 will have something different going on. 

Using a variant allows you to do things like resize, crop, monochrome, and a lot more. For example, this is how you would render a black and white resized image in your view:

<%= image_tag @post.image.variant(resize: '47x47', monochrome: true) %>

How does this work? The Rails Guide for Active Storage says this:

To create a variation of the image, call variant on the Blob. You can pass any MiniMagick supported transformation to the method.

To enable variants, add mini_magick to your Gemfile:

https://guides.rubyonrails.org/active_storage_overview.html#transforming-images

Sweet! MiniMagick is a wrapper around the infamous ImageMagick library. Basically just about anything you can do with ImageMagick directly you should be able to do from a Ruby API provided by MiniMagick… and according to the Rails docs anything minimagick can do we should be able to pass through the variant…  

We will get to that soon.  For now let’s talk about how you’d watermark an image using ONLY ImageMagick.

ImageMagick has a ton of utilities and according to ImageMagick’s docs watermarking is done using the composite command along with the -watermark option.  The ImageMagick docs even give a pretty straight forward example:

composite -watermark 30% -gravity south \
            wmark_image.png  logo.jpg    wmark_watermark.jpg

convert happens to be one of the supported tools by minimagick so we should be able to get this done… right?

Unfortunately after looking at the source code for Active Support it appears that this is straight up impossible. ActiveStorage::Variation#transform does use a MiniMagick::Image object but it also contains a hardcoded a call to mogrify.

mogrify is another ImageMagick command that is useful for a ton of different things like resizing, cropping etc but it does not really offer first class support for watermarking images.  It’s not impossible (!!) but it’s just not as fun.

OK, since we are stuck using mogrify how are we going to get this done? Luckily mogrify supports an option called draw. According to the docsdraw is designed to:

…annotate or decorate an image with one or more graphic primitives. The primitives include shapes, text, transformations, and pixel operations.

https://imagemagick.org/script/command-line-options.php#draw

One of the shape primitives is an image ! 

How would this look if we were using mogrify directly from the command line? Here’s an example that will plop a watermark directly in the center of your image:

mogrify -gravity center \
        -draw 'image Over 0,0 0,0 "watermark.png"' \
        unwatermarked_image.jpg

And finally, how would we use this to watermark an image in our app?  Like this:

image_tag @post.image.variant(
  combine_options: {
    gravity: 'center',
    draw: 'image Over 0,0 0,0 "'+Post::WATERMARK_PATH.to_s+'"'
  }
)

That’s basically it.  I’ve published source code for a demo app that does exactly this on GitHub — check it out here.

If you’re using Administrate (a wonderful Rails engine produced by the folks at thoughtbot) you may have found that your application’s helpers are not available to Administrate’s views.

Exposing your app’s helper methods to Administrate is pretty straight forward — just add the following to your application.rb:

# config/application.rb
module YourApp
  class Application < Rails::Application
    # ... your application's config

    # Load App helpers into administrate
    config.to_prepare do
      Administrate::ApplicationController.helper YourApp::Application.helpers
    end
  end
end

If you’re interested in how this works take a look at this bit of the Rails source code.

Rails’ AbstractController provides a method (specifically modules_for_helpers) for submitting a list of helpers which are processed and added into the engine.  Very cool!

The nice thing here is that is not specific to Administrate and can be used with any Rails engine needing access to your application’s helpers.

Just a quick note — when creating foreign keys in a Rails 5.1+ you should make sure to they are typed as bigint

But why?

Starting in Rails 5.1 primary keys were defaulted to bigintCheck this commit from 2016.  This means that referential foreign keys also need to be bigintIf you’ve gotten into the habit of rails g model Whatever user_id:integer,you need to break it. 

Use user:references or user:belongs_to (which is an alias of references) instead.  These will automatically use bigint and all will be well.

But why???

If you forget to do this or use some kind of gem that generates an integer foreign key your app won’t break yet…

The issue is rooted in this bit I stole from the PostgreSQL docs:

TypeStorage SizeRange
integer4 bytes -2147483648 to +2147483647
bigint8 bytes -9223372036854775808 to 9223372036854775807

bigint is yuge and as a result you could run into a situation where the primary key (a bigint) you’re trying to persist is bigger than the FK column (an integer) can hold.

Active Storage is a pretty nice addition to Rails. Essentially you’re given an out-of-the-box polymorphic solution to managing ANY type of attachment. Rails will handle the processing and storing of attachments along with extra stuff such as metadata extraction. Super cool.

The Problem

Management of Active Storage attachments in a way that feels Railsy is a different story though. Let’s say you have a Profile resource that uses a Picture attachment.  When viewing profile#show wouldn’t it be nice to just delete the picture from there? Where would the actual act of destroying an attachment live?

Attachment management seems strange because it FEELS like it’s just another aspect of your resource but it’s not.

I tried a couple options that didn’t seem to be right…

  • Add logic to the resource controller’s destroy action to delete an instance of the resource OR optionally remove just the resource’s attachment(s)
  • A universal attachment controller.  CRUD requests are sent to AttachmentController and operate using a combination of attachment ID and type

I think both of those options felt wrong because we either end up with controller actions concerned about things they shouldn’t really know about OR we end up with a “universal” controller that will likely become littered with callbacks for many different resources.

A Convention I’ve Adopted

For now when I have a resource using Active Storage attachments I create namespaced controllers (and routes) for each attachment type under the resource they support. 

So in the case of the Profile and Picture example above I’d have something like this: 

  • app/controllers/profiles_controller.rb
  • app/controllers/profile/pictures_controller.rb

This is nice because PicturesController can have callbacks or prerequisites (think authentication) that are specific to the app OR add things that are specific to pictures without polluting intent.

Code + Walk-through

Checkout the walletapp repo for a demo I made using this technique.  For now please check the commit history and shortly I’m going to add a tutorial on how to setup an app from scratch that has attachments and attachment management using the nested controller approach.

Storage.

Top hat, monocle and sweet stashA common gripe I’ve heard people bring up about Ruby is the fact that you can monkey-patch basically anything at any point during runtime. The result is that the world your code expects could potentially change under it and all hell breaks loose.  What if you included a 3rd party library that changed your expectations of .to_s?

Here’s a contrived example:

There is another way to monkey-patch that is safer — refinements!  Refinements have been in Ruby since 2.0 and essentially provides a way to namespace your monkey-patching efforts.

Creating and using refinements is pretty easy and requires:

  1. Create a module to hold your refinements
  2. USE that module inside your code

When making use of refinements the changes are scoped/limited within the scope of the module using said refinements. Here’s an example:

First we can create a module to hold our changes to the Integer class:

Next, via using, we can “use” this refinement in our own code!

Now finally here’s an example of calling CrazyInteger and how using refinements has protected Integer#to_s outside of the scope of CrazyInteger‘s class.

crystal-lang-logoToday I spent a few hours playing around with the Crystal language.  For a project that’s considered “new” the docs are pretty great.  Coming from the world of Ruby writing basic Crystal code is straightforward and most of everything works as expected.

For fun and science I decided to try and throw together a simple program that would allow me to try the following:

  • Flesh out some basic classes
  • Raise exceptions
  • Use public/private methods
  • Import code

What I came up with is a pointless ~30 line program that takes a word, looks up the Wikipedia page for that word and finally prints the page’s title.  The code is extremely easy to read and while putting it together I found that most things just “worked” as they would in Ruby.

The only gotcha I found was that when extracting text from the page body’s title a MatchData object is returned and the matches are accessed via a block.  Ruby also has MatchData . Unlike Crystal, Ruby’s implementation provides MatchData#captures for a super easy way to access an array of the actual matches… which is what you probably care about.

Here’s the code:

I’ve been playing with Docker off and on for almost a year and noticed that I have a lot of abandoned images on my system that are left over from experimenting.  This post contains a few commands you can use to clean up your system.

To illustrate what I mean by “abandoned images”, here’s the output of `docker images` on my machine.

That last image is 11 months old and at this point I have no idea what the hell I was doing with it — time to clean!

The list below is a (mostly) plagiarized repost of an answer left by Ulises Reyes on Stack Overflow and explains how to clean up old images.

 

Delete One Image

Easy!

docker rmi <the_image_id>

 

Delete All Images

The `docker rmi` command can accept a list of image IDs so destroying all images is pretty easy:

docker rmi $(docker images -q)

 

Stop and Delete All Images

Gracefully (SIGTERM) stop all images then delete them.

docker rm $(docker stop $(docker ps -q))

 

KILL and Delete All Images

SIGKILL all images then delete them.

docker rm $(docker kill $(docker ps -q))

 

Delete All Images Except for…

If you want to delete all images except for a select few it can be done using grep.  Basically you will make a call to `docker images` then filter the list using grep.

The example below will delete all images that do NOT have contain the words “dont”, “kill” and “me”.

Note — the `awk {‘print $3}` is what actually prints out the image ID.

docker rmi $(docker images | grep -v 'dont\|kill\|me' | awk {'print $3'})

Here’s a nice snippet that can be used in a PostgreSQL database to output the current database’s size.

SELECT pg_size_pretty(pg_database_size(current_database())) AS human_size, 
       pg_database_size(current_database())                 AS raw_size;

That statement will output something along the lines of:

human_size  | raw_size
------------+--------------
181 GB      | 193841573064

git push origin hamsterAt work we have been using GitLab for a while and it’s been pretty good but we have recently opted to move to GitHub Enterprise.  As a result there are a number of small repos we have hosted on GitLab that now need to be moved over to GitHub.

Additionally, this should generically work for any git repository migration but I’m explicitly calling out “gitlab” and “github” in hopes it makes the post easier to follow — and because it’s my actual use case.

The process is as follows:

  1. Mirror clone the old repo @ GitLab
  2. Mirror push to the new repo @ GitHub with pre-commit hooks skipped
  3. Update the push URL for the mirror cloned repo

Why clone as `mirror`?

Because it’s like `–bare` but with more magic!

This blurb from the git docs pretty much covers it:

–mirror
Set up a mirror of the source repository. This implies –bare. Compared to –bare, –mirror not only maps local branches of the source to local branches of the target, it maps all refs (including remote-tracking branches, notes etc.) and sets up a refspec configuration such that all these refs are overwritten by a git remote update in the target repository.

And from GitHub’s docs:

As with a bare clone, a mirrored clone includes all remote branches and tags, but all local references will be overwritten each time you fetch, so it will always be the same as the original repository.

Why skip pre-commit hooks?

You may not need to do this but the repos I’ve been working on have a lot of pre-commit stuff that runs — tests, static analysis, etc.  For the purpose of just pushing the entire repo to a new location i don’t care about this stuff.  `–no-verify` tells git to push and skip the hooks.

Repo Migration Process

Developer Change Process

For developers who are using the repos being migrated, the process for pointing @ GitHub is a one liner.

EDIT: This post has been updated and retitled for El Capitan.

 

If you have attempted to install the Nokogiri gem and encountered errors similar to the output below… I feel your pain and have a solution.

The Pain

➜ z gem install nokogiri Building native extensions. This could take a while... ERROR: Error installing nokogiri: ERROR: Failed to build gem native extension.
/Users/mariozig/.rvm/rubies/ruby-2.2.1/bin/ruby -r ./siteconf20150318-11691-1hlore5.rb extconf.rb
checking if the C compiler accepts ... yes
checking if the C compiler accepts -Wno-error=unused-command-line-argument-hard-error-in-future... no
Building nokogiri using packaged libraries.
checking for gzdopen() in -lz... yes
checking for iconv using --with-opt-* flags... yes
************************************************************************
IMPORTANT NOTICE:
Building Nokogiri with a packaged version of libxml2-2.9.2
with the following patches applied:
 - 0001-Revert-Missing-initialization-for-the-catalog-module.patch
 - 0002-Fix-missing-entities-after-CVE-2014-3660-fix.patch
Team Nokogiri will keep on doing their best to provide security
updates in a timely manner, but if this is a concern for you and want
to use the system library instead; abort this installation process and
reinstall nokogiri as follows:
gem install nokogiri -- --use-system-libraries
 [--with-xml2-config=/path/to/xml2-config]
 [--with-xslt-config=/path/to/xslt-config]
If you are using Bundler, tell it to use the option:
bundle config build.nokogiri --use-system-libraries
 bundle install
Note, however, that nokogiri is not fully compatible with arbitrary
versions of libxml2 provided by OS/package vendors.
************************************************************************
Extracting libxml2-2.9.2.tar.gz into tmp/x86_64-apple-darwin14.1.0/ports/libxml2/2.9.2... OK
Running patch with /Users/mariozig/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/ports/patches/libxml2/0001-Revert-Missing-initialization-for-the-catalog-module.patch...
Running 'patch' for libxml2 2.9.2... OK
Running patch with /Users/mariozig/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/ports/patches/libxml2/0002-Fix-missing-entities-after-CVE-2014-3660-fix.patch...
Running 'patch' for libxml2 2.9.2... OK
Running 'configure' for libxml2 2.9.2... OK
Running 'compile' for libxml2 2.9.2... OK
Running 'install' for libxml2 2.9.2... OK
Activating libxml2 2.9.2 (from /Users/mariozig/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/ports/x86_64-apple-darwin14.1.0/libxml2/2.9.2)...
************************************************************************
IMPORTANT NOTICE:
Building Nokogiri with a packaged version of libxslt-1.1.28
with the following patches applied:
 - 0001-Adding-doc-update-related-to-1.1.28.patch
 - 0002-Fix-a-couple-of-places-where-f-printf-parameters-wer.patch
 - 0003-Initialize-pseudo-random-number-generator-with-curre.patch
 - 0004-EXSLT-function-str-replace-is-broken-as-is.patch
 - 0006-Fix-str-padding-to-work-with-UTF-8-strings.patch
 - 0007-Separate-function-for-predicate-matching-in-patterns.patch
 - 0008-Fix-direct-pattern-matching.patch
 - 0009-Fix-certain-patterns-with-predicates.patch
 - 0010-Fix-handling-of-UTF-8-strings-in-EXSLT-crypto-module.patch
 - 0013-Memory-leak-in-xsltCompileIdKeyPattern-error-path.patch
 - 0014-Fix-for-bug-436589.patch
 - 0015-Fix-mkdir-for-mingw.patch
Team Nokogiri will keep on doing their best to provide security
updates in a timely manner, but if this is a concern for you and want
to use the system library instead; abort this installation process and
reinstall nokogiri as follows:
gem install nokogiri -- --use-system-libraries
 [--with-xml2-config=/path/to/xml2-config]
 [--with-xslt-config=/path/to/xslt-config]
If you are using Bundler, tell it to use the option:
bundle config build.nokogiri --use-system-libraries
 bundle install
************************************************************************
Extracting libxslt-1.1.28.tar.gz into tmp/x86_64-apple-darwin14.1.0/ports/libxslt/1.1.28... OK
Running patch with /Users/mariozig/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/ports/patches/libxslt/0001-Adding-doc-update-related-to-1.1.28.patch...
Running 'patch' for libxslt 1.1.28... OK
Running patch with /Users/mariozig/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/ports/patches/libxslt/0002-Fix-a-couple-of-places-where-f-printf-parameters-wer.patch...
Running 'patch' for libxslt 1.1.28... OK
Running patch with /Users/mariozig/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/ports/patches/libxslt/0003-Initialize-pseudo-random-number-generator-with-curre.patch...
Running 'patch' for libxslt 1.1.28... OK
Running patch with /Users/mariozig/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/ports/patches/libxslt/0004-EXSLT-function-str-replace-is-broken-as-is.patch...
Running 'patch' for libxslt 1.1.28... OK
Running patch with /Users/mariozig/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/ports/patches/libxslt/0006-Fix-str-padding-to-work-with-UTF-8-strings.patch...
Running 'patch' for libxslt 1.1.28... OK
Running patch with /Users/mariozig/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/ports/patches/libxslt/0007-Separate-function-for-predicate-matching-in-patterns.patch...
Running 'patch' for libxslt 1.1.28... OK
Running patch with /Users/mariozig/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/ports/patches/libxslt/0008-Fix-direct-pattern-matching.patch...
Running 'patch' for libxslt 1.1.28... OK
Running patch with /Users/mariozig/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/ports/patches/libxslt/0009-Fix-certain-patterns-with-predicates.patch...
Running 'patch' for libxslt 1.1.28... OK
Running patch with /Users/mariozig/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/ports/patches/libxslt/0010-Fix-handling-of-UTF-8-strings-in-EXSLT-crypto-module.patch...
Running 'patch' for libxslt 1.1.28... OK
Running patch with /Users/mariozig/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/ports/patches/libxslt/0013-Memory-leak-in-xsltCompileIdKeyPattern-error-path.patch...
Running 'patch' for libxslt 1.1.28... OK
Running patch with /Users/mariozig/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/ports/patches/libxslt/0014-Fix-for-bug-436589.patch...
Running 'patch' for libxslt 1.1.28... OK
Running patch with /Users/mariozig/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/ports/patches/libxslt/0015-Fix-mkdir-for-mingw.patch...
Running 'patch' for libxslt 1.1.28... OK
Running 'configure' for libxslt 1.1.28... ERROR, review '/Users/mariozig/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/ext/nokogiri/tmp/x86_64-apple-darwin14.1.0/ports/libxslt/1.1.28/configure.log' to see what happened.
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers. Check the mkmf.log file for more details. You may
need configuration options.
Provided configuration options:
 --with-opt-dir
 --with-opt-include
 --without-opt-include=${opt-dir}/include
 --with-opt-lib
 --without-opt-lib=${opt-dir}/lib
 --with-make-prog
 --without-make-prog
 --srcdir=.
 --curdir
 --ruby=/Users/mariozig/.rvm/rubies/ruby-2.2.1/bin/$(RUBY_BASE_NAME)
 --help
 --clean
 --use-system-libraries
 --enable-static
 --disable-static
 --with-zlib-dir
 --without-zlib-dir
 --with-zlib-include
 --without-zlib-include=${zlib-dir}/include
 --with-zlib-lib
 --without-zlib-lib=${zlib-dir}/lib
 --enable-cross-build
 --disable-cross-build
/Users/mariozig/.rvm/gems/ruby-2.2.1/gems/mini_portile-0.6.2/lib/mini_portile.rb:279:in `block in execute': Failed to complete configure task (RuntimeError)
 from /Users/mariozig/.rvm/gems/ruby-2.2.1/gems/mini_portile-0.6.2/lib/mini_portile.rb:271:in `chdir'
 from /Users/mariozig/.rvm/gems/ruby-2.2.1/gems/mini_portile-0.6.2/lib/mini_portile.rb:271:in `execute'
 from /Users/mariozig/.rvm/gems/ruby-2.2.1/gems/mini_portile-0.6.2/lib/mini_portile.rb:66:in `configure'
 from /Users/mariozig/.rvm/gems/ruby-2.2.1/gems/mini_portile-0.6.2/lib/mini_portile.rb:109:in `cook'
 from extconf.rb:278:in `block in process_recipe'
 from extconf.rb:177:in `tap'
 from extconf.rb:177:in `process_recipe'
 from extconf.rb:487:in `<main>'
extconf failed, exit code 1
Gem files will remain installed in /Users/mariozig/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2 for inspection.
Results logged to /Users/mariozig/.rvm/gems/ruby-2.2.1/extensions/x86_64-darwin-14/2.2.0/nokogiri-1.6.6.2/gem_make.out

The Problem

For some reason Apple’s Yosemite version of OSX does not have a system accessible installation of libxml2.  Nokogiri requires this in order to compile and luckily Xcode has a version of libxml2 bundled with it — we just need to specify it when installing the gem.  It’s important to get Nokogiri installed correctly because as of right now Rails 4.2.1.rc4 automatically attempts to install it and you will feel pain.

How I Solved It

gem install nokogiri -- --use-system-libraries=true --with-xml2-include=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include/libxml2/