The blog of , a Ruby on Rails development team
Known from Rails LTS, makandropedia and Nanomize

Know your APIs

Last month we unrolled another project: A social media center for one of Germany's largest insurance companies. The R+V Newsroom streams activities by R+V and R+V24 from eight different sources, giving a total of 14 different accounts polled, adding about 9 new posts each day. Until today, about 200,000 imports have been performed.

It's all about APIs

Polling APIs is different from usual single-box coding where errors are usually your own bad. Since we need to handle uncertainties implied by remote systems, our code needs to be particularly robust. Since we're talking to eight different systems, we need to talk eight different API languages. And since we're regularly checking data streams, we need a reliable way to find the point we left off for each of them.

Each network has its own associated importer that knows a) how to fetch new posts and b) how to write them to the database. With API client gems like twitter, koala, yt and xing_api, this is no big deal. Some sources had to be consumed more manually, like Wordpress RSS feeds, and to display press releases we're even scraping their web page.

The key for robustness is asynchronous import scheduled by whenever and performed by Sidekiq. When an import fails (due to an API unavailability like during the the DNS DDoS last month), a clever configuration makes sure it is retried a number of times before we're notified of its failure.

But it's not just flexible input, the R+V Newsroom also provides its data in various formats. Besides the website, it offers widgets that can be embedded into other pages and display a limited, filtered list of posts. Additionally, the post stream is available as RSS and JSON, leaving the choice on how to consume the data to the user.

Great UX with reasonable effort

In the frontend, the R+V Newsroom makes use of a number of great libraries to leverage user experience.

Number one is our more and more established Javascript library Unpoly. Think Rails + intelligent Turbolinks + Angular directives + power modals. Unpoly turns a plain full-page server-side website into a snappy app while requiring only little changes in your code. As a side effect, it leaves the frontend with lean Javascript (~280 lines of Coffeescript) and even has the website still accessible when Javascript should be turned off. You should give it a try!

We further sped up the page by delivering images in the right sizes for each device. Since monitor resolution is ever increasing, images need to be served in higher resolutions, too. At the same time, the traffic from mobile devices increases—devices, that mostly have little bandwidth and usually sport far smaller screens than desktop computers. By using the awesome lazysizes, we could accomplish a flexible and maintainable delivery of "just right" images to whatever device visits the page. Read more in this in-depth blog post.

Another notable improvement was made to file uploading in the backend. You all know these plain file input fields that make any webpage feel like it's 20 years old. It does not have to be this way! jQuery-File-Upload is a mature library that lets you build awesome, snappy file uploads that are customizable and play well with server-side components like CarrierWave. See an in-depth example in the makandropedia.

Hosting by makandra

Having this new platform with a great UX, R+V decided to add the other half of UX: reliable servers. The Newsroom is hosted on makandra's powerful Rails infrastucture. Like web applications from Audi, ABUS and ProSieben, the R+V Newsroom runs on redundant machines that are properly scaled and caringly maintained.

With all this in place, the R+V Newsroom is a solid platform with a snappy UI in both frontend and backend, looking forward to its next 100,000 imports.

Four party cooperation

The R+V Newsroom is the result of a multi-company endeavor. Conception, project management and customer communication was done by d.tales, a content marketing agency from Munich. Christian Ringleb crafted these clean, bluey designs, while we were growing the Newsroom line by line. Thanks for the sweet cooperation!

Introductory interview with Andreas

At makandra we're welcoming 2-3 new colleagues per year. We'll publish short introduction interviews like this one with Andi, who joined our operations team two weeks ago.

Introduce Yourself

I am Andreas Herz, living in Schwabmünchen south of Augsburg. I live there with my little daughter and my wife. In my spare time I like to fool around with my daughter or rebuild the Lego buildings she destroyed. If there is still time left I like to work on technical projects, playing video or board games. Before I joined makandra I worked as a systems developer for Linogate after graduating from Augsburg University, where I studied computer science with physics.

How did you first learn about makandra and what drew you to makandra originally?

I know makandra since the beginning and followed the progress they made. After a nice evening at the summer party I had a great talk with Henning and Thomas about the opportunities at makandra and had the chance to get an valuable insight into the daily work.

What is your role at makandra?

I joined the operations team as senior operations engineer.

What are 3 words to describe makandra?

  • structured
  • organized
  • familiar

What challenges are you looking forward at makandra?

I'm looking forward to learn more about the techniques used at makandra, especially Puppet and Ruby. It's also interesting to see how a more advanced setup with loadbalancer, dedicated servers, failover etc. is working and managed with a huge amount of automatization.

Are you messy or organized?

Sometimes even too organized.

Describe what you were like at age 10.

Imagine a small nerd sitting in front of a PC 24/7 playing video games or trying to fix the PC from other people.

What did you want to be when growing up?

I wanted to be a game developer which also brought me into coding.

Best vacation you've been to?

So far the best vacation was honeymooning in the Berchtesgaden Alps. We had a really nice hotel and liked the relaxing and calm atmosphere there close to nature.

Vim or Emacs? (Be careful here…)

Vim of course :)

What tools are you using that you don't want to miss anymore?

Linux in general (although it's an OS), but without i3wm, zsh, tmux, vim and taskwarrior I would be less productive.


Welcome to makandra, Andi!

Now available: Post Rails Book Bundle

The Post Rails Book Bundle is a collection of books about growing Ruby on Rails applications as they become bigger and more successful.

The bundle is only available until this Friday (July 22nd 2016), so grab your copy while you can!

The following eight books are included at a heavily discounted price:

Fearless Refactoring Rails Controllers

This book guides you through the complicated task of cleaning up the mess that controllers often become in legacy Rails applications.

How do you extract the business essence from the controller, while keeping the HTTP related aspects in it, untouched?

It also touches to topic of introducing explicitness to multiple Rails conventions which simply further refactorings.

Trailblazer - A new architecture for Rails

Trailblazer introduces several new abstraction layers into Rails. It gives developers structure and architectural guidance and finally answers the question of "Where do I put this kind of code?" in Rails.

This book walks you through a realistic development of a Rails application with Trailblazer and discusses every bloody aspect of it.

Rails As She Is Spoke - How Rails gets OOP wrong but it works anyway

Do you want to understand Rails? Do you want to write Rails apps effectively, or see your own open source creations delight their users and enjoy wild success like Rails has? Have you noticed that Rails never quite makes perfect sense according to traditional object-oriented theory, or that the Ruby On Rails Guides never seem to quite explain the realities of cutting-edge Rails development?

Frontend friendly Rails - Better defaults for your sophisticated frontends

Upgrade Rails defaults and introduce cool features that’ll help you with making your apps more maintainable and faster to write.

Take your API to a higher level in terms of maintenance and provide user experience improvements.

Growing Rails Applications in Practice

Discover a simpler way to scale Rails codebases. Instead of introducing new patterns or service-oriented architecture, we will show how to use discipline, consistency and code organization to make your application grow more gently.

Rails TDD Video Class

The real goal of the class is to teach you how to combine Rails with TDD. Other things, like DDD and mutant are also crucial. We don’t want to show the only one and true way. We are going to present more techniques over time. Think of this class as a TV series, which is easy to consume in short episodes every day.

Modular Rails - The complete guide to modular Rails applications

Wait! What's a modular application?!

It's pretty simple. Instead of putting everything into one project, you put your MVC components into specialized Rails engines packaged as gems. Module by module, you can define what your application will be!

Unfuck a Monorail for Great Justice

Monolithic Rails apps – or monorails – are a problem in the world of Rails development. This book doesn't just show you how to get them back on track. It shows you how to get them back on track more cleanly and more swiftly than you would have believed humanly possible.


All in all this is $458 worth of books for only $199. Offer is valid until this Friday (July 22nd 2016). More information at www.railsbookbundle.com

Using responsive images

Back in the web's youth, you'd embed an image with <img src="some/file.jpg" />. Everyone who read your site would download and see that image. After all, it was only a small file a few pixels high and wide.

Times have changed. Web pages contain huge images. People visit web pages from various devices with different screen sizes and resolutions. Moreover, hi-DPI screens have more pixels per area, requiring higher-resolution images.

Web pages need to adapt. But how?

Naive approach

The easiest means developer-wise would be to simply provide images with double (or triple) the traditional resolution.

Unfortunately, this has the drawback of delivering 4x (or 9x) as large image files to everyone. Even people with a 3.5 inch display would need to load the full 4K landing page image over their prepaid mobile connection.

That's bad.

Flexible approach (the future)

What if we gave each client exactly the image size it needed? It would mean more work for the developer, but combine the best user experience with the best possible load time for the client.

That's called a "responsive image". Like a responsive web page, it sort of "adapts" to the client's screen size. Unlike a responsive web page, there is some effort required to achieve this.

Using responsive images is a trade-off between storage and complexity on the server side, and perceived performance and quality on the client side. Offering 100 versions for an image will give each device the optimum image file as quick as possible, but consume lots of space on the server and make rendering and maintenance a nightmare.

Hence, it is about finding the sweet spots of valuable image sizes. Nowadays, browsers are quite good in interpolating images, so you do not actually need to target all resolutions - which, by the way, is impossible. You could never target all mobile device resolutions.

Let's get to the meat. Here's the process (inspired by Matt Wilcox). Do this for each image location in your application (e.g. header banner, image thumbs, gallery, etc.):

  • Determine required image widths
  • Generate them
  • Render them

However, it is not totally as simple as this, so we'll talk about the process in detail.

1. Finish the design

You'll save yourself wasted time when you only start optimizing image performance after all conditions have settled, including responsive design! Remember, premature optimization is the root of all evil.

2. Pick a standard version

You should always have a standard image that's just as large as it needs to be for a normal density desktop screen.

3. Determine responsive image versions

Next, resize the window though all breakpoints and make a list of the widths the image goes through. Then add pixel density variants to their right.

Example:

Breakpoint x1 x1.5 x2 ← Pixel density
lg 1310 1965 2620 Standard image and retina image (see below)
md 1110 1665 2220 The versions above are still fine for these widths
sm+xs 890 1335 1780 1 version for mobile screens, those with Retina can still use 1310

The selected versions are: 890, 1310 and 1965. For an example image, their sizes are 110kB, 212kB and 430kB.

Because width matters way more than height in responsive design, the image sizes are denoted with the new w unit. Thus, the selected image sizes above are called 890w, 1310w and 1965w.

How to choose image versions

How many versions you choose depends on the degree of client optimization you want to achieve. More images means possibly better performance, but also more complexity on the server.

  • When image sizes are close (as in 10%), drop the smaller one. E.g. 1200, 1280 → 1280
  • There are reasonable limits for image size and density factor. Read on.

Reasonable lower limit

Almost all desktop browsers are used at widths beyond 1000px. Most mobile devices sport double-density screens that are 320px or more in width. Thus, images below 640w are not useful for full width images. (Images smaller than full-width on a 320px screen are mostly so small in file size that you generally should not need to create variants. Usually these images are quite small on desktop, too, so you might simply reuse the desktop image on mobile.)

A 640w JPG with 65kB would be only 20kB at 320w. While this looks like two thirds saved (wow!), it's only 45kB. So you'd create an extra image version just to save 45kB per image on one of the few non-retina tiny screens. Probably there are places where you can optimize better.

Reasonable factor limit

The intuitive upper resolution limit is the max size an image can get, multiplied by the screen's pixel density. However, since browsers are quite good at interpolating images nowadays, you do not need to feed every client pixel. Under most circumstances, 1.5 times larger will do fine for retina double-density devices. (I've manually tested this.)

For a 800w JPG and a 2x version of 300kB, the 1.5x version has 187kB. That's a 40% or 113kB reduction, which is quite much. Still, both variants are visually almost indistinguishable with only very sharp edges slightly blurrier in the smaller image. I call it "loss-less enough" ;-).

Considerations

The above limits and suggestions target the "average" case, where you want improve bandwidth usage and retina user experience with little effort. Of course you need to derive your own strategy depending on your audience.

4. Generate image versions

Now that you know which image widths you need, you'll have to generate them.

I'm using Carrierwave to store image uploads and to generate these versions. In the respective uploader, add image versions like this:

version :banner_1965 do
  process resize_to_fill: [1965, 600]
end
version :banner_1310, from_version: :banner_1965 do
  process resize_to_fill: [1310, 400]
end
version :banner_890, from_version: :banner_1310 do
  process resize_to_fill: [890, 272]
end

Affixing the versions with their width makes using them much easier, see below.

5. Render the image

We're almost done. However, the technically hardest part remains: How do we give the browser the right image variant? We cannot reliably know upfront which screen size he has, nor how large the image will be rendered.

Fortunately, a solution is already in the making. It has quite complex powers, but in the form we'll be using it consists of two additional attributes on the img tag: srcset and sizes.

srcset
Contains a list of image URLs together with their width: srcset="large.jpg 1965w, medium.jpg 1310w, small.jpg 890.jpg"
sizes
Contains a list of "reduced media queries" (which work slightly different from CSS) and the corresponding image size: sizes="(max-width: 900px) 500px, 1310px". An entry without a media query is taken as default value.

However, since I'm already using the awesome lazysizes (I can recommend this great library, btw) for lazy image loading, I'll also use it for managing responsive images. Because it lazy loads the image anyway, it can also select the appropriate size on the fly. Thus it makes the sizes attribute dispensable.

We'll use the following markup:

<img src="fallback/image.jpg" # Fallback image for old browsers
  class="lazyload" # Activate lazy image loading
  # A transparent 1px gif that keeps modern browsers from loading the src
  srcset="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw=="
  data-sizes="auto" # Tell lazysizes to manage the `sizes` attribute
  data-srcset="large.jpg 1965w, medium.jpg 1310w, small.jpg 890px" />

To generate this in Rails, we'll use the following helper …

# Expects the version names to end with _<image width>, e.g. banner_1965
def responsive_image_tag(uploader, versions:)
  url_widths = versions.map do |version|
    url = uploader.send(version).url
    width = version[/_(\d+)$/, 1]
  
    [url, width]
  end
  srcset = url_widths.map { |url, width| "#{ url } #{ width }w" }.join(', ')
  
  image_tag url_widths.first[0], # lazysizes recommends src == data-srcset[0]
    class: 'lazyload',
    # This transparent gif keeps modern browsers from early loading the
    # fallback `src` that is only for old browsers
    srcset: 'data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==',
    data: {
      srcset: srcset,
      sizes: 'auto',
    }
end

… and use it like this:

= responsive_image_tag @page.image, versions: %i[banner_1965 banner_1310 banner_890]

Voilà! (This example image does not use lazysizes because it is not installed on this blog. Nontheless it's using the srcset and sizes attributes to have the browser choose the appropriate version.)

Conclusion

We've built a simple system of responsive images that delivers appropriate image sizes to each client. It is as easy as determining the required image widths, defining them in a Carrierwave uploader and rendering them with the supplied helper method.

I hope you enjoyed this post. Send me a note when you've incorporated responsive images somewhere!

Resources

Your Rails 3.2 project should switch to Rails LTS

After many months of work, Rails 5 was released today. Thanks to the almost a thousand committers, this new version brings us exciting new features such as ActionCable or the attributes API.

Unfortunately this also means your trusty Rails 3.2 app is no longer getting new security patches from the open-source community, who now has their hands full maintaining versions 4 and 5.

Rails 3.2 has been on limited maintenance for the past two years. With the Rails 5 release, this support has ended entirely. If your business uses Rails 3.2, you should act now to prevent data breaches and liability.

If your team can't or don't want to upgrade your application now, you can switch to Rails LTS and continue to receive security updates. Rails LTS is a drop-in replacement for the official Ruby on Rails gems.

For more information about Rails LTS visit https://railslts.com/.

Our book in a bundle:
A bunch of books about building and maintaining large-scale Rails applications, all at a discounted price. Featuring best-selling books like Trailblazer, Fearless Refactoring, or Growing Rails Applications in Practice!
Our address:
makandra GmbH
Werner-von-Siemens-Str. 6
86159 Augsburg
Germany
Contact us:
+49 821 58866 180
info@makandra.de
Commercial register court:
Augsburg Municipal Court
Register number:
HRB 24202
Sales tax identification number:
DE243555898
Chief executive officers:
Henning Koch
Thomas Eisenbarth