4 ways to filter has_many associations

You have a model that has_many associations, and you’d like to get a collection of these models with their associations that satisfies a list of conditions both on the model and the association records. You’ve tried a few things, but whatever you’ve tried only “sort of” works (meaning, not really) and leaves you feeling like you need a push in the right direction.

Filtering has_many associations can be one of those problems which makes you scratch your head and want to do some extra reading to brush up on your SQL and ActiveRecord knowledge.

For example, let’s say you have User and Project models in your system, and you want to write a query that retrieves a collection of users with projects that were created in a given date range.

Depending on what you want your query to return, here are 4 different ways to approach this problem:

  1. The simplest thing you can do is to combine the joins and where methods that ActiveRecord gives you, like so:

    User.joins(:projects).where(projects: { zipcode: 30332 })

    This technique is a good choice when you need to filter your records by one or more attributes. This will give you back a collection of User records which all have projects with a zipcode of 30332.

    One gotcha to keep in mind is that since you’re doing a INNER JOIN with the joins method, this can return duplicates. Read on to point 2 to see how you can get around this.

  2. More often than not, you might have scopes defined on your association models which you want to re-use. So when you want to use these scopes to do your filtering, get acquainted with ActiveRecord's merge function. According to the ActiveRecord documentation, merge returns an array representing the intersection of the ActiveRecord::Relation that you call merge on, and the ActiveRecord::Relation that you pass in.

    So, for example, if you have a scope in your Project model called opened_recently, which returns all projects which have been created within the last 10 days, you can do something like:


    This will return to you a list of User objects which all satisfy the condition of having projects which were opened recently.

    Like I said before, one thing to keep in mind when you’re using joins for a has_many association is that since you are doing an INNER JOIN, it is possible that you will get back duplicate User (in our example) objects – basically one object for every ‘match’. It’s easy enough to get around this with a call to the uniq method.


    The uniq method works by changing your query from something like SELECT `users` from ... to something like SELECT DISTINCT `users` from...

  3. If you’re going to be needing to retrieve information from the records you filter, especially information stored in the associations of these records, another option to consider is eager loading and the includes method. As you probably know, the benefit of eager loading is that you get to avoid the N + 1 queries problem.

    It’s pretty straightforward to filter by attributes or scopes on the association:

    User.includes(:projects).where(projects: { zipcode: '30332' })

    This will give us all users who have projects in Atlanta, and also eager load their projects.

    If in your .where you want you use an SQL fragment, then you need to call the references method:

    User.includes(:projects).where('projects.deleted_at IS NOT NULL').references(:projects)

    You can also use the merge method in conjunction with includes and references to make use of any scopes you might have defined in your association models, like so:


    You won’t need to use uniq when you use includes because ActiveRecord will take care of it for you automatically.

  1. Now, what should you do if you want your collection to only have those associations which meet your filter criteria? This situation arose for me when I had to write an API endpoint which let app users pass in parameters to filter associations. There are a couple of ways you can approach this. In my case, since I had to return an object that would eventually be converted to JSON, the simplest way was to manually construct a hash which contained the record and the filtered associations.

So I did something like this in my controller:

def show 
  user_attributes.merge(projects: filtered_projects) 

def user_attributes 
  # returns a hash containing attributes for the given user 
  # something like this: # { id: 1, first_name: 'Rick', last_name: 'Sanchez' } 

def filtered_projects 
  # here's where I used any Project related params to filter down my projects 
  # something like Project.opened_after(params[:project_date]).as_json 
  # I use as_json because I need this method to return a hash 

Note that the merge method used here is the one for Hash, and not the ActiveRecord method. You should be able to extend this to return multiple users as well.

Another way to do this is by defining a scoped has_many association. If you’re interested in seeing how this might be done, ping me in the comment section below and I’ll do a post on this topic specifically.

Hope this was helpful. As always, if you have a question about this topic that isn’t specifically addressed above, leave a comment below, and I’ll do my best to help out!

How to use has_many :through with additional attributes on the join table

A common area where developers can get hung up is when using has_many :through. has_many :through essentially allows you to link two models together with a “join table”. Though it can be pretty simple to work with, things can get confusing when you find yourself having to pass and store additional attributes in this join table.

“I have read every post on the internet pertaining to has_many through with additional attributes on the join table and I am still not getting it”

“how do you store extra info in the join model and extract that out later?”

If you followed along my previous post, at this point you should have a good idea of how to write your models, controllers and views such that you can create, display and edit records which use a has_many :through association. If you’re still stuck on that, leave a comment below and I’ll try to sort you out. Make sure to check out Ryan Bates screencast on the basics, and also checkout Rahoul’s article on when you should has_many in the first place.

So, what do you do when you want to store/access additional attributes in the join table?

You already know how to set up your view form so that it passes in parameters to your controller and correctly creates a new record in the join table. If you’re using checkboxes in your view, the checkbox code probably looks like this:

<% Group.all.each do |group| %>
  <%= check_box_tag "user[group_ids][]", group.id, @user.group_ids.include?(group.id) %>
  <%= group.name %>
  <br />
<% end %>

…assuming that you have a User and Group models and a UserGroup which functions as the join.

And your controller #create action should be very similar to this:

def create

def user_params
  params.require(:user).permit(group_ids: [])

As you can see, we don’t have to do anything special in the controller because group_ids will be accepted as a parameter (it is part of the dynamic programming which happens when you call has_many). You do have to correctly permit the group_ids param though.

Now, let’s say for a given user you want to specify if they are an admin of the group or not, via checkbox.

I’m going to assume that we will have an edit page for each group record, and on this page, we will see all the users that belong to this group along with a checkbox next to each user indicating if they are an admin or not.

The View

In my group edit page, I want to show the users that are in the group and next to each user show a checkbox allowing me to select if the user is going to be an admin.

<%= form_for @group do |f| %>
  <%= @group.inspect %>
  <br />
  <% if @group.user_groups.present? %>
    <%= f.fields_for :user_groups do |ugf| %>
      <% user = ugf.object.user %>
      <%= user.name %>
      <%= ugf.check_box :admin %>
      <br />
    <% end %>
  <% else %>
    No Users in this group yet
  <% end %>
  <%= f.submit 'Update group' %>
<% end %>

Couple of things to note here:

1) I’m using the user_groups association. By using fields_for on this association, I can treat it like any other association of group and build a custom form for it.

2) When submitting this form, the user_groups parameters will be passed in under the user_groups_attributes key in the params hash.

The Model

To be able to pass in a hash with user_groups_attributes to the Group model and call update or save on it, we need to use the accepts_nested_attributes_for method. This method tells Rails and ActiveRecord how to correctly deal with user_groups_attributes being in the params hash.

Our Group would look like this:

class Group < ActiveRecord::Base
  has_many :user_groups
  has_many :users, through: :groups
  accepts_nested_attributes_for :user_groups

So the parameter we pass in to accepts_nested_attributes_for is the model/association we want to accept nested attributes for.

The Controller

Because of the setup we did above in the Group model, we can now use the usual update method with the params that we get from the view/form.

def update
  @group = Group.find(params[:id])

def group_params
  params.require(:group).permit(user_groups_attributes: [:admin, :id])

If you’re using Rails 4 and strong_parameters, you will have to make sure you permit the correct parameters.

And that’s it! You should now be able to update this admin attribute on the UserGroup join table. You can follow the same approach for different types of data as well (like a text field for example). I encourage you to look deeper into what the params hash looks like once it gets to the controller so that you get more comfortable with it. Play with this idea in Rails console as well to increase your confidence.

I’ve posted an example app on github – check it out if you need more info about how exactly to get this to work. If you’re still stuck, or are dealing with a use case which is not similar to this, post in the comments section below and I’ll try to help.

How a has_many :through association works in practice

When you’re beginning to work with Rails and ActiveRecord, it can be tough to wrap your head around the concept of has_many :through associations, and more importantly how they can be used in your Rails app.

“lets say i have group, member, memberships…how do i then actually USE this in my models, controllers and views?”

“I’m using a has_many :through relationship which appears to be working. I now need a form that I can render to have users join or leave the group.”

First off, you should be reasonably confident that a has_many :through association is a good choice for your app design. Assuming you are, here’s how you can put it into practice.

Let’s say you’re building an app with Users and Groups, and you want to build a form where a given user can join or leave any group. You’ve chosen to associate your users and groups through the table Membership.

Your models are structured like so:

class User < ActiveRecord::Base
  has_many :memberships
  has_many :groups, through: :memberships

class Group < ActiveRecord::Base
  has_many :memberships
  has_many :users, through: :memberships

class Membership < ActiveRecord::Base
  belongs_to :user
  belongs_to :group

I always find it helpful before I start writing my controllers or views, to think through what I want to accomplish, and use the Rails console to ensure I’m on the right path. In other words, I try to come up a with an end-goal, and then work backwards with the help of the Rails console to figure out my solution.

So, given that we want to render a form that allows a user to join or leave a group, we should ask ourselves, “what does it mean to have a user join a group?”

As far as the database is concerned, it essentially means that the Membership table will have a row with the user_id and group_id columns corresponding to our given user and group. Our task then is to orchestrate things such when the user clicks ‘submit’ on their form, the Membership table ends up with a new row (or rows) with the correct user_id and group_id values.

In the form/view that we render for the user, we need a way whereby the user can select which groups they want to join. And in the controller, we want to extract the choices that the user made in the form and use those choices to create new rows in the Membership table.

First, imagine you had a user_id and a group_id, how would you go about creating that new row in the Membership table? ActiveRecord makes it easy. What we want to do is, for a given User record, add a ‘membership’ to it, so that when we say @user.groups, we get back a list of groups which includes the group we just added. If you do this:

#assuming @user is your User record
@user.groups << Group.find(group_id)

#if you have a list of group_ids(something like [1, 2]), you can also do:
@user.group_ids << group_ids

This piece of code will automagically create a new row in the Membership table with the right group_id and user_id. And now whenever you use @user.groups, you will get back a list of groups that you added. For added confidence, try the above in your Rails console.

Check out the Rails guide below for more details on what methods ActiveRecord provides you when you use has_many :through.

I’m going to assume you have some type of authentication set up in your app, which gives you access to the current_user method. We will need this method to get our user_id, via a simple current_user.id call. Once you have your user (with something like @user = User.find(current_user.id)), you can use the code above to add groups.

The next question is, how do you write your view such that it passes the group_id values to the controller?

How you write your view will depend wholly on the flow you expect your users to go through and other factors in your app. Let’s say you decide to provide your user with a list of checkboxes listing the groups they can join or leave. Here’s a simple way to accomplish that:

<% Group.all.each do |group| %>
  <%= check_box_tag "user[group_ids][]", group.id, @user.group_ids.include?(group.id) %>
  <%= group.name %>
  <br />
<% end %>

The usage of check_box_tag probably needs some explanation.

The first parameter "user[group_ids][]" tells Rails to group all the checkboxes with this ID together and pass it in to the controller as an Array. What this means is in your controller, you can do params[:user][:group_ids] and get a nice Ruby Array of all the group_ids that the user chose.

The last parameter just ensures that any Group which the user currently belongs to is checked when the form is rendered.

The above should get you going with using has_many :through in your Rails app, but depending on what your app does I’m pretty sure you will run into issues which are not covered in this post. So if you’re still stuck, check out the following:

1) Check out this amazing Rails cast which takes you through the entire gamut of using has_many :through. If you haven’t already, I’d highly recommend you get yourself an account on RailsCasts.

2) Rails ActiveRecord Guide on has_many :through

If you need more help, hit reply in the comment section below, and I’ll do my best to get you going.

Retrieving Random Records in Rails

What is the best way to retrieve one or more random records in Rails?

Given that there are multiple ways to retrieve random records in Rails and Ruby (take a look at this SO post), it can be confusing and frustrating to decide which course of action to take.

“Is there a technique to retrieve random records that is correct/general/Rails way/cross-database compatible? “

In this post, I’ll cover a couple of simple approaches that you can take to figure out what the best way might be for you.

The first priority that you likely have to keep in mind is speed. Most applications have a threshold for what is considered slow – in web apps for example, anything that takes longer than a few hundred milliseconds to load is considered slow. So you don’t want to be choosing a method that will have your users waiting around forever for your app to do something. That being said, there is also no point in sacrificing code readability, ease of re-use and just time in general, to get faster than you have to.

The second thing you have to consider is the size (number of rows) of the database table you are looking to retrieve a random record from. In general, the bigger your table, the more knowledgable you’ll have to be about your database platform (typically Postgres or MySQL) to get to your desired speed.

Take this common solution to retrieve a random record in Rails:  

random_user = User.all.sample

To use the code above requires no internal knowledge of the database you’re running on. The #all method is a well-known ActiveRecord method and #sample is a Ruby method on Array that retrieves a random value from a given array.

The flip side of this approach is that if your database has a sizable number of rows, it will take a long time to execute. Try it out on your Rails console and see! The reason it does so is because User.all retrieves all the users from the database, and then builds ActiveRecord objects from them. So if you have let’s say 10,000 users, building 10,000 ActiveRecord objects takes a while.  If you’re reasonably certain that the table in question will not grow much in size over the lifetime of your app, and the speed at which the above code executes is agreeable to you, then by all means go for it.

Now, if your table is HUGE (has a million+ rows for example), the above method is not going to work for you. Consider using the ActiveRecord #offset and the Ruby rand methods. The #offset method in ActiveRecord specifies how many rows to skip before returning the rest of the rows. The rand method, as expected, generates a random number less than a maximum value – so when you say rand(10), it will generate a random number less than 10.

So something like this:

count = User.count
random_offset = rand(count)
random_user = User.offset(random_offset).first`

…works better (faster) for big tables, because you’re not making the database scan through every single row in the table.


Business Spotlight – Rachio

Business Spotlight brings into focus highly innovative companies in the home automation space and deep dives into what makes these companies tick. We’ll explore the technologies they use, the challenges they face and often successfully solve, and try to gain an insight into what it takes to build and execute a game changing product.
Meet Rachio, the makers of the Iro, a smart sprinkler controller. Founded by an experienced team of software designers and engineers in response to drought conditions in the western US, the Iro is the culmination of relentless research & development into:
  • Understanding the best and most effective ways to manage water in the context of outdoor irrigation systems.
  • Building and deploying a sophisticated cloud architecture that is able to bring to bear years of irrigation research to the consumer’s home and,
  • Does so in a user-friendly way, such that the user is not overwhelmed.

The outcome of this research is an irrigation controller that for the first time, is worthy of controlling access to one of your most precious resources. The Iro is not only easy to use with your smartphone app, but also knows exactly when and how much water to use depending on current and past weather conditions, the type of soil, the slope of your zone, the type of nozzle you have, the type of vegetation and many other variables, greatly increasing the efficiency of your irrigation system and saving costs substantially.

17466169758_d0244d98d1_o Why Rachio was founded Rachio was started in the summer of 2012, by founders Chris Klein and Franz Garsombke. Chris and Franz were in the middle of a serious drought situation in Denver, Colorado when, inspired by a state-wide conversation on tearing out lawns and replacing them with vegetable gardens, they decided to look into how big of issue outdoor irrigation really was. Chris found that outdoor irrigation contributes to a significant amount of water waste in the US (about 8 billion gallons a day), and that for homes with lawns in particular, 50% of water usage went towards outdoor irrigation. After some research on the subject, and being software experts, they ultimately settled on asking the question, “What can software do to solve this problem?” Exploration Phase The first thing they explored was using a wireless outlet to which a standard irrigation controller could be plugged into. The idea was that via WiFi, they could turn the wireless outlet on and off and subsequently control the irrigation schedule. However, preliminary prototyping efforts with a WeMo controller quickly tabled that idea. They found that the closed architecture of the WeMo did not lend itself well to a suitable solution. They then directed a couple of months worth of R&D effort into devising a solution that would not need any additional hardware, ie, something that would be purely software. In the end, they found there was no way around building hardware, so that’s what they did. Biggest Challenges According to Chris, Rachio’s biggest challenges during this early phase mainly concerned their entry into the hardware products space. Being a team of software experts building a hardware company, and the hardware business being a capital intensive one, they found that their initial lack of experience got in the way of raising funds. For example, to know how much funding they would need, they had to accurately model manufacturing costs considering the huge cost variability of various hardware components. Chris says that winning a state-wide competition for innovators and entrepreneurs was a turning point for Rachio. They not only won a substantial amount of prize money, but were also able to befriend a local manufacturer in the audience. Working out of their new friend’s plant they were able to build relationships that helped them successfully navigate the hardware space. All of Rachio’s manufacturing is based in Colorado, which Chris feels gives them a significant competitive advantage. Pitfalls avoided According to Chris, the obvious things like product-market fit, proof of concept, the relationships you build and the connections you make all play a role in ensuring success for your product and company to-be. Manufacturing in the US, with the manufacturing plant located about an hour away, allowed them to ramp up and down according to demand and remain fluid. Rachio also took a software first approach to build the product. Understanding that the hardware was essentially a shell and that any deficiencies in the software such as bad connections or inaccurate algorithms would be perceived by the consumer as a defective product was a key insight that helped Rachio maintain focus on building high quality software for the Iro. Rachio also benefited from Chris’ background in construction systems management. His experience in building hard goods, commercial buildings and structures became increasingly relevant as they began to manufacture the Iro. Familiarity with managing subcontractors, exposure to lead times and the ways dependencies work in the hard goods world (more of a Waterfall approach, as opposed to Agile) all helped Chris and his team at Rachio when the time came to successfully and consistently make progress. 17783811778_895f179e5a_o Tech Stack The main components of Iro’s brain are comprised of:
1. The internal firmware, which resides on board the Iro
2. A mobile app, which the user uses to configure the Iro
3. The cloud app, which lives on Rachio’s servers and contains the bulk of Iro’s intelligence and business logic.

Firmware The firmware that sits atop the Iro is lightweight in nature, and its main job is to maintain connection to the WiFi network and communicate back and forth with the Rachio cloud app. Communication to the Rachio app involves listening for instructions and reporting back on current status. The firmware also has the ability to continue irrigation and maintain a healthy yard if WiFi connection is lost. Rachio uses an Electric Imp Wifi chip to run the firmware, and according to Chris the firmware is written in Squirrel, which is a “Javascript-y” language provided by Electric Imp. The firmware engineering team consists of a full-time firmware engineer, and Franz, Rachio’s co-founder and CTO. Mobile App The mobile app holds a key position in the Rachio ecosystem, by virtue of being the primary point of contact between the Iro and the user. When Chris and his team first designed the app, what made sense at the time was to design it so that the user felt like they were interacting with a piece of hardware. Chris remembers that getting the design right for the app the first time around was very difficult, owing to the fact that the Iro was one of the very first irrigation controllers to offer full configuration to the user via a mobile app. Considering the usability constraints of an irrigation system, such as multiple zones that can’t run on top of each other, calendaring, different soil types, valves, vegetation types, irrigation systems that run on pumps, a lot of thought had to be put in to the design and user experience (UX) so that the user was able to realize the benefits of having the powerful Rachio cloud at their fingertips. As the years and months have gone by, Rachio have gradually expanded and extended the feature set of their mobile app based on user feedback. As they’ve started to understand the IoT space better, Chris and the Rachio team have undergone a shift in their approach to app design. They are realizing that although the way one interacts with hardware might initially influence how the app is designed, ultimately, what yields good design is considering how one wants to interact with the system as a whole. Actively trying to understand the important elements of the user interaction, be it informational or configurational, and determining the level of abstraction at which the user can harness the widest feature-set possible, in as enjoyable a manner as possible, is key. Version 2.0 of the Iro mobile app is on the way, and according to Chris, it will represent a significant change in how users interact with the Iro stack. The Iro Controller Cloud The Rachio cloud contains the bulk of the business logic and is the real brains behind the magic of the Iro. The Rachio cloud is a multi-layered system, and has a public API layer that allows for client interaction with the cloud. Rachio’s own clients, like their iOS and Android apps, use this public API. The public API is also accessible to any other device that wants to integrate with Rachio’s services and the Iro. Given the competitive nature of the space, it isn’t possible to divulge much about the internals of the Rachio cloud other than to say that it is built in a type-safe language. Rachio also uses Node servers that stand in front of the API layer. Irrigation Scheduling An important component of the Rachio cloud is the irrigation scheduling algorithm. The main function of the irrigation scheduling algorithm is to determine, based on weather and zone data, how frequently your lawn should be watered. When the Iro was being built, and Chris and the team at Rachio were looking into irrigation algorithms, they found that prior research results on the subject were tried and true and should be the way to water. The Iro supports two types of irrigation algorithms:
1. The Penman equation
2. Maximal allowable depletion The Rachio cloud also ties into weather services and personal weather stations to obtain data such as daily evapotranspiration rate which are important inputs to the chosen irrigation algorithm. The next version of the Iro app and cloud infrastructure will allow the user to choose which weather station to use.

18718204015_72d2636c89_o Conclusion Chris says that although all the individual pieces of the irrigation solution have been available for a long time (for example, Kansas provides a spreadsheet to help home owners manually figure out irrigation scheduling), Rachio, by building a sophisticated cloud architecture and apps that encourage the user to interact with the cloud, has successfully managed to tie all the pieces together to create a water efficient and user-friendly system.

Rachio is at an exciting point in their evolution. They have had the opportunity to see their system in action for the last few years with real people and homes, and now have a much clearer vision of how the space they are in will evolve. They are hard at work at realizing this vision and are confident that they will continue to provide value in the years to come. Author Notes: I’d like to thank Chris Klein, CEO at Rachio for being so generous with his time.