Disclaimer. Those who are not familiarized with N+1 problem, you can check “What is the N+1 selects problem in ORM(Object-Relational Mapping) with details explaining the concept.

Since associations were introduced in Pop, the N+1 problem became part of it: many queries hit the database in order to load whole model’s associations. Good news is N+1 problem is not a disease without remedy. In this post I will give you some steps I took to solve it by incorporating emergent design.

Step 1: Start Coding By Example

Coding by example helps you to define what’s your input and what’s your output. Code By Example means you define a scenario where  your solution is tested and what is the expected result. The method I used to apply this technique was Test Driven Development (TDD).

I started to code by example for the most simple scenario I could think of. To me, the most simple scenario was to solve N+1 problem for the has_many type association.

=========== Scenario 1 =============================

 Given I have 3 Users with one Book each one related
 Then I would expect the query performed to be:

 select books.* from books where user_id in ($1, $2, $3)

This, translated into code, would be:

  // pop_test.go
  type User struct {
     ...
     Books []Book `has_many: “books”` 
  }

  // pop_test.go
  type Book struct {
     ...
     UserID int `db:”user_id”` 
  }

// preload_associations_test.go
func TestHasManyNPlusOneSolution(t *testing.T) {
   ...

  for i:= 1; i <= 3; i++ {
      user := User{}
      tx.Create(&user)
      
      tx.Create(&Book{
          UserID: user.ID,
      })
   }

  Debug = true

  users := []User{}
  preload(tx, &users)

  a.Equal(3, len(users))
  a.Equal(1, len(users[0].Books)
  a.Equal(users[0].ID, users[0].Books[0].UserID)
}

// preload_association.go
func preload(tx *Connection, model []User) error {
	return nil
}

Step 2: Be Explicit With Your Input And Output.

As soon as I wrote down the scenario, I noticed there are three things that I needed to do in order to produce the desire output:

• First thing is to iterate over every user and store its ID value in a slice.

• Second thing is that user_id column in the expected query result. I need to pull it out from db tag in Book model.

• Third thing is to link all books with their respective users once the query is successful.

Because I’m in a premature start point, and I don’t want to go into the code details about how to read that field tag, In order to make that test to pass as quickly as possible, I hard-coded the solution and be explicit to have that test with a green face.

func preload(tx *Connection, model []User) error {
    // 1) fill in ids.
    ids := []interface{}
    for _, u := range model {
        ids = append(ids, u.ID)
    }
    
    // 2) load associations.
    books := []books{}
    tx.Where(“user_id in (?)”, ids).All(&books)
    
    // 3) fill in model with associations.
    for i := range model {
        for _, book := range books {
            if book.UserID == model[i].ID {
                model[i].Books = append(model[i].Books, book)
            }
        }
    }
}

Step 3: Go From Specific To Generic.

Once I have green light and test is passing, I noticed the solution was composed into a three steps process:

1. fill in ids.
2. load associations.
3. fill in model with associations.

At this stage I need to be less specialized and be more generic with the solution. The reason is this moment preload function only knows how to preload users and books, but models can be from different types.

In order to achieve generalization, I take one step at the time and apply some refactorings techniques until I’ve satisfied the desired generic level.

I started to refactor step one (fill in ids). In this step, the generic level I want to achieve is to get a list of all ids value from model, no matter what type that model is. This pushed me to incorporate some reflections code and go a little deep in details in order to get the expected result. For every tiny change I applied to my code, I ran my test making sure is still getting green.

Once I got the level of abstraction I want for my step one, I move on to step two (load associations) and do same thing. I stablish first which is the generic level I want to go, and then apply all code changes and refactorings necessary to achieve the correct result. Once I’m done with step two I then move on to step three, always trying to achieve generalization. Here is the result of applying this process to the has_many preload function.

Step 4: Repeat, Repeat and Repeat.

Once I feel I’m done with has_many type association, I start to cycle again following same process for has_one type association. I repeat same three steps: Code By Example, Be Explicit and Be Generic (from now on CBB) until I got all four type associations covered. https://github.com/gobuffalo/pop/blob/nplus1/preload_associations.go#L158

Wrapping up

As software developers we tend to find code solutions based on our experience in one project and try to reuse them in other projects where we think they’ll match up. Sometimes, there are cases where uncertainty is a variable, and applying CBB process helped me to overcome that. I’ve testified the good impact this process has. The way that solution design emerges, without a deliberated thoughtful specification in mind. It feels like an organic living growing from its seed. Give it a try and let me know how it works for you.

Thank you for reading this post. I hope you have enjoyed how emergent design helped me  to find solutions to N+1 problem in Buffalo/Go, and hope you apply it into your projects.