Evil Ecto Embeds

I jest. I ❤️ Ecto, but embeds and default values can be tricky. Yesterday, I had to fix an issue in a project where notifications where not being sent to all users. The problem lies in the combination of Ecto embeds_one, Ecto field defaults and database defaults.

I found the issue somewhat intriguing, so I wanted to 1) share a not-obviously-broken example, 2) explain what is wrong and 3) how to fix it.

The Pledge: Here's an example

Give it a quick read-through. Do you spot any mistakes? If the problem is blatantly obvious to you, you can stop reading 🙂.

defmodule UserTest do
  use DataCase

  test "inserting user applies default settings" do
    # given
    attrs = %{}
    changeset = User.changeset(attrs)

    # when
    user = Repo.insert!(changeset)

    # then
    assert %{settings: %{notify?: true}} = user
  end

  test "listing users to notify returns user with default settings" do
    # given
    %{} |> User.changeset() |> Repo.insert!()

    # when
    users = Repo.all(from u in User, where: u.settings["notify?"] == true)

    # then
    assert length(users) == 1
  end
end

defmodule User do
  use Ecto.Schema
  import Ecto.Changeset

  schema "users" do
    embeds_one :settings, Settings do
      field :notify?, :boolean, default: true
    end
  end

  def changeset(user \\ %User{}, attrs) do
    user
    |> cast(attrs, [])
    |> cast_embed(:settings)
  end
end

defmodule Repo.Migrations.CreateUsers do
  use Ecto.Migration

  def change do
    create table(:users) do
      add :settings, :map, default: %{}
    end
  end
end

The Turn: It's broken

For me, this was a surprising turn.

The first test is fine. Looks like the newly inserted user is saved with default settings:

user = %{} |> User.changeset() |> Repo.insert!(changeset)
assert %{settings: %{notify?: true}} = user

But is it really? 😱 Then why does the next test fail?

users = Repo.all(from u in User, where: u.settings["notify?"] == true)
assert length(users) == 1

While the default value is returned by Repo.insert!/1 , it's not present in the database. Rather, the users database table currently looks like this:

id, settings
 1, {}

Ecto did not receive a value for the settings embed, and instructs the database to save settings = NULL. The database default, as specified in the migration, then kicks in: settings = is stored instead of NULL.

Note: Ecto doesn't support embed defaults.

What makes this confusing is that Ecto "hides" this. When reading a user from the database, Ecto "lazily" applies the default values when reading (when casting the empty JSONB object to the Settings schema).

user_with_lazy_defaults = Repo.get!(User, 1)

But a database WHERE query on settings.notify? fails to find the user, since the default notify? value was not written to the database.

empty_list = Repo.all(from u in User, where: u.settings["notify?"] == true)

The Prestige: How to fix it

This is actually not surprising and luckily quite simple. Since this is not a magic trick, I hope you can forgive me for failing to give you a truly captivating Prestige 😉.

%{settings: %{}}
|> User.changeset()
|> Repo.insert!()

Passing an empty map for the embed changeset is enough. Ecto then casts this to the Settings struct and applies the default field values.

We end up with this database users table state:

id, settings
 1, {"notify?": true}

Ecto works as expected

To be clear: This is not an Ecto bug. Ecto is doing exactly what it should. The bug is in my head, because I misinterpret how defaults and embeds work. And because my first test reinforces my understanding.

What made it especially confusing in the project at hand was the fact that it was a regression bug. The notify? flag was previously stored top-level as User.notify?. Everything worked as expected, because Ecto field defaults for top-level fields are saved to the database.

Then, a bunch of user flags was grouped and moved into the Settings embed. Along came the problem: New users suddenly didn't receive notifications.

But wait: Not all users. That would have been easy to spot 😉. Instead, there was a default manual process in place: After creating a user, usually the user settings were manually edited via the UI. As a result, the lazy user.settings.notify? default value was saved to the database

This manual process further masked the issue, because most new users did end up with {"notify?": true} in the database and were thus notified.

Over to you

Is there a better fix? Should we validate_required(:settings)?

Or is there a better mental model to avoid falling into this trap in the first place?

The post When to use field :default in an Ecto schema by Dan Schultzer also touches on the subject.