Evil Ecto Embeds
I jest. I ❤️ Ecto, but embeds and default values can be tricky.
Yesterday, I had to fix an issue in a project where notifications where not being
sent to
all users. The problem lies in the combination of Ecto embeds_one
, Ecto field
defaults and database defaults.
I found the issue somewhat intriguing, so I wanted to 1) share a not-obviously-broken example, 2) explain what is wrong and 3) how to fix it.
The Pledge: Here's an example
Give it a quick read-through. Do you spot any mistakes? If the problem is blatantly obvious to you, you can stop reading 🙂.
defmodule UserTest do
use DataCase
test "inserting user applies default settings" do
# given
attrs = %{}
changeset = User.changeset(attrs)
# when
user = Repo.insert!(changeset)
# then
assert %{settings: %{notify?: true}} = user
end
test "listing users to notify returns user with default settings" do
# given
%{} |> User.changeset() |> Repo.insert!()
# when
users = Repo.all(from u in User, where: u.settings["notify?"] == true)
# then
assert length(users) == 1
end
end
defmodule User do
use Ecto.Schema
import Ecto.Changeset
schema "users" do
embeds_one :settings, Settings do
field :notify?, :boolean, default: true
end
end
def changeset(user \\ %User{}, attrs) do
user
|> cast(attrs, [])
|> cast_embed(:settings)
end
end
defmodule Repo.Migrations.CreateUsers do
use Ecto.Migration
def change do
create table(:users) do
add :settings, :map, default: %{}
end
end
end
The Turn: It's broken
For me, this was a surprising turn.
The first test is fine. Looks like the newly inserted user is saved with default settings:
user = %{} |> User.changeset() |> Repo.insert!(changeset)
assert %{settings: %{notify?: true}} = user
But is it really? 😱 Then why does the next test fail?
users = Repo.all(from u in User, where: u.settings["notify?"] == true)
assert length(users) == 1
While the default value is returned by Repo.insert!/1
, it's not present in the database.
Rather, the users
database table currently looks like this:
id, settings
1, {}
Ecto did not receive a value for the settings embed, and instructs the database to save settings = NULL. The database default, as specified in the migration, then kicks in: settings = is stored instead of NULL.
Note: Ecto doesn't support embed defaults.
What makes this confusing is that Ecto "hides" this. When reading a user from the
database, Ecto "lazily" applies the default values when reading (when casting the
empty JSONB object to the Settings
schema).
user_with_lazy_defaults = Repo.get!(User, 1)
But a database WHERE
query on settings.notify?
fails to find the user, since the
default notify?
value was not written to the database.
empty_list = Repo.all(from u in User, where: u.settings["notify?"] == true)
The Prestige: How to fix it
This is actually not surprising and luckily quite simple. Since this is not a magic trick, I hope you can forgive me for failing to give you a truly captivating Prestige 😉.
%{settings: %{}}
|> User.changeset()
|> Repo.insert!()
Passing an empty map for the embed changeset is enough.
Ecto then casts this to the Settings
struct and applies the default field values.
We end up with this database users
table state:
id, settings
1, {"notify?": true}
Ecto works as expected
To be clear: This is not an Ecto bug. Ecto is doing exactly what it should. The bug is in my head, because I misinterpret how defaults and embeds work. And because my first test reinforces my understanding.
What made it especially confusing in the project at hand was the fact that it
was a regression bug. The notify?
flag was previously stored top-level as
User.notify?
. Everything worked as expected, because Ecto field defaults for
top-level fields are saved to the database.
Then, a bunch of user flags was grouped and moved into the Settings
embed.
Along came the problem: New users suddenly didn't receive notifications.
But wait: Not all users. That would have been easy to spot 😉.
Instead, there was a default manual process in place: After creating a user,
usually the user settings were manually edited via the UI.
As a result, the lazy user.settings.notify?
default value was saved to the database
This manual process further masked the issue, because most new users did end up with
{"notify?": true}
in the database and were thus notified.
Over to you
Is there a better fix? Should we validate_required(:settings)
?
Or is there a better mental model to avoid falling into this trap in the first place?
The post When to use field :default in an Ecto schema by Dan Schultzer also touches on the subject.