Phoenix: Test smarter, not harder

Ensuring feature stability in the face of technical changes. A guide on using phoenix_test and playwright for Elixir projects.

Update 2024-11-13
New PR phoenix_test#145: playwright driver with 0 deps.

Have these thoughts crossed your mind?

  • Maintaining feature tests is too much work.

  • Oh no! Converting this dead view into a live view means I'll have to delete or rewrite all the tests.

  • Immediate feedback via Javascript would result in better UX here. But then I have to rewrite my tests. Nvm, I'll do a full handle_event roundtrip instead.

If so, this post is for you! Even though it uses Elixir and Phoenix as a specific example, the concepts and most of the tools presented apply to other ecosystems, too.

Note: I recently gave a talk on this at CodeBEAM Europe 2024 in Berlin. I'll share the full video as soon as it is available. For now, here are the slides and a written breakdown.

Feature tests ensure value

Feature tests are located somewhere in the middle to top of the test pyramid. A single unit of code rarely delivers real value to your end users. So you should also ensure your core features work, front to back, via tests. This gives peace of mind even for large scale refactorings.

test "core feature", %{conn: conn} do
  conn
  |> visit("/purchase")
  |> select("3", from: "Amount")
  |> fill_in("Credit card number", with: "4242 4242")
  |> submit()
  |> assert_has(".success", text: "Thanks for your order!")
end

Why not ditch the unit tests? Complex logic with many edge cases is best suited for unit tests. But one could argue that apart from that, feature tests should be the default. They are both expressive and robust to refactorings.

The reason is that feature tests are written from a user's point of view. And interact with the system just like a user would. So usually, via a web browser, filling in and asserting on visible elements.

Performance concerns often prevent us from writing feature tests. Classical approaches spin up a browser (tab) for every test and include a slow retry mechanism to account for network delays. Including the network also means there's more room for flakiness.

phoenix_test to the rescue

Announced beginning of 2024 by German Velasco, phoenix_test tilts the playing field.

It is heavily inspired by capybara from Ruby-land. capybara backs the ApplicationSystemTestCases in Rails.

The core idea: Polymorphism. Multple test drivers implement a common protocol (interface). One API to rule them all. The specific driver can be swapped, without rewriting tests. Thus, you can choose the best driver for your needs.

And if your needs change, you just swap the driver without having to rewrite all your tests. If you switch a view from dead (controller based) to live view, you get to keep your tests. If you add a Javscript phx-hook, you get to keep your tests (soon ™️). Wow!

We can group drivers into two types:

Driver typecapybara (rails)phoenix_test
Framework-specificrack_teststatic, live
Browser-basedselenium, cupriteplaywright, wallaby

Framework-specific drivers are limited but performant. They use low-level testing utilities provided by the framework (rails, phoenix). This means their performance is comparable to those unit tests. But they are limited to the same level of expressiveness that the framework allows.

So, rack_test-driven tests can only test static server-rendered views. No Javascript, no Stimulus Reflex.

The Phoenix LiveView model is far more expressive, so we can test dynamic views using the live driver. As long as we stick to LiveView.JS commands, we can even test a limited amount of Javascript, because LiveViewTest can apply these mutations to the test DOM.

phoenix_test switches static and live drivers automatically. It detects which implementation backs the current view. This ensures we can write tets at the feature level and don't have to worry about redirects, live patches and other technical details.

Browser-based feature tests

But when we need to include custom Javascript, both rack_test and static + live drivers are of no help.

Our alternatives:

  1. Work around it (ignore the JS parts or "simulate" them, e.g. by filling in hidden inputs).
  2. Switch to a browser-based test.

If (1) is good enough: wonderful. We stay fast, but we lose confidence.

(2) has a performance impact, but it guarantees we can write feature-complete tests.

The amount of performance impact depends heavily on the technology used. selenium and wallaby are generally slower. More full HTTP roundtrips and out-of-browser wait/retry loops are involved.

playwright and cuprite are faster and have less levels of abstraction. cuprite goes the furthest, by talking straight to chromium via CDP (chome devtools protcol) using a vanilla ruby driver, ferrum. However, this sacrifices flexibility and interoperability. Playwright does more at the cost of increased complexity. It supports multiple browsers and has additional tools you may be interested in using.

Status of playwright and wallaby drivers for phoenix_test

Both drivers are currently work in progress 🚧. Check the open pull requests for details.

Both drivers pull in a library of the same name to interact with the browser. Some notes on the maturity of those upstream projects:

  • wallaby can be considered "done" software. While there are some shortcomings, it has been around for a while and is battle tested.
  • playwright-elixir is WIP. You can use it, but it's far from feature complete. Only a subset of the API offered by the official playwright SDKs for other languages is supported. Only chromium can currently be used. I expect the previous sentences to soon be outdated though.

Can I use phoenix_test today?

Yes! If you don't need JS support today, phoenix_test will put the fun back in feature testing for you. It improves developer experience through little things, such as

  • open_browser for dead views (not just live views),
  • elegant, unified API at feature level and
  • helpful error messages (no digging through the entire HTML string).

In the coming months, I expect at least one of the browser-based drivers to become available. Also, advanced features such as chainable locators (see playwright docs to get an idea) will make phoenix_test more powerful in the future.

Please get in touch if you have feedback, ideas or want to try out the WIP drivers!