Let it Crash

Designing Robust Systems

Complecting code paths meme with three spidermen pointing at each other

Originally, I submitted this talk for the Dev Connect OWL 2025 conference under the title:

Your Stack is too damn Complex - Let it Crash with Elixir.

But during preparation, the focus and thus title of the talk pivoted. The reason: I asked my friend and former colleague Marcus Autenrieth to hop on deck as co-presenter. Marcus originally introduced me to the BEAM and has great insights to share relating to both tech and people skills. Great choice: More fun for me and Marcus (during prep) and more engaging and interesting for the audience. We decided to

  1. Focus on high-level ideas instead of implementation specifics.
  2. Invite audience participation via open questions and bug-hunting.

The demo is based on Saša Jurić's Soul of Erlang and Elixir (thank you Saša 🙏).

Demo with Audience Participation

github.com/ftes/dcon-elixir-demo

A demo with audience participation - what could go wrong? Luckily: Smooth sailing.

We gave Saša's demo a technical overhaul (phoenix 1.8, liveview 1.1) and UI beauty lift.

Math input in the demo app

The audience was invited

  • test the system
  • produce as much load as possible (in addition the synthetic background load)
  • find all the bugs.

Participation was great. All bugs were found by the audience and then fixed by us. The synthetic background load was tuned to ~80% CPU load. Since we had many audience members' processes to kill, we used Parent.Client.shutdown_all(MySystem.Math).

Questions to the Audience

The talk was not for the faint of heart, we actually asked some questions! The goal was to invite active participation, share knowledge, and provide hooks for follow-up discussion.

We added a simple mnesia + pubsub based bulletin board to the demo app (author names redacted):

Bulletin board in the demo app to gather answers from the audience

Here are the answers:

What are your building blocks for robust systems?

  • Redundancy, Health Checks, failover
  • Monitoring
  • JUnit Tests, load tests
  • Keep it simple, stupid
  • Calculate on client -> less overload in backend
  • Circuit Breaker
  • Excel
  • Loose coupling, async processing
  • Tests
  • As few components as possible
  • EJB 👴
  • I would ask you [editor's note: fair enough!]

How do you find a misbehaving part of your software?

Observability meme with blindfolded dog in fire saying 'This is fine'
  • Patience
  • User
  • Sentry
  • Sentry, Firebase
  • Logfiles
  • PRTG
  • Metrics and Tracing
  • Cadvisor
  • Prometheus
  • otel
  • Prometheus node_exporter
  • Grafana
  • Pentest

Slides

The slide deck is pretty generic and meme-heavy. But here it is in case you find it useful. Check out the speaker notes (hotkey s for reveal.js).

Source: github.com/ftes/dcon-elixir

Further Reading