Elixir v1.6 released

Elixir v1.6 includes new features, enhancements, and bug fixes. The main feature in this release is a code formatter. Important improvements can also be found in the standard library and in the Mix build tool.

Code formatter

The big feature in Elixir v1.6 is the addition of a code formatter and an accompanying mix format task that adds automatic formatting to your projects.

The goal of the formatter is to automate the styling of codebases into a unique and consistent layout used across teams and the whole community. Code is now easier to write, as you no longer need to concern yourself with formatting rules. Code is also easier to read, as you no longer need to convert the styles of other developers in your mind.

The formatter also helps new developers to learn the language by giving immediate feedback on code structure, and eases code reviews by allowing teams to focus on business rules and code quality rather than code style.

To automatically format your codebase, you can run the new mix format task. A .formatter.exs file may be added to your project root for rudimentary formatter configuration. The mix task also supports flags for CI integration. For instance, you can make your build or a Pull Request fail if the code is not formatted accordingly by using the --check-formatted flag. We also recommend developers to check their favorite editor and see if it already provides key bindings for mix format, allowing a file or a code snippet to be formatted without hassle.

The Elixir codebase itself has already been fully formatted and all further contributions are expected to contain formatted code. We recommend existing codebases to be formatted in steps. While the formatter will correctly handle long lines and complex expressions, refactoring the code by breaking those into variables or smaller functions as you format them will lead to overall cleaner and more readable codebases.

Dynamic Supervisor

Supervisors in Elixir are responsible for starting, shutting down, and restarting child processes when things go wrong. Most of the interaction with supervisors happens through the Supervisor module and it provides three main strategies: :one_for_one, :rest_for_one and :one_for_all.

However, sometimes the children of a supervisor are not known upfront and are rather started dynamically. For example, if you are building a web server, you have each request being handled by a separate supervised process. Those cases were handled in the Supervisor module under a special strategy called :simple_one_for_one.

Unfortunately, this special strategy changed the semantics of the supervisor in regards to initialization and shutdown. Plus some APIs expected different inputs or would be completely unavailable depending on the supervision strategy.

Elixir v1.6 addresses this issue by introducing a new DynamicSupervisor module, which encapsulates the old :simple_one_for_one strategy and APIs in a proper module while allowing the documentation and API of the Supervisor module to focus on its main use cases. Having a separate DynamicSupervisor module also makes it simpler to add new features to the dynamic supervisor, such as the new :max_children option that limits the maximum number of children supervised dynamically.

@deprecated and @since attributes

This release also introduces two new attributes associated with function definitions: @deprecated and @since. The former marks if a function or macro is deprecated, the latter annotates the version the API was introduced:

@doc "Breaks a collection into chunks"
@since "1.0.0"
@deprecated "Use chunk_every/2 instead"
def chunk(collection, chunk_size) do
  chunk_every(collection, chunk_size)
end

The mix xref task was also updated to warn if your project calls deprecated code. So if a definition is marked as @deprecated and a module invokes it, a warning will be emitted during compilation. This effectively provides libraries and frameworks a mechanism to deprecate code without causing multiple warnings to be printed in runtime and without impacting performance.

Note those attributes are not yet available to tools that generate documentation. Such functionality will be added in Elixir v1.7 once Elixir adopts EEP-48. We still recommend developers to start annotating their APIs so the information is already available when the tooling is updated.

defguard and defguardp

Elixir provides the concepts of guards: expressions used alongside pattern matching to select a matching clause. Let’s see an example straight from Elixir’s home page:

def serve_drinks(%User{age: age}) when age >= 21 do
  # Code that serves drinks!
end

%User{age: age} is matching on a User struct with an age field and when age >= 21 is the guard.

Since only a handful of constructs are allowed in guards, if you were in a situation where you had to check the age to be more than or equal to 21 in multiple places, extracting the guard to a separate function would be less than obvious and error prone. To address those issues, this release introduces defguard/1 and defguardp/1:

defguard is_drinking_age(age) when age >= 21

def serve_drinks(%User{age: age}) when is_drinking_age(age) do
  # Code that serves drinks!
end

IEx improvements

IEx also got its share of improvements. The new code formatter allows us to pretty print code snippets, types and specifications, improving the overall experience when exploring code through the terminal.

The autocomplete mechanism also got smarter, being able to provide context autocompletion. For example, typing t Enum. and hitting TAB will autocomplete only the types in Enum (in contrast to all functions). Typing b GenServer. and hitting TAB will autocomplete only the behaviour callbacks.

Finally, the breakpoint functionality added in Elixir v1.5 has been improved to support pattern matching and guards. For example, to pattern match on a function call when the first argument is the atom :foo, you may do:

iex> break! SomeFunction.call(:foo, _, _)

For more information, see IEx.break!/4.

mix xref

mix xref is a task added in Elixir v1.3 which provides general information about how modules and files in an application depend on each other. This release brings many improvements to xref, extending the reach of the analysis and helping developers digest the vast amount of data it produces.

One of such additions is the --include-siblings option that can be given to all xref commands inside umbrella projects. For example, to find all of the callers of a given module or function of an application in an umbrella:

$ mix xref callers SomeModule --include-siblings

The graph command in mix xref now can also output general statistics about the graph. In the hexpm project, you would get:

$ mix xref graph --format stats
Tracked files: 129 (nodes)
Compile dependencies: 256 (edges)
Structs dependencies: 46 (edges)
Runtime dependencies: 266 (edges)

Top 10 files with most outgoing dependencies:
  * test/support/factory.ex (18)
  * lib/hexpm/accounts/user.ex (13)
  * lib/hexpm/accounts/audit_log.ex (12)
  * lib/hexpm/web/controllers/dashboard_controller.ex (12)
  * lib/hexpm/repository/package.ex (12)
  * lib/hexpm/repository/releases.ex (11)
  * lib/hexpm/repository/release.ex (10)
  * lib/hexpm/web/controllers/package_controller.ex (10)
  * lib/mix/tasks/hexpm.stats.ex (9)
  * lib/hexpm/repository/registry_builder.ex (9)

Top 10 files with most incoming dependencies:
  * lib/hexpm/web/web.ex (84)
  * lib/hexpm/web/router.ex (29)
  * lib/hexpm/web/controllers/controller_helpers.ex (29)
  * lib/hexpm/web/controllers/auth_helpers.ex (28)
  * lib/hexpm/web/views/view_helpers.ex (27)
  * lib/hexpm/web/views/icons.ex (27)
  * lib/hexpm/web/endpoint.ex (23)
  * lib/hexpm/ecto/changeset.ex (22)
  * lib/hexpm/accounts/user.ex (19)
  * lib/hexpm/repo.ex (19)

mix xref graph also got the --only-nodes and --label options. The former asks Mix to only output file names (nodes) without the edges. The latter allows you to focus on certain relationships:

# To get all files that depend on lib/foo.ex
mix xref graph --sink lib/foo.ex --only-nodes

# To get all files that depend on lib/foo.ex at compile time
mix xref graph --label compile --sink lib/foo.ex --only-nodes

# To get all files lib/foo.ex depends on
mix xref graph --source lib/foo.ex --only-nodes

# To limit statistics only to compile time dependencies
mix xref graph --format stats --label compile

Those improvements will help developers better understand the relationship between files and reveal potentially complex parts of their systems.

Other improvements in Mix include better compiler diagnostics for editor integration, support for the --slowest N flag in mix test that shows the slowest tests in your suite, and a new mix profile.eprof task that provides time based profiling, complementing the existing mix profile.cprof (count based) and mix profile.fprof (flame based).

Summing up

The full list of changes is available in our release notes. There are many other exciting changes, such as compiler enhancements to better track dependencies, leading to less files being recompiled whenever there are changes in project, and overall faster compilation.

Work on Elixir v1.7 has already started. We still welcome developers to try out the previously announced StreamData library, that aims to bring data generation and property-based testing to Elixir. The other features scheduled for v1.7 can be found in the issues tracker.

Don’t forget to check the Install section to get Elixir installed and our Getting Started guide to learn more.

Permalink

Functional Mocks with Mox in Elixir

 

If you’re like me, you came over to Elixir from Ruby and quickly found that certain development assumptions so common to Ruby (and object-oriented programming) require some adjustment in this new functional language.

Recently, I was writing an Elixir-based Alexa skill that needed to reach out to an external service, the Alexa Device Address API. I needed to find the right way to abstract away this concern in a testable manner:

However, writing tests for this function is very difficult. It calls a live API, which renders these tests difficult to set up, slow, and potentially brittle. It mixes both domain-specific data transformation functions with calls to an external API system we do not control. How can we manage this complexity?

Push I/O to the edges with wrapped modules

When designing applications with a layered approach, we try to protect our core domain logic from needing to know about the outside world. These actions, like accessing a database, fetching from a remote API, or receiving inputs from a web service request, are only done at the “edges” of the transaction and wrapped in modules that we own. This is often referred to as a “hexagonal” or a “functional” architecture. Doing so allows our domain logic to remain isolated from impure concerns, and our tests to be cleanly written with deterministic expectations.

So let’s push the HTTP concerns to the outer edges of our design and wrap our HTTP service in its own module, allowing the rest of our app code to collaborate with a module that we own instead of needing to access the messy, unpredictable outside world:

That feels better! The AmazonDeviceApi module now has the exclusive responsibility of wrapping communication with an external API, and now all collaborating modules merely have to deal with the wrapper module, which is under our control.

Now, how are we now to write tests for the original Alexa module? The Alexa module becomes a collaborator with the AmazonDeviceApi module, but we have no way to get to it in our tests:

Ruby mocking versus Elixir mocks

If you were like me and came from Ruby, its dynamic nature allowed us to modify classes and objects at runtime and install mocks from within our tests. RSpec allowed (and encouraged) us to gloriously assert method invocations and stub out methods at test runtime.

When many of us in the Ruby ecosystem came over to Elixir, we felt the tug to bring those same habits to our Elixir code. There are many mocking frameworks that help us try to recreate that experience in Elixir, but wait! Not so fast.


Join us in LA for a one-day Elixir Conference

EMPEX LA | February 10, 2018 →


Early last year, José Valim wrote a blog post called “Mocks and Explicit Contracts” where he came out strongly against implementing mocks in a manner that dynamically modified global modules. He comes out against mocking globals, because “…when you use mock as a verb, you are changing something that already exists, and often those changes are global.” The act of rewiring your program with metaprogramming magic for the sake of a test is, to José, considered harmful. Additionally, it prevents tests from running asynchronously and in parallel.

Instead, he advocates creating Elixir Behaviours for your module interface, then creating test mocks that conform to the interfaces that are swapped in during tests. The code is more readable, easier to reason about, and easier to test.

Let’s try that out by first creating a behavior for our HTTP client, then implementing it in our module:

Note how we fetch the correct implementation of the AlexaDeviceApi via a runtime config variable lookup. This will become important as we turn to our tests, where we will supply a mock to the caller context.

Enter mox. Mox is a library that allows you to define test double behavior at test definition time. Let’s see how that works:

Tada! Now our tests inside the domain boundaries run fast because they run up against in-memory mocks. And we can write a simple integration test that verifies that the actual calls to the live API are working.

Additionally, the use of a behaviour enforces that our test and our live implementations never fall out of sync – they are required to conform to the same interface.

Some finer points of mocks and stubs

I always like to refer back to Martin Fowler’s Mocks Aren’t Stubs article for clearer definition on our test components.

Jose’s article refers to the use of a “mock as a noun” – which I would clarify to be a test fake. He advocates creating static, preprogrammed mock modules with canned responses.

The Mox library allows us to write our test objects both as fakes – modules with canned, preprogrammed responses, but also as mocks – modules that verify calls on functions with specific values.

Here we define a mock verification routine in our test, using the Mox mock:

The notable thing here is that the verification of the function call is written into our test.

And here we use a Mox stub, (what Fowler calls a fake):

The notable thing here is that we do not care whether the test function is invoked or not – merely that if it were ever to be called, it should always return a canned value.

Should you use a stub or a mock? It depends on your requirements in your test. Is the called function (collaborating function) a critical part of your module’s behavior, and is the data passed to it important to verify? You should use a mock function. Is calling the collaborating function merely a side effect of calling your subject and not the main verification purpose of your test? You may want to use a stub.

When should I use Mox?

This process is notably nicer than our original approach, but it still requires a lot of ceremony to wire up and get running in your application. There’s a lot of configuration and boilerplate involved. Clearly, this will become a pain if every module is required to define its own mock. So when should you use Mox mocks?

I suggest using mocks at context (system) boundaries – for example, the top-level module of each application context if you’re using a Phoenix context. A mock can hide away complicated implementation details of the entire system behind it, and the usage of a mock can also drive your design to unify behind a coherent, single interface. Collaborating callers can then use it to implement their own collaboration behaviors with this system.

If you are the downstream consumer of a software system that you may not control, you should be implementing a mock at the outermost entry point to that system.

When should I not use Mox?

Mocks are clearly not appropriate for every module you ever write! How, then should we be writing tests for functions that call across module boundaries, but the two modules are closely-related within the same system?

Tune in to our next blog post to discover some other ways to decouple your modules in a lightweight, test-driven fashion. Got comments? Let us know on Twitter!

Permalink

Why Erlang?

Abstract

With web applications today there is an increasing need for systems that can scale up easily and tolerate the network, hardware, and software errors that occur at scale. Building such systems is challenging, if not impossible, to do in many popular programming languages today. Erlang was designed for telephone systems where uptime requirements were law, and as such was designed to solve these challenges. In this talk Trevor will show how Erlang can be used to solve some of the most challenging problems software developers face today. He will introduce concurrency constructs that Erlang provides and show how they can be used to build reliable and scalable applications. This talk will focus mostly on the new concepts introduced by Erlang, but Trevor will also do a few demos and show code samples to illustrate some of the concepts.

Slides

https://speakerdeck.com/stratus3d/why-erlang

Permalink

This Week In Erlang Jan 12

Welcome to another “This week in Erlang” newsletter.

Articles and Blog posts

Library of the week

There has been a lot of talk about blockchain technologies lately. So, why not talk about one here? The Archain project is a new archiving blockchain technology based on Erlang. Sam Williams (@samecwilliams) did a talk about the whole project at Code Mesh 2017:

Archain Talk: https://www.youtube.com/watch?v=YO3zyQvL3Yg

Archain Repo: https://github.com/ArchainTeam/archain

Employment

Announcements

Permalink

The Asymmetry of ++ Validations

found by Fede Bergero

Let’s kick off 2018 with a super simple Erlang oddity that can be summarized in a tweet from Fede Bergero:

I won’t add much to this, but I thought I would write an article just because I think this totally belongs to this blog.

Partial Asymmetry found at an Iranian Tomb Ceiling — Source

The Findings

Let’s try what Fede suggested in a console…

1> [] ++ something.
something
2> something ++ [].
** exception error: bad argument
in operator ++/2
called as something ++ []
3>

Since we already learnt that compiled codeevaluated code, let’s try this within a module…

https://medium.com/media/32d42b4505306e3ce30636d0f657ccbe/href
1> c(plusplus).
plusplus.erl:7: Warning: this expression will fail with a 'badarg' exception
{ok,plusplus}

That’s cool! Erlang compiler warns us, but only when the empty list is the second argument (as expected).

2> plusplus:first().
something
3> plusplus:second().
** exception error: bad argument
in function plusplus:second/0 (plusplus.erl, line 7)
4>

And this time compiled and evaluated codes behave consistently. That’s good.

What’s going on here?

This time it’s not so hard to see: ++ only validates that its first argument is a list. The second argument can be anything, and whatever it is will be added as the tail to the generated list. Check this out…

5> [1,2,3] ++ something.
[1,2,3|something]
6>

It’s just how ++ works. What Fede found is just an edge case where the resulting list has no head, and therefore it’s only its tail. It’s as if we would have constructed [head|something] and requested its tail…

7> tl([head|something]).
something
8>

So, mystery solved. But to be fair, the only slightly relevant documentation I could find about this was the following one:

The list concatenation operator ++ appends its second argument to its first and returns the resulting list.

It might be a good thing to add some more implementation/spec details to the Erlang docs.

In any case, typer (and consequently dialyzer) seems to be aware of all this. Check this module annotated by typer with specs:

https://medium.com/media/1beed7fa5ef821d5f1c416a79479eaaf/href

Note how typer clearly understands that first/0 will return just the atom something since the resulting improper list will have no head, second/0 will fail (therefore the result is none()) and third/0 will return an improper list. Actually, typer is very precise: it will be a non empty improper list with 1|2|3 as its elements and something as its tail.


The Asymmetry of ++ Validations was originally published in Erlang Battleground on Medium, where people are continuing the conversation by highlighting and responding to this story.

Permalink

“Because it works” is a justification of last resort

Reliability is Hard. Oh, there are tools to make it easier, but “easier” is not the same as “easy”. And Reliability is, certainly, not Easy. When you factor in the (geometrical) complexity of distributed components, and the (geometrical) complexities that get layered on top when these components span nodes, and then networks, well, “Reliability is Hard” gets close to being the Understatement of the Year.
And that’s where rigor comes in, rigor in the form of empirical tests, rigor in the form of mathematical certainties, rigor in the form of proofs, models, and battle-testing, in short RIGOR.
Do you, genuinely, understand what your system does?
Can you explain this to others, simply, in a form they can understand (a-la Feynman)?
Most importantly, is there any place in the system where you up with
  • “I hand-tuned this till it worked”
  • “I implemented my own <something in common use, that I didn’t know existed>
  • “It doesn’t matter, it works, and that’s what’s important”
If the above sounds familiar (and it should, I’ve seen this, lo, too many times in my life) you might want to take a step back and examine your priors…

Permalink

Distributed Systems, and Coding for Failure

The world of Distributed Systems is as much about philosophy as it is about implementation — the specific context, architecture, tooling, and developers all come together to make your way different from everybody else’s.
And yet yet, there are overarching themes and patterns that hold true, that you ignore at your own peril. The most important of these is the necessity to architect for failure — everything springs from this. (°)
Complexity grows geometrically with the number of components, and no amount of fakes, strict contracts, mocks, and the like can reduce this. If anything, poor practices will only increase this complexity
The key is to accept this, and align the risk associated with a given service/component with the risk the business is willing to take. In short, make the component reliable, but no more reliable than necessary”
As a developer, what this means is that you must
  • • Understand the operational semantics of the system as a whole
  • • Internalize the characteristics of the dependencies, especially for your component (on both sides! Things it depends on, and things that depend on it!) 
  • • Make Observability your mantra (°°)
Live this, learn from this, pass it on…
(°) Yes #Erlang peoples, shades of “Let It Crash” here. #FaultTolerance is the name of the game when it comes to building out distributed systems, and, to date, there is no better environment to do this in than Erlang…
(°°) #Instrumentation =/= #Observability. It’s a subset thereof. There is also debugging, exploring, logging, metrics, and a s**t-ton more. Go Google this…

Permalink

Copyright © 2016, Planet Erlang. No rights reserved.
Planet Erlang is maintained by Proctor.