Planet = erlang.

How would you describe Elixir?

Permalink

How would you describe Erlang?

Permalink

Carrots, sticks, and making things worse

Permalink

An Erlanger's encounter with Gleam | Bartek Gorny

Permalink

Erlang/OTP 27.1 Release

OTP 27.1

Erlang/OTP 27.1 is the first maintenance patch package for OTP 27, with mostly bug fixes as well as improvements.

Highlights

• The zip module has been updated with support for:

  • zip64 archives - Archives larger than 4GB or with more than 2^32 entries.
  • extended timestamps - Higher resolution and in UTC.
  • UID/GID - Save and extract the original UID/GID.
  • Fixes so that permission mode attributes are correctly read and set for files in archives.
  • zip:list_dir/2 now also returns directories, not only files. (You can disable this behaviour by using the option skip_directories).

• All releases now have .zip versions of the Windows installer that can be used to install Erlang/OTP on Windows when you do not have administrator priviliges. You can download it from erlang.org/downloads or from https://github.com/erlang/otp/releases.

  Various bugs in the original implementation have also been fixed, such as:

  • Correctly encoding and decoding the DOS timestamps for entries within an archive (that is the non-extended timestamp).
  • Fixed DOS timestamps to be set to localtime instead of UTC (use extended timestamps for UTC timestamps).
  • Used the unix file attributes read from disk when creating archives instead of setting everything to 644.

Potential incompatibilities:

• Progress reports for a dynamically started supervisor will now be logged at debug level.

For details about bugfixes and potential incompatibilities see the Erlang 27.1 README

The Erlang/OTP source can also be found at GitHub on the official Erlang repository, https://github.com/erlang/otp

Download links for this and previous versions are found here

• https://www.erlang.org/downloads

Permalink

My Blog Engine is the Erlang Build Tool

{blog3r,[{vars,[{url,[{base,"https://ferd.ca/"},{notes,"https://ferd.ca/notes/"},{img,"https://ferd.ca/static/img/"},...]},%% Main site{index,#{template=>"index.tpl",out=>"index.html",section=>main}},{index,#{template=>"rss.tpl",out=>"feed.rss",section=>main}},%% Notes section{index,#{template=>"index-notes.tpl",out=>"notes/index.html",section=>notes}},{index,#{template=>"rss-notes.tpl",out=>"notes/feed.rss",section=>notes}},%% All sections' pages.{sections,#{main=>{"posts/","./",[{"Mon, 02 Sep 2024 11:00:00 EDT","My Blog Engine is the Erlang Build Tool","blog-engine-erlang-build-tool.md.tpl"},{"Thu, 30 May 2024 15:00:00 EDT","The Review Is the Action Item","the-review-is-the-action-item.md.tpl"},{"Tue, 19 Mar 2024 11:00:00 EDT","A Commentary on Defining Observability","a-commentary-on-defining-observability.md.tpl"},{"Wed, 07 Feb 2024 19:00:00 EST","A Distributed Systems Reading List","distsys-reading-list.md.tpl"},...]},notes=>{"notes/","notes/",[{"Fri, 16 Aug 2024 10:30:00 EDT","Paper: Psychological Safety: The History, Renaissance, and Future of an Interpersonal Construct","papers/psychological-safety-interpersonal-construct.md.tpl"},{"Fri, 02 Aug 2024 09:30:00 EDT","Atomic Accidents and Uneven Blame","atomic-accidents-and-uneven-blame.md.tpl"},{"Sat, 27 Jul 2024 12:00:00 EDT","Paper: Moral Crumple Zones","papers/moral-crumple-zones.md.tpl"},{"Tue, 16 Jul 2024 19:00:00 EDT","Hutchins' Distributed Cognition in The Wild","hutchins-distributed-cognition-in-the-wild.md.tpl"},...]}}}]}.

{% extends "base.tpl" %}
{% block content %}

<p>I like cats. I like food. <br />
   I don't especially like catfood though.</p>

{% markdown %}
### Have a subtitle

And then _all sorts_ of content!

- lists
- other lists
- [links]({{ url.base }}section/page))
- and whatever fits a demo

> Have a quote to close this out

{% endmarkdown %}
{% endblock %}

$ rebar3 compile
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling ferd_ca
===> template error:
    ┌─ /home/ferd/code/ferd-ca/templates/rss.tpl:
    │
 24 │      {% endfor %}
    │         ╰── syntax error before: "endfor"

===> Compiling templates/rss.tpl failed

Permalink

Typing lists and tuples in Elixir

We have been working on a type system for the Elixir programming language. The type system provides sound gradual typing: it can safely interface static and dynamic code, and if the program type checks, it will not produce type errors at runtime.

It is important to emphasize type errors. The type systems used at scale today do not guarantee the absense of any runtime errors, but only typing ones. Many programming languages error when accessing the “head” of an empty list, most languages raise on division by zero or when computing the logarithm of negative numbers on a real domain, and others may fail to allocate memory or when a number overflows/underflows.

Language designers and maintainers must outline the boundaries of what can be represented as typing errors and how that impacts the design of libraries. The goal of this article is to highlight some of these decisions in the context of lists and tuples in Elixir’s on-going type system work.

In this article, the words “raise” and “exceptions” describe something unexpected happened, and not a mechanism for control-flow. Other programming languages may call them “panics” or “faults”.

The head of a list

Imagine you are designing a programming language and you want to provide a head function, which returns the head - the first element - of a list, you may consider three options.

The first option, the one found in many programming languages, is to raise if an empty list is given. Its implementation in Elixir would be something akin to:

$ list(a) -> a
def head([head | _]), do: head
def head([]), do: raise "empty list"

Because the type system cannot differentiate between an empty list and a non-empty list, you won’t find any typing violations at compile-time, but an error is raised at runtime for empty lists.

An alternative would be to return an option type, properly encoding that the function may fail (or not):

$ list(a) -> option(a)
def head([head | _]), do: {:ok, head}
def head([]), do: :none

This approach may be a bit redundant. Returning an option type basically forces the caller to pattern match on the returned option. While many programming languages provide functions to compose option values, one may also get rid of the additional wrapping and directly pattern match on the list instead. So instead of:

case head(list) do
  {:ok, head} -> # there is a head
  :none -> # do what you need to do
end

You could just write:

case list do
  [head | _] -> # there is a head
  [] -> # do what you need to do
end

Both examples above are limited by the fact the type system cannot distinguish between empty and non-empty lists and therefore their handling must happen at runtime. If we get rid of this limitations, we could define head as follows:

$ non_empty_list(a) -> a
def head([head | _]), do: head

And now we get a typing violation at compile-time if an empty list is given as argument. There is no option tagging and no runtime exceptions. Win-win?

The trouble with the above is that now it is responsibility of the language users to prove the list is not empty. For example, imagine this code:

list = convert_json_array_to_elixir_list(json_array_as_string)
head(list)

In the example above, since convert_json_array_to_elixir_list may return an empty list, there is a typing violation at compile-time. To resolve it, we need to prove the result of convert_json_array_to_elixir_list is not an empty list before calling head:

list = convert_json_array_to_elixir_list(json_array_as_string)

if list == [] do
  raise "empty list"
end

head(list)

But, at this point, we might as well just use pattern matching and once again get rid of head:

case convert_json_array_to_elixir_list(json_array_as_string) do
  [head | _] -> # there is a head
  [] -> # do what you need to do
end

Most people would expect that encoding more information into the type system would bring only benefits but there is a tension here: the more you encode into types, the more you might have to prove in your programs.

While different developers will prefer certain idioms over others, I am not convinced there is one clearly superior approach here. Having head raise a runtime error may be the most pragmatic approach if the developer expects the list to be non-empty in the first place. Returning option gets rid of the exception by forcing users to explicitly handle the result, but leads to more boilerplate compared to pattern matching, especially if the user does not expect empty lists. And, finally, adding precise types means there could be more for developers to prove.

What about Elixir?

Thanks to set-theoretic types, we will most likely distinguish between empty lists and non-empty lists in Elixir’s type system, since pattern matching on them is a common language idiom. Furthermore, several functions in Elixir, such as String.split/2 are guaranteed to return non-empty lists, which can then be nicely encoded into a function’s return type.

Elixir also has the functions hd (for head) and tl (for tail) inherited from Erlang, which are valid guards. They only accept non-empty lists as arguments, which will now be enforced by the type system too.

This covers almost all use cases but one: what happens if you want to access the first element of a list, which has not been proven to be empty? You could use pattern matching and conditionals for those cases, but as seen above, this can lead to common boilerplate such as:

if list == [] do
  raise "unexpected empty list"
end

Luckily, it is common in Elixir to use the ! suffix to encode the possibility of runtime errors for valid inputs. For these circumstances, we may introduce List.first! (and potentially List.drop_first! for the tail variant).

Accessing tuples

Now that we have discussed lists, we can talk about tuples. In a way, tuples are more challenging than lists for two reasons:

1. A list is a collection where all elements have the same type (be it a list(integer()) or list(integer() or float())), while tuples carry the types of each element

2. We natively access tuples by index, instead of its head and tail, such elem(tuple, 0)

In the upcoming v1.18 release, Elixir’s new type system will support tuple types, and they are written between curly brackets. For example, the File.read/1 function would have the return type {:ok, binary()} or {:error, posix()}, quite similar to today’s typespecs.

The tuple type can also specify a minimum size, as you can also write: {atom(), integer(), ...} . This means the tuple has at least two elements, the first being an atom() and the second being an integer(). This definition is required for type inference in patterns and guards. After all, a guard is_integer(elem(tuple, 1)) tells you the tuple has at least two elements, with the second one being an integer, but nothing about the other elements and the tuple overall size.

With tuples support merged into main, we need to answer questions such as which kind of compile-time warnings and runtime exceptions tuple operations, such as elem(tuple, index) may emit. Today, we know that it raises if:

1. the index is out of bounds, as in elem({:ok, "hello"}, 3)

2. the index is negative, as in elem({:ok, 123}, -1)

When typing elem(tuple, index), one option is to use “avoid all runtime errors” as our guiding light and make elem return option types, such as: {:ok, value} or :none. This makes sense for an out of bounds error, but should it also return :none if the index is negative? One could argue that they are both out of bounds. On the other hand, a positive index may be correct depending on the tuple size but a negative index is always invalid. From this perspective, encoding an always invalid value as an :none can be detrimental to the developer experience, hiding logical bugs instead of (loudly) blowing up.

Another option is to make these programs invalid. If we completely remove elem/2 from the language and you can only access tuples via pattern matching (or by adding a literal notation such as tuple.0), then all possible bugs can be caught by the type checker. However, some data structures, such as array in Erlang rely on dynamic tuple access, and implementing those would be no longer possible.

Yet another option is to encode integers themselves as values in the type system. In the same way that Elixir’s type system supports the values :ok and :error as types, we could support each integer, such as 13 and -42 as types as well (or specific subsets, such as neg_integer(), zero() and pos_integer()). This way, the type system would know the possible values of index during type checking, allowing us to pass complex expressions to elem(tuple, index), and emit typing errors if the indexes are invalid. However, remember that encoding more information into types may force developers to also prove that those indexes are within bounds in many other cases.

Once again, there are different trade-offs, and we must select one that best fit into Elixir use and semantics today.

What about Elixir?

The approach we are taking in Elixir is two-fold:

• If the index is a literal integer, it will perform an exact access on the tuple element. This means elem(tuple, 1) will work if we can prove the tuple has at least size 2, otherwise you will have a type error

• If the index is not a literal integer, the function will fallback to a dynamic type signature

Let’s expand on the second point.

At a fundamental level, we could describe elem with the type signature of tuple(a), integer() -> a. However, the trouble with this signature is that it does not tell the type system (nor users) the possibility of a runtime error. Luckily, because Elixir will offer a gradual type system, we could encode the type signature as dynamic({...a}), integer() -> dynamic(a). By encoding the argument and return type as dynamic, developers who want a fully static program will be notified of a typing error, while existing developers who rely on dynamic features of the language can continue to do so, and those choices are now encoded into the types.

Overall,

• For static programs (the ones that do not use the dynamic() type), elem/2 will validate that the first argument is a tuple of known shape, and the second argument is a literal integer which is greater than or equal to zero and less than the tuple size. This guarantees no runtime exceptions.

• Gradual programs will have the same semantics (and runtime exceptions) as today.

Summary

I hope this article outlines some of the design decisions as we bring a gradual type system to Elixir. Although supporting tuples and lists is a “table stakes” feature in most type systems, bringing them to Elixir was an opportunity to understand how the type system will interact with several language idioms, as well as provide a foundation for future decisions. The most important take aways are:

1. Type safety is a commitment from both sides. If you want your type system to find even more bugs through more precise types, you will need to prove more frequently that your programs are free of certain typing violations.

2. Given not everything will be encoded as types, exceptions are important. Even in the presence of option types, it would not be beneficial for developers if elem(tuple, index) returned :none for negative indexes.

3. Elixir’s convention of using the suffix ! to encode the possibility of runtime exceptions for a valid domain (the input types) nicely complements the type system, as it can help static programs avoid the boilerplate of converting :none/:error into exceptions for unexpected scenarios.

4. Using dynamic() in function signatures is a mechanism available in Elixir’s type system to signal that a function has dynamic behaviour and may raise runtime errors, allowing violations to be reported on programs that wish to remain fully static. Similar to how other static languages provide dynamic behaviour via Any or Dynamic types.

The type system was made possible thanks to a partnership between CNRS and Remote. The development work is currently sponsored by Fresha (they are hiring!), Starfish*, and Dashbit.

Happy typing!

Permalink

Announcing the official Elixir Language Server team

I am glad to welcome Elixir’s official Language Server team, formed by (in alphabetical order):

• Jonatan Kłosko
• Łukasz Samson
• Mitch Hanberg
• Steve Cohen

The team will work on the code intelligence infrastructure to be used across tools and editors. These efforts are partially funded by Fly.io and Livebook.

A brief history

The Language Server Protocol (LSP) was created by Microsoft as a protocol between IDEs and programming languages to provide language intelligence tools.

The first implementation of said protocol for Elixir was started by Jake Becker, back in 2017, alongside an implementation for Visual Studio Code, and it relied on the ElixirSense project from Marlus Saraiva to extract and provide some of the language intelligence.

As the Language Server Protocol adoption grew as a whole, so did the usage of Elixir’s implementation, which eventually became the main mechanism Elixir users interacted with the language from their editors.

Eventually, Elixir’s language server implementation got its own organization on GitHub, and maintenance reins were given to Łukasz Samson and Jason Axelson.

Over time, the Elixir Language Server has accrued technical debt. Some of it exists due to intrinsic complexities (for example, the Language Server Protocol uses UTF-16 for text encoding, instead of the more widely used UTF-8), while others are a consequence of working on codebase while both the programming language and the protocol themselves were still evolving.

This led to Mitch Hanberg and Steve Cohen to create alternative language server implementations, exploring different trade-offs.

For example, both Next LS and Lexical use Erlang Distribution to isolate the Language Server runtime from the user code.

Next LS also focused on extracting the LSP protocol parts into GenLSP (which can be used by anyone to easily create a language server), single binary distribution with Burrito, and experimenting with SQLite for the symbol index.

Lexical concerned itself with speed and abstractions to deal with documents, ranges, and more.

This means the Elixir community had, for some time, three distinct language server implementations, each with their own strengths.

Looking forward

The current language server maintainers have agreed to move forward with a single Language Server Protocol project, relying on the strengths of each implementation:

• Lexical provides a stable foundation
• ElixirLS, through ElixirSense, provides the most complete implementation and wider functionality
• Next LS, through GenLSP, provides a general interface for LSP implementations and straight-forward packaging via Burrito

The above is a rough outline, as the specific details of how the projects will move forward are still being discussed. While some of the team members also maintain direct integration with some editors, we will continue relying on the community’s help and efforts to get full coverage across all available editors.

And there is still a lot more to do!

Many underestimate the complexity behind implementing the Language Server Protocol. That’s not surprising: we mostly interact with it from an editor, allowing us to freely ignore what makes it tick.

In practice, the Language Server needs, in many ways, to reimplement several parts of the language and its compiler.

If the Elixir compiler sees the code some_value +, it can immediately warn and say: “this expression is incomplete”. However, the Language Server still needs to make sense of invalid code to provide features like completion. And that applies to everything: missing do-end blocks, invalid operators, invoking macros that do not exist, etc. Mitch has made Spitfire, an error tolerant parser to tackle this particular problem.

Some ecosystems have undertaken multi-year efforts to redesign their compilers and toolchains to provide better tools for lexical and semantic code analysis (which most likely took a significant investment of time and resources to conclude). That’s to say some of the problems faced by Language Server implementations will be best tackled if they are also solved as part of Elixir itself.

For example, every Language Server implementation compiles their own version of a project, making it so every application and its dependencies have to be compiled twice in development: once for Mix and once for the Language Server. Wouldn’t it be nice if Elixir and the Language Servers could all rely on the same compilation artifacts?

This is not news to the Elixir team either: almost every Elixir release within the last 3 years has shipped new code analysis APIs, such as Code.Fragment, with the goal of removing duplication across Language Servers, IEx, and Livebook, as well as reduce their reliance on internal Elixir modules. Most recently, Elixir v1.17 shipped with new APIs to help developers emulate the compiler behaviour. Our goal is to make these building blocks available for all Elixir developers, so their benefits are reaped beyond the language server tooling.

Furthermore, as set-theoretic types make their way into Elixir, we also want to provide official APIs to integrate them into our tools.

Sponsorships

Currently, Fly.io is sponsoring Łukasz Samson to work part-time on the Language Server and editor integration. The Livebook project is donating development time from Jonatan Kłosko, creator of Livebook, to improve the Elixir compiler and its code intelligence APIs.

We are grateful to both companies for investing into the community and you should check them out.

As mentioned above, Language Server implementations are complex projects, and unifying efforts is an important step in the right direction. However, we also need community help, and one of the ways to do so is by sponsoring the developers making this possible:

• Łukasz Samson
• Mitch Hanberg
• Steve Cohen

Companies who can afford to sponsor part-time development are welcome to reach out and help us achieve this important milestone.

Progress updates

A new project website and social media accounts will be created soon, and you can follow them to stay up to date with our progress and any interesting developments.

The name of the new project is still in the works as well as many of the decisions we’ll need to make, so please have patience!

In the meantime, you can continue to use the language server of your choice, and we’ll be sure to make the transition to the fourth and final project as smooth as possible.

Thank you!

Permalink

Elixir v1.17 released: set-theoretic data types, calendar durations, and Erlang/OTP 27 support

Elixir v1.17 has just been released. 🎉

This release introduces set-theoretic types into a handful of language constructs. While there are still many steps ahead of us, this important milestone already brings benefits to developers in the form of new warnings for common mistakes. This new version also adds support for Erlang/OTP 27, the latest and greatest Erlang release. You’ll also find a new calendar-related data type (Duration) and a Date.shift/2 function.

Let’s dive in.

Warnings from gradual set-theoretic types

This release introduces gradual set-theoretic types to infer types from patterns and use them to type check programs, enabling the Elixir compiler to find faults and bugs in codebases without requiring changes to existing software. The underlying principles, theory, and roadmap of our work have been outlined in “The Design Principles of the Elixir Type System” by Giuseppe Castagna, Guillaume Duboc, José Valim.

At the moment, Elixir developers will interact with set-theoretic types only through warnings found by the type system. The current implementation models all data types in the language:

• binary(), integer(), float(), pid(), port(), reference() - these types are indivisible. This means both 1 and 13 get the same integer() type.

• atom() - it represents all atoms and it is divisible. For instance, the atom :foo and :hello_world are also valid (distinct) types.

• map() and structs - maps can be “closed” or “open”. Closed maps only allow the specified keys, such as %{key: atom(), value: integer()}. Open maps support any other keys in addition to the ones listed and their definition starts with ..., such as %{..., key: atom(), value: integer()}. Structs are closed maps with the __struct__ key.

• tuple(), list(), and function() - currently they are modelled as indivisible types. The next Elixir versions will also introduce fine-grained support to them.

We focused on atoms and maps on this initial release as they are respectively the simplest and the most complex types representations, so we can stress the performance of the type system and quality of error messages. Modelling these types will also provide the most immediate benefits to Elixir developers. Assuming there is a variable named user, holding a %User{} struct with a address field, Elixir v1.17 will emit the following warnings at compile-time:

• Pattern matching against a map or a struct that does not have the given key, such as %{adress: ...} = user (notice address vs adress).

• Accessing a key on a map or a struct that does not have the given key, such as user.adress.

• Invoking a function on non-modules, such as user.address().

• Capturing a function on non-modules, such as &user.address/0.

• Attempting to call an anonymous function without an actual function, such as user.().

• Performing structural comparisons between structs, such as my_date < ~D[2010-04-17].

• Performing structural comparisons between non-overlapping types, such as integer >= string.

• Building and pattern matching on binaries without the relevant specifiers, such as <<name>> (this warns because by default it expects an integer, it should have been <<name::binary>> instead).

• Attempting to rescue an undefined exception or a struct that is not an exception.

• Accessing a field that is not defined in a rescued exception.

Here’s an example of how the warning for accessing a misspelled field of a struct looks like:

Another example, this time it’s a warning for structural comparison across two Date structs:

These warnings also work natively in text editors, as they are standard Elixir compiler warnings:

These new warnings will help Elixir developers find bugs earlier and give more confidence when refactoring code, especially around maps and structs. While Elixir already emitted some of these warnings in the past, those were discovered using syntax analysis. The new warnings are more reliable, precise, and with better error messages. Keep in mind, however, that the Elixir typechecker only infers types from patterns within the same function at the moment. Analysis from guards and across function boundaries will be added in future releases. For more details, see our new reference document on gradual set-theoretic types.

The type system was made possible thanks to a partnership between CNRS and Remote. The development work is currently sponsored by Fresha (they are hiring!), Starfish*, and Dashbit.

Erlang/OTP support

This release adds support for Erlang/OTP 27 and drops support for Erlang/OTP 24. We recommend Elixir developers to migrate to Erlang/OTP 26 or later, especially on Windows. Support for WERL (a graphical user interface for the Erlang terminal on Windows) will be removed in Elixir v1.18.

You can read more about Erlang/OTP 27 in their release announcement. The bits that are particularly interesting for Elixir developers are the addition of a json module and process labels (proc_lib:set_label/1). The latter will also be available in this Elixir release as Process.set_label/1.

New Duration data type and shifting functions

This Elixir version introduces the Duration data type and APIs to shift dates, times, and date times by a given duration, considering different calendars and time zones.

iex> Date.shift(~D[2016-01-31], month: 2)
~D[2016-03-31]

We chose the name “shift” for this operation (instead of “add”) since working with durations does not obey properties such as associativity. For instance, adding one month and then one month does not give the same result as adding two months:

iex> ~D[2016-01-31] |> Date.shift(month: 1) |> Date.shift(month: 1)
~D[2016-03-29]

Still, durations are essential for building intervals, recurring events, and modelling scheduling complexities found in the world around us. For DateTimes, Elixir will correctly deal with time zone changes (such as Daylight Saving Time). However, provisions are also available in case you want to surface conflicts, such as shifting to a wall clock that does not exist, because the clock has been moved forward by one hour. See DateTime.shift/2 for examples.

Finally, we added a new Kernel.to_timeout/1 function, which helps developers normalize durations and integers to a timeout used by many APIs—like Process, GenServer, and more. For example, to send a message after one hour, you can now write:

Process.send_after(pid, :wake_up, to_timeout(hour: 1))

Learn more

Here are other notable changes in this release:

• There are new Keyword.intersect/2,3 functions to mirror the equivalent in the Map module.

• A new Mix profiler was added, mix profile.tprof, which lets you use the new tprof profiler released with Erlang/OTP 27. This profiler leads to the soft-deprecation of mix profile.cprof and mix profile.eprof.

• We added Kernel.is_non_struct_map/1, a new guard to help with the common pitfall of matching on %{}, which also successfully matches structs (as they are maps underneath).

• Elixir’s Logger now formats gen_statem reports and includes Erlang/OTP 27 process labels in logger events.

For a complete list of all changes, see the full release notes.

Check the Install section to get Elixir installed and read our Getting Started guide to learn more.

Happy learning!

Permalink

The Review Is the Action Item

Permalink