How to use web scraping in Elixir to gather useful data | Erlang Solutions Webinar

Permalink

Best Practice For API Design in Elixir - Part 3 | Erlang Solutions Webinar

Permalink

Best Practice For API Design in Elixir - Part 2 | Erlang Solutions Webinar

Permalink

Erlang Records vs. Elixir Structs

= Erlang Records vs. Elixir Structs

I use Erlang and Elixir regularly, and I often get hung up on the differences between Erlang records and Elixir structs. Both serve the same purpose most of the time but are implemented differently.

In this blog post I will document the differences between these two constructs.

//more

== Similarities

The Erlang documentation defines records as:

[quote, ‘link:https://erlang.org/doc/reference_manual/records.html[Erlang reference manual]’] __ A record is a data structure for storing a fixed number of elements. It has named fields and is similar to a struct in C. __

I couldn’t find a succinct definition of Elixir structs on the Elixir website, but the Elixir School website sums them up nicely:

[quote, ‘link:https://elixirschool.com/en/lessons/basics/modules/#structs[Elixir School, structs]’] __ Structs are special maps with a defined set of keys and default values. A struct must be defined within a module, which it takes its name from. __

Both records and structs are defined within a module, but Elixir structs take on the name of the module they are defined in, whereas Erlang records are defined with a name of their own. Both are defined with a set of named fields and, optionally, default values for those fields.

== Differences

The significant differences between the two constructs are shown below. I’m ignoring syntax here because it doesn’t affect how these constructs are used or behave.

[.data] [options=”header”] |===== ||Erlang records|Elixir structs |Field Default Value|undefined|nil |Evaluation of Field Default|Runtime|Compile-time |Underlying Data Structure|Tuple|Map |Performance|Same as tuples|~30% slower than records |=====

=== Field Default Value

Erlang typically uses the undefined atom as an empty value so records use it as their field default value. Structs were created with the Elixir programming language and they use Elixir’s empty value of nil for their fields.

=== Evaluation of Field Default Expression

The field default value for records and structs can be defined as any valid expression. However, field default value expressions in Elixir structs are evaluated at compile time. This means that the compiled struct only contains literal values for field defaults. It is not possible to have a field default that is evaluated at runtime. If we define a struct like this:

[source, elixir]

defmodule Person do defstruct [ :name, # This expression is evaluated at compile time created_at: DateTime.now!(“Etc/UTC”) ] end —-

The created_at field will be populated with the compilation time and not the time the instance of the person struct was created. Erlang records behave differently. Default field value expressions are evaluated at runtime. This makes it possible to have dynamic defaults. In this case, to have a created_at field that is initialized to the current timestamp when the record is created:

[source, erlang]

-record(person, { name, % This expression is evaluated at the time the record is initialized created_at = os:timestamp(), }). —-

When an instance of this person record is created the os:timestamp/0 function would be invoked and the created_at field would be populated with the return value of the os:timestamp/0 function call. Default value expressions like this are not often used but I think the way Erlang records behave with dynamic defaults is more useful.

=== Underlying Data Structure

Elixir structs are maps with a __struct__ key that contains the name of the struct. Erlang records are translated into tuples during compilation with the first tuple item being the name of the record and any expressions using the record syntax are converted to the equivalent tuple expressions. Both records and structs inherit the performance characteristics of their underlying data structures. This brings us to the next difference.

=== Performance

I copied and modified a benchmark script posted by link:https://elixirforum.com/t/elixir-structs-vs-erlang-records/18630/4[OvermindDL1 on the Elixir Forum]. My final benchmark script benchmarks two operations on three different sizes of records and structs. The script is fairly long so I didn’t include it in this post but it is link:/downloads/records-vs-structs/benchmark_output.txt[available here]. The output from the benchmarking script is also available link:/downloads/records-vs-structs/struct_record_bench.exs[here].

Erlang records are faster by about 50%, but neither construct is particularly costly. Only in rare circumstances should the difference in performance merit choosing records over struct. Performance characteristics for updates and limits are similar, with records averaging about 50% faster overall.

== Conclusion

There are differences between records and structs that you need to be aware of. The most significant difference between the two is the underlying data structure, but as we have seen the difference in performance isn’t enough to justify choosing one over the other in most cases. It’s also good to be aware of the subtle difference in the way field default value expressions are used but this shouldn’t dictate what construct you use. Records and structs serve the same purpose and in most cases you should simply use whichever construct is more convenient. In Elixir structs are more widely supported and you should use them in most cases. Erlang doesn’t have any built in support for structs, so you would be better off sticking with records when in Erlang.

== References

• https://erlang.org/doc/reference_manual/records.html
• https://elixirschool.com/en/lessons/basics/modules/#structs
• https://groups.google.com/g/elixir-lang-talk/c/6kn7J2XnFg8/m/I5poTNCEHwAJ
• https://elixirforum.com/t/elixir-structs-vs-erlang-records/18630/5

Permalink

Building Erlang 23.3.4.3 on Devuan/Debian/Ubuntu

Erlang R24 is out! …but there are, as usual with a version X.0 release, a few rough edges (enough of them around the important-but-annoying WX and OpenGL updates that I’ll be writing another post about that shortly…).

In the meantime, R23.3.4.3 is excellent and quite reliable. The steps for building with kerl are nearly identical as for R22, but it is worth re-posting them with the relevant version updates (or if you like the video version, Dr. Kumar made a few demonstrating his home Erlang + ZX build). Also, if you’re in a part of the world where erlang.org’s docs are occasionally really really sloooooooowwww don’t forget the R23 docs mirror (or my Erlang page with linky links to everything).

sudo apt update
sudo apt upgrade
sudo apt install \
    gcc curl g++ dpkg-dev build-essential automake autoconf \
    libncurses5-dev libssl-dev flex xsltproc libwxgtk3.0-dev \
    wget vim git
mkdir vcs bin
cd vcs
git clone https://github.com/kerl/kerl.git
cd ..
ln -s ~/vcs/kerl/kerl bin/kerl
kerl update releases
kerl build 23.3.4.3 23.3.4.3
kerl install 23.3.4.3 ~/.erts/23.3.4.3
echo '. "$HOME"/.erts/23.3.4.3/activate' >> .bashrc
. ~/.erts/23.3.4.3/activate
wget -q https://zxq9.com/projects/zomp/get_zx && bash get_zx

As usual, the ~/vcs/ directory is just my convention for version-controlled code that my $HOME sync scripts know to ignore, and you might want to install Erlang to some place global on your system like /opt/erts/ or whatever. The steps above work without root privileges with the exception of the apt commands. Remember if you are on Devuan or Debian that you need to perform the sudo commands actually as root unless you configure sudo on your system, then the rest as your normal local user account.

Permalink

Random: The Erlang runtime (aka BEAM) when fed bad data

Permalink

Social virtual spaces with Elixir at Mozilla

Welcome to our series of case studies about companies using Elixir in production. See all cases we have published so far.

Hubs is Mozilla’s take on virtual social experiences. You build your own private spaces and share them with your friends, co-workers, and community. Avatars in this space can move freely in a 3D social environment and watch videos, exchange messages, and talk to other people nearby. All you need is a browser and a microphone!

Hubs is fully open source and you can host it on your infrastructure via Hubs Cloud. Community managers, educators, and event organizers have been using Hubs Cloud to run virtual events and online activities tailored to their specific brandings and needs. All it takes to run your own version of Hubs is one click away - which perhaps makes Hubs the most deployed Phoenix application ever!

From VR to Elixir

The Hubs team started at Mozilla as the Mixed Reality team about 3.5 years ago. Their main goal was to explore ways for online social interaction via avatars and mixed reality.

They quickly focused on building their first proof of concept, where avatars could communicate, move around, and join different rooms, everything running directly in the browser. This was a significant departure from the state of the art of Virtual Reality everywhere, as the getting started experience up to this point was cumbersome and often required steep investment in the form of VR headsets.

The initial prototype was a success and it pushed the team to build a product. However, all communication in the proof of concept was peer-to-peer, which limited the features and experiences they could provide. Therefore the Hubs team knew they needed a capable backend technology to provide fan-out communication and coordinate how all different avatars interact within the virtual spaces. John Shaughnessy, Staff Software Engineer at Mozilla, comments: “When you get a lot of people in the same space, be it virtual or in the real world, there is never only a single conversation going on. Once you get ten or twenty different people in a room, conference calls don’t work well. In Hubs, people transition between multiple simultaneous conversations simply by moving around”.

With this bold vision in hand, they assessed their whole stack. They settled on using JavaScript with Three.js in the front-end and chose the Phoenix web framework for the backend. Greg Fodor, who was an Engineering Manager at Mozilla at the time, explains the choice: “We first listed all of the features we had to implement, from trivial things like REST endpoints, to more complex use cases, such as chat messaging and tracking where avatars are in the virtual world. Once I started to learn Phoenix, I saw all of those features were already there! The application we were building has to manage a large number of connections with real-time low latencies, something we knew the Erlang VM was an excellent fit for”.

In production

Hubs went live in January 2018. Almost everything in Hubs goes through Phoenix. The only exception is the WebRTC voice channels, which are handled by designated voice servers, initially implemented with Janus and later ported to MediaSoup. However, the Phoenix app still manages the voice servers and how connections are assigned to them.

The deployment is orchestrated by Habitat and running on Amazon EC2. Habitat provides packaging and orchestration. When a voice server joins the Habitat ring, the Phoenix services receive a message and start assigning voices to voice servers. Overall they run on 4 Phoenix and 4 voice servers.

The Elixir experience in production has been quite smooth. Dominick D’Aniello, Staff Software Engineer at Mozilla, points out some areas they discussed improving: “the Phoenix application works mostly as a proxy, so we avoid decoding and reencoding the data unless we really need to. But sometimes we have to peek at the payloads and JSON is not the most efficient format to do so.” They have also considered relying more on Elixir processes and the Erlang distribution. Dominick continues: “when a new client joins, it needs to ask all other clients what is their state in the world, what they own, and what they care about. One option is to use Elixir processes in a cluster to hold the state of the different entities and objects in the virtual world”.

Beyond Hubs

With many large companies investing in online communication, the Mozilla team saw the possibility of virtual spaces becoming walled-gardens inside existing social platforms. This led the Hubs team to work on Hubs Cloud, with the mission to commoditize virtual spaces by allowing anyone to run their own version of Hubs with a single click.

Hubs Cloud launched in February 2020 and it has been a hit. New York University did its graduation ceremony on a Hubs Cloud instance. The IEEE Virtual Reality Conference embraced Hubs for a more accessible and sustainable event with talks and poster sessions all happening in virtual rooms, while the Minnesota Twins baseball team launched a Virtual Hall of Fame on top of the platform.

Their cloud version uses Amazon CloudFormation to instantiate Hubs inside the user’s account. This approach brought different challenges to the Hubs team: “we want Hubs Cloud to be as affordable and straightforward as possible. The Phoenix app has already been a massive help on this front. We have also moved some features to Amazon Lambda and made them optional, such as image resizing and video conversion” - details John.

Since Hubs is also open source, developers can run their own Hubs instance in whatever platform they choose or change it however they want. That’s the path Greg Fodor recently took when he announced Jel: “Jel is the video game for work. It is a mashup of Minecraft and Discord, where everything is 3D. My goal is to spark new directions and ideas to get people excited about VR”.

Summing up

Today, the Hubs team has 10 contributors, half of whom are developers. Their engineering team is quite general and learning Elixir happens organically: “you are motivated by the feature you are working on. If it requires changing the backend, you learn Elixir with the help of the team and then make your contribution”.

Overall, the bet on Phoenix was a successful one. Greg Fodor highlights: “The most significant benefit of Phoenix is in using a stack that excels at solving a large variety of problems. Once onboarded to Phoenix, there is a huge surface area our engineers can touch. Any feature they come up with, they can run with it. And because Hubs is open source, our contributors will also have the same experience. Overall, Elixir and Phoenix reduce the effort needed to cause the largest impact possible across our whole product”.

Lately, they have leaned even further into the ecosystem, as they have started exposing Hubs APIs over GraphQL with the help of Absinthe. They have also migrated to Phoenix v1.5 and are using the Phoenix LiveDashboard to provide metrics and instrumentation to Hubs Cloud users.

Permalink

You've got to upgrade Rebar3

request(Url,ETag)->HttpOptions=[{ssl,ssl_opts(Url)},{relaxed,true}|rebar_utils:get_proxy_auth()],casehttpc:request(get,{Url,[{"if-none-match","\""++ETag++"\""}||ETag=/=false]++[{"User-Agent",rebar_utils:user_agent()}]},HttpOptions,[{body_format,binary}],rebar)of% ...

request(Config,Name,Version,ETag)->Config1=Config#{http_etag=>ETag},tryhex_repo:get_tarball(Config1,Name,Version)of% ...

<MononcQc> what the hell where are ssl opts configured in our stack
<MononcQc> fuck I think we might have broken ssl but I'll need to double-check
<MononcQc> shit we do. God fucking damn it

Permalink

Elixir v1.12 released

Elixir v1.12 is out with improvements to scripting, tighter Erlang/OTP 24 integration, stepped ranges, and dozen of new functions across the standard library. Overall this is a small release, which continues our tradition of bringing Elixir developers quality of life improvements every 6 months. Some of these improvements directly relates with the recent efforts of bringing Numerical Computing to Elixir.

Elixir v1.12 requires Erlang/OTP 22+. We also recommend running mix local.rebar after installation to upgrade to the latest Rebar versions, which includes support for Erlang OTP/24+.

Note: this announcement contains asciinema snippets. You may need to enable 3rd-party JavaScript on this site in order to see them. If JavaScript is disabled, noscript tags with the proper links will be shown.

Scripting improvements: Mix.install/2 and System.trap_signal/3

Elixir v1.12 brings new conveniences for those using Elixir for scripting (via .exs files). Elixir has been capable of managing dependencies for a quite long time, but it could only be done within Mix projects. In particular, the Elixir team is wary of global dependencies as any scripts that rely on system packages are brittle and hard to reproduce whenever your system changes.

Mix.install/2 is meant to be a sweet spot between single-file scripts and full-blown Mix projects. With Mix.install/2, you can list your dependencies at the top of your scripts. When you execute the script for the first time, Elixir will download, compile, and cache your dependencies before running your script. Future invocations of the script will simply read the compiled artifacts from the cache:

Mix.install([:jason])
IO.puts(Jason.encode!(%{hello: :world}))

Mix.install/2 also performs protocol consolidation, which gives script developers an option to execute their code in the most performant format possible. Note Mix.install/2 is currently experimental and it may change in future releases.

Furthermore, Mix.install/2 pairs nicely with Livebook, a newly announced project that brings interactive and collaborative notebook projects to Elixir. With Livebook and Mix.install/2, you can bring dependencies into your code notebooks and ensure they are fully reproducible. Watch the Livebook announcement to learn more.

Another improvement to scripting is the ability to trap exit signals via System.trap_signal/3. All you need is the signal name and a callback that will be invoked when the signal triggers. For example, ExUnit leverages this functionality to print all currently running tests when you abort the test suite via SIGQUIT (Ctrl+\\ ). You can see this in action when running tests in the Plug project below:

This is particularly useful when your tests get stuck and you want to know which one is the culprit.

Important: Trapping signals may have strong implications on how a system shuts down and behaves in production and therefore it is extremely discouraged for libraries to set their own traps. Instead, they should redirect users to configure them themselves. The only cases where it is acceptable for libraries to set their own traps is when using Elixir in script mode, such as in .exs files and via Mix tasks.

Tighter Erlang/OTP 24 integration

Erlang/OTP 24 ships with JIT compilation and Elixir developers don’t have to do anything to reap its benefits. There are many other features in Erlang/OTP 24 to look forwards to and Elixir v1.12 provides integration with many of them: such as support for 16bit floats in bitstrings as well as performance improvements in the compiler and during code evaluation.

Another excellent feature in Erlang/OTP 24 is the implementation of EEP 54, which provides extended error information for many functions in Erlang’s stdlib. Elixir v1.12 fully leverages this feature to improve reporting for errors coming from Erlang. For example, in earlier OTP versions, inserting an invalid argument into an ETS table that no longer exists would simply error with ArgumentError:

However, in Elixir v1.12 with Erlang/OTP 24:

Finally, note Rebar v2 no longer works on Erlang/OTP 24+. Mix defaults to Rebar v3 since Elixir v1.4, so no changes should be necessary by the vast majority of developers. However, if you are explicitly setting manager: :rebar in your dependency, you want to move to Rebar v3 by removing the :manager option. Compatibility with unsupported Rebar versions will be removed from Mix in the future.

Stepped ranges

Elixir has had support for ranges from before its v1.0 release. Ranges support only integers and are inclusive, using the mathematic notation a..b. Ranges in Elixir are either increasing 1..10 or decreasing 10..1 and the direction of the range was always inferred from the first and last positions. Ranges are always lazy as its values are emitted as they are enumerated rather than being computed upfront.

Unfortunately, due to this inference, it is not possible to have empty ranges. For example, if you want to create a list of n elements, you cannot express it with a range from 1..n, as 1..0 (for n=0) is a decreasing range with two elements.

Elixir v1.12 supports stepped ranges via the first..last//step notation. For example: 1..10//2 will emit the numbers 1, 3, 5, 7, and 9. You can consider the // operator as an equivalent to “range division”, as it effectively divides the number of elements in the range by step, rounding up on inexact scenarios. Steps can be either positive (increasing ranges) or negative (decreasing ranges). Stepped ranges bring more expressive power to Elixir ranges and they elegantly solve the empty range problem, as they allow the direction of the steps to be explicitly declared instead of inferred.

As of Elixir v1.12, implicitly decreasing ranges are soft-deprecated and warnings will be emitted in future Elixir versions based on our deprecation policy.

then/2 and tap/2

Two new functions have been added to Kernel module, in order to ease working with pipelines. tap/2 passes the given argument to an anonymous function, returning the argument itself. then/2 passes the given argument to an anonymous function, returning the result. The following:

"hello world"
|> tap(&IO.puts/1)
|> then(&Regex.scan(~r/\w+/, &1))

Is equivalent to this:

"hello world"
|> (fn x ->
      IO.puts(x)
      x
    end).()
|> (&Regex.scan(~r/\w+/, &1)).()

Both tap/2 and then/2 are implemented as macros, and compiler improvements available on Erlang/OTP 24 ensure the intermediate anonymous functions is optimized away, which guarantees the idioms above do not have any performance impact on your code.

IEx improvements

IEx got two important quality of life improvements in this release. Hitting tab after a function invocation will show all of the arguments for said function and it is now possible to paste code with pipelines in the shell. See both features in action below:

Additional functions

Elixir v1.12 has also added many functions across the standard library. The Enum module received additions such as Enum.count_until/2, Enum.product/1, Enum.zip_with/2, and more. The Integer module now includes Integer.pow/2 and Integer.extended_gcd/2.

The Code module got a cursor_context/2 function, which is now used to power IEx autocompletion and it is used by projects such as Livebook to provide intellisense.

The EEx application has also been extended to provide metadata on text segments. This has enabled the Surface and Phoenix LiveView teams to implement a new template language called HEEx, which validates both HTML and EEx. Finally, the Registry module supports the :compressed option, which is useful for GraphQL applications managing hundreds of thousands of subscriptions via Absinthe.

For a complete list of all changes, see the full release notes. Check the Install section to get Elixir installed and read our Getting Started guide to learn more.

Have fun!

Permalink

OTP 24.0 Release

img src=http://www.erlang.org/upload/news/

Permalink