I recently published an article on the Code for RentPath blog about running Dialyzer with GitHub Actions.
[quote, ]
__In this blog post I’ll show you how to set up GitHub Actions to perform type analysis on an Elixir project with link:https://erlang.org/doc/man/dialyzer.html[Dialyzer]. I’ll also share optimal settings for Dialyzer PLT caching.
Running automated tests for every commit or pull request has become common practice and is a good way to validate the correctness of your Erlang or Elixir software. Type checking with Dialyzer is another way to ensure the quality of your code changes.
I have previously written about link:http://stratus3d.com/blog/2020/05/01/running-dialyzer-in-jenkins-builds/[how to run Dialyzer in a Jenkins build] and here I’ll share how to do the same with a GitHub workflow.
__
Read the rest of the article on link:https://blog.rentpathcode.com/running-dialyzer-for-elixir-projects-in-github-actions-e0594220b272[blog.rentpathcode.com].
ceverett@icecream:~/Code/python$ cat hello.py
#! /usr/bin/python3
print("Hello, world!")
ceverett@icecream:~/Code/python$ ./hello.py
Hello, world!
ceverett@icecream:~/Code/python$ valgrind ./hello.py
...
==19264== HEAP SUMMARY:
==19264== in use at exit: 320,946 bytes in 166 blocks
==19264== total heap usage: 1,698 allocs, 1,532 frees, 2,505,946 bytes allocated
==19264==
==19264== LEAK SUMMARY:
==19264== definitely lost: 0 bytes in 0 blocks
==19264== indirectly lost: 0 bytes in 0 blocks
==19264== possibly lost: 5,608 bytes in 10 blocks
==19264== still reachable: 315,338 bytes in 156 blocks
==19264== suppressed: 0 bytes in 0 blocks
...
==19264== ERROR SUMMARY: 8163 errors from 116 contexts (suppressed: 0 from 0)
ceverett@icecream:~/Code/python$
Python 2.7
ceverett@icecream:~/Code/python$ cat hello2.py
#! /usr/bin/python
print "Hello, world!"
ceverett@icecream:~/Code/python$ ./hello2.py
Hello, world!
ceverett@icecream:~/Code/python$ valgrind ./hello2.py
...
==19416== HEAP SUMMARY:
==19416== in use at exit: 417,246 bytes in 199 blocks
==19416== total heap usage: 2,958 allocs, 2,759 frees, 3,085,205 bytes allocated
==19416==
==19416== LEAK SUMMARY:
==19416== definitely lost: 0 bytes in 0 blocks
==19416== indirectly lost: 0 bytes in 0 blocks
==19416== possibly lost: 528 bytes in 1 blocks
==19416== still reachable: 416,718 bytes in 198 blocks
==19416== suppressed: 0 bytes in 0 blocks
...
==19416== ERROR SUMMARY: 489 errors from 30 contexts (suppressed: 0 from 0)
ceverett@icecream:~/Code/python$
Erlang R23
(On this one I can include the entire output.)
ceverett@icecream:~/Code/erlang$ cat hello_world
#! /usr/bin/env escript
% Example of an escript
-mode(compile).
main(_) ->
ok = io:setopts([{encoding, unicode}]),
io:format("Hello, world!~n").
ceverett@icecream:~/Code/erlang$ ./hello_world
Hello, world!
ceverett@icecream:~/Code/erlang$ valgrind ./hello_world
==25872== Memcheck, a memory error detector
==25872== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==25872== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==25872== Command: ./hello_world
==25872==
Hello, world!
ceverett@icecream:~/Code/erlang$
Hey, Python! What the actual fsck is going on here?!? Jeebus… The Good Idea Fairy came along and convinced the webtarded among you to start throwing kitchen sink features in (“walrus operator” — yeah, great idea, that, and totally worth losing the sober guidance of Guido over…) along with 8163 memory errors. Sweet! Oh how far the great have fallen…
Honorable mention: Ruby 2.5
I didn’t expect much better from Ruby as the Good Idea Fairy has been forwarding all xer mail there for years, but wow… just wow…
ceverett@icecream:~/Code/ruby$ cat hello_world.rb
#! /usr/bin/ruby
print "Hello, World!\n"
ceverett@icecream:~/Code/ruby$ ./hello_world.rb
Hello, World!
ceverett@icecream:~/Code/ruby$ valgrind ./hello_world.rb
...
==27051== HEAP SUMMARY:
==27051== in use at exit: 2,921,234 bytes in 20,690 blocks
==27051== total heap usage: 65,312 allocs, 44,622 frees, 16,447,358 bytes allocated
==27051==
==27051== LEAK SUMMARY:
==27051== definitely lost: 385,479 bytes in 4,421 blocks
==27051== indirectly lost: 640,953 bytes in 5,808 blocks
==27051== possibly lost: 1,626,429 bytes in 9,663 blocks
==27051== still reachable: 268,373 bytes in 798 blocks
==27051== suppressed: 0 bytes in 0 blocks
...
==27051== ERROR SUMMARY: 162050 errors from 402 contexts (suppressed: 0 from 0)
ceverett@icecream:~/Code/ruby$
162,050 memory errors. For Hello, World. How can there even be 162,050 instructions in the resulting set of commands to the system?
En 2013 comencé un cambio de dinámico a estático en la web comenzando con Octopress y llegando dos años más tarde hasta Lambdapad. No obstante, como digo en este issue, la situación se había vuelto insostenible, ¿qué hacer?
This is the last of three planned release candidates before the OTP 24 release.
The intention with this release is to get feedback from our users. All feedback is welcome, even if it is only to say that it works for you.
Erlang/OTP 24 is a new major release with new features, improvements as well as a few incompatibilities. Some of the new
features are highlighted below.
Highlights rc3
stdlib
Implementation of EEP 56 in supervisor. It adds the concept of significant children as well as the auto_shutdown supervisor flag. See the supervisor manual page for more information.
Highlights rc2
compiler
The compiler will now inline funs that are used only once immediately after their definition.
erts, kernel, stdlib
hex encoding and decoding functions added in the binary module
There is as usual a number of bug fixes and improvements detailed in the readme.
Highlights rc1
erts, kernel, stdlib
The BeamAsm JIT-compiler has been added to Erlang/OTP and will give a significant performance boost for many applications.
The JIT-compiler is enabled by default on most x86 64-bit platforms that have a C++ compiler that can compile C++17.
To verify that a JIT enabled emulator is running you can use erlang:system_info(emu_flavor).
A compatibility adaptor for gen_tcp to use the new socket API has been implemented (gen_tcp_socket).
Extended error information for failing BIF calls as proposed in EEP 54 has been implemented.
Process aliases as outlined by EEP 53 has been introduced.
compiler
Compiler warnings and errors now include column numbers in addition to line numbers.
Variables bound between the keywords 'try' and 'of' can now be used in the clauses following the 'of' keyword
(that is, in the success case when no exception was raised).
ftp
Add support for FTPES (explicit FTP over TLS).
ssl
Support for the "early data" feature for TLS 1.3 servers and clients.
Make TLS handshakes in Erlang distribution concurrent.
wx
The application has been completely rewritten in order
to use wxWidgets version 3 as its base.
Added support for wxWebView.
edoc
EDoc is now capable of emitting EEP-48 doc chunks. This means that, with some configuration, community projects
can now provide documentation for shell_docs the same way that OTP libraries did since OTP 23.0.
This post could easily be 20,000 words… there is just so much shit to wade through to get your book looking just so. What do I mean? Here are my concerns, which are simple ones I think. I want:
Fonts to be crisp and readable, the line-heights appealing to the eye and perhaps a few flourishes here and there like a drop-cap or small-capped chapter intro
Images to layout with proper spacing, a title for accessibility and to be center-aligned
Code samples to be formatted with syntax highlighting
Callouts for … callouts. Like a colored box with a title maybe.
Things like this are easy for word processing tools like Word or Pages but when it comes to digital publishing, not a chance.
I could fill up paragraphs but I won’t. I’ll just summarize by saying: formatting ebooks is a massive pain. Thankfully I’ve figured a few things out.
Have You Tried?
Yes, I’m 99% sure I have. I’ve used:
Scrivener (and I love it). Great for writing, crap for formatting.
Markdown turned into HTML and then formatted with Calibre. Works great - I did this with Take Off with Elixir and it looked amazing. Tweaking Calibre and hand-editing the CSS wasn’t so fun, though.
Michael Hartl’s Softcover which was stellar at formatting things and looked great but the styling choices were lacking. There were ways to edit things but I’m not a LaTex person. Installation was complex but doable… overall I enjoyed this one the most.
A zillion others including iAWriter, Bear, Ulysses, Pages/iBooksAuthor and many others I’m forgetting.
I’ve written 5 books over the last 6 or so years and I’m currently writing 2 more (which I’ll go into in a second). I swear to you I’ve tried just about everything.
When I wrote A Curious Moon I just went back to Word and decided to break the writing process up into two steps: actual writing and then formatting. I knew this is what a lot of writers did, but my process of writing (constantly evolving, “living” ebooks) didn’t lend itself to this. No matter - I can adapt.
It worked out pretty well, too. I wrote everything in Word and then hired an editor. Once editing was done I ported it to InDesign and spent weeks (literally) learning this massive tool.
It was worth it, I think, the book looks amazing…
The Problem is that it’s laborious and just killed my inspiration. Making edits, for instance, means I have to use InDesign’s ultra crap word editor to fix things, which isn’t fun.
Things get really exciting when InDesign bumps versions and everything is completely wrecked because they changed the way images are resolved (which happened)…
OK, enough complaining! Here’s where I’m at today, with a process I really like.
Writing in 2021
In my last post I mentioned that I was writing a book with Troy Hunt. It’s a fun project in which I’m:
Curating posts from his blog that I find interesting
Curating comments from those posts and adding them at the end (anonymously)
Pushing Troy to write retrospectives on these posts, giving some kind of backstory
Once again I decided to use Word and I wonder if that was the right decision. My thinking at the time was that Troy is a Windows person and I could use OneDrive to share the document with him so he could make comments.
There are problems with this, which include:
The document is huge. We’re up to 800+ pages with quite a lot of images. Syncing this thing real time is kind of a bugger.
The formatting looks horrendous and trying to explain to Troy “yeah, nah mate I’ll make it look good” is a bit hard. Especially when he replies “I already did that… have you seen my blog?” and I reply “yeah… ummm…”
Troy writes in Australian and uses words like “flavour”, “favourite” and “whilst”. Word’s spell checker doesn’t like that and YES I’ve reset it to Aussie English but it doesn’t seem to make a difference. Red squiggles are everywhere!
These are interesting problems to solve! For Troy, his blog is a done thing so my formatting woes are a bit ridiculous. I completely understand this, and I think I blew it by pulling things into Word first.
Writing From Scratch: Ulysses
This might come as a shock, but I find Ulysses to be, hands-down, the best writing experience I’ve ever had. In 2018 my choice was Scrivener:
When it comes to assembling your thoughts and structuring your manuscript, there is nothing that beats Scrivener. The functionality it gives you is astounding, and you can write little snippets or big hunks - its all up to you.
It successfully detaches the idea of text from presentation. You can kick up a manuscript and then compile it for various outputs such as epub, mobi, docx, and pdf. The compiler takes time to get used to, but once you do you can have a serviceable book.
This is still true, the keyword being “serviceable”. Writing in Scrivener is an engineer’s dream as it focuses to completely on the process of writing. The aesthetics of it, however, suck and you end up with something… serviceable:
By “serviceable” I mean the text will show on screen as well as the images, and if you’re lucky maybe some fonts will show up. I played with the compiler for days (literally), trying to get my epub layout flow the way I wanted. Line height, title leading, first paragraph non-indent… all of this is tricky or non-existent with Scrivener.
Ulysses, on the other hand, is pure markdown:
When you come from Word and Scrivener, this is amazing. I can’t tell you how many times I have to drop into “invisibles” to correct some janky weird formatting/layout issue in both Word and Scrivener. Things get so utterly ugly that I have to stop writing to fix it, which really makes the writing process suck.
With Ulysses, however, I just write in Markdown and I’m a happy person. It’s more than just a Markdown editor - it’s also a writer’s friend with some amazing support tools. The one I really like is the “Review” feature, which loads up your current page to Language Tool. It takes the feedback (which is free) and shows you where corrections are suggested. There are also ways to annotate your text and leave yourself comments, which I also love.
When I’m ready to preview how things will look, there’s a preview button right there that shows my markdown directly in epub. This is fine, if you’re OK with producing something that’s… “serviceable”. But that’s not what I want with Troy’s book.
Formatting for 2021
There are two apps that will format a book to near pixel-perfection for EPub and PDF that don’t come from Adobe and require a master’s degree:
Apple Pages will open up a Word document, read the styling, and immediately show you a 99% perfect recreation of your Word document. From that point you can start polishing things up and off you go. It really is a great bit of software, but, oddly, it sucks for writing. Not quite sure why.
The winner for me is Vellum. Check this out:
I was able to send my text directly from Ulysses into Vellum and this is the formatting I saw. It is perfect. A downside with Vellum is that it doesn’t support syntax highlighting which was one of my requirements :(. Another downside is that the export themes are slightly customizable, but that’s about it. I don’t mind it - it keeps me from going nuts.
I’m OK with doing screenshots for code and then making sure the reader has a link to a GitHub repo with code in it - that’s what I did for A Curious Moon and it worked out fine. It also looks better and, let’s be honest, no one is going to copy/paste code from an ebook - it’s horrible!
The features in Vellum are tremendous and it’s great for formatting a high-end ebook. It can’t do all that InDesign does because InDesign is all about document designing. But I kind of like that - Vellum is focused on wonderful book formatting, putting the focus on your words and content, no more.
I think it will work really well for Troy’s book - but you tell me.
Live Streaming the Formatting Process
I’ll be live streaming the formatting process with Troy on Monday, April 12th at 2PM PDT. We’ll have a few chapters of our book open in Word and I’m going to pull them into Vellum to see what he thinks. If he doesn’t like it, I’ll pull the book into Apple Pages and see what I can put together there.
If he doesn’t like that I’ll have to go back over to InDesign which isn’t the worst thing in the world, but it’s detailed enough that we’ll have to do another full stream.
We’ll also be discussing titles and cover ideas - so please join us! I would love to hear your thoughts. If you want to be updated on our progress I created a mailing list and we’ll be sending out updates when they happen… maybe twice a week.
A relative newcomer to networking in Erlang, Dr. Ajay Kumar, has started a self-educational project to create a web server from scratch in Erlang to give himself some first-hand insight into how TCP sockets work in Erlang and how web servers work in general. Web servers are a thing almost everyone has written against or for, but few have tried to implement on their own because socket programming sounds scary and time consuming to learn.
This video is quite short but incidentally demonstrates how not scary socket programming is and how easy it is to experiment with networking concepts on your own. Go experiment! Write networky things! It’s fun!
As an aside to this… I have a two-part explanation video that explains everything that is going on inside the service at he is basing his web server on. In the first part I explain what is going on within the default chat server that ZX templates as a network service project (kind of slow, covers basics for Erlang and ZX newcomers), and in the second part I explain how I used that as a basis for creating a telnet chat service that implements global shouts, channels, permissions, and other basic features needed for a real chat service (a bit better paced if you already know your way around Erlang and discusses some higher-level concepts such as service structure and the “service -> worker pattern“).
Welcome to our series of case studies about companies using Elixir in production. See all cases we have published so far.
PepsiCo is one of the world’s leading food and beverage companies serving more than 200 countries and territories around the world. Today Elixir is used at varying capacities inside PepsiCo by six different teams. This article explores how the Search Marketing and Sales Intelligence Platform teams adopted and use Elixir to build internal tools.
The project that would become the first Elixir project and open the door for future Elixir applications inside PepsiCo was started by Jason Fertel back in 2016.
Initially, the application provided workflow automation for managing search marketing operations on multiple web platforms. The product was a success and ended up integrated into PepsiCo in 2018.
Now, the Elixir application plays a central role in a data pipeline that empowers PepsiCo’s marketing and sales teams with tools to query, analyze, and integrate with several search marketing partners.
The pipeline starts with the Data Engineering team, which collects and stores data into Snowflake Data Cloud. The Elixir application reads data from Snowflake’s platform, pre-process, and stores it in two databases: PostgreSQL or Apache Druid, according to the data characteristics. Finally, a Phoenix application serves this data to internal teams and communicates directly with third-party APIs.
Why Elixir?
Elixir helps PepsiCo eCommerce focus and get things done fast. “Elixir allows our team to develop quickly with confidence,” says David Antaramian, a Software Engineering Manager at PepsiCo. “In turn, that lets us deliver value to the business quickly, and it’s the reason we’ve stuck with the language. Whether it’s streaming changes to the front-end or orchestrating concurrent data operations across multiple storage systems, Elixir offers a robust developer experience that translates to a great consumer experience.”
Different Elixir features came together to help the PepsiCo team build compelling development and user experiences. Thanks to its functional and extensible aspects, PepsiCo used Elixir to create a domain-specific language that translates business queries into data structures sent to different stores. This gave them a stable foundation where they can continually add new queries and integrations, even as they grow in complexity.
Furthermore, the reports generated by PepsiCo’s marketing and sales teams often have to query different tables or even separate storages, all while juggling long-running connections to different third-party APIs. Elixir’s programming model, inherited from the Erlang Virtual Machine, makes it trivial to run all of these operations concurrently, leading to fast and rich user interactions while the development team remains focused on delivering features.
Libraries and frameworks
David Antaramian is quick to praise the Erlang runtime and its standard library. He says: “Since we are working with large amounts of data, it is also essential to avoid hitting the database whenever possible. Thankfully, Erlang ships with an in-memory table storage called ETS, which we use to store hundreds of thousands of rows”.
The Erlang standard library was also handy when communicating to some data stores. In particular, the Snowflake platform requires ODBC connections. The PepsiCo team built a library called Snowflex, designed on top of Erlang’s built-in OBDC drivers.
The Elixir ecosystem nicely complements the Erlang one. The front-end, written in React, talks to the server via the Absinthe GraphQL toolkit running on top of the Phoenix web framework. The Ecto database library manages the communication to PostgreSQL. They also use the esaml and Samly libraries to provide authentication within PepsiCo’s organization - another example of leveraging the tools within both Erlang and Elixir communities.
Finally, the team also recognizes the Erlang Ecosystem Foundation’s efforts, which PepsiCo are sponsors of, particularly the Observability Working Group. David remarks: “The adoption of Telemetry by the ecosystem has been a massive help in bringing monitoring visibility and metrics to our system. Now when we see spikes in one place, we can easily correlate them with other system parts”.
Hiring
Today there are approximately 40+ Elixir engineers within PepsiCo distributed across six teams. Eight of those engineers are part of the Search Marketing and Sales Intelligence Platform teams.
While the team recognizes that there aren’t as many Elixir engineers compared to communities like JavaScript, they were effective in hiring qualified Elixir candidates. Chase Gilliam, a Software Engineering Manager at PepsiCo, explains: “We have met many engineers that, like us, found Elixir due to being burned out by previous experiences. So when it came to hiring, many Elixir candidates had a mindset similar to ours, which ultimately sped up the process.”
This initial group of Elixir engineers paved the way for the language’s growth inside PepsiCo. David adds: “At first, we looked for engineers with Elixir experience to help us build a team that could guide other developers. Then we extended the pool to anyone who has a functional programming background and now to developers with either Ruby or Erlang experience. However, if someone is the right candidate, we onboard them even if they have no Elixir experience and train them”. He continues: “We also make extensive use of the learning resources available in the community, such as conferences, books, online courses, and others.”
As the team grew, they adopted best practices and saw the quality of the codebase improve at the same time. Chase concludes: “At the beginning, there were some large modules in our application. Luckily, refactoring in a functional programming language is straightforward, thanks to immutability and limited side-effects. Adopting tools like Credo, ExDoc, and the code formatter was also essential to standardize how we use Elixir internally.” For those interested in learning more about the different use cases for Elixir inside PepsiCo and help continue its growth, they are hiring.
Libros, Youtube y Twitch. Son los proyectos potenciados donde más esfuerzo estoy poniendo durante mi tiempo libre publicando libros como la traducción al inglés del libro de Erlang/OTP o el nuevo libro sobre Phoenix Framework, pero, ¿qué más hay en Youtube y Twitch?
This is the second of three planned release candidates before the OTP 24 release.
The intention with this release is to get feedback from our users. All feedback is welcome, even if it is only to say that it works for you.
Erlang/OTP 24 is a new major release with new features, improvements as well as a few incompatibilities. Some of the new
features are highlighted below.
Highlights rc2
compiler
The compiler will now inline funs that are used only once immediately after their definition.
erts, kernel, stdlib
hex encoding and decoding functions added in the binary module
There is as usual a number of bug fixes and improvements detailed in the readme.
Highlights rc1
erts, kernel, stdlib
The BeamAsm JIT-compiler has been added to Erlang/OTP and will give a significant performance boost for many applications.
The JIT-compiler is enabled by default on most x86 64-bit platforms that have a C++ compiler that can compile C++17.
To verify that a JIT enabled emulator is running you can use erlang:system_info(emu_flavor).
A compatibility adaptor for gen_tcp to use the new socket API has been implemented (gen_tcp_socket).
Extended error information for failing BIF calls as proposed in EEP 54 has been implemented.
Process aliases as outlined by EEP 53 has been introduced.
compiler
Compiler warnings and errors now include column numbers in addition to line numbers.
Variables bound between the keywords 'try' and 'of' can now be used in the clauses following the 'of' keyword
(that is, in the success case when no exception was raised).
ftp
Add support for FTPES (explicit FTP over TLS).
ssl
Support for the "early data" feature for TLS 1.3 servers and clients.
Make TLS handshakes in Erlang distribution concurrent.
wx
The application has been completely rewritten in order
to use wxWidgets version 3 as its base.
Added support for wxWebView.
edoc
EDoc is now capable of emitting EEP-48 doc chunks. This means that, with some configuration, community projects
can now provide documentation for shell_docs the same way that OTP libraries did since OTP 23.0.
