You're Smart Enough for Elixir

A brief glimpse into Elixir from an object oriented perspective.

Concurrency is complicated so Elixir must be complicated.

Learning functional programming means giving up all my knowledge about object oriented programming.

Why Elixir? I haven’t needed it so far. It’s probably only for really complicated computer science-like stuff that I’m not doing.

If you’re looking at Elixir from the perspective of an object oriented scripting language like Ruby, Python, PHP, or even Perl then you’ve probably read, heard, or made statements like those above. Even if just to yourself.

I’m here to tell you that I’ve been there. I’ve professionally programmed in each of those languages over my career. Now I’m coming to Elixir from a recent background in Ruby. Let me assure you — you got this!

Elixir seemed like a scary, fascinating, almost unbelievable world. Serving web requests by creating a new server for each individual request? Madness! No data mutation? Isn’t that what data, you know, does? Processes that supervise other processes? How does that even work?

After a few weeks of using Elixir (and some Erlang) nearly full-time at work, I’ve come to realize my conception of Elixir was much more complicated than Elixir itself. In this post I’ll share what I’ve learned about three areas: the complexity of concurrency, what functional programming in Elixir is like, and why Elixir is interesting.

Concurrency

Concurrency is one of those development approaches that you usually only reach for when you really need to work some magic. The need to scale out is demanding enough that you’re willing to pay the complexity price and take on the risk of getting something wrong. In object oriented programming the danger of ending up with bad (i.e. unexpected) data is very real once you start running things concurrently.

In Ruby you have to start pulling in more complicated objects like Queue and start thinking very hard about all the places in your code that might be trying to update the same data at the same time. If more than one piece of code changes the same data at the same time then you end up with terrible things like race conditions or deadlocks which are no fun at all to debug.

There are some very ambitious projects in Ruby trying to improve the situation but they are hampered by the language itself.

Without a memory model it’s very hard to write concurrent abstractions for Ruby. To write a proper concurrent abstraction it often means to reimplement it more than once for different Ruby runtimes, which is very time-consuming and error-prone.

Concurrent Ruby’s published memory model

What I’ve learned from Elixir is that concurrency wasn’t the hard problem — Ruby’s concurrency was the hard problem. Ruby’s shared memory means you have to start sneaking around the language and being very careful in order to avoid problems.

Elixir works differently! It works so differently that it doesn’t even solve the concurrency problems found in Ruby — it avoids them completely. Instead of sharing memory, blocks of Elixir code run in isolated “processes” that only have their own data. Processes can pass data around, but the data is copied from one process to another and not shared directly. Beam (the underlying virtual machine running the code) makes copying data between processes very lightweight and memory efficient.

Not sharing data means that processes can be working on the same data at the same time without the possibility of unusual problems. At a fundamental level, Elixir is designed to allow parallel work without demanding that programmers figure out how to ensure the data stays correct.

I’ve found that good design using processes has a lot in common with good design using objects. In object oriented languages, objects should communicate by sending messages to other objects and not editing other objects directly. In Elixir the only way processes can communicate is by sending messages.

Functional Programming

This is the aspect of Elixir I thought was going to be the hardest thing to get. Immutable data? Crazy! But in practice it doesn’t really come up. After a couple weeks with Elixir I was surprised to think back and realize I’d never once thought, “This would be easier in Ruby!”

You can think of immutable data as always moving forward. You can’t change the data itself, but you can transform it into a new version of the data. That sounds like it would be an expensive and slow process but, yet again, the virtual machine running Elixir steps in and makes all of the low level mechanics work easily and quickly.

As an example, let’s have a server process that counts the number of times other processes have sent it the “hello” message. It will internally have a count of hello messages like you would expect. When a new “hello” arrives the data value of count is not changed, the count variable itself is reassigned to a new data value that’s count + 1 .

Let’s see a bit more of how this works with a simple comparison between Ruby and Elixir.

Ruby - the array variable is internally mutated

array = [1,2,3]
# => [1, 2, 3]
array.delete_at(0)
# => 1
array
# => [2,3]

Elixir - the array variable is passed through a function that returns new data

array = [1,2,3]
# [1, 2, 3]
List.delete_at(array, 0)
# [2,3]
array
# [1, 2, 3]
array = List.delete_at(array, 0)
# [2,3]

In Ruby we see that the array variable is mutated when we call delete_at. See how the delete_at method is actually part of the data itself. That’s a very common idea in object oriented languages and one of the key differences between object oriented and functional programming.

In Elixir note that the “delete_at” functionality is not a method on the data itself. The data is instead passed through a higher level function that returns the new version of the data. We could decide to assign the new version of the array to the array variable, but even then the data [1,2,3] is not mutated. Think of array as a label that we’ve told Elixir to remove from the data [1,2,3] and apply to the data [2,3].

As a side note: Elixir doesn’t actually have arrays like you’d expect from Ruby. It has “lists” of data that are actually quite different. But let’s stay on target.

Why Elixir?

You don’t need Elixir. Any modern language can fit any number of problems. It’s not that some things aren’t possible without Elixir, but that Elixir makes some traditionally hard things easy.

Consider an example: Elixir provides the concept of Supervisors. These are processes monitoring other processes, restarting them with their initial state if they crash. Supervisors are configured with rules for how many crashes are allowed per time period. If the watched processes pass that threshold, the supervisor itself crashes and the failure handling moves up a level.

Just think of all the concepts that the configuration of a supervisor entails. You give a supervisor rules like “this watched process is allowed 2 restarts in 30 seconds”. You don’t need to write any time checking code or counting logic! The virtual machine running Elixir handles all the details of actual time tracking. Time aside, even the concept of watching another process and getting notification if it crashes would be difficult in many languages. In Elixir it’s as simple as calling Process.monitor.

Layered supervision is one of the key components that allows Elixir applications to reach extremely high availability targets. A well designed system can tolerate failures of its components without itself crashing.

Because process supervision handles the error states it means you can (and should!) write functions that are only concerned with the happy path. You write code that is focused on what should happen and for anything else you let it crash.

Well-behaved Elixir applications don’t assume failure will never happen. Rather, making decisions about failure is built into the language. This goes far beyond exception handling which only allows catching one failure across one block of code. Elixir supervision means a system can fallback to a wide range of working states to keep core functionality available even as networking errors, buggy code, and other real-world problems creep in.

All Elixir processes run on a virtual machine called BEAM (popularly called the Erlang VM). That layer allows the language processes to be optimized in some helpful ways. Because processes involve no shared memory, the Erlang VM can run them on many CPUs or cores in parallel. The Erlang VM also transparently handles networking, so Elixir processes pass messages between each other using the same syntax whether they’re running on the same machine or on multiple machines. Those two features means Elixir can handle both vertical and horizontal scaling. Even better, the Erlang VM can handle much of the scaling process without demanding applications be rewritten specifically to support scaling.

Elixir: a lot of fun!

By far the aspect of Elixir I love the most is that it’s so much fun to write. José Valim, Elixir’s creator, has done a fantastic job of combining a lot of great ideas from a lot of languages. Beautiful declaration syntax from Ruby, list comprehensions and doctests from Python, and of course all the good ideas from Erlang, like pattern matching.

Process supervision means processes can focus on the happy path and let failures be handled at a higher level. Writing less error handling and checking logic is something I’m sure we can all enjoy. It’s also rewarding to see all the CPUs in a system working together to efficiently perform a dozen calculations on the same set of data.

Would you like to write clear and readable code that doesn’t need to jump through convoluted hoops to run in parallel? If so give Elixir a try! I hope you enjoy it.

Resources for learning Elixir

Permalink

Mocks and Explicit Contracts: In Practice w/ Elixir

Writing tests for your code is easy. Writing good tests is much harder. Now throw in requests to external APIs that can return (or not return at all!) a myriad of different responses and we’ve just added a whole new layer of possible cases to our tests. When it comes to the web, it’s easy to overlook the complexity when working with an external API. It’s become so second nature that writing a line of code to initiate an HTTP request can become as casual as any other line of code within your application. However that’s not always the case.

We recently released the first version of our self-service debugging tool. You can see it live at https://debug.spreedly.com. The goal we had in mind for the support application was to more clearly display customer transaction data for debugging failed transactions. We decided to build a separate web application to layer on top of the Spreedly API which could deal with the authentication mechanics as well as transaction querying to keep separate concerns between our core transactional API and querying and displaying data. I should also mention that the support application is our first public facing Elixir application in production!

Since this was a separate service, it meant that we needed to make HTTP requests from the support application to our API in order to pull and display data. Although we had solid unit tests with example responses we’d expect from our API, we wanted to also incorporate remote tests which actually hit the production API so we could occasionally do a real world full stack test. Remote tests aren’t meant to be executed every test run, only when we want that extra ounce of confidence.

As we started looking into testing Elixir applications against external dependencies, we came across José Valim’s excellent blog post, Mocks and Explicit Contracts. If you haven’t read it already, you should check it out. It has a lot of great thoughts and will give this post a little more context. It seemed like a solid approach for building less brittle tests so we thought we implement it ourselves and see how well it would work in reality and if it could provide what we needed to include remote tests along side our unit tests. Here’s how our experience with this approach went…

Pluggable clients

The first thing we needed to do was update the Spreedly API client in the support application to be dynamically selected instead of hardcoded. In production we want to build real HTTP requests, but for unit tests we want to replace that module with a mock module which just returns simulated responses.

# Module wrapping our API requests for transactions
defmodule SupportApp.Transaction do
  @core Application.get_env(:support_app, :core)

  def fetch(token, user, secret) do
    @core.transaction(user, token, access_secret)
  end
end

In line 3 above, you can see that the @core constant is being dynamically set by doing a Application configuration lookup for the value. In our particular case, the lookup will return the module to use for getting Spreedly transaction data.

Once we’ve got that set, now we can configure the module to use in our application config. Notice that the module returned on lookup changes depending on the environment we’re currently running. We really like the explicitness here!

# In config/test.exs
config :support_app, :core, SupportApp.Core.Mock

# In config/config.exs
config :support_app, :core, SupportApp.Core.Api

Enforceable interfaces

So, what do those two modules look like anyway? Well, from our Transaction module above we know that both the HTTP client and the mock will need to have a transaction/3 function which will take care of getting us a transaction whether it be from the Spreedly API or a simulated one we build ourselves.

So in production, we’re using wrapping HTTPotion to make requests.

defmodule SupportApp.Core.Api do
  @behaviour SupportApp.Core
  ...
  def transaction(key, token, secret) do
    path = "..."
    options = ...

    %HTTPotion.Response{body: body} = HTTPotion.get(path, options)
    case Poison.Parser.parse!(body) do
      ...
    end
  end
  ...
end

However, in unit tests we’re going to use a mock module instead which just returns a straight Map of what we’d expect from a production request.

defmodule SupportApp.Core.Mock do
  @behaviour SupportApp.Core

  def transaction(_key, "nonexistent", _), do: nil
  def transaction(_key, token, _) do
    %{
      "transaction": %{
        "succeeded": true,
        "state": "succeeded",
        "token": "7icIbTtxupZpY8SKxwlUAKq8Qiw",
        "message_key": "messages.transaction_succeeded",
        "message": "Succeeded!",
        ...
      }
  end
end

Two things to notice about the modules above:

  1. For the mock module, we pattern matched on the function parameters to alter the response we want. This is incredibly useful when you want to test against different response scenarios since you can define another transaction/3 function with a specific parameter value and then have it return the response with appropriate test data. In our case, we wanted to also test when someone doesn’t enter a transaction token to search (see line 4 in the mock module above).

  2. In both the production and test modules, there’s a line beneath the module definition- @behaviour SupportApp.Core . By having a top level behaviour for these modules, we can be sure that our production API client module and our mock module adhere to the same interface. If we want to add another endpoint, start by adding it to the top level behaviour SupportApp.Core which will then trickle down to all modules and keep us explicitly defining our API contract so our production API client and mock client remain in step.

Here’s a snippet of our behaviour module that ensures all compliant behaviours have a transaction function with the current arity and argument types:

defmodule SupportApp.Core do
  use Behaviour

  @callback transaction(key :: String.t, token :: String.t, secret :: String.t) :: %{}
  ...
end

And that’s it for setup!

Test segmentation

Up to this point we’ve followed the approach as outlined by José’s blog post and created an explicit contract between our modules, allowing us to change underlying implementations depending on the environment we’re running in. That is, a mock module to be used during test and an Spreedly API client in production. However, our original plan was to include remote tests that actually hit our production API so how can we enable that in our tests?

Simple! In the same way we dynamically looked up the module we wanted to use, we can use the test setup block to pass the module we want to use in place of our external dependency client. So for the unit tests we have:

defmodule SupportApp.Core.MockTest do
  use ExUnit.Case, async: true

  setup do
      {:ok, core: Application.get_env(:support_app, :core)} # <-- See config/test.exs
  end

  test "transaction", %{core: core} do
    %{"transaction" => transaction} = core.transaction(...)
    assert txn["token"]
  end
end

But for our remote tests, we change our setup block to drop in a the real HTTP client wrapper. I should also note that we needed to handle creating production API credentials and making them available to our remote tests, but handling that is a bit out of scope for this post.

defmodule SupportApp.Core.ApiTest do
  use ExUnit.Case, async: true
  @moduletag :remote

  setup do
    {:ok, core: SupportApp.Core.Api} # <-- Force the module to be used
  end

  test "transaction", %{core: core} do
   %{"transaction" => txn} = core.transaction(...)
    assert txn["token"]
  end
end

Almost there! Since we don’t want to run our remote tests each time (they’re only around for that extra bit of confidence!) we can just use a @moduletag to exclude them by default and only run them if we explicitly say to do so. We added a line to our test_helper.exs :

ExUnit.configure exclude: [:remote]

Now to run the remote tests, just add an include flag:

$ mix test --include remote

We’re really happy with the setup so far and think it provides us both the confidence of real world full stack tests, with the flexibility of simulating responses within a mock.

Permalink

How do I GenStage?

I’m new to Elixir. And Erlang. And OTP’s GenServer. And GenStage. While I’ve got beginner’s-eye, I’m going to share knowledge for a general audience. So this is a doorknob-simple look at GenStage. With clear examples and I Love Lucy references. Enjoy!

What is a GenStage?

Lucy studies papers Fun and scalable? Interesting…

Erlang is a language invented to solve problems in scalable, fault-tolerant, and distributed ways. Elixir is a language built on top of Erlang that makes solving problems more fun. GenServer is an easy-to-use system for making generic server processes in Erlang (and now in Elixir). GenStage is an easy-to-use system built on top of GenServer to set up processing chains and solve the Lucy Chocolates Problem (see below).

Why would I GenStage?

You’re Elixiring because you have problems that can be split up and run on separate processors, VMs, machines, clusters… planets? (Yes, Elon. Maybe separate planets.)

Q: Let’s say your problem can be split into 2 parts. Are those two parts going to run at the same speed?
A: No, probably not. Part 1 is going to run at part-1-speed, and part 2 is going to run at part-2-speed.

The Lucy Chocolates Problem

For instance, your problem is shipping chocolates. Part 1 is making the treats, and part 2 is wrapping them. As long as your chocolate-wrapper is faster than your chocolate-maker, you’re good. As soon as your chocolate maker is faster than your chocolate wrapper, you have an I Love Lucy situation on your hands.

Lucy begins wrapping chocolates Lucy is overwhelmed by the speed of chocolates production Lucy desperately stuffs chocolates in her mouth and hat (Shown here is Lucille Ball, legendary entertainer, entrepreneur, and person-who-saved-Star-Trek.)

Lucy’s Chocolate Twitter-Feed

Let’s look at a real-life Lucy chocolates problem. Maybe you want to:

  1. pull words from a Twitter feed,
  2. Google image search them, and
  3. auto-create a collage of images based on those tweets.

From Twitter to Google to image generator

Pulling words from a Twitter feed is blazing fast. Google image searching is relatively slow. Photo-editing is glacial.

If you’re running these things in different processes chained together, it won’t be hard for the Twitter-reader process to overwhelm the image-searcher process which in-turn would lock up the collage-maker. Even if you have a bunch of collage-makers, you’d have to round-robin or find some other algorithm to keep them balanced. If you don’t do it right, you might miss some of the sweet sweet Tweets you desperately need to autogenerate your art. This is the same problem Lucy had with the chocolates.

Wouldn’t it be nice to have a way for your collage-maker processes to pull whatever they could handle from your image-searcher processes which could pull what they were able to handle from your tweet-reader?

There is such a way! That’s why GenStage. Because the Lucy Chocolates Problem.

How do I GenStage?

For right now, I’m going to go over the same stuff that was in the GenStage announcement. If you have that stuff down cold, you can skip to the Twitter feed example, scan through, and nod your head sagely.

There are 3 types of GenStage servers: producer , producer_consumer , and consumer . To implement them, you define a module that uses GenStage and create a few callback functions in that module.

Below is the dead-simplest way to set up a three-link chain. It doesn’t even do anything. Producer passes stuff straight through ProducerConsumer to Consumer, which does nothing. It compiles and runs. That’s its claim to fame.

First Glance

    alias Experimental.GenStage

    defmodule Producer do
      use GenStage
      def init(arg) do
        {:producer, :some_kind_of_state}
      end

      def handle_demand(demand, state) do
        {:noreply, things = [:whatever, :you, :want], state}
      end
    end

    defmodule ProducerConsumer do
      use GenStage
      def init(arg) do
        {:producer_consumer, :some_kind_of_state}
      end

      def handle_events(things, from, state) do
        {:noreply, things, state}
      end
    end

    defmodule Consumer do
      use GenStage  
      def init(arg) do
        {:consumer, :some_kind_of_state}
      end

      def handle_events(things, from, state) do
        {:noreply, [], state}
      end
    end

    defmodule Go do
      def go do
        {:ok, producer} = GenStage.start_link(Producer, arg = :nonsense)
        {:ok, prod_con} = GenStage.start_link(ProducerConsumer, arg = :nonsense)
        {:ok, consumer} = GenStage.start_link(Consumer, arg = :nonsense)

        GenStage.sync_subscribe(prod_con, to: producer)
        GenStage.sync_subscribe(consumer, to: prod_con)
      end
    end

In Detail

Great. Let’s see the same thing, but with gobs of annotations!

## Producer
    alias Experimental.GenStage

Development on GenStage is still going gangbusters. It will stay in the Experimental namespace for a while.

    defmodule Producer do

I could have called this thing whatever I wanted. Maybe I should have called it JerryBruckheimer .

      use GenStage

With this one line, we pull in all of GenServer and GenStage and make this module into an unstoppable data processing machine.

      def init(arg) do  

Init is a callback that’s triggered when you start your process. It takes an arg so you can set up your producer with whatever starting info you want. I don’t use the arg here, so I’m going to get a compiler warning.

        {:producer, :some_kind_of_state}
      end

Init needs to return a tuple with an atom and a something-else. The atom has to be :producer , :producer_consumer , or :consumer . That will set some assumptions for GenStage about how this module will behave. The something-else is the state for this server. Like all GenServers, a GenStage module can hold arbitrary data in memory.

      def handle_demand(demand, state) do

This callback is what makes a producer a producer. (Well that and the :producer atom returned by init ). handle_demand is called with the number of “things” a consumer is asking for and the state of the producer at the time the consumer asked for them.

        {:noreply, things = [:whatever, :you, :want], state}
      end
    end

The return value is intended to look like the return value from an asynchronous GenServer cast (or the return value from a call you’re not ready to reply to yet). :noreply indicates that handle_demand is not going to send any information back to whatever called it. This is confusing to me because handle_demand IS sending back information; the very next item in the tuple. But perhaps since it’s not sending information back in the way GenServer usually means, so we’ll let that go. things is the list of what-have-you that you’ve produced.

Things vs. Events - Most examples will use the word events for this. I don’t here because to me an “event” is a particular kind of thing. You could argue that a “thing” passed to a Consumer is an “event” as far as that Consumer is concerned. And I would see your point. …but I’d have to squint. Anyway, call it events everywhere else, but just for now, I’ll call it things.

Note that the things returned here is a list. It has to be a list. It can’t be a not-list. If you try to return something that is not a list, you will get ** (stop) bad return value: {:noreply, :decidedly_not_a_list, :ok} I want to pause here and note that bad return value is not the most helpful error message. Though I will also say that it’s more helpful than bad cast .

Search for bad cast leads to Breaking Bad

## ProducerConsumer

    defmodule ProducerConsumer do
      use GenStage
      def init(arg) do
        {:producer_consumer, :some_kind_of_state}
      end

We’re starting off pretty much the same way here. I could call the module whatever I want. Note it returns :producer_consumer and starting state. Again, :producer_consumer sets GenStage’s assumptions about how this thing will work. And state could be the complete works of Shakespeare; though I wouldn’t recommend it.

      def handle_events(things, from, state) do

This is the consumer’s main callback function. It takes a list of things (the same list sent by producer’s handle_demand above). It also takes from which is how we identify where these events came from, in case we need to. We usually don’t. You won’t often be using from ; you’ll more often call it _from (because underscores denote discarded values which helps avoid compiler warnings). Finally it takes whatever is the current state of the ProducerConsumer.

More about from - In case you’re interested, from (right now) is a tuple with the process identifier of the process the call came from and a reference to the request itself. I say “right now” because the GenServer documentation says that the format of from arguments may change in the future. So it’s best not to {pid, ref} = from if you can avoid it. The request reference seem to be generated by Elixir’s Kernel.make_ref . These references are theoretically unique (like GUUIDs, but not really). In GenServer, you can use from to reply directly to a request. GenServer.reply(from, :my_message) . GenServer makes use of simpler Erlang message passing calls behind the scenes, and tracks all this request reference stuff for you. It’s really cool. Thanks, Erlang.

        {:noreply, things, state}
      end
    end

ProducerConsumer is supposed to be changing that list of things in some way before passing it along to Consumer. But this one isn’t. Because it’s lazy. Here we return a tuple with :noreply ; again I assume because we’re not replying in the traditional GenServer sense. The tuple also has a list (and it must be a list) of things to go to the consumer. Finally, we have the ever-present state .

## Consumer
    defmodule Consumer do
      use GenStage  
      def init(arg) do
        {:consumer, :some_kind_of_state}
      end

By now, we’re pretty familiar with what’s going on here. Or we’ve fallen asleep and scrolled this far with our faces on the trackpad; I won’t judge.

      def handle_events(things, from, state) do

Consumer is handling events (with handle_events ) just the same way ProducerConsumer does, except…

        {:noreply, [], state}
      end
    end

Consumer MUST return an empty list in its tuple. If you return anything else, you’ll get [error] GenStage consumer #PID<0.514.0> cannot dispatch events (an empty list must be returned) This is quite a rebuke! It’s also much more helpful than “bad cast” and “bad return value” both.

    defmodule Go do
      def go do

Now what’s this Go.go() business? I like using a go function in my examples so they don’t automatically run when I use iex -S mix .

        {:ok, producer} = GenStage.start_link(Producer, arg = :nonsense)

Start up a Producer process. Pattern matching with :ok ensures everything went well or makes everything crash. Making something crash when things get a little sketchy is good Elixir practice. Elixir is basically a Windows user in the 1990s, hitting restart every time something goes sideways.

I was skeptical about this practice when I first heard about it. But you know what? I rebooted my way through Microsoft Windows 3.1, 95, 98, ME, and XP. And in-BETWEEN those reboots, I got a lot done. What I’m learning now is that all distributed systems are by-nature as buggy as Windows 3.1 and aspiring to be only as buggy as Windows 95.

        {:ok, prod_con} = GenStage.start_link(ProducerConsumer, arg = :nonsense)
        {:ok, consumer} = GenStage.start_link(Consumer, arg = :nonsense)

Anyway, we grab :ok and the process identifiers for our three GenStage processes. Note I’m passing arguments, because start_link expects me to. Those arguments are passed to the init functions described way above. But we ignore them. Also note that every time I do something like arg = :nonsense , I get a warning about how arg isn’t used. I love ignoring variable names with underscore _ , but when I’m writing examples I want them both descriptive and pretty.

        GenStage.sync_subscribe(prod_con, to: producer)

Aha! Now our ProducerConsumer is ready to grab whatever it can from Producer. It doesn’t grab anything because it’s not the final Consumer and so isn’t running the show.

        GenStage.sync_subscribe(consumer, to: prod_con)
      end
    end

There we go! That’s the final Consumer getting involved. As soon as this happens, Producer gets a call to handle_demand . Based on experience, the demand argument will probably be 500 that first time it gets called, but don’t count on that. Then ProducerConsumer gets a call to handle_events with [:whatever, :you, :want] as its things argument. Then Consumer gets an identical call to handle_events with [:whatever, :you, :want] as its things argument. Then it does nothing! Because again, lazy.

Twitter Feed Example

Fine! Let’s go back to our earlier example. I want to pull tweets from some feed, Google image search the words, and then use an image manipulator to make collages out of those images. I’ve put further description in comments below.

From Twitter to Google to image generator

    # Let's pretend this library is real
    alias AdriansHandWavingLibrary.{Tweet,ImageSearch,Collage}
    alias Experimental.GenStage

    defmodule ReadFromTwitter do
      use GenStage
      def init(twitter_feed) do
        tweets = Tweet.all_to_date(twitter_feed)
        # Note we're setting tweets as the state.
        {:producer, tweets}
      end

      def handle_demand(demand, state) do
        # Pull some tweets out of state. We send those as the events
        # or "things", and we reset state to the remaining tweets.
        # @jacobterpri pointed out the existence of Enum.split/2. Thanks!
        {pulled, remaining} = Enum.split(state, demand)
        {:noreply, pulled, remaining}
      end
    end

    defmodule ConvertToImages do
      use GenStage
      # This step still needs no state.
      def init(_) do
        {:producer_consumer, :ok}
      end

      # Turn a list of tweets into a list of lists of images.
      def handle_events(tweets, _from, _state) do
        image_lists = Enum.map(tweets, &to_list_of_images(&1))
        {:noreply, image_lists, :ok}
      end

      # Do that by splitting the tweets into individual words and running
      # image_for on each word
      defp to_list_of_images(tweet),
        do: tweet
          |> String.split(" ")
          |> Enum.map(fn word -> ImageSearch.image_for(word) end)
    end

    defmodule CollageBackToTwitter do
      use GenStage  
      # Set state to the one thing this needs to keep track of: where to post
      # collages.
      def init(output_twitter_feed) do
        {:consumer, output_twitter_feed}
      end

      # Get the lists of images, collage them together, and send them back out
      # to Twitter. This is definitely the longest step. There's image manipulation.
      # There's uploading. Then there's tweeting. All of that happens in my pretend
      # modules, but go ahead and pretend. That's a lot of time, isn't it?
      # So if we weren't using GenStage, the CollageBackToTwitter module would
      # require a ton of buffering code. The equivalent of Lucy stuffing chocolates
      # in her hat!
      # Thanks, GenStage.
      def handle_events(image_lists, _from, output_twitter_feed) do
        image_lists
        |> Enum.map(&Collage.images_together(&1))
        |> Enum.each(&Tweet.send_image(&1, output_twitter_feed))
        {:noreply, [], output_twitter_feed}
      end
    end

    defmodule Go do
      def go do
        # Note we're sending the Twitter names to pull from and push to.
        {:ok, producer} = GenStage.start_link(ReadFromTwitter, "@madcapulet")
        {:ok, prod_con} = GenStage.start_link(ConvertToImages, arg = :nonsense)
        {:ok, consumer} = GenStage.start_link(CollageBackToTwitter, "@bitcapulet")

        # I'm pretending here that I've tuned this stuff and found the following
        # levels of demand to be optimal. Because, hey look! There are a bunch of
        # settings on the sync_subscribe function. Rad.
        GenStage.sync_subscribe(prod_con, to: producer, max_demand: 50)
        GenStage.sync_subscribe(consumer, to: prod_con, max_demand: 10)
      end
    end

Obligatory Wrap-up Paragraph

If you have a multi-step, multi-process problem, and later steps may be slower than earlier steps, then consider GenStage. It’s simple, fun, and solves this problem thoroughly.

If you want to learn more about this you can read:

Thanks so much to José Valim and his cadre of brilliant and charming collaborators for Elixir and GenStage. Thanks also to the folks who’ve posted previous examples, to Lucille Ball (that human was genius), and to Spreedly for being such a great place to work.

Permalink

From Elixir Mix configuration to release configuration - Alchemy 101 Part 2

Today we will be looking at what happens to your Mix configuration when you perform a release. Take a look at the Set up section and then proceed to Application Configuration.

Set up

You can follow along with the examples. You will require elixir and to perform the following steps.

First create a new mix project.

mix new my_app --module Twitter
cd my_app

 

Next add distillery (for creating releases) to mix.exs as a dependency.

defp deps do
  [{:distillery, "~> 0.10.1"}]
end

 

Then download distillery and create the release configuration.

mix deps.get
mix release.init

 

The rest of the blog assumes that you are in the my_app directory.

Application Configuration

When creating an Elixir OTP Application you will most probably need some configuration. There are 4 ways to supply application configuration on startup.

       1. In your mix.exs file. Here you can specify default application environment variables. This file is used to generate the .app file which is used to start your application. Run 'mix help compile.app' for more information

       2. In your config/config.exs file, this compiles down to sys.config. Alternatively with distillery you can supply your own sys.config file.

       3. With an additional .config file. From what I can see distillery and exrm don't seem to support this out the box. You can find out more here on how to use it.

       4. You can supply it to the Erlang VM when it starts up. Distillery supports this via erl_opts in rel/config.exs. Simply add "-Application Key Value" to it for each application configuration variable e.g. set erl_opts: "-my_app magic_number 42".

From what I have seen most people tend to go with the option 2, supplying configuration via config/config.exs. As configuration is an Elixir script it gives us the potential to be very creative. When creating a release (with exrm or distillery) config.exs (by default) is evaluated and the result is written to rel/$app/releases/$version/sys.config which is picked up by your application on startup. Not knowing this can lead to confusion. Here comes another example where this can happen.

Open lib/twitter_client.ex and add the following to it.

defmodule Twitter do
  require Logger

  def log_twitter_url do
    url = Application.get_env(:my_app, :twitter_url)
    Logger.info("Using #{url}")
  end
end

 

Now add the following to config/config.exs.

config :my_app, twitter_url: System.get_env("TWITTER_URL")

Pretty nice eh? It appears like we can get TWITTER_URL at runtime. Lets create the release and inspect it. Run the following.

export TWITTER_URL="https://api.mock.com"
MIX_ENV=prod mix release
./rel/my_app/bin/my_app console
iex(my_app@127.0.0.1)1> Twitter.log_twitter_url()
17:26:05.270 [info]  Using https://api.mock.twitter.com

 

Perfect! Everything looks good, the url to the mock is being used. Just what I want during development. Now I want to test the integration with the real twitter API, time to change TWITTER_URL.

export TWITTER_URL="https://api.twitter.com/1.1"

 

Start your release in the console and invoke Twitter.log_twitter_url/0.

./rel/my_app/bin/my_app console
iex(my_app@127.0.0.1)1> Twitter.log_twitter_url()
17:26:05.270 [info]  Using https://api.mock.com

 

Strange! It is still using the mock url, but why? As mentioned before when creating a release the configuration is evaluated and written to sys.config. Lets take a look.

cat rel/my_app/releases/0.1.0/sys.config 

[{sasl,[{errlog_type,error}]},
 {my_app,[{twitter_url,<<"https://api.mock.com">>}]}].

 

As you can see twitter_url is "https://api.mock.com". When the release is being created config.exs is evaluated and the results are placed in sys.config. Part of this involved executing System.get_env("TWITTER_URL") and having "https://api.mock.com" returned.

This doesn't have to be a problem though as you can set Application configuration at runtime via Application.put_env/3. Using this you could create a function that reads the OS environmental variable and adds it to the application's configuration.

def os_env_config_to_app_env_config do
  twitter_url = System.get_env("TWITTER_URL")
  Application.put_env(:my_app, :twitter_url, twitter_url)
  :ok
end

 

Note that this function would have to be called before any initialisation in your application takes place. I'm sure there are other ways to handle this scenario, if so feel free to mention them in the comments section.

Hope you enjoyed reading, tune in next time where I'll be talking about what it means to be fault tolerant.

Permalink

Build a complete iOS messaging app using XMPPFramework - Tutorial Part 1

YAXT??! Yet another XMPP tutorial?

 

Well, this is going to be another tutorial, but I'm going to try to make it a little bit different. This is an XMPP tutorial from an iOS developer's perspective. I'll try to answer all the questions I had when I started working in this area. This journey is going to go from no XMPP knowldege at all to having a fully functional instant messaging iOS app using this cool protocol. We are going to be using the super awesome (yet overwhelming at the beginning...) XMPPFramework library, and the idea is also to also mix in some iOS concepts that you are going to need for your app.

What's XMPP?

 

From Wikipedia: Extensible Messaging and Presence Protocol (XMPP) is a communications protocol for message-oriented middleware based on XML.

This basically means XMPP is a protocol for exchanging stuff. What kind of stuff? Messages and presences. We all know what messages are, but what about presences? A presence is just a way of sharing a "status", that's it. You can be 'online', 'offline', 'having lunch', or whatever you want. Also there's another important word: Extensible meaning it can grow. It started as an instant messaging protocol and it has grown into multiple fields for example IoT (Internet of Things). And last, but not least: every piece of information we are going to exchange under this protocol is going to be XML. I can heard you complaining but... Come on, it's not that bad!

Why do we need XMPP? Why not just REST?

 

Well what other options do we have? On the one hand, a custom solution means building everything from scratch, that takes time. On the other hand, we have XMPP, a super tested technology broadly used by millions of people every day, so we can say that's an advantage over a custom approach.

Everytime I talk about XMPP, someone asks me 'Why not just REST?'. Well, there is a misconception here. REST is not a protocol, it's just a way of architecting a networked application; it's just a standarized way of doing something (that I love btw). So let's change the question to something that makes more sense: "Why not just build a custom REST chat application?". The first thing that comes to my mind is what I already explained in the previous paragraph, but there is something else. How do I know when someone has sent me a message? For XMPP this is trivial: we have an open connection all the time so, as soon as a message arrives to the server, it will send us the message. We have a full-duplex. On the other hand, the only solution with REST is polling. We will need to ask the server for new messages from time to time to see if there is something new for us. That sucks. So, we will have to add a mechanism that allows us to receive the messages as soon as they are created, like SSE or WebSockets.

There is one more XMPP advantage over a custom REST chat application. REST uses HTTP, an application level protocol that is built on top of a transport level protocol: TCP. So everytime you want to use your REST solution, you will need HTTP, a protocol that is not always available everywhere (maybe you need to embed this in a cheap piece of hardware?). Besides, we have XMPP built on top of TCP that's going to be always available.

What's the basic stuff I need to know to get started?

 

Well, you know a lot already but let's make a list. Lists are always good:

  • XMPP is built on top of TCP. It keeps an open connection all the time.
  • Client/Server architecture. Messages always go through a server.
  • Everything we send and receive is going to be XML and it's called Stanza.
  • We have three different types of stanzas: iq, message and presence.
  • Every individual on the XMPP network is univocally identified by a JID (Jabber ID).
  • All the stanzas are cointained in a Stream. Let's imagine the Stream as a white canvas where you and the server write the stanzas.
  • Stream, iq, message and presence are the core of XMPP. You can find everything perfectly detailed in RFC6120
  • XMPP can be extended to accomplish different stuff. Each extension is called XEP (XMPP Extension Protocol).

 

What's a JID?

Jabber ID (JID) is how we univocally identify each individual in XMPP. It is the address to where we are going to send our stanzas.

This is how a JID looks like:

  • localpart: This is your username.
  • domainpart: Server name where the localpart resides.
  • resourcepart: This is optional, and it identifies a particular client for the user. For example: I can be logged in with andres@erlang-solutions.com on my iPhone, on my Android and on my mac at the same time... So all these will be the same localpart + domainpart but different resourcepart

I'm sure you have already noticed how similar the JID looks to a standard email address. This is because you can connect multiple servers together and the messages are rooted to the right user in the right server, just as email works. Pretty cool, right?

Sometimes you will see we have a JID with just the domain part. Why?! Because it's also possible to send stanzas to a service instead of a user. A service? What's a service?! Services are different pieces of an XMPP server that offer you some special functionality, but don't worry about this right now, just remember: you can have JIDs without a localpart.

What's a Stanza?

Stanza is the name of the XML pieces that we are going to be sending and receiving. The defined stanzas are: <message/><presence/> and <iq/>.

 

<message/>

This is a basic <message/> stanza. Everytime you want to send a message to someone (a JID), you will have to send this stanza:

<message from='andres@erlang-solutions.com/iphone' to='juana@erlang-solutions.com' type='chat'>
    <body>Hey there!</body>
</message>

 

<iq/>

It stands for Info/Query. It's a query-action mechanism, you send an iq and you will get a response to that query. You can pair the iq-query with the iq-response using the stanza id.

For example, we send an iq to the server to do something (don't pay attention to what we want to do... you just need to know there is an iq stanza and how the mechanism works):

<iq to='erlang-solutions.com' type='get' id='1'>
  <query xmlns='http://jabber.org/protocol/disco#items'/>
</iq>

And we get back another iq with the same id with the result of the previous query:

<iq from='erlang-solutions.com' to='ramabit@erlang-solutions.com/Andress-MacBook-Air' id='1' type='result'>
    <query xmlns='http://jabber.org/protocol/disco#items'>
        <item jid='muc.erlang-solutions.com'/>
        <item jid='muclight.erlang-solutions.com'/>
        <item jid='pubsub.erlang-solutions.com'/>
    </query>
</iq>

 

<presence/>

Used to exchange presence information, as you could have imagined. Usually presences are sent from the client to the server and broadcasted by it. The most basic, yet valid presence, to indicate to the server that a user is avaiable is:

<presence/>

After a sucessfull connection, you are not going to receive any <message/> until you make yourself available sending the previous presence.

If you want to make yourself unavailable, you just have to send:

<presence type="unavailable"></presence>

If we want to make the presences more useful, we can send something like this:

<presence>
      <status>On vacation</status>
</presence>

 

What's a Stream?

Before answering this, let's refresh our mind. What's a Unix socket? From Wikipedia: A socket is a special file used for inter-process communication. These allows communication between two processes. So a socket is a file that can be written by two processes (in the same computer or in different computers in the same network). So the client is going to write to this file and server too.

Ok, but how is a socket related to a Stream? Well, we are going to be connected to a server using a socket, therefore we are going to have a 'shared file' between the client and the server. This shared file is a white canvas where we are going to start writting our XML stanzas. The first thing we are going to write to this file is an opening <stream> tag! And there you go... that's our stream.

Perfect, I understand what a stream is, but I still don't understand how to send a message to the server. Well, the only thing we need to do to send a message is writting a <message/> stanza in our shared file. But what happens when the server wants to send me a message? Simple: it will write the message in the 'shared file'.

Are we ok so far?

 

I'm sure at this point you have questions like:

  • "What?! An active TCP connection open all the time? I'm used to REST! How am I going to do that?!" 

​           Easy, you don't have to care about that any more! That's why we are going to use the library, and it will take care of that.

  • "You said nothing about how to connect to the server!"

           Believe me, you don't have to care about this either. If we start adding all this info, we are going to get crazy. Trust me, I've been there.

  • "What about encrypted messages? We need security! How are we going to handle this?"

          Again, you don't have to care about this at this point. Baby steps!

 

You just need to be able to answer: "What's XMPP?", "How do you send a message?", "How do you change your status in XMPP?", "How do you ask something to the server?", "What's a Stream?". If you can answer all that, you are WAY better than me when I started.

All the concepts we described so far are the core of XMPP.  To find out how to get started with XMPPFramework, how to connect to the server and authenticate a user, go to PART 2!

Permalink

Piping & Based-On

Two new powerful survey features – made simple.

Piping:

  • Easily “pipe” in open ended text or selected responses from any previous question(s) in the survey.
  • The “pipe” can be in the question text or piped into any any option (or within any portion of an answer option).

Based-On:

  • Dynamically show answer options based upon any previous lists show in a survey.
  • Works across matrix, single choice, multiple choice and rating! (e.g., “Earlier you rated the following brands 3 out of 5 starts, for each please tell us why?” – this would show a list of all brands that were rated 3 stars by that respondent with a verbatim option for each.)

The post Piping & Based-On appeared first on icanmakeitbetter - Insight Communities, Market Research, Online Surveys and Focus Groups.

Permalink

Copyright © 2016, Planet Erlang. No rights reserved.
Planet Erlang is maintained by Proctor.