Mocks and Explicit Contracts: In Practice w/ Elixir

Writing tests for your code is easy. Writing good tests is much harder. Now throw in requests to external APIs that can return (or not return at all!) a myriad of different responses and we’ve just added a whole new layer of possible cases to our tests. When it comes to the web, it’s easy to overlook the complexity when working with an external API. It’s become so second nature that writing a line of code to initiate an HTTP request can become as casual as any other line of code within your application. However that’s not always the case.

We recently released the first version of our self-service debugging tool. You can see it live at https://debug.spreedly.com. The goal we had in mind for the support application was to more clearly display customer transaction data for debugging failed transactions. We decided to build a separate web application to layer on top of the Spreedly API which could deal with the authentication mechanics as well as transaction querying to keep separate concerns between our core transactional API and querying and displaying data. I should also mention that the support application is our first public facing Elixir application in production!

Since this was a separate service, it meant that we needed to make HTTP requests from the support application to our API in order to pull and display data. Although we had solid unit tests with example responses we’d expect from our API, we wanted to also incorporate remote tests which actually hit the production API so we could occasionally do a real world full stack test. Remote tests aren’t meant to be executed every test run, only when we want that extra ounce of confidence.

As we started looking into testing Elixir applications against external dependencies, we came across José Valim’s excellent blog post, Mocks and Explicit Contracts. If you haven’t read it already, you should check it out. It has a lot of great thoughts and will give this post a little more context. It seemed like a solid approach for building less brittle tests so we thought we implement it ourselves and see how well it would work in reality and if it could provide what we needed to include remote tests along side our unit tests. Here’s how our experience with this approach went…

Pluggable clients

The first thing we needed to do was update the Spreedly API client in the support application to be dynamically selected instead of hardcoded. In production we want to build real HTTP requests, but for unit tests we want to replace that module with a mock module which just returns simulated responses.

# Module wrapping our API requests for transactions
defmodule SupportApp.Transaction do
  @core Application.get_env(:support_app, :core)

  def fetch(token, user, secret) do
    @core.transaction(user, token, access_secret)
  end
end

In line 3 above, you can see that the @core constant is being dynamically set by doing a Application configuration lookup for the value. In our particular case, the lookup will return the module to use for getting Spreedly transaction data.

Once we’ve got that set, now we can configure the module to use in our application config. Notice that the module returned on lookup changes depending on the environment we’re currently running. We really like the explicitness here!

# In config/test.exs
config :support_app, :core, SupportApp.Core.Mock

# In config/config.exs
config :support_app, :core, SupportApp.Core.Api

Enforceable interfaces

So, what do those two modules look like anyway? Well, from our Transaction module above we know that both the HTTP client and the mock will need to have a transaction/3 function which will take care of getting us a transaction whether it be from the Spreedly API or a simulated one we build ourselves.

So in production, we’re using wrapping HTTPotion to make requests.

defmodule SupportApp.Core.Api do
  @behaviour SupportApp.Core
  ...
  def transaction(key, token, secret) do
    path = "..."
    options = ...

    %HTTPotion.Response{body: body} = HTTPotion.get(path, options)
    case Poison.Parser.parse!(body) do
      ...
    end
  end
  ...
end

However, in unit tests we’re going to use a mock module instead which just returns a straight Map of what we’d expect from a production request.

defmodule SupportApp.Core.Mock do
  @behaviour SupportApp.Core

  def transaction(_key, "nonexistent", _), do: nil
  def transaction(_key, token, _) do
    %{
      "transaction": %{
        "succeeded": true,
        "state": "succeeded",
        "token": "7icIbTtxupZpY8SKxwlUAKq8Qiw",
        "message_key": "messages.transaction_succeeded",
        "message": "Succeeded!",
        ...
      }
  end
end

Two things to notice about the modules above:

  1. For the mock module, we pattern matched on the function parameters to alter the response we want. This is incredibly useful when you want to test against different response scenarios since you can define another transaction/3 function with a specific parameter value and then have it return the response with appropriate test data. In our case, we wanted to also test when someone doesn’t enter a transaction token to search (see line 4 in the mock module above).

  2. In both the production and test modules, there’s a line beneath the module definition- @behaviour SupportApp.Core . By having a top level behaviour for these modules, we can be sure that our production API client module and our mock module adhere to the same interface. If we want to add another endpoint, start by adding it to the top level behaviour SupportApp.Core which will then trickle down to all modules and keep us explicitly defining our API contract so our production API client and mock client remain in step.

Here’s a snippet of our behaviour module that ensures all compliant behaviours have a transaction function with the current arity and argument types:

defmodule SupportApp.Core do
  use Behaviour

  @callback transaction(key :: String.t, token :: String.t, secret :: String.t) :: %{}
  ...
end

And that’s it for setup!

Test segmentation

Up to this point we’ve followed the approach as outlined by José’s blog post and created an explicit contract between our modules, allowing us to change underlying implementations depending on the environment we’re running in. That is, a mock module to be used during test and an Spreedly API client in production. However, our original plan was to include remote tests that actually hit our production API so how can we enable that in our tests?

Simple! In the same way we dynamically looked up the module we wanted to use, we can use the test setup block to pass the module we want to use in place of our external dependency client. So for the unit tests we have:

defmodule SupportApp.Core.MockTest do
  use ExUnit.Case, async: true

  setup do
      {:ok, core: Application.get_env(:support_app, :core)} # <-- See config/test.exs
  end

  test "transaction", %{core: core} do
    %{"transaction" => transaction} = core.transaction(...)
    assert txn["token"]
  end
end

But for our remote tests, we change our setup block to drop in a the real HTTP client wrapper. I should also note that we needed to handle creating production API credentials and making them available to our remote tests, but handling that is a bit out of scope for this post.

defmodule SupportApp.Core.ApiTest do
  use ExUnit.Case, async: true
  @moduletag :remote

  setup do
    {:ok, core: SupportApp.Core.Api} # <-- Force the module to be used
  end

  test "transaction", %{core: core} do
   %{"transaction" => txn} = core.transaction(...)
    assert txn["token"]
  end
end

Almost there! Since we don’t want to run our remote tests each time (they’re only around for that extra bit of confidence!) we can just use a @moduletag to exclude them by default and only run them if we explicitly say to do so. We added a line to our test_helper.exs :

ExUnit.configure exclude: [:remote]

Now to run the remote tests, just add an include flag:

$ mix test --include remote

We’re really happy with the setup so far and think it provides us both the confidence of real world full stack tests, with the flexibility of simulating responses within a mock.

Permalink

How do I GenStage?

I’m new to Elixir. And Erlang. And OTP’s GenServer. And GenStage. While I’ve got beginner’s-eye, I’m going to share knowledge for a general audience. So this is a doorknob-simple look at GenStage. With clear examples and I Love Lucy references. Enjoy!

What is a GenStage?

Lucy studies papers Fun and scalable? Interesting…

Erlang is a language invented to solve problems in scalable, fault-tolerant, and distributed ways. Elixir is a language built on top of Erlang that makes solving problems more fun. GenServer is an easy-to-use system for making generic server processes in Erlang (and now in Elixir). GenStage is an easy-to-use system built on top of GenServer to set up processing chains and solve the Lucy Chocolates Problem (see below).

Why would I GenStage?

You’re Elixiring because you have problems that can be split up and run on separate processors, VMs, machines, clusters… planets? (Yes, Elon. Maybe separate planets.)

Q: Let’s say your problem can be split into 2 parts. Are those two parts going to run at the same speed?
A: No, probably not. Part 1 is going to run at part-1-speed, and part 2 is going to run at part-2-speed.

The Lucy Chocolates Problem

For instance, your problem is shipping chocolates. Part 1 is making the treats, and part 2 is wrapping them. As long as your chocolate-wrapper is faster than your chocolate-maker, you’re good. As soon as your chocolate maker is faster than your chocolate wrapper, you have an I Love Lucy situation on your hands.

Lucy begins wrapping chocolates Lucy is overwhelmed by the speed of chocolates production Lucy desperately stuffs chocolates in her mouth and hat (Shown here is Lucille Ball, legendary entertainer, entrepreneur, and person-who-saved-Star-Trek.)

Lucy’s Chocolate Twitter-Feed

Let’s look at a real-life Lucy chocolates problem. Maybe you want to:

  1. pull words from a Twitter feed,
  2. Google image search them, and
  3. auto-create a collage of images based on those tweets.

From Twitter to Google to image generator

Pulling words from a Twitter feed is blazing fast. Google image searching is relatively slow. Photo-editing is glacial.

If you’re running these things in different processes chained together, it won’t be hard for the Twitter-reader process to overwhelm the image-searcher process which in-turn would lock up the collage-maker. Even if you have a bunch of collage-makers, you’d have to round-robin or find some other algorithm to keep them balanced. If you don’t do it right, you might miss some of the sweet sweet Tweets you desperately need to autogenerate your art. This is the same problem Lucy had with the chocolates.

Wouldn’t it be nice to have a way for your collage-maker processes to pull whatever they could handle from your image-searcher processes which could pull what they were able to handle from your tweet-reader?

There is such a way! That’s why GenStage. Because the Lucy Chocolates Problem.

How do I GenStage?

For right now, I’m going to go over the same stuff that was in the GenStage announcement. If you have that stuff down cold, you can skip to the Twitter feed example, scan through, and nod your head sagely.

There are 3 types of GenStage servers: producer , producer_consumer , and consumer . To implement them, you define a module that uses GenStage and create a few callback functions in that module.

Below is the dead-simplest way to set up a three-link chain. It doesn’t even do anything. Producer passes stuff straight through ProducerConsumer to Consumer, which does nothing. It compiles and runs. That’s its claim to fame.

First Glance

    alias Experimental.GenStage

    defmodule Producer do
      use GenStage
      def init(arg) do
        {:producer, :some_kind_of_state}
      end

      def handle_demand(demand, state) do
        {:noreply, things = [:whatever, :you, :want], state}
      end
    end

    defmodule ProducerConsumer do
      use GenStage
      def init(arg) do
        {:producer_consumer, :some_kind_of_state}
      end

      def handle_events(things, from, state) do
        {:noreply, things, state}
      end
    end

    defmodule Consumer do
      use GenStage  
      def init(arg) do
        {:consumer, :some_kind_of_state}
      end

      def handle_events(things, from, state) do
        {:noreply, [], state}
      end
    end

    defmodule Go do
      def go do
        {:ok, producer} = GenStage.start_link(Producer, arg = :nonsense)
        {:ok, prod_con} = GenStage.start_link(ProducerConsumer, arg = :nonsense)
        {:ok, consumer} = GenStage.start_link(Consumer, arg = :nonsense)

        GenStage.sync_subscribe(prod_con, to: producer)
        GenStage.sync_subscribe(consumer, to: prod_con)
      end
    end

In Detail

Great. Let’s see the same thing, but with gobs of annotations!

## Producer
    alias Experimental.GenStage

Development on GenStage is still going gangbusters. It will stay in the Experimental namespace for a while.

    defmodule Producer do

I could have called this thing whatever I wanted. Maybe I should have called it JerryBruckheimer .

      use GenStage

With this one line, we pull in all of GenServer and GenStage and make this module into an unstoppable data processing machine.

      def init(arg) do  

Init is a callback that’s triggered when you start your process. It takes an arg so you can set up your producer with whatever starting info you want. I don’t use the arg here, so I’m going to get a compiler warning.

        {:producer, :some_kind_of_state}
      end

Init needs to return a tuple with an atom and a something-else. The atom has to be :producer , :producer_consumer , or :consumer . That will set some assumptions for GenStage about how this module will behave. The something-else is the state for this server. Like all GenServers, a GenStage module can hold arbitrary data in memory.

      def handle_demand(demand, state) do

This callback is what makes a producer a producer. (Well that and the :producer atom returned by init ). handle_demand is called with the number of “things” a consumer is asking for and the state of the producer at the time the consumer asked for them.

        {:noreply, things = [:whatever, :you, :want], state}
      end
    end

The return value is intended to look like the return value from an asynchronous GenServer cast (or the return value from a call you’re not ready to reply to yet). :noreply indicates that handle_demand is not going to send any information back to whatever called it. This is confusing to me because handle_demand IS sending back information; the very next item in the tuple. But perhaps since it’s not sending information back in the way GenServer usually means, so we’ll let that go. things is the list of what-have-you that you’ve produced.

Things vs. Events - Most examples will use the word events for this. I don’t here because to me an “event” is a particular kind of thing. You could argue that a “thing” passed to a Consumer is an “event” as far as that Consumer is concerned. And I would see your point. …but I’d have to squint. Anyway, call it events everywhere else, but just for now, I’ll call it things.

Note that the things returned here is a list. It has to be a list. It can’t be a not-list. If you try to return something that is not a list, you will get ** (stop) bad return value: {:noreply, :decidedly_not_a_list, :ok} I want to pause here and note that bad return value is not the most helpful error message. Though I will also say that it’s more helpful than bad cast .

Search for bad cast leads to Breaking Bad

## ProducerConsumer

    defmodule ProducerConsumer do
      use GenStage
      def init(arg) do
        {:producer_consumer, :some_kind_of_state}
      end

We’re starting off pretty much the same way here. I could call the module whatever I want. Note it returns :producer_consumer and starting state. Again, :producer_consumer sets GenStage’s assumptions about how this thing will work. And state could be the complete works of Shakespeare; though I wouldn’t recommend it.

      def handle_events(things, from, state) do

This is the consumer’s main callback function. It takes a list of things (the same list sent by producer’s handle_demand above). It also takes from which is how we identify where these events came from, in case we need to. We usually don’t. You won’t often be using from ; you’ll more often call it _from (because underscores denote discarded values which helps avoid compiler warnings). Finally it takes whatever is the current state of the ProducerConsumer.

More about from - In case you’re interested, from (right now) is a tuple with the process identifier of the process the call came from and a reference to the request itself. I say “right now” because the GenServer documentation says that the format of from arguments may change in the future. So it’s best not to {pid, ref} = from if you can avoid it. The request reference seem to be generated by Elixir’s Kernel.make_ref . These references are theoretically unique (like GUUIDs, but not really). In GenServer, you can use from to reply directly to a request. GenServer.reply(from, :my_message) . GenServer makes use of simpler Erlang message passing calls behind the scenes, and tracks all this request reference stuff for you. It’s really cool. Thanks, Erlang.

        {:noreply, things, state}
      end
    end

ProducerConsumer is supposed to be changing that list of things in some way before passing it along to Consumer. But this one isn’t. Because it’s lazy. Here we return a tuple with :noreply ; again I assume because we’re not replying in the traditional GenServer sense. The tuple also has a list (and it must be a list) of things to go to the consumer. Finally, we have the ever-present state .

## Consumer
    defmodule Consumer do
      use GenStage  
      def init(arg) do
        {:consumer, :some_kind_of_state}
      end

By now, we’re pretty familiar with what’s going on here. Or we’ve fallen asleep and scrolled this far with our faces on the trackpad; I won’t judge.

      def handle_events(things, from, state) do

Consumer is handling events (with handle_events ) just the same way ProducerConsumer does, except…

        {:noreply, [], state}
      end
    end

Consumer MUST return an empty list in its tuple. If you return anything else, you’ll get [error] GenStage consumer #PID<0.514.0> cannot dispatch events (an empty list must be returned) This is quite a rebuke! It’s also much more helpful than “bad cast” and “bad return value” both.

    defmodule Go do
      def go do

Now what’s this Go.go() business? I like using a go function in my examples so they don’t automatically run when I use iex -S mix .

        {:ok, producer} = GenStage.start_link(Producer, arg = :nonsense)

Start up a Producer process. Pattern matching with :ok ensures everything went well or makes everything crash. Making something crash when things get a little sketchy is good Elixir practice. Elixir is basically a Windows user in the 1990s, hitting restart every time something goes sideways.

I was skeptical about this practice when I first heard about it. But you know what? I rebooted my way through Microsoft Windows 3.1, 95, 98, ME, and XP. And in-BETWEEN those reboots, I got a lot done. What I’m learning now is that all distributed systems are by-nature as buggy as Windows 3.1 and aspiring to be only as buggy as Windows 95.

        {:ok, prod_con} = GenStage.start_link(ProducerConsumer, arg = :nonsense)
        {:ok, consumer} = GenStage.start_link(Consumer, arg = :nonsense)

Anyway, we grab :ok and the process identifiers for our three GenStage processes. Note I’m passing arguments, because start_link expects me to. Those arguments are passed to the init functions described way above. But we ignore them. Also note that every time I do something like arg = :nonsense , I get a warning about how arg isn’t used. I love ignoring variable names with underscore _ , but when I’m writing examples I want them both descriptive and pretty.

        GenStage.sync_subscribe(prod_con, to: producer)

Aha! Now our ProducerConsumer is ready to grab whatever it can from Producer. It doesn’t grab anything because it’s not the final Consumer and so isn’t running the show.

        GenStage.sync_subscribe(consumer, to: prod_con)
      end
    end

There we go! That’s the final Consumer getting involved. As soon as this happens, Producer gets a call to handle_demand . Based on experience, the demand argument will probably be 500 that first time it gets called, but don’t count on that. Then ProducerConsumer gets a call to handle_events with [:whatever, :you, :want] as its things argument. Then Consumer gets an identical call to handle_events with [:whatever, :you, :want] as its things argument. Then it does nothing! Because again, lazy.

Twitter Feed Example

Fine! Let’s go back to our earlier example. I want to pull tweets from some feed, Google image search the words, and then use an image manipulator to make collages out of those images. I’ve put further description in comments below.

From Twitter to Google to image generator

    # Let's pretend this library is real
    alias AdriansHandWavingLibrary.{Tweet,ImageSearch,Collage}
    alias Experimental.GenStage

    defmodule ReadFromTwitter do
      use GenStage
      def init(twitter_feed) do
        tweets = Tweet.all_to_date(twitter_feed)
        # Note we're setting tweets as the state.
        {:producer, tweets}
      end

      def handle_demand(demand, state) do
        # Pull some tweets out of state. We send those as the events
        # or "things", and we reset state to the remaining tweets.
        # @jacobterpri pointed out the existence of Enum.split/2. Thanks!
        {pulled, remaining} = Enum.split(state, demand)
        {:noreply, pulled, remaining}
      end
    end

    defmodule ConvertToImages do
      use GenStage
      # This step still needs no state.
      def init(_) do
        {:producer_consumer, :ok}
      end

      # Turn a list of tweets into a list of lists of images.
      def handle_events(tweets, _from, _state) do
        image_lists = Enum.map(tweets, &to_list_of_images(&1))
        {:noreply, image_lists, :ok}
      end

      # Do that by splitting the tweets into individual words and running
      # image_for on each word
      defp to_list_of_images(tweet),
        do: tweet
          |> String.split(" ")
          |> Enum.map(fn word -> ImageSearch.image_for(word) end)
    end

    defmodule CollageBackToTwitter do
      use GenStage  
      # Set state to the one thing this needs to keep track of: where to post
      # collages.
      def init(output_twitter_feed) do
        {:consumer, output_twitter_feed}
      end

      # Get the lists of images, collage them together, and send them back out
      # to Twitter. This is definitely the longest step. There's image manipulation.
      # There's uploading. Then there's tweeting. All of that happens in my pretend
      # modules, but go ahead and pretend. That's a lot of time, isn't it?
      # So if we weren't using GenStage, the CollageBackToTwitter module would
      # require a ton of buffering code. The equivalent of Lucy stuffing chocolates
      # in her hat!
      # Thanks, GenStage.
      def handle_events(image_lists, _from, output_twitter_feed) do
        image_lists
        |> Enum.map(&Collage.images_together(&1))
        |> Enum.each(&Tweet.send_image(&1, output_twitter_feed))
        {:noreply, [], output_twitter_feed}
      end
    end

    defmodule Go do
      def go do
        # Note we're sending the Twitter names to pull from and push to.
        {:ok, producer} = GenStage.start_link(ReadFromTwitter, "@madcapulet")
        {:ok, prod_con} = GenStage.start_link(ConvertToImages, arg = :nonsense)
        {:ok, consumer} = GenStage.start_link(CollageBackToTwitter, "@bitcapulet")

        # I'm pretending here that I've tuned this stuff and found the following
        # levels of demand to be optimal. Because, hey look! There are a bunch of
        # settings on the sync_subscribe function. Rad.
        GenStage.sync_subscribe(prod_con, to: producer, max_demand: 50)
        GenStage.sync_subscribe(consumer, to: prod_con, max_demand: 10)
      end
    end

Obligatory Wrap-up Paragraph

If you have a multi-step, multi-process problem, and later steps may be slower than earlier steps, then consider GenStage. It’s simple, fun, and solves this problem thoroughly.

If you want to learn more about this you can read:

Thanks so much to José Valim and his cadre of brilliant and charming collaborators for Elixir and GenStage. Thanks also to the folks who’ve posted previous examples, to Lucille Ball (that human was genius), and to Spreedly for being such a great place to work.

Permalink

From Riak to Kafka: Part I

In Apache Kafka at Spreedly: Part I – Building a Common Data Substrate Ryan introduced the place Kafka will take in our infrastructure. In this series I’ll describe the implementation details of how we’re reliably and efficiently producing data from Riak to Kafka.

Spreedly’s Data Architecture

At Spreedly we use Riak as our permanent data storage. Riak is a key/value store that provides high availability of both the service and the data. As a key/value store it’s very fast at giving you the value if you have the key, but is not well suited to finding arbitrary values by the data they contain. To address that drawback we use Postgres as an indexing layer. Postgres is given the same set of data and indexes fields to support later queries. Each new piece of data is simultaneously written to both Riak and Postgres.

Spreedly's permanent storage data architecture

Adding Kafka to our Data Flow

We need to find a good way to incrementally introduce Kafka as a data substrate. We saw two potential options:

  1. Write to Kafka directly from our application layer
  2. Write from Riak to Kafka

Options for incorporating Apache Kafka into Spreedly's Data Architecture

The first option of adding Kafka as a new top level data layer component would put all the change work into the application layer. The second option of adding Kafka as a child data layer component of Riak would put all the change work into the data layer. Given that one of the major goals of this effort is to reduce our application complexity we opted to prefer the Riak integration if it proved viable.

Our early spikes proved that connecting Riak to Kafka was indeed technically possible. Riak allows configuring postcommit hooks that will be called with any newly written or deleted data.

Postcommit hooks are called after any successful write, don’t block the response to the client, and are directly given the entire key/value object. A postcommit hook would be the perfect place to have Riak write the data into Kafka. As a primarily Ruby shop the downside was that the Riak postcommit hook had to be written in Erlang.

Writing to Kafka directly from Riak

After some spin up time to get a footing in Erlang, it turned out that writing an Erlang function to receive the data and send it to Kafka turned out to be straightforward. Our first function was very simple: receive data, connect to Kafka, produce data, done.

Data flow diagram for producing to Apache Kafka from Riak

Not too surprisingly, when we measured performance it was clear this approach wouldn’t be acceptable in production. The overhead of the hook receiving the new data, then connecting to Kafka, and then producing the message was too inefficient. We’d be paying the cost to setup a TCP connection to Kafka for every individual message that we produced. We needed to find a way for the connection to Kafka to persist outside of the postcommit hook and be reused for each message.

Reliable Message Delivery

We found we could have Riak start a Kafka client during its boot process that would maintain a persistent and fault-tolerant connection. Now every single piece of data could reuse the same connection to Kafka. Performance issue solved!

After more experimentation we realized that the simple approach of producing to Kafka directly from the postcommit hook was not going to serve us very well. There were simply too many ways for the communication between Riak and Kafka to break down and cause any individual message to fail. Our postcommit hook would need to be smarter and retry delivery if Kafka is unreachable for some period of time.

The straightforward solution would have been to have postcommit hook take on the responsibility of retrying delivery to Kafka until success or some eventual timeout (e.g. five minutes). But that has some problems. As robust as Riak is operationally, it does have its limits. One of those limits is that it only allows a finite number of writing processes (such as the postcommit hook) to be active at any one time. Normally we’re nowhere near close to that limit, but if we were to start having processes potentially take minutes instead of milliseconds then we’d introduce the risk of running into that limit. Riak postcommit hooks are also simple functions without a robust support system. If they error out for any reason they have no infrastructure to try and re-run them.

We could have potentially dealt with both of those issues but the cost would’ve been an increase in complexity of the postcommit hook as well as our Riak configuration. We decided a better approach would be to build a new system component that would own the responsibility of producing messages to Kafka. This would allow the postcommit hook to remain focused on the simple path and require no complex alterations to Riak.

An unknown new component in Spreedly's data flow for producing to Apache Kafka

A System Outside of Riak

We needed a way to produce messages to Kafka that could tolerate Kafka being unavailable. Ideally each message could have its own lifecycle and make delivery attempts before succeeding or giving up after some long term timeout. The system would also need to give a response to the postcommit hook so that the hook would know if it should throw an error or consider the message as passed off successfully.

Erlang provides a framework called OTP that’s used for building systems that robustly handle synchronous and asynchronous requests. Exactly what we need for this system!

We made a few attempts to reasonably integrate an OTP system into Riak. After all, Riak runs in Erlang and it seemed reasonable to bring up another Erlang system alongside it. That approach might have been technically doable, but would’ve added too much complexity to our Riak setup. We needed to bring up a new system, separate from Riak.

The external system would need to do three things:

  1. Be callable from the postcommit hook
  2. Respond to the postcommit hook with success or failure
  3. Have high availability and good fault tolerance

It was clear that OTP would be a great fit to the problem. The only issue was the complexity of building an OTP system directly inside of Riak.

Elixir to the rescue! An Erlang system can directly communicate with an Elixir system using exactly the same OTP communication functions we wished to use. An Elixir system is an Erlang system once it’s running. We already had an interest in using Elixir for new services and projects so using Elixir for this system fit our goals very nicely.

A quick spike in Elixir proved that, yes, our Riak nodes would absolutely be able to communicate with an Elixir OTP application. We spun up that project as the “commitlog”.

A data flow diagram showing the new "commitlog" component that will produce to Apache Kafka

A Simplified Postcommit Hook

With the commitlog concept in place, the postcommit hook could switch back to the simple job of sending messages to an external system. The big difference is that the commitlog is now that external system and not Kafka directly. We can do our utmost to ensure that the commitlog remains highly available to the postcommit hook and the commitlog system takes on the responsibility of ensuring messages reach Kafka.

The refocused postcommit hook does three things:

  1. Transforms the new data into a message formatted for the commitlog
  2. Sends the message to the commitlog
  3. Times and records how long it takes to communicate with the commitlog

At each step, the postcommit hook handles failure and is ready to log enough data to allow us to queue the same data again. If you’d like to check out our approach we’ve open sourced our Riak postcommit hook repository. The src/postcommit_hook.erl file is the Erlang source of the hook itself.

Next time we’ll dive into the commitlog and walk through our design choices to create an Elixir OTP system that allows each individual message to be responsible for producing their own data to Kafka.

Permalink

5 Elixir highlights from Erlang & Elixir Factory 2017

<h1>Erlang &amp; Elixir Factory highlights</h1> <p>The latest Erlang &amp; Elixir Factory San Francisco was definitely worth the trip: new faces, interesting ideas and a buzzing hallway track.</p> <p>As usual, plenty of top-notch technical talks, but personally I found that two important trends emerged: teaching and embedded devices.</p> <p>In that respect, here are some highlights:</p> <h2>Sarah Allen - Language encodes Wisdom</h2> <p>In the opening keynote, Sarah Allen delivered a very powerful idea: the way we encode ideas in a language (whether it’s programming or not, it doesn’t really matter) also defines how we think about the world.</p> <p><div style="text-align:center"><iframe width="560" height="315" src="https://www.youtube.com/embed/_PdcGptErsY?rel=0" frameborder="0" allowfullscreen></iframe></div><br/></p> <p>It’s a solid linguistic perspective that can help us driving new adoption and community growth. Erlang and Elixir have a rich vocabulary that can help solving the big technological challenges of tomorrow, so we should work and capitalise on that.</p> <h2>Michal Slaski - Erlang Performance Lab</h2> <p>Michal Slaski introduced Erlang Performance Lab (EPL), a tool capable of visualising the traffic between nodes and processes in a Erlang/Elixir system.</p> <p>While walking through its features and capabilities, Michal touched on one of the most difficult challenges we face when teaching Erlang/Elixir: “seeing” processes.</p> <p><div style="text-align:center"><iframe width="560" height="315" src="https://www.youtube.com/embed/ncedupb-Rqw?rel=0" frameborder="0" allowfullscreen></iframe></div><br/></p> <p>EPL has an incredible potential as an educational tool that can help in exploring applications and grasping how a process-based system works, especially for people whose previous development experience doesn’t include anything similar.</p> <p>Best of all, this tool is open-source and open to contributions from all levels of experience.</p> <h2>Embedded devices galore</h2> <p>There were quite a few talks related to the embedded device development space.</p> <h3>Carl Hewitt - Concurrency and Strong Types for IoT</h3> <p>On the theoretical front, Carl Hewitt dove into strong types and concurrency to outline a mental model for the development of IoT devices, touching various topics from security to computational models.</p> <p><div style="text-align:center"><iframe width="560" height="315" src="https://www.youtube.com/embed/X-0WJ_COvBY?rel=0" frameborder="0" allowfullscreen></iframe></div><br/></p> <h3>Justin Schneck - Building Devices with Elixir/Erlang using Nerves</h3> <p>In terms of tools, Justin Schneck&rsquo;s talk <em>Building Devices with Elixir/Erlang using Nerves</em> showed that by using Nerves, you can quickly create robust devices from a wide selection of off the shelf hardware and accessories. </p> <p><div style="text-align:center"><iframe width="560" height="315" src="https://www.youtube.com/embed/aIGVOFwYtHE" frameborder="0" allowfullscreen></iframe></div><br/></p> <h3>Peer Stritzinger - Wireless Embedded Erlang Applications with Grisp Hardware Boards and Toolchain</h3> <p>Peer Stritzinger also showed how it&rsquo;s possible to dive into hardware-based development with minimal investment in his talk, <em>Wireless Embedded Erlang Applications with Grisp Hardware Boards and Toolchain</em>. </p> <p>Ever wanted to know how to quickly prototype Erlang Embedded Wireless Applications? Look no further. </p> <p><div style="text-align:center"><iframe width="560" height="315" src="https://www.youtube.com/embed/uQY6yD2Lw-g?rel=0" frameborder="0" allowfullscreen></iframe></div><br/></p> <p>In general, the IoT space seems to be fertile ground for Erlang/Elixir based technologies and these talks reflect the effort the community is putting into creating affordable tools that can help fostering innovation.</p> <h2>The hallway track</h2> <p>Chatting to different people revealed quite a broad range of projects and use cases, from startups to big companies (some of them new in the Elixir/Erlang space).</p> <p>There’s a shared consensus that these technologies are solid and worth investing into, so we can safely say we’re past the phase where people are wondering about the commercial suitability of Erlang/Elixir (particularly the latter).</p> <p>At the same time, there’s a clear need to bridge a knowledge gap between people with multi-year expertise on distributed computing and a significant number of people joining the development community.</p> <p>Conversely, BEAM for a wider audience means more exposure to topics like user-experience, user interfaces, fast prototyping and generally speaking more user-facing software.</p> <p>In short, lots of food for thought.</p> <hr> <p><strong>Catch up on the latest and greatest developments in Elixir at <a href="http://www.elixirconf.eu/elixirconf2017?utm_source=Erlang%20Solutions&amp;utm_medium=EEF17%20blog">ElixirConf.EU</a>, where <a href="http://www.elixirconf.eu/elixirconf2017/claudio-ortolina?utm_source=Erlang%20Solutions&amp;utm_medium=EEF17%20blog">Claudio will discuss availability</a> &gt;</strong></p>

Permalink

DockYard Launches Redesigned Website

Today, DockYard released a new version of their flagship website DockYard.com. This latest iteration provides more indepth information on DockYard’s software design and engineering services, body of work, and pursuit of best practice web app development.

“Our new website design represents a collaborative effort between our design and engineering teams to deliver a clean, fast, and beautiful vision for how we want the world to view DockYard,” says Brian Cardarella, DockYard CEO. “As with all of our internal design and engineering efforts we pushed ourselves to not compromise on the implementation.”

In the coming weeks users will see even more advances on DockYard.com, including better representations on the company's culture, hiring, and team. Their blog—“Reefpoints”—will also receive a refresh with a design that better organizes content and provides a more immersive reading experience. New case studies and exclusive content on Progressive Web Applications (PWAs) will also be updated regularly, providing a more thorough understanding of the solutions that DockYard creates.

“We have many lessons learned that we hope to share with the community as a whole,” continues Cardarella. “One of our primary missions at DockYard is to help improve the eco-system that we work in. Improvements to many of our Ember.js and Elixir libraries came out of this effort, and we’re excited to share these in the coming weeks.”

For more information on DockYard and it’s services please visit DockYard.com/services.

DockYard is a software consultancy driven by creating exceptional user experiences. By approaching software with the user experience at the forefront, DockYard delivers web applications that beat out native applications in any web browser using HTML5, CSS, and JavaScript.

Permalink

Take Control of your RabbitMQ queues

<p><strong>Erlang Solutions offers world-leading RabbitMQ consultancy, support &amp; tuning solutions. <a href="http://www2.erlang-solutions.com/l/23452/2017-04-24/4kcfsn">Learn more &gt;</a></strong></p> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">You’re a support engineer&nbsp;and your organisation uses a 3-node RabbitMQ cluster to manage its inter-application interactions across your network. On a regular basis, different departments in your organisation approach you with requests to integrate their new microservices into the network, for communication with other microservices via RabbitMQ.</p> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">With the organisation being so huge, and offices spread across the globe, onus is on each department’s application developers to handle the integration to the RabbitMQ cluster, only after you&rsquo;ve approved and given them the green light to do so. Along with approving integration requests, you also provide general conventions which you’ve adopted from prior experience. Part of the conventions you enforce is that the connecting microservices must create their own dedicated queues on integration to the cluster, as the best approach to isolating services and easily managing them. Unless of course, the microservices would be seeking to only consume messages from already existing queues.</p> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">So, average message rate across your cluster is almost stable at <strong style="box-sizing:border-box">1k/s</strong>, both from internal traffic, and external traffic which is being generated by some mobile apps publicised by the organisation. Everything is smooth sailing, till you get to a point where you realise that the total number of queues in your cluster is nearing the order of thousands, and one of the three servers seems to be over burdened, using more system resources than rest. Memory utilisation on that server starts reaching alarming thresholds. At this point, you realise that things can only get worse, yet you still have more pending requests for integration of more microservices onto the cluster, but can&rsquo;t approve them without figuring out how to solve the growing imbalance in system resources across your deployment.</p> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;"><img alt="RabbitMQ cluster imbalance" src="https://esl-website-production.s3.amazonaws.com/uploads/content_RabbitCluster.png" /></p> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;"><strong style="box-sizing:border-box">Fig 1. RabbitMQ cluster imbalance illustration</strong></p> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">After digging up on some RabbitMQ <a href="https://www.rabbitmq.com/documentation.html" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">documentation</a>, you come to light with the fact that since you&rsquo;re using HA queues, which you’ve adopted to enforce availability of service, all your message operations only reference your master queues. Microservices have been creating queues on certain nodes at will, implying that the provisioning of queues has been random and unstructured across the cluster. Concentration of HA queue masters on one node significantly surpass that on the other nodes, and as a result, with all message consumptions referencing master queues only, the server with the most queue masters is feeling the operational burden in comparison to the rest.</p> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">Your load balancer hasn&rsquo;t been of much help, since whenever you experience a network partition, or purposefully make one of your nodes unavailable for maintenance work, queue provisioning has proceeded uncontrolled on the remaining running nodes. This retains the queue count imbalance upon cluster restoration. A possible immediate solution would be to purge some of the queues, to relieve the memory footprint on the burdened server(s), but you can&rsquo;t afford to do this as most queued up messages are crucial to all business operations transacting through the cluster. New and existing microservices also can&rsquo;t continue timelessly creating and adding more queues into the cluster until this problem has been addressed. So what do you do?</p> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">Well, as of version <strong style="box-sizing:border-box">3.6.0</strong>, RabbitMQ has introduced a mechanism to grant its users more control in determining a queue master&rsquo;s location in a cluster, on creation. This is based on some predefined rules and strategies, configured prior to the queue declaration operations. If you can relate with the situation above, or would like to plan ahead and make necessary amendments to your RabbitMQ installation before encountering similar problems, then read on, and give this feature a go.</p> <h2 style="box-sizing: border-box; margin-top: 1em; margin-bottom: 16px; line-height: 1.225; font-size: 1.75em; font-weight: bold; padding-bottom: 0.3em; border-bottom-width: 1px; border-bottom-style: solid; border-bottom-color: rgb(238, 238, 238); font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol';">So how does it work?</h2> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">Prior to introducing the <em style="box-sizing:border-box">queue master location</em> mechanism, declaration of queues, by default, had been characterized by the queue master being located on the local node on which the declare operation was being executed on. This is somewhat very limiting, and has been the main reason behind the inefficient imbalance of system resources on a RabbitMQ cluster when the number of queues become significantly large.</p> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">Upon introducing this mechanism, the node on which the queue master will be located is now first computed from a configurable strategy, prior to the queue being created.</p> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;"><em style="box-sizing:border-box">Configurable strategy</em> is key here, as it leverages full control to RabbitMQ users to dictate the distribution of queue masters across their cluster. There are three means by which a queue master location strategy may be configured;</p> <ol style="box-sizing: border-box; padding-left: 2em; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;"> <li style="box-sizing: border-box;"><strong style="box-sizing:border-box">Queue declare arguments</strong>: This is at AMQP level, where the queue master location strategy is defined as part of the queue&rsquo;s declaration arguments</li> <li style="box-sizing: border-box;"><strong style="box-sizing:border-box">Policy</strong>: Here the strategy is defined as a RabbitMQ policy.</li> <li style="box-sizing: border-box;"><strong style="box-sizing:border-box">Configuration file</strong>: Location strategy is defined in the <a href="https://www.rabbitmq.com/configure.html#configuration-file" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;"><strong style="box-sizing:border-box">rabbitmq.config</strong></a> file.</li> </ol> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">Once set, the internal execution order of declaring a queue would be as follows;</p> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;"><img alt="RabbitMQ queue master location execution flow" src="https://esl-website-production.s3.amazonaws.com/uploads/content_FlowChart__1_.png" /></p> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;"><strong style="box-sizing:border-box">Fig 2. Queue master location execution flow</strong></p> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">These are the three ways in which a queue master location strategy may be configured, and how the execution flow is ordered upon queue declaration. Next, you may be asking yourself the following question;</p> <h3 style="box-sizing: border-box; margin-top: 1em; margin-bottom: 16px; line-height: 1.43; font-size: 1.5em; font-weight: bold; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol';">What are these strategies anyway?</h3> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">Queue master location strategies are basically the rules which govern the selection of the node on which the queue master will reside, on declaration. If you’re from an Erlang background, you’d understand when I say these strategies are nothing but callback modules of a certain behaviour pattern in RabbitMQ known as the <strong style="box-sizing:border-box">rabbit_queue_master_locator</strong>. If you aren’t from an Erlang background, no worries, all you need to know is what strategies are available to you, and how to make use of them. Currently, there are three queue master location strategies available;</p> <ol style="box-sizing: border-box; padding-left: 2em; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;"> <li style="box-sizing: border-box;"> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;"><strong style="box-sizing:border-box">Min-Masters</strong>: Selects the master node as the one with the least running master queues. Configured as <em style="box-sizing:border-box">min-masters</em>.</p> </li> <li style="box-sizing: border-box;"> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;"><strong style="box-sizing:border-box">Client-local</strong>: Like previous default node selection policy, this strategy selects the queue master node as the local node on which the queue is being declared. Configured as <em style="box-sizing:border-box">client-local</em>.</p> </li> <li style="box-sizing: border-box;"> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;"><strong style="box-sizing:border-box">Random</strong>: Selects the queue master node based on random selection. Configured as <em style="box-sizing:border-box">random</em>.</p> </li> </ol> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">So in a nutshell, this is the general theory behind controlling and dictating the location of a queue master’s node. Syntax rules differ for each case, depending on whether the strategy is defined as part of the queue’s declare arguments, as a policy, or as part of the <a href="https://www.rabbitmq.com/configure.html#configuration-file" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;"><strong style="box-sizing:border-box">rabbitmq.config</strong></a> file.</p> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;"><strong style="box-sizing:border-box">NOTE</strong>: When both, a <em style="box-sizing:border-box">queue master location</em> strategy and <em style="box-sizing:border-box">HA nodes</em> policy have been configured, a conflict could arise in the resulting queue master node. For instance, if one of the slave nodes defined by the <em style="box-sizing:border-box">HA nodes</em> policy becomes the queue master node computed by the location strategy. In such a scenario, the <em style="box-sizing:border-box">HA nodes</em> policy would always take precedence over the <em style="box-sizing:border-box">queue master location strategy</em>.</p> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">With this knowledge at hand, the <a href="https://gist.github.com/Ayanda-D/303cdd1a994fe75798c9#engineer" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">engineer</a> in the situation mentioned above would simply enforce the use of the <strong style="box-sizing:border-box">min-masters</strong> queue location strategy as part of the queue declaration arguments for all microservices connecting to the RabbitMQ cluster. Or even easier, he’d simply set the <strong style="box-sizing:border-box">min-masters</strong> policy on the cluster nodes, using the match-all wildcard for the queue name match pattern. This would ensure that all newly created queues would be automatically distributed across the cluster until there’s a balance in the number of queue masters per node, and ultimately, a balance in the utilization of system resources across all three servers.</p> <h2 style="box-sizing: border-box; margin-top: 1em; margin-bottom: 16px; line-height: 1.225; font-size: 1.75em; font-weight: bold; padding-bottom: 0.3em; border-bottom-width: 1px; border-bottom-style: solid; border-bottom-color: rgb(238, 238, 238); font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol';">Going forward</h2> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">At the moment, only three location strategies have been implemented, namely; <em style="box-sizing:border-box">min-masters</em>, <em style="box-sizing:border-box">client-local</em> and <em style="box-sizing:border-box">random</em>. More strategies are yet to be brewed up, and if you feel you&rsquo;d like to contribute a rule by which the distribution of queues can be carried out to better improve the performance of a RabbitMQ cluster, please feel free to drop a comment. These will go through some rounds of review, and could possibly be implemented and included in near future releases of <a href="https://github.com/rabbitmq" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">RabbitMQ</a>.</p> <h2 style="box-sizing: border-box; margin-top: 1em; margin-bottom: 16px; line-height: 1.225; font-size: 1.75em; font-weight: bold; padding-bottom: 0.3em; border-bottom-width: 1px; border-bottom-style: solid; border-bottom-color: rgb(238, 238, 238); font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol';">Quick n&rsquo; easy experiment</h2> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">I&rsquo;ll illustrate how the queue master location strategy is put into effect with a simple experiment to carry out on your local machine. We&rsquo;re going make things easy by making most of the management UI, to avoid the whole AMQP setup procedures like opening of connections and channels, creating exchanges, and so forth.</p> <ol style="box-sizing: border-box; padding-left: 2em; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;"> <li style="box-sizing: border-box;"> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;"><a href="https://www.rabbitmq.com/download.html" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">Download and install</a> a RabbitMQ package specific for your platform. If you&rsquo;re on a UNIX based OS, you can just quickly download and extract the <a href="https://www.rabbitmq.com/install-generic-unix.html" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">generic unix package</a>, and navigate to the <strong style="box-sizing:border-box">sbin</strong> directory.</p> <pre style="box-sizing: border-box; overflow: auto; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin-top: 0px; margin-bottom: 16px; font-stretch: normal; line-height: 1.45; padding: 16px; border-radius: 3px; word-wrap: normal; background-color: rgb(247, 247, 247);"> <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0px; margin: 0px; border-radius: 3px; word-break: normal; white-space: pre; border: 0px; display: inline; max-width: initial; overflow: initial; line-height: inherit; word-wrap: normal; background: transparent;">tar xvf rabbitmq-server-generic-unix-3.6.1.tar.xz cd rabbitmq-server-generic-unix-3.6.1/sbin </code></pre> </li> <li style="box-sizing: border-box;"> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">Create a 3-node cluster by executing the following commands;</p> <pre style="box-sizing: border-box; overflow: auto; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin-top: 0px; margin-bottom: 16px; font-stretch: normal; line-height: 1.45; padding: 16px; border-radius: 3px; word-wrap: normal; background-color: rgb(247, 247, 247);"> <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0px; margin: 0px; border-radius: 3px; word-break: normal; white-space: pre; border: 0px; display: inline; max-width: initial; overflow: initial; line-height: inherit; word-wrap: normal; background: transparent;">export RABBITMQ_NODE_PORT=5672 &amp;&amp; export RABBITMQ_NODENAME=rabbit &amp;&amp; ./rabbitmq-server -detached export RABBITMQ_NODE_PORT=5673 &amp;&amp; export RABBITMQ_NODENAME=rabbit_1 &amp;&amp; ./rabbitmq-server -detached export RABBITMQ_NODE_PORT=5674 &amp;&amp; export RABBITMQ_NODENAME=rabbit_2 &amp;&amp; ./rabbitmq-server -detached ./rabbitmqctl -n rabbit_1@hostname stop_app ./rabbitmqctl -n rabbit_1@hostname join_cluster rabbit@hostname ./rabbitmqctl -n rabbit_1@hostname start_app ./rabbitmqctl -n rabbit_2@hostname stop_app ./rabbitmqctl -n rabbit_2@hostname join_cluster rabbit@hostname ./rabbitmqctl -n rabbit_2@hostname start_app </code></pre> </li> <li style="box-sizing: border-box;"> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">Enable the rabbitmq_management plugin;</p> <pre style="box-sizing: border-box; overflow: auto; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin-top: 0px; margin-bottom: 16px; font-stretch: normal; line-height: 1.45; padding: 16px; border-radius: 3px; word-wrap: normal; background-color: rgb(247, 247, 247);"> <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0px; margin: 0px; border-radius: 3px; word-break: normal; white-space: pre; border: 0px; display: inline; max-width: initial; overflow: initial; line-height: inherit; word-wrap: normal; background: transparent;">./rabbitmq-plugins enable rabbitmq_management </code></pre> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">and access the management UI from <a href="http://localhost:15672/" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">http://localhost:15672</a>.</p> </li> <li style="box-sizing: border-box;"> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">Verify that your 3-node cluster was successfully created by checking cluster status;</p> <pre style="box-sizing: border-box; overflow: auto; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin-top: 0px; margin-bottom: 16px; font-stretch: normal; line-height: 1.45; padding: 16px; border-radius: 3px; word-wrap: normal; background-color: rgb(247, 247, 247);"> <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0px; margin: 0px; border-radius: 3px; word-break: normal; white-space: pre; border: 0px; display: inline; max-width: initial; overflow: initial; line-height: inherit; word-wrap: normal; background: transparent;">./rabbitmqctl cluster_status </code></pre> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">Or, from the management UI <a href="http://localhost:15672/#/" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;"><strong style="box-sizing:border-box">Overview</strong></a> page.</p> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;"><img alt="RabbitMQ cluster nodes" src="https://esl-website-production.s3.amazonaws.com/uploads/content_ClusterStatus.png" /></p> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;"><strong style="box-sizing:border-box">Fig 3. RabbitMQ cluster nodes</strong></p> </li> <li style="box-sizing: border-box;"> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">Next, navigate to the <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">Queues</code> tab on your management UI, and create <strong style="box-sizing:border-box">3</strong> qeueues on node <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">rabbit_2@hostname</code>. I prefixed my queues with the node name, i.e. <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">rabbit_2.queue.1</code>, and created them as follows</p> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;"><img alt="Rabbit_2 hostname queues" src="https://esl-website-production.s3.amazonaws.com/uploads/content_AddQueue__1_.png" /></p> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;"><strong style="box-sizing:border-box">Fig 4. Add rabbit_2@hostname queues</strong></p> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">Repeat this procedure for queues <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">rabbit_2.queue.2</code> and <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">rabbit_2.queue.3</code>.</p> </li> <li style="box-sizing: border-box;"> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">Repeat <strong style="box-sizing:border-box">step 5</strong> on nodes <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">rabbit_1@hostname</code> and <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">rabbit@hostname</code>, creating <strong style="box-sizing:border-box">5</strong> and <strong style="box-sizing:border-box">9</strong> qeueues on each, respectively.</p> </li> <li style="box-sizing: border-box;"> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">If you&rsquo;ve carried out <strong style="box-sizing:border-box">steps 5</strong> &amp; <strong style="box-sizing:border-box">6</strong> correctly, your queue listing should be similar to the following;</p> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;"><img alt="created rabbitmq queues" src="https://esl-website-production.s3.amazonaws.com/uploads/content_Queues__1_.png" /></p> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;"><strong style="box-sizing:border-box">Fig 5. Created queues</strong></p> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">Or, from the command line, by executing the following;</p> <pre style="box-sizing: border-box; overflow: auto; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin-top: 0px; margin-bottom: 16px; font-stretch: normal; line-height: 1.45; padding: 16px; border-radius: 3px; word-wrap: normal; background-color: rgb(247, 247, 247);"> <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0px; margin: 0px; border-radius: 3px; word-break: normal; white-space: pre; border: 0px; display: inline; max-width: initial; overflow: initial; line-height: inherit; word-wrap: normal; background: transparent;">./rabbitmqctl list_queues -q name pid state </code></pre> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">resulting in a queue listing alike the following;</p> <pre style="box-sizing: border-box; overflow: auto; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin-top: 0px; margin-bottom: 16px; font-stretch: normal; line-height: 1.45; padding: 16px; border-radius: 3px; word-wrap: normal; background-color: rgb(247, 247, 247);"> <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0px; margin: 0px; border-radius: 3px; word-break: normal; white-space: pre; border: 0px; display: inline; max-width: initial; overflow: initial; line-height: inherit; word-wrap: normal; background: transparent;">Ayandas-MacBook-Pro:sbin ayandadube$ ./rabbitmqctl list_queues -q name pid state rabbit_2.queue.3 &lt;rabbit_2@Ayandas-MacBook-Pro.3.1527.1&gt; running rabbit.queue.3 &lt;rabbit@Ayandas-MacBook-Pro.2.4772.0&gt; running rabbit.queue.2 &lt;rabbit@Ayandas-MacBook-Pro.2.4761.0&gt; running rabbit.queue.4 &lt;rabbit@Ayandas-MacBook-Pro.2.4783.0&gt; running rabbit.queue.6 &lt;rabbit@Ayandas-MacBook-Pro.2.4815.0&gt; running rabbit.queue.7 &lt;rabbit@Ayandas-MacBook-Pro.2.4827.0&gt; running rabbit_1.queue.2 &lt;rabbit_1@Ayandas-MacBook-Pro.2.1433.0&gt; running rabbit_1.queue.4 &lt;rabbit_1@Ayandas-MacBook-Pro.2.1455.0&gt; running rabbit.queue.1 &lt;rabbit@Ayandas-MacBook-Pro.2.4751.0&gt; running rabbit_1.queue.1 &lt;rabbit_1@Ayandas-MacBook-Pro.2.1416.0&gt; running rabbit_2.queue.1 &lt;rabbit_2@Ayandas-MacBook-Pro.3.660.1&gt; running rabbit.queue.9 &lt;rabbit@Ayandas-MacBook-Pro.2.4848.0&gt; running rabbit_1.queue.3 &lt;rabbit_1@Ayandas-MacBook-Pro.2.1444.0&gt; running rabbit_2.queue.2 &lt;rabbit_2@Ayandas-MacBook-Pro.3.896.1&gt; running rabbit.queue.8 &lt;rabbit@Ayandas-MacBook-Pro.2.4838.0&gt; running rabbit_1.queue.5 &lt;rabbit_1@Ayandas-MacBook-Pro.2.1476.0&gt; running rabbit.queue.5 &lt;rabbit@Ayandas-MacBook-Pro.2.4794.0&gt; running </code></pre> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">NOTE: The <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">pid</code>&rsquo;s prefix is a good indicator of each queue&rsquo;s home node.</p> </li> <li style="box-sizing: border-box;"> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">Next, we&rsquo;re going to create a queue with the <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">min-masters</code> strategy configured, and verify that it&rsquo;s been created on the correct and expected node. We&rsquo;ll call our queue <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">MinMasterQueue.1</code>.</p> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">Firstly, lets configure the <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">min-masters</code> queue master location policy. We&rsquo;ll name this policy <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">qml-policy</code>, and set it to be applied to all queues, of names prefixed with <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">MinMasterQueue</code>;</p> <pre style="box-sizing: border-box; overflow: auto; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin-top: 0px; margin-bottom: 16px; font-stretch: normal; line-height: 1.45; padding: 16px; border-radius: 3px; word-wrap: normal; background-color: rgb(247, 247, 247);"> <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0px; margin: 0px; border-radius: 3px; word-break: normal; white-space: pre; border: 0px; display: inline; max-width: initial; overflow: initial; line-height: inherit; word-wrap: normal; background: transparent;">./rabbitmqctl set_policy qml-policy "^MinMasterQueue\." '{"queue-master-locator":"min-masters"}' --apply-to queues </code></pre> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">With our policy is configured, we can now go on and create our <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">MinMasterQueue.1</code> queue.</p> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">So to prove that the <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">min-masters</code> policy does work, we&rsquo;ll create our <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">MinMasterQueue.1</code> queue on the node with the<strong style="box-sizing:border-box">most</strong> number of queues, i.e. <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">rabbit@hostname</code> (9 queues). With the queue master location policy in effect, the set<code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">rabbit@hostname</code> node should be overriden by the node computed by the policy, which has the least number of queues, i.e. <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">rabbit_2@hostname</code>. Let&rsquo;s proceed!</p> </li> <li style="box-sizing: border-box;"> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">So as mentioned in <strong style="box-sizing:border-box">step 8</strong>, let&rsquo;s create <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">MinMasterQueue.1</code> on the node with the most queues, <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">rabbit@hostname</code>, as follows;</p> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;"><img alt="RabbitMQ MinMaster Queue" src="https://esl-website-production.s3.amazonaws.com/uploads/content_MinMasterQueue.png" /></p> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;"><strong style="box-sizing:border-box">Fig 6. MinMasterQueue creation</strong></p> </li> <li style="box-sizing: border-box;"> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">Now the moment of truth; let&rsquo;s verify if the queue was created on the correct node;</p> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;"><img alt="Created MinMasterQueue" src="https://esl-website-production.s3.amazonaws.com/uploads/content_CreatedMinMasterQueue.png" /></p> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;"><strong style="box-sizing:border-box">Fig 7. Min-master queue</strong></p> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">Or, from the command line by executing the following;</p> <pre style="box-sizing: border-box; overflow: auto; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin-top: 0px; margin-bottom: 16px; font-stretch: normal; line-height: 1.45; padding: 16px; border-radius: 3px; word-wrap: normal; background-color: rgb(247, 247, 247);"> <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0px; margin: 0px; border-radius: 3px; word-break: normal; white-space: pre; border: 0px; display: inline; max-width: initial; overflow: initial; line-height: inherit; word-wrap: normal; background: transparent;">./rabbitmqctl list_queues -q name pid </code></pre> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">which should yield the following as part of the full list of displayed queues;</p> <pre style="box-sizing: border-box; overflow: auto; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin-top: 0px; margin-bottom: 16px; font-stretch: normal; line-height: 1.45; padding: 16px; border-radius: 3px; word-wrap: normal; background-color: rgb(247, 247, 247);"> <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0px; margin: 0px; border-radius: 3px; word-break: normal; white-space: pre; border: 0px; display: inline; max-width: initial; overflow: initial; line-height: inherit; word-wrap: normal; background: transparent;">MinMasterQueue.1 &lt;rabbit_2@Ayandas-MacBook-Pro.3.1995.3&gt; </code></pre> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">The results are indeed correct. The home node of <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">MinMasterQueue.1</code> is rightly the one which had the least number queue masters, i.e. <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">rabbit_2@hostname</code>.</p> </li> <li style="box-sizing: border-box;"> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;">You can repeatedly execute <strong style="box-sizing:border-box">step 9</strong>, creating more <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">MinMasterQueue.N</code> queues to see this queue master location strategy in effect. The home node of the queues created will interchange from one node to another, depending on the queue masters&rsquo; count per node, at each given moment of execution.</p> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;"><img alt="Queue MinMasters" src="https://esl-website-production.s3.amazonaws.com/uploads/content_QueueMinMasters__1_.png" /></p> <p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px;"><strong style="box-sizing:border-box">Fig 8. Min-master queues</strong></p> </li> </ol> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">This is a quick illustration of this mechanism at work. In addition to setting a policy from the command line, as in <strong style="box-sizing:border-box">step 8</strong>, there also other means of defining the queue master location strategy which I illustrate in the next section.</p> <h2 style="box-sizing: border-box; margin-top: 1em; margin-bottom: 16px; line-height: 1.225; font-size: 1.75em; font-weight: bold; padding-bottom: 0.3em; border-bottom-width: 1px; border-bottom-style: solid; border-bottom-color: rgb(238, 238, 238); font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol';">Examples</h2> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">Following are some examples of how to configure queue master location strategies.</p> <h4 style="box-sizing: border-box; margin-top: 1em; margin-bottom: 16px; line-height: 1.4; font-size: 1.25em; font-weight: bold; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol';">1. rabbitmq.config</h4> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">Firstly, to set the location strategy from the <a href="https://www.rabbitmq.com/configure.html#configuration-file" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;"><strong style="box-sizing:border-box">rabbitmq.config</strong></a> file, simply add the following configuration entry;</p> <pre style="box-sizing: border-box; overflow: auto; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin-top: 0px; margin-bottom: 16px; font-stretch: normal; line-height: 1.45; padding: 16px; border-radius: 3px; word-wrap: normal; background-color: rgb(247, 247, 247);"> <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0px; margin: 0px; border-radius: 3px; word-break: normal; white-space: pre; border: 0px; display: inline; max-width: initial; overflow: initial; line-height: inherit; word-wrap: normal; background: transparent;">{rabbit,[ . . {queue_master_locator, &lt;&lt;"min-masters"&gt;&gt;}, . . ]}, </code></pre> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;"><strong style="box-sizing:border-box">NOTE</strong>: The strategy is configured as an Erlang <a href="http://erlang.org/doc/reference_manual/data_types.html#id67658" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">binary</a> data type i.e. <strong style="box-sizing:border-box">&lt;&lt;&ldquo;min-masters&rdquo;&gt;&gt;</strong>.</p> <h4 style="box-sizing: border-box; margin-top: 1em; margin-bottom: 16px; line-height: 1.4; font-size: 1.25em; font-weight: bold; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol';">2. Policy</h4> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">As already seen from our experiment, setting the strategy as a <a href="https://www.rabbitmq.com/parameters.html#policies" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;"><strong style="box-sizing:border-box">policy</strong></a> on a UNIX environment can be carried out as follows;</p> <pre style="box-sizing: border-box; overflow: auto; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin-top: 0px; margin-bottom: 16px; font-stretch: normal; line-height: 1.45; padding: 16px; border-radius: 3px; word-wrap: normal; background-color: rgb(247, 247, 247);"> <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0px; margin: 0px; border-radius: 3px; word-break: normal; white-space: pre; border: 0px; display: inline; max-width: initial; overflow: initial; line-height: inherit; word-wrap: normal; background: transparent;">rabbitmqctl set_policy qml-policy ".*" '{"queue-master-locator":"min-masters"}' --apply-to queues </code></pre> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">This creates a <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">min-masters</code> queue location strategy policy, of name <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">qml-policy</code>, which, from the <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0px; margin: 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">".*"</code> wildcard match pattern, will be applied to all queues created on the node/cluster.</p> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">You can find more information on defining policies from the official RabbitMQ <a href="https://www.rabbitmq.com/parameters.html#policies" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;"><strong style="box-sizing:border-box">documentation</strong></a>.</p> <h4 style="box-sizing: border-box; margin-top: 1em; margin-bottom: 16px; line-height: 1.4; font-size: 1.25em; font-weight: bold; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol';">3. Declare arguments</h4> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">I illustrate setting the queue location strategy from declare arguments using three examples; in <strong style="box-sizing:border-box">Erlang</strong>, <strong style="box-sizing:border-box">Java</strong> and <strong style="box-sizing:border-box">Python</strong>.</p> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">In <strong style="box-sizing:border-box">Erlang</strong>, you&rsquo;d simply specify the location strategy as part of the &lsquo;queue.declare&rsquo; record as follows;</p> <pre style="box-sizing: border-box; overflow: auto; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin-top: 0px; margin-bottom: 16px; font-stretch: normal; line-height: 1.45; padding: 16px; border-radius: 3px; word-wrap: normal; background-color: rgb(247, 247, 247);"> <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0px; margin: 0px; border-radius: 3px; word-break: normal; white-space: pre; border: 0px; display: inline; max-width: initial; overflow: initial; line-height: inherit; word-wrap: normal; background: transparent;"> Args = [{&lt;&lt;"x-queue-master-locator"&gt;&gt;, &lt;&lt;"min-masters"&gt;&gt;}], QueueDeclare = #'queue.declare'{queue = &lt;&lt;"microservice.queue.1"&gt;&gt;, auto_delete= true, durable = false, arguments = Args }, #'queue.declare_ok'{} = amqp_channel:call(Channel, QueueDeclare), </code></pre> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">In <strong style="box-sizing:border-box">Java</strong>, just create an arguments map, define the queue master location strategy and declare the queue as follows;</p> <pre style="box-sizing: border-box; overflow: auto; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin-top: 0px; margin-bottom: 16px; font-stretch: normal; line-height: 1.45; padding: 16px; border-radius: 3px; word-wrap: normal; background-color: rgb(247, 247, 247);"> <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0px; margin: 0px; border-radius: 3px; word-break: normal; white-space: pre; border: 0px; display: inline; max-width: initial; overflow: initial; line-height: inherit; word-wrap: normal; background: transparent;">Map args = new HashMap(); args.put("x-queue-master-locator", "min-masters"); channel.queueDeclare("microservice.queue.1", false, false, false, args); </code></pre> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">Similarly, in <strong style="box-sizing:border-box">Python</strong>, using <a href="https://pika.readthedocs.org/en/latest/index.html#" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">Pika</a> AMQP library, you&rsquo;d carry out something similar to the following;</p> <pre style="box-sizing: border-box; overflow: auto; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin-top: 0px; margin-bottom: 16px; font-stretch: normal; line-height: 1.45; padding: 16px; border-radius: 3px; word-wrap: normal; background-color: rgb(247, 247, 247);"> <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; padding: 0px; margin: 0px; border-radius: 3px; word-break: normal; white-space: pre; border: 0px; display: inline; max-width: initial; overflow: initial; line-height: inherit; word-wrap: normal; background: transparent;">queue_name = 'microservice.queue.1' args = {"x-queue-master-locator": "min-masters"} channel.queue_declare(queue = queue_name, durable = True, arguments = args )</code></pre> <p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">You can find complete versions of these snippets <a href="https://github.com/Ayanda-D/rabbitmq-examples" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">here</a>. These are simplified primers which you can build upon. If you have a requirement to implement something more complex, and need some assistance, please don&rsquo;t hesitate to get in <a href="https://gist.github.com/Ayanda-D/rabbitmqsupport@erlang-solutions.com" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">touch</a>!</p> <p style="box-sizing: border-box; margin-top: 0px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 0px !important;">This feature was my first contribution to RabbitMQ as part of the <a href="https://gist.github.com/Ayanda-D/rabbitmqsupport@erlang-solutions.com" style="box-sizing: border-box; color: rgb(64, 120, 192); text-decoration: none; background-color: transparent;">Erlang Solutions RabbitMQ Support team</a>. Thank you everybody!</p> <p>&nbsp;</p> <hr> <p>&nbsp;</p> <p><strong>Erlang Solutions is the world leader in <a href="http://www2.erlang-solutions.com/l/23452/2017-04-24/4kcfsn">RabbitMQ</a> consultancy, development, and support.</strong> </p> <p><strong>We can help you design, set up, operate and optimise a system with RabbitMQ. Got a system with more than the typical requirements? We also offer RabbitMQ customisation and bespoke support.</strong> </p> <p><strong><a href="http://www2.erlang-solutions.com/l/23452/2017-04-24/4kcfsn">Learn more about our work with RabbitMQ &gt;</a></strong></p>

Permalink

RabbitMQ monitoring: WombatOAM and the RabbitMQ Management plugin

<p><strong>Erlang Solutions offers world-leading RabbitMQ consultancy, support &amp; tuning solutions. <a href="http://www2.erlang-solutions.com/l/23452/2017-04-24/4kcfsn">Learn more &gt;</a></strong></p> <h2>1. Introduction</h2> <p>&nbsp;</p> <p>If you&rsquo;re a RabbitMQ user, then you must be accustomed to monitoring and keeping track of the status of your Rabbit installation by use of the native&nbsp;<a href="https://www.rabbitmq.com/management.html">RabbitMQ Management plugin</a>, or, alternatively, using third party monitoring tools such as&nbsp;<a href="https://sensuapp.org/">Sensu</a>, which internally make use of the RabbitMQ Management&nbsp;<a href="https://cdn.rawgit.com/rabbitmq/rabbitmq-management/rabbitmq_v3_6_3/priv/www/api/index.html">API</a>&nbsp;for metrics acquisition, to present them on custom UIs. Regardless of the user&rsquo;s preferred tool, a common aspect which cuts across most of these off-the-shelf tools&nbsp;is full dependency on the RabbitMQ Management plugin. In other words, for you to monitor and manage your RabbitMQ installation via a web interface, the RabbitMQ Management plugin has to be&nbsp;<strong>enabled at all times</strong>&nbsp;on the RabbitMQ nodes. The downside of this approach lies mainly on the overhead the RabbitMQ Management plugin introduces per each node it is enabled on. The following image depicts the accompanying and required applications introduced on a node when the RabbitMQ Management plugin is enabled;</p> <p><img alt="" src="https://esl-website-production.s3.amazonaws.com/uploads/content_WR_ApplicationOverhead_A.png" /></p> <p>A total of&nbsp;<strong>13</strong>&nbsp;additional applications are required by the RabbitMQ Management Plugin, which aren’t related to, or required to run any of the AMQP operations. Internally, the RabbitMQ Management Plugin creates multiple Erlang ETS tables, which are RAM based, in order to store, aggregate and compute statistics and various RabbitMQ specific node metrics. Unless the hardware has been dimensioned to take this into account, it can place a huge demand on the node’s memory, and could potentially contribute to a number of unknown side effects when traffic load patterns vary from peak to peak. Ironically, a common recommendation in the RabbitMQ community for troubleshooting is to disable and re-enable the RabbitMQ Management Plugin!</p> <p>In an ideal world, RabbitMQ nodes should be dedicated to delivery of the AMQP protocol (queueing and message interchange logic between connected clients). All potential burdensome operations like UI monitoring and management should ideally be taken care of on a completely independant node;&nbsp;deployable on a separate physical machine from the RabbitMQ nodes, for all OAM functions. This is how&nbsp;telecoms systems have addressed monitoring and operations for decades. This inflexibility of the RabbitMQ Management plugin and other monitoring tools dependant on its API&nbsp;brings to light the main strengths and advantages of using our tool WombatOAM [4]. Illustrated below, is the application overhead <a href="https://www.erlang-solutions.com/products/wombat-oam.html">WombatOAM</a> introduces on a RabbitMQ node;</p> <p><img alt="" src="https://esl-website-production.s3.amazonaws.com/uploads/content_WR_ApplicationOverhead_B.png" /></p> <p>So what is different about the WombatOAM approach of monitoring RabbitMQ? Firstly, only&nbsp;<strong>1</strong>&nbsp;additional application is introduced, which is the&nbsp;<code>wombat_plugin</code>, unlike the 13 additional applications which are introduced by the RabbitMQ Management plugin. The reason behind this is the fact that WombatOAM only requires a single and very lightweight application, which is the&nbsp;<code>wombat_plugin</code>, on the node it&rsquo;s monitoring, to relay all metrics and events to a separate and independent node, responsible for carrying out all heavy computations and UI related operations. This implies a much less, if not negligible, application overhead on the RabbitMQ node in comparison to that introduced by RabbitMQ Management plugin, which carries out all of its computations and operations on the RabbitMQ node, itself.</p> <p>WombatOAM thus fully leverages distributed Erlang by carrying out its major operations and maintenance functions in a loosely coupled manner, on an independent and separate node. This grants much more freedom to the RabbitMQ nodes to predominantly be focused on carrying out AMQP functions,&nbsp;<strong>only</strong>, with very little or no chance at all, of experiencing any problems relating to UI operations and maintenance functions.</p> <p><strong>NOTE</strong>: The RabbitMQ Management Plugin does attempt to reduce the amount of overhead when used in a cluster setup. Not all nodes in a cluster need the full RabbitMQ Management plugin enabled, but just one. The rest of the nodes need only the&nbsp;<a href="https://github.com/rabbitmq/rabbitmq-management-agent">RabbitMQ Management Agent</a>&nbsp;plugin enabled, which is a single lightweight application with no additional dependencies. This implies that any problems arising from the heavy resource footprint of the RabbitMQ Management Plugin are likely to be only experienced on&nbsp;<strong>1</strong>&nbsp;out of&nbsp;<strong>N</strong>&nbsp;cluster nodes.</p> <p>The following results were captured from a benchmark test executed against a RabbitMQ node; first as a standalone node, next, being monitored by WombatOAM, and finally with the RabbitMQ Management plugin enabled (without WombatOAM). This test was carried out on a 2GHz Intel i7 8 core MacBook Pro, with 16G memory.</p> <table> <thead> <tr> <th>&nbsp;</th> <th>Standalone Node</th> <th>WombatOAM enabled</th> <th>RabbitMQ Management plugin enabled</th> </tr> </thead> <tbody> <tr> <td>Applications Count</td> <td>9</td> <td>10</td> <td>22</td> </tr> <tr> <td>Erlang Process Count</td> <td>257</td> <td>287</td> <td>344</td> </tr> <tr> <td>Memory (MB)</td> <td>145.2</td> <td>152</td> <td>186.8</td> </tr> <tr> <td>Message Rate (msgs/sec)</td> <td>18692</td> <td>18518</td> <td>18321</td> </tr> </tbody> </table> <p>&nbsp;</p> <p>It&rsquo;s clear from the captured results, that the RabbitMQ node experiences less strain, and thus performs much better (higher message rate, for example), when being monitored by WombatOAM, as compared to monitoring using the RabbitMQ Management plugin. Even without the application of traffic, the total process count for example on the RabbitMQ node goes up from 144 to 232 when using the RabbitMQ Management Plugin, as compared to only 174 when using WombatOAM, which is much less overhead. Latency was also captured to be averaging at approximately 237920 microseconds using WombatOAM, as compared to an average of 254705 microseconds with the RabbitMQ Management Plugin enabled. These improvements in message rate and latency become more relevant on an aggregate scale, on cluster wide setups, playing a crucial role on the overall system’s performance.</p> <p>In addition, this same test may also be carried out with WombatOAM being used to monitor a node with the RabbitMQ Management Plugin already enabled, in order to gather and analyse hundreds more related metrics, alarms and notifications from the virtual machine that RabbitMQ runs on. These metrics, such as process memory, atom memory, Mnesia and TCP specific metrics, and much more, are are not available from the RabbitMQ Management Plugin.</p> <p><strong>NOTE</strong>: These tests could also yield varied results depending on the hardware platform on which the test RabbitMQ nodes are installed on.</p> <h2>2. Installation</h2> <p>&nbsp;</p> <p><span style="color:#000000"><strong>You can ask for a free 45 day product trial of the WombatOAM&nbsp;package</strong>&nbsp;</span>by contacting us&nbsp;at&nbsp;<a href="mailto:general@erlang-solutions.com">general@erlang-solutions.com</a>&nbsp;or by filling in the &lsquo;Request a Demo&rsquo; form on the <a href="https://www.erlang-solutions.com/products/wombat-oam.html">product page</a>. &nbsp;The package&nbsp;will be accompanied with the WombatOAM documentation, and easy to follow installation procedures to get it up and running in a very short space of time.</p> <p><img alt="" src="https://esl-website-production.s3.amazonaws.com/uploads/content_WR_WombatOAM.png" style="width:200px" />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;<img alt="" src="https://esl-website-production.s3.amazonaws.com/uploads/content_WR_RabbitMQ.png" style="width:300px" /></p> <h2>3. Features</h2> <p>&nbsp;</p> <p>Next up, I&rsquo;ll discuss following WombatOAM features, in the context of RabbitMQ;</p> <ul class="dotted"> <li><em>Metrics</em></li> <li><em>Notifications, Events &amp; Alarms</em></li> <li><em>Configuration</em></li> <li><em>Control</em></li> <li><em>Explore</em></li> </ul> <p>WombatOAM has a vast number of other features and capabilities, explained and illustrated in greater detail in the&nbsp;WombatOAM documentation.</p> <h3>&nbsp;</h3> <h3>3.1. Metrics</h3> <p>One of the primary purposes of the WombatOAM RabbitMQ plugin is metrics acquisition and processing from the RabbitMQ nodes it’s connected to, and presenting these metrics on the WombatOAM UI. WombatOAM provides a number of RabbitMQ specific metrics such as publish rate, deliver rate, messages ready, messages unacknowledged, connection related metrics, and so forth. A total of&nbsp;<strong>49</strong>&nbsp;RabbitMQ specific metrics are currently provided by WombatOAM, and additional metrics can easily be developed if required by WombatOAM users.</p> <ul class="dotted"> <li>Live metrics</li> </ul> <p><img alt="" src="https://esl-website-production.s3.amazonaws.com/uploads/content_WR_Metrics_3.png" /></p> <ul class="dotted"> <li>Collected metrics</li> </ul> <p><img alt="" src="https://esl-website-production.s3.amazonaws.com/uploads/content_WR_Collected_Metrics_1.png" /></p> <ul class="dotted"> <li>Gauge metrics</li> </ul> <p><img alt="" src="https://esl-website-production.s3.amazonaws.com/uploads/content_WR_Gauge_Metrics_0.png" /></p> <p>In addition, WombatOAM also provides an extremely useful feature of defining upper and lower bound metric thresholds, from which alarms will be raised when these metric limits are exceeded. This means support engineers are never caught by surprise when certain event limits are reached.</p> <h3>3.2. Notifications, Events &amp; Alarms</h3> <p>RabbitMQ users have to manually dig up the log files, copy them across from remote servers, analyse them line by line, in order to get an indication of the events and alarms being raised in their nodes. This can be a tedious and time consuming exercise to carry out, especially when problem resolution is required in a very short space of time on your production environment. However, with WombatOAM installed, events and alarm notifications are presented in a very user friendly manner, on a dashboard, under the&nbsp;<strong>Notifications</strong>&nbsp;tab, with minimal effort from the user, as compared to manually locating and reading up log files.</p> <p><img alt="" src="https://esl-website-production.s3.amazonaws.com/uploads/content_WR_Notifications_1.png" /></p> <p>WombatOAM alarm and event notifications go on to provide even more information than what the native RabbitMQ logs would expose, for example code version discrepancies. Without WombatOAM, many problems, only detectable by using WombatOAM, would only be discovered during post-problem diagnosis, when the RabbitMQ nodes have already suffered from the side effects, and in some cases, may even have gone down. This is the definition of preemptive support, which is a shortcoming of many monitoring systems, except WombatOAM, which excels in this area. The last chapter, 16, of the text,&nbsp;<a href="http://shop.oreilly.com/product/0636920024149.do"><em>Designing for Scalability with Erlang OTP</em></a>&nbsp;explores preemptive support in detail, and why it’s extremely crucial in the operations and maintenance activities of any system, (whether or not Erlang based), such as RabbitMQ. Quoting part of the text:</p> <blockquote> <p>Preemptive support automation gathers data in the form of metrics, events, alarms, and logs for a particular application; analyzes the data; and uses the results to predict service disruptions before they occur. An example is noticing an increase in memory usage, which predicts that the system might run out of memory in the near future unless appropriate corrective actions are taken.</p> </blockquote> <h3>&nbsp;</h3> <h3>3.3. Configuration</h3> <p>Another powerful feature which WombatOAM provides is&nbsp;<strong>configuration management</strong>&nbsp;from the user interface. RabbitMQ is heavily&nbsp;<a href="https://www.rabbitmq.com/configure.html">configurable</a>&nbsp;from the&nbsp;<a href="https://www.rabbitmq.com/configure.html#configuration-file">rabbitmq.config</a>&nbsp;file, and from the application resource files of all its associated plugins. WombatOAM gives full visibility of all application environment variables, which, in RabbitMQ are all the&nbsp;<a href="https://www.rabbitmq.com/configure.html#config-items">configuration parameters</a>&nbsp;defined in&nbsp;<a href="https://www.rabbitmq.com/configure.html#configuration-file">rabbitmq.config</a>&nbsp;file, as illustrated below;</p> <ul class="dotted"> <li>Select the&nbsp;<strong>rabbit</strong>&nbsp;application and start request;</li> </ul> <p><img alt="" src="https://esl-website-production.s3.amazonaws.com/uploads/content_WR_Config_Parameters_A.png" /></p> <ul class="dotted"> <li>Configuration parameters are displayed</li> </ul> <p><img alt="" src="https://esl-website-production.s3.amazonaws.com/uploads/content_WR_Config_Parameters_B.png" /></p> <p>WombatOAM also goes the extra mile, to allow users to make in-memory configuration changes during system runtime.</p> <p>This means that with WombatOAM installed, if you’re, for example, using the RabbitMQ queue master location policies[5], you could update the location policy via the WombatOAM as illustrated below. This will be in effect for the queue master location configuration, in memory.</p> <p><img alt="" src="https://esl-website-production.s3.amazonaws.com/uploads/content_WR_Configuration_A.png" /></p> <p><strong>NOTE</strong>: It must be clarified that&nbsp;<strong>not all</strong>&nbsp;configuration changes will be applied. Some settings are cached, for example, channels retain&nbsp;<code>channel_operation_timeout</code>&nbsp;configuration in their cache. So the new value will not be applied to already created channels, only to new channels created thereafter.</p> <p>In addition, WombatOAM’s&nbsp;<em>configuration management</em>&nbsp;feature allows for global, cluster wide configuration settings. This is a very useful feature which helps avoid inconsistencies between configuration changes carried out on a single node, as compared to configuration settings on the other cluster nodes.</p> <ul class="dotted"> <li>Global configuration changes</li> </ul> <p><img alt="" src="https://esl-website-production.s3.amazonaws.com/uploads/content_WR_Configuration_B.png" /></p> <h3>3.4. Control</h3> <p>WombatOAM grants a high degree of control over the RabbitMQ nodes it’s monitoring by providing an interface to execute Erlang native functions. This is found under the&nbsp;<strong>Services</strong>&nbsp;tab of the WomabatOAM UI. This means that AMQP equivalent operations such as queue and exchange declarations, or, native RabbitMQ operations like mirror policy definitions, and control operations (like listing of queues, exchanges, connections, and so forth) may be executed from the WombatOAM UI.</p> <p><strong>NOTE</strong>: This feature will, in the near future, be replaced by a more user friendly and safer means of executing control commands. This should also help avoid the continuous and repeated re-entry of such commands, if executed in a frequent manner.</p> <ul class="dotted"> <li>Below is an example of a control function being executed to acquire and display all exchanges</li> </ul> <p><img alt="" src="https://esl-website-production.s3.amazonaws.com/uploads/content_WR_Control_A.png" /></p> <ul class="dotted"> <li>Result being displayed</li> </ul> <p><img alt="" src="https://esl-website-production.s3.amazonaws.com/uploads/content_WR_Control_B.png" /></p> <p>WombatOAM also logs these executed control commands as part of its&nbsp;<strong>Notifications</strong>, making it possible to carry out minor and/or major audit trail operations, to investigate which commands were executed, and when. This is very useful when, for example, troubleshooting a problem’s cause resulting from human intervention on the RabbitMQ nodes.</p> <p>Other useful RabbitMQ control functions you could execute from the WombatOAM interface are summarized as follows (you can copy, modify and use these as you like):</p> <table> <thead> <tr> <th>Command</th> <th>Function</th> </tr> </thead> <tbody> <tr> <td>Declare a queue</td> <td>rabbit_amqqueue:declare(rabbit_misc:r(&lt;&lt;&ldquo;/&rdquo;&gt;&gt;, queue, _Queue = &lt;&lt;&ldquo;test.queue&rdquo; &gt;&gt;), false, false, [], none).</td> </tr> <tr> <td>Declare 20 queue</td> <td>begin Queues = 20, L=[{<em>,</em>} = rabbit_amqqueue:declare(rabbit_misc:r(&lt;&lt;&ldquo;/&rdquo;&gt;&gt;, queue, list_to_binary(&ldquo;test.queue.&rdquo;++integer_to_list(N))), false, false, [], none)</td> </tr> <tr> <td>List queues</td> <td>rabbit_amqqueue:list().</td> </tr> <tr> <td>Declare exchange</td> <td>rabbit_exchange:declare({resource, _VHost = &lt;&lt;&ldquo;/&rdquo;&gt;&gt;, exchange, _XName = &lt;&lt;&ldquo;test.exchange&rdquo;&gt;&gt;}, _Type = topic, _Durable = true, _AutoDelete = false, _Internal = false, _Args = []).</td> </tr> <tr> <td>List exchanges</td> <td>rabbit_exchange:list().</td> </tr> <tr> <td>Set mirror policy</td> <td>rabbit_policy:set(_VHost = &lt;&lt;&ldquo;/&rdquo;&gt;&gt;, _PolicyName = &lt;&lt;&ldquo;ha-2-policy&rdquo;&gt;&gt;, _Pattern = &lt;&lt;&ldquo;.*&rdquo;&gt;&gt;, _Definition = [{&lt;&lt;&ldquo;ha-mode&rdquo;&gt;&gt;, &lt;&lt;&ldquo;exactly&rdquo;&gt;&gt;}, {&lt;&lt;&ldquo;ha-params&rdquo;&gt;&gt;, 2}], _Priority = 1, _ApplyTo = &lt;&lt;&ldquo;queues&rdquo;&gt;&gt;).</td> </tr> <tr> <td>Clear policy</td> <td>rabbit_policy:delete(_VHost = &lt;&lt;&ldquo;/&rdquo;&gt;&gt;, _PolicyName = &lt;&lt;&ldquo;ha-2-policy&rdquo;&gt;&gt;).</td> </tr> <tr> <td>Create vhost</td> <td>rabbit_vhost:add(_VHost = &lt;&lt;&ldquo;test.vhost&rdquo;&gt;&gt;).</td> </tr> <tr> <td>Authenticate user</td> <td>rabbit_access_control:check_user_pass_login( &lt;&lt;&ldquo;username&rdquo;&gt;&gt;, &lt;&lt;&ldquo;password&rdquo;&gt;&gt; ).</td> </tr> <tr> <td>Purge queue</td> <td>rabbit_control_main:purge_queue({resource, _VHost = &lt;&lt;&ldquo;/&rdquo;&gt;&gt;, queue, _QName = &lt;&lt;&ldquo;test.queue&rdquo;&gt;&gt;}).</td> </tr> <tr> <td>RabbitMQ Status</td> <td>rabbit:status().</td> </tr> </tbody> </table> <p>&nbsp;</p> <p>Additionally, WombatOAM also provides control operations which allow users to;</p> <ul class="dotted"> <li>Forcefully initiate garbage collection</li> <li>Kill specific processes, using the&nbsp;<em>process identifier</em>&nbsp;(PID) or the&nbsp;<em>registered name</em>&nbsp;as reference</li> <li>Soft purge modules on managed nodes, ensuring no old module versions are held in memory, unless being used by a process.</li> </ul> <p>The image below illustrates these control categories;</p> <p><img alt="" src="https://esl-website-production.s3.amazonaws.com/uploads/content_WR_Control_C.png" /></p> <h3>3.5. Explore</h3> <p>WombatOAM has a powerful feature, extremely useful for RabbitMQ support and development engineers, which allows them to explore node processes and ETS tables, (including the&nbsp;<code>rabbit_queue</code>&nbsp;table, which consists of all alive queues on the nodes). The image below depicts the categories which may be explored;</p> <ul class="dotted"> <li>Different Explore categories</li> </ul> <p><img alt="" src="https://esl-website-production.s3.amazonaws.com/uploads/content_WR_Explore_A.png" /></p> <p>For example, inspecting the process state of an essential process like the&nbsp;<code>vm_memory_monitor</code>&nbsp;is only a matter of navigating to the&nbsp;<code>Explore</code>&nbsp;tab -&gt;&nbsp;<code>Process state</code>&nbsp;category, specifying the process registered name, in this case the<code>vm_memory_monitor</code>, or PID (Process Identifier), and executing the request.</p> <ul class="dotted"> <li>Specifying&nbsp;<code>vm_memory_monitor</code>&nbsp;registered name, and executing request</li> </ul> <p><img alt="" src="https://esl-website-production.s3.amazonaws.com/uploads/content_WR_Explore_B.png" /></p> <ul class="dotted"> <li>Process state of&nbsp;<code>vm_memory_monitor</code>&nbsp;displayed</li> </ul> <p><img alt="" src="https://esl-website-production.s3.amazonaws.com/uploads/content_WR_Explore_C.png" /></p> <h2>4. When to use WombatOAM</h2> <p>&nbsp;</p> <p>A question often raised regarding WombatOAM and the RabbitMQ Management plugin is;</p> <blockquote> <p><em>“&hellip;.which is the right tool to use, and on which occasions?”</em></p> </blockquote> <p>Both these tools may be viewed as complementary to each other. Despite WombatOAM’s superior capabilities to provide more than just RabbitMQ metrics, but also a vast number of events, alarms, Erlang VM node specifics, and so forth, the RabbitMQ Management plugin does also provide useful features, like, for example, manual publishing/receiving of messages from the UI, plugin specific operations like creation of links and upstreams for the Federation plugin, and so forth.</p> <p>Hence, bearing this in mind, recommendation and best practice is to use both tools in the following manner;</p> <ul class="dotted"> <li> <p><strong>WombatOAM</strong>, at&nbsp;<strong>ALL</strong>&nbsp;times, to monitor and keep track of the wellness of the RabbitMQ installation. WombatOAM will introduce very minimal overhead on the RabbitMQ node(s) which it is monitoring, thus chances of end-to-end service interruptions from resource hungry UI operations would be as good as neglible.</p> </li> <li> <p><strong>RabbitMQ Management Plugin</strong>, to be enabled only when certain functions and operations are required. For example, specific plugin operations (e.g. Federation plugin configuration), or easy definition of mirroring policies. All these are once off operations, for which the RabbitMQ Management Plugin need not be enabled at all times for.</p> </li> </ul> <p>&nbsp;</p> <p>With WombatOAM continually enabled, and the RabbitMQ Management plugin enabled on specific occasions only, when required, common RabbitMQ problems such as nodes raising VM High Watermark memory alarms due to excessive memory usage beyond that permissible, would hardly be ever experienced, since these are attributed by UI operations of a high memory footprint. Unless of course, the root cause is something else not directly related to, nor attributed by, memory hungry UI computations and operations, and also, the basis is not on any hardware limitations on which RabbitMQ is installed on. This is just one major example, amongst an abitrary number of many other problems, which would be alleviated by making use of WombatOAM in this manner.</p> <h2>5. Beyond RabbitMQ</h2> <p>&nbsp;</p> <p>Beyond the specific metrics provided by the RabbitMQ plugin, WombatOAM also by default, provides Erlang VM related metrics. This means that WombatOAM gathers more metrics and node specific information from the RabbitMQ nodes like no other monitoring tool available. Additional metrics gathered from the RabbitMQ node are illustrated in the image below;</p> <p><img alt="" src="https://esl-website-production.s3.amazonaws.com/uploads/content_WR_Beyond_A.png" /></p> <p>Each of these metric categories are both rich in the information they provide and crucial in monitoring any RabbitMQ installation, and in essence, any Erlang based system. For example,&nbsp;<strong>Memory</strong>&nbsp;and&nbsp;<strong>Mnesia System metrics</strong>&nbsp;would be critical for any RabbitMQ installation, as all&nbsp;<em>metadata</em>&nbsp;(queue, exchanges, bindings, and so forth) is stored in Mnesia, along with the fact that RabbitMQ nodes are classified as either&nbsp;<strong>DISC</strong>&nbsp;or&nbsp;<strong>RAM</strong>&nbsp;nodes, based on type of Mnesia tables they’re configured to use. Same applies to all other metrics WombatOAM provides; they all play a crucial role for any RabbitMQ (and Erlang system) installation.</p> <p>Adding on,&nbsp;<strong>I/O</strong>&nbsp;metrics are also provided, which can also reveal vital information regarding the number of permissible client connections when specifications such as scalability come into consideration. And off course,&nbsp;<strong>Error Log Notifications</strong>&nbsp;can indicate the rate at which errors/exceptions and warnings are being reported and logged to the SASL logs and general logs, respectively, by the RabbitMQ nodes. The magnitudes of these metrics in particular, can be an easy and direct indicator that the node is experiencing problems which have, or have not yet been to be detected by the user.</p> <h2>6. Conclusion</h2> <p>&nbsp;</p> <p>Rounding up the discussion, its clear that WombatOAM proves an essential and efficient tool for monitoring any RabbitMQ installation. As already pointed out in&nbsp;Section 4, recommendation is to maintain an instance of WombatOAM continually, monitoring your RabbitMQ installation, and have the RabbitMQ Management Plugin on the other hand, due its resource hungry nature, only enabled for certain periods of time, when required for some of its specific control features, to use once off, and disabling it thereafter. This guarantees that during the majority periods of your RabbitMQ installation&rsquo;s uptime, your nodes are dedicated to AMQP operations&nbsp;<strong>only</strong>, and avoid servere UI related operational overheads induced by the RabbitMQ Management Plugin. Per&nbsp;<strong>24hour</strong>&nbsp;monitoring cycle, you could for example, employ the RabbitMQ Management plugin for no more than&nbsp;<strong>2 hours</strong>&nbsp;only, when required, with WombatOAM monitoring continually throughout the entire cycle.</p> <p><strong>If you would like to&nbsp;receive a copy of this post&nbsp;in the form of a PDF Whitepaper, fill in your details <a href="https://www.surveymonkey.co.uk/r/WombatOAMmonitorRabbirMQ">here</a>.&nbsp;</strong></p> <h2>References</h2> <p>[1]&nbsp;<a href="https://www.rabbitmq.com/configure.html">https://www.rabbitmq.com/configure.html</a></p> <p>[2]&nbsp;<a href="https://www.rabbitmq.com/management.html">https://www.rabbitmq.com/management.html</a></p> <p>[3]&nbsp;<a href="http://shop.oreilly.com/product/0636920024149.do">Designing for Scalability with Erlang OTP</a></p> <p>[4]&nbsp;<a href="https://www.erlang-solutions.com/products/wombat-oam.html">https://www.erlang-solutions.com/products/wombat-oam.html</a></p> <p>[5]&nbsp;<a href="https://www.erlang-solutions.com/blog/take-control-of-your-rabbitmq-queues.html">https://www.erlang-solutions.com/blog/take-control-of-your-rabbitmq-queues.html</a></p> <p>&nbsp;</p> <hr> <p>&nbsp;</p> <p><strong>Erlang Solutions is the world leader in <a href="http://www2.erlang-solutions.com/l/23452/2017-04-24/4kcfsn">RabbitMQ</a> consultancy, development, and support.</strong> </p> <p><strong>We can help you design, set up, operate and optimise a system with RabbitMQ. Got a system with more than the typical requirements? We also offer RabbitMQ customisation and bespoke support.</strong> </p> <p><strong><a href="http://www2.erlang-solutions.com/l/23452/2017-04-24/4kcfsn">Learn more about our work with RabbitMQ &gt;</a></strong></p>

Permalink

Copyright © 2016, Planet Erlang. No rights reserved.
Planet Erlang is maintained by Proctor.