Planet = erlang.

José Valim | ElixirConf EU 2017 Keynote

Permalink

Chris McCord | Keynote: Phoenix - Gauging Progress

Permalink

Tetiana Dushenkivska & Keith Salisbury | Keynote: The Age of Elixir

Permalink

Erlang/OTP 20.0 is released

img src=http://www.erlang.org/upload/news/

Erlang/OTP 20.0 is a new major release with new features, quite a few (characteristics) improvements, as well as a few incompatibilities.

There are only minor changes compared to the second release candidate, some of them listed below:

• ERTS:
  • erlang:term_to_binary/1 changed the encoding of all atoms from ATOM_EXT to ATOM_UTF8_EXT and SMALL_ATOM_UTF8_EXT. This is now changed so that only atoms actually containing unicode characters are encoded with the UTF8 tags while other atoms are encoded ATOM_EXT just as before.

Here are some of the most important news in OTP 20:

Potential Incompatibilities

• ERTS:

  • The non SMP Erlang VM is deprecated and not built by default
  • Remove deprecated erlang:hash/2
  • erlang:statistics/1 with scheduler_wall_time now also includes info about dirty CPU schedulers.
  • The new purge strategy introduced in OTP 19.1 is mandatory and slightly incompatible for processes holding funs
    see erlang:check_process_code/3.
  • The NIF library reload is not supported anymore.
  • Atoms can now contain arbitrary unicode characters which means that the DFLAG_UTF8_ATOMS capability in the distribution protocol must be supported if an OTP 20 node should accept the connection with another node or library. Third party libraries which uses the distribution protocol need to be updated with this.

• Asn1: Deprecated module and functions removed (asn1rt, asn1ct:encode/3 and decode/3)

• Ssh: client only option in a call to start a daemon will now fail

Highlights

Erts:

• Dirty schedulers enabled and supported on VM with SMP support.
• support for “dirty” BIFs and “dirty” GC.
• erlang:garbage_collect/2 for control of minor or major GC
• Erlang literals are no longer copied when sending messages.
• Improved performance for large ETS tables, >256 entries (except ordered_set)
• erlang:system_info/1 atom_count and atom_limit
• Reduced memory pressure by converting sub-binaries to heap-binaries during GC
• enif_select, map an external event to message
• Improvements of timers internally in the VM resulting in reduced memory consumption and more efficient administration for timers

Compiler:

• Code generation for complicated guards is improved.
• Warnings for repeated identical map keys. #{'a'=>1, 'b'=>2, 'a'=>3} will warn for the repeated key a.
• By default there is now a warning when export_all is used. Can be disabled
• Pattern matching for maps is optimized
• New option deterministic to omit path to source + options info the BEAM file.
• Atoms may now contain arbitrary unicode characters.
• compile:file/2 has an option to include extra chunks in the BEAM file.

Misc other applications

• Significantly updated string module with unicode support and many new functions
• crypto now supports OpenSSL 1.1
• Unnamed ets tables optimized
• gen_fsm is deprecated and replaced by gen_statem
• A new event manager to handle a subset of OS signals in Erlang
• Optimized sets add_element, del_element and union
• Added rand:jump/0-1
• When a gen_server crashes, the stacktrace for the client will be printed to facilitate debugging.
• take/2 has been added to dict, orddict, and gb_trees.
• take_any/2 has been added to gb_trees
• erl_tar support for long path names and new file formats
• asn1: the new maps option changes the representation of SEQUENCE to be maps instead of records
• A TLS client will by default call public_key:pkix_verify_hostname/2 to verify the hostname
• ssl: DTLS documented in the API, experimental
• ssh: improving security, removing and adding algorithms
• New math:fmod/2

For more details see
http://erlang.org/download/otp_src_20.0.readme

The Erlang/OTP source can also be found at GitHub on the official Erlang repository,
https://github.com/erlang/otp with tag OTP-20.0

Pre built versions for Windows can be fetched here:
http://erlang.org/download/otp_win32_20.0.exe
http://erlang.org/download/otp_win64_20.0.exe

On line documentation can be browsed here:
www.erlang.org/doc/

Thanks to all contributors.

Permalink

How to build a self-sustaining development team

<p><div style="text-align:center"><a href="http://www2.erlang-solutions.com/l/23452/2017-06-20/4plzjl"><img src ="https://s3.amazonaws.com/uploads.hipchat.com/15025/4746287/Zhyv4D9SymUk1I6/longer%202.png"/></a></div><br/></p> <p><strong>This is an excerpt from <em><a href="http://www2.erlang-solutions.com/l/23452/2017-06-20/4plzjl">The route to the successful adoption of non-mainstream programming languages</a></em> by Francesco Cesarini and Mike Williams.</strong></p> <p>So you’ve already built the prototype, failed forwards, and secured approval to move on with engineering the final product. It’s vital that you keep the team that built the prototype and use them to mentor the new recruits. In doing so, you need to make sure your team becomes self-sustainable. </p> <p>It is best if the mentor can sit next to the new recruits, communication needs to be quick and efficient from both sides. As members move to new positions or get assigned new POCs, you need to be able to replace them, be capable of mentoring the new joiners, bringing them up to speed. </p> <p>Our philosophy is to work with internal teams to grow the team&rsquo;s skills, knowledge and capabilities. Create an environment to grow and foster the mindset for success. </p> <p>If you are a small company, a minimal team should consist of <strong>4-5 people</strong>. </p> <p>If you are a large company, <strong>10-12 developers and up</strong>. </p> <p>Not everyone needs to be using or developing in Erlang, but they should be passionate about it, be able to jump in and be productive at short notice, and be capable of rotating other team members out.</p> <h1>Transferable skills</h1> <p>It is always a mistake to advertise for programmers with experience in the language you are adopting; you want good and experienced open-minded software developers who understand different programming paradigms and approaches. They do not need to know the programming language, but they should be capable of understanding what is meant by <strong>using the right tool for the job</strong>. </p> <p><strong><em>Erlang, you can learn easily. Good software practices, less so.</em></strong> </p> <p>If you acquire a company, you will get a pool of great developers who already have the domain knowledge. Back in 2004, a company in Silicon Valley was advertising for Erlang developers with at least a decade of Erlang experience. They had adopted Erlang as a result of an acquisition at a time when there were maybe 20 developers in the world who qualified (including the authors of this blog). </p> <p>Ironically, that company used to employ another ten, but had just shut down their Swedish operations, making them redundant. Don’t get too hung up on location. Focus on talent and productivity. </p> <h1>A team that grows with you</h1> <p>As your software project progresses, the focus of your work will change, at the start your focus will be on developers with good architectural and programming skills; later on configuration management, verification and testing; then deployment, operations, maintenance, and documentation. </p> <p><strong><em>Build a team of developers who can embrace the skills needed across the whole of the development life cycle.</em></strong> </p> <p>You should avoid different people for each category, rather you should have people who themselves can embrace the skills needed for the whole of the development life cycle. Handing over knowledge from person to person is error prone and costly, when a person works across several areas this can be avoided.</p> <h1>Cut out the bad habits</h1> <p>Learning Erlang/OTP is easy, as it is a very compact language and framework. The difficult part is unlearning other programming language models and getting rid of the bad habits they bring with them. </p> <p>This is where mentorship and code reviews become an essential part of the process. Review all code, not only code in the application itself but test cases, simulators and tools. </p> <p><strong><em>Code reviews should be led by experienced developers focusing not only on quality, style, and documentation, but also on educating developers on the system as a whole.</em></strong></p> <p>New recruits often get up to speed fastest when making changes to existing code. Be cautious and review new code written by new recruits carefully.</p> <p>Ericsson did a study in the late 90s looking at the productivity of graduates and experienced developers in large projects after they had attended Erlang and OTP training. They concluded that a graduate straight out of university became productive after about a month, as they could easily pick up new programming models and handle change. </p> <p>It took an experienced software developer who attended the same courses three months. They picked up Erlang easily, but had to spend more time unlearning their previous programming paradigms and embrace the Erlang way. Code reviews and mentorship in this phase were critical. When we say productive, we obviously mean productive in their respective levels of seniority.</p> <h1>Invest in training</h1> <p>When building your team, do not underestimate the value of training and study circles. Where there is significant change being effected within an organisation, there must also be adequate training and support provided to the those impacted, with an effective feedback path to ensure that issues with change do not turn into objections and resistance. </p> <p><strong><em>We often see companies give their developers a book hoping they will become productive. Why not? They are smart, well paid, and as a result, should be capable of picking up a new language on their own.</em></strong> </p> <p>It might work for polyglots who have used other functional and concurrent programming languages, but never, ever, take it for granted. If you want to save money and get your developers productive quickly, invest in a proper training and mentorship, especially if you are ramping up a team of ten or more developers. It will allow them to quickly understand and embrace the new programming paradigm and reduce beginner mistakes. </p> <p>If you go down the book route, don’t be surprised if the project runs into trouble. By being penny wise and pound foolish, you’ll scare away the Erlang/OTP experts, as they’ll burn out spending all their time refactoring and debugging bad code with no time left over to mentor those who really need it.</p> <p>And remember, stakeholders outside of your core development team will need training to sustain the excitement and investment in the project. A tactic that has worked for us in the past is “boot camp” training; we’ve seen it create the right mindset and understanding of “change tools” within change leadership teams. </p> <h1>Industry subject matter expertise</h1> <p>Erlang is just a programming language. That’s the easy part. Learning the application domain is probably the hardest task of any software project. This is where the vertical subject matter expert needs to come to the rescue. </p> <p><strong><em>Learning a programming language is the easy part, never underestimate vertical subject-matter expertise.</em></strong> </p> <p>Accept the fact that you will have to train your new recruits in Erlang and maybe some of the other technologies you use, but also accept the fact that they might not be vertical subject matter experts. They might be shipping code after a few months, but it’s going to take them much longer to learn the necessary domain expertise.</p> <h1>Plan and act with long term adoption in mind</h1> <p>If you work for a large company and inherit a team, remember that you can’t force them to use technology they don’t like. If you are going to use Erlang/OTP or any other non-mainstream language, make sure the team understands the advantages and benefits, and receive the right onboarding.</p> <p>One of the main reasons for projects moving from non-mainstream languages is their inability to make the team self-sustainable. Not having a recruitment pipeline and losing key members of the team puts pressure on the other members. </p> <p>A key aspect of these kind of projects is changing individual mindsets and transitioning from established ways of doing things. This involves creating energy and enthusiasm by showing the way from small safe incremental steps to the new operating model. Incremental is key. I have seen the pace of change play a crucial role in change adoption. If there is too much change then teams can feel overwhelmed. It can turn a “change advocate” into someone who is passive or worse, actively resisting change. This behaviour can also be triggered if change creates more work or duplication. </p> <p style="text-align:center"><img src="http://i.imgur.com/3H82oHl.png" align="centre" height="350" width="450"></p> <p><br/> The Kübler-Ross Change Curve is a powerful tool to understand where people are on the change journey. Those driving change will be further along the change curve than those who have change forced upon them. Prepare a toolbox of techniques to help trigger and stimulate movement through the change curve. For example, “town halls” as a vehicle for ensuring that objections to change are aired so that objectors to change do not get stuck in the “denial” region of the change curve. Remember: projects where the right tool for the job is used will often consist of multiple programming languages. Not everyone is expected to work on Erlang, but they should know it well enough to be able to help out when necessary. Don’t freeze out the naysayers, you may need them down the road. </p> <h1>Join the community</h1> <p>Take on summer interns and thesis students; this is an excellent way to evaluate developers. When they graduate, they might join you. Present at conferences and get your name out. Where possible, if there is a fit with what you are doing, join academic research projects. Your costs are often covered through grants. </p> <p>If you are a large corporation, provide regular courses open to everyone in the company to recruit internally. Run hackathons and study circles and host meetups to generate interest and get a contact network. Use and contribute to open source projects, and where possible, release and maintain components others might find useful. Those who see what you are doing out of curiosity and like what they see, are likely to join the team when an opening appears. </p> <p><strong>Interested in learning more about the fundamentals of building a team that will last? (You should be). <a href="http://www2.erlang-solutions.com/l/23452/2017-06-20/4plzjl">Learn how to successfully deploy new technology and teams</a> for the long term from myself and Mike Williams, a seasoned Erlang veteran who has introduced the language into numerous industries, including online gambling and betting.</strong> </p> <p><br/>† Shameless plug - I have co-authored two books on Erlang - Erlang Programming and Designing for Scalability with Erlang/OTP. </p> <hr> <p><strong><a href="http://www2.erlang-solutions.com/emailpreference">Subscribe to our newsletter</a> for the latest Erlang and Elixir news, or <a href="mailto:general@erlang-solutions.com">contact us directly</a> for more on how to introduce a new technology to your stack.</strong></p>

Permalink

Three reasons to review your online betting tech stack

<p><div style="text-align:center"><a href="http://www2.erlang-solutions.com/l/23452/2017-06-06/4mfx9b"><img src ="https://s3.amazonaws.com/uploads.hipchat.com/15025/4746287/Zhyv4D9SymUk1I6/longer%202.png"/></a></div><br/></p> <p>As CEO at Erlang Solutions, I’ve worked closely with online bookmakers for almost a decade, and have seen the digital marketplace go from SMS bets and WAP sites - remember them! - to sophisticated in-play systems and VR games. </p> <p>We’ve come a long way, but there is still reason to review your <a href="http://www2.erlang-solutions.com/l/23452/2017-06-06/4mfwls">online gambling &amp; betting</a> tech stack. First, punters expect perfection of their digital experiences. Second, strong industry competition gives your dissatisfied customers other options. Last, legacy systems can hamper your innovation and competitiveness in the marketplace. </p> <h1>1. Customers expect more</h1> <p>Today’s players increasingly consume games across multiple channels and devices, day or night. They demand ease-of-use and seamless gameplay. Add to this the fact that they expect personalisation, recommendations, and instant access to their favourite products and games, it’s clear that omnichannel, always-on service delivery will only become more prevalent in future. </p> <p>This huge leap in sophistication and digital engagement sees today’s bookmakers working to grow their revenues, whilst maintaining the high standards their customers expect in terms of product quality and service availability. </p> <h1>2. High competition</h1> <p>All of this needs to happen whilst navigating the perils of what is essentially still a maturing marketing. Just look at the high barriers to entry, and the current environment of M&amp;As which is seeing competition consolidated in online gambling and betting. To stay on top, bookmakers are innovating fast to win the precious few pockets of growth in the sector and it’s hardly a surprise that operators have developed an unblinking, shark-like focus on the end user. </p> <p>Every step of the way, user experience comes first. There is a relentless demand for it to be easier and smoother, regardless of backend challenges. This has left the industry in search of the most efficient and effective ways of directing users to their objective, whether it’s registering, playing a game, placing a bet, cashing out that bet or making a deposit. Systems must be ultra-responsive and able to engage with customers when, where, and how they want. </p> <p>Those who get this ‘right’ in comparison to their competitors enjoy higher customer engagement and retention, and hence profits. And for each operator, this journey to support customer-centricity through the tech stack is slightly different. </p> <h1>3. Legacy systems</h1> <p>Many established operators have legacy systems that are hard to scale or modernise whilst maintaining uptime and quality of customer experience. In our experience, those operators ought to take a step back, review their technology choices and evaluate which of the bewildering range of tech, old and new, can best be deployed in helping them achieve these customer-centric goals. </p> <p>Using the right tools for the job, regardless of legacy systems, has never been more critical for the sector. We work with some of the major players in online betting, helping them create and deploy next generation systems in Erlang, Elixir, and beyond.</p> <p><strong>Interested in finding the right tool for your job? (You should be). <a href="http://www2.erlang-solutions.com/l/23452/2017-06-06/4mfx9b">Learn how to successfully deploy new technology</a> for the long term from Francesco Cesarini and Mike Williams, two Erlang veterans who have introduced to language into numerous industries, including online gambling and betting.</strong> </p> <hr> <p><strong><a href="http://www2.erlang-solutions.com/emailpreference">Subscribe to our newsletter</a> for the latest Erlang and Elixir news, or <a href="mailto:general@erlang-solutions.com">contact us directly</a> for more on how to introduce a new technology to your stack.</strong></p>

Permalink

Analyze This!

I’ve found an edge case for dialyzer. Those things are so rare, I had to write about it.

Analyze This (1999)

The Dialyzer vs. Proplists Situation

Check out this module…

As you can see, all 3 functions look pretty similar. The difference lies in the Expansions list. While in good/0 tuples have lists as second elements, in bad/0 we have atoms and in wat/0 we have a mixture of both.

If we compile the module and try to use the functions, they actually work…

2> expand:good().
[expanded,expanded,too,c]
3> expand:bad().
[expanded,expanded,c]
4> expand:wat().
[expanded,expanded,too,c]
5>

But if we run dialyzer on it…

src/expand.erl
  13: The call proplists:expand(Expansions::[{'a','expanded'} | {'b','expanded'},...], ListIn::['a' | 'b' | 'c',...]) breaks the contract (Expansions, ListIn) -> ListOut when Expansions :: [{Property::property(),Expansion::[term()]}], ListIn :: [term()], ListOut :: [term()]

Dialyzer here is correctly pointing out that the spec for proplists:expand/2 requires all Expansions to be lists. And in our line 13 (that is bad/0) we’re using atoms, which is wrong. But then… why is Dialyzer not complaining about line 18 (in wat/0) where we’re using atoms as well?

What’s going on here?

I’m not 100% sure of this one (hopefully someone with more knowledge will correct me if I’m wrong), but my best guess is this one:

To check if the calls to proplists:expand/2 are valid, Dialyzer is inferring the type of Expansions in each function and then matching it against proplists:expand/2 specification (i.e. [{property(),[term()]}]).

• In good/0, Dialyzer determines that Expansions :: [{atom(), [atom(),…]}] and since atom() is a subtype of property() and [atom(),…] is a subtype of [term()] we’re all good :)
• In bad/0, Dialyzer determines that Expansions :: [{atom(), atom()}] and since atom() is a subtype of property() but atom() is not a subtype of [term()] we’re not good :(
• Finally, in wat/0, Dialyzer determines that Expansions :: [{atom(), atom()|[atom(),…]}]. Here atom() is a subtype of property() and there are perfectly valid values of type atom()|[atom(),…] that are [term()] (namely, all of the [atom(),…] ones), Dialyzer can’t tell (based on the type information alone) that we’re using one instance that doesn’t work well. So, it doesn’t warn us.

Dialyzer is an amazing tool, but it’s not almighty 🙏

But, on the other hand…

Of course, there is something else to notice here: proplists:expand/2 actually works despite of us using the wrong expansion types, right? So, I’m hereby proposing a change to proplists:expand/2 specification to:

-spec expand(Expansions, ListIn) -> ListOut when
      Expansions ::
        [{Property :: property(), Expansion :: term() | [term()]}],
      ListIn :: [term()],
      ListOut :: [term()].

Erlang & Elixir Factory Lite Buenos Aires 2017

Just like last time, I would like to finish this article with a reminder…

The first South American Erlang & Elixir conference (The Erlang & Elixir Factory Lite @ Buenos Aires) is getting closer and closer! As usual, I would like to invite you all to it.

The programme is already online and among other great speakers, you’ll be able to watch Simon Thompson talking about the power of functional programming!

So, all my south-american readers: come join us! It will be awesome!

Analyze This! was originally published in Erlang Battleground on Medium, where people are continuing the conversation by highlighting and responding to this story.

Permalink

Kicking The Tires On This Serverless Thing

I just released a video series about building a serverless application with Firebase, and I thought I would write it up in a blog series as well. I think it’s worth reading and understanding what I went through. This is Part 1 of that series.

The Punchline

Before doing this I spent a few months with AWS Lambda (which you’ll read about below); first with The Serverless Framework (which I quite like - so jealous they got that domain!) and then with ClaudiaJS which didn’t quite fit what I wanted. Finally I rolled my own thing with some shell scripts and a zip file.

I ended up with a mess that cost more money than I wanted it to. It wasn’t easy to figure out and more than once I had to rage-swim at the Y to get over the stress of the day.

I had better luck with Firebase. I had a fun time and built something interesting. At least I think it is. I had to approach the development process in a completely different way than what I was used to… but I like that kind of thing. I know others don’t. The big thing to me, however, is that I was able to build something that I would truly use. In fact I’m using parts of it now in production (more on that in a future post).

This is a long story and will cover many posts. Not sure how many but I’ll try to keep it focused. As you can clearly tell I like Firebase, a lot, and yes I encountered quite a few issues along the way but they were surmountable. I’ll get into all that, I promise.

For now let’s start at the beginning, when I first dug into the “serverless” thing.

AWS, Firebase, Webtask.io…?

Let’s start with Webtask. I know a number of people over at Auth0 (who own and run Webtask.io) and I have a ton of respect for them. Everything you’re about to read has been said to them in person, so don’t think I’m being unfair - I think they would agree with me straight away.

If you head over to the Webtask site you’ll see a simple page with a headline:

It’s a puzzling headline with a rather sparse lede. If you click on anything on this page (the green button or “learn more”) you’re asked to log in. Once you log in you’re taken to this page:

It’s gorgeous, as you can see, but it just adds to the confusion. What “more” am I supposed to learn here? How much does this cost? What language do I use? What… is happening?

I’ve built a few tasks with this tool and they worked great, but from there… I have no idea. If you read their pricing you’ll become more confused, most likely. The documentation looks promising but is, once again, more than a bit sparse.

Verdict: Webtask is a neat idea but doesn’t seem to be the thing I need. In fact I have no idea what it’s supposed to be.

AWS

Amazon introduced AWS Lambda over a year ago and it made a big splash. The pitch was simple to understand:

AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume - there is no charge when your code is not running. With Lambda, you can run code for virtually any type of application or backend service - all with zero administration. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability. You can set up your code to automatically trigger from other AWS services or call it directly from any web or mobile app.

If you’ve worked with AWS before, you know that nothing AWS is ever simple. I went into this with a good dose of skepticism!

Step 1: Pick Your Function

After signing into the AWS console and clicking on “Lambda”, you’re sent to a splash screen with a button that says “Get Started”. You’re then taken here:

Welcome to AWS. It might just be me, but I feel like working with AWS is like wandering around the Mall of America, hoping to find that one store that sells that one thing you’re trying to find.

Note: for foreign readers, the Mall of America is a gigantic structure in Minnesota that embodies everything that is not wonderful about the US.

AWS is conceptually deafening. I am literally exhausted after working with it for a few hours and often I roll out of my chair onto the floor, weeping, as I can’t remember what I just did nor how much it will cost me.

OK maybe a little hyperbolic. There’s just a lot of moving parts is what I’m trying to say. So, where was I …

Step 2: Pick Your Trigger

Right: after you pick you function (good luck with that) you’re given this screen:

Your function is executed based on some type of event. If you know AWS and you know what these services are, you’re in luck! If not, say goodbye to the next few hours.

Step 3: Get Lost In The AWS Wilderness

We want to choose “API Gateway” for this so we can call our function from the outside world over HTTPS. And then…

This is where I’m going to hit the fast-forward button. I remember swearing a lot and feeling lost at this point (not hyperbole). I mean: I know what these things are, but I don’t know the implications fully. We’re talking about security here, and staging environments! These aren’t to be taken lightly.

I tried to figure it out myself for 2 solid days, and then I just gave up and went with a framework.

Serverless Framework

The Serverless Framework is a Node project that automates most of the pain when dealing with AWS. The video on the site is reasonably informative but… clicking on the documentation link (from the home page) gives you a 404, which I think is funny. If you use the menu on the top you’ll be OK.

So, in short, you use their CLI to generate a YML file and a Node file for your function code. You can use Python or Java with Lambda, but for now we’ll stick with Node.

As with most HelloWorld demos, the initial steps look simple. I’m not interested in that, I want to see what complex looks like. Here ya go:

service: bigmachine-checkout

provider:
  name: aws
  runtime: nodejs4.3
  region: us-west-2
  stage: dev
  vpc:
    securityGroupIds:
      - "sg-derptyderp"
    subnetIds:
      - "subnet-herptyherp"
  iamRoleStatements: # permissions for all of your functions can be set here
    - Effect: Allow
      Action:
        - cloudwatch:DescribeAlarms
        - cloudwatch:GetMetricStatistics
        - ec2:DescribeAccountAttributes
        - ec2:DescribeAvailabilityZones
        - ec2:DescribeSecurityGroups
        - ec2:DescribeSubnets
        - ec2:DescribeVpcs
        - ec2:CreateNetworkInterface
        - ec2:DescribeNetworkInterfaces
        - ec2:DetachNetworkInterface
        - ec2:DeleteNetworkInterface
        - sns:ListSubscriptions
        - sns:ListTopics
        - sns:Publish
        - logs:DescribeLogStreams
        - logs:GetLogEvents
      Resource: "*"

functions:
  paypalBraintreeFinalize:
     handler: handler.paypalExpressFinalize
     events:
      - http:
          path: paypal/finalize
          method: post
          cors: true

  paypalBraintreeToken:
     handler: handler.paypalExpressToken
     events:
      - http:
          path: paypal/token
          method: get
          cors: true

  stripeCheckout:
    handler: handler.stripeCharge
    events:
     - http:
         path: charge/stripe
         method: post
         cors: true

All in all, not that bad. You can read through this and probably figure out that I have functions to handle a Stripe charge and some PayPal stuff. They’re exposed via the API Gateway (which the framework does for you) and I’m able to wire up CORS easily.

Unfortunately I also want to talk to a database that’s not DynamoDB. I like using PostgreSQL with Compose.com which means that if I want my Lambda functions to talk to the outside world I need to set up a gateway, a VPC and all the lovely machinery that goes along with it.

The super silly thing is that I need to do this even if I use Amazon’s RDS stuff - you have to have a VPC setup for security reasons. That aint cheap.

VPCs and gateways are not cheap but, in the grand scheme of things, $50/month isn’t so bad either. But if I’m going to pay that… why don’t I just use Heroku?

Back Where I Started

It took me about 10 days to get things running properly. 2 of those days were spent being very, very frustrated trying to figure out all of the moving pieces and dealing with errors like this:

This was the final capper for me:

servless --help

WARNING: You are running v1.7.0. v1.8.0 will include the following breaking changes:
  ...

This doesn’t make me feel excited about the team behind this thing. Breaking things on a point release goes against the whole idea of semantic versioning. It wouldn’t matter but the framework uses Node/NPM which by general agreement follows semver, so it seems a bit silly to ignore it.

A bug report was filed and completely ignored, aside from this email response:

Breaking Changes - We’re no longer following strict semver. You can always find the most recent list of breaking changes in the upcoming milestone or in the Serverless CLI.

So there’s that then. Would you base a business on this framework? I decided it wasn’t for me.

Rolling My Own

Once you understand the machinery that goes into AWS Lambda, rolling your own isn’t too difficult. I took a weekend to dive in and figure out the bits that confused me prior, and eventually I had a few shell scripts rolled together that orchestrated most of what I needed.

If you’re comfortable with scripting, know AWS and can tolerate some head-pounding… AWS Lambda isn’t all that bad. A bit more expensive then I’d like but… not that bad.

Verdict: Every single time I’ve used AWS I’ve had to ditch a weekend or 2 remembering how things worked. It’s a powerful system, but in many ways is overkill when compared to something like Heroku (or the services like Heroku). The problem is that Heroku can get expensive, fast.

Then there’s what I’ve been doing for the last 5 years or so that’s working great: DigitalOcean. It has been my go-to for so long; it takes a lot to justify moving away from them. I have my build scripts down cold and I can whip up a server during lunch, complete with SSL and multiple, managed Node processes. I have the same for Elixir too.

Honestly: serverless with AWS isn’t buying me anything. I don’t want to use DynamoDB (though I’m sure it’s wonderful), I can use a message queue if I want to do things in a “small function” kind of way, and honestly it just takes too much space in my brain.

All of that said, yes I know that new frameworks are popping up daily, so maybe things will change a bit. For now, no AWS Lambda for me.

And Then: Firebase

I remember reading this post with a groan, thinking “not again”…

Today we are excited to announce the beta launch of Cloud Functions for Firebase. It lets you write small pieces of JavaScript, deploy them to Google’s Cloud infrastructure, and execute them in response to events from throughout the Firebase ecosystem.

I’ll admit to a heavy amount of Magpie-ishness. I can’t help it… it’s just me and I’ve learned to let me be me… as opposed to you or anyone else.

The first impression I had when looking over the “new” Firebase site was one of immense relief. The page is elegant, focused, easy on the eyes and easy to understand. The console is wonderfully simple, too:

Clicking through each of the services made sense to me. But what about the functions? How simple would it be to get code up here? What about hosting? Can I SSL this stuff? Outbound networking? Can I use PostgreSQL instead of Firebase or, maybe, together with it?

I found out the answers to all of these questions was, in short: simple, simple, yes, yes and yes. After working with AWS, Firebase was a very welcome change.

I’ll get to all of that in the next post.

See this series as a video

Watch how I built a serverless ecommerce site using Firebase. Over 3 hours of tightly-edited video, getting into the weeds with Firebase. We’ll use the realtime database, storage, auth, and yes, functions. I’ll also integrate Drip for user management. I detest foo/bar/hello-world demos; I want to see what’s really possible. That’s what this video is.

Permalink

¿Qué son los Sistemas Reactivos?

En cada una de mis charlas sobre Erlang siempre comienzo explicando el cambio sucedido hace bastantes años de programación secuencial a programación paralela y concurrente. Cambio que significó repensar el software de otra forma. Un grupo de programadores escribió el Manifiesto de Sistemas Reactivos para indicar pautas a seguir, ¿aún no lo has firmado?

Permalink

Kicking The Tires On This Serverless Thing

I just released a video series about building a serverless application with Firebase, and I thought I would write it up in a blog series as well. I think it's worth reading and understanding what I went through. This is Part 1 of that series.

The Punchline

Before doing this I spent a few months with AWS Lambda (which you'll read about below); first with The Serverless Framework (which I quite like - so jealous they got that domain!) and then with ClaudiaJS which didn't quite fit what I wanted. Finally I rolled my own thing with some shell scripts and a zip file.

I ended up with a mess that cost more money than I wanted it to. It wasn't easy to figure out and more than once I had to rage-swim at the Y to get over the stress of the day.

I had better luck with Firebase. I had a fun time and built something interesting. At least I think it is. I had to approach the development process in a completely different way than what I was used to... but I like that kind of thing. I know others don't. The big thing to me, however, is that I was able to build something that I would truly use. In fact I'm using parts of it now in production (more on that in a future post).

This is a long story and will cover many posts. Not sure how many but I'll try to keep it focused. As you can clearly tell I like Firebase, a lot, and yes I encountered quite a few issues along the way but they were surmountable. I'll get into all that, I promise.

For now let's start at the beginning, when I first dug into the "serverless" thing.

AWS, Firebase, Webtask.io...?

Let's start with Webtask. I know a number of people over at Auth0 (who own and run Webtask.io) and I have a ton of respect for them. Everything you're about to read has been said to them in person, so don't think I'm being unfair - I think they would agree with me straight away.

If you head over to the Webtask site you'll see a simple page with a headline:

It's a puzzling headline with a rather sparse lede. If you click on anything on this page (the green button or "learn more") you're asked to log in. Once you log in you're taken to this page:

It's gorgeous, as you can see, but it just adds to the confusion. What "more" am I supposed to learn here? How much does this cost? What language do I use? What... is happening?

I've built a few tasks with this tool and they worked great, but from there... I have no idea. If you read their pricing you'll become more confused, most likely. The documentation looks promising but is, once again, more than a bit sparse.

Verdict: Webtask is a neat idea but doesn't seem to be the thing I need. In fact I have no idea what it's supposed to be.

AWS

Amazon introduced AWS Lambda over a year ago and it made a big splash. The pitch was simple to understand:

AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume - there is no charge when your code is not running. With Lambda, you can run code for virtually any type of application or backend service - all with zero administration. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability. You can set up your code to automatically trigger from other AWS services or call it directly from any web or mobile app.

If you've worked with AWS before, you know that nothing AWS is ever simple. I went into this with a good dose of skepticism!

Step 1: Pick Your Function

After signing into the AWS console and clicking on "Lambda", you're sent to a splash screen with a button that says "Get Started". You're then taken here:

Welcome to AWS. It might just be me, but I feel like working with AWS is like wandering around the Mall of America, hoping to find that one store that sells that one thing you're trying to find.

Note: for foreign readers, the Mall of America is a gigantic structure in Minnesota that embodies everything that is not wonderful about the US.

AWS is conceptually deafening. I am literally exhausted after working with it for a few hours and often I roll out of my chair onto the floor, weeping, as I can't remember what I just did nor how much it will cost me.

OK maybe a little hyperbolic. There's just a lot of moving parts is what I'm trying to say. So, where was I ...

Step 2: Pick Your Trigger

Right: after you pick you function (good luck with that) you're given this screen:

Your function is executed based on some type of event. If you know AWS and you know what these services are, you're in luck! If not, say goodbye to the next few hours.

Step 3: Get Lost In The AWS Wilderness

We want to choose "API Gateway" for this so we can call our function from the outside world over HTTPS. And then...

This is where I'm going to hit the fast-forward button. I remember swearing a lot and feeling lost at this point (not hyperbole). I mean: I know what these things are, but I don't know the implications fully. We're talking about security here, and staging environments! These aren't to be taken lightly.

I tried to figure it out myself for 2 solid days, and then I just gave up and went with a framework.

Serverless Framework

The Serverless Framework is a Node project that automates most of the pain when dealing with AWS. The video on the site is reasonably informative but... clicking on the documentation link (from the home page) gives you a 404, which I think is funny. If you use the menu on the top you'll be OK.

So, in short, you use their CLI to generate a YML file and a Node file for your function code. You can use Python or Java with Lambda, but for now we'll stick with Node.

As with most HelloWorld demos, the initial steps look simple. I'm not interested in that, I want to see what complex looks like. Here ya go:

service: bigmachine-checkout

provider:  
  name: aws
  runtime: nodejs4.3
  region: us-west-2
  stage: dev
  vpc:
    securityGroupIds:
      - "sg-derptyderp"
    subnetIds:
      - "subnet-herptyherp"
  iamRoleStatements: # permissions for all of your functions can be set here
    - Effect: Allow
      Action:
        - cloudwatch:DescribeAlarms
        - cloudwatch:GetMetricStatistics
        - ec2:DescribeAccountAttributes
        - ec2:DescribeAvailabilityZones
        - ec2:DescribeSecurityGroups
        - ec2:DescribeSubnets
        - ec2:DescribeVpcs
        - ec2:CreateNetworkInterface
        - ec2:DescribeNetworkInterfaces
        - ec2:DetachNetworkInterface
        - ec2:DeleteNetworkInterface
        - sns:ListSubscriptions
        - sns:ListTopics
        - sns:Publish
        - logs:DescribeLogStreams
        - logs:GetLogEvents
      Resource: "*"

functions:  
  paypalBraintreeFinalize:
     handler: handler.paypalExpressFinalize
     events:
      - http:
          path: paypal/finalize
          method: post
          cors: true

  paypalBraintreeToken:
     handler: handler.paypalExpressToken
     events:
      - http:
          path: paypal/token
          method: get
          cors: true

  stripeCheckout:
    handler: handler.stripeCharge
    events:
     - http:
         path: charge/stripe
         method: post
         cors: true

All in all, not that bad. You can read through this and probably figure out that I have functions to handle a Stripe charge and some PayPal stuff. They're exposed via the API Gateway (which the framework does for you) and I'm able to wire up CORS easily.

Unfortunately I also want to talk to a database that's not DynamoDB. I like using PostgreSQL with Compose.com which means that if I want my Lambda functions to talk to the outside world I need to set up a gateway, a VPC and all the lovely machinery that goes along with it.

The super silly thing is that I need to do this even if I use Amazon's RDS stuff - you have to have a VPC setup for security reasons. That aint cheap.

VPCs and gateways are not cheap but, in the grand scheme of things, $50/month isn't so bad either. But if I'm going to pay that... why don't I just use Heroku?

Back Where I Started

It took me about 10 days to get things running properly. 2 of those days were spent being very, very frustrated trying to figure out all of the moving pieces and dealing with errors like this:

This was the final capper for me:

servless --help

WARNING: You are running v1.7.0. v1.8.0 will include the following breaking changes:  
  ...

This doesn't make me feel excited about the team behind this thing. Breaking things on a point release goes against the whole idea of semantic versioning. It wouldn't matter but the framework uses Node/NPM which by general agreement follows semver, so it seems a bit silly to ignore it.

A bug report was filed and completely ignored, aside from this email response:

Breaking Changes - We're no longer following strict semver. You can always find the most recent list of breaking changes in the upcoming milestone or in the Serverless CLI.

So there's that then. Would you base a business on this framework? I decided it wasn't for me.

Rolling My Own

Once you understand the machinery that goes into AWS Lambda, rolling your own isn't too difficult. I took a weekend to dive in and figure out the bits that confused me prior, and eventually I had a few shell scripts rolled together that orchestrated most of what I needed.

If you're comfortable with scripting, know AWS and can tolerate some head-pounding... AWS Lambda isn't all that bad. A bit more expensive then I'd like but... not that bad.

Verdict: Every single time I've used AWS I've had to ditch a weekend or 2 remembering how things worked. It's a powerful system, but in many ways is overkill when compared to something like Heroku (or the services like Heroku). The problem is that Heroku can get expensive, fast.

Then there's what I've been doing for the last 5 years or so that's working great: DigitalOcean. It has been my go-to for so long; it takes a lot to justify moving away from them. I have my build scripts down cold and I can whip up a server during lunch, complete with SSL and multiple, managed Node processes. I have the same for Elixir too.

Honestly: serverless with AWS isn't buying me anything. I don't want to use DynamoDB (though I'm sure it's wonderful), I can use a message queue if I want to do things in a "small function" kind of way, and honestly it just takes too much space in my brain.

All of that said, yes I know that new frameworks are popping up daily, so maybe things will change a bit. For now, no AWS Lambda for me.

And Then: Firebase

I remember reading this post with a groan, thinking "not again"...

Today we are excited to announce the beta launch of Cloud Functions for Firebase. It lets you write small pieces of JavaScript, deploy them to Google's Cloud infrastructure, and execute them in response to events from throughout the Firebase ecosystem.

I'll admit to a heavy amount of Magpie-ishness. I can't help it... it's just me and I've learned to let me be me... as opposed to you or anyone else.

The first impression I had when looking over the "new" Firebase site was one of immense relief. The page is elegant, focused, easy on the eyes and easy to understand. The console is wonderfully simple, too:

Clicking through each of the services made sense to me. But what about the functions? How simple would it be to get code up here? What about hosting? Can I SSL this stuff? Outbound networking? Can I use PostgreSQL instead of Firebase or, maybe, together with it?

I found out the answers to all of these questions was, in short: simple, simple, yes, yes and yes. After working with AWS, Firebase was a very welcome change.

I'll get to all of that in the next post.

See this series as a video

Watch how I built a serverless ecommerce site using Firebase. Over 3 hours of tightly-edited video, getting into the weeds with Firebase. We'll use the realtime database, storage, auth, and yes, functions. I'll also integrate Drip for user management. I detest foo/bar/hello-world demos; I want to see what's really possible. That's what this video is.

Permalink