Benchmarking websocket server performance with Artillery

We recently deployed a custom websocket server for Oh Dear! and wanted to test its ability to handle requests under load. This blogpost will explain how we performed that load test using Artillery.

Installing Artillery

There’s a powerful tool called Artillery that allows you to – among other things – stresstest a websocket server. This includes socket.io as well as regular websocket servers.

First, install it using either npm or yarn. This assumes you have nodejs already installed.

# Via npm
$ npm install -g artillery

# Via Yarn
$ yarn global add artillery

Once installed, it’s time for the fun part.

Creating your scenario for the load test

There are a few ways you can load test your websockets. You can, rather naively, just launch a bunch of requests – much like ab (Apache Bench) would – and see how many hits/second you can get.

This is relatively easy with artillery. First, create a simple scenario you want to launch.

Save this in a configuration YAML file called loadtest1.yml.

config:
  target: "ws://127.0.0.1:6001/app/0a1422fe85698e340c142219e77192e8?protocol=7&client=js&version=4.3.1&flash=false"
  phases:
    - duration: 20  # Test for 20 seconds
      arrivalRate: 10 # Every second, add 10 users
      rampTo: 100 # Ramp it up to 100 users over the 20s period
      name: "Ramping up the load"
scenarios:
  - engine: "ws"
    flow:
      - send: 'hello'
      - think: 5

This will do a few things, as explained by the comments in the file:

• Run the test for 20 seconds
• Every second, it will add 10 users until it reaches 100 users
• Once connected, the user will send a message over the channel with the string hello
• Every user will hold the connection open for 5 seconds

To start this scenario, run this artillery command.

$ artillery run loadtest1.yml
Started phase 0 (Ramping up the load), duration: 20s @ ...

However, this is a fairly naive approach, as your test is sending garbage (the string “hello”) and will just test the connection limits of both your client and the server. It’ll mostly stresstest the TCP stack, not so much the server itself, as it’s not doing anything (besides accepting some connections).

A test like this can quickly give you a few thousand connected users, without too much hassle (assuming you’ve increased your max open files limits).

Testing a real life sample

It would be far more useful if you could test an actual websocket implementation. One that would look like this:

1. Connect to a websocket
2. Make it subscribe to a channel for events
3. Receive the events

Aka: what a real browser would do.

To test this, consider the following scenario. Save this YAML content in a file called loadtest2.yml.

config:
  target: "wss://socket.ohdear.app/app/2420b144fceaca7df5a5?protocol=7&client=js&version=4.3.1&flash=false"
  phases:
    - duration: 60  # Test for 60 seconds
      arrivalRate: 10 # Every second, add 10 users
      rampTo: 100 # And ramp it up to 100 users in total over the 60s period
      name: "Ramping up the load"
    - duration: 120 # Then resume the load test for 120s
      arrivalRate: 100 # With those 100 users we ramped up to in the first phase
      rampTo: 100 # And keep it steady at 100 users
      name: "Pushing a constant load"
scenarios:
  - engine: "ws"
    flow:
      - send: '{"event":"pusher:subscribe","data":{"channel":"public"}}'  # Subscribe to the public channel
      - think: 15 # Every connection will remain open for 15s

This example is similar to our first one, but the big difference is in the message we send: {"event":"pusher:subscribe","data":{"channel":"public"}}. An actual JSON payload that instructs the client to listen to events sent on the channel public.

In our example, this subscribes to the counter of live amount of health checks being run on Oh Dear!. Every second or so, we publish a new number on the public channel, so every socket listening to that channel will receive data. And more importantly: our server is forced to send that data to every connected user.

With an example like this, we’re testing our full websocket stack: data needs to be sent to our websocket and it needs to relay that to every client subscribed to that channel. This is what will cause the actual load (on both client and the server) and this is what you’ll want to test.

Now, it’s a matter of slowly increasing the amount of users in the scenario and finding the breaking point. ;-)