Artillery Load Testing: A YAML Tutorial With Real Output

Artillery load testing means describing a test in a YAML file, with a config section for the target and the load phases and a scenarios section for what each virtual user does, and running it with artillery run. You install it from npm, and it reports response times, request rates, status codes and errors in the terminal as the test runs.

This tutorial writes one Artillery script from scratch, runs it against a small local shop API, and shows the output Artillery 2.0.34 printed, including the three things that surprised us: an undocumented exit code, error rows keyed by full URL, and a report command that no longer works.

Key Takeaways

• Artillery 2.0.34 (August 2026) is open source under MPL-2.0 and asks for Node.js 22.18.0 or later.
• arrivalRate counts new virtual users started per second; how many run at once depends on session length. Our 10 arrivals per second kept about 8 users active at a time.
• Read the per-endpoint rows from the metrics-by-endpoint plugin: the overall median was 1 ms while the product endpoint’s median was 56.3 ms.
• HTTP 500s alone do not fail a run. Add expect checks or ensure thresholds, and set useOnlyRequestNames on both plugins.
• artillery report is gone (removed in 2.0.22). Keep the JSON from --output and read it with jq, or use Artillery Cloud.

What is Artillery?

Artillery is an open-source load testing tool for Node.js, released under the Mozilla Public License 2.0, in which tests are YAML files, or JavaScript and TypeScript for more control. Its engines cover HTTP, WebSocket, Socket.IO and Playwright browsers. The current release is 2.0.34, published on 14 August 2026.

Each virtual user is independent, with its own TCP connections and cookies, and users arrive in phases: a phase says how many new users to start each second, and each one runs one scenario to the end and leaves. In Locust and JMeter, by contrast, a fixed number of users loop until the test stops. If you are still choosing a tool, our k6 vs JMeter vs Gatling vs Locust comparison covers those four tools.

How do you install Artillery?

Install Artillery globally from npm, then check the version it reports. The package declares Node.js 22.18.0 or later in its engines field, and the Artillery docs ask for the most recent LTS release of Node.js, so check node -v first.

node -v
npm install -g artillery@latest
artillery version

We installed 2.0.34 into its own project folder and ran it through npx; npx artillery version printed Artillery: 2.0.34 and Node.js: v23.11.1. The docs advise keeping load tests in a separate project, not as a dev dependency of your application.

How do you write an Artillery test script?

An Artillery script has a config section with the target URL, the load phases and any plugins, and a scenarios section listing what virtual users do. Each scenario is a flow of steps such as get, post and think, and a weight decides how often new users pick it over the other scenarios.

Our test target is a small shop API with four endpoints: a home page, a product endpoint that takes 20 to 80 ms and fails with HTTP 500 about 2% of the time, a login that returns a token, and a cart that needs that token and now and then returns HTTP 200 without its items key. Here is the whole script, shop.yml:

config:
  target: http://127.0.0.1:8080
  phases:
    - name: warm up
      duration: 10
      arrivalRate: 2
      rampTo: 10
    - name: sustained load
      duration: 50
      arrivalRate: 10
  payload:
    path: users.csv
    fields:
      - username
      - password
      - productId
    skipHeader: true
  plugins:
    metrics-by-endpoint:
      useOnlyRequestNames: true
    expect:
      outputFormat: prettyError
      reportFailuresAsErrors: true
    ensure:
      thresholds:
        - http.response_time.p95: 200

scenarios:
  - name: browse
    weight: 3
    flow:
      - get:
          url: /
          name: GET /
      - think: 1
      - get:
          url: /products?id={{ productId }}
          name: GET /products
          expect:
            - statusCode: 200

  - name: cart
    weight: 1
    flow:
      - post:
          url: /login
          name: POST /login
          json:
            username: '{{ username }}'
            password: '{{ password }}'
          capture:
            json: $.token
            as: token
      - get:
          url: /cart
          name: GET /cart
          headers:
            Authorization: 'Bearer {{ token }}'
          expect:
            - statusCode: 200
            - hasProperty: items

What each part does:

• target is the base URL for every request.
• payload loads users.csv and gives each virtual user one row, here a username, a password and a product ID. skipHeader: true skips the header line.
• weight: 3 and weight: 1 make three out of four new users browse and one in four open the cart.
• think: 1 pauses the user for one second.
• capture reads $.token from the login response with JSONPath and stores it as token for the next request. Captures are strict: if nothing matches, that user stops making requests.
• name labels each request, so the per-endpoint plugin groups every product ID under GET /products.
• expect checks each response. A status check is how a 500 gets counted as a failure; hasProperty catches a 200 with the wrong body.
• ensure sets a pass or fail threshold for the whole run, here a 95th percentile under 200 ms.

All three plugins ship inside the artillery package in 2.0.34.

What do arrivalRate and load phases mean in Artillery?

arrivalRate is the number of new virtual users Artillery starts every second during a phase. Each user runs its scenario and leaves, so the number of users active at once depends on how long a scenario takes: arrivals per second multiplied by the average session length.

Our numbers show the difference. In the sustained phase 10 users arrived every second, and the summary’s mean session length was 772.8 ms (a browse session took just over a second because of the one-second think time; a cart session took a few milliseconds). Ten arrivals a second times 0.77 seconds is about 8 users active at any moment, not 10, and nowhere near the 560 users the test created in total.

Artillery has four kinds of phase:

maxVusers caps how many users a phase can have running at once, which stops users piling up on a slow server. A phase’s duration is only how long users keep arriving: the run ends when the last user finishes, so our 60 seconds of phases took 1 minute, 2 seconds. To hit a target from a concurrent users calculation, divide that number by your session length in seconds to get the arrival rate.

How do you run an Artillery test?

Run artillery run with the script name. Add --output to save a JSON report of the run. During the test the CLI prints a metrics block roughly every 10 seconds, and when the last user finishes it prints a summary for the whole run.

artillery run --output run1.json shop.yml

Our 20-second control runs used edited copies of the script, with --overrides to shorten the phases:

artillery run --overrides '{"config":{"phases":[{"duration":20,"arrivalRate":10}]}}' shop.yml

How do you read Artillery results?

The end-of-run summary lists counters, rates and histograms in alphabetical order. Read the per-endpoint response times first, then the status-code counts and the failed checks, and compare vusers.created with vusers.completed to see how many users did not finish. These are the lines that matter from our run, trimmed (histograms also report min, max and mean):

http.codes.200: ................................................ 1112
http.codes.500: ................................................ 8
http.request_rate: ............................................. 19/sec
http.requests: ................................................. 1120
http.response_time:
  median: ...................................................... 1
  p95: ......................................................... 77.5
  p99: ......................................................... 82.3
plugins.expect.failed.hasProperty: ............................. 4
plugins.expect.failed.statusCode: .............................. 8
plugins.metrics-by-endpoint.response_time.GET /:
  median: ...................................................... 1
  p95: ......................................................... 3
plugins.metrics-by-endpoint.response_time.GET /products:
  median: ...................................................... 56.3
  p95: ......................................................... 80.6
  p99: ......................................................... 85.6
vusers.created: ................................................ 560
vusers.completed: .............................................. 548
vusers.failed: ................................................. 12

Four things to take from this output.

Read each endpoint, not http.response_time. The overall median of 1 ms is accurate and useless: 713 of the 1,120 requests went to the fast home page, login and cart, which hid the product endpoint’s 56.3 ms median and 80.6 ms 95th percentile. Our guide to why percentiles beat averages explains why the tail is the number to watch. Without the metrics-by-endpoint plugin, Artillery prints only the overall figures.

Failed users are failed checks. vusers.failed: 12 is the 8 failed status checks plus the 4 failed hasProperty checks, because reportFailuresAsErrors turns each failed expectation into an error for that user. The 8 matches http.codes.500 exactly.

Error rows ignore your request names by default. The metrics-by-endpoint plugin grouped every product request under GET /products, but the error lines did not: our run printed eight separate rows such as errors.Failed expectations for request http://127.0.0.1:8080/products?id=473. The expect plugin has its own useOnlyRequestNames option. With it set, the next run printed errors.Failed expectations for request GET /products as one row.

The request rate follows from the phases. 560 users making two requests each is 1,120 requests in about a minute, the 19 per second Artillery reported.

How do you fail a CI build with Artillery?

Artillery exits with code 0 whenever every virtual user starts and finishes, even if responses were errors. The docs say it “will not” count protocol-level errors such as 500s toward the exit code. To fail a pipeline you need a check that sets a non-zero code: an ensure threshold or condition, or a failed expect check.

We confirmed each case with a separate run:

The docs’ exit codes page does not list 21; the expect plugin sets global.artillery.suggestedExitCode = 21 when any expectation fails (line 199 of its index.ts in 2.0.34). CI treats any non-zero code as a failure, so the build still stops, but a script that tests for exactly 1 will miss it.

ensure takes two forms. thresholds checks that a metric stays under a value. conditions takes an expression over several metrics, for example http.response_time.p95 < 250 and http.request_rate > 1000, and strict: false makes one optional. The docs warn that a metric name that does not exist fails the check, so a typo fails the build instead of passing it silently.

Where is the Artillery HTML report?

There is no longer a local HTML report. The artillery report command was removed in version 2.0.22, although many tutorials still tell you to run it. In 2.0.34 it prints a box saying the command “has been deprecated and is no longer supported”, writes nothing, and exits with code 0, so a script that runs it will not notice.

Instead, the JSON file from --output holds an aggregate section with the summary and an intermediate array with one entry per reporting period, which you can query with jq. Or record runs to Artillery Cloud, the hosted dashboard the docs now point to, with --record and --key.

Can Artillery load test with Playwright?

Yes. Artillery’s Playwright engine, built in since 2.0.0-33, runs real headless browsers, so each virtual user loads pages the way a visitor does, scripts and images included. It records browser metrics such as FCP and LCP for each page, which the HTTP engine cannot see. You write the test as a Playwright function and point a scenario at it with engine: playwright.

We did not run the Playwright engine for this tutorial, so there are no measured numbers here. Per its reference page, each Artillery release pins a specific Playwright version and Chromium binary, and distributed runs on Fargate or ACI are always headless. A browser needs far more CPU and memory than an HTTP client, so expect many fewer virtual users per machine than in a test like ours.

How do you run Artillery from several machines?

Artillery distributes a test from the CLI to workers in your own cloud account: run-lambda for AWS Lambda, run-fargate for AWS Fargate and run-aci for Azure Container Instances. It bundles the script, CSV files and plugins, creates the resources it needs, runs the workers, merges their metrics and removes the resources when the test ends.

You supply the AWS or Azure access, the networking and the bill, and traffic comes from the regions you pick in that account. That suits testing internal services from inside your own network.

Running load tests without your own cloud workers

LoadFocus does not run Artillery scripts. It runs JMeter and k6 scripts, and tests built in the browser, from 25+ cloud regions, with no load generators or cloud account of your own to manage, and keeps a history of runs you can compare over time.

If your Artillery scenarios are a handful of HTTP calls, rebuilding them as a k6 load test takes little time, and k6 tests are JavaScript, which a Node.js team already reads. If YAML scenarios or the Playwright engine are why you picked Artillery, keeping Artillery is a sound choice; the Artillery alternative page lays out that trade-off.

Frequently Asked Questions

Is Artillery free?

Yes. The Artillery CLI is open source under the Mozilla Public License 2.0 and installs from npm. You pay only for the machines that generate load, including any AWS or Azure workers it starts in your own account for distributed tests. Artillery Cloud, the hosted dashboard for stored reports, is a separate service with its own plans.

What does arrivalRate mean in Artillery?

arrivalRate is how many new virtual users Artillery starts each second during a load phase. Each user runs one scenario and then finishes, so the number of users active at the same time is the arrival rate multiplied by the average session length. For example, 10 arrivals per second with sessions averaging 0.77 seconds keep about 8 users active.

Why does Artillery exit with code 21?

Exit code 21 means at least one expect check failed during the run. The expect plugin sets it when any expectation fails, even if every ensure threshold (the run-level pass or fail limits) passed. The Artillery exit codes page does not list it. A failed ensure check produces exit code 1 instead, and a run with no failed checks exits with 0, even when some responses were HTTP 500.

Does Artillery still generate an HTML report?

No. The artillery report command was removed in version 2.0.22. In 2.0.34 it prints a deprecation notice, writes no file and exits with code 0. Save results with artillery run --output report.json and read the JSON with a tool such as jq, or record runs to Artillery Cloud with --record to see them in its dashboard.

What is the difference between Artillery and Locust?

Artillery tests are YAML files run by Node.js, and its load model starts new users at a set rate per second. Locust tests are Python classes, and you set a fixed number of users who keep looping through their tasks. Artillery suits teams who want declarative scenarios and Playwright browser tests; Locust suits teams who want every test written as Python code.

Sources

• Artillery 2.0.34 release date, MPL-2.0 licence and Node.js engines field: registry.npmjs.org/artillery, October 2026.
• Virtual users, load phases and test lifecycle: artillery.io/docs, “Core concepts”, October 2026.
• Phase types, maxVusers, payload, scenario weights: artillery.io/docs, “Test script reference”, October 2026.
• Requests, capture, think, cookies and HTTP metrics: artillery.io/docs, “HTTP engine”, October 2026.
• ensure thresholds and conditions: artillery.io/docs, “ensure” plugin reference, October 2026.
• expect checks, reportFailuresAsErrors and useOnlyRequestNames: artillery.io/docs, “Expectations and Assertions”, October 2026.
• Per-endpoint metrics: artillery.io/docs, “metrics-by-endpoint”, October 2026.
• Command line options and the JSON report: artillery.io/docs, “run” command, October 2026.
• Exit codes: artillery.io/docs, “Exit Codes”, October 2026. Exit code 21: artillery-plugin-expect/index.ts, bundled in artillery 2.0.34.
• Removal of artillery report in 2.0.22: artillery.io/docs, “report” command, October 2026.
• Playwright engine: artillery.io/docs, “Load testing with Playwright and Artillery”, October 2026.
• Distributed tests on Lambda, Fargate and ACI: artillery.io/docs, “Distributed load testing”, October 2026.
• Test runs: Artillery 2.0.34 on Node.js 23.11.1, 60 seconds (10-second ramp from 2 to 10 arrivals per second, then 50 seconds at 10) against a local Python test API, plus two 20-second control runs and an artillery report call, 2026-10-27.

Related reading

• Locust Load Testing: A Python Tutorial With Real Output
• k6 vs JMeter vs Gatling vs Locust: A Practical Comparison for 2026
• Comparing Load Testing Tools for CI/CD Pipelines
• Why are Percentiles More Useful than Averages in Performance Testing
• Calculate concurrent users for load tests