Tag: benchmark

Categories
Swift

Getting started with Benchmark package

There was a blog post on swift.org about a new Swift package called Benchmark at the end of the March. As the name speaks for itself, it is all about measuring the performance of the piece of a code. Since last time I wrote about Merging sorted arrays with duplicates in Swift then it would be useful to know what is the performance of that function.

The function under measurement is just about merging two sorted arrays in a way that it detects duplicates, where the duplicate detection is based on the id.

The function we are going to measure is part of a Swift package. Since Benchmark is a separate package, we’ll need to add a package dependency as the first step. It brings in a couple of additional dependencies as well. 

.package(url: "https://github.com/ordo-one/package-benchmark", .upToNextMajor(from: "1.0.0")),

Interestingly, setting up a separate target for the package is really easy.

swift package --allow-writing-to-package-directory Benchmark init MyNewBenchmarkTarget

Which creates a separate target ready for running benchmarks.

swift package benchmark --target MyNewBenchmarkTarget

Before we run, let’s set it up to benchmark our function mentioned before. Firstly, we need to add a dependency to the generated part of the Package.swift. Since my example package has a target “PackageExampleBenchmark” which contains the function we want to measure, then it needs to be a dependency of the benchmark target.

package.targets += [
    .executableTarget(
        name: "MyNewBenchmarkTarget",
        dependencies: [
            .product(name: "Benchmark", package: "package-benchmark"),
            "PackageExampleBenchmark" // <-- added
        ],
        path: "Benchmarks/MyNewBenchmarkTarget",
        plugins: [
            .plugin(name: "BenchmarkPlugin", package: "package-benchmark")
        ]
    ),
]

import Benchmark
import Foundation
import PackageExampleBenchmark

let benchmarks = {
    let original = BenchmarkData.generate(count: 100000, offset: 0)
    let other = BenchmarkData.generate(count: 10, offset: 70000)

    Benchmark("SomeBenchmark") { benchmark in
        for _ in benchmark.scaledIterations {
            blackHole(original.mergeSorted(with: other, areInIncreasingOrder: { $0.date < $1.date }))
        }
    }
}

enum BenchmarkData {
    static let referenceDate = Date(timeIntervalSince1970: 1711282131)

    struct Item: Identifiable {
        let id: String
        let date: Date

    }

    static func generate(count: Int, offset: Int) -> [Item] {
        let ids = (0..<count)
            .map({ $0 + offset })
        return zip(ids.shuffled(), ids).map({
            Item(id: "\($0)",
                 date: Self.referenceDate.addingTimeInterval(TimeInterval($1)))
        })
    }
}

If we now run the benchmark, we’ll get this:

➜  PackageExampleBenchmark swift package benchmark --target MyNewBenchmarkTarget
Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build complete! (0.18s)
Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build complete! (0.35s)
Build complete!
Building BenchmarkTool in release mode...
Building benchmark targets in release mode for benchmark run...
Building MyNewBenchmarkTarget

==================
Running Benchmarks
==================

100% [------------------------------------------------------------] ETA: 00:00:00 | MyNewBenchmarkTarget:SomeBenchmark

=======================================================================================================
Baseline 'Current_run'
=======================================================================================================

Host 'MacBook-Air.lan' with 8 'arm64' processors with 24 GB memory, running:
Darwin Kernel Version 23.4.0: Wed Feb 21 21:51:37 PST 2024; root:xnu-10063.101.15~2/RELEASE_ARM64_T8112

====================
MyNewBenchmarkTarget
====================

SomeBenchmark
╒═══════════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                        │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞═══════════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Malloc (total) (K) *          │      60 │      60 │      60 │      60 │      60 │      60 │      60 │     178 │
├───────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Memory (resident peak) (K)    │    9077 │    9871 │    9904 │    9912 │    9912 │    9912 │    9912 │     178 │
├───────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Throughput (# / s) (#)        │     188 │     183 │     180 │     178 │     177 │     131 │     113 │     178 │
├───────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Time (total CPU) (μs) *       │    5266 │    5423 │    5517 │    5542 │    5616 │    7598 │    8672 │     178 │
├───────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Time (wall clock) (μs) *      │    5333 │    5460 │    5562 │    5612 │    5648 │    7623 │    8819 │     178 │
╘═══════════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

Great, we’ll see how long does it take. In the example above we were merging an array of 10000 elements with an array of 100 where half of them are duplicates. All of this runs under a 1 ms which is an excellent result.

Only a single data point does not give us a full picture. What I would like to know is how does it run when we have 10, 100, 1000, 10000, 100000 elements in the existing array and what happens if we try to merge 10, 100, 1000, 10000, 100000 elements to it.

let benchmarks = {
    Benchmark.defaultConfiguration = .init(
        metrics: [.wallClock]
    )

    let counts = [10, 100, 1000, 10000, 100000]
    let arrays = counts.map({ BenchmarkData.generate(count: $0, offset: 0) })

    for (originalIndex, originalCount) in counts.enumerated() {
        for (incomingIndex, incomingCount) in counts.reversed().enumerated() {
            let originalData = BenchmarkData.generate(count: originalCount, offset: 0)
            let incomingData = BenchmarkData.generate(count: incomingCount, offset: 0)
            Benchmark("\(originalCount) < \(incomingCount)") { benchmark in
                for _ in benchmark.scaledIterations {
                    blackHole(originalData.mergeSorted(with: incomingData, areInIncreasingOrder: { $0.date < $1.date }))
                }
            }
        }
    }
}

10 < 10
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (μs) * │      13 │      13 │      13 │      13 │      13 │      20 │      38 │   10000 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

10 < 100
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (μs) * │      61 │      61 │      61 │      61 │      64 │      85 │     102 │   10000 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

10 < 1000
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (μs) * │     500 │     503 │     507 │     525 │     528 │     556 │     629 │    1948 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

10 < 10000
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (μs) * │    5292 │    5411 │    5444 │    5476 │    5562 │    6304 │    6349 │     183 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

10 < 100000
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (ms) * │      53 │      54 │      54 │      54 │      54 │      55 │      55 │      19 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

100 < 10
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (μs) * │      46 │      46 │      46 │      47 │      51 │     111 │    1464 │   10000 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

100 < 100
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (μs) * │     115 │     116 │     118 │     118 │     126 │     142 │     529 │    8343 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

100 < 1000
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (μs) * │     543 │     545 │     552 │     568 │     571 │     587 │     677 │    1795 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

100 < 10000
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (μs) * │    5347 │    5444 │    5472 │    5509 │    5554 │    5693 │    5746 │     183 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

100 < 100000
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (ms) * │      51 │      51 │      52 │      52 │      52 │      52 │      52 │      20 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

1000 < 10
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (μs) * │     372 │     384 │     386 │     404 │     409 │     419 │     467 │    2555 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

1000 < 100
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (μs) * │     444 │     456 │     458 │     473 │     481 │     499 │     542 │    2157 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

1000 < 1000
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (μs) * │    1144 │    1166 │    1176 │    1188 │    1202 │    1248 │    1307 │     849 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

1000 < 10000
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (μs) * │    5648 │    5722 │    5743 │    5775 │    5833 │    5939 │    5951 │     174 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

1000 < 100000
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (ms) * │      53 │      54 │      54 │      54 │      54 │      54 │      54 │      19 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

10000 < 10
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (μs) * │    3962 │    4039 │    4057 │    4090 │    4119 │    4174 │    4270 │     247 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

10000 < 100
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (μs) * │    3923 │    4028 │    4059 │    4104 │    4166 │    4502 │    5312 │     245 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

10000 < 1000
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (μs) * │    4671 │    4784 │    4813 │    4846 │    4887 │    4956 │    4995 │     208 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

10000 < 10000
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (ms) * │      10 │      10 │      10 │      10 │      10 │      10 │      10 │     100 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

10000 < 100000
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (ms) * │      57 │      57 │      57 │      58 │      59 │      59 │      59 │      18 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

100000 < 10
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (ms) * │      40 │      41 │      41 │      41 │      41 │      41 │      41 │      25 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

100000 < 100
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (ms) * │      39 │      39 │      40 │      40 │      40 │      41 │      41 │      26 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

100000 < 1000
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (ms) * │      38 │      38 │      38 │      38 │      38 │      39 │      39 │      27 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

100000 < 10000
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (ms) * │      50 │      51 │      51 │      51 │      51 │      51 │      51 │      20 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

100000 < 100000
╒══════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞══════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Time (wall clock) (ms) * │     110 │     110 │     110 │     111 │     111 │     112 │     112 │      10 │
╘══════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

All in all, seems like I can be pretty happy with these results since merging 10000 elements to 10000 elements takes around 10 ms. Merging 100 elements on the other hand, takes around 0.5 ms.

If this was helpful, please let me know on Mastodon@toomasvahter or Twitter @toomasvahter. Feel free to subscribe to RSS feed. Thank you for reading.

Support me on Patreon
Donate with Paypal
Buy me a coffee
Categories
iOS Xcode

Comparing build speeds of Xcode 13 and Xcode 14b2

Xcode 14 beta 2 came with WWDC’22 and one of the highlights of the upcoming Xcode is faster builds. What’s new in Xcode mentions that Xcode 14 can be up to 25% faster, mostly due to improved parallelism. For testing this out, I looked for a large open source iOS app. WordPress for iOS is available on GitHub, and it was pretty easy to set up the local development setup.

My 16″ MBP is from 2019 and has 2,3 GHz 8-Core Intel Core i9 with 32 GB 2667 MHz DDR4 RAM. Benchmarking involved in deleting derived data, letting the Mac idle a couple of minutes to cool a bit, and then building the project. Also, I switched between Xcodes after every build. Let’s look into results.

Xcode 13.4.1Xcode 14 beta 2
293314
288317
279318
Build times for WordPress for iOS project

Results are in, and I have to admit that this is something I did not expect to see. I was expecting to see Xcode 14 doing much better compared to Xcode 13, but it is actually the other way around. Probably this project has many ways to improve the build times and making it better to use the improved parallelism in Xcode 14, but at the moment it turned out to be around 10% slower instead. Fascinating.

If this was helpful, please let me know on Mastodon@toomasvahter or Twitter @toomasvahter. Feel free to subscribe to RSS feed. Thank you for reading.

Support me on Patreon
Donate with Paypal
Buy me a coffee
Categories
Swift Swift Package

Measuring collections with Apple’s Swift Collections Benchmark

Apple’s swift-collections package contains data structure implementations which have more specific use-cases and therefore not fitting into the Foundation. Along with collections Swift package Apple also released swift-collections-benchmark package which implements benchmarking. As like with any other Swift package we can use the benchmarking package for our own purposes as well. Therefore, in this blog post let’s take a look on how to use the benchmarking package.

Step one is to create a new Swift package with executable target which will be used to run benchmarks. In Xcode’s File menu we can select New > Swift Package. In the created package we will want to do some changes in the Package.swift file. We’ll need to add swift-collections-benchmark dependency and change the package type to executable instead of library.

// swift-tools-version:5.4
import PackageDescription
let package = Package(
name: "SwiftBenchmark",
products: [
.executable(
name: "SwiftBenchmark",
targets: ["SwiftBenchmark"]),
],
dependencies: [
.package(url: "https://github.com/apple/swift-collections-benchmark&quot;, from: "0.0.1"),
],
targets: [
.executableTarget(
name: "SwiftBenchmark",
dependencies: [.product(name: "CollectionsBenchmark", package: "swift-collections-benchmark")])
]
)
view raw Package.swift hosted with ❤ by GitHub
Package.swift file after changing the default library template to executable.

Secondly, we need to add main.swift file to the package or use the @main keyword. In this example I went for main.swift. When that is set as well, we can start using the benchmarking package and writing benchmarks for collections. For trying out the benchmarking I am going to use raywenderlicht/swift-algorithms-club’s queue implementations. It provides two examples of queues where one is a thin wrapper around the array type and the other one is an optimized version which looks into improving performance when dequeuing elements from the queue.

Setting up a benchmark requires defining a benchmark object and adding benchmark tasks to it and then calling main() on the benchmark object. Every added task will show up as a separate line on the chart when we render results. The package has some input generators defined and also allows adding more if we need to. Check the registerInputGenerator function on the Benchmark type for more information. But OK, back to benchmarks. The snippet below shows how to define a benchmark for dequeue operation over a variety of input sizes where the max input size is currently limited to 128000.

var queueBenchmark = Benchmark(title: "Queue")
queueBenchmark.add(title: "QueueSimple", input: [Int].self, maxSize: 128_000) { input in
return { timer in
var queue = QueueSimple<Int>()
input.forEach({ queue.enqueue($0) })
timer.measure {
for _ in 0..<input.count {
let value = queue.dequeue()
blackHole(value)
}
}
}
}
queueBenchmark.add(title: "QueueOptimized", input: [Int].self, maxSize: 128_000) { input in
return { timer in
var queue = QueueOptimized<Int>()
input.forEach({ queue.enqueue($0) })
timer.measure {
for _ in 0..<input.count {
let value = queue.dequeue()
blackHole(value)
}
}
}
}
queueBenchmark.main()
view raw main.swift hosted with ❤ by GitHub

When everything is set we can go ahead and run the benchmark. First navigate to the package’s folder and use the swift run command as shown below. When benchmarking finishes we can use the generated file, in this case “results”, to render a chart.

▶ swift run -c release SwiftBenchmark run results –cycles 3
[2/2] Linking SwiftBenchmark
* Build Completed!
Running 2 tasks on 76 sizes from 1 to 1M:
QueueSimple
QueueOptimized
Output file: /Users/toomas/Desktop/SwiftBenchmark/results
Appending to existing data (if any) for these tasks/sizes.
Collecting data:
1.2.4…8…16…32…64…128…256…512…1k…2k…4k…8k…16k…32k…64k…128k…256k…512k…1M — 15.1s
1.2.4…8…16…32…64…128…256…512…1k…2k…4k…8k…16k…32k…64k…128k…256k…512k…1M — 15s
1.2.4…8…16…32…64…128…256…512…1k…2k…4k…8k…16k…32k…64k…128k…256k…512k…1M — 15.1s
Finished in 45.2s
▶ swift run -c release SwiftBenchmark render results chart.png && open chart.png
view raw run.sh hosted with ❤ by GitHub

Like expected, the optimized queue performs much better in larger input sizes. Perfect!

If this was helpful, please let me know on Mastodon@toomasvahter or Twitter @toomasvahter. Feel free to subscribe to RSS feed. Thank you for reading.

Support me on Patreon
Donate with Paypal
Buy me a coffee