Measuring collections with Apple’s Swift Collections Benchmark

Apple’s swift-collections package contains data structure implementations which have more specific use-cases and therefore not fitting into the Foundation. Along with collections Swift package Apple also released swift-collections-benchmark package which implements benchmarking. As like with any other Swift package we can use the benchmarking package for our own purposes as well. Therefore, in this blog post let’s take a look on how to use the benchmarking package.

Step one is to create a new Swift package with executable target which will be used to run benchmarks. In Xcode’s File menu we can select New > Swift Package. In the created package we will want to do some changes in the Package.swift file. We’ll need to add swift-collections-benchmark dependency and change the package type to executable instead of library.

	// swift-tools-version:5.4

	import PackageDescription

	let package = Package(
	name: "SwiftBenchmark",
	products: [
	.executable(
	name: "SwiftBenchmark",
	targets: ["SwiftBenchmark"]),
	],
	dependencies: [
	.package(url: "https://github.com/apple/swift-collections-benchmark", from: "0.0.1"),
	],
	targets: [
	.executableTarget(
	name: "SwiftBenchmark",
	dependencies: [.product(name: "CollectionsBenchmark", package: "swift-collections-benchmark")])
	]
	)

view raw Package.swift hosted with ❤ by GitHub

Package.swift file after changing the default library template to executable.

Secondly, we need to add main.swift file to the package or use the @main keyword. In this example I went for main.swift. When that is set as well, we can start using the benchmarking package and writing benchmarks for collections. For trying out the benchmarking I am going to use raywenderlicht/swift-algorithms-club’s queue implementations. It provides two examples of queues where one is a thin wrapper around the array type and the other one is an optimized version which looks into improving performance when dequeuing elements from the queue.

Setting up a benchmark requires defining a benchmark object and adding benchmark tasks to it and then calling main() on the benchmark object. Every added task will show up as a separate line on the chart when we render results. The package has some input generators defined and also allows adding more if we need to. Check the registerInputGenerator function on the Benchmark type for more information. But OK, back to benchmarks. The snippet below shows how to define a benchmark for dequeue operation over a variety of input sizes where the max input size is currently limited to 128000.

	var queueBenchmark = Benchmark(title: "Queue")
	queueBenchmark.add(title: "QueueSimple", input: [Int].self, maxSize: 128_000) { input in
	return { timer in
	var queue = QueueSimple<Int>()
	input.forEach({ queue.enqueue($0) })
	timer.measure {
	for _ in 0..<input.count {
	let value = queue.dequeue()
	blackHole(value)
	}
	}
	}
	}
	queueBenchmark.add(title: "QueueOptimized", input: [Int].self, maxSize: 128_000) { input in
	return { timer in
	var queue = QueueOptimized<Int>()
	input.forEach({ queue.enqueue($0) })
	timer.measure {
	for _ in 0..<input.count {
	let value = queue.dequeue()
	blackHole(value)
	}
	}
	}
	}
	queueBenchmark.main()

view raw main.swift hosted with ❤ by GitHub

When everything is set we can go ahead and run the benchmark. First navigate to the package’s folder and use the swift run command as shown below. When benchmarking finishes we can use the generated file, in this case “results”, to render a chart.

	▶ swift run -c release SwiftBenchmark run results –cycles 3
	[2/2] Linking SwiftBenchmark

	* Build Completed!
	Running 2 tasks on 76 sizes from 1 to 1M:
	QueueSimple
	QueueOptimized
	Output file: /Users/toomas/Desktop/SwiftBenchmark/results
	Appending to existing data (if any) for these tasks/sizes.

	Collecting data:
	1.2.4…8…16…32…64…128…256…512…1k…2k…4k…8k…16k…32k…64k…128k…256k…512k…1M — 15.1s
	1.2.4…8…16…32…64…128…256…512…1k…2k…4k…8k…16k…32k…64k…128k…256k…512k…1M — 15s
	1.2.4…8…16…32…64…128…256…512…1k…2k…4k…8k…16k…32k…64k…128k…256k…512k…1M — 15.1s
	Finished in 45.2s

	▶ swift run -c release SwiftBenchmark render results chart.png && open chart.png