Performance testing using XCTMetric

XCTMetric enables creating tests with measure blocks collecting information about CPU, memory and disk. In this post we’ll write UI-tests measuring a button tap what triggers writing to disk, allocating larger amount of memory and applying filters what requires CPU to do more work. It should be noted that XCTMetric can also be used in unit-tests.

Method under the test

The method we are going to write performance tests against is a simple method dealing with loading an image, writing data to disk, applying CIFilter and writing processed image to disk. In this example case, everything runs on a main thread what probably would not be a case in a real application.

@IBAction func process(_ sender: Any) {
	let image = UIImage(named: "Image")!
	imageStorage.store(image, filename: "original")
        
	let processedImage = ImageProcessor.processImage(image)
	imageStorage.store(image, filename: "processed")
	processedImageView.image = processedImage
}

XCTClockMetric for measuring taken time

XCTClockMetric is for measuring time taken by the block. Useful for catching regressions in longer running operations.

func testCalculateWithClockMetric() {
	let app = XCUIApplication()
	app.launch()
	measure(metrics: [XCTClockMetric()]) {
		app.buttons["Process"].tap()
	}
}

XCTCPUMetric for measuring CPU utilization

XCTCPUMetric measures CPU activity and output 3 different results: CPU time, CPU cycles and CPU instructions retired. CPUs have a feature called speculative execution what means that more instructions are completed than the actual program flow requires. Retired instructions are the instructions which were actually needed by the flow of the program. This feature speeds up the program execution as CPU can process data ahead of time. Example case would be if else where CPU processes both branches but only one branch is valid in the program flow.

func testCalculateWithCPUMetric() {
	let app = XCUIApplication()
	app.launch()
	measure(metrics: [XCTCPUMetric(application: app)]) {
		app.buttons["Process"].tap()
	}
}

XCTMemoryMetric for measuring allocated memory

XCTMemoryMetric measures allocated physical memory useful for testing operation allocating significant amount of memory (processing images).

func testCalculateWithMemoryMetric() {
	let app = XCUIApplication()
	app.launch()
	measure(metrics: [XCTMemoryMetric(application: app)]) {
		app.buttons["Process"].tap()
	}
}

XCTStorageMetric for measuring disk usage

XCTStorageMetric measures bytes written to the disk.

func testCalculateWithStorageMetric() {
	let app = XCUIApplication()
	app.launch()
	measure(metrics: [XCTStorageMetric(application: app)]) {
		app.buttons["Process"].tap()
	}
}

XCTOSSignpostMetric for measuring time between signposts

Apple provides signpost metric for application launch time for making it easy to add performance test measing launch time. In WWDC’19 session “Optimizing app launch” the suggested goal is 400 ms which is the duration of the app launch animation. XCTOSSignpostMetric has initializer for custom signpost as well.

func testLaunchPerformance() {
	if #available(macOS 10.15, iOS 13.0, tvOS 13.0, *) {
		measure(metrics: [XCTOSSignpostMetric.applicationLaunch]) {
			XCUIApplication().launch()
		}
	}
}

Summary

XCTMetric enables writing performance tests for performance critical parts of the application. We took a look at CPU, memory, storage and signpost metrics.

If this was helpful, please let me know on Twitter @toomasvahter. Feel free to subscribe to RSS feed. Thank you for reading.

Example

MeasuringInTests (Xcode 11.3)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s