Categories
iOS Swift UIKit

Resizing UIImages with aspect fill on iOS

Resizing images is an important topic when we need to display images which do not match with the intended display size. For example, rendering much large images in a small rectangle. UIImageView supports scaling images automatically but that becomes inefficient when dealing with larger images. In this blog post we’ll take a look on how to crop and resize images to fill a target size while keeping the original aspect ratio.

Cropping and scaling UIImages

The end goal of this exercise is to create a small ImageScaler struct which supports cropping the original image and resizing it to the target size. The final instance of the UIImage has smaller size which means that less memory is required for rendering the image.

// Scaling the full image
let originalImage = UIImage(named: "example_photo.jpg")!
var targetSize = CGSize(width: 100, height: 100)
imageView.image = ImageScaler.scaleToFill(originalImage, in: targetSize)
// Scaling only a small part of the image
let originalImage = UIImage(named: "example_photo.jpg")!
var fromRect = CGRect(x: 362.0, y: 449.0, width: 260.0, height: 160.0)
var targetSize = CGSize(width: 100, height: 100)
imageView.image = ImageScaler.scaleToFill(originalImage, in: targetSize, from: fromRect)
view raw Image.swift hosted with ❤ by GitHub
Scaling an image to a target size.

As the first step let’s take a look on how to write the cropping and scaling logic. NSHipster has a great post about the different techniques what we can use for resizing images. We are going to use UIGraphicsImageRenderer for creating the scaled image. One important thing to note is that when we use CGContext for drawing the image then we need to flip the coordinate system because UIImage’s and CGContext’s coordinates do not match (UIImage uses upper left corner, CGContext bottom left corner). Coordinates can be transformed from the UIImage coordinate system to the CGContext coordinate system by combining translation and scale transforms. First we’ll move the image and then flip it in the opposite direction so that the final frame stays in the image rect.

var transform = CGAffineTransform(translationX: 0.0, y: targetSize.height)
transform = transform.scaledBy(x: 1, y: -1)
context.cgContext.concatenate(transform)
view raw CGContext.swift hosted with ❤ by GitHub
Flipping the image vertically in the CGContext .

Next step after applying the affine transform is to crop the image. UIImage has a cgImage property but it can be nil when the instance was initialized with a CIImage backing storage. Therefore, we’ll need to handle both cases. Apple has a convenience drawing method for CIImage which already knows how to handle cropping. On the otherhand CGImage needs to be cropped first and then drawn. The full implementation of the crop and resize becomes:

private static func scale(_ image: UIImage, fromRect: CGRect = .zero, targetSize: CGSize) -> UIImage {
let renderer = UIGraphicsImageRenderer(size: targetSize)
return renderer.image { context in
// UIImage and CGContext coordinates are flipped.
var transform = CGAffineTransform(translationX: 0.0, y: targetSize.height)
transform = transform.scaledBy(x: 1, y: -1)
context.cgContext.concatenate(transform)
// UIImage -> cropped CGImage
let makeCroppedCGImage: (UIImage) -> CGImage? = { image in
guard let cgImage = image.cgImage else { return nil }
guard !fromRect.isEmpty else { return cgImage }
return cgImage.cropping(to: fromRect)
}
if let cgImage = makeCroppedCGImage(image) {
context.cgContext.draw(cgImage, in: CGRect(origin: .zero, size: targetSize))
}
else if let ciImage = image.ciImage {
var transform = CGAffineTransform(translationX: 0.0, y: image.size.height)
transform = transform.scaledBy(x: 1, y: -1)
let adjustedFromRect = fromRect.applying(transform)
let ciContext = CIContext(cgContext: context.cgContext, options: nil)
ciContext.draw(ciImage, in: CGRect(origin: .zero, size: targetSize), from: adjustedFromRect)
}
}
}
view raw Resize.swift hosted with ❤ by GitHub
Crop and resize the image.

Calculating rects for scaled to fill behavior

Now we have cropping and resize logic available. Depending on the target size of the image we’ll need to figure out which parts of the original image should be cropped so that the original aspect ratio does not change. Note that the original image size and the target size can have different aspect ratios: square, portrait, landscape. Therefore, we’ll need to handle all the cases. Let’s start by adding convenience properties to CGSize.

extension CGSize {
enum Aspect {
case portrait, landscape, square
}
var aspect: Aspect {
switch width / height {
case 1.0:
return .square
case 1.0:
return .landscape
default:
return .portrait
}
}
var aspectRatio: CGFloat {
return width / height
}
}
Convenience properties on CGRect.

After that we can add a method on CGRect which calculates a CGRect of the original image, what can be drawn in the target image. There are 9 different combinations what we need to handle. But the core logic stays the same: scale the current rectangle so that it fills the target size while keeping the original aspect ratio. Then chop off the sides which go over the target size and center the image in the target size.

extension CGRect {
func scaled(toFillSize targetSize: CGSize) -> CGRect {
var scaledRect = self
switch (size.aspect, targetSize.aspect) {
case (.portrait, .portrait), (.portrait, .square):
scaledRect.size.height = width / targetSize.aspectRatio
scaledRect.size.width = width
if scaledRect.height > height {
scaledRect.size = size
}
scaledRect.origin.y -= (scaledRect.height height) / 2.0
case (.portrait, .landscape), (.square, .landscape):
scaledRect.size.height = width / targetSize.aspectRatio
scaledRect.size.width = width
if scaledRect.height > height {
scaledRect.size = size
}
scaledRect.origin.y -= (scaledRect.height height) / 2.0
case (.landscape, .portrait), (.square, .portrait):
scaledRect.size.height = height
scaledRect.size.width = height * targetSize.aspectRatio
if scaledRect.width > width {
scaledRect.size = size
}
scaledRect.origin.x -= (scaledRect.width width) / 2.0
case (.landscape, .landscape), (.landscape, .square):
scaledRect.size.height = height
scaledRect.size.width = height * targetSize.aspectRatio
if scaledRect.size.width > width {
scaledRect.size = size
}
scaledRect.origin.x -= (scaledRect.width width) / 2.0
case (.square, .square):
return self
}
return scaledRect.integral
}
}
Scaling rect to the target size with keeping the initial aspect ratio.

Finalizing the ImageScaler

We can proceed with creating a single static method which takes care of scaling the original image to the target size while keeping the aspect ratio. The whole implementation looks like this:


struct ImageScaler {
static func scaleToFill(_ image: UIImage, in targetSize: CGSize, from fromRect: CGRect = .zero) -> UIImage {
let rect = fromRect.isEmpty ? CGRect(origin: .zero, size: image.size) : fromRect
let scaledRect = rect.scaled(toFillSize: targetSize)
return scale(image, fromRect: scaledRect, targetSize: targetSize)
}
private static func scale(_ image: UIImage, fromRect: CGRect = .zero, targetSize: CGSize) -> UIImage {
let renderer = UIGraphicsImageRenderer(size: targetSize)
return renderer.image { context in
// UIImage and CGContext coordinates are flipped.
var transform = CGAffineTransform(translationX: 0.0, y: targetSize.height)
transform = transform.scaledBy(x: 1, y: -1)
context.cgContext.concatenate(transform)
// UIImage -> cropped CGImage
let makeCroppedCGImage: (UIImage) -> CGImage? = { image in
guard let cgImage = image.cgImage else { return nil }
guard !fromRect.isEmpty else { return cgImage }
return cgImage.cropping(to: fromRect)
}
if let cgImage = makeCroppedCGImage(image) {
context.cgContext.draw(cgImage, in: CGRect(origin: .zero, size: targetSize))
}
else if let ciImage = image.ciImage {
var transform = CGAffineTransform(translationX: 0.0, y: image.size.height)
transform = transform.scaledBy(x: 1, y: -1)
let adjustedFromRect = fromRect.applying(transform)
let ciContext = CIContext(cgContext: context.cgContext, options: nil)
ciContext.draw(ciImage, in: CGRect(origin: .zero, size: targetSize), from: adjustedFromRect)
}
}
}
}

Summary

While UIImageView supports cropping and scaling images, we’ll need to do this on our own. It becomes handy especially when dealing with larger images.

If this was helpful, please let me know on Twitter @toomasvahter. Feel free to subscribe to RSS feed. Thank you for reading.

Categories
iOS Swift UIKit

Displaying images efficiently on iOS

Loading an image and displaying it on a screen consists of several steps. Firstly, we need to load the image data into memory, then decoding it to pixel data and finally, telling GPU to display it on screen. The whole process can be as short as two lines of code: creating an instance of UIImage using the name of the image and then assigning it to an UIImageView. Simple, but not so efficient.

Memory consumption impacts

Memory management is important topic as misusing memory can lead to, in worse case, system terminating our app. In addition, using too much memory will  cause high system CPU usage due to it trying to make more memory available by compressing it. Moreover, high CPU will lead to shorter battery life and no-one is happy about it.

High memory usage can be caused by keeping whole images in the memory and letting GPU to downscale it. The more efficient approach is to create a thumbnail with the size of the image view. This approach will use the minimum amount of pixel data and therefore system will use less resources.

Creating a thumbnail

For keeping resource consumption low, lets create UIImage extension for loading and creating the image at URL with specified size.

extension UIImage {
convenience init?(thumbnailOfURL url: URL, size: CGSize, scale: CGFloat) {
let options = [kCGImageSourceShouldCache: false] as CFDictionary
guard let source = CGImageSourceCreateWithURL(url as CFURL, options) else { return nil }
let targetDimension = max(size.width, size.height) * scale
let thumbnailOptions = [kCGImageSourceCreateThumbnailFromImageAlways: true,
kCGImageSourceCreateThumbnailWithTransform: true,
kCGImageSourceShouldCacheImmediately: true,
kCGImageSourceThumbnailMaxPixelSize: targetDimension] as CFDictionary
guard let thumbnail = CGImageSourceCreateThumbnailAtIndex(source, 0, thumbnailOptions) else { return nil }
self.init(cgImage: thumbnail)
}
}

Thumbnail creation consists of a couple of steps. Firstly, we create an instance of CGImageSource and tell it not to load and decode the data immediately (by setting kCGImageSourceShouldCacheImmediately to false). Instead, we pass the source into the thumbnail creation method which will immediately process the image data and scale it to the appropriate size. This approach avoids keeping the whole image in memory and instead, just uses the unscaled version.

private func loadThumbnailImage() {
let size = imageView.bounds.size
let scale = traitCollection.displayScale
let url = Bundle.main.url(forResource: "Wallpaper", withExtension: "jpg")!
DispatchQueue.global(qos: .userInitiated).async {
let image = UIImage(thumbnailOfURL: url, size: size, scale: scale)!
DispatchQueue.main.async { [weak self] in
self?.imageView.image = image
}
}
}

In a bit extreme example: displaying a thumbnail of a 5120 by 2880 pixels JPG image makes the app’s memory usage to be around 7 MB compared to 28 MB when the whole image is in memory. But on the other hand, we can have an app with multiple image views, each of them displaying a much larger image. Depending on the app, the difference can be huge.

Summary

We took a look at issues what can be caused by excessive use of system resources when displaying images. Then, we added an extension to UIImage for loading a larger image and scaling it to the size it is going to displayed. Small change, but has a huge impact.

If this was helpful, please let me know on Twitter @toomasvahter. Feel free to subscribe to RSS feed. Thank you for reading.

Example project

UIImageThumbnail (GitHub) Xcode 10.1, Swift 4.2

Resources

Image and Graphics Best Practices (Apple)

Categories
iOS Swift UIKit

Creating persistent data store on iOS

Storing data persistently on iOS is something what is needed quite often. In this post, we are going to look into how to build a persistent data store and how to store image data.

Initialising the persistent data store

Persistent data store is an object managing a folder on disk. It allows writing and reading data asynchronously.
Firstly, we need to create a folder where to store all the files. As every instance of the data store should manage its own folder, we will add an argument name to the initialiser. Then we can create a folder in user’s documents folder with that name. As writing and reading data is an expensive operation, we are going to offload the work to a concurrent DispatchQueue. Concurrent dispatch queue allows us to read multiple files at the same time (more about it a bit later).

final class PersistentDataStore {
let name: String
private let dataStoreURL: URL
private let queue: DispatchQueue
init(name: String) throws {
self.name = name
queue = DispatchQueue(label: "com.augmentedcode.persistentdatastore", qos: .userInitiated, attributes: .concurrent, autoreleaseFrequency: .workItem)
let documentsURL = try FileManager.default.url(for: .documentDirectory, in: .userDomainMask, appropriateFor: nil, create: false)
dataStoreURL = documentsURL.appendingPathComponent(name, isDirectory: true)
try FileManager.default.createDirectory(at: dataStoreURL, withIntermediateDirectories: true, attributes: nil)
}
}

Storing data asynchronously

Method for storing data on disk consists of closure, identifier and completion handler. This allows us to create a closure what transforms object to data. For example, it could transform UIImage to Data. Secondly, this transformation, possibly slow operation, can be offloaded to the same thread writing the data into a file. Using closure gives us a flexible API what we can extend with convenience methods.

typealias Identifier = String
enum Result {
case failed(Error)
case noData
case success(Identifier)
}
func storeData(_ dataProvider: @escaping () -> (Data?), identifier: Identifier = UUID().uuidString, completionHandler block: @escaping (Result) -> ()) {
queue.async(flags: .barrier) {
let url = self.url(forIdentifier: identifier)
guard let data = dataProvider(), !data.isEmpty else {
DispatchQueue.main.async {
block(.noData)
}
return
}
do {
try data.write(to: url, options: .atomic)
DispatchQueue.main.async {
block(.success(identifier))
}
}
catch {
DispatchQueue.main.async {
block(.failed(error))
}
}
}
}
// Example (adding data to data store with unique identifier):
persistentStore.storeData({ () -> (Data?) in
return image.jpegData(compressionQuality: 1.0)
}) { (result) in
switch result {
case .success(let identifier):
print("Stored data successfully with identifier \(identifier).")
case .noData:
print("No data to store.")
case .failed(let error):
print("Failed storing data with error \(error)")
}
}

Identifier is internally used as a filename and default implementation creates unique identifier. Therefore, when data store consumer would like to replace the current file, it can supply an identifier, otherwise new file is created.
Completion handler contains a Result enum type. Result enum consists of three cases: success, transformation failure and data writing failure. Success’ associated value is identifier, failure contains error object and transformation failure is equal to noData.
Important to note here is that the work item has barrier specified. Barrier means that when DispatchQueue starts to handle the work item, it will wait until all the previous work items have finished running. Meaning, we will never try to update a file on disk when some other request is busy reading it.

Loading data asynchronously

Load data is generic method allowing the data transformation closure to return a specific type (e.g. transforming Data to UIImage). Shortly, load data reads file from disk and transforms it into a different type. As transformation can be a lengthy task, it is yet again running on the background thread and will not cause any hiccups in the UI.

func loadData<T>(forIdentifier identifier: Identifier, dataTransformer: @escaping (Data) -> (T?), completionHandler block: @escaping (T?) -> ()) {
queue.async {
let url = self.url(forIdentifier: identifier)
guard FileManager.default.fileExists(atPath: url.path) else {
DispatchQueue.main.async {
block(nil)
}
return
}
do {
let data = try Data(contentsOf: url, options: .mappedIfSafe)
let object = dataTransformer(data)
DispatchQueue.main.async {
block(object)
}
}
catch {
print("Failed reading data at URL \(url).")
DispatchQueue.main.async {
block(nil)
}
}
}
}
// Example
persistentStore.loadData(forIdentifier: "my_identifier", dataTransformer: { UIImage(data: $0) }) { (image) in
guard let image = image else {
print("Failed loading image.")
return
}
print(image)
}

Removing data asynchronously

Removing a single file or all of the files is pretty straight-forward. As we are modifying files on disk, we will use barrier again and then FileManager’s removeItem(at:) together with contentsOfDirectory(at:includingPropertiesForKeys:options:).

func removeData(forIdentifier identifier: Identifier) {
queue.async(flags: .barrier) {
let url = self.url(forIdentifier: identifier)
guard FileManager.default.fileExists(atPath: url.path) else { return }
do {
try FileManager.default.removeItem(at: url)
}
catch {
print("Failed removing file at URL \(url) with error \(error).")
}
}
}
func removeAll() {
queue.async(flags: .barrier) {
do {
let urls = try FileManager.default.contentsOfDirectory(at: self.dataStoreURL, includingPropertiesForKeys: nil, options: [])
try urls.forEach({ try FileManager.default.removeItem(at: $0) })
}
catch {
print("Failed removing all files with error \(error).")
}
}
}

Extension for storing images

It is easy to extend the PersistentDataStore with convenience methods for storing a specific type of data. This allows us to hide the technical details of transforming image to data and vice-versa. Moreover, calling the method gets easier to read as data transformation closure is not visible anymore.

extension PersistentDataStore {
func loadImage(forIdentifier identifier: Identifier, completionHandler block: @escaping (UIImage?) -> (Void)) {
loadData(forIdentifier: identifier, dataTransformer: { UIImage(data: $0) }, completionHandler: block)
}
func storeImage(_ image: UIImage, identifier: String = UUID().uuidString, completionHandler handler: @escaping (Result) -> ()) {
storeData({ image.jpegData(compressionQuality: 1.0) }, identifier: identifier, completionHandler: handler)
}
}
// Examples:
persistentStore.storeImage(image) { (result) in
print(result)
}
persistentStore.loadImage(forIdentifier: "my_identifier") { (image) -> (Void) in
guard let image = image else {
print("Failed loading image.")
return
}
print(image)
}

Summary

We created a persistent data store what is performant and has a flexible API. API can be extended easily to support any other data transformation. In addition, it uses thread-safe techniques for making sure data never gets corrupted.

Playground

PersistentDataStore (GitHub) Xcode 10, Swift 4.2

References

DispatchQueues (Apple)
dispatch_barrier_async (Apple)