SpriteKit Offline Rendering

,

This project shows how to export Metal-rendered content to an image sequence or H.264 video, using SpriteKit, SKRenderer, Metal, IOSurface, and AVFoundation.

GitHubRepository

Context

The use case I had in mind was recording user-generated content. Here's the scenario:

The project grew into a SpriteKit offline renderer that uses several Apple frameworks and technologies. It demonstrates how to:

The project does not focus on the UX of exporting and sharing media files. Instead, it shows how to build a backend that consumes Metal content and produces flat media files.

Demo

Download SKRenderer Demo from GitHub and open the project in Xcode. The demo app runs in Xcode Live Preview, Simulator, iOS, and Mac Catalyst.

When rendering starts, the live SKView is paused to free resources for the offline renderer. When rendering completes:

Export to video. The app is running in Xcode live canvas. The user selects a duration and presses Render. The offline renderer finishes. The user navigates to the file using the path printed in Xcode console and finds the video.
Export to PNG sequence. The app is running in Xcode live canvas. The user selects PNGs as output format, then presses Render. The user navigates to the folder using the path printed in Xcode console and finds the images.
SKRenderer-Demo-XcodeConsole.png
Xcode console output. During and after rendering, the app prints information to the console: the render settings, the render progress, the output file path, and the existing files in the work folder.

Read below about how the project is set up.

SKRenderer

SKRenderer takes a SpriteKit scene and outputs a Metal texture. The texture can be used in a Metal pipeline:

What SKRenderer is not:

Setup

Below is the boilerplate setup done once when SKRenderer is created:

// Get the GPU

let device = MTLCreateSystemDefaultDevice()

// Factory for creating command buffers, used later each frame
// Command buffers = instructions for the GPU

let commandQueue = device.makeCommandQueue()

// Allocate GPU memory for the texture we'll render into
// Texture = a block of GPU memory holding pixels
// The memory allocation stays constant (let), the pixel data changes each frame

let textureDesc = MTLTextureDescriptor()
textureDesc.width = pixelWidth
textureDesc.height = pixelHeight
let renderTexture = device.makeTexture(descriptor: textureDesc)

// Create an SKRenderer instance and assign a scene to render

let renderer = SKRenderer(device: device)
renderer.scene = scene

Then for each frame, we run code in the following form:

// Update scene
// This calls all SKScene delegate functions, from update to didFinishUpdate

renderer.update(atTime: currentTime)

// Configure the rendering operation for this frame
// Set the created texture as the render target and specifies clear/store actions

let renderPassDescriptor = MTLRenderPassDescriptor()
renderPassDescriptor.colorAttachments[0].texture = renderTexture
renderPassDescriptor.colorAttachments[0].loadAction = .clear
renderPassDescriptor.colorAttachments[0].storeAction = .store

// Create a command buffer to hold this frame's GPU instructions

let commandBuffer = commandQueue.makeCommandBuffer()

// Viewport is required by the API but appears ignored by SKRenderer in this context
// The texture dimensions determine the actual output size

let viewport = CGRect(origin: .zero, size: sceneSize)

// Render the scene into the texture
// SKRenderer writes drawing commands into commandBuffer

renderer.render(
    withViewport: viewport,
    commandBuffer: commandBuffer,
    renderPassDescriptor: renderPassDescriptor
)

// GPU work is asynchronous with CPU work
// commit() submits work but returns immediately
// completion handler ensures GPU work is done for this frame

commandBuffer.addCompletedHandler {
    /// Do something with the completed texture, like image encoding
    encodeFrame(from: texture)
}

// Send the command buffer to GPU for execution

commandBuffer.commit()

Resolution and Scale Factor

A SpriteKit scene is sized in points. A Metal texture is sized in pixels. If a scene is created at 1920x1080 and SKRenderer draws it, the output will be 1920x1080 pixels. In order to get Retina resolution, we must multiply the size of the allocated texture by a scale factor. SKRenderer will handle the mapping between the point-based scene and the pixel-based texture. This is reminiscent of UIView's contentScaleFactor property.

let scene = SKScene(size: CGSize(width: 1920, height: 1080))

// Scale allocated texture before rendering

let textureDesc = MTLTextureDescriptor()
textureDesc.width = Int(1920 * renderScale)  // @3x: 5760 pixels
textureDesc.height = Int(1080 * renderScale)  // @3x: 3240 pixels

renderer.render(...)

Known Issues and Workarounds

HiDPI scaling doesn't work for all nodes. SKShapeNode with antialiasing enabled renders at @1x no matter the resolution of the Metal texture descriptor, and therefore will appear blurry at Retina scales. A workaround is to use supersampling: create shapes upsized by the scale factor, then scale them down:

let scaleFactor: CGFloat = 3 // iPhone Retina scale

let shape = SKShapeNode(rectOf: CGSize(width: 150 * scaleFactor, height: 75 * scaleFactor))
shape.lineWidth = 3 * scaleFactor
shape.setScale(1/scaleFactor)

// Physics body must match final size, not supersampled size

shape.physicsBody = SKPhysicsBody(rectangleOf: CGSize(width: 150, height: 75))

An alternative is to set isAntialiased = false on shape nodes, which will force SKRenderer to draw them at the correct resolution, but curves will appear jagged.

Textures created programmatically should also be scaled to match the Retina target. Pass the scale factor to the generator and scale texture creation accordingly:

let textureSize = CGSize(width: 2 * scaleFactor, height: 2 * scaleFactor)

// Generate a texture with Core Graphics

let cgRenderer = UIGraphicsImageRenderer(size: textureSize)
let squareTexture = SKTexture(image: cgRenderer.image { context in
    SKColor.white.setFill()
    context.fill(CGRect(origin: .zero, size: textureSize))
})

Another issue you may encounter is the renderer crashing when Core Image filters are activated. A workaround is to disable Metal API Validation in Product > Scheme > Edit Scheme > Diagnostics.

Media Export

Here's how each export pipeline works:

Export to PNG

The getBytes() method is convenient, but slow for high-performance needs. For high frame rates or large textures, PNG export is slower than video encoding due to this CPU copy and per-frame PNG compression.

Export to Video

Video export uses IOSurface, which is a low-level memory management framework. IOSurface provides memory that both GPU and CPU can access quickly.

The blit is fast. The memcpy from IOSurface to CVPixelBuffer is faster than getBytes() from a Metal texture. Compared to PNG's getBytes(), this pipeline reduces per-frame overhead from ~5ms to <1ms on iPhone 13 at 1080p.

Update Timestep

SpriteKit's update function takes a current time value, not a delta time. When the scene is updated with SKRenderer update(atTime:) method, the correct value must be passed. I found that time values must start from a CACurrentMediaTime() and not 0. Then, each tick, add a delta time:

var currentTime: TimeInterval = CACurrentMediaTime()
let deltaTime = 1.0 / fps

for frame in 0..<totalFrames {
	currentTime += deltaTime
    renderer.update(atTime: currentTime)
}

This time setup is called "monotonically increasing", i.e. the current time is positive and always increasing. I explored various delta time values to understand SpriteKit's internal clock for update, actions, and physics. Below are my findings. In each of these scenarios, current time starts at CACurrentMediaTime().

Backwards

A delta time is subtracted from the current time each frame:

Frozen

The same current time value is passed every frame:

Speed Control

A multiple of delta time is added every frame, and scene.physicsWorld.speed is set at different values.

Delta time * 10, physicsWorld.speed = 1:

Delta time * 10, physicsWorld.speed = 0.1:

physicsWorld.speed = 0:

Conclusion

Simulation Rollback

In order to record a specific segment of the simulation, the SpriteKit scene must be set up to recover a given state and replay the simulation. Typically this means having a deterministic state initializer + a command pattern on top of SpriteKit:

// Live interaction mutates the scene by issuing commands:

run(Command.create(..))
run(Command.move(..))

// The same interaction can be reproduced later:

history = [
    Action(time: 1.0, command: .create(...)),
    Action(time: 1.5, command: .move(...)),
    // ...
]

A recording pass would be implemented like this:

This enables capturing complex simulations at any resolution and frame rate, fully decoupled from real-time display limits.

Determinism

Interaction and behavior must be deterministic for frame-perfect replay. Consider the figure below: each render is from the same scene, and each image is the 500th frame of a 10 seconds simulation.

SKRenderer-determinism.png
Determinism test. Two runs of the same scene produce diverging results in the bouncing orange balls.

From empirical testing, I found the following to be deterministic:

I found the following to not be deterministic, despite the fixed time step supplied to the renderer:

If your setup depends on precise physics body positions interacting over multiple seconds, use guide rails to direct behavior, such as careful level design and checkpoints.

License

This project is licensed under the Apache License 2.0.

If this project helps your work, attribution or a link back is appreciated: https://www.achrafkassioui.com/spritekit-offline-rendering/