Camera Capture with AVFoundation

Quick Patterns

Basic Session Setup

import AVFoundation

let session = AVCaptureSession()
let photoOutput = AVCapturePhotoOutput()
let sessionQueue = DispatchQueue(label: "camera.session")

sessionQueue.async {
    session.beginConfiguration()
    defer { session.commitConfiguration() }

    session.sessionPreset = .photo

    guard let camera = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: .back),
          let input = try? AVCaptureDeviceInput(device: camera),
          session.canAddInput(input) else { return }
    session.addInput(input)

    guard session.canAddOutput(photoOutput) else { return }
    session.addOutput(photoOutput)
    photoOutput.maxPhotoQualityPrioritization = .quality

    session.startRunning() // Blocking -- never on main thread
}

SwiftUI Camera Preview

struct CameraPreview: UIViewRepresentable {
    let session: AVCaptureSession

    func makeUIView(context: Context) -> PreviewView {
        let view = PreviewView()
        view.previewLayer.session = session
        view.previewLayer.videoGravity = .resizeAspectFill
        return view
    }
    func updateUIView(_ uiView: PreviewView, context: Context) {}

    class PreviewView: UIView {
        override class var layerClass: AnyClass { AVCaptureVideoPreviewLayer.self }
        var previewLayer: AVCaptureVideoPreviewLayer { layer as! AVCaptureVideoPreviewLayer }
    }
}

Rotation Handling (iOS 17+)

let coordinator = AVCaptureDevice.RotationCoordinator(device: camera, previewLayer: previewLayer)
previewLayer.connection?.videoRotationAngle = coordinator.videoRotationAngleForHorizonLevelPreview

let observation = coordinator.observe(\.videoRotationAngleForHorizonLevelPreview, options: [.new]) { [weak previewLayer] coord, _ in
    DispatchQueue.main.async {
        previewLayer?.connection?.videoRotationAngle = coord.videoRotationAngleForHorizonLevelPreview
    }
}

Capture Photo with Rotation

func capturePhoto() {
    var settings = AVCapturePhotoSettings()
    settings.photoQualityPrioritization = .balanced
    if let connection = photoOutput.connection(with: .video) {
        connection.videoRotationAngle = coordinator.videoRotationAngleForHorizonLevelCapture
    }
    photoOutput.capturePhoto(with: settings, delegate: self)
}

Permission Request

func requestCameraAccess() async -> Bool {
    let status = AVCaptureDevice.authorizationStatus(for: .video)
    switch status {
    case .authorized: return true
    case .notDetermined: return await AVCaptureDevice.requestAccess(for: .video)
    case .denied, .restricted: return false
    @unknown default: return false
    }
}
// Info.plist: NSCameraUsageDescription, NSMicrophoneUsageDescription (for video)

---

Decision Tree

What do you need?
|
+-- Just let user pick a photo?
|   -> PHPicker or PhotosPicker, not AVFoundation
|
+-- Simple photo/video with system UI?
|   -> UIImagePickerController (limited customization)
|
+-- Custom camera UI?
|   +-- Photo capture -> AVCaptureSession + AVCapturePhotoOutput
|   +-- Video recording -> AVCaptureSession + AVCaptureMovieFileOutput
|   +-- Both -> AVCaptureSession + both outputs
|
+-- Rotation handling?
|   +-- iOS 17+ -> RotationCoordinator (automatic gravity tracking)
|   +-- Pre-iOS 17 -> Deprecated videoOrientation (manual observation)
|
+-- Capture feels slow?
|   +-- 2+ second delay -> Set photoQualityPrioritization = .speed or .balanced
|   +-- Want zero-shutter-lag -> Enable responsive capture (iOS 17+)
|   +-- Maximum responsiveness -> Enable deferred processing (iOS 17+)
|
+-- Camera freezes/stops?
|   +-- On phone call -> Add interruption observers
|   +-- In Split View -> Camera unavailable with multiple foreground apps
|   +-- After prolonged use -> Thermal pressure
|
+-- Issue / not working?
    -> See Diagnostics section below

---

Anti-Patterns

Calling startRunning() on main thread

startRunning() is blocking (1-3 seconds). Always call on a dedicated serial queue. Blocking main thread freezes UI.

Using deprecated videoOrientation (iOS 17+)

Manual device orientation observation misses edge cases (face-up, face-down). Use AVCaptureDevice.RotationCoordinator which automatically tracks gravity.

Ignoring session interruptions

Without handling .AVCaptureSessionWasInterrupted, camera appears frozen when phone calls arrive, Split View activates, or thermal pressure occurs. Session auto-resumes after interruption ends -- just update UI.

Modifying session without configuration block

Calling addInput/removeInput without beginConfiguration()/commitConfiguration() may leave session in invalid state between calls. Always batch changes atomically.

Creating new AVCaptureSession per capture

Session creation is expensive. Create once, keep reference, reuse across captures and view lifecycle.

Using .photo preset for video

Wrong format for video recording. Use .high, .hd1920x1080, or .hd4K3840x2160 for video.

Ignoring photoQualityPrioritization

Default .quality causes 2+ second capture delay. Use .speed for social/sharing, .balanced for general use.

Not checking canAddInput/canAddOutput

Adding incompatible inputs/outputs fails silently. Always check session.canAddInput() / session.canAddOutput() before adding.

---

Deep Patterns

Session Architecture

AVCaptureSession
    +-- Inputs
    |   +-- AVCaptureDeviceInput (camera)
    |   +-- AVCaptureDeviceInput (microphone, for video)
    +-- Outputs
    |   +-- AVCapturePhotoOutput (photos)
    |   +-- AVCaptureMovieFileOutput (video files)
    |   +-- AVCaptureVideoDataOutput (raw frames)
    +-- Connections (automatic between compatible input/output)

All session configuration happens on a dedicated serial queue, never main thread. Configuration changes wrapped in beginConfiguration()/commitConfiguration() for atomic updates.

Session Presets

Preset	Resolution	Use Case
.photo	Optimal for photos	Photo capture
.high	Highest device quality	Video recording
.medium	VGA quality	Preview, lower storage
.hd1280x720	720p	HD video
.hd1920x1080	1080p	Full HD video
.hd4K3840x2160	4K	Ultra HD video
.inputPriority	Use device format	Custom configuration

Session Notifications

// Interrupted (phone call, Split View, thermal)
NotificationCenter.default.addObserver(forName: .AVCaptureSessionWasInterrupted, object: session, queue: .main) { notification in
    let reason = notification.userInfo?[AVCaptureSessionInterruptionReasonKey] as? Int
}

// Interruption ended (auto-resumes, just update UI)
NotificationCenter.default.addObserver(forName: .AVCaptureSessionInterruptionEnded, object: session, queue: .main) { _ in }

// Runtime error
NotificationCenter.default.addObserver(forName: .AVCaptureSessionRuntimeError, object: session, queue: .main) { notification in
    let error = notification.userInfo?[AVCaptureSessionErrorKey] as? Error
}

Interruption Reasons

Reason	Cause
.videoDeviceNotAvailableInBackground	App went to background
.audioDeviceInUseByAnotherClient	Another app using audio
.videoDeviceInUseByAnotherClient	Another app using camera
.videoDeviceNotAvailableWithMultipleForegroundApps	Split View (iPad)
.videoDeviceNotAvailableDueToSystemPressure	Thermal throttling

Responsive Capture Pipeline (iOS 17+)

Four complementary APIs for maximum capture responsiveness:

Zero Shutter Lag: Ring buffer of recent frames captures the exact moment of tap. Enabled by default for iOS 17+ apps. iPhone XS+ required. Does not apply to flash, manual exposure, or bracketed captures.

photoOutput.isZeroShutterLagSupported // check first
photoOutput.isZeroShutterLagEnabled   // true by default for iOS 17+

Responsive Capture (Overlapping): New capture starts while previous still processing. Increases peak memory.

if photoOutput.isResponsiveCaptureSupported {
    photoOutput.isResponsiveCaptureEnabled = true
}

Fast Capture Prioritization: Adapts quality for rapid sequential captures. Off by default -- reduces quality.

if photoOutput.isFastCapturePrioritizationSupported {
    photoOutput.isFastCapturePrioritizationEnabled = true
}

Readiness Coordinator: Synchronous shutter button state updates without async lag.

let readinessCoordinator = AVCapturePhotoOutputReadinessCoordinator(photoOutput: photoOutput)
readinessCoordinator.delegate = self

// Call BEFORE capturePhoto()
readinessCoordinator.startTrackingCaptureRequest(using: settings)
photoOutput.capturePhoto(with: settings, delegate: self)

Delegate provides .ready, .notReadyMomentarily, .notReadyWaitingForCapture, .notReadyWaitingForProcessing, .sessionNotRunning.

Deferred Photo Processing (iOS 17+)

Returns immediately with proxy image; full Deep Fusion processing happens in background. iPhone 11 Pro+ required.

if photoOutput.isAutoDeferredPhotoDeliverySupported {
    photoOutput.isAutoDeferredPhotoDeliveryEnabled = true
}

Delegate callbacks:

// Standard photo (non-deferred)
func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {
    guard error == nil, let data = photo.fileDataRepresentation() else { return }
    savePhotoToLibrary(data)
}

// Deferred proxy -- save to PhotoKit ASAP before app is backgrounded
func photoOutput(_ output: AVCapturePhotoOutput,
                 didFinishCapturingDeferredPhotoProxy deferredPhotoProxy: AVCaptureDeferredPhotoProxy,
                 error: Error?) {
    guard error == nil, let proxyData = deferredPhotoProxy.fileDataRepresentation() else { return }
    Task {
        try await PHPhotoLibrary.shared().performChanges {
            let request = PHAssetCreationRequest.forAsset()
            request.addResource(with: .photoProxy, data: proxyData, options: nil)
        }
    }
}

Limitations: cannot apply pixel buffer customizations to deferred photos. Use PhotoKit adjustments after processing. Final processing happens on-demand or when device is idle.

Photo Quality Prioritization

Value	Speed	Quality	Use Case
.speed	Fastest	Lower	Social sharing, rapid capture
.balanced	Medium	Good	General photography
.quality	Slowest	Best	Documents, professional

Photo Settings

// Standard JPEG
var settings = AVCapturePhotoSettings()

// HEIF format
settings = AVCapturePhotoSettings(format: [AVVideoCodecKey: AVVideoCodecType.hevc])

// Flash
settings.flashMode = .auto // .off, .on, .auto

// High resolution
settings.isHighResolutionPhotoEnabled = true

// Thumbnail for immediate display
settings.embeddedThumbnailPhotoFormat = [AVVideoCodecKey: AVVideoCodecType.jpeg]

Settings cannot be reused -- create a new instance per capture. Use AVCapturePhotoSettings(from:) to copy.

Camera Switching

func switchCamera() {
    sessionQueue.async { [self] in
        guard let currentInput = session.inputs.first as? AVCaptureDeviceInput else { return }
        let newPosition: AVCaptureDevice.Position = currentInput.device.position == .back ? .front : .back

        guard let newDevice = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: newPosition),
              let newInput = try? AVCaptureDeviceInput(device: newDevice) else { return }

        session.beginConfiguration()
        defer { session.commitConfiguration() }

        session.removeInput(currentInput)
        if session.canAddInput(newInput) {
            session.addInput(newInput)
            // Update RotationCoordinator for new device
            setupRotationCoordinator(device: newDevice, previewLayer: previewLayer)
        } else {
            session.addInput(currentInput) // Fallback: restore old
        }
    }
}

Front camera mirroring: preview is mirrored (user expectation, like a mirror). Captured photo is NOT mirrored (text reads correctly when shared). This is intentional Apple behavior.

Video Recording

let movieOutput = AVCaptureMovieFileOutput()

// Setup: session preset .high, add microphone input + movie output
sessionQueue.async {
    session.beginConfiguration()
    defer { session.commitConfiguration() }

    session.sessionPreset = .high
    if let mic = AVCaptureDevice.default(for: .audio),
       let audioInput = try? AVCaptureDeviceInput(device: mic),
       session.canAddInput(audioInput) { session.addInput(audioInput) }
    if session.canAddOutput(movieOutput) { session.addOutput(movieOutput) }
}

// Start recording
let url = FileManager.default.temporaryDirectory.appendingPathComponent(UUID().uuidString + ".mov")
if let connection = movieOutput.connection(with: .video) {
    connection.videoRotationAngle = coordinator.videoRotationAngleForHorizonLevelCapture
}
movieOutput.startRecording(to: url, recordingDelegate: self)

// Stop recording
movieOutput.stopRecording()

Delegate: AVCaptureFileOutputRecordingDelegate provides didFinishRecordingTo with output URL and optional error.

Device Types

Type	Description
.builtInWideAngleCamera	Standard (1x)
.builtInUltraWideCamera	Ultra-wide (0.5x)
.builtInTelephotoCamera	Telephoto (2x, 3x)
.builtInDualCamera	Wide + telephoto
.builtInTripleCamera	Wide + ultra-wide + telephoto
.builtInTrueDepthCamera	Front TrueDepth (Face ID)
.builtInLiDARDepthCamera	LiDAR depth

Device Configuration

try device.lockForConfiguration()
defer { device.unlockForConfiguration() }

device.focusMode = .continuousAutoFocus       // if supported
device.exposureMode = .continuousAutoExposure // if supported
device.torchMode = .on                        // if hasTorch
device.videoZoomFactor = 2.0                  // zoom

Preview Layer Video Gravity

Value	Behavior
.resizeAspect	Fit entire image, may letterbox
.resizeAspectFill	Fill layer, may crop edges
.resize	Stretch to fill (distorts)

Complete CameraManager (Reference)

@MainActor
class CameraManager: NSObject, ObservableObject {
    let session = AVCaptureSession()
    let photoOutput = AVCapturePhotoOutput()
    private let sessionQueue = DispatchQueue(label: "camera.session")
    private var rotationCoordinator: AVCaptureDevice.RotationCoordinator?
    private var rotationObservation: NSKeyValueObservation?
    @Published var isSessionRunning = false

    func setup() async -> Bool {
        guard await AVCaptureDevice.requestAccess(for: .video) else { return false }
        return await withCheckedContinuation { continuation in
            sessionQueue.async { [self] in
                session.beginConfiguration()
                defer { session.commitConfiguration() }
                session.sessionPreset = .photo

                guard let camera = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: .back),
                      let input = try? AVCaptureDeviceInput(device: camera),
                      session.canAddInput(input) else { continuation.resume(returning: false); return }
                session.addInput(input)

                guard session.canAddOutput(photoOutput) else { continuation.resume(returning: false); return }
                session.addOutput(photoOutput)
                photoOutput.maxPhotoQualityPrioritization = .quality
                continuation.resume(returning: true)
            }
        }
    }

    func start() {
        sessionQueue.async { [self] in
            session.startRunning()
            DispatchQueue.main.async { self.isSessionRunning = self.session.isRunning }
        }
    }

    func stop() {
        sessionQueue.async { [self] in
            session.stopRunning()
            DispatchQueue.main.async { self.isSessionRunning = false }
        }
    }

    func capturePhoto() {
        var settings = AVCapturePhotoSettings()
        settings.photoQualityPrioritization = .balanced
        if let connection = photoOutput.connection(with: .video),
           let angle = rotationCoordinator?.videoRotationAngleForHorizonLevelCapture {
            connection.videoRotationAngle = angle
        }
        photoOutput.capturePhoto(with: settings, delegate: self)
    }
}

extension CameraManager: AVCapturePhotoCaptureDelegate {
    nonisolated func photoOutput(_ output: AVCapturePhotoOutput,
                                  didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {
        guard let data = photo.fileDataRepresentation() else { return }
        // Handle photo data
    }
}

---

Diagnostics

Root Causes (by frequency)

1. Threading (session work on main thread) -- 35%
2. Session lifecycle (not started, interrupted, not configured) -- 25%
3. Rotation (deprecated APIs, missing coordinator) -- 20%
4. Permissions (denied, not requested) -- 15%
5. Configuration (wrong preset, missing input/output) -- 5%

Quick Diagnostic Code

// 1. Session state
print("isRunning: \(session.isRunning)")
print("inputs: \(session.inputs.count), outputs: \(session.outputs.count)")

// 2. Threading
sessionQueue.async {
    print("Setup thread: \(Thread.isMainThread ? "MAIN (bad)" : "Background (good)")")
}

// 3. Permissions
let status = AVCaptureDevice.authorizationStatus(for: .video)
print("Camera permission: \(status.rawValue)") // 0=notDetermined, 1=restricted, 2=denied, 3=authorized

// 4. Interruptions
NotificationCenter.default.addObserver(forName: .AVCaptureSessionWasInterrupted, object: session, queue: .main) { notification in
    if let reason = notification.userInfo?[AVCaptureSessionInterruptionReasonKey] as? Int {
        print("Interrupted: reason \(reason)")
    }
}

Symptom Table

Symptom	Likely Cause	Fix
Black preview	Session not started, permission denied, no input	Check session state + permissions
Preview layer shows nothing	Layer not in hierarchy or frame is zero	Set previewLayer.session, verify frame
UI freezes when opening camera	startRunning() on main thread	Move to dedicated serial queue
Camera freezes on phone call	No interruption handling	Add .AVCaptureSessionWasInterrupted observer
Camera stops in Split View	Multiple foreground apps	Show "Camera unavailable in Split View" message
Camera stops after prolonged use	Thermal pressure	Reduce session preset to .medium, show message
Preview rotated 90 degrees wrong	Not using RotationCoordinator	Create RotationCoordinator, observe preview angle
Captured photo rotated wrong	Rotation angle not applied to output connection	Set connection.videoRotationAngle before capture
Front camera photo not mirrored	Correct behavior	Preview mirrors, photo does not (Apple standard)
Capture takes 2+ seconds	.quality prioritization	Use .speed or .balanced
Shutter button allows double-tap	No readiness tracking	Use ReadinessCoordinator (iOS 17+)
Permission denied with no prompt	Already denied, cannot re-prompt	Show Settings prompt with openSettingsURLString
Crash on older iOS	iOS 17+ APIs without availability check	Guard with if #available(iOS 17.0, *)

---

Related

• ax-vision -- Vision framework (feeds from camera capture for CV tasks)
• ax-foundation-models -- On-device AI (not camera)
• WWDC: 2021-10247, 2023-10105

Checklist Before Shipping

Session:

• All session work on dedicated serial queue
• startRunning() never on main thread
• Configuration changes in beginConfiguration()/commitConfiguration()
• Session preset matches use case

Permissions:

• Camera permission requested before session setup
• NSCameraUsageDescription in Info.plist
• NSMicrophoneUsageDescription if recording audio
• Graceful handling of denied state

Rotation:

• RotationCoordinator used (not deprecated videoOrientation)
• Preview layer rotation observed
• Capture rotation angle applied when taking photos
• Tested in all orientations

Responsiveness:

• photoQualityPrioritization set for use case
• Capture button shows immediate feedback
• Deferred processing considered for maximum speed

Interruptions:

• Session interruption observer registered
• UI feedback shown when interrupted
• Tested with incoming phone call
• Tested in Split View (iPad)

Camera Switching:

• Front/back switch updates rotation coordinator
• Switch happens on session queue with configuration block
• Fallback if new camera unavailable