Skip to content

FrameProcessor protocol #865

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pblazej wants to merge 3 commits into from
Closed

FrameProcessor protocol #865

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
312424f
FrameProcessor protocol
pblazej Dec 9, 2025
6d5d7a6
Move and respect isEnabled value
pblazej Dec 9, 2025
76c1ebe
Alternative solution with typealias
pblazej Dec 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion Sources/LiveKit/Agent/SessionOptions.swift
View file
Open in desktop
Copy link
Contributor Author

@pblazej pblazej Dec 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The minimal implementation is:

    private static let noiseFilter = AICAudioFilter()
    private let session = Session(
        tokenSource: ...,
        options: SessionOptions(audioProcessor: Self.noiseFilter)
    )

Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,19 @@ public struct SessionOptions: Sendable {
public init(
room: Room = .init(),
preConnectAudio: Bool = true,
agentConnectTimeout: TimeInterval = 20
agentConnectTimeout: TimeInterval = 20,
audioProcessor: AudioFrameProcessor? = nil,
videoProcessor: VideoFrameProcessor? = nil
) {
self.room = room
self.preConnectAudio = preConnectAudio
self.agentConnectTimeout = agentConnectTimeout

if let audioProcessor {
room.add(audioFrameProcessor: audioProcessor)
}
if let videoProcessor {
room.add(videoFrameProcessor: videoProcessor)
}
}
}
30 changes: 30 additions & 0 deletions Sources/LiveKit/Core/Room+FrameProcessor.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
/*
* Copyright 2025 LiveKit
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

import Foundation

@objc
public extension Room {
func add(audioFrameProcessor: AudioFrameProcessor) {
add(delegate: audioFrameProcessor)
AudioManager.shared.capturePostProcessingDelegate = audioFrameProcessor
}

func add(videoFrameProcessor: VideoFrameProcessor) {
add(delegate: videoFrameProcessor)
(localParticipant.firstCameraVideoTrack as? LocalVideoTrack)?.capturer.processor = videoFrameProcessor
}
}
10 changes: 9 additions & 1 deletion Sources/LiveKit/Protocols/AudioCustomProcessingDelegate.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,14 @@ internal import LiveKitWebRTC

public let kLiveKitKrispAudioProcessorName = "livekit_krisp_noise_cancellation"

public typealias AudioFrameProcessor = AudioCustomProcessingDelegate & RoomDelegate

/// Used to modify audio buffers before they are sent to the network or played to the user
@objc
public protocol AudioCustomProcessingDelegate: Sendable {
@objc
var isEnabled: Bool { get set }
Copy link
Contributor Author

@pblazej pblazej Dec 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This IS a breaking change for Krisp, we can go for @objc optional but then this requirement does not make much sense to me.

Copy link
Contributor

@lukasIO lukasIO Dec 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the alternative would be to not define it here and keep isEnabled on the processor implementations, right?

Maybe that's preferable if the alternative is a breaking change

Copy link
Contributor Author

@pblazej pblazej Dec 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can also use @objc optional annotation, but then you get Bool? you need to default to ?? true for legacy things like Krisp.

Copy link
Contributor Author

@pblazej pblazej Dec 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The motivation here is obviously to avoid casting in the audio/video thread.

Copy link
Contributor

@lukasIO lukasIO Dec 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but implementations could still implement an isEnabled check in the process function and just return early?

Copy link
Contributor Author

@pblazej pblazej Dec 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or should WE do it internally (not passing frames) (see below)


/// An optional identifier for the audio processor implementation.
/// This can be used to identify different types of audio processing (e.g. noise cancellation).
/// Generally you can leave this as the default value.
Expand Down Expand Up @@ -112,7 +117,10 @@ class AudioCustomProcessingDelegateAdapter: MulticastDelegate<AudioRenderer>, @u

func audioProcessingProcess(audioBuffer: LKRTCAudioBuffer) {
let lkAudioBuffer = LKAudioBuffer(audioBuffer: audioBuffer)
target?.audioProcessingProcess(audioBuffer: lkAudioBuffer)

if let target, target.isEnabled {
target.audioProcessingProcess(audioBuffer: lkAudioBuffer)
}

// Convert to pcmBuffer and notify only if an audioRenderer is added.
if isDelegatesNotEmpty, let pcmBuffer = lkAudioBuffer.toAVAudioPCMBuffer() {
Expand Down
4 changes: 4 additions & 0 deletions Sources/LiveKit/Protocols/VideoProcessor.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,11 @@

import Foundation

public typealias VideoFrameProcessor = RoomDelegate & VideoProcessor

@objc
public protocol VideoProcessor {
var isEnabled: Bool { get set }

func process(frame: VideoFrame) -> VideoFrame?
}
2 changes: 1 addition & 1 deletion Sources/LiveKit/Track/Capturers/VideoCapturer.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -313,7 +313,7 @@ extension VideoCapturer {
}

// Apply processing if we have a processor attached.
if let processor = _state.processor {
if let processor = _state.processor, processor.isEnabled {
guard let processedFrame = processor.process(frame: lkFrame) else {
log("VideoProcessor didn't return a frame, skipping frame.", .warning)
return
Expand Down
4 changes: 3 additions & 1 deletion Sources/LiveKit/VideoProcessors/BackgroundBlurVideoProcessor.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ import os.signpost
/// - Important: This class is not thread safe and will be called on a dedicated serial `processingQueue`.
@available(iOS 15.0, macOS 12.0, tvOS 15.0, visionOS 1.0, *)
@objc
public final class BackgroundBlurVideoProcessor: NSObject, @unchecked Sendable, VideoProcessor, Loggable {
public final class BackgroundBlurVideoProcessor: NSObject, @unchecked Sendable, VideoFrameProcessor, Loggable {
#if LK_SIGNPOSTS
private let signpostLog = OSLog(subsystem: Bundle.main.bundleIdentifier ?? "", category: "BackgroundBlur")
#endif
Expand Down Expand Up @@ -69,6 +69,8 @@ public final class BackgroundBlurVideoProcessor: NSObject, @unchecked Sendable,
private var cachedPixelBuffer: CVPixelBuffer?
private var cachedPixelBufferSize: CGSize?

public var isEnabled: Bool = true

// MARK: Init

/// Initialize the background blur video processor.
Expand Down
Loading