Fix Delayed::Job probe crash on deserialization errors
#492
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
Previously, the `Delayed::Job` probe eagerly deserializes the `payload_object` in order to name the trace. This moves the work ahead of where it would normally happen, and in case of an error during deserialization (e.g. syntax error in the YAML payload, or a deleted database record as in the customer issue), the error will be raised outside of the block where it's normally be caught. This fix defers the naming until a spot where we can guarantee DJ's successful deserialization, preventing the crashes while capturing the deserialization (database load) time within the trace. If an error occurs prior to reaching that step, the trace will have a default "unknown" label. It is possible that this would affect some traces that we previously were able to name successfully – specifically, if an error occurs in a lifecycle hook. If this turns out to be an issue, we can try and opportunistically check for `@payload_object` on the `job` instance during the `:error` and `:failure` hook, but this is reaching too deep into the implementation internals for my taste. And honestly it seems fair enough to categorize anything that never made it to the actual `#perform` as generic "unknown+error" traces. Fixes #491
chancancode
force-pushed
the
deplayed_job
branch
from
c902c62 to
ddca4f7
Compare
zvkemp
approved these changes
Aug 27, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, the
Delayed::Jobprobe eagerly deserializes thepayload_objectin order to name the trace. This moves the work ahead of where it would normally happen, and in case of an error during deserialization (e.g. syntax error in the YAML payload, or a deleted database record as in the customer issue), the error will be raised outside of the block where it's normally be caught.This fix defers the naming until a spot where we can guarantee DJ's successful deserialization, preventing the crashes while capturing the deserialization (database load) time within the trace.
If an error occurs prior to reaching that step, the trace will have a default "unknown" label.
It is possible that this would affect some traces that we previously were able to name successfully – specifically, if an error occurs in a lifecycle hook. If this turns out to be an issue, we can try and opportunistically check for
@payload_objecton thejobinstance during the:errorand:failurehook, but this is reaching too deep into the implementation internals for my taste. And honestly it seems fair enough to categorize anything that never made it to the actual#performas generic "unknown+error" traces.Fixes #491