Skip to content

Fix deadlock in dd-task-scheduler #10096

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
rkennke merged 12 commits into from
Dec 11, 2025
Merged

Fix deadlock in dd-task-scheduler #10096

rkennke merged 12 commits into from
Dec 11, 2025

Conversation

@rkennke
Copy link
Contributor

@rkennke rkennke commented Dec 8, 2025 •
edited
Loading

What Does This Do

This is a fix/workaround that addresses a potential deadlock in JFR. The deadlock has been reported to upstream, and a fix has been proposed there, but it would be good to also fix it in our own code for when the code is running on an unpatched JDK.

Upstream bug: JDK-8371889

Motivation

This change fixes a deadlock in the profiling agent.

Additional Notes

We are loading the JFR handlers class early. This will prevent the deadlock that happens when we call EventType.getEventType() from the SmapEventFactory static initializer. That JFR code would first lock the Utils class lock, and then later try to acquire the Handlers' class initializer lock. If another thread is currently in the static initializer of the Handlers class, then that thread would attempt to acquire the Utils class lock, and thus deadlock. Calling the Handlers static initializer early avoids that scenario by ensuring that the Handlers class is fully initialized before calling into the JFR code that would acquire the Utils lock.

The problem is only present in JDKs < 22, and the Handlers class has been renamed and moved starting in JDK22. That is why the class is loaded reflectively and any errors while loading the class are silently ignored.

Contributor Checklist

Jira ticket: PROF-13025

@rkennke rkennke added comp: profiling Profiling type: bug Bug report and fix labels Dec 8, 2025
@rkennke rkennke marked this pull request as ready for review December 8, 2025 11:52
@rkennke rkennke requested a review from a team as a code owner December 8, 2025 11:52
jbachorik
jbachorik reviewed Dec 8, 2025
View reviewed changes
...-openjdk/src/main/java/com/datadog/profiling/controller/openjdk/events/SmapEntryFactory.java Outdated
// Load JFR Handlers class early, if present (it has been moved and renamed in JDK23+).
// This prevents a deadlock. See PROF-13025.
try {
Class.forName("jdk.jfr.events.Handlers");
Copy link
Contributor

@jbachorik jbachorik Dec 8, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are touching this class only when we have JFR available, so loading the Handlers class should not be disturbing anything.
But - we have smap entry events disabled by default, yet we are getting these issues ... I have a suspicion that we are initing the support regardless of the enablement status. Could you check for PROFILING_SMAP_COLLECTION_ENABLED setting before merging this PR?

Copy link
Contributor Author

@rkennke rkennke Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be slighly more complex. The smap event is registered in datadog.trace.bootstrap.Agent and there we probably don't have easy access to PROFILING_SMAP_COLLECTION_ENABLED (not sure), and it might also affect other events. I filed PROF-13213 for a deeper investigation. I would merge this PR as-is. Is that ok?

Copy link
Contributor

@jbachorik jbachorik Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@pr-commenter
Copy link

pr-commenter bot commented Dec 8, 2025 •
edited
Loading

Benchmarks

⚠️ Warning: Baseline build not found for merge-base commit. Comparing against the latest commit on master instead.

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master PROF-13025
git_commit_date 1765384916 1765389405
git_commit_sha 2b04c87 99ec036
release_version 1.57.0-SNAPSHOT~2b04c87b81 1.57.0-SNAPSHOT~99ec0365ca
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1765391242 1765391242
ci_job_id 1284822789 1284822789
ci_pipeline_id 86006601 86006601
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-1-ob8p4swk 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-1-ob8p4swk 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 58 metrics, 7 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.57.0-SNAPSHOT~99ec0365ca, baseline=1.57.0-SNAPSHOT~2b04c87b81

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.079 s) : 0, 1079007
Total [baseline] (10.886 s) : 0, 10886271
Agent [candidate] (1.079 s) : 0, 1078682
Total [candidate] (10.849 s) : 0, 10848884
section appsec
Agent [baseline] (1.265 s) : 0, 1264719
Total [baseline] (11.183 s) : 0, 11183027
Agent [candidate] (1.264 s) : 0, 1264441
Total [candidate] (11.194 s) : 0, 11193692
section iast
Agent [baseline] (1.236 s) : 0, 1235624
Total [baseline] (11.289 s) : 0, 11289054
Agent [candidate] (1.222 s) : 0, 1222351
Total [candidate] (11.129 s) : 0, 11128717
section profiling
Agent [baseline] (1.203 s) : 0, 1202987
Total [baseline] (11.038 s) : 0, 11038378
Agent [candidate] (1.202 s) : 0, 1201745
Total [candidate] (11.089 s) : 0, 11089496
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.079 s -
Agent appsec 1.265 s 185.713 ms (17.2%)
Agent iast 1.236 s 156.617 ms (14.5%)
Agent profiling 1.203 s 123.98 ms (11.5%)
Total tracing 10.886 s -
Total appsec 11.183 s 296.757 ms (2.7%)
Total iast 11.289 s 402.783 ms (3.7%)
Total profiling 11.038 s 152.107 ms (1.4%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.079 s -
Agent appsec 1.264 s 185.759 ms (17.2%)
Agent iast 1.222 s 143.669 ms (13.3%)
Agent profiling 1.202 s 123.063 ms (11.4%)
Total tracing 10.849 s -
Total appsec 11.194 s 344.808 ms (3.2%)
Total iast 11.129 s 279.834 ms (2.6%)
Total profiling 11.089 s 240.613 ms (2.2%) 
gantt
    title petclinic - break down per module: candidate=1.57.0-SNAPSHOT~99ec0365ca, baseline=1.57.0-SNAPSHOT~2b04c87b81

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.184 ms) : 0, 1184
crashtracking [candidate] (1.179 ms) : 0, 1179
BytebuddyAgent [baseline] (646.618 ms) : 0, 646618
BytebuddyAgent [candidate] (647.221 ms) : 0, 647221
GlobalTracer [baseline] (281.562 ms) : 0, 281562
GlobalTracer [candidate] (281.084 ms) : 0, 281084
AppSec [baseline] (32.477 ms) : 0, 32477
AppSec [candidate] (32.127 ms) : 0, 32127
Debugger [baseline] (68.268 ms) : 0, 68268
Debugger [candidate] (68.217 ms) : 0, 68217
Remote Config [baseline] (627.942 µs) : 0, 628
Remote Config [candidate] (640.27 µs) : 0, 640
Telemetry [baseline] (9.108 ms) : 0, 9108
Telemetry [candidate] (8.992 ms) : 0, 8992
Flare Poller [baseline] (3.76 ms) : 0, 3760
Flare Poller [candidate] (3.775 ms) : 0, 3775
section appsec
crashtracking [baseline] (1.201 ms) : 0, 1201
crashtracking [candidate] (1.215 ms) : 0, 1215
BytebuddyAgent [baseline] (688.488 ms) : 0, 688488
BytebuddyAgent [candidate] (688.887 ms) : 0, 688887
GlobalTracer [baseline] (259.503 ms) : 0, 259503
GlobalTracer [candidate] (259.09 ms) : 0, 259090
AppSec [baseline] (175.444 ms) : 0, 175444
AppSec [candidate] (173.471 ms) : 0, 173471
Debugger [baseline] (66.196 ms) : 0, 66196
Debugger [candidate] (68.061 ms) : 0, 68061
Remote Config [baseline] (680.401 µs) : 0, 680
Remote Config [candidate] (708.373 µs) : 0, 708
Telemetry [baseline] (8.98 ms) : 0, 8980
Telemetry [candidate] (9.026 ms) : 0, 9026
Flare Poller [baseline] (3.818 ms) : 0, 3818
Flare Poller [candidate] (4.02 ms) : 0, 4020
IAST [baseline] (24.85 ms) : 0, 24850
IAST [candidate] (24.456 ms) : 0, 24456
section iast
crashtracking [baseline] (1.201 ms) : 0, 1201
crashtracking [candidate] (1.192 ms) : 0, 1192
BytebuddyAgent [baseline] (798.952 ms) : 0, 798952
BytebuddyAgent [candidate] (790.449 ms) : 0, 790449
GlobalTracer [baseline] (258.398 ms) : 0, 258398
GlobalTracer [candidate] (255.549 ms) : 0, 255549
AppSec [baseline] (35.623 ms) : 0, 35623
AppSec [candidate] (34.421 ms) : 0, 34421
Debugger [baseline] (66.231 ms) : 0, 66231
Debugger [candidate] (66.184 ms) : 0, 66184
Remote Config [baseline] (559.928 µs) : 0, 560
Remote Config [candidate] (544.705 µs) : 0, 545
Telemetry [baseline] (8.503 ms) : 0, 8503
Telemetry [candidate] (8.377 ms) : 0, 8377
Flare Poller [baseline] (3.55 ms) : 0, 3550
Flare Poller [candidate] (3.46 ms) : 0, 3460
IAST [baseline] (27.154 ms) : 0, 27154
IAST [candidate] (26.813 ms) : 0, 26813
section profiling
ProfilingAgent [baseline] (97.109 ms) : 0, 97109
ProfilingAgent [candidate] (97.522 ms) : 0, 97522
crashtracking [baseline] (1.175 ms) : 0, 1175
crashtracking [candidate] (1.186 ms) : 0, 1186
BytebuddyAgent [baseline] (701.539 ms) : 0, 701539
BytebuddyAgent [candidate] (700.31 ms) : 0, 700310
GlobalTracer [baseline] (220.138 ms) : 0, 220138
GlobalTracer [candidate] (220.185 ms) : 0, 220185
AppSec [baseline] (32.304 ms) : 0, 32304
AppSec [candidate] (32.135 ms) : 0, 32135
Debugger [baseline] (67.866 ms) : 0, 67866
Debugger [candidate] (67.736 ms) : 0, 67736
Remote Config [baseline] (648.246 µs) : 0, 648
Remote Config [candidate] (597.266 µs) : 0, 597
Telemetry [baseline] (9.008 ms) : 0, 9008
Telemetry [candidate] (8.916 ms) : 0, 8916
Flare Poller [baseline] (3.739 ms) : 0, 3739
Flare Poller [candidate] (3.731 ms) : 0, 3731
Profiling [baseline] (97.69 ms) : 0, 97690
Profiling [candidate] (98.09 ms) : 0, 98090
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.57.0-SNAPSHOT~99ec0365ca, baseline=1.57.0-SNAPSHOT~2b04c87b81

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.085 s) : 0, 1084993
Total [baseline] (8.787 s) : 0, 8787060
Agent [candidate] (1.081 s) : 0, 1081104
Total [candidate] (8.741 s) : 0, 8741069
section iast
Agent [baseline] (1.22 s) : 0, 1219631
Total [baseline] (9.421 s) : 0, 9421472
Agent [candidate] (1.228 s) : 0, 1227900
Total [candidate] (9.497 s) : 0, 9496861
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.085 s -
Agent iast 1.22 s 134.638 ms (12.4%)
Total tracing 8.787 s -
Total iast 9.421 s 634.412 ms (7.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.081 s -
Agent iast 1.228 s 146.796 ms (13.6%)
Total tracing 8.741 s -
Total iast 9.497 s 755.792 ms (8.6%) 
gantt
    title insecure-bank - break down per module: candidate=1.57.0-SNAPSHOT~99ec0365ca, baseline=1.57.0-SNAPSHOT~2b04c87b81

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.191 ms) : 0, 1191
crashtracking [candidate] (1.19 ms) : 0, 1190
BytebuddyAgent [baseline] (651.916 ms) : 0, 651916
BytebuddyAgent [candidate] (648.189 ms) : 0, 648189
GlobalTracer [baseline] (282.406 ms) : 0, 282406
GlobalTracer [candidate] (283.25 ms) : 0, 283250
AppSec [baseline] (32.699 ms) : 0, 32699
AppSec [candidate] (32.428 ms) : 0, 32428
Debugger [baseline] (67.831 ms) : 0, 67831
Debugger [candidate] (67.214 ms) : 0, 67214
Remote Config [baseline] (659.31 µs) : 0, 659
Remote Config [candidate] (635.775 µs) : 0, 636
Telemetry [baseline] (9.052 ms) : 0, 9052
Telemetry [candidate] (8.965 ms) : 0, 8965
Flare Poller [baseline] (3.786 ms) : 0, 3786
Flare Poller [candidate] (3.733 ms) : 0, 3733
section iast
crashtracking [baseline] (1.18 ms) : 0, 1180
crashtracking [candidate] (1.192 ms) : 0, 1192
BytebuddyAgent [baseline] (788.613 ms) : 0, 788613
BytebuddyAgent [candidate] (794.659 ms) : 0, 794659
GlobalTracer [baseline] (255.246 ms) : 0, 255246
GlobalTracer [candidate] (256.781 ms) : 0, 256781
IAST [baseline] (26.721 ms) : 0, 26721
IAST [candidate] (26.847 ms) : 0, 26847
AppSec [baseline] (34.492 ms) : 0, 34492
AppSec [candidate] (35.724 ms) : 0, 35724
Debugger [baseline] (65.717 ms) : 0, 65717
Debugger [candidate] (64.665 ms) : 0, 64665
Remote Config [baseline] (536.645 µs) : 0, 537
Remote Config [candidate] (549.087 µs) : 0, 549
Telemetry [baseline] (8.385 ms) : 0, 8385
Telemetry [candidate] (8.484 ms) : 0, 8484
Flare Poller [baseline] (3.446 ms) : 0, 3446
Flare Poller [candidate] (3.499 ms) : 0, 3499
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master PROF-13025
git_commit_date 1765384916 1765389405
git_commit_sha 2b04c87 99ec036
release_version 1.57.0-SNAPSHOT~2b04c87b81 1.57.0-SNAPSHOT~99ec0365ca
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1765391725 1765391725
ci_job_id 1284822790 1284822790
ci_pipeline_id 86006601 86006601
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-r2ny3hhq 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-r2ny3hhq 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 3 performance improvements and 1 performance regressions! Performance is the same for 14 metrics, 18 unstable metrics.

scenario Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p95 Δ mean throughput candidate mean agg_http_req_duration_p50 candidate mean agg_http_req_duration_p95 candidate mean throughput baseline mean agg_http_req_duration_p50 baseline mean agg_http_req_duration_p95 baseline mean throughput
scenario:load:insecure-bank:iast:high_load better
[-330.786µs; -174.400µs] or [-12.236%; -6.451%]		 unstable
[-1274.686µs; -271.535µs] or [-16.230%; -3.457%]		 unstable
[-120.415op/s; +236.818op/s] or [-9.045%; +17.788%]		 2.451ms 7.081ms 1389.545op/s 2.703ms 7.854ms 1331.344op/s
scenario:load:insecure-bank:iast_GLOBAL:high_load better
[-285.997µs; -100.174µs] or [-9.764%; -3.420%]		 unsure
[-892.025µs; -79.510µs] or [-10.948%; -0.976%]		 unstable
[-149.776op/s; +170.099op/s] or [-12.024%; +13.655%]		 2.736ms 7.662ms 1255.818op/s 2.929ms 8.147ms 1245.656op/s
scenario:load:petclinic:no_agent:high_load better
[-2.486ms; -1.499ms] or [-13.468%; -8.121%]		 unsure
[-3.357ms; -0.393ms] or [-11.130%; -1.304%]		 unstable
[+2.020op/s; +52.167op/s] or [+0.819%; +21.144%]		 16.466ms 28.283ms 273.812op/s 18.458ms 30.158ms 246.719op/s
scenario:load:petclinic:iast:high_load worse
[+0.564ms; +1.478ms] or [+3.221%; +8.443%]		 unsure
[+0.502ms; +1.817ms] or [+1.740%; +6.296%]		 unstable
[-35.713op/s; +10.963op/s] or [-13.719%; +4.211%]		 18.530ms 30.017ms 247.938op/s 17.509ms 28.858ms 260.312op/s
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.57.0-SNAPSHOT~99ec0365ca, baseline=1.57.0-SNAPSHOT~2b04c87b81
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.226 ms) : 1214, 1238
.   : milestone, 1226,
iast (3.44 ms) : 3389, 3491
.   : milestone, 3440,
iast_FULL (5.869 ms) : 5811, 5927
.   : milestone, 5869,
iast_GLOBAL (3.683 ms) : 3628, 3737
.   : milestone, 3683,
profiling (2.109 ms) : 2089, 2128
.   : milestone, 2109,
tracing (1.827 ms) : 1813, 1842
.   : milestone, 1827,
section candidate
no_agent (1.192 ms) : 1180, 1203
.   : milestone, 1192,
iast (3.191 ms) : 3148, 3234
.   : milestone, 3191,
iast_FULL (5.952 ms) : 5892, 6012
.   : milestone, 5952,
iast_GLOBAL (3.54 ms) : 3485, 3595
.   : milestone, 3540,
profiling (2.042 ms) : 2023, 2061
.   : milestone, 2042,
tracing (1.908 ms) : 1891, 1924
.   : milestone, 1908,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.226 ms [1.214 ms, 1.238 ms] -
iast 3.44 ms [3.389 ms, 3.491 ms] 2.214 ms (180.5%)
iast_FULL 5.869 ms [5.811 ms, 5.927 ms] 4.643 ms (378.7%)
iast_GLOBAL 3.683 ms [3.628 ms, 3.737 ms] 2.457 ms (200.3%)
profiling 2.109 ms [2.089 ms, 2.128 ms] 882.429 µs (72.0%)
tracing 1.827 ms [1.813 ms, 1.842 ms] 600.959 µs (49.0%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.192 ms [1.18 ms, 1.203 ms] -
iast 3.191 ms [3.148 ms, 3.234 ms] 1.999 ms (167.7%)
iast_FULL 5.952 ms [5.892 ms, 6.012 ms] 4.76 ms (399.4%)
iast_GLOBAL 3.54 ms [3.485 ms, 3.595 ms] 2.348 ms (197.0%)
profiling 2.042 ms [2.023 ms, 2.061 ms] 849.929 µs (71.3%)
tracing 1.908 ms [1.891 ms, 1.924 ms] 716.103 µs (60.1%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.57.0-SNAPSHOT~99ec0365ca, baseline=1.57.0-SNAPSHOT~2b04c87b81
    dateFormat X
    axisFormat %s
section baseline
no_agent (18.92 ms) : 18732, 19107
.   : milestone, 18920,
appsec (20.209 ms) : 19997, 20421
.   : milestone, 20209,
code_origins (18.122 ms) : 17941, 18303
.   : milestone, 18122,
iast (17.924 ms) : 17745, 18104
.   : milestone, 17924,
profiling (18.509 ms) : 18321, 18697
.   : milestone, 18509,
tracing (18.07 ms) : 17889, 18251
.   : milestone, 18070,
section candidate
no_agent (17.037 ms) : 16864, 17209
.   : milestone, 17037,
appsec (19.859 ms) : 19656, 20063
.   : milestone, 19859,
code_origins (17.911 ms) : 17730, 18093
.   : milestone, 17911,
iast (18.821 ms) : 18630, 19012
.   : milestone, 18821,
profiling (18.481 ms) : 18296, 18666
.   : milestone, 18481,
tracing (18.2 ms) : 18014, 18385
.   : milestone, 18200,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 18.92 ms [18.732 ms, 19.107 ms] -
appsec 20.209 ms [19.997 ms, 20.421 ms] 1.29 ms (6.8%)
code_origins 18.122 ms [17.941 ms, 18.303 ms] -797.499 µs (-4.2%)
iast 17.924 ms [17.745 ms, 18.104 ms] -995.222 µs (-5.3%)
profiling 18.509 ms [18.321 ms, 18.697 ms] -410.419 µs (-2.2%)
tracing 18.07 ms [17.889 ms, 18.251 ms] -849.739 µs (-4.5%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 17.037 ms [16.864 ms, 17.209 ms] -
appsec 19.859 ms [19.656 ms, 20.063 ms] 2.822 ms (16.6%)
code_origins 17.911 ms [17.73 ms, 18.093 ms] 874.665 µs (5.1%)
iast 18.821 ms [18.63 ms, 19.012 ms] 1.784 ms (10.5%)
profiling 18.481 ms [18.296 ms, 18.666 ms] 1.444 ms (8.5%)
tracing 18.2 ms [18.014 ms, 18.385 ms] 1.163 ms (6.8%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master PROF-13025
git_commit_date 1765384916 1765389405
git_commit_sha 2b04c87 99ec036
release_version 1.57.0-SNAPSHOT~2b04c87b81 1.57.0-SNAPSHOT~99ec0365ca
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1765391459 1765391459
ci_job_id 1284822794 1284822794
ci_pipeline_id 86006601 86006601
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-aittw0v6 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-aittw0v6 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 2 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.57.0-SNAPSHOT~99ec0365ca, baseline=1.57.0-SNAPSHOT~2b04c87b81
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.473 ms) : 1461, 1484
.   : milestone, 1473,
appsec (3.733 ms) : 3515, 3952
.   : milestone, 3733,
iast (2.215 ms) : 2151, 2280
.   : milestone, 2215,
iast_GLOBAL (2.253 ms) : 2189, 2318
.   : milestone, 2253,
profiling (2.085 ms) : 2031, 2139
.   : milestone, 2085,
tracing (2.051 ms) : 2000, 2102
.   : milestone, 2051,
section candidate
no_agent (1.467 ms) : 1455, 1478
.   : milestone, 1467,
appsec (3.695 ms) : 3476, 3914
.   : milestone, 3695,
iast (2.207 ms) : 2143, 2271
.   : milestone, 2207,
iast_GLOBAL (2.269 ms) : 2204, 2334
.   : milestone, 2269,
profiling (2.482 ms) : 2321, 2643
.   : milestone, 2482,
tracing (2.049 ms) : 1998, 2100
.   : milestone, 2049,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.473 ms [1.461 ms, 1.484 ms] -
appsec 3.733 ms [3.515 ms, 3.952 ms] 2.261 ms (153.5%)
iast 2.215 ms [2.151 ms, 2.28 ms] 742.334 µs (50.4%)
iast_GLOBAL 2.253 ms [2.189 ms, 2.318 ms] 780.582 µs (53.0%)
profiling 2.085 ms [2.031 ms, 2.139 ms] 611.897 µs (41.5%)
tracing 2.051 ms [2.0 ms, 2.102 ms] 578.121 µs (39.3%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.467 ms [1.455 ms, 1.478 ms] -
appsec 3.695 ms [3.476 ms, 3.914 ms] 2.228 ms (151.9%)
iast 2.207 ms [2.143 ms, 2.271 ms] 739.753 µs (50.4%)
iast_GLOBAL 2.269 ms [2.204 ms, 2.334 ms] 802.412 µs (54.7%)
profiling 2.482 ms [2.321 ms, 2.643 ms] 1.015 ms (69.2%)
tracing 2.049 ms [1.998 ms, 2.1 ms] 581.987 µs (39.7%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.57.0-SNAPSHOT~99ec0365ca, baseline=1.57.0-SNAPSHOT~2b04c87b81
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.615 s) : 15615000, 15615000
.   : milestone, 15615000,
appsec (14.498 s) : 14498000, 14498000
.   : milestone, 14498000,
iast (18.182 s) : 18182000, 18182000
.   : milestone, 18182000,
iast_GLOBAL (18.288 s) : 18288000, 18288000
.   : milestone, 18288000,
profiling (14.786 s) : 14786000, 14786000
.   : milestone, 14786000,
tracing (14.856 s) : 14856000, 14856000
.   : milestone, 14856000,
section candidate
no_agent (15.078 s) : 15078000, 15078000
.   : milestone, 15078000,
appsec (14.654 s) : 14654000, 14654000
.   : milestone, 14654000,
iast (18.36 s) : 18360000, 18360000
.   : milestone, 18360000,
iast_GLOBAL (17.875 s) : 17875000, 17875000
.   : milestone, 17875000,
profiling (14.651 s) : 14651000, 14651000
.   : milestone, 14651000,
tracing (14.836 s) : 14836000, 14836000
.   : milestone, 14836000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.615 s [15.615 s, 15.615 s] -
appsec 14.498 s [14.498 s, 14.498 s] -1.117 s (-7.2%)
iast 18.182 s [18.182 s, 18.182 s] 2.567 s (16.4%)
iast_GLOBAL 18.288 s [18.288 s, 18.288 s] 2.673 s (17.1%)
profiling 14.786 s [14.786 s, 14.786 s] -829.0 ms (-5.3%)
tracing 14.856 s [14.856 s, 14.856 s] -759.0 ms (-4.9%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.078 s [15.078 s, 15.078 s] -
appsec 14.654 s [14.654 s, 14.654 s] -424.0 ms (-2.8%)
iast 18.36 s [18.36 s, 18.36 s] 3.282 s (21.8%)
iast_GLOBAL 17.875 s [17.875 s, 17.875 s] 2.797 s (18.6%)
profiling 14.651 s [14.651 s, 14.651 s] -427.0 ms (-2.8%)
tracing 14.836 s [14.836 s, 14.836 s] -242.0 ms (-1.6%)

bric3
bric3 reviewed Dec 9, 2025
View reviewed changes
...-openjdk/src/main/java/com/datadog/profiling/controller/openjdk/events/SmapEntryFactory.java Outdated

static {
// Load JFR Handlers class early, if present (it has been moved and renamed in JDK23+).
// This prevents a deadlock. See PROF-13025.
Copy link
Contributor

@bric3 bric3 Dec 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Is there a JDK bug id as well to drop here ?

The deadlock has been reported to upstream, and a fix has been proposed there

@rkennke rkennke requested a review from a team as a code owner December 9, 2025 13:28
@rkennke rkennke requested a review from mhlidd December 9, 2025 13:28
@rkennke rkennke requested a review from jbachorik December 9, 2025 18:23
@rkennke
Copy link
Contributor Author

rkennke commented Dec 9, 2025

@jbachorik can you re-review this? I pulled the loading of Handlers class out to Agent.registerSmapEvent(). I think this is a better place, and it uses AGENT_CLASSLOADER directly and sits nicely besides the code that loads the SmapEntryFactory. WDYT?
(previous CI troubles seem to be resolved, btw)

jbachorik
jbachorik reviewed Dec 10, 2025
View reviewed changes
Copy link
Contributor

@jbachorik jbachorik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good - can you add the JBS bug in the comment here, @bric3 requested?

@rkennke rkennke merged commit 015be6d into master Dec 11, 2025
549 checks passed
@rkennke rkennke deleted the PROF-13025 branch December 11, 2025 11:43
@github-actions github-actions bot added this to the 1.57.0 milestone Dec 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bric3 bric3 bric3 left review comments

@jbachorik jbachorik jbachorik approved these changes

@mhlidd mhlidd Awaiting requested review from mhlidd mhlidd is a code owner automatically assigned from DataDog/apm-java

Assignees

No one assigned

Labels

comp: profiling Profiling type: bug Bug report and fix

Projects

None yet

Milestone

1.57.0

Development

Successfully merging this pull request may close these issues.

4 participants

@rkennke @jbachorik @bric3