Stopped polling PR through cron expression

[Aamir010]

We have a build job which stopped polling suddenly and we have to restart the Jenkins to get this jobs poll.

Is there anything that messed up Jenkins core scheduler?

Jenkins Version :- 2.128
Plugin Version :- 1.4.26

[chrisLeeTW]

I also facing the same problem.

[Aamir010]

@chrisLeeTW did you found any workaround ? We are facing this issue from last 5 days, we haven't change anything or update.

[chrisLeeTW]

@Aamir010 sorry, currently I used same workaround just like you, but I have some assumption about this issue, and install plugin of my own version to further debug logging collection.

Hope my assumption is right & I'll have PR to fix this issue :)

[CodeMonk]

Anything interesting in the logs?

[chrisLeeTW]

@CodeMonk sorry, plugin is still working functionally afterward. I'll keep watching.

[Aamir010]

Anything interesting in the logs?

@CodeMonk There's a couple of exception found on logs because of old BitBucket API version. Can you able to release the fix of it ? Below are the log traces.

java.io.FileNotFoundException: https://api.bitbucket.org/1.0/users/test\vinoth.j
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1890)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
at java.net.URL.openStream(URL.java:1045)
at org.jenkinsci.plugins.api.BitbucketApiService.getUserByUsername(BitbucketApiService.java:148)
at org.jenkinsci.plugins.BitbucketSecurityRealm.loadUserByUsername(BitbucketSecurityRealm.java:181)
at jenkins.security.UserDetailsCache$Retriever.call(UserDetailsCache.java:172)
at jenkins.security.UserDetailsCache$Retriever.call(UserDetailsCache.java:161)
at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
at jenkins.security.UserDetailsCache.loadUserByUsername(UserDetailsCache.java:127)
at hudson.model.User$UserIDCanonicalIdResolver.resolveCanonicalId(User.java:1256)
at hudson.model.User$CanonicalIdResolver.resolve(User.java:1193)
at hudson.model.User.get(User.java:464)
at hudson.model.User.get(User.java:431)
at hudson.plugins.git.GitChangeSet.findOrCreateUser(GitChangeSet.java:388)
at hudson.plugins.git.GitChangeSet.getAuthor(GitChangeSet.java:466)
at com.cloudbees.workflow.rest.external.ChangeSetExt.mapFields(ChangeSetExt.java:186)
at com.cloudbees.workflow.rest.external.ChangeSetExt.create(ChangeSetExt.java:161)
at com.cloudbees.workflow.rest.endpoints.RunAPI.doChangesets(RunAPI.java:112)
at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343)
at com.cloudbees.workflow.util.ServeJson$Processor.invoke(ServeJson.java:30)
at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)
at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)
at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:248)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:248)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:230)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:202)
at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:85)
at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:112)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:531)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at winstone.BoundedExecutorService.lambda$scheduleNext$0(BoundedExecutorService.java:80)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

[CodeMonk]

@Aamir010 That one is actually benign. (Well, technically it's a bug, but, it has no other side effects right now)

I'm looking for the logs right around where the polling happens. They should look something like this:

Oct 01, 2018 3:35:02 AM bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient send
WARNING: Response status: HTTP/1.1 429 Unknown Status Code URI: https://bitbucket.org/api/2.0/repositories/ACCOUNT/REPO/pullrequests/?pagelen=50

[chrisLeeTW]

@CodeMonk @Aamir010 that is the error stack strace from this plugin (https://github.com/mallowlabs/bitbucket-oauth-plugin). this plugin is not calling any user relative bitbucket api for any action.

@CodeMonk my plugin is still working until now. and no any interesting logs for now. 😭

[chrisLeeTW]

@Aamir010 do you still faced this problem again ?

[Aamir010]

Yes, the problem is still continued but I don't think that was because of this plugin. As we have more than 100+ job's which are polling every minute and we changed the polling interval for those jobs and also we distribute Jenkins across multiple small instances which eliminates this issue

[CodeMonk]

Yeah, we have a lot of throttling issues identified. AFter we push out a release, we are planning to add a triggering option, ASAP to avoid hammering bitbucket.

[chrisLeeTW]

We recently still face this problem again & again, but I do not see any relative log about bitbucket api 429 throttling issues.

So after some long time research & reference some jenkins issue (https://issues.jenkins-ci.org/browse/JENKINS-41336, https://issues.jenkins-ci.org/browse/JENKINS-30558, https://issues.jenkins-ci.org/browse/JENKINS-25704, https://issues.jenkins-ci.org/browse/JENKINS-25704), and I found this pr in stash-pullrequest-builder-plugin (nemccarthy/stash-pullrequest-builder-plugin#147), and I think it may be good approach to fix this issue.

So I already build my own patch and continue testing in our production jenkins instnace, and I will keep watching for 1 week.

If It works fine, and then I will submit the pull request asap.

[Dohbedoh]

So I already build my own patch and continue testing in our production jenkins instnace, and I will keep watching for 1 week.
If It works fine, and then I will submit the pull request asap.

@chrisLeeTW Any news about this ? Have you had the chance to propose a PR ?