After a fair bit of debugging, the best explanation of this is that the prior_job list for this library has grown so large (>300) that the linked list we build is so deep (when you factor in try runs and phab objects) that it's exceeding the recurison limit (1000) when we pickle it. But we only ever pickle it in one place: when we memoize it over here.

Now the annoying thing is I could not get this to reproduce locally - I restored a copy of the prod db to dev and ran it locally trying to induce it to exceed the limit. The pickled string was very large (>150k characters) but no recursion issue.

This is still my best guess of the failure though, because it's definitely crashing right when we try to enter SCMProvider.check_for_update