Cleanup: find a better way to handle retryable errors

[BlaineEXE]

Enhancement

Why is this needed?:

Code cleanup and consistency

Describe the solution you'd like in detail:

Some options:

1. Find a way to use errors.Is() with a custom COSI retryable error type
2. Use github.com/siderolabs/gen xerror tagging
3. Keep current code?
4. Something else?

Describe alternatives you've considered:

See above and below

Additional context:

Original discussion: #170 (comment)

[BlaineEXE]

I'm going to experiment with a custom error type and propose a PR for this soon so we can better discuss and compare genericized sidero tagging versus a simpler but hardcoded error type.

Copying my response from #170 (comment) below

I have a few parallel thoughts:

Overall I do think the suggestion and pattern would be a good one for us here. But, considering the other thoughts below, I think we can make our own error type/tag with very little boilerplate and Golang's errors.Is() without adding unnecessary complexity/dependencies. I don't think we need arbitrary tagging when there is only one case/tag we care about.

From a practical lens, I'd prefer to handle this in a follow-up PR because this pattern is already in place in the existing controllers that are merged. If we choose this pattern, I'd like to make sure all controllers are consistent.

As a final thought, I think I'd prefer not to pull in this particular dependency unless it becomes more broadly useful. The https://github.com/siderolabs/gen repo appears to basically be a large util library which we won't use most of, and I'm not much of a fan.

As for why this pattern exists: Originally, I was trying to return an error with reconcile.TerminalError(err) from the helpers, but there is a case where we need to convert the terminal error to a non-terminal error here:

container-object-storage-interface/controller/internal/reconciler/bucketclaim.go

Lines 70 to 80 in d587b32

	if updErr := r.Status().Update(ctx, claim); updErr != nil {
	logger.Error(err, "failed to update BucketClaim status after reconcile error", "updateError", updErr)
	// If status update fails, we must retry the error regardless of the reconcile return.
	// The reconcile needs to run again to make sure the status is eventually be updated.
	return reconcile.Result{}, err
	}
	if !retryError {
	return reconcile.Result{}, reconcile.TerminalError(err)
	}
	return reconcile.Result{}, err