Training with multispectral imagery: Training has diverged

[xabierr]

Getting the following when training with MS imagery (5 bands):

cfg = setup_cfg(base_model, trains, tests, workers=2, eval_period=50, #workers 2 eval 50
max_iter=50000, out_dir=out_dir, imgmode="ms",
num_bands=num_bands)

cfg.SOLVER.IMS_PER_BATCH = 1
cfg.SOLVER.BASE_LR = 0.0001
trainer = MyTrainer(cfg, patience=5)
trainer.resume_or_load(resume=False)
trainer.train()

Error:

File ~\AppData\Local\anaconda3\envs\detectree2\lib\site-packages\detectron2\modeling\proposal_generator\proposal_utils.py:108, in find_top_rpn_proposals(proposals, pred_objectness_logits, image_sizes, nms_thresh, pre_nms_topk, post_nms_topk, min_box_size, training)
106 if not valid_mask.all():
107 if training:
--> 108 raise FloatingPointError(
109 "Predicted boxes or scores contain Inf/NaN. Training has diverged."
110 )
111 boxes = boxes[valid_mask]
112 scores_per_img = scores_per_img[valid_mask]

FloatingPointError: Predicted boxes or scores contain Inf/NaN. Training has diverged.