I can not reproduce your train AUC, & valid AUC scores

[linhduongtuan]

Dear Besbes,
Firstly, I thank for your sharing code. It is very nice. I test with sagittal dataset with anterior cruciate ligament tear. However, I could not reproduce your results like train & valid AUC scores. So could you explain to clarify the issue? Or maybe I acquire a mistake somewhere?
Best regard. Linh

[ahmedbesbes]

Hello, what scores did you get?
Have you downloaded the right MRNet dataset?
Did you run the code properly (as indicated in the readme file)

[linhduongtuan]

My AUC scores for training and validating are 75% and 77%, respectively.
I use the dataset from Stanford. I also know that another MRNet dataset comes from Croatia.
I have read your code and blog throughout and hope no mistake occurs during my implementation
Kindly

[ahmedbesbes]

The scores you are mentioning, are they related to a specific plane or to the global (stacked) model?

[linhduongtuan]

I just tested only ACL diagnosis with Sagittal plane. Because I guess the plane has the most important tasks for the diagnosis, especially for ACL. And this condition should be the highest AUC score? I've read both papers from Stanford Uni (first author: Nicholas Bien) and Croatia + Turkey group. I know you try to reproduce the experiments of Stanford Uni paper.
Do you have any further suggestion?
Best regards. Linh

[ahmedbesbes]

Can you share the code? or the hyperparameters you used?

[linhduongtuan]

I don't have HPC so I use Google Colab to reproduce your experiment! I don't know how do authors of the paper (https://doi. org/10.1371/journal.pmed.1002699) do exact to get their AUC scores? I have test certain codes following the paper. However, I have never seen they reach the scores.
Here your code I manipulate to use Google Colab: https://drive.google.com/file/d/1UbzDfytwy6fY1iykkzU5xhY08ioEoMTZ/view?usp=sharing
Best regards
Linh

[ParthTrehan]

yeah, i also could not reproduce the AUCs. I saw that you are optimizing AUCs but in the papers, they are minimizing the error rate

[therc01]

I don't have HPC so I use Google Colab to reproduce your experiment! I don't know how do authors of the paper (https://doi. org/10.1371/journal.pmed.1002699) do exact to get their AUC scores? I have test certain codes following the paper. However, I have never seen they reach the scores.
Here your code I manipulate to use Google Colab: https://drive.google.com/file/d/1UbzDfytwy6fY1iykkzU5xhY08ioEoMTZ/view?usp=sharing
Best regards
Linh

Hi, im getting an error while building global acl classifier. I used the code from your blog for that.
Im getting error in this loop
` predictions = []

labels = []

with torch.no_grad():

    
     for image, label, _ in tqdm_notebook(train_loader):
        logit = mrnet(image.cuda())
        prediction = torch.sigmoid(logit)
        predictions.append(prediction.item())
        labels.append(label.item())`

particularly in prediction.item()
i used np.squeeze() to achieve the same. I did same for label.item(). It gets resolve here.

now its throwing error at
` X = np.zeros((len(predictions), 3))

X[:, 0] = results['axial']
X[:, 1] = results['coronal']
X[:, 2] = results['sagittal']`

values are not getting assigned here to X. What do i do now? What im doing wrong here.