[P1] difficulty in reproducing mrpc results

[NitayGitHub]

Hi, I am running on the original code task_steer.py file with the arguments below, but struggling to reproduce the same results on the GLUE mrpc dataset. Are these the correct arguments to use:

epochs: int = 40
lr: float = 0.0003
position: str = "f3"
rank: int = 1
dropout: float = 0.05
weight_decay: float = 0.0
warmup_ratio: float = 0.0
reft_intervention: str = "ConditionedSourceLowRankRotatedSpaceIntervention"
layers: str = "all"
batch_size: int = 32
eval_batch_size: int = 32
accumulation_steps: int = 1
max_grad_norm: float = 1.0
logging_steps: int = 20
max_length: int = 256
seeds: list[int] = [42, 43, 44, 45, 46]
task: str = "glue"
train_dataset: str = "mrpc"
model_name: str = "FacebookAI/roberta-base"
test_split: str = "test"
metric: str = "accuracy"

[frankaging]

Hi Nitay,

Thanks for raising the q. Did you take a look at this thread?. As i mentioned before, GLUE (especially smaller tasks like MRPC) is a little unstable. Can you also post your avg. accuracy on MRPC? Thanks.

[NitayGitHub]

I applied the fixes I mentioned here: #177

These are my library versions:

torch : 2.6.0+cu124
torchvision : 0.21.0+cu124
pyvene : 0.1.8
transformers : 4.52.4
protobuf : 3.20.3
matplotlib : 3.10.7
ipywidgets : 8.1.5
plotnine : 0.14.5
huggingface_hub : 0.33.1
numpy : 1.26.4
accelerate : 1.8.1
sentencepiece : 0.2.0
evaluate : 0.4.6
datasets : 3.6.0
wandb : 0.20.1
scikit_learn : 1.2.2
jupyter : ❌ Not installed (PackageNotFoundError)
fsspec : 2025.3.0
ydata_profiling : 4.16.1
seaborn : 0.12.2

After running on seeds 42, 43, 44, I got:
{'test_accuracy': 0.4831981460023175, 'test_f1': 0.35174418604651164, 'test_combined_score': 0.4174711660244146}
{'test_accuracy': 0.7219003476245655, 'test_f1': 0.8070739549839228, 'test_combined_score': 0.7644871513042442}
{'test_accuracy': 0.4264194669756663, 'test_f1': 0.19249592169657423, 'test_combined_score': 0.30945769433612025}

[NitayGitHub]

I tried running train.py in examples with the same arguments and got more stable scores of
{'test_accuracy': 0.8644264194669756, 'test_f1': 0.8997429305912596, 'test_combined_score': 0.8820846750291176}
{'test_accuracy': 0.8586326767091541, 'test_f1': 0.8993399339933994, 'test_combined_score': 0.8789863053512768}
{'test_accuracy': 0.8679026651216686, 'test_f1': 0.9020618556701031, 'test_combined_score': 0.8849822603958859}
{'test_accuracy': 0.828505214368482, 'test_f1': 0.8735042735042735, 'test_combined_score': 0.8510047439363777}

Although this is still not 89 like in the paper

[frankaging]

@NitayGitHub interesting. it is hard for us to dig into details at this point (but i would still encourage you to look for potential changes in our codebase that could cause this discrepancy. i will also try to do this). but based on our Table 14 in the appendix, MRPC has a large variance as reported, 89.2(2.62). i would suggest that you can report the avg along with std with your runs, and see if it is 2sd's away of each other.

i also want to point out that GLUE tasks are pretty small, esp. MRPC.