GRPO training on structured data

[eduardzl]

Please check that this issue hasn't been reported before.

• I searched previous Bug Reports didn't find any similar reports.

Expected Behavior

I am trying to run GRPO training on tool calls data set.
The configuation looks like :

base_model: /opt/ml/model/gpt-oss-20b

use_kernels: false
model_quantization_config: Mxfp4Config
model_quantization_config_kwargs:
  dequantize: true

plugins:
  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin

experimental_skip_move_to_device: true  # prevent OOM by NOT putting model to GPU before sharding

chat_template: tokenizer_default

datasets:
  - path: /opt/ml/model/grpo_rows.jsonl
    type: chat_template
    field_messages: prompt
    message_field_role: role
    message_field_content: content
    field_tools: tools

rl: grpo

trl:
  use_vllm: true
  vllm_server_host: 127.0.0.1
  vllm_server_port: 8000
  vllm_server_timeout: 300
  num_generations: 4
  max_completion_length: 6000
  rollout_func: ged.runtime.rollout.tool_rollout
  reward_funcs:
    - ged.runtime.rewards.r_step_keywords
    - ged.runtime.rewards.r_mock_success
    - ged.runtime.rewards.r_json
  reward_weights: [0.4, 0.4, 0.2]

dataset_prepared_path: last_run_prepared
val_set_size: 0.03
eval_steps: 10

When running the training, I am getting error :

Mapping RL Dataset (num_proc=30):   0%|          | 0/30 [00:10<?, ? examples/s]
multiprocess.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/opt/conda/envs/training-env/lib/python3.12/site-packages/multiprocess/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/training-env/lib/python3.12/site-packages/datasets/utils/py_utils.py", line 586, in _write_generator_to_queue
    for i, result in enumerate(func(**kwargs)):
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/training-env/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3664, in _map_single
    for i, example in iter_outputs(shard_iterable):
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/training-env/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3638, in iter_outputs
    yield i, apply_function(example, i, offset=offset)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/training-env/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3561, in apply_function
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/axolotl/src/axolotl/prompt_strategies/dpo/chat_template.py", line 63, in transform_fn
    chosen_raw = sample[field_chosen]
                 ~~~~~~^^^^^^^^^^^^^^
  File "/opt/conda/envs/training-env/lib/python3.12/site-packages/datasets/formatting/formatting.py", line 283, in _getitem_
    value = self.data[key]
            ~~~~~~~~~^^^^^
KeyError: 'chosen'

Should the data set for GRPO include
field_chosen: chosen
field_rejected: rejected # Required for DPO-style
?

As I undersatand chosen/rejected fields are relevant for DPO, not GRPO.

Thank you

Current behaviour

Error raised when trying to train GRPO on tools data set.

Steps to reproduce

1. Preparing data set with function calls
2. Creating Axolotl configuation as specified with GRPO training.
3. Launching training

Config yaml

Possible solution

No response

Which Operating Systems are you using?

• Linux
• macOS
• Windows

Python Version

3.12

axolotl branch-commit

dd78f2e

Acknowledgements

• My issue title is concise, descriptive, and in title casing.
• I have searched the existing issues to make sure this bug has not been reported yet.
• I am using the latest version of axolotl.
• I have provided enough information for the maintainers to reproduce and diagnose the issue.

[ved1beta]

hey thanks for reporting this , please can you try after Removing type: chat_template in dataset
also if this doesn't work please consider sharing your dataset

[eduardzl]

@ved1beta Thanks fotr reply. After removing type: chat_template I getting another error.

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/workspace/axolotl/src/axolotl/cli/train.py", line 121, in <module>
    fire.Fire(do_cli)
  File "/opt/conda/envs/training-env/lib/python3.12/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/training-env/lib/python3.12/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/training-env/lib/python3.12/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/axolotl/src/axolotl/cli/train.py", line 88, in do_cli
    return do_train(parsed_cfg, parsed_cli_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/axolotl/src/axolotl/cli/train.py", line 43, in do_train
    dataset_meta = load_datasets(cfg=cfg, cli_args=cli_args)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/axolotl/src/axolotl/common/datasets.py", line 59, in load_datasets
    train_dataset, eval_dataset, total_num_steps, prompters = prepare_datasets(
                                                              ^^^^^^^^^^^^^^^^^
  File "/workspace/axolotl/src/axolotl/utils/data/utils.py", line 50, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/axolotl/src/axolotl/utils/data/sft.py", line 65, in prepare_datasets
    return _prepare_standard_dataset(cfg, tokenizer, processor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/axolotl/src/axolotl/utils/data/sft.py", line 98, in _prepare_standard_dataset
    train_dataset, eval_dataset, prompters = loader.load(_load_datasets)
                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/axolotl/src/axolotl/utils/data/lock.py", line 44, in load
    return load_fn()
           ^^^^^^^^^
  File "/workspace/axolotl/src/axolotl/utils/data/sft.py", line 77, in _load_datasets
    train_dataset, eval_dataset, prompters = _load_and_prepare_datasets(
                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/axolotl/src/axolotl/utils/data/sft.py", line 503, in _load_and_prepare_datasets
    dataset, prompters = _load_tokenized_prepared_datasets(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/axolotl/src/axolotl/utils/data/sft.py", line 299, in _load_tokenized_prepared_datasets
    dataset, prompters = _load_raw_datasets(
                         ^^^^^^^^^^^^^^^^^^^
  File "/workspace/axolotl/src/axolotl/utils/data/sft.py", line 331, in _load_raw_datasets
    dataset_wrapper, dataset_prompter = _load_and_process_single_dataset(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/axolotl/src/axolotl/utils/data/sft.py", line 401, in _load_and_process_single_dataset
    dataset_wrapper, dataset_prompter = get_dataset_wrapper(
                                        ^^^^^^^^^^^^^^^^^^^^
  File "/workspace/axolotl/src/axolotl/utils/data/wrappers.py", line 107, in get_dataset_wrapper
    if dataset_config.type.startswith("bradley_terry"):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'startswith'

Here is sample from my data set:

{"task_id": "cpa_reconciliation_018", "task_name": "Bank reconciliation for Cascade Retail Co", "tools": [{"type": "function", "function": {"name": "read_csv", "description": "Read a CSV file and return summary statistics", "parameters": {"type": "object", "properties": {"file_path": {"type": "string", "description": "Path to the CSV file to read"}}, "required": ["file_path"]}}}, {"type": "function", "function": {"name": "read_excel", "description": "Read an Excel file and return summary statistics", "parameters": {"type": "object", "properties": {"file_path": {"type": "string", "description": "Path to the Excel file to read"}, "sheet_name": {"type": "string", "description": "Name of the sheet to read (default: first sheet)"}}, "required": ["file_path"]}}}, {"type": "function", "function": {"name": "reconcile_accounts", "description": "Reconcile two accounts and identify discrepancies within tolerance", "parameters": {"type": "object", "properties": {"account1": {"type": "string", "description": "First account identifier"}, "account2": {"type": "string", "description": "Second account identifier"}, "tolerance": {"type": "number", "description": "Acceptable variance tolerance amount"}}, "required": ["account1", "account2", "tolerance"]}}}, {"type": "function", "function": {"name": "prepare_journal_entry", "description": "Prepare a double-entry journal entry", "parameters": {"type": "object", "properties": {"debit_account": {"type": "string", "description": "Account to debit"}, "credit_account": {"type": "string", "description": "Account to credit"}, "amount": {"type": "number", "description": "Transaction amount"}, "description": {"type": "string", "description": "Description of the transaction"}}, "required": ["debit_account", "credit_account", "amount", "description"]}}}], "messages": [{"role": "system", "content": "You are an CPA assistant. Use the available tools to read data files, perform calculations, and produce accurate financial reports. Think step-by-step and validate your work."}, {"role": "user", "content": "Reconcile accounts for Cascade Retail Co.. Read book records from cascade_retail_co._books_q1_2024.csv and calculate the final reconciled balance."}, {"role": "assistant", "content": "Plan: (1) read data from file, (2) apply reconcile_accounts, then validate and report results."}, {"role": "assistant", "content": "", "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "read_csv", "arguments": "{\"file_path\": \"cascade_retail_co._books_q1_2024.csv\"}"}}]}, {"role": "tool", "tool_call_id": "call_1", "content": "{\"rows\": 20, \"total_amount\": -17569.03, \"total_balance\": 4173835.86}"}, {"role": "assistant", "content": "", "tool_calls": [{"id": "call_2", "type": "function", "function": {"name": "reconcile_accounts", "arguments": "{}"}}]}, {"role": "tool", "tool_call_id": "call_2", "content": "{\"ok\": true, \"result\": \"Success\"}"}, {"role": "assistant", "content": "**Final Answer:** $223,113.58"}], "metadata": {"scenario_id": "cpa_reconciliation_018", "scenario_name": "Bank reconciliation for Cascade Retail Co", "domain": "accounting_general", "complexity": "low", "files": [{"path": "cascade_retail_co._books_q1_2024.csv", "sha256": "be208e4d0deb6f27"}], "created_at": "2025-11-17T09:52:05.492167+00:00", "source": "data_generation/format_for_training_read_tools.py"}}
{"task_id": "cpa_expense_management_006", "task_name": "Expense management and categorization for Atlas Manufactu...", "tools": [{"type": "function", "function": {"name": "read_csv", "description": "Read a CSV file and return summary statistics", "parameters": {"type": "object", "properties": {"file_path": {"type": "string", "description": "Path to the CSV file to read"}}, "required": ["file_path"]}}}, {"type": "function", "function": {"name": "read_excel", "description": "Read an Excel file and return summary statistics", "parameters": {"type": "object", "properties": {"file_path": {"type": "string", "description": "Path to the Excel file to read"}, "sheet_name": {"type": "string", "description": "Name of the sheet to read (default: first sheet)"}}, "required": ["file_path"]}}}, {"type": "function", "function": {"name": "query_general_ledger", "description": "Query the general ledger for account transactions within a date range", "parameters": {"type": "object", "properties": {"account_id": {"type": "string", "description": "Account ID or number to query"}, "start_date": {"type": "string", "description": "Start date in YYYY-MM-DD format"}, "end_date": {"type": "string", "description": "End date in YYYY-MM-DD format"}}, "required": ["account_id", "start_date", "end_date"]}}}, {"type": "function", "function": {"name": "calculate_financial_ratio", "description": "Calculate financial ratios (liquidity, profitability, efficiency, leverage)", "parameters": {"type": "object", "properties": {"ratio_type": {"type": "string", "enum": ["current_ratio", "quick_ratio", "debt_to_equity", "return_on_assets", "return_on_equity", "profit_margin"], "description": "Type of financial ratio to calculate"}, "values": {"type": "object", "description": "Dictionary of values needed for the ratio calculation"}}, "required": ["ratio_type", "values"]}}}, {"type": "function", "function": {"name": "prepare_journal_entry", "description": "Prepare a double-entry journal entry", "parameters": {"type": "object", "properties": {"debit_account": {"type": "string", "description": "Account to debit"}, "credit_account": {"type": "string", "description": "Account to credit"}, "amount": {"type": "number", "description": "Transaction amount"}, "description": {"type": "string", "description": "Description of the transaction"}}, "required": ["debit_account", "credit_account", "amount", "description"]}}}], "messages": [{"role": "system", "content": "You are an CPA assistant. Use the available tools to read data files, perform calculations, and produce accurate financial reports. Think step-by-step and validate your work."}, {"role": "user", "content": "Calculate total travel expenses for Atlas Manufacturing. Read data from atlas_manufacturing_expenses_fy_2024.csv and general_ledger.csv.. Read the expense data from atlas_manufacturing_expenses_fy_2024.csv and sum all amounts in the 'travel' category."}, {"role": "assistant", "content": "Plan: (1) read data from file, (2) apply query_general_ledger, (3) calculate required values, then validate and report results."}, {"role": "assistant", "content": "", "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "read_csv", "arguments": "{\"file_path\": \"atlas_manufacturing_expenses_fy_2024.csv\"}"}}]}, {"role": "tool", "tool_call_id": "call_1", "content": "{\"rows\": 30, \"total_amount\": 64320.03}"}, {"role": "assistant", "content": "", "tool_calls": [{"id": "call_2", "type": "function", "function": {"name": "query_general_ledger", "arguments": "{}"}}]}, {"role": "tool", "tool_call_id": "call_2", "content": "{\"ok\": true, \"result\": \"Success\"}"}, {"role": "assistant", "content": "", "tool_calls": [{"id": "call_3", "type": "function", "function": {"name": "calculate_financial_ratio", "arguments": "{\"values\": [64320.03, 30]}"}}]}, {"role": "tool", "tool_call_id": "call_3", "content": "{\"ok\": true, \"result\": 64320.03}"}, {"role": "assistant", "content": "**Final Answer:** $64,320.03"}], "metadata": {"scenario_id": "cpa_expense_management_006", "scenario_name": "Expense management and categorization for Atlas Manufactu...", "domain": "accounting_general", "complexity": "low", "files": [{"path": "atlas_manufacturing_expenses_fy_2024.csv", "sha256": "ed515b547b9d3b98"}, {"path": "general_ledger.csv", "sha256": "73aeb62d95063c65"}], "created_at": "2025-11-17T09:52:05.493454+00:00", "source": "data_generation/format_for_training_read_tools.py"}}

[ved1beta]

Hey, thanks for sharing the dataset!
the issue is that GRPO requires a custom transform function to modify the labels .
You have the reward functions set up (ged.runtime.rewards.*), but you're missing the transform function.
Create a transform function (similar to your reward functions) in your ged.runtime module
and it should runn fine : )
refer this : https://docs.axolotl.ai/docs/rlhf.html#reward-functions