Development: Run PECV Bench in Artemis
#11661
Conversation
…reating a course, and an exercise. And duplicate all important functionality from quick-course-setup folder
khinevich
changed the title
Hyperion: Run PECV Bench in Artemis
…perion/run-pecv-bench-in-artemis
khinevich
changed the title
Hyperion: Run PECV Bench in ArtemisProgrammin Exercise: Run PECV Bench in Artemis
khinevich
changed the title
Programmin Exercise: Run PECV Bench in ArtemisProgramming Exercise: Run PECV Bench in Artemis
khinevich
changed the title
Programming Exercise: Run PECV Bench in ArtemisProgramming exercises: Run PECV Bench in Artemis
…ted id, tittle, shortName and projectKey of each variant
… better error handling
khinevich
changed the title
Programming exercises: Run PECV Bench in ArtemisDevelopment: Run PECV Bench in Artemis
WalkthroughAdds a PECV‑Bench Hyperion benchmark automation suite: ignore patterns and requirements, README, logging and auth utilities, course create/delete, exercise packaging/import and consistency checks, and an orchestration script to run the end‑to‑end workflow against an Artemis server. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant Orchestrator as run_pecv_bench_in_artemis.py
participant GitRepo as PECV-Bench Repo
participant FS as Filesystem
participant PECVCLI as PECV-Bench CLI
participant Artemis as Artemis Server
User->>Orchestrator: run with config.ini
Orchestrator->>Orchestrator: load config, init logging
Orchestrator->>GitRepo: clone or pull PECV-Bench
Orchestrator->>FS: install/editable dependencies
Orchestrator->>PECVCLI: generate/materialize variants
Orchestrator->>Artemis: authenticate (admin)
Orchestrator->>Artemis: create PECV-Bench course (POST)
par Per-variant (parallel)
Orchestrator->>FS: convert variant -> intermediate zips + final ZIP
Orchestrator->>Artemis: import exercise (multipart upload)
Artemis-->>Orchestrator: return exercise ID
Orchestrator->>FS: store mapping variant -> server ID
end
par Consistency checks (concurrent)
Orchestrator->>Artemis: request consistency check (POST)
Artemis-->>Orchestrator: return consistency results
Orchestrator->>FS: persist per-variant JSON result
end
Orchestrator->>PECVCLI: run analysis / generate plots
Orchestrator->>FS: generate final report and inject summary
Orchestrator->>User: output results directory
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (1)
🧰 Additional context used🧠 Learnings (7)📓 Common learnings📚 Learning: 2024-07-09T19:08:50.316ZApplied to files:
📚 Learning: 2024-07-09T19:07:45.295ZApplied to files:
📚 Learning: 2024-10-08T15:35:42.972ZApplied to files:
📚 Learning: 2024-10-31T20:40:39.930ZApplied to files:
📚 Learning: 2024-10-12T17:52:44.113ZApplied to files:
📚 Learning: 2024-07-05T20:17:25.449ZApplied to files:
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
🔇 Additional comments (1)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 12
🧹 Nitpick comments (14)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/.gitignore (1)
1-5: Consider ignoring the generated results directory.The PR description indicates that the workflow outputs reports to
results/artemis-benchmark/report.md. This is a generated artifact and should likely be added to.gitignoreto prevent committing benchmark reports and intermediate outputs.Apply this diff to add the results directory:
__pycache__/ .env venv .idea/ pecv-bench/ +results/Additionally, consider normalizing
venvtovenv/for clarity that it refers to a directory:-venv +venv/supporting_scripts/course-scripts/hyperion-benchmark-workflow/requirements.txt (1)
1-1: Consider pinning the dependency version for reproducibility.Unpinned dependencies can lead to different behavior across environments. Consider specifying a version (e.g.,
requests>=2.28.0,<3.0.0orrequests==2.32.3).-requests +requests>=2.28.0,<3.0.0supporting_scripts/course-scripts/hyperion-benchmark-workflow/create_pecv_bench_course.py (2)
94-94: Remove extraneousfprefix from string without placeholders.- logging.info(f"Waiting 2 seconds for server cleanup before recreation") + logging.info("Waiting 2 seconds for server cleanup before recreation")
137-137: Remove extraneousfprefix from string without placeholders.- logging.info(f"Sending DELETE request, it can take around 30 seconds to delete a course") + logging.info("Sending DELETE request, it can take around 30 seconds to delete a course")supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py (3)
11-11: Remove unusedTuplefrom imports.-from typing import Dict, Any, List, Tuple +from typing import Dict, Any, List
192-192: Remove extraneousfprefix from string without placeholders.- logging.info(f"Multipart form-data body and content type prepared.") + logging.info("Multipart form-data body and content type prepared.")
241-242: Avoid splitting the same string twice.Minor optimization to reduce redundant string operations.
- exercise = exercise_variant_local_id.split(":")[0] - variant = exercise_variant_local_id.split(":")[1] + exercise, variant = exercise_variant_local_id.split(":")supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py (7)
1-14: Remove unused imports.Static analysis correctly identifies several unused imports that should be removed:
json(line 3)configfromlogging(line 8) — also shadowed by theconfigparservariable on line 17check_exercise_consistency(line 13)Pathfrompathlib(line 14)from concurrent.futures import ThreadPoolExecutor, as_completed import configparser -import json import subprocess import os import sys import requests -from logging import config from typing import Dict, List, Tuple from logging_config import logging from utils import login_as_admin, SERVER_URL from create_pecv_bench_course import create_pecv_bench_course -from manage_programming_exercise import check_exercise_consistency, convert_variant_to_zip, import_programming_exercise, process_variant_consistency_check -from pathlib import Path +from manage_programming_exercise import convert_variant_to_zip, import_programming_exercise, process_variant_consistency_check
30-31: Use parameterpecv_bench_dirinstead of globalPECV_BENCH_PATHin log message.The log message references the global
PECV_BENCH_PATHinstead of the function parameterpecv_bench_dir, which could be misleading if different values are passed.if os.path.exists(pecv_bench_dir): - logging.info(f"Directory {PECV_BENCH_PATH} already exists. Pulling latest changes.") + logging.info(f"Directory {pecv_bench_dir} already exists. Pulling latest changes.")
100-102: Use bareraiseto preserve the original traceback.When re-raising an exception, use
raisewithout specifying the exception name to preserve the full traceback.except Exception as e: logging.exception(f"Failed creating exercise from git patch file: {e}") - raise e + raise
131-132: Remove unused variable assignment.The
pathvariable is assigned but never used.- path = manager.materialize_variant(variant.variant_id, force=True) + manager.materialize_variant(variant.variant_id, force=True)
242-251: Consider refactoring the import verification pattern.The imports of
VariantManagerandExerciseIdentifierat lines 243-244 are unused inmain()(flagged by static analysis). They appear to be a "fail-fast" check before proceeding. A clearer pattern would be to either:
- Add a comment explaining the intent, or
- Move the import check into a dedicated function
+ # Verify pecv-bench imports are available before proceeding try: - from cli.commands.variants import VariantManager - from cli.utils import ExerciseIdentifier - + from cli.commands.variants import VariantManager # noqa: F401 + from cli.utils import ExerciseIdentifier # noqa: F401 + for EXERCISE in EXERCISES: create_all_variants(COURSE, EXERCISE)
300-300: Uselogginginstead ofprint()for consistency.All other output in this file uses the
loggingmodule. Thisprint()statement breaks consistency and may not appear in log files.- print(f"\n{programming_exercises}\n") + logging.debug(f"Programming exercises mapping: {programming_exercises}")
303-305: Track this TODO for dynamic model name configuration.The hardcoded
model_namewith NOTE comments indicates planned work to parse fromapplication-local.yml. This is acceptable for an initial implementation but should be tracked.Would you like me to open an issue to track implementing the YAML parser for extracting the model name from the Spring configuration?
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/config.iniis excluded by!**/*.ini
📒 Files selected for processing (8)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/.gitignore(1 hunks)supporting_scripts/course-scripts/hyperion-benchmark-workflow/README.md(1 hunks)supporting_scripts/course-scripts/hyperion-benchmark-workflow/create_pecv_bench_course.py(1 hunks)supporting_scripts/course-scripts/hyperion-benchmark-workflow/logging_config.py(1 hunks)supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py(1 hunks)supporting_scripts/course-scripts/hyperion-benchmark-workflow/requirements.txt(1 hunks)supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py(1 hunks)supporting_scripts/course-scripts/hyperion-benchmark-workflow/utils.py(1 hunks)
🧰 Additional context used
🧠 Learnings (7)
📓 Common learnings
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 8802
File: src/main/resources/templates/rust/exercise/src/context.rs:1-1
Timestamp: 2024-10-08T15:35:42.972Z
Learning: Code inside the `exercise` directories in the Artemis platform is provided to students and is intended for them to implement. TODO comments in these directories are meant to guide students and should not be addressed in the PR.
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 8802
File: src/main/resources/templates/rust/exercise/src/context.rs:1-1
Timestamp: 2024-08-05T00:11:50.650Z
Learning: Code inside the `exercise` directories in the Artemis platform is provided to students and is intended for them to implement. TODO comments in these directories are meant to guide students and should not be addressed in the PR.
📚 Learning: 2024-08-05T00:11:50.650Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 8802
File: src/main/resources/templates/rust/exercise/src/context.rs:1-1
Timestamp: 2024-08-05T00:11:50.650Z
Learning: Code inside the `exercise` directories in the Artemis platform is provided to students and is intended for them to implement. TODO comments in these directories are meant to guide students and should not be addressed in the PR.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/README.md
📚 Learning: 2024-10-31T20:40:39.930Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 9626
File: src/main/resources/templates/c_sharp/exercise/BubbleSort.cs:9-12
Timestamp: 2024-10-31T20:40:39.930Z
Learning: In the Artemis project, files under the `exercise` directory are incomplete exercises intended to be completed by the student. TODO comments in these files are intentional and should not be implemented.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/README.md
📚 Learning: 2024-10-12T17:52:44.113Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 9440
File: src/main/resources/templates/typescript/test/package.json:9-11
Timestamp: 2024-10-12T17:52:44.113Z
Learning: In the Artemis TypeScript programming exercise templates, the "assignment" workspace directory specified in `src/main/resources/templates/typescript/test/package.json` is copied in by the CI beforehand.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/README.md
📚 Learning: 2024-10-08T15:35:42.972Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 8859
File: src/main/resources/templates/gitlabci/javascript/regularRuns/.gitlab-ci.yml:20-21
Timestamp: 2024-10-08T15:35:42.972Z
Learning: In the Artemis project, `npm audit` is run manually by instructors as part of `npm install` to prioritize speed in the CI process.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/README.md
📚 Learning: 2024-10-31T20:53:33.463Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 9626
File: src/main/resources/templates/c_sharp/exercise/.gitignore:396-402
Timestamp: 2024-10-31T20:53:33.463Z
Learning: In this project, .gitignore files generated by `dotnet new gitignore` should be kept as-is, even if they contain duplicate patterns.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/.gitignore
📚 Learning: 2025-06-10T12:26:42.449Z
Learnt from: Wallenstein61
Repo: ls1intum/Artemis PR: 10989
File: src/main/java/de/tum/cit/aet/artemis/programming/service/ProgrammingExerciseImportFromFileService.java:108-117
Timestamp: 2025-06-10T12:26:42.449Z
Learning: In ProgrammingExerciseImportFromFileService, the current directory traversal logic for sharing imports (walking importExerciseDir before extraction) is working correctly in practice and should not be changed to more complex solutions, even if there are theoretical issues with the approach.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
🪛 GitHub Check: Codacy Static Code Analysis
supporting_scripts/course-scripts/hyperion-benchmark-workflow/requirements.txt
[notice] 1-1: supporting_scripts/course-scripts/hyperion-benchmark-workflow/requirements.txt#L1
Statement seems to have no effect
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
[warning] 3-3: supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py#L3
Unused import sys
[warning] 9-9: supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py#L9
Unused start imported from tracemalloc
[warning] 11-11: supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py#L11
Unused Tuple imported from typing
[notice] 31-31: supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py#L31
Trailing whitespace
[notice] 157-157: supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py#L157
Trailing whitespace
supporting_scripts/course-scripts/hyperion-benchmark-workflow/create_pecv_bench_course.py
[warning] 1-1: supporting_scripts/course-scripts/hyperion-benchmark-workflow/create_pecv_bench_course.py#L1
Unused import sys
[notice] 83-83: supporting_scripts/course-scripts/hyperion-benchmark-workflow/create_pecv_bench_course.py#L83
Trailing whitespace
[notice] 106-106: supporting_scripts/course-scripts/hyperion-benchmark-workflow/create_pecv_bench_course.py#L106
Trailing whitespace
supporting_scripts/course-scripts/hyperion-benchmark-workflow/utils.py
[warning] 4-4: supporting_scripts/course-scripts/hyperion-benchmark-workflow/utils.py#L4
Unused List imported from typing
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py
[warning] 3-3: supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py#L3
Unused import json
[warning] 13-13: supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py#L13
Unused check_exercise_consistency imported from manage_programming_exercise
[warning] 14-14: supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py#L14
Unused Path imported from pathlib
[notice] 36-36: supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py#L36
Trailing whitespace
[notice] 60-60: supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py#L60
Trailing whitespace
[notice] 131-131: supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py#L131
Unused variable 'path'
[notice] 141-141: supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py#L141
Trailing whitespace
[notice] 160-160: supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py#L160
Trailing whitespace
[notice] 161-161: supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py#L161
Trailing whitespace
[warning] 243-243: supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py#L243
Unused VariantManager imported from cli.commands.variants
[warning] 244-244: supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py#L244
Unused ExerciseIdentifier imported from cli.utils
[notice] 370-370: supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py#L370
Trailing whitespace
🪛 LanguageTool
supporting_scripts/course-scripts/hyperion-benchmark-workflow/README.md
[grammar] ~25-~25: Ensure spelling is correct
Context: ...ate a virtual environment in hyperion-benchmarl-workflow folder: ```shell pyt...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
🪛 Ruff (0.14.6)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
78-78: Do not catch blind exception: Exception
(BLE001)
192-192: f-string without any placeholders
Remove extraneous f prefix
(F541)
253-253: Consider moving this statement to an else block
(TRY300)
254-254: Do not catch blind exception: Exception
(BLE001)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/create_pecv_bench_course.py
94-94: f-string without any placeholders
Remove extraneous f prefix
(F541)
100-100: Create your own exception
(TRY002)
100-100: Avoid specifying long messages outside the exception class
(TRY003)
107-110: Create your own exception
(TRY002)
107-110: Avoid specifying long messages outside the exception class
(TRY003)
137-137: f-string without any placeholders
Remove extraneous f prefix
(F541)
144-144: Do not catch blind exception: Exception
(BLE001)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/utils.py
18-18: Do not perform function call requests.Session in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
36-37: Create your own exception
(TRY002)
36-37: Avoid specifying long messages outside the exception class
(TRY003)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py
17-17: Redefinition of unused config from line 8
(F811)
34-34: Starting a process with a partial executable path
(S607)
38-38: Starting a process with a partial executable path
(S607)
42-42: Starting a process with a partial executable path
(S607)
55-55: subprocess call: check for execution of untrusted input
(S603)
56-56: Starting a process with a partial executable path
(S607)
70-70: subprocess call: check for execution of untrusted input
(S603)
102-102: Use raise without specifying exception name
Remove exception name
(TRY201)
131-131: Local variable path is assigned to but never used
Remove assignment to unused variable path
(F841)
133-133: Do not catch blind exception: Exception
(BLE001)
138-138: Do not catch blind exception: Exception
(BLE001)
171-171: Do not catch blind exception: Exception
(BLE001)
225-225: Do not catch blind exception: Exception
(BLE001)
296-296: Do not catch blind exception: Exception
(BLE001)
334-334: Do not catch blind exception: Exception
(BLE001)
352-352: subprocess call: check for execution of untrusted input
(S603)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Codacy Static Code Analysis
🔇 Additional comments (1)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/logging_config.py (1)
1-4: LGTM!Simple and effective centralized logging configuration for the workflow scripts.
supporting_scripts/course-scripts/hyperion-benchmark-workflow/create_pecv_bench_course.py
Outdated
Show resolved
Hide resolved
supporting_scripts/course-scripts/hyperion-benchmark-workflow/create_pecv_bench_course.py
Show resolved
Hide resolved
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
Outdated
Show resolved
Hide resolved
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
Outdated
Show resolved
Hide resolved
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
Outdated
Show resolved
Hide resolved
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py
Outdated
Show resolved
Hide resolved
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py
Outdated
Show resolved
Hide resolved
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py
Show resolved
Hide resolved
supporting_scripts/course-scripts/hyperion-benchmark-workflow/utils.py
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Actionable comments posted: 3
♻️ Duplicate comments (1)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py (1)
81-93:problem_statement_contentmay still be undefined when checked on line 92.The variable
problem_statement_contentis only assigned when the file exists (line 83), but line 92 checksif problem_statement_content is not Nonewithout first initializing it. If the file doesn't exist, this will raiseNameError(not just skip the assignment).Apply this diff to initialize the variable before the conditional:
# Overwrite problem statement, exercise ID, course ID, title and shortName in the config file problem_statement_file_path = os.path.join(variant_path, "problem-statement.md") + problem_statement_content = None if os.path.exists(problem_statement_file_path): problem_statement_content = read_problem_statement(problem_statement_file_path)
🧹 Nitpick comments (9)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py (2)
189-190: Remove unnecessary f-string prefix.This f-string has no placeholders.
body, content_type = urllib3.filepost.encode_multipart_formdata(files_payload) - logging.info(f"Multipart form-data body and content type prepared.") + logging.info("Multipart form-data body and content type prepared.")
239-240: PotentialIndexErrorifexercise_variant_local_iddoesn't contain":".If the format of
exercise_variant_local_idis unexpected (missing:),split(":")[1]will raise anIndexError.Consider adding validation or using safer unpacking:
- exercise = exercise_variant_local_id.split(":")[0] - variant = exercise_variant_local_id.split(":")[1] + parts = exercise_variant_local_id.split(":") + if len(parts) != 2: + logging.error(f"Invalid exercise_variant_local_id format: {exercise_variant_local_id}") + return f"[{exercise_variant_local_id}] error - invalid format" + exercise, variant = partssupporting_scripts/course-scripts/hyperion-benchmark-workflow/utils.py (1)
9-12: Consider adding fallback values for configuration constants.If any of these keys are missing from
config.ini, aNoOptionErrorwill be raised at module import time. Other scripts in this PR usefallback=parameters for safer config loading.-CLIENT_URL: str = config.get('Settings', 'client_url') -SERVER_URL: str = config.get('Settings', 'server_url') -ADMIN_USER: str = config.get('Settings', 'admin_user') -ADMIN_PASSWORD: str = config.get('Settings', 'admin_password') +CLIENT_URL: str = config.get('Settings', 'client_url', fallback="") +SERVER_URL: str = config.get('Settings', 'server_url', fallback="") +ADMIN_USER: str = config.get('Settings', 'admin_user', fallback="") +ADMIN_PASSWORD: str = config.get('Settings', 'admin_password', fallback="")Alternatively, if these are truly required, consider raising a clearer error message when they're missing.
supporting_scripts/course-scripts/hyperion-benchmark-workflow/create_pecv_bench_course.py (3)
93-94: Remove unnecessary f-string prefix.This f-string has no placeholders.
- logging.info(f"Waiting 2 seconds for server cleanup before recreation") + logging.info("Waiting 2 seconds for server cleanup before recreation")
121-122: Add error handling for the course list request.If the GET request fails or returns non-200,
courseResponse.json()may raise an exception or return unexpected data.courseResponse: requests.Response = session.get(f"{SERVER_URL}/core/courses") + if courseResponse.status_code != 200: + logging.error(f"Failed to fetch courses. Status code: {courseResponse.status_code}") + return False courses = courseResponse.json()
139-139: Remove unnecessary f-string prefix.This f-string has no placeholders.
- logging.info(f"Sending DELETE request, it can take around 30 seconds to delete a course") + logging.info("Sending DELETE request, it can take around 30 seconds to delete a course")supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py (3)
100-102: Use bareraiseto preserve the original traceback.When re-raising the same exception, use
raisewithout the exception variable to preserve the full traceback.except Exception as e: logging.exception(f"Failed creating exercise from git patch file: {e}") - raise e + raise
131-131: Remove unused variable assignment.The
pathvariable is assigned but never used.- path = manager.materialize_variant(variant.variant_id, force=True) + manager.materialize_variant(variant.variant_id, force=True)
304-306: Consider extracting model configuration from application config.The TODO/NOTE comments indicate future work to extract model configuration from
application-local.yml. This is a good reminder for follow-up work.Would you like me to open an issue to track implementing the YAML parser for extracting the model configuration from
application-local.yml?
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/README.md(1 hunks)supporting_scripts/course-scripts/hyperion-benchmark-workflow/create_pecv_bench_course.py(1 hunks)supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py(1 hunks)supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py(1 hunks)supporting_scripts/course-scripts/hyperion-benchmark-workflow/utils.py(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- supporting_scripts/course-scripts/hyperion-benchmark-workflow/README.md
🧰 Additional context used
🧠 Learnings (21)
📓 Common learnings
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 8802
File: src/main/resources/templates/rust/exercise/src/context.rs:1-1
Timestamp: 2024-10-08T15:35:42.972Z
Learning: Code inside the `exercise` directories in the Artemis platform is provided to students and is intended for them to implement. TODO comments in these directories are meant to guide students and should not be addressed in the PR.
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 8802
File: src/main/resources/templates/rust/exercise/src/context.rs:1-1
Timestamp: 2024-08-05T00:11:50.650Z
Learning: Code inside the `exercise` directories in the Artemis platform is provided to students and is intended for them to implement. TODO comments in these directories are meant to guide students and should not be addressed in the PR.
📚 Learning: 2024-11-26T20:26:32.835Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 9751
File: src/main/resources/templates/go/solution/bubblesort.go:11-19
Timestamp: 2024-11-26T20:26:32.835Z
Learning: In this project, prefer simple and understandable solutions that demonstrate basic principles over optimized implementations.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py
📚 Learning: 2025-08-11T12:57:51.535Z
Learnt from: ekayandan
Repo: ls1intum/Artemis PR: 11027
File: src/main/java/de/tum/cit/aet/artemis/programming/service/GitService.java:1455-1465
Timestamp: 2025-08-11T12:57:51.535Z
Learning: In the Artemis project, when ekayandan responds with "same as above" to a code review suggestion, it means they want to defer the suggested change to a follow-up PR to keep the current PR scope minimal and focused on the core functionality being implemented.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.pysupporting_scripts/course-scripts/hyperion-benchmark-workflow/utils.py
📚 Learning: 2024-07-05T20:02:19.606Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 8802
File: src/main/resources/templates/gitlabci/rust/regularRuns/.gitlab-ci.yml:22-25
Timestamp: 2024-07-05T20:02:19.606Z
Learning: In the Artemis project, comments should focus on providing reasoning for non-obvious situations rather than describing self-explanatory code.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py
📚 Learning: 2025-06-06T14:45:29.888Z
Learnt from: Wallenstein61
Repo: ls1intum/Artemis PR: 10989
File: src/main/webapp/app/programming/manage/update/programming-exercise-update.component.ts:947-971
Timestamp: 2025-06-06T14:45:29.888Z
Learning: When introducing new features in large PRs, developers may intentionally choose not to refactor existing code to minimize impact on the source codebase, prioritize stability, and keep changes focused and reviewable. This is a valid engineering decision that should be respected.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.pysupporting_scripts/course-scripts/hyperion-benchmark-workflow/utils.py
📚 Learning: 2024-07-02T17:40:10.186Z
Learnt from: bensofficial
Repo: ls1intum/Artemis PR: 8951
File: src/main/java/de/tum/in/www1/artemis/service/connectors/localci/buildagent/BuildAgentSSHKeyService.java:1-1
Timestamp: 2024-07-02T17:40:10.186Z
Learning: Ensure to verify the presence of wildcard imports before commenting on their usage in the code review.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/utils.pysupporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.pysupporting_scripts/course-scripts/hyperion-benchmark-workflow/create_pecv_bench_course.py
📚 Learning: 2024-07-09T19:08:50.316Z
Learnt from: florian-glombik
Repo: ls1intum/Artemis PR: 8858
File: src/main/webapp/app/shared/sidebar/sidebar.component.ts:59-59
Timestamp: 2024-07-09T19:08:50.316Z
Learning: For the PR ls1intum/Artemis#8858, avoid suggesting the removal of trivially inferred type annotations.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/utils.py
📚 Learning: 2024-10-08T15:35:42.972Z
Learnt from: julian-christl
Repo: ls1intum/Artemis PR: 7829
File: src/main/java/de/tum/in/www1/artemis/web/rest/ComplaintResponseResource.java:0-0
Timestamp: 2024-10-08T15:35:42.972Z
Learning: The user has fixed the issue regarding the redundant wildcard import in `ComplaintResponseResource.java` by removing it in commit 7e392e0.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/utils.pysupporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.pysupporting_scripts/course-scripts/hyperion-benchmark-workflow/create_pecv_bench_course.py
📚 Learning: 2024-10-08T15:35:42.972Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 8802
File: src/main/resources/templates/rust/exercise/src/context.rs:1-1
Timestamp: 2024-10-08T15:35:42.972Z
Learning: Code inside the `exercise` directories in the Artemis platform is provided to students and is intended for them to implement. TODO comments in these directories are meant to guide students and should not be addressed in the PR.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/utils.pysupporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2024-10-04T23:23:49.139Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 9261
File: src/main/resources/templates/c_plus_plus/test/testUtils/TestFailedError.py:6-6
Timestamp: 2024-10-04T23:23:49.139Z
Learning: In the Artemis project, when adding new programming language templates like C++, files such as `src/main/resources/templates/c_plus_plus/test/testUtils/TestFailedError.py` may be copied from existing templates without modifications. In such cases, suggesting modifications to these files (e.g., removing unnecessary `pass` statements) may not be necessary.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/utils.pysupporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2024-10-20T18:35:39.224Z
Learnt from: SamuelRoettgermann
Repo: ls1intum/Artemis PR: 9303
File: src/main/java/de/tum/in/www1/artemis/service/exam/StudentExamService.java:266-267
Timestamp: 2024-10-20T18:35:39.224Z
Learning: When reviewing code in this project, avoid suggesting code changes that are outside the scope of the PR.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/utils.py
📚 Learning: 2024-10-31T20:40:30.235Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 9626
File: src/main/resources/templates/c_sharp/exercise/BubbleSort.cs:3-4
Timestamp: 2024-10-31T20:40:30.235Z
Learning: In this project, files under the `exercise` directory are incomplete exercises to be completed by students. Therefore, review comments suggesting implementations in these files may not be necessary.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/utils.pysupporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2024-10-08T15:35:48.767Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 9261
File: src/main/resources/templates/c_plus_plus/test/testUtils/Utils.py:15-35
Timestamp: 2024-10-08T15:35:48.767Z
Learning: In the Artemis project, when adding new programming language templates (e.g., C++), if files like `src/main/resources/templates/c_plus_plus/test/testUtils/Utils.py` are copied from the original C template without significant changes, code style improvements such as adding type annotations are not required in the initial PR introducing them.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/utils.pysupporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2025-06-10T12:26:42.449Z
Learnt from: Wallenstein61
Repo: ls1intum/Artemis PR: 10989
File: src/main/java/de/tum/cit/aet/artemis/programming/service/ProgrammingExerciseImportFromFileService.java:108-117
Timestamp: 2025-06-10T12:26:42.449Z
Learning: In ProgrammingExerciseImportFromFileService, the current directory traversal logic for sharing imports (walking importExerciseDir before extraction) is working correctly in practice and should not be changed to more complex solutions, even if there are theoretical issues with the approach.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2024-10-15T08:00:53.531Z
Learnt from: Hialus
Repo: ls1intum/Artemis PR: 8607
File: src/main/java/de/tum/in/www1/artemis/web/rest/programming/ProgrammingExerciseResource.java:64-64
Timestamp: 2024-10-15T08:00:53.531Z
Learning: For the Artemis project, import statements are automatically managed by formatters and should not be commented on.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.pysupporting_scripts/course-scripts/hyperion-benchmark-workflow/create_pecv_bench_course.py
📚 Learning: 2024-10-12T17:52:44.113Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 9440
File: src/main/resources/templates/typescript/test/package.json:9-11
Timestamp: 2024-10-12T17:52:44.113Z
Learning: In the Artemis TypeScript programming exercise templates, the "assignment" workspace directory specified in `src/main/resources/templates/typescript/test/package.json` is copied in by the CI beforehand.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2024-10-31T20:40:39.930Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 9626
File: src/main/resources/templates/c_sharp/exercise/BubbleSort.cs:9-12
Timestamp: 2024-10-31T20:40:39.930Z
Learning: In the Artemis project, files under the `exercise` directory are incomplete exercises intended to be completed by the student. TODO comments in these files are intentional and should not be implemented.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2024-10-20T18:37:45.365Z
Learnt from: SamuelRoettgermann
Repo: ls1intum/Artemis PR: 9303
File: src/main/java/de/tum/in/www1/artemis/service/exam/StudentExamService.java:296-300
Timestamp: 2024-10-20T18:37:45.365Z
Learning: When reviewing code changes in `StudentExamService.saveSubmission`, if the PR aims to improve readability without changing logic, avoid suggesting changes that alter logic, such as adding exceptions in the default case of switch statements.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2024-10-12T16:13:02.782Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 9440
File: src/main/resources/templates/typescript/exercise/src/mergesort.ts:2-2
Timestamp: 2024-10-12T16:13:02.782Z
Learning: In files under the `exercise` directory, TODO comments are intentionally included for students to complete and should remain as is.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2025-01-25T17:22:31.410Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 10202
File: src/main/resources/templates/ruby/test/test/test_structural.rb:5-18
Timestamp: 2025-01-25T17:22:31.410Z
Learning: When reviewing exercise templates, avoid adding tests that would pass immediately with the starter code. Tests should verify the actual implementation that students need to complete, not just the presence of classes and methods defined in the template.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2024-10-31T20:41:04.600Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 9626
File: src/main/resources/templates/c_sharp/exercise/MergeSort.cs:9-12
Timestamp: 2024-10-31T20:41:04.600Z
Learning: In the C# programming language template, files under the `exercise` directory (e.g., `MergeSort.cs`) are intentionally left incomplete as exercises for students to complete, and should not be flagged for missing implementations during code reviews.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
🧬 Code graph analysis (1)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py (3)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/utils.py (1)
login_as_admin(14-16)supporting_scripts/course-scripts/hyperion-benchmark-workflow/create_pecv_bench_course.py (1)
create_pecv_bench_course(30-109)supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py (4)
check_exercise_consistency(208-224)convert_variant_to_zip(32-140)import_programming_exercise(142-205)process_variant_consistency_check(227-254)
🪛 Ruff (0.14.6)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py
17-17: Redefinition of unused config from line 8
(F811)
34-34: Starting a process with a partial executable path
(S607)
38-38: Starting a process with a partial executable path
(S607)
42-42: Starting a process with a partial executable path
(S607)
55-55: subprocess call: check for execution of untrusted input
(S603)
56-56: Starting a process with a partial executable path
(S607)
70-70: subprocess call: check for execution of untrusted input
(S603)
102-102: Use raise without specifying exception name
Remove exception name
(TRY201)
131-131: Local variable path is assigned to but never used
Remove assignment to unused variable path
(F841)
133-133: Do not catch blind exception: Exception
(BLE001)
138-138: Do not catch blind exception: Exception
(BLE001)
172-172: Do not catch blind exception: Exception
(BLE001)
226-226: Do not catch blind exception: Exception
(BLE001)
297-297: Do not catch blind exception: Exception
(BLE001)
335-335: Do not catch blind exception: Exception
(BLE001)
353-353: subprocess call: check for execution of untrusted input
(S603)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/utils.py
18-18: Do not perform function call requests.Session in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
36-37: Create your own exception
(TRY002)
36-37: Avoid specifying long messages outside the exception class
(TRY003)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
76-76: Do not catch blind exception: Exception
(BLE001)
190-190: f-string without any placeholders
Remove extraneous f prefix
(F541)
251-251: Consider moving this statement to an else block
(TRY300)
252-252: Do not catch blind exception: Exception
(BLE001)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/create_pecv_bench_course.py
93-93: f-string without any placeholders
Remove extraneous f prefix
(F541)
99-99: Create your own exception
(TRY002)
99-99: Avoid specifying long messages outside the exception class
(TRY003)
106-109: Create your own exception
(TRY002)
106-109: Avoid specifying long messages outside the exception class
(TRY003)
139-139: f-string without any placeholders
Remove extraneous f prefix
(F541)
146-146: Do not catch blind exception: Exception
(BLE001)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: Codacy Static Code Analysis
- GitHub Check: Analyse
🔇 Additional comments (4)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/utils.py (1)
14-39: LGTM!The authentication logic is straightforward and includes appropriate logging for success and failure cases.
supporting_scripts/course-scripts/hyperion-benchmark-workflow/create_pecv_bench_course.py (1)
30-109: Course creation and retry logic looks solid.The implementation handles the 400 status (existing course) gracefully by attempting deletion and retry. The flow is clear and well-logged.
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py (2)
27-64: Git operations include appropriate cleanup before pull.The
git reset --hard HEADandgit clean -fdbefore pulling ensures a clean state, which is good practice for automation scripts.
265-299: Thread pool implementation for variant imports looks good.The concurrent import logic properly collects results and handles errors. The dictionary update after
as_completedis thread-safe since it happens in the main thread.
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
Show resolved
Hide resolved
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py
Outdated
Show resolved
Hide resolved
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (2)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py (1)
257-262: Add validation forcourse_idafter creation.While
.get("id")prevents aKeyError, ifcourse_idisNone(e.g., due to an error response), subsequent operations will fail with unclear errors. This was flagged in a previous review but not fully addressed.# Step 5: Create PECV Bench Course response_data = create_pecv_bench_course(session) course_id = response_data.get("id") + if course_id is None: + logging.error("Failed to retrieve course ID from response. Exiting.") + sys.exit(1)supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py (1)
80-93:UnboundLocalErrorifproblem-statement.mddoes not exist.
problem_statement_contentis only assigned when the file exists (line 83), but the check at line 92 assumes it's defined. If the file is missing, this will raiseNameError: name 'problem_statement_content' is not defined.# Overwrite problem statement, exercise ID, course ID, title and shortName in the config file problem_statement_file_path = os.path.join(variant_path, "problem-statement.md") + problem_statement_content = None if os.path.exists(problem_statement_file_path): problem_statement_content = read_problem_statement(problem_statement_file_path)
🧹 Nitpick comments (3)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py (2)
99-101: Use bareraiseinstead ofraise e.When re-raising a caught exception, prefer bare
raiseto preserve the original traceback.except Exception as e: logging.exception(f"Failed creating exercise from git patch file: {e}") - raise e + raise
128-134: Remove unused variable assignment.The return value from
materialize_variantis assigned topathbut never used.try: # 'variant.variant_id' gets the ID string like "001" - path = manager.materialize_variant(variant.variant_id, force=True) + manager.materialize_variant(variant.variant_id, force=True) logging.info(f"Generated {variant.variant_id}") except Exception as e: logging.exception(f"Failed to create variant {variant.variant_id}: {e}") continuesupporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py (1)
189-190: Remove extraneous f-string prefix.The f-string has no placeholders.
- logging.info(f"Multipart form-data body and content type prepared.") + logging.info("Multipart form-data body and content type prepared.")
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py(1 hunks)supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py(1 hunks)
🧰 Additional context used
🧠 Learnings (20)
📓 Common learnings
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 8802
File: src/main/resources/templates/rust/exercise/src/context.rs:1-1
Timestamp: 2024-10-08T15:35:42.972Z
Learning: Code inside the `exercise` directories in the Artemis platform is provided to students and is intended for them to implement. TODO comments in these directories are meant to guide students and should not be addressed in the PR.
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 8802
File: src/main/resources/templates/rust/exercise/src/context.rs:1-1
Timestamp: 2024-08-05T00:11:50.650Z
Learning: Code inside the `exercise` directories in the Artemis platform is provided to students and is intended for them to implement. TODO comments in these directories are meant to guide students and should not be addressed in the PR.
📚 Learning: 2024-11-26T20:26:32.835Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 9751
File: src/main/resources/templates/go/solution/bubblesort.go:11-19
Timestamp: 2024-11-26T20:26:32.835Z
Learning: In this project, prefer simple and understandable solutions that demonstrate basic principles over optimized implementations.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py
📚 Learning: 2025-08-11T12:57:51.535Z
Learnt from: ekayandan
Repo: ls1intum/Artemis PR: 11027
File: src/main/java/de/tum/cit/aet/artemis/programming/service/GitService.java:1455-1465
Timestamp: 2025-08-11T12:57:51.535Z
Learning: In the Artemis project, when ekayandan responds with "same as above" to a code review suggestion, it means they want to defer the suggested change to a follow-up PR to keep the current PR scope minimal and focused on the core functionality being implemented.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py
📚 Learning: 2024-07-05T20:02:19.606Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 8802
File: src/main/resources/templates/gitlabci/rust/regularRuns/.gitlab-ci.yml:22-25
Timestamp: 2024-07-05T20:02:19.606Z
Learning: In the Artemis project, comments should focus on providing reasoning for non-obvious situations rather than describing self-explanatory code.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py
📚 Learning: 2025-06-06T14:45:29.888Z
Learnt from: Wallenstein61
Repo: ls1intum/Artemis PR: 10989
File: src/main/webapp/app/programming/manage/update/programming-exercise-update.component.ts:947-971
Timestamp: 2025-06-06T14:45:29.888Z
Learning: When introducing new features in large PRs, developers may intentionally choose not to refactor existing code to minimize impact on the source codebase, prioritize stability, and keep changes focused and reviewable. This is a valid engineering decision that should be respected.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py
📚 Learning: 2024-10-31T20:40:30.235Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 9626
File: src/main/resources/templates/c_sharp/exercise/BubbleSort.cs:3-4
Timestamp: 2024-10-31T20:40:30.235Z
Learning: In this project, files under the `exercise` directory are incomplete exercises to be completed by students. Therefore, review comments suggesting implementations in these files may not be necessary.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2024-07-02T17:40:10.186Z
Learnt from: bensofficial
Repo: ls1intum/Artemis PR: 8951
File: src/main/java/de/tum/in/www1/artemis/service/connectors/localci/buildagent/BuildAgentSSHKeyService.java:1-1
Timestamp: 2024-07-02T17:40:10.186Z
Learning: Ensure to verify the presence of wildcard imports before commenting on their usage in the code review.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2024-10-15T08:00:53.531Z
Learnt from: Hialus
Repo: ls1intum/Artemis PR: 8607
File: src/main/java/de/tum/in/www1/artemis/web/rest/programming/ProgrammingExerciseResource.java:64-64
Timestamp: 2024-10-15T08:00:53.531Z
Learning: For the Artemis project, import statements are automatically managed by formatters and should not be commented on.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2024-10-08T15:35:42.972Z
Learnt from: julian-christl
Repo: ls1intum/Artemis PR: 7829
File: src/main/java/de/tum/in/www1/artemis/web/rest/ComplaintResponseResource.java:0-0
Timestamp: 2024-10-08T15:35:42.972Z
Learning: The user has fixed the issue regarding the redundant wildcard import in `ComplaintResponseResource.java` by removing it in commit 7e392e0.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2024-10-12T17:52:44.113Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 9440
File: src/main/resources/templates/typescript/test/package.json:9-11
Timestamp: 2024-10-12T17:52:44.113Z
Learning: In the Artemis TypeScript programming exercise templates, the "assignment" workspace directory specified in `src/main/resources/templates/typescript/test/package.json` is copied in by the CI beforehand.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2025-06-10T12:26:42.449Z
Learnt from: Wallenstein61
Repo: ls1intum/Artemis PR: 10989
File: src/main/java/de/tum/cit/aet/artemis/programming/service/ProgrammingExerciseImportFromFileService.java:108-117
Timestamp: 2025-06-10T12:26:42.449Z
Learning: In ProgrammingExerciseImportFromFileService, the current directory traversal logic for sharing imports (walking importExerciseDir before extraction) is working correctly in practice and should not be changed to more complex solutions, even if there are theoretical issues with the approach.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2024-10-08T15:35:42.972Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 8802
File: src/main/resources/templates/rust/exercise/src/context.rs:1-1
Timestamp: 2024-10-08T15:35:42.972Z
Learning: Code inside the `exercise` directories in the Artemis platform is provided to students and is intended for them to implement. TODO comments in these directories are meant to guide students and should not be addressed in the PR.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2024-10-31T20:40:39.930Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 9626
File: src/main/resources/templates/c_sharp/exercise/BubbleSort.cs:9-12
Timestamp: 2024-10-31T20:40:39.930Z
Learning: In the Artemis project, files under the `exercise` directory are incomplete exercises intended to be completed by the student. TODO comments in these files are intentional and should not be implemented.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2024-10-04T23:23:49.139Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 9261
File: src/main/resources/templates/c_plus_plus/test/testUtils/TestFailedError.py:6-6
Timestamp: 2024-10-04T23:23:49.139Z
Learning: In the Artemis project, when adding new programming language templates like C++, files such as `src/main/resources/templates/c_plus_plus/test/testUtils/TestFailedError.py` may be copied from existing templates without modifications. In such cases, suggesting modifications to these files (e.g., removing unnecessary `pass` statements) may not be necessary.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2024-10-20T18:37:45.365Z
Learnt from: SamuelRoettgermann
Repo: ls1intum/Artemis PR: 9303
File: src/main/java/de/tum/in/www1/artemis/service/exam/StudentExamService.java:296-300
Timestamp: 2024-10-20T18:37:45.365Z
Learning: When reviewing code changes in `StudentExamService.saveSubmission`, if the PR aims to improve readability without changing logic, avoid suggesting changes that alter logic, such as adding exceptions in the default case of switch statements.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2024-10-12T16:13:02.782Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 9440
File: src/main/resources/templates/typescript/exercise/src/mergesort.ts:2-2
Timestamp: 2024-10-12T16:13:02.782Z
Learning: In files under the `exercise` directory, TODO comments are intentionally included for students to complete and should remain as is.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2024-10-08T15:35:48.767Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 9261
File: src/main/resources/templates/c_plus_plus/test/testUtils/Utils.py:15-35
Timestamp: 2024-10-08T15:35:48.767Z
Learning: In the Artemis project, when adding new programming language templates (e.g., C++), if files like `src/main/resources/templates/c_plus_plus/test/testUtils/Utils.py` are copied from the original C template without significant changes, code style improvements such as adding type annotations are not required in the initial PR introducing them.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2025-01-25T17:22:31.410Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 10202
File: src/main/resources/templates/ruby/test/test/test_structural.rb:5-18
Timestamp: 2025-01-25T17:22:31.410Z
Learning: When reviewing exercise templates, avoid adding tests that would pass immediately with the starter code. Tests should verify the actual implementation that students need to complete, not just the presence of classes and methods defined in the template.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2024-10-31T20:41:04.600Z
Learnt from: magaupp
Repo: ls1intum/Artemis PR: 9626
File: src/main/resources/templates/c_sharp/exercise/MergeSort.cs:9-12
Timestamp: 2024-10-31T20:41:04.600Z
Learning: In the C# programming language template, files under the `exercise` directory (e.g., `MergeSort.cs`) are intentionally left incomplete as exercises for students to complete, and should not be flagged for missing implementations during code reviews.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
📚 Learning: 2025-04-22T10:22:01.154Z
Learnt from: tobias-lippert
Repo: ls1intum/Artemis PR: 10714
File: src/main/webapp/app/exercise/exercise-create-buttons/exercise-manage-button/exercise-manage-button.component.ts:38-43
Timestamp: 2025-04-22T10:22:01.154Z
Learning: In the Artemis codebase, the ExerciseType enum has uppercase keys (PROGRAMMING, MODELING, etc.) but lowercase string values ('programming', 'modeling', etc.), which are used directly in translation keys and route paths.
Applied to files:
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
🧬 Code graph analysis (1)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py (3)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/utils.py (1)
login_as_admin(14-16)supporting_scripts/course-scripts/hyperion-benchmark-workflow/create_pecv_bench_course.py (1)
create_pecv_bench_course(30-109)supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py (4)
check_exercise_consistency(208-224)convert_variant_to_zip(32-140)import_programming_exercise(142-205)process_variant_consistency_check(227-254)
🪛 Ruff (0.14.6)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py
33-33: Starting a process with a partial executable path
(S607)
37-37: Starting a process with a partial executable path
(S607)
41-41: Starting a process with a partial executable path
(S607)
54-54: subprocess call: check for execution of untrusted input
(S603)
55-55: Starting a process with a partial executable path
(S607)
69-69: subprocess call: check for execution of untrusted input
(S603)
101-101: Use raise without specifying exception name
Remove exception name
(TRY201)
130-130: Local variable path is assigned to but never used
Remove assignment to unused variable path
(F841)
132-132: Do not catch blind exception: Exception
(BLE001)
137-137: Do not catch blind exception: Exception
(BLE001)
171-171: Do not catch blind exception: Exception
(BLE001)
225-225: Do not catch blind exception: Exception
(BLE001)
296-296: Do not catch blind exception: Exception
(BLE001)
334-334: Do not catch blind exception: Exception
(BLE001)
352-352: subprocess call: check for execution of untrusted input
(S603)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py
76-76: Do not catch blind exception: Exception
(BLE001)
190-190: f-string without any placeholders
Remove extraneous f prefix
(F541)
251-251: Consider moving this statement to an else block
(TRY300)
252-252: Do not catch blind exception: Exception
(BLE001)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: Codacy Static Code Analysis
- GitHub Check: Analyse
🔇 Additional comments (14)
supporting_scripts/course-scripts/hyperion-benchmark-workflow/run_pecv_bench_in_artemis.py (7)
1-24: Configuration and imports look good.The configuration loading with fallback values is properly implemented. The
REFERENCEfallback issue from previous reviews has been addressed.
26-64: Repository management logic is sound.The function properly handles both fresh clone and update scenarios with appropriate cleanup (
git reset --hardandgit clean -fd) before pulling. Error handling exits the script on failure, which is appropriate for a workflow script.
65-78: LGTM!Using
sys.executableensures the correct Python interpreter is used, and editable install mode is appropriate for accessing the pecv-bench CLI tools.
140-173: LGTM!The safer dictionary access using
.get("id")has been properly implemented. The function handles both zip creation and import failures gracefully.
175-226: LGTM!The report summarization logic is clean with proper file existence checks and error handling.
264-298: LGTM!The parallel import workflow is well-structured. Results are collected in the main thread after
future.result(), making the dictionary updates thread-safe.
339-368: LGTM!The report generation step properly sequences the analysis commands and exits on failure. The final report summarization ties everything together.
supporting_scripts/course-scripts/hyperion-benchmark-workflow/manage_programming_exercise.py (7)
1-11: Imports are clean.Previous review feedback about unused imports has been addressed.
13-21: LGTM!The sanitization logic correctly handles edge cases like names starting with numbers.
23-30: LGTM!Simple file reading utility. The caller (
convert_variant_to_zip) checks file existence before calling.
111-140: LGTM!The zip packaging logic correctly handles the template-to-exercise rename and includes proper cleanup of intermediate files.
154-156: Case mismatch issue has been addressed.The config file path now correctly uses lowercase
"exercise-details.json", matching whatconvert_variant_to_zipcreates.
208-224: LGTM!The consistency check function is straightforward with proper error logging.
227-254: LGTM!The worker function properly handles missing exercise IDs and writes results to per-variant files. The
Nonecheck forexercise_server_idprevents unnecessary network calls for failed imports.
IcanCUthere
left a comment
IcanCUthere
left a comment
There was a problem hiding this comment.
Tested it locally, seems to work as expected, but got some weird errors when executing the script.
supporting_scripts/course-scripts/hyperion-benchmark-workflow/config.ini
Outdated
Show resolved
Hide resolved
khinevich
requested review from
Elfari1028,
HawKhiem,
IcanCUthere and
sawys777
Updated Python version in installation instructions.
sawys777
left a comment
sawys777
left a comment
There was a problem hiding this comment.
Tested locally, report.md generated as expected
Checklist
General
Server
Motivation and Context
When developing a new Consistency Check approach, we need to evaluate it. This can be done with PECV (Programming Exercise Consistency Varification) Bench.
Provided in this MR solution runs the python script, which establishes a workflow, using PECV Dataset for the Consistency Check.
This approach is necessary for developers to systematically evaluate new approaches. Instead of a continuous process, a developer will trigger this script and observe the approach metric. This capability allows developers to conveniently run the benchmark against a specific version of consistency check approach to measure its performance, so that developers can contribute measurable improvements.
Description
Changes in supporting-scripts/course-scripts folder only
Main method, upon already listed functions, calls consistency check
Steps for Testing
cd supporting_scripts/course-scripts/hyperion-benchmark-workflowsource venv/bin/activate!!! If executions fails due to Course Deletion issue (before it fails, script tries 3 times send Delete request), just rerun the script.
Screenshots
Testserver States
You can manage test servers using Helios. Check environment statuses in the environment list. To deploy to a test server, go to the CI/CD page, find your PR or branch, and trigger the deployment.
Review Progress
Performance Review
Code Review
Manual Tests
Summary by CodeRabbit
New Features
Documentation
Chores
✏️ Tip: You can customize this high-level summary in your review settings.