You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/source/usage.rst
+6-5Lines changed: 6 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,10 +11,11 @@ Hierarchy of generated files
11
11
12
12
In order to understand the contents of the generated folders/files, it's good to know how ``smart-dispatch`` deals with **commands** that a user requests to launch on the cluster:
13
13
14
-
* Each invokation of ``smart-dispatch`` creates a so-called **batch** of **jobs**. Smart Dispatch will do its best to create as many simultaneous jobs so as to effecitvely utilze the allocated resources.
14
+
* Smart Dispatch will distribute commands to jobs such that each of the latter uses an entire node. Jobs may run many commands concurrently if necessary to use a maximum number of cores and GPUs. The distribution is based on number of cores per node / per command and number of GPUs per node / per command.
15
+
15
16
* Each job is basically a single PBS file that is run by the queue management system on the cluster (either ``msub`` or ``qsub``).
16
-
* A job spawns mulitple concurrent **workers** that all cooperate to execute the requested commands.
17
-
* Each worker (basically, a python script) is executing commands sequentially.
17
+
* A job spawns multiple concurrent **workers** that all cooperate to execute the requested commands.
18
+
* Each worker is executing commands sequentially.
18
19
19
20
A typical hierarchy of ``./SMART_DISPATCH_LOGS/{batch_id}/`` should look like this: ::
20
21
@@ -58,7 +59,7 @@ Now let's go through the subdirectories.
58
59
This directory holds generated PBS files (``job_commands_{pbs_index}.sh``) as well as three command lists:
59
60
60
61
``commands.txt``:
61
-
A list pending commands (this is where the workers are taking their next commands to execute from).
62
+
A list of pending commands (this is where the workers are taking their next commands to execute from).
62
63
``running_commands.txt``:
63
64
A list of currently running commands.
64
65
``failed_commands.txt``:
@@ -68,7 +69,7 @@ This directory holds generated PBS files (``job_commands_{pbs_index}.sh``) as we
68
69
``logs/``
69
70
^^^^^^^^^
70
71
71
-
Output and error logs in are saved in this directory. The root level contains logs for actual commands. There are also two additional subfolder:
72
+
Output and error logs are saved in this directory. The root level contains logs for actual commands. There are also two additional subfolders:
0 commit comments