Dakota Job Submission Examples
The following page gives some specific scripting examples for performing Dakota remote job submission using the classic Job Submission dialog.
Prerequisites for Non-Sandia Users
If you use Dakota GUI’s job submission feature outside of Sandia, you will not be provided with any default machine definitions or execution templates, and will need to write your own. However, one machine code, “DAKOTA,” is already provided for you - it can be viewed under “Window > Preferences > Job Submission > Codes.”
Example 1: Job Submission to a Non-Queued Machine with a Basic Dakota Study
Note
Read about the definition of a non-queued machine before proceeding.
Initial Setup
First, in the Project Explorer, right-click the Dakota study you would like to submit to a remote machine. Choose “Run As > Run Configurations.” Then, in the Run Configurations dialog, double-click “Job Submission” in the configuration type list on the left side of the dialog.
If all goes well, your job submission run configuration should default select the “DAKOTA” code, the first machine you defined, and the first execution template you defined, if you defined an execution template.
Resources Tab
In the “Input File” field, verify that your Dakota .in file has been selected.
In the larger “Input Files” field, check the checkboxes for any associated project files that you would like to have sent to the remote machine selected in the “Machine” dropdown.
In the “Output Files” field, create filename patterns for anything that you want the GUI to retrieve from the remote machine after the job has completed. This field accepts wildcards for filename patterns.
Machine Tab
In the “Job Run Directory” field, set the path on the remote machine that you would like the job to run in. Note that the path must be absolute: it should start with something like
/home/${username}/
, where “username” is the username on the remote machine that you would like the job to run under.Check the “Clear job run directory” field if you would like the job to clean up any files in the remote run directory before starting.
Ignore the “Queue” properties, since we are targeting a non-queued machine in this example.
Execution Instructions Tab
Let’s skip the “Code Parameters” tab for now and go to “Execution Instructions.”
Job Commands:
Copy-paste the following script into the “Job Commands” field:
#!/bin/sh
cd ${remote.dir} || exit 9
if [ -f /etc/bashrc ]; then
source /etc/bashrc
fi
module load dakota
dakota -input ${input.deck.base.name}.in 1>${input.deck.base.name}.out 2>&1
Note the presence of ${variable}
syntax. These are variables which we must define as part of the job submission configuration. Some variables are provided for you by the job submission framework,
but others are Dakota-specific and must be manually defined. Assuming that you have not modified the previous script, you will need to define the following variables:
preprocessor
DAKOTA_APPLIC_CONCURRENCY
clean_subdirectory
work_subdirectory
To add a custom variable:
Click on the “Variables…” button to the bottom-right of the “Job Commands” field (take care to click the correct “Variables…” button, as there are multiple.
In the “Select Variable” pop-up that follows, click “Add.”
In the “Variable Definition” pop-up, fill out each field:
Name: The name of your variable; in other words, what goes between
${
and}
.Label: A human-readable label for your variable, which will be displayed on the Code Parameters tab of this job submission configuration.
Description: A written description of the variable, if you wish to provide one.
Required: Whether the job submission configuration should require that this variable be filled out.
Persisted: Whether the variable value is persisted.
Type: The type represented by the variable.
Default Value: A default value for the variable.
Now, fill out custom variable definitions for “preprocessor,” “DAKOTA_APPLIC_CONCURRENCY,” “clean_subdirectory,” and “work_subdirectory.”
Code Parameters Tab
Once you have defined your custom variables on the “Execution Instructions” tab, return to the third tab - “Code Parameters.” You will note that now, there are generated fields in which you can enter values for each custom variable you defined. If you did not provide a default value for any of these variables, you may provide values on this tab (this is admittedly more readable than using variable default values).
Example 2: Job Submission to a Queued Machine
Note
Read about the definition of a queued machine before proceeding.
Initial Setup
First, in the Project Explorer, right-click the Dakota study you would like to submit to a remote machine. Choose “Run As > Run Configurations.” Then, in the Run Configurations dialog, double-click “Job Submission” in the configuration type list on the left side of the dialog.
If all goes well, your job submission run configuration should default select the “DAKOTA” code, the first machine you defined, and the first execution template you defined, if you defined an execution template.
Resources Tab
TODO
Execution Instructions Tab
TODO
Job commands:
cd ${remote.dir} || exit 9
if [ -f /etc/bashrc ]; then
source /etc/bashrc
fi
chmod 744 *.sh
dos2unix *.sh
echo submitting to the ${queue} queue
sbatch -N ${num.nodes} --partition=${queue} ${qos} --time=${job.hours}:${job.minutes}:${job.seconds} -A ${account} ${script.name} ${capture.job.id}
Script contents:
#!/bin/sh
cd ${remote.dir} || exit 9
if [ -f /etc/bashrc ]; then
source /etc/bashrc
fi
export RELATIVE_LOCATION=..
export PREPROCESSOR=${preprocessor}
export APREPRO="module load seacas && aprepro"
export DAKOTA_PPN=${ppn}
export DAKOTA_APPLIC_CONCURRENCY=${evaluation_concurrency}
export DAKOTA_APPLIC_PROCS=$((${num.processors}/DAKOTA_APPLIC_CONCURRENCY))
echo "Checking that constraint between number of processors, number of concurrent runs, and number of processors per run is respected..."
remainder=$((${num.processors}%DAKOTA_APPLIC_CONCURRENCY))
if [ $remainder -gt 0 ];
then
echo "${num.processors} must be a multiple of $DAKOTA_APPLIC_CONCURRENCY"
echo "Exiting..."
exit
fi
echo "INFO: DAKOTA will manage ${DAKOTA_APPLIC_CONCURRENCY} concurrent application jobs, each running on ${DAKOTA_APPLIC_PROCS} cores, with ${DAKOTA_PPN} processes per node"
if [ ${clean_subdirectory} -eq 0 ]; then
num_dirs=`find . -name ${work_subdirectory}.* -type d | wc -l`
if [ $num_dirs -gt 0 ]; then
rm -r ${work_subdirectory}.*
fi
fi
module load dakota
dakota -input ${input.deck.base.name}.in 1>${input.deck.base.name}.out 2>&1
TODO
Code Parameters
TODO