The Dakota Restart Utility

The Dakota restart utility program provides a variety of facilities for managing restart files from Dakota executions. The executable program name is dakota_restart_util and it has the following options, as shown by the usage message returned when executing the utility without any options:

Usage:
  dakota_restart_util command <arg1> [<arg2> <arg3> ...] --options
    dakota_restart_util print <restart_file>
    dakota_restart_util to_neutral <restart_file> <neutral_file>
    dakota_restart_util from_neutral <neutral_file> <restart_file>
    dakota_restart_util to_tabular <restart_file> <text_file>
      [--custom_annotated [header] [eval_id] [interface_id]]
      [--output_precision <int>]
    dakota_restart_util remove <double> <old_restart_file> <new_restart_file>
    dakota_restart_util remove_ids <int_1> ... <int_n> <old_restart_file> <new_restart_file>
    dakota_restart_util cat <restart_file_1> ... <restart_file_n> <new_restart_file>
options:
  --help                       show dakota_restart_util help message
  --custom_annotated arg       tabular file options: header, eval_id,
                               interface_id
  --freeform                   tabular file: freeform format
  --output_precision arg (=10) set tabular output precision

Several of these functions involve format conversions. In particular, the binary format used for restart files can be converted to ASCII text and printed to the screen, converted to and from a neutral file format, or converted to a tabular format for importing into 3rd-party plotting programs. In addition, a restart file with corrupted data can be repaired by value or id, and multiple restart files can be combined to create a master database.

To/From Neutral File Format

A Dakota restart file can be converted to a neutral file format using a command like the following:

dakota_restart_util to_neutral dakota.rst dakota.neu

which results in a report similar to the following:

Writing neutral file dakota.neu
Restart file processing completed: 11 evaluations retrieved.

Similarly, a neutral file can be returned to binary format using a command like the following:

dakota_restart_util from_neutral dakota.neu dakota.rst

which results in a report similar to the following:

Reading neutral file dakota.neu
Writing new restart file dakota.rst
Neutral file processing completed: 11 evaluations retrieved.

The contents of the generated neutral file are similar to the following (from the first two records for the Cylinder example).

6 7 2 1.8000000000000000e+00 intake_dia 1.0000000000000000e+00 flatness 0 0 0 0
NULL 4 2 1 0 3 3 3 3 1 2 obj_fn nln_ineq_con_1 nln_ineq_con_2 nln_ineq_con_3
  -2.4355973813420619e+00 -4.7428486677140930e-01 -4.5000000000000001e-01
   1.3971143170299741e-01 -4.3644298963447897e-01  1.4999999999999999e-01
   1.3855136437818300e-01  0.0000000000000000e+00  0.0000000000000000e+00
   1.4999999999999999e-01  0.0000000000000000e+00 -1.9485571585149869e-01 1
6 7 2 2.1640000000000001e+00 intake_dia 1.7169994018008317e+00 flatness 0 0 0 0
NULL 4 2 1 0 3 3 3 3 1 2 obj_fn nln_ineq_con_1 nln_ineq_con_2 nln_ineq_con_3
  -2.4869127192988878e+00 6.9256958799989843e-01 -3.4245008972987528e-01
   8.7142207937157910e-03 -4.3644298963447897e-01  1.4999999999999999e-01
   2.9814239699997572e+01  0.0000000000000000e+00  0.0000000000000000e+00
   1.4999999999999999e-01  0.0000000000000000e+00 -1.6998301774282701e-01 2

This format is not intended for direct viewing (print should be used for this purpose). Rather, the neutral file capability has been used in the past for managing portability of restart data across platforms (recent use of more portable binary formats has largely eliminated this need) or for advanced repair of restart records (in cases where the remove command was insufficient).

Tabular Format

Conversion of a binary restart file to a tabular format enables convenient import of this data into 3rd-party post-processing tools such as Matlab, TECplot, Excel, etc. This facility is nearly identical to the output activated by the tabular_data keyword in the Dakota input file specification, but with two important differences:

1. No function evaluations are suppressed as they are with tabular_data (i.e., any internal finite difference evaluations are included). 2. The conversion can be performed later, i.e., for Dakota runs executed previously.

An example command for converting a restart file to tabular format is:

dakota_restart_util to_tabular dakota.rst dakota.m

which results in a report similar to the following:

Writing tabular text file dakota.m
Restart file processing completed: 10 evaluations tabulated.

The contents of the generated tabular file are similar to the following (from the gradient-based optimization textbook problem example). Note that while evaluations resulting from numerical derivative offsets would be reported (as described above), derivatives returned as part of the evaluations are not reported (since they do not readily fit within a compact tabular format):

%eval_id interface             x1             x2         obj_fn nln_ineq_con_1 nln_ineq_con_2
1            NO_ID            0.9            1.1         0.0002           0.26           0.76
2            NO_ID        0.90009            1.1 0.0001996404857   0.2601620081       0.759955
3            NO_ID        0.89991            1.1 0.0002003604863   0.2598380081       0.760045
4            NO_ID            0.9        1.10011 0.0002004407265       0.259945   0.7602420121
5            NO_ID            0.9        1.09989 0.0001995607255       0.260055   0.7597580121
6            NO_ID     0.58256179   0.4772224441   0.1050555937   0.1007670171 -0.06353963386
7            NO_ID   0.5826200462   0.4772224441   0.1050386469   0.1008348962 -0.06356876195
8            NO_ID   0.5825035339   0.4772224441   0.1050725476   0.1006991449 -0.06351050577
9            NO_ID     0.58256179   0.4772701663   0.1050283245    0.100743156 -0.06349408333
10           NO_ID     0.58256179   0.4771747219   0.1050828704   0.1007908783 -0.06358517983
...

Controlling tabular format

The command-line options --freeform and --custom_annotated give control of headers in the resulting tabular file. Freeform will generate a tabular file with no leading row nor columns (variable and response values only). Custom annotated format accepts any or all of the options:

  • header: include %-commented header row with labels

  • eval_id: include leading column with evaluation ID

  • interface_id: include leading column with interface ID

For example, to recover Dakota 6.0 tabular format, which contained a header row, leading column with evaluation ID, but no interface ID:

dakota_restart_util to_tabular dakota.rst dakota.m --custom_annotated header eval_id

Resulting in

%eval_id             x1             x2         obj_fn nln_ineq_con_1 nln_ineq_con_2
1                   0.9            1.1         0.0002           0.26           0.76
2               0.90009            1.1 0.0001996404857   0.2601620081       0.759955
3               0.89991            1.1 0.0002003604863   0.2598380081       0.760045
...

Finally, --output_precision integer will generate tabular output with the specified integer digits of precision.

Concatenation of Multiple Restart Files

In some instances, it is useful to combine restart files into a single master function evaluation database. For example, when constructing a data fit surrogate model, data from previous studies can be pulled in and reused to create a combined data set for the surrogate fit. An example command for concatenating multiple restart files is:

dakota_restart_util cat dakota.rst.1 dakota.rst.2 dakota.rst.3 dakota.rst.all

which results in a report similar to the following:

Writing new restart file dakota.rst.all
dakota.rst.1 processing completed: 10 evaluations retrieved.
dakota.rst.2 processing completed: 110 evaluations retrieved.
dakota.rst.3 processing completed: 65 evaluations retrieved.

The dakota.rst.all database now contains 185 evaluations and can be read in for use in a subsequent Dakota study using the -read_restart option to the dakota executable.

Removal of Corrupted Data

On occasion, a simulation or computer system failure may cause a corruption of the Dakota restart file. For example, a simulation crash may result in failure of a post-processor to retrieve meaningful data. If 0’s (or other erroneous data) are returned from the user’s analysis_driver, then this bad data will get recorded in the restart file. If there is a clear demarcation of where corruption initiated (typical in a process with feedback, such as gradient-based optimization), then use of the -stop_restart option for the Dakota executable can be effective in continuing the study from the point immediately prior to the introduction of bad data. If, however, there are interspersed corruptions throughout the restart database (typical in a process without feedback, such as sampling), then the remove and remove_ids options of dakota_restart_util can be useful.

An example of the command syntax for the remove option is:

dakota_restart_util remove 2.e-04 dakota.rst dakota.rst.repaired

which results in a report similar to the following:

Writing new restart file dakota.rst.repaired
Restart repair completed: 65 evaluations retrieved, 2 removed, 63 saved.

where any evaluations in dakota.rst having an active response function value that matches 2.e-04 within machine precision are discarded when creating dakota.rst.repaired.

An example of the command syntax for the remove_ids option is:

dakota_restart_util remove_ids 12 15 23 44 57 dakota.rst dakota.rst.repaired

which results in a report similar to the following:

Writing new restart file dakota.rst.repaired
Restart repair completed: 65 evaluations retrieved, 5 removed, 60 saved.

where evaluation ids 12, 15, 23, 44, and 57 have been discarded when creating dakota.rst.repaired. An important detail is that, unlike the -stop_restart option which operates on restart record numbers, the remove_ids option operates on evaluation ids. Thus, removal is not necessarily based on the order of appearance in the restart file. This distinction is important when removing restart records for a run that contained either asynchronous or duplicate evaluations, since the restart insertion order and evaluation ids may not correspond in these cases (asynchronous evaluations have ids assigned in the order of job creation but are inserted in the restart file in the order of job completion, and duplicate evaluations are not recorded which introduces offsets between evaluation id and record number). This can also be important if removing records from a concatenated restart file, since the same evaluation id could appear more than once. In this case, all evaluation records with ids matching the remove_ids list will be removed.

If neither of these removal options is sufficient to handle a particular restart repair need, then the fallback position is to resort to direct editing of a neutral file to perform the necessary modifications.