Preprocessing with dprepro and pyprepro

Dakota is packaged with two template processing tools that are intended for use in the preprocessing phase of analysis drivers.

The first tool, pyprepro, features simple parameter substitution, setting of immutable (fixed) variable names, and provides full access within templates to all of the Python programming language. As such, templates can contain loops, conditionals, lists, dictionaries, and other Python language features.

The second tool, dprepro, uses the same template engine as pyprepro, and in addition understands Dakota’s parameter file formats. In particular, when using dprepro in an analysis driver, Dakota variables become available for use within templates. dprepro is also integrated with the dakota.interfacing module to provide direct access to Parameters and Results objects within templates (see Section 1.9.3.8) and to provide template processing capability within Python scripts that import dakota.interfacing.

Changes and Updates to dprepro

The version of dprepro described in this section is a replacement for an earlier version that shipped with Dakota releases prior to 6.8. Although the new version offers a wide array of new features, it largely maintains backward compatibility with the old. Users should be aware of two important differences between the two versions.

The earlier version of dprepro was written in Perl, but the new one is written in Python. It is compatible with Python 2 (2.6 and greater) and 3. Some users, especially on Windows, may need to modify existing analysis drivers to invoke dprepro using Python instead of Perl.
Recent versions of Perl dprepro supported per-field output formatting in addition to the global numerical format that could be specified on the command line. This was accomplished by adding a comma- separated format string to individual substitution expressions in templates (e.g. {x1,%5.3f}). Per-field formatting remains a feature of the new dprepro, but the syntax has changed. Python-style string formatting is used, as explained in Section 1.9.5.5. Existing templates that make use of per-field formatting will need to be updated.

Although the old dprepro has been deprecated as of the 6.8 release of Dakota, it is still available in Dakota’s bin/ folder under the name dprepro.perl.

Usage

Running dprepro with the --help option at the command prompt causes its options and arguments to be listed. These are shown in Listing 68.

dprepro accepts three positional command line arguments. They are:

include: The name of a Dakota parameters file (required),
infile: The name of a template file (or a dash if the template is provided on stdin) (required), and
outfile: The name of the output file, which is the result of processing the template. This argument is optional, and output is written to stdout if it is missing.

The remaining options are used to

Set custom delimiters for Python code lines (--code) and blocks (--code-block) and for inline statements that print (--inline). The last of these is equivalent to Perl dprepro’s --left-delimiter and --right-delimiter switches, which also have been preserved to maintain backward compatibility. They default to "{ }".
Insert additional parameters for substitution, either from a JSON file (--json-include) or directly on the command line (--var). Variables that are defined using these options are immutable (Section 1.9.3.7).
Silence warnings (--no-warn)
Set the default numerical output format (--output-format).

Listing 68 dprepro usage

usage: dprepro [-h] [--code CHAR] [--code-block "OPEN CLOSE"]
               [--inline "OPEN CLOSE"] [--simple-parser] [--json-include FILE]
               [--python-include FILE] [--no-warn] [--output-format FMT]
               [--var "var=value"] [-v]
               include infile [outfile]

dprepro -- python-based input deck pre-processor and template engine.

version: 20220601.0

positional arguments:
  include               Include (parameter) file.
  infile                Specify the input file. Or set as `-` to read stdin
  outfile               Specify the output file. Otherwise, will print to
                        stdout

optional arguments:
  --code CHAR           ["%"] Specify the string to delineate a single code
                        line
  --code-block "OPEN CLOSE"
                        ["{% %}"] Specify the open and close of a code block.
                        NOTE: the inner-most character must *not* be any of
                        "{}[]()"
  -h, --help            show this help message and exit
  --inline "OPEN CLOSE"
                        ["{ }"] Specify the open and close of inline
                        code/variables to print
  --json-include FILE   Specify JSON formatted files to load variables
                        directly. As with `--include`, all variables will be
                        immutable. Can specify multiple. See include ordering
  --no-warn             Silence warning messages.
  --output-format FMT   ['%0.10g'] Specify the default float format. Note that
                        this can be easily overridden inline as follows:
                        `{'%3.8e' % param}`. Specify in either %-notation or
                        {}.format() notation.
  --python-include FILE
                        Specify a python formatted file to read and use the
                        resulting environment. NOTE: the file is read a
                        regular python where indentation matters and blocks
                        should *not* have an `end` statement (unlike in coode
                        blocks). As with `--include`, all variables will be
                        immutable and can specify this flag multiple times.
                        See include ordering
  --simple-parser       Always use the simple parser in dprepro rather than
                        dakota.interfacing
  --var "var=value"     Specify variables to predefine. They will be defined
                        as immutable. Use quotes to properly delineate
  -v, --version         Print the version and exit

Fallback Flags
-----------------
Will also accept `--(left/right)-delimiter` as an alias to the 
respective parts of `--inline`. 

Include Ordering
----------------
All include files are read and set as Immutable immediately. They are read
in the following order: include files, json-include, python-include.
Therefore, if a variable is, for example, set in the include file and the 
python-include, the original value will hold.

Sources:
--------
Built from BottlePy's SimpleTemplateEngine[1] with changes to better match
the behavior of APREPRO[2] and DPREPRO[3] and more tuned to simulation 
input files

[1]: https://bottlepy.org/docs/dev/stpl.html
[2]: https://github.com/gsjaardema/seacas
[3]: https://dakota.sandia.gov/

The pyprepro script accepts largely the same command line options. The primary differences are that pyprepro does not require or accept Dakota-format parameters files, and it has just two positional command line arguments, the infile and outfile, both defined as above. In addition, pyprepro accepts one or more --include files. These may be used to set parameters and execute arbitrary Python scripting before template processing occurs (See Section 1.9.3.7).

Template Expressions

This section describes the expressions that are permitted in templates. All examples, except where otherwise noted, use the default delimiters "{ }" for inline printed expressions, % for single-line Python statements, and "{% %}" for Python code blocks.

Expressions can be of three different forms (with defaults)

Inline single-line expressions (rendered): {expression}
Python code single-line (silent): % expression
Python code multi-line blocks (silent): {% expression (that can span many lines) %}

Expressions can contain just about any valid Python code. The only important difference is that indentation is ignored and blocks must end with end. See the examples below.

Inline Expressions

Inline expressions are delineated with {expression} and always display.

Consider:

param1 = {param1 = 10}
param2 = {param1 + 3}
param3 = {param3 = param1**2}

Returns:

param1 = 10
param2 = 13
param3 = 100

In this example, the first and third line both display a value and set the parameter.

Python Single Line Code

A % at the start of a line is used to begin a single-line code expression. These are non-printing. Consider the following example.

% param1 = pi/4
The new value is {sin(param1)}

It returns:

The new value is 0.7071067812

Furthermore, single lines can be used for Python logic and loops. This example demonstrates looping over an array, which is explained in further detail below. As stated previously, unlike ordinary Python, indentation is not required and is ignored. Blocks of Python code are concluded with end.

% angles = [0,pi/4,pi/2,3*pi/4,pi]
% for angle in angles:
cos({angle}) = { cos(angle)}
% end

Returns:

cos(0) = 1
cos(0.7853981634) = 0.7071067812
cos(1.570796327) = 6.123233996e-17
cos(2.35619449) = -0.7071067812
cos(3.141592654) = -1

Code Blocks

Finally, multi-line code blocks may be specified without prepending each Python statement with %. Instead, the entire block is enclosed in {% %}. (Indentation is ignored within code blocks.)

{%
# Can have comments too!
txt = ''
for ii in range(10):
    txt += ' {}'.format(ii)
end
%}
txt: {txt}

returns:

txt:  0 1 2 3 4 5 6 7 8 9

Changing Delimiters

As noted in the --help for dprepro and pyprepro, the delimiters for single-line Python statements, code blocks, and inline printed expressions can be changed. This is useful when the defaults are reserved characters in the output format.

For code blocks (default {% %}), the innermost characters cannot be any of “{}[]()”.

Escaping Delimiters

All delimiters can be escaped with a leading \. A double \\ followed by the delimiter will return \. For example:

{A=5}
\{A=5\}
\\{A=5\\}

Returns:

5
{A=5}
\{A=5\}

Note that escaping the trailing delimiter (e.g. \}) is optional.

Whitespace Control

Expressions span the entire line, which can possibly introduce undesired white space. Ending a line with \\ will prevent the additional space. Consider the following:

BLOCK \\
{%
if True:
    block = 10
else:
    block = 20
end
%}
{block}

Which renders as:

BLOCK 10

Without the trailing \\, the result would instead be:

BLOCK
10

This can also be abused to allow spacing. Consider the following:

I want this to \\
%
render as \\
%
one line

Since the % symbolize a code block (empty in this case), it will render

I want this to render as one line

Immutable Variables

Variables can be fixed such that they cannot be redefined (without explicitly allowing it).

In this example, the attempted reassignment of param to 20 is ignored,

% param = Immutable(10)
% param = 20
{param}

and the output is

because param is Immutable. To explicitly make a variable mutable again, call it with Mutable():

set             : \{ param = Immutable(10) \} : { param = Immutable(10) }
try to reset    : \{ param = 20 \}            : { param = 20 }
make mutable    : \{ param = Mutable(21) \}   : { param = Mutable(21) }
reset           : \{ param = 20 \}            : { param = 20 }

Returns:

set             : { param = Immutable(10) } : 10
try to reset    : { param = 20 }            : 10
make mutable    : { param = Mutable(21) }   : 21
reset           : { param = 20 }            : 20

Note that any variable set on the command line by any of these three means:

--var argument
--include file
--json-include file

is immutable. This listing is in order of precedence; variables set by a --var argument cannot be modified by --include or --json-include files. This feature is useful for overriding defaults set in templates.

Suppose the template file MyTemplate.inp contains:

param1 = {param1 = 10}
param2 = {param2 = pi}

Executing pyprepro MyTemplate.in yields:

param1 = 10
param2 = 3.141592654

However, for pyprepro --var "param1=30" MyTemplate.in:

param1 = 30
param2 = 3.141592654

Or, if an optional --include file that is named MyInclude.inp and contains the following is added:

{param1 = 32}

Then running pyprepro --include MyInclude.inp MyTemplate.inp outputs:

param1 = 32
param2 = 3.141592654

Note that variable definitions set using --var override definitions in --include files.

There is one caveat to variable immutability. While the variable name is reserved, the value can still be changed if it is a mutable Python object (“mutable” has different meanings for Python objects than is used in pyprepro and dprepro templates). For example:

% param = Immutable( [1,2,3])
% param.append(4)   # This will work because it is modifying the object
% param = ['a','b','c']   # This won't because it is redefining
{param}

Will output:

[1, 2, 3, 4]

Preprocessing with dprepro and pyprepro

Changes and Updates to dprepro

Usage

Template Expressions

Inline Expressions

Python Single Line Code

Code Blocks

Changing Delimiters

Escaping Delimiters

Whitespace Control

Immutable Variables

Exceptional service in the national interest