.. _`interfaces:dprepro-and-pyprepro`:
"""""""""""""""""""""""""""""""""""""""
Preprocessing with dprepro and pyprepro
"""""""""""""""""""""""""""""""""""""""
Dakota is packaged with two template processing tools that are intended
for use in the preprocessing phase of analysis drivers.
The first tool, ``pyprepro``, features simple parameter substitution,
setting of immutable (fixed) variable names, and provides full access
within templates to all of the Python programming language. As such,
templates can contain loops, conditionals, lists, dictionaries, and
other Python language features.
The second tool, ``dprepro``, uses the same template engine as
``pyprepro``, and in addition understands Dakota’s parameter file
formats. In particular, when using ``dprepro`` in an analysis driver,
Dakota variables become available for use within templates. ``dprepro``
is also integrated with the ``dakota.interfacing`` module to provide
direct access to ``Parameters`` and ``Results`` objects within templates
(see Section `1.9.3.8 <#interfaces:params-and-results>`__) and to
provide template processing capability within Python scripts that import
``dakota.interfacing``.
.. _`interfaces:dprepro-changes`:
Changes and Updates to dprepro
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The version of ``dprepro`` described in this section is a replacement
for an earlier version that shipped with Dakota releases prior to 6.8.
Although the new version offers a wide array of new features, it largely
maintains backward compatibility with the old. Users should be aware of
two important differences between the two versions.
- The earlier version of ``dprepro`` was written in Perl, but the new
one is written in Python. It is compatible with Python 2 (2.6 and
greater) and 3. Some users, especially on Windows, may need to modify
existing analysis drivers to invoke ``dprepro`` using Python instead
of Perl.
- Recent versions of Perl ``dprepro`` supported per-field output
formatting in addition to the global numerical format that could be
specified on the command line. This was accomplished by adding a
comma- separated format string to individual substitution expressions
in templates (e.g. ``{x1,%5.3f}``). Per-field formatting remains a
feature of the new ``dprepro``, but the syntax has changed.
Python-style string formatting is used, as explained in
Section `1.9.5.5 <#interfaces:per-field-output-formatting>`__.
Existing templates that make use of per-field formatting will need to
be updated.
Although the old ``dprepro`` has been deprecated as of the 6.8 release
of Dakota, it is still available in Dakota’s ``bin/`` folder under the
name ``dprepro.perl``.
.. _`interfaces:dprepro-usage`:
Usage
~~~~~
Running ``dprepro`` with the ``--help`` option at the command prompt
causes its options and arguments to be listed. These are shown in
:numref:`advint:dprepro_usage`.
``dprepro`` accepts three positional command line arguments. They are:
#. ``include``: The name of a Dakota parameters file (*required*),
#. ``infile``: The name of a template file (or a dash if the template is
provided on ``stdin``) (*required*), and
#. ``outfile``: The name of the output file, which is the result of
processing the template. This argument is optional, and output is
written to ``stdout`` if it is missing.
The remaining options are used to
- Set custom delimiters for Python code lines (``--code``) and blocks
(``--code-block``) and for inline statements that print
(``--inline``). The last of these is equivalent to Perl
``dprepro``\ ’s ``--left-delimiter`` and ``--right-delimiter``
switches, which also have been preserved to maintain backward
compatibility. They default to ``"{ }"``.
- Insert additional parameters for substitution, either from a JSON
file (``--json-include``) or directly on the command line
(``--var``). Variables that are defined using these options are
*immutable* (Section `1.9.3.7 <#interfaces:immutable-variables>`__).
- Silence warnings (``--no-warn``)
- Set the default numerical output format (``--output-format``).
.. literalinclude:: ../usingdakota/samples/dprepro_usage
:language: dakota
:tab-width: 2
:caption: ``dprepro`` usage
:name: advint:dprepro_usage
The ``pyprepro`` script accepts largely the same command line options.
The primary differences are that ``pyprepro`` does not require or accept
Dakota-format parameters files, and it has just two positional command
line arguments, the ``infile`` and ``outfile``, both defined as above.
In addition, ``pyprepro`` accepts one or more ``--include`` files. These
may be used to set parameters and execute arbitrary Python scripting
before template processing occurs (See
Section `1.9.3.7 <#interfaces:immutable-variables>`__).
.. _`interfaces:template-expressions`:
Template Expressions
~~~~~~~~~~~~~~~~~~~~
This section describes the expressions that are permitted in templates.
All examples, except where otherwise noted, use the default delimiters
``"{ }"`` for inline printed expressions, ``%`` for single-line Python
statements, and ``"{% %}"`` for Python code blocks.
Expressions can be of three different forms (with defaults)
- Inline single-line expressions (rendered): ``{expression}``
- Python code single-line (silent): ``% expression``
- Python code multi-line blocks (silent):
``{% expression (that can span many lines) %}``
Expressions can contain just about any valid Python code. The only
important difference is that indentation is ignored and blocks must end
with ``end``. See the examples below.
.. _`interfaces:inline-expressions`:
Inline Expressions
^^^^^^^^^^^^^^^^^^
Inline expressions are delineated with ``{expression}`` and *always
display*.
Consider:
::
param1 = {param1 = 10}
param2 = {param1 + 3}
param3 = {param3 = param1**2}
Returns:
::
param1 = 10
param2 = 13
param3 = 100
In this example, the first and third line both display a value *and* set
the parameter.
.. _`interfaces:python-single-line-code`:
Python Single Line Code
^^^^^^^^^^^^^^^^^^^^^^^
A ``%`` at the start of a line is used to begin a single-line code
expression. These are non-printing. Consider the following example.
::
% param1 = pi/4
The new value is {sin(param1)}
It returns:
::
The new value is 0.7071067812
Furthermore, single lines can be used for Python logic and loops. This
example demonstrates looping over an array, which is explained in
further detail below. As stated previously, unlike ordinary Python,
indentation is not required and is ignored. Blocks of Python code are
concluded with ``end``.
::
% angles = [0,pi/4,pi/2,3*pi/4,pi]
% for angle in angles:
cos({angle}) = { cos(angle)}
% end
Returns:
::
cos(0) = 1
cos(0.7853981634) = 0.7071067812
cos(1.570796327) = 6.123233996e-17
cos(2.35619449) = -0.7071067812
cos(3.141592654) = -1
.. _`interfaces:code-blocks`:
Code Blocks
^^^^^^^^^^^
Finally, multi-line code blocks may be specified without prepending each
Python statement with ``%``. Instead, the entire block is enclosed in
``{% %}``. (Indentation is ignored within code blocks.)
::
{%
# Can have comments too!
txt = ''
for ii in range(10):
txt += ' {}'.format(ii)
end
%}
txt: {txt}
returns:
::
txt: 0 1 2 3 4 5 6 7 8 9
.. _`interfaces:changing-delimiters`:
Changing Delimiters
^^^^^^^^^^^^^^^^^^^
As noted in the ``--help`` for ``dprepro`` and ``pyprepro``, the
delimiters for single-line Python statements, code blocks, and inline
printed expressions can be changed. This is useful when the defaults are
reserved characters in the output format.
For code blocks (default ``{% %}``), the innermost characters cannot be
any of “``{}[]()``”.
Furthermore every template can individually reset the command delimiter by
specifying them at the top of the file. From the help:
::
Specify delineators as the the first non-whitespace line. Start with a
comment '//', '#','%', '$' or '' (nothing), then a command, then '=' or
space, followed by the new setting. See examples.
Commands are:
Commands Flags
---------------------------------------
| PYPREPRO_CODE | |
| DPREPRO_CODE | --code |
| PREPRO_CODE | |
| ---------------------|--------------|
| PYPREPRO_CODE_BLOCK | |
| DPREPRO_CODE_BLOCK | --code-block |
| PREPRO_CODE_BLOCK | |
| ---------------------|--------------|
| PYPREPRO_INLINE | |
| DPREPRO_INLINE | --inline |
| PREPRO_INLINE | |
---------------------------------------
.. _`interfaces:escaping-delimiters`:
Escaping Delimiters
^^^^^^^^^^^^^^^^^^^
All delimiters can be escaped with a leading ``\``. A double ``\\``
followed by the delimiter will return ``\``. For example:
::
{A=5}
\{A=5\}
\\{A=5\\}
Returns:
::
5
{A=5}
\{A=5\}
Note that escaping the trailing delimiter (e.g. ``\}``) is optional.
.. _`interfaces:whitespace-control`:
Whitespace Control
^^^^^^^^^^^^^^^^^^
Expressions span the entire line, which can possibly introduce undesired
white space. Ending a line with ``\\`` will prevent the additional
space. Consider the following:
::
BLOCK \\
{%
if True:
block = 10
else:
block = 20
end
%}
{block}
Which renders as:
::
BLOCK 10
Without the trailing ``\\``, the result would instead be:
::
BLOCK
10
This can also be abused to allow spacing. Consider the following:
::
I want this to \\
%
render as \\
%
one line
Since the ``%`` symbolize a code block (empty in this case), it will
render
::
I want this to render as one line
.. _`interfaces:immutable-variables`:
Immutable Variables
^^^^^^^^^^^^^^^^^^^
Variables can be fixed such that they cannot be redefined (without
explicitly allowing it).
In this example, the attempted reassignment of ``param`` to 20 is
ignored,
::
% param = Immutable(10)
% param = 20
{param}
and the output is
::
10
because ``param`` is ``Immutable``. To explicitly make a variable
mutable again, call it with ``Mutable()``:
::
set : \{ param = Immutable(10) \} : { param = Immutable(10) }
try to reset : \{ param = 20 \} : { param = 20 }
make mutable : \{ param = Mutable(21) \} : { param = Mutable(21) }
reset : \{ param = 20 \} : { param = 20 }
Returns:
::
set : { param = Immutable(10) } : 10
try to reset : { param = 20 } : 10
make mutable : { param = Mutable(21) } : 21
reset : { param = 20 } : 20
Note that any variable set on the command line by any of these three
means:
- ``--var`` argument
- ``--include`` file
- ``--json-include`` file
is immutable. This listing is in order of precedence; variables set by a
``--var`` argument cannot be modified by ``--include`` or
``--json-include`` files. This feature is useful for overriding defaults
set in templates.
Suppose the template file ``MyTemplate.inp`` contains:
::
param1 = {param1 = 10}
param2 = {param2 = pi}
Executing ``pyprepro MyTemplate.in`` yields:
::
param1 = 10
param2 = 3.141592654
However, for ``pyprepro --var "param1=30" MyTemplate.in``:
::
param1 = 30
param2 = 3.141592654
Or, if an optional ``--include`` file that is named ``MyInclude.inp``
and contains the following is added:
::
{param1 = 32}
Then running ``pyprepro --include MyInclude.inp MyTemplate.inp``
outputs:
::
param1 = 32
param2 = 3.141592654
Note that variable definitions set using ``--var`` override definitions
in ``--include`` files.
There is one caveat to variable immutability. While the variable name is
reserved, the value can still be changed if it is a mutable Python
object (“mutable” has different meanings for Python objects than is used
in ``pyprepro`` and ``dprepro`` templates). For example:
::
% param = Immutable( [1,2,3])
% param.append(4) # This will work because it is modifying the object
% param = ['a','b','c'] # This won't because it is redefining
{param}
Will output:
::
[1, 2, 3, 4]