Dakota  Version 7.0
Explore and Predict with Confidence
Pydantic Input Grammar: Complete Reference and Procedure

Pydantic Grammar Table of Contents

Overview

The Pydantic models in dakota/python/dakota/spec constitute the canonical specification of Dakota's input grammar. Implement grammar edits and additions in these Python models and, where required, provide matching C++ validation or computed-field implementations so that the shared grammar contract remains consistent between schema generation and runtime.

Most models inherit from DakotaBaseModel (dakota/python/dakota/spec/base.py). Parsed fields are declared with DakotaField and represent user-visible keywords and arguments. ValidationRule subclasses capture checks or mutations that depend on multiple parsed fields. ComputedFieldSpec entries paired with @computed_field properties export derived values in the schema or validated models. This page is self-contained: procedural steps reference the detailed authoring sections that follow; readers may consult any section on demand.

Procedure: Adding a New Keyword

Follow these steps when adding a new keyword. Each step references the authoritative section on this page that documents the required authoring surface and implementation details.

  1. Decide grammar role and one-of semantics. Choose whether the keyword is a scalar or collection argument, a pointer- style keyword that names another block, a Union branch, or a selection- wrapper (method or model family). If the keyword participates in a Union, ensure the Python annotation, default/default_factory, and dakota union_pattern metadata express identical one-of semantics. See How Union fields are processed.

  2. Declare the parsed field on the appropriate model class. Add a typed class attribute to a DakotaBaseModel-derived class and declare it with DakotaField. Provide Pydantic parameters (default, default_factory, description, bounds) as appropriate. Include dakota metadata keys only as required: aliases, argument, block_pointer, anchor, union_pattern, model_default, internal_only, and materialization. For a new method or model selection, follow the selection-wrapper pattern described in DakotaField authoring and Adding a new method or model.

  3. Author materialization mapping to IR. Select the IR key or keys that the field should modify and author the materialization metadata describing the write instructions. Each item must include at minimum ir_key, storage_type, and ir_value_type. Some storage_type values require additional properties (stored_value, secondary_ir_key, secondary_ir_value_type, enum_scope, secondary_literal_value). Prefer reusing existing storage-type handlers for identical IR effects; for domain-specific shapes use an existing model as a template. See Materialization and IR writes.

  4. Select the IR-default source. Choose one canonical source for any IR key's default: a grammar-level default on the Pydantic field, an implementation-level override in default_overrides_registry.json, or a type-policy default in default_policy_registry.json. The IR codegen merges these disjoint sources; generate_ir_tables.py treats collisions as errors. See IR default values.

  5. Add validation rules or computed fields if required. If correctness depends on relationships among fields, author a ValidationRule subclass, attach it via _VALIDATION_RULES on the model, and provide the matching C++ implementation and registration. If a derived value must appear in schema or the validated model, add a ComputedFieldSpec to _COMPUTED_FIELDS and implement the corresponding @computed_field property with a matching C++ registration. Both features require exact string-name agreement between Python metadata and the C++ registry entries. See Validation rules and computed fields.

  6. Implement and register required C++ logic and bindings. Any new ValidationRule or computed-field function requires a C++ declaration and implementation, registration in the appropriate registry constructor, and exposure via pybind so the Python layer can invoke it. Typical files to edit are under packages/dakota_parser/dakota_validation (include, src, and test). Ensure the registered C++ string names match the Python metadata. See Validation rules and computed fields.

  7. Regenerate schema and run IR code generation. Generate JSON schema from the updated Pydantic models and confirm that x-materialization, x-model-validations, x-computed-fields, and x-model-default metadata appear as expected. Run dakota/src/ir_codegen/extract_defaults_from_schema.py and dakota/src/ir_codegen/generate_ir_tables.py and inspect the generated IR contracts and write-instruction tables. Confirm intended ir_keys, ir_value_types, and defaults are present and that no collisions exist. See Materialization and IR writes and IR default values.

  8. Test, document, and publish. Add or update Python unit tests for parsing and schema output and C++ tests for validation and computed-field behavior. Execute integration tests that exercise materialization through ProblemDescDB reads. Update documentation and example inputs to demonstrate the new keyword. Verify CI and style checks succeed before submitting the change.

How Union fields are processed

Dakota recognizes several Union patterns to express one-of semantics. The DakotaBaseModel layer applies a model-level validator to enforce single-branch selection for Union fields represented as single-field wrapper models. The wrapper field name is the user-visible discriminator.

Example:

trend: Union[TrendNone, ExperimentalGPTrendConstant, ExperimentalGPTrendLinear]

Accepted user input is a single-key dictionary matching a branch wrapper, for example {"linear": {...}} or {"none": true}.

Pattern 1: Optional with model default

Python shape: Union[OptionA,...] with a model default_factory.

Semantics: omission yields a meaningful default branch.

Pattern 2: Optional with no default branch

Python shape: Union[OptionA,...] | None with default=None.

Semantics: omission disables the feature.

Pattern 3: Three-state optional with default branch

Python shape: Union[OptionA,...,dict] | None with default_factory.

Semantics: omission => disabled; {} => enabled with defaults; branch object => enabled with explicit configuration.

Pattern 4: Required Union

Python shape: Union[OptionA,...].

Semantics: user must select a branch.

Pattern 5: Three-state optional without grammar default

Python shape: Union[OptionA,...,dict] | None with default=None.

Semantics: omission => disabled; {} => implementation defaults; branch => explicit.

Pattern 6: Required parent with optional subselection

Python shape: Union[OptionA,...,dict].

Semantics: parent must exist; {} is a valid "no subselection" choice.

Runtime validation rules

  • If a Union has multiple branches, exactly one branch must be selected.
  • Validated model instances matching a branch are accepted.
  • Dictionaries with more than one key fail validation.
  • Unrecognized keys fail validation.
  • If dict is a branch, empty-dictionary semantics depend on the pattern.
  • For Pattern 3, omission converts to None while {} instantiates from default_factory, preserving three states.

Authoring guidance

  • Record the intended pattern with union_pattern.
  • Set model_default when the pattern specifies a default branch.
  • Keep the Python annotation, default/default_factory, union_pattern, and model_default consistent.

DakotaField authoring

DakotaField extends pydantic.Field with a dakota dictionary that controls parser, schema, and materialization behavior. Standard Pydantic kwargs (default, default_factory, description, gt, ge, etc.) remain valid.

Common dakota keys:

  • aliases: list[str] or str for accepted alternative spellings.
  • block_pointer: string naming the referenced block family/role.
  • argument: child field treated as inline parameter.
  • anchor: boolean True to attach branches to parent context.
  • union_pattern: integer pattern id for one-of semantics.
  • model_default: default branch name for patterns with defaults.
  • internal_only: boolean to reject user input for the field.
  • materialization: list[dict] describing IR writes and presence effects.

Notes:

  • Use aliases only for backward compatibility.
  • Use block_pointer when the keyword names another block.
  • Use argument when a child field acts as the inline argument.
  • Use anchor=True only when grammar requires attached discriminators.
  • internal_only fields are rejected from user input by DakotaBaseModel.

Example:

directory_tag: Literal[True] | None = DakotaField(
default=None,
description="Tag evaluation directories with identifiers",
dakota={"aliases": ["dir_tag"]},
)
dakota (lowercase) namespace for new Dakota modules
Definition: DakotaSurrogates.hpp:18

Materialization and IR writes

Materialization maps validated Pydantic JSON into Dakota's IR. The models describe valid input, dakota["materialization"] prescribes how input affects IR keys, and the IR codegen emits write-instruction tables.

Pipeline:

  • Models export JSON schema with x-materialization metadata.
  • Codegen scripts under dakota/src/ir_codegen read schema and produce IR contracts and write tables.
  • At runtime, validated JSON is traversed and generated write instructions populate IR keys consumed by ProblemDescDB.

Key scripts:

  • dakota/src/ir_codegen/extract_defaults_from_schema.py
  • dakota/src/ir_codegen/generate_ir_tables.py

Materialization item shape:

  • ir_key: destination IR key.
  • storage_type: write-handler identifier.
  • ir_value_type: Dakota IR type (Real, size_t, StringArray, short, etc.).
  • stored_value: optional literal for presence-driven writes.
  • Additional handler-specific properties as required.

Common materialization cases: These are the storage_type values authors are most likely to use when adding or editing grammar.

storage_type expected fields when to use
DIRECT_VALUE ir_key, ir_value_type Write the field's validated value directly to the IR key. Use for 1:1 scalar or collection values.
ADD_TO_VALUE ir_key, ir_value_type Add the field's value into an existing numeric aggregate instead of replacing it. Use for roll-up counters.
PRESENCE_TRUE ir_key, ir_value_type Set the target IR key to boolean true when the field is present. Use for presence-driven boolean flags.
PRESENCE_FALSE ir_key, ir_value_type Set the target IR key to boolean false when the field is present. Use for presence-driven negating flags.
PRESENCE_LITERAL ir_key, ir_value_type, stored_value[, enum_scope] Write a literal stored_value when the field is present. Use when presence selects a specific enumerated or literal runtime value.
PRESENCE_ENUM ir_key, ir_value_type, stored_value[, enum_scope] Presence selects one of a set of enumerated IR values. Use when the field is a symbolic enumerant.
AUGMENT_ENUM ir_key, ir_value_type, stored_value Augment an enum-like IR setting with an additional symbolic value. Use when presence should extend an enumerated configuration.
TYPE_DATA_COMBINED ir_key, ir_value_type, stored_value, secondary_ir_key, secondary_ir_value_type Composite handler: primary literal selection plus secondary direct value write. Use when a symbolic choice and an argument are required.
METHOD_PIECEWISE ir_key, ir_value_type, stored_value, secondary_ir_key, secondary_ir_value_type, secondary_literal_value Composite handler for piecewise method semantics; expands into primitive writes during codegen. Use for piecewise method configs.

Specialized materialization handlers: These handlers are for established Dakota-specific data shapes. Reuse an existing model that already uses the handler as your template.

storage_type expected fields when to use
ANALYSIS_COMPONENTS ir_key, ir_value_type, handler-specific props TODO
ADJACENCY_MATRIX ir_key, ir_value_type, handler-specific props TODO
CATEGORICAL ir_key, ir_value_type, handler-specific props TODO
CONTINUOUS_INTERVAL_UNCERTAIN ir_key, ir_value_type, handler-specific props TODO
DISCRETE_INTERVAL_UNCERTAIN ir_key, ir_value_type, handler-specific props TODO
DISCRETE_SET_VALUES ir_key, ir_value_type, handler-specific props TODO
DISCRETE_UNCERTAIN_SET_VALUES_PROBS ir_key, ir_value_type, handler-specific props TODO
HISTOGRAM_BIN_UNCERTAIN ir_key, ir_value_type, handler-specific props TODO
HISTOGRAM_POINT_UNCERTAIN ir_key, ir_value_type, handler-specific props TODO
ID_TO_INDEX_SET ir_key, ir_value_type, handler-specific props TODO
INT_SET ir_key, ir_value_type, handler-specific props TODO
RESPONSE_LEVELS_ARRAY ir_key, ir_value_type, handler-specific props TODO
UNCERTAIN_CORRELATION_MATRIX ir_key, ir_value_type, handler-specific props TODO
UNCERTAIN_INIT_POINT_FLAG ir_key, ir_value_type, handler-specific props TODO

For domain-specific handlers marked TODO, consult models under dakota/python/dakota/spec that already use the handler. Those handlers typically require additional, handler-specific metadata beyond the generic ir_key / ir_value_type shape.

IR default values

IR defaults ensure a complete runtime IR even when users omit inputs. The generator merges three disjoint sources:

  • schema-extracted defaults recorded in dakota/src/schema_defaults_extracted.json;
  • explicit override defaults in dakota/src/default_overrides_registry.json; and
  • type-policy defaults in dakota/src/default_policy_registry.json.

The generator reads these via load_schema_defaults(), load_overrides(), and load_policies() and merges them into per-block IR contracts. Place a default in the Pydantic model when it is a grammar-level default. Use default_overrides_registry.json when the default is implementation- specific or not representable in schema. Use the policy registry for ordinary type zero/false/empty defaults. Ensure only one source claims an IR key; collisions cause generate_ir_tables.py to error.

Adding a new method or model

Follow the selection-wrapper pattern when adding a method or model family.

Steps:

  • Create a module in dakota/python/dakota/spec/method or dakota/python/dakota/spec/model.
  • Define a config class for the body; reuse mixins for shared syntax.
  • Declare parsed keywords as typed class attributes with DakotaField.
  • Define a single-field selection wrapper inheriting from MethodSelection or ModelSelection. The wrapper must contain exactly one field named with the user-visible keyword.
  • Import the wrapper in package init.py so it becomes part of the union assembly (MethodUnion or ModelUnion).

Minimal example:

class MyMethodConfig(DakotaBaseModel):
max_iterations: int | None = DakotaField(default=None, description="...")
class MyMethodSelection(MethodSelection):
my_method: MyMethodConfig = DakotaField(...)

Validation rules and computed fields

Validation rules and computed fields constitute part of the shared grammar contract and require coordinated Python and C++ changes. The Python side provides schema metadata and dispatch; the authoritative execution logic resides in the C++ dakota_validation module.

To add a ValidationRule:

  • Implement a ValidationRule subclass in dakota/python/dakota/spec/validation.
  • Call super().__init__ with rule name, context, referenced fields, literals, and error message.
  • Implement call(self,\ instance) to dispatch to C++ via _call_cpp_validator.
  • Add C++ declaration and implementation in packages/dakota_parser/dakota_validation.
  • Register the rule name in ValidatorRegistry::ValidatorRegistry().
  • Expose the function via pybind in bindings.cpp.
  • Add/update C++ tests under packages/dakota_parser/dakota_validation/test.
  • Attach the rule on the model via _VALIDATION_RULES.

Validation call contract:

  • instance: model serialized to dict.
  • fields: paths from validationFields.
  • literals: from validationLiterals.
  • context: descriptive string.
  • return: dict of mutations or empty dict for check-only rules.

To add a computed field:

  • Declare _COMPUTED_FIELDS:\ ClassVar[dict[str,\ ComputedFieldSpec]] on the model.
  • Add a matching @computed_field property that computes the value.
  • Implement C++ declaration and function in computed_fields.hpp and source.
  • Register the function in ComputedFieldRegistry::ComputedFieldRegistry().
  • Ensure pybind bindings expose the function.
  • Add/update C++ tests.

Example:

_COMPUTED_FIELDS: ClassVar[dict[str, ComputedFieldSpec]] = {
"lower_bounds": ComputedFieldSpec(
function="discrete_set_str_lower_bounds",
description="Lower bounds: min element per variable",
return_type=list[str],
),
}
@computed_field
@property
def lower_bounds(self) -> List[str]:
return DiscreteSetStrBounds.lower_bounds(self)

DakotaBaseModel.__init_subclass__ detects _VALIDATION_RULES and _COMPUTED_FIELDS and exports them as x-model-validations and x-computed-fields in schema. Keep Python property names, ComputedFieldSpec function strings, and C++ registry entries in exact agreement.

Block Pointers and API Mode

Overview

Block pointers are string-valued keywords whose arguments name other blocks in a Dakota study. For example, a method block may contain a model_pointer keyword whose value is the id_model of a model block. In a freeform .in input file, these pointers define the study graph — they are how the user wires methods, models, variables, interfaces, and responses together.

Dakota's Pydantic models serve two consumers:

  • Input-file mode — Models export a JSON schema that drives the C++ PEGTL-based freeform parser and the C++ JSON validation path (validate_json_input_to_json, read_json_input_file_to_json). In this mode, pointer keywords that the schema marks as required must be present because the study graph cannot be constructed without them.
  • API mode — The Python API and C++ per-block API (validate_*_block_json_to_json) let users construct individual components and assemble the study graph via dependency injection. Pointer keywords are meaningless in this context because the graph is defined programmatically, not by string references.

Because we maintain a single set of Pydantic models for both consumers, validation must distinguish between the two modes so that required pointers are enforced in input-file mode but relaxed in API mode.

Python: input_file_mode()

In Python, the default behavior is API mode. A model_validator on DakotaBaseModel (_fill_api_mode_pointer_sentinels) inspects every field that has an x-block-pointer annotation. When such a required field is missing from the user's input dict, the validator silently injects a sentinel value (POINTER_SENTINEL) so that Pydantic's normal required-field check passes. Optional pointer fields need no sentinel — Pydantic accepts their absence already.

To switch to input-file mode (where all required pointers are enforced), wrap the validation call in the input_file_mode context manager:

from dakota.spec.base import input_file_mode
with input_file_mode():
# This will raise ValidationError if truth_model_pointer is absent
config = ActiveSubspaceConfig(**user_dict)

This is a Python contextvars-based context manager, so it is thread-safe and re-entrant.

C++: api_mode parameter

The C++ JSON validation path threads a bool api_mode parameter through validate_json_document, walk_and_validate, and validate_definition. When api_mode is true, the required-field check in validate_definition skips any field that the generated metadata identifies as a block pointer.

The distinction is wired at the entry-point level:

  • Whole-file entry points (validate_json_input_to_json, read_json_input_file_to_json, and all freeform read_*_to_json functions) use api_mode=false. All required fields including pointers must be present.
  • Per-block API entry points (validate_method_block_json_to_json, validate_model_block_json_to_json, etc.) use api_mode=true. Required pointer fields are silently skipped; all other required fields are still enforced.

The api_mode parameter is threaded as a function argument (not a global variable) so the validation path is thread-safe and re-entrant.

How pointers are identified

A field is recognized as a block pointer when it carries x-block-pointer metadata in the exported JSON schema. In the Pydantic model, this is specified in the dakota dict of DakotaField:

truth_model_pointer: str = DakotaField(
description="Pointer to a truth model",
"block_pointer": "model",
...
},
)

The block_pointer key is mapped to x-block-pointer in the exported JSON schema. The code generator reads this annotation to build the set of pointer fields per definition, and the runtime required-field check consults it when api_mode is true.

Pointers must not have sub-keywords

Do not add child keywords (sub-keywords) to a pointer field.

In API mode, required pointer fields may be omitted entirely. If a pointer field carried child configuration keywords, that configuration would become inaccessible to API users who (correctly) omit the pointer. The result would be features that are configurable from a freeform input file but unreachable from the Python or C++ API.

If a pointer currently has associated sub-keywords, or a design calls for configuration that logically accompanies a pointer, factor the configuration into a sibling field or a separate model. The pointer should remain a plain str (or list[str]) with no nested structure.

Example of what NOT to do:

# BAD: truth_model_pointer has child configuration
truth_model_pointer: TruthModelPointerWithOptions = DakotaField(
description="...",
dakota={"block_pointer": "model"},
)
class TruthModelPointerWithOptions(DakotaBaseModel):
pointer: str = DakotaField(...)
some_option: int = DakotaField(default=5, ...) # inaccessible in API mode!

Instead, make the option a sibling field on the parent model:

# GOOD: pointer is a plain str; option is a sibling
truth_model_pointer: str = DakotaField(
description="...",
dakota={"block_pointer": "model"},
)
some_option: int = DakotaField(default=5, description="...")

Pointer Groups

Some required model-typed fields wrap a config model that contains required pointer fields plus optional configuration siblings. These are called pointer groups. A typical example is NestedConfig.sub_method_pointer, whose type SubMethodPointer holds a required pointer: str block-pointer field and optional scheduling fields such as iterator_servers.

When to use: A required field is a pointer group when its model contains required block_pointer fields (and optionally other fields with defaults) but the model as a whole must be omittable in API mode.

API-mode semantics: If the pointer-group field is absent, the _fill_api_mode_pointer_sentinels validator injects an empty dict {}. The child model's own before-validator then fills in the pointer sentinel. If the field is present (because the user wants to configure optional fields like iterator_servers), no injection occurs and validation proceeds normally.

Annotation: Add "pointer_group": True to the dakota dict of the DakotaField on the parent model's field:

sub_method_pointer: SubMethodPointer = DakotaField(
default=...,
description="...",
dakota={"argument": "pointer", "pointer_group": True, ...},
)

Origin: In freeform input, these correspond to keywords that take a string argument (the pointer) and also have optional sub-keywords. The Pydantic model splits this into a synthetic pointer field (annotated block_pointer) and sibling config fields. The parent field on the containing model is annotated pointer_group so that API-mode injection can reach it.

Do not annotate a field pointer_group if its wrapped model has required non-pointer fields (e.g., a required type-selection bool or a required method-name string). Such models require real user input and cannot be meaningfully auto-filled with an empty dict.

Pointer Unions

Some required Union fields have at least one branch that is a pointer-only or pointer-group wrapper. These are called pointer unions. A typical example is EmbeddedConfig.sub_method : Union[GlobalMethodName, GlobalMethodPointer], where GlobalMethodPointer is a pure pointer branch and GlobalMethodName has a real method-name string.

When to use: A required Union field is a pointer union when at least one of its branches is all-pointer (i.e., every required field on the branch model is either a block_pointer, a pointer_group, or another pointer_union).

API-mode semantics: If the pointer-union field is absent, the validator introspects the Union branches to find the first all-pointer branch. It then builds a minimal injection dict for that branch (injecting the pointer sentinel into the branch's pointer fields) and injects it. Child validators fill in sentinels bottom-up. If the user provides a branch (e.g., a method-name branch), that is real input and validates normally.

Annotation: Add "pointer_union": True to the dakota dict of the DakotaField on the Union field:

sub_method: Union[GlobalMethodName, GlobalMethodPointer] = DakotaField(
description="...",
dakota={"anchor": True, "union_pattern": 4, "pointer_union": True},
)

Defensive check: If no all-pointer branch is found, a ValueError is raised during validation with a message like: "pointer_union field 'sub_method' on EmbeddedConfig has no all-pointer branch". This catches developer annotation mistakes at validation time rather than silently misbehaving.

Do not annotate SurrogateConfig.category or other real type-selection Unions as pointer_union. Those Unions require the user to always pick a branch; they are not pointer plumbing.

Nesting Limits

The full pointer-relaxation stack supports three levels of nesting:

pointer_union  ->  pointer_group  ->  block_pointer

Deeper than three levels is administratively prohibited. If a future design requires deeper nesting, the developer must find a flatter representation. Deeply nested option hierarchies were tolerable for freeform input files but create ergonomic problems for API users and make automated injection brittle.

Guidance for new grammar: Favor broad, flat specifications. Prefer sibling fields over deeply nested option sub-models. When a pointer and optional configuration must coexist, use the pointer-group pattern (pointer as a sibling of the configuration, not as a child of it).

DakotaFloat

Use DakotaFloat for fields that may carry non-finite values (NaN, inf, -inf). DakotaFloat is an Annotated float with a custom serializer:

  • finite values remain numeric;
  • NaN serializes as "nan";
  • +inf serializes as "inf";
  • -inf serializes as "-inf".

This preserves compatibility between Python JSON and downstream C++ handling. Prefer DakotaFloat for fields that historically used sentinel floating values (e.g., DBL_MAX or -DBL_MAX).