Simulations
DraftA simulation produces numbers, and numbers are persuasive whether or not they are right. The purpose of recording a run properly is to make it possible to tell the difference later.
What to record
| Field | Content |
|---|---|
| Question | What the simulation is meant to determine |
| Model | Governing equations, force model, or system being simulated |
| Assumptions | Every simplification, stated explicitly |
| Tool and version | Which software, which version, which configuration |
| Inputs | Initial conditions, parameters, constants and their sources |
| Method | Integrator, step size, tolerances, convergence criteria |
| Validation | The known case the setup was checked against |
| Results | Outputs, with plots where they help |
| Sensitivity | How the result changes with the uncertain inputs |
| Interpretation | What it means, and what it does not |
The assumptions field is where most of the value is, and it is the one most often left implicit. A result is only meaningful within the assumptions that produced it, and a reader six months later, including yourself, will not reconstruct them.
Validation comes first
A simulation must be checked against something known before its novel results are believed. This is not optional and it is routinely skipped.
| Validation | Example |
|---|---|
| Analytical case | A two-body propagation must conserve energy and reproduce the closed-form solution |
| Known result | Reproduce a published figure with the published inputs |
| Conservation laws | Energy, momentum and angular momentum should be conserved where the physics conserves them |
| Limiting behaviour | The result should approach a known answer as a parameter goes to a limit |
| Independent implementation | A second tool with the same model should agree |
Cross-validation between independently written tools is the most practical check, and the process of finding out why two implementations disagree is how errors are actually located. See Simulation for the professional practice this mirrors.
A run whose validation is not recorded should be treated as unvalidated, regardless of whether it was checked at the time.
Sensitivity
A single run is a point estimate and says nothing about robustness. Varying the uncertain inputs shows which ones matter.
The output of the sensitivity step is the honest form of the result: not a single number, but a number with a statement of what it depends on. A drag decay prediction that varies by a factor of three across plausible atmospheric density should be reported that way rather than as a single date. See Orbital perturbations.
Reproducibility
A simulation result is only useful if it can be regenerated, and the requirements are more demanding than for an experiment because everything is in software and software changes.
- Archive the configuration
- Inputs, parameters and settings stored with the result, not reconstructed from memory.
- Record the version
- Of the tool, its dependencies and any data files such as ephemerides or atmosphere models. A different ephemeris version gives different answers.
- Store the outputs
- Not just the plots. A plot cannot be reanalysed.
- Note the environment
- Compiler, platform and floating point settings occasionally matter, particularly for chaotic systems and long integrations.
Data file versions are the most commonly missed. A propagation using a planetary ephemeris, an atmospheric density model or an Earth orientation parameter file depends on which version of those files was used, and they are updated regularly.
What simulation cannot do
Worth stating, because it is easy to forget when a model runs cleanly.
A simulation reproduces what was modelled. It cannot reveal a physical effect that was not included, and its most dangerous failure mode is producing a plausible answer to a question the model does not actually address.
The corresponding discipline is to state what the model omits alongside what it includes. A thermal model without a term for a heat path that exists will happily produce temperatures, and nothing about the output indicates the omission.
References
Simulations cite their models and sources. This page is a convention.