Verified Is Not Validated: What a Simulation Report Can Claim

A word I see in a lot of simulation reports is "validated". Often the evidence behind it is a mesh study, or a check that the reaction forces add up to the applied load. Both are worth doing, and I ask for them on every job. Neither of them is validation.

A reader who sees "validated" assumes the model has been held up against the real thing and has matched it within some stated margin. If that comparison never happened, the report is claiming more than the work supports, and the person signing off a design or a process change is carrying a risk they cannot see.

1. Two different questions

The definitions most of us work from are the ones NAFEMS and the ASME V&V committees have published. Verification is the process of determining that a computational model accurately represents the underlying mathematical model and its solution. Validation is the process of determining the degree to which a model is an accurate representation of the real world, for its intended use.

The NAFEMS leaflet on the subject puts it simply: verification is the domain of mathematics, and validation the domain of physics. Verification asks whether we solved the equations correctly. Validation asks whether they were the right equations for the question the client actually has.

The chain from the real object to the numbers in a report, after the NAFEMS and ASME V&V definitions. A converged mesh speaks only to the right-hand link.

2. What verification can honestly claim

The same leaflet splits verification in two. Code verification checks that the mathematical model and solution algorithms are working correctly, which is mostly the job of the solver developers and their benchmark suites. Calculation verification checks that the discrete solution of our particular model is accurate, which is our job on every project.

Calculation verification is where the mesh study lives. For CFD, the grid convergence index proposed by Roache and set out as a reporting procedure by Celik and co-authors in the Journal of Fluids Engineering in 2008 gives a disciplined way to put a number on discretisation error, with a factor of safety of 1.25 when three grids are used. I have written before that this check belongs on page one of the report, beside the result it supports, and that a peak which keeps rising with refinement may be a singularity rather than a stress.

A well-converged answer is still an answer to the equations we chose to solve. If a bolted joint was modelled as bonded, refining the mesh simply converges on the wrong number with more confidence. That is why I am careful to write "verified" or "mesh-independent" and stop there when that is all the work has shown.

3. Validation needs a measurement

ASME V&V 20, the standard for verification and validation in CFD and heat transfer, frames validation as a comparison. The comparison error is E = S - D, the simulation result minus the experimental value. Against it sits the validation uncertainty, which combines the numerical uncertainty, the uncertainty from uncertain inputs and the uncertainty in the measurement itself. If E is much larger than the validation uncertainty, the difference is probably the modelling error. If it is smaller, the model error is hidden somewhere inside that uncertainty band, and the comparison cannot say more than that.

I like this framing because it is strict about what has to exist before the word can be used. There has to be a measured quantity with its own uncertainty, and the simulation has to predict that same quantity at the same place and under the same conditions. Someone also has to decide beforehand how close is close enough for the decision at hand. A strain gauge reading on a prototype bracket or a pressure tap on a duct can support a validation statement for the quantity it measured. A colourful contour plot that "looks right" cannot.

4. What I write when there is no test data

Most consulting work happens before anything is built, so there is often nothing to measure. In those cases I think the honest course is to say so plainly and then describe what the confidence does rest on.

Where a published experiment exists with the same physics, such as flow over a backward-facing step or a plate with a hole under tension, the same modelling choices can be run against it first. That is validation of the modelling approach on a benchmark, and I would describe it in exactly those words, with the benchmark named. When the inputs are the weak point, a sensitivity study on the few values nobody is sure of usually tells the client more than any comparison would.

Calibration needs its own word. If a model has been tuned until it matches one site measurement, it may be perfectly useful, but it has not been validated by the same measurement it was tuned on. I would want a second, independent reading before using the stronger word.

The other honest statement is about how the result will be used. A study that ranks three layouts relative to each other needs less from the model than one that predicts an absolute temperature against a hard limit. I would rather a report said "this model is fit to compare the options, and the absolute values carry an uncertainty we have not quantified" than borrow a word it has not earned.

5. The words I try to hold to

I do not see "verified" as the lesser word. A verified model with a clear note of what it has not been compared against is good engineering, and a reviewer can work with it. What I push back on in reviews is the jump from the third row of that table to the fifth without the comparison in between.

Where Shirsh fits

At Shirsh Technosolutions we carry out FEA and CFD with ANSYS Mechanical, ANSYS Fluent, OpenFOAM and Code_Aster, and we review simulation reports prepared by others. If you have a report in hand and would like a second opinion on what it has verified and what it has actually been compared against, please get in touch.

References: NAFEMS and ASME, "What is Verification and Validation?", leaflet of the NAFEMS Analysis Management Working Group (definitions of verification, validation, code verification and calculation verification), referring to ASME V&V 10, Guide for Verification and Validation in Computational Solid Mechanics. ASME V&V 20-2009, Standard for Verification and Validation in Computational Fluid Dynamics and Heat Transfer, as summarised by H. W. Coleman, "An Overview of ASME V&V 20" (comparison error E = S - D, validation uncertainty and its components). I. B. Celik, U. Ghia, P. J. Roache, C. J. Freitas, H. Coleman and P. E. Raad, "Procedure for Estimation and Reporting of Uncertainty Due to Discretization in CFD Applications", Journal of Fluids Engineering, 130(7), 078001, 2008. P. J. Roache, grid convergence index and its factor of safety, as discussed in A. Satyadharma and Harinaldi, Computation 9(2), 10, 2021.