Search waiting | Matsusada Precision

Searching...

Technical Terms

The Precision loss is a type of error that can occur in floating-point arithmetic when the number of significant digits in a value is reduced during a calculation. This often happens due to the finite precision with which computers can store real numbers. Two common scenarios lead to significant precision loss.

The first is "subtractive cancellation," which occurs when subtracting two numbers that are very nearly equal. For example, consider subtracting 1.23456789 from 1.23456780. The result is 0.00000009. The original numbers had nine significant digits of precision, but the result has only one. The leading significant digits have canceled each other out, and the result is dominated by the small rounding errors that were present in the original numbers.

The second scenario is when adding or subtracting numbers of vastly different magnitudes. If you add a very small number to a very large number, the contribution of the small number might be completely lost if it is smaller than the precision of the large number. For instance, (1.0 x 10^10) + 1.0 might just result in 1.0 x 10^10 if the floating-point representation doesn't have enough precision to store the result.

When writing software that processes measurement data from high-resolution instruments, programmers must be aware of potential precision loss and may need to use higher-precision data types (like 64-bit double-precision floats) or rearrange calculations to minimize these effects.

Related words

Information on related articles in Technical Knowledge