About Precision and Accuracy for Numeric Data - 17.1

Inline Quality and Discovery

Version
17.1
Language
English
Product name
Trillium Quality and Discovery
Title
Inline Quality and Discovery

For information about the functions that support this implementation, see Expression Builder Functions.

For numeric operations, a fixed point decimal arithmetic implementation is available. This allows for accurate arithmetic calculations by-passing previous floating point implementations and converting all numbers into integers. Since calculations are effectively done using integer arithmetic giving accurate results, you do not have to factor in floating point rounding errors. For example, you can simply add up 20000 + 20000 to get 40000, not 40000.00001.

For numeric operations, a fixed point decimal arithmetic implementation is available. This allows for accurate arithmetic calculations by-passing previous floating point implementations and converting all numbers into integers. Since calculations are effectively done using integer arithmetic giving accurate results, you do not have to factor in floating point rounding errors. For example, you can simply add up 20000 + 20000 to get 40000, not 40000.00001.

Note:

The Expression Builder supports 20 after the decimal place precision.

For example:

  • d[1].d[49] (result would be rounded to d[1].d[20])
  • 0.d[50] (leading zero are not significant; result would be rounded to 0.d[20])
  • d[50].0 (trailing zero are not significant)
  • d[10].d[40] (result would be rounded to d[10].d[20])
  • d[40].d[10]

where d = a digit from 0 to 9 and the [] indicates the number of digits in the number.

Note:

Significant digits are defined as the total digits in a number, excluding leading and trailing zeroes, before and after the decimal place. Precision is defined as the number of digits after the decimal place.