Precision and Accuracy for Numeric Data - trillium_discovery - 17.1

Trillium Discovery Center

Product type
Software
Portfolio
Verify
Product family
Trillium
Product
Trillium > Trillium Discovery
Version
17.1
Language
English
Product name
Trillium Discovery
Title
Trillium Discovery Center
Topic type
How Do I
Overview
Configuration
Reference
Administration
Installation
First publish date
2008

For numeric operations, a fixed point decimal arithmetic implementation is available. This allows for accurate arithmetic calculations by-passing previous floating point implementations and converting all numbers into integers. Since calculations are effectively done using integer arithmetic giving accurate results, you do not have to factor in floating point rounding errors. For example, you can simply add up 20000 + 20000 to get 40000, not 40000.00001.

Note: The Expression Builder supports 20 after the decimal place precision.

For example:

  • d[1].d[49] (result would be rounded to d[1].d[20])
  • 0.d[50] (leading zero are not significant; result would be rounded to 0.d[20])
  • d[50].0 (trailing zero are not significant)
  • d[10].d[40] (result would be rounded to d[10].d[20])
  • d[40].d[10]

where d = a digit from 0 to 9 and the [] indicates the number of digits in the number.

Note: Significant digits are defined as the total digits in a number, excluding leading and trailing zeroes, before and after the decimal place. Precision is defined as the number of digits after the decimal place.

For information about the functions that support this implementation, see Expression Builder Functions.