Math Error caused by Floor Function?
These 4 boxes have the same math expression. The boxes on the left use a Real attribute, the boxes on the right use Integer. The ones on the top do not use FLOOR, the ones in the bottom use FLOOR (again, all have the same math expression).
The expression is simple: 11111040*1.3/4.5/26 (don't ask why...). As you can see, there is no decimal part to the results (validated by the Real attribute and by Excel), however FLOOR seem to produce a different result.
Here is the project for you to look at: http://www.4shared.com/zip/Obvonc2n/math.html
Anyone encounter something like that?
Thanks,
GR.
Comments
when you ask for a floor() function … in this calculating instance … it rounds the value down
though the ceil() function does not round it up
go figure??? … guess if you want all the same values … eliminate the floor()
just putting the equation itself in the Display Text …
… works well without interveningAttribute
or changing a text type attribute To: the expression
…stores nicely … and can be used in mathematical operations
and, when we get writable tables … you will probably store there
okay … don't know where you are going with this one @EatingMyHat
but, love how you show the first 6 numbers in numerical order!
I put the equation in the Display Text to simplify the test, in my code these are all attributes (REAL). But when I started to get errors, I broke it down and placed the numbers there. Not sure how Writeable tables will help with the math errors.
I checked up to 45 figures after the decimal point... there is no hidden decimal there. Checked it with ROUNDDOWN and TRUNC functions and still, Excel insists that there is no decimal to round here.
Thanks,
GR