Wednesday, June 3, 2015

.NET Rounding Error

A recent project of mine was creating a custom reporting engine in .NET since all of the commercial ones were either too expensive or not flexible enough for the projects requirements.  After creating the engine the natural next step was writing reports to run on it; it was during this process that I found a very interesting error in .net 4.

.NET apparently handles calculations differently depending on whether or not they are implicit or explicit.  For example if you were to run this statement:

If( 3 < (0.6F / 0.2F ))

you would expect .net to do the float calculation, then either cast the integer to a float, or round the float to an integer and perform the calculation.  Apparently it does not as the statement above always evaluates to true, when in reality it is false since the integer 3 is the same value as the float 3.0.  My guess is when performing the division there is some sort of math error in the .net engine which results in something like 3.0000000000000000000001 for 0.6F / 0.2F

The solution to this issue is to explicitly force trimming like this:

Int f = (int)(0.6F / 0.2F)
If( 3 < f )

The above statement casts the float calculation to an integer effectively truncating the extra floating data and allowing a correct comparison.  A better solution would probably be to use Math.Round just in case .net decides to evaluate the calculation to just under 3 rather than just over it.

No comments:

Post a Comment

Please leave your thoughts, I love hearing what you got out of the post. Spam comments will be removed.