We are doing one of the critical projects. The code was working fine at dev, QA, UAT and prod fr sometime. Suddenly, we get a call from product support team, the actual value and the values shown by our code does not match by a cent. It was not a concern as the payments are rounded to next 5 cents. But wanted to confirm that there was nothing wrong in our calculation. We debugged the problem and figured out the sample code
double d = 320.085 // the value comes as input which worked for others
value = d*100
Java does not give the out put as 32008.5 rather it provides output as 32008.499999999996
It is because of the rounding off the values from decimal to binary system. And there are some minor variations. We tested for a range of decimals and their result.
We did the sample program
int hunderdThousand = 1000 * 1000;
double db = 0;
float f = 0;
for(int i=0;i<hunderdThousand;i++){
db += 0.001;
f += 0.001f;
}
System.out.println(db);
System.out.println(f);
Expected output was 1000 for both. but output werer 999.999999983265 and 991.1416 respectively. The double values are close to 1000 with 10 9's accuracy. However float erodes the value by almost 0.9%.
Wednesday, October 14, 2009
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment