CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Results 1 to 5 of 5

Threaded View

  1. #5
    Join Date
    Feb 2017
    Posts
    674

    Re: Float point number simple arithmetic.

    In addition to my post #4.

    Not all decimal numbers have a finite digital representation. This includes 0.1, which requires an infinite number of bits. So when you assign 0.1 to a single-precision FP, there is a truncation error. When you print the FP, you see 0.1000000015, where the 15 is the error. The 15, however, is not part of the significant-digit range and should not be considered. (If you instead assign 0.125, there is no truncation because it has a finite digital representation that fits within the significant-digit range.)

    Now say you assign 0.1000000015 to a double-precision FP. This is a different story because now the 15 is part of the significant-digit range of the FP. If you square this FP, a 3 appears at a certain position determined by the place of the 15. If you instead square 0.1015, the 3 shows up again but in another position. This happens both with single and double precision. It is because 0.1015 is within the significant-digit range of both, as is the 3 in the squared result.
    Last edited by wolle; December 1st, 2021 at 01:24 AM.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  





Click Here to Expand Forum to Full Width

Featured