INT Function Rounding Problems - Page 2
CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Page 2 of 3 FirstFirst 123 LastLast
Results 16 to 30 of 41

Thread: INT Function Rounding Problems

  1. #16
    Join Date
    Jul 2006
    Location
    Germany
    Posts
    3,722

    Re: INT Function Rounding Problems

    Yes, and not yet finished.
    @David, I'm not sure if this explins what happens here.
    Remember the core question is
    Why is Int(tv * 15 + 0.5) correctly 32 and
    why is Int(tv * Val("15") + 0.5) wrongly 31
    Val("15") can only evaluate to 15, but I think it depends what datatype this 15 is gonna be.
    If tv is a double (not a currency) Val("15") as a multiplier evaluates to a double, too, and the result is correct.
    I suspect when tv is a currency, Val("15") becomes a currency, too, as well as 0.5, and then and only then we get the error.
    Only I can't explain then, why the result is ok when you give 15 literally...
    It's really a little disquieting.

  2. #17
    Join Date
    May 2009
    Location
    London
    Posts
    51

    Angry Re: INT Function Rounding Problems

    The Val function returns a double.
    If this is the case then I would assume that the tv variable (if it is a currency) would have to be converted to a double as double variables have a much larger range.
    Converting the result of the val function to a currency may result in overflow errors whereas converting the tv to a double would only affect the accuracy of the number.

    WoF: The question is also why if tv is a currency variable Int(tv * Val("15") + 0.5) returns 31 but (tv * Val("15") + 0.5) returns 32. Basically int(32)=31!

  3. #18
    Join Date
    Jul 2006
    Location
    Germany
    Posts
    3,722

    Re: INT Function Rounding Problems

    Right, right you are.
    And now after this one gave me the willies for days now, I'll add to this some more.
    Code:
    dim tv as Single
    tv = 2.1
    debug.print tv * Val("15") 'prints 31.4999985694885
    debug.print tv * 15           'prints 31.5
    Maybe there is the reason. We get conversion to a single where it is not 31.5 but 31.49...
    Then the rounding of the Int() function gets it and cuts off to 31.
    Then Adding 0.5 makes 31.5 and the final rounding of the Int() function drops bakc to 31.

    Well I'm not sure, but it is something like this.

  4. #19
    Join Date
    Jan 2006
    Location
    Chicago, IL
    Posts
    14,982

    Re: INT Function Rounding Problems

    Didn't you read my post?

    To make them equal, I added cInt()
    Code:
    Private Sub Form_Load()
        Dim tv As Single
        tv = 2.1
        Debug.Print tv * CInt(Val("15"))
        Debug.Print tv * 15
    End Sub
    David

    CodeGuru Article: Bound Controls are Evil-VB6
    2013 Samples: MS CODE Samples

    CodeGuru Reviewer
    2006 Dell CSP
    2006, 2007 & 2008 MVP Visual Basic
    If your question has been answered satisfactorily, and it has been helpful, then, please, Rate this Post!

  5. #20
    Join Date
    Jul 2006
    Location
    Germany
    Posts
    3,722

    Re: INT Function Rounding Problems

    David, I read your post full well.
    Now your introduction of CInt() produces a new aspect of weirdness.
    Try this
    Code:
        Dim tv As Single
        tv = 2.1
        'Debug.Print tv * CInt(Val("15"))
        'Debug.Print tv * 15
        Debug.Print Int(tv * CInt(Val("15") + 0.5))
        Debug.Print Int(tv * CInt(15) + 0.5)
    If you remove the Int function in fornt of the expressions you'll see some binary rounding error for sure.

  6. #21
    Join Date
    Jan 2006
    Location
    Chicago, IL
    Posts
    14,982

    Re: INT Function Rounding Problems

    Adding cInt() does it again. What answer do you want?

    Code:
    Private Sub Form_Load()
        Dim tv As Single
        tv = 2.1
        Debug.Print tv * CInt(Val("15")) ' 31.5
        Debug.Print tv * 15 ' 31.5
        Debug.Print Int(tv * CInt(Val("15") + 0.5)) ' 33
        Debug.Print Int(tv * CInt(15) + 0.5) ' 31
        Debug.Print CInt(tv * CInt(15) + 0.5) ' 32
        Debug.Print (tv * CInt(Val("15") + 0.5)) ' 33.6
        Debug.Print (tv * CInt(15) + 0.5) ' 31.9999985694885
    End Sub
    David

    CodeGuru Article: Bound Controls are Evil-VB6
    2013 Samples: MS CODE Samples

    CodeGuru Reviewer
    2006 Dell CSP
    2006, 2007 & 2008 MVP Visual Basic
    If your question has been answered satisfactorily, and it has been helpful, then, please, Rate this Post!

  7. #22
    Join Date
    Jul 2006
    Location
    Germany
    Posts
    3,722

    Re: INT Function Rounding Problems

    Yes, I admitted already that CInt does it. There were already two more solutions how to avoid the problem, but there was no real explanation, why that darn error happens.

  8. #23
    Join Date
    Jan 2006
    Location
    Chicago, IL
    Posts
    14,982

    Re: INT Function Rounding Problems

    I posted the latest code, along with the link to my group. They didin't understand what the question is? Then they pointed to the link I gave before explaining Banker's Rounding. Int() does not, but Cint() does use it.
    David

    CodeGuru Article: Bound Controls are Evil-VB6
    2013 Samples: MS CODE Samples

    CodeGuru Reviewer
    2006 Dell CSP
    2006, 2007 & 2008 MVP Visual Basic
    If your question has been answered satisfactorily, and it has been helpful, then, please, Rate this Post!

  9. #24
    Join Date
    Jul 2006
    Location
    Germany
    Posts
    3,722

    Re: INT Function Rounding Problems

    Ok, I will not longer insist in this, since the reason of this might be the bankers or no bankers rounding.

    So in memory of this code:
    Code:
    dim tv as Single
    tv = 2.1
    debug.print tv * Val("15") 'prints 31.4999985694885
    debug.print tv * 15           'prints 31.5
    I will issue the warning never to use the Val() function within an expression where rounding takes place.
    Because the full integral value of 15 is 'rounded' down to 14.9999999 (something like that)
    What I wanted to point out is, the Val() function should NOT do any (un)rounding at all, bankers or no.

  10. #25
    Join Date
    Jan 2006
    Location
    Chicago, IL
    Posts
    14,982

    Re: INT Function Rounding Problems

    I think Val() evaluates as a Long, which is the rounding problem.
    David

    CodeGuru Article: Bound Controls are Evil-VB6
    2013 Samples: MS CODE Samples

    CodeGuru Reviewer
    2006 Dell CSP
    2006, 2007 & 2008 MVP Visual Basic
    If your question has been answered satisfactorily, and it has been helpful, then, please, Rate this Post!

  11. #26
    Join Date
    Apr 2009
    Posts
    394

    Re: INT Function Rounding Problems

    ?typename(val("1"))
    Double

  12. #27
    DataMiser is offline Super Moderator Power Poster
    Join Date
    Jul 2008
    Location
    WV
    Posts
    4,830

    Re: INT Function Rounding Problems

    I ran some tests. Val("15") returns 15. The problem only seems to occur here when tv is defined as a single and the val() statement is used without being converted to a specific type. Vb is guessing at what type of var should be returned and apparently in this case it is incorrect.

    If I use a double or currency it returns correctly, if I place the val() inside a csng() or cint() it returns correct. If I place it inside a clng() or Ccur() or Cdbl() it shows the issue.

    It would seem that one should assign the result of the Val() statement into the proper variable type before doing math operations on the data returned by this function.

  13. #28
    Join Date
    Jul 2006
    Location
    Germany
    Posts
    3,722

    Re: INT Function Rounding Problems

    Yes, that's what I concluded, too:
    Do not use Val() inside expressions with different datatypes.

  14. #29
    Join Date
    Jan 2000
    Location
    Saskatchewan, Canada
    Posts
    595

    Re: INT Function Rounding Problems

    Val function is Evil.

    Try
    ? val("3,512.12")


    A good reason not to use it.

  15. #30
    Join Date
    Jul 2006
    Location
    Germany
    Posts
    3,722

    Re: INT Function Rounding Problems

    Yes, well it is known that the val function stops at the next nonnumerical character, which - in this case - is the comma. That's not very good, but it's a well known fact.
    What I did not know is, that it will fail at evaluating an integer to the point, when used within an expression of other numeric data types.
    Since other type conversion functions like CCur(), CInt(), CDbl() take string type arguments as well, your advice not to use Val() is not wrong, though.
    Last edited by WoF; May 27th, 2009 at 05:57 PM.

Page 2 of 3 FirstFirst 123 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


Azure Activities Information Page

Windows Mobile Development Center


Click Here to Expand Forum to Full Width

This is a CodeGuru survey question.


Featured


HTML5 Development Center