Convert the return value from Unicode. This is because your DLL returns BSTR but VB treats all strings (when calling dll function usiing Declare) as single byte which caused the addition of byte 0s in between.
Code:
Form1.Text3.Text = Len(StrConv(FSR(tstr, strlen), vbFromUnicode))
Form1.Text4.Text = Len(StrConv(FSL(tstr, strlen), vbFromUnicode))
Form1.Text5.Text = Len(tFSR(tstr, strlen))
Form1.Text6.Text = Len(tFSL(tstr, strlen))
Open FName For Output As FNum
Print #FNum, StrConv(FSR(tstr, strlen), vbFromUnicode)
Print #FNum, StrConv(FSL(tstr, strlen), vbFromUnicode)
Print #FNum, tFSR(tstr, strlen)
Print #FNum, tFSL(tstr, strlen)
Close FNum
Check also the parameter that you pass (FSString). If the function expects ANSI string, use the Byval. If it is unicode, use the Byref As Any. But I think, the original one (Byval as String) u used before is fine because it was working before except the return is not ok.

Hope it will work for you