There are two “text handling” types in Xojo – String and Text.<\/p>\n\n\n\n
And they vary quite a bit in how they handle textual data.<\/p>\n\n\n\n
While strings use UTF-8 as their default encoding you still have to worry about what form of UTF-8 the characters in the string are in. Strings dont deal with “characters” in the way you and I perceive them. <\/p>\n\n\n\n
For instance if you run this code<\/p>\n\n\n\n
Dim s1 As String = \"\u00fc\"\nDim s2 As String = &u75 + &u308\n\nBreak\n<\/code><\/pre>\n\n\n\nWhat you see in the debugger as the text they hold is the same<\/p>\n\n\n\n
But if you chnage this to<\/p>\n\n\n\n
\nDim s1 As String = \"\u00fc\"\n\nDim s2 As String = &u75 + &u308\n\nIf s1 = s2 Then \n Break\nEnd If\n<\/code><\/pre>\n\n\n\nwhat you will find is that while they appear to you and me to hold the same contents they are not “the same”. And this is because the first one uses one form of UTF-8 (composed characters) and the second uses a different form (decomposed characters)<\/p>\n\n\n\n
And theres no built in mechanism to know one is in one form or the other nor any to convert one into the other \ud83d\ude41<\/p>\n\n\n\n
So years ago the “next” framework was created and a new type added – Text. And it deals with these issues much better. That same code, using text, looks like<\/p>\n\n\n\n
Dim t1 As Text = \"\u00fc\"\nDim t2 As Text = &u75 + &u308\n\nIf t1 = t2 Then\n Break\nEnd If\n<\/code><\/pre>\n\n\n\nbut this time when you run you WILL hit the break point. Text handles the different forms seamlessly and you get the result you expect.<\/p>\n\n\n\n
And the differences go further than this. When you split a string up into “characters” you get different numbers of characters from the two apparently equal strings. Not so with text. <\/p>\n\n\n\n
\nDim s1 As String = \"\u00fc\"\n\nDim s2 As String = &u75 + &u308\n\nIf s1 = s2 Then \n Break \/\/ wont stop here but you might expect it should\nEnd If\n\nDim s1Chars() As String = s1.Split(\"\")\nDim s2Chars() As String = s2.Split(\"\")\n\nBreak \/\/ note that s1chars. ubound < s2chars.ubound\n \/\/ and the contents are totally different\n\nDim s1CodePoints() As UInt32\nFor i As Integer = 1 To s1.LenB\n s1CodePoints.Append AscB(s1.MidB(i,1))\nNext\nDim s2CodePoints() As UInt32\nFor i As Integer = 1 To s2.LenB\n s2CodePoints.Append AscB(s2.MidB(i,1))\nNext\n\nBreak \/\/ again the ubounds are different - this time they should be !\n\nDim t1 As Text = \"\u00fc\"\n\nDim t2 As Text = &u75 + &u308\n\nIf t1 = t2 Then\n Break\nEnd If\n\nDim t1Chars() As Text = t1.Split\nDim t2Chars() As Text = t2.Split\n\nBreak \/\/ note that t1Chars.ubound = t2Chars.ubound\n \/\/ and the chars are \"the same\" !!!!!!\n\nDim t1CodePoints() As UInt32\nFor Each cp As UInt32 In t1.Codepoints\n t1CodePoints.Append cp\nNext\nDim t2CodePoints() As UInt32\nFor Each cp As UInt32 In t2.Codepoints\n t2CodePoints.Append cp\nNext\n\nBreak \/\/ these should differ since one uses one form of utf-8\n \/\/ and one uses a different one<\/code><\/pre>\n\n\n\nText just handles things seamlessly<\/p>\n\n\n\n
With the transition to API 2 it will be a shame if String doesnt adopt some of these capabilities AND there’s no framework provided means to normalize string so they all use UTF-8 composed or decomposed so we can deal with the inconsistencies that can arise.<\/p>\n","protected":false},"excerpt":{"rendered":"
There are two “text handling” types in Xojo – String and Text. And they vary quite a bit in how they handle textual data. While strings use UTF-8 as their default encoding you still have to worry about what form of UTF-8 the characters in the string are in. Strings dont deal with “characters” in … <\/p>\n