View Issue Details

IDProjectCategoryView StatusLast Update
0006380Multi Theft Auto : San AndreasScriptingpublic2011-08-21 22:27
ReporterarranTuna Assigned Toccw  
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
Target Version1.1 
Summary0006380: fileRead has problems reading a UTF-8 file
Description

When editor loads definitions it uses fileRead then loadstring. When it is loading it returns error:

INFO: Error: race/edf/edf_client.lua: [string "edf/edf_client.lua"]:1: '=' expected near '»'

Which causes race checkpoints to not work as they should.

Changing the script to ANSI fixes the problem

Steps To Reproduce

Try and make a race map in 1.1 with the newest resources (they're UTF-8 encoded) and notice the 2 load failures when starting.

TagsNo tags attached.

Relationships

child of 0006375 closed New issues Unicode issues 

Activities

Talidan

2011-08-18 21:05

administrator   ~~0014343

Last edited: 2011-08-18 21:19

Have you tried saving with UTF-8 without BOM?

I would imagine this isn't really a bug in fileRead, but merely the nature of BOM. Anything that reads a BOM needs to implement it's own BOM detection.

Something like this: http://pastebin.com/MN4KiqeA (line 18)

arranTuna

2011-08-18 21:21

manager   ~~0014344

Same problem when using UTF-8 without BOM.

Talidan

2011-08-18 21:40

administrator   ~~0014345

Really? That's very strange. » is a character that's part of BOM, i don't understand how that error would appear if there was no BOM.

arranTuna

2011-08-18 23:23

manager   ~~0014347

Ingame it was a ' and in the log it was that symbol.

MX_Master

2011-08-19 07:31

viewer   ~~0014349

http://pastebin.com/MN4KiqeA : not correct BOM check and cutting


local BOM = utfChar(0xEF)..utfChar(0xBB)..utfChar(0xBF)

if string.sub(script,1,3) == BOM then
script = string.sub(script,4)
end

must be


if string.byte( script, 1 ) == 0xEF then
script = string.sub( script, 4 )
end

arranTuna

2011-08-19 12:38

manager   ~~0014354

I tried both, neither worked.

MX_Master

2011-08-19 14:08

viewer   ~~0014355

can you give any simple script (and all files for it) to reproduce this issue?

arranTuna

2011-08-19 17:29

manager   ~~0014357

Last edited: 2011-08-19 17:31

Try and make a race map in 1.1 with the newest resources (they're UTF-8 encoded) and notice the 2 load failures when starting.

resources[editor]\edf\scriptreader.lua is where the reading takes place.

Load the race definition to see the errors.

MX_Master

2011-08-19 18:05

viewer   ~~0014358

Last edited: 2011-08-19 18:15

just saw this script (r773).. there is no BOM cutting there. It must be inserted here

http://code.google.com/p/mtasa-resources/source/browse/trunk/%5Beditor%5D/edf/scriptreader.lua#53

and here

http://code.google.com/p/mtasa-resources/source/browse/trunk/%5Beditor%5D/edf/scriptreader_client.lua#52

and my check

if string.byte( script, 1 ) == 0xEF then
script = string.sub( script, 4 )
end

must work ideally. And another side of this problem in creating scripts that can't be loaded.

Talidan

2011-08-19 19:09

administrator   ~~0014359

MX_Master, your check only checks a single byte of the BOM. The BOM has three bytes, which is why my script takes 3 bytes. I realise there's a mistake in my code though, it should be this:
local BOM = string.char(0xEF)..string.char(0xBB)..string.char(0xBF)

if string.sub(script,1,3) == BOM then
script = string.sub(script,4)
end

If using string.byte, you'd want something like:

if ( string.byte( script, 1 ) == 0xEF ) and ( string.byte( script, 2 ) == 0xBB ) and ( string.byte( script, 3 ) == 0xBF ) then
script = string.sub( script, 4 )
end

Either way, it appears there's some other issue at hand here.

Talidan

2011-08-19 19:11

administrator   ~~0014360

I'm thinking if this only happens to the clientside script loader, it could be to do with triggerClientEvent corrupting unicode strings. Will have to investigate.

MX_Master

2011-08-19 20:19

viewer   ~~0014361

1-byte check it's very logic and fast solution (: for the full check we can use

if string.find( script, "^\239\187\191" ) then
script = string.sub( script, 4 )
end

arranTuna

2011-08-21 18:13

manager   ~~0014381

None of the above worked still getting

[17:15:43] INFO: Error: race/edf/edf.lua: [string "edf/edf.lua"]:1: '=' expected near '?

arranTuna

2011-08-21 22:11

manager   ~~0014384

Wait, I made a mistake.

local BOM = string.char(0xEF)..string.char(0xBB)..string.char(0xBF)

if string.sub(script,1,3) == BOM then
script = string.sub(script,4)
end

Does work.

Issue History

Date Modified Username Field Change