Unexpected character in unicode tag.

[PeterHo249]

I has an issue with parsing unicode tag.
Input:
<リンク> This a second unicode tag</リンク>

Expected output:
<リンク> This a second unicode tag</リンク>

Real output:
<�リンク> This a second unicode tag</リンク>

I don't know why there is � here.

[geoffmcl]

@PeterHo249, you seem to have exposed an encoding BUG...

Yeah, have no idea at the moment why an input -

3C E3 83 AA E3 83 B3 E3 82 AF 3E
 can become 
26 6C 74 3B EF BF BD E3 83 AA E3 83 B3 E3 82 AF 26 67 74 3B

The 3C to 26 6C 74 3B is to be expected... and same for 3E... but...

Why was EF BF BD added??? the unicode replacement character!!

Now I am sure this is somehow triggered by the preceding <, 3C, char, because when the unicode is presented as text, there is no addition... no problem...

And see the second input with is preceded by a / char... and there is no problem...

Need to setup a debug session, to look at exactly what's going on... unless someone beats me to it...

Meantime, look forward to feedback, comments, patches, PR, etc, ... thanks...

[geoffmcl]

@PeterHo249, I think @jidanni has filed a duplicate, #913, showing this problem...

In his xml sample, where Unicode tags are allowed, it causes a cascading failure...

In brief, since tidy incorrectly encodes a multi-byte Unicode start tag, through ChangeChar, into the lexer, and is copied into the node, the tag later can not be match to the correctly encoded same name multi-byte closing tag, with catastrophic, cascading, errors thereafter...

In lengthy debugging, I can pin point the exact location where the lexer is trashed, lexer.c#L2894...

As can be seen, in the line before, the code fetches a char, c = TY_(ReadChar)(doc->docIn);... where c is a uint, but in the ChangeChar(lexer, (tmbchar)c); service, the size is reduced to a char. Now that is no problems if the character is in the single byte range, but when the uint c is a multi-byte Unicode, this changes the last char in the lexer to an incorrect value...

Take, for example, the XML sample given in #913, <縣市>anything</縣市>, the ReadChar will return 32291, U+7E23, from the utf-8 0xE7 0xB8 0xA3, which will be reduced to a #, 0x23, in ChangeChar... and the trouble begins...

This is residual coding that goes back to the days when tidy defaulted to the latin1 single byte code set, and at this time I am not sure how to re-code it...

Look forward to feedback, comments, patches, PR, etc, ... thanks...

[geoffmcl]

@PeterHo249, @jidanni, after some more debugging, am starting the get a glimmer of what to change...

In this case, and several other cases, a single byte character set is assumed, when adjusting, changing the lexer, and this is no longer true...

For example, when the lexer detects a start tag, it uses this code -

    /* check for start tag */
    if (TY_(IsLetter)(c) || (cfgBool(doc, TidyXmlTags) && TY_(IsXMLNamechar)(c)))
    {
        TY_(UngetChar)(c, doc->docIn);     /* push back letter */
        TY_(UngetChar)('<', doc->docIn);
        lexer->lexsize -= 2;      /* discard "<" + letter */
        lexer->txtend = lexer->lexsize;
        lexer->state = LEX_STARTTAG;         /* ready to read tag name */

Note the lexer->lexsize -= 2; /* discard "<" + letter */, but -= 2 is not correct, if the letter is a multi-byte Unicode code point, like 32291! It would be -= 4, ie discard < + 3 utf-8.

In most cases, this does not cause any real problem, except on occasions, it leaves some dead characters in the lexer buffer...

The idea forming is that the service TY_(AddCharToLexer)(lexer, c); returns the character count it added, and in these cases it is used to backup the lexer, not an assumed number...

And then in the case LEX_STARTTAG:, when this character is retrieved, AddCharToLexer is used to add the code point back, as multi-byte utf-8, at the correct place in the lexer... completely removing the byte oriented ChangeChar, which starts this problem...

Anyway, just an idea at this stage... must write the code to test it out...

Meantime, look forward to feedback, comments, ideas, patches, PR, etc., ... thanks...

[roeyfuchs]

Hi.. trying to solve this one
I started assuming we encoding only UTF8 and added this function to the lexer:

static uint GetNumberOfByte(uint c) {
    // UTF 8:
    uint ret_value = 1;
    if(c >= 0x0 && c <= 0x7F) {
        ret_value = 1;
    }
    else if(c >= 0x80 && c <= 0x7FF) {
        ret_value = 2;
    }
    else if(c >= 0x800 && c <= 0xFFFF) {
        ret_value = 3;
    }
    else if(c >= 0x10000 && c <= 0x10FFFF) {
        ret_value = 4;
    }
    return ret_value;
}

and following @geoffmcl comment, I changed lexer->lexsize -= 2 to lexer->lexsize -= (1 + GetNumberOfByte(c));.
I got diffrenet error message now:

line 1 column 32 - Error: unexpected </ンク> in <ンク>
This document has errors that must be fixed before

I believe we need to change one more thing in the closing tag detection, but Im not sure.
Im not so familiar with tidy's code, so any assistance would be appreciated.. thanks :)