MS "smart tags" are not recognized

[ivanov17]

Hello! I've processed with tidy 5.6.0 some old html pages which have been made in good old times with MS Word. I'd like to clean html code and therefore I've used tidy with option --word-2000 yes. In general tidy makes this work perfectly.

But these pages have MS-specific "smart tags" and there I have a problem. Tidy doesn't processed pages which have tags <o:smarttagtype> and various <st1> tags such as <st1:state>, <st1:place>, and so on.

All errors that I get contains same things:

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 15 column 1 - Error: <o:smarttagtype> is not recognized!
line 17 column 1 - Error: <o:smarttagtype> is not recognized!
line 114 column 39 - Error: <st1:city> is not recognized!
line 114 column 49 - Error: <st1:place> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 11 column 51 - Error: <o:smarttagtype> is not recognized!
line 12 column 76 - Error: <o:smarttagtype> is not recognized!
line 1516 column 15 - Error: <st1:state> is not recognized!
line 1516 column 26 - Error: <st1:place> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 15 column 1 - Error: <o:smarttagtype> is not recognized!
line 17 column 1 - Error: <o:smarttagtype> is not recognized!
line 19 column 1 - Error: <o:smarttagtype> is not recognized!
line 21 column 1 - Error: <o:smarttagtype> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 15 column 1 - Error: <o:smarttagtype> is not recognized!
line 17 column 1 - Error: <o:smarttagtype> is not recognized!
line 19 column 1 - Error: <o:smarttagtype> is not recognized!
line 21 column 1 - Error: <o:smarttagtype> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 14 column 1 - Error: <o:smarttagtype> is not recognized!
line 197 column 3 - Error: <st1:metricconverter> is not recognized!
line 199 column 33 - Error: <st1:metricconverter> is not recognized!
line 232 column 73 - Error: <st1:metricconverter> is not recognized!
line 244 column 28 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 345 column 20 - Error: <st1:metricconverter> is not recognized!
line 346 column 61 - Error: <st1:metricconverter> is not recognized!
line 386 column 14 - Error: <st1:metricconverter> is not recognized!
line 543 column 33 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 156 column 64 - Error: <st1:metricconverter> is not recognized!
line 193 column 28 - Error: <st1:metricconverter> is not recognized!
line 196 column 49 - Error: <st1:metricconverter> is not recognized!
line 198 column 46 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 221 column 74 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 14 column 1 - Error: <o:smarttagtype> is not recognized!
line 194 column 34 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 203 column 60 - Error: <st1:metricconverter> is not recognized!
line 212 column 63 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 142 column 37 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 14 column 1 - Error: <o:smarttagtype> is not recognized!
line 220 column 62 - Error: <st1:metricconverter> is not recognized!
line 230 column 78 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 155 column 18 - Error: <st1:metricconverter> is not recognized!
line 165 column 31 - Error: <st1:metricconverter> is not recognized!
line 175 column 35 - Error: <st1:metricconverter> is not recognized!
line 196 column 66 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 122 column 73 - Error: <st1:metricconverter> is not recognized!
line 137 column 71 - Error: <st1:metricconverter> is not recognized!
line 146 column 65 - Error: <st1:metricconverter> is not recognized!
line 160 column 26 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 14 column 1 - Error: <o:smarttagtype> is not recognized!
line 214 column 10 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 14 column 1 - Error: <o:smarttagtype> is not recognized!
line 691 column 17 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 498 column 40 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 248 column 31 - Error: <st1:metricconverter> is not recognized!
line 275 column 31 - Error: <st1:metricconverter> is not recognized!
line 350 column 16 - Error: <st1:metricconverter> is not recognized!
line 369 column 17 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 14 column 1 - Error: <o:smarttagtype> is not recognized!
line 1144 column 78 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 14 column 1 - Error: <o:smarttagtype> is not recognized!
line 840 column 35 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 15 column 1 - Error: <o:smarttagtype> is not recognized!
line 17 column 1 - Error: <o:smarttagtype> is not recognized!
line 106 column 20 - Error: <st1:date> is not recognized!
line 149 column 19 - Error: <st1:country-region> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 156 column 58 - Error: <st1:metricconverter> is not recognized!
line 158 column 75 - Error: <st1:metricconverter> is not recognized!
line 176 column 1 - Error: <st1:metricconverter> is not recognized!
line 199 column 33 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 158 column 71 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 197 column 64 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 134 column 64 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 143 column 21 - Error: <st1:metricconverter> is not recognized!
line 151 column 42 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 132 column 60 - Error: <st1:metricconverter> is not recognized!
line 201 column 12 - Error: <st1:metricconverter> is not recognized!
line 208 column 32 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 14 column 1 - Error: <o:smarttagtype> is not recognized!
line 207 column 72 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 119 column 76 - Error: <st1:metricconverter> is not recognized!
line 157 column 3 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 14 column 1 - Error: <o:smarttagtype> is not recognized!
line 157 column 62 - Error: <st1:metricconverter> is not recognized!
line 194 column 47 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 232 column 58 - Error: <st1:metricconverter> is not recognized!
line 235 column 47 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 141 column 53 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 15 column 1 - Error: <o:smarttagtype> is not recognized!
line 135 column 71 - Error: <st1:personname> is not recognized!
line 141 column 47 - Error: <st1:metricconverter> is not recognized!
line 218 column 46 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 236 column 15 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 315 column 18 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 169 column 29 - Error: <st1:metricconverter> is not recognized!
line 180 column 24 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 14 column 1 - Error: <o:smarttagtype> is not recognized!
line 257 column 66 - Error: <st1:place> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 14 column 1 - Error: <o:smarttagtype> is not recognized!
line 139 column 10 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 14 column 1 - Error: <o:smarttagtype> is not recognized!
line 160 column 76 - Error: <st1:metricconverter> is not recognized!
line 176 column 37 - Error: <st1:metricconverter> is not recognized!
line 197 column 38 - Error: <st1:metricconverter> is not recognized!
line 206 column 32 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 301 column 44 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 121 column 12 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 224 column 14 - Error: <st1:metricconverter> is not recognized!
line 236 column 32 - Error: <st1:metricconverter> is not recognized!
line 286 column 71 - Error: <st1:metricconverter> is not recognized!
line 288 column 41 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 143 column 9 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 147 column 54 - Error: <st1:metricconverter> is not recognized!
line 163 column 52 - Error: <st1:metricconverter> is not recognized!
line 174 column 63 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 134 column 75 - Error: <st1:metricconverter> is not recognized!
line 327 column 75 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 124 column 3 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 254 column 30 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 396 column 11 - Error: <st1:metricconverter> is not recognized!
line 411 column 25 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 139 column 29 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 14 column 1 - Error: <o:smarttagtype> is not recognized!
line 404 column 14 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 262 column 16 - Error: <st1:metricconverter> is not recognized!
line 292 column 63 - Error: <st1:metricconverter> is not recognized!
line 293 column 80 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 144 column 80 - Error: <st1:metricconverter> is not recognized!
line 215 column 57 - Error: <st1:metricconverter> is not recognized!
line 233 column 67 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 180 column 38 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 119 column 33 - Error: <st1:metricconverter> is not recognized!
line 148 column 32 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 161 column 61 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 116 column 8 - Error: <st1:metricconverter> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.

line 13 column 1 - Error: <o:smarttagtype> is not recognized!
line 724 column 66 - Error: <st1:personname> is not recognized!
This document has errors that must be fixed before
using HTML Tidy to generate a tidied up version.`

I tried other options such as --drop-proprietary-attributes yes and --bare yes, but I've got same errors.

I would like to get advice about it or workaround for this issue.

Thanks.

---

OS: Fedora release 30 (Thirty) x86_64
Kernel: 5.4.7-100.fc30.x86_64
HTML Tidy for Linux version 5.6.0

[ler762]

On 1/16/20, anarchist Ivanov ***@***.***> wrote: Hello! I've processed with tidy 5.6.0 some old html pages which have been made in good old times with MS Word. I'd like to clean html code and therefore I've used tidy with option `--word-2000 yes`. In general tidy makes this work perfectly. But these pages have MS-specific "smart tags" and there I have a problem. Tidy doesn't processed pages which have tags `<o:smarttagtype>` and various `<st1>` tags such as `<st1:state>`, `<st1:place>`, and so on.

<.. snip ..>

I would to get recommendations about it or workaround for this issue.

It'd be nice if someone could say how '--new-inline-tags' was supposed to work. I found this page https://www.farrail.com/pages/touren-engl/Steam-in-china-2017-Sandaoling+Fuxin.php that has various '<st1' tags, so I tried tidy --new-inline-tags st1:city,st1:place,st1:country-region and it mangled the file pretty badly. Removing the tags and marking it as an html 4 doc worked for me: sed -E -e '1,1 s@^<html>$@<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html>@' \ -e 's@</?st1:[Cc]ity[^>]*>@@g' \ -e 's@</?st1:[Pp]lace[^>]*>@@g' \ -e 's@</?st1:[Cc]ountry-region[^>]*>@@g' \ /tmp/x.html |\ tidy -q -indent -wrap 120 --tidy-mark no \ --drop-empty-elements no \ --drop-empty-paras no \ --word-2000 yes --preserve-entities yes Regards, Lee

[geoffmcl]

@ivanov17 thank you for your issue... lots of error messages...

But no sample(s) to try, experiment, test, understand, begin, etc...

And, please, no, I do not want the full word outputs, nor pointers to html in the wild...

Just minimum, to be a word document, and word html snippets that have only one of the tag types...

With such test samples, we can begin to see if a fix is easy, in the extra code run IFF --word-2000 yes is in the config... that code has not been touched in quite a long time... and maybe just a few tweaks... or maybe not!

Or as @ler762 suggests, sed can remove things, before tidy sees it...

So, some small, relevant, sample(s) please... maybe with output expected... thanks...

[ivanov17]

@ler762
Thank you. I used these regular expressions and got rid of errors took place while processing my files:

sed -e ':again;$!N;$!b again; :b; s/<\/\?o:SmartTagType[^<>]*"\?>//g; t b'
sed -e ':again;$!N;$!b again; :b; s/<\/\?st[1-9]:[^<>]*"\?>//g; t b'

I also downloaded and processed file linked to your answer and my regexps worked perfectly with it. I used options --word-2000 yes and --bare yes.
But if I tried to add --new-inline-tags st1:city,st1:place,st1:country-region, tidy saves these tags inside the body of the document, encodes some of angle brackets inside the head and moves some of meta tags and title tag to the body:

<head>
  <meta name="generator" content="HTML Tidy for HTML5 for Linux version 5.6.0">
  <meta http-equiv="content-type" content="text/html; charset=utf-8">
  <meta http-equiv="keywords" name="keywords" content="Steam in China, &lt;st1:place w:st=">
  <title></title>
</head>
<body>
  <st1:city w:st="on">Sandaoling</st1:city>, Fuxin, class JS, 2-8-2 class SY, 2-8-2"&gt;
  <meta name="DESCRIPTION" content="FarRail Tours (+49-177-56 13 999) offers tours to the most interesting railway lines of the world. Tours for photographers and video film makers.">
  <meta name="PAGE-TOPIC" content="Steam in China, &lt;st1:place w:st="><st1:city w:st="on">Sandaoling</st1:city>, Fuxin, railway tours"&gt;
  <meta name="PAGE-TYPE" content="Visit to the last regular steam trains in China">
  <meta name="AUTHOR" content="Bernd Seiler">
  <meta name="PUBLISHER" content="FarRail Tours">
  <meta name="COPYRIGHT" content="Bernd Seiler, FarRail Tours">
  <meta name="ROBOTS" content="INDEX,FOLLOW">
  <title>Last Real Steam in China 2017:</title><st1:place w:st="on"><st1:city w:st="on">Sandaoling</st1:city></st1:place> and Fuxin
  <link rel="stylesheet" href="../../formate/standard-1.css">

[ivanov17]

@geoffmcl

No problem. There are fragments of the file which Tidy has problems with:

<html xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:w="urn:schemas-microsoft-com:office:word"
xmlns:st1="urn:schemas-microsoft-com:office:smarttags"
xmlns="http://www.w3.org/TR/REC-html40">

<head>
<meta http-equiv=Content-Type content="text/html; charset=utf-8">
<meta name=ProgId content=Word.Document>
<meta name=Generator content="Microsoft Word 10">
<meta name=Originator content="Microsoft Word 10">
<title>XVII Congress of Association of Anarchist Movements</title>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
 name="country-region"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
 name="City"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
 name="place"/>

Word's XML data and the <style> tag located here didn't cause processing problems. After that the body begins:

</head>

<body lang=RU style='tab-interval:35.4pt'>

<div class=Section1>

<p class=MsoPlainText style='margin-top:36.0pt'><span lang=EN-US
style='mso-ansi-language:EN-US'>Anarchist Information Bureau<o:p></o:p></span></p>

<p class=MsoPlainText><span lang=EN-US style='mso-ansi-language:EN-US'><o:p>&nbsp;</o:p></span></p>

<p class=MsoPlainText><span lang=EN-US style='mso-ansi-language:EN-US'>On
August, 13-15, 2004 XVII Congress of Association of Anarchist Movements took place
in St. Peterburg. The Congress collected 52 comrades, including 19 participants
of Association from 6 cities – </span><st1:City><st1:place><span lang=EN-US
  style='mso-ansi-language:EN-US'>Kazan</span></st1:place></st1:City><span
lang=EN-US style='mso-ansi-language:EN-US'>, </span><st1:City><st1:place><span
  lang=EN-US style='mso-ansi-language:EN-US'>Moscow</span></st1:place></st1:City><span
lang=EN-US style='mso-ansi-language:EN-US'>, Nizhniy Novgorod, St. Peterburg,
Solnechnogorsk and Yaroslavl – and 33 guests from Russia (Volzhskiy, Moscow,
Petrozavodsk, St. Peterburg, Solnechnogorsk, Tyumen), Finland (Helsinki),
Germany (Berlin) and Italy (Kozence, Rome). During the Congress activity of
local anarchist groups (</span><st1:City><st1:place><span lang=EN-US
  style='mso-ansi-language:EN-US'>Moscow</span></st1:place></st1:City><span
lang=EN-US style='mso-ansi-language:EN-US'>, Nizhniy Novgorod, </span><st1:place><span
 lang=EN-US style='mso-ansi-language:EN-US'>Volga</span></st1:place><span
lang=EN-US style='mso-ansi-language:EN-US'>, </span><st1:City><st1:place><span
  lang=EN-US style='mso-ansi-language:EN-US'>Yaroslavl</span></st1:place></st1:City><span
lang=EN-US style='mso-ansi-language:EN-US'>, St. Peterburg, Kazan,
Petrozavodsk) was discussed and a report about the VII Congress of
International of Anarchist Federations (IFA), taken place this spring in
Besanson , France, that had ratified the Association joining IFA. Congress has
established, that the participants of Council of Revolutionary Groups of
Anarchists, (SRGA) </span><st1:City><st1:place><span lang=EN-US
  style='mso-ansi-language:EN-US'>Yaroslavl</span></st1:place></st1:City><span
lang=EN-US style='mso-ansi-language:EN-US'> should take responsibility for
contacts with anarchist federations in other countries and interaction with
Secretary of IFA <o:p></o:p></span></p>

<p class=MsoPlainText><span lang=EN-US style='mso-ansi-language:EN-US'><o:p>&nbsp;</o:p></span></p>

<p class=MsoPlainText><span lang=EN-US style='mso-ansi-language:EN-US'>Congress
discussed the report «About directions in modern anarchism» and a problem of
interactions between the anarchistic organizations of </span><st1:country-region><st1:place><span
  lang=EN-US style='mso-ansi-language:EN-US'>Russia</span></st1:place></st1:country-region><span
lang=EN-US style='mso-ansi-language:EN-US'>. Claims of members of Federation of
Revolutionary Anarchists to Petersburg Anarchist League of and to participants
of «Independent Action» are considered within the companionable discussions.
The information on separate conflict situations in anarchist movement is taken
into account in connection with proclamations of some participants from </span><st1:City><st1:place><span
  lang=EN-US style='mso-ansi-language:EN-US'>Moscow</span></st1:place></st1:City><span
lang=EN-US style='mso-ansi-language:EN-US'>. Having considered the problem with
the attitude to «the Union of Anarchists of Ukraine», Congress has agreed with
accepting new decisions for this occasion being unnecessary, because problems,
arising in this case, are subject of the «Agreement on interaction» of </span><st1:City><st1:place><span
  lang=EN-US style='mso-ansi-language:EN-US'>ADA</span></st1:place></st1:City><span
lang=EN-US style='mso-ansi-language:EN-US'><o:p></o:p></span></p>

And so on, following text has markup similar to previous.

</div>
</body>
</html>

Hope it will help you.

[geoffmcl]

@ivanov17 thank you for the minimal sample... still large, but understand word's output is very VERBOSE, to say the least ;=))

We do have some sample cases - 1067112, 431721, 470663, maybe others - which I have reviewed, 1 x 1, but your sample introduces several new items, or conditions, problems...

1. new tags in the `head`, like `<o:SmartTagType ... name="country-region"/>`, and
2. new tags in the `body`, like `<st1:place>`, etc

Now, in our previous samples, such new head tags are HIDDEN in a comment, like <!--[if gte mso 9]><xml> <o:DocumentProperties> <o:Author>AusAID</o:Author> ... etc, ending with -->, so these are not seen by tidy, in that they are just a text part of a comment, mostly ignored by tidy... even if not told --word-2000 yes...

In your sample, they will be parsed by tidy, and when it finds the tag <o:SmartTagType ..., and finding virtually any tag, ... will CLOSE the head, and open the body...

So my first question is your sample correct? Did you miss this comment around the xml?

If it is, then I do not think tidy can really do otherwise... other than create a big mess... from a big mess... sorry...

For the second item, it may be possible to add, or modify, code to deal with tags like <st1:xxxxxx..., like we already do for the <o:p> tag...

If you do not add --word-2000 yes, the <o:p> will be treated as an error, you must fix...

But when you turn on word parsing, it will just be a warning, like Warning: <o:p> is not approved by W3C...

Maybe the <st1:xxxx tags could likewise be treated differently if word-2000 is on... just an idea, thought... not tested yet...

@ler762, thank you for your feedback, comments...

I found --new-inline-tags st1:city,st1:place,st1:country-region is a possible work around... it seemed to work for me... on your link... Are you suggesting it failed? What do you mean by mangled the file pretty badly?

Can you supply a simple, small, sample? But this seems a separate issue... outside a word-2000 doc... potentially a NEW issue... thanks...

Will try to find time for some more testing, experimentation, etc... and look for any other feedback, on the issue...

But meantime, @ivanov17, look forward to your confirmation that your sample is correct... and you should try the above new-inline-tags config... thanks...

[ivanov17]

@geoffmcl

So my first question is your sample correct? Did you miss this comment around the xml?

Yes, I deleted from my answer the comment around XML, comment around style block for MS Office and style tag, which contains comment around another CSS block. It takes up 80 lines =)

I have 57 files that contains these “smart tags” and <o:SmartTagType: ... /> tags always places inside the head and outside comments. They are related with <st1: ... > tags and indicates which types of “smart tags” there are in the body.

Microsoft wrote that “smart tags” are deprecated in 2010. If they don't use it in their products a long time ago, there are no reasons for saving it in future for tidy output, I think.

Thank you.

[ler762]

On 2/2/20, Geoff McLane wrote: @ler762, thank you for your feedback, comments... I found `--new-inline-tags st1:city,st1:place,st1:country-region` **is** a possible work around... it seemed to work for me... on your link... Are you suggesting it **failed**? What do you mean by `mangled the file pretty badly`?

I mean that when I compared the original file to the tidy'ed output the difference was so large that I wouldn't try to use an editor to repair the tidy'ed output. But I didn't know that

... finding virtually any tag, ... will CLOSE the head, and open the body...

Now that I understand why the <head> section was truncated so badly it doesn't seem quite as bad as I'd originally thought.. maybe it's worth putting a note in the --new-inline-tags doc that if a new tag is found inside the <head> section that section is closed and the <body> section is started??

Can you supply a simple, **small**, sample? But this seems a **separate** issue... outside a word-2000 doc... potentially a **NEW** issue...

Thanks, but the only files I've seen that would need '--new-inline-tags ...' have those tags in the head. It's easier to clean them up with sed than come up with patches, test cases, etc. Regards Lee

[geoffmcl]

@ivanov17 and @ler762 thanks for the feedback, comments...

@ivanov17, deleting the comment <!-- ... --> around the XML, in the <head>, is NOT ok. This is very significant! Have added it back to my test sample...

Unfortunately, if you have 57 files, with <o:SmartTagType: ... />, in the head, with NO covering comment, then these files may do very badly with tidy... and you may end up with an unusable mess... when you force output... hence the tidy ERROR!

@ler762, as stated, in the head, without <!-- ... -->, virtually any tag, valid or invalid, like even <p>para 1</p>, will CLOSE the <head>... and put the para in the visible <body>, as does a browser, in rendering...

Consider the following sample... with a --new-inline-tags st1:city,st1:place,st1:country-region config...

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<!-- <xml><o:SmartTagType ... /> this is ok </xml> -->
<title>Is. #855-2</title>
<p>Para 1 - head</p>
<style>
<!--
.bg {background-color: coral;}
-->
</style>
</head>
<body>
<p class="bg">The dinosaurs ...  
<st1:place w:st="on"><st1:City w:st="on">Fuxin</st1:City></st1:place>
</p>
</body>
</html>

The clues to this are the following warnings -

line 7 column 1 - Warning: inserting implicit <body> # oops, tidy found the p
line 13 column 1 - Warning: discarding unexpected </head> # because it had already closed head
line 14 column 1 - Warning: discarding unexpected <body> # and was already in body
line 8 column 1 - Warning: moved <style> tag to <head>! fix-style-tags: no to avoid.
line 16 column 1 - Warning: <st1:place> is not approved by W3C
line 16 column 22 - Warning: <st1:city> is not approved by W3C

Now the <style> tag is different...

tidy allows, and keeps, in output, the <!-- ... --> block around the style CSS... you have to read back into html history to understand this... was not uncommon, back in the day... is not used, seen so much now...

But in this case, tidy next finds the <style> tag, now in the body!... moves it to the head... see warning... you can avoid this with --fix-style-tags no... but this is not the point here...

So the tidy sample output is valid html -

<!DOCTYPE html>
<html lang="en">
<head>
<meta name="generator" content=
"HTML Tidy for HTML5 for Windows version 5.7.28">
<meta charset="utf-8">
<!-- <xml><o:SmartTagType ... /> this is ok </xml> -->
<title>Is. #855-2</title>

<style>
<!--
.bg {background-color: coral;}
-->
</style>
</head>
<body>
<p>Para 1 - head</p>
<p class="bg">The dinosaurs ... <st1:place w:st=
"on"><st1:city w:st="on">Fuxin</st1:city></st1:place></p>
</body>
</html>

Still to test, experiment, ... with letting <st1:xxxx ...> be a warning, if --word-2000 yes, like we do for <o:p>... maybe possible... not sure...

Any patches, code, comments on that much appreciated... thanks...

[ivanov17]

@geoffmcl

I'm sorry. Maybe my message wasn't clear enough. I deleted comments with xml and styles that they contained, because they were too long and not related to errors I've got. I saved in my answer only tags no placed in comments.

As I wrote, <o:SmartTagType: ... /> tags that contains in my files, always places inside the head and outside comments. So my sample above is correct.

[geoffmcl]

@ivanov17 also maybe my comments are not clear enough... sorry... to try again... simply...

As stated, I see TWO problems in your sample - 1: [new] tags in the head, 2: new tags in the body... by new here, I mean not approved by the W3C...

You should be able to apply a work-around, namely add to config --new-inline-tags st1:city,st1:place,st1:country-region,o:SmartTagType... adding others as needed...

Maybe trying like new-blocklevel-tags, etc... see Teaching Tidy Options...

What do you get if you use this config, adjusted for every new tag present? Does this work? Does this solve your problem?

It seems to work quite well for me, using your above sample... and tidy 5.7.28... and loading your sample in a browser, looks virtually the same as loading the output of tidy... and the error output ends with Tidy found 32 warnings and 0 errors!...

If not, why not? What more do you expect tidy to do? What is the problem you think can be solved by tidy?

And maybe, why are you processing these files with tidy? What is your aim?

Simply stripping out some of MS Word's stuff is probably better done with other tools, like sed, etc... Is the final product to be published? Or is there some other aim?

Sorry, lots of simple question? Maybe with some answers, we can be of more help...

[ivanov17]

@geoffmcl

Thank you.

You should be able to apply a work-around, namely add to config --new-inline-tags st1:city,st1:place,st1:country-region,o:SmartTagType... adding others as needed...

What do you get if you use this config, adjusted for every new tag present? Does this work?

Yes, it works with tidy 5.6.0. It saves tags to the tidy output and shows no errors.

Does this solve your problem?

Not entirely. Using that approach tags are still persisting in a new file and if I want to get the output without MS tags I should remove them by using another tool, e.g. sed.

What more do you expect tidy to do? What is the problem you think can be solved by tidy?

I expect that using an option named word tidy should recognize all tags which MS Word produced and delete all MS-specific tags from output. I understand that this option completely called word-2000, but my files are over 15 years old.

Of course, I'm able to use sed. But if tidy can work with the majority of MS-specific tags, it would better to teach it to recognize a bit more of old tags.

Or maybe for similar cases, it would be useful to have an option like --new-inline-tags which could also delete specified tags from the output, not saving it.

And maybe, why are you processing these files with tidy? What is your aim?

Is the final product to be published? Or is there some other aim?

Yes, I want to publish the final product. I want to get html5-compatible code without MS-specific tags. I work with a lot of files and I use shell script which processing all files in a directory. At first, I had errors using tidy. Now I process files by sed before calling tidy and all works perfectly.

But tidy's developers may be interested in this case, I think. The ability to recognizing “smart tags” and removing them from output without using other tools would be useful.

Sorry, I have one more question. It's not related to this topic, but it's important for me. It seems that tidy removes from output all meta tags. I want to save some of them, e.g 'author' and 'date'. Is it possible to do using tidy? In a nutshell, please.

I can open new topic if necessary. Thank you again.

[geoffmcl]

@ivanov17 thank you for the good feedback...

Yes, it works with tidy 5.6.0. It saves tags to the tidy output and shows no errors.

That's good... and urged me to continue testing, to learn how <o:p> is excepted, when you use --word-2000 yes...

As suggested, maybe, just MAYBE, it would be possible to except other MS smart tags... But there is a sort of problem...

Here is the code, in static void AdjustConfig( doc ), in config.c, Line 1230 ... extract below...

    /* Word 2000 needs o:p to be declared as inline */
    if ( cfgBool(doc, TidyWord2000) )
    {
        doc->config.defined_tags |= tagtype_inline;
        TY_(DefineTag)( doc, tagtype_inline, "o:p" );
    }

So a fix, to now except say o:SmartTagType is as simple as adding TY_(DefineTag)( doc, tagtype_inline, "o:smarttagtype" );, extending the above hard coded list... with more hard code...

Then further extending the hard list with st1:city,st1:place,st1:country-region,......

Is that the complete MS Smart Tags list? Have we got yhem ALL?

Just adding 4/5 code lines does not seems too much of a problem... just for a --word-2000 yes config...

But is it an open list?

what do you think? Is it worth it? After all this time... maybe... 50:50 ... but can be persuaded...

You talk about pre-processing a file with sed... but... instead...

Why not just export HTML_TIDY=/path/word.conf, containing/with 1 line new-inline-tags: st1:city,st1:place,st1:country-region,o:smarttagtype ... and your done... just saying...

Do not have time to reply to the new issues you raise - 1. removing tags, and 2. meta deletions - ... will try to get to those... but as you suggest, maybe new topic is a better way to go, with a simple sample(s), more explanation, etc...

Look forward to feedback... thanks...