[Resources] To index an not to index

[jpradocueva]

Resources that crawlers should not be allowed to index:

• https://www.openmobilealliance.org/Technical/?
• https://www.openmobilealliance.org/release_program/?
• https://oma-knowledge-based.openmobilealliance.org/?
• https://temp.openmobilealliance.org/?

Resources to expose to crawlers so they can be indexed:

• http://member.openmobilealliance.org/FTP/Public_documents/
• https://www.openmobilealliance.org/WorkProgram/WIDs/
• https://www.openmobilealliance.org/events/?
• https://www.openmobilealliance.org/release/?
• https://www.openmobilealliance.org/tech/?
• https://www.openmobilealliance.org/test_events/?

[rubystream]

From the list Resources that crawlers should not be allowed to index the following ones cannot be resolved with robots.txt but needs to be addressed by the web server rooter rules:

• https://oma-knowledge-based.openmobilealliance.org/?
• https://temp.openmobilealliance.org/?

From the list Resources to expose to crawlers so they can be indexed the following required users authorization:

• http://www.member.openmobilealliance.org/FTP/Public_documents/?
• http://www.openmobilealliance.org/WorkProgram
• https://www.openmobilealliance.org/WorkProgram/?

The above ones cannot be resolved using sitemap.xml or robots.txt and should be excluded from this issue.

[jpradocueva]

@rubystream, I will take Nathan to the links that the sitemap cannot resolve.

Also, how can we maintain these sitemaps after updating the pages or adding new ones?

[rubystream]

As mentioned in the PR #410

Used XML-Sitemaps.com to create additional sitemaps.
Afterwards add the node so the new generated ones share same CSS.

In most of the cases the generation of the additional sitemap.xml file has taken couple of minutes.

[Evadon-Nathan]

For: https://oma-knowledge-based.openmobilealliance.org
I will see if I can set up the reverse proxy again so the traffic comes to our server first. In which case I can serve a separate robot.txt

https://temp.openmobilealliance.org/ doesn't exist any more so you can ignore that one

For these URLs, they are wrong and I have opened another issue for that (#424)
http://www.member.openmobilealliance.org/FTP/Public_documents/?
http://www.openmobilealliance.org/WorkProgram
https://www.openmobilealliance.org/WorkProgram/?

[jpradocueva]

@rubystream The original list was updated with the correct URLs

[rubystream]

The #427 should address the URLs provided in #424 so this issue should be closed.