Material Ingestor – GitHub #1154
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| uri = URI(url) | ||
| return nil unless uri.host =~ /github\.com|github\.io/i | ||
|
|
||
| if uri.host.end_with?('github.io') |
Check failure
Code scanning / CodeQL
Incomplete URL substring sanitization High
|
|
||
| if uri.host.end_with?('github.io') | ||
| github_api_from_io(uri) | ||
| elsif uri.host.end_with?('github.com') |
Check failure
Code scanning / CodeQL
Incomplete URL substring sanitization High
| # frozen_string_literal: true | ||
|
|
||
| module Ingestors | ||
| module Concerns |
There was a problem hiding this comment.
I've created 'Concerns' for the sake of linting with Rubocop and passing the ABC metrics
This way we do not have a +200 lines in GithubIngestor Class
| # return doi if doi | ||
|
|
||
| # doi = fetch_doi_from_file(full_name, 'CITATION.md') | ||
| # return doi if doi |
There was a problem hiding this comment.
I left these in case someone wants to use these two other files to fetch DOI – most of the time if a DOI is provided, it is in README.md through a badge.
| category: :materials, | ||
| user_agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0' | ||
| } | ||
| end |
There was a problem hiding this comment.
Not really sure about those, I copied this one from bioschemas_ingestor.rb
|
|
||
| def resolve_url(repo_data) | ||
| homepage_nil_or_empty = repo_data['homepage'].nil? || repo_data['homepage'].empty? | ||
| url = homepage_nil_or_empty ? repo_data['html_url'] : get_redirected_url(repo_data['homepage']) |
There was a problem hiding this comment.
get_redirected_url is a method I added in the Ingestor Class to be able to follow links (through a 30X response or a meta[http-equiv="Refresh"] tag) as the open_url method was not sufficient to follow all redirections.
I stumbled upon a jupyter notebook which was automatically redirecting to another path that the github.io page (e.g., instead of visitng me.github.io/myrepo/ it directly redirected to me.github.io/myrepo/introduction/)
There was a problem hiding this comment.
This is the github page : https://se-for-sci.github.io/
It automatically redirects to : https://se-for-sci.github.io/content/intro.html
Summary of changes
TTL) before re-updating the material. Here is the list of the API calls (n = 4 per Github repo)Motivation and context
Checklist
to license it to the TeSS codebase under the
BSD license.