Suggestions for Streamlining .janno Files

[Tlkhi]

I think it would be better for the .janno files to be as simple as the .anno file used in the AADR.
Like this:

Genetic ID
Master ID
Year data from this individual was first published
Publication abbreviation
doi for publication of this representation of the data
Link to the most permanent repository hosting these data
Method for Determining Date
Date mean in BP in years before 1950 CE
Date standard deviation in BP
Full Date One of two formats. (Format 1) 95.4% CI calibrated radiocarbon age. (Format 2) Archaeological context range, e.g. 2500-1700 BCE
Age at death, Morphological sex from physical anthropology
Group ID
Locality
Political Entity
Lat.
Long.
Pulldown Strategy
Suffices (indicating data types used for sources which can be a subset of that in bam)
Data type
No. Libraries
SNPs hit on autosomal targets (Computed using easystats on 1240k snpset)
Molecular Sex
Sum total of ROH segments >4 cM
Sum total of ROH segments >20cM
Y haplogroup
mtDNA coverage (merged data)
mtDNA haplogroup if >2x or published
mtDNA match to consensus if >2x (merged data)
Damage rate in first nucleotide on sequences overlapping 1240k targets (merged data)
Sex ratio [Y/(Y+X) counts] (merged data)
ANGSD MOM 95% CI truncated at 0 (only if male and >=200 SNPs)
hapConX 95% CI truncated at 0 (only if male and >=2000 SNPs covered on X chromosome)
Library type (minus=no.damage.correction, half=damage.retained.at.last.position, plus=damage.fully.corrected, ds=double.stranded.library.preparation, ss=single.stranded.library.preparation)
Libraries
endogenous by library (computed on shotgun data)
ASSESSMENT
ASSESSMENT WARNING

[nevrome]

Please note the .janno column specification here: https://github.com/poseidon-framework/poseidon-schema/blob/master/janno_columns.tsv

Many of the columns you mention from the AADR have a direct equivalent in our schema. Some don't because of the different structure and purpose of the datasets. Some could have, but we have not yet come around to specify them. The AADR does not maintain a rigorous specification of its columns that we could simply copy.

Could you please clarify which columns exactly you would like to add/modify to/in the .janno file specification?

[Tlkhi]

Could you please clarify which columns exactly you would like to add/modify to/in the .janno file specification?

You might want to add and/or modify some columns, *(It would be ideal if the columns follow the same order as AADR):

ASSESSMENT

Sum total of ROH segments >4 cM

Sum total of ROH segments >20cM

Full Date One of two formats. (Format 1) 95.4% CI calibrated radiocarbon age (Conventional Radiocarbon Age BP, Lab number) e.g. 2624-2350 calBCE (3990±40 BP, Ua-35016). (Format 2) Archaeological context range, e.g. 2500-1700 BCE

Method for Determining Date

Year data from this individual was first published

Pulldown Strategy

Suffices (indicating data types used for sources which can be a subset of that in bam)

[nevrome]

Ah - OK - thanks. That is more concrete 👍
Why do you think a change of the column order would be a sensible decision? I'm a bit confused about what that would achieve? Note that the AADR is a moving target itself, where every release can come with new/different columns.

• ASSESSMENT: To my knowledge this is based on the subjective judgement of the AADR team. But scientists apply different criteria to decide if a sample should be added to a given analysis or not. Maybe there would be a way to automatically derive a data-quality-based recommendation with the minotaur workflow. Feel free to make a suggestion. If we add a column to the schema, then it must be well specified.
• Sum total of ROH segments >4 cM, Sum total of ROH segments >20cM: This may be something we could include, yes.
• Full Date One of two formats: We have a way to encode age information, that is imho cleaner and more easy to parse/use than the AADR encoding. What specifically are you missing?
• Method for Determining Date: see Date_Note.
• Year data from this individual was first published: That is indeed useful. But I think it would be even more useful to pair this with the information where exactly an individual was first published. Maybe a second Publication column specifically for this purpose would be the way to go.
• Pulldown Strategy: What exactly does this column mean? I'm admitting my ignorance here. The most common entry in that column in v62 is Native Pulldown on 3.2M snpset. That sounds like some internal, intermediate data processing step for the AADR. How would that generalise to a universal data format like the Poseidon .janno file?
• Suffices: Which information encoded in suffixes is not already available in other .janno columns?

[Tlkhi]

Ah - OK - thanks. That is more concrete 👍 Why do you think a change of the column order would be a sensible decision? I'm a bit confused about what that would achieve? Note that the AADR is a moving target itself, where every release can come with new/different columns.

I think if the columns in the .janno file are in the same order as those in the .anno file, it will make it easier for users (including myself and others) to access and understand the data when working with both .anno and .janno files. While the AADR might introduce minor changes to the .anno file over time, these changes are unlikely to be extensive.

• ASSESSMENT: To my knowledge this is based on the subjective judgement of the AADR team. But scientists apply different criteria to decide if a sample should be added to a given analysis or not. Maybe there would be a way to automatically derive a data-quality-based recommendation with the minotaur workflow. Feel free to make a suggestion. If we add a column to the schema, then it must be well specified.

This is used to indicate whether a sample is contaminated or not (I hope I’m not completely mistaken). Although there are already columns related to contamination, it might be better to rename or group them under a heading like Assessment to make it easier to understand.

• Full Date One of two formats: We have a way to encode age information, that is imho cleaner and more easy to parse/use than the AADR encoding. What specifically are you missing?

In my opinion, the "Full Date" column in the anno offers a simpler and more straightforward approach to representing dates. This single column could effectively replace multiple related columns in the .janno file. For example:

• For samples dated based on archaeological context:
  3000-2000 BC
• For samples that are directly dated:
  470-630 calCE

• Pulldown Strategy: What exactly does this column mean? I'm admitting my ignorance here. The most common entry in that column in v62 is Native Pulldown on 3.2M snpset. That sounds like some internal, intermediate data processing step for the AADR. How would that generalise to a universal data format like the Poseidon .janno file?

This refers to the method used to enrich and sequence specific regions of the genome. For example, the most common entry in AADR v62 is "Native Pulldown on 3.2M snpset" which is a targeted enrichment approach focusing on a predefined set of SNPs. Could add a column like Genotyping_Method to capture this information.

• Suffices: Which information encoded in suffixes is not already available in other .janno columns?

While I know that Capture_Type is already included in the .janno file, a column for Suffices (e.g., SG for Shotgun sequencing, AG for Agilent (1240K), etc.) could also be added. Including a dedicated column for Suffices would streamline the identification of the sequencing method used for each sample.

[nevrome]

I think if the columns in the .janno file are in the same order as those in the .anno file, it will make it easier for users (including myself and others) to access and understand the data when working with both .anno and .janno files. While the AADR might introduce minor changes to the .anno file over time, these changes are unlikely to be extensive.

Well - fair enough. I don't think Poseidon needs to immitade the AADR, but I guess a similar column order could be convenient in some situations. I'm not sure if this is worth a breaking change, though.

This is used to indicate whether a sample is contaminated or not (I hope I’m not completely mistaken). Although there are already columns related to contamination, it might be better to rename or group them under a heading like Assessment to make it easier to understand.

The AADR v62 version of the .anno file has a ASSESSMENT WARNING column that seems to justify some of the choices for ASSESSMENT. I see entries like damage.ds.half=0.026, sexratio=0.03, Chalcolithic.outlier, mtcontam=[0.851,0.887], manual inspection of mtDNA finds only 5 discordant alleles out of 264 (1.9% contamination), ANGSD=[0.062,0.121], this.would.normally.be.called.highly.contaminated.but.ANGSD.may.have.malfunctioned.because.of.the.unique.largely.Khoesan.related.ancestry, damage=0.034-0.319, mtmatchmax=0.814-0.838, hapConX=[0.136,0.158] or date doesn't match European ancestry signature. They point to a more involved, manual judgement, that is difficult to formalize or even automise. But I'm open to suggestions. Would be really cool to have a reliable automatic process.

In my opinion, the "Full Date" column in the anno offers a simpler and more straightforward approach to representing dates. This single column could effectively replace multiple related columns in the .janno file.

In principle I agree - such a little DSL to encode age information is neat, and maybe I would argue for that if we would design Poseidon from scratch. I love writing little parsers 😄 . But consider the consequences: every user who wants to work with the data has to parse this mini-language. A simple column-, or list-column-based structure as in the .janno file is far more accessible for quantitative work. And Poseidon is not just a human-readable data storage system, but also machine-readable data analysis tool. Also every user would have to learn this DSL to encode information! The AADR has many consistency issues in this column, I assume because users struggle to add the data consistently in the right form. Again a column-based system is more reliable here.

This refers to the method used to enrich and sequence specific regions of the genome. For example, the most common entry in AADR v62 is "Native Pulldown on 3.2M snpset" which is a targeted enrichment approach focusing on a predefined set of SNPs. Could add a column like Genotyping_Method to capture this information.

But we're talking about computational processing here, right? I know enrichment only in the context of capture techniques. Could you point me to a paper where this targeted enrichment using a 3.2M Snp set is explained?

While I know that Capture_Type is already included in the .janno file, a column for Suffices (e.g., SG for Shotgun sequencing, AG for Agilent (1240K), etc.) could also be added. Including a dedicated column for Suffices would streamline the identification of the sequencing method used for each sample.

Well - suffices are not a purpose on their own, right? Suffices encode information that should also be properly stored elsewhere. I would understand if you would like Poseidon to embrace suffices in the Poseidon_ID. But I'm not in favour of that. These suffices become hidden semantics for new users. I think it's better to store information explicitly in specific columns. And there I'm open to additional columns, if something is not properly covered yet 👍

[nevrome]

I should perhaps point out that other members of the Poseidon core team may be more in favor of suffixes in Poseidon_IDs or group names. It's specifically me who prefers these identifiers concise and free of any additional baggage.

[Tlkhi]

Well - fair enough. I don't think Poseidon needs to immitade the AADR, but I guess a similar column order could be convenient in some situations. I'm not sure if this is worth a breaking change, though.

I agree that Poseidon doesn’t need to imitate the AADR exactly, but aligning the column order where possible could improve usability, especially for users who work with both '.janno' & 'anno'.

The AADR v62 version of the .anno file has a ASSESSMENT WARNING column that seems to justify some of the choices for ASSESSMENT. I see entries like damage.ds.half=0.026, sexratio=0.03, Chalcolithic.outlier, mtcontam=[0.851,0.887], manual inspection of mtDNA finds only 5 discordant alleles out of 264 (1.9% contamination), ANGSD=[0.062,0.121], this.would.normally.be.called.highly.contaminated.but.ANGSD.may.have.malfunctioned.because.of.the.unique.largely.Khoesan.related.ancestry, damage=0.034-0.319, mtmatchmax=0.814-0.838, hapConX=[0.136,0.158] or date doesn't match European ancestry signature. They point to a more involved, manual judgement, that is difficult to formalize or even automise. But I'm open to suggestions. Would be really cool to have a reliable automatic process.

You’re right. However, having a standardized Assessment column (or something similar) in the '.janno' could still be useful, even if it’s just a high-level summary (e.g., 'not contam,' 'possible contam,' 'contam,' and 'outlier').

In principle I agree - such a little DSL to encode age information is neat, and maybe I would argue for that if we would design Poseidon from scratch. I love writing little parsers 😄 . But consider the consequences: every user who wants to work with the data has to parse this mini-language. A simple column-, or list-column-based structure as in the .janno file is far more accessible for quantitative work. And Poseidon is not just a human-readable data storage system, but also machine-readable data analysis tool. Also every user would have to learn this DSL to encode information! The AADR has many consistency issues in this column, I assume because users struggle to add the data consistently in the right form. Again a column-based system is more reliable here.

I see your point about the challenges of using a DSL for date encoding, especially for machine readability and consistency. Perhaps a compromise could be to include both a 'Full Date' column (for human readability) and separate columns for 'Date_Mean', 'Date_Start', 'Date_End', and 'Date_Type' (for machine readability and analysis). This way, users who prefer the simplicity of the AADR format can still use the 'Full Date' column, while others can rely on the structured columns for quantitative work.

But we're talking about computational processing here, right? I know enrichment only in the context of capture techniques. Could you point me to a paper where this targeted enrichment using a 3.2M Snp set is explained?

The 'Pulldown Strategy' refers to computational processing for targeted enrichment. In the AADR, the term 'Native Pulldown on 3.2M snpset' is used to describe a method where a predefined set of 3.2 million SNPs are targeted for enrichment and sequencing. For more details on this approach, you can refer to 'Rohland et al. 2022 (doi:10.1101/gr.276728.122)'.
Adding a 'Genotyping_Method' column to the '.janno' could be very useful for explicitly capturing this information, making it easier for users to understand how the data was generated and ensuring compatibility across datasets.

Well - suffices are not a purpose on their own, right? Suffices encode information that should also be properly stored elsewhere. I would understand if you would like Poseidon to embrace suffices in the Poseidon_ID. But I'm not in favour of that. These suffices become hidden semantics for new users. I think it's better to store information explicitly in specific columns. And there I'm open to additional columns, if something is not properly covered yet 👍

I understand your preference for keeping Poseidon_ID concise and free of additional information. Since there’s already a 'Capture_Type' column, if the team prefers, suffixes could be appended to the end of each sample ID (e.g., ABC001.SG or ABC002.AG) in the .fam files.

[stschiff]

Hi, thanks also from my side for these suggestions, @Tlkhi. Some quick (and selective) comments:

• ROH -> Yes, interesting, but infeasable to supply that for all existing packages in the Community-Archive (as the community archive is in principle and author-first archive, we don't perform analyses usually). For Minotaur this is a good idea, but it's currently not part of our Minotaur-Pipeline. Would require @TCLamnidis comments.

• Assessment: Yes, we've thought about this before, too. Again, this might useful for Minotaur, but for the Commuity-Archive we're unlikely to make such an assessment. We don't want to be the authority to decide this, I think.

• Date column: This is more like a "computed view", as I think of it, as it is essentially a condensed human-readable string computed from various date-source columns. What we can consider is for our websites to show dates in such a concise way, that is then "computed" on the fly from the source columns. That sounds quite useful for me. I am skeptical to add this as an explicit column, as it would require a more complicated validation system that would have to keep the source columns and this "view" column in sync. The redundancy hurts is in the long-run, I think.

• Pulldown strategy: We already have Capture_Type and (in the YAML file) snpSet. Aren't these two already covering all needs?

Now, for columns that make sense for Minotaur (Assessment and ROH), my approach would be to first implement them in Minotaur, and then add these columns to the Minotaur archive without a schema update. Then, if they turn out to be useful, we can incorporate in them in the schema. The nice thing about our schema and the tooling is that it explicitly allows for arbitrary additional columns.

[Tlkhi]

Hi, @stschiff

Yes, my suggestions are specifically for Minotaur, not for the Community-Archive. I understand that applying changes to all datasets in the Community-Archive would be challenging. Additionally, many of those datasets are already included in the AADR. As for Minotaur, the focus is often on new and/or non-existing datasets that are not included in the AADR. And you’re right about the pulldown strategy—Capture_Type and snpSet already cover the necessary requirements. I was being a bit picky.

[TCLamnidis]

Late to the party, but wanted to add my two cents.

Minotaur is limited to just reprocessing the raw data into genotypes, by design. I think adding downstream processing directly within it would become cumbersome to maintain effectively. Any additional analyses (past sex determination, which is packaged within nf-core/eager already), would ideally be undertaken downstream. But I am open to including such columns within the PMA, if they are calculated in a reproducible and documented manner.

I am working on a downstream pipeline that would run some standard analyses, and I could see hapROH being one of them. The results could then be filled into the janno, and hence populate the archive.