[io] don't miss writing a histogram that is only in a last file with option -n 2

[ferdymercury]

This Pull request:

Changes or fixes:

Fixes #9022

Explores solution suggested by @jblomer

fyi @will-cern

Checklist:

• tested changes locally
• updated the docs (if necessary)

[github-actions]

Test Results

    19 files      19 suites   3d 17h 10m 42s ⏱️
 2 745 tests  2 745 ✅ 0 💤 0 ❌
50 721 runs  50 721 ✅ 0 💤 0 ❌

Results for commit f312f3b.

♻️ This comment has been updated with latest results.

[silverweed]

Thank you for tackling this long-standing issue!
I left a small comment.

[silverweed]

LGTM but I'd wait for @pcanal 's approval as well

[pcanal]

Why is it && and not || ?

[pcanal]

Or more exactly, I don't understand yet why the fact the histogram can be found means that it is not written in the end ...

[ferdymercury]

Check out:

https://github.com/ferdymercury/root/blob/1531153ea11a7b54f4eb3c170bbd28e9bc46447f/io/io/src/TFileMerger.cxx#L808-L817

so if canBeFound is true, there is an optimization that spare some write cycles.

We use && to avoid it being true, ie to force writing to file. Using || would go in the different direction.

Do you want me to rename canBeFound to skipPartialWriting ?

[pcanal]

Do you want me to rename canBeFound to skipPartialWriting ?

Not yet. I am still confused.

The original canBeFound meant (in the context of incremental merge) 'histogram can be found in the source' while the new version is histogram can be found in the source and in the target.

The optimization is indeed 'skip partial writing if we can found the histogram again'.

So I don't understand (yet) the semantic of the change. ie. Why is the new criteria the right choice? Is the new criteria instead 'just' making canBeFound always false?

Another avenue of inquiry is 'the original code assume that if canBeFound is true then there will be another change to write the histogram. Why is it no true anymore (it does not seem to be realted to 'can not be found in target')? Is there other variation of the example that also fails (how does the -n X value relates to the number of files in the input list and how many are 'missing' the histograms).

Related: is it possible that the alternative if that at the refresh boundary there needs to be a flush/write as if we were at the end?

[ferdymercury]

Thanks for the review. I am not sure about these questions; I just followed jblomer's suggestion. My (limited) understanding is that this change just forces an extra partial write the first time that a new histogram appear in any file. So it does not really harm, but is suboptimal since, if all files have exactly the same histograms, then we could have waited until the end. But it makes it work if there are some files with and some without, independently of the chosen N.

[pcanal]

Yes but when going through the first file, it would not be (yet) in both the input and output, so it would be (spuriously) written, or am I still missing something?

[ferdymercury]

Hmm well, this is the result I get in debug mode:

Info in TFileMerger::MergeOne: Writing partial result of h1 into target
Info in TFileMerger::MergeOne: Writing partial result of h3 into target
Info in TFileMerger::MergeOne: Writing partial result of h2 into target

So... are you referring to the second line Writing partial result of h3 into target when you say spurious write?

[pcanal]

I think so. Since h3 is in every file, shouldn't it not need any partial write?

[ferdymercury]

Behaviour prior to this PR:

Info in <TFileMerger::MergeOne>: Writing partial result of h1 into target
Info in <TFileMerger::MergeOne>: Writing partial result of h3 into target

This happens because for the first file, (type & kIncremental) is false.

Behavior after this PR:

Info in <TFileMerger::MergeOne>: Writing partial result of h1 into target
Info in <TFileMerger::MergeOne>: Writing partial result of h3 into target
Info in <TFileMerger::MergeOne>: Writing partial result of h2 into target

So, this PR is only improving things, right?

If I understand you correctly, on top of this improvement, you would like to additionally see h3 not partially written at all:

Info in <TFileMerger::MergeOne>: Writing partial result of h1 into target
Info in <TFileMerger::MergeOne>: Writing partial result of h2 into target

to be even more performant, but what you suggest would require to traverse all the files in advance to know which key should be not partially written at all if it's in all files. I think that would be less efficient than one extra partial write in the first occurrence of a key throughout the list of files; right?

[ferdymercury]

Ok, let's do a more elaborate example:

auto filename0 = "f0_9022.root";
auto filename1 = "f1_9022.root";
auto filename2 = "f2_9022.root";
auto outname = "file9022mergeroutput.root";
TFile f0(filename0, "RECREATE");
TH1F h0("h0", "h0", 1, 0, 1);
h0.Write();
f0.Close();
TFile f1(filename1, "RECREATE");
TH1F h("h1", "h1", 1, 0, 1);
h.Write();
TH1F h3("h3", "h3", 1, 0, 1);
h3.Write();
f1.Close();
TFile f2(filename2, "RECREATE");
TH1F h2("h2", "h2", 1, 0, 1);
h2.Write();
h3.Write();
f2.Close();
TFileMerger filemerger{false, false};
filemerger.SetMaxOpenedFiles(2);
filemerger.OutputFile(std::unique_ptr<TFile>{TFile::Open(outname, "RECREATE")});
filemerger.AddFile(filename0);
filemerger.AddFile(filename1);
filemerger.AddFile(filename2);
gDebug = 1;
filemerger.Merge();

So before the PR we get:

Info in <TFileMerger::MergeOne>: Writing partial result of h0 into target

(because for first type, type&incremental is false).

And the output file only contains histogram h0.

After the PR:

Info in <TFileMerger::MergeOne>: Writing partial result of h0 into target
Info in <TFileMerger::MergeOne>: Writing partial result of h1 into target
Info in <TFileMerger::MergeOne>: Writing partial result of h3 into target
Info in <TH1Merger::ExamineHistogram>: Examine histogram h3 - labels 1 - same limits 1 - axis found 1
Info in <TH1Merger::ExamineHistogram>: Examine histogram h3 - labels 1 - same limits 1 - axis found 1
Info in <TH1Merger::LabelMerge>: Merging histogram h3 into h3
Info in <TFileMerger::MergeOne>: Writing partial result of h2 into target

And the output file contains all histograms.

So the question is: why do you think it is a problem to have a first write whenever it appears? It's the only way to make it work so that the next appearance can call Merge. It's true that with a higher value of SetMaxOpenedFiles and with all files containing exactly the same hists, it's less optimum with this change, but still, this version is safer I guess?...