Race Conditions in Parallel Builds Corrupt Artifacts

[Ekam219]

Hi @DonggeLiu, What's your view on this?

Problem Summary

A race condition in the build pipeline leads to corrupted build artifacts when multiple threads or processes concurrently access shared resources. This occurs because the current implementation lacks synchronization mechanisms for critical sections of the build process.

Affected Components

• File I/O Operations: Concurrent writes to shared files (e.g., build.sh, target.o, coverage reports).
• OSS-Fuzz Configuration: Simultaneous modifications to project configurations.
• Artifact Generation: Race conditions during binary compilation and coverage data aggregation.

Consequences

• Corrupted coverage reports skew fuzzing effectiveness analysis.
• Prevents deployment of distributed fuzzing (requires stable builds).
• Flaky builds.

[Ekam219]

I have created draft pr for now

[DonggeLiu]

Hi @Ekam219, thanks for noticing this.
Would you mind listing all the places (lines/functions) you found that conduct concurrent writes? This allows us to minimize changes needed for the solutions by reuse existing code.

For example, we have a very simple solution to address concurrent write to fuzz target: Each trial will create and use their own replica of the project directory, e.g., oss-fuzz/projects/<project_a-benchmark-trial>. I reckon reusing this approach is simpler and more reliable than semaphore:

1. It has lower complexity and reuses existing code
2. All trials starts with a clean copy of the project. Trials will not be affected by the new content written to Dockerfile/build.sh by other trials.

However, there may be places we did not use this approach properly. If we can identify all concurrent writes, we can think of a universal solution for all cases.

[Ekam219]

Hi @DonggeLiu,

Thank you for the guidance! Here's a consolidated solution leveraging the project isolation pattern you suggested, extended to address all concurrency risks:

---

Concurrency Fixes via Trial Isolation

1. Project Directories

• Problem: create_ossfuzz_project() uses shared projects/<name>
• Fix:

  # evaluator.py  
  def create_ossfuzz_project(self, trial_id: int):  
      unique_name = f"{self.benchmark.id}-trial{trial_id}"  
      generated_path = f"oss-fuzz/projects/{unique_name}"  
      shutil.copytree(original_project, generated_path)  # Fresh copy per trial  

2. Docker Images

• Problem: Shared gcr.io/oss-fuzz/{project} tags
• Fix:

  # builder_runner.py  
  image_name = f"gcr.io/oss-fuzz/{generated_project}-trial{trial_id}"  
  command = ['docker', 'build', '-t', image_name, ...]  

3. Build Artifacts

• Problem: Shared build/out/<project>
• Fix:

  # oss_fuzz_checkout.py  
  def get_build_artifact_dir(..., trial_id: int):  
      return f"build/{build_artifact}/{generated_project}-trial{trial_id}"  

4. Corpus/Log Paths

• Problem: Shared corpora/{sample_id}, logs/build/...
• Fix:

  # workdir.py  
  def corpus(self, sample_id, trial_id):  
      return f"corpora/trial{trial_id}-{sample_id}"  
  
  def build_logs_target(self, ..., trial_id: int):  
      return f"logs/build/trial{trial_id}-{generated_target_name}-F{iteration}.log"  

---

Thanks again for the review! Let me know your thoughts :)

Best regards,
EkamJot

[DonggeLiu]

Hi @Ekam219, thanks for listing them.
I will comment my opinion on each, please let me know if you think otherwise.

Concurrency Fixes via Trial Isolation

1. Project Directories

• Problem: create_ossfuzz_project() uses shared projects/<name>

I don't think this alone will cause race condition.
The function copies the original project directory into a new unique dir.
Reading from the original dir is fine because we should never modify the original project dir.
Writing to the new dir name is also fine if it's truly unique.

• Fix:

  evaluator.py

  def create_ossfuzz_project(self, trial_id: int):
  unique_name = f"{self.benchmark.id}-trial{trial_id}"
  generated_path = f"oss-fuzz/projects/{unique_name}"
  shutil.copytree(original_project, generated_path) # Fresh copy per trial

I reckon this solution is the same as this code block?

2. Docker Images

• Problem: Shared gcr.io/oss-fuzz/{project} tags
• Fix:

  builder_runner.py

  image_name = f"gcr.io/oss-fuzz/{generated_project}-trial{trial_id}"
  command = ['docker', 'build', '-t', image_name, ...]

IIRC, this has been done at here and here.

3. Build Artifacts

• Problem: Shared build/out/<project>

• Fix:

  oss_fuzz_checkout.py

  def get_build_artifact_dir(..., trial_id: int):
  return f"build/{build_artifact}/{generated_project}-trial{trial_id}"
  4. Corpus/Log Paths

• Problem: Shared corpora/{sample_id}, logs/build/...

• Fix:

  workdir.py

  def corpus(self, sample_id, trial_id):
  return f"corpora/trial{trial_id}-{sample_id}"

  def build_logs_target(self, ..., trial_id: int):
  return f"logs/build/trial{trial_id}-{generated_target_name}-F{iteration}.log"

Similar to the ones above, I reckon this is already supported:

oss-fuzz-gen/experiment/builder_runner.py

Lines 611 to 612 in 358aff9

	outdir = get_build_artifact_dir(generated_project, 'out')
	workdir = get_build_artifact_dir(generated_project, 'work')