Skip to content

Conversation

senbaikang
Copy link

In the prediction step, some outputs in parquet format are saved locally in a non-blocking way using dask's persist function. Afterwards these files are accessed, but there's no guarantee that they have been fully saved on disk. I added a simple fix to it, though there might be a better way.

@EliHei2
Copy link
Owner

EliHei2 commented Jan 13, 2025

Thanks for the contribution @senbaikang! I'm reviewing this, this is similar to the idea of segger.prediction.predict_multigpu. It looks fine but I have to test it thoroughly. will do soon!

@senbaikang
Copy link
Author

Thank you for reviewing it! For the original code I encounter an error of inexistent output files for the prediction step, hence the small fix. Hope it helps and feel free to improve!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants