How to reduce the startup time of model

[qizzhao]

Hi,
Thank you for taking your time and take a look for my question.

My dataset is kinda large with 20 features and 1000000+ point. And I have set that parallelism=:multiprocessing and numprocs=20 and batch=True and batch_size=10000 and turbo=True. But it takes forever to wait before the symbolic regression actually starts. The script stuck at :
Info: Copying definition of mse_loss to workers...
Info: Finished!

How could I faster the startup with large dataset.

Thank you

[MilesCranmer]

Can you downsample the dataset? Normally symbolic regression doesn’t need so much data to converge

[qizzhao]

Dear Miles,
Thank you for your reply. The reason why I am using large dataset is for improving the expression's generalization and avoiding overfitting on train set. What is the ideal number of data point you suggest ?

[MilesCranmer]

It depends on the problem but symbolic regression is much less flexible, so you need less data than for other ML algorithms. Basically, it’s harder to overfit!

[qizzhao]

Thanks. I am going to try on smaller dataset about 500,000 points and close the discussion.