[Scikit-learn-general] Problem with parallel processing in randomSearch

Discussion:

Stylianos Kampakis

2016-02-23 19:15:04 UTC

Hi everyone,

Sometimes, when I am using random search with n_jobs>1 the processing
stops. I am on a Mac. I went through some discussions on Github where
people said it relates joblib and this problem is more common on Mac.
However, I couldn't find the answer to two questions I have:

1) Why the processing stops only some times and not every single time?

2) Have any people managed to find a workaround?

Thank you all in advance,

Joel Nothman

2016-02-24 00:38:35 UTC

Permalink

What estimator(s) are you searching over? How big is your data?

On 24 February 2016 at 06:15, Stylianos Kampakis <

Post by Stylianos Kampakis
Hi everyone,
Sometimes, when I am using random search with n_jobs>1 the processing
stops. I am on a Mac. I went through some discussions on Github where
people said it relates joblib and this problem is more common on Mac.
1) Why the processing stops only some times and not every single time?
2) Have any people managed to find a workaround?
Thank you all in advance,
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

Stylianos Kampakis

2016-02-24 16:42:08 UTC

Permalink

Hi Joel,

I ran lots of tests today and I realized that it is XGBRegressor (a wrapper
for XGboost) that is guilty. However, it doesn't always stop. It does only
some times. Any ideas why this might be happening?

Post by Joel Nothman
What estimator(s) are you searching over? How big is your data?
On 24 February 2016 at 06:15, Stylianos Kampakis <

Sebastian Raschka

2016-02-24 18:51:54 UTC

Permalink

I am not using XGBoost and honestly don't much about the way its implemented. However, have you checked if memory swapping could be an issue if you are doing multiprocessing? Maybe it's using so much RAM due to creating multiple copies that it just feels like it stopped? Just a thought :P

Post by Stylianos Kampakis
Hi Joel,
I ran lots of tests today and I realized that it is XGBRegressor (a wrapper for XGboost) that is guilty. However, it doesn't always stop. It does only some times. Any ideas why this might be happening?
What estimator(s) are you searching over? How big is your data?
Hi everyone,
1) Why the processing stops only some times and not every single time?
2) Have any people managed to find a workaround?
Thank you all in advance,
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 <http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140>
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general <https://lists.sourceforge.net/lists/listinfo/scikit-learn-general>
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 <http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140>
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general <https://lists.sourceforge.net/lists/listinfo/scikit-learn-general>
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

Stylianos Kampakis

2016-02-24 20:27:25 UTC

Permalink

Could be, but I don't think so in this particular case. The dataset is like
a few thousand rows and around 20 features.

Post by Sebastian Raschka
I am not using XGBoost and honestly don't much about the way its
implemented. However, have you checked if memory swapping could be an issue
if you are doing multiprocessing? Maybe it's using so much RAM due to
creating multiple copies that it just feels like it stopped? Just a thought
:P
On Feb 24, 2016, at 11:42 AM, Stylianos Kampakis <
Hi Joel,
I ran lots of tests today and I realized that it is XGBRegressor (a
wrapper for XGboost) that is guilty. However, it doesn't always stop. It
does only some times. Any ideas why this might be happening?

Post by Joel Nothman
What estimator(s) are you searching over? How big is your data?
On 24 February 2016 at 06:15, Stylianos Kampakis <

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general