Discussion:
[Scikit-learn-general] cross_val_score crashes python every time
c TAKES
2014-10-03 23:34:47 UTC
Permalink
I'm trying to use cross_val_score inside a lambda function to take full
advantage of my processors - especially because previously I was having
problems when setting cross_val_score's built in n_jobs > 1. I have
successfully done something similar before, though in a little bit
different way, so I'm surprised I'm having trouble with it.

But I've troubleshooted enough at this point that I'm convinced either
there's something weird happening inside cross_val_score that I don't know
about (I noticed sklearn has a lengthy parallelism handling file) or
there's some nuance about my current approach that I'm simply overlooking.

Below is a shortened, concise version of the code causing problems.
Attached is a slightly longer and better commented (but still shortened)
version of the code. If I set workers to 1, the code works just fine.
Once I go greater than 1, it crashes the python kernel every time once it
hits cross_val_score.

I'm running this on Windows 7 with Anaconda and Python 2.7:
'2.7.6 |Anaconda 1.9.2 (64-bit)| (default, Nov 11 2013, 10:49:15) [MSC
v.1500 64 bit (AMD64)]'

Anyone know what the deal is? Here is the code in question:




class Job(object): # struct for holding job info
pass
workers = 2
jobs_q = Queue()
results_q = Queue()

# pseudo code for loop here
job = Job()
job.estimator = MultinomialNB()
job.x_csrmat = csr_matrix(x_csrmat, copy=True)
job.y_a = np.array(y_a)
job.kfold = KFold(len(y_a), nfolds, random_state=6, shuffle=True)
jobs_q.put(job)
# end pseudo code for loop

def executeWorker():
while True:
job = jobs_q.get()
if job is None:
break
# try/except never catches error, and code never proceeds beyond it
try:
cvScore = mean(cross_validation.cross_val_score(job.estimator,
job.x_csrmat, y=job.y_a, cv=job.kfold))
except Exception as e:
with open(tsOutDir + 'error.txt', 'w') as f:
f.write(traceback.format_exception(*sys.exc_info()) + "\n\nI/O
error({0}): {1}".format(e.errno, e.strerror))
raise
results_q.put(results_df)
workers_list = [Thread(target=executeWorker) for _ in xrange(workers)]
for thread in workers_list:
thread.daemon = True
thread.start()
for _ in xrange(workers):
# jobs_q already full. Add None-s so workers know to stop
jobs_q.put(None)
for thread in workers_list:
thread.join()
Olivier Grisel
2014-10-06 08:53:58 UTC
Permalink
There might be a problem with running multiprocessing (that is used
internally by cross_val_score with n_jobs=-1) in concurrent Python
threads.

BTW, why do you use threads in the first place?
--
Olivier
Gael Varoquaux
2014-10-06 13:39:43 UTC
Permalink
Post by c TAKES
I'm trying to use cross_val_score inside a lambda function to take full
advantage of my processors -
Try not using a lambda function, but a fully-feldged function. Parallel
computing has problems with lambda functions.

Gaël
c TAKES
2014-10-07 19:01:03 UTC
Permalink
Gael - I tried separating out the lambda function into a standalone
function. Unfortunately no luck - same result :(.

Olivier - I'm using the default, which is n_jobs=1, so hopefully that
shouldn't be a problem.

The reason I'm using the threads is that I want to be able to train models
using a variety of different parameters that aren't actually parameters of
the classifier/regressor. These parameters involve feature construction
and things like that. I want to be able to train separate models testing
each of these parameters while making as much use of my 8 processors as I
can.

Previously when I was trying to use cross_val_score with n_jobs > 1, it
would always hang. Since then I think it has started working - I have no
clue why it's working now but wasn't before. I may simply go back to
trying that approach, though with 4 cv folds, I still won't be able to use
all 8 processors.

On Mon, Oct 6, 2014 at 8:39 AM, Gael Varoquaux <
Post by Gael Varoquaux
Post by c TAKES
I'm trying to use cross_val_score inside a lambda function to take full
advantage of my processors -
Try not using a lambda function, but a fully-feldged function. Parallel
computing has problems with lambda functions.
Gaël
------------------------------------------------------------------------------
Slashdot TV. Videos for Nerds. Stuff that Matters.
http://pubads.g.doubleclick.net/gampad/clk?id=160591471&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
Olivier Grisel
2014-10-08 13:26:22 UTC
Permalink
Generic python cannot use all cores efficiently when using threads
because of the Global Interpreter Lock [1]. This is why people use
multiprocessing instead of threading.

multiprocessing can be quite complex to use properly though. I would
advise you to use the joblib wrapper instead. Please see the
documentation here:

https://pythonhosted.org/joblib/parallel.html

That will require you to reorganize your code a bit though.

[1] http://en.wikipedia.org/wiki/Global_Interpreter_Lock
--
Olivier
Loading...