Oh yes, I get that now. All this while I was thinking there was an issue
https://github.com/scikit-learn/scikit-learn/issues/5115.
Thanks a lot for clearing this up. I am going to change the loop and see
if I can run the parallel implementation on mac.
It was probably running on server since its has many more processors..
Post by Sebastian RaschkaI am not that much into the multi-processing implementation in
scikit-learn / joblib, but I think this could be one issue why your mac
hangs⊠Iâd say that itâs probably the safest approach to only set the
n_jobs parameter for the innermost object.
E.g., if you 4 processors, you said the GridSearch to 2 and a k-fold loop
to e.g., 5, I can imagine that it would blow up because you are suddenly
trying to run 10 processes on 4 processors if it makes sense!?
Post by Amita MisraI had not thought about the n_jobs parameter, mainly because it does not
run on my mac and the system just hangs if i use it.
Post by Amita MisraThe same code runs on linux server though.
I have one more clarification to seek.
I was running it on server with this code. Would this be fine or may I
move the n_jobs=3 to GridSearchCV
Post by Amita Misragrid_search = GridSearchCV(pipeline,
param_grid=param_grid,scoring=scoringcriteria,cv=5)
Post by Amita Misrascores = cross_validation.cross_val_score(grid_search, X_train,
Y_train,cv=cvfolds,n_jobs=3)
Post by Amita MisraThanks,
Amita
You are welcome, and I am glad to hear that it works :). And âyour"
approach is definitely the cleaner way to do it ⊠I think you just need to
be a bit careful about the n_jobs parameter in practice, I would only set
it to n_jobs=-1 in the inner loop.
Post by Amita MisraBest,
Sebastian
Post by Amita MisraThanks.
Actually there were 2 people running the same experiments and the
other person was doing as you have shown above.
Post by Amita MisraPost by Amita MisraWe were getting the same results but since methods were different I
wanted to ensure that I am doing it the right way.
Post by Amita MisraPost by Amita MisraThanks,
Amita
On Thu, May 12, 2016 at 2:43 PM, Sebastian Raschka <
I see; thatâs what I thought. At first glance, the approach (code)
looks correct to me but I havenâ t done it this way, yet. Typically, I use
a more âmanualâ approach iterating over the outer folds manually (since I
Post by Amita MisraPost by Amita Misrags_est = ⊠your gridsearch, pipeline, estimator with param grid and
cv=5
Post by Amita MisraPost by Amita Misraskfold = StratifiedKFold(y=y_train, n_folds=5, shuffle=True,
random_state=123)
Post by Amita MisraPost by Amita Misrags_est.fit(X_train[outer_train_idx], y_train[outer_train_idx])
y_pred = gs_est.predict(X_train[outer_valid_idx])
acc = accuracy_score(y_true=y_train[outer_valid_idx],
y_pred=y_pred)
Post by Amita MisraPost by Amita Misraprint(' | inner ACC %.2f%% | outer ACC %.2f%%' %
(gs_est.best_score_ * 100, acc * 100))
Post by Amita MisraPost by Amita Misracv_scores[name].append(acc)
However, it should essentially do the same thing as your code if I see
it correctly.
Post by Amita MisraPost by Amita MisraPost by Amita MisraActually I do not have an independent test set and hence I want to
use it as an estimate for generalization performance. Hence my classifier
is fixed SVM and I want to learn the parameters and also estimate an
unbiased performance using only one set of data.
Post by Amita MisraPost by Amita MisraPost by Amita MisraI wanted to ensure that my code correctly does a nested 10*5 CV and
the parameters are learnt on a different set and final evaluation to get
the predicted score is on a different set.
Post by Amita MisraPost by Amita MisraPost by Amita MisraAmita
On Thu, May 12, 2016 at 1:24 PM, Sebastian Raschka <
I would say there are 2 different applications of nested CV. You
could use it for algorithm selection (with hyperparam tuning in the inner
loop). Or, you could use it as an estimate of the generalization
performance (only hyperparam tuning), which has been reported to be less
biased than the a k-fold CV estimate (Varma, S., & Simon, R. (2006). Bias
in error estimation when using cross-validation for model selection. BMC
Bioinformatics, 7, 91. http://doi.org/10.1186/1471-2105-7-91)
Post by Amita MisraPost by Amita MisraPost by Amita MisraBy "you could use it as an estimate of the generalization
performance (only hyperparam tuning)â I mean as a replacement for k-fold on
the training set and evaluation on an independent test set.
Post by Amita MisraPost by Amita MisraPost by Amita MisraPost by ÐлекÑей ÐÑалÑHi Amita,
As far as I understand your question, you only need one CV loop to
===
pipeline=Pipeline([('scale',
preprocessing.StandardScaler()),('filter',
SelectKBest(f_regression)),('svr', svm.SVR())]
gamma_range,'svr__C': C_range}]
scoring=scoring_function)
http://scikit-learn.org/stable/modules/grid_search.html#grid-search
http://scikit-learn.org/stable/modules/grid_search.html#gridsearch-scoring
and also evaluate the final model.
way to do it. I have the code but still I want to be sure that I am not
overfitting any way.
preprocessing.StandardScaler()),('filter',
SelectKBest(f_regression)),('svr', svm.SVR())]
gamma_range,'svr__C': C_range}]
Y_pred=cross_validation.cross_val_predict(grid_search, X_train,
Y_train,cv=10)
Post by Amita MisraPost by Amita MisraPost by Amita MisraPost by ÐлекÑей ÐÑалÑcorrelation= numpy.ma.corrcoef(Y_train,Y_pred)[0, 1]
please let me know if my understanding is correct.
This is 10*5 nested cross validation. Inner folds CV over training
data involves a grid search over hyperparameters and outer folds evaluate
the performance.
Post by Amita MisraPost by Amita MisraPost by Amita MisraPost by ÐлекÑей ÐÑалÑThanks,
Amita--
Amita Misra
Graduate Student Researcher
Natural Language and Dialogue Systems Lab
Baskin School of Engineering
University of California Santa Cruz
------------------------------------------------------------------------------
who
of MDM
only the
untouched!
Post by Amita MisraPost by Amita MisraPost by Amita MisraPost by ÐлекÑей ÐÑалÑhttps://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
--
Yours sincerely,
Alexey A. Dral
------------------------------------------------------------------------------
who
of MDM
only the
untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j_______________________________________________
Post by Amita MisraPost by Amita MisraPost by Amita MisraPost by ÐлекÑей ÐÑалÑScikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
Post by Amita MisraPost by Amita MisraPost by Amita MisraMobile security can be enabling, not merely restricting. Employees
who
Post by Amita MisraPost by Amita MisraPost by Amita Misrabring their own devices (BYOD) to work are irked by the imposition
of MDM
Post by Amita MisraPost by Amita MisraPost by Amita Misrarestrictions. Mobile Device Manager Plus allows you to control only
the
Post by Amita MisraPost by Amita MisraPost by Amita Misraapps on BYO-devices by containerizing them, leaving personal data
untouched!
Post by Amita MisraPost by Amita MisraPost by Amita Misrahttps://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
--
Amita Misra
Graduate Student Researcher
Natural Language and Dialogue Systems Lab
Baskin School of Engineering
University of California Santa Cruz
------------------------------------------------------------------------------
Post by Amita MisraPost by Amita MisraPost by Amita MisraMobile security can be enabling, not merely restricting. Employees
who
Post by Amita MisraPost by Amita MisraPost by Amita Misrabring their own devices (BYOD) to work are irked by the imposition
of MDM
Post by Amita MisraPost by Amita MisraPost by Amita Misrarestrictions. Mobile Device Manager Plus allows you to control only
the
Post by Amita MisraPost by Amita MisraPost by Amita Misraapps on BYO-devices by containerizing them, leaving personal data
untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j_______________________________________________
Post by Amita MisraPost by Amita MisraPost by Amita MisraScikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
Post by Amita MisraPost by Amita MisraMobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of
MDM
Post by Amita MisraPost by Amita Misrarestrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data
untouched!
Post by Amita MisraPost by Amita Misrahttps://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
--
Amita Misra
Graduate Student Researcher
Natural Language and Dialogue Systems Lab
Baskin School of Engineering
University of California Santa Cruz
------------------------------------------------------------------------------
Post by Amita MisraPost by Amita MisraMobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of
MDM
Post by Amita MisraPost by Amita Misrarestrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data
untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j_______________________________________________
Post by Amita MisraPost by Amita MisraScikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
Post by Amita MisraMobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data
untouched!
Post by Amita Misrahttps://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
--
Amita Misra
Graduate Student Researcher
Natural Language and Dialogue Systems Lab
Baskin School of Engineering
University of California Santa Cruz
------------------------------------------------------------------------------
Post by Amita MisraMobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data
untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j_______________________________________________
Post by Amita MisraScikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general