Discussion:
OOB score in gradient boosting models
(too old to reply)
Yanir Seroussi
2013-03-22 10:39:42 UTC
Permalink
Hi,

I'm new to the mailing list, so I apologise if this has been asked before.

I want to use the oob_score_ in GradientBoostingRegressor to determine the
optimal number of iterations without relying on an external validation set,
so I set the subsample parameter to 0.5 and trained the model. However,
I've noticed that oob_score_ improves in a similar manner to the in-bag
scores (train_score_). That is, it goes down very fast, and keeps improving
regardless of the number of iterations.

Digging through the code in ensemble/gradient_boosting.py, it seems like
the cause is that oob_score_[i] includes previous trees that were trained
on the OOB instances of the i-th sample. Isn't the OOB score supposed to be
calculated for each OOB instance using only trees that where this instance
wasn't used for training (as done for random forests)?

Cheers,
Yanir
Andreas Mueller
2013-03-22 10:55:21 UTC
Permalink
Hi Yanir.
I was not aware that GradientBoosting had oob scores.
Is that even possible / sensible? It definitely does not do what it
promises :-/

Peter, any thoughts?

Cheers,
Andy
Post by Yanir Seroussi
Hi,
I'm new to the mailing list, so I apologise if this has been asked before.
I want to use the oob_score_ in GradientBoostingRegressor to determine
the optimal number of iterations without relying on an external
validation set, so I set the subsample parameter to 0.5 and trained
the model. However, I've noticed that oob_score_ improves in a similar
manner to the in-bag scores (train_score_). That is, it goes down very
fast, and keeps improving regardless of the number of iterations.
Digging through the code in ensemble/gradient_boosting.py, it seems
like the cause is that oob_score_[i] includes previous trees that were
trained on the OOB instances of the i-th sample. Isn't the OOB score
supposed to be calculated for each OOB instance using only trees that
where this instance wasn't used for training (as done for random forests)?
Cheers,
Yanir
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
Peter Prettenhofer
2013-03-22 11:21:56 UTC
Permalink
Hi Yanir,

thanks for raising this issue.
I've implemented this feature without much though; furthermore, I
haven't used OOB estimates in my work yet.
I need to think more deeply about the issue - will come back to you.

You propose to update ``y_pred`` only for the in-bag samples, correct?

best,
Peter
Post by Andreas Mueller
Hi Yanir.
I was not aware that GradientBoosting had oob scores.
Is that even possible / sensible? It definitely does not do what it promises
:-/
Peter, any thoughts?
Cheers,
Andy
Hi,
I'm new to the mailing list, so I apologise if this has been asked before.
I want to use the oob_score_ in GradientBoostingRegressor to determine the
optimal number of iterations without relying on an external validation set,
so I set the subsample parameter to 0.5 and trained the model. However, I've
noticed that oob_score_ improves in a similar manner to the in-bag scores
(train_score_). That is, it goes down very fast, and keeps improving
regardless of the number of iterations.
Digging through the code in ensemble/gradient_boosting.py, it seems like the
cause is that oob_score_[i] includes previous trees that were trained on the
OOB instances of the i-th sample. Isn't the OOB score supposed to be
calculated for each OOB instance using only trees that where this instance
wasn't used for training (as done for random forests)?
Cheers,
Yanir
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
--
Peter Prettenhofer
Andreas Mueller
2013-03-22 12:14:57 UTC
Permalink
We should open an issue in the issue tracker.
Peter Prettenhofer
2013-03-22 12:27:34 UTC
Permalink
I've opened an issue for this:
https://github.com/scikit-learn/scikit-learn/issues/1802
Post by Andreas Mueller
We should open an issue in the issue tracker.
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
--
Peter Prettenhofer
Yanir Seroussi
2013-03-22 12:38:16 UTC
Permalink
Thanks for the quick response. Good to see that I'm not imagining things :-)

Before posting this question, I had a look at Friedman's paper and ESLII
and the R gbm documentation, but I couldn't find a clear description of how
OOB estimates are calculated. I think it makes sense to have a separate
y_oob_pred.
I'll probably try fixing it locally over the weekend (unless you beat me to
it). I'll let you know how it goes.

Cheers,
Yanir
Post by Peter Prettenhofer
https://github.com/scikit-learn/scikit-learn/issues/1802
Post by Andreas Mueller
We should open an issue in the issue tracker.
------------------------------------------------------------------------------
Post by Andreas Mueller
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
--
Peter Prettenhofer
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
Peter Prettenhofer
2013-03-22 12:42:54 UTC
Permalink
Post by Yanir Seroussi
Thanks for the quick response. Good to see that I'm not imagining things :-)
Before posting this question, I had a look at Friedman's paper and ESLII and
the R gbm documentation, but I couldn't find a clear description of how OOB
estimates are calculated. I think it makes sense to have a separate
y_oob_pred.
I'll probably try fixing it locally over the weekend (unless you beat me to
it). I'll let you know how it goes.
If you manage to fix it, a PR would be much appreciated!
Please keep me posted about your progress.

thanks,
peter
Post by Yanir Seroussi
Cheers,
Yanir
Post by Peter Prettenhofer
https://github.com/scikit-learn/scikit-learn/issues/1802
Post by Andreas Mueller
We should open an issue in the issue tracker.
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
--
Peter Prettenhofer
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
--
Peter Prettenhofer
Continue reading on narkive:
Loading...