Discussion:
feature selection
(too old to reply)
Pagliari, Roberto
2014-10-20 19:20:07 UTC
Permalink
I'm not sure if I correctly understood the feature selection algorithms.

Basically, accuracy, or any other scoring function is used to determine whether to keep a specific feature or not? If so, how is the optimal subset of features determined? Bruteforce would be exponential in complexity.

Thanks,
George Bezerra
2014-10-20 20:08:37 UTC
Permalink
Not an expert, but I think the idea is that you remove (or add) features
one by one, starting from the ones that have the least (or most) impact.

E.g., try removing a feature, if performance improves, keep it that way and
move on to the next feature. It's a greedy approach; not optimal, but
avoids exponential complexity.

George.
I’m not sure if I correctly understood the feature selection algorithms.
Basically, accuracy, or any other scoring function is used to determine
whether to keep a specific feature or not? If so, how is the optimal subset
of features determined? Bruteforce would be exponential in complexity.
Thanks,
------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
--
George Bezerra
Lars Buitinck
2014-10-20 20:29:19 UTC
Permalink
Not an expert, but I think the idea is that you remove (or add) features one
by one, starting from the ones that have the least (or most) impact.
E.g., try removing a feature, if performance improves, keep it that way and
move on to the next feature. It's a greedy approach; not optimal, but avoids
exponential complexity.
No. That would be backward stepwise selection. Neither that, nor its
forward cousin (find most discriminative feature, then second-most,
etc.) are implemented in scikit-learn.

The feature selection in sklearn.feature_selection computes a score
per feature (in practice always a significance test, but the API is
set up so that other scores are possible), then keeps the k best
features, the p% best, or the ones that don't exceed some
threshold/p-value.
Andy
2014-11-02 21:09:08 UTC
Permalink
Post by Lars Buitinck
Not an expert, but I think the idea is that you remove (or add) features one
by one, starting from the ones that have the least (or most) impact.
E.g., try removing a feature, if performance improves, keep it that way and
move on to the next feature. It's a greedy approach; not optimal, but avoids
exponential complexity.
No. That would be backward stepwise selection. Neither that, nor its
forward cousin (find most discriminative feature, then second-most,
etc.) are implemented in scikit-learn.
Isn't RFE the backward step selection using a maximum number of features?

------------------------------------------------------------------------------
Lars Buitinck
2014-11-02 21:15:03 UTC
Permalink
Post by Andy
Post by Lars Buitinck
No. That would be backward stepwise selection. Neither that, nor its
forward cousin (find most discriminative feature, then second-most,
etc.) are implemented in scikit-learn.
Isn't RFE the backward step selection using a maximum number of features?
Yes, and I already admitted that mistake, but I'm willing to do so
again if you like :)
Maybe we should add this to the docstring.

------------------------------------------------------------------------------
Andy
2014-11-02 21:19:27 UTC
Permalink
Post by Lars Buitinck
Post by Andy
Post by Lars Buitinck
No. That would be backward stepwise selection. Neither that, nor its
forward cousin (find most discriminative feature, then second-most,
etc.) are implemented in scikit-learn.
Isn't RFE the backward step selection using a maximum number of features?
Yes, and I already admitted that mistake, but I'm willing to do so
again if you like :)
Maybe we should add this to the docstring.
Yeah I just read the rest of the thread. Sorry for the noise.
Adding it to the docstring is probably a good idea for searchability,
and the doc page
http://scikit-learn.org/dev/modules/feature_selection.html#recursive-feature-elimination
should probably also contain the word "backward selection"

------------------------------------------------------------------------------
Manoj Kumar
2014-10-20 20:35:54 UTC
Permalink
Hi,

No expert here, either but there are also feature selection classes which
compute the score per feature.

A simple example would be the f_classif, which in a very broad way measures
how a certain feature varies across all the classes to how a feature varies
in a particular class (a naive explanation).
Intuitively if this ratio is lower for a feature as compared to the other
features it means that the feature is noisy (it varies a lot in a
particular class itself) and can be excluded.
You can have a look here for more details http://goo.gl/jsmEqd

HTH
Manoj
Joel Nothman
2014-10-21 02:14:16 UTC
Permalink
I assume Robert's query is about RFECV.
Post by Manoj Kumar
Hi,
No expert here, either but there are also feature selection classes which
compute the score per feature.
A simple example would be the f_classif, which in a very broad way
measures how a certain feature varies across all the classes to how a
feature varies in a particular class (a naive explanation).
Intuitively if this ratio is lower for a feature as compared to the other
features it means that the feature is noisy (it varies a lot in a
particular class itself) and can be excluded.
You can have a look here for more details http://goo.gl/jsmEqd
HTH
Manoj
------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
Joel Nothman
2014-10-21 02:14:26 UTC
Permalink
*Roberto
Post by Joel Nothman
I assume Robert's query is about RFECV.
Post by Manoj Kumar
Hi,
No expert here, either but there are also feature selection classes which
compute the score per feature.
A simple example would be the f_classif, which in a very broad way
measures how a certain feature varies across all the classes to how a
feature varies in a particular class (a naive explanation).
Intuitively if this ratio is lower for a feature as compared to the other
features it means that the feature is noisy (it varies a lot in a
particular class itself) and can be excluded.
You can have a look here for more details http://goo.gl/jsmEqd
HTH
Manoj
------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
Lars Buitinck
2014-10-21 07:27:21 UTC
Permalink
Post by Joel Nothman
I assume Robert's query is about RFECV.
Oh wait, RFE = backward subset selection. I'm an idiot, sorry.
Dayvid Victor
2014-10-21 09:49:57 UTC
Permalink
There are feature selection algorithms based on Evolutionary Algorithms,
so, despite the exponential space of search, you can fix a number of
evaluations.

Experimentally, this approach have found optimal solutions on
Instace/Feature/Classifier selection, without exploring the whole search
space.

via Mobile.

Dayvid V. R. Oliveira
PhD candidate in Computer Science - UFPE
MSc in Computer Science - UFPE
Computer Engineer - UFPE
Post by Lars Buitinck
Post by Joel Nothman
I assume Robert's query is about RFECV.
Oh wait, RFE = backward subset selection. I'm an idiot, sorry.
------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
Continue reading on narkive:
Loading...