Discussion:
GSoC 2015: Global optimization based Hyper parameter optimization (SMAC)
(too old to reply)
hamzeh alsalhi
2015-03-27 15:49:46 UTC
Permalink
I have revised my proposal to focus only on SMAC and to prioritize SMAC RF
because it can be worked on independently GP.

Thank you for meeting with me in person Vlad, and for giving me feedback on
ways to improve my proposal.

https://github.com/scikit-learn/scikit-learn/wiki/GSoC-2015-Proposal:-Global-optimization-based-Hyper-parameter-optimization-(SMAC-RF-and-GP)-Hamzeh-Alsalhi
--
Hamzeh Alsalhi
Computer Science
Cornell University 2015
T: (850) 319.5607 | E: ***@cornell.edu
Gael Varoquaux
2015-03-28 17:29:34 UTC
Permalink
Sorry for the slow reply,
Post by hamzeh alsalhi
I have revised my proposal to focus only on SMAC and to prioritize SMAC RF
because it can be worked on independently GP. 
I actually believed that GP were an easier route forward.

The way I would have tackled such a project would have been:

1. Improve GPs
2. In parallel, built a Bayesian optimization example (example not
module)
3. Use the work above to make a hyper-parameter selection object.

In addition to giving us 3, 1 and 2 would be good to have.

Cheers,

Gaël
Andreas Mueller
2015-03-31 15:29:41 UTC
Permalink
To do the Bayesian optimization, I think it needs more than an example.
Step two is really non-trivial.
Why do you think the GP route is easier?
Post by Gael Varoquaux
Sorry for the slow reply,
Post by hamzeh alsalhi
I have revised my proposal to focus only on SMAC and to prioritize SMAC RF
because it can be worked on independently GP.
I actually believed that GP were an easier route forward.
1. Improve GPs
2. In parallel, built a Bayesian optimization example (example not
module)
3. Use the work above to make a hyper-parameter selection object.
In addition to giving us 3, 1 and 2 would be good to have.
Cheers,
Gaël
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
Gael Varoquaux
2015-03-31 18:01:36 UTC
Permalink
Post by Andreas Mueller
To do the Bayesian optimization, I think it needs more than an example.
Maybe I am naive, but I thought that it wasn't that bad.
Post by Andreas Mueller
Why do you think the GP route is easier?
Because we already have GPs. Also, I am worried that the description in
the SMAC paper is quite vague, and that we will have to dig in referenced
papers and code to understand what is really being done. Maybe that
simply reflects my lack of understanding of the field :).

Gaël
Post by Andreas Mueller
Post by Gael Varoquaux
Sorry for the slow reply,
Post by hamzeh alsalhi
I have revised my proposal to focus only on SMAC and to prioritize SMAC RF
because it can be worked on independently GP.
I actually believed that GP were an easier route forward.
1. Improve GPs
2. In parallel, built a Bayesian optimization example (example not
module)
3. Use the work above to make a hyper-parameter selection object.
In addition to giving us 3, 1 and 2 would be good to have.
Cheers,
Gaël
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
--
Gael Varoquaux
Researcher, INRIA Parietal
Laboratoire de Neuro-Imagerie Assistee par Ordinateur
NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France
Phone: ++ 33-1-69-08-79-68
http://gael-varoquaux.info http://twitter.com/GaelVaroquaux
Vlad Niculae
2015-03-31 19:05:39 UTC
Permalink
Hi Gael,
Post by Gael Varoquaux
Post by Andreas Mueller
Why do you think the GP route is easier?
Because we already have GPs.
Well, we already have random forests too.

Both cases would need quite a bit of machinery on top, and I don’t know the extent of it, but I thought it would be interesting to have a chance to explore both alleys, since Cristof’s proposal seems to cover Spearmint’s approach thoroughly. By reading [1], it also seemed like SMAC can be a good fit regarding discrete parameters and cross-validation support (though supporting this might just be an implementation detail vs. a limitation of the algorithm, I am not sure.)

[1] Towards an empirical foundation for assessing bayesian optimization of hyperparameters https://www.cs.ubc.ca/~hoos/Publ/EggEtAl13.pdf
Post by Gael Varoquaux
Also, I am worried that the description in
the SMAC paper is quite vague, and that we will have to dig in referenced
papers and code to understand what is really being done. Maybe that
simply reflects my lack of understanding of the field :).
Gaël
Post by Andreas Mueller
Post by Gael Varoquaux
Sorry for the slow reply,
Post by hamzeh alsalhi
I have revised my proposal to focus only on SMAC and to prioritize SMAC RF
because it can be worked on independently GP.
I actually believed that GP were an easier route forward.
1. Improve GPs
2. In parallel, built a Bayesian optimization example (example not
module)
3. Use the work above to make a hyper-parameter selection object.
In addition to giving us 3, 1 and 2 would be good to have.
Cheers,
Gaël
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
--
Gael Varoquaux
Researcher, INRIA Parietal
Laboratoire de Neuro-Imagerie Assistee par Ordinateur
NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France
Phone: ++ 33-1-69-08-79-68
http://gael-varoquaux.info http://twitter.com/GaelVaroquaux
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
Mathieu Blondel
2015-03-31 23:30:21 UTC
Permalink
Post by Vlad Niculae
Hi Gael,
Post by Gael Varoquaux
Post by Andreas Mueller
Why do you think the GP route is easier?
Because we already have GPs.
We have a GP implementation but it's being rewritten...
Post by Vlad Niculae
Well, we already have random forests too.
SMAC needs the variance of predictions so we'll need to get this PR merged
https://github.com/scikit-learn/scikit-learn/pull/3645

I would really like #3645 to get merged in any case.
Post by Vlad Niculae
Both cases would need quite a bit of machinery on top, and I don’t know
the extent of it, but I thought it would be interesting to have a chance to
explore both alleys, since Cristof’s proposal seems to cover Spearmint’s
approach thoroughly. By reading [1], it also seemed like SMAC can be a
good fit regarding discrete parameters and cross-validation support (though
supporting this might just be an implementation detail vs. a limitation of
the algorithm, I am not sure.)
In order to support discrete parameters, our tree implementation would need
to support categorical variables though.

I personally like the simplicity of SMAC. It could potentially be better
than GPs for highly non-smooth hyper-parameter surfaces.

Mathieu
Post by Vlad Niculae
[1] Towards an empirical foundation for assessing bayesian optimization of
hyperparameters https://www.cs.ubc.ca/~hoos/Publ/EggEtAl13.pdf
Post by Gael Varoquaux
Also, I am worried that the description in
the SMAC paper is quite vague, and that we will have to dig in referenced
papers and code to understand what is really being done. Maybe that
simply reflects my lack of understanding of the field :).
Gaël
Post by Andreas Mueller
Post by Gael Varoquaux
Sorry for the slow reply,
Post by hamzeh alsalhi
I have revised my proposal to focus only on SMAC and to prioritize
SMAC RF
Post by Gael Varoquaux
Post by Andreas Mueller
Post by Gael Varoquaux
Post by hamzeh alsalhi
because it can be worked on independently GP.
I actually believed that GP were an easier route forward.
1. Improve GPs
2. In parallel, built a Bayesian optimization example (example not
module)
3. Use the work above to make a hyper-parameter selection object.
In addition to giving us 3, 1 and 2 would be good to have.
Cheers,
Gaël
------------------------------------------------------------------------------
Post by Gael Varoquaux
Post by Andreas Mueller
Post by Gael Varoquaux
Dive into the World of Parallel Programming The Go Parallel Website,
sponsored
Post by Gael Varoquaux
Post by Andreas Mueller
Post by Gael Varoquaux
by Intel and developed in partnership with Slashdot Media, is your hub
for all
Post by Gael Varoquaux
Post by Andreas Mueller
Post by Gael Varoquaux
things parallel software development, from weekly thought leadership
blogs to
Post by Gael Varoquaux
Post by Andreas Mueller
Post by Gael Varoquaux
news, videos, case studies, tutorials and more. Take a look and join
the
Post by Gael Varoquaux
Post by Andreas Mueller
Post by Gael Varoquaux
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
Post by Gael Varoquaux
Post by Andreas Mueller
Dive into the World of Parallel Programming The Go Parallel Website,
sponsored
Post by Gael Varoquaux
Post by Andreas Mueller
by Intel and developed in partnership with Slashdot Media, is your hub
for all
Post by Gael Varoquaux
Post by Andreas Mueller
things parallel software development, from weekly thought leadership
blogs to
Post by Gael Varoquaux
Post by Andreas Mueller
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
--
Gael Varoquaux
Researcher, INRIA Parietal
Laboratoire de Neuro-Imagerie Assistee par Ordinateur
NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France
Phone: ++ 33-1-69-08-79-68
http://gael-varoquaux.info
http://twitter.com/GaelVaroquaux
------------------------------------------------------------------------------
Post by Gael Varoquaux
Dive into the World of Parallel Programming The Go Parallel Website,
sponsored
Post by Gael Varoquaux
by Intel and developed in partnership with Slashdot Media, is your hub
for all
Post by Gael Varoquaux
things parallel software development, from weekly thought leadership
blogs to
Post by Gael Varoquaux
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
Andreas Mueller
2015-03-31 23:43:43 UTC
Permalink
Post by Mathieu Blondel
SMAC needs the variance of predictions so we'll need to get this PR merged
https://github.com/scikit-learn/scikit-learn/pull/3645
I would really like #3645 to get merged in any case.
I have not even seen that one! Shame on me!
It looks pretty cool indeed.
Vlad Niculae
2015-04-01 01:01:08 UTC
Permalink
In order to support discrete parameters, our tree implementation would need to support categorical variables though.
Ah, good point, I didn’t think about that. But we could use the usual hacks (integer or one-hot encoding). I wonder how that compares to using GPs and rounding when it comes to acquisition.
[1] Towards an empirical foundation for assessing bayesian optimization of hyperparameters https://www.cs.ubc.ca/~hoos/Publ/EggEtAl13.pdf
Post by Gael Varoquaux
Also, I am worried that the description in
the SMAC paper is quite vague, and that we will have to dig in referenced
papers and code to understand what is really being done. Maybe that
simply reflects my lack of understanding of the field :).
Gaël
Post by Gael Varoquaux
Post by Gael Varoquaux
Sorry for the slow reply,
Post by hamzeh alsalhi
I have revised my proposal to focus only on SMAC and to prioritize SMAC RF
because it can be worked on independently GP.
I actually believed that GP were an easier route forward.
1. Improve GPs
2. In parallel, built a Bayesian optimization example (example not
module)
3. Use the work above to make a hyper-parameter selection object.
In addition to giving us 3, 1 and 2 would be good to have.
Cheers,
Gaël
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
--
Gael Varoquaux
Researcher, INRIA Parietal
Laboratoire de Neuro-Imagerie Assistee par Ordinateur
NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France
Phone: ++ 33-1-69-08-79-68
http://gael-varoquaux.info http://twitter.com/GaelVaroquaux
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
Loading...