StratifiedKFold will keep the class distribution the same for you:
http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.StratifiedKFold.html#sklearn.cross_validation.StratifiedKFold
I was looking at this, it is written:
This cross-validation object is a variation of KFold, which returns stratified folds. The folds are made by preserving the percentage of samples for each class.
But I don't know how he could manage since that I pass to him just the training set and I don't also how to set this percentage for each class. Do I miss something?
I have done one simple test code (see below) where I have my two dataset (class A and class B). I add a cicle for where I select the 20% for each as test and the other 80% as training. I concatenate the train and the test. I scale my training and my test. I found the best C and gamma for my RBF SVM. Train the SVM and use on my test set. The results are in a list with the score values. I think that I'm doing something wrong cause I get my score always 0.5 (in this case I tried always with range(3)).
I think that I will take a look on the metrics that you wrote to me! thanks for that!! Do you think that the StratifiedKFold is better than train_test_split ? Could you see some conceptual mistake on the code below?
#TEST
X_noscaled_A = X_noscaled_A[0:100,:]y_A = y_A[0:100,:]X_noscaled_B = X_noscaled_B[0:100,:]y_B = y_B[0:100,:]
#Define a list for the resultsscores = list()
for i in range(3): #Split keeping the ratio X_train_noscal_A, X_test_noscal_A, y_train_A, y_test_A = train_test_split(X_noscaled_A, y_A, test_size = 0.20) X_train_noscal_B, X_test_noscal_B, y_train_B, y_test_B = train_test_split(X_noscaled_B, y_B, test_size = 0.20)
#Concatenate in order to have just one vector for train and one vector for test X_train_noscal = numpy.concatenate((X_train_noscal_A, X_train_noscal_B)) y_train = numpy.concatenate((y_train_A,y_train_B)) X_test_noscal = numpy.concatenate((X_test_noscal_A,X_test_noscal_B)) y_test = numpy.concatenate((y_test_A,y_test_B))
#Scale the training set scaler = preprocessing.StandardScaler().fit(X_train_noscal) X_train = scaler.transform(X_train_noscal)
#Scale the test set using the values obtained from the test set X_test = scaler.transform(X_test_noscal)
#Optimization of C and gamma C_range = 10.0 ** numpy.arange(-3, 7) gamma_range = 10.0 ** numpy.arange(-5, 3) param_grid = dict(gamma=gamma_range, C=C_range) svr = svm.SVC() clfopt = grid_search.GridSearchCV(svr,param_grid) clfopt.fit(X_train, y_train)
print clfopt.best_estimator_.C print clfopt.best_estimator_.gamma
#Define a SVM using the best parameters C and gamma clf = svm.SVC(gamma = clfopt.best_estimator_.gamma, C = clfopt.best_estimator_.C) clf.fit(X_train, y_train)
#Write the result in the list scores.append(clf.score(X_test,y_test))
#See the resultsprint scores
From: ***@gmail.com
Date: Fri, 21 Jun 2013 10:57:58 -0400
To: scikit-learn-***@lists.sourceforge.net
Subject: Re: [Scikit-learn-general] SVM: select the training set randomly
StratifiedKFold will keep the class distribution the same for you:
http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.StratifiedKFold.html#sklearn.cross_validation.StratifiedKFold
There are lots of metrics (score functions, etc.) available:
http://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics
http://scikit-learn.org/stable/modules/model_evaluation.html#model-evaluation
See the docs for a particular estimator to find out what the score method returns (which is generally the score function used in optimizing the model). For instance
http://jaquesgrobler.github.io/Online-Scikit-Learn-stat-tut/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC.score
-Roban
On Fri, Jun 21, 2013 at 10:20 AM, Gianni Iannelli <***@msn.com> wrote:
>
> Thank You very much for the link!! It does closely what I wanna do!
>
> In my case I have two classes that are for example 0 and 1. I wanna keep the distribution (in the training set and so also the test set) between them similar. And I also need that are choosen randomly, I don't care if in one case I get the same index for training and test set. Well, to select randomly I think that the sklearn.cross_validation.ShuffleSplit() does what I want and I will investigate on that. To keep the distribution equally between the two classes I was thinking to:
>
> split the two classes
> apply for each of them the separation in training and test using the ShuffleSplit()
> concatenate now the two classes again (they will have the same size before the split)
> add to one of the two index vector the size of one of the two class (depends how I will concatenate the two)
> apply my SVM classification
>
>
> What do you think? Do you think is ok?
>
> I have one another question. How score works? What it computes? I searched around but I found this:
>
> sklearn.metrics.classification_report(y_true, y_pred, labels=None, target_names=None)
>
> That maybe could give me back a confusion matrix where I could compute (maybe, I'm just guessing) an Overall Accuracy and a Kappa Coefficient.
>
> Is it correct?
>
> Thank You Very Much!!!
>
> ________________________________
> Date: Fri, 21 Jun 2013 10:59:13 +1000
> From: ***@student.usyd.edu.au
> To: scikit-learn-***@lists.sourceforge.net
> Subject: Re: [Scikit-learn-general] SVM: select the training set randomly
>
>
> Please see http://scikit-learn.org/stable/tutorial/statistical_inference/model_selection.html
>
>
> On Fri, Jun 21, 2013 at 10:31 AM, Gianni Iannelli <***@msn.com> wrote:
>
> Dear All,
>
> I have one question. I have a dataset of 100 vector each with some features. Of this 100 I already know the classification of all of them. What I wanna do is select randomly in this 100 a subset to use as training set and the rest as test set. There is something already implemented in scikit that do it automatically or I have to use an index method? For index method I mean to separate the two classes, for example I have 40 (class A) and 60 (class B) for each of them I select 10 number randomly for each class and put set these 20 vector as training set. After that I select the other 80 vectors (also using the index of the main matrix) and classify.
>
> Do you think this is too crazy and there is something simple? There is also a validation of the result that could tell me how the classification is good? I know that this is not a real case because I know the classification result but I just wanna see what happens changing the number of features, number of training elements, and so on.
>
> Thanks All!!!
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-***@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
>
> ------------------------------------------------------------------------------ This SF.net email is sponsored by Windows: Build for Windows Store. http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________ Scikit-learn-general mailing list Scikit-learn-***@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-***@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>