Discussion:
tSNE assertion errors
(too old to reply)
L***@web.de
2016-04-18 12:28:05 UTC
Permalink
------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
Alexander Fabisch
2016-04-18 12:44:36 UTC
Permalink
Hi Sarah,

t-SNE does not support incremental training. Your model will be
retrained every time you fit a new batch of data (see
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/manifold/t_sne.py#L664).
That means you might have found a dataset that reveals an error in
implementation. Could you provide a small script that reproduces the error?

Best regards,

Alexander

Am 18.04.2016 um 14:28 schrieb ***@web.de:
> Hey everyone!
> I am new to Python and the scikit learn package so I hope someone can
> help me with the two issues I encountered during use of the
> sklearn.manifold implementation of the t-SNE algorithm. First a little
> bit of context: I am repeatedly feeding batches of dimensionality
> 500x784 to the algorithm for visualization. However, before my script
> finishes, one of the two following error messages occurs:
> AssertionError:[t-SNE]Insertionfailed
> or
> AssertionError:Treeconsistency failed:unexpected number of
> points=499at root node=500
> Furthermore, these messages do not occur at a fixed time but their
> behaviour seems rather non-deterministic. Hopefully someone came
> across this problem before and can help me to fix it.
> Best,
> Sarah
>
>
> ------------------------------------------------------------------------------
> Find and fix application performance issues faster with Applications Manager
> Applications Manager provides deep performance insights into multiple tiers of
> your business applications. It resolves application problems quickly and
> reduces your MTTR. Get your free trial!
> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
>
>
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-***@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

--
Von meinem iPhone gesendet
L***@web.de
2016-04-18 13:28:38 UTC
Permalink
------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
Andreas Mueller
2016-04-21 15:19:55 UTC
Permalink
Can you please report this on the issue tracker? Thanks!

On 04/18/2016 09:28 AM, ***@web.de wrote:
> Thanks for your response Alexander! Here is a simplified version of my
> script applied to the MNIST data set. It wasn't clear from my first
> mail but I don't want to train it incrementally but instead apply tsne
> to each batch within the data set (for several epochs). This works for
> an unspecified number of epochs/batches until the program crashes.
> for epoch in range(20)
> for batch in batch_iterator(mnist_data):
> # reshape from (500, 1, 28, 28) to (500, 784)
> data = batch.reshape(batch.shape[0], -1)
> tsne = TSNE(n_components=2, random_state=0, init='pca',
> verbose=0).fit_transform(data)
> # continue with scatter plot visualization...
> plt.scatter(tsne[:,0], tsne[:,1], c=labels)
> plt.show()
> *Gesendet:* Montag, 18. April 2016 um 14:44 Uhr
> *Von:* "Alexander Fabisch" <***@informatik.uni-bremen.de>
> *An:* scikit-learn-***@lists.sourceforge.net
> *Betreff:* Re: [Scikit-learn-general] tSNE assertion errors
> Hi Sarah,
>
> t-SNE does not support incremental training. Your model will be
> retrained every time you fit a new batch of data (see
> https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/manifold/t_sne.py#L664).
> That means you might have found a dataset that reveals an error in
> implementation. Could you provide a small script that reproduces the
> error?
>
> Best regards,
>
> Alexander
> Am 18.04.2016 um 14:28 schrieb ***@web.de:
>
> Hey everyone!
> I am new to Python and the scikit learn package so I hope someone
> can help me with the two issues I encountered during use of the
> sklearn.manifold implementation of the t-SNE algorithm. First a
> little bit of context: I am repeatedly feeding batches of
> dimensionality 500x784 to the algorithm for visualization.
> However, before my script finishes, one of the two following error
> messages occurs:
> AssertionError:[t-SNE]Insertionfailed
> or
> AssertionError:Treeconsistency failed:unexpected number of
> points=499at root node=500
> Furthermore, these messages do not occur at a fixed time but their
> behaviour seems rather non-deterministic. Hopefully someone came
> across this problem before and can help me to fix it.
> Best,
> Sarah
>
> ------------------------------------------------------------------------------
> Find and fix application performance issues faster with Applications Manager
> Applications Manager provides deep performance insights into multiple tiers of
> your business applications. It resolves application problems quickly and
> reduces your MTTR. Get your free trial!
> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
>
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-***@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
> --
> Von meinem iPhone gesendet
> ------------------------------------------------------------------------------
> Find and fix application performance issues faster with Applications
> Manager Applications Manager provides deep performance insights into
> multiple tiers of your business applications. It resolves application
> problems quickly and reduces your MTTR. Get your free trial!
> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z_______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-***@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
> ------------------------------------------------------------------------------
> Find and fix application performance issues faster with Applications Manager
> Applications Manager provides deep performance insights into multiple tiers of
> your business applications. It resolves application problems quickly and
> reduces your MTTR. Get your free trial!
> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
>
>
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-***@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
L***@web.de
2016-04-22 08:44:44 UTC
Permalink
Alexander already reported it and it seemes that the bleeding-edge version of sklearn from github resolves the issue. However, I have trouble now to install it on my Windows machine (using Python 2.7 and Anaconda 4.0.5) via pip command. Somehow it fails to find the required DLLs during programme execution, even though sklearn appears under the list of installed packages.

Did anyone experience this problem before and can possibly help me with it?

Best,
Sarah

 
 
 
 

Gesendet: Donnerstag, 21. April 2016 um 17:19 Uhr
Von: "Andreas Mueller" <***@gmail.com>
An: scikit-learn-***@lists.sourceforge.net
Betreff: Re: [Scikit-learn-general] tSNE assertion errors

Can you please report this on the issue tracker? Thanks!
 
On 04/18/2016 09:28 AM, ***@web.de wrote:

Thanks for your response Alexander! Here is a simplified version of my script applied to the MNIST data set. It wasn't clear from my first mail but I don't want to train it incrementally but instead apply tsne to each batch within the data set (for several epochs). This works for an unspecified number of epochs/batches until the program crashes.
 
for epoch in range(20)
    for batch in batch_iterator(mnist_data):
        # reshape from (500, 1, 28, 28) to (500, 784)

        data = batch.reshape(batch.shape[0], -1)
        tsne = TSNE(n_components=2, random_state=0, init='pca', verbose=0).fit_transform(data)
   
        # continue with scatter plot visualization...
        plt.scatter(tsne[:,0], tsne[:,1], c=labels)
        plt.show()
 

Gesendet: Montag, 18. April 2016 um 14:44 Uhr
Von: "Alexander Fabisch" <***@informatik.uni-bremen.de>[***@informatik.uni-bremen.de]
An: scikit-learn-***@lists.sourceforge.net[scikit-learn-***@lists.sourceforge.net]
Betreff: Re: [Scikit-learn-general] tSNE assertion errors

Hi Sarah,

t-SNE does not support incremental training. Your model will be retrained every time you fit a new batch of data (see https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/manifold/t_sne.py#L664[https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/manifold/t_sne.py#L664]). That means you might have found a dataset that reveals an error in implementation. Could you provide a small script that reproduces the error?

Best regards,

Alexander
 
Am 18.04.2016 um 14:28 schrieb ***@web.de:

Hey everyone!
 
I am new to Python and the scikit learn package so I hope someone can help me with the two issues I encountered during use of the sklearn.manifold implementation of the t-SNE algorithm. First a little bit of context: I am repeatedly feeding batches of dimensionality 500x784 to the algorithm for visualization. However, before my script finishes, one of the two following error messages occurs:
 
AssertionError: [t-SNE] Insertion failed
or
AssertionError: Tree consistency failed: unexpected number of points=499 at root node=500
 
Furthermore, these messages do not occur at a fixed time but their behaviour seems rather non-deterministic. Hopefully someone came across this problem before and can help me to fix it.
 
Best,
Sarah     
------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!https://ad.doubleclick.net/ddm/clk/302982198;130105516;z[https://ad.doubleclick.net/ddm/clk/302982198;130105516;z]
     
_______________________________________________
Scikit-learn-general mailing listScikit-learn-***@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/scikit-learn-general
 
--
Von meinem iPhone gesendet
------------------------------------------------------------------------------ Find and fix application performance issues faster with Applications Manager Applications Manager provides deep performance insights into multiple tiers of your business applications. It resolves application problems quickly and reduces your MTTR. Get your free trial! https://ad.doubleclick.net/ddm/clk/302982198;130105516;z_______________________________________________[https://ad.doubleclick.net/ddm/clk/302982198;130105516;z_______________________________________________] Scikit-learn-general mailing list Scikit-learn-***@lists.sourceforge.net[Scikit-learn-***@lists.sourceforge.net] https://lists.sourceforge.net/lists/listinfo/scikit-learn-general     
------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!https://ad.doubleclick.net/ddm/clk/302982198;130105516;z[https://ad.doubleclick.net/ddm/clk/302982198;130105516;z]
     
_______________________________________________
Scikit-learn-general mailing listScikit-learn-***@lists.sourceforge.net[Scikit-learn-***@lists.sourceforge.net]https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------ Find and fix application performance issues faster with Applications Manager Applications Manager provides deep performance insights into multiple tiers of your business applications. It resolves application problems quickly and reduces your MTTR. Get your free trial! https://ad.doubleclick.net/ddm/clk/302982198;130105516;z_______________________________________________[https://ad.doubleclick.net/ddm/clk/302982198;130105516;z_______________________________________________] Scikit-learn-general mailing list Scikit-learn-***@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/scikit-learn-general[https://lists.sourceforge.net/lists/listinfo/scikit-learn-general]
Loading...