Sunday, 29 September 2013

Numpy memory error creating huge matrix

Numpy memory error creating huge matrix

I am using numpy and trying to create a huge matrix. While doing this, I
receive a memory error
Because the matrix is not important, I will just show the way how to
easily reproduce the error.
a = 10000000000
data = np.array([float('nan')] * a)
not surprisingly, this throws me MemoryError
There are two things I would like to tell:
I really need to create and to use a big matrix
I think I have enough RAM to handle this matrix (I have 24 Gb or RAM)
Is there an easy way to handle big matrices in numpy?
Just to be on the safe side, I previously read these posts (which sounds
similar):
Python Numpy Very Large Matrices
Python/Numpy MemoryError
Processing a very very big data set in python - memory error

No comments:

Post a Comment