0

I am working on a model which reflects the spread of innovations through a society of N individuals. I have an adjacency matrix A of size NxN, note that this adjacency matrix is sparse.

I want to do simulations for NxN=10^7. I first tried Matlab, but unfortunately Matlab cannot handle NxN>10^4.

Is it possible to use NumPy for my simulations?

LH10
  • 17
  • 3
  • Numpy and Matlab work on very similar C routines under the hood. If Matlab can't handle it, I seriously doubt numpy can. Please look at an alternative using big data tools. PySpark is a good please to start. – cs95 Sep 13 '17 at 11:12
  • N*N = 10^14......... You probably need a different representation; how many individuals are there in your sparse matrix? – Reblochon Masque Sep 13 '17 at 11:12
  • follow this link https://stackoverflow.com/questions/14525344/whats-the-maximum-size-of-a-numpy-array – Amrit Sep 13 '17 at 11:13
  • Sorry I meant NxN = 10^7, I edited my question. – LH10 Sep 14 '17 at 10:48

1 Answers1

1

Scipy can handle NxN sparse matrices with N=10^7

import scipy.sparse as sparse
N = 10e7
sparse.bsr_matrix((N, N))

Output:

<100000000x100000000 sparse matrix of type '<class 'numpy.float64'>'
    with 0 stored elements (blocksize = 1x1) in Block Sparse Row format>

Whether it is suitable for your simulations depends on a number of things that we don't know. You might need to use a different sparse matrix class.

Stop harming Monica
  • 12,141
  • 1
  • 36
  • 56
  • `scipy.sparse` store element indices as integers. For smaller matrices it uses `np.int32` but can use `np.int64` if needed. So that allows very large shapes. But the number of nonzero elements is limited by memory. Unless very sparse, calculations will be slow compared to dense arrays. – hpaulj Sep 13 '17 at 17:25