ndarray indexing issues

[ashishlal]

I am new to mxnet. I just installed mxnet 1.0.0 and python 3.5 on a Ubuntu 14.04 machine with CUDA 8.0 and cudnn 7.0.5.

My code is given below. I am trying to store image data in an ndarray. (see https://github.com/ypwhs/DogBreed_gluon/blob/master/get_features_v3.ipynb for the original code) -

X_224 = nd.zeros((n, 3, 224, 224))
X_299 = nd.zeros((n, 3, 299, 299))

mean = np.array([0.485, 0.456, 0.406])
std = np.array([0.229, 0.224, 0.225])

for i, (fname, breed) in tqdm(df.iterrows(), total=n):
    img = cv2.imread('data/train/%s.jpg' % fname)
    img_224 = ((cv2.resize(img, (224, 224))[:, :, ::-1] / 255.0 - mean) / std).transpose((2, 0, 1))
    img_299 = ((cv2.resize(img, (299, 299))[:, :, ::-1] / 255.0 - mean) / std).transpose((2, 0, 1))

    X_224[i] = nd.array(img_224) <-- I get error in this line
    X_299[i] = nd.array(img_299)

ValueError: Indexing NDArray with index=0 and type=class 'numpy.int64' is not supported.

[anirudh2290]

Issue is that isinstance is not working as expected by the code and returns false in python3. Workaround is to force the type of i to be int instead of numpy.int64. I think this should also be fixed on the mxnet side. The problem is that we are calling isinstance with int and in the check isinstance(np.int64, int) is not reliable (numpy/numpy#2951). Please see: https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/ndarray/ndarray.py#L2039. The suggested fix is to check with np.integer for numpy data types. WDYT @reminisce ?

[piiswrong]

@reminisce

[reminisce]

The problem is that in MXNet, integer_types are defined as native python integer types: int and long, while np.int32 and np.int64 are different classes. That's why isinstance check failed. This happens for both python2 and python3.

@piiswrong Is there any concern of adding np.int32 and np.int64 to the definition of integer_types here?
https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/base.py#L40

[anirudh2290]

@reminisce I wasnt able to reproduce the issue for python 2. The int type changed in python3 and the type is not related to numpy classes

[reminisce]

@anirudh2290 Have you tried the following?

Python 2.7.14 |Anaconda custom (64-bit)| (default, Oct  5 2017, 02:28:52) 
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import mxnet as mx
/Users/jwum/anaconda2/lib/python2.7/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
/Users/jwum/anaconda2/lib/python2.7/site-packages/mxnet-1.1.0-py2.7.egg/mxnet/optimizer.py:136: UserWarning: WARNING: New optimizer mxnet.optimizer.NAG is overriding existing optimizer mxnet.optimizer.NAG
  Optimizer.opt_registry[name].__name__))
imp/Users/jwum/anaconda2/lib/python2.7/site-packages/urllib3/contrib/pyopenssl.py:46: DeprecationWarning: OpenSSL.rand is deprecated - you should use os.urandom instead
  import OpenSSL.SSL
>>> import numpy as np
>>> a = np.array([0], dtype='int32')
>>> b = mx.nd.array(a)
>>> b[a[0]]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jwum/anaconda2/lib/python2.7/site-packages/mxnet-1.1.0-py2.7.egg/mxnet/ndarray/ndarray.py", line 504, in __getitem__
    % (str(key), str(type(key))))
ValueError: Indexing NDArray with index=0 and type=<type 'numpy.int32'> is not supported

[anirudh2290]

@reminisce away from laptop. Will give that a try. Tried with np.int64

[anirudh2290]

@reminisce verified that it works for np.int64 on python2, doesn't work for np.int32. For python3, it doesnt work for either np.int32 or np.int64. This is probably because in python2, the int is 64 bit because of python build on my machine and isinstance check passed for np.int64. In python3 since int is changed and is not related to np.int32 or np.int64, it fails isinstance checks for both np.int32 and np.int64.

[anirudh2290]

@ashishlal related pr merged. can this be closed ?

[indhub]

#10434 fixes this