[WIN] Stochastic bug: Minc2 data may load as np.longdouble in h5py < 2.10

[effigies]

These AppVeyor failures are strange and stochastic. The failures are consistent:

======================================================================
ERROR: autogenerated test from validate_data_deprecated
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python34\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\python34\lib\site-packages\nibabel\tests\test_api_validators.py", line 23, in meth
    validator(self, imaker, params)
  File "c:\python34\lib\site-packages\nibabel\tests\test_image_api.py", line 380, in validate_data_deprecated
    assert_data_similar(img._data, params)
  File "c:\python34\lib\site-packages\nibabel\tests\test_helpers.py", line 80, in assert_data_similar
    real_arr = np.asarray(arr)
  File "c:\python34\lib\site-packages\numpy\core\numeric.py", line 501, in asarray
    return array(a, dtype, copy=False, order=order)
  File "c:\python34\lib\site-packages\nibabel\minc1.py", line 261, in __array__
    return self.minc_file.get_scaled_data()
  File "c:\python34\lib\site-packages\nibabel\minc2.py", line 139, in get_scaled_data
    return self._normalize(raw_data, sliceobj)
  File "c:\python34\lib\site-packages\nibabel\minc1.py", line 189, in _normalize
    dmin, dmax = self._get_valid_range()
  File "c:\python34\lib\site-packages\nibabel\minc2.py", line 98, in _get_valid_range
    info = np.iinfo(ddt.type)
  File "c:\python34\lib\site-packages\numpy\core\getlimits.py", line 516, in __init__
    raise ValueError("Invalid integer data type.")
ValueError: Invalid integer data type.
======================================================================
ERROR: nibabel.tests.test_minc2.TestMinc2File.test_array_proxy_slicing
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python34\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\python34\lib\site-packages\nibabel\tests\test_minc1.py", line 200, in test_array_proxy_slicing
    arr = img.get_data()
  File "c:\python34\lib\site-packages\nibabel\dataobj_images.py", line 202, in get_data
    data = np.asanyarray(self._dataobj)
  File "c:\python34\lib\site-packages\numpy\core\numeric.py", line 553, in asanyarray
    return array(a, dtype, copy=False, order=order, subok=True)
  File "c:\python34\lib\site-packages\nibabel\minc1.py", line 261, in __array__
    return self.minc_file.get_scaled_data()
  File "c:\python34\lib\site-packages\nibabel\minc2.py", line 139, in get_scaled_data
    return self._normalize(raw_data, sliceobj)
  File "c:\python34\lib\site-packages\nibabel\minc1.py", line 189, in _normalize
    dmin, dmax = self._get_valid_range()
  File "c:\python34\lib\site-packages\nibabel\minc2.py", line 98, in _get_valid_range
    info = np.iinfo(ddt.type)
  File "c:\python34\lib\site-packages\numpy\core\getlimits.py", line 516, in __init__
    raise ValueError("Invalid integer data type.")
ValueError: Invalid integer data type.
======================================================================
ERROR: nibabel.tests.test_minc2.TestMinc2File.test_load
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python34\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\python34\lib\site-packages\nibabel\tests\test_minc1.py", line 187, in test_load
    data = img.get_data()
  File "c:\python34\lib\site-packages\nibabel\dataobj_images.py", line 202, in get_data
    data = np.asanyarray(self._dataobj)
  File "c:\python34\lib\site-packages\numpy\core\numeric.py", line 553, in asanyarray
    return array(a, dtype, copy=False, order=order, subok=True)
  File "c:\python34\lib\site-packages\nibabel\minc1.py", line 261, in __array__
    return self.minc_file.get_scaled_data()
  File "c:\python34\lib\site-packages\nibabel\minc2.py", line 139, in get_scaled_data
    return self._normalize(raw_data, sliceobj)
  File "c:\python34\lib\site-packages\nibabel\minc1.py", line 189, in _normalize
    dmin, dmax = self._get_valid_range()
  File "c:\python34\lib\site-packages\nibabel\minc2.py", line 98, in _get_valid_range
    info = np.iinfo(ddt.type)
  File "c:\python34\lib\site-packages\numpy\core\getlimits.py", line 516, in __init__
    raise ValueError("Invalid integer data type.")
ValueError: Invalid integer data type.
======================================================================
ERROR: nibabel.tests.test_minc2.TestMinc2File.test_mincfile_slicing
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python34\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\python34\lib\site-packages\nibabel\tests\test_minc1.py", line 171, in test_mincfile_slicing
    data = mnc.get_scaled_data()
  File "c:\python34\lib\site-packages\nibabel\minc2.py", line 139, in get_scaled_data
    return self._normalize(raw_data, sliceobj)
  File "c:\python34\lib\site-packages\nibabel\minc1.py", line 189, in _normalize
    dmin, dmax = self._get_valid_range()
  File "c:\python34\lib\site-packages\nibabel\minc2.py", line 98, in _get_valid_range
    info = np.iinfo(ddt.type)
  File "c:\python34\lib\site-packages\numpy\core\getlimits.py", line 516, in __init__
    raise ValueError("Invalid integer data type.")
ValueError: Invalid integer data type.
======================================================================
ERROR: nibabel.tests.test_processing.test_spatial_axes_check
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python34\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\python34\lib\site-packages\numpy\testing\_private\decorators.py", line 155, in skipper_func
    return f(*args, **kwargs)
  File "c:\python34\lib\site-packages\nibabel\tests\test_processing.py", line 366, in test_spatial_axes_check
    out = resample_from_to(img, img, mode='nearest')
  File "c:\python34\lib\site-packages\nibabel\processing.py", line 180, in resample_from_to
    cval=cval)
  File "c:\python34\lib\site-packages\scipy\ndimage\interpolation.py", line 416, in affine_transform
    filtered = spline_filter(input, order, output=numpy.float64)
  File "c:\python34\lib\site-packages\scipy\ndimage\interpolation.py", line 113, in spline_filter
    spline_filter1d(input, order, axis, output=output)
  File "c:\python34\lib\site-packages\scipy\ndimage\interpolation.py", line 82, in spline_filter1d
    _nd_image.spline_filter1d(input, order, axis, output)
RuntimeError: array type dtype('float64') not supported
======================================================================
FAIL: autogenerated test from validate_data_interface
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python34\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\python34\lib\site-packages\nibabel\tests\test_api_validators.py", line 23, in meth
    validator(self, imaker, params)
  File "c:\python34\lib\site-packages\nibabel\tests\test_image_api.py", line 208, in validate_data_interface
    self._check_proxy_interface(imaker, meth_name)
  File "c:\python34\lib\site-packages\nibabel\tests\test_image_api.py", line 253, in _check_proxy_interface
    assert_true(data is data_again)
AssertionError: False is not true
======================================================================
FAIL: autogenerated test from validate_dtype
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python34\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\python34\lib\site-packages\nibabel\tests\test_api_validators.py", line 23, in meth
    validator(self, imaker, params)
  File "c:\python34\lib\site-packages\nibabel\tests\test_image_api.py", line 178, in validate_dtype
    assert_equal(img.get_data_dtype().type, params['dtype'])
AssertionError: <class 'numpy.float64'> != <class 'numpy.float64'>
======================================================================
FAIL: nibabel.tests.test_minc2.TestMinc2File.test_mincfile
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python34\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\python34\lib\site-packages\nibabel\tests\test_minc1.py", line 159, in test_mincfile
    assert_equal(mnc.get_data_dtype().type, tp['dtype'])
AssertionError: <class 'numpy.float64'> != <class 'numpy.float64'>

And I've seen them show up in Python 3.4, Python 3.4-x64, Python 3.5 and Python 3.5-x64, but in none of these cases do they consistently fail.

Example builds:

• https://ci.appveyor.com/project/nipy/nibabel/build/1.0.476
• https://ci.appveyor.com/project/nipy/nibabel/build/1.0.477

[matthew-brett]

Hum - but that's worrying - no? What was the conclusion about what was causing it?

[effigies]

Unclear. It's totally stochastic. Reverting to old versions of numpy didn't resolve it. My best guess is something broken in the Python builds used specifically on AppVeyor. By moving to Miniconda, we test against possibly the most widely distributed build of Python for Windows.

[matthew-brett]

Well - possibly - but do you think Appveyor is installing a not-standard Python? Or is this the build from Python.org? If it's the Python.org build, then it is widely used.

[effigies]

I'm not really sure. We have a thread with support, but I haven't had time to follow up lately. Hence this quick-and-dirty attempt at a fix, here.

I know you're busy, too, so not trying to shove this on you, but if you have any time for tracking this down, you can take over that conversation with my blessing. Yarik and I spent a pretty fruitless week a couple months back.

[effigies]

Note that this is still occasionally happening as of 2018.12.17. https://ci.appveyor.com/project/nipy/nibabel/builds/21076767/job/bh4mt7vahjwfr37l

[effigies]

Since it looks like numpy is running into this too (numpy/numpy#13263), I'll leave this open. If it's a numpy problem, then this can serve as a test case.

Also flagging numpy/numpy#12096 where we initially reported this.

[matthew-brett]

I guess you already tried setting the PYTHONHASHSEED to test that?

[effigies]

I did not. Looks like there isn't a way to get Python to spit out the effective hash seed, so some trial and error will be necessary before we can reliably reproduce the issue. Fortunately, moving to Azure, we'll have a lot more parallel jobs we can throw at it.

To maintain the thread, this was in response to #783 (comment):

I think its stochasticity is an effect of dict ordering, which is why it only shows up in Python < 3.6, so I guess if we just wait for one more end-of-life the tests will be fixed.

[effigies]

Returning discussion here from #790:

Looks like this was actually h5py/h5py#1051, fixed in h5py/h5py#1134, just released a couple days ago.

I don't think it was ever at risk of producing bad data, although it did fail cache checks. Do you suppose we ought to require h5py >= 2.10?

[matthew-brett]

Good spot. Ouch. But glad to see it will likely be fixed.

[yarikoptic]

Do you suppose we ought to require h5py >= 2.10?

hard requirement might make backporting etc more difficult on distribution based platforms. And as always -- newer is not necessarily "less buggy". May be, because IIRC it was Windows specific, such versioned dependency should be only for windows?

[effigies]

It's also specific to Python3 versions <3.6. We could update the optional package check to only check h5py versions on Windows + Python < 3.6.

[effigies]

I guess the reason this showed up suddenly in #638 (June 2018) was h5py 2.8 was released in May 2018, which included h5py/h5py#926. We could strictly warn on 2.8 <= h5py.__version__ < 2.10.