tensorflow version - NEON.h5 (file signature not found)

[aloui-mathias]

Hi,
Since the Pytorch-Release, I'am having issue with the tesorflow version.
I tried using my script to make a prediction but when loading the model I had an error.
I tried to reload the model by changing the name in current_release.csv to force a new download but still the same error.
Thankfully I have an old backup virtual environment. I replace the NEON.h5 file with the one from this old env and it worked.

Here is an example of my error :

In [1]: from deepforest import deepforest
Using TensorFlow backend.

In [2]: model = deepforest.deepforest()
Reading config file: C:\Users\MA\anaconda3\envs\DeepForestTourne\lib\site-packages\deepforest\data\deepforest_config.yml
A blank deepforest object created. To perform prediction, either train or load an existing model.

In [3]: model.use_release()
Model from DeepForest release https://github.com/weecology/DeepForest/releases/tag/1.0.0 was already downloaded. Loading model from file.
Loading pre-built model: https://github.com/weecology/DeepForest/releases/tag/1.0.0
C:\Users\MA\anaconda3\envs\DeepForestTourne\lib\site-packages\deepforest\data/NEON.h5

OSError Traceback (most recent call last)
in
----> 1 model.use_release()

~\anaconda3\envs\DeepForestTourne\lib\site-packages\deepforest\deepforest.py in use_release(self, gpus)
169 # Suppress compilte warning, not relevant here
170 warnings.filterwarnings("ignore", category=UserWarning)
--> 171 self.model = utilities.read_model(self.weights, self.config)
172
173 # Convert model

~\anaconda3\envs\DeepForestTourne\lib\site-packages\deepforest\utilities.py in read_model(model_path, config)
48 # warnings.simplefilter('ignore', UserWarning)
49 print(model_path)
---> 50 model = models.load_model(model_path, backbone_name='resnet50')
51
52 return model

~\anaconda3\envs\DeepForestTourne\lib\site-packages\deepforest\keras_retinanet\models_init_.py in load_model(filepath, backbone_name)
81 """
82 import keras.models
---> 83 return keras.models.load_model(filepath, custom_objects=backbone(backbone_name).custom_objects)
84
85

~\anaconda3\envs\DeepForestTourne\lib\site-packages\keras\engine\saving.py in load_wrapper(*args, **kwargs)
490 os.remove(tmp_filepath)
491 return res
--> 492 return load_function(*args, **kwargs)
493
494 return load_wrapper

~\anaconda3\envs\DeepForestTourne\lib\site-packages\keras\engine\saving.py in load_model(filepath, custom_objects, compile)
581
582 if H5Dict.is_supported_type(filepath):
--> 583 with H5Dict(filepath, mode='r') as h5dict:
584 model = _deserialize_model(h5dict, custom_objects, compile)
585 elif hasattr(filepath, 'write') and callable(filepath.write):

~\anaconda3\envs\DeepForestTourne\lib\site-packages\keras\utils\io_utils.py in init(self, path, mode)
189 self._is_file = False
190 elif isinstance(path, six.string_types) or _is_path_instance(path):
--> 191 self.data = h5py.File(path, mode=mode)
192 self._is_file = True
193 elif isinstance(path, dict):

~\anaconda3\envs\DeepForestTourne\lib\site-packages\h5py_hl\files.py in init(self, name, mode, driver, libver, userblock_size, swmr, rdcc_nslots, rdcc_nbytes, rdcc_w0, track_order, **kwds)
406 fid = make_fid(name, mode, userblock_size,
407 fapl, fcpl=make_fcpl(track_order=track_order),
--> 408 swmr=swmr)
409
410 if isinstance(libver, tuple):

~\anaconda3\envs\DeepForestTourne\lib\site-packages\h5py_hl\files.py in make_fid(name, mode, userblock_size, fapl, fcpl, swmr)
171 if swmr and swmr_support:
172 flags |= h5f.ACC_SWMR_READ
--> 173 fid = h5f.open(name, flags, fapl=fapl)
174 elif mode == 'r+':
175 fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)

h5py_objects.pyx in h5py._objects.with_phil.wrapper()

h5py_objects.pyx in h5py._objects.with_phil.wrapper()

h5py\h5f.pyx in h5py.h5f.open()

OSError: Unable to open file (file signature not found)

[bw4sz]

Thanks. I am making a document today about how best to update. I think the cleanest thing is to manually load the desired tensorflow release model.

Download here and then load weights

reloaded = deepforest.deepforest(weights="example_save_weights.h5")

https://github.com/weecology/DeepForest/releases/tag/v0.3.0

I just had another user report that the performance was different than expected between the use_release() and the download model. This is extremely surprising since they are the same model. #195

Please let me know if you experience any change in performance. Close when ready.

[aloui-mathias]

Thanks for the quick response.

Yes I also have a problem of performance with the pytorch version. I'm using DeepForest in urban area (and I'm going to train it on new urban data to have a better model) and the tensorflow version is pretty good but the pytorch version is really not great and is very sensitive to changes in patch_size and overlap.

[bw4sz]

can you show some example images? The underlying data and algorithm is basically identical.

…

On Thu, Jun 10, 2021 at 4:23 AM Mathias Aloui ***@***.***> wrote: Thanks for the quick response. Yes I also have a problem of performance with the pytorch version. I'm using DeepForest in urban area (and I'm going to train it on new urban data to have a better model) and the tensorflow version is pretty good but the pytorch version is really not great and is very sensitive to changes in patch_size and overlap. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#198 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJHBLEFQ7BHJXBYVLP4K2LTSCOB7ANCNFSM46LV4CGQ> .

-- Ben Weinstein, Ph.D. Postdoctoral Fellow University of Florida http://benweinstein.weebly.com/

[aloui-mathias]

Here is an example with the same image and the same parameters (patch_size = 400, patch_overlap = 0.15, iou_threshold=0.15) :

The first with tensorflow :

The second with pytorch :

Note : pytorch is way faster than tensorflow

[bw4sz]

can you pass me that raw image? This is surprising, there must be an additional parameter change somewhere, minimum score threshold between the models? We test against the benchmark here: https://github.com/weecology/NeonTreeEvaluation and the pytorch model is within 1% of the tensorflow of the model with no visible differences. The training data are the same between those two releases. I just retrained a model to add new annotations, including a small urban tile, I will upload. Would you be willing to share your annotations and i'll add them in for future users? I'm going to have a look at your image, but something doesn't quite make sense. I'm going to test it on predict_image first, to make sure it isn't some tiling method between the old and new versions. That seems more likely than the actual model weights.

…

On Thu, Jun 10, 2021 at 1:13 PM Mathias Aloui ***@***.***> wrote: Here is an example with the same image and the same parameters (patch_size = 400, patch_overlap = 0.15, iou_threshold=0.15) : The first with tensorflow : [image: tensorflow] <https://user-images.githubusercontent.com/43454650/121590723-f1e18480-ca38-11eb-9142-06317a06fb67.png> The second with pytorch : [image: pytorch] <https://user-images.githubusercontent.com/43454650/121590722-f148ee00-ca38-11eb-8a7e-3d8d79d37aed.png> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#198 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJHBLBTPT35MQVHV7CAGDTTSEMHTANCNFSM46LV4CGQ> .

-- Ben Weinstein, Ph.D. Postdoctoral Fellow University of Florida http://benweinstein.weebly.com/

[aloui-mathias]

Here is the image I used :

I didn't check if other parameters were different (I assumed that all default parameters were the same).

The annotations I have are for the challenge Detect Trees for the Hack4Nature (https://hack4nature.slite.com/p/note/Fs34nEyzDG61edEM5oHnUF/Challenge-4-Detect-Tree). We have a wiki Slite with everything we use and soon we will put a link to a web page for online annotations and all the data we use. I can keep you posted on our advancement ;-)

[bw4sz]

Great, thanks, keep my posted. I'm downloading that image and playing with it. I just trained a new model. I really can't imagine the tensorflow and pytorch versions should be meaningfully different. It must be in preprocessing. I'll update.

…

On Fri, Jun 11, 2021 at 12:04 AM Mathias Aloui ***@***.***> wrote: Here is the image I used : [image: image] <https://user-images.githubusercontent.com/43454650/121643179-d191e400-ca91-11eb-88d0-011a9b8b3e52.png> I didn't check if other parameters were different (I assumed that all default parameters were the same). The annotations I have are for the challenge Detect Trees for the Hack4Nature (https://www.hackfornature.com/blog/challenge-4-detect-tree <http://url>). We have a wiki Slite with everything we use and soon we will put a link to a web page for online annotations and all the data we use. I can keep you posted on our advancement ;-) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#198 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJHBLABPEGD4D3N6TSAQEDTSGYRHANCNFSM46LV4CGQ> .

-- Ben Weinstein, Ph.D. Postdoctoral Fellow University of Florida http://benweinstein.weebly.com/

[bw4sz]

Just posting a few sample images as a try to reproduce

from deepforest import main
from matplotlib import pyplot as plt
m=main.deepforest()
m.use_release()
a = m.predict_image(path = "/Users/benweinstein/Downloads/test_image.png",return_plot=True)
plt.imshow(a[:,:,::-1])

[aloui-mathias]

Just a little follow-up on the performance issue :

I saw your pretty good prediction with predict_image and I wondered what about the evaluation methode.
I tested the evaluation methode on the prediction from predict_tile and predict_image.
For predict_image everything is perfect, everything at 100%.
But for predict_tile it's a mess. class precision and recall are 100% but box recall and precision are not.
I think like you said there is a problem with the preprocessing, with the tiling process.

A comment on how evaluation works :

Right now the evaluation method main.deepforest.evaluate works like predict_image to make the prediction. Which explains why everything works on predict_image predictions and not on predict_tile predictions.
If the preprocessing is fixed, we will still need an evaluation method that works like predict_tile to make prediction otherwise for big image the evaluation will not be accurate.

My own prediction :

I used the exact same code you used and I have a slightly different prediction, strange ?

My tests :

Here is the notebook I used for my evaluation tests

pytorch.zip

Edit :

I found out that changing the patch_size change drastically the performance for pytorch predict_tile

[bw4sz]

Changing patch_size will definitely have performance for predict tile, that's a feature not a bug. For every input resolution, there will be an optimal patch size. There is no way to anticipate for all users. The question is why did earlier versions of deepforest predict_tile work better for you? I will check the default patch size and try to see what has changed.

I cannot fathom why using the exact same code would produce different results. That seems like an important thing to investigate. Thanks for following up.

[bw4sz]

I'm still trying to wrap my head around this. I think you are not using the code quite as intended. Let's figure out how to make the docs better. The evaluate method is meant for when you have a .csv of annotations for each image. You are 100% right there is not an evaluate method directly for a tile. You would need to cut the tile into pieces using preprocess.split_raster, save the output and then evaluate on this. I can make an example in the docs. Maybe make a quick video in evaluate.

[bw4sz]

Evaluating tiles

The evaluation method uses deepforest.predict_image for each of the paths supplied in the image_path column. This means that the entire image is passed for prediction. This will not work for large orthomosaics. The deepforest.predict_tile method does a couple things under hood that need to be repeated for evaluation.

psuedo_code:

output_annotations = deepforest.preprocess.split_raster(
    path_to_raster = <path>,
    annotations_file = <original_annotation_path>,
    base_dir = <location to save crops>
    patch_size = <size of each crop>
    
)

output_annotations.to_csv("new_annotations.csv")

results = model.evaluate(
    csv_file="new_annotations.csv",
    root_dir=<base_dir from above>
)

[Mashood3624]

@bw4sz How to download weights for "Alive" and "Dead" labels ?

[Mashood3624]

model ****