ISIS Data Area MRO / MEX

[jlaura]

Hi,
I used ISIS3 a few years back before it became ISIS 4. Anyhow, I tried to install with anaconda which failed because my glibc version was 2.3.1. So I tried miniconda3 and was successful in installing ISIS 7.2. Trouble started with the Data area. I used the downloadIsisData script and configured rclone. I ran into a few issues. I started by downloading the base folder which may have worked fine. I haven't checked yet. Next came the mro folder. My internet connection failed several times. I was able to restart but files that started downloading were skipped and the download started again with files that had not yet started their download. So I had a lot of incomplete downloaded files. After a day, the script said it was finished. The properties tab reported a folder with 6667 files at 305 gib. But when I tried running the downloadIsisData script again for any updates, the script tried downloading another 200+ gib's. I tried to modify the downloadIsisData script to replace the rclone command copy with sync. It is currently about 38% finished. The partion I have isis installed is 1tib. That partition has about 280 gib free. I also have lro and mex installed too. The properties tab for the mro folder says it has 7235 files at 307.2 gib's. This is what it also said before I started. Does this sound right?
cartrite

Copied from #5168.

[cartrite]

I just wanted to report strange behavior downloading data for isis with this new method. I thought that was where I should report it but this is probably better here. I did read that others were having issues as well. So.....
cartrite

[cartrite]

This may show what I am talking about.
out.txt

[cartrite]

out.txt viewable here.

(base) cartrite@localhost:/srv/disk1/isis3/miniconda3/envs> conda activate isis
(isis) cartrite@localhost:/srv/disk1/isis3/miniconda3/envs> updateIsisData mro $ISISDATA
2023-04-03 06:22:31 ERROR : Local file system at /srv/disk1/isis3/miniconda3/envs/isis/data/mro/kernels: Ignoring --track-renames as the source and destination do not have a common hash
Transferred: 1.496 GiB / 1.496 GiB, 100%, 4.589 MiB/s, ETA 0s
Checks: 6799 / 6799, 100%
Deleted: 91 (files), 2 (dirs)
Transferred: 176 / 176, 100%
Elapsed time: 5m45.0s
Transferred: 82.882 GiB / 210.292 GiB, 39%, 2.974 MiB/s, ETA 12h11m11s
Checks: 1770 / 1770, 100%
Renamed: 1
Transferred: 536 / 1199, 45%
Elapsed time: 5h40m45.0s
Transferring:

•    kernels/ck/mro_sc_psp_130827_130902.bc: 92% /232.747Mi, 211.916Ki/s, 1m25s
  

•    kernels/ck/mro_sc_psp_130917_130923.bc: 99% /231.991Mi, 314.396Ki/s, 6s
  

•    kernels/ck/mro_sc_psp_130924_130930.bc: 67% /230.505Mi, 191.425Ki/s, 6m38s
  

•    kernels/ck/mro_sc_psp_131001_131007.bc: 85% /229.675Mi, 284.369Ki/s, 1m57s
  

•    kernels/ck/mro_sc_psp_131008_131014.bc: 61% /235.795Mi, 304.596Ki/s, 5m8s
  

•    kernels/ck/mro_sc_psp_131015_131021.bc: 39% /233.185Mi, 296.516Ki/s, 8m7s
  

•    kernels/ck/mro_sc_psp_131022_131028.bc: 39% /235.182Mi, 609.404Ki/s, 3m58s
  

•    kernels/ck/mro_sc_psp_131029_131104.bc: 10% /236.513Mi, 167.297Ki/s, 21m28s
  

•    kernels/ck/mro_sc_psp_131105_131111.bc: 15% /236.552Mi, 286.704Ki/s, 11m56s
  

•    kernels/ck/mro_sc_psp_131112_131118.bc: 11% /237.274Mi, 496.938Ki/s, 7m11s
  

[Kelvinrr]

Make sure whatever pipelines you are using work fine and I think we can resolve this.

[cartrite]

I just had a internet interruption. Starting the command again seemed to pick up where it left off. I'm using the "pipelines" provided in your rclone-conf. Correct?

[Kelvinrr]

yeas, you want to use that configuration file with the command I had. That should do a better job to recover from interruptions that the script.

[cartrite]

It looks like success.
(base) cartrite@localhost:/srv/disk1/isis3> conda activate isis
(isis) cartrite@localhost:/srv/disk1/isis3> rclone sync --config "/srv/disk1/isis3/rclone-conf" mro: "/srv/disk1/isis3/mro/" --progress
Transferred: 166.919 GiB / 166.919 GiB, 100%, 3.641 MiB/s, ETA 0s
Checks: 7320 / 7320, 100%
Transferred: 756 / 756, 100%
Elapsed time: 12h34m52.3s
(isis) cartrite@localhost:/srv/disk1/isis3>
Thank You

I'm not sure how anyone should download the other folders though.

[cartrite]

A long time ago I was able to accomplish this running on Celestia which is a work in progress. This was done with Isis3 ,mro and mex files and probably Gdal too. https://www.youtube.com/watch?v=zmrJYZxJmtY&t=51s

I wish to start continuing my work and am thankful for your help.