Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NMRL-299 Replacement of FileField by BlobField #100

Merged
merged 8 commits into from
May 26, 2017

Conversation

Espurna
Copy link

@Espurna Espurna commented May 25, 2017

This pull request replaces all FileFields in the system to FileFields from Blob library.
An upgrade util has been created in order to migrate the content in those old FileFields to the new Blob ones.

Steps to migrate:

  • If you have a bushy blobstorage: Just run the upgrade step and then pack the database.
  • If you have a lawn blobstorage: First migrate it following this, then run the upgrade step and pack.
  • In some cases, blobstorage has both bushy and lawn layouts at the same time. This couldn't be possible and gives an error during the packing or blobstorage migration process. The best way to solve this could be to delete one of the files types, and make sure that .layout file is in accordance with your selection. Then follow the previous steps.

The content_types with FileFields that has been migrated are:
"Attachment",
"ARImport",
"Instrument",
"InstrumentCertification",
"Method",
"Multifile",
"Report",
"SamplePoint"

Warning! While the migration process goes as expected, the following wired incident is happening:
Before migration:

  • filestorage -> 9.1Gb
    -- Data.fs -> 4.6Gb
  • blobstorage -> 7.2Mb

After migration:

  • filestorage -> 9.1Gb
    -- Data.fs -> 4.6Gb
  • blobstorage -> 4.1Gb

After packing:

  • filestorage -> 5.1Gb
    -- Data.fs -> 465.1Mb
  • blobstorage -> 54.9Mb

The files are being duplicated even if in upgrade/utils.pyL173 the migrator is deleting the file in filestorage... WHY?!

UPDATE Packing will solve this.... But packing fails...

2017-05-25 19:56:56 ERROR ZODB.DB packing
Traceback (most recent call last):
  File "/home/pau/dev/naralabs/nmrl/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/DB.py", line 812, in pack
    self.storage.pack(t, self.references)
  File "/home/pau/dev/naralabs/nmrl/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/blob.py", line 800, in pack
    self._packUndoing(packtime, referencesf)
  File "/home/pau/dev/naralabs/nmrl/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/blob.py", line 755, in _packUndoing
    remove_committed(filepath)
OSError: [Errno 21] Is a directory: '/home/pau/dev/naralabs/nmrl/zinstance/var/blobstorage/0x00/0x00'
2017-05-25 19:56:56 ERROR Zope.SiteErrorLog 1495735016.730.846204863413 http://localhost:8080/Control_Panel/Database/main/manage_pack
Traceback (innermost last):
  Module ZPublisher.Publish, line 138, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 48, in call_object
  Module <string>, line 3, in manage_pack
  Module AccessControl.requestmethod, line 70, in _curried
  Module App.ApplicationManager, line 403, in manage_pack
  Module ZODB.DB, line 812, in pack
  Module ZODB.blob, line 800, in pack
  Module ZODB.blob, line 755, in _packUndoing
OSError: [Errno 21] Is a directory: '/home/pau/dev/naralabs/nmrl/zinstance/var/blobstorage/0x00/0x00'

UPDATE2
https://stackoverflow.com/questions/25733112/plone-migrate-blob-data-to-bushy-layout-ioerror-errno-21
https://stackoverflow.com/questions/5604321/how-and-why-to-set-the-layout-type-for-plone-app-blob-lawn-v-bushy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants