Skip to content
This repository was archived by the owner on Mar 4, 2020. It is now read-only.

Complement Step does not update all objects #39

Merged
merged 14 commits into from
Apr 11, 2018

Conversation

nihadness
Copy link
Contributor

@nihadness nihadness commented Apr 4, 2018

Current behavior

  1. Objects (and their dependencies) are created and updated at the same time.
  2. While retrieving recently modified objects during Complement Step, system doesn't behave properly. Although some of the objects are handled correctly, some are missing in the destination instance. It can be because of 'http' connection problem or etc.

Expected Behavior after this PR

  1. Instead of creating, updating and handling dependencies of objects, all the recently created objects will be created in the destination. After that, recently modified objects will be updated without handling their dependencies.
  2. Instead of yielding items by trusting 'next' link of JSON response (which means if http fails once, no more objects will be retrieved because next link won't be in the JSON response), we can retrieve objects from uid_catalog page by page with the window of 500.

@nihadness nihadness added the bug label Apr 4, 2018
@nihadness nihadness self-assigned this Apr 4, 2018
@nihadness nihadness requested a review from xispa April 4, 2018 11:08
Inserting/ Updating new record to soup table should follow the steps
below:
1. If object's UID exists in the table, just update the path if
   necessary.
2. If object's UID doesn't exist in the table, try to insert it. If
   insert fails, it is because of duplicate 'path'. Wait for duplicate
   record to be updated and then insert the Object.
query["b_start"] = start_from
items = self.get_items(**query)
# Try one more time if items are not retrieved
if not items:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest to add two variables at the top of the class to make this more flexible. Sometimes, depending on the infrastructure, one might find useful to be able to set the maximum number of retries and the sleep time in seconds between 2 retries as well. I suggest to set the variables with the following values by default:

MAX_RETRIES=5
SLEEP_MS_RETRIES=5

@xispa xispa merged commit e01a550 into senaite:master Apr 11, 2018
@nihadness nihadness deleted the complement-step branch May 4, 2018 09:33
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Development

Successfully merging this pull request may close these issues.

2 participants