feat: introduce the copy by chunk for replication #202
Conversation
|
@goharbor/all-maintainers Hi, please review this proposal, thanks! |
|
|
||
| #### Phase 2 Implementation | ||
|
|
||
| The key point of phase 2 is **`Breakpoint & Resume`**. |
There was a problem hiding this comment.
because we decided to do phase 1 at this stage, and the details in the phase 2 have not been fully discussed, it may not be very appropriate.
chlins
force-pushed
the
feat/replication-chunk-transfer
branch
from
13d4e87 to
9cca0f1
Compare
| ```go | ||
| type Client interface { | ||
| // PullBlobChunk pulls the specified blob, but by chunked | ||
| PullBlobChunk(repository, digest string, start, end int64) (size int64, blob io.ReadCloser, err error) |
There was a problem hiding this comment.
is size needed in the return value since it can be calculated via start and end?
There was a problem hiding this comment.
yes, but just return this as unify with original PullBlob method.
| *update policy* | ||
|
|
||
| ```rest | ||
| PUT /replication/policies |
There was a problem hiding this comment.
I believe it has the policy ID in the URI?
There was a problem hiding this comment.
Sure, hide the existed column.
|
|
||
| **Scope:** | ||
|
|
||
| Chunk resuming crossing the job/execution/policy and multiple times execution(e.g. job retry), cache chunk location and last end range in redis. |
There was a problem hiding this comment.
I think this is cool but a little unsure from the cost-effective perspective
|
|
||
| The retry for chunk adopts the same strategy with blob, default 5 times and can be configured by environment. | ||
|
|
||
| #### Phase 2 Implementation |
There was a problem hiding this comment.
IMHO, If we want to implement phase 1 in next minor release and spark more conversations around phase 2, would it be better we split them into two proposals?
There was a problem hiding this comment.
I think we can keep the phase2 idea in this proposal for understand, and when we plan to do phase2 in the future, then we can re-design the entire phase2 and push a new proposal for review.
|
|
||
| The key point of phase 2 is **`Breakpoint & Resume`**. | ||
|
|
||
| From the process of chunk API, we need to store the location for next chunk push and the last pushed chunk end range, so we need to define common interface for easily integration and adapter in the future. |
There was a problem hiding this comment.
I need more explanation for the concepts like breakpoint and location to understand the idea....
There was a problem hiding this comment.
Just initialize idea, we can discuss this for more details when we do phase2.
|
@chlins @wy65701436 the Phase1 and Phase2 are not defined at least I didn't find anything. |
|
@chlins my confusion comes from "Phase 1" and "Phase1" |
chlins
force-pushed
the
feat/replication-chunk-transfer
branch
from
9cca0f1 to
fc9140a
Compare
Signed-off-by: chlins <[email protected]>
chlins
force-pushed
the
feat/replication-chunk-transfer
branch
from
fc9140a to
fe62c84
Compare
Signed-off-by: chlins [email protected]