storage: CopierFrom large files, progress resets and copy never finishes

[miguelb]

We are copying a large file from a regional bucket to a multi-regional bucket, and seeing some strange behaviors.

• If you do not specify the context timeout, the copier goes into an endless error/retry. Before we noticed this issue, we went through a Petabyte of egress in 1-2 weeks.
• We then added a timeout, copying continues to retry until it reaches the timeout period.
• We also added a retry strategy of "never" but it still to continues to retry.

Client
GCS

Environment
golang:1.19.1-alpine

Go Environment
1.19.1

Code

func newGCPClient(ctx context.Context) (*storage.Client, error) {
	options := []option.ClientOption{option.WithCredentialsJSON([]byte(config.Env.GoogleCredentials))}
	return storage.NewClient(ctx, options...)
}

func CopyFile(fromBucket string, fromKey string, toBucket string, toKey string) error {
	ctx := context.Background()
	client, err := newGCPClient(ctx)
	if err != nil {
		return err
	}
	defer client.Close()

	src := client.Bucket(fromBucket).Object(fromKey)
	dst := client.Bucket(toBucket).Object(toKey)

	ctx, cancel := context.WithTimeout(ctx, 5*time.Minute)
	defer cancel()

	copier := dst.CopierFrom(src)
	copier.ProgressFunc = func(copiedBytes, totalBytes uint64) {
		fmt.Println(fmt.Sprintf("progress: %d / %d", copiedBytes, totalBytes))
	}
	if _, err := copier.Run(ctx); err != nil {
		return err
	}
	return nil
}

func main() {
	config.LoadVariables()
	fromBucket := ""
	fromKey := ""
	toBucket := ""
	toKey := ""
	if err := CopyFile(fromBucket, fromKey, toBucket, toKey); err != nil {
		fmt.Println("ERROR: ", err)
		return
	}
	fmt.Println("SUCCESS")
}

Expected behavior
Progress increments and eventually finishes

Actual behavior
Progress seems to reset and it never finishes

Here are logs of the progress:

progress: 1862270976 / 8209196233
progress: 1904214016 / 8209196233
progress: 1929379840 / 8209196233
progress: 2231369728 / 8209196233
progress: 2038431744 / 8209196233
progress: 1551892480 / 8209196233
progress: 2298478592 / 8209196233
progress: 1929379840 / 8209196233
progress: 2449473536 / 8209196233
progress: 2021654528 / 8209196233
ERROR:  context deadline exceeded
Exiting.

Also, if we run the same code on two GCS buckets that are both multi-regional. The copy happens almost immediately.

Here are the logs

progress: 8209196233 / 8209196233
SUCCESS
Exiting.

[tritone]

Hi @miguelb , thanks for filing the issue. Could you let us know what release of the storage client you are using? That will help us reproduce. Also, if I understand correctly, this particular use case (regional to multi region bucket) never worked; it's not something that started at a certain date or after updating the client version -- can you confirm?

[miguelb]

This is our first time running into this issue, this is a new codebase. It doesn't work if buckets are different region configuration. If the GCS buckets have the same region configuration, copy happens immediately between the buckets.

cloud.google.com/go/storage v1.27.0

[miguelb]

I ran our test against previous versions and cloud.google.com/go/storage v1.24.0 does work. Its slow, but eventually does finish.

Here are the logs:

progress: 2759852032 / 8209196233
progress: 4236247040 / 8209196233
progress: 5494538240 / 8209196233
progress: 7004487680 / 8209196233
progress: 8209196233 / 8209196233
SUCCESS
Exiting.

[tritone]

Thanks very much for the details and for confirming that this worked in an earlier library version. We made a major refactor of the client internals in v1.25.0 so it's likely that there was a bug introduced at that point for your specific use case.

I'll work on a repro and fix for the issue.

[tritone]

We found a bug that was introduced in Copier during the refactor that was preventing the Copy operation from completing when the object is large enough to require multiple requests to the API. We've merged a fix which will go out with the next release of storage (sometime next week). Thanks again for bringing this to our attention, and apologies for the inconvenience.

We'll also work on expanding our integration test coverage of this case. If you have concerns about the cost of data egress you incurred due to this bug, please contact your account manager and have them get in touch with me (they can find me internally via my github handle).