Allow for shallow updates

[KaSroka]

For verification purposes, it would be nice to init and fetch only versions from the manifest file.
So issuing:

west init
west update --shallow

or

west init --shallow
west update

Would create a minimal version of repository saving a lot of time for big projects.

[mbolivar]

For verification purposes, it would be nice to init and fetch only versions from the manifest file.

This is possible if you use a branch for the project revision and set the clone-depth attribute in the project.

However, if you use a SHA, you can't do this as easily, because not all Git servers let you fetch a SHA directly. GitHub is one of these servers.

One potential workaround is to add a new project attribute that says "the SHA in the revision field is in this branch over here", (#344) but that's a little tricky because the branch could get ahead of the SHA in the revision field, e.g. (for nrf) after merging a PR in the the zephyr project before updating nrf/west.yml to the new SHA. So I agree this would be nice, but it's not totally trivial.

[KaSroka]

Yes but clone-depth is useless if you want to have full repo for development and shallow for verification.

Also I'm getting this when using clone-depth: 1:

=== updating proj (some/dir):
--- proj: fetching, need revision master with --depth 1
error: unknown option `tags--depth=1'

So what would be nice to have an option to override clone-depth from west init or from west update.

[mbolivar]

Also I'm getting this when using clone-depth: 1:

This was fixed in master last week, sorry about that.

So what would be nice to have an option to override clone-depth from west init or from west update.

I understand and agree. You get what I'm saying about why that isn't really possible to do 100% effectively with SHA revisions, though, right?

[KaSroka]

Yes, I understand it and wonder if we can somehow make a fallback here - if it failed with limited depth try to do a full fetch or something like that.

[mbolivar]

I think something like a combination of specifying the upstream branch plus a fallback strategy would probably be needed, indeed. That makes the feature complex to implement (and test) and less predictable in its results (and thus less useful IMO), which is why it hasn't been a priority, but I do agree it's a valid use case.

[koffes]

I just closed #399 as a dupe of this issue. This should work as long as the git installation is >= 2.5.0 on both the client and server. Agree with @KaSroka that as long as we have a fallback, this would make a great feature.

[marc-hb]

One potential workaround is to add a new project attribute that says "the SHA in the revision field is in this branch over here"

For a repo with many branches this would be a useful and hopefully simple optimization even before looking at more complex shallow optimizations. #139 (comment)

BTW which refs does west clone and fetch by default? All branches or just a --single-branch default? I guess the former yet there seems to be no trace of that left in any branch or /refs/, it's a pity.

I feel like this has already been answered somewhere buried in the long #331 discussion but I'm afraid I just fell asleep trying to understand what @marc-hb wrote there at the time. It should probably be documented?

[marc-hb]

I think something like a combination of specifying the upstream branch plus a fallback strategy would probably be needed, indeed.

Just for fun if you're a git nerd (if not why are you reading this page?), here's another case where shallow optimization fails and needs some kind of CI/validation fallback; when too shallow and missing the merge base! thesofproject/linux@53b27a20cd5d45544c3

[mbolivar-nordic]

Hi, since multiple people are asking for this, I will try to prioritize it.

One potential workaround is to add a new project attribute that says "the SHA in the revision field is in this branch over here"

For a repo with many branches this would be a useful and hopefully simple optimization even before looking at more complex shallow optimizations. #139 (comment)

There was an RFC for this that I never got around to finishing. I will see about resurrecting it:

#344

One thing I just realized is that it's pretty useless for branches that may rebase, but that's not everybody, so it may be useful.

BTW which refs does west clone and fetch by default? All branches or just a --single-branch default?

It fetches everything, including tags. Information on getting the details in:

https://docs.zephyrproject.org/latest/guides/west/repo-tool.html#troubleshooting

I guess the former yet there seems to be no trace of that left in any branch or /refs/, it's a pity.

That's because we use refs/west/ as a staging ground for temporary fetches, and clean it up when we're done.

I feel like this has already been answered somewhere buried in the long #331 discussion but I'm afraid I just fell asleep trying to understand what @marc-hb wrote there at the time. It should probably be documented?

It's currently deliberately left as an implementation detail to try to avoid boxing us into a particular strategy.

[marc-hb]

There was an RFC for this that I never got around to finishing. I will see about resurrecting it: #344

Doh! How did I miss #344? Because it was closed? Thanks, it also happens to be much easier to read than #331 ... but still very long, IMHO better start a brand new issue that resumes where #331 left. It's possible because #331 did stop with at a fairly complete proposal.

One thing I just realized is that it's pretty useless for branches that may rebase, but that's not everybody, so it may be useful.

I recommend "volatile branch or volatile SHA1" as a ligher and better name than "branches-that-may-rebase" or "rewritable history". When I'm amending or discarding a git commit I'm conceptually not rebasing anything yet the relevant net effect on the SHA1s is the same. In fact I don't even need to use the "git rebase" command when amending the last commit - yet the relevant net effect is again the same: I just changed SHA1s / rewrote history.

"git rebase" is yet another poorly named git command because it describes only one use case. Should have been "git rewrite" or "git replay". So better not stretch this already bad name to even worse things like "rebasable branch".

Back to your point then yes: if the user tries to "catch" a specific SHA1 by pointing at a volatile branch that keeps jumping around the place then for sure it's likely to fail at some point. This is IMHO just PBKC and the best that can be done then is to provide a good error message. The whole revision-in idea assumes that the desired SHA1 is a parent of the revision-in. Need a good error message/hint when that's wrong in any case.

Or were you referring to something different?

Maybe let's discuss this in a new issue?

That's because we use refs/west/ as a staging ground for temporary fetches, and clean it up when we're done.

Yet west does NOT directly or indirectly invoke git gc, correct? I feel another déjà vu sorry.

[mbolivar-nordic]

Yet west does NOT directly or indirectly invoke git gc, correct? I feel another déjà vu sorry.

No, it lets the user policy on that rule the day.

[mbolivar-nordic]

Maybe let's discuss this in a new issue?

Yep, this is getting out of scope.

[carlescufi]

Now that GitHub supports fetching arbitrary references, the plan is to implement this in the near future.

[marc-hb]

Just FYI: Shallow cloning can create pull requests with one million commits #2556
thesofproject/linux#2556

[czeslawmakarski]

The current proposition for the implementation of shallow cloning in west.

0. Define the 'old approach' and the 'new approach'. Let the 'old approach' be the currently supported in west tool approach to clone git repo, while the new aproach is the following:

   $ git init

   $ git remote add origin <origin-url>

   $ git fetch --depth 1 origin <SHA>

   $ git checkout -b manifest-rev FETCH_HEAD

1. Add the 'shallow' field to the global, 'remotes' and 'project' sections, implicitly defined as 'false', i.e. (on the example of 'remotes' section):

  remotes:
    # nRF Connect SDK GitHub organization.
    # NCS repositories are hosted here.
    - name: ncs
      url-base: https://github.com/nrfconnect
      shallow: true
    # Third-party repository sources:
    - name: zephyrproject
      url-base: https://github.com/zephyrproject-rtos
      shallow: false
    - name: throwtheswitch
      url-base: https://github.com/ThrowTheSwitch
      # No 'shallow' field present, so shallow=false in this case.

2. Add the '--shallow' command line flag to every west command which initializes the git repositories (i.e. 'west init' and 'west update').
   When the '--shallow' flag shall be passed, all implicitly defined 'shallow' fields shall be treated as true, i.e. if there is no 'shallow' field present in the remotes section, it is treated as 'true'.

3. The 'shallow' field of the sections 'global', 'remotes', 'projects' overwrite each other. The latter takes precedence if defined explicitly.

4. When fetching the repositories, if the 'shallow' field is set to true (explicitly or implicitly), try fetching with the new fast scheme, if it fails, fall back to the classic one and emit a warning.

The following pythonish pseudocode explains what should happen when west tool shall try to clone the git repository:

def west_clone_repo(is_shallow_flag_passed, global_field_shallow=None, remote_field_shallow=None, project_field_shallow=None):
    is_shallow_clone = is_shallow_flag_passed
    if global_field_shallow is not None:
        is_shallow_clone = is_shallow_clone and (global_field_shallow == True)
    if remote_field_shallow is not None:
        is_shallow_clone = is_shallow_clone and (remote_field_shallow == True)
    if project_field_shallow is not None:
        is_shallow_clone = is_shallow_clone and (project_field_shallow == True)

    if is_shallow_clone:
        try:
            clone_with_new_approach()
        except e:
            warning("New approach has failed, fallback to the old approach")
            clone_with_old_approach()
    else:
        clone_with_old_approach()

[marc-hb]

I don't understand the rationale for calling shallow cloning "new approach" and unlimited depth "old approach": is there any other difference between them besides the --depth? Moreover some users will still clone some repos with unlimited depth even after this option is available, so the existing cloning approach will never get "old".

Must the depth really be hardcoded to 1? With a bit of depth it's much easier to make the difference between an original commit and its cherry-picks and generally easier to "locate" the current HEAD without looking at SHAs. It also allows per-commit CI checks for small enough pull requests.

How does this interact with the existing clone-depth attribute?

Yes but clone-depth is useless if you want to have full repo for development and shallow for verification.

You can git fetch --unshallow

I'm also wondering what happens when you repeatedly fetch a moving repo with a depth of 1... does that leave gaps? Admittedly a pure git, not-west question but related to this.

[carlescufi]

I don't understand the rationale for calling shallow cloning "new approach" and unlimited depth "old approach": is there any other difference between them besides the --depth?

I think @czeslawmakarski means fetching a particular ref, note this:
git fetch --depth 1 origin <SHA>
vs what west actually does today:
git fetch -f --tags -- <remote> 'refs/heads/*:refs/west/*'
i.e. today west fetches all refs/heads/ instead of a particular ref.

[marc-hb]

Thanks @carlescufi for reminding me that shallow cloning of a SHA depends on fetching SHAs directly which I think is not designed yet. Implementing the former before the latter would feel like jumping the gun to me.

Fetching SHAs directly seems simple but it's not, see "fallback" discussion above and long discussions in #344

PS: with some repos, fetching SHAs directly can offer large performance improvements even before shallow cloning.

[tejlmand]

1. Add the 'shallow' field to the global, 'remotes' and 'project' sections, implicitly defined as 'false', i.e. (on the example of 'remotes' section):

Is this actually something we should specify in a manifest file ?
I mean

• developer A might want everything
• developer B might want everything on project X, Y, and Z and shallow on the rest
• CI might want shallow on everything

A project maintainer cannot in any sensible way determine the right use-case for everyone, making it hard to write a proper manifest file.
Also, users should not be expected to modify a manifest file, just because they want a specific behavior for project X.

To me, this looks like something that should be a west configuration setting, perhaps like (all settings are True / False):

west config manifest.shallow True                     # Shallow clone for this west workspace,
west config manifest.remote.<name>.shallow True       # Shallow clone for this remote
west config manifest.project.<name>.shallow False     # Shallow clone for this project
west config manifest.project.<name>.shallow True

Project setting takes precedence over remote setting, remote setting takes precedence over workspace setting.

That way a user wanting to do shallow cloning on all projects in remote MyRemote except projB can do:

west config manifest.repo.MyRemote.shallow True
west config manifest.project.projB.shallow False

The decision on using shallow cloning is probably quite constant, based on the users preference, and not a use-case where a user want to do shallow update today, and tomorrow decides to update everything, just to do shallow cloning the third day.

2. Add the '--shallow' command line flag to every west command which initializes the git repositories (i.e. 'west init' and 'west update').

I think, in this particular case, a west config will probably be more suitable.

3. The 'shallow' field of the sections 'global', 'remotes', 'projects' overwrite each other. The latter takes precedence if defined explicitly.

Placing this in manifest actually means we start imposing a specific workflow.
Of course there could be use-cases where a specific project/repo contains binaries (not good in git), and could benefit from defaulting to shallow cloning, but in such case, being able to specify it in manifest on a per-project basis should be sufficient.

---

With the above suggestion, the workflow for a user wanting to do shallow clone on everything (example CI) could then be:

west init
west config manifest.shallow True
west update

Edit: repo changed to remote

[czeslawmakarski]

Is this actually something we should specify in a manifest file ?

The intention was rather to state the possibility of specifing such fields in the manifest file, not the obligation. The proposed west config approach is also feasible and in some cases would be even more straightforward to use.

I envisioned the possibility to place shallow fields in the manifest if, for example, some custom 3rd-party remote does not support the direct SHA fetching, so it could be 'marked' by such to always enforce full cloning.

Also please note, that in your approach one has to call first west init to subsequently call west config manifest.shallow True, so the original git repository (containing the original west.yml) shall be cloned the old-way. If someone could pass the west init --shallow than even the original git repo shall be fetched faster - and in many cases the original git repo is the one which fetches for a long time.

[tejlmand]

The intention was rather to state the possibility of specifing such fields in the manifest file, not the obligation.

yes, I understand. But with only a manifest possibility or the --shallow flag that is command specific, and not project specific, then the manifest possibility becomes the only way.
Having the global, remote, and project granularity bears the risk of leading users into a sub-optimal workflow of modifying the manifest file directly for their desired behavior.

By only having shallow available for the projects in the manifest, then we can handle the git binary repo case, while at the same time lead the users to a better workflow, as I don't believe a user will start adding shallow: True on all projects in a manifest 😉

Also please note, that in your approach one has to call first west init to subsequently call west config manifest.shallow True, so the original git repository (containing the original west.yml) shall be cloned the old-way.

except for the fact that the first time a user updates the manifest repo using git pull or git fetch he will fetch everything unless he knows the corresponding git command.

That is:

west init --shallow
west update

git -C<manifest-repo> pull

will still fetch everything, just a bit later.
So user anyway needs to know a bit more on git.

So maybe the shallow init usecase is better handled using git, that is:

git clone --depth 1
west init -l <manifest-repo>
west config manifest.shallow True
west update

cause that will then correspond to later manifest fetches using:

git -C<manifest-repo> fetch --depth 1
git -C<manifest-repo> checkout FETCH_HEAD

or what you need to update to.

Note: if having a --shallow parameter, I believe this makes the most sense on the west init command.

[marc-hb]

I envisioned the possibility to place shallow fields in the manifest if, for example, some custom 3rd-party remote does not support the direct SHA fetching, so it could be 'marked' by such to always enforce full cloning.

I find the idea of using the manifest to tell west about the various allow*SHA1InWant capabilities[*] interesting because I'm afraid it's not possible to query these dynamically from the server. However this per remote clone information is related but very different from shallow cloning preferences which are user-specific.

[*] https://git-scm.com/docs/git-fetch-pack

Having the global, remote, and project granularity bears the risk of leading users into a sub-optimal workflow of modifying the manifest file directly for their desired behavior.

Like @tejlmand, I think it's a bad idea to add user preferences in the manifest because it will pollute the local "git status". Very soon after someone commits their personal preferences and shares them (intentionally or not; many people use the evil "git commit -a") and then everyone has to either pollute their git status too or have very long arguments about default values that not everybody can ever agree on because different people and different bots have different use cases.

.west/config seems like a good, possibly fine grained place. A --depth n argument seems OK too and it wouldn't even need any documentation: could be just passed to git as is. No need to be inconsistent with git and have a --shallow option; the existing git terminology in this area is very clear (for a change).

The intention was rather to state the possibility of specifing such fields in the manifest file, not the obligation.

IMHO good tools and languages are better defined by how they stop users from shooting in each other's foot than by how many possibilities (and lack of test coverage) they have.

So user anyway needs to know a bit more on git.

If you don't know at least "a bit" about git then you should not get close to a west manifest, you should not be setting up any automation and you should probably not get close to shallow cloning either cause it's a hack. I've answered many more git questions than than I asked in my life yet this shallow cloning weirdness cost me days of troubleshooting and investigation: thesofproject/linux#2556

tl;dr: these are relatively advanced git optimizations and topics: their user interface can assume basic git knowledge.

and in many cases the original git repo is the one which fetches for a long time.

Good news: in this particular case you can already shallow clone the manifest repo and save a lot of time without any new west feature.

Speaking of automation, an alternative, 100% safe and conceptually simple optimization is a mirror/cache. git makes mirroring very easy. A bit off-topic sorry: how convenient is it to override remote.url-base, I mean is there a way that does not require changing the manifest? This has allow*SHA1InWant implications.

[tejlmand]

So user anyway needs to know a bit more on git.

If you don't know at least "a bit" about git then you should not get close to a west manifest, you should not be setting up any automation and you should probably not get close to shallow cloning either cause it's a hack.

Well, even less experienced west or git users are expected to do west init / update for their development.
west does a lot of great things easier for less experienced users, and should continue to do so.

Shallow cloning is probably better kept to more experienced users, which is exactly why I showed that using plain git an experienced user can shallow clone the manifest repo today, thus making the need for a west init --shallow small.
We avoid users doing west init --shallow when they shouldn't, by not having that argument in the first place.

[mbolivar-nordic]

shallow cloning of a SHA depends on fetching SHAs directly which I think is not designed yet. Implementing the former before the latter would feel like jumping the gun to me.

I've posted a PR which allows us to fetch SHAs directly, to start progress on this: #475

[marc-hb]

@mbolivar-nordic just abandoned #475 and I'm not sure why.

I think giving #475 and #344 a name should help clarify concepts, I suggest "narrow". I see roughly 4 classes of git clones in general use (with and without west)

• LARGE
  git clone; git fetch --tags
  Slowest, biggest and best for airplane trips mentioned in Fetch even SHA revisions directly #475. Current west default.

• git's DEFAULT
  git clone; git fetch
  All remote branches. "Opportunistic" tags only.

• NARROW
  git clone --no-tags --single-branch
git fetch ref_name_or_SHA
  Maximum optimization without breaking any git command. Savings can be huge in projects with many branches.

• DEPTH1 a.k.a. "shallow"
  git clone --depth 1
  Maximum savings and most git commands are broken by design. For CI and repetitive throw-away clones only, best avoided for persistent clones. In git's documentation "shallow" means "truncated", it does not mean depth 1. Git terminology can be confusing enough, better not diverge from it when not necessary.

Thanks to custom refspecs, intermediate --depth, different clone+fetch combinations, different git versions etc. the reality is not just these 4 points but actually a spectrum. However I think these 4 points should capture most real-world use cases and that these are the concepts west should present, not just DEPTH1.

The big difference fetch-pack's uploadpack.allow*SHA1InWant makes: it simply forbids NARROW and DEPTH1 and forces a fallback on LARGE or DEFAULT (with a warning?) when what you want is a SHA1 instead of a tag or branch. #344 explored a revision-in trick to fall back on NARROW.

[mbolivar-nordic]

@mbolivar-nordic just abandoned #475 and I'm not sure why.

Now reopened

[mbolivar-nordic]

@marc-hb west actually does not use git clone ever. It uses git init followed by git fetch --tags. Whether that fetch uses a single revision or not depends on whether the revision might be a SHA. If not, it fetches the revision only (narrow). If so, it syncs the entire remote ref space (large? wide? full?).

I think we are headed towards a world with the following possibilities:

• FULL: git fetch --tags just-one-ref + fallback to git fetch --tags all-the-refs on error (will be the new default)
• NARROW: git fetch just-one-ref with no fallback
• DEPTH1: git fetch --depth 1 just-one-ref with no fallback

[marc-hb]

west actually does not use git clone ever.

I was starting from plain, familiar git use cases.

I think we are headed towards a world with the following possibilities:

I mentioned DEFAULT because it's the git default. I agree the other 3 seem enough for west, I actually suggested that in #475

[mbolivar-nordic]

I mentioned DEFAULT because it's the git default.

Gotcha, I misunderstood you. I thought you were saying this was the west default.

I agree the other 3 seem enough for west, I actually suggested that in #475

Awesome, that makes more sense now. I was confused about why there seemed to be more possibilities here than there. I'm totally with you.

Seems like we're on the same page then and the FULL/NARROW/DEPTH1 breakdown in #319 (comment) is what we'd like to have.

Comments from anyone else following this thread?

[mbolivar-nordic]

@czeslawmakarski I would like your help in testing the results of this optimization in your environment.

Here are two scripts, fetch_full.sh and fetch_depth1.sh: https://gist.github.com/mbolivar-nordic/f610f1234c2cb8a6cb716c75e44b6c8f

Could you please run them in the environment you are creating your workspaces in as follows:

$ ./fetch_full.sh 2>results_full.txt; ./fetch_depth1.sh 2>results_depth1.txt

Then share the results_full.txt and results_depth1.txt files with me? This is a simulation which gets the code for NCS v1.5 with and without --depth 1. 10 trials are conducted.

In my environment (1 GbE network connection and an SSD), I get somewhat surprising results:

• results_full.txt: real time μ=68.5sec σ=5.9sec
• results_depth1.txt real time μ=42.9sec σ=1.7sec

This is "only" a 37% improvement: significant to be sure, but not an earth-shattering improvement.

I'm curious if the speedup in your environment is worth it to you or if setting up some caching as discussed elsewhere would be better for solving the performance problems you're having.

[marc-hb]

I tried something slightly different: I hardcoded --depth 1 into src/west/app/project.py and ran that.
https://gist.github.com/marc-hb/85d7bad9f04ededa4f46273004545819
That produced some errors but only after cloning.

With a fast, business network connection it made little difference. Most of the difference (10s) was when west init clones zephyr.git. Does any CI need to repeatedly clone on a slow connection, without any local cache/mirror? (which would make a much bigger difference on a slow network connection)

I think this is a combination of:

• git has good compression. Only repos with very large churn benefit from --depth 1, small patches cost nothing.
• A lot of time is not spent downloading but checking out / decompressing the code.
• west runs one git command at a time.

Then I ran a few du commands:

2.5G grand total with full clones including --tags

850M all .git/ folders - full clones
250M all .git/ folders, single branch depth1
=> -600M total disk savings

400M zephyr/.git
60M zephyr/.git single branch depth1
=> 340M saved with just zephyr.git That's more than half the savings without any code change in west.

The next biggest repos:
espressif: -70M saved
openthread: -60M saved
stm32: -30M saved

[czeslawmakarski]

@mbolivar Thanks for the scripts - I'll give them a go and shall come back with the results.

[czeslawmakarski]

HI @mbolivar-nordic,

First, I've done the test on my local machine (ISP Downstream about 300 Mbps, NVME storage):
and the results are:
Full: 106.99s w/ stdev 13.88s
Depth1: 78.18 w/ stdev 7.09s
This gives about 26% of time boost, similar to your case.

Then we've run the tests on our Jenkins environment and the results were different on different nodes - ranging from 16% to 58% - you can see the image included.

AFAIK the nodes were executing only your bash scripts and the same time, so probably the reason could be in the differences in ISP bandwidth allocation to the corresponding nodes.

Tagging @dawidprzybylo for visibility.

[marc-hb]

Jenkins has a number of git optimizations:
https://plugins.jenkins.io/git/#wipe-out-repository-and-force-clone
https://plugins.jenkins.io/git/#user-content-honor-refspec-on-initial-clone (shallow cloning)
https://support.cloudbees.com/hc/en-us/articles/115001728812-Using-a-Git-reference-repository

Shallow cloning and reference repositories won't work for west subprojects (only for git submodules) but if Zephyr - the largest repo by far - is used as the manifest then it would already save a lot.

Not cloning from scratch every time and cleaning the repo instead should Just Work. That's the most basic and obvious git thing to do.

[mbolivar-nordic]

OK, I did a bit more benchmarking.

So far I concluded from this that, at least in my environment:

• the "real win" for performance is to use an existing workspace as a cache when creating a new one
• --depth 1 may not be much of a win
• more testing is needed

---

This time I compared:

• NARROW updates as defined here
• NARROW + --depth 1
• using a pre-fetched workspace as a cache for initializing missing repositories during west update (it would be straightforward to prepare a Docker image for CI which contains such a cache)

I used this west branch during testing: https://github.com/mbolivar-nordic/west/tree/update-optimizations (commit 867b42).

With this branch, you can:

• use an existing workspace as a cache via a new west update --path-cache /my/cache/workspace
• do a NARROW update with west update --narrow
• add a --depth 1 fetch via west update --narrow --o=--depth=1

---

WARNING: THE SCRIPT IN THE FOLLOWING LINK DELETES GIT REPOSITORIES WHILE IT RUNS. DO NOT RUN IT IN A WORKSPACE YOU CARE ABOUT.

I also wrote this benchmarking script: https://gist.github.com/mbolivar-nordic/4e50065a724ed579eb0b430f26473736

---

You can run it safely like this, after editing the EXISTING_WORKSPACE variable in the script to suit your environment.

mkdir west-update-testing
cd west-update-testing
git clone https://github.com/zephyrproject-rtos/zephyr   # can use sdk-nrf if you want
west init -l zephyr # or nrf
./benchmark.sh

Results go to the shell running benchmark.sh. Logs go to benchmark.log (or LOG_FILE if set).

Here are box-and-whisker plots of my results on the same 1 GbE network + SSD environment from earlier:

As you can see, the results are consistent between runs, and using a cache is the clear winner in terms of performance. Paradoxically, --depth 1 takes a bit longer than just running a plain narrow fetch, perhaps because of extra git overhead. I'm guesing that network and file system overhead dominate with no cache, and asking git to do extra work adds a bit of time. But I need to reproduce a bit more.

I'm also curious if others (@marc-hb , @czeslawmakarski ) can reproduce these results.

I understand that maintaining a cache is a burden for CI environment maintainers, though, so I will prepare a PR with all of these options available once I write some more test cases. I still think that even a stale cache which is missing some git refs is likely to be much faster than either --narrow or --narrow -o=--depth=1.

Thanks for everyone's patience on this. I think we are closing in on a good solution.

[mbolivar-nordic]

Final benchmark results after re-running with a few new options for apples/apples comparison:

TL;DR: even with a fast network connection, caching is the clear performance win for updating a Zephyr workspace.

Shallow updates are slower than using a cache, even if the cache is 'stale' and contains very out of date clones of the workspace's projects. Oddly, shallow updates are slower than just fetching the exact project revision at full depth (at least in this workspace + my environment).

Explanation of each box-and-whiskers plot (n=10 in each case):

• 'new-cache': west update --path-cache FOO where FOO is an existing workspace that contains all the SHAs needed by the update
• 'old-cache': west update --path-cache BAR where BAR is an existing workspace that is "over 6 months old", containing only the SHAs available as of Zephyr v2.4.0, released 27 September 2020
• 'narrow+old': west update --path-cache BAR --narrow, i.e. 'old-cache' but skipping fetches for irrelevant branches and tags
• 'narrow+depth1': west update --narrow -o=--depth=1, i.e. no caches and shallow updates as requested by this issue
• 'narrow': west update --narrow
• 'no-opts': plain west update

In each case, zephyr is the manifest repository and commit 72595a334cdbcc964aeb9bdba37b84e21ac08b07 is used. I used my home's 1GbE<->fiber connection on an otherwise quiet network. (I used Zephyr both for wider applicability and since it's a larger workspace than NCS.)

I've posted a PR with the new west options used above: #496

Please review and of course feel free to run your own tests to determine what is best for your environment.

[czeslawmakarski]

I've been experimenting with the cached version also, and my results were similar to yours - much faster than the 'shallow' cloning. The only problem we can envision is that the old-cache version of the NCS (most probably latest 1.x.0 release) would be needed to be included in the Docker image.

[mbolivar-nordic]

Final benchmark results

Famous last words 😄

@carlescufi asked me why the results are so different between the above benchmarks and the rough script I posted about a month ago, which showed a ~37% improvement for NCS if using --depth=1.

I re-ran the benchmarks with NCS:

TL;DR: caching is still the clear winner, but shallow updates do make a difference for our downstream Zephyr distro. However, even an old cache that lags behind by 2 releases is still 20% faster than shallow updates when combined with --narrow, and a new cache is way faster, as before.

Explanation of each box-and-whiskers plot (n=10 in each case):

• 'new-cache': west update --path-cache FOO where FOO is an existing workspace that contains all the SHAs needed by the update
• 'old-cache': west update --path-cache BAR where BAR is an existing workspace that contains the objects from NCS v1.4.0, released 30 Oct 2020
• 'narrow+old': west update --path-cache BAR --narrow, i.e. 'old-cache' but skipping fetches for irrelevant branches and tags
• 'narrow+depth1': west update --narrow -o=--depth=1, i.e. no caches and shallow updates as requested by this issue
• 'narrow': west update --narrow
• 'no-opts': plain west update

In each case, sdk-nrf is the manifest repository and commit 61fa66c78 is used. As before, I used my home's 1GbE fiber connection on an otherwise quiet network.

These results are consistent with my earlier NCS results, so NCS really is different.

Interestingly, much of the performance difference just comes from zephyr itself:

$ time west update --narrow -o=--depth=1 zephyr
[...]
real    0m6.658s
user    0m3.252s
sys    0m1.294s
$ time west update zephyr
[...]
real    0m21.744s
user    0m26.042s
sys    0m4.732s

[mbolivar-nordic]

The only problem we can envision is that the old-cache version of the NCS (most probably latest 1.x.0 release) would be needed to be included in the Docker image.

@czeslawmakarski that is not 100% certain IMO. You could keep it out of the image but volume-mount it into the running container, since updating from a cache is a read-only operation from the cache's perspective.

[mbolivar-nordic]

you could keep it out of the image but volume-mount it into the running container,

This assumes you've got a bare metal server or some other way to do persistent storage, of course.

It also has the advantage that you can define a cron job to run on the weekend that keeps the cache fresh.

[mbolivar-nordic]

Closed by #496.

[marc-hb]

If you have any interest in cloning performance then try cloning with west update --fetch-opt=--filter=tree:0. It's very, very promising: high speedup without any of the git breakages from shallow clones.

• Consider adding config option for treeless clones (--fetch-opt=--filter=...) #638
• Support for Treeless clones actions/checkout#1152 (upvote this)

cc: @aborisovich, @wszypelt , @keqiaozhang, @greg-intel