ibazel watch stops working after 2 refreshes

[alexeagle]

Originally reported at:
angular/angular#21770

Has been observed by several engineers on the Angular team.

[achew22]

Is it consistent on every other save it stops working? Is it consistent across editors? The other report says OS X (which I don't have one of unfortunately), does it happen on your linux workstations the same?

[IgorMinar]

I'm able to repro it consistently across any shell/editor on MacOS. All of the repros of this that I'm aware were on MacOS.

[achew22]

I have a work lap'n'top that is a mac. I'll test it out on there when I get back to the states. Sorry for slow response on this one -- it'll take me a couple of days to get a test rig set up.

[alexeagle]

No problem, we're in this for the long term. +1 that I haven't observed it on Linux. Sorry this project is so platform-finicky

[clintharrison]

Has anyone had a chance to look into this further? I'm running into the same issue, but if anyone has made any progress I'd appreciate not duplicating that work.

[jjudd]

We're running into this on Ubuntu 16.04. We're going to dive in and attempt to fix it in the next few weeks. Anyone have any additional info on this? We think it might be related to the open files/inotify watches errors issues in this project, but that's a guess right now.

[borkaehw]

We have dived into this issue for pass few days on Ubuntu 16.04. It turned out we found a bug on our custom version of bazel-watcher, it should be unrelated to the issue here.

We are not able to reproduce the problem on Linux, it seems to happen only on macOS. @clintharrison What OS are you running?

We appreciate if anyone can provide more information.

[clintharrison]

I'm also on macOS, 10.12.6 currently. I unfortunately haven't had time to look into this, so I don't have any more details to provide :\

[joprice]

This may affect adoption especially for web projects, where a livereload workflow is expected. Is there a working commit to use before this issue was introduced?

[jjudd]

@joprice which OS are you using? We spent quite a bit of time trying to reproduce this on Ubuntu. We found a bug in code we wrote that isn't on master, but otherwise couldn't reproduce this issue.

[joprice]

I'm on an osx 10.12.6 using bazel 0.15.2

[joprice]

Not sure if it's related, but I added a listener for errors on SourceFileWatcher and I see the following error after the first update:

Error writing success to stdin: write |1: broken pipe

go func() {
    for err := range handler.SourceFileWatcher.Errors {
      fmt.Fprintf(os.Stderr, "Error %s\n", err)
    }
  }()

[joprice]

I just noticed that changes to build files continue to trigger. Perhaps it has to do with how many files are being watched? I ran sourceQuery and buildQuery locally and got 10x more files for sourceQuery.

[clintharrison]

In my testing on a single-file py_binary, I am reproducing this when I edit the file with vim's writebackup enabled. With nowritebackup, ibazel receives WRITE events from fsnotify and rebuilds. With writebackup on, there will be a single RENAME event, after which modifications fail to trigger a rebuild.

[joprice]

Setting nowritebackup works. I'm still confused why the BUILD file continues to work under the same conditions though.

[achew22]

@joprice, can you tell me what version of iBazel you're using? That sounds like a very old bug (#10) that we fixed. Vim writes out a temporary file and mv's it into place instead of opening the file and writing it since mv operations in UNIX are atomic. I was under the impression that PR fixed it.

If you wanted to add a test to demonstrate this, it would be pretty easy to do in https://github.com/bazelbuild/bazel-watcher/blob/master/e2e/simple/simple_test.go

[clintharrison]

@achew22 I have this happening on master. Since macOS uses the kqueue-based implementation of fsnotify, it behaves differently. There is a fsnotify.Rename event, not a Remove one. Unfortunately, adding Rename to source_event_handler is not quite a fix, because the re-add of the original file will run before the file exists (so you get "no such file or directory").

[achew22]

@clintharrison, we are now in a much better world CI wise where we actually have a mac machine we can run tests on through BuildKite/Travis. Is that something you would be able to reproduce in a test case?

[joprice]

I’m using a locally built version off master.

[achew22]

I think this is fixed now and is released to NPM. homebrew is in review.

[flolu]

I still have this issue 😮 (Ubuntu 19.10)

[achew22]

@flolu can you construct an example repo/testcase that this fails consistently in?

[flolu]

@achew22 Unfortunately not until now. For the moment it seems to occur randomly.

But it seems to occur often when compiler has thrown errors and I fixed them.
Then I get something like Error writing success to stdin: write |1: broken pipe.
When I then press CTRL + C and restart the command my live server boots almost instantly. So compiling was probably done before the error occurred.