Slow performance loading some web pages #481
Comments
Sure. I think in this case the problem will be the blocklists, I've experienced similar issues on some websites, did you try it out without the blocklists rules? I'll take a look at it. |
Awesome. With OpenSnitch running and unrestricted firefox rules (as was Tor in the 7 sec load result), it took 4 seconds to load WU. |
|
Yes, I've reproduced it, and it's the same problem I already saw. I think the problem is that we're just dropping the requests instead of rejecting them , so in some cases the apps wait until the DNS resolution times out. I didn't find a way of rejecting them with libnetfilter-queue. One way of fixing this issue could be replying a DNS answer poiting to localhost. |
|
That makes sense, I did see a host request name bubble in firefox sit there for a long time. I wouldn't know what to suggest other than close the socket, but that may not be supported for all protocols. I've used |
|
well, mmmh, now that you mention killing requests, I guess that we could kill connections via netlink.conntrack .. interesting. So conman.Parse() could return an action along with the properties of a connection and kill it on demand. I'll write a PoC some day in the near future 👍 edit: |
|
hey @Techtonictools , could you test it with this daemon? Stop the running service, rename the daemon to opensnitchd if you wish, and launch it from a terminal: I think it's working as expected, killing the outgoing requests. WU loads much faster. |
|
Absolutely. On my laptop I edited the debian version: |
|
oops, I see, you need to launch it as root: |
|
When running the private as root, I didn't notice a perf change. It took 38 seconds to load WU on the laptop with the private bin and on my main pc on the same gateway with the release version it took 37 seconds. How long does it take for it to load on your host? |
|
Without killing connections (Action: deny) it takes 1' to just display some text of the page and 1'40" for the loading progress bar to stop (I'm on a really slow connection, 50-500KB/s). Killing connections (Action: reject) the loading progress bar stops in about 10-22s, almost at the same time that any text is displayed. Although it seems to load faster, I can see that it keeps loading resources in background, and it takes 40s in total +-. I think I'll push these changes to a new branch in order to save it, and maybe add the "reject" Action for the v1.5.0. In any case, we could also reply to DNS queries pointing blocked domains to localhost. If someone wants to work on this and make a PoC it'd be more than welcome. |
When blocking a connection via libnetfilter-queue using NF_DROP the connection is discarded. If the blocked connection is a DNS query, the app that initiated it will wait until it times out, which is ~30s. This behaviour can for example cause slowdowns loading web pages: #481 This change adds the option to reject connections by killing the socket that initiated it. Just set a rule's Action to "reject". Denying: $ time telnet 1.1.1.1 22 Trying 1.1.1.1... telnet: Unable to connect to remote host: Connection timed out real 2m10,039s Rejecting: $ time telnet 1.1.1.1 22 Trying 1.1.1.1... telnet: Unable to connect to remote host: Software caused connection abort real 0m0,005s TODO: add the reject option to the GUI (preferences, pop-up and rules dialog)
|
for anyone interested and future reference: 1b8ec16 feedback welcome by the way. |
I'd like to look at the code, but I'm not sure I can fix it and for sure can't do it right away. Where in the source is the logic that determines if a request will be allowed or denied? |
|
There're 2 points in the code that you can look at:
If the domain is in a block/allowlist the response could be crafted here.
Modifying the packet here wouldn't have much sense I guess, since we would have to wait for the response. SetVerdict*() funcs are defined in netfilter/packet.go and they send the verdict through a channel (netfilter/queue.go), optionally |
|
Thanks for laying that out, gustavo. I started to read the code path going backwards from onPacket for this issue and also for issue #482 . All these types, libraries and golang are new to me. I want to read up on all the libraries and learn how this program works so I can understand it. |
|
I'm starting to feel confident with the Action: reject, it seems to work fine (if it improves web pages loading times is another story). Maybe I'll add it for this release: deny: reject: I don't know if you mentioned it already, but could you reproduce this problem using "proc" monitor method? I want to discard that it's not due to the ebpf module. |
|
That looks like a substantial improvement. I set the Process monitor
method in the UI to Proc just now and re-ran the scenario. I didn't see
a performance change. WU took longer than a minute time to begin to
load.
…On Mon, 2021-09-06 at 06:22 -0700, Gustavo Iñiguez Goia wrote:
I'm starting to feel confident with the Action: reject, it seems to
work fine (if it improves web pages loading times is another story).
Maybe I'll add it for this release:
deny:
$ time curl duckduckgo.com
curl: (6) Could not resolve host: duckduckgo.com
real 0m20,049s
user 0m0,007s
sys 0m0,012s
reject:
$ time curl duckduckgo.com
curl: (6) Could not resolve host: duckduckgo.com
real 0m0,043s
user 0m0,012s
sys 0m0,006s
I don't know if you mentioned it already, but could you reproduce
this problem using "proc" monitor method? I want to discard that it's
not due to the ebpf module.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
|
When blocking a connection via libnetfilter-queue using NF_DROP the connection is discarded. If the blocked connection is a DNS query, the app that initiated it will wait until it times out, which is ~30s. This behaviour can for example cause slowdowns loading web pages: #481 This change adds the option to reject connections by killing the socket that initiated them. Denying: $ time telnet 1.1.1.1 22 Trying 1.1.1.1... telnet: Unable to connect to remote host: Connection timed out real 2m10,039s Rejecting: $ time telnet 1.1.1.1 22 Trying 1.1.1.1... telnet: Unable to connect to remote host: Software caused connection abort real 0m0,005s
|
hey! I think this issue is resolved in latest RC2: https://github.com/evilsocket/opensnitch/releases/tag/v1.5.0-rc.2 So I'm closing this now. If the issue remains, please reopen it and we'll take a look. |
I've been using OpenSnitch starting on a laptop with infrequent use and now I'm using it on my main PC and laptop on a daily basis. It is such a great tool. I've never used or seen anything like it (I don't run mac now) and I'm using it to block Microsoft from trying to update my VM through their back doors and hacks they added so that they can force-feed updates and control my PC. It is also useful to block data mining of my activity on the web and data mining on the PC itself. OpenSnitch additionally enables me to reduce being manipulated through ads on the web and it brings awareness about what apps want to phone home, giving me control of deciding who can phone home and who cannot.
Describe the bug
I have OpenSnitch configured so that everything is blocked except for domains in an exception list. Then I maintain that list of approved sites. The issue I've been hitting is it takes a long time to load some of the heavy websites. Sites that seem to have lots of libraries, requests to complete and code to load like weather underground (WU) does, those sites reproduce a performance issue. Some sites load quickly enough and there isn't any perf issue, like when using old.reddit.com, posting, searching, etc.
I had previously used a tool called Fiddler on Windows to block all http requests that respected the system proxy, except for websites in a list, and I'd surf the web using it for protection. When I was using that setup and loaded WU, it did not have a slow down. I know that Fiddler is not a full software proxy like OpenSnitch, but I think that when using Fiddler in that way to load WU, it demonstrated that failing requests doesn't make the page take a long time to load.
The wait time I'm experiencing to load WU with OpenSnitch is about 68 seconds before seeing any rendering at all and before load completes. With OpenSnitch off and clearing the browser cache, the page loaded in 4 seconds. Tor (unrestricted in OpenSnitch) loads the page in 7 seconds. Tor-like speed would be great. Could there be performance improvements made for the web surfing scenario in OpenSnitch?
To Reproduce
1. Modify the attached rules to include a correct directory path to store the exception list in
2. Create the exception list from using the domains listed below and stick it in that path in step 1
3. Launch Firefox and clear the cache if the WU site has been visited before
4. Enter WU in the address bar, invoke it and begin timing https://www.wunderground.com/forecast/us/wa/seattle
5. End timing when you see the page beginning to render
Expected behavior (optional)
1. Some fonts won't load, there will be spinners that never die and the page may look incomplete.
2. The page will have what looks like an HTML canvas drawing of the 10 day forecast that is completely legible and usable with a mouse-over.
3. Less than 15 second load time. I don't know what is reasonable since I don't know what work is being done for filtering the requests. 15 seconds is just an arbitrary number.
OS (please complete the following information):
The perf issue reproduces on my Macbook Pro 17", but it also reproduces on a custom PC that is much faster. Both have an unused wired NIC and WiFi. Both use the same internet connection/gateway. The config:
Additional context
Traceroute
Exception list for WU
rules.zip
The text was updated successfully, but these errors were encountered: