Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request handler should be able to timeout blocking sync code in user defined handler #908

Open
Pijukatel opened this issue Jan 15, 2025 · 1 comment
Labels
bug Something isn't working. t-tooling Issues with this label are in the ownership of the tooling team.

Comments

@Pijukatel
Copy link
Contributor

Current purely async implementation of request handler is not capable of triggering timeout for blocking sync code. Such code is created by users and we have no control over it. So we can't expect only async blocking code.

Following simple user defined handler can't currently trigger timeout even if it should.

@crawler.router.default_handler
async def handler(context: BasicCrawlingContext) -> None:
    time.sleep(some_big_time)
@Pijukatel Pijukatel added bug Something isn't working. t-tooling Issues with this label are in the ownership of the tooling team. labels Jan 15, 2025
Pijukatel added a commit that referenced this issue Jan 17, 2025
…e handler (#909)

### Description

Fix crawler not retrying user handler if there was timeout in the
handler.
Add parametrized test with one parameter set skipped with reason
#908

### Issues

- Closes: #907
@janbuchar
Copy link
Collaborator

It is almost impossible to handle this cleanly AFAIK - we cannot interrupt sync code that blocks the event loop. Some options I see:

  1. Run request handler in a separate event loop - might break things like playwright in unusual ways
  2. Detect when something blocks the event loop for too long and notify the user

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working. t-tooling Issues with this label are in the ownership of the tooling team.
Projects
None yet
Development

No branches or pull requests

2 participants