Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add integration test for Scrapy integration #148

Closed
vdusek opened this issue Dec 2, 2023 · 0 comments · Fixed by #390
Closed

Add integration test for Scrapy integration #148

vdusek opened this issue Dec 2, 2023 · 0 comments · Fixed by #390
Assignees
@vdusek
Labels
debt Code quality improvement or decrease of technical debt. t-tooling Issues with this label are in the ownership of the tooling team.
Milestone
108th sprint - To...

Comments

@vdusek
Copy link
Contributor

vdusek commented Dec 2, 2023

https://github.com/apify/apify-sdk-python/tree/v1.3.0/src/apify/scrapy
@vdusek vdusek added debt Code quality improvement or decrease of technical debt. t-tooling Issues with this label are in the ownership of the tooling team. labels Dec 2, 2023
@vdusek vdusek self-assigned this Dec 2, 2023
@vdusek vdusek mentioned this issue Dec 4, 2023
Merged
@vdusek vdusek closed this as completed Jan 15, 2024
@vdusek vdusek reopened this Jan 15, 2024
@vdusek vdusek mentioned this issue Jan 23, 2024
Merged
@vdusek vdusek removed their assignment Feb 13, 2024
@vdusek vdusek changed the title Create integration tests for Scrapy module Add integration test for Scrapy integration Oct 31, 2024
@vdusek vdusek mentioned this issue Feb 6, 2025
Merged
@vdusek vdusek self-assigned this Feb 6, 2025
@vdusek vdusek added this to the 107th sprint - Tooling team milestone Feb 6, 2025
vdusek added a commit that referenced this issue Feb 13, 2025
@vdusek
feat: Unify Apify and Scrapy to use single event loop & remove `nest-…
96949be 
…asyncio` (#390)

### Description

- Apify (asyncio) and Scrapy (Twisted) now run on a single event loop.
  - `nest-asyncio` has been completely removed.
  - It seems that this change also improved the performance.
- The `ApifyScheduler`, which is synchronous, now executes asyncio
coroutines (communication with RQ) in a separate thread with its own
asyncio event loop.
- Logging setup has to be adjusted and I moved to a dedicated file in
the SDK.
- The try-import functionality for optional dependecies from Crawlee was
added to `scrapy` subpackage.
- A new integration test for Scrapy Actor has been added.

### Issues

- Closes: #148
- Closes: #176
- Closes: #392
- Relates: apify/actor-templates#303
- This issue will be closed once the corresponding PR in
`actor-templates` is merged.

### Tests

- A new integration test for Scrapy Actor has been added.
- And of course, it was tested manually using the Actor from
guides/templates.

### Next steps

- Update Scrapy Actor template in `actor-templates`.
- Update [Actor Scrapy Books
Example](https://github.com/apify/actor-scrapy-books-example).
- Add HTTP cache storage for KVS, @honzajavorek will provide his
implementation.

### Follow-up issues

- There are still a few issues to be resolved.
- #391
- #395
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vdusek vdusek

Labels
debt Code quality improvement or decrease of technical debt. t-tooling Issues with this label are in the ownership of the tooling team.
Projects
None yet
Milestone
108th sprint - Tooling team
Development

Successfully merging a pull request may close this issue.

feat: Unify Apify and Scrapy to use single event loop & remove nest-asyncio apify/apify-sdk-python
1 participant
@vdusek