-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
community[minor]: Spider Document Loader #5415
Merged
+232
−0
Merged
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit
Hold shift + click to select a range
92225b1
files
WilliamEspegren c189f09
format
WilliamEspegren 13115d5
update yarn.lock
WilliamEspegren cf18235
Merge branch 'langchain-ai:main' into main
WilliamEspegren 82d6800
new version of spider-client
WilliamEspegren a5171a5
new spider-client version
WilliamEspegren 7a7df15
spider constructor
WilliamEspegren ee29a75
Merge branch 'langchain-ai:main' into main
WilliamEspegren 871895b
Merge branch 'langchain-ai:main' into main
WilliamEspegren 054ce47
correct doc
WilliamEspegren 5415d7b
change ref link
WilliamEspegren 5be9067
commented out params in example
WilliamEspegren 4fce948
updated spider-client version
WilliamEspegren 8202839
Merge branch 'main' into main
bracesproul 070d6c5
changed location of files
WilliamEspegren 9b5f255
Merge remote-tracking branch 'origin/main'
WilliamEspegren 1398a41
change location of files
WilliamEspegren 70b0ba1
changed file location
WilliamEspegren cadda77
Merge branch 'langchain-ai:main' into main
WilliamEspegren a16b265
updated correct imports
WilliamEspegren 233cae5
Update spider.ts
jacoblee93 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
format
commit c189f0904a79becf94a043ba110ea4faba9fbdbd
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,13 +1,13 @@ | ||
import { SpiderLoader } from "langchain/document_loaders/web/spider"; | ||
|
||
const loader = new SpiderLoader({ | ||
url: "https://spider.cloud", // The URL to scrape | ||
apiKey: process.env.SPIDER_API_KEY, // Optional, defaults to `SPIDER_API_KEY` in your env. | ||
mode: "scrape", // The mode to run the crawler in. Can be "scrape" for single urls or "crawl" for deeper scraping following subpages | ||
params: { | ||
// optional parameters based on Spider API docs | ||
// For API documentation, visit https://spider.cloud/docs/api | ||
}, | ||
}); | ||
const docs = await loader.load(); | ||
url: "https://spider.cloud", // The URL to scrape | ||
apiKey: process.env.SPIDER_API_KEY, // Optional, defaults to `SPIDER_API_KEY` in your env. | ||
mode: "scrape", // The mode to run the crawler in. Can be "scrape" for single urls or "crawl" for deeper scraping following subpages | ||
params: { | ||
// optional parameters based on Spider API docs | ||
// For API documentation, visit https://spider.cloud/docs/api | ||
}, | ||
}); | ||
|
||
const docs = await loader.load(); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -1241,7 +1241,7 @@ | |
"@mendable/firecrawl-js": "^0.0.13", | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hey there! 👋 I noticed that a new package "@spider-cloud/spider-client" has been added to the dependencies in the package.json file. This change is flagged for your review to ensure it aligns with the project's dependency management strategy. Keep up the great work! 🚀 |
||
"@notionhq/client": "^2.2.10", | ||
"@pinecone-database/pinecone": "^1.1.0", | ||
"@spider-cloud/spider-client": "^0.0.10", | ||
"@spider-cloud/spider-client": "^0.0.11", | ||
"@supabase/supabase-js": "^2.10.0", | ||
"@swc/core": "^1.3.90", | ||
"@swc/jest": "^0.2.29", | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey there! 👋 I've reviewed the code and flagged a change in the PR for your attention. The addition of
apiKey: process.env.SPIDER_API_KEY
accesses an environment variable, so please take a look when you get a chance. Let me know if you need further assistance!