v2.0.2

• Fix serialization issues in CheerioCrawler caused by parser conflicts in recent versions of cheerio.

v2.0.1

• Use got-scraping 2.0.1 until fully compatible.

v2.0.0

We're releasing SDK 2 ahead of schedule, because we need state of the art HTTP2 support for scraping and with Node.js versions <15.10, HTTP2 is not very reliable. We bundled in 2 more potentially breaking changes that we were waiting for, but we expect those to have very little impact on users. Migration should therefore be super simple. Just bump your Node.js version.

If you're waiting for full TypeScript support and new features, those are still in the works and will be released in SDK 3 at the end of this year.

• BREAKING: Require Node.js >=15.10.0 because HTTP2 support on lower Node.js versions is very buggy.
• BREAKING: Bump cheerio to 1.0.0-rc.10 from rc.3. There were breaking changes in cheerio between the versions so this bump might be breaking for you as well.
• Remove LiveViewServer which was deprecated before release of SDK v1.
• We no longer tag beta releases.

v1.3.4

1.3.4 / 2021/08/04

• Fix issues with TS builds caused by incomplete browser-pool rewrite

v1.3.3

1.3.3 / 2021/08/04

• Fix public URL getter of key-value stores

v1.3.2

• Fix headerGeneratorOptions not being passed to got-scraping in requestAsBrowser.

v1.3.1

1.3.1 / 2021/07/13

• Fix client /v2 duplication in apiBaseUrl.

v1.3.0

Navigation hooks in CheerioCrawler

CheerioCrawler downloads the web pages using the requestAsBrowser utility function.
As opposed to the browser based crawlers that are automatically encoding the URLs, the
requestAsBrowser function will not do so. We either need to manually encode the URLs
via encodeURI() function, or set forceUrlEncoding: true in the requestAsBrowserOptions,
which will automatically encode all the URLs before accessing them.

We can either use forceUrlEncoding or encode manually, but not both - it would
result in double encoding and therefore lead to invalid URLs.

We can use the preNavigationHooks to adjust requestAsBrowserOptions:

preNavigationHooks: [
    (crawlingContext, requestAsBrowserOptions) => {
        requestAsBrowserOptions.forceUrlEncoding = true;
    }
]

Apify class and Configuration

Adds two new named exports:

• Configuration class that serves as the main configuration holder, replacing explicit usage of
  environment variables.
• Apify class that allows configuring the SDK. Env vars still have precedence over the SDK configuration.

When using the Apify class, there should be no side effects.
Also adds new configuration for WAL mode in ApifyStorageLocal.

As opposed to using the global helper functions like main, there is an alternative approach using Apify class.
It has mostly the same API, but the methods on Apify instance will use the configuration provided in the constructor.
Environment variables will have precedence over this configuration.

const { Apify } = require('apify'); // use named export to get the class

const sdk = new Apify({ token: '123' });
console.log(sdk.config.get('token')); // '123'

// the token will be passed to the `call` method automatically
const run = await sdk.call('apify/hello-world', { myInput: 123 });
console.log(`Received message: ${run.output.body.message}`);

Another example shows how the default dataset name can be changed:

const { Apify } = require('apify'); // use named export to get the class

const sdk = new Apify({ defaultDatasetId: 'custom-name' });
await sdk.pushData({ myValue: 123 });

is equivalent to:

const Apify = require('apify'); // use default export to get the helper functions

const dataset = await Apify.openDataset('custom-name');
await dataset.pushData({ myValue: 123 });

Full list of changes:

• Add Configuration class and Apify named export, see above.
• Fix proxyUrl without a port throwing an error when launching browsers.
• Fix maxUsageCount of a Session not being persisted.
• Update puppeteer and playwright to match stable Chrome (90).
• Fix support for building TypeScript projects that depend on the SDK.
• add taskTimeoutSecs to allow control over timeout of AutoscaledPool tasks
• add forceUrlEncoding to requestAsBrowser options
• add preNavigationHooks and postNavigationHooks to CheerioCrawler
• deprecated prepareRequestFunction and postResponseFunction methods of CheerioCrawler
• Added new event aborting for handling gracefully aborted run from Apify platform.

v1.2.1

• Fix requestAsBrowser behavior with various combinations of json, payload legacy options. closes: #1028

v1.2.0

This release brings the long awaited HTTP2 capabilities to requestAsBrowser. It could make HTTP2 requests even before, but it was not very helpful in making browser-like ones. This is very important for disguising as a browser and reduction in the number of blocked requests. requestAsBrowser now uses got-scraping.

The most important new feature is that the full set of headers requestAsBrowser uses will now be generated using live data about browser headers that we collect. This means that the "header fingeprint" will always match existing browsers and should be indistinguishable from a real browser request. The header sets will be automatically rotated for you to further reduce the chances of blocking.

We also switched the default HTTP version from 1 to 2 in requestAsBrowser. We don't expect this change to be breaking, and we took precautions, but we're aware that there are always some edge cases, so please let us know if it causes trouble for you.

Full list of changes:

• Replace the underlying HTTP client of utils.requestAsBrowser() with got-scraping.
• Make useHttp2 true by default with utils.requestAsBrowser().
• Fix Apify.call() failing with empty OUTPUT.
• Update puppeteer to 8.0.0 and playwright to 1.10.0 with Chromium 90 in Docker images.
• Update @apify/ps-tree to support Windows better.
• Update @apify/storage-local to support Node.js 16 prebuilds.