[twitter] only download first page #2514

afterdelight · 2022-04-20T10:03:30Z

twitter with adult account only download first 25 photos. pls fix

Hrxn · 2022-04-20T11:34:17Z

I just downloaded thousands of pics, like, yesterday?

Fair to assume you're doing something wrong..
Steps to reproduce? gallery-dl verbose log? Link that's causing problems?

AlttiRi · 2022-04-20T12:14:12Z

The first 25 images, or 25 images total?

With the "recent" Twitter changes it now requires to use an account to download NSFW media.

You need to use the "auth_token" cookie from the browser where you are logged in (or "username" and "password") in the config file:

        "twitter": {
            "cookies": {
                "auth_token": "XXXABCDEFXXX"
            }
        }

Take a look at the similar issues.

afterdelight · 2022-04-20T14:33:48Z

Okay after changing the url to https://twitter.com/vaha0121w4/media it download 1146 photos while the account has about 66000 tweets. source: https://twisave.com/vaha0121w4

How to download all medias from the account excluding retweets?

AlttiRi · 2022-04-20T14:51:32Z

/username and /username/media is limited by ~1000 last posts as far I know.

Download the result of "the search page" https://twitter.com/search-advanced.

For example: gallery-dl "https://twitter.com/search?q=from:username".

Everything:
https://twitter.com/search?q=from:vaha0121w4

~~From 2014-03-03 to 2018-08-15:~~
~~https://twitter.com/search?q=from:vaha0121w4+since:2014-03-03+until:2018-08-15~~
It does not seem to work in gallery-dl. I get [twitter][info] No results for ....

Also it looks that &src=typed_query is not needed for gallery-dl, while Twitter shows only a part without it.
Just to compare:
https://twitter.com/search?q=from:vaha0121w4&src=typed_query
~~https://twitter.com/search?q=from:vaha0121w4+since:2014-03-03+until:2018-08-15&f=image&src=typed_query~~

afterdelight · 2022-04-20T15:06:34Z

okay, now its downloading more pics. we will see how many it got

afterdelight · 2022-04-20T15:08:30Z

will 'https://twitter.com/search?q=from:vaha0121w4' sufficient to download all images and videos?

AlttiRi · 2022-04-20T15:32:57Z

If you use an account credentials, then it should do.

You can run additionally /username/media after /search?q=from:username just in case.

afterdelight · 2022-04-20T20:13:44Z

okay, does https://twitter.com/search?q=from:vaha0121w4 include download retweets?

Hrxn · 2022-04-20T22:44:39Z

Yes, if you set the "retweets" option accordingly.,

AlttiRi · 2022-04-20T22:54:39Z

Just checked:

/search?q=from:username does not includes retweets. Even if "retweets" is used (true/"original").
It works only when you download /username endpoint.

Also with "retweets": "original" the filename will use the original tweet id and author name, instead of the reposter's ones.

(With false value (by default) it does not download retweets at all from /username endpoint.)

AlttiRi · 2022-04-20T23:29:19Z

The interested thing is that /username (timeline) has some posts missed even within the first ~1000 posts.

For example, I have run gallery-dl https://twitter.com/DPMaker_ (mostly NSFW 3D)

Then I run again gallery-dl https://twitter.com/DPMaker_/media:

It (the first run) has gaps even for the recent posts.

For example, it displays in /media:
https://twitter.com/DPMaker_/status/1512576718120255492

However, in timeline I only see the artist's two reply to this post. But not the main post with the image.

UPD: Note: this post is pinned.

UPD2: This is a media reply https://twitter.com/DPMaker_/status/1488751448624013312 (also is missed in timeline)

So, I think, if you don't interested in retweets, don't use /username at all.
Use only /username/media and /search?q=from:username for the large count of tweets.

UPD3:
Timeline does not includes media from replies, a pinned post (at least with "retweets": false), the posts are out of the ~1000 recent (while whey can be in /media), ...something more?

UPD4: Also I did not tested "text-tweets": true, + postprocessors with /media.

UPD5: /media does not includes "cards" ("cards": true,).

afterdelight · 2022-04-21T06:58:34Z

That's nice to know. Btw you have a user script to display total media posts on twitter user page and if possible all total tweets too.

afterdelight · 2022-04-21T07:42:21Z

Also I want to know how to seperate downloaded retweets to subfolder using '/search?q=from:username' and 'retweets": "original'

example:
E://main-download-folder/retweets/downloaded-pics

nisehime · 2022-04-22T21:08:23Z

Timeline does not includes media from replies, a pinned post (at least with "retweets": false), the posts are out of the ~1000 recent (while whey can be in /media), ...something more?

Have you set the pinned option to true?
To download timeline with replies you should use appropriate link. For your example: https://twitter.com/DPMaker_/with_replies. However, with replies you will also get potentially unneeded media, which are neither target user's tweets, nor his retweets (so tweets he replied to). I explained that here. Unfortunately, my suggested fix wasn't done.

/search?q=from:username does not includes retweets.

You should put include:nativeretweets in the search query. You can also put filter:media or filter:links, to speed up the process in exchange to accuracy.

It does not seem to work in gallery-dl. I get [twitter][info] No results for ....

Works for me. Twitter's search is not consistent. It doesn't return all results, very random and sometimes decides to not return anything older than X. Nothing really can be done about that other than trying again over some time, Hitomi Downloader dev suggested that trying different user-agents may help.

will 'https://twitter.com/search?q=from:vaha0121w4' sufficient to download all images and videos?

No. You should always do timeline/media and then search.

afterdelight · 2022-04-23T07:53:15Z

i only want the user's medias both in his timeline and his replies. not other users. so to use with media filter, i put it like
'https://twitter.com/search?q=from:vaha0121w4&filter:media' ?

Also can you answer both of my questions on top of your reply? thanks

nisehime · 2022-04-23T12:08:06Z

i only want the user's medias both in his timeline and his replies.

I suggest using /media first, then continue with search. You can copy the id of the last downloaded tweet and put it in the search query with max_id:{id} parameter. For example, the last id from /media page of your account is 1335006214657196032, so the search link would be:
https://twitter.com/search?q=from:vaha0121w4 max_id:1335006214657196032 filter:media
However, as I type, it didn't work (works in Hitomi btw). Not because I do something wrong, it's just twitter search, as I explained, is inconsistent. Removing filter:media or using filter:links (basically the same as media but also includes tweets with links and quoted tweets) worked though in this case specifically.

Anyway, twitter's search is a pain. It never gives you all tweets and requires various approaches to work with. Use issues search on this rep, it was discussed many times.

you have a user script to display total media posts on twitter user page and if possible all total tweets too

Display where? In a browser? It's already displayed there, no scripts needed. Gallery-dl also have that info in the metadata (see -K).

AlttiRi · 2022-04-23T12:59:26Z

i put it like
'https://twitter.com/search?q=from:vaha0121w4&filter:media' ?

Not /search?filter:media&q=from:username, but /search?q=filter:media+from:username.
It's the one query param "q" with two "options" separated by a space.
"+" (or "%20") sign is an alias for " " (space) character in a URL.
Using a space (" ") as is would require to quote the URL (as well as when there is "&" character).

afterdelight · 2022-04-23T17:52:13Z

okay thanks for the clarfication, i got
[twitter][info] No results for https://twitter.com/search?q=filter:media+from:vaha0121w4
but i got success with
https://twitter.com/search?filter:media&q=from:vaha0121w4

afterdelight · 2022-04-23T17:52:43Z

also i dont have any info about tweets numbers or media numbers on the username display page tho

AlttiRi · 2022-04-23T19:07:33Z

okay thanks for the clarfication, i got
[twitter][info] No results for https://twitter.com/search?q=filter:media+from:vaha0121w4
but i got success with
https://twitter.com/search?filter:media&q=from:vaha0121w4

Because of it's the same thing as https://twitter.com/search?q=from:vaha0121w4.
Wrong using of a search param (?q=from:vaha0121w4&filter:media=), or using of a fictional param (filter:media) has no effect.

Currently /search?q=filter:media+from:username does not work. 8 hours ago it worked fine.

AlttiRi · 2022-04-23T19:09:32Z

Gallery-dl also have that info in the metadata (see -K).

It's count of posts with media.

afterdelight · 2022-04-23T19:27:31Z

thanks now i can see the number. Now i need to know how to download profile picture and profile banner to sub folders.

afterdelight · 2022-06-30T16:53:20Z

hey, how to get medias included in replies and retweets? whats the command? ty

afterdelight · 2022-06-30T18:33:35Z

i got it
https://twitter.com/search?q=from:vaha0121w4+include:nativeretweets&f=live

afterdelight mentioned this issue May 16, 2022

Help with Twitter profile scraper command line ytdl-org/youtube-dl#30932

Open

nisehime mentioned this issue Jun 5, 2022

[Twitter] Unsearchable users (NSFW) #2658

Closed

nisehime mentioned this issue Jul 19, 2022

[Twitter] How to bypass twitter limit? #2766

Closed

mikf closed this as completed Dec 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[twitter] only download first page #2514

[twitter] only download first page #2514

afterdelight commented Apr 20, 2022

Hrxn commented Apr 20, 2022

AlttiRi commented Apr 20, 2022 •

edited

Loading

afterdelight commented Apr 20, 2022

AlttiRi commented Apr 20, 2022 •

edited

Loading

afterdelight commented Apr 20, 2022

afterdelight commented Apr 20, 2022

AlttiRi commented Apr 20, 2022 •

edited

Loading

afterdelight commented Apr 20, 2022

Hrxn commented Apr 20, 2022

AlttiRi commented Apr 20, 2022 •

edited

Loading

AlttiRi commented Apr 20, 2022 •

edited

Loading

afterdelight commented Apr 21, 2022 •

edited

Loading

afterdelight commented Apr 21, 2022

nisehime commented Apr 22, 2022 •

edited

Loading

afterdelight commented Apr 23, 2022

nisehime commented Apr 23, 2022 •

edited

Loading

AlttiRi commented Apr 23, 2022 •

edited

Loading

afterdelight commented Apr 23, 2022 •

edited

Loading

afterdelight commented Apr 23, 2022

AlttiRi commented Apr 23, 2022 •

edited

Loading

AlttiRi commented Apr 23, 2022 •

edited

Loading

afterdelight commented Apr 23, 2022 •

edited

Loading

afterdelight commented Jun 30, 2022

afterdelight commented Jun 30, 2022

[twitter] only download first page #2514

[twitter] only download first page #2514

Comments

afterdelight commented Apr 20, 2022

Hrxn commented Apr 20, 2022

AlttiRi commented Apr 20, 2022 • edited Loading

afterdelight commented Apr 20, 2022

AlttiRi commented Apr 20, 2022 • edited Loading

afterdelight commented Apr 20, 2022

afterdelight commented Apr 20, 2022

AlttiRi commented Apr 20, 2022 • edited Loading

afterdelight commented Apr 20, 2022

Hrxn commented Apr 20, 2022

AlttiRi commented Apr 20, 2022 • edited Loading

AlttiRi commented Apr 20, 2022 • edited Loading

afterdelight commented Apr 21, 2022 • edited Loading

afterdelight commented Apr 21, 2022

nisehime commented Apr 22, 2022 • edited Loading

afterdelight commented Apr 23, 2022

nisehime commented Apr 23, 2022 • edited Loading

AlttiRi commented Apr 23, 2022 • edited Loading

afterdelight commented Apr 23, 2022 • edited Loading

afterdelight commented Apr 23, 2022

AlttiRi commented Apr 23, 2022 • edited Loading

AlttiRi commented Apr 23, 2022 • edited Loading

afterdelight commented Apr 23, 2022 • edited Loading

afterdelight commented Jun 30, 2022

afterdelight commented Jun 30, 2022

AlttiRi commented Apr 20, 2022 •

edited

Loading

AlttiRi commented Apr 20, 2022 •

edited

Loading

AlttiRi commented Apr 20, 2022 •

edited

Loading

AlttiRi commented Apr 20, 2022 •

edited

Loading

AlttiRi commented Apr 20, 2022 •

edited

Loading

afterdelight commented Apr 21, 2022 •

edited

Loading

nisehime commented Apr 22, 2022 •

edited

Loading

nisehime commented Apr 23, 2022 •

edited

Loading

AlttiRi commented Apr 23, 2022 •

edited

Loading

afterdelight commented Apr 23, 2022 •

edited

Loading

AlttiRi commented Apr 23, 2022 •

edited

Loading

AlttiRi commented Apr 23, 2022 •

edited

Loading

afterdelight commented Apr 23, 2022 •

edited

Loading