Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chunk at offset 0 has invalid #348

Closed
kiroInn opened this issue Nov 22, 2024 · 30 comments
Closed

chunk at offset 0 has invalid #348

kiroInn opened this issue Nov 22, 2024 · 30 comments

Comments

@kiroInn
Copy link

kiroInn commented Nov 22, 2024

when i download video, I got the error message

Error executing command: Command failed: youtubedr download -d /Users/xx/Code/creator -q 137 https://www.youtube.com/shorts/videoId 2024/11/22 17:49:47 download to directory /Users/xx/Code/video-creator time=2024-11-22T17:49:47.025+08:00 level=INFO msg="Downloading video" id=xxxx quality=hd1080 mimeType="video/mp4; codecs=\"avc1.640028\"" chunk at offset 0 has invalid size: expected=6753827 actual=0

Has anyone come across it?

@greentornado
Copy link

me too, seems youtube changed something

@shuntan
Copy link

shuntan commented Nov 22, 2024

me too, waiting for some solution

@greentornado
Copy link

Please check @corny if you have time, thanks

@prologic
Copy link

Yup same here:

chunk at offset 0 has invalid size: expected=2022330 actual=0

@prologic
Copy link

can't really tell what's going on here...

INFO[0002] downloading video                             id=YpiK1FMy2Mg
2024/11/23 08:09:25 INFO Downloading url="https://rr5---sn-ntq7yned.googlevideo.com/videoplayback?expire=1732334964&ei=FAFBZ4HjNuCBjuMPy8qq6QY&ip=159.196.9.199&id=o-AOn6AyK0S6hNo7NaHwQbwkuuA5ApAgQmWb3H01gauBUJ&itag=18&source=youtube&requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&met=1732313364%2C&mh=VD&mm=31%2C29&mn=sn-ntq7yned%2Csn-ntqe6nes&ms=au%2Crdu&mv=m&mvi=5&pl=22&rms=au%2Cau&initcwndbps=2632500&bui=AQn3pFR_puuxwQOcK5k-2reNAzKyQEm_nKATjI8ogy-181z_w9z7Yq7_OEUJVuR0oBBUFYL2j2v2STDY&spc=qtApAdc0eOZ-xM6q44o52We5IbFEebYd6UCVAwAO_azv8Zc6ebfgkMZ_si2tPOxjag&vprv=1&svpuc=1&mime=video%2Fmp4&ns=bf_R8a9MK0tz9SUTRwJOHAwQ&rqh=1&gir=yes&clen=66150607&ratebypass=yes&dur=975.423&lmt=1732139977265329&mt=1732312877&fvip=5&fexp=51326932%2C51331020%2C51335594&c=MWEB&sefc=1&txp=4538434&n=9DsMhlCdfzp5s1&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cxpc%2Cbui%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cns%2Crqh%2Cgir%2Cclen%2Cratebypass%2Cdur%2Clmt&sig=AJfQdSswRAIgenb1pymZ1gK_GInep0y6ZiNGWlno7MwaRO4L73JoOCsCIHKJzmOoLsuSzaBB1CSBv_sVv6hX1SaxgE9S2GriAlp1&lsparams=met%2Cmh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Crms%2Cinitcwndbps&lsig=AGluJ3MwRgIhAOuQJqZxsEHEx_jvGdthIjRtT0PmH116K-9SxXzP9uQjAiEA3ssnh3weIV3a_D5MNcd7jAuLCxHyA85Lvr68k3RBt9g%3D"
2024/11/23 08:09:25 INFO Downloading contentLength=66150607
ERRO[0003] error downloading video                       error="error writing video to file \"data/YpiK1FMy2Mg.mp4\": chunk at offset 31457280 has invalid size: expected=10485760 actual=0" id=YpiK1FMy2Mg
2024/11/23 08:09:25 ([::1]:60147 -) "GET /stream/YpiK1FMy2Mg HTTP/1.1" 200 0 795.713292ms

After adding some debug logs to the package. I also tested curling the url and I get a 403 Forbidden?

@prologic
Copy link

So what how does a HTML5 Video tag work with a blob: src? 🤔

<video tabindex="-1" class="video-stream html5-main-video" controlslist="nodownload" style="width: 1145px; height: 644px; left: 228px; top: 0px;" src="blob:https://www.youtube.com/9b425976-0036-4cc9-b1ce-0ed447422016"></video>

@prologic
Copy link

youtubedr also fails:

$ youtubedr download 'https://www.youtube.com/watch?v=YpiK1FMy2Mg'
2024/11/23 09:01:12 download to directory .
time=2024-11-23T09:01:12.946+10:00 level=INFO msg="Downloading video" id=YpiK1FMy2Mg quality=medium mimeType="video/mp4; codecs=\"av01.0.01M.08\""
chunk at offset 0 has invalid size: expected=10485760 actual=0

As does yt-dl:

$ youtube-dl 'https://www.youtube.com/watch?v=YpiK1FMy2Mg'
[youtube] YpiK1FMy2Mg: Downloading webpage
WARNING: unable to extract uploader id; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
ERROR: unable to download video data: HTTP Error 403: Forbidden

At this point I'm fairly confident that most/all? Youtube frontends are now broken :/

@greentornado
Copy link

Yt-dlp is working

@greentornado
Copy link

the problem should be in the client.go file, but i dont know how to refine it

@prologic
Copy link

The "problem" being what exactly? I haven't been able to work it out myself :/

@greentornado
Copy link

Examined the last commit with 403, the code point to that file

@PylotLight
Copy link

Decoding technique likely changed again... hopefully this gets updated soon.

@greentornado
Copy link

Anyone or fork has a solution yet ? Thanks

@shuntan
Copy link

shuntan commented Nov 24, 2024

Hey bro, I found some alternative methods to try. The current configuration of Android seems to be temporarily unavailable, It is strongly recommended to use a proxy, otherwise you will be banned easily. @greentornado @PylotLight @prologic,

Reference sources: https://github.com/jordibruin/YouTubeKit/blob/ad584bf7966b565f8332ee51b3df84938331a3d2/Sources/YouTubeKit/InnerTube.swift#L21

// In client.go, add IOS device
var DefaultClient = IOSClient

/*--------------------------struct-----------------------------*/
IOSClient = models.ClientInfo{
		Name:      "IOS",
		Version:   "17.33.2",
		Key:       "AIzaSyB-63vPrdThhKuerbB2N_l7Kwwcxj6yUAc",
		UserAgent: "com.google.ios.youtube/17.33.2 (iPhone14,3; U; CPU iOS 15_6 like Mac OS X)",
}

//Add `Screen` and `PlayerParams` fields
type ClientInfo struct {
	Name           string
	Key            string
	Version        string
	UserAgent      string
	AndroidVersion int
	Screen         string
	PlayerParams   string
}

//Add `ThirdParty` and `Params` fields
type InnertubeRequest struct {
	VideoID         string            `json:"videoId,omitempty"`
	BrowseID        string            `json:"browseId,omitempty"`
	Continuation    string            `json:"continuation,omitempty"`
	Context         InntertubeContext `json:"context"`
	ThirdParty      *ThirdParty       `json:"thirdParty,omitempty"`
	PlaybackContext *PlaybackContext  `json:"playbackContext,omitempty"`
	ContentCheckOK  bool              `json:"contentCheckOk,omitempty"`
	RacyCheckOk     bool              `json:"racyCheckOk,omitempty"`
	Params          string            `json:"params,omitempty"`
}

//Add `EmbedUrl` field
type ThirdParty struct {
	EmbedUrl string `json:"embedUrl,omitempty"`
}

//Add `ClientScreen` field and change to omitempty
type InnertubeClient struct {
	HL                string `json:"hl"`
	GL                string `json:"gl"`
	ClientName        string `json:"clientName"`
	ClientVersion     string `json:"clientVersion"`
	AndroidSDKVersion int    `json:"androidSDKVersion,omitempty"`
	UserAgent         string `json:"userAgent,omitempty"`
	TimeZone          string `json:"timeZone"`
	UTCOffset         int    `json:"utcOffsetMinutes"`
	ClientScreen      string `json:"clientScreen,omitempty"`
}

/*--------------------------function-----------------------------*/
//Some modifications
func (c *Client) videoDataByInnertube(ctx context.Context, id string) ([]byte, error) {
	data := models.InnertubeRequest{
		VideoID:        id,
		Context:        prepareInnertubeContext(*c.client),
		ContentCheckOK: false,
		RacyCheckOk:    false,
		Params:         client.PlayerParams,
		PlaybackContext: &models.PlaybackContext{
			ContentPlaybackContext: models.ContentPlaybackContext{
				//SignatureTimestamp: 19250,
				HTML5Preference: "HTML5_PREF_WANTS",
			},
		},
		ThirdParty: &models.ThirdParty{
			EmbedUrl: "https://www.youtube.com/",
		},
	}

	return c.httpPostBodyBytes(ctx, client, "https://www.youtube.com/youtubei/v1/player?key="+c.client.Key, data)
}

//Add some headers
func (c *Client) httpPost(ctx context.Context, url string, body interface{}) (*http.Response, error) {
	data, err := json.Marshal(body)
	if err != nil {
		return nil, err
	}

	req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(data))
	if err != nil {
		return nil, err
	}

	req.Header.Set("X-Youtube-Client-Name", "3")
	req.Header.Set("X-Youtube-Client-Version", client.Version)
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("Accept", "*/*")
	req.Header.Set("Accept-Language", "de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6")
	req.Header.Set("Accept-Encoding", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
	req.Header.Set("Referer", "https://youtube.com/")
	req.Header.Set("Origin", "https://youtube.com")
	req.Header.Set("Host", "https://youtube.com")
	req.Header.Set("User-Agent", client.UserAgent)

	resp, err := c.httpDo(req)
	if err != nil {
		return nil, err
	}

	if resp.StatusCode != http.StatusOK {
		resp.Body.Close()
		return nil, ErrUnexpectedStatusCode(resp.StatusCode)
	}

	return resp, nil
}

//ClientScreen field is optional
func prepareInnertubeContext(clientInfo models.ClientInfo) InntertubeContext {
	return inntertubeContext{
		Client: inntertubeContext{
			HL:                "en",
			GL:                "US",
			TimeZone:          "UTC",
			ClientName:        clientInfo.Name,
			ClientVersion:     clientInfo.Version,
			AndroidSDKVersion: clientInfo.AndroidVersion,
			UserAgent:         clientInfo.UserAgent,
			ClientScreen:      clientInfo.Screen,
		},
	}
}

@greentornado
Copy link

greentornado commented Nov 25, 2024

thanks! will try and let you know the result

@sirt102
Copy link

sirt102 commented Nov 25, 2024

I meet the same issue :<

@greentornado
Copy link

greentornado commented Nov 26, 2024

it still doesnt work on my end @shuntan, do you have good result ? Can I use your implementation ?

@GhiaC
Copy link

GhiaC commented Nov 26, 2024

Did anyone find a solution?

@amimimor
Copy link

amimimor commented Dec 1, 2024

I also tried the suggested change and still getting 403 from yt

@corny
Copy link
Collaborator

corny commented Dec 4, 2024

I've improved status code checking (see #350), but I currently don't have take a closer look into this issue.

@arnoldhao
Copy link

arnoldhao commented Dec 4, 2024

I used "parse" to get the URL and tried downloading it another way, but still got a 403 error.
Seems like YouTube changed something in how the download URL is parsed.

@greentornado
Copy link

anyone has a working version yet, thanks

@PylotLight
Copy link

anyone has a working version yet, thanks

If they did, I imagine we would all know about it. Could we kindly stop spamming with follow-ups?

@ruizlenato
Copy link
Contributor

ruizlenato commented Dec 7, 2024

Yt-dlp is working

They fixed the problem.
yt-dlp/yt-dlp#11750

@gcottom
Copy link

gcottom commented Dec 8, 2024

Looks like the s sig is working correctly, maybe. After removing the optional return, I get further with the dl. File size shows in the terminal. Nsig has changed. Yt-dlp has the new function and I’ve tried to use it in the go code. JS is returning its input value and not doing the transformations. My JS knowledge is pretty limited and the obscuration that google uses doesn’t help. I’ll add the new function sig to the comments when I get home. I’ve tried using regexp2 with the regex from yt-dip. Took some mods but after changing the grouping names to numbered groupings, made some progress in identifying the function that does the transforms.

@gcottom
Copy link

gcottom commented Dec 13, 2024

MM = function(a) {
        if (a.B) {
            if (!wE(a.B) && !a.B.startsWith("local"))
                throw new g.jt("Untrusted URL",a.B);
            var b = g.Fk(a.B);
            a.scheme = b.G;
            a.C = b.j + (b.C != null ? ":" + b.C : "");
            var c = b.B;
            if (c.startsWith("/videoplayback"))
                a.path = "/videoplayback",
                c = c.slice(14);
            else if (c.startsWith("/initplayback"))
                a.path = "/initplayback",
                c = c.slice(13);
            else if (c.startsWith("/api/manifest")) {
                var d = c.indexOf("/", 12)
                  , e = c.indexOf("/", d + 1);
                d > 0 && e > 0 ? (a.path = c.slice(0, e),
                c = c.slice(e + 1)) : (a.path = c,
                c = "")
            } else
                a.path = c,
                c = "";
            d = a.j;
            a.j = xEa(c);
            Object.assign(a.j, yEa(b.D.toString()));
            Object.assign(a.j, d);
            a.j.file === "index.m3u8" && (delete a.j.file,
            a.path += "/file/index.m3u8");
            a.B = "";
            a.url = "";
            a.D && (b = "nn"[+a.D],
            MM(a),
            c = a.j[b] || null) && (c = zEa[0](c),
            a.set(b, c))
        }
    }
    ;

the variable names could vary.

@chickenfresh
Copy link

func (config playerConfig) decodeNsig(encoded string) (string, error) {
	//fmt.Println(encoded)
	//fBody, err := config.getNFunction()
	//if err != nil {
	//	return "", err
	//}
	fBody := `function(a){var b=a.split(a.slice(0,0)),c=[-1762978610,224018618,372078021,-1584087523,1631548760,-1523401041,function(d,e){e.length!=0&&(d=(d%e.length+e.length)%e.length,e.splice(0,1,e.splice(d,1,e[0])[0]))},
	   function(){for(var d=64,e=[];++d-e.length-32;)switch(d){case 58:d=96;continue;case 91:d=44;break;case 65:d=47;continue;case 46:d=153;case 123:d-=58;default:e.push(String.fromCharCode(d))}return e},
	   -1875214889,-1221130857,1929975707,-1762978610,-1987230004,1562453898,function(){for(var d=64,e=[];++d-e.length-32;){switch(d){case 91:d=44;continue;case 123:d=65;break;case 65:d-=18;continue;case 58:d=96;continue;case 46:d=95}e.push(String.fromCharCode(d))}return e},
	   -460348276,1141163327,-1209828986,642821393,1673505824,function(d,e){e=(e%d.length+d.length)%d.length;d.splice(-e).reverse().forEach(function(f){d.unshift(f)})},
	   1562453898,1039712780,198668272,733991645,/[;]'([,/,59,/]){/,-1775324511,-1662739629,-331919071,-504717341,function(d,e){if(e.length!=0){d=(d%e.length+e.length)%e.length;var f=e[0];e[0]=e[d];e[d]=f}},
	   -1514776029,-2107135749,function(d,e,f,h,l,m){return e(h,l,m)},
	   null,function(d,e){for(e=(e%d.length+d.length)%d.length;e--;)d.unshift(d.pop())},
	   function(d,e){d=(d%e.length+e.length)%e.length;e.splice(d,1)},
	   1994577881,function(d,e,f){var h=d.length;f.forEach(function(l,m,n){this.push(n[m]=d[(d.indexOf(l)-d.indexOf(this[m])+m+h--)%d.length])},e.split(""))},
	   1581593411,-81352162,-393381805,-1818317157,1582095365,-1669922702,-1501675767,function(d){for(var e=d.length;e;)d.push(d.splice(--e,1)[0])},
	   283239203,2026246890,"pop",438638805,function(){for(var d=64,e=[];++d-e.length-32;)switch(d){case 46:d=95;default:e.push(String.fromCharCode(d));case 94:case 95:case 96:break;case 123:d-=76;case 92:case 93:continue;case 58:d=44;case 91:}return e},
	   1885980169,function(d,e,f,h,l){return e(f,h,l)},
	   -1325120929,1598665835,1852467764,null,-998626056,-1916462760,b,null,42490944,-367416562,-1334988290,1212718012,b,424069788,function(d){d.reverse()},
	   -331919071,1145271056,1410038943,-886548250,1791824832,-418171758,function(){for(var d=64,e=[];++d-e.length-32;){switch(d){case 58:d-=14;case 91:case 92:case 93:continue;case 123:d=47;case 94:case 95:case 96:continue;case 46:d=95}e.push(String.fromCharCode(d))}return e},
	   394991787,189178273,/(\)],);[\]]/,b,-861027642,938992366,1617953070,-1390903909,-2068775914,1776751202];c[34]=c;c[57]=c;c[61]=c;try{try{c[24]>10&&((((0,c[53])((0,c[6])(c[85],c[60]),c[6],c[44],c[79]),c[20])(c[61],c[82]),c[17])(c[16],c[44]),1)||((((0,c[52])(c[24],c[80]),c[82])(c[41],c[21]),c[82])(c[25],c[26]),c[82])(c[40],c[20]),c[new Date("1969-12-31T18:46:01.000-05:15")/1E3]!=-8&&(c[30]==4&&(((((0,c[84])((0,c[53])(),c[9],c[39]),c[13])((0,c[13])((0,c[66])(c[20],c[19]),c[84],(0,c[35])(),c[9],c[39]),
	       c[81],c[80],c[62]),((((0,c[82])(c[54],c[39]),c[28])(c[80]),(0,c[76])(c[70],c[21]),c[84])((0,c[11])(),c[9],c[20]),c[28])(c[21]),c[28])(c[39]),((0,c[82])(c[47],c[26]),c[84])((0,c[35])(),c[9],c[26]),c[82])(c[68],c[39]),[])||((((((0,c[13])(((0,c[81])(c[20],c[34]),c[76])(c[73],c[80]),c[82],c[49],c[39]),c[81])(c[new Date("1970-01-01T08:29:50.000+08:30")/1E3+49],c[75]),c[76])(c[14],c[39]),c[76])(c[83],c[80]),c[81])(c[39],c[224+new Date("1970-01-01T03:42:21.000+03:45")/1E3]),c[84])((0,c[60])(),c[9],c[26])<=
	   (0,c[79])((0,c[66])(c[26],c[37]),c[76],((0,c[13])((0,c[82])(c[50],c[39]),c[82],c[72],c[20]),c[76])(c[27],c[21]),c[77],c[80])),c[40]<-7&&(((0,c[76])(c[36],c[0]),(0,c[66])(c[17],c[44]),c[38])(c[75]),"null")||((0,c[62])((0,c[69])(c[55]),c[85],c[14],c[36]),c[79])(c[6],c[55]),c[48]!=-6&&(c[68]<=6&&(((0,c[79])(c[19],c[81]),c[85])(c[16],c[49]),1)||((0,c[47])(c[58]),c[0])(c[68],c[49])),c[new Date("1969-12-31T22:30:10.000-01:30")/1E3]!==0&&(c[47]!==9||(((((0,c[8])((0,c[63])(),c[19],c[36]),c[8])((0,c[63])(),
	   c[19],c[30]),(((0,c[6])(c[14],c[30]),c[6])(c[17],c[36]),c[6])(c[42],c[4]),c[8])((0,c[21])(),c[19],c[36]),c[16])(c[4]),0))&&(((((0,c[5])(c[27],c[22]),(0,c[59])(c[64],c[1]),(0,c[59])(c[50],c[33]),c[83])(c[17],c[72]),c[83])(c[25],c[1]),c[73])(c[27],c[85]),c[2])(c[72],c[37])}catch(d){(0,c[new Date("1970-01-01T06:15:39.000+06:15")/1E3])(c[59]),(0,c[46])((0,c[85])(c[66],c[59])*(0,c[85])(c[49],c[12]),c[28],c[53],c[38]),(0,c[29])(c[new Date("1970-01-01T06:00:11.000+06:00")/1E3],c[59])}try{c[17]!=-8?(0,c[46])((((0,c[28])(c[72],
	   c[45]),c[23])(c[48],c[50]),c[13])(c[27],c[56]),c[67],c[81],c[64]):((((0,c[61])(c[40],c[35]),c[85])(c[60],c[62]),c[7])((0,c[20])(),c[18],c[35]),c[7])((0,c[69])(),c[18],c[62]),c[23]<=4&&(((0,c[5])(c[10],c[62]),c[5])(c[83],c[35]),(0,c[61])(c[17],c[35]),c[61])(c[2],c[3]),c[19]<=-10?((0,c[new Date("1970-01-01T01:45:37.000+01:45")/1E3])(c[35]),c[7])((0,c[29])(),c[18],c[62]):(0,c[22])((0,c[61])(c[8],c[62]),c[75],c[3],c[57])}catch(d){(0,c[31])((0,c[58])(c[64],c[29]),c[4],(0,c[4])(c[2]),c[41])}}catch(d){return"enhanced_except_3JwBo-P-_w8_"+
	   a}return b.join("")}`
	return evalJavascript(fBody, encoded)
}

hardcoded but it helps. I was not able to rewrite yt-dlp regular expression to golang https://github.com/yt-dlp/yt-dlp/blob/2037a6414f81db8080ca724dca506fde91974c5d/yt_dlp/extractor/youtube.py#L3121

@corny
Copy link
Collaborator

corny commented Dec 21, 2024

Is this issue solved with the new version I've released yesterday?

@shoce
Copy link

shoce commented Dec 21, 2024

Is this issue solved with the new version I've released yesterday?

yes, fixed for me

@corny corny closed this as completed Dec 22, 2024
@natxo
Copy link

natxo commented Dec 24, 2024

yes, works very well, thanks to all involved fixing this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests