We are automating a file download — it works fine when running headed, but headless execution doesn’t manage to log in. Proxying the requests through Fiddler show that several JavaScript pages download unexpected content.

I’ve added a user-agent to the request, but I’ve noticed that the ChromeDriver also sets sec-ch-* headers … I expect the null sec-ch-ua causes the web server to refuse our request. I don’t see any issues in the ChromeDriver repo for the sec-ch-* headers … and I don’t really want to walk back versions until I find one that doesn’t try setting this header value. Firefox’s GeckoDriver, though, doesn’t set them … so I moved the script over to use Firefox instead of Chrome and am able to download the file.

Headed run:

GET /o/telx-theme/css/A.bootstrap.css+slick,,_slick.css,Mcc.JKqfH-juDS.css.pagespeed.cf.ZO22sEGAvO.css HTTP/1.1
Host: example.com
Connection: keep-alive
sec-ch-ua: “Chromium”;v=”92″, ” Not A;Brand”;v=”99″, “Google Chrome”;v=”92″
sec-ch-ua-mobile: ?0
User-Agent: “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36
Accept: text/css,*/*;q=0.1
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: no-cors
Sec-Fetch-Dest: style
Referer: https://example.com/web/guest/login
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cookie: JSESSIONID=0330C2C988F31010790779A126EA6F55.node1; COOKIE_SUPPORT=true; GUEST_LANGUAGE_ID=en_US; AWSELB=039B496118DDEAD697B2B51C93852940763289C324F9E7C7223F953330AF5506573D13C4D5599541FD3CADB645303C1CAEB6D26992826965DA6C8BEDBDE9C297AE26CD76ED; AWSELBCORS=039B496118DDEAD697B2B51C93852940763289C324F9E7C7223F953330AF5506573D13C4D5599541FD3CADB645303C1CAEB6D26992826965DA6C8BEDBDE9C297AE26CD76ED; TS0194d418=01092b79076749232d762d2a6c232e015d103453fbeda3826bd3d20e1d937f5a90cabe03655c97a79198969eea539e4c2e7fc426216092c78ccda85763d52300ce05672704e45b4fc25516d2c24279656db7b0242f7c8b9c8bfed35b7608afb0c54bbc33d489f431059d048094c1e707a20d28031885ca6c61f81613ac299044f0c2b9ba36

 

Headless run:

GET /o/telx-theme/css/A.bootstrap.css+slick,,_slick.css,Mcc.JKqfH-juDS.css.pagespeed.cf.ZO22sEGAvO.css HTTP/1.1
Host: example.com
Connection: keep-alive
sec-ch-ua:
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36
Accept: text/css,*/*;q=0.1
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: no-cors
Sec-Fetch-Dest: style
Referer: https://example.com/web/guest/login
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US
Cookie: JSESSIONID=F4293ECE33B134CC368C0E62D6923B48.node1; COOKIE_SUPPORT=true; GUEST_LANGUAGE_ID=en_US; AWSELB=039B496118DDEAD697B2B51C93852940763289C324A5AB24AE470C70960B5319A93C181302D27B4C9425A4AA05795334C4404D491FBCC8E6A9B809746A802EAC2EC8C2FBFA; AWSELBCORS=039B496118DDEAD697B2B51C93852940763289C324A5AB24AE470C70960B5319A93C181302D27B4C9425A4AA05795334C4404D491FBCC8E6A9B809746A802EAC2EC8C2FBFA; TS0194d418=01ba3b12a4ef612e3839114024b5082fd19d56b17293c914ff867740ad37ae362e385934695ad3fc275074bfd1ee24c7d1591b146ad39d153a8758aecc8eb44d374dc1c689e540deca9566f723df65e9f5ad26551e25bacd5df14e4e6104a91a0ecdb59a65176bd5a0ebed284847e0e6618a05ed1d9db6b544e195d8e1f41164e7199a6596