Python Selenium Headed v/s Headless

Avatar
by Lisa

We are automating a file download — it works fine when running headed, but headless execution doesn’t manage to log in. Proxying the requests through Fiddler show that several JavaScript pages download unexpected content.

I’ve added a user-agent to the request, but I’ve noticed that the ChromeDriver also sets sec-ch-* headers … I expect the null sec-ch-ua causes the web server to refuse our request. I don’t see any issues in the ChromeDriver repo for the sec-ch-* headers … and I don’t really want to walk back versions until I find one that doesn’t try setting this header value. Firefox’s GeckoDriver, though, doesn’t set them … so I moved the script over to use Firefox instead of Chrome and am able to download the file.

Headed run:

GET /o/telx-theme/css/A.bootstrap.css+slick,,_slick.css,Mcc.JKqfH-juDS.css.pagespeed.cf.ZO22sEGAvO.css HTTP/1.1
Host: example.com
Connection: keep-alive
sec-ch-ua: “Chromium”;v=”92″, ” Not A;Brand”;v=”99″, “Google Chrome”;v=”92″
sec-ch-ua-mobile: ?0
User-Agent: “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36
Accept: text/css,*/*;q=0.1
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: no-cors
Sec-Fetch-Dest: style
Referer: https://example.com/web/guest/login
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cookie: JSESSIONID=0330C2C988F31010790779A126EA6F55.node1; COOKIE_SUPPORT=true; GUEST_LANGUAGE_ID=en_US; AWSELB=039B496118DDEAD697B2B51C93852940763289C324F9E7C7223F953330AF5506573D13C4D5599541FD3CADB645303C1CAEB6D26992826965DA6C8BEDBDE9C297AE26CD76ED; AWSELBCORS=039B496118DDEAD697B2B51C93852940763289C324F9E7C7223F953330AF5506573D13C4D5599541FD3CADB645303C1CAEB6D26992826965DA6C8BEDBDE9C297AE26CD76ED; TS0194d418=01092b79076749232d762d2a6c232e015d103453fbeda3826bd3d20e1d937f5a90cabe03655c97a79198969eea539e4c2e7fc426216092c78ccda85763d52300ce05672704e45b4fc25516d2c24279656db7b0242f7c8b9c8bfed35b7608afb0c54bbc33d489f431059d048094c1e707a20d28031885ca6c61f81613ac299044f0c2b9ba36

 

Headless run:

GET /o/telx-theme/css/A.bootstrap.css+slick,,_slick.css,Mcc.JKqfH-juDS.css.pagespeed.cf.ZO22sEGAvO.css HTTP/1.1
Host: example.com
Connection: keep-alive
sec-ch-ua:
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36
Accept: text/css,*/*;q=0.1
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: no-cors
Sec-Fetch-Dest: style
Referer: https://example.com/web/guest/login
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US
Cookie: JSESSIONID=F4293ECE33B134CC368C0E62D6923B48.node1; COOKIE_SUPPORT=true; GUEST_LANGUAGE_ID=en_US; AWSELB=039B496118DDEAD697B2B51C93852940763289C324A5AB24AE470C70960B5319A93C181302D27B4C9425A4AA05795334C4404D491FBCC8E6A9B809746A802EAC2EC8C2FBFA; AWSELBCORS=039B496118DDEAD697B2B51C93852940763289C324A5AB24AE470C70960B5319A93C181302D27B4C9425A4AA05795334C4404D491FBCC8E6A9B809746A802EAC2EC8C2FBFA; TS0194d418=01ba3b12a4ef612e3839114024b5082fd19d56b17293c914ff867740ad37ae362e385934695ad3fc275074bfd1ee24c7d1591b146ad39d153a8758aecc8eb44d374dc1c689e540deca9566f723df65e9f5ad26551e25bacd5df14e4e6104a91a0ecdb59a65176bd5a0ebed284847e0e6618a05ed1d9db6b544e195d8e1f41164e7199a6596

Math Time – Delta Edition

Avatar
by Lisa

An update to my previous mathematical analysis of covid transmission now that I’ve seen R0 estimates for this delta variant …

The R0 value for the delta variant seems to be between 5 and 8. Looks like just over 46% of the US population is vaccinated. The vaccines are published as being 90-something percent effective. That makes an effective transmission rate between (5 * (1- (0.46 * 0.95))) and (8 * (1- (0.46 * 0.9))). Between 2.9 and 4.7 — somewhat surprising given the R0 of slightly under 3 that was published at the start of the SARS-CoV-2 outbreak. That means that, as health orders and mandates are lifted, we’re basically exactly where we were a year ago even though about half the population is vaccinated.

A mathematically interesting thing — if you could get the vaccine efficacy up to 100% (a third shot, a tenth shot, a different vaccine, whatever)? We’d still have an effective transmission rate between 2.7 and 4.3 — the value goes down, but not significantly. On the other hand, increasing the percentage of fully vaccinated individuals by 10% gives us an effective rate of transmission between 2.5 and 4.0. Having 70% of the population vaccinated would yield an effective rate of transmission between 1.8 and 3.0. We’d need to get somewhere between 90 and 98% of the population vaccinated to bring the delta variant’s effective rate down below 1 (the point where it would die out naturally)!

That tells me this virus is going to be around for a long time — especially since the R0 for some upcoming variants might be higher. Also, I’m curious to see if the government authorizes a third dose given the minimal impact increasing efficacy has on the effective rate of spread.

Fortify on Demand Remediation: Command Injection

Avatar
by Lisa

Any time user input is used to shell out and execute a command, you risk the user executing more than you want. I can string together commands in DOS using &, in Unix using ; … and stringing together commands and then executing them can blow things up spectacularly.

You can add any sort of filter to the user input to sort this … however, it doesn’t absolutely mean the vulnerability doesn’t exist. If your “user” input is trusted (in this case, it’s an automated process where some code calls some other code … so “passing” is good enough), no big. But if there are actual users involved, you should also filter out any characters that are used to string commands together.

Estados Unidos Mexicanos -v- US Gun Manufacturers

Avatar
by Lisa

Interesting approach, especially considering that the American government has a fairly long history of saying “hey, y’all — your drug cartels are a huge problem that you need to get sorted” — Mexico has sued a list of US gun manufacturers in US District Court in Mass (1:21-cv-11269). “Defendants have a duty not to supply the criminal market in Mexico” and details how Mexico feels the defendants have failed in this duty.

Fortify on Demand Remediation — Cross-Site Scripting DOM (JS)

Avatar
by Lisa

This vulnerability occurs when you accept user input or gather input from a AJAX call to another web site and then use that input in output. The solution is to sanitize the input, but Fortify on Demand seems to object strenuously to setting innerHTML … so filtering alone may not be sufficient depending on how you subsequently use the data.

To sanitize a string in JavaScript, use a function like this:

/**
 * Sanitize and encode all HTML in a string
 * @param  {string} str  The input string
 * @return {string} –    The sanitized string
 */
 var sanitizeHTML = function (str) {
    return str.replace(/&/g, ‘&amp;’).replace(/</g, ‘&lt;’).replace(/>/g, ‘&gt;’);
};

This will replace ampersands and the < and > from potential HTML tags with the HTML-encoded equivalents. To avoid using innerHTML, you might need to get a little creative. In many cases, I have a span where the results are displayed. I color-code the results based on success/failure … in that case, I an replace innerHTML with a combination of setting the css color style element to ‘green’ or ‘red’ then setting the innerText to my message string.

I can bold an entire element using a similar method. Changing some of the text, however … I haven’t come up with anything other than breaking the message into multiple HTML elements. E.g. a span for “msgStart”, one for “msgMiddle”, and one for “msgEnd” – I can then bold “msgMiddle” and set innerText for all three elements.

On Questioning Science

Avatar
by Lisa
While science is based on questioning, “questioning” means “questioning, then developing a plan to test your new hypothesis, carrying out your test, documenting and publishing your results, then discussing those results with the scientific community”.
Questioning cannot just stop with a gut feeling, some one-off event you witnessed, or something you’re neighbor’s dog-walker’s friend overheard whilst riding the bus. You cannot just believe that the acceleration of gravity on Earth is -1.5 m/s^2. You believe it, design an experiment to measure the acceleration of gravity, measure it, and … well, find out that you’re wrong.
I have a quip that I use with Anya — she knows you’re not supposed to break laws. And she knows there are “laws of physics”. So she put it together and announced proudly that we may not break the laws of physics. (And, I expect, that meant that there were some physics police wandering around ready to fine you). I tell her she’s welcome to break the laws of physics, but then she needs to publish her proposed ‘new laws of physics’ that explain what she was able to do in a peer-reviewed journal. Because they’re not laws like a group of random politicians decided something is illegal. They’re laws like the scientific community believes it is impossible. And most of us are thrilled to learn we’re wrong and gain a better understanding of the world around us.