Author: Lisa

Springfield

Governor DeWine had an editorial in the NY Times this morning — https://www.nytimes.com/2024/09/20/opinion/springfield-haitian-migrants-ohio.html — he starts out OK. Essentially I’ve got a history in that area, I know the area, and I know this nonsense to be untrue. But then he says he still supports Trump because of the horrible problems with immigration elsewhere.

Without mentioning that a bipartisan attempt to do something got scuttled by the very guy I am telling you will fix it.

Worse – without thinking maybe some of those reports are made up or embellished like this situation has been? If I know first-hand what is going on in Springfield, and I know that Trump’s portrayal of it is wildly inaccurate? Why wouldn’t I question how accurate his depiction of Colorado or even the Mexican border is?!?

They just let you steal!

This is another case of simplifying a complex situation to make someone look bad: Cali lets you walk right out if you are stealing $950 or less. And ignoring the real problem (if you want to have all of these crimes investigated and prosecuted, your local/state taxes need to go up so departments can staff accordingly and enough jails can be built to house all of these criminals).

Every state has a law that differentiates between misdemeanor theft and felony theft. e.g. https://law.justia.com/codes/arkansas/title-5/subtitle-4/chapter-36/subchapter-1/section-5-36-103/ for Arkansas at $1000. I think most people would agree that someone who steals ten dollars worth of merchandise and someone who steals three grand worth of merchandise should get different punishments. Punishments are generally defined along with the classification of the crime. And each state’s law reflects this. Different states have different dollar amounts — and $950 sounds like a lot of money. But compared to the other states? California’s demarcation is pretty middle of the pack.

Unfortunately the entire criminal justice system is overloaded. Police may be too busy to deal with your theft complaint. The prosecutor may not get around to filing charges. And what happens if they do get a conviction? Now we need to find somewhere to detain the thief. And, again, if there’s a person stealing cars, a person kidnapping minors, and a person stealing the latest video game … ideally, we’d have resources to punish all of them. But we don’t. And I’d be way more upset if the kidnapper skated because they had a task force down at the GameStop gathering video from all the surrounding stores to track down this thief.

Prop 47 absolutely has some flaws. There’s a Prop 36 this year that seems like an attempt to fix some of the unintended consequences — two or more theft charges under $950 would be a felony. Stealing from multiple places that add up to over $950 would be a felony. Won’t know if that passes until November, but it’s not like all the liberals are dancing around saying this was 100% perfect and everyone else should do it too.

 

Alabama: $500
Alaska: $750
Arizona: $1,000
Arkansas: $1,000
California: $950
Colorado: $2,000
Connecticut: $2,000
Delaware: $1,500
Florida: $750
Georgia: $1,500
Hawaii: $750
Idaho: $1,000
Illinois: $500
Indiana: $750
Iowa: $300
Kansas: $1,500
Kentucky: $1,000
Louisiana: $1,000
Maine: $1,000
Maryland: $1,500
Massachusetts: $1,200
Michigan: $1,000
Minnesota: $1,000
Mississippi: $1,000
Missouri: $750
Montana: $1,500
Nebraska: $1,500
Nevada: $1,200
New Hampshire: $1,000
New Jersey: $200
New Mexico: $500
New York: $1,000
North Carolina: $1,000
North Dakota: $1,000
Ohio: $1,000
Oklahoma: $1,000
Oregon: $1,000
Pennsylvania: $2,000
Rhode Island: $1,500
South Carolina: $2,000
South Dakota: $1,000
Tennessee: $1,000
Texas: $2,500
Utah: $1,500
Vermont: $900
Virginia: $1,000
Washington: $750
West Virginia: $1,000
Wisconsin: $2,500
Wyoming: $1,000

PostgreSQL 12 — Cascading Replication

I’ve got replicated PostgreSQL database pairs that each have some 50TB of data. The server operating systems need to be upgraded, but there is a constraint: no in-place upgrades. I don’t get to veto that constraint (i.e. the fact that we could just cross our fingers and upgrade a replica … and, if it fails, built new and pull the data again doesn’t matter). Unfortunately, trying to add a second replica delays the existing replication. Since all write operations to to the RW server and reads to to the read-only replica … having the read-only copy a day or two out of sync whilst this secondary replica comes online is a non-starter.

Fortunately, you can cascade replication — seed the new replica from the current read-only replica. Create a new replication slot — here new-pg-ro-replica-pgdata. You need to verify the new server is in the pg_hba.conf file to authenticate with the replication account.

pg_basebackup -h pg-ro-replica.example.net -D /pgdata -U replicatorID -v -P --wal-method=stream --slot=new-pg-ro-replica-pgdata

Wait … wait … wait. It’ll finish eventually. Then tweak your recovery.conf

standby_mode = 'on'
primary_conninfo = 'host=pg-rw-replica.example.net port=5432 user=replicatorID password=your_password' sslmode=require
primary_slot_name = 'new-pg-ro-replica-pgdata'

And
touch /pgdata/standby.signal

Finally, start the server

pg_ctl start -D /pgdata

Voila — a second read-only replica. Now they can decom the old server.

OpenSearch 2.x CACerts Permission Error

In my dev OpenSearch 2.x environment, I get a strange error indicating that the application cannot read the cacerts file — except the file is world readable, selinux is disabled, and there’s nothing actually preventing access from the OS level.

[2024-09-17T12:48:52,666][ERROR][c.a.d.a.h.j.AbstractHTTPJwtAuthenticator] [linux1569.mgmt.windstream.net] Error creating JWT authenticator. JWT authentication will not work
com.amazon.dlic.util.SettingsBasedSSLConfigurator$SSLConfigException: Error loading trust store from /opt/elk/opensearch/jdk/lib/security/cacerts
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.initFromKeyStore(SettingsBasedSSLConfigurator.java:338) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.configureWithSettings(SettingsBasedSSLConfigurator.java:196) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLContext(SettingsBasedSSLConfigurator.java:117) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLConfig(SettingsBasedSSLConfigurator.java:131) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.getSSLConfig(HTTPJwtKeyByOpenIdConnectAuthenticator.java:65) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.initKeyProvider(HTTPJwtKeyByOpenIdConnectAuthenticator.java:47) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
        at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.<init>(AbstractHTTPJwtAuthenticator.java:89) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.<init>(HTTPJwtKeyByOpenIdConnectAuthenticator.java:26) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
        at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.lambda$newInstance$1(DynamicConfigModelV7.java:432) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) [?:?]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.newInstance(DynamicConfigModelV7.java:430) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.buildAAA(DynamicConfigModelV7.java:329) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.<init>(DynamicConfigModelV7.java:102) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.securityconf.DynamicConfigFactory.onChange(DynamicConfigFactory.java:288) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.configuration.ConfigurationRepository.notifyAboutChanges(ConfigurationRepository.java:570) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.configuration.ConfigurationRepository.notifyConfigurationListeners(ConfigurationRepository.java:559) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:554) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.configuration.ConfigurationRepository.loadConfigurationWithLock(ConfigurationRepository.java:538) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:531) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.configuration.ConfigurationRepository.initalizeClusterConfiguration(ConfigurationRepository.java:284) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.configuration.ConfigurationRepository.lambda$initOnNodeStart$10(ConfigurationRepository.java:439) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/opt/elk/opensearch/jdk/lib/security/cacerts" "read")
        at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488) ~[?:?]
        at java.base/java.security.AccessController.checkPermission(AccessController.java:1071) ~[?:?]
        at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
        at java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:742) ~[?:?]
        at java.base/sun.nio.fs.UnixPath.checkRead(UnixPath.java:789) ~[?:?]
        at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:49) ~[?:?]
        at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:171) ~[?:?]
        at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?]
        at java.base/java.nio.file.spi.FileSystemProvider.readAttributesIfExists(FileSystemProvider.java:1270) ~[?:?]
        at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributesIfExists(UnixFileSystemProvider.java:191) ~[?:?]
        at java.base/java.nio.file.Files.isDirectory(Files.java:2319) ~[?:?]
        at org.opensearch.security.support.PemKeyReader.checkPath(PemKeyReader.java:214) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.support.PemKeyReader.resolve(PemKeyReader.java:290) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.support.PemKeyReader.resolve(PemKeyReader.java:276) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.initFromKeyStore(SettingsBasedSSLConfigurator.java:327) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
        ... 25 more

Looks like Java has its own security mechanism — the java.policy needed to be updated to allow read access to cacerts (why!?!?!?)

vi /opt/elk/opensearch/jdk/conf/security/java.policy

# Add this grant:

    permission java.io.FilePermission "/opt/elk/opensearch/jdk/lib/security/cacerts", "read";


Why so militant

Someone asked why some feminists are so anti-anything-feminine. I think a silly analogy makes it easier to understand:

If there were a law that required everyone to eat pizza for dinner every day, and a whole freedom movement evolved to ensure we could all pick our own dinner? I expect some people would be so adverse to pizza as to never eat it again. Electing to eat pizza (because I love pizza, just not every day) could be seen as an insult to the Dinner Liberation movement.

Viewing the movement as action so you never had to eat pizza again rather than action so you could chose from the entire world of options including the one that had formerly been forced upon you.

Trusting Science

Kinda hard question for me, as a scientist, if I trust science or trust experts. Few who ask are honestly curious – they’ve got an agenda. I generally trust Science and Experts. *But* I also know that Science and Experts aren’t always right. They are generally right with the information they had available at the time, the measuring tools they had available at the time, etc. It’s surprisingly easy to do nothing wrong and still manage to arrive at the wrong conclusion. There are some things that have remained consistent over enough time and testing that they’re generally accepted as true (scientific theories). But even that name … scientists aren’t out there claiming it’s the complete, never changing truth. It’s the current theory.

What I don’t trust second-hand accounts of science or experts. There is generally a peer-reviewed publication that makes a cumbersome read. With a lot of details you don’t really need. But! It’s also exactly what was studied, how it was studied, what conclusions the researchers drew, how statistically significant the findings were, and other factors that should be included in future studies. A newspaper article claiming researchers say XYZ? I’ll use my internet search engine of choice to find the actual article if I’m interested in the claim. It’s a newspaper’s summary of a PR guy’s summary of the abstract written by an expert to explain something that requires domain knowledge to understand well.

Removing and Recreating a ZFS Pool

In testing out various ways to achieve disk compression on our PostgreSQL servers, I ended up with a server build with a version of ZFS newer that the package distribution. Which means I needed to recreate the pool to use an older version of ZFS that would be updated as part of the routine patching. Beyond backing up and restoring the data …

# Get rid of existing pool

zpool export pgpool
zpool destroy pgpool
zpool list # this still shows a pool on sdb

# Clear the label

zpool labelclear /dev/sdb

# Didn’t work, so blow away everything on sdb

dd if=/dev/zero of=/dev/sdb bs=1M count=10
wipefs -a /dev/sdb

# Uninstall custom built zfs

cd /root/zfs
make uninstall

Install new ZFS

yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
yum install kernel-devel

yum install https://zfsonlinux.org/epel/zfs-release-2-3$(rpm –eval “%{dist}”).noarch.rpm
dnf config-manager –disable zfs
dnf config-manager –enable zfs-kmod
yum install zfs

# Sign kernel modules

/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/$(uname -r)/extras/zfs/avl/zavl.ko

/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/$(uname -r)/extras/zfs/zfs/zfs.ko

# Reboot

init 6

# And start over — recreate the pool

zpool create pgpool sdb
zfs create pgpool/pgdata
zfs set compression=lz4 pgpool/pgdata
df -h /pgpool/pgdata/

Outnumbered

Any political system requires “buy in” from a very large percentage of those being governed. If the 600,000 people who live in Wyoming thought they had absolutely no say in how the country ran because there are like 8 million people in NYC, 3.5 million in LA. Or even the just about 600,000 people who live in Harrisburg, PA … they don’t have much incentive to peaceful participation in the federal government.

I hear folks in rural parts of Ohio saying the same thing — cannabis will be legalized because of the three C’s (Cinci, Columbus, Cleveland) and they get no say in it. Which made me curious — just how “outnumbered” are these “rural” folks. So I grabbed a list of cities in Ohio with population numbers. Columbus is huge, almost a million people! But there are almost 12 million people in Ohio. So Columbus is just under 8% of the population. Add in Cleveland, and you are up to almost 11% of the population. Keep going — add Cinci to get 13.5%. So the “three C’s” are only 13.5% of the entire population. Toledo gets us over 15%. But these are hardly “bossing everyone else around” percentages. I was down to the 143rd largest city in Ohio — Fostoria with just over 13,000 people — before “cities” account for 50% of the state’s population.

And that assumes 100% of people in urban areas are voting against whatever 100% of these rural people want to see happen. Which is absurd. If 80% of the people in these cities were voting against “the rural way”, we’re adding cities with just under 4,000 residents before we reach 50%.

If 75% are voting against “the rural way”, we’re down to cities with just under 2,000 residents.

While I think Wyoming is probably right — there are enough liberal voters nationally that conservatives would be “outnumbered” without creative districting, over-representation in the senate, and over-representation in the electoral college … the same doesn’t seem to hold true in Ohio.

Hot and Fast Smoked Ribs

Unfortunately, I don’t think we’re going to be able to replicate this exactly — had a bunch of bad temperature probes, so we didn’t realize how hot our smoker was going. We usually do the 3-2-1 thing at like 225F … but the smoker chamber was over 300 degrees. So I did a quick search for “what happens if I smoke my pork ribs at 300” and discovered the hot and fast method. Absolutely no idea that was even a thing!

So we put the ribs on at about 6PM, and it was about 9PM by the time we were eating. I coated them with some spice rubs about an hour before we started cooking. We brushed them with a butter/spice mixture after about an hour of cooking and again about half an hour before pulling them. They were tender but not mush, crispy, tasty (a little too much salt in places, but that’s just the rub), moist, and incredible. While I have no idea what the temperature we cooked these things at, we’re going to try to use this method again after we get some good temp probes.