Author: Lisa

PostgreSQL 12 — Cascading Replication

I’ve got replicated PostgreSQL database pairs that each have some 50TB of data. The server operating systems need to be upgraded, but there is a constraint: no in-place upgrades. I don’t get to veto that constraint (i.e. the fact that we could just cross our fingers and upgrade a replica … and, if it fails, built new and pull the data again doesn’t matter). Unfortunately, trying to add a second replica delays the existing replication. Since all write operations to to the RW server and reads to to the read-only replica … having the read-only copy a day or two out of sync whilst this secondary replica comes online is a non-starter.

Fortunately, you can cascade replication — seed the new replica from the current read-only replica. Create a new replication slot — here new-pg-ro-replica-pgdata. You need to verify the new server is in the pg_hba.conf file to authenticate with the replication account.

pg_basebackup -h pg-ro-replica.example.net -D /pgdata -U replicatorID -v -P --wal-method=stream --slot=new-pg-ro-replica-pgdata

Wait … wait … wait. It’ll finish eventually. Then tweak your recovery.conf

standby_mode = 'on'
primary_conninfo = 'host=pg-rw-replica.example.net port=5432 user=replicatorID password=your_password' sslmode=require
primary_slot_name = 'new-pg-ro-replica-pgdata'

And
touch /pgdata/standby.signal

Finally, start the server

pg_ctl start -D /pgdata

Voila — a second read-only replica. Now they can decom the old server.

OpenSearch 2.x CACerts Permission Error

In my dev OpenSearch 2.x environment, I get a strange error indicating that the application cannot read the cacerts file — except the file is world readable, selinux is disabled, and there’s nothing actually preventing access from the OS level.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
[2024-09-17T12:48:52,666][ERROR][c.a.d.a.h.j.AbstractHTTPJwtAuthenticator] [linux1569.mgmt.windstream.net] Error creating JWT authenticator. JWT authentication will not work
com.amazon.dlic.util.SettingsBasedSSLConfigurator$SSLConfigException: Error loading trust store from /opt/elk/opensearch/jdk/lib/security/cacerts
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.initFromKeyStore(SettingsBasedSSLConfigurator.java:338) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.configureWithSettings(SettingsBasedSSLConfigurator.java:196) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLContext(SettingsBasedSSLConfigurator.java:117) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLConfig(SettingsBasedSSLConfigurator.java:131) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.getSSLConfig(HTTPJwtKeyByOpenIdConnectAuthenticator.java:65) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.initKeyProvider(HTTPJwtKeyByOpenIdConnectAuthenticator.java:47) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
        at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.<init>(AbstractHTTPJwtAuthenticator.java:89) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.<init>(HTTPJwtKeyByOpenIdConnectAuthenticator.java:26) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
        at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.lambda$newInstance$1(DynamicConfigModelV7.java:432) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) [?:?]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.newInstance(DynamicConfigModelV7.java:430) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.buildAAA(DynamicConfigModelV7.java:329) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.<init>(DynamicConfigModelV7.java:102) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.securityconf.DynamicConfigFactory.onChange(DynamicConfigFactory.java:288) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.configuration.ConfigurationRepository.notifyAboutChanges(ConfigurationRepository.java:570) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.configuration.ConfigurationRepository.notifyConfigurationListeners(ConfigurationRepository.java:559) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:554) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.configuration.ConfigurationRepository.loadConfigurationWithLock(ConfigurationRepository.java:538) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:531) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.configuration.ConfigurationRepository.initalizeClusterConfiguration(ConfigurationRepository.java:284) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.configuration.ConfigurationRepository.lambda$initOnNodeStart$10(ConfigurationRepository.java:439) [opensearch-security-2.15.0.0.jar:2.15.0.0]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/opt/elk/opensearch/jdk/lib/security/cacerts" "read")
        at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488) ~[?:?]
        at java.base/java.security.AccessController.checkPermission(AccessController.java:1071) ~[?:?]
        at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
        at java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:742) ~[?:?]
        at java.base/sun.nio.fs.UnixPath.checkRead(UnixPath.java:789) ~[?:?]
        at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:49) ~[?:?]
        at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:171) ~[?:?]
        at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?]
        at java.base/java.nio.file.spi.FileSystemProvider.readAttributesIfExists(FileSystemProvider.java:1270) ~[?:?]
        at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributesIfExists(UnixFileSystemProvider.java:191) ~[?:?]
        at java.base/java.nio.file.Files.isDirectory(Files.java:2319) ~[?:?]
        at org.opensearch.security.support.PemKeyReader.checkPath(PemKeyReader.java:214) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.support.PemKeyReader.resolve(PemKeyReader.java:290) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
        at org.opensearch.security.support.PemKeyReader.resolve(PemKeyReader.java:276) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.initFromKeyStore(SettingsBasedSSLConfigurator.java:327) ~[opensearch-security-2.15.0.0.jar:2.15.0.0]
        ... 25 more

Looks like Java has its own security mechanism — the java.policy needed to be updated to allow read access to cacerts (why!?!?!?)

1
2
3
4
5
vi /opt/elk/opensearch/jdk/conf/security/java.policy
 
# Add this grant:
 
    permission java.io.FilePermission "/opt/elk/opensearch/jdk/lib/security/cacerts", "read";

Why so militant

Someone asked why some feminists are so anti-anything-feminine. I think a silly analogy makes it easier to understand:

If there were a law that required everyone to eat pizza for dinner every day, and a whole freedom movement evolved to ensure we could all pick our own dinner? I expect some people would be so adverse to pizza as to never eat it again. Electing to eat pizza (because I love pizza, just not every day) could be seen as an insult to the Dinner Liberation movement.

Viewing the movement as action so you never had to eat pizza again rather than action so you could chose from the entire world of options including the one that had formerly been forced upon you.

Trusting Science

Kinda hard question for me, as a scientist, if I trust science or trust experts. Few who ask are honestly curious – they’ve got an agenda. I generally trust Science and Experts. *But* I also know that Science and Experts aren’t always right. They are generally right with the information they had available at the time, the measuring tools they had available at the time, etc. It’s surprisingly easy to do nothing wrong and still manage to arrive at the wrong conclusion. There are some things that have remained consistent over enough time and testing that they’re generally accepted as true (scientific theories). But even that name … scientists aren’t out there claiming it’s the complete, never changing truth. It’s the current theory.

What I don’t trust second-hand accounts of science or experts. There is generally a peer-reviewed publication that makes a cumbersome read. With a lot of details you don’t really need. But! It’s also exactly what was studied, how it was studied, what conclusions the researchers drew, how statistically significant the findings were, and other factors that should be included in future studies. A newspaper article claiming researchers say XYZ? I’ll use my internet search engine of choice to find the actual article if I’m interested in the claim. It’s a newspaper’s summary of a PR guy’s summary of the abstract written by an expert to explain something that requires domain knowledge to understand well.

Removing and Recreating a ZFS Pool

In testing out various ways to achieve disk compression on our PostgreSQL servers, I ended up with a server build with a version of ZFS newer that the package distribution. Which means I needed to recreate the pool to use an older version of ZFS that would be updated as part of the routine patching. Beyond backing up and restoring the data …

# Get rid of existing pool

zpool export pgpool
zpool destroy pgpool
zpool list # this still shows a pool on sdb

# Clear the label

zpool labelclear /dev/sdb

# Didn’t work, so blow away everything on sdb

dd if=/dev/zero of=/dev/sdb bs=1M count=10
wipefs -a /dev/sdb

# Uninstall custom built zfs

cd /root/zfs
make uninstall

Install new ZFS

yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
yum install kernel-devel

yum install https://zfsonlinux.org/epel/zfs-release-2-3$(rpm –eval “%{dist}”).noarch.rpm
dnf config-manager –disable zfs
dnf config-manager –enable zfs-kmod
yum install zfs

# Sign kernel modules

/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/$(uname -r)/extras/zfs/avl/zavl.ko
…
/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/$(uname -r)/extras/zfs/zfs/zfs.ko

# Reboot

init 6

# And start over — recreate the pool

zpool create pgpool sdb
zfs create pgpool/pgdata
zfs set compression=lz4 pgpool/pgdata
df -h /pgpool/pgdata/

Outnumbered

Any political system requires “buy in” from a very large percentage of those being governed. If the 600,000 people who live in Wyoming thought they had absolutely no say in how the country ran because there are like 8 million people in NYC, 3.5 million in LA. Or even the just about 600,000 people who live in Harrisburg, PA … they don’t have much incentive to peaceful participation in the federal government.

I hear folks in rural parts of Ohio saying the same thing — cannabis will be legalized because of the three C’s (Cinci, Columbus, Cleveland) and they get no say in it. Which made me curious — just how “outnumbered” are these “rural” folks. So I grabbed a list of cities in Ohio with population numbers. Columbus is huge, almost a million people! But there are almost 12 million people in Ohio. So Columbus is just under 8% of the population. Add in Cleveland, and you are up to almost 11% of the population. Keep going — add Cinci to get 13.5%. So the “three C’s” are only 13.5% of the entire population. Toledo gets us over 15%. But these are hardly “bossing everyone else around” percentages. I was down to the 143rd largest city in Ohio — Fostoria with just over 13,000 people — before “cities” account for 50% of the state’s population.

And that assumes 100% of people in urban areas are voting against whatever 100% of these rural people want to see happen. Which is absurd. If 80% of the people in these cities were voting against “the rural way”, we’re adding cities with just under 4,000 residents before we reach 50%.

If 75% are voting against “the rural way”, we’re down to cities with just under 2,000 residents.

While I think Wyoming is probably right — there are enough liberal voters nationally that conservatives would be “outnumbered” without creative districting, over-representation in the senate, and over-representation in the electoral college … the same doesn’t seem to hold true in Ohio.

Hot and Fast Smoked Ribs

Unfortunately, I don’t think we’re going to be able to replicate this exactly — had a bunch of bad temperature probes, so we didn’t realize how hot our smoker was going. We usually do the 3-2-1 thing at like 225F … but the smoker chamber was over 300 degrees. So I did a quick search for “what happens if I smoke my pork ribs at 300” and discovered the hot and fast method. Absolutely no idea that was even a thing!

So we put the ribs on at about 6PM, and it was about 9PM by the time we were eating. I coated them with some spice rubs about an hour before we started cooking. We brushed them with a butter/spice mixture after about an hour of cooking and again about half an hour before pulling them. They were tender but not mush, crispy, tasty (a little too much salt in places, but that’s just the rub), moist, and incredible. While I have no idea what the temperature we cooked these things at, we’re going to try to use this method again after we get some good temp probes.

Figs

Came across someone referencing the “God Hates Figs” bumper sticker as presumed “religious nuts” online today.

Those are generally folks against religious nuts – Westboro Baptist Church had a domain for god hates fags. There was a coordinated effort online to overrun their message board (they shut it down eventually) and god hates figs came up as a joke. You can find all sorts of biblical citations showing that god hates figs. Far more than those showing that god hates people of a specific sexual orientation. Someone bought a domain, and there you have it. God hates figs.

Not saying there aren’t people who took it seriously – but it’s meant like the “dangers of dihydro-monoxide” thing. A bit of a riff on people who use the Bible to justify hatred.

The Best Medal?

I’m not sure if this is tripling down, quadrupling down, or what … but Trump’s made more statements about how he’d prefer a Medal of Freedom because Medal of Honor recipients are gravely injured or even killed.

Beyond the incredibly absurd notion that he’d qualify for a Medal of Honor — his assertion boils down to this: a Medal of Honor winner gave something significant of themselves. They put themselves into an incredibly dangerous situation to allow the rest of us to enjoy our Constitutionally protected freedoms. They earned that medal, often with their own body or even their own life.

Medal of Freedom winners also did something — but the person to whom Trump was referring? Was a doctor, who did what many other doctors do without being awarded medals. The ‘extra’ that got the medal? They gave money. Some of which went to Republican campaigns and conservative causes. Now, I presume even DonOld realized you couldn’t actually award a Medal of Freedom to someone for donations to his campaign. She (and her husband) also gave money to research centers fighting substance abuse, medical research, and Holocaust remembrance — not a bad thing, but there’s a huge difference between giving your life for something and giving your time and money for something.

Of course DonOld would rather be lauded for handing over some money (or, in his case, creating a charity to allow him to get praise for handing over other people’s money to beneficial causes) than actually have to give something of himself.

 

https://trumpwhitehouse.archives.gov/medaloffreedom/

The following individuals received the Presidential Medal of Freedom from President Trump on November 16, 2018:

MIRIAM ADELSON. Miriam Adelson is a committed doctor, philanthropist, and humanitarian. She has practiced internal and emergency medicine, studied and specialized in the disease of narcotic addiction, and founded two research centers committed to fighting substance abuse. With her husband, Sheldon, she also established the Adelson Medical Research Foundation, which supports research to prevent, reduce, or eliminate disabling and life-threatening illness. As a committed member of the American Jewish community, she has supported Jewish schools, Holocaust memorial organizations, Friends of the Israel Defense Forces, and Birthright Israel, among other causes.