Tag: system recovery

Recovering A Seriously Screwed Up Fedora System

The graphical interface on a Fedora 28 laptop was unavailable — buggered up video device/driver. Change to what used to be called run level 3, and we could not log in! We know the root password, but it would not take it. Single user is password protected too — and we were unable to log in there.

Normal recovery process:

Get to the grub menu, highlight the kernel you want to boot, and hit ‘e’ to edit it. Scroll down. On line that starts with linux16, change “rhgb quiet” to say “rd.break enforcing=0”
ctrl-x to boot

Once you get a shell:
mount -o remount,rw /sysroot
chroot /sysroot

Voila, you’ve got access to your files. Use vi to edit whatever has the box seriously screwed up (passwd if your problem is that you don’t know the root password) and you’re set. We reset the root password just in case. Aaaand … we still couldn’t log in on init 1 or init 3! And at this point I was feeling stubborn about getting logged into the box.

Now you can tweak up the system so it is not using sulogin when booting into single user mode but that isn’t a good way to install network-sourced packages. For some reason, we had to disable selinux before we could log into anything other than the graphical target. I’m sure there is a policy we could have tweaked, but it was far easier to disable the thing, boot into the multi-user target, sort the video driver, and then boot into the graphical target.

USN Rollback

I had to recover my domain controller from the Hyper-V image backup. There’s some protection build into AD which prevents just randomly reverting a server. When you’ve got a larger domain, the built-in protection after unsupported restoration procedures serves a purpose. Pausing netlogon avoids having users log on against bad data. Disabling replication avoids propagating bad information out to the remainder of the network. The solution is simple – demote the DC, promote it again, and the DC returns to service. But when you have a single domain controller in a single domain in a single forest … well, there’s no other data around. What the recovered DC has is as good as it’s going to get (i.e. a change from 2AM is lost when I revert to my 10PM backup). And taking the entire domain down and building it overkill. You can, instead, basically tell AD to go with it. From the MS documentation:

To restore a previous version of a virtual domain controller VHD without system state data backup

  1. Using the previous VHD, start the virtual domain controller in DSRM, as described in the previous section. Do not allow the domain controller to start in normal mode. If you miss the Windows Boot Manager screen and the domain controller begins to start in normal mode, turn off the virtual machine to prevent it from completing startup. See the previous section for detailed instructions for entering DSRM.
  2. Open Registry Editor. To open Registry Editor, click Start, click Run, type regedit, and then click OK. If the User Account Control dialog box appears, confirm that the action it displays is what you want, and then click Yes. In Registry Editor, expand the following path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters. Look for a value named DSA Previous Restore Count. If the value is there, make a note of the setting. If the value is not there, the setting is equal to the default, which is zero. Do not add a value if you do not see one there.
  3. Right-click the Parameters key, click New, and then click DWORD (32-bit) Value.
  4. Type the new name Database restored from backup, and then press ENTER.
  5. Double-click the value that you just created to open the Edit DWORD (32-bit) Value dialog box, and then type 1 in the Value data box. The Database restored from backup entry option is available on domain controllers that are running Windows 2000 Server with Service Pack 4 (SP4), Windows Server 2003 with the updates that are included in article 875495 (http://go.microsoft.com/fwlink/?LinkId=137182) in the Microsoft Knowledge Base installed, and Windows Server 2008.
  6. Restart the domain controller in normal mode.
  7. When the domain controller restarts, open Event Viewer. To open Event Viewer, click Start, click Control Panel, double-click Administrative Tools, and then double-click Event Viewer.
  8. Expand Application and Services Logs, and then click the Directory Services log. Ensure that events appear in the details pane.
  9. Right-click the Directory Services log, and then click Find. In Find what, type 1109, and then click Find Next.
  10. You should see at least an Event ID 1109 entry. If you do not see this entry, proceed to the next step. Otherwise, double-click the entry, and then review the text confirming that the update was made to the InvocationID:

 

  • Active Directory has been restored from backup media, or has been configured to host an application partition. 
The invocationID attribute for this directory server has been changed. 
The highest update sequence number at the time the backup was created is <time>

InvocationID attribute (old value):<Previous InvocationID value>
InvocationID attribute (new value):<New InvocationID value>
Update sequence number:<USN>

The InvocationID is changed when a directory server is restored from backup media or is configured to host a writeable application directory partition.
  • Close Event Viewer.
  • Use Registry Editor to verify that the value in DSA Previous Restore Count is equal to the previous value plus one. If this is not the correct value and you cannot find an entry for Event ID 1109 in Event Viewer, verify that the domain controller’s service packs are current. You cannot try this procedure again on the same VHD. You can try again on a copy of the VHD or a different VHD that has not been started in normal mode by starting over at step 1.
  • Close Registry Editor.

 

After following the instructions from Microsoft, I still had a problem — my DC has replication turned off & netlogon comes up paused. In regedit, locate HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters and delete the “Dsa Not Writable” key (value: dword:00000004). In a command prompt, run the following:

 

repadmin /options dchostname.example.com -DISABLE_OUTBOUND_REPL
repadmin /options dchostname.example.com -DISABLE_INBOUND_REPL

Reboot the DC. When it starts, netlogon should be running and replication.