A NixOS Major Upgrade Without Relying on Luck: 25.11 to 26.05

The machine I upgraded was not a disposable test box.

It is my daily GNOME desktop and development environment, but it also runs PostgreSQL, Redis, Docker, Ollama, Traefik, Open WebUI, and several other services. Moving from NixOS 25.11 to 26.05 changes systemd, D-Bus, Docker, GNOME, NetworkManager, glibc, and more. If I had simply run nixos-rebuild switch, a failure would have left me wondering whether the operating system, a database, a container image, or the live desktop session was responsible.

The most valuable result of this upgrade was therefore not a list of renamed NixOS options. It was a safer upgrade process: reduce the number of moving parts, build everything in advance, prepare the rollback before deployment, and validate the new system with real services and real data.

TL;DR

This was the final upgrade sequence:

Six principles are worth carrying into future upgrades:

system.stateVersion is not the currently installed NixOS release. Do not bump it automatically.
Keep the operating-system upgrade separate from application upgrades. Temporarily pin container images to exact digests.
A successful nix eval does not prove that the system can be built. Build the complete system closure.
When several foundational components change at once, prefer a boot deployment over a live switch.
System generations and data backups recover from different classes of failure. You need both.
Validation must outlive the first few minutes after boot. Timers and delayed jobs may fail later.

Why I Did Not Run `nixos-rebuild switch`

NixOS has excellent declarative configuration and generation rollback, but neither eliminates every runtime risk.

The candidate system included all of these changes at once:

D-Bus moved from dbus-daemon to dbus-broker.
The initrd switched to systemd.
systemd, Docker, GNOME, and NetworkManager crossed release boundaries.
The glibc update affected PostgreSQL collation metadata.
Home Manager required several option migrations.
Containers still using latest or main could have upgraded their applications during the reboot.

switch activates services inside the currently running system. That is convenient for a small configuration change. It is less attractive when foundational services, the desktop session, and stateful applications all change together. The result can be an awkward intermediate state: the target system on disk is new, some processes still belong to the old environment, and other services have already restarted.

I used the prebuilt result instead:

bash
sudo ./result-26.05/bin/switch-to-configuration boot

This installs the new generation as the next boot target without hot-switching every service in the current session. The old system stays intact until the backups and checks are complete, and the reboot creates a clean boundary: a complete old system before it, and a complete new system after it.

Step 1: Capture a Baseline Before Changing Versions

Before touching the Flake, I recorded:

the output of nixos-version, plus the kernel, Nix, and systemd versions;
available space on / and /boot;
failed systemd units;
versions and health of PostgreSQL, Redis, Docker, and other stateful services;
running containers and their resolved image digests;
the current system generation and at least one older bootable generation.

This may not feel like progress, but it determines whether two important questions can be answered later: What actually changed, and did this problem already exist before the upgrade?

Do not bump `stateVersion`

I deliberately kept the existing values:

nix
system.stateVersion = "25.05";
home.stateVersion = "24.05";

stateVersion is not a display field for the installed release. It declares which historical compatibility behavior the system should preserve, and it can affect service data directories, defaults, and data formats.

Updating nixpkgs to 26.05 does not mean that system.stateVersion should also become 26.05. Unless you intend to review and perform the associated state migrations, leaving it unchanged is usually the safe choice.

Protect the rollback path first

My maintenance job previously ran this every week:

bash
nix-collect-garbage -d

The -d flag deletes every non-current generation, which directly conflicts with the goal of retaining the old system after an upgrade. I changed it to:

bash
nix-collect-garbage --delete-older-than 30d

NixOS can create rollback generations, but that does not mean your garbage-collection policy will preserve them. Rollback capability is itself something that must be maintained.

Step 2: Freeze Applications to Reduce the Number of Variables

Before the upgrade, Open WebUI, Dockhand, and Traefik used main, latest, or ordinary version tags. If Docker restarted after those tags had moved in the registry, an operating-system upgrade could silently become three application upgrades as well, potentially including application database migrations.

I resolved the image used by each running container and pinned that exact digest in the configuration:

nix
image = "ghcr.io/example/app@sha256:<digest>";

After reboot, the applications would therefore run the same images that had already been working on the old system.

The lesson is not that tags should never be used. It is that one change window should deal with one layer of the stack. Finish and validate the operating-system upgrade first, then upgrade the applications as separate, reviewable changes. Otherwise, the troubleshooting space grows exponentially.

Step 3: Migrate the Configuration, Not Just the Option Names

The direct part of the 25.11-to-26.05 migration was updating the Flake releases:

nix
nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";

home-manager = {
  url = "github:nix-community/home-manager/release-26.05";
  inputs.nixpkgs.follows = "nixpkgs";
};

I then followed evaluation errors and deprecation warnings through the configuration. A few cases were particularly instructive.

Move `systemd-resolved` to structured settings

The old extraConfig, dnssec, and fallbackDns options became structured settings:

nix
services.resolved.settings.Resolve = {
  DNSSEC = false;
  DNSOverTLS = "no";
  MulticastDNS = "no";
  LLMNR = "yes";
  Cache = "yes";
  FallbackDNS = [
    "1.1.1.1"
    "8.8.8.8"
  ];
};

This looks like a syntax-only migration, but it still requires runtime verification. After reboot, I used resolvectl status to inspect the effective resolver, per-link DNS configuration, and resolv.conf mode.

Select the Ollama CPU package explicitly

The previous configuration used a Boolean acceleration setting:

nix
services.ollama.acceleration = false;

The new release expresses that choice through the package:

nix
services.ollama.package = pkgs.ollama-cpu;

When migrating configuration, preserve the original intent instead of merely searching for a new option with a similar name. The behavior I needed to retain here was “do not enable GPU acceleration.”

Pin the PostgreSQL server and client to the same major version

The PostgreSQL service was already pinned to version 15:

nix
services.postgresql.package = pkgs.postgresql_15;

The system package list, however, still contained the unversioned postgresql package. Under the new nixpkgs release, that resolved to PostgreSQL 17. The full system build then found file collisions between the PostgreSQL 15 and 17 outputs in system-path.

The fix was to pin the CLI tools as well:

nix
environment.systemPackages = with pkgs; [
  postgresql_15
];

Pinning a service does not necessarily pin every related tool in the system. For databases, compilers, and language runtimes with major-version compatibility boundaries, inspect the closure for accidental second versions.

Remove Flake outputs that are no longer delivered

The ThinkPad configuration and Home Manager package both built successfully, but nix flake check --no-build still found an obsolete NUC/Proxmox configuration that no longer evaluated under 26.05.

It could not stop the ThinkPad from booting, but it proved that the Flake was no longer internally consistent. After confirming that the machine had been retired, I removed its output, input, and host configuration.

An output with no owner and no continuous validation is not a harmless archive for possible future use. It is hidden maintenance cost waiting for the next upgrade.

Step 4: Build Everything—Do Not Treat `eval` as Acceptance

I first confirmed that the target system evaluated:

bash
nix eval --raw \
  .#nixosConfigurations.nixos.config.system.build.toplevel.drvPath

Then I performed a real build of the complete system:

bash
nix build \
  .#nixosConfigurations.nixos.config.system.build.toplevel \
  --print-build-logs \
  -o result-26.05

I also built the standalone Home Manager activation package:

bash
nix build \
  .#homeConfigurations.cheverjohn.activationPackage \
  --no-link

An evaluation proves that module merging and expression evaluation succeed. It does not catch every problem, including:

two packages installing the same file into system-path;
an unavailable upstream download for proprietary software;
a Home Manager activation package that fails to build;
an error while generating a service script;
a NAR missing from a regional binary cache with no fallback substituter.

This system required more than a thousand derivations, and its closure grew from roughly 31.6 GiB to 34.0 GiB. A full build costs more than an evaluation, but that cost is best paid while the old system is still healthy—not after rebooting into the new one.

Diff the closures, not just the source code

Once the build completed, I ran:

bash
nix store diff-closures /run/current-system ./result-26.05

A Git diff tells me what I wrote in the configuration. A closure diff tells me what will actually run. It quickly exposed:

major-version changes in the kernel, systemd, and Docker;
whether PostgreSQL had moved unexpectedly;
replacements of runtimes such as Node.js;
the actual changes to D-Bus, the desktop, and firmware.

In a declarative system, the final closure—not the source diff—is the deployment artifact.

Step 5: Back Up the Data Before Installing the Boot Entry

An old generation can roll back system configuration and software. It cannot undo database writes or an application data migration. Before deployment, I therefore backed up:

PostgreSQL with pg_dumpall;
Redis after checking persistence and saving an RDB snapshot;
bind-mounted state for file-based applications such as Open WebUI, with the relevant containers stopped;
the current configuration, lock file, and important state inventories.

There is an important boundary here:

generation rollback handles a system that does not boot or has broken configuration;
data backup handles a new service that has already modified persistent state.

Preparing only one of them is not a complete recovery plan.

Only after the backups were complete did I install the prebuilt result into systemd-boot. I also verified that the boot menu still contained a NixOS 25.11 generation.

Step 6: Validate the Rebooted System in Layers

Reaching the desktop only proves that the machine booted. It does not prove that the upgrade is finished. I divided acceptance testing into several layers.

System identity

bash
nixos-version
uname -r
readlink -f /run/current-system
systemctl is-system-running
systemctl --failed

The first task is to confirm that the running system is the intended store path, rather than an old generation selected by mistake.

Stateful services

bash
pg_isready -U postgres
redis-cli ping
curl http://127.0.0.1:11434/api/version

An active process is not enough. I checked versions, existing data, persistence status, and real API responses.

Containers

For each container, I checked:

health status;
whether .Config.Image still contained the pinned digest;
restart count;
the external HTTP endpoint;
application logs for migration failures.

Desktop and network

This layer covered the GNOME session, Home Manager user services, Wi-Fi, DNS, Bluetooth, and external connectivity. A desktop system cannot be accepted by testing system services alone: user units and desktop portals can also break across releases.

Two Problems That Appeared Only After Reboot

PostgreSQL collation warnings after the glibc update

The new system uses glibc 2.42, while the existing PostgreSQL cluster recorded collation version 2.40. Connections produced a collation version mismatch warning.

I did not react to the warning by immediately changing metadata. First, I performed read-only checks of the recorded and actual versions for each database and confirmed that there were no user tables or indexes. With a pre-upgrade backup already available, I refreshed the metadata:

sql
ALTER DATABASE postgres REFRESH COLLATION VERSION;
ALTER DATABASE template1 REFRESH COLLATION VERSION;

If a database contains indexes that depend on locale-aware ordering, merely refreshing the version number is not enough. The affected objects must first be assessed and rebuilt. REFRESH COLLATION VERSION does not rebuild indexes for you.

`ExecStart` is not a shell command line

The system initially looked healthy after reboot. At midnight, however, a timer activated drop-caches.service; the service failed and changed the overall system state to degraded. Its existing configuration was:

nix
serviceConfig.ExecStart =
  "${pkgs.coreutils}/bin/sync; echo 3 > /proc/sys/vm/drop_caches";

systemd does not pass ExecStart= through a shell by default. The ; and > characters do not mean command sequencing and redirection. systemd treated sync; as part of the executable name and returned 203/EXEC.

If shell behavior is genuinely required, use a NixOS service script:

nix
systemd.services.drop-caches = {
  serviceConfig.Type = "oneshot";
  script = ''
    ${pkgs.coreutils}/bin/sync
    echo 3 > /proc/sys/vm/drop_caches
  '';
};

The better solution is probably to remove this timer entirely. Linux already manages the page cache, and regularly forcing it to be dropped usually does little beyond reducing cache hit rates.

NixOS 26.05 did not introduce this mistake. The broken configuration had existed for some time; the real timer execution simply fell outside my earlier observation windows. The lesson is that upgrade acceptance must cover delayed timers, not just the system state immediately after boot.

At the time of writing, this service remains a known follow-up item. I did not turn “the core services work” into the less honest claim that “the system has no problems.”

Do Not Let Every Red Log Line Drive the Upgrade

The new boot also logged duplicate D-Bus service names from dbus-broker, an Fcitx/PAM warning, and several BIOS, ACPI, and PMC messages.

They were worth recording, but the word error alone was not enough to classify them as blocking. I used five questions instead:

Does the message correspond to a failed unit?
Does an independent functional check fail?
Does the error repeat or trigger retries?
Was it introduced by the upgrade, or did it already exist?
Would fixing it expand the scope of the current change?

For example, dbus-broker printed many duplicate-name warnings, but GNOME, portals, and the keyring all worked. I recorded the messages as non-blocking noise instead of restructuring the entire desktop package set during the upgrade window.

Being rigorous does not mean changing configuration whenever a log line turns red. It means supporting every conclusion with state and functional evidence.

A Reusable Upgrade Checklist

Before the upgrade

Create a dedicated upgrade branch.
Record the system, kernel, Nix, systemd, and database versions.
Check /, /boot, failed units, and current generations.
Inventory every stateful service and its data directory.
Record the exact digest of every running container image.
Keep system.stateVersion and home.stateVersion unchanged.
Confirm that garbage collection will not remove the rollback generation too soon.

Configuration and build

Match the nixpkgs and Home Manager release branches.
Handle removed, renamed, and semantically changed options.
Pin database server and CLI tools to the intended major version.
Freeze container images so applications do not upgrade at the same time.
Build the complete NixOS system closure.
Build the standalone Home Manager activation package.
Run the formatter, git diff --check, and Flake checks.
Review the effective version changes with nix store diff-closures.

Deployment and recovery

Back up PostgreSQL, Redis, and file-based application state.
Verify that backups are readable and record which services must stop before restoration.
Deploy with boot to avoid a mixed live transition of foundational services.
Confirm that an old generation remains in the boot menu.
Define the conditions for system rollback and data restoration.

After reboot

Confirm that /run/current-system points to the target closure.
Check system and user units.
Verify database connectivity, versions, data, and persistence.
Verify container digests, health, restart counts, and HTTP endpoints.
Test the desktop, Wi-Fi, DNS, Bluetooth, and external connectivity.
Check the glibc/PostgreSQL collation version.
Check failed units again after timers have had time to run.
Keep backups and the old generation until the observation period ends.

Closing Thoughts

The NixOS 25.11-to-26.05 upgrade ultimately succeeded. The new kernel, systemd, dbus-broker, GNOME, Docker, PostgreSQL, Redis, Ollama, and Home Manager all work, while the old generation and pre-upgrade backups remain available.

What gives me confidence is not merely that this particular upgrade did not fail. It is that the process did not depend on luck:

the build happened while the old system was still usable;
application upgrades were moved out of this change window;
both system rollback and data recovery existed before deployment;
every post-reboot conclusion was supported by a command, a state check, or a real request;
the one failed unit remained visible instead of being ignored for the sake of a clean conclusion.

The strength of NixOS is not just that configuration can be written as code. More importantly, it lets us decompose a risky upgrade into a sequence of reviewable, buildable, verifiable, and reversible state transitions. The tooling makes that possible; whether the upgrade is truly safe still depends on how we design the process.