It turns out that I had to do a couple of tweaks to the RPi configuration. The first is to edit /etc/dhcp/dhclient not to request a NTP server in the DHCP request because then ntpd ignores /etc/ntp.conf. In fact later I should switch to a static IP address so that the RPi not reliant on the DHCP server on the LAN being up.

I also discovered that I had to set the NTP config in DietPi to custom, otherwise every morning it would kill the ntp daemon

It also helped me discover that my LAN's NTP server wasn't configured to accept NTP requests due to a change in the shipped default configuration. I had to edit out the noquery parameter on the IPv4 and IPv6 restrict lines.