Safely change firewall rules remotely

One of the NTP Pool server operators emailed recently and told that his server was unavailable for a few hours because he had changed some firewall rules remotely and managed to lock everyone out. Oops!

There's an easy way to do safe remote firewall changes.

He should have used the at daemon to automatically recover! If you don't have a console server then you absolutely must get in the habit of using it before doing anything that can lock you out of your server.

The basic concept is to setup something that in say 5 minutes will undo whatever you're about to do. So if you are messing with the firewall rules, the easy way is to tell it to disable the firewall in five minutes. For example on Fedora and RHEL you can use service iptables stop to stop the firewall, so as root enter:

at "now + 5 minutes"
service iptables stop

You can type a whole list of commands and when you're done, press ctrl-d to stop.

It will look something like:

# at "now + 5 minutes"
service iptables stop
job 6 at Tue Sep 18 17:53:03 2007
#

Another variation to "program" the at daemon would be echo service iptables stop | at "now + 5 minutes".

Anyway, now you go mess with the firewall for 5 minutes and when you're done you delete the at job. You use the atq command to see a list of queued jobs and then atrm to remove it

# atq
6       Tue Sep 18 17:53:00 2007
# atrm 6

Editing firewall rules is a great use case for this, but other neat ways to use at could be:

You mess with the network configuration. Make the system just reboot in case you get it wrong (be sure that the working configuration files are put back; or that you don't edit the configuration files until you've gotten it working with ifconfig).

You take something down for maintenance; use at to schedule it being started again in an hour. So if you forget to restart it, it'll come back in an hour at the latest.

For example, start MySQL replication at 8pm:

$ at 20:00
mysql -uroot -e 'start slave;'
^D
job 8 at Tue Sep 18 20:00:00 2007

Send yourself a reminder:

$ at 18:32
mail -s "put money in parking meter!" ask@example.com < /dev/null
^D
job 8 at Tue Sep 18 18:32:00 2007

You have to make sure atd is running of course, but on most Unix systems it should be (it's installed on OS X, but isn't running by default).