Debian Linux Server Setup

bios settings

Get good security from the beginning, upgrade the bios, get defaults, and change a few basic options:

Boot options:
First: Hard Drive
Second: Disabled
Third: Disabled

Disable Parallel port, Serial ports, and any unused RAID chipset

Enable auto startup after power failure

System password to enter the bios, but not to boot the system:
(Enter a good and easy to remeber password)

debian install

Boot with kernel bf24, and follow the guidance:

-Network install
-base-config
 -No GMT, Europe/Oslo
 -MD5 passwords
 -Add dk or se debian mirror
 -Do not run tasksel or dselect 
 -Remove pcmcia package

filesystem settings

I like ext3 cause it's widely supported and fairly fail proof when using the journal.

Create the necessary filesystems, if boot disk; /boot = 100MB, /var = 4000MB, swap = 512 MB minimum, / = 500 MB, and /usr = rest of the drive:

#p - view file system
#w - write partition table
#c - create file system
fdisk /dev/hdx

Create the actual file systems:

# -j - journal
# -m 1 - percentage used to super-user
# -c - check for badblocks
mke2fs -c -j -m 1 /dev/hdx1

network settings

IP configuration in /etc/network/interfaces:

auto eth0
iface eth0 inet static
        address xxx.xxx.xxx.xxx
        netmask 255.255.255.xxx
        network xxx.xxx.xxx.0
        broadcast xxx.xxx.xxx.255
        gateway xxx.xxxx.xxx.1

Domain name system settings in /etc/resolv.conf:

search x-pec.com
nameserver 193.75.75.75
nameserver 193.212.1.10

portmapper control access for ssh, ftp, pop3, snmp:
/etc/hosts.deny

ALL: ALL@ALL, PARANOID

/etc/hosts.allow

ALL:127.0.0.1
ALL:xxx.xxx.xxx.xxx, xxx.xxx.xx.
in.ftpd:ALL
ipop3d:ALL

Enter only hosts and nets that you are connecting from, like work and home. Select one system that you will relay all the other connections to the inner servers, on those systems only allow the parent system nothing else.

startup settings

Make sure that /etc/motd don't get overwritten on evert startup and selects yes on disk scan prompts, edit /etc/default/rcS:

# Set EDITMOTD to "no" if you don't want /etc/motd to be editted automatically
EDITMOTD=no
# Set FSCKFIX to "yes" if you want to add "-y" to the fsck at startup.
FSCKFIX=yes
Or add a new file in /etc/update-motd.d/20-ivc (chmod +x /etc/update-motd.d/20-ivc) with the current /etc/motd data, even add colors and fetch other data

Limit regular teminal logins and ctrl+alt+del binds in /etc/inittab:

# What to do when CTRL-ALT-DEL is pressed.
##ca:12345:ctrlaltdel:/sbin/shutdown -t1 -a -r now

# /sbin/getty invocations for the runlevels.
#
1:2345:respawn:/sbin/getty 38400 tty1
#2:23:respawn:/sbin/getty 38400 tty2
#3:23:respawn:/sbin/getty 38400 tty3
#4:23:respawn:/sbin/getty 38400 tty4
#5:23:respawn:/sbin/getty 38400 tty5
#6:23:respawn:/sbin/getty 38400 tty6

Limit where and how the root user can login directly in /etc/securetty:

# /etc/securetty: list of terminals on which root is allowed to login.
# See securetty(5) and login(1).
console

# Standard consoles
tty1
#tty2
#tty3
...
#tty12

# Same as above, but these only occur with devfs devices
vc/1
#vc/2
#vc/3
...
#vc/12

Remove the silly inetd daemons in /etc/inetd.conf:

#discard               stream  tcp     nowait  root    internal
#discard               dgram   udp     wait    root    internal
#daytime               stream  tcp     nowait  root    internal
#time          stream  tcp     nowait  root    internal

deb packages

Default install is around 180 MB, add the recommended packages after the networks is online.

apt-get install <package>

packages:
ssh
hdparm
mtr
iptraf
lynx
wget
ncftp
bzip2
unzip
zip
make
gcc (gcc-2.95)
build-essential (and all dependencies)
ncurses-dev
kernel-package
modules-init-tools (not just upgrade, needs to be installed too)
file-rc (obsolete, sysv-rc takes this role)
snmpd
lm-sensors
postfix
(obsolete) ide-smart
smartmontools
dnsutils
ntpdate
lsof
netcat, nmap, tcpdump (optional)
rsync
uptimed
sysstat (sar system details)
vnstat

and add the necessary backports from backuports.org

Remove the unneseccary packages if they're not already out:

apt-get remove --purge <package>:

ppp
aptitude and tasksel

lilo

For easy kernel upgrade with fallback to the prior kernel and boot security, edit /etc/lilo.conf:

# You can set a password here, and uncomment the `restricted' lines
# in the image definitions below to make it so that a password must
# be typed to boot anything but a default configuration. 

password=XXX

# Specifies the number of deciseconds (0.1 seconds) LILO should
# wait before booting the first image.
#
delay=20

# Boot up Linux by default.
#
#default=Linux

image=/vmlinuz
        label=Linux
        read-only
        restricted

image=/vmlinuz.old
        label=LinuxOLD
        read-only
        restricted

And finally:

chmod 600 /etc/lilo.conf
lilo -v

grub2

Grub superceds lilo and should be used as the default boot mananger.

sshd

Restricting sshd and using host.allow filtering is creating a pretty good control over who can connect to the system, /etc/ssh/sshd_config:

PermitRootLogin no

su restriction

Restrict who can acutally su to another user (root) in /etc/pam.d/su:

# Uncomment this to force users to be a member of group root
# before than can use `su'. You can also add "group=foo" to
# to the end of this line if you want to use a group other
# than the default "root".
# (Replaces the `SU_WHEEL_ONLY' option from login.defs)
auth       required   pam_wheel.so

Add each user that is allowed into the root group in /etc/group:

root:x:0:ivc
...

file-rc

Or keep using sysv-rc by configuring it using sysv-rc-conf.

Get rid of that old and unconvinient rc startup system, get file-rc instead.

apt-get install file-rc

To create your own startup script add this line to /etc/runlevel.conf, runs after all the other daemons:

89      -       2,3,4,5         /etc/init.d/rc.local

Restrict access to the file:

chmod 700 /etc/init.d/rc.local

performance

The first easy performance tweak is to enable DMA and 32-bit I/O support on the hard drives. Make sure you have the IDE chipset(s) enabled in the kernel, else you will get error messages.

Put these commands in /etc/init.d/rc.local:

# Hard Drive - DMA and 32-bit
hdparm -d 1 -c 1 /dev/hda

move home directory

Why do we need a seperate /home partition, move it to /usr like the bsd unices:

mv /home /usr/
ln -s /usr/home /home

Do the same for /tmp:

mv /tmp /usr/
ln -s /usr/tmp /tmp

Edit passwd to reflect the change, don't rely on the symlink:

pico /etc/passwd

Update /etc/adduser.conf:

# The DHOME variable specifies the directory containing users' home
# directories.
DHOME=/usr/home

Make the home directories more secure, restrict to entering but not listing:

# If DIR_MODE is set, directories will be created with the specified
# mode. Otherwise the default mode 0755 will be used.
DIR_MODE=0711

And for the current user directories change them:

chmod 711 <user dir>

shell settings

Add usefull shell aliases, /etc/bash.bashrc and /etc/profile:

alias ll="ls -lh --color"
alias pico="nano -w"
alias rm="rm -i"

To keep the file system permission tight change the umask:

umask 022

For root keep it serious tight and auto logout, /root/.bashrc:

umask 077
TMOUT=3600

logrotate

Keep the size down on log files by using compression in /etc/logrotate.conf, but look out for the rotate setting if you want to keep backlogs (hint: logrotate.d/):

# keep 4 weeks worth of backlogs
rotate 999

# uncomment this if you want your log files compressed
compress

postfix

Keep a small and strict postfix local mail server config, /etc/postfix/main.cf:

# see /usr/share/postfix/main.cf.dist for a commented, fuller
# version of this file.

# Do not change these directory settings - they are critical to Postfix
# operation.
command_directory = /usr/sbin
daemon_directory = /usr/lib/postfix
program_directory = /usr/lib/postfix

smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
setgid_group = postdrop
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no
myhostname = x-pec.com
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = x-pec.com, localhost.localdomain, localhost
relayhost =
mynetworks = 127.0.0.0/8
mailbox_command =
mailbox_size_limit = 0
recipient_delimiter = +

Add yourself to the end of /etc/aliases to get all system mails generated on the system:

root: ivc
...
ivc: xxx@yourmail.com

Reload the db file:

postalias /etc/aliases

ntpdate

Keep the date and time correct using ntpdate and a stable date server.

Add this line to /etc/default/ntp-servers:

NTPSERVERS="time.windows.com"

To keep it check and update the time daily, create this script in /etc/cron.daily/ntpdate:

#!/bin/sh

/usr/sbin/ntpdate 0.debian.pool.ntp.org 1.debian.pool.ntp.org 2.debian.pool.ntp.org 3.debian.pool.ntp.org

Make it executable:

chmod 700 /etc/cron.daily/ntpdate

snmpd

Create cool mrtg graphs with data gathered through snmp. Control mechanism and some custom values from lm-sensors in /etc/snmp/snmpd.conf:

##       sec.name  source          community
com2sec local     localhost           XXX
com2sec network 212.125.228.0/27      XXX

##     group.name sec.model  sec.name
group MyRWGroup  v1        local
group MyRWGroup  v2c        local
group MyRWGroup  usm        local
group MyROGroup  v1        network
group MyROGroup  v2c        network
group MyROGroup  usm        network

##           incl/excl subtree                          mask
view all    included  .1                               80

##                context sec.model sec.level prefix read   write  notif
access MyROGroup ""      any       noauth    exact      all    none   none
access MyRWGroup ""      any       noauth    exact      all    all    all

syslocation Planet X-Pec
syscontact Admin <admin@planetx-pec.com>

exec cputemp1 /usr/home/ivc/bin/snmp-temp.sh /sys/bus/i2c/devices/0-0290/temp2_input
exec systemp1 /usr/home/ivc/bin/snmp-temp.sh /sys/bus/i2c/devices/0-0290/temp1_input

Content of snmp-temp.sh:

#!/bin/sh

cat $1 | sed 's/-//' | head -c 2
exit 0

sysctl

Strengthen the network security by disabling remote routing exploits and log miscellaneous bits in /etc/sysctl.conf:

# Uncomment the next two lines to enable Spoof protection (reverse-path filter)
# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1

# Uncomment the next line to enable TCP/IP SYN cookies
# See http://lwn.net/Articles/277146/
# Note: This may impact IPv6 TCP sessions too
net.ipv4.tcp_syncookies=1

# Do not accept ICMP redirects (prevent MITM attacks)
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0

# Do not send ICMP redirects (we are not a router)
net.ipv4.conf.all.send_redirects = 0

# Do not accept IP source route packets (we are not a router)
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0

# Log Martian Packets
net.ipv4.conf.all.log_martians = 1

# Enable ignoring broadcasts request
net.ipv4.icmp_echo_ignore_broadcasts = 0

# Enable bad error message Protection
net.ipv4.icmp_ignore_bogus_error_responses = 1

upclient

Superseded by uptimed. Keep track of the uptime the machine generates. Download the linux upclient at http://uptimes.hostingwired.com/

Unpack and compile by changing dir to src/

cd src/
make linux
mv products/ ../../upclient/

Go to http://uptimes.hostingwired.com/ and creat/login to your account, add system, enter the info you want to have publicly availble, click add.
Edit the upclient.conf file and paste the md5sum into where is says Authkey, also remove the path on the PidFile line and change SendOSVersion to 0:

PidFile = upclient.pid # Location of pid file
AuthKey = xxxxxxxxxxxxxxxxxxxxxxxxx # Your Authorization key
SendOSVersion = 0 # Send version of your OS

Make sure it stays in the background on boot-up by adding a line in /etc/init.d/rc.local:

#uptime client
cd /usr/home/ivc/bin/upclient; ./upclient &

(obsolete) ide-smart

Install ide-smart to watch the drives for prefailures, check that no line has "Failed" at the end.

Add a cron entry pointing to this shell script:

0 6 * * * root /usr/home/ivc/bin/ide-smart.sh /dev/hda &

The content of ide-smart.sh is:

#!/bin/bash
host=`hostname`
smart=`/usr/sbin/ide-smart $1`
/usr/sbin/ide-smart $1 -q
if [ $? != 0 ] ; then
 echo "
 ide-smart detected a failed SMART test!
 Please check the $1 drive.

 Result:
 $smart

 -$host

 "
fi

smartmontools

Smartmontools can execute routinely short or long smar tests and send email alerts when a disk problem has accoured. Use Spinrite regularly to refresh the surface and magnetic properties.

pico /etc/smartd.conf
/dev/hda -S on -o on -a -I 194 -s (S/../.././02|L/../../6/03) -m ivcatx-pec.com
/dev/hdb -S on -o on -a -I 194 -s (S/../.././02|L/../../6/03) -m ivcatx-pec.com

lm-sensors

Get the temperature of the cpu and system chipset, speed of various fans, and voltage status from monitor chips.

Kernel 2.6 options:
Device Drivers -< i2c support
<M> I2C support
<M>   I2C device interface                                    

I2C Algorithms  --->
<M> I2C bit-banging interfaces 
<M> I2C PCF 8584 interfaces

I2C Hardware Bus support  --->
(Every option as modules)

Hardware Sensors Chip support  --->
(Every option as modules)

Other I2C Chip support  --->
<M> EEPROM reader

Compile kernel, install, and reboot:

make-kpkg kernel-image; dpkg -i ../kernel-image-2.6.x.deb; shutdown -r now

lm-sensors require you to have a /sys filesystem, by default it does not exists, create it:

mkdir /sys

pico /etc/fstab:
sysfs           /sys            sysfs   defaults                0       0

mount /sys

Detect the available sensors on the motherboard. Follow the easy instructions:

sensors-detect
...
(copy the last lines into clipboard)

pico /etc/modules
/etc/init.d/modutil

pico /etc/modutil/local
update-modules

sensors

Remove the unnecessary modules in the kernel and recompile. Either reboot right away or wait till next schedueled reboot.

kernel upgrade

When a new kernel comes out once in a while there are usually new config options available. Filter those options out before you recompile:

make oldconfig

fsck disk maintenance

Doing proper disk maintenance is critical. To handle bad blocks first dump them then load fsck.

fsck.ext3 -f -v -c /dev/hdb1

dumpe2fs -b /dev/hdb1 > ./badblocks_hdb.txt
fsck.ext3 -f -v -L ./badblocks_hdb.txt /dev/hdb1

foreign characters

Proper console listing of Norwegian characters:
apt-get install locales
Enable option no_NO in configuration.
Put export LANG=no_NO and export LC_MESSAGES POSIX in /etc/profile

Re-login...

And in smb.conf put unix charset = iso-8859-1

bash default

Default /bin/sh is a symlink to /bin/dash, however /bin/bash is necessary, not /bin/dash. Therefore do this:

dpkg-reconfigure dash

rsync backup/mirror

To duplicate an old drive to a new one, you can use rsync. This is normally done on a seperate machine booting Live CD or Debian Netinstall disc.

mount /dev/hda5 /old/
mount /dev/hda1 /old/boot/
mount /dev/hda2 /old/var/
mount /dev/hda6 /old/usr/

mount /dev/hdb5 /new/
mount /dev/hdb1 /new/boot/
mount /dev/hdb2 /new/var/
mount /dev/hdb6 /new/usr

rsync -avl /old/ /new/
To install the bootloader, preferably GRUB, find the new root drive (/) UUID by issuing:
chroot /newdrive/
blkid /dev/hda5
 UUID=2fac4....
Insert the new id into /boot/grub/menu.lst, replace all the old UUID entries with the new.

To install the bootsector and updating grub, issue:
grub-install /dev/hda
update-grub
If the system is not booting, press 'e' on the bootloader screen to edit the grub boot command-line, and change root to root=/dev/hda5 to see if it will boot.
Alternatively change /etc/fstab from md[0..6] to hda[0..6] signgle drive system to set-up the raid array later on.
Also make sure to update the initramfs-image and install udev if it's not installed:
update-initramfs -u
apt-get install udev
If the boot fails, either wait for the Buzybox command-line or boot the Debian Netinstall disc and select Rescue. Then grub and lilo can be used to fix the bootloader.

Password less login

Login directly without being prompted to enter a password.

Openwrt, dropbear:
dropbearkey -t dss -f db_key -s 1024
dropbearkey -y -f db_key > db_key.pub 

Copy the db_key.pub data to .ssh/authorized_keys2 on the server, append user@host and maybe prepend a command= parmeter to auto excecute a script or binary.

Use the alternate ssh client to login:
dbclient -i db_key user@host

Search and replace multiple files

Using perl to replace a string in many files:

perl -pi -w -e 's/search/replace/g;' *.php

Using find and sed:

find . -name "*.c" -exec sed -i "s/oldWord/newWord/g" '{}' \;

truecrypt

There is no truecrypt debian package, so a new one has to be compiled from the source. Fortunately, there is a script to lighten this prodecure:

  1. Download the latest debian script package: http://www.unchartedbackwaters.co.uk/pyblosxom/static/truecrypt_debian_packaging
  2. Download truecrypt source from http://www.truecrypt.org/
  3. Read the README.Debian to move and run the package tools to cover all the dependencies

maintenance

Keep the system updated:

apt-get update
apt-get upgrade
Find SUID/SGID programs:
find / -type f \( -perm -04000 -o -perm -02000 \)
Find world-writable files:
find / -perm -2 ! -type l -ls
Find unowned files:
find / \( -nouser -o -nogroup \) -print

Creating software RAID

To create a simple RAID1 array from two drives, setup one drive and then add the second.

mdadm --create /dev/md0 --metadata=0.90 --raid-devices=2 -level=1 /dev/sdb1 missing
mdadm --create /dev/md1 --metadata=0.90 --raid-devices=2 -level=1 /dev/sdb2 missing
mdadm --create /dev/md2 --metadata=0.90 --raid-devices=2 -level=1 /dev/sdb5 missing
mdadm --create /dev/md3 --metadata=0.90 --raid-devices=2 -level=1 /dev/sdb6 missing
mkfs.ext3 -j -m 1 /dev/md0
mkfs.ext3 -j -m 1 /dev/md1
mkfs.ext3 -j -m 1 /dev/md2
mkfs.ext3 -j -m 1 /dev/md3
mkswap /dev/sdb3

Copy the system over to the new array by rebooting from a NetInstall, use Automatic array detection to assembly the drives. Use rsync to duplicate the drive, reboot and add the last drive to the array. Shown below.

Adding/replacing md drive

To rebuild/add/replace drive. Mark the failed drive, remove it from the array, change the physical drive, copy paritio and add it to the raid.

cat /proc/mdstat # check for failed drive
mdadm /dev/md0 -f /dev/hdb1 # set as failed
mdadm /dev/md0 -r /dev/hdb1 # remove
blkid # list and find swap partition
remove swap line in pico /etc/fstab
shutdown -h now # replace the /dev/hdb drive with an identical
sfdisk -d /dev/hda | sfdisk /dev/hdb # duplicate/create partitions on new drive
fdisk -l # check partitions
mdadm /dev/md0 -a /dev/hdb1 # add
cat /proc/mdstat # check the health and start of rebuild
dpkg-reconfigure grub-pc # install bootloader on all raid drives, hda and hdb
mkswap /dev/sdb3 # create swap device
add line in pico /etc/fstab with provided new UUID
swapon /dev/sdb3 # turn on
swapon --show # check active swap partitions, should be two

Details http://really.slacking.net/~modat7/howto/RAID-mdadm-Setup.txt and http://www.howtoforge.com/replacing_hard_disks_in_a_raid1_array

pcspkr boot warning

To remove the pcspkr module already loaded message add this line to /etc/modprobe.d/blacklist:

blacklist snd-pcsp

Replicate hard drive

To upgrade a system drive to a larger or new drive, a few essential steps has to be performed in the proper squence.

  1. Do a proper test and reliability test of the new drives, using for instance SpinRite
  2. Partition the new drives and format using ext3 filesystem
  3. To limit downtime, rsync data over the network from the working system to get the majority of the replication done live
  4. Disassemble software RAID to single disks, in hda1 hdb1 assembly, remove hdb1
  5. Shutdown and disconnect one of the drive (be aware that all changed data on the removed drive will not be updated)
  6. Set-up another machine with the current system drive (primary master) and the new drive (secondary master)
  7. Boot from USB or DVD Debian Netboot to partition the new drive, fdisk /dev/sdb and change the partition id to fd for Linux SoftRAID
  8. Mount all the partitions on both drives;
  9. Copy over all the files, rsync -avHP --delete /mnt/old/ /mnt/new/
  10. Bind the system devices and blocks:
  11. Change root to the new OS drive, chroot /mnt/new
  12. Install bootloader, grub-install --recheck /dev/sdb (new drive, rechekc updates devicemap), then the kernel update-grub
  13. Fix fstab partitions by editing /etc/fstab and add the real /dev/md[0..3]] devices instead of block ids
  14. Unmount and shutdown the machine, disconnect the drives
  15. Install to the new hard drives in the server and boot up in recovery (single mode) to test the new drive setup
  16. Reboot, add the missing devices to the raid array

References

Encrypting external drive

When using a USB drive for backup, it is a good idea to encrypt the drive so the content stays safe if the drive goes missing.

apt-get install cryptsetup
cryptsetup -y -v luksFormat --type luks1 /dev/sdX
cryptsetup luksOpen /dev/sdX backupdrive
cryptsetup -v status backupdrive
cryptsetup luksDump /dev/sdX
mkfs.ext4 -j -m 1 /dev/mapper/backupdrive

mkdir /mnt/backup
mount /dev/mapper/backupdrive /mnt/backup

umount /mnt/backup
fsck -vy /dev/mapper/backupdrive
cryptsetup luksClose backupdrive

References

Links and references

Securing and Optimizing RH 7.3
Hvordan få norske tegn i skallet
Perfect Setup
LAMP on Sarge
Hdd Resolution & recovery Free Secondary NS providers