Help Centre
Home Server & Cloud Infrastructure Dedicated Server

Working with the Linux Rescue System

For Dedicated Servers, Server Power Deals and Value Dedicated Servers managed in the server administration of the IONOS account

In this article you will learn how to use the rescue system of your Dedicated Server. The Rescue System is an important tool for diagnosing and repairing issues with your server’s system.

Note

If you  have purchased a dedicated server managed through the cloud panel, the following articles provide information about the Rescue system of your server:

Booting the Dedicated Server to the Linux Rescue System

Booting the Dedicated Server to the Windows Rescue System

Working with the Linux Rescue System (Hardware RAID)

Using Windows Rescue System on a Dedicated Server Windows

Important Information on Available Rescue Systems

The following rescue systems are available for  Dedicated Servers purchased under the Server Power Deals : 

  • Linux 64-bit mini rescue system(debian/stable)

  • Linux 64-bit rescue system(debian/oldstable)

  • Linux 64-bit rescue system(debian/stable)


Rescue System

With the Rescue System you can, among other things:

  • backup the data of your server

  • repair a damaged file system

  • call up log files to identify sources of error

  • assign a new root password

  • repair the network configuration of your server


Mini Rescue System

The Mini Rescue System does not have all the programs that the Rescue System includes. Therefore, use the Mini-Rescue System only if your server has little RAM and cannot boot.

Note

The StorCLI, PERCCLI, and ARCCONF CLI utilities cannot be accessed via the Mini-Rescue System. 

Booting the Server into the Rescue System

How to  boot your server into the rescue system is explained here: 

Booting Linux Server Into the Rescue System

Identify Hardware Controller

To check which hardware controller is installed on your server, you can  use the program lshw.  This program shows detailed information about the hardware components. 

Display Hardware Information with lshw 

To view a summary of hardware information, type the following command:

lshw -short

To write the hardware information to a text file, type the following command:

lshw > lshw_output.txt

RAID Controller Management Programs

The software required to manage each RAID controller is installed by default. If necessary, you can download the software for the respective controller here:


Adaptec RAID Controller

https://storage.microsemi.com/en-us/downloads


Broadcom RAID Controller

https://www.broadcom.com/support/download-search


Dell RAID Controller

https://www.dell.com/support/home/de-de?app=products


3ware RAID Controller

https://www.broadcom.com/support/knowledgebase/1211161499498/operating-systems-supported-for-the-lsi-3ware-and-megaraid-contr


Areca RAID Controller

https://www.areca.com.tw/support/downloads.html

Check the Status of the Hardware RAID

Use the commands listed below to access hardware RAID and configuration information:

Adaptec RAID Controller
COMMAND EXPLANATION
arcconf GETCONFIG 1 This command displays information about the RAID controller, configuration, RAID arrays, available disks and their current state.
arcconf getconfig 1 PD Use this command to display information about the existing hard disks.
arcconf getlogscontroller-number device ¦ dead ¦ event [clear ¦ tabular] Use this command to display the log entries. You can use the following parameters to specify the type of log entries displayed:
Example: arcconf getlogs 1 device tabular DEVICE: Shows all log entries with errors the controller could detect on all connected devices.
  DEAD: Shows all log entries where device and hard disk failures were recorded.
  EVENT: Shows all log entries with information about special events (e.g. rebuilds, LDMs, etc.)
  You can also specify the following parameters:
  tabular: Displays the log in table format.
  clear: Deletes the specified controller protocol.
arcconf getlogs controller-number device ¦ dead ¦ event [clear ¦ tabular] > FILENAME.log This command writes the log entries to a file.
Example: arcconf getlogs 1 device tabular > device.log  

You can find more information here:

User's Guide ARCCONF Command Line Utility

Broadcom RAID Controller
COMMAND EXPLANATION
storcli64 /c0 show all This command displays information about the RAID controller and the configuration.
storcli64 /c0/eall/sall show Use this command to display information about the existing hard disks.
storcli64 /c0/eALL/sALL show all This command displays information about the available disks and their current status (serial number, temperature, etc.).
storcli /cx show eventloginfo Use this command to display the log entries.

You can find more information here:

12Gb/s MegaRAID Tri-ModeStorCLI User Guide

DELL-PERC-RAID Controller
COMMAND EXPLANATION
perccli64 /c0/e32/s0 show all Use this command to display information about disk 0.
perccli64 /c0/e32/s1 show all Use this command to display information about disk 1.
perccli64 /c0 show Use this command to display information about the existing hard disks.
perccli /c0 show This command displays the log entries for controller 0.

You can find more information here:

Dell EMC PowerEdge RAID Controller CLI Reference Guide

3ware RAID Controller

For instructions on how to use tw_cli to access hardware RAID and configuration information, see the following article:

Monitor / rebuild hardware RAID (Linux)

For more information on installation, configuration and use, please refer to the 3ware documentation. You can find it here:

https://www.broadcom.com/support/knowledgebase/1211161499541/lsi-3ware-user-documentation-downloads 

Areca RAID Controller

For instructions on how to use the Command Line Interface to access hardware RAID and configuration information, see the following article

Monitor / rebuild hardware RAID (Linux)

More information about the command line interface can be found here:

https://areca.starline.de/RaidCards/Documents/Manual_Spec/Software

Check the Status of the Software RAID

To check the status of the software RAID, enter the following command in the shell:

rescue:~#cat /proc/mdstat

Listing Hard Drive Partitions

In the Rescue System you can partition the internal hard disk, check the file system and faulty incorrect settings.

To list the file systems and identify the file system in question, enter the command df -h. The abbreviation df stands for disk free. With the parameter -h you can display the number of occupied blocks in machine readable form in giga, mega or kilobytes.

rescue on 87.106.16.168:~$ df -h
Filesystem Size Used Availability Use% Mounted on
/dev/root 372M 356M 17M 96% /
devtmpfs 32G 0 32G 0% /dev
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 32G 102M 32G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 32G 0 32G 0% /sys/fs/cgroup

To list the existing hard disks, partitions, and logical volumes, enter the command fdisk -l. Fdisk is a command line program for the partitioning of hard disks. With this program you can list, create and remove partitions. 

rescue:~# fdisk –l

The following example lists information from a server with a software raid. The information was gathered by the rescue system. 

Note

The output of fdisk -l may differ according to server model and/or configuration. 

rescue:~#  fdisk -l

Disk /dev/sda: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 1 123 987966 fd Linux raid autodetect
/dev/sda2 124 367 1959930 82 Linux swap / Solaris
/dev/sda4 368 30401 241248105 5 Extended
/dev/sda5 368 976 4891761 fd Linux raid autodetect
/dev/sda6 977 3409 19543041 fd Linux raid autodetect
/dev/sda7 3410 4018 4891761 fd Linux raid autodetect
/dev/sda8 4019 30401 211921416 fd Linux raid autodetect

Disk /dev/sdb: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 123 987966 fd Linux raid autodetect
/dev/sdb2 124 367 1959930 82 Linux swap / Solaris
/dev/sdb4 368 30401 241248105 5 Extended
/dev/sdb5 368 976 4891761 fd Linux raid autodetect
/dev/sdb6 977 3409 19543041 fd Linux raid autodetect
/dev/sdb7 3410 4018 4891761 fd Linux raid autodetect
/dev/sdb8 4019 30401 211921416 fd Linux raid autodetect

Disk /dev/md8: 217.0 GB, 217007456256 bytes
2 heads, 4 sectors/track, 52980336 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md8 doesn't contain a valid partition table

Disk /dev/md7: 5009 MB, 5009047552 bytes
2 heads, 4 sectors/track, 1222912 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md7 doesn't contain a valid partition table

Disk /dev/md6: 20.0 GB, 20012007424 bytes
2 heads, 4 sectors/track, 4885744 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md6 doesn't contain a valid partition table

Disk /dev/md5: 5009 MB, 5009047552 bytes
2 heads, 4 sectors/track, 1222912 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md5 doesn't contain a valid partition table

Disk /dev/md1: 1011 MB, 1011548160 bytes
2 heads, 4 sectors/track, 246960 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md1 doesn't contain a valid partition table

If running fdisk does not result in a list, there may be a problem with the controller, the hard disks or the cabling. In this case, please contact support.

File System Check

Before you mount the partition or logical volume in question, you should check the file system for consistency with a File System Check (FSCK). 

Caution

A File System Check must not be performed with mounted and/or encrypted partitions, as this may cause data loss. If you check the file system of a Logical Volume, it must not be mounted either. Therefore make sure that the partition or logical volume is not mounted or encrypted before you check the file system with the File System Check.

To check a partition or logical volume with the File System Check, enter the following command:

rescue:~# fsck -f /dev/md1

The following check is performed on the partition md1. The result shows that the file system is in order. 

fsck -f /dev/md1
fsck 1.40-WIP (07-Apr-2007)
e2fsck 1.40-WIP (07-Apr-2007)
/: recovering journal
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/: ***** FILE SYSTEM WAS MODIFIED *****
/: 7628/123648 files (0.5% non-contiguous), 83136/246960 blocks

md5, md6 and md7 are XFS and do not need to be checked. If you still want to check them, you can use the command below. 

rescue:~# xfs_check /dev/mdX

Example:

rescue:~# xfs_check /dev/md5 

If you receive an error message, mount and unmount the partition and run the check again. The following example shows a file system check with an error message.  

xfs_check /dev/md5
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed. Mount the filesystem to replay the log, and unmount it before
re-running xfs_check. If you are unable to mount the filesystem, then use
the xfs_repair -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.
mount /dev/md5 /mnt
umount /dev/md5
xfs_check /dev/md5

If the check again fails, the partition can be repaired with xfs_repair. 

xfs_repair -L /dev/mdx
Caution

Repairing the partition with the command xfs_repair -L /dev/mdx may cause data loss under certain circumstances. You should therefore always create a backup beforehand.

Mounting Internal Hard Disk

Now you can mount the internal hard disk to the /mnt directory:

rescue:/# mount /PATH/Partition /mnt

Example:

rescue:/# mount /dev/md1 /mnt

Note

The displayed partitions or logical volumes may differ depending on server type or server model.  

Displaying a list of filesystems

To display the folders and filesystems in the /etc/fstab file, type the following command:

cat /mnt/etc/fstab
/dev/md1 / ext3 defaults 1 1
/dev/sda2 none swap sw
/dev/sdb2 none swap sw
/dev/vg00/usr /usr xfs defaults 0 2
/dev/vg00/var /var xfs defaults,usrquota 0 2
/dev/vg00/home /home xfs defaults,usrquota 0 2
devpts /dev/pts devpts gid=5,mode=620 0 0
none /proc proc defaults 0 0
none /tmp tmpfs defaults 0 0

Mounting Remaining Partitions

Now mount the partitions to the folders specified in the fstab file, within the /mnt directory.

rescue:~# mount /dev/vg00/usr /mnt/usr
rescue:~# mount /dev/vg00/var /mnt/var
rescue:~# mount /dev/vg00/home /mnt/home

Then switch to the system located on it.

Images without LVM until December 2010

For operating systems installed before Dec 2010, Logical Volume Manager (LVM) was not used and so the path to your partitions will be different:

rescue:~# mount /dev/md1 /mnt
mount /dev/md5 /mnt/usr
mount /dev/md6 /mnt/var
mount /dev/md7 /mnt/home

Changing the Root Directory (chroot)

After mounting the partitions, you can use chroot to switch the root directory from that of the rescue system, to the root directory of the server's hard drive. The /mnt folder will then serve as the root directory,  and any commands or programs run from the command line will be run from your server's hard drive, rather than from the rescue system.

To use chroot to switch to the system installed on the hard drives, type the following command:

chroot /mnt

Now you have access to your complete system and can repair and/or configure your root server system.

pwd
/
ls -l
total 112
drwxr-xr-x 2 root root 4096 Jun 15 17:05 bin
drwxr-xr-x 4 root root 4096 Jun 15 17:07 boot
drwxr-xr-x 10 root root 12288 Jun 15 17:05 dev
drwxr-xr-x 75 root root 4096 Jun 30 06:32 etc
drwxr-xr-x 6 root root 66 Jun 30 12:25 home
drwxr-xr-x 9 root root 4096 Jul 2 2008 lib
drwx------ 2 root root 49152 Jun 15 17:07 lost+found
drwxr-xr-x 2 root root 4096 Sep 22 2007 mnt
drwxr-xr-x 4 root root 4096 Jul 2 2008 opt
drwxr-xr-x 2 root root 4096 Jul 9 2008 proc
drwx------ 10 root root 4096 Jun 16 13:35 root
drwxr-xr-x 3 root root 4096 Jun 19 10:13 sbin
drwxr-xr-x 2 root root 4096 Jun 15 16:52 srv
drwxr-xr-x 2 root root 4096 Aug 14 2008 sys
drwxrwxrwt 2 root root 4096 Jun 15 17:07 tmp
drwxr-xr-x 12 root root 141 Jun 15 17:05 usr
drwxr-xr-x 17 root root 4096 Jun 15 17:07 var
lrwxrwxrwx 1 root root 12 Jun 15 17:05 vmlinuz -> boot/vmlinuz
lrwxrwxrwx 1 root root 16 Jun 15 17:05 vmlinuz.old -> boot/vmlinuz.old

Starting and Stopping Services in the chroot Environment

You can also start services using the Start/Stop scripts using /etc/init.d/ in the chroot environment. For example, start the MySQL database system with /etc/init.d/mysql start. After the work, stop the services again with the argument stop instead of start.

To stop the services again, use the argument stop. In the following example the MySQL database system is stopped again:

/etc/init.d/mysql stop

Note

Not all services can be run in the chroot environment. You should only start services in the chroot environment for repair purposes.

Backup Files

In some cases, a new installation of the operating system is necessary. In this case, you have the following options for backing up the data on your server:

  • You back up the data to your local computer using WinSCP.

  • You save the data on another server using Secure Copy.

  • You save the data on a local computer using Secure Copy.


After backing up your data, you can install a new image on the server or restore a backup of your system. Then can copy the backed up data to the server.


Save data to a local computer with Microsoft Windows operating system using WinSCP

With WinSCP, you can easily copy the data of your server by drag and drop into a directory that is located on your local computer. 

You can download WinSCP from the following page:

https://winscp.net/eng/download.php

For instructions on how to use WinSCP to store your server data on a local computer running Microsoft Windows, click here:

Saving backups on a local computer (server with Linux)

Copy data to another server with Linux operating system using Secure Copy

Secure Copy is a protocol for encrypted data transmission based on Secure Shell. To copy your server's data to another server using Secure Copy, type the following command in the shell:

scp -r /PATH /DIRECTORY username@IP ADDRESS:/PATH/target directory

Example:

[root@localhost ~]# scp -r /mnt root@82.165.69.130:/home/backup

Save data from a local computer with Linux operating system

The following command allows you to save a directory located on the server to your local computer:

scp remote_username@SERVERIP:/DIRECTORY_REMOVAL_FROM_SERVER /LOKAL_DIRECTORY

Example:

[root@localhost ~]# scp root@217.160.44.208:/home /opt/backup

In the example above, the /home directory is copied to the /opt/backup directory located on the local computer.

To save a single file on your local computer, type the following command:

scp remote_username@SERVERIP:/remote/DIRECTORY_ON_THE_SERVER/FILENAME.txt /local_directory 

Example:

[root@localhost ~]# scp root@217.160.44.208:/remote/home/test.txt /TEMP

Use the following command to copy a directory that is on your local computer to your server:

scp -r /LOKALS /DIRECTORY remote_username@SERVERIP:/DIRECTORY_ON_THE_SERVER

Example:

[root@localhost ~]# scp -r /home/backup root@217.160.44.208:/home

To copy a single file, enter the following command:

scp -r /DIRECTORY/FILENAME remote_username@SERVERIP:/DIRECTORY_ON_THE_SERVER

Example:

[root@localhost ~]# scp -r /home/test.txt root@217.160.44:/home

Note

If scp is not found, you can install the program with the following command:

CentOS

[root@localhost ~]# yum install openssh-clients

Debian / Ubuntu

[root@localhost ~]# apt -y install openssh-server

Unmounting the file systems, ending the maintenance session

After you have finished repairing your server, exit the chroot environment and unmount the file systems in reverse order.

Caution

If you perform a restart with mounted file systems, this can lead to data loss. Therefore, make sure that the file systems are unmounted before restarting your server.

exit
umount /mnt/home
umount /mnt/var
umount /mnt/usr
umount /mnt

Running a Memory Test

The memory test memtest86 is also available in the rescue system, which allows you to check the integrity of the server memory.

Note

Please be aware that since the rescue system already uses a certain amount of server memory, this occupied memory cannot be checked with memtest86.

memtest all
[...]
_

Switching the Boot Mode to Normal

Make sure that the boot mode of your server is switched back to normal. To do this, please proceed as follows:

  • Log in to your IONOS account.

  • Click in the product overview on the Menu > Server & Cloud.
    Optional: If you have more than one contract, select the desired Server & Cloud contract.

    The Cloud Panel then opens.

  • In the Server Administration section, click Recovery Tool.

  • Under Recovery Action, select Reboot now and select Normal system.

  • Click Next.

  • Klick OK.

  • Click Go To Overview.

Restarting the Server

It is generally safer to restart the server via the command line, than to force a restart via the Recovery Tool. First change the boot mode in the Recovery Tool as instructed above, then issue the reboot command:

shutdown -r now
shutdown: sending all processes the TERM signal...
Connection to example.com closed by remote host.

Content