All partitions of our Linux-based server suddenly went read-only, all write operations failed with “Read-only filesystem” error. I had never seen such strange problem before, after some Google search, I reached the conclusion that it might be caused by some serious hard disk issues.
It’s 3:00 am, and I had no choices but go to data center to fix that server by myself. It’s such a terrible experience, luckily, the problem got fixed finally.
Here are the steps I took:
1) I guess the problem was caused by some hard disk issues, and running fsck may have a chance fixing it
2) pressing the ‘reset’ button, after a few seconds, SUSE(SUSE Linux Enterprise Server 11) boot failed, and dropped a prompt asking for root password
3) inputting root password to enter safe-mode, we can run fsck to fix the filesystem of all partitions. we use ext3, so run fsck -t ext3 /dev/sda[n] on each partition.
4) Luckily, all filesystems were repaired successfully, Ctrl+D to exit the safe-mode shell, system rebooted.
5) everything went OK.
Many various issues may lead to such problem, these steps work for my particular situation, I can’t guarantee them work for yours. There is NO WARRANTY, run it on your own risk.