Replacing a Faulty Drive in a Software RAID under Linux

At the end of summer, one of the hard drives in my Intel D945GSEJT server failed. With the drive under warranty, and being part of an encrypted RAID 1, it was replaced at no expense from my side, and with no data lost or leaked. With the excellent software RAID under Linux, adding the replacement incurred only a few minutes of lost uptime. I followed a great guide on Linux software RAID management, and these are the few simple steps needed to replace the failing drive:

1. Remove the relevant partitions from the RAID, e.g. if /dev/sdb has failed and the RAID consists of /dev/md0 and /dev/md1:
   mdadm /dev/md0 –remove /dev/sdb1
   mdadm /dev/md1 –remove /dev/sdb2
2. Power down, replace the drive, and power up
3. Copy the partition table from the drive still in the RAID to the new drive, being very careful to get the commands right, e.g.:
   sfdisk –d /dev/sda | sfdisk /dev/sdb
4. Add the new drive to the RAID, e.g.:
   mdadm –add /dev/md0 /dev/sdb1
   mdadm –add /dev/md1 /dev/sdb2

A Test Drive of Moblin 2 Beta

Yesterday I was finally able to test drive Moblin, Intel's Linux distribution for netbooks. Moblin has received acclaim for its fast boot time and its innovative GUI, and I have been eager to try it out. Since Intel has tailored the distribution to run only on systems with a SSE3-capable Intel processor and an Intel GPU, my only choice was to let my Atom-powered home server double as a desktop system for a while.

Installing the Moblin Live on a USB stick was a breeze, with clear instructions available on the site. Booting into the desktop was fast, though not as fast as demonstrated on Phoronix, where Moblin has been tested on a Samsung NC10 with a very fast SSD.

As for the user interface, I cannnot but agree with the reviewer mentioned, that it is very appealing and easy to navigate. On my system, animations were slightly choppy, which might have been due to my relatively slow USB flash drive, or the high resolution at which I was running the system (1920 x 1080).

Moblin is a very promising distribution for netbooks. Even during my very brief testing, I found quite some points where improvements could be made, I'm sure in time these minor flaws will be corrected. It seems like quite some OSes are intending to compete for netbook dominance (Cloud, Google Chrome OS, etc). It will be interesting to follow the development.

A Word of Caution When Connecting Multiple Drives to the D945GSEJT On-Board Power Connector

Some caution has to be taken when connecting storage devices to the on-board power connector of the D945GSEJT. Credit goes to mcnels1, who made a very observant comment yesterday, on my D945GSEJT home server setup with two 1TB Western Digital Caviar Green (1TWDCG) hard drives.

As mcnels1 pointed out, it seems hazardous to connect two 1TWDCG, which possibly require more than double the power that the board is able to deliver. The maximum power draw of a 1TWDCG is, according to the specifications (last page), 1.671A at 12VDC (which translates to ~20W). The D945GSEJT specifications however state (page 19, last paragraph) that the on-board power connector delivers a maximum of 1.5A at 12V (18W).

However, I have so far experienced no problems with the setup, that could be referred to the power supply being inadequate to power the hard drives. I performed numerous reboots (which is when the hard drives would be expected to draw the most power), due to the complications with the setup concerning the Ethernet controller, without any failures. Also, massive read/write operations have been performed without any problems. For example, due to a defective sector on one of the hard drives, the drive was taken offline from the RAID1, after which I tested every sector on the drive, and then resynced the entire 1TB of the encrypted LVM volume. Also, the drives undergo weekly SMART testing, lasting for almost 4 hours, which has so far always succeeded.

This apparent contradiction between theory and practice might be due to the power consumption of the 1TWDCG being far less than the stated maximal 20W. In a product review that I found yesterday, the reviewer has a D945GSEJT system with an IDE flash drive and one 1TWGCG, and states (translated with some modifications for better conformity with English):

... I have an AC-DC adapter from Seasonic which is supposed to have 80% efficiency and, measured before the AC-DC adapter, the system draws about 20W with only the mother board and the flash drive connected. With the 1TB WD Green added, the computer idles at 23W, which rises to 25-28W when it reads/writes.

This would indicate that the maximum power consumption of the 1TWDCG is really 80% of 8W, which is 6.4W. If this is indeed the case, the on-board power connector should have no problems supplying the hard drives.

Nevertheless, a word of caution is in order. As a late reply to one of my readers who considered connecting four drives to his D945GSEJT, I would advise against using the on-board power connector, taking the above information into account.

By Request: a Look Inside the Intel D945GSEJT Server

I received a request today to snap a picture of the D945GSEJT board mounted inside the mini-ITX case, so here it is:


In the picture above, I have marked some of the connectors (and left some out, e.g. the USB ports):

1. Ethernet port
2. DVI connector
3. 12VDC power connector
4. SODIMM connector
5. Power switch
6. SATA connectors
7. On-board power connector for hard drives

The lower third of the machine as shown in the picture is occupied by the mini-ITX PSU, which I do not use. Because of the design of the case, with the hard drives attached to the lid of the case (the lid is flipped on its side in the picture), there is no room for a PCI card. It would be possible to have a PCI card though, if the machine only needs one hard drive. In that case, the PCI card would be attached to a PCI riser attached to the PCI slot on the board.

Update: Debian Lenny 5.02 Installer and Intel D945GSEJT

When I installed Debian Lenny on my Intel D945GSEJT home server, I reported that the Debian installer did not recognize the network adapter. With the latest version of the installer (5.02), the r8169 module is loaded, and at least on my system, DHCP autoconfiguration of the network connection succeeds. This is a step up from the previous installer, but r8169 is in fact the wrong kernel module, as reported in this post. What you want is the r8168 module, and you can find instructions on compiling that module in the previous link.

Retrieving a List of User/Manually Installed Packages in Debian/Ubuntu

If you are like me and have been running a Linux system for a while, you inevitably come to a point sooner or later where you would really like to know what packages you have been installing - yourself. In the case of Debian (and friends like Ubuntu), the installer pulls in a lot of packages, and there is no special indication (AFAIK) which packages were installed by the installer, and which packages were installed manually by the user. You might want the list in order to quickly get up to speed on a new installation, or, even more probably, clean up a bit among all those packages that you installed because you thought you needed them, or needed them only once or twice.

I have seen quite a lot of suggestions that you should work with the entire list of installed packages, but that is extremely cumbersome if your intention is to clean up, and if you are dealing with a new installation, you risk pulling in obsolete or conflicting packages. What I wanted was a list with just the packages that I had chosen to install myself.

My solution is a bit brute force, but it works very well. Using a virtual machine manager (I use VirtualBox since it is free), I set up a virtual machine with a similar installation as the one for which I wanted to get the list of installed packages, only in the case of the virtual machine installation, I will never install any packages myself, only keep the base installation up to date. On the virtual machine, I created a list of installed packages with the following command (borrowed from here with slight modification):

aptitude search ~i | grep -v "^i A" | cut -d " " -f 4 > clean.txt

aptitude search ~i lists the installed packages, grep -v "^i A" removes the lines starting with "i A" (automatically pulled in dependencies), and cut -d " " -f 4 > clean.txt filters out the package names (starting at position four and ending with a space). The result is written to the file clean.txt. Perform a similar command on the machine for which you want to retrieve the list of installed packages, substituting clean.txt with another file name, say modified.txt. With the two files in the same directory (on whichever machine), perform the following command to get the list of packages that are listed in modified.txt, but not in clean.txt (here is a reference to set operations available from the command line, and other useful commands):

join -v1 modified.txt clean.txt

If anyone has a simpler solution to the problem, please let me know, but once set up, this works like a charm.

Lessons from the D945GSEJT Server

I was very happy to discover a comment on the setup of my Intel D945GSEJT home server today. I am glad to hear that my post helped you, Jules.

The home server works very well. I have learned a lot in the few days that the system has been running, and I felt it was time to share some of the knowledge I have gathered so far.

My choice of building the system in a fanless case with a built-in power supply was not the best. A power supply inevitably produces some heat, and with the whole setup being fanless, the harddrives reported temperatures close to 60 degrees Celcius. The specifications of the drives said that this was slightly above the upper safe temperature limit for operation. I changed to an external 12VDC adapter, and got the temperature of the harddrives down to about 40 degrees Celcius, which is perfectly acceptable.

The D945GSEJT board offers two power connectors, a 12VDC connector on the back panel, and a 4-pin 12V ATX internal connector. The Technical Product Specification of the board says that the 12VDC connector is "preferred", but does not in any way explain why that would be so, which is very strange. Connecting a mini-ITX power supply (with one 4-pin connector and one 20-pin connector) requires you to short two pins on the 20-pin connector, since this is the signal that the motherboard normally gives to the power supply to make it start and deliver the correct voltages on all pins. This is not anything I recommend in any way, but it was what I had to do to be able to use the built-in power supply of my mini-ITX case. I suspect that Intel had the idea that you would use some special power supply which is always on and only has the 4-pin connector, but I have not found any such supplies. My case gave me small electrical shocks using the internal power supply, which might have been caused by my unorthodox solution, but it might just have been the power supply that was dodgy. I am glad I switched to the external power supply, which has none of these worries.

Getting acquainted with mdadm for managing the RAID, and smartmontools for monitoring hard drive health was an essential first step in learning to administer my new home server. It turned out that one of my hard drives had a bad sector, and I got to apply this knowledge almost immediately. Emails from smartd started pouring in, telling me about the problem on the drive. The selftests that I had scheduled for the hard drives had fortunately located the error to a sector which was towards the end of the drive, which was probably not occupied by any data. The solution I ended up using was hdrecover, an extremely simple (293 lines of C code) program that tests the readability of each sector on a drive. If an unreadable sector is found, the program writes some data to it, forcing the drive to mark the sector as bad, an relocate it. In my case, being fairly certain that the defective sector was not occupied by any file, this was a solution as good as any. If you want to locate the file that occupies a certain sector, this can be a bit tricky. I found a howto on this subject, but I am not sure how I would get it to work in my case, with partitions on an encrypted LVM partition.

Setting Up a D945GSEJT Server with Debian Lenny

I previously blogged about the QNAP Turbo Station as a possible candidate for a low power home server. The big problem with that solution was the loud fan, which seemed to be a real show stopper. Searching for an alternative solution, I found the recently released Intel D945GSEJT motherboard. With a TDP below 20W (apparently ~10W when idle) and being totally fanless, it is ideal for the task. I combined this with two 1TB Western Digital Caviar Green discs (cool, quiet and with low energy consumption) in an Emko EM-153 case with a built-in, fanless 60W power supply.

The resulting build is a completely passively cooled, extremely low power, fully functional home server. I installed Debian Lenny, the currently stable release of Debian, which is known to be very robust, and will allow me to easily configure the machine as an SSH server, web server with LAMP, etc. With the disks set up as RAID1 and an encrypted LVM partition containing all but /boot, it is first and foremost the perfect solution for secure file storage.

This setup took quite some effort, which is why I will document the critical points I encountered, for my own reference, and hopefully to help anyone who comes across this entry dealing with a similar setup.

The first problem was that of setting up the OS. I had more or less decided on using Debian, since I am running Ubuntu on my desktop machines. With no optical disc drive connected to the system, I considered installing Debian from USB. The problem with this solution is that the Debian Lenny installer is unable to recognize the Realtek 8111DL Ethernet controller (previous versions of the Debian Lenny installer, read this update about the latest version of the installer). I realized after the install was finished that it would probably be possible to load the correct kernel module manually during the install, which is of course a more elegant solution than the one I ended up using. Ubuntu 9.04 (Jaunty) Server, which I considered as an alternative, also had problems with the NIC (I found a reference that this would work, but I had no success myself). Intel lists some Linux distributions that they have tested on the board, one of which is Ubuntu 8.10 (Intrepid). The installer for Ubuntu 8.10 Server indeed recognized the NIC, so beside the fact that I would settle for an earlier distribution, and not solve the network problem permanently, it could have worked.

However, a problem with pre-Lenny Debian installers, like the one for Ubuntu 8.10, is that the partition editor has a serious flaw in that it cannot correctly handle encrypted partitions set up on a RAID device. There is a workaround for this, which I tried, and unfortunately did not get to work.

My brute force solution, growing tired of the problems I encountered, was to hook up the hard drives to my AMD 64 X2 4850e desktop computer, and perform the installation of Debian Lenny using the DVD drive and network card (another Realtek NIC that the installer could handle) of that machine. A small bump in the road was that the installer hanged when booting, which I solved by specifying the option acpi=off to the kernel. I found a reference to another person having the same problem, in which case the problem was solved by specifying pci=off. For me, this resulted in being able to boot, but prevented the detection of my SATA DVD burner. I now was able to set up the partitioning correctly, and let the installer percolate for about 24 hours, since the 1TB encrypted RAID partition had to be initialized with random data.

With the discs still wired to the desktop computer, I booted into the new system, and used the working network connection to dowload and compile the correct kernel module for the Ethernet adapter. The instructions for doing this can be found here. Although in that case the Intel D945GCLF2 board was used, which has the RTL8111C NIC, the D945GSEJT features the RTL8111DL Ethernet controller, which is handled by the same kernel module (r8168), as can be seen on the Realtek download page for the drivers. After following all of the advice given in the mentioned blog entry, including the module blacklisting, the kernel kept loading the wrong (ie r8169) module when I moved the discs to the Intel Atom board. I tracked this issue down to the file /etc/udev/rules.d/70-persistent-net.rules, which listed the r8169 module, which was the correct module for the desktop AMD computer NIC. Commenting out all the lines in this file and rebooting, udev reconfigured the network setup, and I finally had everything working correctly on the mini-ITX server.

One artefact from the installation procedure was that the acpi=off option made its way from the installer to the final install. Removing this option worked fine on the Intel board, and enabled me to use frequency scaling, which of course is a very desirable feature for an always-on system as this one.

So there you have it. Lots of obstacles along the way, but the end result was well worth it. I hope this can serve to ease the burden of anyone else planning on a similar setup.

Back to Xandros

I've gone full circle, back to using the factory installed "Xandros" distro on my Asus Eee 900.
I had some hurdles to jump on the way. First of all, I had misplaced the Eee packaging including the support and recovery discs. Then, as some sort of bizarre irony, you need to run a Windows program from the support CD to create the reinstallation USB stick from the recovery DVD (it is of course possible to do it manually in Linux, but I wanted to do it the legit way). The only Windows installation I have is the one on my quadruple-boot Dell Inspiron, which cannot read DVD-R. Back to my main computer, where I made ISO images of the support and recovery discs and set up Samba to transfer the files to the Inspiron. On the Inspiron, I used Deamon tools to mount the isos, and was finally able to make the reinstallation USB stick.
As a pleasant surprise, the factory Linux has had quite a lot of updates since I last tried it in June last year. Actually, this distro is perfect for my needs right now. All the programs I would typically use on a trip (which is what I bought the Eee for in the first place) are there, mainly a web browser and instant messengers (Skype without the audio cracking up like in Easy Peasy and EeeBuntu, at last!). One addition since June is a large directory of programs available through the so called "Eee download". Better still, even if the developers have tried to hide as much as possible behind the main interface, you can access a terminal with Ctrl-T in the file manager. Now things get interesting. Updates are available through apt-get, ssh turns out to be installed by default, etc. There is actually a decent catalogue of programs available through apt-get from http://update.eeepc.asus.com/, to which /etc/apt/sources.list points. This, and the fact that the OS boots in a matter of seconds (as opposed to other distros for the Eee) makes me forget about transgressions like the XP-like theme or that USB sticks are auto-mounted as D:. This machine has risen from the ashes, and now I'm ready for my upcoming trips to Spain and Italy!

Asus Eee PC 900 Recovery Blues

I have tried quite some distros on my Asus Eee PC 900, but I have finally done full circle, and decided to reinstall the "Xandros" OS that was installed on delivery. With the Ubuntu-based distros for the Eee that I have tried (Easy Peasy, eeebuntu, and Ubuntu itself) the sound in Skype cracks up. This is a common issue, but I have not been able to get it to work, despite all the tips and tricks I have found and tried. The closest I have come to a working setup on the Eee is with FluxFlux-Eee, where I get perfect sound in Skype. One major drawback of all these compared to the factory distro is the looong boot time, and that is one of the main reasons I have decided to revert. An alternative would be Moblin, but it is still only in alpha, and only works on SSE3-capable processors (read: Atom - the Celeron in the 900 does not have SSE3).

Unfortunately, I have no idea where I have put the recovery CD that came with the Eee PC, or if I even have it. The recovery CD is of course not downloadable from Asus (but the source code is, wonder how many people have actually tried to compile it). For some odd reason, there is a recovery image available on SourceForge, but only for the 701. Using that image actually works on my 900, but I don't get any sound (another confirmed issue without solution). As a last resort I could buy a replacement for the recovery cd from ASUSparts, but to shell out another €15 on the 900 is grim, since I am aldready beating myself up for not waiting for the Atom-based netbooks.

Finally... Clean URLs in Drupal with Debian Lenny

I managed to find out how to enable clean URLs in Drupal (I did a manual installation of Drupal 6.10) on Debian 5.0 (Lenny). What bliss. The correct advice was found among the comments to the "clean URLs with Debian" page that I referred to before. In order to save some potential readers the trouble, and as a reminder to myself, here is the recipe (everything must be run with root privileges):

1. Execute a2enmod rewrite to enable mod_rewrite. You can run apache2ctl -M to verify that the module is loaded (shows up as rewrite_module in the list) after performing step 3 below.
2. Change the file /etc/apache2/sites-available/default so that the section Directory /var/www/ includes the statement AllowOverride All instead of AllowOverride None.
3. Reload apache modules by issuing /etc/init.d/apache2 force-reload.
   

After this you can enable clean URLs through the administration menu in Drupal.

QNAP TS-209 II Research

My research about the QNAP TS-209 II continues. It turns out that people are very satisfied with all aspects of this device, except the 70mm fan, that one person described as being louder than his first full tower PC from 1984. Fortunately, I found some creative advice on how to modify the TS-209 to make it quieter. The way to do it seems to be to use an 80mm fan (there are some quiet ones from Noctua for example) together with a 70mm to 80mm adapter. These adapters are not too common, but I found some at Kustom (UK, no chance of getting them here in Sweden it seems).

At the same time, I'm studying up on Drupal to be able to set up a website if and when I get the TS-209. The LAMP stack and Drupal are installed on the Dell Inspiron now, which gives me a nice base to experiment on.

I promised some ranting, and I didn't have to wait long to find something to get frustrated about. In Drupal, there is a possibility to get "clean URLs" for your website, even a page with instructions for Debian. Sounds easy? Well, the line "Note: This article needs to be updated for Debian "Etch" and "Lenny" releases. (as of March 2009)" should have given me a hint that it was bound not to work. There is just too much outdated information out there. Mental note: write down how I did it if I do get it to work, so that someone is spared from the same insane forum hunt.

DI-624 not the culprit?

Last week I was struggling with my SSH server on my Dell Inspiron running Gentoo. The connection was reset randomly, and so frequently that it was practically unusable. I thought my DI-624 router was to blame, since various posts accused that router model for being plagued with problems. As I often do, I grew tired of Gentoo and switched to experimenting on the Debian installation on the same computer (I switched Ubuntu Intrepid for Debian Lenny last week). Strangely enough, with Debian I don't experience any connection resets. As usual with Linux, I can't help but wonder what was wrong with the OpenSSH setup on Gentoo.

I often wonder if there is no better way to organize Linux support than forums. It's not often that you find something usable browsing forum posts. All too often the posts are outdated, or everyone is as confused as you are. Try finding information about whether a specific wireless network card is supported under Linux for example. I think that rant will be the topic of the next post.

Connection Reset by Peer

I got the Laptop-based Gentoo server replacement for a future QNAP TS-209 up and running. The only problem is that SSH connections to the server are extremely unreliable and I keep getting disconnected, which of course is very annoying. How do you track down an error like this? My main suspect is my router, though. I have understood from posts on other sites that the DI-624 is notorious for dropping connections, locking up, being sluggish, you name it. Thinking about it, I have experienced quite a lot of problems downloading larger package updates over the years. Some days have been worse than others, and I've actually believed it was my ISP. Apparently Linksys are renowned for making good routers, so maybe it is time to upgrade to something more stable. If so, I will have to ask myself if it is time to upgrade to Gigabit ethernet as well...

Hardware Recycling

As I wrote yesterday, I would really like to get my hands on a QNAP TS-209 II. It seems like an ideal home server; it has extremely low power consumption, and you can install Linux on it to tailor it to your own needs. With other NAS devices, it seems what you get is a (buggy) proprietary firmware with a limited array of services.

Waiting for the TS-209 to make it to the top of my gadget shopping list, I decided to make do with what I have. The revamped Dell Inspiron 2650 that was the subject of my very first post is a quadruple-boot machine these days (Windows XP, Ubuntu, Linux From Scratch and Gentoo). I decided to tailor the Gentoo installation to act as a test home server, since it's very customizable, and fun to tweak.

Gentoo is what I started out with when I was getting to know Linux. In Gentoo, you compile all of the software on your machine. It might take you a couple of days to compile all the packages needed to get a Gnome desktop, but once you do, it is compiled specifically for your processor type, and exactly with the features that you have selected. Gentoo is a lot of fun if you don't mind having all-night compile sessions on your computer once in a while, or pulling your hair over some bizarre error that you solve ten days later realizing it was due to some version inconsistency deep down in the dependency tree. After a while, the hair pulling was too much for me. Since I switched to Ubuntu on the machines I use routinely, the amount of time spent on maintenance has decreased dramatically. But for a project like this, when you want to learn how all the bits and pieces fit together, Gentoo is perfect.

I have come so far as to administer the system remotely via SSH, and it is very satisfying to be able to log in on my lunch break and see 'emerge -e world' chew away on the packages, optimizing the system for my old Celeron.

Before I sign off, I have to make a note that GNU Screen is probably one of the most overlooked packages there is. I had never heard of it until a few days ago, but it is that package that makes it possible to do what I just described - to detatch from a terminal session and attach to it at a later time. Be sure to check it out.

The Quest for a Media Server

I have been on the lookout for a "Media Server" for quite some time now - a computer with very low power consumption that could run Linux and had accelerated video decoding with HDMI output. Media players similar to the TViX are finally being launched by other companies (Western Digital, QNAP). Hopefully we will see the segment grow into the main stream user community, which will put more pressure on manufacturers to come up with firmwares that are less buggy.

Another interesting range of products is the NAS segment. Network Attached Storage like the QNAP Turbo Station units have grown beyond acting as simple storage units and offer services such as BitTorrent clients, FTP and HTTP servers, in fact complete Linux systems. In the case of the QNAP Turbo Station, you can in fact install Debian on it, as it seems without too much hassle.

With their ~20W footprints, these products are great replacements for a HTPC in the case of the TViX-like media players, and for a regular ATX-based home server in the case of the NAS. But what if you could have both? Would it be impossible to equip an ARM-based device like the Turbo Station with a graphics processor like the Sigma Designs chip in the TViX?

As an alternative, it seems that Asus are releasing the second generation of eee Boxes, and while the "low-end" models are not powerful enough to play HD material, maybe the "high-end" b208 is, with its dual-core Atom processor. Since they are equipped with ATI Radeon chips, one would think that video decoding offloading to the graphics processor should be no problem, at least in Windows, which still seems to be the main focus of graphics card manufacturers when it comes to driver development. On Linux, only nVidia seem to have a working framework for video decoding (VDPAU, still too fresh to have made its way into most Linux distibutions it seems). AMD/ATI have a great open source initiative going on (as opposed to nVidia), but so far there seems to be no way of offloading video decoding in Linux with the proprietary Catalyst driver or the open source Radeon and RadeonHD drivers for ATI cards. The Quest continues...