I have had a pretty dreadful NAS solution for the last few years - a USB drive attached to the Asus router, accessible through a Samba share. The target user for the router is probably someone who stores some family pictures to the drive, and surfs to a few web pages now and then. I am sure it works fine for that use case. I had a 2 TB and a 500 GB drive attached, both more or less full, with Crashplan running on a server to back everything up to their data centers. It turned out the shares disappeared every now and then, which made the Crashplan backend crash (no pun intended). Moderately heavy traffic on the router also caused the router itself to crash, since it probably ran out of memory. This in turn caused the file systems on the USB drives to be damaged, leading to loss of files when the file systems were repaired.
It is pretty amazing that you can develop and sell a typical consumer router/switch/access point for less than $50, but you should not be surprised that cracks pretty fast under pressure. It is time to put together a better solution - more about that in coming posts.

A Guide to Open Source AVCHD .mts Conversion

Having recently bought a Sony Handycam video camera, I was suddenly faced with an unanticipated need to manually convert the recorded movies into a usable format using open source tools available for Linux. Sony assumes that you run their proprietary software (of course available for Windows and OS X only) for copying recorded movies to your hard drive and burning them to DVD or Blu-Ray discs. For burning HD movies, Sony's solution is the AVCHD format, which is a Blu-Ray-like format for DVDs. A far from optimal solution, considering I do not and do not plan to install Windows or OS X, and that the AVCHD format seems poorly standardized, so that an AVCHD disc working on a Sony Blu-Ray player might not work on Blu-Ray players from other manufacturers. My solution to this problem is converting the movies to a more standardized format.

In many cases, you can convert the .mts files with ffmpeg directly, so you should try that first. However, with some clips from my camera, ffmpeg has problems interpreting the file. In that case, it seems you have to take the long way home, converting the video to a raw YUV format, before recoding it to your desired output format.

To begin with, you need to extract the movies stored on the camera. When mounting the camera as a USB mass storage device, a large directory structure is revealed, in parts similar to the AVCHD directory structure. The recorded movies can be found as .mts files in the AVCHD/BDMV/STREAM folder. Copy these files to a working directory.

We will need to compile some tools for the conversion. The first of these is xporthdmv. Download the source package (xport.zip) from the developers web page and unzip it. Then compile it:
gcc -o xporthdmv xport.c
Now adjust file permissions and copy the executable to somewhere in your path as needed.

The next tool is ldecod. Download the source package (latest version, which at the time of writing is jm17.0.zip) from the developers web page, and unpack it. The source can be compiled with the following commands, executed from the JM folder:
chmod 755 unixprep.sh
./unixprep.sh
cd ldecod
sed -i s/-march=pentium4//g Makefile
make
Now adjust the file permissions of the resulting ldecod.exe executable (located in the JM/bin directory) and copy it as ldecod to somewhere in your path as needed.

You will also need ffmpeg, but since this is probably available through repositories for your Linux distribution, I will leave that installation safely in your hands.

Now we are ready to start converting. First, the .mts file will need to be demuxed into the corresponding audio and video files:
xporthdmv -nh movie.mts 1 1 1
This will result in two files bits0001.mpv (H.264 video) and bits0001.mpa (AC3 audio). The reason why we are using xporthdmv to demux, and not using ffmpeg directly, is that ffmpeg has trouble reading some of the .mts files that the Sony Handycam generates.

Next, we will use ldecod to convert the video to YUV. It might be possible to use the demuxed video directly with ffmpeg, but in my attempts, ffmpeg has been having problems reading the file, presumably since it is interlaced 50 fps video, while the container indicates that it is 25 fps. The resulting file is huge, so make sure you have enough space. This is how to do the conversion:
ldecod -i bits0001.mpv -o movie.yuv

We can now used the demuxed and converted video and audio files as input to ffmpeg. Even if recoding is not generally recommended, I wanted to have the movies in a standard-compliant mp4 container, and get rid of the horrible, horrible interlacing. Here is how to generate a deinterlaced mp4 with H.264 video and AAC audio:
/usr/bin/ffmpeg -r 25 -s 1440x1080 -i movie.yuv -i bits0001.mpa -acodec libfaac -ar 48000 -ab 160k -ac 2 -vcodec libx264 -vpre hq -b 3000k -aspect 16:9 -deinterlace movie.mp4
The options could need to be adjusted (for example the first and/or second -r 25, the first one indicating that I have a PAL camera, and the second that I want to keep that framerate for the output) if you have a different input, or if you want other bitrates etc, for the output.

In the next example, we are generating a low resolution AVI file with XViD video and MP3 audio, since this is one of the most widely supported formats for standalone players:
/usr/bin/ffmpeg -r 25 -s 1440x1080 -i movie.yuv -i bits0001.mpa -f avi -vcodec libxvid -vtag XVID -s 320x270 -r 25 -aspect 16:9 -b 700k -qmin 3 -qmax 5 -bufsize 4096 -mbd 2 -bf 2 -flags +4mv -trellis -aic -cmp 2 -subcmp 2 -g 300 -acodec libmp3lame -ar 48000 -ab 128k -ac 2 movie.avi
Again, adust the options if you want higher resolution, higher bitrate, etc.

References:

1. How to recover video from AVCHD (using open source software
2. HOWTO: ffmpeg & x264 presets
   

Replacing a Faulty Drive in a Software RAID under Linux

At the end of summer, one of the hard drives in my Intel D945GSEJT server failed. With the drive under warranty, and being part of an encrypted RAID 1, it was replaced at no expense from my side, and with no data lost or leaked. With the excellent software RAID under Linux, adding the replacement incurred only a few minutes of lost uptime. I followed a great guide on Linux software RAID management, and these are the few simple steps needed to replace the failing drive:

1. Remove the relevant partitions from the RAID, e.g. if /dev/sdb has failed and the RAID consists of /dev/md0 and /dev/md1:
   mdadm /dev/md0 –remove /dev/sdb1
   mdadm /dev/md1 –remove /dev/sdb2
2. Power down, replace the drive, and power up
3. Copy the partition table from the drive still in the RAID to the new drive, being very careful to get the commands right, e.g.:
   sfdisk –d /dev/sda | sfdisk /dev/sdb
4. Add the new drive to the RAID, e.g.:
   mdadm –add /dev/md0 /dev/sdb1
   mdadm –add /dev/md1 /dev/sdb2

A Test Drive of Moblin 2 Beta

Yesterday I was finally able to test drive Moblin, Intel's Linux distribution for netbooks. Moblin has received acclaim for its fast boot time and its innovative GUI, and I have been eager to try it out. Since Intel has tailored the distribution to run only on systems with a SSE3-capable Intel processor and an Intel GPU, my only choice was to let my Atom-powered home server double as a desktop system for a while.

Installing the Moblin Live on a USB stick was a breeze, with clear instructions available on the site. Booting into the desktop was fast, though not as fast as demonstrated on Phoronix, where Moblin has been tested on a Samsung NC10 with a very fast SSD.

As for the user interface, I cannnot but agree with the reviewer mentioned, that it is very appealing and easy to navigate. On my system, animations were slightly choppy, which might have been due to my relatively slow USB flash drive, or the high resolution at which I was running the system (1920 x 1080).

Moblin is a very promising distribution for netbooks. Even during my very brief testing, I found quite some points where improvements could be made, I'm sure in time these minor flaws will be corrected. It seems like quite some OSes are intending to compete for netbook dominance (Cloud, Google Chrome OS, etc). It will be interesting to follow the development.

A Word of Caution When Connecting Multiple Drives to the D945GSEJT On-Board Power Connector

Some caution has to be taken when connecting storage devices to the on-board power connector of the D945GSEJT. Credit goes to mcnels1, who made a very observant comment yesterday, on my D945GSEJT home server setup with two 1TB Western Digital Caviar Green (1TWDCG) hard drives.

As mcnels1 pointed out, it seems hazardous to connect two 1TWDCG, which possibly require more than double the power that the board is able to deliver. The maximum power draw of a 1TWDCG is, according to the specifications (last page), 1.671A at 12VDC (which translates to ~20W). The D945GSEJT specifications however state (page 19, last paragraph) that the on-board power connector delivers a maximum of 1.5A at 12V (18W).

However, I have so far experienced no problems with the setup, that could be referred to the power supply being inadequate to power the hard drives. I performed numerous reboots (which is when the hard drives would be expected to draw the most power), due to the complications with the setup concerning the Ethernet controller, without any failures. Also, massive read/write operations have been performed without any problems. For example, due to a defective sector on one of the hard drives, the drive was taken offline from the RAID1, after which I tested every sector on the drive, and then resynced the entire 1TB of the encrypted LVM volume. Also, the drives undergo weekly SMART testing, lasting for almost 4 hours, which has so far always succeeded.

This apparent contradiction between theory and practice might be due to the power consumption of the 1TWDCG being far less than the stated maximal 20W. In a product review that I found yesterday, the reviewer has a D945GSEJT system with an IDE flash drive and one 1TWGCG, and states (translated with some modifications for better conformity with English):

... I have an AC-DC adapter from Seasonic which is supposed to have 80% efficiency and, measured before the AC-DC adapter, the system draws about 20W with only the mother board and the flash drive connected. With the 1TB WD Green added, the computer idles at 23W, which rises to 25-28W when it reads/writes.

This would indicate that the maximum power consumption of the 1TWDCG is really 80% of 8W, which is 6.4W. If this is indeed the case, the on-board power connector should have no problems supplying the hard drives.

Nevertheless, a word of caution is in order. As a late reply to one of my readers who considered connecting four drives to his D945GSEJT, I would advise against using the on-board power connector, taking the above information into account.

By Request: a Look Inside the Intel D945GSEJT Server

I received a request today to snap a picture of the D945GSEJT board mounted inside the mini-ITX case, so here it is:


In the picture above, I have marked some of the connectors (and left some out, e.g. the USB ports):

1. Ethernet port
2. DVI connector
3. 12VDC power connector
4. SODIMM connector
5. Power switch
6. SATA connectors
7. On-board power connector for hard drives

The lower third of the machine as shown in the picture is occupied by the mini-ITX PSU, which I do not use. Because of the design of the case, with the hard drives attached to the lid of the case (the lid is flipped on its side in the picture), there is no room for a PCI card. It would be possible to have a PCI card though, if the machine only needs one hard drive. In that case, the PCI card would be attached to a PCI riser attached to the PCI slot on the board.

Update: Debian Lenny 5.02 Installer and Intel D945GSEJT

When I installed Debian Lenny on my Intel D945GSEJT home server, I reported that the Debian installer did not recognize the network adapter. With the latest version of the installer (5.02), the r8169 module is loaded, and at least on my system, DHCP autoconfiguration of the network connection succeeds. This is a step up from the previous installer, but r8169 is in fact the wrong kernel module, as reported in this post. What you want is the r8168 module, and you can find instructions on compiling that module in the previous link.