Saturday, 1 October 2011

Update to the fancontrol script

One final annoyance when starting up the qnap was that my /etc/rc.local never finished and so the led at the front of the box would never settle down to all green.

Now that I am happy with the scripts I thought I would make them into a daemons. A quick google led to Tony's Cafe blog who described beautifully how to create a debian script for the fancontrol. So I modified his instructions slightly:

Copy /etc/init.d/skeleton to /etc/init.d/fancontrol

Edit /etc/init.d/fancontrol

Change:
NAME=fancontrol
DAEMON=/usr/local/sbin/$NAME
DAEMON_ARGS=""
All seemed to be fine except that when I ran /etc/init.d/fancontrol the script would just hang... A bit more googling made me realise that I was no longer running '/usr/local/sbin/fancontrol &' I was running it without the & flag. A quick modification to the start of my fancontol script to have this at the top (take from linuxquestions) :
if [ "x$1" != "x--" ]; then
$0 -- 1> /dev/null 2> /dev/null &
exit 0 fi

Sunday, 3 July 2011

Dell Studio 1558 fan problems

So we have a dell studio 1558. It's a beast of a laptop, quad core, radeon hd 5400, and in windows land can play quite a few games at a respectable fps.

Now therein lies the problem - this is one hell of a power hungry laptop

The Debian install has been a bit of a nightmare: the fan was on all the time, frequent hard lockups especially when on battery power and add to that the flaky broadcom wifi drivers. It has been an interesting year with this beast.

The only stable configuration seemed to be to run the laptop with the closed source fglrx and leave the laptop plugged into the wall. Anything else was tempting fate.

Recently kernel 2.6.39 hitting testing I though I would try the open source radeon KMS drivers. Last time there was no power control so the gpu was running hot so the fan was always flat out. Installing the flgrx drivers fixed the overheating problem, but the laptop was still quite flakey.

I uninstalled the fglrx drivers and enabled KMS by changing /ect/modprobe.d/radeon-kms.conf to:
options radeon modeset=1


I rebooted and once again got the dreaded maxed out fan. This time I knew it was most probably the kernel running the gpu at maximum speed. A quick look at the following (mount -t debugfs debugfs /sys/kernel/debug/,cat /sys/kernel/debug/dri/0/radeon_pm_info) showed:

default engine clock: 750000 kHz
current engine clock: 750000 kHz
default memory clock: 800000 kHz
current memory clock: 800000 kHz
voltage: 900 mV
PCIE lanes: 16

The gpu was running flat out. A quick google later gave me the following commands to stick into my rc.local:
echo profile > /sys/class/drm/card0/device/power_method
echo low > /sys/class/drm/card0/device/power_profile
Now cat /sys/kernel/debug/dri/0/radeon_pm_info gives:
default engine clock: 750000 kHz
current engine clock: 156990 kHz
default memory clock: 800000 kHz
current memory clock: 199960 kHz
voltage: 900 mV
PCIE lanes: 16
Now the computer seems to be running a lot cooler - I have yet to get it to crash on battery power and if I do have any problems I will hopefully get some help. I did a quick test with fullscreen flash video and there was no significant screen fps problems to hopefully the majority of our problems with this machine are over.

I'm going to do a bit more googling to see if I can reduce the number of PCIE lanes or even the voltage but for the time being at least I have a useable laptop.

The only unsolvable problem seems to be if I try and change the brightness too quickly I get a total crash....

Saturday, 2 July 2011

Replacing a hard drive on my QNAP debian install

My replacement drive arrived - time to add it to the existing raid. From my exisiting hard drive I get the following output:

Partition Table: gpt
Number  Start   End     Size    File system  Name  Flags
 1      17.4kB  3000MB  3000MB               Root  raid
 2      3000MB  4000MB  1000MB               Var   raid
 3      4000MB  4512MB  512MB                Swap  raid
 4      4512MB  2000GB  1996GB               Data  raid
Here are the commands I issued in parted:
mktable gpt
mkpart Root 17.4kB 3000MB
toggle 1 raid true
mkpart Var 3000MB 4000MB
toggle 2 raid true
mkpart Swap 4000MB 4512MB
toggle 3 raid true
mkpart Data 4512MB 100%
toggle 4 raid true
To add the partitions to the existing, degraded arrays you have to first drop the old drive and add the new one using mdadm. Double check which drive you are replacing and type the following:
mdadm /dev/md0 --fail /dev/sdb1 --remove /dev/sdb1
mdadm --manage --add /dev/md0 /dev/sdb1
mdadm /dev/md1 --fail /dev/sdb2 --remove /dev/sdb2
mdadm --manage --add /dev/md1 /dev/sdb2
mdadm /dev/md2 --fail /dev/sdb3 --remove /dev/sdb3
mdadm --manage --add /dev/md2 /dev/sdb3
mdadm /dev/md3 --fail /dev/sdb4 --remove /dev/sdb4
mdadm --manage --add /dev/md3 /dev/sdb4
I found that for some reason the /dev/md2 (swap partition) didn't seem to be happy - so I totally deleted /dev/md2: and created a new one from scratch
swapoff /dev/md2
mdadm --stop /dev/md2
mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3
mkswap /dev/md2
swapon /dev/md2
I used the new UUID information to alter the /etc/fstab info to amke sure swap came on every reboot.

Thursday, 30 June 2011

Read only root and dhclient

To make my read only root viable I have a separate /var directory and /tmp mapped to /var/tmp

In my quest to get my read only root I encountered a problem with the dhclient trying to write to resolv.conf - unfortunately what the standard dhclient does is create a new /etc/resolv.conf.dhclient-new then uses mv to overwrite the existing resolv.conf.

I'm not a great fan of hacking debian scripts as 2 years down the line an upgrade will nuke said script and then I'm back to square one. My solution was to unlock the root fs for a short time during the dhclient magic.

I already had two existing scripts in /usr/local/sbin called lock and unlock which basically run:

lock:
mount -o remount,ro /
and unlock:
mount -o remount,rw /
Add some symlinks in /etc/dhcp/dhclient-enter-hooks.d/ and  /etc/dhcp/dhclient-exit-hooks.d/ as follows:
ln -s /usr/local/sbin/unlock /etc/dhcp/dhclient-enter-hooks.d/aunlock
ln -s /usr/local/sbin/unlock /etc/dhcp/dhclient-exit-hooks.d/zlock
I guessed that the dhclient script runs the hooks in alphabetical order and it seems to be the case, but I could not find any definitive proof for this.

Wednesday, 29 June 2011

QNAP debian temperature control

Tonight was a hot night and I started to get worried about the temperature of the hard drives. Luckily there is a program called hddtemp which can be used to monitor the temperature of the drives.

Qnap debian comes with a mostly finished deamon called qcontrol - which monitors the gpio and controls such things as the triggers for the power switch. Reading around qcontrol is not production ready - it seems to run a lua script, but I tried to do a few things and failed. Time to write a few scripts!

Note: I stuck both these scripts in /usr/local/sbin/ as they require root to be able to run
#!/bin/dash
#Where hddtemp is located
HDDTEMP=/usr/sbin/hddtemp
#List of harddrives
HDDLIST="/dev/sda /dev/sdb"
#Temperature for silent
TEMPSILENT=30
#Temperature for low
TEMPLOW=35
#Temperature for meduim
TEMPMEDIUM=40
#Temperature for fast
TEMPFAST=45
#Hysteriesis for temp
#Set the initial temperature to be 0
TEMPERATURE=0
#Loop through the hard drives and get the highest Temperature
for HDD in $HDDLIST
do
  #Test to see if the hard drive exists
  if [ ! -e $HDD ]; then
    echo $HDD" does not exist skipping"
    continue
  fi
  #Test the hdd
  TESTTEMP=$($HDDTEMP $HDD | awk 'BEGIN {FS=":";} {print $3;}' | awk 'BEGIN  {FS="°C";} {print $1;}')
  if [ $TESTTEMP -ge $TEMPERATURE ]; then
    TEMPERATURE=$(($TESTTEMP))
  fi
done
#Echo the highest temperature
echo $TEMPERATURE
#Kill exisiting qcontrol instances then start
killall qcontrol 2> /dev/null
rm /var/run/qcontrol.sock 2> /dev/null
/usr/sbin/qcontrol -d 1> /dev/null &
#Wait 1 sec for qcontrol to start
sleep 1
# Now alter the fan based on the temperature
if [ $TEMPERATURE -le $TEMPSILENT ]; then
  qcontrol fanspeed silence
elif [ $TEMPERATURE -le $TEMPLOW ]; then
  qcontrol fanspeed low
elif  [ $TEMPERATURE -le $TEMPMEDIUM ]; then
  qcontrol fanspeed medium
else
  qcontrol fanspeed fast
fi
exit 0
I called this fanspeed and created another script that called this one every 60 seconds called fancontrol.
#!/bin/sh
while true;
do
  #call the fanspeed control
  /usr/local/sbin/fanspeed 1> /dev/null
  #Sleep for a minute
  sleep 60
done

Headless panic - no ip address

The whole point of a two disk raid1 setup is that I should be able to lose one hard drive right?

With great confidence I removed the defective disk from my QNAP TS210 and ran the machine on only one hard drive.

Complete success - I could still access all the files and everything looked good!

Everything worked fine. So, time to fit the new hard drive.... or maybe not. In my excitement I forgot to power down the machine properly, and just pulled the power out of the back of the machine. I had yet to do the whole read only root magic and so the machine booted up with the dreaded file system errors, and fstab was still in the default remount=ro for the root partition so the machine failed to get a valid ip address from the router.

Argh!

I quickly removed the hard drive from the nas, put it into my usb sata drive adaptor on my desktop and ran fsck on the root drive there. After a bit of back and forth and reading the /var/log/daemon.log I could see that the dhclient was trying to get an ip address from the router but something was not quite right.

Onto the topic at hand - I quickly (i.e. asked my awesome partner to remind me syntax of awk) made a script to make sure that something like this should never happen again.

And here it is - I called it backupip and call it from the /etc/rc.local script:
#!/bin/bash


# Sleep for 100 seconds
sleep 100


# Parse ifconfig and see if we have a valid ip address
a=$(/sbin/ifconfig eth0 | grep "inet addr" | awk 'BEGIN {FS=":"}{print $2}' | awk '{print $1}')


# If we have no ip address give the nas this ipaddres
if [ -z $a ]
then
  echo "no ip address"
  ifconfig eth0 down
  ifconfig eth0 192.168.1.2 netmask 255.255.255.0 up
  route add default gw 192.168.1.1 eth0
fi
Not exactly the best script in the world - but it's simple and now I managed to ssh into my system and get to the bottom of the problem. (which was a failure of my separate /var drive starting up soon enough.)

Note: I'm pretty sure I can get the script to only use awk but I was in a bit of a rush!

Tuesday, 28 June 2011

Broken hard drive.

As luck would have it, half way through configuring my debian install I started to get a bit suspicious that one of the hard drives was making some odd noises.A quick dmesg gave the following result:
[ 1102.483303] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[ 1102.490402] ata2.00: failed command: FLUSH CACHE EXT
[ 1102.495443] ata2.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
[ 1102.495453]          res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 1102.509100] ata2.00: status: { DRDY }
[ 1102.512800] ata2: hard resetting link
[ 1105.963276] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl F300)
[ 1109.663376] ata2.00: configured for UDMA/133
[ 1109.667680] ata2.00: device reported invalid CHS sector 0
[ 1109.673125] ata2: EH complete
[ 1185.483296] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[ 1185.490387] ata2.00: failed command: FLUSH CACHE EXT
[ 1185.495428] ata2.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
I quickly installed smartctl to check what was up and got the following results for the second hard drive:
smartctl 5.40 2010-07-12 r3124 [armv5tel-unknown-linux-gnueabi] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Device Model:     ST2000DL003-9VT166
Serial Number:    5YD2K2G9
Firmware Version: CC32
User Capacity:    2,000,398,934,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Thu Jun 23 10:56:00 2011 BST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                 ( 623) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 255) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x30b7) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   100   100   006    Pre-fail  Always       -       1996128
  3 Spin_Up_Time            0x0003   093   093   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       26
  5 Reallocated_Sector_Ct   0x0033   092   092   036    Pre-fail  Always       -       5696
  7 Seek_Error_Rate         0x000f   100   253   030    Pre-fail  Always       -       358966
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       129
10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       19
183 Runtime_Bad_Block       0x0032   097   097   000    Old_age   Always       -       3
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   099   099   000    Old_age   Always       -       1
188 Command_Timeout         0x0032   099   098   000    Old_age   Always       -       64425492496
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   064   061   045    Old_age   Always       -       36 (Lifetime Min/Max 31/36)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       18
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       26
194 Temperature_Celsius     0x0022   036   040   000    Old_age   Always       -       36 (0 22 0 0)
195 Hardware_ECC_Recovered  0x001a   100   100   000    Old_age   Always       -       1996128
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       39
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       39
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       170926813478951
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       2046017289
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       3928948219

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Lots of Old_age notices from a hard drive fresh out of the box. Time to return it!

Monday, 27 June 2011

Installing debian on my QNAP TS-210

I bought two QNAP TS-210 and four 2TB hard drives as a split site backup strategy for our important files. I knew that it was possible to install debian on the qnaps but wanted to try the default firmware for a while.

I was a great fan of the default firmware for a few days but started to get annoyed with little things:

  • I couldn't tweak the nfs settings to my liking.
  • The rsync backup setup was pretty basic and I wanted to use something more intelligent like unison 
It is possible to tweak things using some clever scripts that replaced config files, but then you start being unable to use the web based system to configure your system so in the end I decided to go with the more familiar debian experience.

I started out my linux career with debian and after a happy relationship with ubuntu I just kept on returning to debian for my server and now I have 4 debian computers and plan to convert the last ubuntu box over pretty soon. I'm a big fan of debian and I'm actually used to the location of files and the ability to apt-get anything out there so I thought I would give installing Debian a go.

I followed the guide at http://www.cyrius.com/debian/kirkwood/qnap/ and was up and running in no time.

I was planning on trying out a read only root filesystem with a separate /var partition so went with the following partitions:

3GB - ext3 root
1GB - ext3 var
512MB - swap
1.8TB - data mounted on /home.

This is what my 'parted -l' looks like:
Partition Table: gpt
Number Start End Size File system Name Flags
1 17.4kB 3000MB 3000MB Root raid
2 3000MB 4000MB 1000MB Var raid
3 4000MB 4512MB 512MB Swap raid
4 4512MB 2000GB 1996GB Data raid
When setting up the machine for the first time I recommend that you do not set up the 1.8TB data drive otherwise you will have to wait 4 hours for the 1.8TB /home partition to be built. Best to finish the debian install then create the big partition. The resync can happen while you are not sshed into the system.