Oct 27

Hello Large-Scale Production Engineers!

Here’s the link for summarize_jsh

to test, try:

for i in foo bar baz; do for j in `seq 9 13`; do for m in larry moe curley ; do  echo ${i}-${j}: $m; done; done; done | summarize_jsh

Output looks like:


host summary [count 15]:

written by admin

Aug 01

# less the youngest file in the dir that is not a dot file.
function lless () {
less `ls -Art1|tail -1`

written by admin

Aug 01

Hyperthreading is back in Intel Processors. We use MatLab which apparently doesn’t like to see 16 cores when there are really only 8.

Here’s a script to flip procs 8-and-up  on and off. Change the perl ‘>= 8′   part to set it for your environment.

Thanks to http://dag.wieers.com/blog/is-hyper-threading-enabled-on-a-linux-system for

if [ ! -z $1 ]
 # fancy bash block method of doing if else;
 $(echo $1 | grep -qi "^on$") && {
 echo turning CPU from 8 up ON ;

$(echo $1 | grep -qi "^off$") &&  {
 echo turning CPU from 8 up OFF ;

for i in $(find /sys/devices/system/node -name "cpu[0-9]*" |perl -ne  '($i)= m{u(\d+)$}xms; if ($i >= 8){print}; undef $i;' )
 if [ -d $i  ]
 echo $i
 echo "$CPU_VALUE"  > $i/online 2> /dev/null
 echo $i is not a dir


 echo "must give an argument of 'on' or 'off'"

echo Running processors
cat /proc/cpuinfo|egrep '^processor'

written by admin

Mar 02
## gold txmouse setup for fixing the hightlight-paste annoyance in vnc and vmware

I‘m a huge fan of the Windows app TXMouse

By default it avoids a set of apps it doesn’t interact well with. As of early 2010 it does not work correctly with an X11 session running on RealVNC. The symptoms are:

- double-pastes in xterms

- anything mouse highlighted in emacs immediately pastes into the buffer

The TXMouse site has a way to figure out what is going on, here’s what I found for newer RealVNC viewer:


written by admin

Sep 17

I was talking with some friends about automating MySQL backups – here’s a script to backup my wordpress mysql database and mail it to a gmail account for safekeeping. It cleans up after itself too.

I run it from cron every 4th day:

0 4 */4 * * /usr/local/bin/wordpress_backup.bash



# This requires 'nail' which takes attachments from the command line. Available
# with "sudo yum install -y nail" 


# if the dir does not exist create it
if [ ! -d $DUMP_DIR  ]
 mkdir -p $DUMP_DIR

# generate a file name with todays date
DUMP_FILE=$DUMP_DIR/${BASE_FILE_NAME}_`date +%Y-%m-%d`.sql.bz2

# run the mysqldump, pipe it through bzip2 and redirect it to the filename
mysqldump --add-drop-table -h localhost -u nnnnnn -pxxxxxx wp_dbase |
 bzip2  -c > ${DUMP_FILE}

# generate an email body, pipe to nail with the bz2 dump attached
(echo; echo; echo "#########"; echo "###" ;date +%Y-%m-%d; echo "autogenerated blog dumpfile $DUMP_FILE" ) |
nail -a ${DUMP_FILE} -s "blog dumpfile $DUMP_FILE" MAILID+blogbackup@gmail.com

if [ ! -z ${BASE_FILE_NAME} -o ! -d $DUMP_DIR ]
 cd ${DUMP_DIR}
 # look for files that match the dump files that are older than 3 days
 # and remove them
 find $DUMP_DIR -name "${BASE_FILE_NAME}*" -mtime +3 |xargs -i rm -v {}
 echo "not doing a find/delete"

written by admin

Aug 01

Whenever I start a new gig I have to key all the boxes – it’s tedious at best. Here’s a bash function to put a key on a far box. I always make sure I have an “eval `ssh-agent`” and “ssh-add” in my shell before running ‘keybox’.

keybox [user@]box
keybox root@hammerforge

Uses ‘mktemp’ which might not be on every OS. Assumes a POSIXy shell at both ends. Tested with bash and zsh.

# Thu May 28 13:57:25 2009
# will key a box for you
# use like "keybox foobar" OR "keybox user@foobar" it will take
# ~/.ssh/id_dsa.pub (change for your keytype) and write it to the far
# $USER/.ssh/authorized_keys file. Assumes you are using ssh-agent, ssh-add
# for passwdless logins.
#-Tony Thu May 28 16:59:22 2009

function keybox () {

# correct for your keytype

# see is we got user@box, key the 'user' (ie root@hammerforge will
# have your key added to ~root/.ssh/authorized_Keys)
if (echo $1|grep -q @)
USER_TO_KEY=`echo $1|cut -f1 -d'@'`
MACHINE_TO_KEY=`echo $1|cut -f2 -d'@'`

if [ -f ${SSH_PUB_KEY} ]
echo found ${SSH_PUB_KEY}
echo "did not file a public key, generating a new dsa key"
ssh-keygen -t dsa
echo "continuing to keybox $MACHINE_TO_KEY"


TEMP_FILE=`mktemp -u`
# use some ssh options so it doesn't complain about known_hosts.
ssh -o StrictHostKeyChecking=no ${USER_TO_KEY}@${MACHINE_TO_KEY} "cp ~${USER_TO_KEY}/${AUTH_KEYS} ~${USER_TO_KEY}/.ssh/hold_authorized_keys ;echo \"$LOCAL_KEY\" > $TEMP_FILE ; cat $TEMP_FILE >> ~${USER_TO_KEY}/${AUTH_KEYS}"
# do another ssh to confirm you can get back to the box
# Batchmode causes ssh it quit if the autologin doesn't work
ssh -o BatchMode=yes ${USER_TO_KEY}@${MACHINE_TO_KEY} "hostname; date ; rm -v $TEMP_FILE"


written by admin

Apr 14

I love the idea of Netomata! I haven’t used it yet, but have often lamented the lack of structure around networking configs. This is not just a great idea for the implementation level, but also for management. If you run your shop with this, a director/manager can learn the tool and get visibility into the entire networking infrastructure rather than having to trace through the decentralized networking equipment configs.

It’s also makes the networking piece of Disaster Recovery significantly easier.

The benefits and why pages are great summaries of why to use centrally generated configs for all machine management. One of the points is “Providing a limited kind of process documentation.” This massivly sells the process short. It would be better to say “Provides unequivocally and 100% repeatable process documentation.”

If you’ve got experience with it, please post a trip report.


written by admin

Mar 26

A necessary piece of operations is riding herd on home grown applications and projects from the corporate wilds. These things come to you late in their lifecycle with little to say about how their technology or composition. Often the expectation is that you’ll just take them over and “make them work.” Sometimes that’s doable, but most time there are support limitations.

Here’s the interview and explanation process I use to work with groups outside of Ops to set realistic expectations and about what can and and can’t do for them. It is step 0 of a project plan work. I like to avoid surprised and clearly set expectations about Operations can and can’t do.

How to have Ops take ownership for systems or processes or programs:

  1. What is the business justification for this process?
  2. Who sponsors the process (outside of operations)?
  3. When will the process be turned over to operations?
  4. How will your group know the process is in place and being monitored?
  5. What are Operations obligations and responsibilities?
  6. What are the sponsoring groups obligations and responsibilities?

System category:

  1. Requires full/half/quarter time staff member.
  2. Existing process needs monitoring and response plan.
  3. Trivial process that doesn’t require monitoring.
  4. Trivial process that needs monitoring.
  5. Ops can monitor but not trouble shoot.
  6. Ops can troubleshoot at level 1/2/3 but cannot fix.

Why would ops decline to accept your system, process or program:

  1. There may be no way to support the process (for instance it involves on-going manual work – in this case the process likely needs to start at Engineering).
  2. It will incur resource costs beyond reasonable levels (i.e. network usage beyond our current capacity, etc.)
  3. The sponsoring group does not provide ongoing budgetary support.

What you should expect from us.

  1. Integrity and discipline in all our work.
  2. A consulting approach to putting your process into production. This means being an organization that is committed to your success and wants to put your work into production.
  3. A “closed loop” system that has clear responsibility, reporting, troubleshooting and escalation procedures.

written by admin

Jan 20

I setup dhcpd and tfpt just infrequently enough to forget the details. I’m putting my gottchas here so I don’t forget them.

syslinux package ‘pxelinux’:
pxelinux loads and gets the right IP, then it fails trying to
getting the error “tftp server does not support tsize option”


in file /etc/dhcpd.conf:

# absolutly critical to have the next-server line for tftp booting
# when you get "tftp server does not support tsize option" error,
#it's because your missing the config line, Double check with:
#          grep next-server     /etc/dhcpd.conf
#    - Tony 10/17/08


1] for setting up tftpd you have to make sure there are not entries like
this in /etc/hosts file      joust.famemobile.com joust

if so you have to change them to this.   joust.famemobile.com joust

2] Using tcpdump for tftp trouble shooting

The fact that loading pxelinux.0 succeeds made me think everything else should work.

The pxelinux.0 loads fine, but the config file ‘pxelinux.cfg/01-00-0c-29-c4-b0-5a’ does not.

05:27:20.882329 IP (tos 0×0, ttl 20, id 2, offset 0, flags [none], proto: UDP (17), length: 55) > [udp sum ok] 27 RRQ “pxelinux.0″ octet tsize 0
05:27:20.893400 IP (tos 0×0, ttl 20, id 4, offset 0, flags [none], proto: UDP (17), length: 60) > [udp sum ok] 32 RRQ “pxelinux.0″ octet blksize 1456
05:27:20.953322 IP (tos 0×0, ttl 20, id 29, offset 0, flags [none], proto: UDP (17), length: 91) > 63 RRQ “pxelinux.cfg/01-00-0c-29-c4-b0-5a” octet tsize 0 blks
… stuff cut out…
05:27:20.972168 IP (tos 0×0, ttl 18, id 44911, offset 0, flags [none], proto: UDP (17), length: 54) > [udp sum ok] 26 ERROR tftp-err-#8 ” tsize option required”

The “” is the indicator there is something wrong.

written by admin

Jan 19

I use blogger and host the files on my server, after I edit a post it has to sftp the files so they appear here. This is the process for adding them.

Adding the Blogger sftp servers to iptables.

Blogger.com lists their outbound ip’s here. (It was current Jan 19, 2009)

# always check the addresses are correct and the link above.
for i in  64.233.178/28
             echo iptables -A INPUT -i eth0 -s $i -p tcp --dport ssh -j ACCEPT
### Output
iptables -A INPUT -i eth0 -s -p tcp --dport ssh -j ACCEPT
iptables -A INPUT -i eth0 -s -p tcp --dport ssh -j ACCEPT
iptables -A INPUT -i eth0 -s -p tcp --dport ssh -j ACCEPT
iptables -A INPUT -i eth0 -s 64.233.178/28 -p tcp --dport ssh -j ACCEPT

######## Other notes
I cheated and used ipcalc to the get the subnet calculations:

  ipcalc -

written by admin