General Commands


Create files with a number sequence

$for i in `seq -w 01 31`; do touch log/tomcat_access_2011-08-$i.log;done

-w pads out with zeros. I've found in some systems that -w doesn't and -f (print format) is needed. I.e. `seq -f "%03g" 001 009

Swap out file extensions

for i in *.wma; do mv "$i" "$(basename "$i" .wma).mp3"; done

add in a time extension
for i in *.bz2; do mv "$i" "$(basename "$i" .bz2).$(date +%s).bz2"; done

total the amount of memory a daemon is using

If there are a number of the processes started (e.g. apache), calculating the total can be done by using the following command:
unset SUM;for i in `pstree -p apache|cut -d \) -f1 |grep -o -E '[0-9].{0,}'`;do SUM=$((SUM+`pmap -x $i|grep total|awk '{print $3}'`));done;echo $SUM"KB"

This finds all processes running as apache takes the total memory usage for each and totals it all together. [NOTE: This includes shared memory]

To get a closer idea of actual memory:
unset SUM;for i in `ps -eAo rss,command|grep httpd|grep -v grep|awk '{print $1}'`;do SUM=$((SUM+$i));done;echo $SUM"KB"

x2vnc –direction ip address:screen number - Connect to another x11 server. Direction is where the screen will be and is north, east etc. Screen number is 0 unless remote machine has more than 1.

top – shows repeating utilisation including load. top -Hc will show threads; and press z on the console to show in colour

ps list running process details. ps -eLf | less -+S to show threads and complete commands, then wrap lines

Load is misleading if the box is multi-proccessor and it is better to use mpstat 5 5 or prstat (Solaris) to see if the cpu is really maxed

killall program name - as it says. Program name must be as it is stated in ”ps”

cat file | mail email_address - prints file contents to standard output, then pipes to email address

echo "some text" > filename - overwrites file contents with some text

echo "some text" >> filename - appends some text to filename

>filename - empties the file contents and keeps redirect links

zcat zip_file - This command concatenates a files that is gzipped

which file - like locate but doesn’t use a database to lookup. It uses the path environment variable. E.g. It shows which ps is being used if simply typing “$ ps” (there may be more than one ps on the system).

Whereis filename – like locate but only uses the path

netstat –an | grep port_number - shows which protocol uses this port.

df and du give details of disk usage. df an overall summary and du a pef file or directory total.

du -sh /{\.*,*} - accounts for shell regex issues and checks hidden directories. a summarises files and directory sizes in human readable form (without recursing directories) in Solaris.

du –d1 –kh - same as above but for FreeBSD (–d1 is depth to 1 directory)

why du and df don't match

dd– RAW data copy

rc-update<init.script> add<runlevel> - Gentoo command to add a process to boot

chkconfig –list --Redhat (and derivatives)

chkconfig --add ntpd

chkconfig --level 3 ntpd on

update-rc.d --Debian (and derivatives)

mount–o <file> <directory>

awk- programming language. To grab certain parts from stdout, separated by commas, typical use: awk -F \, '{print $4}'

cut -c1-5– cuts characters from the end or beginning of a line.

zgrep greps a zipped file

nmap -sP more detailed broadcast ping

cat locale_filename | gzip - | ssh remote_ip 'cat - > remote_filename' copies and zips to a remote machine without using any local diskspace.


for lots of files to a remote tar
tar cvzf - * | ssh remote_ip " cat - > remote_tar_name"
or lots of files to lots of files
tar czf - * | ssh remote_ip " tar xvf - -C remote_dir "
should work for any other unix archivers eg cpio, as long as they can accept

stdin and stdout
==grab from a remote machine:==

ssh remote_machine ‘cd remote/dir; tar -cf – some.tar.gz’ | tar -xf -

dd if=/dev/urandom of=20M_file bs=1024 count=20000– creates a data file called 20M_file which is 20MB of random binary data

sockstat – show current connections and relating process. Tcpkill can be used to drop these connections.

cat /proc/mdstat– show raid information.

Personalities : raid0raid1

md13 : active raid1 sdac11sdo10

71553408 blocks 2/2UU<< a missing U here shows dead disk

dmidecode – gives hardware information

vmstat -- report virtual memory statistics

lsof/ fstat - list open files

stat <filename> - gives verbose details on the files open status

iostat -- report I/O statistics

systat -[iostat|netstat|vmstat|icmp]– graphs and orders relevant output

sysctl -a | grep cpu– displays cpu specific hardware info
script <file>- logs everything done in a terminal to the file.

/usr/sbin/grub-md5-crypt– creates and md5 crypt

dumpfs and tunefs view and change filesystem parameters.

sudo su - root -c "crontab -l"- run crontab as root

pkill <expression>– kill processes based on pgrep. BSD requires -f

mount -t iso9660 -o loop <Image_File> <Mount_Point>- mounts a cdrom

Networking commands

Networking Commands

time tcpdump -s0 -i interface host hostname -w filename.dmp - create a raw dump for Wireshark

Using find

case insensitive filenames

find . –iname "*csid*"

exec Vs xargs

an option to run a command on the results is -exec. As in
find . –iname "*csid*" –exec grep “string” {} \; -print

(The space between the braces and backslash is important). However it is often more efficient to pipe the results into an xargs command. I.e.
find . –iname "*csid*" | xargs grep “string”

To pass values in the middle of the xargs induced command use {}'s. E.g.
egrep -lm1 "(amazon|google|apple|)" *| xargs -I '{}' mv '{}' ../cur/

searching for files with REGEX

find . -regextype posix-extended -regex ".+(originserver\\.log\\.|access_)[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}(\\.log)?$" -mtime +5 -type f -exec mv {} old_logs/ \; && gzip old_logs/*

Searching for files modified between 2 times

find . -type f -newermt "2013-01-10 11:45:00" ! -newermt "2013-01-10 12:45:00" -printf "%t %h/%f\n"

The -printf format prints the modification timestamp, directory path and filename

Using Tail

tail -c 0 --retry --follow=name /var/log/hue/jobsubd.log --pid 21209

-c                     Output the last 'n' bytes.
--retry                Keep trying to open a file even if it is inaccessible when tail starts or if it becomes  inaccessible later; useful when following by name,
                        i.e., with --follow=name
--follow[={name|descriptor}]  Output appended data as the file grows;
-f, --follow, and --follow=descriptor are equivalent
--pid=PID           With -f, terminate after process ID, PID dies

Using grep

useful flags

-I ignore binary files
-r recurse directories
-Dskip skip devices

Show only the part of a matching line that matches PATTERN

grep -o -E '$PATTERN.{0,5}'

To search for and print UP TO 5 characters after PATTERN
grep -o -E '.{5}$PATTERN'

To search for and print ONLY 5 characters before PATTERN

$ pgrep -lf Dprogram.name=run.sh
5961 /usr/java/jdk1.5.0_17/bin/java -Dprogram.name=run.sh -server -Xms256m -XX:PermSize=128m -XX:MaxPermSize=256m -Xmx3072m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -verbosegc -XX:+PrintGCDetails -Xloggc:/tmp/gc.log -XX:+PrintGCTimeStamps -Djavax.management.builder.initial=org.jboss.system.server.jmx.MBeanServerBuilderImpl -Djboss.platform.mbeanserver -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=2765 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dfile.encoding=UTF-8 -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/crash -Djava.endorsed.dirs=/usr/lib/jboss/lib/endorsed -classpath /usr/lib/jboss/bin/run.jar:/usr/java/jdk1.5.0_17/lib/tools.jar org.jboss.Main -Dcom.tc.productkey.path=/usr/lib/jboss/server/musicstation/conf/product.key -Dmusicstation.jms.state=enabled -Dsolr.master.enable=false -Dsolr.slave.enable=false -Dsolr.master.host=vm-pat4-app03.ribob01.net -Djboss.server.log.dir=/mnt/log/vm_pat4_app01/jboss_server_logs -Djava.net.preferIPv4Stack=true -Djboss.partition.name:DefaultPartition=pat4 -Djboss.partition.udpGroup: -Djboss.partition.udpGroup: -Djboss.partition.udpGroup: -Dappcontrol.block.incoming.requests=true -Dappcontrol.server.group.name=pat4 -Dapp.control.server.list=vm-pat4-app01.ribob01.net vm-pat4-app02.ribob01.net vm-pat4-app03.ribob01.net -Dapp.control.controller.list=vm-pat4-app01.ribob01.net vm-pat4-app02.ribob01.net vm-pat4-app03.ribob01.net -Dapp.control.jms.list=vm-pat4-app01.ribob01.net -b -c musicstation

$ pgrep -lf Dprogram.name=run.sh| grep -o -E 'Xmx.{5}' 

Look ahead regular expression to exclude some results

grep -P "--regex=^(?=.*${SEARCH_STRING})((?!${EXCLUDE_LIST}).)*$" ${SEARCH_LOG})

Split a file into multiple files and rename

split -l 91747 full.txt guvera-sniffed-requests-thread- --additional-suffix=.txt -d --suffix-length=3
for FILE in `ls`; do mv $FILE `echo $FILE | sed -e 's:^guvera-sniffed-requests-thread-0*:guvera-sniffed-requests-thread-:'`; done