SED

NOTE
different versions of SED have different REGEX rules and commands

Contents

NAME

sed - stream editor for filtering and transforming text

SYNOPSIS

sed [OPTION]... {script-only-if-no-other-script} [input-file]...
       -e combine commands
       -n don't print output
       -i replace entries in the input file on the fly
       -f <file> run <file> as a script. "#!/bin/sed -f" can be added to the start of 
       the file to allow it to run on its own

Commands and Special Characters

s substitute. As in ‘s/this/that’
w write output to. As in ‘s/this/that/w outfile’
p print out changes
d delete matched condition
\1 (\2...) first (and second etc) caught condition using \(\)
& the condition being searched for
q quit sed after finding match
a append a given line, as a whole line after the search pattern. I.e. ‘/search/ a add this line next’
i same as a but insert before pattern
c same as a and i but change the swap the line out
y change to case
l prints extra characters for debug
/ field delimiter. This can be changed on the fly by simply substituting for something else. E.g. change to | when working with paths

Known variations

The MacOS version requires that when using -i, a suffix is specified, even if it is blank -i ''. For compatibility, it’s a good idea to use something like -i .swp, then deleting the file if it’s successful

Regular expressions

When using REGEX in checking strings in sed, it’s important to note that some special characters will need escaping. For example:

$sed -n 's/^build = .\([a-z]\{3,4\}\)[0-9]\{1,\}.*/\1/p' file.txt 
dev

Alternatively, extended REGEX can be enabled using -E. So the above expression then becomes:

$sed -En 's/^build = .([a-z]{3,4})[0-9]{1,}.*/\1/p' file.txt
dev

Also worth noting is that boolean OR and capture groups are the same. For example:

$sed -En 's/^build = .((dev|prod))[0-9]{1,}.*/\1/p' file.txt
dev

is the same as:

$sed -En 's/^build = .(dev|prod)[0-9]{1,}.*/\1/p' file.txt 
dev

Examples

CR/LF removal/addition

unix2dos:  sed -i "s/$/\r/" textfile
dos2unix:  sed -i "s/\r$//" textfile

removing -i sends to stdout instead of editing the file.

changing ERROR in log file

sed -i "s/ERROR/EDITED/g" /usr/lib/jboss/server/javaApp/log/server.log

Replace underscores with Spaces in filenames

for i in `ls` ; do j="`echo $i | sed 's/_/\ /g'`"; mv $i "$j";done

Select only numbers between square braces

sed "s/.*\[\([0-9]*\)\].*/\1/" /var/log/messages

The same but also print all characters before the word “primary”. In this case, the datetime

sed "s/\(.*\)primary.*\[\([0-9]*\).*/\1 - \2/" /var/log/messages

If there are multiple records of the same search criteria but an unknown number of times, “grep -o” is better brought into the equation

switch two words around

sed 's/\([a-z]*\) \([a-z]*\)/\2 \1/'

detect duplicated words, you can use

sed -n '/\([a-z][a-z]*\) \1/p'

Eliminate them

sed 's/\([a-z]*\) \1/\1/'

Print PIDs of all sshd entries in /var/log/messages (and using POSIX)

sed 's/.*\(sshd\).*\[\([[:digit:]]\{1,\}\).*/\1 - \2/' /var/log/messages

leave the first CHANGE word alone alone, but change the second, third, etc. to DELETED, use /2g:

sed 's/\<CHANGE\>/DELETED /2g'

add a colon after the 80th character in each line, you could type:

sed 's/./&:/80'

Write changed lines to a new file in stead of STDOUT (-n surpresses, /w writes)

sed -n 's/^[0-9]*[02468] /&/w filename

Working with line numbers (line 3)

replace a line using line number

sed -ie '43 c some_string//' /path/to/somefile

specify a range 3,53 or 3,$ to the end of file

sed -n '31,35 p' /path/to/file

remove all lines except between the two keywords:

sed -e '1,/start_keyword/ s/#.*//' -e '/stop_keyword/,$ s/#.*//'

Print lines between 2 dates/times (delete the others)

sed  -e '1,/Apr 26 17:45/ d' -e '/Apr 26 19:00/,$ d'

Substitute a whole matched line for just a specific part

(I.e. “\(a-zA-Z0-9\+\)”)

sed -n '/searchString/ { s/^.*someBitOfText\([a-zA-Z0-9]\+\)someOtherText.*$/\1/; p };' my.log

print expression between 2 tags

sed -n 's/.*<someTag>//;s/<\/someTag>.*//p'

OPTIMIZING FOR SPEED

If execution speed needs to be increased (due to
large input files or slow processors or hard disks), substitution will
be executed more quickly if the “find” expression is specified before
giving the “s/.../.../” instruction. Thus:

sed 's/foo/bar/g' filename         # standard replace command
sed '/foo/ s/foo/bar/g' filename   # executes more quickly
sed '/foo/ s//bar/g' filename      # shorthand sed syntax