Substituting file contents in place with BASH and sed

Substituting in place with sed

# Usage : sed_sub_in_place search replace file 
# ^[\].*$ escaped in search (Basic Regular Expressions)
# / and & escaped in replace ( escape BRE special characters: ^[\].*$ with \ if present)
# makes copy of the original with the name file.tmp

sed_sub_in_place() {
  s=${1}
  s=${s//\\/\\\\}
  s=${s//\//\\\/}
  s=${s//\^/\\\^}
  s=${s//\[/\\\[}
  s=${s//\]/\\\]}
  s=${s//\./\\\.}
  s=${s//\*/\\\*}
  s=${s//\$/\\\$}
  s_w=${2//\//\\\/}
  s_w=${s_w//\&/\\\&}
  f="$3"
  s_e="s/$s/$s_w/g"
  sed --in-place=.tmp "$s_e" "$f"  
}

Run it:

# replacing < with &lt; in data.txt
sed_sub_in_place '<' '&lt;' data.txt

# replacing > with &gt; in data.txt
sed_sub_in_place '>' '&gt;' data.txt

# replacing " with &quot; in data.txt
sed_sub_in_place '"' '&quot;' data.txt 

# replacing ' with &apos; in data.txt
sed_sub_in_place "'" '&apos;' data.txt

# replacing '[div]' with '<div>' in data.txt
sed_sub_in_place '[div]' '<div>' data.txt

# replacing '[/div]' with '</div>' in data.txt
sed_sub_in_place '[/div]' '</div>' data.txt

# replacing SetEnvIfNoCase User-Agent (&lt;|<|%3C)(%20|s)*script(s|%20|>|&gt;|%3E) keep_out with REPLACED in data.txt

sed_sub_in_place 'SetEnvIfNoCase User-Agent (&lt;|<|%3C)(%20|s)*script(s|%20|>|&gt;|%3E) keep_out' 'REPLACED' data.txt

Substituting with BASH

A function below, bash_sub_in_place, escapes \*[] and ?. Escaping in replace is not necessary. Also, the function reads in whole file at one go, so it is not suitable for making substitutions in very large files..

# Usage: bash_sub_in_place search replace file
# \*[]? escaped in search, due to their special meaning for BASH
# nothing escaped in replace
# makes copy with the name file.timestamp.nanoseconds.tmp

bash_sub_in_place() {
  local s=${1//\\/\\\\}
  s=${s//\*/\\\*}
  s=${s//\[/\\\[}
  s=${s//\]/\\\]}
  s=${s//\?/\\\?}
  local s_w="$2"
  local f="$3"
  local d=$(date +"%s.%N")
  cp "$f" "$f.$d.tmp"
  local f_c=$(< $f )
  local fcr=${f_c//${s}/${s_w}}
  echo "$fcr" > "$f"
}

This function works in the same manner like sed_sub_in_place. Substitute function names and run examples above if in doubt.

More complex substitutions

Both functions can be used in more sophisticated manner. One can, for example, make them substitute special characters in HTML for their predefined entities and back.

Here are examples:

# sed_htmlize takes one argument, file to make substitutions in
# substitutes &,<,>,",' for HTML character entities
sed_htmlize(){
  sed_sub_in_place '&' '&amp;' $1
  sed_sub_in_place '<' '&lt;' $1
  sed_sub_in_place '>' '&gt;' $1
  sed_sub_in_place '"' '&quot;' $1
  sed_sub_in_place "'" '&apos;' $1
}

# sed_unhtmlize takes one argument, file to make substitutions in
# substitutes HTML predefined entities for &,<,>,",'
sed_unhtmlize(){
  sed_sub_in_place '&amp;' '&' $1
  sed_sub_in_place '&lt;' '<' $1
  sed_sub_in_place '&gt;' '>' $1
  sed_sub_in_place '&quot;' '"'  $1
  sed_sub_in_place '&apos;' "'" $1
} 

As for use:

sed_htmlize data.txt

to turn &,<,>,",' into HTML predefined entities, and

sed_unhtmlize() data.txt

to get the original version of data.txt file back.

Safety of substituting in place

In short, substituting in place is neither safe nor recommended in most of cases. For the two functions presented here, sed_sub_in_place and bash_sub_in_place, make copy of the original file contents in file.timestamp.nanoseconds.tmp file. Make sure you check that your substitutions produce desired effects before deleting temporary files. Even better, work on copies of original files.