Segmentation Fault: Core dumped..;-)

HOWTO: Performing global search and replace across files on your Linux machine

Posted : July 6, 2004 at 1:40 pm [America/Los_Angeles]

Was assigned a simple task this morning:

...
<a href="http://indrayam.com">Back to Home</a>
</body>
</html>

There are lots of files in a directory that have a pattern similar to the one shown above. My assignment was to replace the URL in the href entries with a new one.

Sounds like a fairly common and simple problem. However, I soon realized that I did not have a canned utility or shell script that I could quickly use. After some googling around, here’s what I came up with:

Bash/Sed version:

#!/bin/bash

for i in $( ls *.html ); do
   sed "s/\"http://indrayam.com\"/\"http://indrayam.com/it/\"/g" $i > $i.new
   mv -f $i.new $i
done

If you want to peruse through content a few levels deep, try this:

#!/bin/bash

for i in $( find . -name '*.html' ); do
   sed "s/\"http://indrayam.com\"/\"http://indrayam.com/it/\"/g" $i > $i.new
   mv -f $i.new $i
done

Here are a few pointers, in case you do decide on using the script:

1. For starters, restrict the $(ls *.<file-extension>) section of the shell script to a single file by replacing it with $(ls <filename>). In other words, test the script out on a single file first.

2. Always run the “double-whammy” test - Run the shell script once and then run the script again without any modifications!. If the targetted file looks ok after this double execution, you have wisely selected your initial-pattern and new-pattern in sed’s s/<initial-pattern>/<new-pattern>/g expression.

   For example, let’s say the sed expression in the shell script shown above looked something like this instead:

   sed "s/http://indrayam.com/http://indrayam.com/it//g" $i > $i.new
   

   Do you see a problem here? Since our initial-pattern is now http://indrayam.com (as opposed to “http://indrayam.com” in the shell script above), executing the script against the same file twice will result in href looking something like:

   <a href="http://indrayam.com/it//it/">Back to Home</a>
   

   Of course, this double-whammy problem will bite you primarily if your initial-pattern is a subset of the new-pattern

3. Only after you’ve tested for these two things should you consider opening up the spigot by resetting $(ls <filename>) to $(ls *.<file-extension>)

#!/bin/bash

for i in $( ls *.html ); do
   cp $i $i.bak
   sed "s/\"http://indrayam.com\"/\"http://indrayam.com/it/\"/g" $i > $i.new
   mv -f $i.new $i
done

Perl One-liner:

If you like Perl, here’s a really, really neat one-liner:

(unix prompt)>perl -i.bak -p -e 's#"http://indrayam.com"#"http://indrayam.com/it/"#ig' *.html

If you want to peruse through content a few levels deep, try this:

(unix prompt)>perl -i.bak -p -e 's#"http://indrayam.com"#"http://indrayam.com/it/"#ig' {.,*,*/*,*/*/*}/*.html

Nice!

For more information, check out this site Thanks to MS (see comments below) for pointing this out.

- Anand

Category: How do I?