Replace a word with multiple lines using sed?

Question

I'm working on a bash-script that has to prepare an E-Mail for being sent to a user.

It aggregates some data, which ends up being multiple lines of stuff. For the example stored in $DATA.

Now, after a bit of stfw I found a few things like sed -ei "s/_data_/${DATA}/g" mail.tpl and also sed replace with variable with multiple lines. None of them work.

Now the question is, how do I get sed to replace something with multiple lines of text?

(Alternatives to sed are also welcome!)

@ZsoltBotykai You dont say? :D I even mentioned it in my question... — Cobra_Fast, Apr 11 '12 at 14:07
As an aside, don't use uppercase for your private shell variables. — tripleee, Mar 11 '18 at 12:23
Because that's how you avoid inadvertent clashes with reserved variables, which are uppercase. There's no consensus on what *exactly* this means but your script's private variables are definitely not "system variables" as intended in this clause in POSIX. See https://stackoverflow.com/q/673055/874188 — tripleee, Mar 11 '18 at 15:54
@Cobra_Fast any chance you could consider accepting a different answer? — tripleee, Mar 11 '18 at 15:55

score 31 · Answer 1 · edited Apr 02 '19 at 18:33

31

You can do this with AWK using variable substitution. We can set a variable in AWK using -v, and then use AWK's gsub function to substitute all occurrences of a regular expression with that variable.

For example, if the file test has the following contents ...

foo
bar
blah _data_and_data_
foo
_data_ foobar _data_ again

... and the Bash variable $DATA is ...

... then awk -v r=$DATA '{gsub(/_data_/,r)}1' test replaces all occurrences of the regular expression _data_ in the file test with the contents of $DATA, resulting in the following:

foo
bar
blah 1
2
3
4
5and1
2
3
4
5
foo
1
2
3
4
5 foobar 1
2
3
4
5 again

edited Apr 02 '19 at 18:33

Jivan Pal

182
10

answered Apr 11 '12 at 14:11

Kent

189,393
32
233
301

6

Oh my god, this WORKS! I wish I could +1 you several more times, you saved me man! – Camilo Martin Dec 21 '13 at 00:11
4

Hey Kent, this is an old one, but you seem to be missing quotes around $DATA. confirm? – glenn jackman Feb 13 '14 at 01:53
2

should add quotes indeed: awk -v r="$DATA" '{gsub(/_data_/,r)}1' test – Alec Istomin Sep 06 '17 at 18:39
3

Apparently some (?) Awks will dislike this and say `"newline in string"` (though the answer without double quotes around `DATA` will fail in different ways). – tripleee Mar 11 '18 at 12:45
a short explanation for awk-noobs would be nice – lucidbrot May 08 '18 at 07:01
1

For people confused about the seemingly extraneous `1` after the substitution function, see [this question](https://unix.stackexchange.com/questions/63891/what-is-the-meaning-of-1-at-the-end-of-an-awk-script). Short version: leave the `1` there. – Ti Strga Aug 18 '22 at 17:31

anubhava · Answer 2 · 2019-07-04T14:39:40.287

18

I would suggest simply replacing sed with perl command like this:

perl -i.bak -pe 's/_data_/$ENV{"DATA"}/g' mail.tpl

edited Jul 04 '19 at 14:39

answered Apr 11 '12 at 14:31

anubhava

761,203
64
569
643

2

This is the only that worked for me on OSX. All the other sed and awk based approches failed fo me. – Cédric Vidal Dec 01 '17 at 13:45
4

This will fail in unsettling ways if `DATA` contains a slash. But you can `export DATA` and access it from inside Perl as `$ENV{"DATA"}` without getting code and data mixed up. So `'s/_data_/$ENV{"DATA"}/g'` simply. – tripleee Mar 11 '18 at 12:22
1

@anubhava and @tripleee, you saved my day. This is the only solution that worked for me: neither `sed` nor `awk` could do the trick. Thanks. – Dmytro Titov Jul 04 '19 at 14:37

ring bearer · Accepted Answer · 2012-04-11T14:18:16.387

16

If you build your multiple line text with "\n"s, this will work with a simple sed command as:

DATA=`echo ${DATA} | tr '\n' "\\n"`
#now, DATA="line1\nline2\nline3"
sed "s/_data_/${DATA}/" mail.tpl

edited Apr 11 '12 at 14:18

answered Apr 11 '12 at 14:13

ring bearer

20,383
7
59
72

6

I'm getting `sed: -e expression #1, char 32: unterminated 's' command` from that method. – Cobra_Fast Apr 11 '12 at 14:20
2

This is not entirely portable; not all `sed` dialects accept the `\n` digraph to represent a literal newline. In some other dialects, you will probably need to use a backslash before a literal newline. Other answers on this page show you how to do that in more detail. – tripleee Jul 06 '17 at 04:11
11

Looking at this again, I don't think there is *any* version of `tr` which can replace one character with two characters; it's simply the wrong tool for this job. The newlines are lost by accident because you incorrectly fail to quote the argument to `echo`, but this has multiple other undesirable side effects; and the `tr` simply does nothing at all here. **This should not be the accepted answer.** – tripleee Mar 11 '18 at 12:26
Yes does not work for me either. Try [this solution (below)](https://stackoverflow.com/questions/10107459/replace-a-word-with-multiple-lines-using-sed/#22901380) – TechupBusiness Apr 15 '19 at 15:26
1

Given that the shell is Bash, then there's no need for the (broken) `tr` invocation - [just use `${DATA//$'\n'/\\n}`](/a/39269241/4850040). – Toby Speight Aug 14 '19 at 13:08

score 14 · Answer 4 · edited May 23 '17 at 12:26

ring bearer's answer didn't work for me; I think the usage of tr there is wrong, and the way it's written, it simply strips away newlines by use of echo.

Instead, I used sed. I used code from another answer to replace newlines (credit: Zsolt Botykai). I also expected some dollar signs ($) in my input so I took care of that too. You might need to add other input handling. Note the use of double quotes in echo to preserve newlines.

DATA="$(cat whatever)"
ESCAPED_DATA="$(echo "${DATA}" | sed ':a;N;$!ba;s/\n/\\n/g' | sed 's/\$/\\$/g')"

Then you can use ${ESCAPED_DATA} in sed:

cat input | sed 's/one liner/'"${ESCAPED_DATA}"'/' > output

Just thought I'd share.

This works well also when multi-line strings have been built in a `bash` script with `$'\n'` construct (the `cat` in the beginning is, of course, then not needed since the string is already in a variable). — Ville, Jul 30 '17 at 21:43

score 13 · Answer 5 · edited Apr 27 '15 at 19:29

I tried it and sed 's/pattern/\na\nb\nc/g' but it does not work on all systems. What does work is putting a \ followed by a newline in the replace pattern, like this:

sed 's/pattern/a\
b\
c/g'

This appends a line containing b and a line containing c when the pattern is seen.

To put it in a variable, use double backslashes:

export DATA="\\
a\\
b\\
c"

and then:

sed "s/pattern/${DATA}/g"

Note the double quotes.

score 8 · Answer 6 · answered Oct 28 '15 at 17:52

8

Echo variable into temporary text file.

Insert text file into mail.tpl and delete data from mail.tpl

echo ${DATA} > temp.txt    
sed -i -e "/_data_/r temp.txt" -e "//d" mail.tpl

answered Oct 28 '15 at 17:52

Ben Pingilley

729
1
6
13

1

This is the only one that worked for me after trying many other solutions. – Kevin Oct 06 '20 at 21:43

score 4 · Answer 7 · answered Jan 08 '18 at 17:53

4

Escaping all the newlines with a \ (except the last one) worked for me. The last newline must not be escaped not to break the s command.

Example :

DATA="a
b
c"

ESCAPED=$(echo "${DATA}" | sed '$!s@$@\\@g')
echo "${ESCAPED}" 
a\
b\
c

sed "s/pattern/${ESCAPED}/" file

answered Jan 08 '18 at 17:53

mgraff

41
1

I had to use double slashes. – Stonecraft Oct 04 '21 at 01:40

score 3 · Answer 8 · answered Sep 09 '22 at 08:45

As per the gnu documentation about multiline techniques:

sed '/./{H;$!d;}; x; s/REGEXP/REPLACEMENT/'

The first expression, /./{H;$!d;} operates on all non-empty lines, and adds the current line (in the pattern space) to the hold space. On all lines except the last, the pattern space is deleted and the cycle is restarted.

The x command fetches the accumulated lines from the hold space back to the pattern space.

The s/// command then operates on all the text in the paragraph (including the embedded newlines).

score 2 · Answer 9 · answered Apr 11 '12 at 14:19

2

You can put your data in a temp file and run:

$ sed '/_data_/r DATA_FILE' mail.tpl | sed '/_data_/d'> temp; mv temp mail.tpl

answered Apr 11 '12 at 14:19

Mihai

2,125
2
14
16

score 1 · Answer 10 · answered Aug 02 '16 at 05:17

1

Not sure if you have tried to put "\n" in the replace part

sed 's/[pattern]/\
[line 1]\n\
[line 2]\n\
[line n]\n\
/g' mail.tpl

The first line has /\ for readibility reasons. Each line after that is a stand-alone line like you would find in a text editor. Last line is stand-alone, once again for readability reasons. You can make all of this one line if needed. Works on Debian Jessie when I tested it.

answered Aug 02 '16 at 05:17

lsu_guy

1,525
15
12

This is so sed.:) – Barney Szabolcs Jan 06 '21 at 23:02

score 1 · Answer 11 · edited Jun 22 '22 at 06:45

1

DATA=`echo ${DATA} | tr '\n' "\n"`

sed -z "s/_data_/${DATA}/g" -i mail.tpl

Try this...

edited Jun 22 '22 at 06:45

flyingfishcattle

1,817
3
14
25

answered Jun 21 '22 at 16:10

Kalyan Kv

11
2

Replace a word with multiple lines using sed?

11 Answers11

Linked

Related