Using a Bash variable in URL results in "curl: (3) Illegal characters found in URL"

Question

Problem

I'm trying to use a variable when calling a cURL command but it's including the the literal $line instead of the actual value.

while read line; do
  curl "https://x.com/v1/PhoneNumbers/$line?Type=carrier" -u "x:x" 
done < "${1:-/dev/stdin}"

Context

I'm passing a list of numbers to the script trying to read them line by line.

What output do you expect? $line to be translated to a shell variable value or something else? — Chen A., Aug 27 '17 at 11:57
if you go on https://www.twilio.com/lookup you see the command there I'm simply trying to make it read from a file a bunch of phone numbers — Linda Roesch, Aug 27 '17 at 12:14

Mateusz Piotrowski · Accepted Answer · 2017-08-27T13:29:09.780

From the comments we know that you're getting the following error:

curl: (3) Illegal characters found in URL

If formatted this way:

while IFS="$IFS"$'\r' read line; do
  curl "https://x.com/v1/PhoneNumbers/$line?Type=carrier" -u "x:x" 
done < "${1:-/dev/stdin}"

your command should work.

The problem is that your appending \r at the end of your input lines (so that every line of your input ends with a \r\n sequence). By default read does not strip trailing r. If we want read to trim those characters we have to add this character to the IFS environmental variable for read like this: IFS="$IFS"$'\r')" read ....

Here's a great comment from Charles Duffy:

Personally, I'd suggest IFS=$' \t\n\r', not referring to the old $IFS value -- why make your code's behavior contextually dependent?

Another valuable comment; this time from chepner:

Granted, a valid line probably isn't going to contain a \r, but conceptually, you don't want to treat a carriage return as whitespace; you just want to strip the \r that is part of the \r\n line ending. Instead of modifying IFS, read the line normally, then strip it with line=${line%$'\r'} before calling curl.

Granted, a valid line probably isn't going to *contain* a `\r`, but conceptually, you don't want to treat a carriage return as whitespace; you just want to strip the `\r` that is part of the `\r\n` line ending. Instead of modifying `IFS`, read the line normally, then strip it with `line=${line%$'\r'}` before calling `curl`. — chepner, Aug 27 '17 at 13:25

Chen A. · Answer 2 · 2017-08-27T12:09:53.637

0

You try to address a variable and a text right after it, you should use ${VAR} syntax:

curl "https://x.com/v1/PhoneNumbers/${line}?Type=carrier" -u "x:x" done < "${1:-/dev/stdin}"

The reason is bash tries to expand the parameter $line?Type .. instead of $line.

For example

[root@ ~]# line=1234
[root@ ~]# echo $line567

[root@ ~]# echo ${line}567
1234567

edited Aug 27 '17 at 12:09

answered Aug 27 '17 at 11:55

Chen A.

10,140
3
42
61

Unfortunately I'm getting curl: (3) Illegal characters found in URL – Linda Roesch Aug 27 '17 at 12:15
This is not the case as `?` cannot be a part of the variable name so it's perfectly OK to omit `{}` (at least in this case). – Mateusz Piotrowski Aug 27 '17 at 12:17
Can you add echo statement of the string before executing curl and show it here? `echo https://x.com/v1/PhoneNumbers/$line?Type=carrier` – Chen A. Aug 27 '17 at 12:17
2

`echo` isn't nearly as useful for showing nonprintable characters as `printf '%q\n' "https://x.com/v1/PhoneNumbers/$line?Type=carrier"` (which will properly quote and escape a trailing carriage return, f/e, which is what the accepted answer proposes) – Charles Duffy Aug 27 '17 at 13:00

Using a Bash variable in URL results in "curl: (3) Illegal characters found in URL"

2 Answers2