I have .sh script that, among other things, wgets currency exchange rates from Google as follows:
printf 'Bash: Going to get exchange rates'
echo
wget -qO- "" | sed '/res/!d;s/<[^>]*>//g' > exrates
wget -qO- "" | sed '/res/!d;s/<[^>]*>//g' >> exrates
wget -qO- "" | sed '/res/!d;s/<[^>]*>//g' >> exrates
wget -qO- "" | sed '/res/!d;s/<[^>]*>//g' >> exrates
wget -qO- "" | sed '/res/!d;s/<[^>]*>//g' >> exrates
wget -qO- "" | sed '/res/!d;s/<[^>]*>//g' >> exrates
mv /home/stan/perl/2014/scripts/exrates /home/stan/perl/2014/exrates/exrates
printf 'Bash: Got exchange rates'
echo
Occasionally script hangs here, however. I don't mind not updating these rates every time it runs, if it hangs I'd like to skip this step altogether, but how?
What should I put in "if" statement to check if wget can fetch data promptly or will take forever? A little more verbosity in wget execution wouldn't hurt either.
Btw, I don't know why wget hangs. Browser opens those pages okay, and same commands run from terminal line by line work, too.
3 Answers
I assume that it hangs because you have a number of HTTP requests being sent to a single host in a script. The host in question doesn't like that too much and it starts to block requests from your IP address.
A simple workaround would be to put a sleep in between the requests. You could also make use of a function:
getExchangeRates() {
wget -qO- "" | sed '/res/!d;s/<[^>]*>//g' >> exrates
sleep 10 # Adding a 10 second sleep
}
and invoke it by passing a parameter to the function:
getExchangeRates aud
The function could also be invoked in a loop for various currencies:
for currency in aud jpy hkd nzd eur gpb; do
getExchangeRates $currency
done
use timeout in wget statement only
wget --timeout 10 <URL>
timeout is in seconds and put some sleep in between two wgets
wget has various timeout options. From the man page
--timeout=seconds
Set the network timeout to seconds seconds. This is equivalent to
specifying --dns-timeout, --connect-timeout, and --read-timeout,
all at the same time.
So you can simply set --timeout, or if you believe it's one of the other factors alone you can set a specific timeout
