Home » Food Marketing Corner » Bash String with Hex to Decimal

Bash String with Hex to Decimal

Alexander Ross • • 10 min read
Bash String with Hex to Decimal

I have struggled with some similar problems already I believe discussed but really I do not get the way out: What if your file has data for example as below (hex values can be anything but fixed to 1 byte) and you want to convert each hex to decimal and keep it as in below format so instead of hex you display decimal and still comma separated? - Please note number of data lines is not fixed it can by any number.

Example 1:

0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51
0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a
0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0
0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51
0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a
0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0
0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51
0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a
0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0

Example 2: same as above but lets say it has data name in front:

X:0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51
Y:0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a
Z:0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0
X:0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51
Y:0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a
Z:0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0
X:0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51
Y:0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a
Z:0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0

So I have spent some time to understand the bash grep, sed etc.. as I am completely not Software guy I manage to play and filter out some text files in the correct format next I would like to understand how to operate on strings with more than one value and above examples will really help . Thanks in advance.

I did try printf, echo and I am able to use them when each string in text file has one value for example : if I use data file which has data

0xff
0xff
0xff
0xff
0xff
.
.
.
.
.
.
..
0xff

all works fine and can be converted with printf function: this was found in other thread:

while read line

do

     printf '%d' $line

done < file

Above works great for the one hex value in line: Can this be adjusted to work with my examples or there is any better way?

1

6 Answers

You can use (GNU) Awk and it's strtonum function, which converts strings that start with 0x to decimal numbers.

  • Just splitting at , and starting from column 1:
awk 'BEGIN {FS=OFS=","} {for (i=1;i<=NF;i++) $i=strtonum($i); print}' file.txt

81,81,81,81,81,81,81,81,81,81,81,81,81,81
90,90,90,90,90,90,90,90,90,90,90,90,90,90
240,240,240,240,240,240,240,240,240,240,240,240,240,240
81,81,81,81,81,81,81,81,81,81,81,81,81,81
90,90,90,90,90,90,90,90,90,90,90,90,90,90
240,240,240,240,240,240,240,240,240,240,240,240,240,240
81,81,81,81,81,81,81,81,81,81,81,81,81,81
90,90,90,90,90,90,90,90,90,90,90,90,90,90
240,240,240,240,240,240,240,240,240,240,240,240,240,240
  • Splitting at , or : and starting from column 2:
awk 'BEGIN {FS=",|:"; OFS=","} {for (i=2;i<=NF;i++) $i=strtonum($i); print}' file.txt

X,81,81,81,81,81,81,81,81,81,81,81,81,81,81
Y,90,90,90,90,90,90,90,90,90,90,90,90,90,90
Z,240,240,240,240,240,240,240,240,240,240,240,240,240,240
X,81,81,81,81,81,81,81,81,81,81,81,81,81,81
Y,90,90,90,90,90,90,90,90,90,90,90,90,90,90
Z,240,240,240,240,240,240,240,240,240,240,240,240,240,240
X,81,81,81,81,81,81,81,81,81,81,81,81,81,81
Y,90,90,90,90,90,90,90,90,90,90,90,90,90,90
Z,240,240,240,240,240,240,240,240,240,240,240,240,240,240
  • This way, however, also the : will be converted to ,. If this is not desired, you could print the columns 1, 2, and the rest separately:
awk 'BEGIN {FS=",|:"} {printf "%s:%s", $1, strtonum($2); for (i=3;i<=NF;i++) printf ",%s", strtonum($i); printf "\n"}' file.txt

X:81,81,81,81,81,81,81,81,81,81,81,81,81,81
Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90
Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240
X:81,81,81,81,81,81,81,81,81,81,81,81,81,81
Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90
Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240
X:81,81,81,81,81,81,81,81,81,81,81,81,81,81
Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90
Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240
1

A generalized awk solution -

awk -F'[:,]' -v OFS=, '{for(i=1;i<=NF;i++){if($i~/^0x/){$i=strtonum($i)}}if($1!~/^[0-9]/)sub(",",":");print}' file

Saved to a file, run with each of your example sets:

$: cat x2d
#! /usr/bin/awk -f
BEGIN{ FS="[:,]"; OFS="," }
{ for ( i=1; i<=NF; i++ ) { if ( $i ~ /^0x/ ) { $i=strtonum($i) } }
  if ( $1 !~ /^[0-9]/ ) sub(",",":");
  print;
}

$: ./x2d withName
X:81,81,81,81,81,81,81,81,81,81,81,81,81,81
Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90
Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240
X:81,81,81,81,81,81,81,81,81,81,81,81,81,81
Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90
Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240
X:81,81,81,81,81,81,81,81,81,81,81,81,81,81
Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90
Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240

$: ./x2d noName
81,81,81,81,81,81,81,81,81,81,81,81,81,81
90,90,90,90,90,90,90,90,90,90,90,90,90,90
240,240,240,240,240,240,240,240,240,240,240,240,240,240
81,81,81,81,81,81,81,81,81,81,81,81,81,81
90,90,90,90,90,90,90,90,90,90,90,90,90,90
240,240,240,240,240,240,240,240,240,240,240,240,240,240
81,81,81,81,81,81,81,81,81,81,81,81,81,81
90,90,90,90,90,90,90,90,90,90,90,90,90,90
240,240,240,240,240,240,240,240,240,240,240,240,240,240

Just for the practice, the same in basic bash.

$: cat x2d
#! /bin/bash
while IFS=:, read -ra field
do for f in "${field[@]}"
   do case $f in 0x*) printf %d, $f;; *) printf %s: $f;; esac
   done
   printf $'\n'
done < $1

$: ./x2d withName
X:81,81,81,81,81,81,81,81,81,81,81,81,81,81,
Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90,
Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240,
X:81,81,81,81,81,81,81,81,81,81,81,81,81,81,
Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90,
Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240,
X:81,81,81,81,81,81,81,81,81,81,81,81,81,81,
Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90,
Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240,

$: ./x2d noName
81,81,81,81,81,81,81,81,81,81,81,81,81,81,
90,90,90,90,90,90,90,90,90,90,90,90,90,90,
240,240,240,240,240,240,240,240,240,240,240,240,240,240,
81,81,81,81,81,81,81,81,81,81,81,81,81,81,
90,90,90,90,90,90,90,90,90,90,90,90,90,90,
240,240,240,240,240,240,240,240,240,240,240,240,240,240,
81,81,81,81,81,81,81,81,81,81,81,81,81,81,
90,90,90,90,90,90,90,90,90,90,90,90,90,90,
240,240,240,240,240,240,240,240,240,240,240,240,240,240,

For data sets this small, the difference is irrelevant.
If the inputs are big, the awk will be faster.

4

I'd use perl. For your first case:

$ perl -F, -lane 'print join(",", map(hex, @F))' ex1.txt
81,81,81,81,81,81,81,81,81,81,81,81,81,81
90,90,90,90,90,90,90,90,90,90,90,90,90,90
240,240,240,240,240,240,240,240,240,240,240,240,240,240
81,81,81,81,81,81,81,81,81,81,81,81,81,81
90,90,90,90,90,90,90,90,90,90,90,90,90,90
240,240,240,240,240,240,240,240,240,240,240,240,240,240
81,81,81,81,81,81,81,81,81,81,81,81,81,81
90,90,90,90,90,90,90,90,90,90,90,90,90,90
240,240,240,240,240,240,240,240,240,240,240,240,240,240

and the second, with the prefix:

$ perl -F'[:,]' -lane 'print $F[0], ":", join(",", map(hex, @F[1..$#F]))' ex2.txt
X:81,81,81,81,81,81,81,81,81,81,81,81,81,81
Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90
Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240
X:81,81,81,81,81,81,81,81,81,81,81,81,81,81
Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90
Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240
X:81,81,81,81,81,81,81,81,81,81,81,81,81,81
Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90
Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240
2

A straightforward Bash solution would be to just let Bash parse integers like it normally does. For Example 2:

to_dec() {
  local -r IFS=,:
  local -a line
  local -ai nline
  while read -ra line; do
    nline=("${line[@]:1}")
    printf '%s:%s\n' "${line[0]}" "${nline[*]}"
  done
}

Then to_dec < file gives the expected output for Example 2. A to_dec for Example 1 will be much simpler, because there’s no need for an auxiliary array in that case; just read the numbers:

to_dec() {
  local -r IFS=,
  local -ai nline
  while read -ra nline; do
    printf '%s\n' "${nline[*]}"
  done
}

As for the math operations for Example 2, requested in a comment, in pure Bash one could create functions named after the line prefixes and then execute them on the arrays. In any case, it won’t be “efficient”; it’s Bash. (Also, presumably, Bash has integer division only; to circumvent that one could use a fixed-point representation of some sort, but that’s way too hacky in Bash.)

#!/bin/bash

X() { local -n a="$1"; local -i i
      for i in "${!a[@]}"; do ((a[i] /= 2)); done; }
Y() { local -n a="$1"; local -i i
      for i in "${!a[@]}"; do ((a[i] *= 2)); done; }
Z() { local -n a="$1"; local -i i
      for i in "${!a[@]}"; do ((a[i] += 2)); done; }

to_dec() {
  local -r IFS=,:
  local -a line
  local -ai nline
  while read -ra line; do
    nline=("${line[@]:1}")
    "${line[0]}" 'nline'  # transform the array
    printf '%s:%s\n' "${line[0]}" "${nline[*]}"
  done
}

to_dec  # takes the script's default stdin / stdout

For a bit more efficiency and floating-point arithmetics, awk can help:

#!/usr/bin/awk -f

BEGIN { FS = "[:,]" }

function X(numbers) { for (i in numbers) numbers[i] /= 2 }
function Y(numbers) { for (i in numbers) numbers[i] *= 2 }
function Z(numbers) { for (i in numbers) numbers[i] += 2 }

{
  for (i = 2; i <= NF; ++i) numbers[i] = strtonum($i)
  f = $1
  @f(numbers)
  output = $1 ":" numbers[2]
  for (i = 3; i <= NF; ++i) output = output "," numbers[i]
  print output
}

For large data processing, a proper implementation in a more low-level programming language would be more efficient.

5

GNU AWK

awk -F'[,:]' -v OFS=, '
    /:/{ withName=1 }
    {
        for(i=1;i<=NF;i++) $i=(strtonum($i) ? strtonum($i) : $i)
    }
    withName{
        sub(/,/,":")
        withName=0
    }1
' file1 file2

81,81,81,81,81,81,81,81,81,81,81,81,81,81
90,90,90,90,90,90,90,90,90,90,90,90,90,90
240,240,240,240,240,240,240,240,240,240,240,240,240,240
81,81,81,81,81,81,81,81,81,81,81,81,81,81
90,90,90,90,90,90,90,90,90,90,90,90,90,90
240,240,240,240,240,240,240,240,240,240,240,240,240,240
81,81,81,81,81,81,81,81,81,81,81,81,81,81
90,90,90,90,90,90,90,90,90,90,90,90,90,90
240,240,240,240,240,240,240,240,240,240,240,240,240,240
X:81,81,81,81,81,81,81,81,81,81,81,81,81,81
Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90
Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240
X:81,81,81,81,81,81,81,81,81,81,81,81,81,81
Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90
Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240
X:81,81,81,81,81,81,81,81,81,81,81,81,81,81
Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90
Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240

echo 
'0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51
 0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a
 0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0
 0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51
 0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a
 0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0
 0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51
 0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a
 0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0' | 

gawk -nb '$_ += ORS = RT' RS='\n|,' 

81,81,81,81,81,81,81,81,81,81,81,81,81,81
90,90,90,90,90,90,90,90,90,90,90,90,90,90
240,240,240,240,240,240,240,240,240,240,240,240,240,240
81,81,81,81,81,81,81,81,81,81,81,81,81,81
90,90,90,90,90,90,90,90,90,90,90,90,90,90
240,240,240,240,240,240,240,240,240,240,240,240,240,240
81,81,81,81,81,81,81,81,81,81,81,81,81,81
90,90,90,90,90,90,90,90,90,90,90,90,90,90
240,240,240,240,240,240,240,240,240,240,240,240,240,240

 gawk -nMb '$++NF = +$_' OFS='\f'

0x5151515151515151515151515151
                              1649317825652239364356745892811089
0x5a5a5a5a5a5a5a5a5a5a5a5a5a5a
                              1832575361835821515951939880901210
0xf0f0f0f0f0f0f0f0f0f0f0f0f0f0
                              4886867631562190709205173015736560
0x5151515151515151515151515151
                              1649317825652239364356745892811089
0x5a5a5a5a5a5a5a5a5a5a5a5a5a5a
                              1832575361835821515951939880901210
0xf0f0f0f0f0f0f0f0f0f0f0f0f0f0
                              4886867631562190709205173015736560
0x5151515151515151515151515151
                              1649317825652239364356745892811089
0x5a5a5a5a5a5a5a5a5a5a5a5a5a5a
                              1832575361835821515951939880901210
0xf0f0f0f0f0f0f0f0f0f0f0f0f0f0
                              4886867631562190709205173015736560

echo '
X:0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51
Y:0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a 
Z:0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0 
X:0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51 
Y:0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a 
Z:0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0 
X:0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51 
Y:0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a 
Z:0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0' | 

gawk -nb 'NF == +(ORS = RT) || $_ += _' RS='\n?[A-Z]:|[\n,]' 

X:81,81,81,81,81,81,81,81,81,81,81,81,81,81
Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90
Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240
X:81,81,81,81,81,81,81,81,81,81,81,81,81,81
Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90
Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240
X:81,81,81,81,81,81,81,81,81,81,81,81,81,81
Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90
Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Alexander Ross
Author

Alexander Ross

Alexander Ross has covered the video game industry for a decade, writing deep dives on game design, esports tournaments, VR developments, and gaming culture.