Close

Polishing and more bash hacking

A project log for Libre Gates

A Libre VHDL framework for the static and dynamic analysis of mapped, gate-level circuits for FPGA and ASIC. Design For Test or nothing !

Yann Guidon / YGDESYann Guidon / YGDES 11/12/2020 at 11:359 Comments

Before digging too deep into the netlist extractor, I wanted to review and massage the files a bit. @llo helped and installed the latest GHDL on Fedora over WSL: several warnings uncovered some variable naming issues, now solved. Thanks Laura !

The archive seems to be almost easy to use, but speed could still be better. Building the library is essentially linear, but the various tests could be run in parallel. It does not matter much because they are pretty short, except for the longest step in ALU8 where all the 500+ faults are injected, and it takes a while... The fault verification of INC8 could also use less time if all 4 cores could run simultaneously in this "trivially parallel" application.

I still don't want to use make. There are other paralleling helpers such as xargs or GNU parallel but they create too many complications for passing information back and forth. I also want to avoid semaphores, critical sections or locks... and I finally found the right ingredients using only bash! Here is the structure:

# if NBTHREADS is not set or invalid, then query the system
[[ $(( NBTHREADS + 0 )) -eq 0 ]] &&
  NBTHREADS=$( { which nproc > /dev/null 2>&1 ; } && nproc ||
     grep -c ^processor /proc/cpuinfo )

echo "using $NBTHREADS parallel threads"

function TheWorkload () {
  echo "starting workload #$1"
  sleep $(( (RANDOM % 5) + 1 ))
  echo "end of workload #$1"
}

{
  threads=0
  for i in $( seq 1 20 )
  do
    threads=$(( $threads + 1 ))
    echo "loop #$i, $threads running threads"
    TheWorkload $i &
    if [[ $threads -ge $NBTHREADS ]] ; then
      echo "waiting ..."
      wait -n
      threads=$(( $threads - 1 ))
    fi
  done

  wait
  echo "End of script !"
} 2> /dev/null # hide bash's termination messages

Something is still missing: the loop must stop if one workload fails. It's quite delicate with more sophisticated tools but in bash, it's as easy as detecting the fault and break the loop. This leads to this new version:

[[ $(( NBTHREADS + 0 )) -eq 0 ]] &&
  NBTHREADS=$( { which nproc > /dev/null 2>&1 ; } && nproc ||
     grep -c ^processor /proc/cpuinfo )
echo "using $NBTHREADS parallel threads"

err_flag=0

function TheWorkload () {
  echo "starting workload #$1"
  sleep $(( (RANDOM % 5) + 1 ))
  echo "end of workload #$1"
  # make the 12th run break the loop
  [[ "$1" -eq "12" ]] && {
    err_flag=1
    echo "$1 reached !"
  }
}

{
  threads=0
  for i in $( seq 1 20 )
  do
    threads=$(( $threads + 1 ))
    echo "loop #$i, $threads running threads"
    TheWorkload $i &
    if [[ $threads -ge $NBTHREADS ]] ; then
      echo "waiting ..."
      wait -n && {
        echo "Got an error !" ; err_flag=1 ; break ; }
      [[ $err_flag -eq "1" ]] && {
        echo "found an error !" ; break ; }
      threads=$(( $threads - 1 ))
    fi
  done

  wait
  [[ $err_flag -eq "0" ]] && { echo "it worked."
           } ||   echo "premature termination"
} 2> /dev/null # hide bash's job termination messages

Since everything is controlled by a single loop in the script, and the flag can only be written by the workload, there is no risk of race condition, so no need of semaphore or what else... The only tricky part is to ensure that the remaining tasks, have to complete and check their errors too.

I have seen other similar examples that use the jobs command to list the spawned threads but there is the risk of catching sub-sub-threads if the workload forks... Using a thread counter is faster (no fork-exec of other tools) and limits the scope of the test, which removes interferences.

There are more issues to solve such as adapting the result of each invocation of the GHDL-generated binary. But another place where significant time can be saved is during the simulation setup: the binary is restarted all the time despite only tiny changes are made. The VHDL simulation system must also be updated...

--------------------------------------------------------------------------

Hmmm it seems I botched the script and made wrong assumptions. It is not possible to send a value back to the main script once a workload has forked. Yet this code works because the return value is available from wait:

err_flag=0

function TheWorkload () {
  echo "starting workload #$1"
  sleep $(( (RANDOM % 5) + 1 ))
  echo "end of workload #$1"
  [[ "$1" -eq "19" ]] && {
    echo "$1 reached !"
  } # the return value here indicates success
}

{
  threads=0
  for i in $( seq 1 20 )
  do
    threads=$(( $threads + 1 ))
    TheWorkload $i &
    if [[ $threads -ge $NBTHREADS ]] ; then
      wait -n && { echo "Got an error !" ; err_flag=1 ; break ; }
      threads=$(( $threads - 1 ))
    fi
  done

  wait && { echo "Got a late error !" ; err_flag=1 ; }
} 2> /dev/null # hide bash's job termination messages

  [[ $err_flag -eq "0" ]] && { echo "it worked."
           } ||   echo "premature termination"

It is able to check for "late errors" but I can't easily send other values back from the workload. The err_flag is updated only from the main loop. Yet I'd like to collect statistics from the workloads.

--------------------------------------------------------------------------

sam. nov. 14 06:45:42 CET 2020

The new archive reduces the whole test time from 75s to 42s ! It makes use of all available CPUs for several parallel tests. More needs to be done but I'm glad I significantly shortened the tests.

Go get the latest archive !

Discussions

Yann Guidon / YGDES wrote 11/14/2020 at 06:36 point

@Ken Yap I have a question that you might help answer :

In one case, I have >500 subprocesses that generate a number and I want to accumulate them, or log them. So far I have not found a fast, reasonable and safe way to perform this. I have no success with flock. Do you have any hint ?

  Are you sure? yes | no

Yann Guidon / YGDES wrote 11/14/2020 at 23:57 point

Thanks for the tip !
I thought about pipes but...
In the end, the chunk of data is very short and the order is not relevant so I just >> to a common file and it seems to work... I just have to add an extra loop at the end to process the common file and the results seem consistent.

  Are you sure? yes | no

Ken Yap wrote 11/15/2020 at 00:11 point

A pipe doesn't guarantee order either. Appending to a file should be ok as long as the writes are as soon as possible after open, and atomic so that there's no chance of interleaving data.

Edit: Make sure you do it this way:

result=$(longprogram)

echo $result >> log

Not:

longprogram >> log

  Are you sure? yes | no

Yann Guidon / YGDES wrote 11/15/2020 at 09:23 point

Good one !

  Are you sure? yes | no

Ken Yap wrote 11/12/2020 at 22:22 point

There are several simplifications available in modern bash.

Instead of for i in $(seq 1 20) try for i in {1..20}

$ isn't required inside $(( )) arithmetic expressions.

And for parameter missing or unset try ${parameter:-word} though word may be complicated in your case.

  Are you sure? yes | no

Yann Guidon / YGDES wrote 11/14/2020 at 05:40 point

Thanks for the tips :-)

Now I have a problem with the enum syntax : for i in {1..4} ;  works but for i in {1..$a} fails expansion... The seq version is clearer and allows expansion.

bash has such an intricate syntax... but it gets stuff done :-D

  Are you sure? yes | no

Ken Yap wrote 11/14/2020 at 05:48 point

Yes, it's because brace expansion happens before variable expansion. Just like expressions like foo{a,b,c,d}, a variable doesn't work inside the brace. I've explained it here: https://green-possum-today.blogspot.com/2020/04/bash-ranges.html which contain other examples, like increments, backward ranges, non-numeric ranges, prefixes and suffixes.

The bash man page is incredibly concise and often multiple readings and experimentation are required to comprehend a feature.

  Are you sure? yes | no

Yann Guidon / YGDES wrote 11/14/2020 at 05:55 point

yes, the bash man page is ... well... how do I put it ? :-D
I've read it so many times and still have to return to it.
Anyway, I am quite proud of my latest achievement, it will be useful for many projects.

  Are you sure? yes | no