Progress Bars

knoxg | October 31, 2013 | Programming | 2 Comments

Some background information

The progress bar.

Not just the name of a sadly defunct bar in London or a sign depicting the prevention of egress, progress bars serve a useful purpose to inform the user just how long they’re going to have to sit around with a thumb up their arse whilst they perform their highly skilled non-DMCA-based white-collar workloads.

They look a bit like this:
or this:
or this:
or this:

As you may be aware, Microsoft will be dropping support for several classes of progress bar in 2014, for reasons of security to the Microsoft revenue base.

A lot of tasks don’t have an easily-defined halting condition, so you get slightly different progress bars that move a little to the right, and then a little to the left, or look like barberpoles if you use one of those computers that they look like barberpoles on.

They look a bit like this:
or this:
or this:

These days of course, it’s more modern to expose your end-users to “throbbers” instead, which give a pleasing sense of going round in circles, and are probably a good indication of what limbo is going to end up being like, if you’re religiously inclined.

They appear in many guises, such as

this…	or this…	or this…	or this…	or this…	or this….

or this:

Some foreground information

So anyway, I’m not one to stand in the way of progress (ha), and now that I think you might know what a progress bar is, here’s something to make your scp task generate some javascript which you can dump into a browser to make a little rectangle get larger as the copy process takes place.

People generally expect to see this sort of thing during long periods of what appears to be inactivity, so you might as well give it to them so as not to surprise them, which apparently is a usability nightmare, unless you’re trying to cram the next great evolution in gesture-based interfaces down people’s throats.

The program itself is called scpwrap, and it’s vaguely similar to that previous post about piping stdout/stderr streams in VBA, since that’s what it does, but this implementation is in C, which is much more self-respecting. You could probably drop it into a bash script without people thinking they should wrap it in a chef recipe or a puppet whatever-they-call-their-module-components [1].

It’s slightly more complicated than it should be because scp won’t normally display progress information if it’s not running in a terminal, hence the forkpty pseudo-terminal malarky on line 319. The code is based on (and is backwards-compatible with) the scp-to-zenity wrapper written by Clayton Shepard, but also handles non-standard stdout/stderr streams, has customisable output templates, and deals with error conditions in a relatively sane way.

Here’s the sourcecode for the scpwrap program, and the groff source for the manpage, which took almost as long to write up.

There’s also a link to the randomnoun apt repository at the bottom of this post, if you want to install it automatically.

Show codeHide code

scpwrap.c

/* (c) 2013 randomnoun. All Rights Reserved. This work is licensed under a
 * BSD Simplified License. (http://www.randomnoun.com/bsd-simplified.html)
 */
 
/* scpwrap.c
 * 
 * $Id: scpwrap.c,v 1.19 2013-11-03 22:24:42 knoxg Exp $
 *
 * Wrapper for scp which translates progress text, e.g:
 * 
 *   something.tar.gz                                1% 2112KB   2.1MB/s   00:50 ETA   
 * 
 * into javascript, which can then display and update a progress bar.
 *
 * Call this thing from a bash script or other process/thread that will dump its output 
 * directly into an iframe which is surrounded by <script> elements.
 *
 * COMPILING
 * 
 * Remember to include the util library when compiling !
 * i.e. gcc -lutil scpwrap.c -oscpwrap
 *
 * as of 2013, you could try doing this instead, for reasons that I don't particularly understand:
 *      gcc -Wl,--no-as-needed -lutil scpwrap.c -oscpwrap
 * 
 * Modified from http://cwshep.blogspot.com/2009/06/showing-scp-progress-using-zenity.html
 *
 * This program relies on the fact that even if stdout performs checks to see whether 
 * it is running on a tty, these checks are not perfomed for stderr ( search for 'isatty' in 
 * http://www.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/scp.c?rev=1.178;content-type=text%2Fx-cvsweb-markup )
 *
 * see http://www.linuxquestions.org/questions/programming-9/problem-with-child-process-658750/#post3229496
       http://www.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/scp.c?rev=1.178;content-type=text%2Fx-cvsweb-markup
       http://stackoverflow.com/questions/2605130/redirecting-exec-output-to-a-buffer-or-file
       http://stackoverflow.com/questions/903864/how-to-exit-a-child-process-and-return-its-status-from-execvp
 */
 
#include <unistd.h>
#include <sys/types.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdarg.h>
#include <sys/select.h>
#include <getopt.h>
 
/* let's say that the things we're going to replace in here are:
 * %f - filename
 * %p - progress (0-100) and "%" character
 * %t - transfer size ("2112KB")
 * %s - speed ("2.1MB/s" or however progressmeter.c does things)
 * %e - ETA ("--:--" or "05:23"), hh:mm:ss, or however progressmeter.c does things)
 */
 
// if we get more than this number of bytes on stdout/stderr without a newline,
// then they will be emitted in >1 stdout/stderr template 
#define STDERR_BUFSIZE 1024
#define STDOUT_BUFSIZE 1024
 
// enabled by --js option; stdout/stderr will be javascript-String escaped
static int ESCAPE_JS = 0;
 
static char* TXT_STDOUT_TEMPLATE = "";
static char* TXT_STDERR_TEMPLATE = "";
static char* TXT_START_TEMPLATE = "";
static char* TXT_PROGRESS_TEMPLATE = "%p\n";
static char* TXT_END_TEMPLATE = "";
 
static char* JS_STDOUT_TEMPLATE = "ui.addOutput(\"%s\");\n";
static char* JS_STDERR_TEMPLATE = "ui.addOutputError(\"%s\");\n";
static char* JS_START_TEMPLATE = "var sp = ui.startScpProgress();\n";
static char* JS_PROGRESS_TEMPLATE = "sp.setProgress(\"%f\", %p, \"%t\", \"%s\", \"%e\");\n";
static char* JS_END_TEMPLATE = "ui.stopScpProgress(%c);\n";
 
/** convert ASCII to javascript escape.
 
    see http://rishida.net/tools/conversion
 */ 
void printjs (char *str) {
    int i=0;
    for (i=0; str[i]!=0; i++) {
        switch (str[i]) {
            case 0: printf("\\0"); break;
            case 8: printf("\\b"); break;
            case 9: printf("\\t"); break;
            case 10: printf("\\n"); break;
            case 13: printf("\\r"); break;
            case 11: printf("\\v"); break;
            case 12: printf("\\f"); break;
            case 34: printf("\\\""); break;
            case 39: printf("\\\'"); break;
            case 92: printf("\\\\"); break;
            default:
                if (str[i]>0x1f && str[i]<0x7F) { 
                    printf("%c", str[i]); 
                } else { 
                    printf("\\u%04x", (int) str[i]); 
                }
        }       
    }
}
 
/** Take a format string ("template") containing 0 or more tokens in the form 
   "{n}", and replace each token with the n'th optional parameter to this function. 
   The string after token replacement is sent to stdout. 
 
   The nparam must be set to the number of optional parameters supplied.
   All optional parameters must be of type char*.
 
   e.g. 
     printfmt(3, "first: {0}, second: {1}, third: {2}", "a", "b", "c");
   produces the output
     "first: a, second: b, third: c"
 
   C-like escapes ("\n", "\t" etc) in the format string will be converted into their appropriate character     
   if JS_ESCAPE is true, then values in the optional parameters will be Javascript-escaped
     (i.e. the character '\n' will be converted to the two-character "\n" escape) 
 */  
int printfmt (int nparam, const char *format, ...) {
    va_list ap;
    int i = 0;
    int inBrace = 0, inEscape = 0;
    int paramIdx = 0;
 
    char *params[nparam];
    va_start(ap, format);
    for (i=0; i<nparam; i++) { 
        params[i] = va_arg(ap, char *);
    }  
    va_end(ap);
 
    for (i=0; format[i]!=0; i++) {
        if (inEscape) {
            inEscape = 0;
            switch (format[i]) {
                case 'n': printf("\n"); break;
                case 'r': printf("\r"); break;
                case 't': printf("\t"); break;
                case '{': printf("{"); break;
                case '\\': printf("\\"); break;
                default: /* unknown escape, just print character */ printf("%c", format[i]);
            }
        } else if (inBrace) {
            switch (format[i]) {
                case '0': case '1': case '2': case '3': case '4': 
                case '5': case '6': case '7': case '8': case '9':
                    paramIdx = paramIdx*10 + (format[i]-'0');
                    break;
                case '}': 
                    inBrace = 0; 
                    if (ESCAPE_JS) { 
                        printjs(params[paramIdx]); 
                    } else { 
                        printf("%s", params[paramIdx]); 
                    }
                    break;
                default:
                    // weird thing in brace
                    return -1;
            }      
        } else {      
            switch (format[i]) {
                case '{':  paramIdx = 0; inBrace = 1; break;
                case '\\': inEscape = 1; break;
                default: printf("%c", format[i]); 
            }
        }
    }
}
 
/** send usage information to stdout */ 
void usage() {
    printf("usage: scpwrap [options] -- scp-options \n"
      "Where options are:\n" 
      "  --js                   use javascript default templates, and javascript-escape output strings\n"
      "  --stdoutTemplate txt   template to use for unrecognised stdout text\n"
      "  --stderrTemplate txt   template to use for unrecognised stderr text\n"
      "  --startTemplate txt    text to display before the first progressTemplate appears\n"
      "  --progressTemplate txt template to use for copy progress output\n"
      "  --endTemplate txt      template to use after copy completes\n"
      "The following placeholders can be used in progress templates:\n"
      "  %%f  filename\n"
      "  %%p  progress amount (0-100)\n"
      "  %%t  transfer size (e.g. \"2112KB\")\n"
      "  %%s  speed (e.g. \"2.1MB/s\")\n"
      "  %%e  ETA (e.g. \"--:--\" or \"05:23\")\n"
      "The following placeholder can be used in stdout/stderr templates:\n"  
      "  %%s  text string\n"
      "The following placeholder can be used in the endTemplate:\n"  
      "  %%c  exit code\n"
      "\n"
      "See the 'scpwrap' and 'scp' man page for more options. Example usage:\n"
      "  scpwrap --js -- -i identityfile user@host1:file1 user@host2:file2\n"
      );
 
    // This is a program that wraps scp and prints out the numeric progress on separate lines.\n");
    fflush(stdout);
}
 
 
/** replace all instances of a substring in a string with a relacement string.
   memory for the returned string will be reserved via malloc
 
   see http://coding.debuntu.org/c-implementing-str_replace-replace-all-occurrences-substring
 */   
char *str_replace ( const char *string, const char *substr, const char *replacement ){
    char *tok = NULL;
    char *newstr = NULL;
    char *oldstr = NULL;
    /* if either substr or replacement is NULL, duplicate string and let caller handle it */
    if ( substr == NULL || replacement == NULL ) { 
        return strdup (string); 
    }
    newstr = strdup (string);
    while ( (tok = strstr ( newstr, substr ))) {
        oldstr = newstr;
        newstr = malloc ( strlen ( oldstr ) - strlen ( substr ) + strlen ( replacement ) + 1 );
        /*failed to alloc mem, free old string and return NULL */
        if ( newstr == NULL ) {
            free (oldstr);
            return NULL;
        }
        memcpy ( newstr, oldstr, tok - oldstr );
        memcpy ( newstr + (tok - oldstr), replacement, strlen ( replacement ) );
        memcpy ( newstr + (tok - oldstr) + strlen( replacement ), tok + strlen ( substr ), strlen ( oldstr ) - strlen ( substr ) - ( tok - oldstr ) );
        memset ( newstr + strlen ( oldstr ) - strlen ( substr ) + strlen ( replacement ) , 0, 1 );
        free (oldstr);
    }
    return newstr;
}
 
/** main */
main(int argc, char **argv) {
    char *startTemplate = TXT_START_TEMPLATE;
    char *stdoutTemplate = TXT_STDOUT_TEMPLATE;
    char *stderrTemplate = TXT_STDERR_TEMPLATE;
    char *progressTemplate = TXT_PROGRESS_TEMPLATE;
    char *endTemplate = TXT_END_TEMPLATE;
 
    int shownStartTemplate = 0;       // set to 1 when startTemplate is printed
 
    char stderrBuf[STDERR_BUFSIZE];   // stderr capture buffer
    char stdoutBuf[STDOUT_BUFSIZE];   // stdout capture buffer
    char fieldBuf[STDOUT_BUFSIZE];    // as per stdout, with spaces replaced with \0x0
 
    int stderrIdx = 0, stdoutIdx = 0; // indices within stderrBuf/stdoutBuf
    ssize_t stderrc, stdoutc;         // count of bytes read into stderrBuf/stdoutBuf
    int doneStderr = 0, doneStdout=0; // set to 1 when stderr/stdout file descriptors have closed
 
    int stdoutPtyFd;                  // file descriptor for the master side of the stdout pseudoterminal    
    int stderrPipeFd[2];              // pipe used to read stderr
 
    pid_t pid;                        // pid of scp child process
    int exitStatus = 0;               // scp child process exist status
 
    // parse options
    int c;
    int digit_optind = 0;
    while (1) {
        int this_option_optind = optind ? optind : 1;
        int option_index = 0;
        static struct option long_options[] = {
            {"js",               no_argument,       0,  0 },
            {"startTemplate",    required_argument, 0,  0 },
            {"stdoutTemplate",   required_argument, 0,  0 },
            {"stderrTemplate",   required_argument, 0,  0 },
            {"progressTemplate", required_argument, 0,  0 },
            {"endTemplate",      required_argument, 0,  0 },
            {0,         0,                 0,  0 }
        };
 
        c = getopt_long(argc, argv, "?", long_options, &option_index);
        if (c == -1) { break; }
        switch (c) {
            case 0:
                switch (option_index) {
                    case 0: 
                        ESCAPE_JS = 1;
                        startTemplate = JS_START_TEMPLATE;
                        stdoutTemplate = JS_STDOUT_TEMPLATE;
                        stderrTemplate = JS_STDERR_TEMPLATE;
                        progressTemplate = JS_PROGRESS_TEMPLATE;
                        endTemplate = JS_END_TEMPLATE;
                        break;
                    case 1: startTemplate = optarg; break;
                    case 2: stdoutTemplate = optarg; break;
                    case 3: stderrTemplate = optarg; break;
                    case 4: progressTemplate = optarg; break;
                    case 5: endTemplate = optarg; break;
                    default:
                        fprintf(stderr, "getopt returned option_index %d\n", option_index);
                        exit(1);   
                }
                break;
            case '?':
                usage(); 
                exit(1);
                break;
            default:
                fprintf(stderr, "getopt returned character code 0%o '%c'\n", c, c);
                exit(1);
        }
    }
    if (optind >= argc) {
       fprintf(stderr, "You must supply options to 'scp' after the '--' command line-argument\n");
       usage();
       exit(1);
    }
 
    // replace user-specified placeholders (e.g. %s) with positional placeholders (e.g. {0})
    /* str_replace calls malloc for each result string */
    stdoutTemplate = str_replace(stdoutTemplate, "%s", "{0}");
    stderrTemplate = str_replace(stderrTemplate, "%s", "{0}");
    progressTemplate = str_replace(progressTemplate, "%f", "{0}"); // feel free to tell me that this leaks memory on each str_replace
    progressTemplate = str_replace(progressTemplate, "%p", "{1}");
    progressTemplate = str_replace(progressTemplate, "%t", "{2}");
    progressTemplate = str_replace(progressTemplate, "%s", "{3}");
    progressTemplate = str_replace(progressTemplate, "%e", "{4}");
    endTemplate = str_replace(endTemplate, "%c", "{0}");
    // TODO: check return values for NULL (malloc error)
 
    // create pipe for stderr
    pipe(stderrPipeFd);
 
    // get a pseudoterminal
    pid = forkpty (&stdoutPtyFd, NULL, NULL, NULL);
    if (pid == 0) {
        /* CHILD */
        close(stderrPipeFd[0]);    // close reading end in the child
        // dup2(stderrPipeFd[1], 1);  // send stdout to the pipe (unused; stdout is the pseudoterminal fd)
        dup2(stderrPipeFd[1], 2);  // send stderr to the pipe
        close(stderrPipeFd[1]);    // this descriptor is no longer needed
 
        // pass arguments "scp" then argv[optind] to argv[argc]
        // argv[optind-1] should be pointing to the '--' argument so we replace it with "scp"
        argv[optind-1] = "scp";       
        execvp("scp", &argv[optind-1]);
        perror("execvp");
        _exit (2);
    } else if (pid == -1) {
        /* ERROR */
        perror("forkpty");
        exit (1);
    } else {
        /* PARENT */
        close(stderrPipeFd[1]);  // close the write end of the pipe in the parent
        fd_set selectFds;
        int retval;
 
        /* Watch stdout (stdoutPtyFd) or stderr (stderrPipeFd[0]) to see when it has input. */
        FD_ZERO(&selectFds);
        FD_SET(stdoutPtyFd, &selectFds);
        FD_SET(stderrPipeFd[0], &selectFds);
 
        while (!(doneStdout || doneStderr)) {
            // NB: first param isn't the number of file descriptors, it's the highest file descriptor + 1
            // FD_SETSIZE may not be terribly efficient, but should work here 
            retval = select(FD_SETSIZE, &selectFds, NULL, NULL, NULL);
            if (retval==-1) { 
                perror("select()"); exit(1);
            } else {
                if (FD_ISSET(stderrPipeFd[0], &selectFds) && !doneStderr) {
                    stderrc = read(stderrPipeFd[0], &stderrBuf[stderrIdx], 1);
                    //printf ("this shouldnt be more than 1: %d, %c\n", stderrc, stderrBuf[stderrIdx]);
                    if (stderrc <= 0) {
                        doneStderr=1;
                    } else {
                        stderrIdx += stderrc; 
                        if (stderrBuf[stderrIdx-1]=='\r' || stderrBuf[stderrIdx-1]=='\n' || stderrIdx==STDERR_BUFSIZE-1) {
                            stderrBuf[stderrIdx]=0;
                            printfmt(1, stderrTemplate, stderrBuf);
                            stderrIdx=0; 
                        }
                    }  
                }
 
                if (FD_ISSET(stdoutPtyFd, &selectFds) && !doneStdout) {
                    stdoutc = read(stdoutPtyFd, &stdoutBuf[stdoutIdx], 1);
                    // printf ("this shouldnt be more than 1: %d, %c\n", stdoutc, stdoutBuf[stdoutIdx]);
                    if (stdoutc <= 0) {
                        doneStdout=1; ;
                    } else {
                        stdoutIdx += stdoutc;
                        if (stdoutBuf[stdoutIdx-1]=='\r' || stdoutBuf[stdoutIdx-1]=='\n' || stdoutIdx==STDOUT_BUFSIZE-1) {
                            stdoutBuf[stdoutIdx]=0;
 
                            // stdoutBuf will be something like
                            // something.tar.gz                                1% 2112KB   2.1MB/s   00:50 ETA
                            // unless we exceed BUFSIZE. which will never happen. unless we get a horrendously long filename perhaps
 
                            // fieldBuf is set to stdoutBuf, but with spaces replaced with char(0)s
                            // fields[n] are the character positions of the nth field in fieldBuf
                            strncpy(fieldBuf, stdoutBuf, STDOUT_BUFSIZE);
                            int fieldIdx=0, fields[5], i=0;
                            for (i=0; i<stdoutIdx; i++) {
                                if (fieldBuf[i]==' ' || fieldBuf[i]=='\r') { 
                                    fieldBuf[i]=0; 
                                } else {
                                    if (fieldIdx<5 && (i==0 || fieldBuf[i-1]==0)) {
                                        fields[fieldIdx++]=i; 
                                    }
                                }
                            }
 
                            // right number of fields and a percentage character in the right place? 
                            if (fieldIdx==5 && fieldBuf[fields[1] + strlen(&fieldBuf[fields[1]])-1]=='%') {
                                fieldBuf[fields[1] + strlen(&fieldBuf[fields[1]])-1]=0;
                                if (!shownStartTemplate) {
                                    shownStartTemplate = 1;
                                    printf("%s", startTemplate);
                                }  
                                printfmt(5, progressTemplate, &fieldBuf[fields[0]],
                                  &fieldBuf[fields[1]],&fieldBuf[fields[2]],&fieldBuf[fields[3]],&fieldBuf[fields[4]]);
                            } else {
                                // don't generate empty lines on stdout
                                if ((strncmp(stdoutBuf, "\n", STDOUT_BUFSIZE)!=0) &&
                                    (strncmp(stdoutBuf, "\r", STDOUT_BUFSIZE)!=0) ) {
                                    printfmt(1, stdoutTemplate, stdoutBuf);
                                }   
                            }
                            fflush(stdout);
 
                            stdoutIdx=0; 
                        }
                    }  
                }  
            }
 
            // reset set
            FD_ZERO(&selectFds);
            if (!doneStdout) { FD_SET(stdoutPtyFd, &selectFds); }
            if (!doneStderr) { FD_SET(stderrPipeFd[0], &selectFds); }
        }  
 
        // both stdout & stderr have been closed, dump whatever's left
        if (stderrIdx!=0) { stderrBuf[stderrIdx]=0; printfmt(1, stderrTemplate, stderrBuf); }
        if (stdoutIdx!=0) { stdoutBuf[stdoutIdx]=0; printfmt(1, stdoutTemplate, stdoutBuf); }
        fflush(stdout);
 
        // get the exit status of the scp process. 
        pid_t w = waitpid (pid, &exitStatus, 0);
        if (w==-1) { perror("waitpid"); exit(1); }
 
        if (WIFEXITED(exitStatus)) {
          sprintf(stdoutBuf, "%d", WEXITSTATUS(exitStatus));
          printfmt(1, endTemplate, stdoutBuf);
          exit (WEXITSTATUS(exitStatus));  // propagate exitStatus
 
        } else if (WIFSIGNALED(exitStatus)) {
          // display a signal as -(signal number)
          sprintf(stdoutBuf, "-%d", WTERMSIG(exitStatus));
          printfmt(1, endTemplate, stdoutBuf); 
          exit (1);
 
        } else {
          // something weird
          exit (1);
        }
    }
 
    // this should should never be executed 
    exit (0);
}

/* (c) 2013 randomnoun. All Rights Reserved. This work is licensed under a * BSD Simplified License. (http://www.randomnoun.com/bsd-simplified.html) */ /* scpwrap.c * * $Id: scpwrap.c,v 1.19 2013-11-03 22:24:42 knoxg Exp $ * * Wrapper for scp which translates progress text, e.g: * * something.tar.gz 1% 2112KB 2.1MB/s 00:50 ETA * * into javascript, which can then display and update a progress bar. * * Call this thing from a bash script or other process/thread that will dump its output * directly into an iframe which is surrounded by <script> elements. * * COMPILING * * Remember to include the util library when compiling ! * i.e. gcc -lutil scpwrap.c -oscpwrap * * as of 2013, you could try doing this instead, for reasons that I don't particularly understand: * gcc -Wl,--no-as-needed -lutil scpwrap.c -oscpwrap * * Modified from http://cwshep.blogspot.com/2009/06/showing-scp-progress-using-zenity.html * * This program relies on the fact that even if stdout performs checks to see whether * it is running on a tty, these checks are not perfomed for stderr ( search for 'isatty' in * http://www.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/scp.c?rev=1.178;content-type=text%2Fx-cvsweb-markup ) * * see http://www.linuxquestions.org/questions/programming-9/problem-with-child-process-658750/#post3229496 http://www.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/scp.c?rev=1.178;content-type=text%2Fx-cvsweb-markup http://stackoverflow.com/questions/2605130/redirecting-exec-output-to-a-buffer-or-file http://stackoverflow.com/questions/903864/how-to-exit-a-child-process-and-return-its-status-from-execvp */ #include <unistd.h> #include <sys/types.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <stdarg.h> #include <sys/select.h> #include <getopt.h> /* let's say that the things we're going to replace in here are: * %f - filename * %p - progress (0-100) and "%" character * %t - transfer size ("2112KB") * %s - speed ("2.1MB/s" or however progressmeter.c does things) * %e - ETA ("--:--" or "05:23"), hh:mm:ss, or however progressmeter.c does things) */ // if we get more than this number of bytes on stdout/stderr without a newline, // then they will be emitted in >1 stdout/stderr template #define STDERR_BUFSIZE 1024 #define STDOUT_BUFSIZE 1024 // enabled by --js option; stdout/stderr will be javascript-String escaped static int ESCAPE_JS = 0; static char* TXT_STDOUT_TEMPLATE = ""; static char* TXT_STDERR_TEMPLATE = ""; static char* TXT_START_TEMPLATE = ""; static char* TXT_PROGRESS_TEMPLATE = "%p\n"; static char* TXT_END_TEMPLATE = ""; static char* JS_STDOUT_TEMPLATE = "ui.addOutput(\"%s\");\n"; static char* JS_STDERR_TEMPLATE = "ui.addOutputError(\"%s\");\n"; static char* JS_START_TEMPLATE = "var sp = ui.startScpProgress();\n"; static char* JS_PROGRESS_TEMPLATE = "sp.setProgress(\"%f\", %p, \"%t\", \"%s\", \"%e\");\n"; static char* JS_END_TEMPLATE = "ui.stopScpProgress(%c);\n"; /** convert ASCII to javascript escape. see http://rishida.net/tools/conversion */ void printjs (char *str) { int i=0; for (i=0; str[i]!=0; i++) { switch (str[i]) { case 0: printf("\\0"); break; case 8: printf("\\b"); break; case 9: printf("\\t"); break; case 10: printf("\\n"); break; case 13: printf("\\r"); break; case 11: printf("\\v"); break; case 12: printf("\\f"); break; case 34: printf("\\\""); break; case 39: printf("\\\'"); break; case 92: printf("\\\\"); break; default: if (str[i]>0x1f && str[i]<0x7F) { printf("%c", str[i]); } else { printf("\\u%04x", (int) str[i]); } } } } /** Take a format string ("template") containing 0 or more tokens in the form "{n}", and replace each token with the n'th optional parameter to this function. The string after token replacement is sent to stdout. The nparam must be set to the number of optional parameters supplied. All optional parameters must be of type char*. e.g. printfmt(3, "first: {0}, second: {1}, third: {2}", "a", "b", "c"); produces the output "first: a, second: b, third: c" C-like escapes ("\n", "\t" etc) in the format string will be converted into their appropriate character if JS_ESCAPE is true, then values in the optional parameters will be Javascript-escaped (i.e. the character '\n' will be converted to the two-character "\n" escape) */ int printfmt (int nparam, const char *format, ...) { va_list ap; int i = 0; int inBrace = 0, inEscape = 0; int paramIdx = 0; char *params[nparam]; va_start(ap, format); for (i=0; i<nparam; i++) { params[i] = va_arg(ap, char *); } va_end(ap); for (i=0; format[i]!=0; i++) { if (inEscape) { inEscape = 0; switch (format[i]) { case 'n': printf("\n"); break; case 'r': printf("\r"); break; case 't': printf("\t"); break; case '{': printf("{"); break; case '\\': printf("\\"); break; default: /* unknown escape, just print character */ printf("%c", format[i]); } } else if (inBrace) { switch (format[i]) { case '0': case '1': case '2': case '3': case '4': case '5': case '6': case '7': case '8': case '9': paramIdx = paramIdx*10 + (format[i]-'0'); break; case '}': inBrace = 0; if (ESCAPE_JS) { printjs(params[paramIdx]); } else { printf("%s", params[paramIdx]); } break; default: // weird thing in brace return -1; } } else { switch (format[i]) { case '{': paramIdx = 0; inBrace = 1; break; case '\\': inEscape = 1; break; default: printf("%c", format[i]); } } } } /** send usage information to stdout */ void usage() { printf("usage: scpwrap [options] -- scp-options \n" "Where options are:\n" " --js use javascript default templates, and javascript-escape output strings\n" " --stdoutTemplate txt template to use for unrecognised stdout text\n" " --stderrTemplate txt template to use for unrecognised stderr text\n" " --startTemplate txt text to display before the first progressTemplate appears\n" " --progressTemplate txt template to use for copy progress output\n" " --endTemplate txt template to use after copy completes\n" "The following placeholders can be used in progress templates:\n" " %%f filename\n" " %%p progress amount (0-100)\n" " %%t transfer size (e.g. \"2112KB\")\n" " %%s speed (e.g. \"2.1MB/s\")\n" " %%e ETA (e.g. \"--:--\" or \"05:23\")\n" "The following placeholder can be used in stdout/stderr templates:\n" " %%s text string\n" "The following placeholder can be used in the endTemplate:\n" " %%c exit code\n" "\n" "See the 'scpwrap' and 'scp' man page for more options. Example usage:\n" " scpwrap --js -- -i identityfile user@host1:file1 user@host2:file2\n" ); // This is a program that wraps scp and prints out the numeric progress on separate lines.\n"); fflush(stdout); } /** replace all instances of a substring in a string with a relacement string. memory for the returned string will be reserved via malloc see http://coding.debuntu.org/c-implementing-str_replace-replace-all-occurrences-substring */ char *str_replace ( const char *string, const char *substr, const char *replacement ){ char *tok = NULL; char *newstr = NULL; char *oldstr = NULL; /* if either substr or replacement is NULL, duplicate string and let caller handle it */ if ( substr == NULL || replacement == NULL ) { return strdup (string); } newstr = strdup (string); while ( (tok = strstr ( newstr, substr ))) { oldstr = newstr; newstr = malloc ( strlen ( oldstr ) - strlen ( substr ) + strlen ( replacement ) + 1 ); /*failed to alloc mem, free old string and return NULL */ if ( newstr == NULL ) { free (oldstr); return NULL; } memcpy ( newstr, oldstr, tok - oldstr ); memcpy ( newstr + (tok - oldstr), replacement, strlen ( replacement ) ); memcpy ( newstr + (tok - oldstr) + strlen( replacement ), tok + strlen ( substr ), strlen ( oldstr ) - strlen ( substr ) - ( tok - oldstr ) ); memset ( newstr + strlen ( oldstr ) - strlen ( substr ) + strlen ( replacement ) , 0, 1 ); free (oldstr); } return newstr; } /** main */ main(int argc, char **argv) { char *startTemplate = TXT_START_TEMPLATE; char *stdoutTemplate = TXT_STDOUT_TEMPLATE; char *stderrTemplate = TXT_STDERR_TEMPLATE; char *progressTemplate = TXT_PROGRESS_TEMPLATE; char *endTemplate = TXT_END_TEMPLATE; int shownStartTemplate = 0; // set to 1 when startTemplate is printed char stderrBuf[STDERR_BUFSIZE]; // stderr capture buffer char stdoutBuf[STDOUT_BUFSIZE]; // stdout capture buffer char fieldBuf[STDOUT_BUFSIZE]; // as per stdout, with spaces replaced with \0x0 int stderrIdx = 0, stdoutIdx = 0; // indices within stderrBuf/stdoutBuf ssize_t stderrc, stdoutc; // count of bytes read into stderrBuf/stdoutBuf int doneStderr = 0, doneStdout=0; // set to 1 when stderr/stdout file descriptors have closed int stdoutPtyFd; // file descriptor for the master side of the stdout pseudoterminal int stderrPipeFd[2]; // pipe used to read stderr pid_t pid; // pid of scp child process int exitStatus = 0; // scp child process exist status // parse options int c; int digit_optind = 0; while (1) { int this_option_optind = optind ? optind : 1; int option_index = 0; static struct option long_options[] = { {"js", no_argument, 0, 0 }, {"startTemplate", required_argument, 0, 0 }, {"stdoutTemplate", required_argument, 0, 0 }, {"stderrTemplate", required_argument, 0, 0 }, {"progressTemplate", required_argument, 0, 0 }, {"endTemplate", required_argument, 0, 0 }, {0, 0, 0, 0 } }; c = getopt_long(argc, argv, "?", long_options, &option_index); if (c == -1) { break; } switch (c) { case 0: switch (option_index) { case 0: ESCAPE_JS = 1; startTemplate = JS_START_TEMPLATE; stdoutTemplate = JS_STDOUT_TEMPLATE; stderrTemplate = JS_STDERR_TEMPLATE; progressTemplate = JS_PROGRESS_TEMPLATE; endTemplate = JS_END_TEMPLATE; break; case 1: startTemplate = optarg; break; case 2: stdoutTemplate = optarg; break; case 3: stderrTemplate = optarg; break; case 4: progressTemplate = optarg; break; case 5: endTemplate = optarg; break; default: fprintf(stderr, "getopt returned option_index %d\n", option_index); exit(1); } break; case '?': usage(); exit(1); break; default: fprintf(stderr, "getopt returned character code 0%o '%c'\n", c, c); exit(1); } } if (optind >= argc) { fprintf(stderr, "You must supply options to 'scp' after the '--' command line-argument\n"); usage(); exit(1); } // replace user-specified placeholders (e.g. %s) with positional placeholders (e.g. {0}) /* str_replace calls malloc for each result string */ stdoutTemplate = str_replace(stdoutTemplate, "%s", "{0}"); stderrTemplate = str_replace(stderrTemplate, "%s", "{0}"); progressTemplate = str_replace(progressTemplate, "%f", "{0}"); // feel free to tell me that this leaks memory on each str_replace progressTemplate = str_replace(progressTemplate, "%p", "{1}"); progressTemplate = str_replace(progressTemplate, "%t", "{2}"); progressTemplate = str_replace(progressTemplate, "%s", "{3}"); progressTemplate = str_replace(progressTemplate, "%e", "{4}"); endTemplate = str_replace(endTemplate, "%c", "{0}"); // TODO: check return values for NULL (malloc error) // create pipe for stderr pipe(stderrPipeFd); // get a pseudoterminal pid = forkpty (&stdoutPtyFd, NULL, NULL, NULL); if (pid == 0) { /* CHILD */ close(stderrPipeFd[0]); // close reading end in the child // dup2(stderrPipeFd[1], 1); // send stdout to the pipe (unused; stdout is the pseudoterminal fd) dup2(stderrPipeFd[1], 2); // send stderr to the pipe close(stderrPipeFd[1]); // this descriptor is no longer needed // pass arguments "scp" then argv[optind] to argv[argc] // argv[optind-1] should be pointing to the '--' argument so we replace it with "scp" argv[optind-1] = "scp"; execvp("scp", &argv[optind-1]); perror("execvp"); _exit (2); } else if (pid == -1) { /* ERROR */ perror("forkpty"); exit (1); } else { /* PARENT */ close(stderrPipeFd[1]); // close the write end of the pipe in the parent fd_set selectFds; int retval; /* Watch stdout (stdoutPtyFd) or stderr (stderrPipeFd[0]) to see when it has input. */ FD_ZERO(&selectFds); FD_SET(stdoutPtyFd, &selectFds); FD_SET(stderrPipeFd[0], &selectFds); while (!(doneStdout || doneStderr)) { // NB: first param isn't the number of file descriptors, it's the highest file descriptor + 1 // FD_SETSIZE may not be terribly efficient, but should work here retval = select(FD_SETSIZE, &selectFds, NULL, NULL, NULL); if (retval==-1) { perror("select()"); exit(1); } else { if (FD_ISSET(stderrPipeFd[0], &selectFds) && !doneStderr) { stderrc = read(stderrPipeFd[0], &stderrBuf[stderrIdx], 1); //printf ("this shouldnt be more than 1: %d, %c\n", stderrc, stderrBuf[stderrIdx]); if (stderrc <= 0) { doneStderr=1; } else { stderrIdx += stderrc; if (stderrBuf[stderrIdx-1]=='\r' || stderrBuf[stderrIdx-1]=='\n' || stderrIdx==STDERR_BUFSIZE-1) { stderrBuf[stderrIdx]=0; printfmt(1, stderrTemplate, stderrBuf); stderrIdx=0; } } } if (FD_ISSET(stdoutPtyFd, &selectFds) && !doneStdout) { stdoutc = read(stdoutPtyFd, &stdoutBuf[stdoutIdx], 1); // printf ("this shouldnt be more than 1: %d, %c\n", stdoutc, stdoutBuf[stdoutIdx]); if (stdoutc <= 0) { doneStdout=1; ; } else { stdoutIdx += stdoutc; if (stdoutBuf[stdoutIdx-1]=='\r' || stdoutBuf[stdoutIdx-1]=='\n' || stdoutIdx==STDOUT_BUFSIZE-1) { stdoutBuf[stdoutIdx]=0; // stdoutBuf will be something like // something.tar.gz 1% 2112KB 2.1MB/s 00:50 ETA // unless we exceed BUFSIZE. which will never happen. unless we get a horrendously long filename perhaps // fieldBuf is set to stdoutBuf, but with spaces replaced with char(0)s // fields[n] are the character positions of the nth field in fieldBuf strncpy(fieldBuf, stdoutBuf, STDOUT_BUFSIZE); int fieldIdx=0, fields[5], i=0; for (i=0; i<stdoutIdx; i++) { if (fieldBuf[i]==' ' || fieldBuf[i]=='\r') { fieldBuf[i]=0; } else { if (fieldIdx<5 && (i==0 || fieldBuf[i-1]==0)) { fields[fieldIdx++]=i; } } } // right number of fields and a percentage character in the right place? if (fieldIdx==5 && fieldBuf[fields[1] + strlen(&fieldBuf[fields[1]])-1]=='%') { fieldBuf[fields[1] + strlen(&fieldBuf[fields[1]])-1]=0; if (!shownStartTemplate) { shownStartTemplate = 1; printf("%s", startTemplate); } printfmt(5, progressTemplate, &fieldBuf[fields[0]], &fieldBuf[fields[1]],&fieldBuf[fields[2]],&fieldBuf[fields[3]],&fieldBuf[fields[4]]); } else { // don't generate empty lines on stdout if ((strncmp(stdoutBuf, "\n", STDOUT_BUFSIZE)!=0) && (strncmp(stdoutBuf, "\r", STDOUT_BUFSIZE)!=0) ) { printfmt(1, stdoutTemplate, stdoutBuf); } } fflush(stdout); stdoutIdx=0; } } } } // reset set FD_ZERO(&selectFds); if (!doneStdout) { FD_SET(stdoutPtyFd, &selectFds); } if (!doneStderr) { FD_SET(stderrPipeFd[0], &selectFds); } } // both stdout & stderr have been closed, dump whatever's left if (stderrIdx!=0) { stderrBuf[stderrIdx]=0; printfmt(1, stderrTemplate, stderrBuf); } if (stdoutIdx!=0) { stdoutBuf[stdoutIdx]=0; printfmt(1, stdoutTemplate, stdoutBuf); } fflush(stdout); // get the exit status of the scp process. pid_t w = waitpid (pid, &exitStatus, 0); if (w==-1) { perror("waitpid"); exit(1); } if (WIFEXITED(exitStatus)) { sprintf(stdoutBuf, "%d", WEXITSTATUS(exitStatus)); printfmt(1, endTemplate, stdoutBuf); exit (WEXITSTATUS(exitStatus)); // propagate exitStatus } else if (WIFSIGNALED(exitStatus)) { // display a signal as -(signal number) sprintf(stdoutBuf, "-%d", WTERMSIG(exitStatus)); printfmt(1, endTemplate, stdoutBuf); exit (1); } else { // something weird exit (1); } } // this should should never be executed exit (0); }

Show codeHide code

scpwrap.groff

.\" 
.\" (c) 2013 randomnoun. All Rights Reserved. This work is licensed under a
.\" BSD Simplified License. (http://www.randomnoun.com/bsd-simplified.html)
.\" 
.\" scpwrap.groff
.\" $Id: scpwrap.groff,v 1.5 2013-10-31 06:35:47 knoxg Exp $
.\"
.\" Man page for the scpwrap utility
.\" 
.\" Process this file with
.\" 
.\"   visual pager:         groff -man -Tascii scpwrap.groff | less
.\"   HTML:                 groff -man -Thtml scpwrap.groff > scpwrap.html
.\"   HTML with hyperlinks: man2html -H code.randomnoun.com -M /man2html scpwrap.groff > scpwrap.html
.\"   fixed width HTML:     groff -man -Tascii scpwrap.groff | perl -i -pe 's!(.)\x08\g1!<b>\1</b>!g; s!</b>(\s*)<b>!\1!g; s!_\x08(.)!<u>\1</u>!g;; s!</u>(\s*)<u>!\1!g;' > scpwrap.html
.\"
.TH SCPWRAP 1 "OCTOBER 2013" vmaint "User Commands"
.SH NAME
scpwrap \- a wrapper for scp to convert its output into machine-readable format
.SH SYNOPSIS
.nh \" no hyphenating
.na \" no right-margin adjustment
.B scpwrap [--js] [--stdoutTemplate 
.I template
.B ] [--stderrTemplate 
.I template
.B ] [--startTemplate
.I template
.B ] [--progressTemplate
.I template
.B ] [--endTemplate
.I template
.B ] --
.I scp-options
.B ...
.ad \" re-enable right-margin adjustment (i.e. full justification)
.SH DESCRIPTION
.B scpwrap
wraps the 
.BR scp (1)
command such that the output generated by that command is
converted into a format more easily processed by other
commands.  
.P
.BR scp (1) 
progress information is captured and converted into a  
machine-readable format for use in more sophisticated user interfaces
(which in turn would require even more effort to turn back into
a machine-readable format). And so the circle of life continues. 
.P
The built-in text and javascript output templates are suitable for 
.BR zenity (1),
or some as-yet-to-be-documented javascript processor, respectively.
See the \fBEXAMPLES\fR section to see the general syntax of these defaults. 
.SH OPTIONS
.IP \fB--js\fR
Use the default javascript templates rather than the default
text templates (see the \fBDEFAULTS\fR section below).
 
This switch also enables javascript-output escaping
(see the \fBJAVASCRIPT OUTPUT ESCAPES\fR section below). 
.IP "\fB--stdoutTemplate\fR \fItemplate\fR"
Use the supplied 
.I template
to generate machine-readable text when unrecognised output is 
detected on stdout. The placeholder
\fB%s\fR in the template will be replaced by the text received on stdout.
.IP "\fB--stderrTemplate\fR \fItemplate\fR"
Use the supplied 
.I template
to generate machine-readable text when any output is
detected on stderr (e.g. to display text contained in the remote system's 
.B sshd (8) 
Banner file). The placeholder
\fB%s\fR in the template will be replaced by the text received on stderr.
.IP "\fB--startTemplate\fR \fItemplate\fR"
Use the supplied 
.I template
to generate machine-readable text as scp progress begins.
.IP "\fB--progressTemplate\fR \fItemplate\fR"
Use the supplied 
.I template
to generate machine-readable text as scp progresses. 
The placeholders \fB%f\fR, \fB%p\fR, \fB%t\fR,
\fB%s\fR and \fB%e\fR are available (see the \fBPLACEHOLDERS\fR section below) 
.IP "\fB--endTemplate\fR \fItemplate\fR"
Use the supplied 
.I template
when scp completes. The \fI%c\fR placeholder is available.
.IP \fIscp-options\fR
these command-line options are passed directly to 
.BR scp (1)
and should conform to the expected syntax of that command.
.SH PLACEHOLDERS
Placeholders can (and should) be embedded in template strings, which will
be replaced by the appropriate value when the stdout/stderr of  
.BR scp (1)
is parsed by this utility. Most placeholders are only available
in specific template types.
.P
Recognised placeholders are:
.TP 5
\fB%s\fR
Free-form stdout or stderr text
.TP
\fB%f\fR
The name of the file being copied (e.g. \fBtest.txt\fR)
.TP
\fB%p\fR
The percentage progress of the current file being copied (e.g. \fB13\fR)
.TP
\fB%t\fR
The transfer size of the current file being copied (e.g. \fB2112KB\fR)
.TP
\fB%s\fR
The transfer speed of the current file being copied (e.g. \fB2.1MB/s\fR)
.TP
\fB%e\fR
The ETA of the current file being copied (e.g. \fB--:-- or \fB05:23\fR)
.TP
\fB%c\fR
The exit code of the 
.BR scp (1)
process (e.g. \fB0\fR), or a negative number if the process was 
terminated by a signal (the absolute value of this number corresponds to 
the signal number received).
.SH PLACEHOLDER ESCAPES
The following escape sequences are recognised in placeholder strings supplied
on the command-line:
.TP 5
\fB\\n\fR
Newline
.TP
\fB\\r\fR
Carriage return
.TP
\fB\\t\fR
Tab character
.TP
\fB\\\\\fR
Backslash
.SH JAVASCRIPT OUTPUT ESCAPES
If the text generated by scp contains characters that would cause 
javascript parsing errors, and the \fB--js\fR command-line option is in
effect, then the strings substituted into the \fB%s\fR placeholder will
be javascript-escaped (e.g. newlines will be replaced by \fB\\n\fR, 
characters above ASCII codepoint 127 will be replaced by \fB\\u0000\fR-style
sequences.)  
.SH DEFAULTS
The following defaults are used, in the absence of any template overrides:
.SS Text output (without --js parameter)
Note that many templates are simply "" (i.e. the empty string, 
therefore no output is generated)
.TP 20
\fB--stdoutTemplate\fR
""
.TP
\fB--stderrTemplate\fR
""
.TP
\fB--startTemplate\fR
""
.TP
\fB--progressTemplate\fR
"%p\\n"
.TP
\fB--endTemplate\fR
"" 
.SS Javascript output (with --js parameter)
The default javascript output relies on a script-visible 'ui' object, 
as shown below.
.TP 20
\fB--stdoutTemplate\fR
"ui.addOutput(\\"%s\\");\\n"
.TP
\fB--stderrTemplate\fR
"ui.addOutputError(\\"%s\\");\\n";
.TP
\fB--startTemplate\fR
"var sp = ui.startScpProgress();\\n";
.TP
\fB--progressTemplate\fR
.nf
"sp.setProgress(\\"%f\\", %p, \\"%t\\", \\"%s\\", \\"%e\\");\\n";
.fi
.TP
\fB--endTemplate\fR
"ui.stopScpProgress(%c);\\n";
.SH EXIT STATUS
The \fBscpwrap\fR command will return the same exit code as the child
\fBscp\fR process; i.e. it exits 0 on success, and >0 if an error occurs. 
.P
If a signal interrupts processing of the child process, then \fBscpwrap\fR 
terminates with an exit status of 1. The signal number can be determined 
using the \fB%c\fR placeholder to the \fB--endTemplate\fR template.   
.SH EXAMPLES
.SS Example 1 (text output)
The command
 
.RS
.nf
scpwrap -- -i key.pem somefile.tar.gz \\
  user@somehost:/home/user/somefile.tar.gz
.fi
.RE
 
might produce output something similar to the following:
 
.RS
.nf
0
45
47
49
 
  ... 20 lines omitted ...
 
98
100
.fi
.RE
.P
 
.SS Example 2 (javascript output)
The command
 
.RS
.nf
scpwrap --js -- -i key.pem somefile.tar.gz \\
  user@somehost:/home/user/somefile.tar.gz \\
.fi
.RE
 
might produce output something similar to the following:
 
.RS
.nf
ui.addOutputError("NOTICE TO USERS\\n");
ui.addOutputError("\\n");
ui.addOutputError("This service is for authorised clients only.\\n");
ui.addOutputError("\\n");
ui.addOutputError("This computer system is the private property of its owner, whether\\n");
ui.addOutputError("individual, corporate or government.  It is for authorized use only.\\n");
ui.addOutputError("Users (authorised or unauthorised) have no explicit or implicit\\n");
ui.addOutputError("expectation of privacy.\\n");
ui.addOutputError("\\n");
ui.addOutputError("It is a criminal offence to:\\n");
ui.addOutputError("  i. Obtain access to data without authority\\n");
ui.addOutputError("       (Penalty 2 years imprisonment)\\n");
ui.addOutputError("  ii Damage, delete, alter or insert data without authority\\n");
ui.addOutputError("       (Penalty 10 years imprisonment)\\n");
ui.addOutputError("\\n");
ui.addOutputError("For more information, see http://www.randomnoun.com/login-banner.html\\n");
var sp = ui.startScpProgress();
sp.setProgress("somefile.tar.gz", 0, "0", "0.0KB/s", "--:--");
sp.setProgress("somefile.tar.gz", 45, "2112KB", "2.1MB/s", "00:01");
sp.setProgress("somefile.tar.gz", 47, "2208KB", "1.9MB/s", "00:01");
sp.setProgress("somefile.tar.gz", 49, "2320KB", "1.7MB/s", "00:01");
sp.setProgress("somefile.tar.gz", 52, "2448KB", "1.5MB/s", "00:01");
sp.setProgress("somefile.tar.gz", 54, "2576KB", "1.4MB/s", "00:01");
 
  ... 20 lines omitted ...
 
sp.setProgress("somefile.tar.gz", 98, "4624KB", "266.7KB/s", "00:00");
sp.setProgress("somefile.tar.gz", 100, "4693KB", "187.7KB/s", "00:25");
ui.stopScpProgress(0);
.fi
.RS
 
.SS Example 3 (custom javascript output)
The command
 
.RS
.nf
scpwrap --js --stderrTemplate '' --stdoutTemplate '' \\
  --startTemplate '' --endTemplate '' \\
  --progressTemplate 'setProgress(%p);\\n' 
  -- -i key.pem somefile.tar.gz \\
  user@somehost:/home/user/somefile.tar.gz 
.fi
.RE
 
might produce output something similar to the following:
 
.RS
.nf
setProgress(0);
setProgress(45);
setProgress(47);
 
   ... 20 lines omitted ...
 
setProgress(98);
setProgress(100);
.fi
.RE
.SH BUGS
.P
It might be preferable to get the default strings from something in /etc
.P
The whole thing's a bit pointless
.SH AUTHOR
Greg Knox <knoxg at randomnoun dot com>
.SH LICENCE
(c) 2013 randomnoun. All Rights Reserved. This work is licensed under a
BSD Simplified License. (http://www.randomnoun.com/bsd-simplified.html)
.SH "SEE ALSO"
.BR vmaint (1),
http://www.randomnoun.com/wp/2013/10/31/progress-bars/

.\" .\" (c) 2013 randomnoun. All Rights Reserved. This work is licensed under a .\" BSD Simplified License. (http://www.randomnoun.com/bsd-simplified.html) .\" .\" scpwrap.groff .\" $Id: scpwrap.groff,v 1.5 2013-10-31 06:35:47 knoxg Exp $ .\" .\" Man page for the scpwrap utility .\" .\" Process this file with .\" .\" visual pager: groff -man -Tascii scpwrap.groff | less .\" HTML: groff -man -Thtml scpwrap.groff > scpwrap.html .\" HTML with hyperlinks: man2html -H code.randomnoun.com -M /man2html scpwrap.groff > scpwrap.html .\" fixed width HTML: groff -man -Tascii scpwrap.groff | perl -i -pe 's!(.)\x08\g1!\1!g; s!(\s*)!\1!g; s!_\x08(.)!\1!g;; s!(\s*)!\1!g;' > scpwrap.html .\" .TH SCPWRAP 1 "OCTOBER 2013" vmaint "User Commands" .SH NAME scpwrap \- a wrapper for scp to convert its output into machine-readable format .SH SYNOPSIS .nh \" no hyphenating .na \" no right-margin adjustment .B scpwrap [--js] [--stdoutTemplate .I template .B ] [--stderrTemplate .I template .B ] [--startTemplate .I template .B ] [--progressTemplate .I template .B ] [--endTemplate .I template .B ] -- .I scp-options .B ... .ad \" re-enable right-margin adjustment (i.e. full justification) .SH DESCRIPTION .B scpwrap wraps the .BR scp (1) command such that the output generated by that command is converted into a format more easily processed by other commands. .P .BR scp (1) progress information is captured and converted into a machine-readable format for use in more sophisticated user interfaces (which in turn would require even more effort to turn back into a machine-readable format). And so the circle of life continues. .P The built-in text and javascript output templates are suitable for .BR zenity (1), or some as-yet-to-be-documented javascript processor, respectively. See the \fBEXAMPLES\fR section to see the general syntax of these defaults. .SH OPTIONS .IP \fB--js\fR Use the default javascript templates rather than the default text templates (see the \fBDEFAULTS\fR section below). This switch also enables javascript-output escaping (see the \fBJAVASCRIPT OUTPUT ESCAPES\fR section below). .IP "\fB--stdoutTemplate\fR \fItemplate\fR" Use the supplied .I template to generate machine-readable text when unrecognised output is detected on stdout. The placeholder \fB%s\fR in the template will be replaced by the text received on stdout. .IP "\fB--stderrTemplate\fR \fItemplate\fR" Use the supplied .I template to generate machine-readable text when any output is detected on stderr (e.g. to display text contained in the remote system's .B sshd (8) Banner file). The placeholder \fB%s\fR in the template will be replaced by the text received on stderr. .IP "\fB--startTemplate\fR \fItemplate\fR" Use the supplied .I template to generate machine-readable text as scp progress begins. .IP "\fB--progressTemplate\fR \fItemplate\fR" Use the supplied .I template to generate machine-readable text as scp progresses. The placeholders \fB%f\fR, \fB%p\fR, \fB%t\fR, \fB%s\fR and \fB%e\fR are available (see the \fBPLACEHOLDERS\fR section below) .IP "\fB--endTemplate\fR \fItemplate\fR" Use the supplied .I template when scp completes. The \fI%c\fR placeholder is available. .IP \fIscp-options\fR these command-line options are passed directly to .BR scp (1) and should conform to the expected syntax of that command. .SH PLACEHOLDERS Placeholders can (and should) be embedded in template strings, which will be replaced by the appropriate value when the stdout/stderr of .BR scp (1) is parsed by this utility. Most placeholders are only available in specific template types. .P Recognised placeholders are: .TP 5 \fB%s\fR Free-form stdout or stderr text .TP \fB%f\fR The name of the file being copied (e.g. \fBtest.txt\fR) .TP \fB%p\fR The percentage progress of the current file being copied (e.g. \fB13\fR) .TP \fB%t\fR The transfer size of the current file being copied (e.g. \fB2112KB\fR) .TP \fB%s\fR The transfer speed of the current file being copied (e.g. \fB2.1MB/s\fR) .TP \fB%e\fR The ETA of the current file being copied (e.g. \fB--:-- or \fB05:23\fR) .TP \fB%c\fR The exit code of the .BR scp (1) process (e.g. \fB0\fR), or a negative number if the process was terminated by a signal (the absolute value of this number corresponds to the signal number received). .SH PLACEHOLDER ESCAPES The following escape sequences are recognised in placeholder strings supplied on the command-line: .TP 5 \fB\\n\fR Newline .TP \fB\\r\fR Carriage return .TP \fB\\t\fR Tab character .TP \fB\\\\\fR Backslash .SH JAVASCRIPT OUTPUT ESCAPES If the text generated by scp contains characters that would cause javascript parsing errors, and the \fB--js\fR command-line option is in effect, then the strings substituted into the \fB%s\fR placeholder will be javascript-escaped (e.g. newlines will be replaced by \fB\\n\fR, characters above ASCII codepoint 127 will be replaced by \fB\\u0000\fR-style sequences.) .SH DEFAULTS The following defaults are used, in the absence of any template overrides: .SS Text output (without --js parameter) Note that many templates are simply "" (i.e. the empty string, therefore no output is generated) .TP 20 \fB--stdoutTemplate\fR "" .TP \fB--stderrTemplate\fR "" .TP \fB--startTemplate\fR "" .TP \fB--progressTemplate\fR "%p\\n" .TP \fB--endTemplate\fR "" .SS Javascript output (with --js parameter) The default javascript output relies on a script-visible 'ui' object, as shown below. .TP 20 \fB--stdoutTemplate\fR "ui.addOutput(\\"%s\\");\\n" .TP \fB--stderrTemplate\fR "ui.addOutputError(\\"%s\\");\\n"; .TP \fB--startTemplate\fR "var sp = ui.startScpProgress();\\n"; .TP \fB--progressTemplate\fR .nf "sp.setProgress(\\"%f\\", %p, \\"%t\\", \\"%s\\", \\"%e\\");\\n"; .fi .TP \fB--endTemplate\fR "ui.stopScpProgress(%c);\\n"; .SH EXIT STATUS The \fBscpwrap\fR command will return the same exit code as the child \fBscp\fR process; i.e. it exits 0 on success, and >0 if an error occurs. .P If a signal interrupts processing of the child process, then \fBscpwrap\fR terminates with an exit status of 1. The signal number can be determined using the \fB%c\fR placeholder to the \fB--endTemplate\fR template. .SH EXAMPLES .SS Example 1 (text output) The command .RS .nf scpwrap -- -i key.pem somefile.tar.gz \\ user@somehost:/home/user/somefile.tar.gz .fi .RE might produce output something similar to the following: .RS .nf 0 45 47 49 ... 20 lines omitted ... 98 100 .fi .RE .P .SS Example 2 (javascript output) The command .RS .nf scpwrap --js -- -i key.pem somefile.tar.gz \\ user@somehost:/home/user/somefile.tar.gz \\ .fi .RE might produce output something similar to the following: .RS .nf ui.addOutputError("NOTICE TO USERS\\n"); ui.addOutputError("\\n"); ui.addOutputError("This service is for authorised clients only.\\n"); ui.addOutputError("\\n"); ui.addOutputError("This computer system is the private property of its owner, whether\\n"); ui.addOutputError("individual, corporate or government. It is for authorized use only.\\n"); ui.addOutputError("Users (authorised or unauthorised) have no explicit or implicit\\n"); ui.addOutputError("expectation of privacy.\\n"); ui.addOutputError("\\n"); ui.addOutputError("It is a criminal offence to:\\n"); ui.addOutputError(" i. Obtain access to data without authority\\n"); ui.addOutputError(" (Penalty 2 years imprisonment)\\n"); ui.addOutputError(" ii Damage, delete, alter or insert data without authority\\n"); ui.addOutputError(" (Penalty 10 years imprisonment)\\n"); ui.addOutputError("\\n"); ui.addOutputError("For more information, see http://www.randomnoun.com/login-banner.html\\n"); var sp = ui.startScpProgress(); sp.setProgress("somefile.tar.gz", 0, "0", "0.0KB/s", "--:--"); sp.setProgress("somefile.tar.gz", 45, "2112KB", "2.1MB/s", "00:01"); sp.setProgress("somefile.tar.gz", 47, "2208KB", "1.9MB/s", "00:01"); sp.setProgress("somefile.tar.gz", 49, "2320KB", "1.7MB/s", "00:01"); sp.setProgress("somefile.tar.gz", 52, "2448KB", "1.5MB/s", "00:01"); sp.setProgress("somefile.tar.gz", 54, "2576KB", "1.4MB/s", "00:01"); ... 20 lines omitted ... sp.setProgress("somefile.tar.gz", 98, "4624KB", "266.7KB/s", "00:00"); sp.setProgress("somefile.tar.gz", 100, "4693KB", "187.7KB/s", "00:25"); ui.stopScpProgress(0); .fi .RS .SS Example 3 (custom javascript output) The command .RS .nf scpwrap --js --stderrTemplate '' --stdoutTemplate '' \\ --startTemplate '' --endTemplate '' \\ --progressTemplate 'setProgress(%p);\\n' -- -i key.pem somefile.tar.gz \\ user@somehost:/home/user/somefile.tar.gz .fi .RE might produce output something similar to the following: .RS .nf setProgress(0); setProgress(45); setProgress(47); ... 20 lines omitted ... setProgress(98); setProgress(100); .fi .RE .SH BUGS .P It might be preferable to get the default strings from something in /etc .P The whole thing's a bit pointless .SH AUTHOR Greg Knox <knoxg at randomnoun dot com> .SH LICENCE (c) 2013 randomnoun. All Rights Reserved. This work is licensed under a BSD Simplified License. (http://www.randomnoun.com/bsd-simplified.html) .SH "SEE ALSO" .BR vmaint (1), http://www.randomnoun.com/wp/2013/10/31/progress-bars/

And here’s the manpage itself:

SCPWRAP(1)                       User Commands                      SCPWRAP(1)



NAME
       scpwrap - a wrapper for scp to convert its output into machine-readable
       format

SYNOPSIS
       scpwrap [--js] [--stdoutTemplate template ] [--stderrTemplate template
       ] [--startTemplate template ] [--progressTemplate template ]
       [--endTemplate template ] -- scp-options ...

DESCRIPTION
       scpwrap wraps the scp(1) command such that the output generated by that
       command  is  converted  into  a  format  more easily processed by other
       commands.

       scp(1) progress information is captured and converted into  a  machine-
       readable format for use in more sophisticated user interfaces (which in
       turn would require even more  effort  to  turn  back  into  a  machine-
       readable format). And so the circle of life continues.

       The  built-in  text  and  javascript  output templates are suitable for
       zenity(1),  or  some  as-yet-to-be-documented   javascript   processor,
       respectively.   See  the  EXAMPLES section to see the general syntax of
       these defaults.

OPTIONS
       --js   Use the default javascript templates  rather  than  the  default
              text templates (see the DEFAULTS section below).

              This  switch  also  enables  javascript-output escaping (see the
              JAVASCRIPT OUTPUT ESCAPES section below).

       --stdoutTemplate template
              Use the supplied template to generate machine-readable text when
              unrecognised output is detected on stdout. The placeholder %s in
              the template will be replaced by the text received on stdout.

       --stderrTemplate template
              Use the supplied template to generate machine-readable text when
              any output is detected on stderr (e.g. to display text contained
              in the remote system's sshd (8) Banner file). The placeholder %s
              in the template will be replaced by the text received on stderr.

       --startTemplate template
              Use  the  supplied template to generate machine-readable text as
              scp progress begins.

       --progressTemplate template
              Use the supplied template to generate machine-readable  text  as
              scp  progresses.   The  placeholders  %f,  %p, %t, %s and %e are
              available (see the PLACEHOLDERS section below)

       --endTemplate template
              Use the supplied template when scp completes. The %c placeholder
              is available.

       scp-options
              these  command-line  options  are  passed directly to scp(1) and
              should conform to the expected syntax of that command.

PLACEHOLDERS
       Placeholders can (and should) be embedded in  template  strings,  which
       will  be  replaced  by  the appropriate value when the stdout/stderr of
       scp(1) is parsed by this utility. Most placeholders are only  available
       in specific template types.

       Recognised placeholders are:

       %s   Free-form stdout or stderr text

       %f   The name of the file being copied (e.g. test.txt)

       %p   The percentage progress of the current file being copied (e.g. 13)

       %t   The transfer size of the current file being copied (e.g. 2112KB)

       %s   The transfer speed of the current file being copied (e.g. 2.1MB/s)

       %e   The ETA of the current file being copied (e.g. --:-- or 05:23)

       %c   The exit code of the scp(1) process (e.g. 0), or a negative number
            if the process was terminated by a signal (the absolute  value  of
            this number corresponds to the signal number received).

PLACEHOLDER ESCAPES
       The  following  escape  sequences are recognised in placeholder strings
       supplied on the command-line:

       \n   Newline

       \r   Carriage return

       \t   Tab character

       \\   Backslash

JAVASCRIPT OUTPUT ESCAPES
       If the text generated by  scp  contains  characters  that  would  cause
       javascript  parsing  errors,  and  the  --js  command-line option is in
       effect, then the strings substituted into the %s  placeholder  will  be
       javascript-escaped  (e.g.  newlines  will be replaced by \n, characters
       above ASCII codepoint 127 will be replaced by \u0000-style sequences.)

DEFAULTS
       The following defaults  are  used,  in  the  absence  of  any  template
       overrides:

   Text output (without --js parameter)
       Note  that  many  templates  are  simply  ""  (i.e.  the  empty string,
       therefore no output is generated)

       --stdoutTemplate    ""

       --stderrTemplate    ""

       --startTemplate     ""

       --progressTemplate  "%p\n"

       --endTemplate       ""

   Javascript output (with --js parameter)
       The default javascript output relies on a script-visible  'ui'  object,
       as shown below.

       --stdoutTemplate    "ui.addOutput(\"%s\");\n"

       --stderrTemplate    "ui.addOutputError(\"%s\");\n";

       --startTemplate     "var sp = ui.startScpProgress();\n";

       --progressTemplate
                           "sp.setProgress(\"%f\", %p, \"%t\", \"%s\", \"%e\");\n";

       --endTemplate       "ui.stopScpProgress(%c);\n";

EXIT STATUS
       The  scpwrap  command  will  return the same exit code as the child scp
       process; i.e. it exits 0 on success, and >0 if an error occurs.

       If a signal interrupts processing of the child  process,  then  scpwrap
       terminates  with  an  exit  status  of  1.  The  signal  number  can be
       determined using the %c placeholder to the --endTemplate template.

EXAMPLES
   Example 1 (text output)
       The command

              scpwrap -- -i key.pem somefile.tar.gz \
                user@somehost:/home/user/somefile.tar.gz

       might produce output something similar to the following:

              0
              45
              47
              49

                ... 20 lines omitted ...

              98
              100

   Example 2 (javascript output)
       The command

              scpwrap --js -- -i key.pem somefile.tar.gz \
                user@somehost:/home/user/somefile.tar.gz \

       might produce output something similar to the following:

              ui.addOutputError("NOTICE TO USERS\n");
              ui.addOutputError("\n");
              ui.addOutputError("This service is for authorised clients only.\n");
              ui.addOutputError("\n");
              ui.addOutputError("This computer system is the private property of its owner, whether\n");
              ui.addOutputError("individual, corporate or government.  It is for authorized use only.\n");
              ui.addOutputError("Users (authorised or unauthorised) have no explicit or implicit\n");
              ui.addOutputError("expectation of privacy.\n");
              ui.addOutputError("\n");
              ui.addOutputError("It is a criminal offence to:\n");
              ui.addOutputError("  i. Obtain access to data without authority\n");
              ui.addOutputError("       (Penalty 2 years imprisonment)\n");
              ui.addOutputError("  ii Damage, delete, alter or insert data without authority\n");
              ui.addOutputError("       (Penalty 10 years imprisonment)\n");
              ui.addOutputError("\n");
              ui.addOutputError("For more information, see http://www.randomnoun.com/login-banner.html\n");
              var sp = ui.startScpProgress();
              sp.setProgress("somefile.tar.gz", 0, "0", "0.0KB/s", "--:--");
              sp.setProgress("somefile.tar.gz", 45, "2112KB", "2.1MB/s", "00:01");
              sp.setProgress("somefile.tar.gz", 47, "2208KB", "1.9MB/s", "00:01");
              sp.setProgress("somefile.tar.gz", 49, "2320KB", "1.7MB/s", "00:01");
              sp.setProgress("somefile.tar.gz", 52, "2448KB", "1.5MB/s", "00:01");
              sp.setProgress("somefile.tar.gz", 54, "2576KB", "1.4MB/s", "00:01");

                ... 20 lines omitted ...

              sp.setProgress("somefile.tar.gz", 98, "4624KB", "266.7KB/s", "00:00");
              sp.setProgress("somefile.tar.gz", 100, "4693KB", "187.7KB/s", "00:25");
              ui.stopScpProgress(0);


   Example 3 (custom javascript output)
       The command

              scpwrap --js --stderrTemplate '' --stdoutTemplate '' \
                --startTemplate '' --endTemplate '' \
                --progressTemplate 'setProgress(%p);\n'
                -- -i key.pem somefile.tar.gz \
                user@somehost:/home/user/somefile.tar.gz

       might produce output something similar to the following:

              setProgress(0);
              setProgress(45);
              setProgress(47);

                 ... 20 lines omitted ...

              setProgress(98);
              setProgress(100);

BUGS
       It might be preferable to get the default  strings  from  something  in
       /etc

       The whole thing's a bit pointless

AUTHOR
       Greg Knox 

LICENCE
       (c) 2013 randomnoun. All Rights Reserved. This work is licensed under a
       BSD Simplified License. (http://www.randomnoun.com/bsd-simplified.html)

SEE ALSO
       vmaint(1), https://www.randomnoun.com/wp/2013/10/31/progress-bars/



vmaint                           OCTOBER 2013                       SCPWRAP(1)

If you want to install it in binary form, I've created an apt repository for a couple of ubuntu distribution/architecture combinations (maverick/amd64 and precise/i386), which you can add to your /etc/apt/sources.list file automatically using these commands:

$ sudo add-apt-repository "deb http://packages.randomnoun.com/ubuntu $(lsb_release -sc) main"
$ gpg --keyserver  hkp://pool.sks-keyservers.net --recv-key A21A1486 && gpg --export --armor A21A1486 | sudo apt-key add -
gpg: requesting key A21A1486 from hkp server pool.sks-keyservers.net
gpg: key A21A1486: "Greg Knox " not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
OK
$ sudo apt-get update
Get:1 http://packages.randomnoun.com maverick Release.gpg [198B]
Get:2 http://packages.randomnoun.com maverick Release [2,242B]
Ign http://packages.randomnoun.com maverick/main Sources
Ign http://packages.randomnoun.com maverick/main amd64 Packages
...
$ sudo apt-get install scpwrap
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  scpwrap
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 12.0kB of archives.
After this operation, 69.6kB of additional disk space will be used.
Get:1 http://packages.randomnoun.com/ubuntu/ maverick/main scpwrap amd64 1.0-1 [12.0kB]
Fetched 12.0kB in 0s (113kB/s)
Selecting previously deselected package scpwrap.
(Reading database ... 59957 files and directories currently installed.)
Unpacking scpwrap (from .../scpwrap_1.0-1_amd64.deb) ...
Processing triggers for man-db ...
Setting up scpwrap (1.0-1) ...

If you don't have add-apt-repository installed in your OS, then try sudo apt-get install python-software-properties first (or you could manually edit /etc/apt/sources.list instead).

Update 4/11/2013: Added instructions for the packages.randomnoun.com apt repository; added BSD licence to source code

[1] Something using modern philosophies, no doubt.

Tags:c