Progress Bars

Some background information

The progress bar.

Not just the name of a sadly defunct bar in London or a sign depicting the prevention of egress, progress bars serve a useful purpose to inform the user just how long they’re going to have to sit around with a thumb up their arse whilst they perform their highly skilled non-DMCA-based white-collar workloads.

They look a bit like this: bar1

or this: bar2

or this: bar4

or this: bar6

As you may be aware, Microsoft will be dropping support for several classes of progress bar in 2014, for reasons of security to the Microsoft revenue base.

A lot of tasks don’t have an easily-defined halting condition, so you get slightly different progress bars that move a little to the right, and then a little to the left, or look like barberpoles if you use one of those computers that they look like barberpoles on.

They look a bit like this: bar2b

or this: bar5

or this: bar7

These days of course, it’s more modern to expose your end-users to “throbbers” instead, which give a pleasing sense of going round in circles, and are probably a good indication of what limbo is going to end up being like, if you’re religiously inclined.

They appear in many guises, such as

this… or this… or this… or this… or this… or this….

throb1

throb3

or this

throb5

throb6

throb7

or this:

press-any-key

Some foreground information

So anyway, I’m not one to stand in the way of progress (ha), and now that I think you might know what a progress bar is, here’s something to make your scp task generate some javascript which you can dump into a browser to make a little rectangle get larger as the copy process takes place.

People generally expect to see this sort of thing during long periods of what appears to be inactivity, so you might as well give it to them so as not to surprise them, which apparently is a usability nightmare, unless you’re trying to cram the next great evolution in gesture-based interfaces down people’s throats.

The program itself is called scpwrap, and it’s vaguely similar to that previous post about piping stdout/stderr streams in VBA, since that’s what it does, but this implementation is in C, which is much more self-respecting. You could probably drop it into a bash script without people thinking they should wrap it in a chef recipe or a puppet whatever-they-call-their-module-components [1].

It’s slightly more complicated than it should be because scp won’t normally display progress information if it’s not running in a terminal, hence the forkpty pseudo-terminal malarky on line 319. The code is based on (and is backwards-compatible with) the scp-to-zenity wrapper written by Clayton Shepard, but also handles non-standard stdout/stderr streams, has customisable output templates, and deals with error conditions in a relatively sane way.

Here’s the sourcecode for the scpwrap program, and the groff source for the manpage, which took almost as long to write up.

There’s also a link to the randomnoun apt repository at the bottom of this post, if you want to install it automatically.

scpwrap.c
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
/* (c) 2013 randomnoun. All Rights Reserved. This work is licensed under a
 * BSD Simplified License. (http://www.randomnoun.com/bsd-simplified.html)
 */
 
/* scpwrap.c
 * 
 * $Id: scpwrap.c,v 1.19 2013-11-03 22:24:42 knoxg Exp $
 *
 * Wrapper for scp which translates progress text, e.g:
 * 
 *   something.tar.gz                                1% 2112KB   2.1MB/s   00:50 ETA   
 * 
 * into javascript, which can then display and update a progress bar.
 *
 * Call this thing from a bash script or other process/thread that will dump its output 
 * directly into an iframe which is surrounded by <script> elements.
 *
 * COMPILING
 * 
 * Remember to include the util library when compiling !
 * i.e. gcc -lutil scpwrap.c -oscpwrap
 *
 * as of 2013, you could try doing this instead, for reasons that I don't particularly understand:
 *      gcc -Wl,--no-as-needed -lutil scpwrap.c -oscpwrap
 * 
 * Modified from http://cwshep.blogspot.com/2009/06/showing-scp-progress-using-zenity.html
 *
 * This program relies on the fact that even if stdout performs checks to see whether 
 * it is running on a tty, these checks are not perfomed for stderr ( search for 'isatty' in 
 * http://www.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/scp.c?rev=1.178;content-type=text%2Fx-cvsweb-markup )
 *
 * see http://www.linuxquestions.org/questions/programming-9/problem-with-child-process-658750/#post3229496
       http://www.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/scp.c?rev=1.178;content-type=text%2Fx-cvsweb-markup
       http://stackoverflow.com/questions/2605130/redirecting-exec-output-to-a-buffer-or-file
       http://stackoverflow.com/questions/903864/how-to-exit-a-child-process-and-return-its-status-from-execvp
 */
 
#include <unistd.h>
#include <sys/types.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdarg.h>
#include <sys/select.h>
#include <getopt.h>
 
/* let's say that the things we're going to replace in here are:
 * %f - filename
 * %p - progress (0-100) and "%" character
 * %t - transfer size ("2112KB")
 * %s - speed ("2.1MB/s" or however progressmeter.c does things)
 * %e - ETA ("--:--" or "05:23"), hh:mm:ss, or however progressmeter.c does things)
 */
 
// if we get more than this number of bytes on stdout/stderr without a newline,
// then they will be emitted in >1 stdout/stderr template 
#define STDERR_BUFSIZE 1024
#define STDOUT_BUFSIZE 1024
 
// enabled by --js option; stdout/stderr will be javascript-String escaped
static int ESCAPE_JS = 0;
 
static char* TXT_STDOUT_TEMPLATE = "";
static char* TXT_STDERR_TEMPLATE = "";
static char* TXT_START_TEMPLATE = "";
static char* TXT_PROGRESS_TEMPLATE = "%p\n";
static char* TXT_END_TEMPLATE = "";
 
static char* JS_STDOUT_TEMPLATE = "ui.addOutput(\"%s\");\n";
static char* JS_STDERR_TEMPLATE = "ui.addOutputError(\"%s\");\n";
static char* JS_START_TEMPLATE = "var sp = ui.startScpProgress();\n";
static char* JS_PROGRESS_TEMPLATE = "sp.setProgress(\"%f\", %p, \"%t\", \"%s\", \"%e\");\n";
static char* JS_END_TEMPLATE = "ui.stopScpProgress(%c);\n";
 
/** convert ASCII to javascript escape.
 
    see http://rishida.net/tools/conversion
 */ 
void printjs (char *str) {
    int i=0;
    for (i=0; str[i]!=0; i++) {
        switch (str[i]) {
            case 0: printf("\\0"); break;
            case 8: printf("\\b"); break;
            case 9: printf("\\t"); break;
            case 10: printf("\\n"); break;
            case 13: printf("\\r"); break;
            case 11: printf("\\v"); break;
            case 12: printf("\\f"); break;
            case 34: printf("\\\""); break;
            case 39: printf("\\\'"); break;
            case 92: printf("\\\\"); break;
            default:
                if (str[i]>0x1f && str[i]<0x7F) { 
                    printf("%c", str[i]); 
                } else { 
                    printf("\\u%04x", (int) str[i]); 
                }
        }       
    }
}
 
/** Take a format string ("template") containing 0 or more tokens in the form 
   "{n}", and replace each token with the n'th optional parameter to this function. 
   The string after token replacement is sent to stdout. 
 
   The nparam must be set to the number of optional parameters supplied.
   All optional parameters must be of type char*.
 
   e.g. 
     printfmt(3, "first: {0}, second: {1}, third: {2}", "a", "b", "c");
   produces the output
     "first: a, second: b, third: c"
 
   C-like escapes ("\n", "\t" etc) in the format string will be converted into their appropriate character     
   if JS_ESCAPE is true, then values in the optional parameters will be Javascript-escaped
     (i.e. the character '\n' will be converted to the two-character "\n" escape) 
 */  
int printfmt (int nparam, const char *format, ...) {
    va_list ap;
    int i = 0;
    int inBrace = 0, inEscape = 0;
    int paramIdx = 0;
 
    char *params[nparam];
    va_start(ap, format);
    for (i=0; i<nparam; i++) { 
        params[i] = va_arg(ap, char *);
    }  
    va_end(ap);
 
    for (i=0; format[i]!=0; i++) {
        if (inEscape) {
            inEscape = 0;
            switch (format[i]) {
                case 'n': printf("\n"); break;
                case 'r': printf("\r"); break;
                case 't': printf("\t"); break;
                case '{': printf("{"); break;
                case '\\': printf("\\"); break;
                default: /* unknown escape, just print character */ printf("%c", format[i]);
            }
        } else if (inBrace) {
            switch (format[i]) {
                case '0': case '1': case '2': case '3': case '4': 
                case '5': case '6': case '7': case '8': case '9':
                    paramIdx = paramIdx*10 + (format[i]-'0');
                    break;
                case '}': 
                    inBrace = 0; 
                    if (ESCAPE_JS) { 
                        printjs(params[paramIdx]); 
                    } else { 
                        printf("%s", params[paramIdx]); 
                    }
                    break;
                default:
                    // weird thing in brace
                    return -1;
            }      
        } else {      
            switch (format[i]) {
                case '{':  paramIdx = 0; inBrace = 1; break;
                case '\\': inEscape = 1; break;
                default: printf("%c", format[i]); 
            }
        }
    }
}
 
/** send usage information to stdout */ 
void usage() {
    printf("usage: scpwrap [options] -- scp-options \n"
      "Where options are:\n" 
      "  --js                   use javascript default templates, and javascript-escape output strings\n"
      "  --stdoutTemplate txt   template to use for unrecognised stdout text\n"
      "  --stderrTemplate txt   template to use for unrecognised stderr text\n"
      "  --startTemplate txt    text to display before the first progressTemplate appears\n"
      "  --progressTemplate txt template to use for copy progress output\n"
      "  --endTemplate txt      template to use after copy completes\n"
      "The following placeholders can be used in progress templates:\n"
      "  %%f  filename\n"
      "  %%p  progress amount (0-100)\n"
      "  %%t  transfer size (e.g. \"2112KB\")\n"
      "  %%s  speed (e.g. \"2.1MB/s\")\n"
      "  %%e  ETA (e.g. \"--:--\" or \"05:23\")\n"
      "The following placeholder can be used in stdout/stderr templates:\n"  
      "  %%s  text string\n"
      "The following placeholder can be used in the endTemplate:\n"  
      "  %%c  exit code\n"
      "\n"
      "See the 'scpwrap' and 'scp' man page for more options. Example usage:\n"
      "  scpwrap --js -- -i identityfile user@host1:file1 user@host2:file2\n"
      );
 
    // This is a program that wraps scp and prints out the numeric progress on separate lines.\n");
    fflush(stdout);
}
 
 
/** replace all instances of a substring in a string with a relacement string.
   memory for the returned string will be reserved via malloc
 
   see http://coding.debuntu.org/c-implementing-str_replace-replace-all-occurrences-substring
 */   
char *str_replace ( const char *string, const char *substr, const char *replacement ){
    char *tok = NULL;
    char *newstr = NULL;
    char *oldstr = NULL;
    /* if either substr or replacement is NULL, duplicate string and let caller handle it */
    if ( substr == NULL || replacement == NULL ) { 
        return strdup (string); 
    }
    newstr = strdup (string);
    while ( (tok = strstr ( newstr, substr ))) {
        oldstr = newstr;
        newstr = malloc ( strlen ( oldstr ) - strlen ( substr ) + strlen ( replacement ) + 1 );
        /*failed to alloc mem, free old string and return NULL */
        if ( newstr == NULL ) {
            free (oldstr);
            return NULL;
        }
        memcpy ( newstr, oldstr, tok - oldstr );
        memcpy ( newstr + (tok - oldstr), replacement, strlen ( replacement ) );
        memcpy ( newstr + (tok - oldstr) + strlen( replacement ), tok + strlen ( substr ), strlen ( oldstr ) - strlen ( substr ) - ( tok - oldstr ) );
        memset ( newstr + strlen ( oldstr ) - strlen ( substr ) + strlen ( replacement ) , 0, 1 );
        free (oldstr);
    }
    return newstr;
}
 
/** main */
main(int argc, char **argv) {
    char *startTemplate = TXT_START_TEMPLATE;
    char *stdoutTemplate = TXT_STDOUT_TEMPLATE;
    char *stderrTemplate = TXT_STDERR_TEMPLATE;
    char *progressTemplate = TXT_PROGRESS_TEMPLATE;
    char *endTemplate = TXT_END_TEMPLATE;
 
    int shownStartTemplate = 0;       // set to 1 when startTemplate is printed
 
    char stderrBuf[STDERR_BUFSIZE];   // stderr capture buffer
    char stdoutBuf[STDOUT_BUFSIZE];   // stdout capture buffer
    char fieldBuf[STDOUT_BUFSIZE];    // as per stdout, with spaces replaced with \0x0
 
    int stderrIdx = 0, stdoutIdx = 0; // indices within stderrBuf/stdoutBuf
    ssize_t stderrc, stdoutc;         // count of bytes read into stderrBuf/stdoutBuf
    int doneStderr = 0, doneStdout=0; // set to 1 when stderr/stdout file descriptors have closed
 
    int stdoutPtyFd;                  // file descriptor for the master side of the stdout pseudoterminal    
    int stderrPipeFd[2];              // pipe used to read stderr
 
    pid_t pid;                        // pid of scp child process
    int exitStatus = 0;               // scp child process exist status
 
    // parse options
    int c;
    int digit_optind = 0;
    while (1) {
        int this_option_optind = optind ? optind : 1;
        int option_index = 0;
        static struct option long_options[] = {
            {"js",               no_argument,       0,  0 },
            {"startTemplate",    required_argument, 0,  0 },
            {"stdoutTemplate",   required_argument, 0,  0 },
            {"stderrTemplate",   required_argument, 0,  0 },
            {"progressTemplate", required_argument, 0,  0 },
            {"endTemplate",      required_argument, 0,  0 },
            {0,         0,                 0,  0 }
        };
 
        c = getopt_long(argc, argv, "?", long_options, &option_index);
        if (c == -1) { break; }
        switch (c) {
            case 0:
                switch (option_index) {
                    case 0: 
                        ESCAPE_JS = 1;
                        startTemplate = JS_START_TEMPLATE;
                        stdoutTemplate = JS_STDOUT_TEMPLATE;
                        stderrTemplate = JS_STDERR_TEMPLATE;
                        progressTemplate = JS_PROGRESS_TEMPLATE;
                        endTemplate = JS_END_TEMPLATE;
                        break;
                    case 1: startTemplate = optarg; break;
                    case 2: stdoutTemplate = optarg; break;
                    case 3: stderrTemplate = optarg; break;
                    case 4: progressTemplate = optarg; break;
                    case 5: endTemplate = optarg; break;
                    default:
                        fprintf(stderr, "getopt returned option_index %d\n", option_index);
                        exit(1);   
                }
                break;
            case '?':
                usage(); 
                exit(1);
                break;
            default:
                fprintf(stderr, "getopt returned character code 0%o '%c'\n", c, c);
                exit(1);
        }
    }
    if (optind >= argc) {
       fprintf(stderr, "You must supply options to 'scp' after the '--' command line-argument\n");
       usage();
       exit(1);
    }
 
    // replace user-specified placeholders (e.g. %s) with positional placeholders (e.g. {0})
    /* str_replace calls malloc for each result string */
    stdoutTemplate = str_replace(stdoutTemplate, "%s", "{0}");
    stderrTemplate = str_replace(stderrTemplate, "%s", "{0}");
    progressTemplate = str_replace(progressTemplate, "%f", "{0}"); // feel free to tell me that this leaks memory on each str_replace
    progressTemplate = str_replace(progressTemplate, "%p", "{1}");
    progressTemplate = str_replace(progressTemplate, "%t", "{2}");
    progressTemplate = str_replace(progressTemplate, "%s", "{3}");
    progressTemplate = str_replace(progressTemplate, "%e", "{4}");
    endTemplate = str_replace(endTemplate, "%c", "{0}");
    // TODO: check return values for NULL (malloc error)
 
    // create pipe for stderr
    pipe(stderrPipeFd);
 
    // get a pseudoterminal
    pid = forkpty (&stdoutPtyFd, NULL, NULL, NULL);
    if (pid == 0) {
        /* CHILD */
        close(stderrPipeFd[0]);    // close reading end in the child
        // dup2(stderrPipeFd[1], 1);  // send stdout to the pipe (unused; stdout is the pseudoterminal fd)
        dup2(stderrPipeFd[1], 2);  // send stderr to the pipe
        close(stderrPipeFd[1]);    // this descriptor is no longer needed
 
        // pass arguments "scp" then argv[optind] to argv[argc]
        // argv[optind-1] should be pointing to the '--' argument so we replace it with "scp"
        argv[optind-1] = "scp";       
        execvp("scp", &argv[optind-1]);
        perror("execvp");
        _exit (2);
    } else if (pid == -1) {
        /* ERROR */
        perror("forkpty");
        exit (1);
    } else {
        /* PARENT */
        close(stderrPipeFd[1]);  // close the write end of the pipe in the parent
        fd_set selectFds;
        int retval;
 
        /* Watch stdout (stdoutPtyFd) or stderr (stderrPipeFd[0]) to see when it has input. */
        FD_ZERO(&selectFds);
        FD_SET(stdoutPtyFd, &selectFds);
        FD_SET(stderrPipeFd[0], &selectFds);
 
        while (!(doneStdout || doneStderr)) {
            // NB: first param isn't the number of file descriptors, it's the highest file descriptor + 1
            // FD_SETSIZE may not be terribly efficient, but should work here 
            retval = select(FD_SETSIZE, &selectFds, NULL, NULL, NULL);
            if (retval==-1) { 
                perror("select()"); exit(1);
            } else {
                if (FD_ISSET(stderrPipeFd[0], &selectFds) && !doneStderr) {
                    stderrc = read(stderrPipeFd[0], &stderrBuf[stderrIdx], 1);
                    //printf ("this shouldnt be more than 1: %d, %c\n", stderrc, stderrBuf[stderrIdx]);
                    if (stderrc <= 0) {
                        doneStderr=1;
                    } else {
                        stderrIdx += stderrc; 
                        if (stderrBuf[stderrIdx-1]=='\r' || stderrBuf[stderrIdx-1]=='\n' || stderrIdx==STDERR_BUFSIZE-1) {
                            stderrBuf[stderrIdx]=0;
                            printfmt(1, stderrTemplate, stderrBuf);
                            stderrIdx=0; 
                        }
                    }  
                }
 
                if (FD_ISSET(stdoutPtyFd, &selectFds) && !doneStdout) {
                    stdoutc = read(stdoutPtyFd, &stdoutBuf[stdoutIdx], 1);
                    // printf ("this shouldnt be more than 1: %d, %c\n", stdoutc, stdoutBuf[stdoutIdx]);
                    if (stdoutc <= 0) {
                        doneStdout=1; ;
                    } else {
                        stdoutIdx += stdoutc;
                        if (stdoutBuf[stdoutIdx-1]=='\r' || stdoutBuf[stdoutIdx-1]=='\n' || stdoutIdx==STDOUT_BUFSIZE-1) {
                            stdoutBuf[stdoutIdx]=0;
 
                            // stdoutBuf will be something like
                            // something.tar.gz                                1% 2112KB   2.1MB/s   00:50 ETA
                            // unless we exceed BUFSIZE. which will never happen. unless we get a horrendously long filename perhaps
 
                            // fieldBuf is set to stdoutBuf, but with spaces replaced with char(0)s
                            // fields[n] are the character positions of the nth field in fieldBuf
                            strncpy(fieldBuf, stdoutBuf, STDOUT_BUFSIZE);
                            int fieldIdx=0, fields[5], i=0;
                            for (i=0; i<stdoutIdx; i++) {
                                if (fieldBuf[i]==' ' || fieldBuf[i]=='\r') { 
                                    fieldBuf[i]=0; 
                                } else {
                                    if (fieldIdx<5 && (i==0 || fieldBuf[i-1]==0)) {
                                        fields[fieldIdx++]=i; 
                                    }
                                }
                            }
 
                            // right number of fields and a percentage character in the right place? 
                            if (fieldIdx==5 && fieldBuf[fields[1] + strlen(&fieldBuf[fields[1]])-1]=='%') {
                                fieldBuf[fields[1] + strlen(&fieldBuf[fields[1]])-1]=0;
                                if (!shownStartTemplate) {
                                    shownStartTemplate = 1;
                                    printf("%s", startTemplate);
                                }  
                                printfmt(5, progressTemplate, &fieldBuf[fields[0]],
                                  &fieldBuf[fields[1]],&fieldBuf[fields[2]],&fieldBuf[fields[3]],&fieldBuf[fields[4]]);
                            } else {
                                // don't generate empty lines on stdout
                                if ((strncmp(stdoutBuf, "\n", STDOUT_BUFSIZE)!=0) &&
                                    (strncmp(stdoutBuf, "\r", STDOUT_BUFSIZE)!=0) ) {
                                    printfmt(1, stdoutTemplate, stdoutBuf);
                                }   
                            }
                            fflush(stdout);
 
                            stdoutIdx=0; 
                        }
                    }  
                }  
            }
 
            // reset set
            FD_ZERO(&selectFds);
            if (!doneStdout) { FD_SET(stdoutPtyFd, &selectFds); }
            if (!doneStderr) { FD_SET(stderrPipeFd[0], &selectFds); }
        }  
 
        // both stdout & stderr have been closed, dump whatever's left
        if (stderrIdx!=0) { stderrBuf[stderrIdx]=0; printfmt(1, stderrTemplate, stderrBuf); }
        if (stdoutIdx!=0) { stdoutBuf[stdoutIdx]=0; printfmt(1, stdoutTemplate, stdoutBuf); }
        fflush(stdout);
 
        // get the exit status of the scp process. 
        pid_t w = waitpid (pid, &exitStatus, 0);
        if (w==-1) { perror("waitpid"); exit(1); }
 
        if (WIFEXITED(exitStatus)) {
          sprintf(stdoutBuf, "%d", WEXITSTATUS(exitStatus));
          printfmt(1, endTemplate, stdoutBuf);
          exit (WEXITSTATUS(exitStatus));  // propagate exitStatus
 
        } else if (WIFSIGNALED(exitStatus)) {
          // display a signal as -(signal number)
          sprintf(stdoutBuf, "-%d", WTERMSIG(exitStatus));
          printfmt(1, endTemplate, stdoutBuf); 
          exit (1);
 
        } else {
          // something weird
          exit (1);
        }
    }
 
    // this should should never be executed 
    exit (0);
}
scpwrap.groff
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
.\" 
.\" (c) 2013 randomnoun. All Rights Reserved. This work is licensed under a
.\" BSD Simplified License. (http://www.randomnoun.com/bsd-simplified.html)
.\" 
.\" scpwrap.groff
.\" $Id: scpwrap.groff,v 1.5 2013-10-31 06:35:47 knoxg Exp $
.\"
.\" Man page for the scpwrap utility
.\" 
.\" Process this file with
.\" 
.\"   visual pager:         groff -man -Tascii scpwrap.groff | less
.\"   HTML:                 groff -man -Thtml scpwrap.groff > scpwrap.html
.\"   HTML with hyperlinks: man2html -H code.randomnoun.com -M /man2html scpwrap.groff > scpwrap.html
.\"   fixed width HTML:     groff -man -Tascii scpwrap.groff | perl -i -pe 's!(.)\x08\g1!<b>\1</b>!g; s!</b>(\s*)<b>!\1!g; s!_\x08(.)!<u>\1</u>!g;; s!</u>(\s*)<u>!\1!g;' > scpwrap.html
.\"
.TH SCPWRAP 1 "OCTOBER 2013" vmaint "User Commands"
.SH NAME
scpwrap \- a wrapper for scp to convert its output into machine-readable format
.SH SYNOPSIS
.nh \" no hyphenating
.na \" no right-margin adjustment
.B scpwrap [--js] [--stdoutTemplate 
.I template
.B ] [--stderrTemplate 
.I template
.B ] [--startTemplate
.I template
.B ] [--progressTemplate
.I template
.B ] [--endTemplate
.I template
.B ] --
.I scp-options
.B ...
.ad \" re-enable right-margin adjustment (i.e. full justification)
.SH DESCRIPTION
.B scpwrap
wraps the 
.BR scp (1)
command such that the output generated by that command is
converted into a format more easily processed by other
commands.  
.P
.BR scp (1) 
progress information is captured and converted into a  
machine-readable format for use in more sophisticated user interfaces
(which in turn would require even more effort to turn back into
a machine-readable format). And so the circle of life continues. 
.P
The built-in text and javascript output templates are suitable for 
.BR zenity (1),
or some as-yet-to-be-documented javascript processor, respectively.
See the \fBEXAMPLES\fR section to see the general syntax of these defaults. 
.SH OPTIONS
.IP \fB--js\fR
Use the default javascript templates rather than the default
text templates (see the \fBDEFAULTS\fR section below).
 
This switch also enables javascript-output escaping
(see the \fBJAVASCRIPT OUTPUT ESCAPES\fR section below). 
.IP "\fB--stdoutTemplate\fR \fItemplate\fR"
Use the supplied 
.I template
to generate machine-readable text when unrecognised output is 
detected on stdout. The placeholder
\fB%s\fR in the template will be replaced by the text received on stdout.
.IP "\fB--stderrTemplate\fR \fItemplate\fR"
Use the supplied 
.I template
to generate machine-readable text when any output is
detected on stderr (e.g. to display text contained in the remote system's 
.B sshd (8) 
Banner file). The placeholder
\fB%s\fR in the template will be replaced by the text received on stderr.
.IP "\fB--startTemplate\fR \fItemplate\fR"
Use the supplied 
.I template
to generate machine-readable text as scp progress begins.
.IP "\fB--progressTemplate\fR \fItemplate\fR"
Use the supplied 
.I template
to generate machine-readable text as scp progresses. 
The placeholders \fB%f\fR, \fB%p\fR, \fB%t\fR,
\fB%s\fR and \fB%e\fR are available (see the \fBPLACEHOLDERS\fR section below) 
.IP "\fB--endTemplate\fR \fItemplate\fR"
Use the supplied 
.I template
when scp completes. The \fI%c\fR placeholder is available.
.IP \fIscp-options\fR
these command-line options are passed directly to 
.BR scp (1)
and should conform to the expected syntax of that command.
.SH PLACEHOLDERS
Placeholders can (and should) be embedded in template strings, which will
be replaced by the appropriate value when the stdout/stderr of  
.BR scp (1)
is parsed by this utility. Most placeholders are only available
in specific template types.
.P
Recognised placeholders are:
.TP 5
\fB%s\fR
Free-form stdout or stderr text
.TP
\fB%f\fR
The name of the file being copied (e.g. \fBtest.txt\fR)
.TP
\fB%p\fR
The percentage progress of the current file being copied (e.g. \fB13\fR)
.TP
\fB%t\fR
The transfer size of the current file being copied (e.g. \fB2112KB\fR)
.TP
\fB%s\fR
The transfer speed of the current file being copied (e.g. \fB2.1MB/s\fR)
.TP
\fB%e\fR
The ETA of the current file being copied (e.g. \fB--:-- or \fB05:23\fR)
.TP
\fB%c\fR
The exit code of the 
.BR scp (1)
process (e.g. \fB0\fR), or a negative number if the process was 
terminated by a signal (the absolute value of this number corresponds to 
the signal number received).
.SH PLACEHOLDER ESCAPES
The following escape sequences are recognised in placeholder strings supplied
on the command-line:
.TP 5
\fB\\n\fR
Newline
.TP
\fB\\r\fR
Carriage return
.TP
\fB\\t\fR
Tab character
.TP
\fB\\\\\fR
Backslash
.SH JAVASCRIPT OUTPUT ESCAPES
If the text generated by scp contains characters that would cause 
javascript parsing errors, and the \fB--js\fR command-line option is in
effect, then the strings substituted into the \fB%s\fR placeholder will
be javascript-escaped (e.g. newlines will be replaced by \fB\\n\fR, 
characters above ASCII codepoint 127 will be replaced by \fB\\u0000\fR-style
sequences.)  
.SH DEFAULTS
The following defaults are used, in the absence of any template overrides:
.SS Text output (without --js parameter)
Note that many templates are simply "" (i.e. the empty string, 
therefore no output is generated)
.TP 20
\fB--stdoutTemplate\fR
""
.TP
\fB--stderrTemplate\fR
""
.TP
\fB--startTemplate\fR
""
.TP
\fB--progressTemplate\fR
"%p\\n"
.TP
\fB--endTemplate\fR
"" 
.SS Javascript output (with --js parameter)
The default javascript output relies on a script-visible 'ui' object, 
as shown below.
.TP 20
\fB--stdoutTemplate\fR
"ui.addOutput(\\"%s\\");\\n"
.TP
\fB--stderrTemplate\fR
"ui.addOutputError(\\"%s\\");\\n";
.TP
\fB--startTemplate\fR
"var sp = ui.startScpProgress();\\n";
.TP
\fB--progressTemplate\fR
.nf
"sp.setProgress(\\"%f\\", %p, \\"%t\\", \\"%s\\", \\"%e\\");\\n";
.fi
.TP
\fB--endTemplate\fR
"ui.stopScpProgress(%c);\\n";
.SH EXIT STATUS
The \fBscpwrap\fR command will return the same exit code as the child
\fBscp\fR process; i.e. it exits 0 on success, and >0 if an error occurs. 
.P
If a signal interrupts processing of the child process, then \fBscpwrap\fR 
terminates with an exit status of 1. The signal number can be determined 
using the \fB%c\fR placeholder to the \fB--endTemplate\fR template.   
.SH EXAMPLES
.SS Example 1 (text output)
The command
 
.RS
.nf
scpwrap -- -i key.pem somefile.tar.gz \\
  user@somehost:/home/user/somefile.tar.gz
.fi
.RE
 
might produce output something similar to the following:
 
.RS
.nf
0
45
47
49
 
  ... 20 lines omitted ...
 
98
100
.fi
.RE
.P
 
.SS Example 2 (javascript output)
The command
 
.RS
.nf
scpwrap --js -- -i key.pem somefile.tar.gz \\
  user@somehost:/home/user/somefile.tar.gz \\
.fi
.RE
 
might produce output something similar to the following:
 
.RS
.nf
ui.addOutputError("NOTICE TO USERS\\n");
ui.addOutputError("\\n");
ui.addOutputError("This service is for authorised clients only.\\n");
ui.addOutputError("\\n");
ui.addOutputError("This computer system is the private property of its owner, whether\\n");
ui.addOutputError("individual, corporate or government.  It is for authorized use only.\\n");
ui.addOutputError("Users (authorised or unauthorised) have no explicit or implicit\\n");
ui.addOutputError("expectation of privacy.\\n");
ui.addOutputError("\\n");
ui.addOutputError("It is a criminal offence to:\\n");
ui.addOutputError("  i. Obtain access to data without authority\\n");
ui.addOutputError("       (Penalty 2 years imprisonment)\\n");
ui.addOutputError("  ii Damage, delete, alter or insert data without authority\\n");
ui.addOutputError("       (Penalty 10 years imprisonment)\\n");
ui.addOutputError("\\n");
ui.addOutputError("For more information, see http://www.randomnoun.com/login-banner.html\\n");
var sp = ui.startScpProgress();
sp.setProgress("somefile.tar.gz", 0, "0", "0.0KB/s", "--:--");
sp.setProgress("somefile.tar.gz", 45, "2112KB", "2.1MB/s", "00:01");
sp.setProgress("somefile.tar.gz", 47, "2208KB", "1.9MB/s", "00:01");
sp.setProgress("somefile.tar.gz", 49, "2320KB", "1.7MB/s", "00:01");
sp.setProgress("somefile.tar.gz", 52, "2448KB", "1.5MB/s", "00:01");
sp.setProgress("somefile.tar.gz", 54, "2576KB", "1.4MB/s", "00:01");
 
  ... 20 lines omitted ...
 
sp.setProgress("somefile.tar.gz", 98, "4624KB", "266.7KB/s", "00:00");
sp.setProgress("somefile.tar.gz", 100, "4693KB", "187.7KB/s", "00:25");
ui.stopScpProgress(0);
.fi
.RS
 
.SS Example 3 (custom javascript output)
The command
 
.RS
.nf
scpwrap --js --stderrTemplate '' --stdoutTemplate '' \\
  --startTemplate '' --endTemplate '' \\
  --progressTemplate 'setProgress(%p);\\n' 
  -- -i key.pem somefile.tar.gz \\
  user@somehost:/home/user/somefile.tar.gz 
.fi
.RE
 
might produce output something similar to the following:
 
.RS
.nf
setProgress(0);
setProgress(45);
setProgress(47);
 
   ... 20 lines omitted ...
 
setProgress(98);
setProgress(100);
.fi
.RE
.SH BUGS
.P
It might be preferable to get the default strings from something in /etc
.P
The whole thing's a bit pointless
.SH AUTHOR
Greg Knox <knoxg at randomnoun dot com>
.SH LICENCE
(c) 2013 randomnoun. All Rights Reserved. This work is licensed under a
BSD Simplified License. (http://www.randomnoun.com/bsd-simplified.html)
.SH "SEE ALSO"
.BR vmaint (1),
http://www.randomnoun.com/wp/2013/10/31/progress-bars/

And here’s the manpage itself:

SCPWRAP(1)                       User Commands                      SCPWRAP(1)



NAME
       scpwrap - a wrapper for scp to convert its output into machine-readable
       format

SYNOPSIS
       scpwrap [--js] [--stdoutTemplate template ] [--stderrTemplate template
       ] [--startTemplate template ] [--progressTemplate template ]
       [--endTemplate template ] -- scp-options ...

DESCRIPTION
       scpwrap wraps the scp(1) command such that the output generated by that
       command  is  converted  into  a  format  more easily processed by other
       commands.

       scp(1) progress information is captured and converted into  a  machine-
       readable format for use in more sophisticated user interfaces (which in
       turn would require even more  effort  to  turn  back  into  a  machine-
       readable format). And so the circle of life continues.

       The  built-in  text  and  javascript  output templates are suitable for
       zenity(1),  or  some  as-yet-to-be-documented   javascript   processor,
       respectively.   See  the  EXAMPLES section to see the general syntax of
       these defaults.

OPTIONS
       --js   Use the default javascript templates  rather  than  the  default
              text templates (see the DEFAULTS section below).

              This  switch  also  enables  javascript-output escaping (see the
              JAVASCRIPT OUTPUT ESCAPES section below).

       --stdoutTemplate template
              Use the supplied template to generate machine-readable text when
              unrecognised output is detected on stdout. The placeholder %s in
              the template will be replaced by the text received on stdout.

       --stderrTemplate template
              Use the supplied template to generate machine-readable text when
              any output is detected on stderr (e.g. to display text contained
              in the remote system's sshd (8) Banner file). The placeholder %s
              in the template will be replaced by the text received on stderr.

       --startTemplate template
              Use  the  supplied template to generate machine-readable text as
              scp progress begins.

       --progressTemplate template
              Use the supplied template to generate machine-readable  text  as
              scp  progresses.   The  placeholders  %f,  %p, %t, %s and %e are
              available (see the PLACEHOLDERS section below)

       --endTemplate template
              Use the supplied template when scp completes. The %c placeholder
              is available.

       scp-options
              these  command-line  options  are  passed directly to scp(1) and
              should conform to the expected syntax of that command.

PLACEHOLDERS
       Placeholders can (and should) be embedded in  template  strings,  which
       will  be  replaced  by  the appropriate value when the stdout/stderr of
       scp(1) is parsed by this utility. Most placeholders are only  available
       in specific template types.

       Recognised placeholders are:

       %s   Free-form stdout or stderr text

       %f   The name of the file being copied (e.g. test.txt)

       %p   The percentage progress of the current file being copied (e.g. 13)

       %t   The transfer size of the current file being copied (e.g. 2112KB)

       %s   The transfer speed of the current file being copied (e.g. 2.1MB/s)

       %e   The ETA of the current file being copied (e.g. --:-- or 05:23)

       %c   The exit code of the scp(1) process (e.g. 0), or a negative number
            if the process was terminated by a signal (the absolute  value  of
            this number corresponds to the signal number received).

PLACEHOLDER ESCAPES
       The  following  escape  sequences are recognised in placeholder strings
       supplied on the command-line:

       \n   Newline

       \r   Carriage return

       \t   Tab character

       \\   Backslash

JAVASCRIPT OUTPUT ESCAPES
       If the text generated by  scp  contains  characters  that  would  cause
       javascript  parsing  errors,  and  the  --js  command-line option is in
       effect, then the strings substituted into the %s  placeholder  will  be
       javascript-escaped  (e.g.  newlines  will be replaced by \n, characters
       above ASCII codepoint 127 will be replaced by \u0000-style sequences.)

DEFAULTS
       The following defaults  are  used,  in  the  absence  of  any  template
       overrides:

   Text output (without --js parameter)
       Note  that  many  templates  are  simply  ""  (i.e.  the  empty string,
       therefore no output is generated)

       --stdoutTemplate    ""

       --stderrTemplate    ""

       --startTemplate     ""

       --progressTemplate  "%p\n"

       --endTemplate       ""

   Javascript output (with --js parameter)
       The default javascript output relies on a script-visible  'ui'  object,
       as shown below.

       --stdoutTemplate    "ui.addOutput(\"%s\");\n"

       --stderrTemplate    "ui.addOutputError(\"%s\");\n";

       --startTemplate     "var sp = ui.startScpProgress();\n";

       --progressTemplate
                           "sp.setProgress(\"%f\", %p, \"%t\", \"%s\", \"%e\");\n";

       --endTemplate       "ui.stopScpProgress(%c);\n";

EXIT STATUS
       The  scpwrap  command  will  return the same exit code as the child scp
       process; i.e. it exits 0 on success, and >0 if an error occurs.

       If a signal interrupts processing of the child  process,  then  scpwrap
       terminates  with  an  exit  status  of  1.  The  signal  number  can be
       determined using the %c placeholder to the --endTemplate template.

EXAMPLES
   Example 1 (text output)
       The command

              scpwrap -- -i key.pem somefile.tar.gz \
                user@somehost:/home/user/somefile.tar.gz

       might produce output something similar to the following:

              0
              45
              47
              49

                ... 20 lines omitted ...

              98
              100

   Example 2 (javascript output)
       The command

              scpwrap --js -- -i key.pem somefile.tar.gz \
                user@somehost:/home/user/somefile.tar.gz \

       might produce output something similar to the following:

              ui.addOutputError("NOTICE TO USERS\n");
              ui.addOutputError("\n");
              ui.addOutputError("This service is for authorised clients only.\n");
              ui.addOutputError("\n");
              ui.addOutputError("This computer system is the private property of its owner, whether\n");
              ui.addOutputError("individual, corporate or government.  It is for authorized use only.\n");
              ui.addOutputError("Users (authorised or unauthorised) have no explicit or implicit\n");
              ui.addOutputError("expectation of privacy.\n");
              ui.addOutputError("\n");
              ui.addOutputError("It is a criminal offence to:\n");
              ui.addOutputError("  i. Obtain access to data without authority\n");
              ui.addOutputError("       (Penalty 2 years imprisonment)\n");
              ui.addOutputError("  ii Damage, delete, alter or insert data without authority\n");
              ui.addOutputError("       (Penalty 10 years imprisonment)\n");
              ui.addOutputError("\n");
              ui.addOutputError("For more information, see http://www.randomnoun.com/login-banner.html\n");
              var sp = ui.startScpProgress();
              sp.setProgress("somefile.tar.gz", 0, "0", "0.0KB/s", "--:--");
              sp.setProgress("somefile.tar.gz", 45, "2112KB", "2.1MB/s", "00:01");
              sp.setProgress("somefile.tar.gz", 47, "2208KB", "1.9MB/s", "00:01");
              sp.setProgress("somefile.tar.gz", 49, "2320KB", "1.7MB/s", "00:01");
              sp.setProgress("somefile.tar.gz", 52, "2448KB", "1.5MB/s", "00:01");
              sp.setProgress("somefile.tar.gz", 54, "2576KB", "1.4MB/s", "00:01");

                ... 20 lines omitted ...

              sp.setProgress("somefile.tar.gz", 98, "4624KB", "266.7KB/s", "00:00");
              sp.setProgress("somefile.tar.gz", 100, "4693KB", "187.7KB/s", "00:25");
              ui.stopScpProgress(0);


   Example 3 (custom javascript output)
       The command

              scpwrap --js --stderrTemplate '' --stdoutTemplate '' \
                --startTemplate '' --endTemplate '' \
                --progressTemplate 'setProgress(%p);\n'
                -- -i key.pem somefile.tar.gz \
                user@somehost:/home/user/somefile.tar.gz

       might produce output something similar to the following:

              setProgress(0);
              setProgress(45);
              setProgress(47);

                 ... 20 lines omitted ...

              setProgress(98);
              setProgress(100);

BUGS
       It might be preferable to get the default  strings  from  something  in
       /etc

       The whole thing's a bit pointless

AUTHOR
       Greg Knox 

LICENCE
       (c) 2013 randomnoun. All Rights Reserved. This work is licensed under a
       BSD Simplified License. (http://www.randomnoun.com/bsd-simplified.html)

SEE ALSO
       vmaint(1), http://www.randomnoun.com/wp/2013/10/31/progress-bars/



vmaint                           OCTOBER 2013                       SCPWRAP(1)

If you want to install it in binary form, I've created an apt repository for a couple of ubuntu distribution/architecture combinations (maverick/amd64 and precise/i386), which you can add to your /etc/apt/sources.list file automatically using these commands:
$ sudo add-apt-repository "deb http://packages.randomnoun.com/ubuntu $(lsb_release -sc) main"
$ gpg --keyserver  hkp://pool.sks-keyservers.net --recv-key A21A1486 && gpg --export --armor A21A1486 | sudo apt-key add -
gpg: requesting key A21A1486 from hkp server pool.sks-keyservers.net
gpg: key A21A1486: "Greg Knox " not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
OK
$ sudo apt-get update
Get:1 http://packages.randomnoun.com maverick Release.gpg [198B]
Get:2 http://packages.randomnoun.com maverick Release [2,242B]
Ign http://packages.randomnoun.com maverick/main Sources
Ign http://packages.randomnoun.com maverick/main amd64 Packages
...
$ sudo apt-get install scpwrap
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  scpwrap
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 12.0kB of archives.
After this operation, 69.6kB of additional disk space will be used.
Get:1 http://packages.randomnoun.com/ubuntu/ maverick/main scpwrap amd64 1.0-1 [12.0kB]
Fetched 12.0kB in 0s (113kB/s)
Selecting previously deselected package scpwrap.
(Reading database ... 59957 files and directories currently installed.)
Unpacking scpwrap (from .../scpwrap_1.0-1_amd64.deb) ...
Processing triggers for man-db ...
Setting up scpwrap (1.0-1) ...

If you don't have add-apt-repository installed in your OS, then try sudo apt-get install python-software-properties first (or you could manually edit /etc/apt/sources.list instead).

Update 4/11/2013: Added instructions for the packages.randomnoun.com apt repository; added BSD licence to source code

[1] Something using modern philosophies, no doubt.
Tags:
2 Comments

Add a Comment

Your email address will not be published. Required fields are marked *