www.catonmat.net/blog/unix-utilities-pipe-viewer
subscribe to 'good coders code, great reuse' posts via feed good coders code, great reuse More computing sins are committed in the name of efficiency (without necessarily achieving it) than for any other single reason - including blind stupidity.
A Unix Utility You Should Know About: Pipe Viewer Unix Utilities Hi all. This one is going to be about Unix utilities that you should know about. I'll try to write a good introduction to the tool and give as many examples as I can think of. Before I start, I want to clarify one thing - Why am I starting so many article series? The answer is that I want to write about many topics simultaneously and switch between them as I feel inspired. The first post in this series is going to be about not so well known Unix program called Pipe Viewer or pv for short. Pipe viewer is a terminal-based tool for monitoring the progress of data through a pipeline. It can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion.
Ok, let's start with some really easy examples and progress to more complicated ones. log" that is a few gigabytes in size and contains web logs. gz As the file is so huge (several gigabytes), you have no idea how long to wait. By using pv you can precisely time how long it will take.
Here we specified the "-N" parameter to pv to create a named stream. The "-c" parameter makes sure the output is not garbaged by one pv process writing over the other. Notice how the gzip does not include how much data is left or how fast it will finish. It's because the pv process after gzip has no idea how much data gzip will produce (it's just outputting compressed data from input stream). The first pv process, however, knows how much data is left, because it's reading it. Another similar example would be to pack the whole directory of files into a compressed tarball: $ tar -czf - .
In this example pv shows just the output rate of "tar -czf" command. Not very interesting and it does not provide information about how much data is left. We need to provide the total size of data we are tarring to pv, it's done this way: $ tar -cf - .
Next we specify the size "-s" to pv of all files in current dir. This way "pv" knows how much data is still left to be processed and shows us that it will totally take 4 mins 49 secs to finish. Another fine example is copying large amounts of data over network by using help of "nc" utility that I will write about some other time. Suppose you have two computers A and B You want to transfer a directory from A to B very quickly. The fastest way is to use tar and nc, and time the operation with pv. All the files in /path/to/dir on computer A will get transferred to computer B, and you'll be able to see how fast the operation is going.
Have fun measuring your pipes with pv, and until next time! A question to my readers: what other not so well known Unix utilities do you use and/or know know about?
February 2nd, 2009 at 6:15 pm @Daniel Watkins, yes, obviously because there is one more process in the pipline, but judging by Alexandru's example it won't slow things down by much.
February 2nd, 2009 at 9:30 pm Before I found pv, I used "buffer -z 512K" to get pipe flow feedback. It doesn't have a concept of total size, so you end up doing the ETA calculations yourself. Are there any performance ramifications of inserting this into the stream?
February 2nd, 2009 at 10:19 pm Wow thanks for the great post. I've been a UNIX user for 12 years and although I've needed PV many times, I didn't know it existed.
more like lames anyway lol bc they are lame games (= lames) Also, Dos is way more secure and it is the basis of unix, making it a more developed language. Even linus torvalds has admitted to using dos for all important work.
February 3rd, 2009 at 12:57 pm I found pv a while back, the problem was, I kept forgetting to include it in the original slow command. After the gzip, disk image or what ever had been running for a while, I wanted a progress bar, but not to have to restart the operation. A handy little function to solve this problem is: rate () { tailf $1 | pv > /dev/null } It tails the end of a file through pv. It won't give "how long to go" information, but it will show the throughput, and that can often be enough to figure out how long the operation will take.
February 3rd, 2009 at 4:34 pm Not really a utility, but the -exec flag of find is incredibly useful, especially when you find yourself looping over a list of files and performing some operation on them in bash.
February 3rd, 2009 at 10:36 pm This was quite an interesting and useful article, Peteris - thanks. Also, it looks like there is more good info in the many comments, which I must read. Some points by me: Like some other commenters above, I recommend xargs, lsof and fuser as useful tools. find and xargs together are a powerful combination - find lets you find all files under some directory tree that match some criteria, and xargs lets you execute a command on all those found files. Of course the command executed can be a shell script, which means that many commands (in the script) can be executed on each of those files. Mig: >I'd like to read an article about more obscure utilities like od', nm' or objdump'. Do they have any use in general sys administration od definitely is useful for both system administrators as well as developers and general users. A common use of it is to display the contents of a file in one of many possible formats like: - as characters - as bytes in octal or hexadecimal - as words in decimal - etc. This is useful to: - see what the file contains, particularly if you don't have a "native" viewer app for it. One common use of it is to display / dump the names of the symbols defined in object files.
Leave a Comment Name Mail (will not be published) Website XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <pre> <em> <i> <strike> <strong> To leave code snippets use the <pre> tag! Please note: Comment moderation is enabled and may delay your comment.
|
http://csua.com/feed/