6/21 What's the currently accepted "best" fastest way to write a lot of
little data to a file in Unix? Is it still mmap/memcpy, or is there
something more advanced nowadays? Maybe send me e-mail? -- Marco
\_ I'm curious as well, so please post responses here. --darin
\_ I think you will find that it depends entirely on the hardware
platform and in most cases the programming style only affects
a few percent on the fastest RAID drive arrays. this is because
RAM and CPU are so much faster than disks. trying to do high
bandwidth network I/O, on the other hand can be tricky and
interesting if you are into that sort of thing... zero-copy
asynchronous bulk I/O.
\- i agree, a lot of little details matter. is it totally
concurrency domainated, what kind of device is being
written to, is it ok to write to cache or do you need
to flush to metal, can you use some hack like immediate data
[veritas], is locking an issue? does your application only
use traditional sematics [supercomputing uses have special
ways of doing large i/o], can you choose your file system,
is byte range locking an issue? is the rate at which you
are given inodes an issue? etc. it would be interesting to
see how IBM/GPFS would do at this. |