.\" $NetBSD: vndcompress.1,v 1.15 2020/01/24 20:20:35 pgoyette Exp $ .\" .\" Copyright (c) 2013 The NetBSD Foundation, Inc. .\" All rights reserved. .\" .\" This documentation is derived from text contributed to The NetBSD .\" Foundation by Taylor R. Campbell. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" .\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS .\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED .\" TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR .\" PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS .\" BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR .\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF .\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS .\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN .\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) .\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE .\" POSSIBILITY OF SUCH DAMAGE. .\" .Dd January 24, 2020 .Dt VNDCOMPRESS 1 .Os .Sh NAME .Nm vndcompress , .Nm vnduncompress .Nd compress and uncompress disk images in cloop2 format .Sh SYNOPSIS .Nm .Op Fl c .Op Fl rR .Op Fl b Ar blocksize .Op Fl k Ar checkpoint-blocks .Op Fl l Ar length .Op Fl p Ar partial-offset .Op Fl w Ar winsize .Ar image .Ar compressed-image .Op Ar blocksize .Nm vnduncompress .Op Fl d .Op Fl w Ar winsize .Ar compressed-image .Ar image .Sh DESCRIPTION The .Nm utility compresses disk images in cloop2 format, which the kernel's .Xr vnd 4 device can interpret as read-only disk devices using the .Fl z option to .Xr vndconfig 8 . .Pp By default, .Nm vndcompress compresses an image, and .Nm vnduncompress uncompresses an image, but the .Fl c and .Fl d options can control whether either utility compresses or decompresses. .Pp The following options are available for the compression operation: .Bl -tag -width indent .It Fl b Ar blocksize Set the compression block size to .Ar blocksize , which must be a multiple of 512 and must be no more than 4294966784, or 0xfffffe00. (On 32-bit systems, the limit may be smaller, limited by the available virtual address space.) .Pp For compatibility with the old version of .Nm , the compression block size may instead be specified at the end of the command line. .It Fl k Ar checkpoint-blocks Write a checkpoint after every .Ar checkpoint-blocks blocks of output. If interrupted, .Nm can restart at the last checkpoint with the .Fl r option. You can also request a checkpoint at any time by sending .Dv SIGUSR2 to the .Nm process. .It Fl l Ar length Specify the length of the input, so that the input may be a pipe, socket, or FIFO. Otherwise, the input must know its size, i.e. must have its size reported in .Fa st_size by .Xr fstat 2 . .It Fl p Ar partial-blocks Stop after writing .Ar partial-blocks blocks of output. This option is mainly useful for automatic testing of restarts. .It Fl R With the .Fl r option, if restarting fails, then abort right now and don't touch the output file. .It Fl r Try to restart a partial compression from the last checkpoint. If restarting fails, and the .Fl R option is not specified, then truncate the output file and start compression afresh. Restarting may fail for various reasons: if there have been no checkpoints, or if the output file has been corrupted in some easily recognizable ways, or if the input file's size has changed, or if the block size does not match, and so on. .El .Pp The following option is available for both compression and decompression: .Bl -tag -width indent .It Fl w Ar winsize Use an in-memory window of .Ar winsize entries into the table of compressed block offsets. If .Ar winsize is zero, .Nm will use memory proportional to the number of blocks in the uncompressed image, namely 64 bits or 8 bytes per block. If .Ar winsize is nonzero, .Nm will use memory proportional to .Ar winsize , and independent of the size of the uncompressed image. .Pp A nonzero .Ar winsize requires the compressed image to be a seekable file, which compression requires anyway, in order to record the offsets of compressed blocks once they are compressed and written, but which is a limitation for decompression. Thus, decompressing from a pipe is incompatible with a nonzero .Ar winsize . .Pp By default, .Nm uses a fixed window size, unless decompressing with nonseekable input. .El .Sh EXIT STATUS .Ex -std .Sh EXAMPLES Compress an ISO 9660 CD-ROM image: .Bd -literal -offset indent # vndcompress cdrom.iso cdrom.izo .Ed .Pp Send a 59 GB disk partition over a local network with netcat (don't do this over the internet!): .Bd -literal -offset indent # nc 10.0.1.2 12345 < /dev/rcgd1h .Ed .Pp Receive it and save it compressed on another host, showing a progress bar interactively, restarting if possible, and taking a checkpoint every 16 MB (i.e., every 256 compression blocks, which are 64 KB by default): .Bd -literal -offset indent # nc -l 12345 | progress -l 59g \e vndcompress -l 59g -k 256 -r /dev/stdin disk.cloop2 .Ed .Pp If the process is interrupted and you rewire your network and disks so that the receiving host now has the disk you want to image, you can start up where you left off, using the .Fl R option to keep .Nm from clobbering the partial transfer if anything goes wrong: .Bd -literal -offset indent # vndcompress -l 59g -k 256 -rR /dev/rcgd1h disk.cloop2 .Ed .Pp Mount the disk with .Xr vnd 4 , assuming your kernel was built with the .Dv VND_COMPRESSION option enabled: .Bd -literal -offset indent # vndconfig -z vnd0 disk.cloop2 # mount /dev/vnd0d /mnt .Ed .Sh SIGNALS .Nm responds to the following signals: .Bl -tag -width indent .It Dv SIGINFO , Dv SIGUSR1 Report progress to standard error. .It Dv SIGUSR2 Write a checkpoint to disk now. .El .Sh FORMAT The cloop2 format consists of a header, an offset table, and a sequence of compressed blocks. The header is described by the following packed structure: .Bd -literal -offset indent struct cloop2_header { char cl2h_magic[128]; uint32_t cl2h_blocksize; uint32_t cl2h_n_blocks; } __packed; .Ed .Pp The .Fa cl2h_magic field is an arbitrary sequence of 128 bits, the .Fa cl2h_blocksize field is a big-endian integer number of bytes per compression block, and the .Fa cl2h_n_blocks field is a big-endian integer number of the compression blocks in the image. .Pp The offset table is a sequence of one more than .Fa cl2h_n_blocks big-endian 64-bit integers specifying the offset of each compression block relative to the start of the file. The extra offset specifies the end of the last compression block, which may be truncated if the uncompressed image's size is not a multiple of the compression block size. .Sh SEE ALSO .Xr vnd 4 , .Xr vndconfig 8 .Sh HISTORY The .Nm command first appeared in .Nx 3.0 . It was rewritten to be more robust, to support restarting partial transfers, and to support bounded memory usage in .Nx 7.0 .