Category Archives: CLI

CLI – rsync

The rsync command is a utility common to Linux, Unix, BSD and macOS. Versions of rsync are now available on Windows systems as well.

This command is used to synchronize files and directories. This can be done between locally attached storage, or between two different network connected systems. Design of the rsync algorithm minimizes the network usage, while still maintaining file synchronization.

The rsync daemon can be run on a machine to allow other remote machines to copy file to or from it. The rsync command requires arguments indicating the source and destination locations.

There are many different command line options available to rsync. The general format is: rsync options source destination

One of the nice features of rsync is that it only copies files that have changed since the last time they were transferred. Check the reference list at the bottom of this article for suggestions of the best set for your application. Some of the simplest are:

  • rsync -avh /home/usr/dir/ /media/disk/backup/
    – this copies everything in the directory /home/usr/dir/ to /media/disk/backup/
  • rsync -avh –delete /home/user/dir/ /media/disk/backup – does the same as above except that files deleted from /home/user/dir/ will also be deleted from /media/disk/backup
  • rsync –progress -avh /home/usr/dir/ /media/disk/backup/ – does the same as the first example, but show how much of the copy is remaining

When I run rsync -h on my macOS 10.11.6 system I get the following list of options:

rsync is a file transfer program capable of efficient remote update
 via a fast differencing algorithm.

Usage: rsync [OPTION]... SRC [SRC]... DEST
 or   rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
 or   rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST
 or   rsync [OPTION]... SRC [SRC]... rsync://[USER@]HOST[:PORT]/DEST
 or   rsync [OPTION]... [USER@]HOST:SRC [DEST]
 or   rsync [OPTION]... [USER@]HOST::SRC [DEST]
 or   rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]

The ':' usages connect via remote shell, while '::' & 'rsync://' usages connect to an rsync daemon, and require SRC or DEST to start with a module name.

Options

-v, --verbose               increase verbosity
 -q, --quiet                 suppress non-error messages
 --no-motd               suppress daemon-mode MOTD (see manpage caveat)
 -c, --checksum              skip based on checksum, not mod-time & size
 -a, --archive               archive mode; same as -rlptgoD (no -H)
 --no-OPTION             turn off an implied OPTION (e.g. --no-D)
 -r, --recursive             recurse into directories
 -R, --relative              use relative path names
 --no-implied-dirs       don't send implied dirs with --relative
 -b, --backup                make backups (see --suffix & --backup-dir)
 --backup-dir=DIR        make backups into hierarchy based in DIR
 --suffix=SUFFIX         set backup suffix (default ~ w/o --backup-dir)
 -u, --update                skip files that are newer on the receiver
 --inplace               update destination files in-place (SEE MAN PAGE)
 --append                append data onto shorter files
 -d, --dirs                  transfer directories without recursing
 -l, --links                 copy symlinks as symlinks
 -L, --copy-links            transform symlink into referent file/dir
 --copy-unsafe-links     only "unsafe" symlinks are transformed
 --safe-links            ignore symlinks that point outside the source tree
 -k, --copy-dirlinks         transform symlink to a dir into referent dir
 -K, --keep-dirlinks         treat symlinked dir on receiver as dir
 -H, --hard-links            preserve hard links
 -p, --perms                 preserve permissions
 --executability         preserve the file's executability
 --chmod=CHMOD           affect file and/or directory permissions
 -o, --owner                 preserve owner (super-user only)
 -g, --group                 preserve group
 --devices               preserve device files (super-user only)
 --specials              preserve special files
 -D                          same as --devices --specials
 -t, --times                 preserve times
 -O, --omit-dir-times        omit directories when preserving times
 --super                 receiver attempts super-user activities
 -S, --sparse                handle sparse files efficiently
 -n, --dry-run               show what would have been transferred
 -W, --whole-file            copy files whole (without rsync algorithm)
 -x, --one-file-system       don't cross filesystem boundaries
 -B, --block-size=SIZE       force a fixed checksum block-size
 -e, --rsh=COMMAND           specify the remote shell to use
 --rsync-path=PROGRAM    specify the rsync to run on the remote machine
 --existing              skip creating new files on receiver
 --ignore-existing       skip updating files that already exist on receiver
 --remove-source-files   sender removes synchronized files (non-dirs)
 --del                   an alias for --delete-during
 --delete                delete extraneous files from destination dirs
 --delete-before         receiver deletes before transfer (default)
 --delete-during         receiver deletes during transfer, not before
 --delete-after          receiver deletes after transfer, not before
 --delete-excluded       also delete excluded files from destination dirs
 --ignore-errors         delete even if there are I/O errors
 --force                 force deletion of directories even if not empty
 --max-delete=NUM        don't delete more than NUM files
 --max-size=SIZE         don't transfer any file larger than SIZE
 --min-size=SIZE         don't transfer any file smaller than SIZ
 --partial               keep partially transferred files
 --partial-dir=DIR       put a partially transferred file into DIR
 --delay-updates         put all updated files into place at transfer's end
 -m, --prune-empty-dirs      prune empty directory chains from the file-list
 --numeric-ids           don't map uid/gid values by user/group name
 --timeout=TIME          set I/O timeout in seconds
 -I, --ignore-times          don't skip files that match in size and mod-time
 --size-only             skip files that match in size
 --modify-window=NUM     compare mod-times with reduced accuracy
 -T, --temp-dir=DIR          create temporary files in directory DIR
 -y, --fuzzy                 find similar file for basis if no dest file
 --compare-dest=DIR      also compare destination files relative to DIR
 --copy-dest=DIR         ... and include copies of unchanged files
 --link-dest=DIR         hardlink to files in DIR when unchanged
 -z, --compress              compress file data during the transfe
 --compress-level=NUM    explicitly set compression level
 -C, --cvs-exclude           auto-ignore files the same way CVS does
 -f, --filter=RULE           add a file-filtering RULE
 -F                          same as --filter='dir-merge /.rsync-filter'
 repeated: --filter='- .rsync-filter'
 --exclude=PATTERN       exclude files matching PATTERN
 --exclude-from=FILE     read exclude patterns from FILE
 --include=PATTERN       don't exclude files matching PATTERN
 --include-from=FILE     read include patterns from FILE
 --files-from=FILE       read list of source-file names from FILE
 -0, --from0                 all *-from/filter files are delimited by 0s
 --address=ADDRESS       bind address for outgoing socket to daemon
 --port=PORT             specify double-colon alternate port number
 --sockopts=OPTIONS      specify custom TCP options
 --blocking-io           use blocking I/O for the remote shell
 --stats                 give some file-transfer stats
 -8, --8-bit-output          leave high-bit chars unescaped in output
 -h, --human-readable        output numbers in a human-readable format
 --progress              show progress during transfer
 -P                          same as --partial --progress
 -i, --itemize-changes       output a change-summary for all updates
 --out-format=FORMAT     output updates using the specified FORMAT
 --log-file=FILE         log what we're doing to the specified FILE
 --log-file-format=FMT   log updates using the specified FMT
 --password-file=FILE    read password from FILE
 --list-only             list the files instead of copying them
 --bwlimit=KBPS          limit I/O bandwidth; KBytes per second
 --write-batch=FILE      write a batched update to FILE
 --only-write-batch=FILE like --write-batch but w/o updating destination
 --read-batch=FILE       read a batched update from FILE
 --protocol=NUM          force an older protocol version to be used
 -E, --extended-attributes   copy extended attributes
 --cache                 disable fcntl(F_NOCACHE)
 -4, --ipv4                  prefer IPv4
 -6, --ipv6                  prefer IPv6
 --version               print version number
 (-h) --help                  show this help (-h works with no other options)

References:

  1. How to Backup Linux? 15 rsync Command Examples
  2. Do-It-Yourself Backup System Using Rsync
  3. Back up like an expert with rsync

See my other Command Line articles


 

CLI – cat

The cat command is used to ‘concatenate and print files’. Here, the term ‘print’ simply means to display the content of a file to the
standard output device, typically the console. This command is only useful with common text files.

An example of its use is:

Johns-Mac-mini:~
jpurvis$ ls Desktop/
BitTorrent Sync    Scott       jpurvis alias
IMG_0204.jpg    cat.txt       w.jpg
Johns-Mac-mini:~ jpurvis$ cat Desktop/cat.txt
This is a test of the ‘cat’ command

This command is useful to display the content of a file.

The Apple Man page for the ln command can be seen here.


See my other CLI articles


CLI – ln

The ln command allows you to make links in the filesystem. Specifically, it creates a local directory that points to the target file or directory. This allows you to

  • have a file or directory be virtually present in multiple locations, yet only have one copy taking up space and requiring maintenance
  • allow for content spread across multiple disk drives to appear as contiguous in the main drive.

The general use is with the -s option to specify a symbolic link. In the example below I create a symbolic link to the Pictures folder on the external drive D2, then show how it appears when the ls command is run.

Johns-Mac-mini:~ jpurvis$ sudo ln -s /Volumes/D2/Pictures Pictures
Johns-Mac-mini:~ jpurvis$ ls -l
total 8
drwx——   3 jpurvis  staff   102 Dec 28 14:32 Applications
drwxr-xr-x  71 jpurvis  staff  2414 Feb 19 23:43 Calibre Library
drwx——+ 13 jpurvis  staff   442 Mar 30 12:49 Desktop
drwx——+ 15 jpurvis  staff   510 Mar 29 16:38 Documents
drwx——+ 88 jpurvis  staff  2992 Mar 30 14:26 Downloads
drwx——@ 10 jpurvis  staff   340 Nov  6 11:22 Dropbox
drwx——@  8 jpurvis  staff   272 Mar 23 18:40 Google Drive
drwx——+ 71 jpurvis  staff  2414 Mar  6 09:24 Library
drwx——+ 12 jpurvis  staff   408 Nov  6 09:54 Movies
drwx——+  9 jpurvis  staff   306 Dec 15  2008 Music
lrwxr-xr-x   1 root     staff    20 Mar 30 14:45 Pictures -> /Volumes/D2/Pictures

The Apple Man page for the ln command can be seen here.


See my other CLI articles


CLI – sudo

The sudo command (substitute user do) allows a user to execute a command with the privileges and identity of another user. Many Linux systems and OS X come with sudo pre-configured allowing any user designated to be an administrative user to use the sudo command to execute commands with root authority.

The flat file /etc/sudoers is used to configure which users are allowed to execute which commands on the system.

While there are other options to the sudo command, most users will be using it to run a command with root authority, such as

sudo chgrp wheel Users

The user will be prompted for their user password before being allowed to run the command

See Apple’s Mac OS X Manual Pages for further command details.


See my other CLI articles


CLI – ls

The ls command is used to ‘list directory contents’. The syntax of the command is

ls [options]… [file]…

What are some of the options that can be used:

  • -a, –all  do not hide entries starting with .
  • -d, –directory  list directory entries instead of contents
  • -l   use a long listing format
  • -R, –recursive   list subdirectories recursively
  • -s, –size   print size of each file, in blocks
  • -S   sort by file size

More options for the ls command can be found here.

This command is used to get a list of the files and directories contained in the current directory. The most common usage is going to ls or ls -al. The simplest version of just ls will list all of the files and directories by name. If you choose to use the more refined ls -al, you will see the files and directories, but you will also see far more information: permissions, ownership, size and when last modified. Unless another option is selected, the files will be listed in alphabetical ascending order by default. You can see the difference between the two in the image below.

CLI - ls
See Apple’s Mac OS X Manual Pages for further command details.

References:

  1. 16 Practical Examples of Linux LS command for Beginners

See my other CLI articles


CLI – cd

Another of the most fundamental commands that the user can enter via CLI is cd.

The cd command changes the directory that the users is currently located in. The proper syntax is ‘cd destination‘. What the destination is can vary widely. For example if you enter:

  • cd – with no destination will return the user to their home
    directory
  • cd .. – will move you up one level in the directory structure
  • cd dir1 – will move you to the directory dir1 within the current directory
  • cd /etc – will move you to the /etc directory

CLI - cd

See Apple’s Mac OS X Manual Pages for further command details.


See my other CLI articles


CLI – man

One of the most fundamental, yet useful commands that the user can enter via CLI is ‘man’. The ‘man’ command calls up the manual page for the designated command. Unix/Linux OS X systems are loaded in most cases with a set of ‘manual pages’ for the various commands that the respective systems support.

CLI - man

So for instance if I go to the Terminal on my Mac (Go >> Utilities >> Terminal) and type in ‘man man’, I will get the manual page for the man command. To leave the man page display, just enter ‘q’.

CLI - man-1

This provides a quick way to find out what a command does, verify
the syntax to use and see what options are available.


See my other CLI articles