Gitslave Home Page: http://gitslave.sf.net
gits - The git slave repository tool for multi-repository management
gits [-p|--parallel COUNT] [-v|--verbose]+ [--quiet] [--help] [--version] [-n|--no-pager] [--paginate] [--eval-args] [--exclude SLAVE-REGEXP] [--keep-going] [--no-commit] [--no-hide] [--no-progress] [--no-master] [--with-ifpresent|--just-ifpresent] SUBCOMMAND [ARGS]...
gits is a program that assists in assembling a meta-project from a number of individual git repositories which operate (when using gits) as if they were one git repository instead of many, similar to the way that CVS works by default and svn (v1.5) can be coerced to work. Some of these individual git repositories may be part of other meta-projects as well.
Unfortunately, the functionality provided by git-submodule is not sufficient for this mode of operation. Most git commands, like checkout or commit, do not recursively descend into the submodules so you are forced to do execute all git commands N+1 times (leading to pain and mistakes), also, submodules revisions are tracked in the supermodule so that changes to a submodule made outside the supermodule are not automatically seen. Since git does not allow partial checkouts, we are left with little alternative.
Thus, to solve these problems gits was born. Complexity pain is still involved, but the hope is that it is minimized compared to all of the other alternatives.
The basic theory is that there are a few sub-commands (prepare, attach, populate) which help set up the meta-project for gits operations. Then, except for git commands with specific filenames, like git add FILENAME; git reset FILENAME; etc., you should use "gits" instead of "git", and the command will run on all repositories in the project.
In the following example, we will have the following master repositories:
ssh://sourcemaster/src/repos/super.git ssh://sourcemaster/src/repos/lib1.git ssh://sourcemaster/src/repos/lib2.git
The desired working layout of directories with .git in them on disk is:
..../super ..../super/lib1 ..../super/lib2
gits clone [--[no-]fromcheckout] [--nohooks] [--reference[checkout]=PATH] SUPERPROJECT-URL
This command clones an existing superproject and all associated slave repositories. It runs `git clone` on the arguments you provide (allowing you to change the name of the checked-out repository for example) and then runs `gits populate` inside that newly checked out repository. If you are cloning from an existing gitslave checkout instead of the master layout (the master layout is specified in the gits attach commands), you would want to use the --fromcheckout argument.
By default, if there is a git-hooks directory present in the superproject, hooks there will be symlinked into the .git/hooks directory in all related repositories. Adding --nohooks will disable this management.
-------------------------------------------------- gits clone ssh://sourcemaster/src/repos/super.git super2 cd super2 --------------------------------------------------
The gits option --with-ifpresent can be used with this command to populate all conditional slave repositories (those marked with ifpresent flag).
The clone options --reference=PATH or --referencecheckout=PATH work like the `git clone --reference` option, modifying the path based on the normal rules (the --fromcheckout rules are used with --referencecheckout) for upstream repository construction. You must use equals (=) to separate the PATH argument from the option, you cannot use whitespace as allowed by git.
The purpose of the --reference options is to avoid lengthy network copies if you already have a local repository. Note that the use of shared repositories created by --reference is potentially dangerous; see the notes on --shared in the git-clone man page and consider running `gits repack -a` to remove these linkages.
gits prepare
You run this command in the git directory which will be your top level master repository (super here). Typically you would clone this top level master from some other location which has all of your git projects.
-------------------------------------------------- git clone ssh://sourcemaster/src/repos/super.git super cd super gits prepare --------------------------------------------------
gits attach [--recursive=FILENAME] [--reference=PATH] [--adminonly] REPOSITORY LOCALPATH [FLAGS]
Clone the named git repository into the named local directory, and set it up for further gits operations.
Typically LOCALPATH would be a path relative to the top level working directory, for example, a subdirectory in the top level.
REPOSITORY can also be relative (relative to the URL the top level master checkout was cloned from). It may also be an absolute URL but if so, `gits remote add` is not going to be happy. If the URL starts with ^, it will use only the method and hostname from the master's URL. Otherwise, it will be relative to the fully qualified path. We will show both in operation in the example.
The only [flags] currently supported is "ifpresent" which will be set for this slave repository. Other people cloning or populating the superproject will not check out this subrepository if this flag is set, unless they add --with-ifpresent or otherwise arrange for the LOCALPATH to be created.
If the --recursive=FILENAME argument is present (and you must use the --recursive=FILENAME style, not "--recursive FILENAME") the repository you are attaching will be treated as a recursive gitslave master underneath the top-level gitslave master. The filename will typically be ".gitslave". It is relative to the localpath you checked out and must not have any / in it.
The attach option --reference=PATH works exactly like the `git clone --reference` option. In fact this option will be passed as-is to the `git clone` command. You must use equals (=) to separate the PATH argument from the option, you cannot use whitespace as allowed by git.
The purpose of the --reference option is to avoid lengthy network copies if you already have a local repository. Note that the use of shared repositories created by --reference is potentially dangerous; see the notes on --shared in the git-clone man page and consider running `gits repack -a` to remove these linkages.
If the --adminonly option is given, this tells gitslave to set up the management files and to NOT clone. You must clone through a subsequent prepare or some manual action.
-------------------------------------------------- gits attach ../lib1.git lib1 gits attach ^/src/repos/lib2.git lib2 gits attach --recursive=.gitslave ../../super2 supersub gits push --------------------------------------------------
The push in the example is to share the attach with other users.
gits populate [--[no-]fromcheckout] [--nohooks] [--reference[checkout]=PATH] [SLAVES]...
Go through the list of configured slaves and check out (clone) any which have not already been retrieved.
With --fromcheckout, assume that the remote repository is a gits checkout instead of in standard repository layout. With --no-fromcheckout, assume that the remote repository has the standard layout. If either option is given, sets the default repository layout, which is used when no explicit option is provided.
The gits option --with-ifpresent can be used with this command to populate all conditional slave repositories (those marked with ifpresent flag). You can also populate particular conditional slave repositories by listing them on the command line.
By default, if there is a git-hooks directory present in the superproject, hooks there will be symlinked into the .git/hooks directory in all related repositories. Adding --nohooks will disable this management.
The populate options --reference=PATH or --referencecheckout=PATH work like the `git clone --reference` option, modifying the path based on the normal rules (the --fromcheckout rules are used with --referencecheckout) for upstream repository construction. You must use equals (=) to separate the PATH argument from the option, you cannot use whitespace as allowed by git.
The purpose of the --reference options is to avoid lengthy network copies if you already have a local repository. Note that the use of shared repositories created by --reference is potentially dangerous; see the notes on --shared in the git-clone man page and consider running `gits repack -a` to remove these linkages.
-------------------------------------------------- gits populate --------------------------------------------------
gits release [-n] --all | --just-ifpresent | SUPERPROJECT | SLAVES...
Go through the list of configured slaves (or slave arguments) and confirm that there are no unresolved conflicts, untracked (and not ignored) files, uncommitted changes (including staged or stashed changes), unpushed commits, or unmerged private branches, and that there is a tracking branch in each repository. If these conditions are met in all of the selected slaves, the selected slave directories are removed (rm -rf).
The --no-commit (or -n) option prevents repositories from being removed. (Note that if -n is passed to 'gits' instead of the 'release' subcommand, it also disables automatic pagination as --no-pager). This option is implied if the superproject is the root directory (/) or if paths to any slave repositories start with dot (.), contain '..' (parent directory) components, or are absolute (start with /).
The --just-ifpresent option can be used to release only conditional slave repositories (marked with ifpresent flag). This option cannot be used with slave repository arguments or --all.
The --all option releases the entire current superproject; unless -n, --no-commit, or --no-master is used, the current directory is removed, breaking relative paths until you change directory. Instead, you can pass one superproject as an argument with an absolute path (starting with /); this will work from any directory.
Other arguments are treated as the names of slave repositories to release. Only one recursive superproject can be specified, and only as an absolute path. Slave arguments cannot start with dot (.) or contain certain shell metacharacters (semicolon, newline, or quotes). Slave arguments cannot be combined with --all or --just-ifpresent options.
The --force option can be used to force removal even if the state of some repositories does not meet the conditions noted above. This is EXTREMELY DANGEROUS and roughly equivalent to rm -rf on the arguments; for this reason the --force option cannot be combined with --all or a superproject argument.
-------------------------------------------------- gits release --just-ifpresent gits release lib1 gits release -n --all gits release /path/to/super2 --------------------------------------------------
gits detach [--force] SLAVE
Do a `gits release` on SLAVE; if this succeeds, besides removing the slave repository from the local filesystem, also remove the slave from .gitignore and the .gitslave management file, so that it will not be used in subsequent gits activities.
The --force option can be used to override the gits release check; if this is done, the repository will not be removed - only .gitignore and .gitslave will be affected.
The --no-commit option will only remove the slave from .gitignore and .gitslave; those changes will not be committed, and the slave repository will not be removed from the local filesystem.
-------------------------------------------------- gits detach lib1 gits push --------------------------------------------------
The push in the example is to share the detach with other users.
gits pulls [pull args]
For each branch being tracked by the superproject, go through the list of configured slaves, check yourself out to the branch, perform a pull, then switch back to the branch you were on.
Please see SUBSTITUTION for information on how to replace part of the command with the slave repository name for each executed command.
In the common case where no repository or refspec is provided as an argument, pulls will perform a fetch once (in the current branch of each slave) and then rebase (with --preserve-merges) or merge in each branch. This reduces redundant network overhead in the common case where all branches of a repository have the same remote, but if that is not the case, only the current remote will be fetched. If you want to force the slower operation of a pull in each branch, pass the -- argument, e.g. gits pulls --rebase --
-------------------------------------------------- gits pulls --rebase --------------------------------------------------
gits pull [pull args]
Go through the list of configured slaves and perform a pull, Note that even though only the current branch HEAD will be advanced, the commits on other branches will still be fetched.
Please see SUBSTITUTION for information on how to replace part of the command with the slave repository name for each executed command.
-------------------------------------------------- gits pull --rebase -------------------------------------------------- gits pull otherhost:/src/work/wb/%%dir%% --------------------------------------------------
gits logs [log args]
For each branch being tracked by the superproject, generate a list of the "log" messages as specified by the log args, and output them in a fixed format, with related commits grouped together and ordered by time. Do not provide git log options that modify the output format, as they will break the ordering and grouping functionality; only arguments that control the selection of commits should be used. (Note that using the --date={relative,local,default,iso,rfc,short} option is okay, and will modify the displayed date format).
The related commit grouping will group together all commits (in any sub-project) within an 8-hour period that have the same author e-mail and commit message.
-------------------------------------------------- gits logs HEAD...Product-3.1.1 --------------------------------------------------
gits exec COMMAND [ARGS]
For each slave being tracked by the superproject and the superproject itself, cd to the slave directory and execute the listed command.
Please see SUBSTITUTION for information on how to replace part of the command with the slave repository name for each executed command.
-------------------------------------------------- gits exec gitk -------------------------------------------------- gits exec git diff --------------------------------------------------
gits resolve [--[no-]fromcheckout] REPOSITORY LOCAL_RELPATH [REMOTE]
Go through the same process that gits uses for resolving relative repository URLs into absolute URLs, for debugging and certain porting efforts. You may specify the remote of the superproject you wish the relative repository to be in relation to: by default it uses origin.
With --fromcheckout, resolve URLs as if the repository were a clone of another gits checkout. With --no-fromcheckout, resolve URLs as if it were not a clone of another gits checkout. By default it uses the saved repository layout from gits populate or update-remote-url; however gits resolve does not change the saved default even if --fromcheckout or --no-fromcheckout is given.
-------------------------------------------------- gits resolve ../otherpos otherpos gits resolve ../otherpos otherpos otherremote --------------------------------------------------
gits update-remote-url [--[no-]fromcheckout] REMOTENAME NEWURL
Update the superproject's remote.REMOTENAME.url to be NEWURL. Then go though each slave repository and update its remote.REMOTENAME.url using the normal relative url mechanism.
The --fromcheckout option supports using a gits checkout as the remote repository, adjusting the repository paths from their default. The --no-fromcheckout option assumes a normal (as specified in the gits attach commands) repository layout for the remote. If neither option is given, the saved repository layout from the most recent gits populate or update-remote-url command is used; however, this is not generally correct, as the repository layout of the new remote need not be the same as the old one. If either --fromcheckout or --no-fromcheckout is given, sets the default repository layout accordingly.
You can use this command after a clone of a local repository (or local gits checkout, using gits populate --fromcheckout) so that the new repository uses the same remote origin as the first one. (The local clone/populate is much faster than performing a remote clone/populate.)
-------------------------------------------------- git clone /home/user/work/wb /home/user/work/newwb cd /home/user/work/newwb gits populate --fromcheckout gits update-remote-url --no-fromcheckout origin ssh://git/src/git/wb --------------------------------------------------
gits remote add [--[no-]fromcheckout] GIT-REMOTE-ADD-OPTS REMOTENAME REMOTEURL
Adds a remote named REMOTENAME for the repository at REMOTEURL (as modified by the relative URL rules from `gits attach` and `gits update-remote-url` and the --fromcheckout argument). The command gits fetch REMOTENAME can then be used to create and update remote-tracking branches matching REMOTENAME/*.
The superproject remote URL will be set to REMOTEURL, which should be an absolute URL; the slave repositories will have a modified version of that URL (the layout must match that of the existing remotes).
Please note that we do not currently support `gits remote set-url` in a useful way. See `gits update-remote-url` for an alternate method which will satisfy most use cases.
-------------------------------------------------- gits remote add fred --fromcheckout /home/fred/src/foo -------------------------------------------------- gits remote add backup ssh://backups/src/git/foo --------------------------------------------------
gits push [--quick] [push args]
This is the standard git push command--with the addition of a --quick option which will only attempt to push for branches which have outstanding changes. For slow connections to large number of slave repositories, the overhead of an empty push can be large.
However, --quick only checks to see if the current branch needs to push data. If you have changes on other branches, a slow push is still required, as it is if you are pushing to a repository other than the standard origin.
Please see SUBSTITUTION for information on how to replace part of the command with the slave repository name for each executed command.
-------------------------------------------------- gits push --quick -------------------------------------------------- gits push otherhost:/src/work/wb/%%dir%% --------------------------------------------------
gits statuses [-m] [status args]
For each branch being tracked by the superproject, go through the list of configured slaves, check yourself out to the branch, get the git status, then switch back to the branch you were on. The output is summarized for each branch to merge the lists of files in each section.
The -m option will attempt to "move" any uncommitted changes, which may prevent failures checking out other branches, at the risk of creating conflicts, which are then moved as well
Other args supported by `git status` are also supported; these are the same options supported by `git commit`.
-------------------------------------------------- gits statuses --------------------------------------------------
gits archive GIT-ARCHIVE-ARGS
This is the standard git archive command--with the addition of a new --format option of "gits-tar". When you select the gits-tar option, you must supply an on-disk --output file and you cannot use any tar compression options. With gits-tar, the output tar archive will be a unified archive of the entire project, superproject and slaves. Any existing prefix will be treated as a directory prefix (e.g. --prefix foo and --prefix foo/ are the same) and all slave repositories will be unpacked in their corresponding superproject locations.
If you choose a format other than gits-tar, you will probably want to use one of the substitutions like %%basename%% or %%dir%% in the output filename.
-------------------------------------------------- gits archive --format gits-tar -o /tmp/foo.tar master -------------------------------------------------- gits archive --format tar -o /tmp/foo-%%basename%%.tar master --------------------------------------------------
All other commands are passed directly though to git, with one command being run per repository. Output summarizing is performed so that multiple repositories with the same git output will only have the git output shown once. `git status` has a more aggressive summarizing to merge the lists of files in each section.
Examples: -------------------------------------------------- gits commit -a -m "This is a change" gits push gits pull gits branch testing gits checkout testing gits diff master testing gits status gits .... --------------------------------------------------
All normal git commands are supported (plus any potential future commands) but not all commands make sense to run with gits. One good example is git-daemon.
Specify the number of parallel git operations you wish to execute. Parallelism is only activated for push and pull(s) subcommands. This can speed up your processing significantly for large numbers of slave repositories.
If remote repositories are accessed over ssh, you may also wish to activate ssh "ControlMaster" multiplexing.
ssh_config
--------------------------------------------------
Host git
ControlMaster auto
ControlPath ~/.ssh/master-%r@%h:%p
--------------------------------------------------
However, there currently is a "auto" race condition so the first batch of peers do not necessarily take advantage of the multiplexing and there have occasionally been spurious errors with this enabled.
Ask for more information about what is happening. You may repeat the flag multiple times to get more information. One level of verbosity (-v) will print some minor warnings and will specify every repository and its output explicitly. Two levels of verbosity (-vv) will print the underlying commands being executed and three levels (-vvv) will print the data being returned from them.
Ask for less information, which currently means discarding the STDERR of some of the administrative git commands which are executed.
Do not go through output summariziation ("On repo" and leading spaces) for most gitslave commands. Note that the output may currently not be in the same repository order as it was originally. Implies --no-hide
Often used for post-processing output. Example:
rsync -aR `gits --rawout exec sh -c 'git ls-tree -r --name-only HEAD | sed "s:^:%%dir%%/:"'` otherhost:/otherdir
Print gits usage summary and details of gits options and substitutions from this documentation, then exit.
Print version information for gits, git, and Perl.
Disable or re-enable default pagination of gits output using a pager. The pager (default is less) and its options are configured just as for git itself (using core.pager in the master repository or global settings, or environment variables PAGER or GIT_PAGER, with the same precedence).
Pagination is only enabled if standard output is a tty (and there is a controlling tty for the pager to take input from). Pagination is disabled if the configured pager (from config and environment) is set to the value "cat" or the empty string.
The -n option will disable pagination, but will be overridden by any --paginate or --no-pager arguments present on the command line, even if the -n option is given later.
When quoting gits arguments, do not quote dollar '$' and backtick '`' characters, to allow interpolation in the slave environment (but still quote double-quote and backslash). This is mostly useful for exec, where you might want something like the command below to do something useful.
gits --eval-args exec echo Directory is '`pwd`'
Provide a regular expression which excludes those slaves from consideration from gits commands which it matches.
Do not abort when any subsidiary git command fails - instead print a warning and continue processing. Some git command failures will still be considered fatal and cause gits to abort.
This flag requests that gits-internal sub-commands, such as prepare or attach, should not commit their changes after they make them.
This flag requests that gits not hide information when similar (but not identical) output such as commit hashes is output for slave repositories.
This flag requests that gits NOT print a progress bar, which it does by default for slow operations if Term::ProgressBar is loaded. Slow operations are the checkout, fetch, pull(s), and push subcommands. You may use this flag for all operations.
This flag requests that gits only run the listed command on the slave repositories and NOT the super/master/top repository.
Operate also on those slave repositories which are marked as "ifpresent" even if they are not present. This is mostly useful for `gits populate` and `gits checkout`.
Operate only on those slave repositories which are marked as "ifpresent" whether they are present or not. This is mostly useful for `gits release`. Note that this implies --no-master, and overrides --with-ifpresent.
Run the specified git command (with associated arguments) on the repository and all slave repositories. Typically they are git commands run over each slave, but there are gits specific commands such as: pulls, prepare, attach, populate, resolve, exec, logs, and update-remote-url. See OVERVIEW for more information on specific subcommands.
Before execution, essentially all commands running over all of the repositories will go through a substitution phase where certain magic tokens will be replaced with information about the repository in question. These are most often used with `gits exec` and `gits archive`.
%%dir%% represents the on-disk .gitslave-relative of the repository (e.g. the second field in .gitslave) with the superproject getting the value of ".".
%%path%% represents the fully qualified path to the repository in question.
%%basename%% represents the basename of the %%path%%, which is typically the last component of %%dir%% except for the superproject for which it is whatever the last directory name of the path to the superproject.
%%upstream%% represents the URL to the origin repository for the repository in question.
%%upstream_base%% represents the basename to the URL to the origin repository for the repository in question.
# Create a shadow set of bare repositories locally w/o massive transfers from origin gits exec git clone --bare --reference=%%path%% %%upstream%% /tmp/r/%%upstream_base%%
Also see --eval-args for an option to support standard shell `cmd` and $VARIABLE expansion where it might otherwise be quoted. Run the following commmands to see the difference:
gits exec echo '`ls -ld $PWD`'
gits --eval-args exec echo '`ls -ld $PWD`'
gits changes directory to the directory where .gitslave exists. Likewise, when executing most git commands, gits changes directory to the root of the git slave, so any pathnames passed as arguments to gits must be absolute, not relative. Generally this is only a concern for pre-generated commit messages or things like that; you should NOT be passing gits the pathnames of files checked into git slaves--you will likely get the wrong result.
No coding has been performed yet to handle `gits remote set-url` or `gits branch --set-upstream`. See `gits update-remote-url` for a supported method to perform this operation. Support could be added if necessary.
You can have partial success, failure, and repositories on which the operation was never tried and you must recover from such manually. This is usually not very complicated. See --keep-going.
Programs like gitk will not show the global system history.
Special care may be needed if one or more of the repositories is a third party repository and you plan to have a complex branch/tag management strategy, plan to do (public or private) development on the third party repository, or might sometimes not want the absolute latest code on the third party branch. See the gitslave home page for more information on workarounds.
The behavior when different branches have different slave repositories associated with them and you checkout back and forth is probably not ideal (nor are any of the options we have thought of completely ideal).
The file containing the list of slave repositories (possibly in relative form) and the directories relative to the master root where they should be checked out.
The format of this file is:
"possibly-relative-repository-path" "top-level-checkout-relative-path"[ flags]
The flags, which are optional, currently can be the value "ifpresent" which indicates that gits will only process this repository if the top-level-checkout-relative-path is already present.
The GITSLAVE environment variable specifies alternate location(s) of the .gitslave file. Note that the .gitslave file must still exist even if it is not used for this particular operation). GITSLAVE can be a filename or a list of filenames separated by comma and space; if a list of filenames is specified, it has the same effect as if the files were concatenated.
An example of a list of filenames, GITSLAVE=".gitslave, .gitslave-extras" would allow adding a supplemental list of slaves for unusual activity (e.g. release tagging) to the normal list.
You can also use an alternate .gitslave file with just a subset of the slave repositories when you don't want to run commands on all of them.
Note that if you are using recursive gitslave superprojects, the GITSLAVE environment variable overrides .gitslave at the top-level only. Only if the alternate .gitslave file(s) #include .gitslave (or alternate) in recursive superprojects will their slave repositories be included.
perl 5 (probably almost any version of perl 5)
git 1.6 or later (git 1.7 or later preferred)
Optionally uses Parallel::Iterator and Term::ProgressBar if available
Seth Robertson
Report bugs to http://sourceforge.net/projects/gitslave
Copyright (c) 2008 Seth Robertson. License is similar to the GNU Lesser General Public License version 2.1, see LICENSE.TXT for more details.
git(1), git-submodule(1), git-subtree(google)
Gitslave Home Page: http://gitslave.sf.net