3.3 Advanced Git procedures

It is possible to work with several branches on the same local Git repository; this is especially useful for translators who may have to deal with both translation and a stable branch, e.g. stable/2.12.

Some Git commands are introduced first, then a workflow with several Git branches of LilyPond source code is presented.

3.3.1 Merge conflicts

To be filled in later, and/or moved to a different section. I just wanted to make sure that I had a stub ready somewhere.

3.3.2 Advanced Git concepts

A bit of Git vocabulary will be explained below. The following is only introductory; for a better understanding of Git concepts, you may wish to read Other Git documentation.

The git pull origin command above is just a shortcut for this command:

git pull git://git.sv.gnu.org/lilypond.git/ branch:origin/branch

where branch is typically master or translation; if you do not know or remember, see Downloading remote branches to remember which commands you issued or which source code you wanted to get.

A commit is a set of changes made to the sources; it also includes the committish of the parent commit, the name and e-mail of the author (the person who wrote the changes), the name and e-mail of the committer (the person who brings these changes into the Git repository), and a commit message.

A committish is the SHA1 checksum of a commit, a number made of 40 hexadecimal digits, which acts as the internal unique identifier for this commit. To refer to a particular revision, don’t use vague references like the (approximative) date, simply copy and paste the committish.

A branch is nothing more than a pointer to a particular commit, which is called the head of the branch; when referring to a branch, one often actually thinks about its head and the ancestor commits of the head.

Now we will explain the two last commands you used to get the source code from Git—see Downloading individual branches.

git remote add -ft branch -m branch \
  origin git://git.sv.gnu.org/lilypond.git/

git checkout -b branch origin/branch

The git remote has created a branch called origin/branch in your local Git repository. As this branch is a copy of the remote branch web from git.sv.gnu.org LilyPond repository, it is called a remote branch, and is meant to track the changes on the branch from git.sv.gnu.org: it will be updated every time you run git pull origin or git fetch origin.

The git checkout command has created a branch named branch. At the beginning, this branch is identical to origin/branch, but it will differ as soon as you make changes, e.g. adding newly translated pages or editing some documentation or code source file. Whenever you pull, you merge the changes from origin/branch and branch since the last pulling. If you do not have push (i.e. “write”) access on git.sv.gnu.org, your branch will always differ from origin/branch. In this case, remember that other people working like you with the remote branch branch of git://git.sv.gnu.org/lilypond.git/ (called origin/branch on your local repository) know nothing about your own branch: this means that whenever you use a committish or make a patch, others expect you to take the latest commit of origin/branch as a reference.

Finally, please remember to read the man page of every Git command you will find in this manual in case you want to discover alternate methods or just understand how it works.

3.3.3 Resolving conflicts

Occasionally an update may result in conflicts – this happens when you and somebody else have modified the same part of the same file and git cannot figure out how to merge the two versions together. When this happens, you must manually merge the two versions.

If you need some documentation to understand and resolve conflicts, see paragraphs How conflicts are presented and How to resolve conflicts in git merge man page.

If all else fails, you can follow the instructions in Reverting all local changes. Be aware that this eliminates any changes you have made!

3.3.4 Reverting all local changes

Sometimes git will become hopelessly confused, and you just want to get back to a known, stable state. This command destroys any local changes you have made in the currently checked-out branch, but at least you get back to the current online version:

git reset --hard origin/master

3.3.5 Working with remote branches

Fetching new branches from git.sv.gnu.org

To fetch and check out a new branch named branch on git.sv.gnu.org, run from top of the Git repository

git config --add remote.origin.fetch \
  +refs/heads/branch:refs/remotes/origin/branch

git checkout --track -b branch origin/branch

After this, you can pull branch from git.sv.gnu.org with:

git pull

Note that this command generally fetches all branches you added with git remote add (when you initialized the repository) or git config --add, i.e. it updates all remote branches from remote origin, then it merges the remote branch tracked by the current branch into the current branch. For example, if your current branch is master, origin/master will be merged into master.

Local clones, or having several working trees

If you play with several Git branches, e.g. master, translation, stable/2.12), you may want to have one source and build tree for each branch; this is possible with subdirectories of your local Git repository, used as local cloned subrepositories. To create a local clone for the branch named branch, run

git checkout branch
git clone -lsn . subdir
cd subdir
git reset --hard

Note that subdir must be a directory name which does not already exist. In subdir, you can use all Git commands to browse revisions history, commit and uncommit changes; to update the cloned subrepository with changes made on the main repository, cd into subdir and run git pull; to send changes made on the subrepository back to the main repository, run git push from subdir. Note that only one branch (the currently checked out branch) is created in the subrepository by default; it is possible to have several branches in a subrepository and do usual operations (checkout, merge, create, delete...) on these branches, but this possibility is not detailed here.

When you push branch from subdir to the main repository, and branch is checked out in the main repository, you must save uncommitted changes (see git stash) and do git reset --hard in the main repository in order to apply pushed changes in the working tree of the main repository.

3.3.6 Git log

The commands above don’t only bring you the latest version of the sources, but also the full history of revisions (revisions, also called commits, are changes made to the sources), stored in the ‘.git’ directory. You can browse this history with

git log     # only shows the logs (author, committish and commit message)
git log -p  # also shows diffs
gitk        # shows history graphically

3.3.7 Applying remote patches

TODO: Explain how to determine if a patch was created with git format-patch.

Well-formed git patches created with git format-patch should be committed with the following command:

git am patch

Patches created without git format-patch can be applied in two steps. The first step is to apply the patch to the working tree and the index:

git apply --index patch

The second step is to commit the changes and give credit to the author of the patch. This can be done with the following command:

git commit --author="John Smith <john@example.com>"

Please note that using the --index option for patching is quite important here and cannot reliably be replaced by using the -a option when committing: that would only commit files from the working tree that are already registered with git, so every file that the patch actually adds, like a regtest for a fixed bug, would get lost. For the same reason, you should not use the git-independent ‘patch’ program for applying patches.

3.3.8 Cleaning up multiple patches

If you have been developing on your own branch for a while, you may have more commmits than is really sensible. To revise your work and condense commits, use:

git rebase origin/master
git rebase -i origin/master

3.3.9 Commit access

Most contributors are not able to commit patches directly to the main repository—only members of the LilyPond development team have commit access. If you are a contributor and are interested in joining the development team, contact the Project Manager through the mailing list (lilypond-devel@gnu.org). Generally, only contributors who have already provided a number of patches which have been pushed to the main repository will be considered for membership.

If you have been approved by the Project Manager, use the following procedure to obtain commit access:

1. If you don’t already have one, set up a GitLab user account at https://gitlab.com/users/sign_in.
2. After registering, if you are not logged in automatically, login at the same page, and confirm your email address.
3. Navigate to https://gitlab.com/lilypond and ‘Request access’ to the group. Make sure that your account can be related to your activity on the mailing list. If in doubt, please post the username after requesting access.

   Note that you will not have commit access until the Project Manager activates your membership. Once your membership is activated, LilyPond should appear under the heading “Groups” on your profile page.

4. Generate and register an SSH key pair. Excellent instructions are provided in GitLab’s documentation.
5. Configure Git to use the SSH protocol (instead of the GIT or HTTP protocols). From your local Git repository, enter:

   git remote set-url origin git@gitlab.com:lilypond/lilypond.git

6. After your membership has been activated and you’ve configured Git to use SSH, test the connection with:

   git pull --verbose

   SSH should issue the following warning:

   The authenticity of host 'gitlab.com' can't be established. ECDSA key fingerprint is SHA256:HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw. Are you sure you want to continue connecting (yes/no/[fingerprint])?

   Make sure the key fingerprint displayed matches the one above or one of the others published by GitLab. If it doesn’t, respond “no” and check that you configured Git properly in the previous step. If it does match, respond “yes”. SSH should then issue another warning:

   Warning: Permanently added 'gitlab.com' (ECDSA) to the list of known hosts.

   The list of known hosts is stored in the file ‘~/.ssh/known_hosts’.

   At this point, you are prompted for your passphrase if you have one, then Git will attempt a pull.

   If git pull --verbose fails, you should see error messages like these:

   Permission denied (publickey). fatal: The remote end hung up unexpectedly

   If you get the above error, you may have made a mistake when registering your SSH key. If the key is properly registered and it still doesn’t work after an hour, ask for help on the mailing list.

   If git pull --verbose succeeds, the output will include a ‘From’ line that shows ‘ssh’ as the protocol:

   From git@gitlab.com:lilypond/lilypond.git

7. Test your commit access with a dry run:

   Note: Do not push directly to master; instead, push to a private development branch.

   git push --dry-run --verbose

   Note that recent versions of Git (Git 1.6.3 or later) will issue a big warning if the above command is used. The simplest solution is to tell Git to push all matching branches by default:

   git config push.default matching

   Then git push should work as before. For more details, consult the git push man page.

8. Repeat the steps from generating an SSH key through to testing your commit access, for each machine from which you will be making commits, or you may simply copy the files from your local ‘~/.ssh’ folder to the same folder on the other machine.

Technical details

• The git remote set-url command above should modify your local repository’s ‘.git/config’ file. These lines:

  [remote "origin"] url = git://git.sv.gnu.org/lilypond.git/

  should now be changed to:

  [remote "origin"] url = git@gitlab.com:lilypond/lilypond.git

• Similarly, the git config push.default matching command should add these lines to ‘.git/config’:

  [push] default = matching

Known issues and warnings

Encryption protocols, including ssh, generally do not permit packet fragmentation to avoid introducing a point of insecurity. This means that the maximum packet size must not exceed the smallest MTU (Maximum Transmission Unit) set in the routers along the path. This smallest MTU is determined by a procedure during call set-up which relies on the transmission over the path of ICMP packets. If any of the routers in the path block ICMP packets this mechanism fails, resulting in the possibility of packets being transmitted which exceed the MTU of one of the routers. If this happens the packet is discarded, causing the ssh session to hang, timeout or terminate with the error message

ssh: connect to host <host ip addr> port 22: Bad file number
fatal: The remote end hung up unexpectedly

depending on precisely when in the proceedings the first large packet is transmitted. Most routers on the internet have MTU set to 1500, but routers installed in homes to connect via broadband may use a slightly smaller MTU for efficient transmission over ATM. If this problem is encountered a possible work-around is to set the MTU in the local router to 1500.

3.3.10 Merging to master

Do not push directly to the git master branch. Instead, all changes need to be ‘merged’ with a merge request on GitLab. Before allowing the merge, GitLab ensures the following:

1. The merge must be fast-forward. In most cases, this can be achieved by ‘rebasing’ the branch with the most recent commits from master. This can be handled via GitLab, if no conflicts arise. Otherwise, or if preferred, the operation can be performed locally. Afterwards, the branch needs to be pushed with the option --force to overwrite commits.

   Note: Be careful when performing these operations manually: They are potentially dangerous and could result in losing commits if issued incorrectly!

2. The (possibly rebased) changes must have passed automatic testing. This ensures that the master branch is always clean and ready for development and translation.

After rebasing, GitLab will immediately start the automatic testing pipeline. At the moment, all steps may take up to one hour to complete. If you are confident about the rebased result of your changes, you may click “Merge when pipeline succeeds” to avoid waiting for the tests. On failure, the merge will be aborted and no harm is done to the master branch.

Because GitLab enforces fast-forward merges, this means only one set of changes can be rebased and merged at once. A second merge request would be rejected later on because it does not contain the commit(s) merged first. To avoid wasting testing resources, please avoid this situation and check first if a pipeline with a scheduled merge is already running.

How to merge a branch without rebasing

It is generally recommended to rebase commits before merging to get a linear history. However, this is not always possible or wanted. This particularly holds for the translation branch and release/unstable which cannot be force-pushed. For these cases, use the following procedure:

1. Merge the branch manually using the command line. The example assumes no pending changes in the local master branch and merges the translation branch:

   git checkout master git pull git merge translation git push origin HEAD:translation

2. Open a merge request at GitLab. This will immediately trigger automatic testing as described above.
3. Accept the merge request once the testing finishes, or use the button to “Merge when pipeline succeeds”.

This document was generated by gub on July 13, 2020 using texi2html 1.82.