LilyPond — Contributor’s Guide

Appendices

1. Introduction to contributing

This chapter presents a quick overview of ways that people can help LilyPond.

1.1 Help us

We need you!

Thank you for your interest in helping us — we would love to see you get involved! Your contribution will help a large group of users make beautifully typeset music.

Even working on small tasks can have a big impact: taking care of them allows experienced developers work on advanced tasks, instead of spending time on those simple tasks.

For a multi-faceted project like LilyPond, sometimes it’s tough to know where to begin. In addition to the avenues proposed below, you can send an e-mail to the lilypond-devel@gnu.org mailing list, and we’ll help you to get started.

Simple tasks

No programming skills required!

• Mailing list support: answer questions from fellow users. (This may entail helping them navigate the online documentation; in such cases it may sometimes be appropriate to point them to version-agnostic URL paths such as /latest/ or /stable/, which are automatically redirected.)
• Bug reporting: help users create proper (web-big-page)Bug reports section ‘Bug reports’ in General Information, and/or join the Bug Squad to organize (contributor-big-page)Issues section ‘Issues’ in Contributor’s Guide.
• Documentation: small changes can be proposed by following the guidelines for (contributor-big-page)Documentation suggestions section ‘Documentation suggestions’ in Contributor’s Guide.
• LilyPond Snippet Repository (LSR): create and fix snippets following the guidelines in (contributor-big-page)Adding and editing snippets section ‘Adding and editing snippets’ in Contributor’s Guide.
• Discussions, reviews, and testing: the developers often ask for feedback about new documentation, potential syntax changes, and testing new features. Please contribute to these discussions!

Advanced tasks

These jobs generally require that you have the source code and can compile LilyPond.

Contributors using Linux or FreeBSD may also use Lilydev, but if they prefer their own development environment, they should read (contributor-big-page)Working with source code section ‘Working with source code’ in Contributor’s Guide, and (contributor-big-page)Compiling section ‘Compiling’ in Contributor’s Guide.

Begin by reading (contributor-big-page)Summary for experienced developers section ‘Summary for experienced developers’ in Contributor’s Guide.

• Documentation: for large changes, see (contributor-big-page)Documentation work section ‘Documentation work’ in Contributor’s Guide.
• Website: the website is built from the normal documentation source. See the info about documentation, and also (contributor-big-page)Website work section ‘Website work’ in Contributor’s Guide.
• Translations: see (contributor-big-page)Translating the documentation section ‘Translating the documentation’ in Contributor’s Guide, and (contributor-big-page)Translating the website section ‘Translating the website’ in Contributor’s Guide.
• Bugfixes or new features: read (contributor-big-page)Programming work section ‘Programming work’ in Contributor’s Guide.

1.2 Overview of work flow

Git is a version control system that tracks the history of a program’s source code. The LilyPond source code is maintained as a Git repository, which contains:

• all of the source files needed to build LilyPond, and
• a record of the entire history of every change made to every file since the program was born.

The ‘official’ LilyPond Git repository is hosted by the GNU Savannah software forge at https://git.sv.gnu.org. The server provides two separate interfaces for viewing the LilyPond Git repository online: cgit and gitweb.

However, the main development takes place at https://gitlab.com/lilypond/lilypond/, which also hosts the project’s issues. Automatic mirroring ensures that ‘important’ branches (such as master and stable/*) are up-to-date on the ‘official’ repository at GNU Savannah, so you can also base your development on a clone from there.

Compiling (‘building’) LilyPond allows developers to see how changes to the source code affect the program itself. Compiling is also needed to package the program for specific operating systems or distributions. LilyPond can be compiled from a local Git repository (for developers), or from a downloaded tarball (for packagers). Compiling LilyPond is a rather involved process, and most contributor tasks do not require it.

Contributors can contact the developers through the ‘lilypond-devel’ mailing list. The mailing list archive is located at http://lists.gnu.org/archive/html/lilypond-devel/. If you have a question for the developers, search the archives first to see if the issue has already been discussed. Otherwise, send an email to lilypond-devel@gnu.org. You can subscribe to the developers’ mailing list here: http://lists.gnu.org/mailman/listinfo/lilypond-devel.

1.3 Summary for experienced developers

If you are already familiar with typical open-source tools, here’s what you need to know:

• ‘official’ source repository: hosted by GNU Savannah

  https://git.savannah.gnu.org/gitweb/?p=lilypond.git

• development platform: hosted by GitLab; also includes the issue tracker (see Issues)

  https://gitlab.com/lilypond/lilypond/

• environment variables: many maintenance scripts, and many instructions in this guide rely on predefined Environment variables.
• mailing lists: given on (web-big-page)Contact section ‘Contact’ in General Information.
• Git branches:
  • master: always base your work from this branch, but never push directly to it. Instead, use GitLab to merge changes after they have passed automatic testing (see below).
  • translation: Translators should base their work on this branch only and push any translation patches directly to it as well.
  • dev/foo: feel free to push any new branch name under dev/.
• regression tests: also known as “regtests”. A collection of more than a thousand .ly files that are used to track LilyPond’s engraving output between released stable and unstable versions as well as checked for all patches submitted for testing.

  If a patch introduces any unintentional changes to any of the regtests it is very likely it will be rejected (to be fixed) – always make sure that, if you expect any regression test changes, that they are explained clearly as part of the patch description when submitting for testing. For more information see Regression tests.

• reviews: after finishing work on a patch or branch:
  1. Commit the changes and create a merge request. More information on this can be found in the section Uploading a patch for review.
  2. Patches are generally tested within 24 hours of submission. Once it has passed the basic tests – make check, make, make doc – the tracker will be updated and the patch’s status will change to Patch::review for other developers to examine.
  3. Every third day, the “Patch Meister” will examine all merge requests currently under review, looking for any comments by other developers. Depending on what has been posted, the patch will be either; “moved on” to the next patch status (Patch::countdown); set back to Patch::needs_work; or if more discussion is needed, left at Patch::review. In all cases the merge request will be updated by the Patch Meister accordingly.
  4. Once another three days have passed, any patch that has been given Patch::countdown status will be changed to Patch::push, the merge request is updated, and the developer can now rebase and merge to the master branch (or ask one of the other developers to merge it for you).

  Advanced note: This process does means that most patches will take about a week before finally being merged into master. With the limited resources for reviewing patches available and a history of unintended breakages in the master branch (from patches that have not had time to be reviewed properly), this is the best compromise we have found.

1.4 Mentors

We have a semi-formal system of mentorship, similar to the medieval “journeyman/master” training system. New contributors will have a dedicated mentor to help them “learn the ropes”.

Contributor responsibilities

1. Ask your mentor which sections of the CG you should read.
2. If you get stuck for longer than 10 minutes, ask your mentor. They might not be able to help you with all problems, but we find that new contributors often get stuck with something that could be solved/explained with 2 or 3 sentences from a mentor.
3. If you have been working on a task much longer than was originally estimated, stop and ask your mentor. There may have been a miscommunication, or there may be some time-saving tips that could vastly simply your task.
4. Send patches to your mentor for initial comments.
5. Inform your mentor if you’re going to be away for a month, or if you leave entirely. Contributing to lilypond isn’t for everybody; just let your mentor know so that we can reassign that work to somebody else.
6. Inform your mentor if you’re willing to do more work – we always have way more work than we have helpers available. We try to avoid overwhelming new contributors, so you’ll be given less work than we think you can handle.

Mentor responsibilities

1. Respond to questions from your contributor(s) promptly, even if the response is just “sorry, I don’t know” or “sorry, I’m very busy for the next 3 days; I’ll get back to you then”. Make sure they feel valued.
2. Inform your contributor(s) about the expected turnaround for your emails – do you work on lilypond every day, or every weekend, or what? Also, if you’ll be unavailable for longer than usual (say, if you normally reply within 24 hours, but you’ll be at a conference for a week), let your contributors know. Again, make sure they feel valued, and that your silence (if they ask a question during that period) isn’t their fault.
3. Inform your contributor(s) if they need to do anything unusual for the builds, such as doing a “make clean / doc-clean” or switching git branches (not expected, but just in case...)
4. You don’t need to be able to completely approve patches. Make sure the patch meets whatever you know of the guidelines (for doc style, code indentation, whatever), and then send it on to -devel for more comments. If you feel confident about the patch, you can push it directly (this is mainly intended for docs and translations; code patches should almost always go to -devel before being pushed).
5. Keep track of patches from your contributor. Either open merge requests yourself, or help and encourage them to upload the patches themselves.
6. Encourage your contributor to review patches, particularly your own! It doesn’t matter if they’re not familiar with C++ / scheme / build system / doc stuff – simply going through the process is valuable. Besides, anybody can find a typo!
7. Contact your contributor at least once a week. The goal is just to get a conversation started – there’s nothing wrong with simply copy&pasting this into an email:

   Hey there, How are things going? If you sent a patch and got a review, do you know what you need to fix? If you sent a patch but have no reviews yet, do you know when you will get reviews? If you are working on a patch, what step(s) are you working on?

2. Quick start

Want to submit a patch for LilyPond? Great! Never created a patch before? Never compiled software before? No problem! This chapter is for you and will help you do this as quickly and easily as possible.

2.1 LilyDev

“LilyDev” is a custom GNU/Linux operating system which includes all the necessary software and tools to compile LilyPond, the documentation and the website (also see Website work). It is also prepared for building the Grand Unified Builder (GUB), though this is an involved process and may require further tweaking.

While compiling LilyPond on Mac OS and Windows is possible, both environments are complex to set up. LilyDev can be easily run inside a ‘virtual machine’ on either of these operating systems relatively easily using readily available virtualization software. We recommend using VirtualBox as it is available for all major operating systems and is very easy to install & configure.

LilyDev comes in two ‘flavours’: containers and a standard disk image. Windows or Mac OS users should choose the Debian disk image (to be run in a virtual machine), that is the file named ‘LilyDev-VERSION-debian-vm.zip’. GNU/Linux users are recommended to choose one of the containers (currently Debian or Fedora), which are smaller in size, lightweight and easier to manage. The Fedora disk image has currently not been released, you can create it from the sources located in the ‘/mkosi’ subdirectory of the LilyDev repository, however.

Download the appropriate file from here:

https://github.com/fedelibre/LilyDev/releases/latest

If you are not familiar with GNU/Linux, it may be beneficial to read a few “introduction to Linux” type web pages.

Installing LilyDev in VirtualBox

This section discusses how to install and use LilyDev with VirtualBox.

1. Download VirtualBox from here:

   http://www.virtualbox.org/wiki/Downloads

   Note: In virtualization terminology, the operating system where VirtualBox is installed is known as the host. LilyDev will be installed ‘inside’ VirtualBox as a guest.

2. The zip archive you downloaded contains the raw disk image and its SHA256 checksum. You can verify the integrity of the downloaded archive with any hashing tool your OS does support. On Linux, run the following command in the directory where you have extracted the files (this may take some time):

   sha256sum -c SHA256SUMS

   For Windows, look for the tools FCIV or certutil to compute the archive’s hash.

3. As VirtualBox does not support the raw format, you have to extract it and then convert it to VDI format. Make sure that ‘VBoxManage’ is in your PATH or call it from your VirtualBox installation directory:

   VBoxManage convertfromraw LilyDev-VERSION-debian-vm.img \ LilyDev-VERSION-debian-vm.vdi

   Note: You need a fair amount of disk space (around 30GB) to extract the raw image. After converting to a dynamic VirtualBox image it will take up much less space (only the amount of space that is actually allocated by the guest filesystem)

4. Start the VirtualBox software and click ‘New’ to create a new “virtual machine”.

   The ‘New Virtual Machine Wizard’ walks you through setting up your guest virtual machine. Choose an appropriate name for your LilyDev installation and select the ‘Linux’ operating system. When selecting the ‘version’ choose ‘Debian (64-bit)’. If you do not have that specific option choose ‘Linux 2.6/3.x/4.x (64-bit)’.

5. Select the amount of RAM you allow the LilyDev guest to use from your host operating system when it is running. If possible, use at least 1GB of RAM; the more RAM you can spare from your host the better
6. In the ‘Hard Disk’ step, you use the VDI file you have previously created. You may move it within the virtual machine’s folder already created by the wizard (in GNU/Linux the default should be ‘~/VirtualBox VMs/NAME’). Click on ‘Use an existing virtual hard disk file’ and browse to the VDI file.
7. Verify the summary details and click ‘Create’ as soon as you are satisfied. Your new guest shall be displayed in the VirtualBox window now.
8. Enable EFI within the virtual machine’s settings – click on System → Motherboard and select ‘Extended features: Enable EFI’. Otherwise, you won’t be able to boot the image.
9. VirtualBox ‘guest additions’, which are installed by default in the debian image, provide some additional features such as being able to dynamically resize the LilyDev window, allow seamless interaction with your mouse pointer on both the host and guest, and let you copy/paste between your host and guest if needed. It seems that dynamic window resizing works only with the ‘VBoxVGA’ graphics controller, which you can choose in Display → Graphics Controller. To enable clipboard sharing between guest and host, choose General → Advanced → Shared Clipboard → Bidirectional.
10. Click the ‘Start’ button and wait until the login screen appears. Log in as dev user then; type the password lilypond. Before starting any work, be sure to complete the next steps.

    Note: Since the default keyboard layout is US (American), you may have to type the password differently if you are using another layout, like ‘lilzpond’ on a German keyboard, for example.

11. Open a terminal by clicking Applications → Terminal at the upper left of the screen. You may want to change the password of user ‘dev’ before doing further work with the command passwd.
12. You might need to change the keyboard layout from default US (American) to your national layout. Therefore open a terminal and run

    sudo dpkg-reconfigure keyboard-configuration

    Note: You need superuser rights to change certain aspects of the system configuration. The sudo tool allows to gain superuser rights temporarily. It does show you a warning message on its first use that reminds you to use your extended rights carefully.

    At first, you are prompted for the model of your keyboard. Press Enter to show further models. In most cases, it is sufficient to choose ‘Generic, 105 keys’. After that, choose your keyboard layout. Now, you can customize the function of your <AltGr> key. Normally, the default layout settings fit well, so take number 1. The same holds for the question of whether you want to configure a ‘compose’ key. At last, you are asked if you want to configure <Ctrl+Alt+Backspace> as a shortcut to terminate the X server. Presumably, you do not need this, so you can safely type ‘no’.

13. To set up your system language (charset, localized messages etc.), continue with

    sudo dpkg-reconfigure locales

    Note: Restarting is required in order to take the changes into effect.

14. Finally, you should run a setup script. If you are on the command line already, simply type ./setup.sh to run the interactive script that does set up git and downloads all the repositories needed to build LilyPond.

Configuring LilyDev in VirtualBox

• In the settings for the virtual machine, set the network to Bridged mode to allow you to access shared folders when using Windows hosts.
• Set up any additional features, such as ‘Shared Folders’ between your main operating system and LilyDev. This is distinct from the networked share folders in Windows. Consult the external documentation for this.

  Some longtime contributors have reported that ‘shared folders’ are rarely useful and not worth the fuss, particularly since files can be shared over a network instead.

• Pasting into a terminal is done with Ctrl+Shift+v.
• Right-click allows you to edit a file with the text editor (default is Leafpad).

Known issues and warnings

Not all hardware is supported in all virtualization tools. In particular, some contributors have reported problems with USB network adapters. If you have problems with network connection (for example Internet connection in the host system is lost when you launch virtual system), try installing and running LilyDev with your computer’s built-in network adapter used to connect to the network. Refer to the help documentation that comes with your virtualization software.

2.2 Compiling with LilyDev

LilyDev is our custom GNU/Linux which contains all the necessary dependencies to do LilyPond development; for more information, see LilyDev.

Preparing the build

To prepare the build directory, enter (or copy&paste) the below text. This should take less than a minute.

cd $LILYPOND_GIT
sh autogen.sh --noconfigure
mkdir -p build/
cd build/
../configure

Building lilypond

Compiling LilyPond will take anywhere between 1 and 15 minutes on most ‘modern’ computers – depending on CPU and available RAM. We also recommend that you minimize the terminal window while it is building; this can help speed up on compilation times.

cd $LILYPOND_GIT/build/
make

It is possible to run make with the -j option to help speed up compilation times even more. See Compiling LilyPond

You may run the compiled lilypond with:

cd $LILYPOND_GIT/build/
out/bin/lilypond my-file.ly

Building the documentation

Compiling the documentation is a much more involved process, and will likely take 2 to 10 hours.

cd $LILYPOND_GIT/build/
make
make doc

The documentation is put in ‘out-www/offline-root/’. You may view the html files by entering the below text; we recommend that you bookmark the resulting page:

firefox $LILYPOND_GIT/build/out-www/offline-root/index.html

Installing

Don’t. There is no reason to install LilyPond within LilyDev. All development work can (and should) stay within the ‘$LILYPOND_GIT’ directory, and any personal composition or typesetting work should be done with an official GUB release.

Problems and other options

To select different build options, or isolate certain parts of the build, or to use multiple CPUs while building, read Compiling.

In particular, contributors working on the documentation should be aware of some bugs in the build system, and should read the workarounds in Generating documentation.

2.3 Now start work!

LilyDev users may now skip to the chapter which is aimed at their intended contributions:

• Documentation work
• Translating the documentation
• Website work
• Regression tests
• Programming work

These chapters are mainly intended for people not using LilyDev, but they contain extra information about the “behind-the-scenes” activities. We recommend that you read these at your leisure, a few weeks after beginning work with LilyDev.

• Working with source code
• Compiling

3. Working with source code

Advanced contributors will find this material quite useful, particularly if they are working on major new features.

3.1 Starting with Git

3.1.1 Setting up

Installing Git

If you are using a Unix-based machine, the easiest way to download and install Git is through a package manager such as rpm or apt-get – the installation is generally automatic. The only required package is (usually) called git-core, although some of the auxiliary git* packages are also useful (such as gitk).

Alternatively, you can visit the Git website (http://git-scm.com/) for downloadable binaries and tarballs.

Initializing a repository

Once Git is installed, get a copy of the source code:

git clone git://git.sv.gnu.org/lilypond.git ~/lilypond-git

The above command will put the it in ‘~/lilypond-git’, where ~ represents your home directory.

Technical details

This creates (within the ‘$LILYPOND_GIT’ directory) a subdirectory called ‘.git/’, which Git uses to keep track of changes to the repository, among other things. Normally you don’t need to access it, but it’s good to know it’s there.

Configuring Git

Before working with the copy of the main LilyPond repository, you should configure some basic settings with the git config command. Git allows you to set both global and repository-specific options.

To configure settings that affect all repositories, use the ‘--global’ command line option. For example, the first two options that you should always set are your name and email, since Git needs these to keep track of commit authors:

git config --global user.name "John Smith"
git config --global user.email john@example.com

To configure Git to use colored output where possible, use:

git config --global color.ui auto

The text editor that opens when using git commit can also be changed. If none of your editor-related environment variables are set ($GIT_EDITOR, $VISUAL, or $EDITOR), the default editor is usually vi or vim. If you’re not familiar with either of these, you should probably change the default to an editor that you know how to use. For example, to change the default editor to nano, enter:

git config --global core.editor nano

Finally, and in some ways most importantly, let’s make sure that we can easily see the state of our working copy, without the need of typing git status repeatedly. If you’re not using LilyDev, add the following lines to your ‘~/.bashrc’:

export PS1="\u@\h \w\$(__git_ps1)$ "
export GIT_PS1_SHOWDIRTYSTATE=true
export GIT_PS1_SHOWUNTRACKEDFILES=true
export GIT_PS1_SHOWUPSTREAM=auto

The first line will show the branch we’re on. The other lines will use some symbols next to the branch name to indicate some kind of state. “*” means that there are unstaged changes, “+” indicates staged changes; if there are untracked files, a “%” will appear. Finally, we can also see if our HEAD is behind (“<”) or ahead (“>”) of its upstream, and if they have diverged (“<>”) or they are synced (“=”).

You may need to install the additional bash-completion package, but it is definitely worth it. After installation you must log out, and then log back in again to enable it.

Technical details

Git stores the information entered with git config --global in the file ‘.gitconfig’, located in your home directory. This file can also be modified directly, without using git config. The ‘.gitconfig’ file generated by the above commands would look like this:

[user]
        name = John Smith
        email = john@example.com
[color]
        ui = auto
[core]
        editor = nano

Using the git config command without the ‘--global’ option configures repository-specific settings, which are stored in the file ‘.git/config’. This file is created when a repository is initialized (using git init), and by default contains these lines:

[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true

However, since different repository-specific options are recommended for different development tasks, it is best to avoid setting any now. Specific recommendations will be mentioned later in this manual.

3.1.2 Git for the impatient

Since our review process delays patches by 60-120 hours, and you want to be able to work on other stuff while your previous work is getting reviewed, you’re going to use branches.

You can think of a branch as being a separate copy of the source code. But don’t worry about it.

Start work: make a new branch

Let’s pretend you want to add a section to the Contributor’s Guide about using branches.

Start by updating the repository, then making a new branch. Call the branch anything you want as long as the name starts with dev/. Branch names that don’t begin with dev/ are reserved for special things in lilypond.

git checkout master
git pull -r origin master
git branch dev/cg

Switch to that branch

Nothing has happened to the files yet. Let’s change into the new branch. You can think of this as “loading a file”, although in this case it’s really “loading a directory and subdirectories full of files”.

git checkout dev/cg

Your prompt now shows you that you’re on the other branch:

gperciva@LilyDev:~/lilypond-git (dev/cg)$

To be able to manage multiple lilypond issues at once, you’ll need to switch branches. You should have each lilypond issue on a separate branch. Switching branches is easy:

git checkout master
git checkout origin/release/unstable
git checkout dev/cg

Branches that begin with origin/ are part of the remote repository, rather than your local repository, so when you check them out you get a temporary local branch. You should never make changes directly on a branch beginning with origin/. You get changes into the remote repository by making them in local branches, and then Merging to master.

Make your changes

Edit files, then commit them.

git commit -a

Remember how I said that switching to a branch was like “loading a directory”? Well, you’ve just “saved a directory”, so that you can “load” it later.

When you create a new file, you need to add it to git, then commit it:

git add input/regression/avoid-crash-on-condition.ly
git commit -a

Edit more files. Commit them again. Edit yet more files, commit them again. Go eat dinner. Switch to master so you can play with the latest changes from other developers. Switch back to your branch and edit some more. Commit those changes.

At this stage, don’t worry about how many commits you have.

Save commits to external files

Branches are nerve-wracking until you get used to them. You can save your hard work as individual ‘.patch’ files. Be sure to commit your changes first.

git commit -a
git format-patch master

I personally have between 4 and 20 of those files saved in a special folder at any point in time. Git experts might laugh as that behavior, but I feel a lot better knowing that I’ve got those backups.

Prepare your branch for review

After committing, you can update your branch with the latest master:

git commit -a
git checkout master
git pull -r origin master
git checkout dev/cg
git rebase master

Due to the speed of lilypond development, sometimes master has changed so much that your branch can no longer be applied to it. In that happens, you will have a merge conflict. Stop for a moment to either cry or have a stiff drink, then proceed to Merge conflicts.

Upload your branch

Finally, you’re finished your changes. Time to upload for review. Make sure that you’re on your branch, then upload:

git checkout dev/cg
git push ...

Afterwards create a merge request to indicate the branch is ready for review.

Wait for reviews

While you’re waiting for a countdown and reviews, go back to master, make a dev/doc-beams branch, and start adding doc suggestions from issue 12345 from the tracker. Or make a dev/page-breaks and fix bug in page breaking. Or whatever. Don’t worry, your dev/cg is safe.

Combining commits (optional unless you have broken commits)

Does the history of your branch look good?

gitk

If you have a lot of commits on your branch, you might want to combine some of them. Alternately, you may like your commits, but want to edit the commit messages.

git rebase -i master

Follow instructions on the screen.

Merge to master

When you’ve got the coveted Patch::push status, time to merge your changes as described in Merging to master.

Delete your branch

After the merge request has passed testing and was merged to master, you can update and then try removing your branch:

git checkout master
git pull -r origin master
git branch -d dev/cg

The last command will fail if the commit hashes of dev/cg are not present in origin/master. This can be because you used GitLab to rebase your branch, which modifies the commit data and changes the hash. If you are sure that the branch is not needed anymore, replace the -d on the final line with a -D instead.

3.1.3 Other repositories

We have a few other code repositories.

lilypond-extra

There is a separate repository for general administrative scripts, as well as pictures and media files for the website. People interested in working on the website should download this repository, and set their $LILYPOND_WEB_MEDIA_GIT environment variable to point to that repository.

https://github.com/gperciva/lilypond-extra

To configure an environment variable in bash (the default for most GNU/Linux distributions),

export LILYPOND_WEB_MEDIA_GIT=$HOME/dir/of/lilypond-extra/

Be aware that lilypond-extra is the definitive source for some binary files - in particular PDF versions of papers concerning LilyPond. To add further PDFs of this sort, all that is necessary is to add the PDF to lilypond-extra and then add a reference to it in the documentation. The file will then be copied to the website when make website is run.

However, pictures that are also used in the documentation build are mastered in the main git repository. If any of these is changed, it should be updated in git, and then the updates copied to lilypond-extra.

Grand Unified Builder (GUB)

Another item of interest might be the Grand Unified Builder, our cross-platform building tool. Since it is used by other projects as well, it is not stored in our gub repository. For more info, see https://lilypond.org/gub.

The location for the repository is:

http://github.com/gperciva/gub

LilyPad

Our binary releases on MacOS X and Windows contain a lightweight text editor.

To make any modifications the Windows editor, you will need to do the following:

1. Clone the git repository from https://github.com/gperciva/lilypad
2. Make changes to the source, and check it compiles. In a Windows environment MinGW provides both a Git installation and a gcc compiler. This can be obtained from http://www.mingw.org/
3. Update the version which is contained in the ‘rsrc.rc’. Check this compiles, too.
4. Commit the changes with an informative commit message.
5. Push the changes to github. You will need to use syntax similiar to this:

   git push https://UserName@github.com/gperciva/lilypad.git

   You will need to have push access to the git repository for this to be successful.

6. Make a tarball of the source code to be used by GUB by pulling the updated repository from GitHub. Ensure that the tarball has the correct Version number.
7. Copy the tarball to https://lilypond.org/downloads/gub-sources/lilypad/. You will need to have SSH access to lilypond.org. If you do not, contact the Release Manager via the lilypond-devel mailing list.
8. Update GUB to make it use the new tarball by editing ‘gub/specs/lilypad.py’ and changing the source = line to point to the new source.
9. Push this updated ‘lilypad.py’ version to the GUB repository on GitHub.
10. Test the changes with a new GUB compile.

yet more repositories

There are a few other repositories floating around, which will hopefully be documented in the near future.

3.1.4 Downloading remote branches

Organization of remote branches

The main LilyPond repository is organized into branches to facilitate development. These are often called remote branches to distinguish them from local branches you might create yourself (see Using local branches).

The master branch contains all the source files used to build LilyPond, which includes the program itself (both stable and development releases), the documentation (and its translations), and the website. Generally, the master branch is expected to compile successfully.

The translation branch is a side branch that allows translators to work without needing to worry about compilation problems. Periodically, the Translation Meister (after verifying that it doesn’t break compilation), will merge this branch into master to incorporate recent translations. Similarly, the master branch is usually merged into the translation branch after significant changes to the English documentation. See Translating the documentation for details.

LilyPond repository sources

The recommended source for downloading a copy of the main repository is:

git://git.sv.gnu.org/lilypond.git

However, if your internet router filters out connections using the GIT protocol, or if you experience difficulty connecting via GIT, you can try these other sources:

ssh://git.sv.gnu.org/srv/git/lilypond.git
http://git.sv.gnu.org/r/lilypond.git

The SSH protocol can only be used if your system is properly set up to use it. Also, the HTTP protocol is slowest, so it should only be used as a last resort.

Downloading individual branches

Once you have initialized an empty Git repository on your system (see Initializing a repository), you can download a remote branch into it. Make sure you know which branch you want to start with.

To download the master branch, enter the following:

git remote add -ft master -m master \
  origin git://git.sv.gnu.org/lilypond.git/

To download the translation branch, enter:

git remote add -ft translation -m \
  translation origin git://git.sv.gnu.org/lilypond.git/

The git remote add process could take up to ten minutes, depending on the speed of your connection. The output will be something like this:

Updating origin
remote: Counting objects: 235967, done.
remote: Compressing objects: 100% (42721/42721), done.
remote: Total 235967 (delta 195098), reused 233311 (delta 192772)
Receiving objects: 100% (235967/235967), 68.37 MiB | 479 KiB/s, done.
Resolving deltas: 100% (195098/195098), done.
From git://git.sv.gnu.org/lilypond
 * [new branch]      master     -> origin/master
From git://git.sv.gnu.org/lilypond
 * [new tag]         flower/1.0.1 -> flower/1.0.1
 * [new tag]         flower/1.0.10 -> flower/1.0.10
⋮
 * [new tag]         release/2.9.6 -> release/2.9.6
 * [new tag]         release/2.9.7 -> release/2.9.7

When git remote add is finished, the remote branch should be downloaded into your repository—though not yet in a form that you can use. In order to browse the source code files, you need to create and checkout your own local branch. In this case, however, it is easier to have Git create the branch automatically by using the checkout command on a non-existent branch. Enter the following:

git checkout -b branch origin/branch

where branch is the name of your tracking branch, either master or translation.

Git will issue some warnings; this is normal:

warning: You appear to be on a branch yet to be born.
warning: Forcing checkout of origin/master.
Branch master set up to track remote branch master from origin.
Already on 'master'

By now the source files should be accessible—you should be able to edit any files in the ‘$LILYPOND_GIT’ directory using a text editor of your choice. But don’t start just yet! Before editing any source files, learn how to keep your changes organized and prevent problems later—read Basic Git procedures.

Technical Details

The git remote add command should add some lines to your local repository’s ‘.git/config’ file:

[remote "origin"]
        url = git://git.sv.gnu.org/lilypond.git/
        fetch = +refs/heads/master:refs/remotes/origin/master

Downloading all remote branches

To download all remote branches at once, you can clone the entire repository:

git clone git://git.sv.gnu.org/lilypond.git

Other branches

Most contributors will never need to touch the other branches. If you wish to do so, you will need more familiarity with Git; please see Other Git documentation.

• dev/XYZ: These branches are for individual developers. They store code which is not yet stable enough to be added to the master branch.
• stable/XYZ: The branches are kept for archival reasons.
• archive/XYZ: The branches are kept for archival reasons.

3.2 Basic Git procedures

3.2.1 The Git contributor’s cycle

Here is a simplified view of the contribution process on Git:

1. Update your local repository by pulling the most recent updates from the remote repository.
2. Edit source files within your local repository’s working directory.
3. Commit the changes you’ve made to a local branch.
4. Generate a patch to share your changes with the developers.

3.2.2 Pulling and rebasing

When developers push new patches to the git.sv.gnu.org repository, your local repository is not automatically updated. It is important to keep your repository up-to-date by periodically pulling the most recent commits from the remote branch. Developers expect patches to be as current as possible, since outdated patches require extra work before they can be used.

Occasionally you may need to rework some of your own modifications to match changes made to the remote branch (see Resolving conflicts), and it’s considerably easier to rework things incrementally. If you don’t update your repository along the way, you may have to spend a lot of time resolving branch conflicts and reconfiguring much of the work you’ve already done.

Fortunately, Git is able to resolve certain types of branch conflicts automatically with a process called rebasing. When rebasing, Git tries to modify your old commits so they appear as new commits (based on the latest updates). For a more involved explanation, see the git-rebase man page.

To pull without rebasing (recommended for translators), use the following command:

git pull    # recommended for translators

If you’re tracking the remote master branch, you should add the ‘-r’ option (short for ‘--rebase’) to keep commits on your local branch current:

git pull -r # use with caution when translating

If you don’t edit translated documentation and don’t want to type ‘-r’ every time, configure the master branch to rebase by default with this command:

git config branch.master.rebase true

If pull fails because of a message like

error: Your local changes to 'Documentation/learning/tutorial.itely'
would be overwritten by merge.  Aborting.

or

Documentation/learning/tutorial.itely: needs update
refusing to pull with rebase: your working tree is not up-to-date

it means that you have modified some files in you working tree without committing changes (see Commits); you can use the git stash command to work around this:

git stash      # save uncommitted changes
git pull -r    # pull using rebase (translators omit "-r")
git stash pop  # reapply previously saved changes

Note that git stash pop will try to apply a patch, and this may create a conflict. If this happens, see Resolving conflicts.

TODO: I think the next paragraph is confusing. Perhaps prepare the reader for new terms ‘committish’ and ‘head’? -mp

TODO: when committishes automatic conditional update have been tested and documented, append the following to the warning above: Note that using update-committishes make target generally touches committishes.

Technical details

The git config command mentioned above adds the line rebase = true to the master branch in your local repository’s ‘.git/config’ file:

[branch "master"]
        remote = origin
        merge = refs/heads/master
        rebase = true

3.2.3 Using local branches

Creating and removing branches

Local branches are useful when you’re working on several different projects concurrently. To create a new branch, enter:

git branch name

To delete a branch, enter:

git branch -d name

Git will ask you for confirmation if it sees that data would be lost by deleting the branch. Use ‘-D’ instead of ‘-d’ to bypass this. Note that you cannot delete a branch if it is currently checked out.

Listing branches and remotes

You can get the exact path or URL of all remote branches by running:

git remote -v

To list Git branches on your local repositories, run

git branch     # list local branches only
git branch -r  # list remote branches
git branch -a  # list all branches

Checking out branches

To know the currently checked out branch, i.e. the branch whose source files are present in your working tree, read the first line of the output of

git status

The currently checked out branch is also marked with an asterisk in the output of git branch.

You can check out another branch other_branch, i.e. check out other_branch to the working tree, by running

git checkout other_branch

Note that it is possible to check out another branch while having uncommitted changes, but it is not recommended unless you know what you are doing; it is recommended to run git status to check this kind of issue before checking out another branch.

Merging branches

To merge branch foo into branch bar, i.e. to “add” all changes made in branch foo to branch bar, run

git checkout bar
git merge foo

If any conflict happens, see Resolving conflicts.

There are common usage cases for merging: as a translator, you will often want the Translations meister to merge master into translation; on the other hand, the Translations meister wants to merge translation into master whenever he has checked that translation builds successfully.

3.2.4 Commits

Understanding commits

Technically, a commit is a single point in the history of a branch, but most developers use the term to mean a commit object, which stores information about a particular revision. A single commit can record changes to multiple source files, and typically represents one logical set of related changes (such as a bug-fix). You can list the ten most recent commits in your current branch with this command:

git log -10 --oneline

If you’re using an older version of Git and get an ‘unrecognized argument’ error, use this instead:

git log -10 --pretty=oneline --abbrev-commit

More interactive lists of the commits on the remote master branch are available at http://git.sv.gnu.org/gitweb/?p=lilypond.git;a=shortlog and http://git.sv.gnu.org/cgit/lilypond.git/log/.

How to make a commit

Once you have modified some source files in your working directory, you can make a commit with the following procedure:

1. Make sure you’ve configured Git properly (see Configuring Git). Check that your changes meet the requirements described in Code style and/or Documentation policy. For advanced edits, you may also want to verify that the changes don’t break the compilation process.
2. Run the following command:

   git status

   to make sure you’re on the right branch, and to see which files have been modified, added or removed, etc. You may need to tell Git about any files you’ve added by running one of these:

   git add file # add untracked file individually git add . # add all untracked files in current directory

   After git add, run git status again to make sure you got everything. You may also need to modify ‘GNUmakefile’.

3. Preview the changes about to be committed (to make sure everything looks right) with:

   git diff HEAD

   The HEAD argument refers to the most recent commit on the currently checked-out branch.

4. Generate the commit with:

   git commit -a

   The ‘-a’ is short for ‘--all’ which includes modified and deleted files, but only those newly created files that have previously been added.

Commit messages

When you run the git commit -a command, Git automatically opens the default text editor so you can enter a commit message. If you find yourself in a foreign editing environment, you’re probably in vi or vim. If you want to switch to an editor you’re more familiar with, quit by typing :q! and pressing <Enter>. See Configuring Git for instructions on changing the default editor.

In any case, Git will open a text file for your commit message that looks like this:

# Please enter the commit message for your changes.  Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#	modified:   working.itexi
#

Your commit message should begin with a one-line summary describing the change (no more than 50 characters long), and if necessary a blank line followed by several lines giving the details:

Doc: add Baerenreiter and Henle solo cello suites

Added comparison of solo cello suite engravings to new essay with
high-res images, fixed cropping on Finale example.

Commit messages often start with a short prefix describing the general location of the changes.

• Doc: and Doc-**: If a commit affects the documentation in English (or in several languages simultaneously) the commit message should be prefixed with “Doc: ”. If the commit affects only one of the translations, the commit message should be prefixed with “Doc-**: ”, where ** is the two-letter language code.
• Web: and Web-**: Commits that affect the website should use “Web: ” for English, and “Web-**: ” for other languages.
• CSS: Commits that change CSS files should use “Web: CSS: ” or “Doc: CSS: ” depending on whether they affect the website or the documentation/manuals.
• Changes to a single file are often prefixed with the name of the file involved.

Visit the links listed in Understanding commits for examples.

3.2.5 Patches

How to make a patch

If you want to share your changes with other contributors and developers, you need to generate patches from your commits. We prefer it if you follow the instructions in Uploading a patch for review. However, we present an alternate method here.

You should always run git pull -r (translators should leave off the ‘-r’) before doing this to ensure that your patches are as current as possible.

Once you have made one or more commits in your local repository, and pulled the most recent commits from the remote branch, you can generate patches from your local commits with the command:

git format-patch origin

The origin argument refers to the remote tracking branch at git.sv.gnu.org. This command generates a separate patch for each commit that’s in the current branch but not in the remote branch. Patches are placed in the current working directory and will have names that look something like this:

0001-Doc-Fix-typos.patch
0002-Web-Remove-dead-links.patch
⋮

Send an email (must be less than 64 KB) to lilypond-devel@gnu.org briefly explaining your work, with the patch files attached. Translators should send patches to translations@lilynet.net. After your patches are reviewed, the developers may push one or more of them to the main repository or discuss them with you.

Emailing patches

The default x-diff MIME type associated with patch files (i.e., files whose name ends in .patch) means that the encoding of line endings may be changed from UNIX to DOS format when they are sent as attachments. Attempting to apply such an inadvertently altered patch will cause git to fail with a message about ‘whitespace errors’.

The solution to such problems is surprisingly simple—just change the default file extension of patches generated by git to end in .txt, for example:

git config format.suffix '.patch.txt'

This should cause email programs to apply the correct base64 encoding to attached patches.

If you receive a patch with DOS instead of UNIX line-endings, it can be converted back using the dos2unix utility.

Lots of useful information on email complications with patches is provided on the Wine wiki at http://wiki.winehq.org/GitWine.

3.2.6 Uploading a patch for review

Any non-trivial change should be reviewed as a merge request:

https://gitlab.com/lilypond/lilypond/-/merge_requests

Ensure your changes are committed in a separate branch, which should differ from the reference branch to be used (usually origin/master) by just the changes to be uploaded. Checkout the branch with the changes:

git checkout some-branch-with-changes

If the reference branch is to be origin/master, ensure that the branch containing the changes is up-to-date with it. Use git rebase or git pull -r to rebase the branch to the head of origin/master. For example:

git pull -r origin master

To upload a patch for review, the changes must be pushed. If you have commit access, this may use a dev/ branch. Otherwise, or at your convenience, you may use a private fork.

Afterwards create a merge request to start the review cycle. There are multiple options for this as outlined in GitLab’s documentation at https://docs.gitlab.com/ee/user/project/merge_requests/. This will also ask you for a message that will accompany your patch.

If you are not a member of the team and create the merge request from a fork, consider enabling the box to “Allow commits from members who can merge to the target branch”. This makes it possible for somebody with permissions to rebase your changes and merge them for you. Please refer to Merging to master for more details.

Revisions

As revisions are made in response to comments, successive patch sets for the same issue can be uploaded by pushing to the same branch. GitLab automatically keeps track of all pushed commits and allows to compare revisions with each other.

3.2.7 The patch review cycle

Your patch will be available for reviews for the next few hours or days. Three times a week, patches with no known problems are gathered into a “patch countdown” and their status changed to Patch::countdown. The countdown is a 48-hour waiting period in which any final reviews or complaints should be made.

During the countdown, your patch may be set to Patch::needs_work, indicating that you should fix something (or at least discuss why the patch needs no modification). If no problems are found, the patch will be set to patch-push.

Once a patch has Patch::push, it should be sent to your mentor for uploading. If you have git push ability, look at Merging to master.

• Patches get added by opening a merge request. A bot automatically adds the Patch::new label to it.
• GitLab triggers automated testing which ensures that the patch completes make, make test, and make doc. Afterwards, the Patch Meister manually runs make check to verify regression tests. If both steps pass, the merge request is labelled Patch::review. On failure, the status is updated to Patch::needs_work.
• The Patch Meister reviews the tracker periodically, to list patches which have been on review for at least 24 hours. The list is found at

  https://gitlab.com/lilypond/lilypond/-/merge_requests?label_name[]=Patch%3A%3Areview

• For each patch, the Handler reviews any discussion on the merge request, to determine whether the patch can go forward. If there is any indication that a developer thinks the patch is not ready, the Handler marks it Patch::needs_work and makes a comment regarding the reason, referring to the comment if needed.
• Patches with explicit approval, or at least no negative comment, can be updated to Patch::countdown. When saving the tracker item, clear the “send email” box to prevent sending notification for each patch.
• The Patch Meister sends an email to the developer list, with a fixed subject line, to enable filtering by email clients:

  PATCH: Countdown to 20130113

  The text of the email sets the deadline for this countdown batch. At present, batches are done on Tuesday, Thursday and Sunday evenings.

• On the scheduled countdown day, the Patch Meister reviews the previous list of patches on countdown, with the same procedure and criteria as before. Patches with no controversy can be set to “patch-push” with a courtesy message added to the comment block.
• Roughly at six month intervals, the Patch Meister can list the patches which have been set to “patch-needs-work” and send the results to the developer list for review. In most cases, these patches should be marked “patch-abandoned” but this should come from the developer if possible.
• As in most organisations of unpaid volunteers, fixed procedures are useful in as much as they get the job done. In our community, there is room for senior developers to bypass normal patch handling flows, particularly now that the testing of patches is largely automated. Similarly, the minimum age of 24 hours can reasonably be waived if the patch is minor and from an experienced developer.

3.3 Advanced Git procedures

It is possible to work with several branches on the same local Git repository; this is especially useful for translators who may have to deal with both translation and a stable branch, e.g. stable/2.12.

Some Git commands are introduced first, then a workflow with several Git branches of LilyPond source code is presented.

3.3.1 Merge conflicts

To be filled in later, and/or moved to a different section. I just wanted to make sure that I had a stub ready somewhere.

3.3.2 Advanced Git concepts

A bit of Git vocabulary will be explained below. The following is only introductory; for a better understanding of Git concepts, you may wish to read Other Git documentation.

The git pull origin command above is just a shortcut for this command:

git pull git://git.sv.gnu.org/lilypond.git/ branch:origin/branch

where branch is typically master or translation; if you do not know or remember, see Downloading remote branches to remember which commands you issued or which source code you wanted to get.

A commit is a set of changes made to the sources; it also includes the committish of the parent commit, the name and e-mail of the author (the person who wrote the changes), the name and e-mail of the committer (the person who brings these changes into the Git repository), and a commit message.

A committish is the SHA1 checksum of a commit, a number made of 40 hexadecimal digits, which acts as the internal unique identifier for this commit. To refer to a particular revision, don’t use vague references like the (approximative) date, simply copy and paste the committish.

A branch is nothing more than a pointer to a particular commit, which is called the head of the branch; when referring to a branch, one often actually thinks about its head and the ancestor commits of the head.

Now we will explain the two last commands you used to get the source code from Git—see Downloading individual branches.

git remote add -ft branch -m branch \
  origin git://git.sv.gnu.org/lilypond.git/

git checkout -b branch origin/branch

The git remote has created a branch called origin/branch in your local Git repository. As this branch is a copy of the remote branch web from git.sv.gnu.org LilyPond repository, it is called a remote branch, and is meant to track the changes on the branch from git.sv.gnu.org: it will be updated every time you run git pull origin or git fetch origin.

The git checkout command has created a branch named branch. At the beginning, this branch is identical to origin/branch, but it will differ as soon as you make changes, e.g. adding newly translated pages or editing some documentation or code source file. Whenever you pull, you merge the changes from origin/branch and branch since the last pulling. If you do not have push (i.e. “write”) access on git.sv.gnu.org, your branch will always differ from origin/branch. In this case, remember that other people working like you with the remote branch branch of git://git.sv.gnu.org/lilypond.git/ (called origin/branch on your local repository) know nothing about your own branch: this means that whenever you use a committish or make a patch, others expect you to take the latest commit of origin/branch as a reference.

Finally, please remember to read the man page of every Git command you will find in this manual in case you want to discover alternate methods or just understand how it works.

3.3.3 Resolving conflicts

Occasionally an update may result in conflicts – this happens when you and somebody else have modified the same part of the same file and git cannot figure out how to merge the two versions together. When this happens, you must manually merge the two versions.

If you need some documentation to understand and resolve conflicts, see paragraphs How conflicts are presented and How to resolve conflicts in git merge man page.

If all else fails, you can follow the instructions in Reverting all local changes. Be aware that this eliminates any changes you have made!

3.3.4 Reverting all local changes

Sometimes git will become hopelessly confused, and you just want to get back to a known, stable state. This command destroys any local changes you have made in the currently checked-out branch, but at least you get back to the current online version:

git reset --hard origin/master

3.3.5 Working with remote branches

Fetching new branches from git.sv.gnu.org

To fetch and check out a new branch named branch on git.sv.gnu.org, run from top of the Git repository

git config --add remote.origin.fetch \
  +refs/heads/branch:refs/remotes/origin/branch

git checkout --track -b branch origin/branch

After this, you can pull branch from git.sv.gnu.org with:

git pull

Note that this command generally fetches all branches you added with git remote add (when you initialized the repository) or git config --add, i.e. it updates all remote branches from remote origin, then it merges the remote branch tracked by the current branch into the current branch. For example, if your current branch is master, origin/master will be merged into master.

Local clones, or having several working trees

If you play with several Git branches, e.g. master, translation, stable/2.12), you may want to have one source and build tree for each branch; this is possible with subdirectories of your local Git repository, used as local cloned subrepositories. To create a local clone for the branch named branch, run

git checkout branch
git clone -lsn . subdir
cd subdir
git reset --hard

Note that subdir must be a directory name which does not already exist. In subdir, you can use all Git commands to browse revisions history, commit and uncommit changes; to update the cloned subrepository with changes made on the main repository, cd into subdir and run git pull; to send changes made on the subrepository back to the main repository, run git push from subdir. Note that only one branch (the currently checked out branch) is created in the subrepository by default; it is possible to have several branches in a subrepository and do usual operations (checkout, merge, create, delete...) on these branches, but this possibility is not detailed here.

When you push branch from subdir to the main repository, and branch is checked out in the main repository, you must save uncommitted changes (see git stash) and do git reset --hard in the main repository in order to apply pushed changes in the working tree of the main repository.

3.3.6 Git log

The commands above don’t only bring you the latest version of the sources, but also the full history of revisions (revisions, also called commits, are changes made to the sources), stored in the ‘.git’ directory. You can browse this history with

git log     # only shows the logs (author, committish and commit message)
git log -p  # also shows diffs
gitk        # shows history graphically

3.3.7 Applying remote patches

TODO: Explain how to determine if a patch was created with git format-patch.

Well-formed git patches created with git format-patch should be committed with the following command:

git am patch

Patches created without git format-patch can be applied in two steps. The first step is to apply the patch to the working tree and the index:

git apply --index patch

The second step is to commit the changes and give credit to the author of the patch. This can be done with the following command:

git commit --author="John Smith <john@example.com>"

Please note that using the --index option for patching is quite important here and cannot reliably be replaced by using the -a option when committing: that would only commit files from the working tree that are already registered with git, so every file that the patch actually adds, like a regtest for a fixed bug, would get lost. For the same reason, you should not use the git-independent ‘patch’ program for applying patches.

3.3.8 Cleaning up multiple patches

If you have been developing on your own branch for a while, you may have more commmits than is really sensible. To revise your work and condense commits, use:

git rebase origin/master
git rebase -i origin/master

3.3.9 Commit access

Most contributors are not able to commit patches directly to the main repository—only members of the LilyPond development team have commit access. If you are a contributor and are interested in joining the development team, contact the Project Manager through the mailing list (lilypond-devel@gnu.org). Generally, only contributors who have already provided a number of patches which have been pushed to the main repository will be considered for membership.

If you have been approved by the Project Manager, use the following procedure to obtain commit access:

1. If you don’t already have one, set up a GitLab user account at https://gitlab.com/users/sign_in.
2. After registering, if you are not logged in automatically, login at the same page, and confirm your email address.
3. Navigate to https://gitlab.com/lilypond and ‘Request access’ to the group. Make sure that your account can be related to your activity on the mailing list. If in doubt, please post the username after requesting access.

   Note that you will not have commit access until the Project Manager activates your membership. Once your membership is activated, LilyPond should appear under the heading “Groups” on your profile page.

4. Generate and register an SSH key pair. Excellent instructions are provided in GitLab’s documentation.
5. Configure Git to use the SSH protocol (instead of the GIT or HTTP protocols). From your local Git repository, enter:

   git remote set-url origin git@gitlab.com:lilypond/lilypond.git

6. After your membership has been activated and you’ve configured Git to use SSH, test the connection with:

   git pull --verbose

   SSH should issue the following warning:

   The authenticity of host 'gitlab.com' can't be established. ECDSA key fingerprint is SHA256:HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw. Are you sure you want to continue connecting (yes/no/[fingerprint])?

   Make sure the key fingerprint displayed matches the one above or one of the others published by GitLab. If it doesn’t, respond “no” and check that you configured Git properly in the previous step. If it does match, respond “yes”. SSH should then issue another warning:

   Warning: Permanently added 'gitlab.com' (ECDSA) to the list of known hosts.

   The list of known hosts is stored in the file ‘~/.ssh/known_hosts’.

   At this point, you are prompted for your passphrase if you have one, then Git will attempt a pull.

   If git pull --verbose fails, you should see error messages like these:

   Permission denied (publickey). fatal: The remote end hung up unexpectedly

   If you get the above error, you may have made a mistake when registering your SSH key. If the key is properly registered and it still doesn’t work after an hour, ask for help on the mailing list.

   If git pull --verbose succeeds, the output will include a ‘From’ line that shows ‘ssh’ as the protocol:

   From git@gitlab.com:lilypond/lilypond.git

7. Test your commit access with a dry run:

   Note: Do not push directly to master; instead, push to a private development branch.

   git push --dry-run --verbose

   Note that recent versions of Git (Git 1.6.3 or later) will issue a big warning if the above command is used. The simplest solution is to tell Git to push all matching branches by default:

   git config push.default matching

   Then git push should work as before. For more details, consult the git push man page.

8. Repeat the steps from generating an SSH key through to testing your commit access, for each machine from which you will be making commits, or you may simply copy the files from your local ‘~/.ssh’ folder to the same folder on the other machine.

Technical details

• The git remote set-url command above should modify your local repository’s ‘.git/config’ file. These lines:

  [remote "origin"] url = git://git.sv.gnu.org/lilypond.git/

  should now be changed to:

  [remote "origin"] url = git@gitlab.com:lilypond/lilypond.git

• Similarly, the git config push.default matching command should add these lines to ‘.git/config’:

  [push] default = matching

Known issues and warnings

Encryption protocols, including ssh, generally do not permit packet fragmentation to avoid introducing a point of insecurity. This means that the maximum packet size must not exceed the smallest MTU (Maximum Transmission Unit) set in the routers along the path. This smallest MTU is determined by a procedure during call set-up which relies on the transmission over the path of ICMP packets. If any of the routers in the path block ICMP packets this mechanism fails, resulting in the possibility of packets being transmitted which exceed the MTU of one of the routers. If this happens the packet is discarded, causing the ssh session to hang, timeout or terminate with the error message

ssh: connect to host <host ip addr> port 22: Bad file number
fatal: The remote end hung up unexpectedly

depending on precisely when in the proceedings the first large packet is transmitted. Most routers on the internet have MTU set to 1500, but routers installed in homes to connect via broadband may use a slightly smaller MTU for efficient transmission over ATM. If this problem is encountered a possible work-around is to set the MTU in the local router to 1500.

3.3.10 Merging to master

Do not push directly to the git master branch. Instead, all changes need to be ‘merged’ with a merge request on GitLab. Before allowing the merge, GitLab ensures the following:

1. The merge must be fast-forward. In most cases, this can be achieved by ‘rebasing’ the branch with the most recent commits from master. This can be handled via GitLab, if no conflicts arise. Otherwise, or if preferred, the operation can be performed locally. Afterwards, the branch needs to be pushed with the option --force to overwrite commits.

   Note: Be careful when performing these operations manually: They are potentially dangerous and could result in losing commits if issued incorrectly!

2. The (possibly rebased) changes must have passed automatic testing. This ensures that the master branch is always clean and ready for development and translation.

After rebasing, GitLab will immediately start the automatic testing pipeline. At the moment, all steps may take up to one hour to complete. If you are confident about the rebased result of your changes, you may click “Merge when pipeline succeeds” to avoid waiting for the tests. On failure, the merge will be aborted and no harm is done to the master branch.

Because GitLab enforces fast-forward merges, this means only one set of changes can be rebased and merged at once. A second merge request would be rejected later on because it does not contain the commit(s) merged first. To avoid wasting testing resources, please avoid this situation and check first if a pipeline with a scheduled merge is already running.

How to merge a branch without rebasing

It is generally recommended to rebase commits before merging to get a linear history. However, this is not always possible or wanted. This particularly holds for the translation branch and release/unstable which cannot be force-pushed. For these cases, use the following procedure:

1. Merge the branch manually using the command line. The example assumes no pending changes in the local master branch and merges the translation branch:

   git checkout master git pull git merge translation git push origin HEAD:translation

2. Open a merge request at GitLab. This will immediately trigger automatic testing as described above.
3. Accept the merge request once the testing finishes, or use the button to “Merge when pipeline succeeds”.

3.4 Git on Windows

TODO: Decide what to do with this... Pare it down? Move paragraphs next to analogous Unix instructions? -mp

3.4.1 Background to nomenclature

Git is a system for tracking the changes made to source files by a distributed set of editors. It is designed to work without a master repository, but we have chosen to have a master repository for LilyPond files. Editors hold a local copy of the master repository together with any changes they have made locally. Local changes are held in a local ‘branch’, of which there may be several, but these instructions assume you are using just one. The files visible in the local repository always correspond to those on the currently ‘checked out’ local branch.

Files are edited on a local branch, and in that state the changes are said to be ‘unstaged’. When editing is complete, the changes are moved to being ‘staged for commit’, and finally the changes are ‘committed’ to the local branch. Once committed, the changes (called a ‘commit’) are given a unique 40-digit hexadecimal reference number called the ‘Committish’ or ‘SHA1 ID’ which identifies the commit to Git. Such committed changes can be sent to the master repository by ‘pushing’ them (if you have write permission) or by sending them by email to someone who has, either as a complete file or as a ‘diff’ or ‘patch’ (which send just the differences from the master repository).

3.4.2 Installing git

Obtain Git from https://git-for-windows.github.io/.

Note that most users will not need to install SSH. That is not required until you have been granted direct push permissions to the master git repository.

Start Git by clicking on the desktop icon. This will bring up a command line bash shell. This may be unfamiliar to Windows users. If so, follow these instructions carefully. Commands are entered at a $ prompt and are terminated by keying a newline.

3.4.3 Initialising Git

Decide where you wish to place your local Git repository, creating the folders in Windows as necessary. Here we call the folder to contain the repository [path]/Git, but if you intend using Git for other projects a directory name like lilypond-git might be better. You will need to have space for around 100Mbytes.

Start the Git bash shell by clicking on the desk-top icon installed with Git and type

cd [path]/Git

to position the shell at your new Git repository.

Note: if [path] contains folders with names containing spaces use

cd "[path]/Git"

Then type

git init

to initialize your Git repository.

Then type (all on one line; the shell will wrap automatically)

git remote add -ft master origin git://git.sv.gnu.org/lilypond.git

to download the lilypond master files.

We now need to generate a local copy of the downloaded files in a new local branch. Your local branch needs to have a name. It is usual to call it ‘master’ and we shall do that here.

To do this, type

git checkout -b master origin/master

This creates a second branch called ‘master’. You will see two warnings (ignore these), and a message advising you that your local branch ‘master’ has been set up to track the remote branch. You now have two branches, a local branch called ‘master’, and a tracking branch called ‘origin/master’, which is a shortened form of ‘remotes/origin/master’.

Return to Windows Explorer and look in your Git repository. You should see lots of folders. For example, the LilyPond documentation can be found in [path]/Git/Documentation/.

The Git bash shell is terminated by typing exit or by clicking on the usual Windows close-window widget.

3.4.4 Git GUI

Almost all subsequent work will use the Git Graphical User Interface, which avoids having to type command line commands. To start Git GUI first start the Git bash shell by clicking on the desktop icon, and type

cd [path]/Git
git gui

The Git GUI will open in a new window. It contains four panels and 7 pull-down menus. At this stage do not use any of the commands under Branch, Commit, Merge or Remote. These will be explained later.

The top panel on the left contains the names of files which you are in the process of editing (Unstaged Changes), and the lower panel on the left contains the names of files you have finished editing and have staged ready for committing (Staged Changes). At present, these panels will be empty as you have not yet made any changes to any file. After a file has been edited and saved the top panel on the right will display the differences between the edited file selected in one of the panels on the left and the last version committed on the current branch.

The panel at bottom right is used to enter a descriptive message about the change before committing it.

The Git GUI is terminated by entering CNTL-Q while it is the active window or by clicking on the usual Windows close-window widget.

3.4.5 Personalising your local git repository

Open the Git GUI, click on

Edit -> Options

and enter your name and email address in the left-hand (Git Repository) panel. Leave everything else unchanged and save it.

Note that Windows users must leave the default setting for line endings unchanged. All files in a git repository must have lines terminated by just a LF, as this is required for Merge to work, but Windows files are terminated by CRLF by default. The git default setting causes the line endings of files in a Windows git repository to be flipped automatically between LF and CRLF as required. This enables files to be edited by any Windows editor without causing problems in the git repository.

3.4.6 Checking out a branch

At this stage you have two branches in your local repository, both identical. To see them click on

Branch -> Checkout

You should have one local branch called ‘master’ and one tracking branch called ‘origin/master’. The latter is your local copy of the ‘remotes/origin/master’ branch in the master LilyPond repository. The local ‘master’ branch is where you will make your local changes.

When a particular branch is selected, i.e., checked out, the files visible in your repository are changed to reflect the state of the files on that branch.

3.4.7 Updating files from ‘remote/origin/master’

Before starting the editing of a file, ensure your local repository contains the latest version of the files in the remote repository by first clicking

Remote -> Fetch from -> origin

in the Git GUI.

This will place the latest version of every file, including all the changes made by others, into the ‘origin/master’ branch of the tracking branches in your git repository. You can see these files by checking out this branch, but you must never edit any files while this branch is checked out. Check out your local ‘master’ branch again.

You then need to merge these fetched files into your local ‘master’ branch by clicking on

Merge -> Local Merge

and if necessary select the local ‘master’ branch.

Note that a merge cannot be completed if you have made any local changes which have not yet been committed.

This merge will update all the files in the ‘master’ branch to reflect the current state of the ‘origin/master’ branch. If any of the changes conflict with changes you have made yourself recently you will be notified of the conflict (see below).

3.4.8 Editing files

First ensure your ‘master’ branch is checked out, then simply edit the files in your local Git repository with your favourite editor and save them back there. If any file contains non-ASCII characters ensure you save it in UTF-8 format. Git will detect any changes whenever you restart Git GUI and the file names will then be listed in the Unstaged Changes panel. Or you can click the Rescan button to refresh the panel contents at any time. You may break off and resume editing any time.

The changes you have made may be displayed in diff form in the top right-hand panel of Git GUI by clicking on the file name shown in one of the left panels.

When your editing is complete, move the files from being Unstaged to Staged by clicking the document symbol to the left of each name. If you change your mind it can be moved back by clicking on the ticked box to the left of the name.

Finally the changes you have made may be committed to your ‘master’ branch by entering a brief message in the Commit Message box and clicking the Commit button.

If you wish to amend your changes after a commit has been made, the original version and the changes you made in that commit may be recovered by selecting

Commit -> Amend Last Commit

or by checking the Amend Last Commit radio button at bottom right. This will return the changes to the Staged state, so further editing made be carried out within that commit. This must only be done before the changes have been Pushed or sent to your mentor for Pushing - after that it is too late and corrections have to be made as a separate commit.

3.4.9 Sending changes to ‘remotes/origin/master’

If you do not have write access to ‘remotes/origin/master’ you will need to send your changes by email to someone who does.

First you need to create a diff or patch file containing your changes. To create this, the file must first be committed. Then terminate the Git GUI. In the git bash shell first cd to your Git repository with

cd [path]/Git

if necessary, then produce the patch with

git format-patch origin

This will create a patch file for all the locally committed files which differ from ‘origin/master’. The patch file can be found in [path]/Git and will have a name formed from the commit message.

3.4.10 Resolving merge conflicts

As soon as you have committed a changed file your local master branch has diverged from origin/master, and will remain diverged until your changes have been committed in remotes/origin/master and Fetched back into your origin/master branch. Similarly, if a new commit has been made to remotes/origin/master by someone else and Fetched, your local master branch is divergent. You can detect a divergent branch by clicking on

Repository -> Visualise all branch history

This opens up a very useful new window called ‘gitk’. Use this to browse all the commits made by yourself and others.

If the diagram at top left of the resulting window does not show your master tag on the same node as the remotes/origin/master tag your branch has diverged from origin/master. This is quite normal if files you have modified yourself have not yet been Pushed to remotes/origin/master and Fetched, or if files modified and committed by others have been Fetched since you last Merged origin/master into your local master branch.

If a file being merged from origin/master differs from one you have modified in a way that cannot be resolved automatically by git, Merge will report a Conflict which you must resolve by editing the file to create the version you wish to keep.

This could happen if the person updating remotes/origin/master for you has added some changes of his own before committing your changes to remotes/origin/master, or if someone else has changed the same file since you last fetched the file from remotes/origin/master.

Open the file in your editor and look for sections which are delimited with ...

[to be completed when I next have a merge conflict to be sure I give the right instructions -td]

3.4.11 Other actions

The instructions above describe the simplest way of using git on Windows. Other git facilities which may usefully supplement these include

• Using multiple local branches (Create, Rename, Delete)
• Resetting branches
• Cherry-picking commits
• Pushing commits to remote/origin/master
• Using gitk to review history

Once familiarity with using git on Windows has been gained the standard git manuals can be used to learn about these.

3.5 Repository directory structure

Prebuilt Documentation and packages are available from:

    http://www.lilypond.org

LilyPond development is hosted at:

    http://savannah.gnu.org/projects/lilypond

Here is a simple explanation of the directory layout for
LilyPond's source files.


.                        Toplevel READMEs, ChangeLog,
|                          build bootstrapping, patches
|                          for third party programs
|
|-- Documentation/       Top sources for most of the manuals
|   |
|   |
|   |   INDIVIDUAL CHAPTERS FOR EACH MANUAL:
|   |     Note: "Snippets" and "Internals Reference" are
|   |     auto-generated during the Documentation Build process.
|   |
|   |
|   |-- contributor/     Contributor's Guide
|   |-- essay/           Essay on automated music engraving
|   |-- extending/       Extending the functionality of LilyPond
|   |-- learning/        Learning Manual
|   |-- notation/        Notation Reference
|   |-- usage/           Runnning the programs that come with LilyPond
|   |-- web/             The website
|   |
|   |
|   |   TRANSLATED MANUALS:
|   |     Each language's directory can contain...
|   |       1) translated versions of:
|   |          * top sources for manuals
|   |          * individual chapters for each manual
|   |       2) a texidocs/ directory for snippet translations
|   |
|   |-- ca/              Catalan
|   |-- cs/              Czech
|   |-- de/              German
|   |-- es/              Spanish
|   |-- fr/              French
|   |-- hu/              Hungarian
|   |-- it/              Italian
|   |-- ja/              Japanese
|   |-- nl/              Dutch
|   |-- pt/              Portuguese
|   |-- zh/              Chinese
|   |
|   |
|   |   MISCELLANEOUS DOC STUFF:
|   |
|   |-- css/             CSS files for HTML docs
|   |-- included/        .ly files used in the manuals
|   |-- logo/            Web logo and "note" icon
|   |-- ly-examples/     .ly files for the "Examples" webpage
|   |-- misc/            Old announcements, ChangeLogs and NEWS
|   |-- pictures/        Images used (eps/jpg/png/svg)
|   |   `-- pdf/         (pdf)
|   |-- po/              Translated build/maintenance scripts
|   |-- snippets/        Auto-generated from the LSR and from ./new/
|   |   `-- new/         Snippets too new for the LSR
|   `-- topdocs/         AUTHORS, INSTALL, README
|
|
|   C++ SOURCES:
|
|-- flower/              A simple C++ library
|-- lily/                C++ sources for the LilyPond binary
|
|
|   LIBRARIES:
|
|-- ly/                  .ly \include files
|-- mf/                  MetaFont sources for Emmentaler fonts
|-- ps/                  PostScript library files
|-- scm/                 Scheme sources for LilyPond and subroutine files
|-- tex/                 TeX and texinfo library files
|
|
|   SCRIPTS:
|
|-- config/              Autoconf helpers for configure script
|-- python/              Python modules, MIDI module
|   `-- auxiliar/        Python modules for build/maintenance
|-- scripts/             End-user scripts (--> lilypond/usr/bin/)
|   |-- auxiliar/        Maintenance and non-essential build scripts
|   `-- build/           Essential build scripts
|
|
|   BUILD PROCESS:
|   (also see SCRIPTS section above)
|
|-- make/                Specific make subroutine files
|-- stepmake/            Generic make subroutine files
|
|
|   REGRESSION TESTS:
|
|-- input/
|   `-- regression/      .ly regression tests
|       |-- abc2ly/      .abc regression tests
|       |-- lilypond-book/  lilypond-book regression tests
|       |-- midi/        midi2ly regression tests
|       `-- musicxml/    .xml and .itexi regression tests
|
|
|   MISCELLANEOUS:
|
|-- elisp/               Emacs LilyPond mode and syntax coloring
|-- vim/                 Vi(M) LilyPond mode and syntax coloring
`-- po/                  Translations for binaries and end-user scripts

3.6 Other Git documentation

• Official git man pages: http://www.kernel.org/pub/software/scm/git/docs/
• More in-depth tutorials: http://git-scm.com/documentation
• Book about git: Pro Git
• Github help: http://help.github.com/ (very highly recommended by Graham)

4. Compiling

This chapter describes the process of compiling the LilyPond program from source files.

4.1 Overview of compiling

Compiling LilyPond from source is an involved process, and is only recommended for developers and packagers. Typical program users are instead encouraged to obtain the program from a package manager (on Unix) or by downloading a precompiled binary configured for a specific operating system. Pre-compiled binaries are available on the (web-big-page)Download section ‘Download’ in General Information page.

Compiling LilyPond from source is necessary if you want to build, install, or test your own version of the program.

A successful compile can also be used to generate and install the documentation, incorporating any changes you may have made. However, a successful compile is not a requirement for generating the documentation. The documentation can be built using a Git repository in conjunction with a locally installed copy of the program. For more information, see Building documentation without compiling.

Attempts to compile LilyPond natively on Windows have been unsuccessful, though a workaround is available (see (contributor-big-page)LilyDev section ‘LilyDev’ in Contributor’s Guide).

4.2 Requirements

4.2.1 Requirements for running LilyPond

This section contains the list of separate software packages that are required to run LilyPond.

• DejaVu fonts These are normally installed by default.
• FontConfig Use version 2.4.0 or newer.
• Freetype Use version 2.1.10 or newer.
• Ghostscript Use version 9.03 or newer.
• Guile Use version 1.8.8. Version 2.x of Guile is not currently supported.
• Pango User version 1.12 or newer.
• Python Use version 3.5 or newer.
• International fonts. For example:

  Fedora:

  fonts-arabic fonts-hebrew fonts-ja fonts-xorg-truetype taipeifonts ttfonts-ja ttfonts-zh_CN

  Debian based distributions:

  emacs-intl-fonts fonts-ipafont-gothic fonts-ipafont-mincho xfonts-bolkhov-75dpi xfonts-cronyx-75dpi xfonts-cronyx-100dpi xfonts-intl-.*

  These are normally installed by default and are required only to create music with international text or lyrics.

4.2.2 Requirements for compiling LilyPond

This section contains instructions on how to quickly and easily get all the software packages required to build LilyPond.

Most of the more popular Linux distributions only require a few simple commands to download all the software needed. For others, there is an explicit list of all the individual packages (as well as where to get them from) for those that are not already included in your distributions’ own repositories.

Fedora

The following instructions were tested on ‘Fedora’ versions 22 & 23 and will download all the software required to both compile LilyPond and build the documentation.

• Download and install all the LilyPond build-dependencies (approximately 700MB);

  sudo dnf builddep lilypond --nogpgcheck

• Download and install additional ‘build’ tools required for compiling;

  sudo dnf install autoconf gcc-c++

• Download texi2html 1.82 directly from: http://download.savannah.gnu.org/releases/texi2html/texi2html-1.82.tar.gz;

  texi2html is only required if you intend to compile LilyPond’s own documentation (e.g. to help with any document writing). The version available in the Fedora repositories is too new and will not work. Extract the files into an appropriate location and then run the commands;

  ./configure make sudo make install

  This should install texi2html 1.82 into /usr/local/bin, which will normally take priority over /usr/bin where the later, pre-installed versions gets put. Now verify that your operating system is able to see the correct version of texi2html.

  texi2html --version

• Although not ‘required’ to compile LilyPond, if you intend to contribute to LilyPond (codebase or help improve the documentation) then it is recommended that you also need to install git.

  sudo dnf install git

  Also see (contributor-big-page)Starting with Git section ‘Starting with Git’ in Contributor’s Guide.

sudo dnf install texlive-xetex

Linux Mint

The following instructions were tested on ‘Linux Mint 17.1’ and ‘LMDE - Betsy’ and will download all the software required to both compile LilyPond and build the documentation..

• Enable the sources repository;
  1. Using the Software Sources GUI (located under Administration).
  2. Select Official Repositories.
  3. Check the Enable source code repositories box under the Source Code section.
  4. Click the Update the cache button and when it has completed, close the Software Sources GUI.
• Download and install all the LilyPond build-dependencies (approximately 200MB);

  sudo apt-get build-dep lilypond

• Download and install additional ‘build’ tools required for compiling;

  sudo apt-get install autoconf fonts-texgyre texlive-lang-cyrillic

• Although not ‘required’ to compile LilyPond, if you intend to contribute to LilyPond (codebase or help improve the documentation) then it is recommended that you also need to install git.

  sudo apt-get install git

  Also see (contributor-big-page)Starting with Git section ‘Starting with Git’ in Contributor’s Guide.

sudo apt-get install texlive-xetex

OpenSUSE

The following instructions were tested on ‘OpenSUSE 13.2’ and will download all the software required to both compile LilyPond and build the documentation.

• Add the sources repository;

  sudo zypper addrepo -f \ "http://download.opensuse.org/source/distribution/13.2/repo/oss/" sources

• Download and install all the LilyPond build-dependencies (approximately 680MB);

  sudo zypper source-install lilypond

• Download and install additional ‘build’ tools required for compiling;

  sudo zypper install make

• Although not ‘required’ to compile LilyPond, if you intend to contribute to LilyPond (codebase or help improve the documentation) then it is recommended that you also need to install git.

  sudo zypper install git

  Also see (contributor-big-page)Starting with Git section ‘Starting with Git’ in Contributor’s Guide.

sudo zypper install texlive-xetex

Ubuntu

The following commands were tested on Ubuntu versions 14.04 LTS, 14.10 and 15.04 and will download all the software required to both compile LilyPond and build the documentation.

• Download and install all the LilyPond build-dependencies (approximately 200MB);

  sudo apt-get build-dep lilypond

• Download and install additional ‘build’ tools required for compiling;

  sudo apt-get install autoconf fonts-texgyre texlive-lang-cyrillic

• Although not ‘required’ to compile LilyPond, if you intend to contribute to LilyPond (codebase or help improve the documentation) then it is recommended that you also need to install git.

  sudo apt-get install git

  Also see (contributor-big-page)Starting with Git section ‘Starting with Git’ in Contributor’s Guide.

sudo apt-get install texlive-xetex

Other

The following individual software packages are required just to compile LilyPond.

• GNU Autoconf
• GNU Bison

  Use version 2.0 or newer.

• GNU Compiler Collection

  Use version 3.4 or newer (4.x recommended).

• Flex
• FontForge

  Use version 20060125 or newer (we recommend using at least 20100501); it must also be compiled with the ‘--enable-double’ switch, else this can lead to inaccurate intersection calculations which end up with poorly-rendered glyphs in the output.

• GNU gettext

  Use version 0.17 or newer.

• GNU Make

  Use version 3.78 or newer.

• MetaFont

  The mf-nowin, mf, mfw or mfont binaries are usually packaged along with TeX.

• MetaPost

  The mpost binary is also usually packaged with TeX.

• Perl
• Texinfo

  Use version 6.1 or newer.

• Type 1 utilities

  Use version 1.33 or newer.

• Cyrillic fonts

  Often packaged in repositories as texlive-lang-cyrillic.

• TeX Gyre ‘OTF’ font packages. As of LilyPond version 2.19.26, the previous default serif, san serif and monospace fonts now use Tex Gyre’s Schola, Heros and Cursor fonts respectively. Also See (notation-big-page)Fonts section ‘Fonts’ in Notation Reference.

  Some distributions do not always provide ‘OTF’ font files in the Tex Gyre packages from their repositories. Use the command fc-list | grep texgyre to list the fonts available to your system and check that the appropriate *.otf files are reported. If they are not then download and manually extract the ‘OTF’ files to either your local ~/.fonts/ directory or use the configure command and the --with-texgyre-dir=/path_to_otf_files/ option.

  The following font families are required:

  Schola, Heros and Cursor.

4.2.3 Requirements for building documentation

The entire set of documentation for the most current build of LilyPond is available online at https://lilypond.org/doc/v2.19/Documentation/web/development, but you can also build them locally from the source code. This process requires some additional tools and packages.

• Everything listed in Requirements for compiling LilyPond
• ImageMagick
• Netpbm
• gzip
• rsync
• Texi2HTML

  Use version 1.82. Later versions will not work.

  Download texi2html 1.82 directly from: http://download.savannah.gnu.org/releases/texi2html/texi2html-1.82.tar.gz;

  Extract the files into an appropriate location and then run the commands;

  ./configure make sudo make install

  Now verify that your operating system is able to see the correct version of texi2html.

  texi2html --version

• Fonts required to build the documentation in addition to those required to run LilyPond:

  gsfonts fonts-linuxlibertine fonts-liberation fonts-dejavu fonts-freefont-otf ttf-bitstream-vera texlive-fonts-recommended ttf-xfree86-nonfree

4.3 Getting the source code

Downloading the Git repository

In general, developers compile LilyPond from within a local Git repository. Setting up a local Git repository is explained in (contributor-big-page)Starting with Git section ‘Starting with Git’ in Contributor’s Guide.

Downloading a source tarball

Packagers are encouraged to use source tarballs for compiling.

The tarball for the latest stable release is available on the (web-big-page)Source section ‘Source’ in General Information page.

The latest source code snapshot is also available as a tarball from the GNU Savannah Git server.

All tagged releases (including legacy stable versions and the most recent development release) are available here:

https://lilypond.org/download/source/

Download the tarball to your ‘~/src/’ directory, or some other appropriate place.

Unpack the tarball with this command:

tar -xzf lilypond-x.y.z.tar.gz

This creates a subdirectory within the current directory called ‘lilypond-x.y.z/’. Once unpacked, the source files occupy about 40 MB of disk space.

Windows users wanting to look at the source code may have to download and install the free-software 7zip archiver to extract the tarball.

4.4 Configuring make

4.4.1 Running ./autogen.sh

After you unpack the tarball (or download the Git repository), the contents of your top source directory should be similar to the current source tree listed at http://git.sv.gnu.org/gitweb/?p=lilypond.git;a=tree.

Next, you need to create the generated files; enter the following command from your top source directory:

./autogen.sh --noconfigure

This will generate a number of files and directories to aid configuration, such as ‘configure’, ‘README.txt’, etc.

Next, create the build directory with:

mkdir build/
cd build/

We heavily recommend building lilypond inside a separate directory with this method.

4.4.2 Running ../configure

Configuration options

The ../configure command (generated by ./autogen.sh) provides many options for configuring make. To see them all, run:

../configure --help

Checking build dependencies

When ../configure is run without any arguments, it will check to make sure your system has everything required for compilation:

../configure

If any build dependency is missing, ../configure will return with:

ERROR: Please install required programs:  foo

The following message is issued if you are missing programs that are only needed for building the documentation:

WARNING: Please consider installing optional programs:  bar

If you intend to build the documentation locally, you will need to install or update these programs accordingly.

Configuring target directories

If you intend to use your local build to install a local copy of the program, you will probably want to configure the installation directory. Here are the relevant lines taken from the output of ../configure --help:

By default, ‘make install’ will install all the files in ‘/usr/local/bin’, ‘/usr/local/lib’ etc. You can specify an installation prefix other than ‘/usr/local’ using ‘‘--prefix’’, for instance ‘‘--prefix=$HOME’’.

A typical installation prefix is ‘$HOME/usr’:

../configure --prefix=$HOME/usr

Note that if you plan to install a local build on a system where you do not have root privileges, you will need to do something like this anyway—make install will only succeed if the installation prefix points to a directory where you have write permission (such as your home directory). The installation directory will be automatically created if necessary.

The location of the lilypond command installed by this process will be ‘prefix/bin/lilypond’; you may want to add ‘prefix/bin/’ to your $PATH if it is not already included.

It is also possible to specify separate installation directories for different types of program files. See the full output of ../configure --help for more information.

If you encounter any problems, please see Problems.

4.5 Compiling LilyPond

4.5.1 Using make

LilyPond is compiled with the make command. Assuming make is configured properly, you can simply run:

make

‘make’ is short for ‘make all’. To view a list of make targets, run:

make help

TODO: Describe what make actually does.

See also

Generating documentation provides more info on the make targets used to build the LilyPond documentation.

4.5.2 Saving time with the ‘-j’ option

If your system has multiple CPUs, you can speed up compilation by adding ‘-jX’ to the make command, where ‘X’ is one more than the number of cores you have. For example, a typical Core2Duo machine would use:

make -j3

If you get errors using the ‘-j’ option, and ‘make’ succeeds without it, try lowering the X value.

Because multiple jobs run in parallel when ‘-j’ is used, it can be difficult to determine the source of an error when one occurs. In that case, running ‘make’ without the ‘-j’ is advised.

4.5.3 Useful make variables

make normally echoes each command, but LilyPond makefiles suppress this behavior by default. The goal is to show progress without hiding warnings and errors in the noise of long commands.

To enable echoing commands, and to increase the verbosity of some of the commands, set VERBOSE=1 on the command line or in ‘local.make’ at the top of the build tree.

Similarly, to reduce the verbosity, set SILENT=1. Because of the way these options are implemented, make -s does not serve this purpose.

4.6 Post-compilation options

4.6.1 Installing LilyPond from a local build

If you configured make to install your local build in a directory where you normally have write permission (such as your home directory), and you have compiled LilyPond by running make, you can install the program in your target directory by running:

make install

If instead, your installation directory is not one that you can normally write to (such as the default ‘/usr/local/’, which typically is only writeable by the superuser), you will need to temporarily become the superuser when running make install:

sudo make install

or…

su -c 'make install'

If you don’t have superuser privileges, then you need to configure the installation directory to one that you can write to, and then re-install. See Configuring target directories.

4.6.2 Generating documentation

Documentation editor’s edit/compile cycle

• Initial documentation build:

  make [-jX] make [-jX CPU_COUNT=X] doc ## can take an hour or more make [-jX CPU_COUNT=X] doc-stage-1 ## to build only PDF documentation

• Edit/compile cycle:

  ## edit source files, then… make [-jX] ## needed if editing outside ## Documentation/, but useful anyway ## for finding Texinfo errors. make [-jX CPU_COUNT=X] doc ## usually faster than initial build.

• Reset:

  It is generally possible to remove the compiled documentation from your system with ‘make doc-clean’, but this method is not 100% guaranteed. Instead, if you want to be sure you have a clean system, we recommend that you delete your ‘build/’ directory, and begin compiling from scratch. Since the documentation compile takes much longer than the non-documentation compile, this does not increase the overall time by a great deal.

Building documentation

After a successful compile (using make), the documentation can be built by issuing:

make doc

or, to build only the PDF documentation and not the HTML,

make doc-stage-1

After this initial build, make doc only makes changes to the documentation where needed, so it may only take a minute or two to test changes if the documentation is already built.

If make doc succeeds, the HTML documentation tree is available in ‘out-www/offline-root/’, and can be browsed locally. Various portions of the documentation can be found by looking in ‘out/’ and ‘out-www’ subdirectories in other places in the source tree, but these are only portions of the docs. Please do not complain about anything which is broken in those places; the only complete set of documentation is in ‘out-www/offline-root/’ from the top of the source tree.

make doc sends the output from most of the compilation to logfiles. If the build fails for any reason, it should prompt you with the name of a logfile which will provide information to help you work out why the build failed. These logfiles are not deleted with make doc-clean. To remove all the logfiles generated by the compilation process, use:

find -name '*.log' -delete

make doc compiles the documents for all languages. To save some compile time, the English language documents can be compiled on their own with:

make LANGS='' doc

Similarly, it is possible to compile a subset of the translated documentation by specifying their language codes on the command line. For example, the French and German translations are compiled with:

make LANGS='de fr' doc

Note that this will also compile the English version.

Compilation of documentation in Info format with images can be done separately by issuing:

make info

An issue when switching branches between master and translation is the appearance/disappearance of translated versions of some manuals. If you see such a warning from make:

No rule to make target `X', needed by `Y'

Your best bet is to delete the file Y.dep and to try again.

Building a single document

It’s possible to build a single document. For example, to rebuild only ‘contributor.pdf’, do the following:

cd build/
cd Documentation/
touch ../../Documentation/contributor.texi
make out=www out-www/contributor.pdf

If you are only working on a single document, test-building it in this way can give substantial time savings - recreating ‘contributor.pdf’, for example, takes a matter of seconds.

Saving time with CPU_COUNT

The most time consuming task for building the documentation is running LilyPond to build images of music, and there cannot be several simultaneously running lilypond-book instances, so the ‘-j’ make option does not significantly speed up the build process. To help speed it up, the makefile variable ‘CPU_COUNT’ may be set in ‘local.make’ or on the command line to the number of .ly files that LilyPond should process simultaneously, e.g. on a bi-processor or dual core machine:

make -j3 CPU_COUNT=3 doc

The recommended value of ‘CPU_COUNT’ is one plus the number of cores or processors, but it is advisable to set it to a smaller value unless your system has enough RAM to run that many simultaneous LilyPond instances. Also, values for the ‘-j’ option that pose problems with ‘make’ are less likely to pose problems with ‘make doc’ (this applies to both ‘-j’ and ‘CPU_COUNT’). For example, with a quad-core processor, it is possible for ‘make -j5 CPU_COUNT=5 doc’ to work consistently even if ‘make -j5’ rarely succeeds.

Installing documentation

The HTML, PDF and if available Info files can be installed into the standard documentation path by issuing

make install-doc

This also installs Info documentation with images if the installation prefix is properly set; otherwise, instructions to complete proper installation of Info documentation are printed on standard output.

To install the Info documentation separately, run:

make install-info

Note that to get the images in Info documentation, install-doc target creates symbolic links to HTML and PDF installed documentation tree in ‘prefix/share/info’, in order to save disk space, whereas install-info copies images in ‘prefix/share/info’ subdirectories.

It is possible to build a documentation tree in ‘out-www/online-root/’, with special processing, so it can be used on a website with content negotiation for automatic language selection; this can be achieved by issuing

make WEB_TARGETS=online doc

and both ‘offline’ and ‘online’ targets can be generated by issuing

make WEB_TARGETS="offline online" doc

Several targets are available to clean the documentation build and help with maintaining documentation; an overview of these targets is available with

make help

from every directory in the build tree. Most targets for documentation maintenance are available from ‘Documentation/’; for more information, see (contributor-big-page)Documentation work section ‘Documentation work’ in Contributor’s Guide.

The makefile variable QUIET_BUILD may be set to 1 for a less verbose build output, just like for building the programs.

Building documentation without compiling

The documentation can be built locally without compiling LilyPond binary, if LilyPond is already installed on your system.

From a fresh Git checkout, do

./autogen.sh   # ignore any warning messages
cp GNUmakefile.in GNUmakefile
make -C scripts && make -C python
nice make LILYPOND_EXTERNAL_BINARY=/path/to/bin/lilypond doc

Please note that this may break sometimes – for example, if a new feature is added with a test file in input/regression, even the latest development release of LilyPond will fail to build the docs.

You may build the manual without building all the ‘input/*’ stuff (i.e. mostly regression tests): change directory, for example to ‘Documentation/’, issue make doc, which will build documentation in a subdirectory ‘out-www’ from the source files in current directory. In this case, if you also want to browse the documentation in its post-processed form, change back to top directory and issue

make out=www WWW-post

Known issues and warnings

You may also need to create a script for pngtopnm and pnmtopng. On GNU/Linux, I use this:

export LD_LIBRARY_PATH=/usr/lib
exec /usr/bin/pngtopnm "$@"

On MacOS X with fink, I use this:

export DYLD_LIBRARY_PATH=/sw/lib
exec /sw/bin/pngtopnm "$@"

On MacOS X with macports, you should use this:

export DYLD_FALLBACK_LIBRARY_PATH=/opt/local/lib
exec /opt/local/bin/pngtopnm "$@"

4.6.3 Testing LilyPond binary

LilyPond comes with an extensive suite that exercises the entire program. This suite can be used to test that the binary has been built correctly.

The test suite can be executed with:

make test

If the test suite completes successfully, the LilyPond binary has been verified.

More information on the regression test suite is found at (contributor-big-page)Regression tests section ‘Regression tests’ in Contributor’s Guide.

4.7 Problems

For help and questions use lilypond-user@gnu.org. Send bug reports to bug-lilypond@gnu.org.

Bugs that are not fault of LilyPond are documented here.

Compiling on MacOS X

Here are special instructions for compiling under MacOS X. These instructions assume that dependencies are installed using MacPorts. The instructions have been tested using OS X 10.5 (Leopard).

First, install the relevant dependencies using MacPorts.

Next, add the following to your relevant shell initialization files. This is ~/.profile by default. You should create this file if it does not exist.

export PATH=/opt/local/bin:/opt/local/sbin:$PATH
export DYLD_FALLBACK_LIBRARY_PATH=/opt/local/lib:$DYLD_FALLBACK_LIBRARY_PATH

Now you must edit the generated ‘config.make’ file. Change

FLEXLEXER_FILE = /usr/include/FlexLexer.h

to:

FLEXLEXER_FILE = /opt/local/include/FlexLexer.h

At this point, you should verify that you have the appropriate fonts installed with your ghostscript installation. Check ls /opt/local/share/ghostscript/fonts for: ’c0590*’ files (.pfb, .pfb and .afm). If you don’t have them, run the following commands to grab them from the ghostscript SVN server and install them in the appropriate location:

svn export http://svn.ghostscript.com/ghostscript/tags/urw-fonts-1.0.7pre44/
sudo mv urw-fonts-1.0.7pre44/* /opt/local/share/ghostscript/fonts/
rm -rf urw-fonts-1.07pre44

Now run the ./configure script. To avoid complications with automatic font detection, add

--with-fonts-dir=/opt/local/share/ghostscript/fonts

FreeBSD

To use system fonts, dejaview must be installed. With the default port, the fonts are installed in ‘usr/X11R6/lib/X11/fonts/dejavu’.

Open the file ‘$LILYPONDBASE/usr/etc/fonts/local.conf’ and add the following line just after the <fontconfig> line. (Adjust as necessary for your hierarchy.)

<dir>/usr/X11R6/lib/X11/fonts</dir>

International fonts

On Mac OS X, all fonts are installed by default. However, finding all system fonts requires a bit of configuration; see this post on the lilypond-user mailing list.

On Linux, international fonts are installed by different means on every distribution. We cannot list the exact commands or packages that are necessary, as each distribution is different, and the exact package names within each distribution changes. Here are some hints, though:

Red Hat Fedora

    taipeifonts fonts-xorg-truetype ttfonts-ja fonts-arabic \
         ttfonts-zh_CN fonts-ja fonts-hebrew

Debian GNU/Linux

   apt-get install emacs-intl-fonts xfonts-intl-.* \
        fonts-ipafont-gothic  fonts-ipafont-mincho \
        xfonts-bolkhov-75dpi xfonts-cronyx-100dpi xfonts-cronyx-75dpi

Using lilypond python libraries

If you want to use lilypond’s python libraries (either running certain build scripts manually, or using them in other programs), set PYTHONPATH to ‘python/out’ in your build directory, or ‘…/usr/lib/lilypond/current/python’ in the installation directory structure.

4.8 Concurrent stable and development versions

It can be useful to have both the stable and the development versions of LilyPond available at once. One way to do this on GNU/Linux is to install the stable version using the precompiled binary, and run the development version from the source tree. After running make all from the top directory of the LilyPond source files, there will be a binary called lilypond in the out directory:

<path to>/lilypond/out/bin/lilypond

This binary can be run without actually doing the make install command. The advantage to this is that you can have all of the latest changes available after pulling from git and running make all, without having to uninstall the old version and reinstall the new.

So, to use the stable version, install it as usual and use the normal commands:

lilypond foobar.ly

To use the development version, create a link to the binary in the source tree by saving the following line in a file somewhere in your $PATH:

exec <path to>/lilypond/out/bin/lilypond "$@"

Save it as Lilypond (with a capital L to distinguish it from the stable lilypond), and make it executable:

chmod +x Lilypond

Then you can invoke the development version this way:

Lilypond foobar.ly

TODO: ADD

- other compilation tricks for developers

4.9 Build system

We currently use make and stepmake, which is complicated and only used by us. Hopefully this will change in the future.

Version-specific texinfo macros

• made with scripts/build/create-version-itexi.py and
  scripts/build/create-weblinks-itexi.py
• used extensively in the WEBSITE_ONLY_BUILD version of the website (made with ‘website.make’, used on lilypond.org)
• not (?) used in the main docs?
• the numbers in VERSION file: MINOR_VERSION should be 1 more than the last release, VERSION_DEVEL should be the last online release. Yes, VERSION_DEVEL is less than VERSION.

5. Documentation work

There are currently 11 manuals for LilyPond, not including the translations. Each book is available in HTML, PDF, and info. The documentation is written in a language called texinfo – this allows us to generate different output formats from a single set of source files.

To organize multiple authors working on the documentation, we use a Version Control System (VCS) called Git, previously discussed in Starting with Git.

5.1 Introduction to documentation work

Our documentation tries to adhere to our Documentation policy. This policy contains a few items which may seem odd. One policy in particular is often questioned by potential contributors: we do not repeat material in the Notation Reference, and instead provide links to the “definitive” presentation of that information. Some people point out, with good reason, that this makes the documentation harder to read. If we repeated certain information in relevant places, readers would be less likely to miss that information.

That reasoning is sound, but we have two counter-arguments. First, the Notation Reference – one of five manuals for users to read – is already over 500 pages long. If we repeated material, we could easily exceed 1000 pages! Second, and much more importantly, LilyPond is an evolving project. New features are added, bugs are fixed, and bugs are discovered and documented. If features are discussed in multiple places, the documentation team must find every instance. Since the manual is so large, it is impossible for one person to have the location of every piece of information memorized, so any attempt to update the documentation will invariably omit a few places. This second concern is not at all theoretical; the documentation used to be plagued with inconsistent information.

If the documentation were targeted for a specific version – say, LilyPond 2.10.5 – and we had unlimited resources to spend on documentation, then we could avoid this second problem. But since LilyPond evolves (and that is a very good thing!), and since we have quite limited resources, this policy remains in place.

A few other policies (such as not permitting the use of tweaks in the main portion of NR 1+2) may also seem counter-intuitive, but they also stem from attempting to find the most effective use of limited documentation help.

Before undertaking any large documentation work, contributors are encouraged to contact the Documentation Meister.

5.2 \version in documentation files

Every documentation file which includes LilyPond code must begin with a \version statement, since the build procedure explicitly tests for its presence and will not continue otherwise. The \version statement should reference a version of LilyPond consistent with the syntax of the contained code.

Since the \version statement is not valid Texinfo input it must be commented out like this:

@c \version "2.19.1"

So, if you are adding LilyPond code which is not consistent with the current version header, you should

1. run convert-ly on the file using the latest version of LilyPond (which should, if everybody has done proper maintenance, not change anything);
2. add the new code;
3. modify the version number to match the new code.

5.3 Documentation suggestions

Small additions

For additions to the documentation,

1. Tell us where the addition should be placed. Please include both the section number and title (i.e. "LM 2.13 Printing lyrics").
2. Please write exact changes to the text.
3. A formal patch to the source code is not required; we can take care of the technical details.
4. Send the suggestions to the bug-lilypond mailing list as discussed in (web-big-page)Contact section ‘Contact’ in General Information.
5. Here is an example of a perfect documentation report:

   To: bug-lilypond@gnu.org
   From: helpful-user@example.net
   Subject: doc addition
   
   In LM 2.13 (printing lyrics), above the last line ("More options,
   like..."), please add:
   
   ----
   To add lyrics to a divided part, use blah blah blah.  For example,
   
   \score {
     \notes {blah <<blah>> }
     \lyrics {blah <<blah>> }
     blah blah blah
   }
   ----
   
   In addition, the second sentence of the first paragraph is
   confusing.  Please delete that sentence (it begins "Users
   often...") and replace it with this:
   ----
   To align lyrics with something, do this thing.
   ----
   
   Have a nice day,
   Helpful User
   

Larger contributions

To replace large sections of the documentation, the guidelines are stricter. We cannot remove parts of the current documentation unless we are certain that the new version is an improvement.

1. Ask on the lilypond-devel mailing list if such a rewrite is necessary; somebody else might already be working on this issue!
2. Split your work into small sections; this makes it much easier to compare the new and old documentation.
3. Please prepare a formal git patch.

Contributions that contain examples using overrides

Examples that use overrides, tweaks, customer Scheme functions etc. are (with very few exceptions) not included in the main text of the manuals; as there would be far too many, equally useful, candidates.

The correct way is to submit your example, with appropriate explanatory text and tags, to the LilyPond Snippet Repository (LSR). Snippets that have the “docs” tag can then be easily added as a selected snippet in the documentation. It will also appear automatically in the Snippets lists. See Introduction to LSR.

Snippets that don’t have the “docs” tag will still be searchable and viewable within the LSR, but will be not be included in the Snippets list or be able to be included as part of the main documentation.

Generally, any new snippets that have the “docs” tag are more carefully checked for syntax and formatting.

Announcing your snippet

Once you have followed these guidelines, please send a message to lilypond-devel with your documentation submissions. Unfortunately there is a strict ‘no top-posting’ check on the mailing list; to avoid this, add:

> I'm not top posting

(you must include the > ) to the top of your documentation addition.

We may edit your suggestion for spelling, grammar, or style, and we may not place the material exactly where you suggested, but if you give us some material to work with, we can improve the manual much faster.

Thanks for your interest!

5.4 Texinfo introduction and usage policy

5.4.1 Texinfo introduction

The language is called Texinfo; you can see its manual here:

http://www.gnu.org/software/texinfo/manual/texinfo/

However, you don’t need to read those docs. The most important thing to notice is that text is text. If you see a mistake in the text, you can fix it. If you want to change the order of something, you can cut-and-paste that stuff into a new location.

5.4.2 Documentation files

All manuals live in ‘Documentation/’.

In particular, there are four user manuals, their respective master source files are ‘learning.tely’ (LM, Learning Manual), ‘notation.tely’ (NR, Notation Reference), ‘music-glossary.tely’ (MG, Music Glossary), and ‘lilypond-program’ (AU). Each chapter is written in a separate file, ending in ‘.itely’ for files containing lilypond code, and ‘.itexi’ for files without lilypond code, located in a subdirectory associated to the manual (‘learning/’ for ‘learning.tely’, and so on); list the subdirectory of each manual to determine the filename of the specific chapter you wish to modify.

Developer manuals live in ‘Documentation/’ too. Currently there is only one: the Contributor’s Guide ‘contrib-guide.texi’ you are reading.

Snippet files are part of documentation, and the Snippet List (SL) lives in ‘Documentation/’ just like the manuals. For information about how to modify the snippet files and SL, see LSR work.

5.4.3 Sectioning commands

The Notation Reference uses section headings at four, occasionally five, levels.

• Level 1: @chapter
• Level 2: @section
• Level 3: @subsection
• Level 4: @unnumberedsubsubsec
• Level 5: @subsubsubheading

The first three levels are numbered in HTML, the last two are not. Numbered sections correspond to a single HTML page in the split HTML documents.

The first four levels always have accompanying nodes so they can be referenced and are also included in the ToC in HTML.

Most of the manual is written at level 4 under headings created with

@node Foo
@unnumberedsubsubsec Foo

Level 3 subsections are created with

@node Foo
@subsection Foo

Level 4 headings and menus must be preceded by level 3 headings and menus, and so on for level 3 and level 2. If this is not what is wanted, please use:

@subsubsubheading Foo

Please leave two blank lines above a @node; this makes it easier to find sections in texinfo.

Do not use any @ commands for a @node. They may be used for any @sub... sections or headings however.

not:
@node @code{Foo} Bar
@subsection @code{Foo} Bar

but instead:
@node Foo Bar
@subsection @code{Foo} Bar

No punctuation may be used in the node names. If the heading text uses punctuation (in particular, colons and commas) simply leave this out of the node name and menu.

@menu
* Foo Bar::
@end menu

@node Foo Bar
@subsection Foo: Bar

Backslashes must not be used in node names or section headings. If the heading text should include a backslash simply leave this out of the node name and menu and replace it with @bs{} in the heading text.

@menu
* The set command
@end menu

@node The set command
@subsection The @code{@bs{}set} command

References to such a node may use the third argument of the @ref command to display the texually correct heading.

@ref{The set command,,The @code{@bs{}set command}

With the exception of @ commands, \ commands and punctuation, the section name should match the node name exactly.

Sectioning commands (@node and @section) must not appear inside an @ignore. Separate those commands with a space, ie @n ode.

Nodes must be included inside a

@menu
* foo::
* bar::
@end menu

construct. These can be constructed with scripts: see Stripping whitespace and generating menus.

5.4.4 LilyPond formatting

• Most LilyPond examples throughout the documentation can be produced with:

  @lilypond[verbatim,quote]

  If using \book{} in your example then you must also include the papersize=X variable, where X is a defined paper size from within ‘scm/paper.scm’. This is to avoid the default a4 paper size being used and leaving too much unnecessary whitespace and potentially awkward page breaks in the PDFs.

  The preferred papersizes are a5, a6 or a8landscape.

  a8landscape works best for a single measure with a single title and/or single tagline:

  @lilypond[papersize=a8landscape,verbatim] \book { \header { title = "A scale in LilyPond" } \relative { c d e f } } @end lilypond

  and can also be used to easily show features that require page breaks (i.e. page numbers) without taking large amounts of space within the documentation. Do not use the quote option with this paper size.

  a5 or a6 paper sizes are best used for examples that have more than two measures of music or require multiple staves (i.e. to illustrate cross-staff features, RH and LH parts etc.) and where \book{} constructions are required or where a8landscape produces an example that is too cramped. Depending on the example the quote option may need to be omitted.

  In rare cases, other options may be used (or omitted), but ask first.

• Please avoid using extra spacing either after or within the @lilypond parameters.

  not: @lilypond [verbatim, quote, fragment] but instead: @lilypond[verbatim,quote,fragment]

• Inspirational headwords are produced with:

  @lilypondfile[quote,ragged-right,line-width=16\cm,staffsize=16] {pitches-headword.ly}

• LSR snippets are linked with:

  @lilypondfile[verbatim,quote,ragged-right,texidoc,doctitle] {filename.ly}

• Use two spaces for indentation in lilypond examples (no tabs).
• All engravers should have double-quotes around them:

  \consists "Spans_arpeggio_engraver"

  LilyPond does not strictly require this, but it is a useful convention to follow.

• Try to avoid using #' or #` when describing context or layout properties outside of an @example or @lilypond, unless the description explicitly requires it.

  i.e. “...setting the transparent property leaves the object where it is, but makes it invisible.”

• If possible, only write one bar per line.
• If you only have one bar per line, omit bar checks. If you must put more than one bar per line (not recommended), then include bar checks.
• Tweaks should, if possible, also occur on their own line.

  not: \override TextScript.padding = #3 c1^"hi" but instead: \override TextScript.padding = #3 c1^"hi"

  excepted in Templates, where ‘doctitle’ may be omitted.

• Avoid long stretches of input code. Nobody is going to read them in print. Create small examples. However, this does not mean it has be minimal.
• Specify durations for at least the first note of every bar.
• If possible, end with a complete bar.
• Comments should go on their own line, and be placed before the line(s) to which they refer.
• For clarity, always use { } marks even if they are not technically required; i.e.

  not: \context Voice \repeat unfold 2 \relative c' { c2 d } but instead: \context Voice { \repeat unfold 2 { \relative c' { c2 d } } }

• Add a space around { } marks; i.e.

  not: \chordmode{c e g} but instead: \chordmode { c e g }

• Use { } marks for additional \markup format commands; i.e.

  not: c^\markup \tiny\sharp but instead: c^\markup { \tiny \sharp }

• Remove any space around < > marks; i.e.

  not: < c e g > 4 but instead: <c e g>4

• Beam, slur and tie marks should begin immediately after the first note with beam and phrase marks ending immediately after the last.

  a8\( ais16[ b cis( d] b) cis4~ b' cis,\)

• If you want to work on an example outside of the manual (for easier/faster processing), use this header:

  \paper { indent = 0\mm line-width = 160\mm - 2.0 * 0.4\in line-width = #(- line-width (* mm 3.000000)) } \layout { }

  You may not change any of these values. If you are making an example demonstrating special \paper{} values, contact the Documentation Editor.

5.4.5 Text formatting

• Lines should be less than 72 characters long. (We personally recommend writing with 66-char lines, but do not bother modifying existing material). Also see the recommendations for fixed-width fonts in the Syntax survey.
• Do not use tabs.
• Do not use spaces at the beginning of a line (except in @example or @verbatim environments), and do not use more than a single space between words. ‘makeinfo’ copies the input lines verbatim without removing those spaces.
• Use two spaces after a period.
• In examples of syntax, use @var{musicexpr} for a music expression.
• Don’t use @rinternals{} in the main text. If you’re tempted to do so, you’re probably getting too close to “talking through the code”. If you really want to refer to a context, use @code{} in the main text and @rinternals{} in the @morerefs.

5.4.6 Syntax survey

Comments

• @c … — single line comment. ‘@c NOTE:’ is a comment which should remain in the final version. (gp only command ;)
• @ignore — multi-line comment:

  @ignore … @end ignore

Cross references

Enter the exact @node name of the target reference between the brackets (eg. ‘@ref{Syntax survey}’). Do not split a cross-reference across two lines – this causes the cross-reference to be rendered incorrectly in HTML documents.

• @ref{…} — link within current manual.
• @rchanges{…} — link to Changes.
• @rcontrib{…} — link to Contributor’s Guide.
• @ressay{…} — link to Engraving Essay.
• @rextend{…} — link to Extending LilyPond.
• @rglos{…} — link to the Music Glossary.
• @rinternals{…} — link to the Internals Reference.
• @rlearning{…} — link to Learning Manual.
• @rlsr{…} — link to a Snippet section.
• @rprogram{…} — link to Application Usage.
• @ruser{…} — link to Notation Reference.
• @rweb{…} — link to General Information.

External links

• @email{…} — create a mailto: E-mail link.
• @uref{URL[, link text]} — link to an external url. Use within an @example ... @end example.

  @example @uref{URL [, link text ]} @end example

Fixed-width font

• @code{…}, @samp{…} —

  Use the @code{…} command when referring to individual language-specific tokens (keywords, commands, engravers, scheme symbols, etc.) in the text. Ideally, a single @code{…} block should fit within one line in the PDF output.

  Use the @samp{…} command when you have a short example of user input, unless it constitutes an entire @item by itself, in which case @code{…} is preferable. Otherwise, both should only be used when part of a larger sentence within a paragraph or @item. Do not use @code{…} or @samp{…} inside an @example block, and do not use either as a free-standing paragraph; use @example instead.

  A single unindented line in the PDF has space for about 79 fixed-width characters (76 if indented). Within an @item there is space for about 75 fixed-width characters. Each additional level of @itemize or @enumerate shortens the line by about 4 columns.

  However, even short blocks of @code{…} and @samp{…} can run into the margin if the Texinfo line-breaking algorithm gets confused. Additionally, blocks that are longer than this may in fact print nicely; it all depends where the line breaks end up. If you compile the docs yourself, check the PDF output to make sure the line breaks are satisfactory.

  The Texinfo setting @allowcodebreaks is set to false in the manuals, so lines within @code{…} or @samp{…} blocks will only break at spaces, not at hyphens or underscores. If the block contains spaces, use @w{@code{…}} or @w{@samp{…}} to prevent unexpected line breaks.

  The Texinfo settings txicodequoteundirected and txicodequotebacktick are both set in the manuals, so backticks (`) and apostrophes (') placed within blocks of @code, @example, or @verbatim are not converted to left- and right-angled quotes (‘ ’) as they normally are within the text, so the apostrophes in ‘@w{@code{\relative c''}}’ will display correctly. However, these settings do not affect the PDF output for anything within a @samp block (even if it includes a nested @code block), so entering ‘@w{@samp{\relative c''}}’ wrongly produces ‘\relative c’’’ in PDF. Consequently, if you want to use a @samp{…} block which contains backticks or apostrophes, you should instead use ‘@q{@code{…}}’ (or ‘@q{@w{@code{…}}}’ if the block also contains spaces). Note that backslashes within @q{…} blocks must be entered as ‘@bs{}’, so the example above would be coded as ‘@q{@w{@code{@bs{}relative c''}}}’.

• @command{…} — Use when referring to command-line commands within the text (eg. ‘@command{convert-ly}’). Do not use inside an @example block.
• @example — Use for examples of program code. Do not add extraneous indentation (i.e. don’t start every line with whitespace). Use the following layout (notice the use of blank lines). Omit the @noindent if the text following the example starts a new paragraph:

  …text leading into the example… @example … @end example @noindent continuation of the text…

  Individual lines within an @example block should not exceed 74 characters; otherwise they will run into the margin in the PDF output, and may get clipped. If an @example block is part of an @item, individual lines in the @example block should not exceed 70 columns. Each additional level of @itemize or @enumerate shortens the line by about 4 columns.

  For long command line examples, if possible, use a trailing backslash to break up a single line, indenting the next line with 2 spaces. If this isn’t feasible, use ‘@smallexample … @end smallexample’ instead, which uses a smaller fontsize. Use @example whenever possible, but if needed, @smallexample can fit up to 90 characters per line before running into the PDF margin. Each additional level of @itemize or @enumerate shortens a @smallexample line by about 5 columns.

• @file{…} — Use when referring to filenames and directories in the text. Do not use inside an @example block.
• @option{…} — Use when referring to command-line options in the text (eg. ‘@option{--format}’). Do not use inside an @example block.
• @verbatim — Prints the block exactly as it appears in the source file (including whitespace, etc.). For program code examples, use @example instead. @verbatim uses the same format as @example.

  Individual lines within an @verbatim block should not exceed 74 characters; otherwise they will run into the margin in the PDF output, and may get clipped. If an @verbatim block is part of an @item, individual lines in the @verbatim block should not exceed 70 columns. Each additional level of @itemize or @enumerate shortens the line by about 4 columns.

Indexing

• @cindex … — General index. Please add as many as you can. Don’t capitalize the first word.
• @funindex … — is for a \lilycommand.

Lists

• @enumerate — Create an ordered list (with numbers). Always put ‘@item’ on its own line. As an exception, if all the items in the list are short enough to fit on single lines, placing them on the ‘@item’ lines is also permissible. ‘@item’ and ‘@end enumerate’ should always be preceded by a blank line.

  @enumerate @item A long multi-line item like this one must begin on a line of its own and all the other items in the list must do so too. @item Even short ones @end enumerate

  @enumerate @item Short item @item Short item @end enumerate

• @itemize — Create an unordered list (with bullets). Use the same format as @enumerate. Do not use ‘@itemize @bullet’.

Special characters

(define (foo x)
  "The @code{\\foo} command..."
  ...)

• --, --- — Create an en dash (–) or an em dash (—) in the text. To print two or three literal hyphens in a row, wrap one of them in a @w{…} (eg. ‘-@w{-}-’).
• @@, @{, @} — Create an at-sign (@), a left curly bracket ({), or a right curly bracket (}).
• @bs{} — Create a backslash within a @q{…}, @qq{…}, or @warning{…} block. This is a custom LilyPond macro, not a builtin @-command in Texinfo. Texinfo would also allow ‘\\’, but this breaks the PDF output.
• @tie{} — Create a variable-width non-breaking space in the text (use ‘@w{ }’ for a single fixed-width non-breaking space). Variables or numbers which consist of a single character (probably followed by a punctuation mark) should be tied properly, either to the previous or the next word. Example: ‘The letter@tie{}@q{I} is skipped’

Miscellany

• @notation{…} — refers to pieces of notation, e.g. ‘@notation{clef}’. Also use for specific lyrics (‘the @notation{A - men} is centered’). Only use once per subsection per term.
• @q{…} — Single quotes. Used for ‘vague’ terms. To get a backslash (\), you must use ‘@bs{}’.
• @qq{…} — Double quotes. Used for actual quotes (“he said”) or for introducing special input modes. To get a backslash (\), you must use ‘@bs{}’.
• @var{…} — Use for metasyntactic variables (such as foo, bar, arg1, etc.). In most cases, when the @var{…} command appears in the text (and not in an @example block) it should be wrapped with an appropriate texinfo code-highlighting command (such as @code, @samp, @file, @command, etc.). For example: ‘@code{@var{foo}}’, ‘@file{@var{myfile.ly}}’, ‘@samp{git checkout @var{branch}}’, etc. This improves readability in the PDF and HTML output.
• @version{} — Return the current LilyPond version string. Use ‘@w{@version{}}’ if it’s at the end of a line (to prevent an ugly line break in PDF); use ‘@w{"@version{}"}’ if you need it in quotes.
• @w{…} — Do not allow any line breaks.
• @warning{…} — produces a “Note: ” box. Use for important messages. To get a backslash (\), you must use ‘@bs{}’.

5.4.7 Other text concerns

• References must occur at the end of a sentence, for more information see the texinfo manual. Ideally this should also be the final sentence of a paragraph, but this is not required. Any link in a doc section must be duplicated in the @morerefs section at the bottom.
• Introducing examples must be done with

  . (i.e. finish the previous sentence/paragraph) : (i.e. `in this example:') , (i.e. `may add foo with the blah construct,')

  The old “sentence runs directly into the example” method is not allowed any more.

• Abbrevs in caps, e.g., HTML, DVI, MIDI, etc.
• Colon usage
  1. To introduce lists
  2. When beginning a quote: “So, he said,...”.

     This usage is rarer. Americans often just use a comma.

  3. When adding a defining example at the end of a sentence.
• Non-ASCII characters which are in utf-8 should be directly used; this is, don’t say ‘Ba@ss{}tuba’ but ‘Baßtuba’. This ensures that all such characters appear in all output formats.

5.5 Documentation policy

5.5.1 Books

There are four parts to the documentation: the Learning Manual, the Notation Reference, the Program Reference, and the Music Glossary.

• Learning Manual:

  The LM is written in a tutorial style which introduces the most important concepts, structure and syntax of the elements of a LilyPond score in a carefully graded sequence of steps. Explanations of all musical concepts used in the Manual can be found in the Music Glossary, and readers are assumed to have no prior knowledge of LilyPond. The objective is to take readers to a level where the Notation Reference can be understood and employed to both adapt the templates in the Appendix to their needs and to begin to construct their own scores. Commonly used tweaks are introduced and explained. Examples are provided throughout which, while being focussed on the topic being introduced, are long enough to seem real in order to retain the readers’ interest. Each example builds on the previous material, and comments are used liberally. Every new aspect is thoroughly explained before it is used.

  Users are encouraged to read the complete Learning Manual from start-to-finish.

• Notation Reference: a (hopefully complete) description of LilyPond input notation. Some material from here may be duplicated in the Learning Manual (for teaching), but consider the NR to be the "definitive" description of each notation element, with the LM being an "extra". The goal is _not_ to provide a step-by-step learning environment – do not avoid using notation that has not be introduced previously in the NR (for example, use \break if appropriate). This section is written in formal technical writing style.

  Avoid duplication. Although users are not expected to read this manual from start to finish, they should be familiar with the material in the Learning Manual (particularly “Fundamental Concepts”), so do not repeat that material in each section of this book. Also watch out for common constructs, like ^ - _ for directions – those are explained in NR 3. In NR 1, you can write: DYNAMICS may be manually placed above or below the staff, see @ref{Controlling direction and placement}.

  Most tweaks should be added to LSR and not placed directly in the ‘.itely’ file. In some cases, tweaks may be placed in the main text, but ask about this first.

  Finally, you should assume that users know what the notation means; explaining musical concepts happens in the Music Glossary.

• Application Usage: information about using the program lilypond with other programs (lilypond-book, operating systems, GUIs, convert-ly, etc). This section is written in formal technical writing style.

  Users are not expected to read this manual from start to finish.

• Music Glossary: information about the music notation itself. Explanations and translations about notation terms go here.

  Users are not expected to read this manual from start to finish.

• Internals Reference: not really a documentation book, since it is automagically generated from the source, but this is its name.