[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

12.4 Doc build

12.4.1 The function of make doc  
12.4.2 Building a bibliography  
[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

12.4.1 The function of make doc

The following is a set of notes on how make doc functions.

Preliminary question to be answered some time: where do all the GNUmakefiles come from. They’re in the build directory, but this is not part of source. Must be the configure script. And it looks like this comes from autogen.sh. Must at some point kill the whole git directory, repull and see what is created when.

Anyway, here’s how make doc progresses:

This is the build dependency tree from ‘stepmake/stepmake/generic-targets.make’:

 
doc: doc-stage-1
  doc-stage-1:
        $(MAKE) -C $(depth)/scripts/build out=
        $(MAKE) out=www WWW-1
          WWW-1: local-WWW-1
        $(LOOP)
          $(MAKE) out=www WWW-2
          WWW-2: local-WWW-2
            $(LOOP)
          $(MAKE) out=www WWW-post
 
MAKE = make
-C = Change to directory before make

doc-stage-1 does lots of opening and looking in files, but no processing.

 
Variable LOOP =

+ make -C python
&& make -C scripts
&&  make -C flower
&&  make -C lily
&&  make -C mf
&&  make -C ly
&&  make -C tex
&&  make -C ps
&&  make -C scm
&&  make -C po
&&  make -C make
&&  make -C elisp
&&  make -C vim
&&  make -C input
&&  make -C stepmake
&&  make -C Documentation
&& true

From git grep:

stepmake/stepmake/generic-vars.make has this:

 
LOOP=+$(foreach i, $(SUBDIRS), $(MAKE) -C $(i) $@ &&) true

$@ is the name of the target - WWW-1 in this case.

In GNUmakefile.in we find:

 
SUBDIRS = python scripts \
        flower lily \
        mf ly \
        tex ps scm \
        po make \
        elisp vim \
        input \
        stepmake $(documentation-dir)

So that’s how we get the main make loop...

That loop expands like this:

 
make -C python WWW-1 &&
make -C scripts WWW-1 &&
make -C flower WWW-1 &&
make -C lily WWW-1 &&
make -C mf WWW-1 &&
make -C ly WWW-1 &&
make -C tex WWW-1 &&
make -C ps WWW-1 &&
make -C scm WWW-1 &&
make -C po WWW-1 &&
make -C make WWW-1 &&
make -C elisp WWW-1 &&
make -C vim WWW-1 &&
make -C input WWW-1 &&
make -C stepmake WWW-1 &&
make -C Documentation WWW-1 &&
true

The directories up to and including vim produce no effect with make in non-debug mode, although debug does show lots of action.

‘git/build/input/GNUmakefile’ is:

 
depth=../
include $(depth)/config.make
include $(configure-srcdir)/./input/GNUmakefile
MODULE_INCLUDES += $(src-dir)/$(outbase)

The first include is:

 
..//config.make

(note the // which is strictly wrong)

which has lots of variables to set, but no action occurs.

The second is:

 
lilypond-git/./input/GNUmakefile

which similarly doesn’t create any actual action.

An error message at the end of build/input/GNUmakefile stops make processing before it moves on to regression - so where does that come from?

And the answer is - make processes all directories in the directory it’s entered (with some exceptions like out and out-www) and so it changes to /regression.

It then seems to consider whether it needs to make/remake loads of makefiles. Don’t understand this yet. Possibly these are all the makefiles it’s processing, and it always checks they’re up to date before processing other files?

Could be correct - some of this output is:

 
Must remake target `../../make/ly-inclusions.make'.
Failed to remake target file `../../make/ly-inclusions.make'.

Having decided that, it then leaves the directory and re-executes:

 
make -C regression WWW-1

The top of this make is:

 
This program built for i486-pc-linux-gnu
Reading makefiles...
Reading makefile `GNUmakefile'...
Reading makefile `../..//config.make' (search path) (no ~ expansion)...

which looks like it’s re-reading all its known makefiles to check they’re up to date.

(From the make manual:

To this end, after reading in all makefiles, make will consider each as a goal target and attempt to update it. If a makefile has a rule which says how to update it (found either in that very makefile or in another one) or if an implicit rule applies to it (see Chapter 10 [Using Implicit Rules], page 103), it will be updated if necessary. After all makefiles have been checked, if any have actually been changed, make starts with a clean slate and reads all the makefiles over again. (It will also attempt to update each of them over again, but normally this will not change them again, since they are already up to date.)

So my assumption seems correct)

There appear to be about 74 of them. After all the makefile checking, we get this:

 
Updating goal targets....
Considering target file `WWW-1'.
File `WWW-1' does not exist.
Considering target file `local-WWW-1'.
File `local-WWW-1' does not exist.
Considering target file `out-www/collated-files.texi'.
File `out-www/collated-files.texi' does not exist.
Looking for an implicit rule for `out-www/collated-files.texi'.
Trying pattern rule with stem `collated-files.texi'.
Trying implicit prerequisite `collated-files.texi.in'.
Trying pattern rule with stem `collated-files.texi'.
Trying implicit prerequisite `collated-files.texi.in'.
Trying pattern rule with stem `collated-files'.
Trying implicit prerequisite `collated-files.tely'.
Trying pattern rule with stem `collated-files'.
Trying implicit prerequisite `out-www/collated-files.tely'.
Trying rule prerequisite `out-www/version.itexi'.
Found prerequisite `out-www/version.itexi' as VPATH `/home/phil/lilypond-git/input/regression/out-www/version.itexi'

grep finds this if searching for local-WWW-1:

 
make/lysdoc-targets.make:
  local-WWW-1: $(outdir)/collated-files.texi $(outdir)/collated-files.pdf

which means that local-WWW-1 depends on coll*.texi and coll*.pdf and so these will need to be checked to see if they’re up to date. So make needs to find rules for both of those and (as it says) it certainly needs to make coll*.texi, since it doesn’t exist.

In ly-rules.make we have:

 
.SUFFIXES: .doc .tely .texi .ly

which I’ll work out at some point, and also this rule:

 
$(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
  LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<

Note that the recipe is a very long line - it could probably benefit from splitting. The same makefile also has:

 
$(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
  LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<

which seems to be an almost exact duplicate. Whatever, the first one is executed first. Have not checked if the second executes.

The first recipe translates as this:

 
LILYPOND_VERSION=2.15.0 /usr/bin/python   --process=' ' \
  --output=./out-www --format= --lily-output-dir \
    /home/phil/lilypond-git/build/out/lybook-db

if we stop the build with an $(error), but I think this is because we need to allow it to process the dependencies first. It looks like foo.texi is shown as being dependent on foo.tely, plus a load of other files.

 
DOCUMENTATION_LOCALE_TARGET is blank
INIT_LY_SOURCES = /home/phil/lilypond-git/scm/auto-beam.scm \
  /home/phil/lilypond-git/scm/autochange.scm

plus 10s (100s?) of other .scm files.

 
SCHEME_SOURCES = /home/phil/lilypond-git/ly/Welcome-to-LilyPond-MacOS.ly \
  /home/phil/lilypond-git/ly/Welcome_to_LilyPond.ly

ditto .ly files. This does seem a teency bit wrong - it looks like the .ly and .scm files have been interchanged. ly-vars.make has these 2 lines:

 
INIT_LY_SOURCES = $(wildcard $(top-src-dir)/scm/*.scm)
SCHEME_SOURCES = $(wildcard $(top-src-dir)/ly/*.ly)

Looks like a bug.....

So it now works its way through all these files, checking if they need to be remade. This is 100s of lines of the debug listing, although none in the normal list. Clearly none has to be made since they’re source files. It concludes:

 
Must remake target `out-www/collated-files.tely'

‘lysdoc-rules.make’ has this:

 
$(outdir)/collated-files.tely: $(COLLATED_FILES)
        $(LYS_TO_TELY) --name=$(outdir)/collated-files.tely --title="$(TITLE)" --author="$(AUTHOR)" $^

‘lysdoc-vars.make’ has:

 
COLLATED_FILES = $(sort $(TEXINFO_SOURCES) $(LY_FILES) $(OUT_LY_FILES) )

We find that:

 
TEXINFO_SOURCES = AAA-intro-regression.tely
OUT_LY_FILES is empty

so LY_FILES has the big long list of all the .ly files in the regression directory.

This kicks off

 
/home/phil/lilypond-git/build/scripts/build/out/lys-to-tely

with a list of all the files in the regression test directory. This should (I believe) create the file collated-files.tely.

So the next rule in make is for ‘version.itexi’, and make duly checks this. There’s a rule in ‘doc-i18n-root-rules.make’ that this depends on ‘git/VERSION’:

 
$(outdir)/version.%: $(top-src-dir)/VERSION
        $(PYTHON) $(top-src-dir)/scripts/build/create-version-itexi.py > $ 

This causes create-version-itexi.py to run and create version.itexi.

Once that’s done, all the other *.scm and *.ly files are checked and since they have no rules associated, they aren’t remade (just as well for source files, really). Since version.itexi was remade make concludes that collated-files.texi must be remade. To do this, it runs lilypond-book.py on collated-files.tely, as below:

 
LILYPOND_VERSION=2.15.0
  /usr/bin/python
  /home/phil/lilypond-git/scripts/lilypond-book.py
    -I /home/phil/lilypond-git/input/regression/
    -I ./out-www -I /home/phil/lilypond-git/input
    -I /home/phil/lilypond-git/Documentation
    -I /home/phil/lilypond-git/Documentation/snippets
    -I /home/phil/lilypond-git/input/regression/
    -I /home/phil/lilypond-git/Documentation/included/
    -I /home/phil/lilypond-git/build/mf/out/
    -I /home/phil/lilypond-git/build/mf/out/
    -I /home/phil/lilypond-git/Documentation/pictures
    -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
    --process='/home/phil/lilypond-git/build/out/bin/lilypond
    -I /home/phil/lilypond-git/input/regression/
    -I ./out-www
    -I /home/phil/lilypond-git/input
    -I /home/phil/lilypond-git/Documentation
    -I /home/phil/lilypond-git/Documentation/snippets
    -I /home/phil/lilypond-git/input/regression/
    -I /home/phil/lilypond-git/Documentation/included/
    -I /home/phil/lilypond-git/build/mf/out/
    -I /home/phil/lilypond-git/build/mf/out/
    -I /home/phil/lilypond-git/Documentation/pictures
    -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
    -dbackend=eps
    --formats=ps,png,pdf
    -dinclude-eps-fonts
    -dgs-load-fonts
    --header=doctitle
    --header=doctitlecs
    --header=doctitlede
    --header=doctitlees
    --header=doctitlefr
    --header=doctitlehu
    --header=doctitleit
    --header=doctitleja
    --header=doctitlenl
    --header=doctitlezh
    --header=texidoc
    --header=texidoccs
    --header=texidocde
    --header=texidoces
    --header=texidocfr
    --header=texidochu
    --header=texidocit
    --header=texidocja
    --header=texidocnl
    --header=texidoczh
    -dcheck-internal-types
    -ddump-signatures
    -danti-alias-factor=2'
    --output=./out-www
    --format=texi-html
    --verbose
    --lily-output-dir /home/phil/lilypond-git/build/out/lybook-db
    out-www/collated-files.tely

So - lilypond-book runs on:

 
input/regression/out-www/collated-files.tely

Note the –verbose flag - this is from the make variable LILYPOND_BOOK_VERBOSE which is added to the make variable LILYPOND_BOOK_FLAGS.

Now found the invocation to write some of the image files. It’s like this:

 
/home/phil/lilypond-git/build/out/bin/lilypond
  -I /home/phil/lilypond-git/input/regression/
  -I ./out-www -I /home/phil/lilypond-git/input
  -I /home/phil/lilypond-git/Documentation
  -I /home/phil/lilypond-git/Documentation/snippets
  -I /home/phil/lilypond-git/input/regression/
  -I /home/phil/lilypond-git/Documentation/included/
  -I /home/phil/lilypond-git/build/mf/out/
  -I /home/phil/lilypond-git/build/mf/out/
  -I /home/phil/lilypond-git/Documentation/pictures
  -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
  -dbackend=eps
  --formats=ps,png,pdf
  -dinclude-eps-fonts
  -dgs-load-fonts
  --header=doctitle
  --header=doctitlecs
  --header=doctitlede
  --header=doctitlees
  --header=doctitlefr
  --header=doctitlehu
  --header=doctitleit
  --header=doctitleja
  --header=doctitlenl
  --header=doctitlezh
  --header=texidoc
  --header=texidoccs
  --header=texidocde
  --header=texidoces
  --header=texidocfr
  --header=texidochu
  --header=texidocit
  --header=texidocja
  --header=texidocnl
  --header=texidoczh
  -dcheck-internal-types
  -ddump-signatures
  -danti-alias-factor=2
  -I  "/home/phil/lilypond-git/build/out/lybook-db"
  -I  "/home/phil/lilypond-git/build/input/regression"
  -I  "/home/phil/lilypond-git/input/regression"
  -I  "/home/phil/lilypond-git/build/input/regression/out-www"
  -I  "/home/phil/lilypond-git/input"
  -I  "/home/phil/lilypond-git/Documentation"
  -I  "/home/phil/lilypond-git/Documentation/snippets"
  -I  "/home/phil/lilypond-git/input/regression"
  -I  "/home/phil/lilypond-git/Documentation/included"
  -I  "/home/phil/lilypond-git/build/mf/out"
  -I  "/home/phil/lilypond-git/build/mf/out"
  -I  "/home/phil/lilypond-git/Documentation/pictures"
  -I  "/home/phil/lilypond-git/build/Documentation/pictures/out-www"
  --formats=eps
  --verbose
  -deps-box-padding=3.000000
  -dread-file-list
  -dno-strip-output-dir
  "/home/phil/lilypond-git/build/out/lybook-db/snippet-names--415419468.ly"'

Note the –verbose. This causes 100s of lines of Lily debug output. But at present I can’t work out where the flag comes from. Later.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

12.4.2 Building a bibliography

Bibliography files contain a list of citations, like this:

 
@Book{vinci,
  author = {Vinci, Albert C.},
  title = {Fundamentals of Traditional Music Notation},
  publisher = {Kent State University Press},
  year = {1989}
}

There are a variety of types of citation (e.g. Book (as above), article, publication). Each cited publication has a list of entries that can be used to identify the publication. Bibliograpies are normally stored as files with a .bib extension. One part of the doc-build process is transforming the bibliography information into texinfo files. The commands to do this are in the ‘GNUmakefile’ in the ‘Documentation’ directory.

A typical line of the makefile to translate a single bibliography is:

 
$(outdir)/colorado.itexi:
        BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
                -s $(top-src-dir)/Documentation/lily-bib \
                -o $(outdir)/colorado.itexi \
                $(src-dir)/essay/colorado.bib

Line by line:

 
$(outdir)/colorado.itexi:

We’re making the file ‘colorado.itexi’ and so this is the make instruction.

 
        BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \

It’s in the ‘essay’ directory and we want to run the bib2texi.py script against it.

 
                -s $(top-src-dir)/Documentation/lily-bib \

The style template is ‘lily-bib.bst’ and is found in the ‘Documentation’ directory.

 
                -o $(outdir)/colorado.itexi \

The output file in ‘colorado.itexi’.

 
                $(src-dir)/essay/colorado.bib

The input file is ‘colorado.bib’ in the ‘essay’ directory.

The bib2texi Python script used to be used with a variety of options, but now is always called using the same options, as above. Its job is to create the file containing the options for bibtex (the program that actually does the translation), run bibtex, and then clean up some temporary files. Its main "value add" is the creation of the options file, using this code:

 
open (tmpfile + '.aux', 'w').write (r'''
\relax
\citation{*}
\bibstyle{%(style)s}
\bibdata{%(files)s}''' % vars ())

The key items are the style file (now always lily-bib for us) and the input file.

The style file is written in its own specialised language, described to some extent at

 
http://amath.colorado.edu/documentation/LaTeX/reference/faq/bibtex.pdf

The file ‘lily-bib.bst’ also has fairly extensive commenting.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]

This document was generated by gub on July 13, 2020 using texi2html 1.82.