The rivendell Git server provides repositories for all of the *LFS projects (and some others). The repository which we are interested in for LFS editing is (unsurprisingly) the LFS repository. A complete list of the modules which are available can be found using the Trac source browser interface at https://wiki.linuxfromscratch.org/lfs/browser.

There are two types of Git access to the LFS tree. First, there is anonymous read-only access which anyone can use. Second, there is read-write access granted to active editors.

To get anonymous access, simply use the following command (note that this assumes you are using bash or a similar shell:

git clone https://git.linuxfromscratch.org/lfs.git lfs-git

This will copy the public repository to a subdirectory named .git in the directory lfs-git and then checkout the default branch to that directory. It will also set the local repository to track the default branch of the public repository so that you can update your local repository by simply running (after changing to directory lfs-git):

git pull

Note that for the LFS book, the default branch is named trunk.

For editors, access is slightly more complicated. You first need to generate an ssh key-pair. To generate the keys run:

ssh-keygen -t ed25519

When prompted where to save them, it's probably best to leave them in .ssh (as id_ed25519 and id_ed25519.pub). When prompted for a passphrase just press enter unless you want to give the phrase every time you synchronize to the server. However, since the same passphrase will be used when you log in to rivendell over ssh, it may be advisable to have some security in place.

Having generated your keys, send the ~/.ssh/id_ed25519.pub to an LFS administrator in order to have it added to ~git/.ssh/authorized_keys on rivendell. If you will be obtaining a login account on the server, the administrator will use the same key to allow you ssh access.

Your local copy of id_ed25519 and id_ed25519.pub should remain untouched by this process.

Once this process is set up, try to checkout the latest LFS book revision by running (from your local machine):

git clone [email protected]:lfs.git lfsbook

If all goes well you will download a copy of the current repository to lfsbook/.git and you will check out the default branch, which is named trunk. You will also have write access so from now on be extra careful. Note that no changes will be made until you issue a git push command.

The above is fine for getting the default branch, but you may want to work on a different public branch. To set up a local branch that tracks a public branch named new_branch, just issue:

git checkout --track origin/<new_branch>

If you need to work on the default branch named trunk, just commit your changes (if any) to the new branch (the commit is local), and switch back to trunk with:

git checkout trunk

Since the new branch is already set up, if you want again to work on the new branch, just do:

git checkout <new_branch>

As with anonymous access, you can update your local repository by simply cd'ing into the LFS directory and running:

git pull

Let's get familiar with the basic set of commands which all editors will use on an almost daily basis. There are many more options available than the ones listed here, so you will want to read the Git documentation (Pro Git) at some point. git help also provides a useful quick reference for Git commands. Sometimes git status shows a couple of commands that an user might want to run in the current state of the repository.

git clone. This command is used to copy a public repository to the local machine and checkout the default (trunk for LFS) branch. You should only need to do this once. Various options allow creation of a “shallow” repository that does not contain the full history or all the branches. See git help clone.

Examples

To clone the LFS repository with only read access:

git clone https://git.linuxfromscratch.org/lfs.git lfsbook

To clone with read/write access:

git clone [email protected]:lfs.git lfsbook

git pull. This command synchronizes your local repository. If you have made local changes, Git will try to merge any changes on the server with the changes you have committed on your machine. If the changes on the server overlap with local uncommitted changes, the merge is cancelled, and the work tree will remain untouched.

Unlike svn up, when Git merges your local commits with the changes on the server, it will produce a merge commit. Too many merge commits may cause the history to be confusing. As an alternative, one can pass --rebase to git pull, telling Git to rebase your local commits onto the changes on the server. Read git rebase for the details about rebasing.

To make a clean history, editors should use rebasing instead of merging for synchronizing the changes on the server if you have some local commits. To make --rebase the default for git pull, issue: git config pull.rebase true.

You should always do a manual git pull before trying to push changes in order to ensure that there are no conflicts with changes that have been made since you started your work. Note that git push will warn you if there is a conflict and you forget to perform a git pull.

git add [modified-files]. This stages the modifications to all the modified-files (including newly created files) to the index. If you specify a directory in modified-files, all the files modified in this directory or its subdirectories are staged. The change is not in the local repository until you do a git commit. We will touch on this more as we continue in the chapter.

This command can be used in a very flexible way by using the -p option. You can select each hunk of the diff that is staged into the index. This allows testing a full change and then divides commits into small “atomic” pieces that are easier to review and understand.

git rm. This removes a file and lets Git know about it. Various options allow selecting whether the file is only deleted in the index or both in the index and the working directory. Note that you need to commit (and possibly push) for the removal to appear in the repository.

git mv. This renames or moves a file and lets Git know about it. Again, you need to commit (and possibly push) for the change to appear in the repository.

git status. This is probably the command you'll be using most often. It prints the files that have been modified in the working directory, and that are not in the index, or that are in the index and not committed yet. It also outputs the state of the local repository respective to the remote one.

A very handy feature is that it outputs a list of commands that can be used in the current context.

git commit. This command stores your changes to the local repository. Normally it commits changes staged in the index, but various options allow you to bypass staging. Note that nothing is changed on the server until you issue git push. The -m and -F options can be used to pass a log message to the command. If you don't specify a -m "MESSAGE" or -F "Filename" option, Git will open the default editor and ask you to type in a log message. The default editor is specified by the environment variables, GIT_EDITOR or VISUAL or EDITOR (checked in that order) or by a configuration parameter in ~/.git/config. For example, to set your default editor to vim, run:

export GIT_EDITOR=vim

or:

git config --global core.editor vim

Do not use empty log messages (see later in this document on the policy which governs them).

git log. Outputs the commit log of the branch currently in the working directory. Various options allow customizing the level of information given for each commit. Editors are strongly advised to run this command before pushing to the remote repository or merging. If the history is too convoluted, it may be necessary to run git rebase in order to simplify it. Your co-editors will hate you if you change the history after pushing it.

git push. This command synchronizes the public repository on the server with your local one. You should make sure you want to publish all the commits you have made. Sometimes it may be useful to run git rebase -i in order to simplify the history. Also, to prevent conflicts when pushing, it is better to first use git pull prior to pushing and resolve any conflicts locally.

git checkout. This command is used to make a branch of the local repository visible in the working directory.

git merge. This is useful to apply changes from another branch (or several branches) to the current one. By default the resulting changes are committed. This can be prevented with the --no-commit option, which gives you a chance to inspect and further tweak the merge result before committing.

In some cases, merging results in conflicts that have to be manually resolved. Note that edits should merge only to a clean branch, that is, after committing all changes.

git rebase This is useful to have the current branch start from another point in the history. For example if you have:

                              B1---B2---B3    my_branch
                             /
                            A---C1---C2       trunk

rebasing my_branch onto trunk will produce:

                              B'1---B'2---B'3 my_branch
                             /
                   A---C1---C2                trunk

where the state reached after commit B'3 is the same as it was at B3.

But the most useful command is git rebase -i that allows you to rewrite history when you have made messy commits.

Here is a summary list of things to do before committing changes:

You should normally restrict a commit to one set of changes, so if you are updating the versions of three packages you will usually do that in three separate commits. Clearly, what forms a "set" is a matter for your judgement. Always think that a set will have to be merged to (at least) the multilib branch. Pushing is not mandatory after each change if you plan to soon make another one. Once again, editors should run git pull before pushing in order to resolve any conflict locally.

The editors of the LFS book prefer that changes to the date in general.ent are made as part of a real change.

This may seem really obvious but it's very easy to make a typo in installation command changes which causes the installation to break. We've all done it, you'll probably do it too eventually. But double check to minimize the chance.

The following elements should be uncommented and updated in the general.ent file when a release (including -rc releases) is made, and the two lines about version.ent should be commented out:

<!ENTITY version         "10.2-rc1">
<!ENTITY versiond        "10.2-rc1">
<!ENTITY releasedate     "August 26th, 2021">
<!ENTITY copyrightdate   "1999-2021">

For development snapshots, these entities are automatically generated from the Git commit info. So general.ent does not need to be updated.

Changelog updates should always be provided with the exception of small typo fixes. You don't need to add "fixed small typo in XXX" to the changelog otherwise it will grow too much.

Changelog updates need to be in the following format:

<listitem>
  <para>Month Day, Year</para>
  <itemizedlist>
    <listitem>
      <para>[username] - What you changed.</para>
    </listitem>
<!-- if it only applies to the sysv version (adapt for systemd) -->
    <listitem revision="sysv">
      <para>[username] - What you changed only for sysv version.</para>
    </listitem>
    <listitem>
      <para>[username] - Previous changelog entry from the same day,
      by you or another editor.</para>
    </listitem>
  <itemizedlist>
</listitem>

Example:

<listitem>
  <para>March 3, 2006</para>
  <itemizedlist>
    <listitem>
      <para>[renodr] - Update to attr-2.5.1. Fixes
      <ulink url="&lfs-ticket-root;4833">#4833</ulink>.</para>
    </listitem>
  </itemizedlist>
</listitem>

Changelog entries are always on top of the previously added changelog entry.

If you are adding files, you need to run a git add command on each of them. When you remove files, you should always do that with git rm. Moving or renaming files is done with git mv. Adding a directory and all its children is done with git add <dirname>.

If you think you're ready to commit, run git status to see the state of the working copy. The normal process is to first git add the files that are modified to the index, then run git commit to store the set of changes into the local repository. The ouput of git status looks like this:

On branch trunk
Your branch is up to date with 'origin/trunk'.

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        modified:   README

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   TODO

where the "Changes to be committed" appear in green, while the "Changes not staged for commit" appear in red.

Sometimes, a change will make something no longer appropriate, but you think that it might be needed in the future. If you comment it out in the normal way, and start the comment with a FIXME: marker, that will be highlighted whenever the file is edited, at least if you use vim.

If you encounter a FIXME: in a comment, do not automatically assume that it can be deleted because the following instructions have now been commented out.

Note that there is no example here - the rendering of html strips out comments, and an empty box looks silly. As always, render your local copy after you comment something out, so that you can see the result!

Once you are sure that everything renders and that you know which files you wish to commit, you're ready. First commit locally:

Now push your work: git push. A transaction log of all the commits that make up the push will be emailed to the <[email protected]> mailinglist so other editors can see right away what you did. The commit emails contain some basic info (log, changes to which files) including a diff -u format output.

About log messages: Due to the way git log displays log messages, please keep the size of each line below 72 characters. A good commit message is composed of a short summary (not more than fifty characters), a blank line, then details, reasons, etc on less than 72 characters lines. Log messages should never be empty. Even if the message is just 'small typo fix', that'll do. Other usual messages are 'update to package-x.y.z' or 'fixed installation instructions of package foo'. That last one usually needs also explanation lines as said above.

Example: If you have modified general.ent, the changelog, and fixed a typo in chapter 6 bash installation, the workflow could be:

git add general.ent chapter01/changelog.xml chapter06/bash.xml
git commit -m 'fixed a typo in chapter 6 bash'

The final part of updating the book is to update the ticket. This is usually as easy as going to the wiki (https://wiki.linuxfromscratch.org/lfs), going to the ticket and choosing resolve, changing resolution to fixed.

This chapter covers the things you need to do when you are using Trac for entering new bugs into the system and fixing/updating outstanding bugs.

We assume you have already logged into Trac before doing anything outlined in the following sections.

You found a bug, or somebody else found a bug and you decided to add it to Trac. Or you want to log a request for a feature that somebody has asked for or a task of some other type.

When you are ready to start working on an issue that is in Trac, step one is to assign it to yourself. This informs all the editors that you are working on the issue.

Once you have fixed an issue and have committed the relevant changes into the Subversion repository, you need to mark the ticket fixed in Trac.

Updating a package in the LFS-Book consists of the following:

When a new package is released by its maintainers, we mark this in Trac. You don't need to update the book right after a package is released, but at least announce it.

When you're going to update the book by adding a new package to it, here's how you do it:

Now that you're done, make sure the book renders properly and check the links in Chapter 3 to make sure the new package links are valid.

If this checks out, then you're done with the package update. For more information about how to build the HTML files from the book sources, refer to Chapter 8.

Commit your changes to the Git repository.

In both LFS and BLFS we try to advise our users of security vulnerabilities. Sometimes vulnerabilities are mentioned in a release announcement, at other times they are noticed later.

The current advisories point to the development books, with past advisories pointing to the numbered books (rather than to'stable' ). You will probably need to create some symlinks in wherever you render the books so that both versions of the current LFS and BLFS books point to your local copies so that you can check them before pushing. If you have local renders of the current released books you can also link to those. Because consolidated.html is located in the BLFS book, you will need to make a special symlink to get to that from your local LFS advisories.

For historical reasons, the links at rivendell point to 'svn' and 'systemd' for the current books.

Normally we will raise a security advisory once we have a tested fix in the appropriate book (LFS or BLFS), i.e. a newer version, or a patch. However sometimes a vulnerability remains open for a long time and we may choose to suggest a workaround such as "do not use this feature of the package".

We try to provide enough details to enable users to decide if they need to update as soon as possible, including a vulnerability assessment and any details we can find. If upstream provides an assessment of the severity we will normally use that. Otherwise we must search around for details. If details of a CVE are publically visible there might be a rating at https://nvd.nist.gov/vuln/search although sometimes that site regards issues as Moderate when a rating of High might be appropriate. If in doubt, our default rating is High.

In theory, a user of a recent past version of the book can look at our current and previous vulnerabilities and see every fixed vulnerability which affects a package. In practice it is not always clear that a newer version has fixed a vulnerability - if that comes to light after we have made a release with a newer version of the package such items might be ignored on the grounds that most users have already looked at the previous vulnerabilities and are unlikely to notice new additions - for example a vulenrability in flac-3.2 was fixed in flac-3.3 but not publically mentioned. In such cases we can add a 'Late Advisories' section between the advisories for the current book and the previous advisories - see 10.0-102 in consolidated.html and the corresponding entry in 10.0.html. In these cases, the best way to alert users is to announce the advisory on the lfs-support and blfs-support lists.

From time to time we may become aware that the details of an advisory should be changed, e.g. it might be that it is invalid for some reason (e.g. never in a released version, or only applicable to windows). When this happens we can modify the existing item and change the 'Date' entry to 'Updated' with the current date.

For day to day updates to advisories the files are organized into:

There are two parts to creating an advisory:

Much of any new item will be copied from a template or from an earlier advisory. It is important to take your time when reviewing what you have created, to see that it reads and links correctly.

The consolidated page provides a list of our advisories since we began to create them. They are numbered within releases, in each case newest first. This page lives in blfs/advisories but it is common to both LFS and BLFS.

Create the details for the next available advisory number. The top part of the page contains a lot of commented items which can be used as a template, but if there are previous advisories for this package it may be easier to copy details from one of those.

As well as the number, and the sa-NN.N-nnn link, each entry has a date and a severity. Where upstream has assigned a severity we normally go with that. If a source such as NVD has assigned a severity, consider if it is appropriate (our default is 'High' even for Denial of Service on non-server programs).

Where one or more CVEs have been issued, it may be useful to link to NVD if that provides details, or else to any other reliable source which explains the vulnerability (i.e. google for it). If the details are not public, you might not be able to find a suitable external link. But for packages which provide their own advisories it is good to link to those.

Render the page, check that the links (external to applications, and internal to the sysv and systemd books) work correctly and that what you have written and pasted makes sense.

It might be beneficial to commit the change to consolidated.html at this point, so that you own the advisory number (e.g. if you think other people may also be updating advisories). In that case it is probably best to do only one if you have several, and to then update the page for the current release.

There are separate pages for LFS and BLFS where we list our advisories since our last release. These pages are in alphabetical order, with newer advisories coming before older ones for the same package.

In both LFS and BLFS the page is labelled with the number of our current release, so 10.1.html from March 2021 until we release our next version.

You will see there is a commented <h3>PackageName<h3> as a guide. The <h4> line should be copied from the entry you created in consolidated.html. Follow that with a short paragraph summarising the problem and how to fix it (or in exceptional cases other actions to work around the problem). Add a sentence linking to the advisory in consolidated.html.

Some packages get extra information, e.g. Thunderbird has an italic paragraph explaining how the vulnerabilities are in a browser-like context. If this is the first advisory for Thunderbird in this release, copy that part from the previous advisories. Similarly, in LFS glibc gets different information because the only supported upgrade path in LFS is to build a new LFS: give information to let users decide how serious the vulnerabilit{y,ies} is/are for their use-case (many are quite old).

When a new release of the books is ready (i.e. the new book directories for this version have been populated), there are notes in the Notes-re-release.txt files:

As well as updating index.html for the new release, and creating a new page for the new version - initially stating "There are no known security vulnerabilities" - the advisories for what is now the previous release need to be amended.

That is because the packages in that release have been tested up to the versions and packages in that release, and from time to time packages will later drop out, or be replaced by forked versions, or even move to a different part of the book. So the links should be changed so that instead of pointing to the development book, they now point to the version we have just released.

This chapter describes the processes on how to convert the DocBook XML sources to HTML which are presented to the users on the website. The processes are basically same on a local machine when the user clones the XMLs to a local repository and wants to create the HTML files there.

Prerequisites are a small set of tools, namly make, DocBook XML, DocBook XSLT and the xsltproc tools. This tools are used to convert DocBook XML to other formats, most used target format is HTML which can be used to present the books in a nicely formatted way to the user. Other formats might be PDF, but this is not part of this chapter.

This chapter is devided in following parts: