AUTHOR: Matthias S. Benkmann <article at winterdrache dot de>

DATE: 2004-12-24

LICENSE: Creative Commons Attribution-NoDerivs 2.0 
         (http://creativecommons.org/licenses/by-nd/2.0/)

SYNOPSIS: Install LFS next to an existing OS, not on a fresh partition (v1.0)

DESCRIPTION:
This hint presents a simple and safe way to install one or more LFS systems 
on a pre-existing partition that may already contain data from other (even
non-Linux) operating systems. The different systems will be bootable 
independent from each other and run with no loss of performance. 
If you want to build an LFS system but can not or do not want to give it its
own dedicated partition, this hint is for you.

ATTACHMENTS:
http://www.linuxfromscratch.org/hints/downloads/attachments/lfs_next_to_existing_systems/lfs_next_helpers.tar.bz2

PREREQUISITES:
None.

HINT:

########################################################################### 
 Contents
###########################################################################

1. The LFS Partition
  1.1 Choosing the LFS Partition
  1.2 Creating the LINUX_FROM_SCRATCH Directory
  1.3 Matching Up Hint and Book

2. Building the LFS System

3. Making the LFS System Bootable
  3.1 First Read Then Proceed
  3.2 The Problem
  3.3 The Solution: pre-init
  3.4 The Boot Manager is Special
  3.5 Making the LFS System Bootable
  3.6 /dev/console

4. Accessing the Full LFS Partition

5. Elevating the LFS system to /
  5.1 Introduction
  5.2 Don't Saw the Branch You're Sitting On
  5.3 Don't Break Your Boot Manager
  5.4 Don't Touch Special Directories and Files

A. Acknowledgements and Changelog


########################################################################### 
 Before You Begin
###########################################################################

Just to avoid misunderstandings let me make clear that this hint is *not*
about cross-compiling an LFS system from a non-Linux operating system. This
hint addresses only the aspects relevant to building and booting an LFS system
next to other data on an existing partition rather than using a fresh, empty, 
dedicated partition. This hint is mostly independent of the actual instructions 
used to build the LFS system, so you may use this hint in conjuction with a
cross-compiling hint. However, this hint is written under the assumption that 
you are following the LFS book, i.e. you're using a Linux system as host for 
the build, and any shell commands given in this hint are meant to be executed
on a Linux host.


########################################################################### 
 1. The LFS Partition
###########################################################################

 1.1 Choosing the LFS Partition
 ------------------------------

The LFS book explains how to create a new partition for your LFS system and
how to set it up properly. When you use this hint you don't need to do this.
Instead you pick one of your existing partitions to install on. There are
some conditions that a partition needs to fulfill in order to be usable with
this hint. If your chosen partition belongs to a Linux system (e.g. your host
distro), condition 1 is probably the only thing you need to worry about. But
if you want to install to a partition from a non-Linux OS, read them all 
carefully.

1) The partition needs to have enough free space.

   This restriction should be obvious. See the LFS book for an estimate of how
   much space you are going to need.

2) The filesystem used on the partition must offer the usual Linux/Unix 
   features such as user/group/other permissions, hard and symbolic links.
   
   This hint does not use loopback-mounted files or other tricks to install
   one filesystem on another filesystem. Therefore the target partition's 
   filesystem needs to support the features necessary for a Linux installation 
   directly. And of course these features need to be supported by the Linux 
   driver for that filesystem. Unfortunately this rules out VFAT and NTFS, 
   so installing LFS on a Windows partition is not possible with this hint.
   But partitions from other *nixes may work (e.g. the HFS+ filesystem from 
   Mac OS X has been confirmed to support all features, including device
   nodes). 
   If you are unsure about your chosen partition, contact me or ask on the 
   lfs-support mailing list.
   Note that Linux device nodes don't need to be supported on the target
   partition. A workaround for partitions that don't support Linux device nodes
   is offered later in this hint.

3) The partition needs to be usable as a root partition.

   This restriction is usually a non-issue. It just means that the kernel
   needs to be able to use the chosen partition as root filesystem. Partitions
   that satisfy condition 2 should also satisfy this condition, but there may 
   be some filesystem that the kernel can't use as root fs. 

4) If there is a file /dev/console on the partition, attempting to open it
   must be safe.
   
   The Linux kernel attempts to open /dev/console on the root partition during
   the boot. If this file does not exist, is a regular file or an actual 
   Linux device node for the console this is harmless, but if it is something
   else (e.g. a device node for a non-Linux OS), you should make sure that
   the kernel's attempt to open it is safe. Contact me or ask on lfs-support
   if you do indeed have a weird /dev/console on your target partition and
   are unsure about its safety. BTW, it is trivial to patch the Linux kernel
   to not open /dev/console and this will be described later in this hint, so
   even if you should have an unsafe /dev/console, this need not be a 
   show-stopper.

   NOTE: 
         To find out if you do have a /dev/console on your chosen target 
         partition, you need to boot with a system that does not use the /dev 
         directory on that partition (e.g. boot from a rescue disk). Many 
         systems use virtual /dev directories that contain files only while the
         system is running. These virtual files don't matter, because they 
         won't exist when you boot your LFS system.
         

 1.2 Creating the LINUX_FROM_SCRATCH Directory
 ---------------------------------------------

After you've chosen a target partition, mount it if it isn't already.
To keep the files of the new LFS system separate from those of other operating
systems, create a new directory LINUX_FROM_SCRATCH somewhere on
the target partition. It does not need to be a top-level directory and the
name is arbitrary. This hint will refer to this directory as 
/.../LINUX_FROM_SCRATCH. Substitute the appropriate path to your LFS directory
whenever you see this.


 1.3 Matching Up Hint and Book
 -----------------------------

The LFS book is written under the assumption that the new LFS system will be
installed to the root directory / of the target partition. But with this hint
you will install into the /.../LINUX_FROM_SCRATCH directory instead.
Take for instance the following fragment from the LFS book:

      Throughout this book, the environment variable LFS will be used 
      several times. It is paramount that this variable is always 
      defined. It should be set to the mount point chosen for the LFS 
      partition. Check that the LFS variable is set up properly with:

          echo $LFS

      Make sure the output shows the path to the LFS partition's mount 
      point, which is /mnt/lfs if the provided example was followed. 

If you followed these instructions exactly and set the variable LFS to the 
mount point of the LFS partition, you'd install to the / directory of the 
target partition. To match up the LFS book's instructions and this hint, 
create a directory /mnt/lfs and use the following command:

    mount --bind /.../LINUX_FROM_SCRATCH /mnt/lfs

to do a virtual mount of /.../LINUX_FROM_SCRATCH on /mnt/lfs. After doing this
your situation will match that in the LFS book exactly. You have a mount 
point /mnt/lfs and it's where your LFS system will be built.
As a consequence you can set your LFS variable to /mnt/lfs just as done in the
standard book instructions.

Of course you could just as well skip the mount --bind command and substitute
/.../LINUX_FROM_SCRATCH whenever the book talks about "the mount point of
the LFS partition".

NOTE: 
    When the LFS book does not talk about the mount point, but about the 
    partition itself or its corresponding block device /dev/[xxx], the book's
    instructions apply unchanged, i.e. "/dev/[xxx]" remains "/dev/[xxx]" and
    does *not* become "/.../LINUX_FROM_SCRATCH/dev/[xxx]" or whatever.


########################################################################### 
 2. Building the LFS System
###########################################################################

Except for the points discussed in the previous chapter with respect to the
LFS partition and its mount point, the LFS book's instructions apply unchanged 
up to and including the last package of Chapter 6. After you've installed that,
extract the lfs_next_helpers.tar.bz2 archive and install it (from inside the
chroot environment, just like the other Chapter 6 packages) using the following
command:

    make install

This will install the program /sbin/pre-init, which allows booting of a Linux
system whose filesystem root is not the root directory of its partition (as is
the case when building according to this hint).

After installing lfs_next_helpers, continue with the book up to but not
including the chapter "Making the LFS System Bootable".


########################################################################### 
 3. Making the LFS System Bootable
###########################################################################

 3.1 First Read Then Proceed
 ---------------------------

If you're not already familiar with it, read the LFS book's chapter
"Making the LFS System Bootable" in its entirety now before you do anything
else. Then read the rest of this chapter in the hint. After that, follow
the instructions from the LFS book, modifying them as appropriate.


 3.2 The Problem
 ---------------

What would happen if you attempted to boot your LFS system with the 
instructions from the book? Let's leave aside the boot manager related 
problems for the moment. These are relatively minor and will be discussed
later. The major problem will manifest itself after the kernel has finished
loading and wants to start /sbin/init. init is the parent of all processes, 
responsible for starting the boot scripts, for instance. The LFS book's
instructions build init dynamically linked by default. This means that for
loading init the kernel needs the dynamic linker whose path is hardwired in
the init binary (e.g. to "/lib/ld-linux.so.2" on the i386-pc-linux-gnu 
platform).
Here's the first problem. The hardwired path starts with "/lib", but our
LFS system's lib directory is not /lib but /.../LINUX_FROM_SCRATCH/lib. So
unless you have another /lib directory on the target partition (e.g. from your
host distro) that contains a compatible dynamic linker, the kernel won't be
able to load init.
But the problems don't end there. init and the boot scripts want to read their
config files from the /etc directory, but the proper directory to read from is
/.../LINUX_FROM_SCRATCH/etc.


 3.3 The Solution: pre-init
 --------------------------

The solution to the problem above is similar to the way Chapter 6 is built.
Chapter 6 uses the chroot command to create an environment whose filesystem
root is /.../LINUX_FROM_SCRATCH. For solving the boot problem the same
method is used, but in a more extreme way. The new LFS system shall be
self-sufficient, so booting up a host system and then using chroot on that
host system to start the LFS system is unsatisfactory. Therefore the chroot
command has to be inserted into the boot process, before init is even started.
This way init itself will already work with the changed filesystem root and
because init is the parent of all processes, all other processes see only the
changed root, too. Furthermore LFS's own boot scripts will be responsible for
bringing up the system and not the scripts from a host OS.

The tool to make all of this happen is the pre-init program which you installed
from the lfs_next_helpers archive. pre-init uses the chroot(2) system call
(not the /bin/chroot binary) to change the filesystem root, performs some 
other preparations necessary for a successful boot and then replaces itself 
with the init from the LFS system, which will start up and work as if it were 
installed on its own partition. Because pre-init is statically linked and 
does not use any external binaries it has no dependencies on a host system.


 3.4 The Boot Manager is Special
 -------------------------------

Before you can set up your system to boot via pre-init, it is important that
you give special consideration to some aspects of the boot manager.

The first thing that you have to consider is that there can be only one
master boot manager in the master boot record (MBR) of a hard disk and only 
one non-master boot manager per partition. The fact that you're using this 
hint suggests that you have a special interest in keeping your other operating
system(s) untouched. Therefore you should think well about the consequences
before you make the decision to install a new boot manager. If you already
have a boot manager that can boot Linux systems (which is probably the case
if you're booting your host system from disk) there is usually no need to
install a new one. Don't forget that the boot manager has no influence on the
performance (or the "purity") of your LFS system. If your existing boot manager
can handle it, the safest and simplest course of action is to use it for
booting the new LFS system.

The other important detail you have to consider is that the boot manager is
loaded before pre-init can change the filesystem root. The boot manager always
sees a physical view of the disk and partitions. All paths it uses at
run-time will refer to the physical directory structure of the partitions
with no consideration of any mounts or changes in filesystem root that may be
active once the system is booted. An example should make this clear:

Example: 
    Your host system has a separate partition for /usr and you have chosen
    that partition to install the new LFS system on. 
    You have created the directory /usr/LINUX_FROM_SCRATCH as the root 
    directory for the LFS system. 
    You have chosen to use the mount --bind trick suggested in this hint to 
    make /usr/LINUX_FROM_SCRATCH available as /mnt/lfs under the host system.
    
    Viewed from inside Chapter 6's chroot environment, the stage* files of
    the boot manager GRUB are found in /boot/grub.
    
    Viewed from your host system outside chroot, the GRUB files are found in
    /mnt/lfs/boot/grub as well as /usr/LINUX_FROM_SCRATCH/boot/grub.
    
    The boot loader sees the files at (hd?,?)/LINUX_FROM_SCRATCH/boot/grub.

This issue is the most likely source of confusion when using this hint. 
If booting doesn't work as expected, always check your paths. A typical
mistake would be to copy a new kernel to the directory /boot from inside
the new LFS system, to specify (hd?,?)/boot/<kernelname> in GRUB's menu.lst 
and to be confused when GRUB complains that it can't find the kernel image. 
Another typical mistake would be to add a new entry to /boot/grub/menu.lst 
inside the new LFS system and to wonder why it doesn't show up in the boot 
menu.

NOTE:
    If you use LILO instead of GRUB you have to keep in mind that LILO does
    not load config files at boot time and does not use the directory structure
    to find the kernel image at boot time. It's all pre-computed at the time 
    you run /sbin/lilo. Therefore, when you use LILO, only the paths that you 
    pass on the kernel command line (e.g. the "init=..." parameter) have to 
    reflect the physical directory structure. The paths that are only read by 
    /sbin/lilo are ordinary user space paths.
    Disclaimer: 
      I know little about LILO. Take the above with a grain of salt.


 3.5 Making the LFS System Bootable
 ----------------------------------

To make your LFS system bootable follow the instructions in the LFS book 
(taking the issues mentioned in the previous section into account, especially
regarding the paths). When you come to the point of adding an entry for your
new LFS system to your boot manager's configuration, there is one small change
you have to make. Just add 

    init=/.../LINUX_FROM_SCRATCH/sbin/pre-init
    
to the kernel parameters. Make sure that /.../LINUX_FROM_SCRATCH reflects the
*physical* location of the directory on the partition. My entry in GRUB's 
menu.lst looks like this

    title LFS system in /lfs on /dev/hda2
    root (hd1,1)
    kernel /boot/lfskernel-x.y.z root=/dev/hda2 init=/lfs/sbin/pre-init

Note that my kernel is /boot/lfskernel-x.y.z and not /lfs/boot/lfskernel-x.y.z.
/boot is the directory at the top-level of /dev/hda2, not the one in /lfs.
I take care to make sure that /boot and /lfs/boot are always in sync. In fact
I have set up /boot as a symlink to lfs/boot (Notice that the other way around, 
i.e. /lfs/boot as symlink to ../boot wouldn't work!).

pre-init understands some options that you can pass on the kernel command line:

  lfsdir=<dir>  
                 Overrides the directory to use as filesystem root. The default
                 is determined by stripping the suffix "/sbin/..." from the 
                 location of the pre-init binary as passed to the kernel with
                 the "init=..." argument.
  
  realinit=<path> 
                 Overrides the program to execute after pre-init has completed
                 its preparations. The default is "/sbin/init". This option
                 is useful for instance when you want to boot with /bin/sh as 
                 init for maintenance work. 
                 <path> may be specified with or without a leading <lfsdir>
                 prefix.
  
  makedev        Tells pre-init to mount a ramfs on <lfsdir>/dev and to create
                 device nodes for console, null and tty1 on this ramfs. This
                 allows you to boot from a partition that does not support
                 Linux device nodes. Note that this is *not* redundant, even
                 if your boot scripts contain commands to set up /dev.
                 Without a working <lfsdir>/dev/console (or alternatively
                 <lfsdir>/dev/null) the normal init used by LFS won't even get
                 to the point of starting your boot scripts.

pre-init will pass any command line arguments that it doesn't interpret on to 
the real init.


 3.6 /dev/console
 ----------------

Before the kernel starts pre-init, it tries to open /dev/console as standard 
input/output/error for it. It does this, because normal programs, including
the usual inits (but not pre-init), require that these standard file 
descriptors are open.
If your LFS partition does not have a /dev/console at the top-level 
(which it probably doesn't if you're not installing to the root partition of
another Linux system), then you'll get the following message:

    Warning: unable to open an initial console.

This warning is harmless, because one of the things pre-init does when
preparing to boot the LFS system is to open the proper console for the new
LFS system, which is /.../LINUX_FROM_SCRATCH/dev/console.
The only problem caused by a missing /dev/console (aside from the annoying
warning message) is that if pre-init encounters an error that occurs before it
can open /.../LINUX_FROM_SCRATCH/dev/console (or while opening it), is that you
won't see pre-init's error message. The system will just appear to hang
and you have to guess at the problem (which is most likely an incorrect path 
used somewhere). 

To overcome this problem there are several possibilities. The easiest one is
to just create /dev/console (e.g. by using `cp -a' to copy the console device
node from your host Linux distro to directory /dev on the target partition). 

Another solution is to patch the kernel. Responsible for opening /dev/console 
is the following statement in the file init/main.c of the kernel sources:

        if (sys_open((const char __user *) "/dev/console", O_RDWR, 0) < 0)
                printk("Warning: unable to open an initial console.\n");
                
Just replace "/dev/console" with "/.../LINUX_FROM_SCRATCH/dev/console" and
recompile the kernel. Make sure that /.../LINUX_FROM_SCRATCH is the *physical*
path on the partition.


########################################################################### 
 4. Accessing the Full LFS Partition
###########################################################################

The method described in this hint is designed to give the illusion that the
LFS system has its own dedicated partition. It's intended to look as
much as possible like a standard by-the-book LFS system. For that reason, the
other files on the partition outside of the /.../LINUX_FROM_SCRATCH directory
are not made visible in the LFS system by default. You should treat the 
partition as if it was not mounted already. If you want to make the other
files on it available, mount it somewhere (e.g. /mnt) from inside the
LFS system. 

Although it's no problem to have the same partition mounted several 
times, there is one small issue that you should be aware of. If you mount
the LFS partition from inside the LFS system, you will not only get access to 
the files outside of /.../LINUX_FROM_SCRATCH, you will also see a 2nd copy
of every file from your LFS system. This means, for example, that the command 
`find / -name bash' will list your LFS system's bash shell twice, one time as
"/bin/bash" and a 2nd time as "/mnt/.../LINUX_FROM_SCRATCH/bin/bash" (assuming
you mounted the partition on /mnt). This can be annoying and may have
undesirable effects, such as archiving files twice when making a full backup.
There is a simple yet effective trick to get rid of the redundant directory
entries for the LFS system. Simply hide /.../LINUX_FROM_SCRATCH behind a
ramfs like this:

    mount -t ramfs ramfs /mnt/.../LINUX_FROM_SCRATCH

This will make all the files in /mnt/.../LINUX_FROM_SCRATCH disappear. If you
automount your LFS partition at boot via an entry in /etc/fstab, e.g. because 
you want to share a directory (e.g. /home) among different systems, you can
put a ramfs line corresponding to the above into fstab, too. Just make sure 
it's listed after the line for the LFS partition.

TIP:
     Keep the /etc/passwd and /etc/group files synchronized among your LFS
     and other systems that access the same partition. Otherwise a file created
     by user foo:bar under the new LFS system may appear to be owned by sna:fu
     when accessing the partition from another system, which can be a 
     security problem.


########################################################################### 
 5. Elevating the LFS system to /
###########################################################################

 5.1 Introduction
 ----------------

At some point you may decide that you no longer want your LFS system confined
to /.../LINUX_FROM_SCRATCH, but want to move it up to the / level in the
directory hierarchy. In principle that's an easy task:

    1. mkdir /old-root
    2. mv <contents of /> /old-root 
    3. mv <contents of /.../LINUX_FROM_SCRATCH> /
    4. Remove "init=.../pre-init" from kernel command line
    5. Enjoy

However, there are a few gotchas that you have to watch out for.

 
 5.2 Don't Saw the Branch You're Sitting On
 ------------------------------------------

The worst mistake you can make is to attempt to perform the above moving from
inside the LFS system, by just mounting the partition to /mnt without the
hide-behind-ramfs trick, and performing the moves inside /mnt. The problem 
with this might not be obvious at first glance, but you must not forget that
if you mount your LFS partition to /mnt from inside the LFS system, what you
see in /mnt/.../LINUX_FROM_SCRATCH is the actual / directory of your LFS system
and all of the changes you make there will immediately affect /. Now assume
that you do `mv /mnt/.../LINUX_FROM_SCRATCH/lib /mnt/lib', what happens is 
that instantly your /lib directory disappears. It should be obvious that this
is not good. 
 
While there are certain tricks that can be used to perform the move from inside
the LFS system, the safest way is to boot a different system and use
that for the move. It should be obvious that this system must not have any 
crucial files on the LFS partition, so if your LFS partition is the /usr 
partition of another system, for example, you can't use that system for doing
the move. The best thing to do in that case is to use a boot floppy or boot CD
(e.g. Knoppix).

 
 5.3 Don't Break Your Boot Manager
 ---------------------------------

Don't forget that your boot manager sees the physical directory structure of
the partition. Dependining on how you set things up, you may not want to move
around /boot and/or /.../LINUX_FROM_SCRATCH/boot.
 
 
 5.4 Don't Touch Special Directories and Files
 ---------------------------------------------

Some directories and files play a special role and must not be moved around.
An example of this is the lost+found directory on an ext2fs filesystem. Make
sure you don't accidentally move a special file or directory.
Remember: Glob patterns are your enemy and * is the worst of the bunch. 


###########################################################################
 Appendix A: Acknowledgements and Changelog
###########################################################################

ACKNOWLEDGEMENTS:
  * Martin Schaffner for his valuable feedback.


CHANGELOG:

2004-11-24
  * Released version 1.0

2004-11-17
  * Hint completely rewritten.
  * Changed license to CC by-nd 2.0.
  * Changed versioning scheme to <major>.<minor> (starting at 1.0)