Setting up FreeBSD Jails as Virtual Servers

Category: FreeBSD Jails
Subject: by: Khairil Yusof
last modified 2025-09-12 17:17:45 — by: BSDwatch development
Contributors: Khairil Yusof, Tom McLaughlin, Radek Kozlowski

This is an article on how to setup FreeBSD jails as virtual servers for general development use. It's introductory in nature. Many parts of it have be reworked into a section of the FreeBSD Handbook.

DRAFT
NOTICE This article still contains inaccuracies!
I am re-writing it as you read it now. You have been warned.

FreeBSD Jails

FreeBSD jails are self contained environments, which share resources on the host system but have their own ip address, configuration and programs. Essentially a jail can be used to run a complete virtual FreeBSD system (also known as virtual servers) without high resource requirements of complete virtual environments such as VMWare.

Since they are self contained environments, jails are useful when there is a need for different server configurations that do not conflict with each other or, not to mess up a system when testing development versions of ports or programs. Currently the focus of this article are jails for a development environment, and will not concentrate too much on security. The setup will be that of a full FreeBSD distribution, which has everything the default base system has.

Examples of jails as virtual servers usage

Development versions of ports which can conflict with stable versions such as Gnome development ports.
Different conflicting versions of servers or programs.
Ports testing, source installs
Where mistakes or lack of a packaging can result in a messy state. It is easy to wipe out a jail directory without affecting the host system.
Shared hosting or sharing a physical server within a company.
This article does not deal with security just yet to setup secure virtual servers

Creating a `jail`

Building and Installing

The jail(8) man page has good clear documentation on this, and this is the instruction from the man page on setting up a directory tree containing an entire FreeBSD distribution.

For this example the host environment will have an ip address of 192.168.1.1. Jail directory names are usually set as IP addresses while another naming convention that can be used is hostnames.

The jail will be setup with an IP address of 192.168.1.2, and in the directory /usr/jail/192.168.1.2

The following commands should be run as the root user or by a regular user with the proper sudo(8) permissions.

For csh/tcsh:

            setenv D /here/is/the/jail

For sh/bash/ksh users:

            export D=/here/is/the/jail

On 5.X or newer:

            cd /usr/src
            mkdir -p $D
            make buildworld DESTDIR=$D
            cd etc
            make distribution DESTDIR=$D
            mount_devfs devfs $D/dev
            cd $D
            ln -sf dev/null kernel

On 4.X:

            cd /usr/src
            mkdir -p $D
            make buildworld DESTDIR=$D
            cd etc
            make distribution DESTDIR=$D
            cd $D
            ln -sf dev/null kernel
            mkdir $D/stand
            cp /stand/sysinstall $D/stand

Notes

If you have already built world, you can replace make buildworld, with make installworld on the third line, to avoid recompiling everything again.

WARNING

With make distribution, be very careful that the DESTDIR environment variable is set. If it is not, it defaults to overwriting your host system's base setup (rc.conf, passwd, etc.). If in doubt, type it out manually.

Setting up the host environment

Setup daemons to listen only on host IP address

On the host, any daemons that listen on all ports, need to be removed if they will affect those in the jails. This is because, jail daemons all bind onto the same network interface through use of network aliases on the host. We need to either disable some services, or configure them to listen only on the host IP address.

             inetd_flags="-wW -a 192.168.1.1"
             sendmail_enable="NO"

On 5.x or newer

            rpcbind_enable="NO"

On 4.x

            portmap_enable="NO"

Where 192.168.1.1 is the host IP address, all services running out of inetd will now automatically only listen on that address. We also have to set the sendmail_enable flag to prevent sendmail from listening on external host interfaces. This will still allow a sendmail process to start and bind to the localhost address of the host environment so as not to interrupt programs and scripts which mail their output, such as the standard nightly cron jobs. Lastly, the RPC port mapper has been disabled. The jail host cannot be an NFS server, or client, because it cannot be configured to listen to a specific IP address.

We will then need to list out any other daemons that are listening on all IP addresses.

            sockstat -l4

Any services that have *: and not a specific IP will be listening on all available IP addresses. This is not good, as it may conflict with one of the services listening in our jails.

`sshd`

In /etc/ssh/sshd_config, add or change the ListenAddress as below, substituting the IP address with that of your host IP.

                ListenAddress 192.168.1.1

The following services are optional services that may be running on your machine, though they are not enabled by default. Here is how to have them listen to a single IP address by default.

`bind`

In /etc/namedb/named.conf, add the listen-on option to only listen on specific IP addresses.

               listen-on { 192.168.1.1; }

Starting and setting up the `jail` environment

Starting the `jail` for the first time.

Now that the host environment has been setup, it must be told to accept network traffic addressed to our jail's IP address. Once that is accomplished we can startup the jail for the first time.

            ifconfig ed0 inet alias 192.168.1.2/32
            jail /usr/jail/192.168.1.2 gnome 192.168.1.2 /bin/csh

In the ifconfig(8) command, replace the ed0 with the name of your network interface on the host system. This will allow the host environment to accept packets that are addressed to your jail's address.

The jail command will start your jail, give it the hostname gnome, the IP address 192.168.1.2, and drop you to a csh prompt. You will now be interacting directly with the jail host.

Configuring the `jailed` system.

The next few steps are the required post-install configurations that are usually done at the end of a standard installation.

We need to create an empty /etc/fstab
```
                touch /etc/fstab
```
Run newaliases(1) to quell sendmail(8) warnings.
Configure /etc/resolv.conf so that name resolution will work properly. For this example, set it to the host IP address where we have a dns server running.
```
                nameserver 192.168.1.1
```
In rc.conf put these variables in. Enabling SSH will make it easier for us to access, and run programs on the virtual server.
```
                sshd_enable="YES"
                network_interfaces=""
```
The following options may also be added if these services are not relevant to the operation of the jail.
```
                sendmail_enable="NO"
                inetd_flags="NO"
```
On 5.x or newer:
```
                rpcbind_enable="NO"
```
On 4.x:
```
                portmap_enable="NO"
```
We prevented sendmail from receiving mail from all but the localhost, in this case, the jail's own localhost. If you are not going to be providing network services from your jail, you might as well turn off inetd. SSH will allow you to use SFTP for file transfers since FTP will now be shut off. The RPC port mapper has been disabled, but a jail itself MAY still use, or serve NFS services if actually needed, unlike the host environment.

The last steps can be done from either the command line or through sysinstall(8).
- Set password for root with passwd(8)
- Add accounts for users with adduser(8)
- Configure your timezone

Installing basic `ports` and `packages`

At this point you can begin to add programs through packages or via the ports system that are needed for the functionality of your jail. You cannot use sysinstall(8) from within a jail to add packages or install the ports tree from remote locations.

To install from `packages`:

For csh/tcsh:

                    setenv PACKAGEROOT ftp5.us.FreeBSD.org

For sh/bash/ksh users:

                    export PACKAGEROOT=ftp5.us.FreeBSD.org

Installing the packages:

                    pkg_add -r portupgrade
                    pkg_add -r pdksh
                    etc...

Change the PACKAGEROOT to our desired FTP mirror site. By setting that, and using pkg_add(8) with the -r option, pkg_add will download the latest package for your release.

To install from the latest ports:

                fetch ftp://ftp5.us.freebsd.org/pub/FreeBSD/ports/ports/ports.tar.gz
                cat ports.tar.gz | tar --unlink -xpzf - -C /usr

WARNING

If you have installed packages, and then decide to start installing from the latest ports, run portupgrade -uarR directly after you have created the ports tree, so that your currently installed package versions will be in sync with your ports tree.

Managing `Jails`

Using system `rc` scripts to start, and stop `jails`

In 5.x, or newer, /etc/defaults/rc.conf has a well documented section on configuring jails. We can control our jails from the host system using this script after we have added the appropriate lines in /etc/rc.conf on the host system.

`rc.conf`

For our example we will add the following lines to /etc/rc.conf (on host)

                jail_enable="yes"    # Set to NO to disable starting of any jails
                jail_list="gnome"    # Space separated list of jail names

                jail_gnome_rootdir="/usr/jail/192.168.1.2"   # Jail's root directory
                jail_gnome_hostname="gnome.domain.com"       # Jail's hostname
                jail_gnome_ip="192.168.1.2"                  # Jail's IP address
                jail_gnome_exec="/bin/sh /etc/rc"            # command to execute in jail
                jail_gnome_devfs_enable="YES"                # mount devfs in the jail
                jail_gnome_fdescfs_enable="YES"              # mount fdescfs in the jail
                jail_gnome_procfs_enable="YES"               # mount procfs in jail
                #jail_gnome_devfs_ruleset="ruleset_name"     # devfs ruleset to apply to jail

        You should also do the IP aliasing automatically by adding
        this line in your rc.conf
                ifconfig_fxp0_alias0="inet 192.168.1.2 netmask 0xffffffff"

Starting the `jail`

Now to start the jail, you only need to issue the command. If more than one jail is defined in the startup list, all will be started.

                /etc/rc.d/jail start

To start a specific jail

                /etc/rc.d/jail start gnome

Stopping the `jail`

To stop all jails

                /etc/rc.d/jail stop

To stop a specific jail

                /etc/rc.d/jail stop gnome

Checking that your `jail` is running

To get a list of running jails use the jls(8) command

jls

Reusing and Sharing directories

Reusing `jails`

The steps above to build and setup a jail can, of course, take a while. This is not prefferable if you are looking to do a limited test of a few specific scenarios, and then discard the jail once you are done. You may find it useful to build a basic jail having gone through the steps above, make any additional configuration edits, and install any basic ports that you require. From that point you can repeatedly copy that jail in order to create new ones. To do this, cp -R will not work. Instead, do the following while the original jail is not running.

            mkdir /usr/jail/new
            cd /usr/jail/old
            tar -cpf - . | tar -C /usr/jail/new -xpf -

Sharing a filesystem between jails

Often you want to share directories over different jails, such as distfiles for ports to save disk space. One way to do it is via nullfs.

mount_nullfs(8)

            mount_nullfs /shared/distfiles /usr/jail/192.168.1.2/usr/ports/distfiles

You can automatically mount it at boot time by adding an entry in /etc/fstab on the host system.

            /shared/distfiles /usr/jail/192.168.1.2/usr/ports/distfiles nullfs rw 0 0

Network Access

FIXME

Gnome/X11 Desktop specific

Forwarding X through ssh

There are some issues with forwarding X applications via ssh. Change the entry in /etc/ssh/sshd_config (jail) and set

            PermitRootLogin yes
            X11UseLocalhost no

Notes

I still had some problems with certain applications like gvim. Also there are X authentification errors if I su and try to run an X app. If you know of a solution, please let me know. These problems may only be occuring due to having host X display, on the same box.

Gnome Desktop and `GDM` via secondary X Server and the `XDMCP`

`GDM` and the `XDMCP`

ssh in as root to your virtual server and run gdmsetup. Enable XDMCP.
Run gdm to start up the gdm service.
switch to a free console and you can now login to your jail desktop on a secondary X Server.
```
                    X :1 -query gnome
```
To automatically start up gdm in your jail, copy
/usr/local/etc/rc.d/gdm.sh.sample

to

/usr/local/etc/rc.d/gdm.sh

Gnome Development testing

For those interested in helping out with FreeBSD Gnome testing, you should visit the FreeBSD Gnome Development FAQ.

There are additional jail and build scripts, available from the development cvs, at marcuscom.com

Authors

Khairil Yusof
4.x specific content and further editing by tmclaugh@sdf.lonestar.org
BSDwatch (this copy)

Additional FreeBSD related Articles, and man pages.

BSDwatch

Articles

Setting up FreeBSD Jails as Virtual Servers

FreeBSD Jails

Examples of jails as virtual servers usage

Creating a jail

Building and Installing

On 5.X or newer:

On 4.X:

Notes

Setting up the host environment

Setup daemons to listen only on host IP address

On 5.x or newer

On 4.x

sshd

bind

Starting and setting up the jail environment

Starting the jail for the first time.

Configuring the jailed system.

On 5.x or newer:

On 4.x:

Installing basic ports and packages

To install from packages:

Managing Jails

Using system rc scripts to start, and stop jails

rc.conf

Starting the jail

Stopping the jail

Checking that your jail is running

Reusing and Sharing directories

Reusing jails

Sharing a filesystem between jails

Network Access

Gnome/X11 Desktop specific

Forwarding X through ssh

Notes

Gnome Desktop and GDM via secondary X Server and the XDMCP

GDM and the XDMCP

Gnome Development testing

Authors

Creating a `jail`

`sshd`

`bind`

Starting and setting up the `jail` environment

Starting the `jail` for the first time.

Configuring the `jailed` system.

Installing basic `ports` and `packages`

To install from `packages`:

Managing `Jails`

Using system `rc` scripts to start, and stop `jails`

`rc.conf`

Starting the `jail`

Stopping the `jail`

Checking that your `jail` is running

Reusing `jails`

Gnome Desktop and `GDM` via secondary X Server and the `XDMCP`

`GDM` and the `XDMCP`