How to use jails with epair with FreeBSD 9.0


January 2012.

Introduction


System used:

FreeBSD testjails 9.0-RELEASE FreeBSD 9.0-RELEASE #1: Thu Jan 19 10:47:03 CET 2012 root@testjails:/usr/obj/usr/src/sys/JAILKERNEL i386

Build and install a kernel including VIMAGE, epair and briged interfaces


Configure the kernel: (/usr/src/sys/i386/conf/JAILKERNEL)

include GENERIC

cpu I686_CPU
ident JAILKERNEL

# Virtual networking for jail
options VIMAGE
device epair
device if_bridge

# The nullFS to mount local directory
options NULLFS

Make and install the kernel:

cd /usr/src
make buildkernel KERNCONF=JAILKERNEL
make installkernel KERNCONF=JAILKERNEL
reboot

Create the jail's environement



mkdir /jails/somejail/
cd /usr/src
make buildworld
make installworld DESTDIR=/jails/somejail/
make distribution DESTDIR=/jails/somejail/
echo 'hostname="testjail.example.com"' >> /jails/somejail/etc/rc.conf
echo 'nameserver 8.8.8.8"' >> /jails/somejail/etc/resolv.conf
mkdir /jails/somejail/usr/ports

Start and use the jail manually


Start the jail:

# jail -c vnet name=testjail host.hostname=testjail path=/jails/somejail persist
# jls
JID IP Address Hostname Path
1 - testjail /jails/somejail

Create the virtual ethernet cable:

# ifconfig epair0 create
epair0a
# ifconfig epair0b vnet testjail

Create the bridge:

# ifconfig bridge0 create
# ifconfig bridge0 addm em0 addm epair0a
# ifconfig bridge0 up

Configure the network on both sides:

# ifconfig em0 192.168.1.1/24
# ifconfig
em0: flags=8943 metric 0 mtu 1500
options=98
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=29
media: Ethernet autoselect (1000baseT )
status: active
epair0a: flags=8943 metric 0 mtu 1500
options=8
nd6 options=29
media: Ethernet 10Gbase-T (10Gbase-T )
status: active
bridge0: flags=8843 metric 0 mtu 1500
nd6 options=29
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: epair0a flags=143
ifmaxaddr 0 port 5 priority 128 path cost 14183
member: em0 flags=143
ifmaxaddr 0 port 1 priority 128 path cost 20000
# jexec testjail ifconfig epair0b 192.168.1.2/24
# jexec testjail ifconfig
lo0: flags=8008 metric 0 mtu 16384
options=3
nd6 options=21
epair0b: flags=8843 metric 0 mtu 1500
options=8
inet6 fe80::c0:a4ff:fe00:60b%epair0b prefixlen 64 scopeid 0x2
inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=21
media: Ethernet 10Gbase-T (10Gbase-T )
status: active


Test the connection:

# jexec testjail ping -c 1 192.168.1.1
PING 192.168.1.1 (192.168.3.223): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.795 ms

--- 192.168.1.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.795/0.795/0.795/0.000 ms

# jexec testjail route add default 192.168.1.250
add net default: gateway 192.168.1.250

# jexec testjail ping -c 1 example.net
PING example.net (192.0.43.10): 56 data bytes
64 bytes from 192.0.43.10: icmp_seq=0 ttl=243 time=92.568 ms

--- example.net ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 92.568/92.568/92.568/0.000 ms

Stop the jail:

# jail -r testjail
enceinte# jls
JID IP Address Hostname Path

Clean the network:

# ifconfig epair0a destroy
# ifconfig bridge0 destroy

Start the jail automatically


As of 2011/08/01, /etc/rc.d/jail has problems using the new jail command (v2), so you must patch it if you want to use vnets:
http://www.freebsd.org/cgi/query-pr.cgi?pr=142972
Also, you need this patch to be able to put the epair interface in the jail's vnet after it is created, but before starting the jail's rc.script (via earlypoststart entries in rc.conf).

See also:
http://lists.freebsd.org/pipermail/freebsd-stable/2012-January/065556.html

My original /etc/rc.d/jail script was revision 1.47.

Here is the modified patch that worked for me:
fbsd9_rcdjail147_vnet_patch

Then, in /etc/rc.conf:

#
# Jails networking
#
cloned_interfaces="bridge0"
ifconfig_bridge0="addm em0"

#
# Jails configuration
#
jail_enable="YES"
jail_v2_enable="YES"
jail_list="testjail"

jail_testjail_name="testjail"
jail_testjail_hostname="testjail.example.net"
jail_testjail_devfs_enable="YES"
jail_testjail_rootdir="/jails/somejail"
jail_testjail_vnet_enable="YES"
jail_testjail_exec_prestart0="ifconfig epair0 create"
jail_testjail_exec_prestart1="ifconfig bridge0 addm epair0a"
jail_testjail_exec_prestart2="ifconfig epair0a up"
jail_testjail_exec_earlypoststart0="ifconfig epair0b vnet testjail"
jail_testjail_exec_afterstart0="ifconfig lo0 127.0.0.1"
jail_testjail_exec_afterstart1="ifconfig epair0b 192.168.1.2/24 up"
jail_testjail_exec_afterstart2="route add default 192.168.1.250"
jail_testjail_exec_afterstart3="/bin/sh /etc/rc"
jail_testjail_exec_poststop0="ifconfig bridge0 deletem epair0a"
jail_testjail_exec_poststop1="ifconfig epair0a destroy"


Then:

# /etc/rc.d/jail onestart testjail
Configuring jails:.
Starting jails:epair0a
add net default: gateway 192.168.1.250
Setting hostname: testjail.example.com.
Creating and/or trimming log files.
Starting syslogd.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout
Clearing /tmp (X related).
Updating motd:.
Starting cron.

Thu Jan 19 11:45:24 UTC 2012
testjail.example.net.
testjails# jls
JID IP Address Hostname Path
1 - testjail.example.com /jails/somejail
# jexec testjail ifconfig
lo0: flags=8049 metric 0 mtu 16384
options=3
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
nd6 options=21
epair0b: flags=8843 metric 0 mtu 1500
options=8
inet6 fe80::c0:a4ff:fe00:70b%epair0b prefixlen 64 scopeid 0x2
inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=21
media: Ethernet 10Gbase-T (10Gbase-T )
status: active
# jexec testjail ping -c 1 example.net
PING example.net (192.0.43.10): 56 data bytes
64 bytes from 192.0.43.10: icmp_seq=0 ttl=243 time=92.799 ms

--- example.net ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 92.799/92.799/92.799/0.000 ms


Sources: