Welcome to zewaren.net. This site presents myself and mostly archives the solutions to some problems I once had.

Resetting the firmware of a MegaRAID LSI SAS-9266-8i

Not so frequently asked questions and stuff: 



Issue: I bought a spare SAS-9266-8i on eBay. When I plugged it in, it woudln't stop beeping and complained that its firmware is in fault state.

The following message was displayed during POST:

Version XXXX (Build YYYY ZZ, WWWW)
Copyright(c) WWWW LSI Corporation
Host Adapter - Bus 6 Dev 0:
F/W is in Fault State - MFI Register State 0xF001002

Adapter at Baseport is not responding
No MegaRAID Adapter Installed

This card clearly needed its firmware reconfigured.

What I've tried

I've booted a FreeBSD live CD and tried to inspect the controller using mfiutil, but it couldn't do anything.

dmesg confirms that the firmware is not ready:

# dmesg | grep mfi
mfi0: Megaraid SAS driver Ver XXXX
mfi0: <Invader> port 0x2000-0x20ff mem 0xb1b00000-0xb1b0ffff,0xb1a00000-0xb1afffff irq 16 at device 0.0 on pci2
mfi0: Using MSI
mfi0: Megaraid SAS driver Ver XXXX
mfi0: Firmware fault
mfi0: Firmware not in READY state, error 6

I've tried to use storcli on FreeBSD but it couldn't see the card either.

I've tried to use storcli on a EFI shell, but the executable didn't work: it froze instantly when I started it.

I've tried to use the LSI Pre boot CD to flash the card, but it wasn't detected either by any of the tools.

What worked

The key to reset the card was to use megarec.

What I ended up doing was prepare a FreeDOS boot disk including megarec and its dependencies, and boot it. Then I could use the megarec commands to flash the card.

The first thing I did was erase the current firmware:

> megarec -cleanflash 0

Once rebooted, the card stopped beeping, which was very nice for my ears.

Then I tried flashing the recovery image from the LSI CD:

> megarec -m0flash 0 2208_16.rom

The flashing went well, but once rebooted, the card was back in F/W fault state, destroying my ears again.

I then decided to flash the latest firmware from the LSI website:

> megarec -m0flash 0 mr2208fw.rom

That was the solution. Now the card is working perfectly. I could go into its config tool and setup my RAIDs.

Version (Buil July 22, 2015)
Copyright(c) 2015 AVAGO Technologies
F/W Initializing Devices 100%
HA -0 (Bus 3 Dev 0) LSI MegaRAID SAS 9266-8i
PCI Slot Number: 255

ID LUN VENDOR    PRODUCT                     REVISION          CAPACITY
-- --- ------    -------                     --------          --------
       AVAGO     LSI MegaRAID SAS 9266-8i    3.460.114-6464    0MB
25 0   ATA       CT500MX500SSD1              020               476940MB
25 0   ATA       CT500MX500SSD1              020               476940MB
   0   AVAGO     Virtual Drive               RAID1             476940MB

All good!

Here are some reference articles that gave me all the information I needed:

Build your own FreeBSD ports and make packages out of them using jails, poudriere and portshaker

The FreeBSD logo


You make custom ports of your applications. You would like to use pkg to install them on your production and development machines.


Build a poudriere server with portshaker!


  • Custom ports come in the form of a git port tree
  • Everything should run in a jail. Git has many dependencies you don't want on your host systems (I'm looking a you bash).
  • Everything should use ZFS as much as possible

Setting up the base system

Make sure the following kernel modules are loaded:

  • tmpfs
  • linux
  • linux64
  • nullfs
  • procfs
  • fdescfsu
  • linprocfs

Load them now:

# kldload tmpfs
# kldload linux
# kldload linux64
# kldload nullfs
# kldload procfs
# kldload fdescfsu
# kldload linprocfs

Make sure they're loaded on boot:

# sysrc -f /boot/loader.conf tmpfs_load="YES"
# sysrc -f /boot/loader.conf linux_load="YES"
# sysrc -f /boot/loader.conf nullfs_load="YES"
# sysrc -f /boot/loader.conf procfs_load="YES"
# sysrc -f /boot/loader.conf fdescfsu_load="YES"
# sysrc -f /boot/loader.conf linux64_load="YES"
# sysrc -f /boot/loader.conf linprocfs_load="YES"

Make sure jails can use IPC

# sysctl security.jail.sysvipc_allowed=1
# sysrc jail_sysvipc_allow=YES

Create a ZFS dataset for the poudriere system:

# zfs create zroot/poudriere

Create a jail, and make sure the jail is allowed to create build jails and manage its ZFS dataset. Here the relevant configuration for a ezjail jail:

# cat /usr/local/etc/ezjail/poudriere
export jail_poudriere_zfs_datasets="zroot/poudriere"
export jail_poudriere_parameters="children.max=20 allow.mount allow.mount.tmpfs allow.mount.devfs allow.mount.procfs allow.mount.zfs allow.mount.nullfs allow.mount.linprocfs allow.raw_sockets allow.socket_af allow.sysvipc allow.chflags enforce_statfs=1 ip6=inherit ip4=inherit"

Setting up poudriere in the jail

Install poudriere and dialog4ports:

# make -C /usr/ports/ports-mgmt/dialog4ports install clean
# make -C /usr/ports/ports-mgmt/poudriere install clean

Set the moundpoint of the ZFS dataset:

# zfs set mountpoint=/poudriere zroot/poudriere

Make sure the ZFS dataset for the ports and the distfiles exists:

# zfs create zroot/poudriere/ports
# zfs create zroot/poudriere/ports/distfiles

Configure poudriere in /usr/local/etc/poudriere.conf:

ZPOOL: zroot
ZROOTFS: /poudriere
FREEBSD_HOST: "ftp.XXX.freebsd.org"
DISTFILES_CACHE: /usr/local/poudriere/ports/distfiles

Create a port tree, but don't populate it. It will be done by portshaker later:

# poudriere ports -c -F -f none -M /usr/local/poudriere/ports/main -p main

Create a build jail:

# poudriere jail -c -j 111x64 -v 11.1-RELEASE -a amd64
# poudriere jail -l
111x64   11.1-RELEASE amd64 ftp    2017-12-08 14:58:09 /usr/local/poudriere/jails/111x64

Hehe, we have a jail in a jail.

Setting up portshaker

We're going to build a port tree consisting of the regular FreeBSD main one (using portsnap) merged with our own (using git).

Portskaker is an awesome tool. It will merge different port tree, handling conflicts intelligently, and even merging the UIDs/GIDs files properly.

This example will only build a single port tree, but you can configure as many as you want.

Install portshaker:

# make -C /usr/ports/ports-mgmt/portshaker install clean

Install git:

# Make -C /usr/ports/devel/git install clean

Write /usr/local/etc/portshaker.conf:

main_merge_from="portsnap custom"

Write the portshaker tree source for portsnap, in /usr/local/etc/portshaker.d/portsnap:

. /usr/local/share/portshaker/portshaker.subr
if [ "$1" != '--' ]; then
  err 1 "Extra arguments"
run_portshaker_command $*

Write the portshaker tree source for custom, in /usr/local/etc/portshaker.d/custom:

. /usr/local/share/portshaker/portshaker.subr
if [ "$1" != '--' ]; then
  err 1 "Extra arguments"
run_portshaker_command $*

Make sure both files are executable.

Fetch both port trees:

# portshaker -U

You should see portsnap and a git clone being executed.

Merge the port trees:

# portshaker -M

You should have a complete port tree in /usr/local/poudriere/ports/main/. Look for your custom ports and check that they're here.

Setting up nginx

We want to use poudriere's web interface.

Install nginx:

# make -C /usr/ports/www/nginx install clean

Add a server in /usr/local/etc/nginx/nginx.conf

server {
    listen 80 default;
    server_name _;
    root /usr/local/share/poudriere/html;

    location /data {
        alias /usr/local/poudriere/data/logs/bulk;
        autoindex on;

    location /packages {
        root /usr/local/poudriere/data;
        autoindex on;

That's it! No CGI, no proxy, no whatever. Just plain files to serve.

Test it all

Write a list of package into a pkglist file (/usr/local/etc/poudriere.d/someports-pkglist):


Set the port options:

poudriere options -j 111x64 -p main -f /usr/local/etc/poudriere.d/someports-pkglist

Start the build:

# poudriere bulk -J 8 -j 110x64 -p main -f /usr/local/etc/poudriere.d/someports-pkglist -v -v -v
[00:00:00] ====>> Creating the reference jail... done
[00:00:00] ====>> Mounting system devices for 111x64-main
[00:00:00] ====>> Mounting ports/packages/distfiles
[00:00:00] ====>> Using packages from previously failed build
[00:00:00] ====>> Mounting packages from: /usr/local/poudriere/data/packages/111x64-main
[00:00:00] ====>> Copying /var/db/ports from: /usr/local/etc/poudriere.d/111x64-options
[00:00:00] ====>> Appending to make.conf: /usr/local/etc/poudriere.d/111x64-make.conf
/etc/resolv.conf -> /usr/local/poudriere/data/.m/111x64-main/ref/etc/resolv.conf
[00:00:00] ====>> Starting jail 111x64-main
[00:00:00] ====>> Logs: /usr/local/poudriere/data/logs/bulk/111x64-main/2017-12-08_16h59m32s
[00:00:00] ====>> Loading MOVED
[00:00:01] ====>> Calculating ports order and dependencies
Creating repository in /tmp/packages: 100%
Packing files for repository: 100%
[00:16:07] ====>> Committing packages to repository
[00:16:07] ====>> Removing old packages
[00:16:07] ====>> Built ports: ports-mgmt/pkg devel/automake-wrapper devel/autoconf-wrapper print/indexinfo devel/pkgconf devel/pcre devel/gettext-runtime www/nginx lang/perl5.24 devel/gettext-tools devel/p5-Locale-gettext devel/gmake misc/help2man print/texinfo devel/m4 devel/libtool devel/autoconf devel/automake devel/libevent sysutils/tmux
[111x64-main] [2017-12-08_16h59m32s] [committing:] Queued: 20 Built: 20 Failed: 0  Skipped: 0  Ignored: 0  Tobuild: 0   Time: 00:16:07
[00:16:07] ====>> Logs: /usr/local/poudriere/data/logs/bulk/111x64-main/2017-12-08_16h59m32s
[00:16:07] ====>> Cleaning up
[00:16:07] ====>> Unmounting file systems

Connect to the web interface:
Look at the build lists:
Look at the build report:

Use your pkg repository on other systems

Now that our packages are built, let's use them on other systems:

Make sure the config directory exists:

# mkdir -p /usr/local/etc/pkg/repos

Write a pkg repository config file in /usr/local/etc/pkg/repos/poudriere.conf:

poudriere: {
    url: "http://pkg.example.com/packages/111x64-someports/",
    mirror_type: "http",
    enabled: yes,
    priority: 100

Writing a secure version of this file is left as an exercise for the reader.

Update the local database:

# pkg update

Install your custom ports:

# pkg install custom-port

Enjoy your new custom ports infrastructure!

Installing sympa on FreeBSD

Not so frequently asked questions and stuff: 

The FreeBSD logo Image

Let's install sympa with postfix on a FreeBSD jail.

Install sympa

Install the port:

# make -C /usr/ports/mail/sympa install clean

Make sure the syslog entries of sympa are stored into a separate log file:

# mkdir -p /usr/local/etc/syslog.d
# echo "local1.*       /var/log/sympa" > /usr/local/etc/syslog.d/sympa
# service syslogd reload

Make sure sympa is allowed to write into its data and bounce directory:

# chown root:sympa /usr/local/share/sympa/list_data
# chmod 770 /usr/local/share/sympa/list_data
# chown root:sympa /usr/local/share/sympa/bounce
# chmod 770 /usr/local/share/sympa/bounce

Make sure sympa is allowed to write into its config directory:

# chown root:sympa /usr/local/etc/sympa
# chmod 770 /usr/local/etc/sympa
# chown root:sympa /usr/local/etc/sympa/sympa.conf

In my opinion, this is plain stupid. Running software should not be able to edit its config files.

Make sure sympa is allowed to write into its shared directory:

# chown root:sympa /usr/local/share/sympa
# chmod 705 /usr/local/share/sympa

Also, I think this is bad practice.

Make sure sympa is allowed to write into the system mail config directory:

# chown root:sympa /etc/mail
# chmod 770 /etc/mail

Do I like this? Absolutely not.
In particular, sympa needs to write into /etc/mail/aliases.db. If that file already exists on your system, give it a good chown as well.

Edit /usr/local/etc/sympa/sympa.conf to your need. Here's a basic configuration:

domain  sympa.example.com
listmaster  admin@example.com
wwsympa_url http://sympa.example.com/wws
cookie add69524c8a75f47c05ccb47f96c9195
db_type mysql
db_name sympa
db_user sympa
db_passwd Ujzz6YVdEXppajM
static_content_path /usr/local/share/sympa
static_content_url /static-sympa
sendmail_aliases /usr/local/share/sympa/list_data/sympa_aliases

Configuring the database of sympa is out of the scope of this article. Here I'm simply using MySQL.

Enable the service:

# sysrc sympa_enable="YES"

Start the service:

# service sympa start

Installing and configuring postfix

Install postfix:

# Make -C /usr/ports/mail/postfix install clean

Edit the main config (/usr/local/etc/postfix/main.cf) and configure a transport and a map file for sympa:

myhostname = sympa.example.com
mydomain = sympa.example.com
mydestination = localhost, sympa.example.com
recipient_delimiter = +
mailbox_size_limit = 0
transport_maps = regexp:/usr/local/etc/postfix/transport_regexp_sympa
sympa_destination_recipient_limit = 1
sympabounce_destination_recipient_limit = 1
alias_maps = hash:/etc/mail/aliases,hash:/usr/local/share/sympa/list_data/sympa_aliases
alias_database = hash:/etc/mail/aliases,hash:/usr/local/share/sympa/list_data/sympa_aliases


/^.*\-owner@sympa\.example\.com$/ sympabounce:
/^.*\@sympa\.example\.com$/       sympa:

Edit the transport file (/usr/local/etc/postfix/master.cf) and add transports to sympa:

sympa   unix        -   n   n   -   -   pipe
    flags=R user=sympa argv=/usr/local/libexec/sympa/queue ${recipient}
sympabounce unix    -   n   n   -   -   pipe
    flags=R user=sympa argv=/usr/local/libexec/sympa/bouncequeue ${recipient}

Enable and start postfix:

# sysrc postfix_enable="YES"
# service postfix start

Installing the web interface (wwsympa)

Install dependencies:

# make -C /usr/ports/databases/p5-DBD-mysql install clean
# make -C /usr/ports/www/p5-CGI-Fast install clean

Install nginx and fcgiwrap:

# make -C /usr/ports/www/nginx install clean
# make -C /usr/ports/www/fcgiwrap install clean

Configure a profile in /etc/rc.conf and enable the service:


Start the service and check that it's listening correctly:

# service fcgiwrap start
# sockstat -l | grep fcgiwrap
sympa    fcgiwrap   79058 0  stream /tmp/fcgiwrap_sympa.sock

Add a server to the configuration files of nginx:

server {
   listen 80;
   server_name sympa.example.com;
   access_log /var/log/nginx/sympa-access.log;
   error_log /var/log/nginx/sympa-error.log;
   index wws/;
   location /static-sympa {
       alias /usr/local/share/sympa;
       access_log off;

   location / {
     gzip off;
     fastcgi_pass   unix:/tmp/fcgiwrap_sympa.sock;
     fastcgi_split_path_info ^(/wws)(.+)$;
     fastcgi_param  QUERY_STRING       $query_string;
     fastcgi_param  REQUEST_METHOD     $request_method;
     fastcgi_param  CONTENT_TYPE       $content_type;
     fastcgi_param  CONTENT_LENGTH     $content_length;
     fastcgi_param  PATH_INFO          $fastcgi_path_info;
     fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
     fastcgi_param  REQUEST_URI        $request_uri;
     fastcgi_param  DOCUMENT_URI       $document_uri;
     fastcgi_param  DOCUMENT_ROOT      $document_root;
     fastcgi_param  SERVER_PROTOCOL    $server_protocol;
     fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
     fastcgi_param  SERVER_SOFTWARE    nginx;
     fastcgi_param  REMOTE_ADDR        $remote_addr;
     fastcgi_param  REMOTE_PORT        $remote_port;
     fastcgi_param  SERVER_ADDR        $server_addr;
     fastcgi_param  SERVER_PORT        $server_port;
     fastcgi_param  SERVER_NAME        $server_name;
     fastcgi_param  REMOTE_USER        $remote_user;
     fastcgi_param  SCRIPT_FILENAME    /usr/local/libexec/sympa/wwsympa-wrapper.fcgi;
     fastcgi_param  HTTP_HOST          sympa.example.com;
     fastcgi_intercept_errors on;

Enable and start nginx:

# sysrc nginx_enable="YES"
# service nginx start

The web interface is now available.
WWSympa running on FreeBSD

If everything went well, you can now create a list:

WWSympa showing a newly created list

You can check that the alias file was updated correctly:

# cat /usr/local/share/sympa/list_data/sympa_aliases
## This aliases file is dedicated to Sympa Mailing List Manager
## You should edit your sendmail.mc or sendmail.cf file to declare it
#------------------------------ test_list: list alias created 22 Aug 2017
test_list: "| /usr/local/libexec/sympa/queue test_list@sympa.example.com"
test_list-request: "| /usr/local/libexec/sympa/queue test_list-request@sympa.example.com"
test_list-editor: "| /usr/local/libexec/sympa/queue test_list-editor@sympa.example.com"
#test_list-subscribe: "| /usr/local/libexec/sympa/queue test_list-subscribe@sympa.example.com"
test_list-unsubscribe: "| /usr/local/libexec/sympa/queue test_list-unsubscribe@sympa.example.com"
test_list-owner: "| /usr/local/libexec/sympa/bouncequeue test_list@sympa.example.com"

Test everything

Send a test email:

echo "Est-elle brune, blonde ou rousse ? - Je l'ignore." | mail -s "Test email" test_list@sympa.example.com

Check /var/log/maillog:

Dec  7 21:49:16 sympa1 postfix/qmgr[47576]: 987058465: from=<user1@example.com>, size=1378, nrcpt=1 (queue active)
Dec  7 21:49:17 sympa1 postfix/pipe[76920]: 987058465: to=<test_list@sympa.example.com>, relay=sympa, delay=1.1, delays=0.02/0/0/1.1, dsn=2.0.0, status=sent (delivered via sympa service)
Dec  7 21:49:17 sympa1 postfix/qmgr[47576]: 987058465: removed
Dec  7 21:49:17 sympa1 postfix/smtpd[76916]: connect from unknown[]
Dec  7 21:49:18 sympa1 postfix/qmgr[47576]: 1119E8478: from=<test_list-owner@sympa.example.com>, size=2274, nrcpt=1 (queue active)
Dec  7 21:49:18 sympa1 postfix/smtp[76926]: 1119E8478: to=<user2@example.com>, relay=mail1.example.net[]:25, delay=0.77, delays=0.32/0.01/0.08/0.36, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 7E4E45EA7)
Dec  7 21:49:18 sympa1 postfix/qmgr[47576]: 1119E8478: removed

Check /var/log/messages:

Dec 07 21:49:16 sympa1 sympa_msg[47458]: notice Sympa::Spindle::ProcessIncoming::_twist() Processing Sympa::Message <test_list@sympa.example.com.1512683356.76921>; envelope_sender=user1@example.com; message_id=eme4db733a-c660-45fd-bdf5-37968cdbdc1e@rasa; sender=user1@example.com
Dec 07 21:49:16 sympa1 sympa_msg[47458]: notice Sympa::Spool::store() Sympa::Message <test_list@sympa.example.com.1512683356.76921> is stored into Sympa::Spool::Archive as <1512683356.1512683356.964608.test_list@sympa.example.com,47458,8945>
Dec 07 21:49:16 sympa1 sympa_msg[47458]: notice Sympa::Spool::store() Sympa::Message <test_list@sympa.example.com.1512683356.76921> is stored into Sympa::Spool::Digest <test_list@sympa.example.com> as <1512683356.1512683356.978756,47458,7735>
Dec 07 21:49:16 sympa1 sympa_msg[47458]: notice Sympa::Bulk::store() Message Sympa::Message <test_list@sympa.example.com.1512683356.76921> is stored into bulk spool as <5.5.1512683356.1512683356.985672.test_list@sympa.example.com_z,47458,6615>
Dec 07 21:49:16 sympa1 sympa_msg[47458]: notice Sympa::Spindle::ToList::_send_msg() No VERP subscribers left to distribute message to list Sympa::List <test_list@sympa.example.com>
Dec 07 21:49:18 sympa1 bulk[47461]: notice Sympa::Mailer::store() Done sending message Sympa::Message <5.5.1512683356.1512683356.985672.test_list@sympa.example.com_z,47458,6615/z> for Sympa::List <test_list@sympa.example.com> (priority 5) in 2 seconds since scheduled expedition date

Running Jabber/XMPP client Kaiwa on FreeBSD

Not so frequently asked questions and stuff: 

The FreeBSD logo Image
Let's install Kaiwa on FreeBSD.

Installing Kaiwa

Install nodejs:

# make -C /usr/ports/www/node install clean

Install npm:

# make -C /usr/ports/www/npm install clean

Create a directory to host the code and go there:

# mkdir /usr/local/www/kaiwa
# cd /usr/local/www/kaiwa

Clone the repository:

# git clone https://github.com/digicoop/kaiwa.git .

If you don't have git, you can download the zip file and extract it.

Install the dependencies:

# npm install

Configuring ejabberd

My Jabber/XMPP server is ejabberd.

Here's the relevant part: we must ensure there's a path wired to handler ejabberd_http_ws in /usr/local/etc/ejabberd/ejabberd.yml.

  ## To handle XML-RPC requests that provide admin credentials:
  ## -
  ##   port: 4560
  ##   module: ejabberd_xmlrpc
    port: 5280
    module: ejabberd_http
    ## request_handlers:
    ##   "/pub/archive": mod_http_fileserver
      "/websocket": ejabberd_http_ws
    web_admin: true
    http_poll: true
    http_bind: true
    ## register: true
    captcha: true

Make sure port 5280 is open in your firewall.

Configure Kaiwa

Copy the example config file:

# cp dev_config.example.json dev_config.json

Here's what I put in there:

    "isDev": true,
    "http": {
        "host": "localhost",
        "port": 8000
    "session": {
        "secret": "wSPwBucqnCY4JHEENMY6NM4UsfycNz"
    "server": {
        "name": "Example",
        "domain": "example.com",
        "wss": "ws://example.com:5280/websocket/",
        "muc": "",
        "startup": "groupchat/example%40chat.example.com",
        "admin": "admin"

Running Kaiwa

Create a user to run the service:

# pw useradd kaiwa

Allow the user to feel at home in the code directory:

# chown -R kaiwa:kaiwa /usr/local/www/kaiwa

You can check if everything is working by running the service manually:

# sudo -u kaiwa node server

If everything is fine, install forever to make sure the service is always running:

# npm install forever -g

Create the logs, give them to the right user and configure newsyslog to rotate them:

# touch /var/log/kaiwa.log
# touch /var/log/kaiwa-error.log
# touch /var/log/kaiwa-forever.log
# chown kaiwa:kaiwa /var/log/kaiwa*

echo "/var/log/kaiwa.log kaiwa:kaiwa 640 3 100 * JC /var/run/kaiwa.pid" >> /etc/newsyslog.con
echo "/var/log/kaiwa-error.log kaiwa:kaiwa 640 3 100 * JC /var/run/kaiwa.pid" >> /etc/newsyslog.con
echo "/var/log/kaiwa-forever.log kaiwa:kaiwa 640 3 100 * JC /var/run/kaiwa.pid" >> /etc/newsyslog.con

For thing for the pid file location:

# mkdir /var/run/kaiwa
# chown kaiwa:kaiwa /var/run/kaiwa

Create a rc script in /usr/local/etc/rc.d/kaiwa and make sure it's executable.


# In addition to kaiwa_enable, the following rc variables should be defined:

# kaiwa_msg      The name of your program, printed at start. Defaults to "kaiwa".
# kaiwa_dir      The directory where your node files live. Must be defined.
# kaiwa_logdir   The directory for logfiles. Defaults to ${kaiwa_dir}/logs.
# kaiwa_user     Sudoed before running. Defaults to "kaiwa".
# kaiwa_app      Application main script. Defaults to "server.js" (relative
#               to kaiwa_user's home
# kaiwa_forever  forever binary file path. Defaults to "/usr/local/bin/forever".
# kaiwa_local_forever    use local forever binary
#               (ie. kaiwa_user's home/kaiwa_modules/.bin/forever)
# kaiwa_forever_log      forever log file. Defaults to /var/log/forever.log.

# PROVIDE: kaiwa
# KEYWORD: shutdown

. /etc/rc.subr



# kaiwa executable

# forever needs a path for each command

# get rc vars
load_rc_config $name
: ${kaiwa_enable:="no"}
: ${kaiwa_msg:="kaiwa"}
: ${kaiwa_logdir:="/var/log"}
: ${kaiwa_user:="kaiwa"}
: ${kaiwa_app:="server.js"}
: ${kaiwa_forever:="/usr/local/bin/forever"}
: ${kaiwa_local_forever:="no"}
: ${kaiwa_forever_log:="/var/log/kaiwa-forever.log"}

case ${kaiwa_local_forever} in

# make sure we're pointing to the right place

# any other checks go here
    echo "$kaiwa_msg starting"

    ${kaiwa_forever} start -a -l ${kaiwa_forever_log} -o ${kaiwa_logdir}/kaiwa.log \
        -e ${kaiwa_logdir}/kaiwa-error.log --minUpTime 3000 --pidFile ${pidfile} \
        --workingDir ${kaiwa_dir} ${kaiwa_dir}/${kaiwa_app}

    # kill kaiwa nicely -- node should catch this signal gracefully
    ${kaiwa_forever} stop --killSignal SIGTERM `cat ${pidfile}`

run_rc_command "$1"

Enable the daemon in /etc/rc.conf:


Start Kaiwa and enjoy

Start the service:

# service kaiwa start

Check that the daemon is listening:

# sockstat -4l | grep kaiwa
kaiwa node       54296 11 tcp4       *:*


Building a continuous-integration Android build server on FreeBSD: Part three: configuring Jenkins for on-demand builds

Not so frequently asked questions and stuff: 

The FreeBSD logo Image Image

In this series of blog posts, we're going to create a Android build server for continuous integration on FreeBSD.

  • Part one will explain how to build Android APKs using Gradle on FreeBSD using the Linux emulation.
  • Part two will explain how to configure Gitlab-CI to be able to run builds automatically for each commit.
  • Part three (this post) will explain how to configure Jenkins to be able to run builds and email the APKs to people.


We want people from our project to be able to build APKs of a Android app and get them by email once they're built. People should be able to configure their app with some Jenkins parameters.


This post does not explain the basics of Jenkins. You should read the documentation if it's your first time using it.

You will need the following plugins:

Creating the node

In this scenario, I'm going to assume our FreeBSD build server is not on the same host as the Jenkins.

In consequence, we need to link the two systems by installing a Jenkins slave-node on the build system.

Preparing the server

On your Android build system, create a new user:

# pw group add -n jenkins-slave
# pw user add -n jenkins-slave -g jenkins-slave

Create a SSH key pair:

# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/jenkins-slave/.ssh/id_rsa): jenkins-slave-id_rsa
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in jenkins-slave-id_rsa.
Your public key has been saved in jenkins-slave-id_rsa.pub.

Add the public key to your SSH's authorized_keys. That will be how Jenkins connect to the slave.

# mkdir ~/.ssh/
# cat jenkins-slave-id_rsa.pub >> ~/.ssh/authorized_keys

Create the node on Jenkins

Go to "Manage Jenkins" -> "Manage credentials", and add a new "SSH Username with private key".
Put the private key you generated earlier there and configure the rest.

You used SSH keys, don't you?

Go to "Manage Jenkins" -> "Manage Nodes" and create a new node.

Configure the node with the right host, port and home directory. Set the location of the java executable, or Jenkins will only try to find it in /bin or /usr/bin and it won't work.
That won't connect. Did you forget about your firewall?

If everything goes well, the master node should connect to the server, install its JAR and start the slave node.

Nodes! More nodes!

# ls
jenkins-slave-pepper-id_rsa     jenkins-slave-pepper-id_rsa.pub slave.jar                       workspace

# ps -U jenkins-slave
66474  -  IJ    0:08.67 sshd: jenkins-slave@notty (sshd)
66478  -  IsJ  11:51.54 /usr/local/openjdk8/bin/java -jar slave.jar

Configuring the job

Create a new job, give it a nice name, and start the configuration.

The parameters

Click on "This build is parameterized", and add as many parameters as you want. You will be able to use these parameters as Jenkins variable everywhere later in the build.

I have another one with, like, 10 different parameters

Here I have two parameters:

  • One to specify what kind of build I want: debug, staging, production, etc.
  • One that specify where I want to APK to be sent once it's built.

Fetching the code

Configure Jenkins to fetch the source code of the app where it's located.

Gimme the code!
Here I'll fetch it from git.

Building the app

Add the required environment variables to the build.

This is a good environment, don't you think?

Add a gradle build script, and invoke the tasks to build your app. Here you can use the parameters you set up at the beginning of the config.

You can build without gradle if you want

You can build without the gradle plugin if you want. It only displays the build nicer but is strictly non essential.

Email the APKs once they're built

Add a post build step: "Editable Email Notification".

Put the recipient chosen by the user from the parameters into the list of recipients, customize the sender, reply-to, subject, content, etc.

Add the APK files as attachments.

Don't forget to configure the triggers to send the email if the build succeeds. By default, emails are only sent on failure.

Such a complicated configuration


Start a build. If everything goes well, you should receive the resulting APK by email a few seconds after the build is done.


Building a continuous-integration Android build server on FreeBSD: Part two: configuring Gitlab-CI

Not so frequently asked questions and stuff: 

The FreeBSD logo Image Image Image

In this series of blog posts, we're going to create a Android build server for continuous integration on FreeBSD.

  • Part one will explain how to build Android APKs using Gradle on FreeBSD using the Linux emulation.
  • Part two (this post) will explain how to configure Gitlab-CI to be able to run builds automatically for each commit.
  • Part three will explain how to configure Jenkins to be able to run builds and email the APKs to people.

Step 1: Install, configure and register the Gitlab-CI runner

Most of the information of that step comes from the original documentation.

Create a nice user for the runner:

# pw group add -n gitlab-runner
# pw user add -n gitlab-runner -g gitlab-runner -s /usr/local/bin/bash
# mkdir /home/gitlab-runner
# chown gitlab-runner:gitlab-runner /home/gitlab-runner

Download the runner executable (or build it yourself (see the end of the post)), and mark it as such:

# fetch -o /usr/local/bin/gitlab-ci-multi-runner https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-ci-multi-runner-freebsd-amd64
# chmod +x /usr/local/bin/gitlab-ci-multi-runner

Create and chmod the log file:

# touch /var/log/gitlab_runner.log && sudo chown gitlab-runner:gitlab-runner /var/log/gitlab_runner.log

Create the rc-script (/usr/local/etc/rc.d/gitlab_runner):

# PROVIDE: gitlab_runner

. /etc/rc.subr


load_rc_config $name

command="/usr/local/bin/gitlab-ci-multi-runner run"


    export USER=${user}
    export HOME=${user_home}
    if checkyesno ${rcvar}; then
        cd ${user_home}
        /usr/sbin/daemon -u ${user} -p ${pidfile} ${command} > /var/log/gitlab_runner.log 2>&1

    if [ -f ${pidfile} ]; then
        kill `cat ${pidfile}`

    if [ ! -f ${pidfile} ] || kill -0 `cat ${pidfile}`; then
        echo "Service ${name} is not running."
        echo "${name} appears to be running."

run_rc_command $1

Set the rc script as being executable:

chmod +x /usr/local/etc/rc.d/gitlab_runner

Register the runner with Gitlab-CI:

# sudo -u gitlab-runner -H /usr/local/bin/gitlab-ci-multi-runner register

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/ci):

Please enter the gitlab-ci token for this runner:

Please enter the gitlab-ci description for this runner:
[androidbuild]: Example Android build machine

Please enter the gitlab-ci tags for this runner (comma separated):
Registering runner... succeeded                     runner=sqzGW7_T

Please enter the executor: docker-ssh+machine, docker, docker-ssh, parallels, shell, ssh, virtualbox, docker+machine:

Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Enable the runner service and start it:

# sysrc -f /etc/rc.conf "gitlab_runner_enable=YES" 
# service gitlab_runner start

Step 2: add the CI configuration to your Android project

Create a file named .gitlab-ci.yml and configure it to your needs.

  - export LD_LIBRARY_PATH="/home/androidbuild/android-sdk-linux/build-tools/23.0.2/lib/"
  - export ANDROID_HOME="/home/androidbuild/android-sdk-linux/"

    - master
    - ./gradlew assembleRelease
      - Example/build/outputs/
    expire_in: 1 week

This example file will trigger a build when branch master is updaded, and it will save the artifacts for one week.


Bonus: compiling the runner ourselves

If there's something I don't like, it's downloading executables from the internet when I can build them myself.
Let's build that Gitlab runner ourselves.

Install golang:

# make -C /usr/ports/lang/go install clean

Create a working go directory structure:

% mkdir -p ~/gowork/src/gitlab.com/gitlab-org/
% cd ~/gowork/src/gitlab.com/gitlab-org/
% setenv GOPATH ~/gowork
% setenv PATH "${PATH}:${GOPATH}/bin"

Get the code from Gitlab (checkout the version you want to build):

% git clone https://gitlab.com/gitlab-org/gitlab-ci-multi-runner.git
% cd gitlab-ci-multi-runner/
% git checkout tags/v1.5.2


% gmake deps docker
% gmake build_simple
% file "out/binaries/gitlab-ci-multi-runner"
out/binaries/gitlab-ci-multi-runner: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), statically linked, not stripped

Install (as root):

# install -m 755 out/binaries/gitlab-ci-multi-runner /usr/local/bin/gitlab-ci-multi-runner


Building a continuous-integration Android build server on FreeBSD: Part one: building APKs using Linux emulation

Not so frequently asked questions and stuff: 

Image The FreeBSD logo

In this series of blog posts, we're going to create a Android build server for continuous integration on FreeBSD.

  • Part one (this post) will explain how to build Android APKs using Gradle on FreeBSD using the Linux emulation.
  • Part two will explain how to configure Gitlab-CI to be able to run builds automatically for each commit.
  • Part three will explain how to configure Jenkins to be able to run builds and email the APKs to people.

I'll be using a normal 10.2-RELEASE:

# uname -a
FreeBSD androidbuild 10.2-RELEASE-p11 FreeBSD 10.2-RELEASE-p11 #0: Wed Jan 27 15:56:01 CET 2016     root@androidbuild:/usr/obj/usr/src/sys/GENERIC  amd64

Parts of this posts are duplicate from my old post on how to build APKs on FreeBSD.

Step 1: installing the system dependencies

We're going to install many packages, including a lot of GNU-style one. So if you care about having your systems cleans, I suggest your do all this in a jail.

Install gradle:

# make -C /usr/ports/devel/gradle/ install clean

Go drink the longest coffee you can brew. That stuff installs a lot of dependencies.

Install bash and link it to /bin/bash:

# make -C /usr/ports/shells/bash install clean
# ln -s /usr/local/bin/bash /bin/bash

Install git:

# make -C /usr/ports/devel/git install clean

Install python:

# make -C /usr/ports/lang/python install clean

Create a nice user to execute the builds:

# pw useradd androidbuild
# mkdir /home/androidbuild
# chown androidbuild:androidbuild /home/androidbuild

Load FreeBSD's Linux emulation system and install a base CentOS 6 system:

# kldload linux
# cd /usr/ports/emulators/linux_base-c6 && make install distclean

The latest version of the build tools (24.0.0) use 64bit binaries and libraries. If you want to be able to build your apks with them, you'll also need the 64bit Linux emulation. In that case, you must be running a FreeBSD version >= 10.3. In that case, you should do instead:

# kldload linux
# kldload linux64
echo "OVERRIDE_LINUX_BASE_PORT=c6_64" >> /etc/make.conf
echo "OVERRIDE_LINUX_NONBASE_PORTS=c6_64" >> /etc/make.conf
# cd /usr/ports/emulators/linux_base-c6 && make install distclean

If you alread had the 32 bit CentOS base, uninstall the port, remove the /compat/linux files and reinstall the port again after setting the two entries in make.conf.

Step 2: setting-up the Android SDK

Download the SDK

Fetch and extract the latest version of the Linux SDK:

% fetch 'https://dl.google.com/android/android-sdk_r24.4.1-linux.tgz'
% tar xzf android-sdk_r24.4.1-linux.tgz
% setenv ANDROID_HOME /home/androidbuild/android-sdk-linux/

Patch the SDK

As it is now, the SDK will download build tools for Windows, since it obviously won't recognize our FreeBSD.
Since we're going to use the Linux emulation, we need to path the system so that it downloads Linux binaries:

Download the source of the SDK base:

% git clone https://android.googlesource.com/platform/tools/base

Apply the following patch:

diff -r -u a/common/src/main/java/com/android/SdkConstants.java b/common/src/main/java/com/android/SdkConstants.java
--- a/common/src/main/java/com/android/SdkConstants.java        2016-09-06 07:56:56.325948102 +0000
+++ b/common/src/main/java/com/android/SdkConstants.java        2016-09-06 07:58:10.721944140 +0000
@@ -635,6 +635,8 @@
             return PLATFORM_WINDOWS;
         } else if (os.startsWith("Linux")) {                //$NON-NLS-1$
             return PLATFORM_LINUX;
+        } else if (os.startsWith("FreeBSD")) {              //$NON-NLS-1$
+            return PLATFORM_LINUX;

         return PLATFORM_UNKNOWN;
diff -r -u a/sdklib/src/main/java/com/android/sdklib/internal/repository/archives/ArchFilter.java b/sdklib/src/main/java/com/android/sdklib/internal/repository/archives/ArchFilter.java
--- a/sdklib/src/main/java/com/android/sdklib/internal/repository/archives/ArchFilter.java      2016-09-06 07:56:56.828948347 +0000
+++ b/sdklib/src/main/java/com/android/sdklib/internal/repository/archives/ArchFilter.java      2016-09-06 07:58:35.160941890 +0000
@@ -216,6 +216,8 @@
             hostOS = HostOs.WINDOWS;
         } else if (os.startsWith("Linux")) {                //$NON-NLS-1$
             hostOS = HostOs.LINUX;
+        } else if (os.startsWith("FreeBSD")) {                //$NON-NLS-1$
+            hostOS = HostOs.LINUX;

         BitSize jvmBits;

Rebuild the patched files.

% javac common/src/main/java/com/android/SdkConstants.java
% javac sdklib/src/main/java/com/android/sdklib/internal/repository/archives/ArchFilter.java -cp "sdklib/src/main/java:common/src/main/java:annotations/src/main/java"

Replace the files inside the jar.

% cd sdklib/src/main/java/ && jar uf ${ANDROID_HOME}/tools/lib/sdklib.jar com/android/sdklib/internal/repository/archives/ArchFilter.class
% cd common/src/main/java && jar uf ${ANDROID_HOME}/tools/lib/common.jar com/android/SdkConstants.class

See this patch on the Android website: https://android-review.googlesource.com/#/c/100271/

Download the SDK packages and set-up the build tools

Go to the tool directory:

% cd ${ANDROID_HOME}/tools/

In the example, we're going to build an APK thats uses API version 23.
Let's download the right packages:

% ./android list sdk -u -a
7- Android SDK Build-tools, revision 23.0.3
31- SDK Platform Android 6.0, API 23, revision 3
119- Google APIs, Android API 23, revision 1
% ./android update sdk -u -a -t tools,platform-tools,7,31,119

Let's find the Linux binaries in the build tools, and brand them as such:

% find build-tools/23.0.3/ -maxdepth 1 -type f -print0 | xargs -0 -n 10 file | grep "ELF"
build-tools/23.0.3/mipsel-linux-android-ld:  ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.8, stripped
build-tools/23.0.3/arm-linux-androideabi-ld: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.8, stripped
build-tools/23.0.3/llvm-rs-cc:               ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=8a4ffbc0e197147c4e968722e995605e1d06ea88, not stripped
build-tools/23.0.3/i686-linux-android-ld:    ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.8, stripped
build-tools/23.0.3/bcc_compat:               ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=d565d03b7bafd03d335bdd246832bb31c7cca527, not stripped
build-tools/23.0.3/aapt:                     ELF 32-bit LSB shared object, Intel 80386, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=cfb63b4ad11d0c2d59f10329f0116706e99bf72e, not stripped
build-tools/23.0.3/aidl:                     ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=3cbd3d742040d61877a3c6544778bf4701b2f62d, not stripped
build-tools/23.0.3/aarch64-linux-android-ld: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.8, stripped
build-tools/23.0.3/split-select:             ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=b3bfb153d0ffaef6ed84c316ff682381ba8d75b2, not stripped
build-tools/23.0.3/dexdump:                  ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=a678a8163a2483107d285ffdc94fdde0d4fb2178, not stripped
build-tools/23.0.3/zipalign:                 ELF 32-bit LSB shared object, Intel 80386, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=7d64216663df9fd3b4048952d095fbd07cb4284f, not stripped

% find build-tools/24.0.3/ -maxdepth 1 -type f -print0 | xargs -0 -n 10 file | grep "ELF" | awk 'BEGIN { FS = ":" } ; {print $1}' | xargs brandelf -t Linux
% find build-tools/24.0.3/ -maxdepth 1 -type f -print0 | xargs -0 -n 10 file | grep "ELF" | awk 'BEGIN { FS = ":" } ; {print $1}' | xargs chmod +x

The SDK is now configured and we should be able to build our apps there.

Step 3: building the APK

Fetch the sources of your app:

% git clone 'https://git.example.org/android-app'
% cd android-app

Be sure to have set environment variable ANDROID_HOME to the SDK location.
Start gradle and see if it complains.

% gradle tasks
All tasks runnable from root project

Android tasks
androidDependencies - Displays the Android dependencies of the project.
signingReport - Displays the signing info for each variant.
sourceSets - Prints out all the source sets defined in this project.
To see all tasks and more detail, run gradle tasks --all

To see more detail about a task, run gradle help --task <task>


Total time: 14.839 secs

Gradle is happy, let's try a build:

% gradle assembleRelease

Total time: 2 mins 24.784 secs

Let's look at the build output:

% ls Truc/build/outputs/apk/
truc-app-release-unaligned.apk   truc-app-release.apk


How to advertise a device on the network using Python

Not so frequently asked questions and stuff: 

Python logo Image


You built some IoT device running a Python compliant OS. You want to be able to plug it anywhere, and be able to find it in the network view of Windows (or the equivalent thing of other systems).

Solution: SSDP/UPnP

I've created some code that implements a SSDP server and a HTTP server, in order to notify the network that a device is here.

The base of this code was the SSDP module of coherence.
I took it, converted it to Python 3, and kept only the interesting parts for this project: the parts that responds to `MSEARCH` queries.


The code is attached to this post, and is available as a mirror on Github.

Technical details

First, a SSDP client will try to search devices on the network by issuing a MSEARCH * query:

    M-SEARCH * HTTP/1.1

lib/ssdp.py will handle it and reply with a 200:

    HTTP/1.1 200 OK
    SERVER: ZeWaren example SSDP Server
    USN: uuid:e427ce1a-3e80-43d0-ad6f-89ec42e46363::upnp:rootdevice
    CACHE-CONTROL: max-age=1800
    last-seen: 1477147409.432466
    ST: upnp:rootdevice
    DATE: Sat, 22 Oct 2016 14:44:26 GMT

The client will then fetch the device description:

    GET /jambon-3000.xml HTTP/1.1
    Cache-Control: no-cache
    Connection: Keep-Alive
    Pragma: no-cache
    Accept: text/xml, application/xml
    User-Agent: FDSSDP

And lib/upnp_http_server.py will build and serve that file:

HTTP/1.0 200 OK
Server: BaseHTTP/0.6 Python/3.4.3
Date: Sat, 22 Oct 2016 14:44:26 GMT
Content-type: application/xml

        <friendlyName>Jambon 3000</friendlyName>
        <manufacturer>Boucherie numrique SAS</manufacturer>
        <modelDescription>Jambon Appliance 3000</modelDescription>

The client now knows (nearly) everything about the device.

Relevant links

How to stop the watchdog timer of a BeableBone Black running Linux

Not so frequently asked questions and stuff: 

A Beable Bone Black with an operator setting its watchdog timer

The BeagleBone Black's SoC (AM335x) includes a watchdog timer, that will reset the whole board is it isn't pingged regularly.

Let's see if we can stop that thing running the latest Debian GNU/Linux to date.

# uname -a
Linux beaglebone 4.4.9-ti-r25 #1 SMP Thu May 5 23:08:13 UTC 2016 armv7l GNU/Linux
root@beaglebone:~# cat /etc/debian_version

Ever since this commit, the OMAP watchdog driver has the magic close feature enabled. This means that closing the timer's device won't stop the timer from ticking. The only way to stop it is to send to it the magic character 'V' (a capital 'v').

# wdctl /dev/watchdog
wdctl: write failed: Invalid argument
Device:        /dev/watchdog
Identity:      OMAP Watchdog [version 0]
Timeout:       120 seconds
Timeleft:      119 seconds
KEEPALIVEPING  Keep alive ping reply          0           0
MAGICCLOSE     Supports magic close char      0           0
SETTIMEOUT     Set timeout (in seconds)       0           0

This feature is particularly useful if you want the watchdog timer to only be active when a specific application is running, and if you then want it to be stopped when the application is stopped normally.

Unfortunately, the kernel can be configure with a mode called "no way out", which means that even tough the magic close feature of the driver is enabled, it won't be honored at all, and you are doomed to ping your timer until the end of time once you opened the device.

# cat /proc/config.gz | gunzip | grep CONFIG_WATCHDOG_NOWAYOUT

On a kernel version 3.8, the feature was not enabled:

$ cat /proc/config.gz | gunzip | grep CONFIG_WATCHDOG_NOWAYOUT

So, how do we stop that thing?

Well, you can see in the code of the driver that the default value of the kernel can be overridden by a module param:

static bool nowayout = WATCHDOG_NOWAYOUT;
module_param(nowayout, bool, 0);
MODULE_PARM_DESC(nowayout, "Watchdog cannot be stopped once started "

Edit the boot configuration in /boot/uEnv.txt and add set that parameter to 0 in the cmdline:

cmdline=coherent_pool=1M quiet cape_universal=enable omap_wdt.nowayout=0

Reboot the board, and check that the loaded command line was changed correctly:

# cat /proc/cmdline
console=tty0 console=ttyO0,115200n8 root=/dev/mmcblk0p1 rootfstype=ext4 rootwait coherent_pool=1M quiet cape_universal=enable omap_wdt.nowayout=0

That's it. Now if you send a 'V' to the watchdog right before closing it, it will be stopped.

How to debug python code using GDB on FreeBSD without compromising your system

Not so frequently asked questions and stuff: 

GDB's logo Image The FreeBSD logo


We want to be able to use GDB to debug python code efficiently.

Let's say we have the following code:

from threading import Event
import random
from time import sleep

def blocking_function_one():
	while True:

def blocking_function_two():
	e = Event()

if random.random() > 0.5:

That code will block, and since it doesn't output anything, we have no way of knowing if we went into blocking_function_one or blocking_function_two. Or do we?

For reference, I'm running a 10.2-RELEASE:

# uname -a
FreeBSD bsdlab 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 15:26:37 UTC 2015     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

Step 1: installing a debug version of python

We're going to work in a separate directory, in order not to alter our installation. If we need this in production, we want to be able to let the system as clean as possible when we leave.

# mkdir /usr/local/python-debug

Build python 3.4 from the port collection and set it to be installed in the directory we just created:

# cd /usr/ports/lang/python34
# make install PREFIX=/usr/local/python-debug OPTIONS_FILE_SET+=DEBUG BATCH=1

Normally we would have used NO_PKG_REGISTER=1 to install the package without registering it on the system. Unfortunately, this option is not working anymore (see bug https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=182347).

So, let's copy the files ourselves:

# cp -r work/stage/usr/local/python-debug/* /usr/local/python-debug/

Let's try to run that new python installation:

# setenv LD_LIBRARY_PATH /usr/local/python-debug/lib
# /usr/local/python-debug/bin/python3.4dm
Python 3.4.5 (default, Sep  4 2016, 00:42:59)
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.

The "d" in "dm" means "debug".

Building python also produced an important file that we need to save for later before cleaning the work tree:

# cp work/Python-3.4.5/python-gdb.py ~/

Step 2: building a python aware GDB

Build GDB from the port collection:

  • Making sure the python extensions are enabled
  • Telling the configure script where to find our python installation
  • Telling the configure and build scripts where to find the relevant headers and libraries
# ln -s python3.4 /usr/local/python-debug/bin/python
# cd /usr/ports/devel/gdb
# make PREFIX=/usr/local/python-debug \
PYTHON_CMD=/usr/local/python-debug/bin \
CFLAGS+="-I/usr/local/python-debug/include -L/usr/local/python-debug/lib" \
CXXFLAGS+="-I/usr/local/python-debug/include -L/usr/local/python-debug/lib"

Also copy that installation manually to our special directory:

# cp -r work/stage/usr/local/python-debug/* /usr/local/python-debug/

Let's check that it's working and has the python extensions:

# /usr/local/python-debug/bin/gdb
GNU gdb (GDB) 7.11.1 [GDB v7.11.1 for FreeBSD]
(gdb) python
>import gdb

Step 3: wire it all together

Now we have:

  • A version of Python that integrates debug information.
  • A version of GDB that can run higher level GDB scripts written in Python.
  • A python-gdb script to add commands and macros.

Copy the GDB script somewhere where Python can load it:

# mkdir ~/.python_lib
# mv ~/python-gdb.py ~/.python_lib/python34_gdb.py

Let's run our stupid blocking script:

# setenv PATH "/usr/local/python-debug/bin/:${PATH}"
# python where-am-i-blocking.py

In another shell, find the PID of the script, and attach GDB there.

# ps auxw | grep python
root     24226   0.0  0.7  48664  15492  3  I+    3:00AM     0:00.13 python where-am-i-blocking.py (python3.4)
root     24235   0.0  0.1  18824   2004  4  S+    3:00AM     0:00.00 grep python

# setenv PATH "/usr/local/python-debug/bin/:${PATH}"
# gdb python 24226
GNU gdb (GDB) 7.11.1 [GDB v7.11.1 for FreeBSD]
[Switching to LWP 100160 of process 24226]
0x00000008018e3f18 in _umtx_op () from /lib/libc.so.7

Load the GDB python script:

(gdb) python
>import sys
>import python34_gdb

The python macros are now loaded:

(gdb) py
py-bt               py-down             py-locals           py-up               python-interactive
py-bt-full          py-list             py-print            python

Let's see where we are:

(gdb) py-bt
Traceback (most recent call first):
  <built-in method acquire of _thread.lock object at remote 0x80075c2a8>
  File "/usr/local/python-debug/lib/python3.4/threading.py", line 290, in wait
  File "/usr/local/python-debug/lib/python3.4/threading.py", line 546, in wait
    signaled = self._cond.wait(timeout)
  File "where-am-i-blocking.py", line 11, in blocking_function_two
  File "where-am-i-blocking.py", line 16, in <module>

We're in blocking_function_two.

Let's check the wait's frame local variables:

(gdb) bt
#0  0x00000008018e3f18 in _umtx_op () from /lib/libc.so.7
#1  0x00000008018d3604 in sem_timedwait () from /lib/libc.so.7
#2  0x0000000800eb0421 in PyThread_acquire_lock_timed (lock=0x802417590, microseconds=-1, intr_flag=1) at Python/thread_pthread.h:352
#3  0x0000000800eba84f in acquire_timed (lock=0x802417590, microseconds=-1) at ./Modules/_threadmodule.c:71
#4  0x0000000800ebab82 in lock_PyThread_acquire_lock (self=0x80075c2a8, args=(), kwds=0x0) at ./Modules/_threadmodule.c:139
#5  0x0000000800cfa963 in PyCFunction_Call (func=<built-in method acquire of _thread.lock object at remote 0x80075c2a8>, arg=(), kw=0x0)
    at Objects/methodobject.c:99
#6  0x0000000800e31716 in call_function (pp_stack=0x7fffffff5a00, oparg=0) at Python/ceval.c:4237
#7  0x0000000800e29fc0 in PyEval_EvalFrameEx (
    f=Frame 0x80245d738, for file /usr/local/python-debug/lib/python3.4/threading.py, line 290, in wait (self=<Condition(_lock=<_thread.l
ock at remote 0x80075c510>, _waiters=<collections.deque at remote 0x800b0e9d8>, release=<built-in method release of _thread.lock object a
t remote 0x80075c510>, acquire=<built-in method acquire of _thread.lock object at remote 0x80075c510>) at remote 0x800a6fc88>, timeout=No
ne, waiter=<_thread.lock at remote 0x80075c2a8>, saved_state=None, gotit=False), throwflag=0) at Python/ceval.c:2838
#25 0x0000000800eb4e86 in run_file (fp=0x801c1e140, filename=0x802418090 L"where-am-i-blocking.py", p_cf=0x7fffffffe978)
    at Modules/main.c:319
#26 0x0000000800eb3ab7 in Py_Main (argc=2, argv=0x802416090) at Modules/main.c:751
#27 0x0000000000400cae in main (argc=2, argv=0x7fffffffeaa8) at ./Modules/python.c:69

(gdb) frame 7
#7  0x0000000800e29fc0 in PyEval_EvalFrameEx (
    f=Frame 0x80245d738, for file /usr/local/python-debug/lib/python3.4/threading.py, line 290, in wait (self=<Condition(_lock=<_thread.lock at remote 0x80075c510>, _waiters=<collections.deque at remote 0x800b0e9d8>, release=<built-in method release of _thread.lock object at remote 0x80075c510>, acquire=<built-in method acquire of _thread.lock object at remote 0x80075c510>) at remote 0x800a6fc88>, timeout=None, waiter=<_thread.lock at remote 0x80075c2a8>, saved_state=None, gotit=False), throwflag=0) at Python/ceval.c:2838
2838                res = call_function(&sp, oparg);

(gdb) py-locals
self = <Condition(_lock=<_thread.lock at remote 0x80075c510>, _waiters=<collections.deque at remote 0x800b0e9d8>, release=<built-in method release of _thread.lock object at remote 0x80075c510>, acquire=<built-in method acquire of _thread.lock object at remote 0x80075c510>) at remote 0x800a6fc88>
timeout = None
waiter = <_thread.lock at remote 0x80075c2a8>
saved_state = None
gotit = False

If you don't want to or can't attach to the running process, you can do the same thing with a core dump:

# gcore 24226
# gdb python core.24226

If you don't want to add the lib directory to the path of python everytime you use GDB, add it to your profile's GDB init script:

cat > ~/.gdbinit <<EOF
import sys

You'll only need to import the module (python import python34_gdb) and you'll be good to go.

More ressources

Bonus problem: loading Debian's libc's debug info on a armhf

I've done the exact same thing on a Beagle Bone Black system running Debian.

Unfortunately GDB was complaining that the stack was corrupt.

# gdb /usr/local/opt/python-3.4.4/bin/python3.4dm core.18513
GNU gdb (GDB) 7.11
Reading symbols from /usr/local/opt/python-3.4.4/bin/python3.4dm...done.
[New LWP 18513]
[New LWP 18531]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Core was generated by `python3.4dm'.
#0  0xb6f7d7e0 in recv () from /lib/arm-linux-gnueabihf/libpthread.so.0
[Current thread is 1 (Thread 0xb6fac000 (LWP 18513))]
(gdb) bt
#0  0xb6f7d7e0 in recv () from /lib/arm-linux-gnueabihf/libpthread.so.0
#1  0xb6f7d7d4 in recv () from /lib/arm-linux-gnueabihf/libpthread.so.0
#2  0xb64ae6f8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

A search on the internet indicated that it was because I was missing package libc6-dbg, but that package was installed.

# apt-get install libc6-dbg
Reading package lists... Done
Building dependency tree
Reading state information... Done
libc6-dbg is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

The problem was that using my custom installation directory had GDB look for these files in the wrong place.

(gdb) show debug-file-directory
The directory where separate debug symbols are searched for is "/usr/local/opt/python-3.4.4/lib/debug".

Setting that variable in the init file solves the problem:

cat >> ~/.gdbinit <<EOF
set debug-file-directory /usr/lib/debug
[Current thread is 1 (Thread 0xb6fac000 (LWP 18513))]
(gdb) bt
#0  0xb6f7d7e0 in recv () at ../sysdeps/unix/syscall-template.S:82
#1  0xb68fe18e in sock_recv_guts (s=0xb64ae6f8, cbuf=0x50e2e8 '\313' <repeats 199 times>, <incomplete sequence \313>..., len=65536, flags=0)
    at /tmp/Python-3.4.4/Modules/socketmodule.c:2600
#38 0x00025752 in PyRun_AnyFileExFlags (fp=0x32bcc8, filename=0xb6be5310 "main.py", closeit=1, flags=0xbed1db20) at Python/pythonrun.c:1287
#39 0x0003b1ee in run_file (fp=0x32bcc8, filename=0x2c99f0 L"main.py", p_cf=0xbed1db20) at Modules/main.c:319
#40 0x0003beb8 in Py_Main (argc=2, argv=0x2c9010) at Modules/main.c:751
#41 0x000208d8 in main (argc=2, argv=0xbed1dd14) at ./Modules/python.c:69


Subscribe to Front page feed