devel.reinikainen.net

Table of contents

1. Introduction

1.1. Concepts

1.2. Revision history

1.3. Copyright

1.4. Disclaimer

1.5. Acknowledgments and Thanks

2. Tough choices

2.1. Choosing a server platform

2.2. Choosing a database backend

3. Theory

3.1. Account theory

3.2. Partitioning theory

3.3. Security theory

3.4. Directory structuring theory

4. Theory in practice

4.1. FreeBSD Installation

4.2. Jail network setup

4.3. Creating the jails

4.4. Updating ports

4.5. Jailing MySQL

4.6. libnss-mysql Installation

APPENDIX A GNU Free Documentation License

1. Introduction

1.1 Concepts

My objective is to put together a highly secure and usable server
environment, that uses a SQL backend as much as possible for
“everyday” routines, such as authentication, as well as storage for user
account information. My aim was also to do all this with a very low
budget, because I really didn’t have a lot of money to spend. I’m well aware
that you could do most of this by just using the ports collection, but I
prefer to compile evrything by hand; it gives me more control over the
process.

While there are some very good directory access protocols, such as
LDAP, I’ve always preferred SQL databases, due to the fact that they are
relatively widely supported and do an excellent job at storing and
retrieving data.

The basic assumption here is that “everything is insecure by default”.
One might say this is overly paranoid, but sometimes that’s a good
thing.

If you’re going to attempt a similar setup (of pure madness!), I’ll
assume that you have knowledge of compiling software from source,
installing FreeBSD, how jails work in FreeBSD and basic TCP/IP networking.

I am by no means a guru of any of the softwares mentioned here, and therefore
suggestions and comments are always welcome.

Throughout this document, you will see either hash symbols (#) or dollar signs
($) in front of the commands. These indicate whether the command should be run
as a regular user or with superuser privileges (i.e. root). Do not include these
in the commands! And by the way, copy-pasting commands is not such a great
idea. I also provide my configuration files here for reference only.
Please do not just copy them over to your server without fully understanding
what they do. Additionally, please do not use the authors of this documents
as a technical support, which we are most certainly not. Instead, you should
consult the respective softwares’ manual pages, documentations, forums and
mailing lists.

For brevity’s sake, I will use “e.g.” as a shorthand for “for instance”, and
“i.e.” as a shorthand for “that is” or “in other words” throughout this
document.

1.3. Copyright

Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.2
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
A copy of the license is included in the section entitled “GNU
Free Documentation License”.

1.4. Disclaimer

Use the information in this document at your own risk. I disavow any potential
liability for the contents of this document. Use of the concepts, examples,
and/or other content of this document is entirely at your own risk.

All copyrights are owned by their owners, unless specifically noted otherwise.
Use of a term in this document should not be regarded as affecting the
validity of any trademark or service mark.

Naming of particular products or brands should not be seen as endorsements.

You are strongly recommended to make a backup of your system before major
installation and should make backups at regular intervals.

1.5. Acknowledgments and Thanks

Thanks to everyone who gave comments as I was writing this.

2. Tough choices

2.1. Choosing a server platform

I used to prefer Linux as a server platform, that is, until the day I
came in contact with FreeBSD. While Linux is not a bad alternative, some areas
of it are still very under-developed. I once tried setting up quotas on an
ext3 filesystem and a stable 2.4.x series kernel, only to find out that you
would have needed plenty of kernel patches at the time. Not an option in my
books.

Windows was never an option for me, due to the fact that it can be very
inflexible on some areas. The stability I won’t even bother to go in on here.

While Linux still remains my number one workstation operating system, I’ve
come to think of FreeBSD as the ultimate low-budget server operating system.
It offers rock-solid stability most of the time, has really nice security
features such as jailcells and kernel security levels, and has a bunch of
useful performance-enhanching tools, such as the Vinum volume manager, which
implements software RAID.

I was also considering OpenBSD, which is said to be “secure by default”.
OpenBSD does however not include the ‘jail’ functionality (other than ‘chroot’,
that is). OpenBSD does, however, include a lot of seurity patches, claming that
a ‘jail’-like functionality is not needed in this case. I feel, that the ‘jail’
functionality is not equal to security, but more of a fail-safe against
configuration error or coding mistakes, which could quite possibly happen.
Unfortunately, OpenBSD lacks functional PAM and nsswitch implementations, thus
making it hard if not impossible to authenticate against a SQL database.

In my books, NetBSD falls between OpenBSD and FreeBSD, not really offering
anything new.

2.2. Choosing a database backend

This is most certainly not an easy choice to make, as a reliable database
backend is the foundation for the server, and it’s very difficult to change
the backend when the system is running a production environment.

At the moment, MySQL seems to have the best support of the SQL databases.
MySQL is also relatively easy to setup, but at the same time a bit more
limited than, for example, PostgreSQL. In an ideal situation, the MySQL
server should be replicated synchronously, but unfortunately – while such
solutions do exist – that is a solution that comes at a quite heavy price
tag.

PostgreSQL offeres open-source synchronous replication through their
Postgres-R project.

However, there are several other aspects one must consider. One of these
is software support. One must remember, that while for example the Exim MTA
supports both MySQL and PostgreSQL, the Qpopper POP3 daemon can only be patched
for MySQL support. Of course, at this point, it might be good to check out the
competition, in order to see if someone else has implemented support for the
database backend you’re going to use.

3. Theory

3.1. Account theory

The basic services I wish to offer are the following:

• FTP
• HTTP
• SSH
• SQL
• SMTP
• POP3
• IMAP4

FTP is mostly here for legacy reasons only. I’ve been meaning to replace it with SSH2’s SFTP for a long time, but now that you can do TLS/SSL over FTP, the FTP service seems to die hard. In a high-security server, TLS/SSL is the ONLY way to properly do FTP, if it’s non-anonymous.

These services will all need some sort of user accounts. Organising the accounts in a reasonable way is a challenging task of it’s own. As my focus is on SQL database backend, I wanted to store all user accounts in a SQL server. I did some research on some PAM (Pluggable Authentication Module) solutions, but discovered, that they were too inflexible for a solution such as this, due to the fact that they only provide an authentication method, whereas I wanted to get rid of the whole flat-file /etc/users and /etc/passwd system. I stumbled upon NSS modules while searching for a solution, and found libnss-mysql, which is perfect for the task.

The problem with flat-file /etc/users and /etc/passwd is that it’s difficult to replicate the user base to another server. This problem is easily circumvented when using SQL replication with NSS-level authentication and account management. However, this leads to another problem. What happens if the SQL server is unreachable? The whole account system will become unusable, effectively crippling the offered services; hence my need for a replicated SQL server environment.

I have divided the accounts the above services require into four basic account levels:

Jailed system accounts:

• Provide shell access (SSH) to website files (i.e. webspace).
• Provide access to SQL server.
• Provide quotas for website files.
• Stored in MySQL using libnss-mysql.

MySQL accounts:

• Provide access to MySQL databases.
• Stored in MySQL’s internal table.

Jailed FTP accounts:

• Provide access to webspace.
• Same as system accounts.

Mail accounts:

• Provide access to mailboxes (both via POP3 and IMAP4).
• Provide access to SMTP.
• Provide quotas for maildirs.
• Stored in a custom SQL table.
• One mail login/mail account, no system accounts.

From the SQL point of view, the above setup will generate three tables; one for authenticating system (SSH/FTP) users, one for authenticating mail (SMTP/POP3/IMAP4) users, and finally one, which actually already exists in the default MySQL installation – the internal MySQL user database.

3.2. Partitioning theory

By placing /tmp and /var/log on their own partitions, we can easily avoid ‘log-owerflow’ attacks. In these types of attacks, the idea is to make attacked deamon fill up the disk partition with logs, simply by submitting requests to it in a rapid pace – for exmaple – each second, for an extended amount of time. At first I though about creating separate partitions for all jailed /tmp and /var/log directories, in the following fashion:

Mount point
swap
/
/usr
/tmp
/var/jail/www_subsystem/tmp
/var/jail/www_subsystem/var/log
/var/jail/mail_subsystem/tmp
/var/jail/mail_subsystem/var/log
/var/jail/sql_subsystem/tmp
/var/jail/sql_subsystem/var/log
/var/jail/bnc_subsystem
/var/log
/var

However, the clear disadvantage of this approach is the scattered log files and – obviously – the need for multiple, relatively small partitions. Also, managing this many partitions using bsdlabel is cumbersome. After some researching, I found that the -l argument of syslogd would do the trick, thus allowing me to use the following partition layout:

Mount point
swap
/
/usr
/tmp
/var/log
/var

The root filesystem of the host system should be placed on it’s own partiton, due to the fact that any media errors that may occur during disk I/O to user files will corrupt the filesystem containing vital system files as well. One might even go as far as to create separate root partitions for all jails as well, but I think that’s overkill.

3.3. Security theory

I have divided my server setup into several parts, which I will call subsystmes. Each subsystem is actually a FreeBSD jailcell, running as few services as possible. Ideally, the jail/service ratio would be 1:1, but this is not possible in practice (read below why). Creating the jailcells is quite a bit of work, but pays off in terms of added security. The basic subsystems I’ll need for the above services are as follows:

Webserver subsystem

This is by far the largest subsystem, due to the fact that many of the daemons need fileystem access to the same files. The daemons I need inside this jailcell are as follows:

• Apache HTTPd. The HTTPd needs to have filesystem access to the files it’s serving. Apache is in my opinion pretty much the de facto standard in today’s Internet. Now that the HTTPd version 2 is quite stable, I decided to upgrade to that, and while I was at it, add support for a few more scripting languages. Communication with SQL is done via TCP.
• Tomcat. As Java has begun to emerge from the deep deapths of Sun Microsystems to a usable product, I wanted support for JSP as well. For this task I chose Tomcat, from the Apache Jakarta team. The Apache HTTPd and Tomcat play quite nicely together.

• PHP4. I do most of my web development in PHP, thus this is a must for me. PHP will run as an Apache module, and the module must reside somewhere where the HTTPd daemon finds it. The PHP language has native SQL support.
• Perl. I use Perl quite extensively as the ‘glue’ that links all my services together. It also provides support for CGI scripts.
• PureFTPd. As the FTP daemon also needs access to the actual files, this must also be placed inside this same jailcell.
• Quota. This must be run inside this jailcell, as the FTP server must be subject to quota, in order to prevent the users from running amok with all the free space. While for example ProFTPd has a separate mod_sql module that provides quotas, it’s irrelevant to my setup, as I can simply use FreeBSD’s internal quotas, thanks to the NSS authentication system.
• AwStats. I use AwStats to build the web site statistics. AwStats need access to the web server’s log files, so in that goes as well. To build the statistics automatically, I’ll also need cron. Communication with the SQL server is done over TCP using my AwStatsSQLBatch Perl script, which in turn uses Perl’s DBI.

The setup of this subsystem is documented in detail in my Mini How-To.

Mail subsystem

This is the second largest subsystem right after the webserver subsystem. The separation of the webserver files and maildirs allows me to specify separate quotas for both. The quota and account settings for mail are stored in a custom SQL table, which the following daemons will manipulate:

• Exim. This is my choice of MTA, as it’s reliable, handles Maildirs and both MySQL and PostgreSQL flawlessly. It needs filesystem access to the Maildirs. Exim provides the SMTP service.
• Courier-IMAPd. Courier-IMAPd provides the IMAP4 service and has native MySQL and PostgreSQL support. The IMAP daemon needs filesystem access to the Maildirs.
• . Provides the POP3 service. The POP3 daemon requires filesystem access to the Maildirs. Qpopper can be patched for MySQL and Maildirs, Courier-IMAPd’s POP3 daemon has native MySQL and PostgreSQL support.

The setup of this subsystem is documented in detail in my
Exim, Amavisd-new, Courier-IMAP with TLS+MySQL Auth Mini How-To.

Amavisd-new subsystem

Amavisd-new is placed in it’s own chroot on the host system (no need for a full jail), as communication with Exim is done via SMTP and communication with SQL via TCP. Amavisd-new has native SQL support though Perl’s DBI. The anti-virus scanner(s) must also reside inside this chroot. I will use F-Prot as my anti-virus scanner, because it’s simple, gets the job done and mostly because I happen to have a license for it.

The setup of this subsystem is discussed also in my .

SQL subsystem

This subsystem will contain the heart and brains of the server, namely the SQL server. Communication with the ‘outside world’ is done via TCP.

The one big problem with MySQL is, that it hasn’t got quota support. However, since MySQL lives inside the UFS file system, you can implement quotas by chown’ing databases to appropriate users and activating user quotas for the disk partition which the MySQL databases live on. You don’t need to worry about this leaving your database in an
inconsistent state when the quota is exceeded, as MySQL should then behave as in a “disk full” situation. The exact behaviour is documented in the MySQL manual.

IDENTIFIED BY 'goodsecret' REQUIRE SSL;

Note the IP address! Even though we are connecting from 10.0.0.5 (the host system), MySQL sees it as ‘localhost’, as it’s an alias for fxp0. The rest of the command should be pretty self-explanatory.

We’ll need to GRANT access from our webserver subsystem as well:

GRANT SELECT ON auth.* TO


 IDENTIFIED BY 'anothergoodsecret' REQUIRE SSL;

Tip: The ‘id’ command is quite useful for testing the setup.

‘make’ will not work – you need ‘gmake’.

Tags: bsd, linux, sql

This entry was posted on Saturday, October 3rd, 2009 at 11:06 and is filed under Linux/BSD Server Solutions. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.