Categories
Software

Installing Graylog 3 in a TrueNAS 12.0 Jail (FreeBSD)

Graylog is an enterprise log aggregation and management framework similar to Splunk. Graylog, the company, is based in Houston, Texas (yay!) and boasts over 40k installations. What am I trying to accomplish by using Graylog at home? Well, my initial requirement is to collect logs from my network routers and incorporate that into my existing network monitoring using InfluxDB and Grafana. I’m a client of Comcast Xfinity and my network performance has really been poor in mid to late 2020. After multiple calls to Comcast and several tech visits, I’ve setup my own monitoring to “show” Comcast that I’m not getting the bandwidth they are billing me for. Log retention is low on my routers and cable modem (data rolls off quickly), a goal of this project is to retain at least 30-90 days of log messages from all network devices.

While I didn’t find this other post until after I wrote my post; I will cross link to this other post on essentially the same thing 😉 https://sign13.wordpress.com/setting-up-graylog-in-a-freenas-jail/

In TrueNAS Core 12.0, use the add Jail wizard

Step 1: Create the Graylog Jail

I’ll setup Graylog in a TrueNAS Core 12.0 Jail which will be running FreeBSD 12.2 release. The jail configuration isolates the software and configuration from other applications running on TrueNAS and it provides a stable environment with a large storage pool for log retention. In the Jails menu on TrueNAS Core 12.0, select the add button and the wizard will prompt you for input to create the Jail. Name your jail graylog and select the FreeBSD software release, then hit Next. I chose DHCP for my Networking configuration which will allow me easy SSH access to the “host”. Select Next and then select Submit to create the Jail. This tutorial will assume you have an understanding of basic Unix and SSH commands. After the wizard completes, you will want to click on the greater-than symbol in the graylog jail row in TrueNAS to show the jail details including the IP address. In this instance my graylog jail is 192.168.1.32 (take note!). In my network router I configured DHCP to give the same IP4 address to the MAC address for the graylog jail and to resolve the DNS hostname of graylog (so I don’t need to remember the IP4 address)

Test that the jail is up by pinging the “host”

rich@eragon:~$ ping graylog
PING graylog.pavlovs.ky (192.168.1.32) 56(84) bytes of data.
64 bytes from graylog.pavlovs.ky (192.168.1.32): icmp_seq=1 ttl=64 time=0.691 ms
64 bytes from graylog.pavlovs.ky (192.168.1.32): icmp_seq=2 ttl=64 time=0.619 ms
64 bytes from graylog.pavlovs.ky (192.168.1.32): icmp_seq=3 ttl=64 time=0.684 ms
64 bytes from graylog.pavlovs.ky (192.168.1.32): icmp_seq=4 ttl=64 time=0.752 ms
64 bytes from graylog.pavlovs.ky (192.168.1.32): icmp_seq=5 ttl=64 time=0.537 ms
^C
--- graylog.pavlovs.ky ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 0.537/0.656/0.752/0.073 ms

Step 2: Setup SSH access and a local user

You will need to ssh into your TrueNAS box and use the iocage or the jexec command to remote into the jail on your first access. This is because you won’t have a local user account yet and the SSH daemon isn’t running. SSH into your TrueNAS (mine is named kidney) and type the JLS command to see the JID number of your jail. This will change with every restart so don’t write it down or memorize it. Use the jexec command to remote in.

kidney% jls
   JID  IP Address      Hostname                      Path
    12                  graylog                       /mnt/PavPool/iocage/jails/graylog/root
kidney% sudo jexec 12 /bin/tcsh 

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

Password:
root@graylog:/ # 

You will notice that you are now in a shell (a tcsh shell if you followed my above instructions) in the jail. Please note bash is not installed yet. You will be logged in as root. Update the packages and install bash first; you will need bash when you create your local user. You will also install the sudo package and configure it. sudo allows non-root users to run privileged commands.

root@graylog:/ # pkg update
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
root@graylog:/ # pkg install bash sudo

Now you need to configure sudo by executing the visudo command. This command will edit the sudoers file to tell the system that your local account is allowed to run priveliged commands. Scroll down in the vi window until you see User Privilege

##
## User privilege specification
##
root ALL=(ALL) ALL

You should add a new row by putting cursor under the row with root like above and hitting the i key to insert. Type your user row as follows:


rich ALL=(ALL) ALL

Substitute username rich for whatever your username is. You may wonder that you haven’t created the user yet and that is ok! We will create the user next. Type SHIFT ZZ (that is hold the shift key and then type Z twice) to save the file.

Then, add your local user to enable easy logins as something other than root (not good to remote into things as root). In this example my local login name is rich with unix id of 1000 on all my boxes including TrueNAS <<< this is important. Type adduser to create the rich user with uid of 1000.

root@graylog:/ # adduser
Username: rich_
Full name: Rich P
Uid (Leave empty for default): 1000
Login group [rich_]: 
Login group is rich_. Invite rich_ into other groups? []: 
Login class [default]: 
Shell (sh csh tcsh bash rbash git-shell nologin) [sh]: bash
Home directory [/home/rich_]: 
Home directory permissions (Leave empty for default): 
Use password-based authentication? [yes]: 
Use an empty password? (yes/no) [no]: 
Use a random password? (yes/no) [no]: 
Enter password: 

Next, edit the /etc/rc.conf file and add this line to the end of the file

sshd_enable="YES"

Next, start the sshd service to test that it works

root@graylog:/ # service sshd start

Now, try to ssh into your TrueNAS jail from your localhost

rich@eragon:~$ ssh graylog
Password for rich@graylog:
Last login: Fri Dec  4 18:11:47 2020 from eragon.pavlovs.ky
FreeBSD 12.2-RC3 7c4ec6ff02c(HEAD) TRUENAS 

Welcome to FreeBSD!

To make it more secure you can add an SSO component or add ssh keys to each of your jails (i.e. no more typing your password). You should have convenient SSH access to your jail. Remember, with great power comes great responsibility. Also remember that you are no longer logging in as root so you’ll need to run sudo on privileged commands. Sudo should be all setup and we’ll test it in the next step.

Step 3: Install Graylog Packages

Type the following to install the graylog, elasticsearch and mongodb packages.

[rich@graylog ~]$ sudo pkg update
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.

[rich@graylog ~]$ sudo pkg install graylog elasticsearch6 mongodb44

Update 14-Dec: Updated post based on comments/feedback. I had made a mistake on the packages I used, which was corrected by a reader.

It may take a few mins to download and install those packages. MongoDB will be used for the graylog configuration only. All log data will be stored in the elasticsearch cluster. These packages are downloaded from FreeBSD fresh ports which is a very extensive software pkg repository. Please note as of this writing in Dec 2020, version 3.3.1 is the latest FreeBSD graylog package available but graylog itself is shipping version 4.0.2 on the graylog.org website.

Step 4: Configure Software

None of the above software will be running after it’s installation. Don’t start the services yet. First, edit /etc/rc.conf to tell TrueNAS to autostart the software when the jail starts. Add these lines to the end of the file.

graylog_enable="YES"
elasticsearch_enable="YES"
mongod_enable="YES"

Next, edit the /usr/local/etc/elasticsearch/elasticsearch.yml file to change a few parameters. Add the following to the bottom of the file and save it.

cluster.name: graylog

Start the mongoDB and the elasticsearch components by typing:

[rich@graylog ~]$ sudo service elasticsearch start

[rich@graylog ~]$ sudo service mongod start

Notice the output of the above service commands, if there are any errors starting the commands it should prompt you. You can check if both services are running by typing a netstat command.

[rich@graylog ~]$ netstat -an | grep -i listen

tcp4       0      0 127.0.0.1.9200         *.*                    LISTEN     
tcp6       0      0 ::1.9200               *.*                    LISTEN     
tcp4       0      0 127.0.0.1.9300         *.*                    LISTEN     
tcp6       0      0 ::1.9300               *.*                    LISTEN     
tcp4       0      0 *.22                   *.*                    LISTEN     
tcp6       0      0 *.22                   *.*                    LISTEN     
tcp4       0      0 127.0.0.1.27017        *.*                    LISTEN  

This command shows which TCP ports are being listened to by server processes like mongo and elasticsearch. You will notice that 9200 and 9300 are elasticsearch. 22 is SSHD and 27017 is mongo. If your output from a netstat is similar then you are doing good so far. Now, we need to configure graylog and then start the service for the first time. The graylog config will be in the /usr/local/etc/graylog directory.

[rich@graylog ]$ cd /usr/local/etc/graylog

[rich@graylog ]$ sudo cp graylog.conf.example graylog.conf

[rich@graylog ]$ sudo cp log4j2.xml.example log4j2.xml   

[rich@graylog ]$ sudo mkdir server 

[rich@graylog ]$ sudo touch server/node-id 

[rich@graylog ]$ cd ..

[rich@graylog ]$ sudo chown -R graylog:graylog graylog

This will setup the initial config directory and change the owner of the files to graylog user and graylog group. That user and group should have been created when you pkg installed graylog. If you don’t have that user or group then go back to the package install step and check you did that correctly. Change directory back to /usr/local/etc/graylog and edit the graylog.conf file and ensure the variables are set as such.

is_master = true
node_id_file = /usr/local/etc/graylog/server/node-id
root_username = admin
root_timezone = UTC
bin_dir = /usr/local/share/graylog
data_dir = /usr/local/var/lib/graylog
plugin_dir = /usr/local/share/graylog/plugin

http_bind_address = 192.168.1.32:9000

message_journal_dir = /usr/local/var/lib/graylog/journal

The above config parameters are scattered throughout the config file. Your http_bind_address will be the jail ip4 address. Ensure that all those directories like the journal directory exist and are owned by graylog:graylog. Next, set the hashed admin password and seed by running these two commands on the command line.

pwgen -N 1 -s 96

echo -n yourpassword | shasum -a 256

Edit the graylog.conf file and follow the instructions to paste the password_seed and root_password_sha2 to set those values and then save the file. There are instructions inline in the conf file on the password seed and root password if you are confused. Make sure the file is only readable to the graylog user and group. Now you can finally start the Graylog service

[rich@graylog ]$ sudo service graylog start

[rich@graylog ]$ sudo tail -f /var/log/graylog/server.log

Tail the /var/log/graylog/server.log file to ensure it starts correctly and no fatal errors. It takes a minute or two to startup.

Step 5: Test Jail Restart to Ensure Auto-Launch

If graylog started up correctly, go ahead and stop the jail in the TrueNAS Jails menu. Click the Edit Jail menu item under the graylog jail properties and in the Basic Properties sheet you will check the Auto-start checkbox and click save. Next you can click the start button on the graylog jail and wait a few min for everything to start up. You then should be able to access the graylog UI at http://<your ip address>:9000 (please note that you need to fill in your ip address and it will be different than mine.)

Further tutorials will include hardening + adding SSL and configuring syslog inputs.

By Rich

Lover of science, technology, ice hockey and the outdoors. Houston is home.

10 replies on “Installing Graylog 3 in a TrueNAS 12.0 Jail (FreeBSD)”

couple bugs I noticed here:

graylog 3.x doesn’t support elasticsearch 7.x. For graylog 3.x, need to install elasticsearch 6.8.x. I also noticed that the “mongod” package didn’t exist. In a FreeBSD 12.2-p2 jail, the pkg install command would be:

pkg install graylog elasticsearch6 mongodb44

The elasticsearch.yml config is a little off it seems as well:

Caused by: java.lang.IllegalArgumentException: the [action.auto_create_index] setting value [false] is too restrictive. disable [action.auto_create_index] or set it to [.watches,.triggered_watches,.watcher-history-*]

Simply omitting that option seems to work.

Cheers!

Thanks, I’ll review and amend the instructions. I spend a portion of the weekend getting graylog pipeline processing for my Wifi router logs so there is much to learn!

I’ve made several edits and linked to another wordpress post that was very similar. It’s funny cause when I was writing this blog post originally I was like, “which version of elasticsearch did I use?” and I guessed, lol. It’s always best to do the work, write the steps on how the work was done and then do the work again in a fresh jail to verify the steps. I didn’t do the last part this time. Thank for finding the errors in my write up!

I had to set a password with “echo -n | shasum -a 256.
Otherwise I would get a POST error on the initial login.
Using Geaylog 3.3 on FreeBSD 13.0-CURRENT

Hi, thanks for the nice tutorial.
My server is up and running but I don’t get the password part.
What’s the password and have to use in order to login on graylog webui? I assume it not the output of “pwgen -N 1 -s 96”, is it?

Best

Thanks. I’ve updated the text in the article slightly to make it more clear. The first command which you can run in FreeBSD or on Linux is to create a password seed. Once you run that you can copy and paste the output into the graylog.conf file. The SECOND commands is for your root password and you are running SHA2 to hash that password. You will save that hash in your graylog.conf file. Then, when you login to the web interface you will type admin (that’s the root login) and then type your password. Graylog will hash what you typed in and compare with what’s in the config file.

Hi,
Thanks a lot for this page!

I just installed in a Truenas 12 jail with the updated Graylog 4.1 package and discovered that the instructions need tweaking. The log file complains about not finding /usr/local/var. There is actually no /usr/local/var so I put the message journal under share which existed already. So mkdir /usr/local/share/graylog/journal, chown -R graylog:graylog that same directory, and lastly in graylog.conf: message_journal_dir = /usr/local/share/graylog/journal

I also left the default for the data directory: data_dir = data

I haven’t got around to create any streams yet, but at least now the web interface loads and can be logged into.

Best regards,
Peter

Good question! I created this before that was available. If I was to do it again now I would probably try the plug-in to see if it works for you before trying this manual approach. Please note that older versions of Graylog are impacted by the Log4j vulnerability. The FreeBSD package management system is updated with the version of Graylog that patches the vulnerability. Please see Graylog’s log4j page to understand which of the versions are safe to use. The FreeBSD pkg, which I’m assuming the plugin uses, is only for the opensource version of Graylog (the enterprise jars/modules are missing). With some hacking you can easily add them in; I was thinking about adding a post about my hacking in this space.

BTW, over the holiday’s I got myself a cheapo used Dell from Goodwill and wiped it and installed Proxmox on it. I’m thinking about moving most of my jails and services over to Proxmox from TrueNAS. Now that I have a bigger homelab this allows me to dedicate infra for better performance.

Leave a Reply to RichCancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.