How To Set Up Prometheus To Monitor Your Ubuntu Server

15th March, 2016 by

There are many packages for setting up server monitoring and logging your dedicated Bare Metal or Virtual Server. One of the newer monitoring applications is Prometheus.

It was designed by SoundCloud as a tool to monitor their systems and features a time series database with multi-dimensional data storage coupled to a powerful query language to enable system administrators to define their metrics and generate reports.

Prometheus consists of a couple of different elements, the Prometheus server itself and the Node Exporter.

Getting Started With Prometheus

Unfortunately whilst Prometheus is included in the repositories for Ubuntu 16.04, as that isn’t released yet we’ll need to be installing it from the binary releases on GitHub.  So first we’ll need to get the release from the Prometheus GitHub repository and extract it.

    mkdir ~/Prometheus
    cd ~/Prometheus
prometheus-0.17.0.linux-amd64.tar.gz    mkdir server
    cd server
    tar -xvzf ../prometheus-0.17.0.linux-amd64.tar.gz

This then creates a directory named after the archive with the Prometheus server files inside.  So we’ll need to change to this directory to test it.

    cd prometheus-0.17.0.linux-amd64
    ./prometheus -version 

Given that everything went fine so far you should have an output from Prometheus with its version and the version of Go it is using.  As Prometheus was designed with the primary purpose of monitoring web services it doesn’t natively monitor your local systems.  This is where the Node Exporter part comes in, this takes the information from your system and passes it into Prometheus.  So the next step is to download and install the Node Exporter:

    cd ~/Prometheus
    mkdir node_exporter
    cd node_exporter
    tar -xvzf ../

Now once started the node exporter will run in the background and export the metrics to a web interface that can be used to capture them.  To make it tie in with how services are normally handled on Ubuntu we can create an Upstart script to start the service at bootup, so first, create a link for the node exporter in /usr/bin

    sudo ln -s ~/Prometheus/node_exporter/node_exporter /usr/bin

Next, we need to create the upstart script, so open it as below, note that I’m using nano to edit the file here, other text editors are available:

     sudo nano /etc/init/node_exporter.conf

Then paste the following lines into that file:

    #Prometheus Node Exporter Upstart script
    start on startup
    end script

That was a nice simple script. Save and exit and the service command can now be used to start and stop the node exporter as required.

    service node_exporter start

You can check that it works by connecting to your server on port 9100 with a web browser and looking at the metrics file, this will display loads of metrics about your server that Prometheus will log.

Now the next thing to do is get Prometheus listening to the metrics from the node exporter, for this the first thing that is needed is to create a configuration file for Prometheus.

    sudo nano /etc/prometheus.yml

Now add the following config:

    - job_name: “node”
      scrape_interval: “15s”
    - targets: [’localhost:9100’]

Save and exit the file and all should be ready to go.  We’ve called the job name here “node” as that picks up some default configuration for the node exporter and saves a lot of time filling in the configuration file.

Testing If Monitoring Is Working

Now the moment of truth, if you start Prometheus it should pick up the config file and start up without any errors.  So that’s easily done by running the main Prometheus executable:

    prometheus -config.file /etc/prometheus.yml

You should see some status messages scroll up the screen, the important ones to note are that there’s one that says “Loading configuration file prometheus.yml”, “Target manager started.” and “Listening on :9090”.  If you can see all of those and there are no errors then all should be working well.  You should now be able to connect to port 9090 on your server to see your metrics.

With all this working you can then use ctrl+c to stop Prometheus from running as logging in to your server and manually running it every time the server reboots isn’t very practical.  This means the next step is to create an upstart file for Prometheus itself.  First, we’ll create a link for Prometheus in /usr/bin as before:

    ln -s ~/Prometheus/server/prometheus-0.17.0.linux-amd64/prometheus /usr/bin

Then open the upstart configuration file for Prometheus:

     nano /etc/init/prometheus.conf

With the Upstart configuration file open, paste in the following text.

     #Prometheus Upstart Script
     start on startup
       /usr/bin/prometheus -config.file /etc/prometheus.yml > /var/log/prometheus.log 2>&1 &
     end script

Now you can start Prometheus as a service:

    service prometheus start

The output from Prometheus is now logged to the /var/log/prometheus file, and Prometheus is now running in the background logging your server’s metrics, it will also start up when your server reboots.  If you browse around the Prometheus web interface you should see it monitoring your server’s status and also a selection of graphs detailing showing that data.

(Visited 354 times, 2 visits today)