It was designed by SoundCloud as a tool to monitor their systems and features a time series database with multi-dimensional data storage coupled to a powerful query language to enable system administrators to define their metrics and generate reports.
Prometheus consists of a couple of different elements, the Prometheus server itself and the Node Exporter.
Getting Started With Prometheus
Unfortunately whilst Prometheus is included in the repositories for Ubuntu 16.04, as that isn’t released yet we’ll need to be installing it from the binary releases on GitHub. So first we’ll need to get the release from the Prometheus GitHub repository and extract it.
mkdir ~/Prometheus cd ~/Prometheus wget https://github.com/prometheus/prometheus/releases/download/0.17.0/ prometheus-0.17.0.linux-amd64.tar.gz mkdir server cd server tar -xvzf ../prometheus-0.17.0.linux-amd64.tar.gz
This then creates a directory named after the archive with the Prometheus server files inside. So we’ll need to change to this directory to test it.
cd prometheus-0.17.0.linux-amd64 ./prometheus -version
Given that everything went fine so far you should have an output from Prometheus with its version and the version of Go it is using. As Prometheus was designed with the primary purpose of monitoring web services it doesn’t natively monitor your local systems. This is where the Node Exporter part comes in, this takes the information from your system and passes it into Prometheus. So the next step is to download and install the Node Exporter:
cd ~/Prometheus wget https://github.com/prometheus/node_exporter/releases/download/0.11.0/ node_exporter-0.11.0.linux-amd64.tar.gz mkdir node_exporter cd node_exporter tar -xvzf ../https://github.com/prometheus/node_exporter/releases/download/ 0.11.0/node_exporter-0.11.0.linux-amd64.tar.gz
Now once started the node exporter will run in the background and export the metrics to a web interface that can be used to capture them. To make it tie in with how services are normally handled on Ubuntu we can create an Upstart script to start the service at bootup, so first, create a link for the node exporter in /usr/bin
sudo ln -s ~/Prometheus/node_exporter/node_exporter /usr/bin
Next, we need to create the upstart script, so open it as below, note that I’m using nano to edit the file here, other text editors are available:
sudo nano /etc/init/node_exporter.conf
Then paste the following lines into that file:
#Prometheus Node Exporter Upstart script start on startup script /usr/bin/node_exporter end script
That was a nice simple script. Save and exit and the service command can now be used to start and stop the node exporter as required.
service node_exporter start
You can check that it works by connecting to your server on port 9100 with a web browser and looking at the metrics file, this will display loads of metrics about your server that Prometheus will log.
Now the next thing to do is get Prometheus listening to the metrics from the node exporter, for this the first thing that is needed is to create a configuration file for Prometheus.
sudo nano /etc/prometheus.yml
Now add the following config:
scrape_configs: - job_name: “node” scrape_interval: “15s” target_groups: - targets: [’localhost:9100’]
Save and exit the file and all should be ready to go. We’ve called the job name here “node” as that picks up some default configuration for the node exporter and saves a lot of time filling in the configuration file.
Testing If Monitoring Is Working
Now the moment of truth, if you start Prometheus it should pick up the config file and start up without any errors. So that’s easily done by running the main Prometheus executable:
prometheus -config.file /etc/prometheus.yml
You should see some status messages scroll up the screen, the important ones to note are that there’s one that says “Loading configuration file prometheus.yml”, “Target manager started.” and “Listening on :9090”. If you can see all of those and there are no errors then all should be working well. You should now be able to connect to port 9090 on your server to see your metrics.
With all this working you can then use ctrl+c to stop Prometheus from running as logging in to your server and manually running it every time the server reboots isn’t very practical. This means the next step is to create an upstart file for Prometheus itself. First, we’ll create a link for Prometheus in /usr/bin as before:
ln -s ~/Prometheus/server/prometheus-0.17.0.linux-amd64/prometheus /usr/bin
Then open the upstart configuration file for Prometheus:
With the Upstart configuration file open, paste in the following text.
#Prometheus Upstart Script start on startup script /usr/bin/prometheus -config.file /etc/prometheus.yml > /var/log/prometheus.log 2>&1 & end script
Now you can start Prometheus as a service:
service prometheus start
The output from Prometheus is now logged to the /var/log/prometheus file, and Prometheus is now running in the background logging your server’s metrics, it will also start up when your server reboots. If you browse around the Prometheus web interface you should see it monitoring your server’s status and also a selection of graphs detailing showing that data.