Linux palvelimena – Homework 7 – Server optimization

Week 7 – Server optimization

The seventh and final week had us install and test Varnish, a reverse proxy.

Course homepage: http://terokarvinen.com/2012/aikataulu-%E2%80%93-linux-palvelimena-ict4tn003-3-ja-ict4tn003-5-kevaalla-2012

Assignment

Install Varnish
Try Firebug for Firefox

Environment

I’m using Ubuntu 12.04 @ Virtualbox. Username is ‘antero’. Virtualbox has access to 2 CPU cores (of my Core 2 Quad @ 3.4GHz) and 2048MB of RAM.

Sources

Everything done in this post is based on our previous class (see course homepage above).

Varnish

Varnish is a very useful reverse proxy, which can greatly improve web server performance. We make Varnish listen to TCP port 80, so that it will serve the requested websites instead of Apache. (Varnish does get them from Apache but greately reduces load.) Pages are essentially served as static websites thanks to this, which makes everything *very* fast.

I’m picking things up from last week, when I installed WordPress. https://a1100320.wordpress.com/2012/10/01/linux-palvelimena-homework-6-wordpress/

Prerequisite steps

First I’m going to change Apache’s listening port to 8080.

sudo nano /etc/apache2/ports.conf

If you’ve configured any virtual hosts, be sure to edit their files in /etc/apache2/sites-available like so: <VirtualHost *:8080>

I restart apache2 and try to access my WordPress.

Oh no, it doesn’t appear to work anymore. Luckily installing Varnish will fix that.

Installing Varnish

The metapackage ‘varnish’ installs libvarnishapi1 and varnish, which are exactly the things I need.

sudo apt-get install varnish

I need to edit /etc/default/varnish. One of the configurations therein is uncommented by default, which is what I’ll use. DAEMON_OPTS has numerous flags and values. The only one which needs to be changed is -a, since we don’t want varnish to listen to port 6081 but port 80.

sudo nano /etc/default/varnish

See above for the changes. After that, I need to restart varnish.

sudo service varnish restart

Varnish is now set up and WordPress works once more. We’ll get to actual performance tests later, but first…

Firebug

Time to install Firebug, the Firefox addon. I open Firefox and go to Tools -> Add-ons. (Or just press Shift-Ctrl-A). There’s a search field in the upper right corner:

Typing ‘firebug’ finds the addon I’m after.

Once the addon is installed, I go to my WordPress’ “Hello world” post while having Firebug open, and take a look at how fast various parts of the page load.

HTML generally seems to load quite fast:

Images take a bit longer: (Also, what’s 0.gravatar.com?)

AB

Apache Bench is a tool for testing web server performance. It can create huge amounts of traffic, so be sure to try it out on your localhost only, unless you want to appear like you’re attempting to ddos attack a website.

I’ll create a small HTML page for testing purposes, called fast.html, whose contents are:

<!doctype html>
<html>
<head>
<title>Lightning-fast static webpage</title>
<meta charset="utf-8" />
</head>
<body>
This page ought to be very lightweight!
</body>
</html>

I also have index.php, which has phpinfo(); on it. So essentially I’ll run the same ab command for the sites at port 80, which means Varnish, and 8080 which means plain Apache.

PHP with Varnish

ab -n 1000 -c 10 http://localhost/~antero/index.php

No failed requests. ~1590 requests per second, longest request 46ms.

Static page with Varnish

ab -n 1000 -c 10 http://localhost/~antero/fast.html

No failed requests, ~3500 requests per second, longest request 15ms.

Php without Varnish

ab -n 1000 -c 10 http://localhost:8080/~antero/index.php

No failed requests, ~700 requests per second, longest request ~300ms.

Static page without Varnish

ab -n 1000 -c 10 http://localhost:8080/~antero/fast.html

No failed requests, ~1790 requests per second, longest request 42ms.

Analysis

Here’s the data for easy comparison, with rounded numbers:

index.php Varnish No failed requests. 1600 requests per second, longest request 50ms
index.php Apache  No failed requests, 700  requests per second, longest request 300ms

fast.html Varnish No failed requests, 3500 requests per second, longest request 15ms
fast.html Apache  No failed requests, 1800 requests per second, longest request 40ms

Varnish was consistently faster, but moreso when it comes to PHP. Static pages are rather light to begin with, but still provides a considerable boost in performance.

Best for last: AB meets Varnish meets a WordPress post

I thought it’d be interesting to see a proper, conclusive result, so I tried benchmarking my WordPress’ performance with and without Varnish. I modify /etc/default/varnish and change its listening port, and temporarily switch off Varnish off.

(Changing the listening port might not be necessary, now that I think of it, seeing as it won’t be listening to anything anyway.)

sudo service varnish stop

And modify /etc/apache2/ports.conf to its original state, so that WordPress works normally with port 80, and restart Apache:

sudo service apache2 restart

To clarify: If one doesn’t stop Varnish first, port 80 won’t be available for apache2 to listen.

WordPress without Varnish

ab -n 1000 -c 10 http://localhost/~antero/wordpress/2012/10/hello-world/

Still no failed requests, but the test tooks ages. Requests per second 10, longest request 1998ms. Note: time taken for tests 98 seconds!

WordPress with Varnish

I re-enable Varnish and run the test again.

ab -n 1000 -c 10 http://localhost/~antero/wordpress/2012/10/hello-world/

No failed requests. 1333 requests per second. Longest request 51ms. Time taken for tests 0.75 seconds.

No Varnish
Requests per second: 10    Longest request (ms): 1998  Time taken for tests: 98 seconds

Varnish
Requests per second: 1333  Longest request (ms): 51    Time taken for tests:  0.75 seconds

In a nutshell: Varnish results in roughly 100X performance when it comes to heavier sites like WordPress, which use SQL and PHP.

Posted in Linux palvelimena ICT4TN003-4 | Leave a comment

Linux palvelimena – Homework 6 – WordPress

Week 6 – WordPress

The sixth week had us install and configure WordPress

Course homepage: http://terokarvinen.com/2012/aikataulu-%E2%80%93-linux-palvelimena-ict4tn003-3-ja-ict4tn003-5-kevaalla-2012

Sources used:

http://codex.wordpress.org/Using_Themes

http://wordpress.org/support/topic/solved-permalinks-working-in-apache2-ubuntu-1010

Assignment

Use a live CD for a fresh experience

Install WordPress
Enable PermaLinks
Install a theme of your choosing
Install a plugin of your choosing

Environment

I’m using Ubuntu 12.04 @ Virtualbox. I loaded an old snapshop in order to emulate a clean installation from a live CD. (I opted for Virtualbox to get a shared clipboard between Ubuntu and Windows.) Username is ‘antero’ during the whole exercise, seeing as how I didn’t create a separate user for WordPress.

Prerequisite steps

Time to install LAMP. Install apache2 (with mods userdir, php5 and rewrite), mysql-server and php5 and phpmyadmin. For this week’s report, I’m going to omit the usual installation logs for better readability, and concentrate on essential steps. Also, I decided to drop the “let’s/we” narrative. Also, I enable ufw before installing any servers.

Long story short:

antero@VirtualBox:~$ sudo ufw enable

antero@VirtualBox:~$ sudo apt-get install apache2
antero@VirtualBox:~$ sudo apt-get install php5

antero@VirtualBox:~$ sudo a2enmod userdir
antero@VirtualBox:~$ sudo a2enmod php5

antero@VirtualBox:~$ sudo service apache2 restart

antero@VirtualBox:~$ sudo apt-get install mysql-server
antero@VirtualBox:~$ sudo apt-get install phpmyadmin

About phpmyadmin:

I think I ought to enable php for userdir, by commenting out the appropriate lines in /etc/apache2/mods-available/php5.conf:

<IfModule mod_php5.c>
    <FilesMatch "\.ph(p3?|tml)$">
        SetHandler application/x-httpd-php
    </FilesMatch>
    <FilesMatch "\.phps$">
        SetHandler application/x-httpd-php-source
    </FilesMatch>
    # To re-enable php in user directories comment the following lines
    # (from <IfModule ...> to </IfModule>.) Do NOT set it to On as it
    # prevents .htaccess files from disabling it.
    #<IfModule mod_userdir.c>
    #    <Directory /home/*/public_html>
    #        php_admin_value engine Off
    #    </Directory>
    # </IfModule>
</IfModule>

Time to log in to phpmyadmin with root and the password given during install. Then, Privileges -> Add new user. I enter the information shown in the following screenshot, and make sure that “Create database with same name and grant all privileges” is checked. After clicking “Create user” I close phpmyadmin.

Provided I haven’t made a mistake somewhere, I should be all set for installing WordPress.

Installing WordPress

I download the latest WordPress archive:

antero@VirtualBox:~/public_html$ wget http://wordpress.org/latest.tar.gz

I extract the downloaded file with the built-in archive manager, File Roller, in the graphical file manager. It extracts to public_html/wordpress, which I deem satisfactory. If I recall correctly, the tar.gz includes proper permissions, so I won’t have to modify those. We’ll see.

antero@VirtualBox:~/public_html/wordpress$ ls
index.php        wp-admin              wp-config-sample.php  wp-links-opml.php  wp-settings.php
license.txt      wp-app.php            wp-content            wp-load.php        wp-signup.php
readme.html      wp-blog-header.php    wp-cron.php           wp-login.php       wp-trackback.php
wp-activate.php  wp-comments-post.php  wp-includes           wp-mail.php        xmlrpc.php

I moved the wp-config-sample.php to wp-config.php, which in retrospect was less handy than using cp.

antero@VirtualBox:~/public_html/wordpress$ mv wp-config-sample.php wp-config.php

I then filled in the proper values:

Time for first run, then. I open http://localhost/~antero/wordpress. Lo and behold:

I call my site ‘Fenix Funk’, username ‘admin’ and the password is omitted in this report. I also enter my secondary e-mail address and make sure “Allow search engines to index this site” is checked.

Result:

Enabling nice permalinks

Permalink Settings are easily found, but WordPress says my .htaccess isn’t writable. Instead, it gives me the corresponding settings for the “Month and name” style, so let’s add those.

antero@VirtualBox:~/public_html/wordpress$ nano .htaccess

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /~antero/wordpress/
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /~antero/wordpress/index.php [L]
</IfModule>

Well, that didn’t do anything. Googling helps, though. According to http://wordpress.org/support/topic/solved-permalinks-working-in-apache2-ubuntu-1010 I have to a2enmod rewrite (and restart apache2)

antero@VirtualBox:~/public_html/wordpress$ sudo a2enmod rewrite
antero@VirtualBox:~/public_html/wordpress$ sudo service apache2 restart

That’s better! The “Hello World!” post now has a pretty permalink. Witness the magnificence:

Themes

According to http://codex.wordpress.org/Using_Themes I should download themes from http://wordpress.org/extend/themes and save them to wp-content/themes/themename and then enable it from the WordPress control panel. A theme called Montezuma looks nice. Let’s give that one a go.

antero@VirtualBox:~/public_html/wordpress/wp-content/themes$ wget http://wordpress.org/extend/themes/download/montezuma.1.1.3.zip

Previously used method for extracting the archive applies here. The Montezuma theme is now visible in the WP appearance settings:

The next screenshot will have it on display, when we take a look a plugin.

Plugins

Plugins are installed like themes. I found an amusing plugin called “Stargate Quotes” at http://wordpress.org/extend/plugins

According to its description:

This plugin will randomly display quotes from the Stargate franchise on every page of your user dashboard. Based on the Hello Dolly plugin by Matt Mullenweg.

I don’t particularily need it, but it’s probably OK for testing plugins in general.

antero@VirtualBox:~/public_html/wordpress/wp-content/plugins$ wget http://downloads.wordpress.org/plugin/stargate-quotes.1.5.zip

I’ll take a look whether it shows up in the WP settings after extracting.

Yes! Let’s activate it.

Bummer, there’s something wrong with it.

Let’s just activate the included Hello Dolly and call it a day for now:

Posted in Linux palvelimena ICT4TN003-4 | Leave a comment

Linux palvelimena – Homework 5 – Apache

Week 5 – Apache

The fifth week had us install and configure the apache2 web server.

Course homepage: http://terokarvinen.com/2012/aikataulu-%E2%80%93-linux-palvelimena-ict4tn003-3-ja-ict4tn003-5-kevaalla-2012

Sources used:

http://www.linuxquestions.org/questions/linux-networking-3/turn-off-disable-php-for-some-virtual-hosts-in-apache-412577

http://httpstatus500.com

http://en.wikipedia.org/wiki/List_of_HTTP_status_codes

Assignment

Create a name based virtual host.

Create three of them:
– One with PHP enabled
– One with PHP disabled
– One with a setting of your choosing (see apache2-doc)

Cause the following http status codes to appear in the log files:  200, 404, 403, 500. Analyze the lines containing them. (Bonus: 304).

Voluntary bonus: SSL/TLS in Apache. (Challenging bonus exercise: acquire a free, official certificate.)

Environment

Ubuntu 12.04 @ Virtualbox, username: antero, computer: VirtualBox

Creating name based virtual hosts

Time to get started.

Installing required packages

Let’s install Apache and PHP first.

antero@VirtualBox:~$ sudo apt-get install apache2
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  apache2-mpm-worker apache2-utils apache2.2-bin apache2.2-common libapr1 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap
Suggested packages:
  apache2-doc apache2-suexec apache2-suexec-custom
The following NEW packages will be installed:
  apache2 apache2-mpm-worker apache2-utils apache2.2-bin apache2.2-common libapr1 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap
0 upgraded, 9 newly installed, 0 to remove and 0 not upgraded.
Need to get 1,817 kB of archives.
After this operation, 5,220 kB of additional disk space will be used.
Do you want to continue [Y/n]? y
...
Setting up apache2 (2.2.22-1ubuntu1) ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
antero@VirtualBox:~$

Then to install PHP. I’ll just get the metapackage ‘php5’, which installs the required lib for apache2.

antero@VirtualBox:~$ sudo apt-get install php5
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  apache2-mpm-prefork libapache2-mod-php5 php5-cli php5-common
Suggested packages:
  php-pear php5-suhosin
The following packages will be REMOVED:
  apache2-mpm-worker
The following NEW packages will be installed:
  apache2-mpm-prefork libapache2-mod-php5 php5 php5-cli php5-common
0 upgraded, 5 newly installed, 1 to remove and 0 not upgraded.
Need to get 6,359 kB of archives.
After this operation, 17.3 MB of additional disk space will be used.
Do you want to continue [Y/n]? y
...
Setting up php5 (5.3.10-1ubuntu3.4) ...
Setting up php5-cli (5.3.10-1ubuntu3.4) ...

Creating config file /etc/php5/cli/php.ini with new version
update-alternatives: using /usr/bin/php5 to provide /usr/bin/php (php) in auto mode.
antero@VirtualBox:~$

This might be an apt time to enable userdir and php5 in Apache.

antero@VirtualBox:~$ sudo a2enmod userdir
Enabling module userdir.
To activate the new configuration, you need to run:
  service apache2 restart
antero@VirtualBox:~$ sudo a2enmod php5
Module php5 already enabled
antero@VirtualBox:~$ sudo service apache2 restart
 * Restarting web server apache2                                                                                                              apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1 for ServerName
 ... waiting apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1 for ServerName
                                                                                                                                       [ OK ]
antero@VirtualBox:~$

Preliminary steps

In order to create the virtual hosts, we need to do prepare a bit.

I’m going to create hosts called bansh.ee, haunt.er and marr.ow. I want them to point to localhost, I’ll add them to /etc/hosts, the result of which looks like this:

127.0.0.1       localhost
127.0.1.1       VirtualBox
127.0.0.1       bansh.ee
127.0.0.1       haunt.er
127.0.0.1       marr.ow

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

I’ll be creating their DocumentRoots to /home/antero/banshee, haunter and marrow. However, if we want php to work there, we’ll need to comment out some lines in /etc/apache2/mods-available/php5.conf, after which it’ll look like this:

<IfModule mod_php5.c>
    <FilesMatch "\.ph(p3?|tml)$">
        SetHandler application/x-httpd-php
    </FilesMatch>
    <FilesMatch "\.phps$">
        SetHandler application/x-httpd-php-source
    </FilesMatch>
    # To re-enable php in user directories comment the following lines
    # (from <IfModule ...> to </IfModule>.) Do NOT set it to On as it
    # prevents .htaccess files from disabling it.
    #<IfModule mod_userdir.c>
    #    <Directory /home/*/public_html>
    #        php_admin_value engine Off
    #    </Directory>
    # </IfModule>
</IfModule>

Might as well create the sites’ directories now:

antero@VirtualBox:~$ mkdir banshee
antero@VirtualBox:~$ mkdir haunter
antero@VirtualBox:~$ mkdir marrow
antero@VirtualBox:~$ ls
banshee  Desktop  Documents  Downloads  examples.desktop  haunter  marrow  Music  Pictures  Public  Templates  Videos
antero@VirtualBox:~$

I’ll create an index.html (which might later be renamed to index.php) into each directory, with the site’s name as the content. Then we’ll create the sites’ files in /etc/apache2/sites-available. Result:

antero@VirtualBox:/etc/apache2/sites-available$ cat haunter
<VirtualHost *:80> 
        ServerName haunt.er
        ServerAlias www.haunt.er
        DocumentRoot /home/antero/haunter
</VirtualHost>

antero@VirtualBox:/etc/apache2/sites-available$ cat banshee
<VirtualHost *:80> 
        ServerName bansh.ee
        ServerAlias www.bansh.ee
        DocumentRoot /home/antero/banshee
</VirtualHost>

antero@VirtualBox:/etc/apache2/sites-available$ cat marrow
<VirtualHost *:80> 
        ServerName marr.ow
        ServerAlias www.marr.ow
        DocumentRoot /home/antero/banshee
</VirtualHost>

antero@VirtualBox:/etc/apache2/sites-available$

Let’s use a2ensite to enable the freshly created sites. Note how we use apache2 restart instead of reload.

antero@VirtualBox:/etc/apache2/sites-available$ sudo a2ensite banshee
Enabling site banshee.
To activate the new configuration, you need to run:
  service apache2 reload
antero@VirtualBox:/etc/apache2/sites-available$ sudo service apache2 restart
 * Restarting web server apache2                                                                                                              apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1 for ServerName
 ... waiting apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1 for ServerName
                                                                                                                                       [ OK ]
antero@VirtualBox:/etc/apache2/sites-available$

Navigating to http://bansh.ee with Firefox brings up the index.html in /home/antero/banshee. Success!

Also, I made a mistake earlier. Marrow’s DocumentRoot pointed to /banshee. It’s fixed now.

PHP

Now, let’s see if PHP works. I’ll create an index.php (as opposed to renaming the previously created index.html) in /home/antero/banshee. Its contents:

antero@VirtualBox:~/banshee$ cat index.php 
<!doctype html>
<html>
<head>
<title>Banshee's php page</title>
<meta charset="utf-8" />
</head>
<body>

Testing php!

<?php
phpinfo();
?>

Let’s navigate to http://bansh.ee/index.php with Firefox. Result:

Excellent! Now, this means that PHP works for each of our three sites. What if I want to disable PHP on bansh.ee? We add “php_value engine off” in /etc/apache2/sites-available/banshee. Source: http://www.linuxquestions.org/questions/linux-networking-3/turn-off-disable-php-for-some-virtual-hosts-in-apache-412577/

Let’s restart apache2 and try navigating to index.php again. Nope, doesn’t work anymore. Here’s a screenshot:

We’ll want to make sure that PHP still works for haunt.er and marr.ow, so let’s just copy the index.php to their respective directories:

antero@VirtualBox:~/banshee$ cp index.php ../haunter
antero@VirtualBox:~/banshee$ cp index.php ../marrow

PHP still works for the two other sites. Notice that the page title is still from banshee’s index.php due to me not having changed the contents of the file.

Custom setting

Time to try a setting of my choice. I’ll set marrow’s errorlog to /home/antero/marrow/err0r_log (I dislike leetspeak, but it’s very distinctive and thus fitting for this exercise):

<VirtualHost *:80>
        ServerName marr.ow
        ServerAlias www.marr.ow
        DocumentRoot /home/antero/marrow
        ErrorLog /home/antero/marrow/err0r_log
</VirtualHost>

Now, let’s navigate to some non-existent marr.ow urls: http://marr.ow/lalaa and http://marr.ow/lalaa.html. Not found, eh? Did our errorlog setting work?

antero@VirtualBox:~/marrow$ cat err0r_log 
[Mon Sep 24 12:28:11 2012] [error] [client 127.0.0.1] File does not exist: /home/antero/marrow/lalaa
[Mon Sep 24 12:28:18 2012] [error] [client 127.0.0.1] File does not exist: /home/antero/marrow/lalaa.html
antero@VirtualBox:~/marrow$ 

It certainly did.

HTTP Status codes

Let’s see if we can generate the called-for errors 200, 403, 404 and 500. We just did 404, so that leaves 200, 403 and 500. Let’s wikipedia up what those status codes mean:

200 OK
Standard response for successful HTTP requests. The actual response will depend on the request method used. In a GET request, the response will contain an entity corresponding to the requested resource. In a POST request the response will contain an entity describing or containing the result of the action

403 Forbidden
The request was a valid request, but the server is refusing to respond to it.[2] Unlike a 401 Unauthorized response, authenticating will make no difference.[2] On servers where authentication is required, this commonly means that the provided credentials were successfully authenticated but that the credentials still do not grant the client permission to access the resource (e.g. a recognized user attempting to access restricted content).

500 Internal Server Error
A generic error message, given when no more specific message is suitable.

Source: http://en.wikipedia.org/wiki/List_of_HTTP_status_codes

I except 200 to be fairly easy to produce.In fact, it’s probably already present because of previous page loads. Let’s take a look at /var/log/apache2/other_vhosts_access.log:

bansh.ee:80 127.0.0.1 - - [24/Sep/2012:11:32:43 +0300] "GET / HTTP/1.1" 200 363 "-" "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:15.0) Gecko/201$

The line tells the host, port and ip, time of access, method (GET), status code, browser info and OS info. That’s another one down. 403 and 500 remain. Let’s see if we get 403 if we remove all permissions for marrow’s index.html, restart apache2 (and empty Firefox’s cache):

antero@VirtualBox:~/marrow$ chmod a-rwx index.html
antero@VirtualBox:~/marrow$ sudo service apache2 restart
 * Restarting web server apache2                                                                                                              
 apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1 for ServerName
 ... waiting apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1 for ServerName
                                                                                                                                       [ OK ]

Well, navigating to http://marr.ow does now yield a Forbidden page, like so:

Let’s take a look at err0r_log this time:

antero@VirtualBox:~/marrow$ cat err0r_log 
[Mon Sep 24 12:28:11 2012] [error] [client 127.0.0.1] File does not exist: /home/antero/marrow/lalaa
[Mon Sep 24 12:28:18 2012] [error] [client 127.0.0.1] File does not exist: /home/antero/marrow/lalaa.html
[Mon Sep 24 12:50:59 2012] [error] [client 127.0.0.1] (13)Permission denied: file permissions deny server access: /home/antero/marrow/index.html

The browser does display a 403 Forbidden page as well. Let’s try another method. We’ll create /home/antero/haunter/.htaccess, whose contents look like this:

antero@VirtualBox:~/haunter$ cat .htaccess 
deny from all
antero@VirtualBox:~/haunter$

It causes http://haunt.er to display the 403 Forbidden page. Let’s see if we can find a reference to 403 in the logs. /var/log/apache2/other_vhosts_access.log contains:

haunt.er:80 127.0.0.1 - - [24/Sep/2012:13:00:22 +0300] "GET / HTTP/1.1" 403 492 "-" "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:15.0) Gecko/201$

Yes. That leaves us with code 500. Code 500 seems to be more of a “something’s wrong” type of error code. So, let’s break something. How about inserting a mishmash of garbled data into haunt.er’s .htaccess file? Source: http://httpstatus500.com/

antero@VirtualBox:~/haunter$ cat .htaccess 
asdasgdernherhysetrh sftrnsrom all
antero@VirtualBox:~/haunter$

Navigating to http://haunt.er results in:

which counts as a success for the purposes of this exercise. Also, /var/log/apache2/other_vhosts_access.log now contains the line:

haunt.er:80 127.0.0.1 - - [24/Sep/2012:13:09:34 +0300] "GET / HTTP/1.1" 500 632 "-" "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:15.0) Gecko/201$

That’s all the error codes down. Breaking apache2 isn’t that easy, which is nice.

 

Posted in Linux palvelimena ICT4TN003-4 | 1 Comment

Linux palvelimena – Homework 4 – Package handling

Week 4 – Metapackages and repositories

The fourth week had us dabble with packages. Sources used, if not explicitly stated otherwise, are:

http://terokarvinen.com/2011/create-deb-metapackage-in-5-minutes http://terokarvinen.com/2011/update-all-your-computers-with-a-deb-repository

Course homepage is: http://terokarvinen.com/2012/aikataulu-%E2%80%93-linux-palvelimena-ict4tn003-3-ja-ict4tn003-5-kevaalla-2012

Assignment

Release your own metapackage in your own repository.

Voluntary exercises
– sign the packages
– create metapackages with dependicer

Environment

Ubuntu 12.04 @ Virtualbox, username: antero, computer: VirtualBox

Creating a metapackage

Let’s start by creating a metapackage. A metapackage contains other packages, and it’s handy for installing your favourite software in one go.

We need equivs:

antero@VirtualBox:~$ sudo apt-get install equivs 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
...and so on

The package will contain two roguelike games, namely nethack and dungeon crawl, so I’ll make a working directory to keep organized. In the directory, we create a skeleton file for the package with equivs-control:

antero@VirtualBox:~$ pwd
/home/antero
antero@VirtualBox:~$ mkdir mymeta-games
antero@VirtualBox:~$ cd mymeta-games/
antero@VirtualBox:~/mymeta-games$ equivs-control mymeta-games.cfg
antero@VirtualBox:~/mymeta-games$ ls
mymeta-games.cfg
antero@VirtualBox:~/mymeta-games$

The file is now created. Next we edit it, but first a screenshot of the contents:

Notice the highlighted line about reasonable defaults. I love reasonable defaults. Anyway, time to edit the file a bit. We changed the package name, version number (to 0.1), depends (to include the games) and the description:

Time to build the deb package, then:

antero@VirtualBox:~/mymeta-games$ equivs-build mymeta-games.cfg 
dh_testdir
dh_testroot
dh_prep
dh_testdir
dh_testroot
dh_install
dh_installdocs
dh_installchangelogs
dh_compress
dh_fixperms
dh_installdeb
dh_gencontrol
dh_md5sums
dh_builddeb
dpkg-deb: building package `mymeta-games' in `../mymeta-games_0.1_all.deb'.

The package has been created.
Attention, the package has been created in the current directory,
not in ".." as indicated by the message above!
antero@VirtualBox:~/mymeta-games$

Let’s see if the deb was created:

antero@VirtualBox:~/mymeta-games$ ls
mymeta-games_0.1_all.deb  mymeta-games.cfg

It was. Now to see whether it works, with gdebi. Which, it seems, we have to install first:

The program 'gdebi' is currently not installed.  You can install it by typing:
sudo apt-get install gdebi-core
antero@VirtualBox:~/mymeta-games$ sudo apt-get install gdebi-core 
...
Processing triggers for man-db ...
Setting up gdebi-core (0.8.5build1) ...
antero@VirtualBox:~/mymeta-games$

Doing ‘man gdebi’ describes it as a “Simple tool to install deb files”, which sounds apt (as in fitting).

antero@VirtualBox:~/mymeta-games$ sudo gdebi mymeta-games_0.1_all.deb 
Reading package lists... Done
Building dependency tree        
Reading state information... Done
Building data structures... Done 
Building data structures... Done 

Requires the installation of the following packages: 
crawl  nethack-common  nethack-console 
Nethack and Crawl in one go.
long description and info
.
second paragraph
Do you want to install the software package? [y/N]:y
Get:1 http://fi.archive.ubuntu.com/ubuntu/ precise/universe nethack-common i386 3.4.3-12.2 [453 kB]
Get:2 http://fi.archive.ubuntu.com/ubuntu/ precise/universe nethack-console i386 3.4.3-12.2 [882 kB]
Fetched 1335 kB in 0s (0 B/s)                                                                   
Preconfiguring packages ...
Preconfiguring packages ...
Selecting previously unselected package crawl.
(Reading database ... 170795 files and directories currently installed.)
Unpacking crawl (from .../crawl_2%3a0.9.1-1_i386.deb) ...
Selecting previously unselected package nethack-common.
Unpacking nethack-common (from .../nethack-common_3.4.3-12.2_i386.deb) ...
Selecting previously unselected package nethack-console.
Unpacking nethack-console (from .../nethack-console_3.4.3-12.2_i386.deb) ...
Processing triggers for man-db ...
Processing triggers for ureadahead ...
ureadahead will be reprofiled on next reboot
Setting up crawl (2:0.9.1-1) ...
Setting up nethack-common (3.4.3-12.2) ...
Setting up nethack-console (3.4.3-12.2) ...
update-alternatives: using /usr/lib/games/nethack/nethack-console.sh to provide /usr/games/nethack (nethack) in auto mode.
Selecting previously unselected package mymeta-games.
(Reading database ... 170848 files and directories currently installed.)
Unpacking mymeta-games (from mymeta-games_0.1_all.deb) ...
Setting up mymeta-games (0.1) ...
antero@VirtualBox:~/mymeta-games$

Success! Although the lines “long description and info” and “second paragraph” look ugly. Now, to see if it passes lintian (a static analysis tool for Debian packages). We install lintian with sudo apt-get install lintian, and then:

antero@VirtualBox:~/mymeta-games$ man lintian
antero@VirtualBox:~/mymeta-games$ lintian mymeta-games_0.1_all.deb 
E: mymeta-games: debian-changelog-file-contains-invalid-email-address antero@VirtualBox
E: mymeta-games: maintainer-address-malformed Antero <antero@VirtualBox>
antero@VirtualBox:~/mymeta-games$

Looks like we have some tinkering to do. I’ll edit the cfg file to contain Erkki Esimerkki <erkki@esimerkki.com> and change the version number to 0.2 to prevent future errors. Actually, let’s add the long description as well, as its lack bothers me to no end:

Let's build it again, and retry lintian.
antero@VirtualBox:~/mymeta-games$ nano mymeta-games.cfg 
antero@VirtualBox:~/mymeta-games$ equivs-build mymeta-games.cfg 
dh_testdir
dh_testroot
dh_prep
dh_testdir
dh_testroot
dh_install
dh_installdocs
dh_installchangelogs
dh_compress
dh_fixperms
dh_installdeb
dh_gencontrol
dh_md5sums
dh_builddeb
dpkg-deb: building package `mymeta-games' in `../mymeta-games_0.2_all.deb'.

The package has been created.
Attention, the package has been created in the current directory,
not in ".." as indicated by the message above!
antero@VirtualBox:~/mymeta-games$ ls
mymeta-games_0.1_all.deb  mymeta-games_0.2_all.deb  mymeta-games.cfg
antero@VirtualBox:~/mymeta-games$ lintian mymeta-games_0.2_all.deb 
antero@VirtualBox:~/mymeta-games$

Let’s see if we can sign it. Remember how we set the maintainer to Erkki Esimerkki <erkki@esimerkki.com>? We need to remember that now, as we generate keys with gpg –gen-key. We insert Erkki’s info when asked, and set the key to expire in 1 week, and otherwise just press enter whenever asked something. I wouldn’t usually include all sensitive data on a report, but since this is a temporary environment with fake info, I’ll let it slip.

Did our key get saved succesfully?

antero@VirtualBox:~/mymeta-games$ gpg --list-secret-keys 
/home/antero/.gnupg/secring.gpg
-------------------------------
sec   1024R/25DB1429 2012-09-18 [expires: 2012-09-25]
uid                  Erkki Esimerkki <erkki@esimerkki.com>
ssb   1024R/C953C374 2012-09-18

Looks like it did. Next stop is to rebuild the package once again, this time with the –full parameter. Also, I’ll change to description a bit and increase the version number. (Latter steps not specifically reported anymore.) –full makes it use Erkki’s key. Quote from man equivs-build:

–full | -f

Do a complete build. debuild will be called, that is, a full package will be built and signed, suitable for upload to the Debian servers.The ID used to signed is taken from, in that order, the user from the last entry of a supplied changelog, the Maintainer: field in the equivs control file, or the local username.

It is now a signed package, unless I’ve misunderstood something along the way.

Repository

Next up, setting up a repository. We’re going to need to install Apache web server and enable userdir, so let’s get that out of the way.

antero@VirtualBox:~/mymeta-games$ sudo apt-get install apache2
...

We use a handy tool called a2enmod to enable userdir, and then restart the server:

antero@VirtualBox:~/mymeta-games$ sudo a2enmod userdir
Enabling module userdir.
To activate the new configuration, you need to run:
service apache2 restart
antero@VirtualBox:~/mymeta-games$ sudo service apache2 restart
* Restarting web server apache2                                                                 
apache2: Could     not reliably determine the server's fully 
qualified domain name, using 127.0.1.1 for ServerName
... waiting .apache2: Could not reliably determine the server's 
fully qualified domain name, using 127.0.1.1 for ServerName

It couldn’t reliably determine the domain name, maybe because it’s running on Virtualbox. I’ll have to look into that later.

Next we create and configure an empty repository:

antero@VirtualBox:~$ pwd
/home/antero
antero@VirtualBox:~$ cd public_html/
antero@VirtualBox:~/public_html$ mkdir -p repository/conf

Then we create a file called distributions and edit it to contain:

antero@VirtualBox:~/public_html/repository/conf$ cat distributions 
Codename: precise
Components: main
Suite: precise
Architectures: i386 amd64 source

(The example I’m following used codename and suite ‘lucid’, but I suppose ‘precise’ is appropriate for Ubuntu 12.04.) Then we add our previously created deb package into the repository, with reprepro, which we’ll also have to install.

antero@VirtualBox:~/public_html/repository$ reprepro
The program 'reprepro' is currently not installed.  You can install it by typing:
sudo apt-get install reprepro
antero@VirtualBox:~/public_html/repository$ sudo apt-get install reprepro
Reading package lists... Done
Building dependency tree
...

Now, to actually add the deb:

antero@VirtualBox:~/public_html/repository$ reprepro -VVVV -b /home/antero/public_html/repository/ includedeb precise /home/antero/mymeta-games/mymeta-games_0.3

Notice the -VVVV. More V flags mean more verbose output. -b is used to define the repository’s base dir. As the included screenshot shows, the output is verbose indeed:

Let’s add our repository to the sources.list file:

antero@VirtualBox:~/public_html/repository$ sudo nano /etc/apt/sources.list

and add:

## My test repository
deb http://localhost/~antero/repository precise main

Then we apt-get update and try to install the mymeta-games package we created earlier:

antero@VirtualBox:~/public_html/repository$ sudo apt-get install mymeta-games 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  mymeta-games
0 upgraded, 1 newly installed, 0 to remove and 52 not upgraded.
Need to get 2,088 B of archives.
After this operation, 26.6 kB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  mymeta-games
Install these packages without verification [y/N]? y
Get:1 http://localhost/~antero/repository/ precise/main mymeta-games all 0.3 [2,088 B]
Fetched 2,088 B in 0s (209 kB/s)          
Selecting previously unselected package mymeta-games.
(Reading database ... 172206 files and directories currently installed.)
Unpacking mymeta-games (from .../mymeta-games_0.3_all.deb) ...
Setting up mymeta-games (0.3) ...
antero@VirtualBox:~/public_html/repository$

It works! But it couldn’t be authenticated, which might indicate a problem with the gpg key stuff. I’ll probably update this post regarding that later.

Posted in Linux palvelimena ICT4TN003-4 | Leave a comment

Roguelikes ca. 2012

“Is x a roguelike?”

A paraphrase of the (quite pompously named) Berlin Interpretation, with introduction

http://roguebasin.roguelikedevelopment.org/index.php/Berlin_Interpretation

On the term ‘roguelike’

The term roguelike is used very loosely today. To each their own, but this might not make sense, because:

If everything’s called a roguelike, the term loses its meaning.

We already have the terms hack and slash and dungeon crawler. They are perfectly good, descriptive terms, and don’t refer to a overly specific thing.

The term roguelike refers to a certain type of game, such as NetHack, ADOM, Dungeon Crawl or Angband, to name but a few. These games have a lot in common with each other, and their core gameplay and appearance are reminiscent of Rogue.

If a game shares merely one or two features with these, it might not be a roguelike itself, but a game with some features found in roguelikes. Which, incidentally, is completely okay.

Still, this is an important distinction. Bad analogy:

An ornithologist might get quite annoyed if people started calling fish birds, because both have two eyes and require oxygen to survive, and are commonly eaten by humans.

What separates a roguelike from any old hack and slash, then?

A roguelike typically has most of the following high value factors. The factors are paraphrased from the Berlin Interpretation:

  • Random environment generation
  • Permanent death
  • Turn-based
  • Grid-based
  • Non-modal (Every action available at any point, ie. no separate exploration and fight modes etc.)
  • Complexity (Plenty of possible item/player/monster/world interactions at any given time)
  • Resource management (Scarcity of items, limited inventory, possible item destruction)
  • Hack’n’slash
  • Exploration and discovery (Random environment and unidentified items for every new player character)

It would likely have some of the other factors as well:

  • Enemies are similar to players (ie. have inventory, equipment, use items and cast spells)
  • Tactical challenge (learning curve, trial and error, accumulated knowledge of the game world)
  • ASCII display (traditional, but many roguelikes also have tile graphics available)
  • Dungeons (rooms connected by corridors)
  • Numbers (used to describe character etc, and are deliberately shown)

So?

It’s good to remember that for many people a roguelike is a specific type of game, one still relevant today. When Binding of Isaac is called one on the basis that it has a degree of randomity and permanent death, it might well cause the occasional frown, even if both sides enjoy the game.

On the other hand, dismissing the previous generation of gamers as “butthurt dwellers” because they refuse to call a real-time 3D shooter a roguelike is just ignorant.

Note that appearance alone isn’t the key; Dungeons of Dredmor is blatantly graphical, and yet fulfills a large amount of the aforementioned factors, whereas Dwarf Fortress boasts nice ASCII graphics, but its gameplay is fulfills a precious few. (Save for the adventure mode, reportedly – haven’t tried it myself yet.)

Sources

http://roguebasin.roguelikedevelopment.org/index.php/Berlin_Interpretation
http://www.roguetemple.com/roguelike-definition

Posted in Uncategorized | Tagged , , , , , , , , , | Leave a comment

Linux palvelimena – Homework 3 – Villains and decent folk

This is the third piece of homework for the course ICT4TN003-4 Linux palvelimena (Linux as a server):
http://terokarvinen.com/2012/aikataulu-%E2%80%93-linux-palvelimena-ict4tn003-3-ja-ict4tn003-5-kevaalla-2012

It will most likely not be finished by the deadline, which is something I’m not happy about, and realize this will likely impact my grade.

Assignment

Solve HoneyNet: Scan of the Month 15. http://old.honeynet.org/scans/scan15/

Additional tasks:
– Can you find additional information on the attacker (using legal means)?
– Analyze the rootkit line by line

Examine the partition to your heart’s content. Use nothing but legal methods, ie. don’t portscan remote addresses. Be careful with the image.

Notes

I’ll use Virtualbox for this assignment. I’ve had my eye on Lubuntu for a while, so the guest OS image I’ll be using is lubuntu-12.04-desktop-i386.iso. After downloading the Scan of the month 15 image and installing autopsy, I’ll disable networking functionality from the guest, and use the host for searches and writing this report.

In Virtual Lubuntu:

Computer name: luba
Username is: mrf

Also, I’ve been advised to include information on my system for compatibility reference.
Hardware:

Mobo:     Asus P5Q (BIOS 1208)
CPU:     Intel Core 2 Quad Q9550 @ 3.4GHz
RAM:    8GB 800MHz DDR2 (4x2GB)
GPU:    Nvidia Geforce GTX 470 (Gigabyte branding)
HDD:    Samsung Spinpoint F1 HD753LJ 750GB
PWR:    Antec Earthwatts 500W
Sound:    Creative SB X-Fi (EMU20K1 chip)
Wlan:    ASUS PCE-N15 11n
DVD:    TSSTcorp SH-S223F
KB:        Costar tenkeyless Cherry MX Black (PS/2)
Mouse:    1000Hz 500DPI (USB)

Software:

Windows 7 Professional x64 SP1
Windows Server 2008 R2
Xubuntu 12.04 LTS x86
Ubuntu Server 14.04 LTS x64

Oracle Virtualbox (VT-x disabled, as I only ever use one core for VMs and Virtualbox’s own virtualization seems to work better)

Every piece of kit seems to work well in Xubuntu 12.04. Nvidia’s restricted driver provides the occasional headache, though.

Autopsy install and setup

We’ve covered the basic apt-get install routine in earlier posts, so from here on out I will just include the commands without the output, unless there is something exceptional worth reporting. (Provided our teacher agrees.) Let’s start by installing autopsy, a forensic browser.

mrf@luba:~$ sudo apt-get install autopsy

We also need the Scan of the month 15 partition image from http://old.honeynet.org/scans/scan15 The image is thus found in /home/mrf/Downloads/honeynet/honeypot.hda8.dd

Autopsy must be started as sudo:

mrf@luba:~$ sudo autopsy

Now we can use Chromium browser to open http://localhost:9999/autopsy

“WARNING: Your browser currently has Java Script enabled.” Time to disable Javascript from Chromium:

chrome://settings/content -> Javascript -> Do not allow any site to run JavaScript.

All set! Time to click the  “New Case” button. Case name is “scan15”, description is empty, Investigator name is a. Antero. Autopsy then creates a directory and configuration for the case, located in:

Case directory (/var/lib/autopsy/scan15/) created
Configuration file (/var/lib/autopsy/scan15/case.aut) created

Next we add a new host, which will be the previously downloaded image file. I choose to call  it host1, which is the default.

Host Directory (/var/lib/autopsy/scan15/host1/) created
Configuration file (/var/lib/autopsy/scan15/host1/host.aut) created

We must now import an image file for this host -> Add Image. Let’s insert /home/mrf/Downloads/honeynet/honeypot.hda8.dd as the location, select Partition for Type, and Copy for Import Method.

In the Image File Details page, we ignore the hash value, accept the default mount point of /1/ and ext file system.

Testing partitions
Copying image(s) into evidence locker (this could take a little while)
Image file added with ID img1
Volume image (0 to 0 - ext - /1/) added with ID vol1

Let’s create a File Activity timeline (Create data file, create timeline, view timeline)

That’s done. We can now access it if need be. We close the timeline view with the Close button and return to autopsy’s gallery. Time to take a look at the contents of the image. We press the Analyze button and enter the File Analysis view.

Analysis

I made some progress in class earlier, so I’ll be picking up where I left off, informationwise.

First off, there’s a deleted file that doesn’t seem to belong: /1/lk.tgz. Let’s click it and use the Export feature and see what it contains.

The archive certainly contains interesting stuff. I extracted the file to /home/mrf/Downloads/last. Let’s take a look at ‘cleaner’ with the strings-command:

mrf@luba:~/Downloads/last$ strings cleaner |less

This thing seems to be a bash script that cleans log files, and contains odd stuff in german. How about ‘install’?

mrf@luba:~/Downloads/last$ strings install |less

The strange boasting in the beginning of the file is pretty incriminating. I’d say that lk.tgz contains the rootkit.

Further analysis

Let’s take a look at some other interesting files.

mrf@luba:~/Downloads/last$ strings logclear |less

killall -9 linsniffer
rm -rf tcp.log
touch tcp.log
./linsniffer >tcp.log &

File ‘ssh’ contains a lot of lines relating to man-in-the-middle attacks and other warnings. It reads like a logfile.

File sshd_config mentions PidFile /dev/ida/.drag-on/pidfile. There are also keys present (ssh_host_key and ssh_host_key.pub)

I’ll take a closer look a bit later on. Let’s check out the other directories on the partition now.

…they seem to be mostly empty. For future reference: /etc and $OrphanFiles are interesting. /etc because it contains system settings in human readable form, although the attacker did have the cleaner script, and $OrphanFiles because it might contain interesting loose files.

Notes on orphanfiles

OrphanFile-2060 seemed to contain suspicious urls:

But then again, they’re commented out, and the urls seem like example placeholder stuff.  (Allow friend, deny lowsecurity…)

OrphanFile-2041 is the previously mentioned install script. Based on this, it has been deployed on the server. Its last line “rm -rf last lk.tgz computer lk.tar.gz” removes lk.tar.gz, last, computer and lk.tgz, which, incidentally, is the archive we extracted earlier. There’s some romanian in there as well.

The file mentions two e-mail addresses: last@linuxmail.org and bidi_damm@yahoo.com, which might be forensically useful in the future.

OrphanFile-2043 is the logfile cleaner script. The script’s called sauber, which is german for clean.
OrphanFile-2044 is inetd.conf, which mentions a user called ‘cyrus’.
OrphanFile-2045 is bash script that mentions mkxfs and linsniffer (remember to write more about this later)
OrphanFile-2047 is a perl script that sorts the output from LinSniffer.
OrphanFile-2050 contains a long string of numbers and an e-mail address: root@dil2.datainfosys.net
OrphanFile-2059 kills linsniffer and removes tcp.log

I had to export most of the orphanfiles for closer inspection with the strings-command.

2039: Similar content with the previously mentioned ssh-file
2049: Mentions root@dil2.datainfosys.net again

Conclusions so far

The Scan 15 challenge asks:

Show step by step how you identify and recover the deleted rootkit from the / partition.
What files make up the deleted rootkit?

We have located lk.tgz and recovered it. Its contents seem to make up the deleted rootkit.

It includes Linsniffer: “linsniffer is an ethernet sniffer. It sits and listens on a network and grabs every packet it sees” (Source: http://www.mail-archive.com/redhat-list@redhat.com/msg02383.html)

It also includes a bash script called Sauber, which cleans the system’s logfiles.

Can you find additional information on the attacker (using legal means)?
Analyze the rootkit line by line.

I would seek additional information on the attacker by the aforementioned e-mail addresses: last@linuxmail.org and bidi_damm@yahoo.com.

I have analyzed the contents of lk.tgz as well I can in the given timeframe and reported the things I deem most noteworthy.

Posted in Linux palvelimena ICT4TN003-4 | Leave a comment

Linux palvelimena – Homework 2 – Running processes and system status

Assignment

The second batch of homework consisted of four main assignments:

1) Install munin.

2) Examine a process of your choice with tools taught in class: top, iotop, netstat, lsof, ps. Finish by killing the process.

3) Stress the system with the ‘stress’ tool. Measure the stress brought on different resources.

4) Examine data collected by munin.

Sources

Unless explicitly stated otherwise, these exercises are based on lessons in Linux palvelimena ICT4TN003-4: http://terokarvinen.com/2011/aikataulu-linux-palvelimena-ict4tn003-2

1) Install munin

man munin – “Munin is as group of programs to gather data from hosts, graph them, create html-pages, and optionally warn contacts about any off-limit values”

First thing’s first. Let’s find and install munin with:

sudo apt-cache search munin

antero@xub12:~$ sudo apt-cache search munin
munin - network-wide graphing framework (grapher/gatherer)
munin-common - network-wide graphing framework (common)
munin-node - network-wide graphing framework (node)
mailping - monitor email service availability and functioning
mumble-django - Mumble-Server web interface
munin-java-plugins - network-wide graphing framework (java plugins for node)
munin-libvirt-plugins - Munin plugins using libvirt
munin-plugins-extra - network-wide graphing framework (user contributed plugins for node)
virt-goodies - A collection of helpful virtualisation related tools

The first result looks like the correct one. Let’s install it with:

sudo apt-get install munin

antero@xub12:~$ sudo apt-get install munin
<<Wall of output ensues, here's the tail>>
Setting up munin (1.4.6-3ubuntu3.1) ...
Adding system user `munin' (UID 115) ...
Adding new group `munin' (GID 124) ...
Adding new user `munin' (UID 115) with group `munin' ...
Not creating home directory `/var/lib/munin'.
Setting up munin-node (1.4.6-3ubuntu3.1) ...
Initializing plugins..done.
munin-node start/running, process 9010
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
antero@xub12:~$

Why is it not creating home directory /var/lib/munin? I sense problems in the near future. Nevertheless, let’s assume it is now gathering data silently.

(At this juncture I restarted the computer.)

2) Examine a process of your choice with tools taught in class: top, iotop, netstat, lsof, ps. Finish by killing the process.

Now then, time to select a process to observe. I’ve been thinking of trying the Spotify Linux beta, so let’s see about installing that. I decided to choose it because of two things:

1) It’s quite heavy on p2p connections, so it might provide interesting netstat data.

2) It allows me to pipe in to some music.

Spotify has kindly provided users with detailed installation instructions, found in: http://www.spotify.com/fi/download/previews/

First we add Spotify’s repository into /etc/apt/sources.list:

antero@xub12:~$ sudo nano /etc/apt/sources.list

(must be run as su since otherwise there’s no permission to write to the file)

Added the repository in the end of the file:

## Spotify repository
deb http://repository.spotify.com stable non-free

I want to verify the download packages, so I did as instructed:

sudo apt-key adv –keyserver keyserver.ubuntu.com –recv-keys 94558F59

antero@xub12:~$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 94558F59
[sudo] password for antero: 
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --secret-keyring /tmp/tmp.lvtxBUkOTA --trustdb-name /etc/apt/trustdb.gpg --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver keyserver.ubuntu.com --recv-keys 94558F59
gpg: requesting key 94558F59 from hkp server keyserver.ubuntu.com
gpg: key 94558F59: public key "Spotify Public Repository Signing Key <operations@spotify.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

Everything seems to be in order, as far as I can tell. After that the program is just a simple apt-get update and install away. Let’s look for it, and install it:

antero@xub12:~$ sudo apt-cache search spotify
spotify-client - Spotify desktop client
spotify-client-gnome-support - Transitional package for spotify-client
spotify-client-qt - Transitional package for spotify-client

antero@xub12:~$ sudo apt-get install spotify-client

<<Wall of text>>
Setting up spotify-client (1:0.8.4.103.g9cb177b.260-1) ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place

Let’s start it up from the Applications menu. Here’s a screenshot of the eventual result:

top

man top: “top – display Linux tasks”

Spotify is now playing random tracks until forever. Time to see what top has to say about that. Let’s run it:

antero@xub12:~$ top

Pressing shift-m shows running processes organized by memory use, from most hungry to least. Shift-p does the same for processor time. Affixed here are screenshots of the system’s CPU and memory usage while Spotify is playing music:

It appears to using 5% of cpu resources. What a hog. Next up is the RAM usage view:

Using 3.4% of the computer’s random access memory. I’d say that’s within reasonable limits.

netstat

man netstat: “netstat – Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships”

Time to check out netstat, with the -pea flags and -n, with no sudo.

antero@xub12:~$ netstat -pena --inet

Here is a screenshot of netstat’s output for your convenience:

Spotify has 3 TCP ports listening, one established TCP connection and 3 UDP connections.

For future reference, just add -n with the other flags, resulting in “netstat -pena –inet”. As far as mnemonics go, “penan internet” is quite dumb, but still helpful.

ps

man ps: “ps – report a snapshot of the current processes”

Next up is ps, started with the waux parameters. Note how there’s no ‘-‘ before the flags – the command is from the days of yore, when hyphens were nowhere to be seen. (As taught in class.) To avoid a wall of output, we pipe ‘grep spotify’.

antero@xub12:~$ ps waux |grep spotify
antero    7715  5.2  3.4 447844 105272 ?       Sl   17:59   1:14 spotify
antero   10095  0.0  0.0   4368   836 pts/2    S+   18:22   0:00 grep --color=auto spotify

Well, the program is certainly running. Interestingly, grep is also shown, since I used it just then.

lsof

man lsof: “lsof – list open files”

I wonder which files Spotify has open. We use the ‘lsof’ command here, which, if invoked without bells and whistles, lists WAY too much output to be pasted here. Let’s grep it a bit.

antero@xub12:~$ lsof |grep spotify

Still too many lines for comfort. Let’s just add a bunch of |grep -v’s for words present in the most numerous entry types. (-v excludes lines with the keyword present.)

antero@xub12:~$ lsof |grep spotify |grep -v '.so' |grep -v 'cache' |grep -v 'pipe' |grep -v 'inode'

iotop

man iotop: “iotop – simple top-like I/O monitor”

Next up is iotop, which is used to monitor I/O activity. It’s not installed by default in Xubuntu 12.04. Here’s the installation output:

antero@xub12:~$ sudo apt-get install iotop
[sudo] password for antero: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  iotop
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 22.2 kB of archives.
After this operation, 123 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu/ precise/universe iotop i386 0.4.4-4 [22.2 kB]
Fetched 22.2 kB in 0s (37.7 kB/s)
Selecting previously unselected package iotop.
(Reading database ... 156655 files and directories currently installed.)
Unpacking iotop (from .../iotop_0.4.4-4_i386.deb) ...
Processing triggers for man-db ...
Setting up iotop (0.4.4-4) ...

‘man iotop’ tells that the program is a “simple top-like I/O monitor”, so let’s try running it without any parameters.

antero@xub12:~$ iotop
Netlink error: Operation not permitted (1)
iotop requires root or the NET_ADMIN capability.

Foiled again! Good thing we have sudo.

Odd. I would have expected Spotify’s disk cache to show up. Gazing upon iotop’s manpage I found a flag that makes it only show processes with actual I/O activity. However, with just -o the window was mostly blank, with the occasional process briefly flashing up and disappearing. I added -b, which is batch mode, with which iotop outputs a feed of activity, which eliminates the flashing. Here’s the output:

antero@xub12:~$ man iotop
antero@xub12:~$ sudo iotop -o
antero@xub12:~$ man iotop
antero@xub12:~$ sudo iotop -ob
Total DISK READ:       0.00 B/s | Total DISK WRITE:       0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
Total DISK READ:       0.00 B/s | Total DISK WRITE:       0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
Total DISK READ:       0.00 B/s | Total DISK WRITE:       0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
Total DISK READ:       0.00 B/s | Total DISK WRITE:       0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
 7718 be/4 antero      0.00 B/s  499.92 K/s  0.00 %  0.00 % spotify
Total DISK READ:       0.00 B/s | Total DISK WRITE:       0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
Total DISK READ:       0.00 B/s | Total DISK WRITE:       0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
Total DISK READ:       0.00 B/s | Total DISK WRITE:      27.23 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
  266 be/3 root        0.00 B/s    3.89 K/s  0.00 %  4.10 % [jbd2/sda6-8]
Total DISK READ:       0.00 B/s | Total DISK WRITE:       0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
Total DISK READ:       0.00 B/s | Total DISK WRITE:       0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
Total DISK READ:       0.00 B/s | Total DISK WRITE:       0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
Total DISK READ:       0.00 B/s | Total DISK WRITE:       0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
Total DISK READ:       0.00 B/s | Total DISK WRITE:       0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
^Cantero@xub12:~$

killall

man killall: “killall – kill processes by name”

Everything must come to an end. It’s time to kill Spotify. The obvious way would be to killall spotify. Let’s see if that works.

antero@xub12:~$ killall spotify

I no longer hear music. Also, the Spotify icon is gone from the taskbar and tray. So that’s that.

stress

man stress: “stress – tool to impose load on and stress test systems”

Enough with peace and tranquility, time for some stress. Oh, but we do need to install it first.

antero@xub12:~$ stress
The program 'stress' is currently not installed.  You can install it by typing:
sudo apt-get install stress
antero@xub12:~$ sudo apt-get install stress
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  stress
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 17.0 kB of archives.
After this operation, 76.8 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu/ precise/universe stress i386 1.0.1-1build1 [17.0 kB]
Fetched 17.0 kB in 0s (92.1 kB/s) 
Selecting previously unselected package stress.
(Reading database ... 156682 files and directories currently installed.)
Unpacking stress (from .../stress_1.0.1-1build1_i386.deb) ...
Processing triggers for install-info ...
Processing triggers for man-db ...
Setting up stress (1.0.1-1build1) ...
antero@xub12:~$

Just running ‘stress’ doesn’t do anything except print info on what the user is expected to do. It also gives an example of a 10 second test, let’s try that.

Example: stress –cpu 8 –io 4 –vm 2 –vm-bytes 128M –timeout 10s.

antero@xub12:~$ stress --cpu 8 --io 4 --vm 2 --vm-bytes 128M --timeout 10s
stress: info: [13621] dispatching hogs: 8 cpu, 4 io, 2 vm, 0 hdd
stress: info: [13621] successful run completed in 10s

Well, that was unimpressive. The laptop became slightly noisier for a while, though.

The assignment told us to measure the stress imposed on the system, but man stress doesn’t really shed light into this. I assume munin will take care of that. In order to get a clearer reading, I’ll just run a 25 second test.

antero@xub12:~$ stress --cpu 8 --io 4 --vm 2 --vm-bytes 128M --timeout 25s
stress: info: [13685] dispatching hogs: 8 cpu, 4 io, 2 vm, 0 hdd
stress: info: [13685] successful run completed in 25s
antero@xub12:~$

I took a break from tinkering with technology, not realizing that it might make munin graphs harder to read. Let’s see.

munin’s data

(Munin proved to be a bit too much for me to handle at this point.)

I think I ought to munin-update first.

antero@xub12:~$ munin-update
munin-update: command not found

Right. The manpage *does* list such a command. Apparently it’s not completely up-to-date. There does exist, however, munin-check.

antero@xub12:~$ munin-check
Check /var/cache/munin/www
check /var/lib/munin/datafile
check /var/lib/munin/limits
check /var/lib/munin/localdomain
check /var/lib/munin/munin-graph.stats
check /var/lib/munin/munin-update.stats
Check miscelaneous
# /var/lib/munin/datafile : Wrong permissions (664 != 644)
# /var/lib/munin/limits : Wrong permissions (664 != 644)
# /var/lib/munin/munin-graph.stats : Wrong permissions (664 != 644)
# /var/lib/munin/munin-update.stats : Wrong permissions (664 != 644)
# /var/lib/munin/plugin-state : Wrong owner (munin != nobody)
# /etc/munin/plugin-conf.d : Wrong permissions (750 != 755)
Check done.  Please note that this script only checks most things,
not all things.

Please also note that this script is very new and may be buggy.

You don’t say. Googling “Wrong permissions (664 != 644)” yields one result, a pastebin paste with similar output. Unfortunately, it is of little help. sudoing the command doesn’t help either.

Bash autocomplete does list various interesting programs though:

antero@xub12:~$ munin
munin-check           munin-cron            munindoc              munin-node            munin-node-configure  munin-run

Perhaps we need to configure a node or something. I guesstimate it might help.

According to http://manpages.ubuntu.com/manpages/natty/man8/munin-check.8.html munin-check should be run as superuser, and it checks required permissions. However, it doesn’t work for me:

antero@xub12:~$ sudo munin-check
Check /var/cache/munin/www
check /var/lib/munin/datafile
check /var/lib/munin/limits
check /var/lib/munin/localdomain
check /var/lib/munin/munin-graph.stats
check /var/lib/munin/munin-update.stats
Check miscelaneous
# /var/lib/munin/datafile : Wrong permissions (664 != 644)
# /var/lib/munin/limits : Wrong permissions (664 != 644)
# /var/lib/munin/munin-graph.stats : Wrong permissions (664 != 644)
# /var/lib/munin/munin-update.stats : Wrong permissions (664 != 644)
# /var/lib/munin/plugin-state : Wrong owner (munin != nobody)
# /etc/munin/plugin-conf.d : Wrong permissions (750 != 755)
Check done.  Please note that this script only checks most things,
not all things.

Please also note that this script is very new and may be buggy.

munin-check’s manpage says -f fixes permissions:

antero@xub12:~$ sudo munin-check -f
Check /var/cache/munin/www
check /var/lib/munin/datafile
check /var/lib/munin/limits
check /var/lib/munin/localdomain
check /var/lib/munin/munin-graph.stats
check /var/lib/munin/munin-update.stats
Check miscelaneous
# /var/lib/munin/datafile : Wrong permissions (664 != 644)
# /var/lib/munin/limits : Wrong permissions (664 != 644)
# /var/lib/munin/munin-graph.stats : Wrong permissions (664 != 644)
# /var/lib/munin/munin-update.stats : Wrong permissions (664 != 644)
# /var/lib/munin/plugin-state : Wrong owner (munin != nobody)
# /etc/munin/plugin-conf.d : Wrong permissions (750 != 755)
Check done.  Please note that this script only checks most things,
not all things.

Please also note that this script is very new and may be buggy.

Yes, it may be buggy, I get it. Others have noticed this behaviour as well: https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=834055

A braver man would go on setting permissions manually, but even running the risk of looking like a lazy student, I prefer to wait for the developers to fix their tools.

*edit* Munin does create graphs in browser-friendly form without any specific configuration, which can be found in /var/cache/munin/www

Posted in Linux palvelimena ICT4TN003-4 | Leave a comment