Awstats only builds main report

I am using awstats on my blog to generate visitor statistics from my web server logs. I installed awstats from the ubuntu repositories, and that package creates a scheduled cron job for building the static report pages.

The cron job executes /usr/share/awstats/tools/buildstatic.sh which in turn executes awstats_buildstaticpages.pl.

Awstats_buildstaticpages.pl is a script which is supposed to create all report pages, but there seems to be an issue with this script. It seems the script does not load the default values for the Show* config options.

The Show config values are used to set which report pages to generate.

So in order to force the script to build all the reports, you need to explicitly add the Show* options to your awstats config file.

Get used diskspace on all servers

I had the need to get the used disk space on all our servers today. VMware only shows the peak disk usage and don't account for when files are removed and disk usage is lowered, those metrics was not useful.

To solve this I used SaltStack which we use to configure our servers. The following command returns the information about each disk on the server, like size and used space. The output is saved to disk_usage.txt.

salt "*" disk.usage > disk_usage.txt

The output does not give a good overview of the total disk usage on the server since each disk is displayed individually, and the output format is not ready to be imported in a spreadsheet.

So I created this Python script which parses the disk_usage.txt file and returns a list of server name and the total used space.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import re

with open("disk_usage.txt", "r") as f:
    result = {}
    hostname = ""
    used_space = 0
    get_next_line = False
    current_mount = None
    for line in f:
        if get_next_line:
            size = '\n'.join(line.splitlines()).lstrip(' ')
            get_next_line = False
            if re.match("[0-9]+", size):
                if current_mount.startswith('/'):
                    used_space += int(size) / 1024 / 1024
                else:
                    used_space += int(size) / 1024 / 1024 / 1024

        if re.match("^[a-zA-Z0-9].*", line):
            if hostname is not "":
                print("{}\t{}".format(hostname, used_space))
            hostname = '\n'.join(line.splitlines()).rstrip(':')
            used_space = 0
        if re.match("^[  ]{4}[A-Z/]+", line):
            current_mount = line.lstrip(' ')
        if re.match("^[  ]{8}used:$", line):
            get_next_line = True

The script has to take into account that Salt seems to return the used disk space in kilobytes on Linux servers, but in just bytes on Windows servers.

Gitlab HTTPS authentication returns exit code 128

We are running Gitlab at work as our source code repository. It has been working fine for most users, but some users has had trouble using HTTPS authentication. There was no issues to login to the Gitlab web ui, but when using a git client to clone a repository over HTTPS would result in an error message saying

exit code 128

We are using LDAP authentication in Gitlab against our Active Directory.

The issue

It turns out that this was only affecting users with an "å" character in their distinguishedName attribute in Active Directory.

When creating users in Active directory the users Full name is a combination of the first and last name, unless you manually change it. The full name is then used to set both the displayName and distinguishedName attribute.

This causes a problem when the user has an "å" in their first or last name, since that character will then be included in the distinguishedName attribute, and will cause LDAP authentication againt gitlab from git to fail.

Solution

Rename the user account and change the Full name to use an "a" instead of "å" this will fix the distinguishedName attribute while leaving the displayName untouched.

After this change the git client should now be able to authenticate.

But this also caused another issue, after the rename the user is unable to login to the Gitlab web ui.

This is because Gitlab stores the external identities (the distinguishedName of the users) in it's database. And after the rename it no longer matches the value in the database.

So to fix this I needed to alter the database and manually update the external identifier to match.

Connect to your gitlab database (we are using Postgresql for our database server)

sudo -u postgres -H psql gitlabhq_production

Then select from identities to see the current values

select * from identities;

Then update the values that no longer match

update identities set extern_uid = '<new distinguished name>' where user_id = <user id>

to exit psql type

\q

The user should now be able to authenticate both with git and the gitlab web ui.

MySQL slow select on information_schema

I noticed that selects on the information_schema table was very slow. Found this guide on how to fix it

http://www.percona.com/blog/2011/12/23/solving-information_schema-slowness/

So the magic that fixed it was

set global innodb_stats_on_metadata=0;

That improved my SQL query from taking 15 seconds down to 0.03 seconds.