Category Archives: mysql

Rosemary Recovery 2020

There was a fail event reported on rosemary from one side of a pair of 60GB SSDs, which hold Rosemary_Data. Typical of my installations this mirror set holds the stuff the system needs beyond the os install: /home, databases, certificates, repositories, mail, samba, local bin, etc. Its a mirror set with an encrypted container, containing a btrfs filesystem. The older versions of these setups contain separate btrfs subvolumes for the different directories, newer ones have only one subvolume for that, and another for snaps. This is an old one.

Rosemary doesn’t have an extensive set of services – really only the /home and the databases. No real need for much of anything else. The local bin comes out of the repo anyway, there is no mail, no repository, no certs. However, without that volume the system won’t come up in a usable way. So lesson one learned here was when you get a fail event, attend to it. I let it go for a few days, because I knew I was going to have to pull the case out of the rack mount to get at the SSDs.

Continue reading Rosemary Recovery 2020

Percona Toolkit

This is to help me remember how to get percona toolkit, so that I can get the grants information out of the mysql database – the database in mysql which keeps tracks of users, databases, and (in particular) grants, i.e. permissions.

I originally dug this up for use when I was migrating tarragon. But now I am working on migrating oregano from fedora to arch, and I will have to transfer the databases. I have finally learned (I admit I am slow) that one cannot expect to be able to move the database directory even from one release to another, let alone from one distribution to another. The only way to do this properly is to dump the databases, reinstall mariadb/mysql in the new place, and set it up, and then reimport the dumped databases.

Continue reading Percona Toolkit

Moving Oregano to arch

It is by no means certain that I will succeed with this effort, but I’m spending some time trying to get Oregano up on Arch.

The first step was just to get Arch booted up on oregano. My previous installation on a laptop didn’t involve an encrypted root, didn’t have raid arrays, didn’t have separate filesystems for things like /home and /var, didn’t run a web server, etc., so the first challenge is to get the system up with all that stuff.

Continue reading Moving Oregano to arch

Tarragon Rebuild 2019

This server, on Amazon, hosts my website and a dozen others, provides mail service for several people’s email including my own with postfix, dovecot, opendkim, amavis, spamassassin and clamd, provides contacts and calendar service using radicale, provides vpn service with openvpn, provides a tor relay, provides nextcloud service, and hosts my svn repository.

The server was last rebuilt in 2017. Long, long ago when I built the first version of it, I was most familiar with Red Hat/Fedora, and since then it has been easiest just to upgrade it with Fedora, always grumbling to myself that someday I’m going to change it. The problem with being on Fedora, of course, is that Fedora changes every 6 months, so I’m constantly behind. And after a year I’m at end of life. This is dumb for a server that I don’t want to be messing with all the time.

Continue reading Tarragon Rebuild 2019

Mysql 5.7 on Fedora 25

Had a fair amount of trouble getting mysqld to run properly on Fedora 25.

One issue was that mysql 5.7 introduces expiration of passwords, and ALL my passwords expired, including root. I couldn’t do anything, including change my password. Catch 22.

Also, the tables were out of date, and I needed a mysql_upgrade. But I couldn’t do that, because my password was expired.

Eventually figured out that I could change the password with mysqladmin password “goop”

After this, I was able to get in as root, and eventually get all the passwords marked as legitimate.

I was also able to update all the tables with mysql_upgrade.

In researching and troubleshooting this, I discovered a bunch of errors on cinnamon, with innodb generating errors of the form:

InnoDB: Ignoring tablespace `xbmc_video90/directorlinktvshow` because it could not be opened.

I’m pretty sure what happened here was that I removed the files for old xmbc databases from /var/lib/mysql, but didn’t actually drop the databases first. This is a bad habit left over from pre-Innodb when you could get rid of a mysql database just by removing the files.

I made a lot of attempts to fix this with setting in mysqld.conf:

innodb_force_recovery = 1 (or 2 or 3)

but it never worked. In the end I managed to get rid of these errors by :

mysqladmin create xbmc_videoxx (or xbmc_musicxx)

and then

mysqladmin drop xbmc_videoxx

mysqladmin generates a whole bunch of errors and warnings about not being able to find the files, but it does seem to remove the dreck from the innodb files.



Making MySQL serve UTF8 correctly

If the MySQL server decides that its default environment is UTF8, and that its client actually wants Latin1, it will translate the return values.

I’ve never before had to be careful of the distinction. Perhaps once in a blue moon I would have a record with a “real” quotation mark or a character with an accent, but if it didn’t work correctly, it was never much of a bother.

Once it became important, I had to understand what was happening. I have a table that has filenames in it, and some of those filenames contain characters (a acute, e grave, o umlaut, etc). The actual files on disk have the names encoded in utf8. The records in the database are also recorded in utf8. But the records were being translated by mysql from utf8 to latin1 as they came in. So “Mamá” was recorded on disk as Mam\xc3\xa1 in the directory, and in the database, but when I got the row into memory, the filename field said Mam\xe1. The difference between latin1 and utf8 for this purpose, is that all these many “western/latin special characters” were actually mapped in latin1 to values within the 256 characters available with 1 byte. So the first 128 in the latin1 codespace were ordinary ascii, and the high order 128 had as many of the western/latin diacriticals as possible crammed in there. And in latin1, e9 is a-acute.

But on the web these days, utf8 is much preferred. Latin1 is ok if all you want is the carefully selected subset of 128 characters that can be shoehorned into the high end of the code-point space. But utf8 is a far more general solution. Using a multibyte sequence to represent over a million characters and special symbols.

Turns out mysql has a bunch of variables to control character set and collating sequence. With phpmyadmin, one can look at database->Variables and see characters_set_database, character_set_filesystem, character_set_result, character_set_server, character_set_system, and a bunch more. Or in mysql client one can show varaiables like ‘%character_set%’;

My problem was that the server had come up believing that some of these were set to utf8 and some were set to latin1. I haven’t tried to figure out the logic of how it figures out its default – I don’t want it to default, I want to tell it what I want. So the solution was to add the directive: “character-set-server=utf8” to the mysql configuration file (on cinnamon it was /etc/mysql/mysql.conf.d/mysqld.conf. After restarting all of the relevant character_set_xxx variables come up as utf8.

Update: 8/9/17

I used these changes also on oregano and tarragon, but it results in a different problem for me on blogforacure data. The blogforacure database, built a long time ago, has lots of tables in latin1. There are not a lot of non-ascii characters, but there are a few. One frequent source is people typing double space after a period, which the ckeditor tries to preserve by creating a non-breaking space, which is hex A0 in latin 1. When the site reads this back, if mysql is told that the database is actually utf8, then it displays this as Â. So if I see a bunch of A circumflex in the output, it means I actually have latin 1 characters in the database, which I am interpreting as if they were utf-8.

Removing the specification of character-set-server=utf8 causes the negotiation to give the right result, and the latin1 non-breaking space appears correctly in the output.

MYSQL on Ubuntu 15.10

I haven’t researched whether this has changed, in 15.10, or whether it has been this way since ubuntu switched to systemd, which is probably the case.

Under systemd, ubuntu no longer uses the /etc/init.d/mysql script, but instead uses a systemd unit in /lib/systemd/system/mysql.service which invokes /usr/bin/mysqld_safe to start and stop mysqld.

I have had a lot of trouble with this, and had to do a lot of debugging to figure out what is going on. Probably I would not have had trouble if I were not trying to port over a running mysql installation manually, i.e. if I just installed mysql-server and proceeded to create new databases, new entries in mysql etc.

One issue is that a mysql install creates a file in /etc/mysql called debian.cnf which contains a user/password for user debian-sys-maint with a generated password, and this is put into the mysql users table, to enable various operations to be performed by mysqladmin using these credentials.

The first problem was that when I copied over the mysql table from the previous installation, I was copying in the old password for debian-sys-main, which didn’t match the debian.cnf file which was installed when I did the apt-get install mysql-server. So I had to read the debian.cnf file, extract the password and change the password in the mysql table.
Continue reading MYSQL on Ubuntu 15.10

Using pam_mysql for authentication

On the occasion of moving the server to amazon, I decided to stop using LDAP. I was making very little use of it, having started to keep my contacts elsewhere. All it was really doing was providing the authentication file for apache logins. And there were so few of those it was quite easy to manage with an htpasswd file.

But recently I’ve begun providing mail to some friends, and while I am happy to provide them with system accounts the problem is that they don’t really WANT system accounts – they just want mail. So the husband asked me, how do I change my password? And the only answer I have is, you have to log on. Worse than that of course, I can’t really even allow him to log on without making him set up for public key authentication.

I would like to enable people to use the server for mail, and to be able to authenticate with apache, and I would like them to manage their own passwords for this, without making them log on via ssh, which would require that they have a key pair registered with me. I looked into being able to change the system password via a webpage, but it looks very messy, and seriously – system passwords are supposed to require a human being – that is intentional.

So I looked into authenticating the mail, and apache, using a database. I found a pam module called pam_mysql which I can use with saslauthd. I set /etc/pam.d/imap and smtp to point to a new pam entry called mail, which uses pam.mysql to autheticate against the database (and still also authenticate against system accounts as well if there is no entry in the database).
Continue reading Using pam_mysql for authentication