more spam improvements

Over the last couple of weeks I have made the following improvements in spam checking for mail handling on tarragon. Tarragon handles mail for about 20 domains, although only about a dozen have any mail to speak of.

I used to have entries in the amavis whitelist file, but this is/was a weakness. It is easy to fake sender addresses. Use of the amavis sendermaps feature is preferable as that way one can give a spamassassin bump to a known address or domain, but the value of the bump can be small enough not to overcome other attributes of the message. So egregious spam that claims to come from my own domain will still be caught. Also, I can have sendermaps for each separate email domain, instead of a whitelist applying to everyone. The file /etc/amavis/conf.d/56-sendermaps now has all the sendermaps.

A second improvement was to enable spamassassin thresholds separately for each email domain. In the file /etc/amavis/conf.d/52-spamchecks, in addition to the global spamassassin values for when to mark spam, when to reject, etc. there are now tables indexed by recipient which allow setting different thresholds for different recipients. I have used this to tighten down the settings for my own domain without running the risk of false positives for others.

A third improvement was to quarantine spam rather than discarding it, at least for spam scores below a certain cutoff level. This provides a couple of benefits. First, if I do screen out something that is wanted, it can be recovered. Second, I can do a periodic review of messages that were rejected with the recipients. I did the first round of that today. I captured information about all the mail quarantined in the last month, separated by recipient, and sent each person a list of the from addresses for review. This uncovered a few senders that needed to be added to the sendermaps.

I did this, basically by just grepping the mail log for the string ‘{DiscardedInbound,Quarantined}’ and dividing it up by destination domain, and then capturing the relevant bits of the message (the date the stored spam name, and the sender) with:

awk '{print $1 " " $2 " " $16 " from " $12}'

The results are sent to the recipient to check over.

There were a small number of emails which the intended recipient asked to be recovered. It turns out that the simple way to do that is with amavis itself.

I’ve told amavis to store the quarantined mail in /mail/quarantine. When he rejects spam the log entry is like this:

Mar 30 11:16:24 tarragon amavis[2965]: (02965-05) Blocked SPAM {DiscardedInbound,Quarantined}, []:41628 [] <> -> <>, quarantine: E/spam-EP-Jl2yDSezM.gz, Queue-ID: B1B59202F7, Message-ID: <>, mail_id: EP-Jl2yDSezM, Hits: 3.204, size: 93142, 710 ms

The name of the spam message here is E/spam-EP-J12yDSezM.gz, which is indexed by first letter, so that it is stored in: /mail/quarantine/E/spam-EP-J12yDSezM.gz.

There is a command amavisd_release, used (by root) as follows:

amavisd_release E/spam-EP-J12yDSezM.gz

This causes amavis to turn the mail over to postfix for delivery.

Spamassassin change

I seemed to have more spam getting through. When I look at those messages which I think should have been caught, I observe that many/most/almost all of them contain in the X-Spam-Status the value: RCVD_IN_DNSWL_HI=-5. Spamassassin is giving them a whopping -5 whole points if the dns source of the message appears in the High Reliability list of the site, which according to what I read, is one of those sites that maintains reputation lists, and says of the High list:

“Recommended Usage: Skip spam filtering for medium and high ranked IPs. These are trusted to send spam rarely enough that they are not worth filtering.”

There is some discussion on the net, others too seem to think they are getting a lot of spam because of this, suggesting that a site on the dnswl high list can be induced to forward spam. I know little of all of this, but I have added a rule to /etc/spamassassin/

score RCVD_IN_DNSWL_HI 0 -0.1 0 -0.1

This changes the value from -5 to -0.1. If I set it to 0 (as I originally did) then I can’t tell in X-Spam-Status whether the rule applied or not. Now I see the rule in X-Spam-Status with a small value.

So far this seems to have helped. Encouraged by this, I’ve added another couple of specifications to /etc/spamassassin/, to wit:

ok_languages en fr
ok_locales en fr

Which should act to increase the “spaminess” score of emails in other languages and character sets. A couple of mail users are French speakers, but AFAIK nobody using tarragon for mail speaks any other language or/and receives mail in another language.

IPv6 implementation

A few weeks back (10 Feb) my friend Mr. G and I exchanged an email in which he said of a possible project “…but this would be an opportunity to learn IPv6”, reminding me that I have for a long time wanted to learn more about IPv6. Part of the genesis of that email conversation was a recent switch by my brother-in-law to a new ISP that employed CGN, so called Carrier Grade Nat, which had disrupted arrangements I had in place for reaching my brother-in-law’s home network. Mr G. opined that the move towards CGN, and other things the ISPs were doing, raised the specter that someday, perhaps sooner than we expect, anyone desiring to do more with their network than occasionally use a browser would find ourselves having to move to ipv6.

More, I have actually wanted to use IPv6 for a long while, but had been under the impression (erroneously) that Comcast really wasn’t ready for this, that all they would give me was a 6to4 tunnel, which I barely understood anyway.

Continue reading IPv6 implementation

Rosemary Recovery 2020

There was a fail event reported on rosemary from one side of a pair of 60GB SSDs, which hold Rosemary_Data. Typical of my installations this mirror set holds the stuff the system needs beyond the os install: /home, databases, certificates, repositories, mail, samba, local bin, etc. Its a mirror set with an encrypted container, containing a btrfs filesystem. The older versions of these setups contain separate btrfs subvolumes for the different directories, newer ones have only one subvolume for that, and another for snaps. This is an old one.

Rosemary doesn’t have an extensive set of services – really only the /home and the databases. No real need for much of anything else. The local bin comes out of the repo anyway, there is no mail, no repository, no certs. However, without that volume the system won’t come up in a usable way. So lesson one learned here was when you get a fail event, attend to it. I let it go for a few days, because I knew I was going to have to pull the case out of the rack mount to get at the SSDs.

Continue reading Rosemary Recovery 2020

Stop pulseaudio startup under gdm

GDM is the display manager I’m using under Arch. I think it is the default DM in Arch, but I don’t really remember if I’ve changed it. In any case, the issue that has arisen is that when GDM is started by systemd after a reboot, it is launched with its own pulseaudio daemon. Then when I log in, as dee, I get a second pulseaudio daemon for dee (which is actually the desired one).

Most of the time this doesn’t matter, I guess. But I’m interested in enabling network sources in pulseaudio so I can (perhaps) have the boxes in the basement send their sound upstairs to oregano. Right now avahi is broadcasting two different network sound services on oregano, one from dee and one from gdm.

I want to stop pulseaudio from launching under gdm.

I thought this would be a simple matter of turning off autospawn in the client conf. The client conf is located in ~/.config/pulse (I think it used to be in ~/.pulse and was moved to be more “correct”). And ~ for gdm is /var/lib/gdm, so I tried client.conf in both .config/pulse/client.conf and .pulse/client.conf, but neither worked.

Poked around a little more and discovered that /usr/lib/systemd/user has a pulseaudio.service and pulseaudio.socket so these are actually being launched there by systemd. After a little reading I found that one could mask them by creating a local user override in ~/.config/systemd (for gdm this is /var/lib/gdm/.config/systemd), so I put in a /var/lib/gdm/.config/systemd/user/pulseaudio.socket and symlinked it to /dev/null.

And when I rebooted, sure enough I got not pulseaudio daemon for gdm.

Raspberry pi updates

It has become a constant annoyance that every time I build a new Raspberry Pi, the thing which is the most difficult is doing updates. I have scripts written to set everything up, but the first thing the script does is attempt to update software. Before I even reach that point, the first thing the Pi itself does is ask to update software. And it always fails.

I think this may be because of where I live. I think that the Raspbian (now Raspberry Pi OS apparently) mirrors must have one locally that is unreliable, and it gives me that one.

Two things I need to do. One is that it tries to connect using ipv6, and fails. I have to tell it to use ipv4. So one thing I end up having to do is tell apt not to use ipv6:

echo 'Acquire::ForceIPv4 "true";' > /etc/apt/apt.conf.d/99-useipv4

The second thing is to change the location of the mirror. One can look up the mirrors at, and pick one nearby, and replace the entry in /etc/apt/sources.list.

Installing Mac OS on KVM


I first tried to install Mac OS onto a VM back in 2010 or 2011 I think. I’ve come back to the task from time to time and have never been successful, but truthfully until recently all the virtual machines I’ve created on Cinnamon had been too slow to be useful anyway. Now that I have Cinnamon doing virtual machines well, I came back to the Mac – at the same time I was doing other VMs, and I didn’t keep careful track of what I did to get it up. This document is meant to record what I remember of what I did the first time, and then to do it again and keep better records.

Continue reading Installing Mac OS on KVM

Packagekit adjustments in ubuntu

When I tried to upgrade Cinnamon to Focal, I began to experience a lot of odd problems with VNC and GDM and the whole collection of machinery associated with getting a graphical environment particularly in a remote window. Cinnamon is physically downstairs in a rack in the basement, and my usual way of working is 80-90% ssh/command line and occasionally vinagre in an adjacent monitor with screens for a dozen or so places that I occasionally need to see graphically, including Cinnamon, but Vinagre usually stays pointed at Rosemary (also in the basement).

Continue reading Packagekit adjustments in ubuntu

Timemachine on Gateway pi

Some people for whom I provide some kinds of support with gateway pis, use Macs. For the pc folk – at least for those on Windows 10, I’ve been seting up to do the filehistory thing, and putting the filehistory onto the /backup drive on the gateway pi. Then it gets sent here overnight. I wanted to do the same for the folks who have Macs, of which there are several.

Continue reading Timemachine on Gateway pi

libmtp udev rules

After updating Arch today I found I could no longer mount usb storage devices – usb sticks. The log shows the device coming up, but then messages from MTP saying:

mtp-probe: bus: 2, device: 5 was not an MTP device

I have seen this before. It is irritating, I am doing something else, my head full of whatever it is, and suddenly usb sticks won’t mount.

Continue reading libmtp udev rules