Update: Notifying users of processes killed by the in-kernel OOM killer

Posted on Mon 01 May 2023 in Sysadmin • Tagged with oom memory notification email

I've updated the Notifying users of processes killed by the in-kernel OOM killer page. The code has been updated is now in GitLab.

Notable changes:

  • Uses Rsyslog's omprog facility to run the script
  • That allows you to pass the configuration file to use as a command line
    argument
  • Mail deliver …

Continue reading

Shell Script Best Practices

Posted on Thu 27 October 2022 in Sysadmin • Tagged with Bash

These are some very good things to consider when writing shell scripts.

I am not sure about point #1 though. If you are writing a script that would be in a software package you need to consider whether the package should require Bash or not. Some minimal systems only have …


Continue reading

Remote console access to a VirtualBox VM running on Linux

Posted on Thu 08 September 2022 in Sysadmin • Tagged with virtualbox

Assumptions:

  • You have a remote Linux machine with a VirtualBox VM and you want to access the VM console remotely.
  • The remote machine is named "vv000" and is accessible only via SSH.
  • The VM is named "jammy"

Procedure:

  1. SSH to the remote machine where the VM is located.
  2. Cleanly shut …


Continue reading

Notifying users of processes killed by the in-kernel OOM killer

Posted on Wed 31 August 2022 in Sysadmin • Tagged with oom, python, hacks

Updates:

  • 2023-05-01: Moved code to GitLab. Corrections to reflect how new version works.
  • 2022-09-09: Updated script to add Bcc option.

We've got servers with up to 512GB of RAM. And we've got users who periodically use it all up. Then their process gets killed by the kernel and the user …


Continue reading

Using SSH Jump Hosts

Posted on Tue 17 May 2022 in Sysadmin • Tagged with ssh

We have a server that is used as a gateway SSH server, meaning the server is used to gain access to our research servers and other machines for users who are on an external network.

The SSH gateway server responds on port 22 to internal networks and on another port …


Continue reading

Merging dicts in Ansible

Posted on Wed 27 April 2022 in Sysadmin • Tagged with ansible

I spent way to much time figuring out this simple problem. In Ansible I wanted to have a top-level dict variable in the inventory that could be overridden by a host-level dict.

Example inventory file:

all:
  vars:
    dict1:
      thing1: stuff
      thing2: more stuff
  children:
    some_machines:
      hosts:
        server-a:
        server-b:
          dict1:
            thing1 …

Continue reading

Setting a global MAILTO environment variable in Cron

Posted on Wed 27 April 2022 in Sysadmin • Tagged with linux

On Debian/Ubuntu sysetems you can set MAILTO in /etc/environment(nor in /etc/default/cron, nor in systemd unit files with DefaultEnvironment) and have it apply to all of your crontabs. MAILTO(among other Cron "environment" variables) must be set within the crontab itself. This dissapoints.

If there is …


Continue reading