DeepBSD’s Blog

Learning Bash

2021-09-23T18:05:27+00:00

How Learning Bash Helps you Learn Linux

When you set out to “Learn Linux” you’re actually embarking on a long journey of UNIX mastery. You’re joining a group of geeks whose ancestry goes back to the 1960’s, and perhaps even before, if you consider that UNIX didn’t grow out of nothing. Learning the Bourne Again SHell will teach you long-standing principles that have served programmers, admins, and geeks well since before 64k was a lot of memory up until now.

By the way, “UNIX” is a copyrighted product, but for many of us, “Unix” is a way of life. So I frequently use “Unix” to mean “Unix-like operating system” such as Linux.

The Terminal (Working In Your Head, Not the Display)

In this modern day and age, most people are used to graphical interfaces or GUIs. The terminal feels like a foreign planet to most computer users nowadays. But graphical layers depend on many other programs and devices working correctly. If any of those go wrong, only a console or terminal might be available. Even very simple problems can cause a graphical layer to go away. If you depend on only the graphical layer, you will experience more downtime and won’t be able to fix many not-so-difficult problems.

The early computers didn’t have the horsepower to show graphical layers. The first computers didn’t even have keyboards or monitors, let alone mice. Computer programs used to be stored on perforated paper. Even large floppy disks were an improvement. And people got lots of work done in those days by doing mental work in their heads instead of on the screen. So really, a terminal is a luxury!

To make sense of the terminal, you have to understand how computers are built and how they work. You won’t be seeing pictures of a hard disk or power supply or keyboard. You have to understand the basic hardware and ideas that make things work. So this article assumes you have that. If you don’t know what a network connection or memory card or disk drive is, you probably are not yet ready for Linux or Bash. (Even the term “card” and “disk” is starting to sound rather old.)

Keep Pieces Small, But Small Pieces Together Can Form Big Pieces

One of the first principles of Unix is that programs are designed to be small and do very specific things, but that these small programs can be joined together to do large, sophisticated jobs using Unix plumbing.

UNIX plumbing is a way of talking about redirecting input and output on the terminal. The notion of data flowing in a direction from one thing to another thing is not new. But in the terminal we constantly thing of STD_IN and STD_OUT and something else called STD_ERR. This is simply data flowing in, data flowing out, and then something going wrong with the flow of data.

Mind you, these small programs we’re talking about need data to work on. Let’s create a simple command at the terminal and print the word “Linux” to the display: echo 'Linux' at the terminal will cause the STD_OUT to be “Linux”.

To see an example of STD_IN you could take the STD_OUT of this command and “pipe” it (UNIX plumbing term) so it becomes the STD_IN of another program (we’ll use tr): echo 'Linux' | tr a-z A-Z

The end result of this is ‘LINUX’ at the prompt. echo produces and STD_OUT of ‘Linux’. Because of the pipe symbol ‘|’, this output now becomes the input to another program called tr. This is a terrific little program which I encourage you to read about. It simply “translates” or “modifies” some textual input to some kind of textual output. It can do different things to characters in a string. In this case, we’re replacing all lowercase characters with uppercase characters. The STD_OUT of one program has become the STD_IN of another program using Unix plumbing. This is a very, very useful concept that happens a lot in Linux and Unix.

You can stack up the commands in a sequence: echo "Linux" | tr a-z A-Z | sed 's/LINU/UNI/' sed is a terrific and powerful program that edits streams of text. Here, tr translates the text to uppercase and sed substitutes ‘LINU’ with ‘UNI’ to output the word ‘UNIX’ from the original ‘Linux’.

We have taken STD_OUT from one program and piped into STD_IN of another program, and then done the same thing again with that output so it becomes STD_IN for yet another program (sed) to produce yet another example of STD_OUT: ‘UNIX’.

There are many, many more examples of plumbing with Linux with more operators. The pipe symbol is just one. The point is, we’re using very small things to accomplish large tasks by gluing the small things together. This idea is fundamental to Bash and Unix/Linux.

Reusing Small and Useful Tools

The plumbing examples above show off another very important principle: reusing common tools.

Imagine you had a screwdriver that had an unlimited number of bits for it. You could use the same screwdriver on slot head screws as well as phillips head screws as well as Torx head screw as well as anything else. These small Unix/Linux tools are very much like that.

Let’s use some plumbing together with the tr command and show how flexible small and reusable tools can be when used together: echo 'I Love Linux' | tr a-z A-Z | tr -s ' ' '\n' | tr -d 'X' > ~/ilike.txt

This example reuses the tr command to uppercase a string, to substitute newlines for blank spaces, and to remove a character from the final string, before redirecting the output to a filename in the user’s home directory (which is another example of Unix/Linux plumbing).

When you use this command, you won’t see anything on STD_OUT, because that output has been redirected to a filename. If you type cat ~/ilike.txt then you’ll see the output. tr has been reused three times in the same command, each with a unique result.

When you start using Bash, you may easily underestimate how flexible and re-usable each of the simple commands can be in your day-to-day work. Many of these small commands will become indispensible tools you cannot live without each day.

Pre-eminance of Text

You’ll notice that each of these commands uses text in STD_OUT and STD_IN. That’s by intention. Linux/Unix depends on text files, not binary files so much. There can be individual programs in Linux/Unix that use binary files, but most of Linux/Unix just use plain old text files. You’ll edit your Bash programs with a simple text editor. It will create simple, flat text files. The advantage to this is you can easily fix things by altering text files.

Computers break rather often. Even when just doing routine maintenance, your system will be in some sort of broken or semi-broken state. If you depend on flat text files to reconfigure and fix things, those are easy to repair even with a broken system. Text files can almost always be edited, even under severe breakage of a system. No special tools required. This has been one of the most beloved aspects of Unix: text files are key.

This implies another of the fundamental princples: Keep it simple. Text is simple.

Silence is Golden

With Bash, if everything went okay, the terminal will not say anything extra. When a Bash program has nothing surprising to report, it stays mum. If you type echo 'Hello World' and execute it, there’s no notice that your command succeeded. There’s only the output your program created.

Beyond that, you can see if you got a non-zero or a zero exit status: echo $?. Basically, if there’s a problem, Linux/Unix will probably tell you. It just assumes you know what you’re doing. It will only throw some kind of error if you made a syntax error or something. Otherwise, Bash will simply try to give you exactly what you asked for. And this is consistent with Linux/Unix design.

Keep it Simple

Cleaverness kills in programming. Just be clear. Not only will you be following Unix philosophy, but people will be able to read your Bash scripts. Also, whenever you’re ready to put a script into production, ask yourself if you could understand what you’re doing even if you have a really bad hangover.

It’s easy to indulge our egos when we’re feeling cleaver and show off how smart we were that day in our script. But that “successful” feeling can turn into annoyance as soon as you stop feeling very cleaver. Write the script so you can follow your own work on your dumb days too, not just your cleaver days.

Whiptail Progress Gauges

2021-04-30T15:16:27+00:00

Creating Progress Gauges with Whiptail

Whiptail is a dandy TUI (terminal user interface) toolkit that helps you make bash scripts for non-technical users. Or for people who just don’t like terminals. The terminal is a huge source of information. Sometimes it’s too much information. When it is, a toolkit like Whiptail can make your scripts seem more friendly. In particular, though, progress gauges can seem a little problematic, because you must provide the indication of progress yourself. The --gauge switch doesn’t possess any magical qualities that help it to know how much more a process must operate before it completes. In fact, it has absolutely no way to know the progress of a process toward completion. You must program that progress yourself. I’m going to talk about that in this article.

The Basic Syntax

It’s worth reading the man page for all the switches for Whiptail. But the switches you’ll use a lot are --title, --infobox, --msgbox, --yesno, --radiolist, --inputbox, --textbox, --passwordbox, --menu, --checklist, and --gauge. Followed by the dimensions of the box itself in lines and columns. Example:

name=$(whiptail --title "Sample input" --inputbox "What is your name?" 8 60 3&>1 1>&2 2&>3)

Some of the switches need to redirect STDIN and STDOUT, so we need to redirect it back so that everything works. Here we capture user input into a variable that we can use later. You’ll have to read up more on that redirection elsewhere, because I’m going to focus on the --gauge switch.

The Problem with Progress Gauges

As I mentioned, you need to solve the problem of displaying a stream of numbers between 1 and 100 that meaningfully show the progress a UNIX process is making. The --gauge switch has no built-in way of doing that. That means that whatever process you’re measuring, you need to find a way of providing --gauge a way of knowing its progress.

In most tutorials for whiptail I see something like the following:

{
    for ((i=0; i<=100; i+=1)); do
        sleep 0.1
        echo $i
    done
} | whiptail --backtitle "PROGRESS GAUGE" --title "Calculating Result" --gauge "Please wait for calculation" 8 50 0

This will work. It will display a progress bar while some process has been executed, but there will probably be no correlation between the process’s progress and the gauge. What are some other possibilities?

Let’s assume for now that our process is abstract enough that we cannot measure the physical size of a target file and a destination file. That would be easy enough to show in a progress bar. And if your gauge just needs to measure a shrinking differential between two physical objects like that, then your calculation is simple. But what if all you have is a process that doesn’t easily lend itself to being measured? Where all you have is a PID?

First, we’ll have to launch the process in the background which means spawning a subshell. Otherwise, we couldn’t measure the process in the same script. So, remember that! We need to launch the process to the background and capture its PID, then watch for the process to disappear from the PID table. How on Earth do we make that happen?

Well, I would recommend passing the name of the function to execute in the background as a parameter to the processgauge function. We need three workers here. So that would mean something like this:

The Calculator, The Caller and the Gauge

calculate(){                                                    ## The Calculator! 'How many angels on a pidhead?'
    echo `Highly technical, time-consuming process here...`
    sleep 600  # sleep for 10 minutes
}

showprogress(){                                        ## The Gauge:  Produce the number stream
    start=$1; end=$2; shortest=$3; longest=$4

    for n in $(seq $start $end); do
        echo $n
        pause=$(shuf -i ${shortest:=1}-${longest:=3} -n 1)  # random wait between 1 and 3 seconds
        sleep $pause
    done
}

processgauge(){                                         ## The Caller:  Start the gauge and watch the PID
    process_to_measure=$1
    message=$2
    backmessage=$3
    eval $process_to_measure &
    thepid=$!
    num=25
    while true; do
        showprogress 1 $num 1 3
        sleep 2
        while $(ps aux | grep -v 'grep' | grep "$thepid" &>/dev/null); do
            if [[ $num -gt 97 ]] ; then num=$(( num-1 )); fi
            showprogress $num $((num+1))
            num=$((num+1))
        done
        showprogress 99 100 3 3
    done  | whiptail --backtitle "$backmessage" --title "Progress Gauge" --gauge "$message" 6 70 0
}

This idea relies on capturing the PID of the process to measure. Then producing numbers up until the process drops out of the PID table. We get a guaranteed number stream up to 25, but you can set num to whatever you want. You can also make the sleep period longer by offering larger numbers for the shortest and longest sleep values. So one advantage of this approach is that it is flexible.

This type of approach works for short processes and long processes. You can set longer sleep times in process gauge (The Caller) for a longer process or shorter sleep times. You can make consistent sleep times if you like. Unfortunately, shuf will not accept decimal arguments. They must be integers, even though sleep will accept decimal arguments. You could probably find another way of managing that problem if you like. For now, this works for me. Perhaps in the future I’ll generate shorter sleep times than one second.

So, from a menu to run calculate, you could call

processgauge calculate "Calculating Maximum Beer Intake" "Calculating MAXBEERS..."

You’ll have to experiment with different calculations to see how you like the progress bar with different scenarios. On shorter processes, the progress bar may jump from 25 to 100 percent really quickly. On longer processes, the bar may go back and forth between 97 and 98 percent for a long time. I chose this vacilation back and forth because at least you know the process hasn’t gotten hung up. At least it’s still running. You could put something else in calculate that outputs to a logfile and then tail -f that logfile to show any errors encountered or other issues. For example:

calculate(){
    logfile="./logfile"
    length=600
    [[ -f $logfile ]] && rm $logfile
    echo "=== START ===" &>$logfile
    for ((i=0; i<=$length; i+=1)); do
        echo "$i:" &>>$logfile
        date +"%D::%N" &>>$logfile
    done
    echo "=== END ===" &>>$logfile
}

Now for longer processes in the background you can watch a logfile to see how it’s doing.

Conclusion

This is only one beginning. I’m sure there are lots more approaches to creating a progress gauge with Whiptail. I like the flexibility. I like how easily I can adapt it. But this approach got me going for today. I’m back in business with Whiptail!

Arch Linux Install Script

2021-02-24T02:58:29+00:00

Arch Linux Install Script

Many new Linux users eventually gravitate toward a more minimal Linux distro where they essentially get to build their own distro, according to their own preferences rather than Canonical’s or another big distro sponsor. Arch Linux is a terrific destination for these intermediate Linux users, because there is a lot of freedom and choice. You get to build your own distro, since you install it by hand. After you’ve installed Arch by hand a few times, you’ll probably want to create your own Arch install script.

Since you’ve already installed the distro by hand several times, you probably already have a list of preferred applications and ways of installing Arch. This script will reflect the simplest possible steps that are automated yet easily comprehended by someone not totally brand new to bash scripting.

This script, named simplest.sh, is mainly a skeleton that you can modify and that can grow with your growing scripting skills. WARNING: I’m using an MBR disk label here. This will probably not work with a UEFI Motherboard or EFI bios. For that you should really use a GPT disk label with an EFI partition formatted with FAT-32. My other install scripts use this format, or they are capable of branching to install with GPT disk label and EFI-friendly partitioning scheme according to whether a fully compliant EFI bios exists. I’ll revisit this later.

When you boot up the archiso disk image from your thumbdrive or ISO file, you’ll need to fetch this script. From a terminal, you can get it by typing curl -O https://raw.githubusercontent.com/deepbsd/farchi/master/simplest.sh The simplest.sh file will appear in your root directory. You should edit the variables to conform to your preferences. For example, if you have a 20G virtual drive (or hard drive) and 2G of RAM, you’ll perhaps want 10G of root partition and 4G of swap partition. You’ll want your own hostname and so on. You’ll need to edit the script before you run it, obviously. For testing on a VM I usually create a 30G virtual disk with 1G of RAM.

You can grab the Farchi script like this:
curl -O https://raw.githubusercontent.com/deepbsd/farchi/master/farchi.sh Obviously, my username on Github is deepbsd. The name of the repo is farchi. And you’ll want to snag the master branch of that repo.

Here’s my simplest.sh script, which should just give you a starting place for your own script. This script does not install X. You can have that as a future exercise! Or you can use my install_x.sh script at the same location if you need ideas. But if you just want to install a basic, vanilla Arch system, this script takes about 5-7 minutes on my i5-4690k with 32G of RAM and an NVME SSD, depending on network speed, mostly. Because of the number of dependencies for X, installing X takes about twice that long.

#!/usr/bin/env bash
   
##  This is the simplest possible Arch Linux install script I think...
HOSTNAME="marbie1"
#VIDEO_DRIVER="xf86-video-vmware"
IN_DEVICE=/dev/sda
BOOT_DEVICE="${IN_DEVICE}1"
ROOT_DEVICE="${IN_DEVICE}2"
SWAP_DEVICE="${IN_DEVICE}3"
HOME_DEVICE="${IN_DEVICE}4"
   
BOOT_SIZE=512M
SWAP_SIZE=2G
ROOT_SIZE=13G
HOME_SIZE=    # Take whatever is left over after other partitions
TIME_ZONE="America/New_York"
LOCALE="en_US.UTF-8"
#KEYBOARD="us"    # change if you need to
FILESYSTEM=ext4
   
use_bcm4360(){ return 1; }  # return 0 for "truthy" and 1 for "falsy"
   
if $(use_bcm4360) ; then
   WIRELESSDRIVERS="broadcom-wl-dkms"
else
   WIRELESSDRIVERS=""
fi
   
BASE_SYSTEM=( base base-devel linux linux-headers linux-firmware dkms vim iwd )
   
devel_stuff=( git nodejs npm npm-check-updates ruby )
printing_stuff=( system-config-printer foomatic-db foomatic-db-engine gutenprint cups cups-pdf cups-filters cups-pk-helper ghostscript gsfonts )
multimedia_stuff=( brasero sox eog shotwell imagemagick sox cmus mpg123 alsa-utils cheese )
   
# VERIFY BOOT MODE
efi_boot_mode(){
   [[ -d /sys/firmware/efi/efivars ]] && return 0
   return 1
}
   
# All purpose error
error(){ echo "Error: $1" && exit 1; }
   
###############################
###  START SCRIPT HERE
###############################
   
### Check that reflector is done
clear
echo "Waiting until reflector has finished updating mirrorlist..."
while true; do
   pgrep -x reflector &>/dev/null || break
   echo -n '.'
   sleep 2
done
   
### Test internet connection
clear
echo "Testing internet connection..."
$(ping -c 3 archlinux.org &>/dev/null) || (error "Not Connected to Network!!!")
echo "Good!  We're connected!!!" && sleep 3
   
## Check time and date before installation
timedatectl set-ntp true
echo && echo "Date/Time service Status is . . . "
timedatectl status
sleep 4
   
$(efi_boot_mode) && error "You have a UEFI Bios; Please use the Farchi or Darchi script for installation"
   
####  Could just use cfdisk to partition drive
#cfdisk "$IN_DEVICE"    # for non-EFI VM: /boot 512M; / 13G; Swap 2G; Home Remainder
   
###  NOTE: Drive partitioning is one of those highly customizable areas where your
###        personal preferences and needs will dictate your choices.  Many options
###        exist here.  An MBR disklabel is very old, limited, and may well inspire
###        you to investigate other options, which is a good exercise.  But, MBR is pretty
###        simple and reliable, within its constraints.  Bon voyage!
   
   
   
# Using sfdisk because we're talking MBR disktable now...
cat > /tmp/sfdisk.cmd << EOF
$BOOT_DEVICE : start= 2048, size=+$BOOT_SIZE, type=83, bootable
$ROOT_DEVICE : size=+$ROOT_SIZE, type=83
$SWAP_DEVICE : size=+$SWAP_SIZE, type=82
$HOME_DEVICE : type=83
EOF
   
sfdisk "$IN_DEVICE" < /tmp/sfdisk.cmd 
   
#####  Format filesystems
mkfs.ext4 "$BOOT_DEVICE"    # /boot
mkfs.ext4 "$ROOT_DEVICE"    # /
mkswap "$SWAP_DEVICE"       # swap partition
mkfs.ext4 "$HOME_DEVICE"    # /home
   
#### Mount filesystems
mount "$ROOT_DEVICE" /mnt
mkdir /mnt/boot && mount "$BOOT_DEVICE" /mnt/boot
swapon "$SWAP_DEVICE"
mkdir /mnt/home && mount "$HOME_DEVICE" /mnt/home
   
lsblk && echo "Here're your new block devices. (Type any key to continue...)" ; read empty
   
   
###  Install base system
clear
echo && echo "Press any key to continue to install BASE SYSTEM..."; read empty
pacstrap /mnt "${BASE_SYSTEM[@]}"
echo && echo "Base system installed.  Press any key to continue..."; read empty
   
# GENERATE FSTAB
echo "Generating fstab..."
genfstab -U /mnt >> /mnt/etc/fstab
cat /mnt/etc/fstab
echo && echo "Here's your fstab. Type any key to continue..."; read empty
   
## SET UP TIMEZONE AND LOCALE
clear
echo && echo "setting timezone to $TIME_ZONE..."
arch-chroot /mnt ln -sf /usr/share/zoneinfo/"$TIME_ZONE" /etc/localtime
arch-chroot /mnt hwclock --systohc --utc
arch-chroot /mnt date
echo && echo "Here's the date info, hit any key to continue..."; read empty
   
## SET UP LOCALE
clear
echo && echo "setting locale to $LOCALE ..."
arch-chroot /mnt sed -i "s/#$LOCALE/$LOCALE/g" /etc/locale.gen
arch-chroot /mnt locale-gen
echo "LANG=$LOCALE" > /mnt/etc/locale.conf
export LANG="$LOCALE"
cat /mnt/etc/locale.conf
echo && echo "Here's your /mnt/etc/locale.conf. Type any key to continue."; read empty
   
## HOSTNAME
clear
echo && echo "Setting hostname..."; sleep 3
echo "$HOSTNAME" > /mnt/etc/hostname
   
cat > /mnt/etc/hosts <<HOSTS
127.0.0.1      localhost
::1            localhost
127.0.1.1      $HOSTNAME.localdomain     $HOSTNAME
HOSTS
   
echo && echo "/etc/hostname and /etc/hosts files configured..."
echo "/etc/hostname . . . "
cat /mnt/etc/hostname 
echo "/etc/hosts . . ."
cat /mnt/etc/hosts
echo && echo "Here are /etc/hostname and /etc/hosts. Type any key to continue "; read empty
   
## SET ROOT PASSWD
clear
echo "Setting ROOT password..."
arch-chroot /mnt passwd
   
## INSTALLING MORE ESSENTIALS
clear
echo && echo "Enabling dhcpcd, pambase, sshd and NetworkManager services..." && echo
arch-chroot /mnt pacman -S git openssh networkmanager dhcpcd man-db man-pages pambase
arch-chroot /mnt systemctl enable dhcpcd.service
arch-chroot /mnt systemctl enable sshd.service
arch-chroot /mnt systemctl enable NetworkManager.service
arch-chroot /mnt systemctl enable systemd-homed
echo && echo "Press any key to continue..."; read empty
   
## ADD USER ACCT
clear
echo && echo "Adding sudo + user acct..."
sleep 2
arch-chroot /mnt pacman -S sudo bash-completion sshpass
arch-chroot /mnt sed -i 's/# %wheel/%wheel/g' /etc/sudoers
arch-chroot /mnt sed -i 's/%wheel ALL=(ALL) NOPASSWD: ALL/# %wheel ALL=(ALL) NOPASSWD: ALL/g' /etc/sudoers
echo && echo "Please provide a username: "; read sudo_user
echo && echo "Creating $sudo_user and adding $sudo_user to sudoers..."
arch-chroot /mnt useradd -m -G wheel "$sudo_user"
echo && echo "Password for $sudo_user?"
arch-chroot /mnt passwd "$sudo_user"
   
## INSTALL WIFI
$(use_bcm4360) && arch-chroot /mnt pacman -S "$WIRELESSDRIVERS"
[[ "$?" -eq 0 ]] && echo "Wifi Driver successfully installed!"; sleep 5
   
## Not installing X in this script...
   
## INSTALL GRUB
clear
echo "Installing grub..." && sleep 4
arch-chroot /mnt pacman -S grub os-prober
   
## We're not checking for EFI; We're assuming MBR
arch-chroot /mnt grub-install "$IN_DEVICE"
   
echo "configuring /boot/grub/grub.cfg..."
arch-chroot /mnt grub-mkconfig -o /boot/grub/grub.cfg
[[ "$?" -eq 0 ]] && echo "mbr bootloader installed..."
   
echo "Your system is installed.  Type shutdown -h now to shutdown system and remove bootable media, then restart"
read empty

Notes About Simplest.sh

As I mentioned, I’m using an MBR disk label. You’ll want to change this if you’re installing to a newer motherboard or BIOS.

The first thing to do is set up my variable names as preferences. Things like partition names and sizes, hostname, wifi driver, video chipset driver, swap space size, root partition size (I assume a 30G VM disk to start with), timezone, keyboard layout (US is assumed), and locale and filesystem format (I assume ext4).

I normally include a few extra packages with dependencies, like printing and networking utilities, a few multimedia things and some development goodies. I could easily include X and an appropriate display manager and desktop environment or window manager. I happen to like Cinnamon, but I normally install XFCE and i3wm as well, just to have options (sometimes I like to switch things up!). For this script, I opted out of installing X, but you can check out my Farchi script or my Darchi script. You can see how I branch conditionally from preferences. I figure you’ll want to just see if the script works for you.

If you want to modify the script to create create GPT partitions, you could use sgdisk.
That’s what I use in the Farchi and Darchi scripts. You could create all the partitions with simply using cfdisk, but I think it’s a good exercise to automate it. Plus, sfdisk and sgdisk are really fast. You can create partitions and volume groups in 3 seconds or less! It’s wicked fast!

You could use a phrase something like this to create partitions using sgdisk:

 if $(efi_boot_mode); then
    sgdisk -Z "$IN_DEVICE"
    sgdisk -n 1::+"$EFI_SIZE" -t 1:ef00 -c 1:EFI "$IN_DEVICE"
    sgdisk -n 2::+"$ROOT_SIZE" -t 2:8300 -c 2:ROOT "$IN_DEVICE"
    sgdisk -n 3::+"$SWAP_SIZE" -t 3:8200 -c 3:SWAP "$IN_DEVICE"
    sgdisk -n 4 -c 4:HOME "$IN_DEVICE"
 fi

Of course, these variables would have to be defined, and so on. You’d have to define the efi_boot_mode function something like I did in the simplest.sh script. And you’d have to complete the rest of the conditional accordingly.

Further, you’d need to install the EFI boot manager instead of a normal non-efi boot loader. I typically use GRUB with the efi-bootmanager package and the --target and --efi-directory flags. Works fine for me so far.

When you’re installing packages and their dependencies, you’ll want to pause at certain times to see if any packages installed with a non-zero exit status. You’ll want to take note of any errors and determine whether they are significant. Normally they are not. For example, sometimes a package gets installed twice, but this is not a problem. Pacman is happy to reinstall packages usually. But I’ve included some read statements to allow seeing the final notes of a process before the screen clears and any errors or information scrolls past, away from your notice.

I like to use ‘here’ documents. You’ll see me using arrays and associative arrays in bash. Also I use && instead of if/then where it saves time and doesn’t obscure readability. I like to pipe unneccesary output to /dev/null when I’m checking exit statuses for functions. Finally, I’m still experimenting with systemd-homed and pambase. I’ve had some problems when installing X: I have trouble using sudo from an X terminal. Not sure why that is, but reinstalling pambase and systemd-homed solved the problem. I’m still not sure how this problem got started in the first place, but I assumed there was a package change by the maintainers. More study for me in the future! You should probably experiement with this situation and see what works best for you.

Conclusion

Hopefully this will get you started with installing Arch linux automatically. When I first started, I simply used cfdisk /dev/sda to create my partitions. This script was much rougher and more ragged. But it evolved. The Farchi and Darchi scripts are more evolved. You can see that in just more than 150 lines of Bash, an entire system get installed. If you wanted to install X, that’s probably another 50-75 lines. So you can see it doesn’t take a whole lot of scripting to get your basic Arch system up and running! I’m hoping you’ll be able to borrow this script to get started on your own journey. Bon voyage!

Remove Underline From Vim hi CursorLineNr

2020-11-21T21:18:37+00:00

Removing Underline from Vim Hilighted CursorLineNr

If you’re like me, you work hard to get your editor to look and act just as you like it.
When something changes without your permission, and now your editor has started looking like it’s gone rogue, that’s bad. I’m not sure what drove this change, but some time ago my hi CursorLineNr value in my color theme started displaying an underline in my editor. It kinda drove me nuts. Even now, when I log into a new system where my syntax coloring files have not been updated, I have to make the change to de-gunk my line number on the cursor line. Here’r are some notes.

What files to change

In your .vimrc, you normally refer to a syntax color file. Mine is xoria256.vim. It’s located in $HOME/.vim/colors/xoria256.vim. (Your installation may vary.) In your .vimrc there should be a line that goes something like colorscheme xoria256 or whatever the name of your color theme is. But what really needs modifying is the colorscheme file itself.

Here is what mine looks like:

" Vim color file
"
" Name:       xoria256.vim
" Version:    1.6
" Maintainer:	Dmitriy Y. Zotikov (xio) <xio@ungrund.org>
"
" Should work in recent 256 color terminals.  88-color terms like urxvt are
" NOT supported.
"
" Don't forget to install 'ncurses-term' and set TERM to xterm-256color or
" similar value.
"
" Color numbers (0-255) see:
" http://www.calmar.ws/vim/256-xterm-24bit-rgb-color-chart.html
"
" For a specific filetype highlighting rules issue :syntax list when a file of
" that type is opened.
"
" TODO: link colours instead of setting values explicitly

" Initialization {{{
if &t_Co != 256 && ! has("gui_running")
 echoerr "Please use GUI or a 256-color terminal (which sets t_Co=256)."
 finish
endif

set background=dark

hi clear

if exists("syntax_on")
 syntax reset
endif

let colors_name = "xoria256"
"}}}
" Colours {{{1
"" General {{{2
hi Normal       ctermfg=252 guifg=#d0d0d0 ctermbg=234 guibg=#1c1c1c cterm=none gui=none
hi Cursor                                 ctermbg=214 guibg=#ffaf00
hi CursorColumn                           ctermbg=238 guibg=#444444
"hi CursorLine                             ctermbg=237 guibg=#3a3a3a cterm=none gui=none 
hi CursorLine                             ctermbg=237 ctermfg=grey guibg=#3a3a3a cterm=none gui=none 
hi CursorLineNr                             ctermfg=white ctermbg=magenta guibg=#3a3a3a cterm=none gui=none 
hi Error        ctermfg=15  guifg=#ffffff ctermbg=1   guibg=#800000
hi ErrorMsg     ctermfg=15  guifg=#ffffff ctermbg=1   guibg=#800000
hi FoldColumn   ctermfg=247 guifg=#9e9e9e ctermbg=233 guibg=#121212
hi Folded       ctermfg=255 guifg=#eeeeee ctermbg=60  guibg=#5f5f87
hi IncSearch    ctermfg=0   guifg=#000000 ctermbg=223 guibg=#ffdfaf cterm=none gui=none
hi LineNr       ctermfg=247 guifg=#9e9e9e ctermbg=233 guibg=#121212
hi MatchParen   ctermfg=188 guifg=#dfdfdf ctermbg=68  guibg=#5f87df cterm=bold gui=bold
" TODO
" hi MoreMsg
hi NonText      ctermfg=247 guifg=#9e9e9e ctermbg=233 guibg=#121212 cterm=bold gui=bold
hi Pmenu        ctermfg=0   guifg=#000000 ctermbg=250 guibg=#bcbcbc
hi PmenuSel     ctermfg=255 guifg=#eeeeee ctermbg=243 guibg=#767676
hi PmenuSbar                              ctermbg=252 guibg=#d0d0d0
hi PmenuThumb   ctermfg=243 guifg=#767676
hi Search       ctermfg=0   guifg=#000000 ctermbg=149 guibg=#afdf5f
hi SignColumn   ctermfg=248 guifg=#a8a8a8
hi SpecialKey   ctermfg=77  guifg=#5fdf5f
" hi SpellBad     ctermfg=160 guifg=fg      ctermbg=bg                cterm=underline               guisp=#df0000
hi SpellBad                               ctermbg=238                                               guisp=#df0000
hi SpellCap     ctermfg=189 guifg=#dfdfff ctermbg=bg  guibg=bg      cterm=underline gui=underline
hi SpellRare    ctermfg=168 guifg=#df5f87 ctermbg=bg  guibg=bg      cterm=underline gui=underline
hi SpellLocal   ctermfg=98  guifg=#875fdf ctermbg=bg  guibg=bg      cterm=underline gui=underline
hi StatusLine   ctermfg=15  guifg=#ffffff ctermbg=239 guibg=#4e4e4e cterm=bold gui=bold
hi StatusLineNC ctermfg=249 guifg=#b2b2b2 ctermbg=237 guibg=#3a3a3a cterm=none gui=none
hi TabLine      ctermfg=fg  guifg=fg      ctermbg=242 guibg=#666666 cterm=none gui=none
hi TabLineFill  ctermfg=fg  guifg=fg      ctermbg=237 guibg=#3a3a3a cterm=none gui=none
" FIXME
hi Title        ctermfg=225 guifg=#ffdfff
hi Todo         ctermfg=0   guifg=#000000 ctermbg=184 guibg=#dfdf00
hi Underlined   ctermfg=39  guifg=#00afff                           cterm=underline gui=underline
hi VertSplit    ctermfg=237 guifg=#3a3a3a ctermbg=237 guibg=#3a3a3a cterm=none gui=none
" hi VIsualNOS    ctermfg=24  guifg=#005f87 ctermbg=153 guibg=#afdfff cterm=none gui=none
" hi Visual       ctermfg=24  guifg=#005f87 ctermbg=153 guibg=#afdfff
hi Visual       ctermfg=255 guifg=#eeeeee ctermbg=96  guibg=#875f87
" hi Visual       ctermfg=255 guifg=#eeeeee ctermbg=24  guibg=#005f87
hi VisualNOS    ctermfg=255 guifg=#eeeeee ctermbg=60  guibg=#5f5f87
hi WildMenu     ctermfg=0   guifg=#000000 ctermbg=150 guibg=#afdf87 cterm=bold gui=bold

"" Syntax highlighting {{{2
hi Comment      ctermfg=244 guifg=#808080
hi Constant     ctermfg=229 guifg=#ffffaf
hi Identifier   ctermfg=182 guifg=#dfafdf                           cterm=none
hi Ignore       ctermfg=238 guifg=#444444
hi Number       ctermfg=180 guifg=#dfaf87
hi PreProc      ctermfg=150 guifg=#afdf87
hi Special      ctermfg=174 guifg=#df8787
hi Statement    ctermfg=110 guifg=#87afdf                           cterm=none gui=none
hi Type         ctermfg=146 guifg=#afafdf                           cterm=none gui=none

"" Special {{{2
""" .diff {{{3
hi diffAdded    ctermfg=150 guifg=#afdf87
hi diffRemoved  ctermfg=174 guifg=#df8787
""" vimdiff {{{3
hi diffAdd      ctermfg=bg  guifg=bg      ctermbg=151 guibg=#afdfaf    100	"hi diffDelete   ctermfg=bg  guifg=bg      ctermbg=186 guibg=#dfdf87 cterm=none gui=none    101	hi diffDelete   ctermfg=bg  guifg=bg      ctermbg=246 guibg=#949494 cterm=none gui=none    102	hi diffChange   ctermfg=bg  guifg=bg      ctermbg=181 guibg=#dfafaf    103	hi diffText     ctermfg=bg  guifg=bg      ctermbg=174 guibg=#df8787 cterm=none gui=none    104	""" HTML {{{3    105	" hi htmlTag      ctermfg=146  guifg=#afafdf    106	" hi htmlEndTag   ctermfg=146  guifg=#afafdf    107	hi htmlTag      ctermfg=244    108	hi htmlEndTag   ctermfg=244    109	hi htmlArg	ctermfg=182  guifg=#dfafdf    110	hi htmlValue	ctermfg=187  guifg=#dfdfaf    111	hi htmlTitle	ctermfg=254  ctermbg=95    112	" hi htmlArg	ctermfg=146    113	" hi htmlTagName	ctermfg=146    114	" hi htmlString	ctermfg=187    115	""" XML {{{3    116	hi link xmlTagName	Statement    117	" hi link xmlTag		Comment    118	" hi link xmlEndTag	Statement    119	hi link xmlTag		xmlTagName    120	hi link xmlEndTag	xmlTag    121	hi link xmlAttrib	Identifier    122	""" django {{{3    123	hi djangoVarBlock ctermfg=180  guifg=#dfaf87    124	hi djangoTagBlock ctermfg=150  guifg=#afdf87    125	hi djangoStatement ctermfg=146  guifg=#afafdf    126	hi djangoFilter ctermfg=174  guifg=#df8787    127	""" python {{{3    128	hi pythonExceptions ctermfg=174    129	""" NERDTree {{{3    130	hi Directory      ctermfg=110  guifg=#87afdf    131	hi treeCWD        ctermfg=180  guifg=#dfaf87    132	hi treeClosable   ctermfg=174  guifg=#df8787    133	hi treeOpenable   ctermfg=150  guifg=#afdf87    134	hi treePart       ctermfg=244  guifg=#808080    135	hi treeDirSlash   ctermfg=244  guifg=#808080    136	hi treeLink       ctermfg=182  guifg=#dfafdf    137	""" rst #{{{3    138	hi link rstEmphasis Number    139	    140	""" VimDebug {{{3    141	" FIXME    142	" you may want to set SignColumn highlight in your .vimrc    143	" :help sign    144	" :help SignColumn    145	    146	" hi currentLine term=reverse cterm=reverse gui=reverse    147	" hi breakPoint  term=NONE    cterm=NONE    gui=NONE    148	" hi empty       term=NONE    cterm=NONE    gui=NONE    149	    150	" sign define currentLine linehl=currentLine    151	" sign define breakPoint  linehl=breakPoint  text=>>    152	" sign define both        linehl=currentLine text=>>    153	" sign define empty       linehl=empty    154	""" vimHelp {{{3    155	hi link helpExample Number    156	hi link helpNumber String    157	hi helpURL ctermfg=110 guifg=#87afdf                           cterm=underline gui=underline    158	hi link helpHyperTextEntry helpURL

You can see on lines 42-44 above, I have replaced the hi CursorLineNr command. In Vim script double quotes are comments. So I commented out the old command and copied the same command with a new argument. When you try to go back in time and see what you changed, this might help you. In the CursorLine value I set the foreground color to grey. This shows up nicely against the dark grey of xoria256 without being distracting. To remove the underline from the line number, I need to define a background and foreground color for the CursorLineNr value. I chose a background of magenta with a foreground of white. It looks pretty good.
The important thing is that it doesn’t distract me, either with too much color or too little, and I can find my cusor on a large display without going hunting.

Note that the color choices must be from the normal 8-bit color choices you have available in your terminal.

Conclusion

Find your color theme file from your callout in your .vimrc. Edit that file (the color file) to define your hi CursorLine and hi CursorLineNr values. Define your cterm values for CursorLine and your ctermfg and ctermbg values for your hi CursorLineNr lines. Then close and reopen vim. That should do it! Voila! Annoying underline is now removed and replaced with a nice color combo of your chosing.

Bash Pomodoro

2020-02-24T04:08:53+00:00

Creating a Pomodoro in Bash

I originally wanted to create a pomodoro timer that could run in i3status or i3blocks. I haven’t quite made it that far, but it does run in bash. For those of you who haven’t heard about it, see the Wikipedia article on the Pomodoro Technique.

I chose to create a work period of 25 minutes, with 5 minute short breaks and 20 minute long breaks. I decided to measure the time using a date object measured in seconds. Then I figured I needed three functions: 1) a compute function that takes an argument of seconds to start from and prints the number of seconds elapsed from that date object (measured in seconds), 2) a “convert seconds” function that takes a date object in seconds and returns the formated string of minutes and seconds that can be displayed to the user, and 3) a “seconds remaining” function that takes two arguments: the number of seconds when the period started and the “now” time in seconds, and that returns a printed string of remaining minutes and seconds in the period.

Further, I needed a function to run the breaks between work periods. This function would start a period for a certain length, and that length would be this functions only required parameter.

Once all this was working, I would need some kind of audible bell or something to tell the user when the Pomodoro period or break was done, and when to take a break or return to work. So that was another function.

Because I not only wanted this to run in a shell terminal, I wrote a function that would display the work periods (pomodoros) and break periods in a single line that could be displayed on a status bar like i3status or i3blocks for my favorite tiling window manager, i3wm. This required that I actually have two functions for showing a pomodoro, one that displays inside a terminal and one that runs inside a bar. Well, the “bar” function works such that it displays a line of text that could run inside a status bar, but I haven’t gotten it to actually work inside either i3bar or i3blocks. It displays inside a terminal on a single line just fine. And the other functions works fine inside a terminal window; it takes up about 5 lines of text, separated by line spaces in-between, for a total of ten line spaces.

Here’s the script:

#!/usr/bin/env bash

#################################
#  Easy little pomodoro time mgmt
#  program to help you not waste time.
#  This runs in a terminal, so you can use it 
#  with a tiling window manager if desired.
#  Requires 'play' is installed (from sox)
#################################

##  Initial variables
short_break=5
long_break=20
pomodoro=25
## Forgot that all terminals are not UTF
#goal="⌚⌚⌚⌚⌚⌚⌚⌚"
goal="********"
completed_pomodoros=""
start_sound=./sounds/Ship_Bell-Mike_Koenig-1911209136.wav
end_sound=./sounds/foghorn-daniel_simon.wav
((durationinmins=${pomodoro}*3600/60))

## Run the i3bar version or regular version?
[[ "$@" =~ 'b' ]] && runinbar=true || unset runinbar

##  Set the start time in seconds
start_time=$(date +%s)

## Play start or end sounds for periods
play_sound(){
   ## make sure 'play' is installed...
   command -v play >/dev/null 2>&1 || { echo >&2 "play (from sox app) is required but not installed. aborting..."; exit 1; } 
   file=${1}
   play -q $file 2>/dev/null
}

## Play the start sound
play_sound $start_sound

## Computes seconds elapsed from after 1st argument
## Expects argument in seconds
compute(){
  start_secs=${1}
  now_time=$(date +%s)
  elapsed=$( expr $now_time - $start_secs )
  printf "%d " "${elapsed}"
}

## Converts seconds to hours, minutes, seconds
## Just displays minutes and seconds
convertsecs(){
   ((h=${1}/3600))
   ((m=(${1}%3600)/60))
   ((s=${1}%60))
   #printf "%02d:%02d:%02d" $h $m $s
   printf "%02d:%02d" $m $s
}

## Computes minutes and seconds remaining from start time
## and now time,  Expects now_seconds then start_time
remainingsecs(){
   ((m=${2}-1-${1}%3600/60))
   ((s=(60-${1}%60)))
   [ $m -eq -1 ] && m=0
   printf "%02d:%02d" $m $s
}

## For showing a list of asterisks and completed Pomodoros
completed(){
   echo '⌚' 
}

##  Runs a break, either a short one or long one
##  Expects an argument in minutes
run_break(){
   play_sound $end_sound
   start=$(date +%s)
   now=$(date +%s)
   length=${1}
   elapsed_secs=$(expr $now - $start)
   minutes=$((elapsed_secs/60))



   while [ $minutes -le $length ]; do
       now=$(date +%s)
       pomo=$(compute $start)
       minutes=$((elapsed_secs/60))

       ## exit and start a new Pomodoro when break is spent
       if (($minutes >= $length)) ; then 
           return 0
       fi

       elapsed_secs=$(expr $now - $start)
       remaining=$(remainingsecs $elapsed_secs $length)
       elapsed=$(convertsecs $pomo)
       if [ $length -ne $short_break ]; then
           break_type="Long Break"
       else
           break_type="Short Break"
       fi

       clear
## Hot pink may not work well on light terminal but it does on dark terminal
cat <<EOBreak


       $(echo -e "\033[38;5;205m")
       //////////////////////   BREAK TIME  //////////////////////////////

       Break: ${break_type}    Elapsed: ${elapsed}   Remaining:  ${remaining}
       $(echo -e "\033[m")

EOBreak

       sleep 1
   done
}

show_bar(){
   count=0
   period=${1}
   length=${2}
   start=$(date +%s)
   spinner=('\' '|' '/' '—' '\' '|' '/' '—')


   while true; do
       elapsedsecs=$(compute $start)
       [ ${elapsedsecs} -gt $((length*60)) ] && return
       pomo=$(compute $start)
       elapsed=$(convertsecs $pomo)
       remaining=$(remainingsecs $pomo $length)
       clear
       printf "%s > %s %ss left %d/%d done" ${period} ${spinner[$count]} ${remaining} ${#completed_pomodoros} ${#goal} 
       sleep 1
       if [ $count -lt 7 ]; then
           ((count++))
       else
           count=0
       fi
   done

}

##  This is the main pomodoro screen
show_pom(){
   period=${1}
   length=${2}
   start=$(date +%s)

   while true; do
       pomo=$(compute $start)
       elapsedsecs=$(compute $start)
       [ ${elapsedsecs} -gt $((length*60)) ] && return
       elapsed=$(convertsecs $pomo)
       remaining=$(remainingsecs $pomo $length)

   clear

cat <<EOF


                   Welcome to MyPomodoro!

       ///////////////  Work Time  ///////////////////////

       Short break: ${short_break}     Elapsed: ${elapsed}

       Long break: ${long_break}     Remaining: ${remaining}

       Goal: ${goal}     Completed: ${completed_pomodoros}
EOF
   sleep 1
   done
}
   

##  This is the main loop
while true; do
   [ "$runinbar" ] && show_bar "Work" ${pomodoro} || show_pom "Work" ${pomodoro}

   completed_pomodoros+=$(completed)
   if  [[  $(( ${#completed_pomodoros} % 4 )) == 0 ]] && [[ ${#completed_pomodoros} -ne 0 ]]; then
       [ "$runinbar" ] && play_sound $end_sound && show_bar "Long_Break" ${long_break} || run_break ${long_break}
   else
       [ "$runinbar" ] && play_sound $end_sound && show_bar "Short_Break" ${short_break} || run_break ${short_break}
   fi
   start_time=$(date +%s)
   play_sound $start_sound
done

Thoughts on Script

I’m not sure this script will stand, but it seems to work for now. The idea was to pass a ‘-b’ argument if it should run in the bar, and pass no arguments if it should just run in the terminal. So far, it just runs in the terminal, either in a single line or on multiple lines.

Also, I thought I could paste in little watch glyphs using Unicode characters, but then those didn’t show up on some of my simpler terminal emulators. So I went back to just using asterisks.

There was also the need to check whether the Sox ‘play’ command was installed and issue a warning if it wasn’t. The command -v command was of use here.

The entire script relies on a while loop at the bottom of the script, which calls the other functions as required.

Pause?

Strictly speaking, I don’t think I need a pause command. The whole idea of Pomodoro is to concentrate on what you’re doing and not get side tracked. If the phone rings, let the caller leave a message. If wifey texts, look at it during a break. If your kid is stuck by the side of the road, then handle it during a break. If your boss interrupts you, well, then it’s up to you how you handle it. But it seems to me that adding a pause function is counter-intuitive to what the Pomodoro Technique is all about. What do you think?

Going Forward

I really want to get this running in i3blocks. I’m not sure whether it’s possible to get it running in i3status. So far no luck. I’ll keep at it.

I’ve still got some comments in the code to remind me to think about things, like the ‘goal’ string variable. Is there a way to display the unicode characters for hourglass or watch in Urxvt for example? Not sure yet.

I think Sox is a pretty universal sound player. It rarely comes pre-packaged with Linux distributions anymore, but I think it’s available in almost all repos, and has few if any dependencies.

This little application has already taken a surprising amount of time to complete thus far. And it still doesn’t run in i3blocks! But I’ve been using it for my productivity in using i3wm, and it’s easy to run inside of a narrow terminal.

Hopefully, I’ll get it running inside i3blocks real soon now!

Idea for Teaching Programming

2019-12-23T13:58:52+00:00

Idea for Tutoring in Programming

I I just had this idea in a dream, and it sounds pretty great, so I’m going to flesh it out a bit.

Several Different Aspects To Course

My own journey is broad and varied, and the student can get a job at any point along the way he/she is qualified.

The course will commence on using UNIX, because that is what I can teach. If another person is a Windows programmer, rock on. But I can teach using Linux.

Elements

Learning Linux
Learn HTML, CSS, JS and Front-end
Get set up and productive with an editor and git
Good programmer habits
Intro to Web Devel
Intro to CSS and Styling
Installing a Database
Installing Wordpress
Intro to JS and jQuery
Intro to talking to a Database in JS/jQuery
Writing a Simple Backend in Node
Writing a Simple React Client
Writing a Simple Backend in Ruby Rails
Writing a Simple Backend in Python/Flask and Python/Django

Linux

Installing Linux
Basic Linux Desktop Survival Basic Desktop Customizing
Basic Linux Distribution Survival and Usage
Basic Linux System Admin
Basic Terminal Commands
Basic Vi Editor
Basic BASH programs
More In Depth Linux System Admin
More In Depth BASH Programming
Intermediate Vi Editor
Intro to Python, Ruby, PHP, and Javascript and Node
Installing Git, Getting Comfortable
Intro to Servers and Back End Programming Installing Wordpress, Ruby, Node and Python if necessary
Programming Pathways: Frontend, Backend, Wordpress, DevOps

Intro to Web Development

First concepts and programs
Our Project: A Portfolio
Focusing on Structure and Positioning
The Psychology of Web Dev: Colors and Flow
Finding Teaching Assets on the Web
Find other peoples’ projects and portfolios and ideas
Solve problems for yourself

Intro to CSS

What is Style?
Good style and bad style
Your toolbox
Where to learn about styling
Who are the experts of style and where do you fit?
Responsive Style
Styling Frameworks
SASS
BEM
Solving some CSS Problems

Intro to Javascript and jQuery

Building a front end project from your own idea
Getting Started and Gitting going
Working with an API
…

Building a Wordpress Site

Installing and First Concepts
Styling and Theming Your Site
Writing a Wordpress Theme
Writing a Wordpress Plugin
Being a Wordpress Developer

Writing a Backend in Node

Postman and similar clients
Understanding HTTP, REST and Databases
This is the world of UNIX and Linux
More classes on this subject
Ideas for Projects
Walking Through Using a Simple MongoDB Instance
Walking Through Using a Simple Postgres Instance

Writing a Simple React Client

Writing a Backend in Ruby/Rails

Writing a Backend in Python/Flask

Writing a Backend in Python/Django

Diving Deeper into React on the Front End

The Computer Museum of America

2019-11-21T16:32:36+00:00

Computer Museum in Georgia

I was truly surprised at how awesome this museum is. It’s a fairly new museum, less than three years old, and yet it has a massive collection of not just personal machines, but also super computers and mainframe computers. There’s also a collection of early DEC computers, such as the PDP-8 and others, that really bring back memories. Those were computers that I had always heard about but had never seen up close. Those were computers the elite hackers used and worked on. In fact, I believe it was a PDP-7 that Dennis Ritchie and Ken Thompson wrote UNIX on mostly. Then later they ported it to the new PDP-11 at Bell labs. Later they rewrote Unix in the new C language.

You can read about it here.

The website for the Computer Museum of America is here.

As you can see from the website, there’s a lot to see. There are a lot of old personal computers and calculators that might be of interest (they were to me), but what you cannot tell from the website is the depth of the individual exhibits. It’s one thing to see a picture of a Cray XMP, like we all did in the computer magazines, but to stand right next to one and look inside a glass side panel is a completely different experience. There are a number of Cray’s there in the super computer exhibit. There are probably 20 or 30 machines from the late 80s up til the early 2000s, and a few more recent machines.

I thought I was computer literate, but looking at those larger machines, especially the super computers, I felt like I knew nothing. My experience has been with microcomputers. And after looking at those super computers, I feel like my experience has been very micro.

For example, I was looking at a heat exchanger from the early 90s, and now I see that liquid cooling is nothing new on computers. This heat exchanger was the size of a small trailer, and it would have been part of the cooling solution. But it had see-through panels to you could at least get an idea of what was involved. I thought I was cool because I used a liquid AIO to cool my overclocked i7-6700K. Well, never mind. I’ll close my mouth and sit quietly after seeing some of these machines.

My One Criticism

I could spend days just with the super computers, or perhaps one or two. It would be nice to have a kiosk handy with some YouTubes about some of these machines. The proprietors are not able to leave the panels off of the machines. And there cannot always be someone around to answer questions. But there are so many questions about these machines, it would be nice to have a FAQ about them.

Or perhaps there are some more placards on the wall I just haven’t seen that explain many of these questions.

One of the questions I have is How do you interface with these machines? I’m guessing you log in through a terminal and basically get a terminal prompt, or perhaps something like an ncurses-type interface. Occasionally I have logged into a mainframe, and that’s how it has been with them. BTW, that’s how it was back in the 80s too I think. I didn’t get a lot of mainframe time then, but I got some, and my exposure was through a TTY emulator. Perhaps that has not changed?

Hands On?

My guess is that it takes a lot to get one of these things up and going. And to keep it going. It would probably be easier to get a terminal to a vm running on a connected Linux box that looks like a supercomputer. Most people would not imagine that there’s a difference.

But some of us are programmers of one kind or another. I would get a kick out of writing a fizzbuzz program and running it on a supercomputer. I imagine others would too?

In any event, these supercomputers are like alien technology. It would be nice to have a little more background on how they actually work. Maybe fire one up for a few minutes? Solve a complicated math problem on it and compare it to a PCs number crunching ability? There’d have to be a really impossible problem to do on a PC that I could understand and appreciate how much power it would take to solve it on the supercomputer. (Understanding it is the key ingredient there.)

I’ll Be Back

If a measure of a museum’s greatness is your intention to go back, then give CMOA a bunch of stars! I could spend days in any of the sections of the museum. One of the fascinating factums is that what you see in the museum (hundreds of machines) only represents a small fraction of the owned exhibits. There are far more computers that are unseen than there are seen in this museum. I’m not sure where they are all stored. My guess is that the owners are very financially well off, and they can afford to not only buy pet supercomputers and mainframes, but also they can afford to store them. I’m guessing that would be akin to collecting vintage race cars or airplanes or large military vehicles. It’s a rich man’s hobby.

So, not only will I keep going back to see new things. I will also go back to keep trying to plough deeper into the simplest of exhibits there. I didn’t pay close enough attention to the devices when I first saw them. I didn’t realize how cool they were when they were new. Now that they old, I’m fascinated by them!

Debian Testing

2019-11-20T21:05:08+00:00

Running Debian Testing as a Daily Driver

I am told this is not a great idea. However, I’m already running several rolling release distros, such as Tumbleweed, Manjaro, and Arch. My thinking is that Debian Testing branch is no more unstable than these distros. Besides, I’ve been at this a while. I’ve seen a lot of update-induced problems and have fixed most of them. Almost all of them in fact. So since Testing is not even as full of land mines as Debian Unstable, I thought I would try it out, and it’s been maybe three months? So far so good.

What About Stable Releases?

Not a problem. When an Testing release gets moved into the Stable branch, Unstable gets moved into Testing. You can read about how the release system works in the Debian Faq.

Basically, you’ve got the Stable branch. It is absolutely stable, but if you’re a desktop user who wants to use very recent software, this branch will leave you feeling left behind when it comes to the latest software release versions. For this reason, some people follow the Testing branch, which provides more recent versions. However, prior to a new Debian release, the Testing branch is “frozen”, meaning that little to no changes are admitted into the Testing branch of Debian. During this period, developers are pounding the snot out of the Testing branch to ensure that it is as absolutely stable as it can possibly be. Only after this phase does the Testing branch get moved into Stable in preparation for a new release.

Why The Risk?

It’s purely about running more recent software. There are many people who do not need the latest software, and these people often choose to run the Stable branch. They couldn’t care less about running the latest software, and frankly, they couldn’t tell the difference if you showed them, usually. That’s fine. That’s just what you want for running a rack full of servers. Those boxes get security updates and that’s about it. Sometimes they will have uptimes measured in years. Or at least they used to before Virtual Machines and containers. But if you’re running Linux on a home server and you just want it to do its job for years, then you probably are running the Stable branch.

However, I want the latest LibreOffice, the latest Nodejs and Python, the latest Cinnamon desktop, and so on. I want the latest Nvidia drivers, the latest browsers, and the latest whizbang goodies. That’s just me.

Why Not Run Another Distro Instead?

I do run other distros. But Debian is one of my old favorites. I feel I need to stay on top of that one. Debian is the launchpad for all of the Ubuntu distros. And lots of other Linux distros too. Staying on top of Debian is essential for keeping your finger on the upstream pulse of all of those related distros for which Debian is the ancestor.

The main objection folks have against Debian is that it tends to lag behind on software versions. But that’s if you’re running on the Stable branch. Not so much if you’re running on Testing.

Another objection can be related to non-free software. If you want to run non-free software, you simply enable non-free repos in your /etc/apt/sources.list file. If you want to run software like Google Chrome and VLC, you can do this.

Another option to the non-free problem is to use Snap packages. I prefer snap packages when running a Debian/Ubuntu distro and Flatpaks when using an RPM distro. If I’m running an Arch distro, I normally can find everything I need in the AUR.

Spotify

This is one of those packages that can be kind of confusing, especially if you play files locally a lot. Basically, you need legacy versions of ffmpeg decoding software to translate locally saved files into a version compatible with the latest Spotify version. Since I play a lot of local files from my Music library, I encounter the can't play local files bug a lot. It hasn’t been a problem with a Snap package on Debian so far. The standard Debian package did not include the proprietary codecs I needed to play all my files. But the Snap did, right off the bat.

Normally, on other distros, it takes some tinkering to play local files, because each distro will probably handle these proprietary codecs differently. But Debian, being one of the most conservative distros when it comes to non-free software, was a perfect candidate for the Spotify Snap package.

Problems

There have been a few problems. One fairly large problem was a result of damaging a file system from repeated overclocking attempts that left the filesystem with lots of errors. I cannot blame that on Debian or even Linux. I basically didn’t take enough precautions with my overclocking (such as maintaining a separate rescue partition).

Another problem happened when I was upgrading a module from the Nvidia driver. It turned out that one of the kernel modules wouldn’t build, and it quietly bombed out and resulted in a driver module that couldn’t be inserted into the current kernel.

This was an easy fix, once I read the errors and googled them. Fortunately, this time, that particular error was the result of a change in how the linux kernel manages video drivers. After a certain kernel version, a version check must be made to see whether it is appropriate to build a module dependency or not. Prior to that kernel version, it was the default behavior to build that module. After that version, you would not build the module with the same dependencies. This change resulted in placing an if/then statement around the block of code where this module dependency was built before. I think it was four lines of code that needed to be added. The github issues page I was reading contained the four lines of code, so I added them to my nvidia module’s source tree. Poof! The modules built and installed flawlessly after that. No problems.

A week or two later, this fix was included in a formal update for the Testing branch. I would have been good to go in about a week and a half anyway. Fortunately, I didn’t have to wait that long.

Who Should Run Testing?

I would say run testing if you need the latest software versions. But be advised that you’ll get problems. If you’re okay with working through problems, then fine.

Be sure to keep a backup of your system!!!

That said, I have really just had these two little problems. Well, the first one resulted in a reinstall, because I didn’t keep backups on this machine! (It’s a machine that I overclock on a fair amount!)

The nvidia problem was a typical problem that you might get on the Testing branch. There could be other ones, and those problems might not get resolved with 10 minutes of Googling. I can take longer. I’ve been fortunate. But this is how I treat any rolling release distribution. I normally don’t get many problems from them, but when they happen, they are not fun. If you do not have backups or have alternate machines you can use to get work done, it ain’t fun. I have lots of alternative machines I can use. If one breaks, it’s not a problem for me to use another until I get the broken machine fixed. In fact, I normally have at least one machine that is broken.

So, the Testing branch is not a problem for me. I still can curse when it breaks, but I realize I have said yes to these problems when I run a bleeding edge release.

For absolute stability, I recommend an LTS release of Ubuntu which are based on Debian Stable. Also, Linux Mint tends to be based on an LTS release and inherits great stability from Debian. And then running Debian Stable itself is about as stable as you can get.

If you really don’t mind running software versions that are not the lastest and greatest, you should try out Debian Stable. If it turns out it’s not recent enough, try running Debian Testing and see what happens! All the best in your journey!

Overclocking On Linux

2019-11-15T14:25:10+00:00

Linux as an Overclocking Platform

I’ve recently gotten into overclocking some of my computers. I’ve always been weary to do this, because it seemed like a great way to ruin perfectly good hardware and throw money down the drain. So, no overclocking for me. Until now.

I looked around and saw that I simply had too much hardware. And it has finally begun to dawn on me that there is quite a bit of headroom built into CPUs these days. No one can predict the performance potential of a randomly binned CPU. So there is a sort of CPU lottery. Maybe you win, or maybe you don’t. But there is going to be at least some potential performance your CPU in your computer will have beyond what it’s currently doing if you are willing to push it a little bit. And, frankly, I got curious. That’s how these things always start for me. I get curious.

There is one more aspect to all this. I made a promise to myself to not buy anymore computers. I probably have about 20 or more towers at my house, mostly all running linux. I figured if I’m not buying any more new CPUs, I’ve got to stay busy with what I’ve got, and my attention turned to performance. If I can get some of these boards and CPUs to wear out or something, then I might buy new stuff???

What is Overclocking Anyway?

There’s really nothing special about overclocking. It’s simply pushing your CPU a little harder for greater speed, and that usually requires more voltage. Since CPUs and the chips on your motherboard are so small, they are vulnerable to heat, so small increases in voltage can place your motherboard and CPU at some risk if you don’t think about what you’re doing. After all, you could have paid many hundreds of dollars for those components, so why would you want to cook them inside of your tower case?

Well depending on your specific CPU and motherboard, you could get a 10-20% performance bump in your system. For free, basically. I’ve finally had some systems that are 15 plus years old die on me. Either a drive controller or mobo component dies or something, but this has been long after that system has been a highly useful system for me. My point is, why not get more use out of that hardware since you’re going to replace it long before it physically dies anyway? Overclocking is one way to do that.

Specifically, to overclock, you must have an unlocked CPU (one that allows you to change speeds and voltages) and a motherboard that supports overclocking. Likewise, you’re going to need a good power supply with at least a Bronze level rating, and also a good cooling solution, probably an after market cooler, to dissipate the increased heat.

An example of an overclock is taking an i5-4790K cpu on an ASRock Z97 Extreme6 board (a little more expensive motherboard that is good at overclocking) and changing the default multiplier in the BIOS so that the CPU speed goes from 3.5GHz to 4.5GHz. Core voltage could need to go up an extra 0.125 volts or so to keep this clock stable. So instead of an idle temperature of, say, 30 degrees Centigrade, perhaps you’re now running closer to 40 degrees at idle. You haven’t spent a dime, unless you’ve upgraded some of your components.

This boost from 3.5GHz to 4.5GHz is almost a 30% increase in CPU speed. And it’s costing you perhaps an extra few pennies in your monthly energy bill. Can you begin to see why the idea might be attractive to people?

But is it Practical?

That depends. If all you do is surf the web and read email, then probably not. If you just want to watch YouTube videos on your machine, then overclocking probably won’t be of interest to you. But, if you use your CPU cycles on things, such as playing intensive computer games or rendering videos or compiling source trees or something, then overclocking your CPU might be something of interest to you.

For me, sometimes I write computer programs that require compute power. For example, if I’m searching for prime numbers, I can easily bring a system to its knees. These are just intensive calculations, and some operations can take a very long time. In fact, there’s a “Great Search for Mersenne Primes” on the internet, and it requires so much compute power that there are thousands (or more?) volunteers sharing their computers’ extra CPU cycles in the search. The software you download is also great for stress testing your system for stability. If you can run the stress test and keep your system stable at higher clocks and voltages, searching for prime numbers is a great way to test your experimental settings. This software is called Prime95.

First and foremost, it’s fun. Overclocking is like hot rodding your computer. You may or may not notice the performance gains you’ll realize.

Is It Safe?

I believe that depends on you. Are you safe? Can you experiment safely? If you can trust yourself to be careful and pay attention and be patient, then yes, overclocking is safe. If you tend to do things like run around with sharp knives or drive too fast or hit live ammunition with hammers just for fun, then overclocking your computer may not be for you. However, if you can do research and learn the meanings of BIOS/UEFI settings for your motherboard, then you can safely realize these performance gains from overclocking.

Basically, you’ll need the things I mentioned earlier:

An unlocked CPU
A motherboard that supports overclocking
A good power supply (80 plus Bronze or better)
A good cooling solution

Is Water Cooling Necessary?

Water cooling is not necessary but it can be more efficient than air cooling. Water can dissipate thermal energy more efficiently than air, or so I hear. But if you’re comparing a small 120mm water cooler to a huge heat sink with multiple fans moving air through the heat sink, then the air cooler will be better.

TDP or “thermal design profile” is a measurement of the maximum heat a CPU or a Cooler can handle. It can be a basic indicator for how much cooling capability a cooler has. If your air cooler can handle 150W TDP and your 120mm AIO can handle 130W TDP, then your air cooler is a better solution. But is it so big that it doesn’t fit into your box? Maybe the biggest heat sink that fits into your box has a TDP of 120W.

This brings up an interesting point: there are a LOT of great cooling solutions these days in either water or liquid coolers. In general, air coolers must sit on top of the CPU, which limits the size of the heat sink that can be used. Water coolers can use tubes to move the hot water through radiators that can be placed anywhere in the computer. You can even use multiple radiators if you so desire. So liquid coolers have more potential in terms of how much heat they can dissipate in your system. But all this depends on your use case. Do you just want to run a massive overclock and build a custom loop cooling solution just so you can show off your high clocks to your friends? You’ll probably want to build a custom loop with multiple radiators, and perhaps you can even overclock your GPU (graphics card) also and include that in your custom loop?

But if all your after is a 10-15% increase in clock speed, there are lots of air coolers that will keep your system simple and do the cooling job reliably. There are also exceptional air coolers with smaller form factors. And there are AIOs for nearly every case. But, size will be a factor in your solution.

Why Linux?

Well for me, I use Linux. Linux is as familiar to me as Mac and Windows are to many people. But most people who post on overclocking forums seem to be using Windows. I’m not aware of anyone who overclocks on Mac, unless we’re talking about Hackintoshes. And I’m not sure Apple hardware even has a BIOS that you can change settings on. So that would prevent overclocking on Apple hardware I think. But since most people who overclock do it on Windows, you just need the Linux equivalents of those software choices for your Linux machine:

Prime95
CPU-Z for linux
HWMonitor for Linux

Slight problem. CPU-Z and HWMonitor (available for Windows at cpuid.com) are not available for Linux. However, there are alternatives for these options on Linux. It’s not worth it for me to run Windows just so I can run these pieces of software.

Prime95 is available as a CLI program on Linux. I can monitor fan speeds, temperatures, clock cycles and load percentages also with CLI programs or scripts. If you’re willing to do that, then you can get about the same information on Linux as you can on Windows. You can google “CPU-Z for Linux” or “HWMonitor for Linux” and find lots of alternatives.

If you’re running Ubuntu you may have a few graphical options available that haven’t been packaged to other distros.

I frequently just use GKrellM or Conky.

First Steps

It’s going to start with your particular hardware. Are you using hardware that is capable of overclocking? If not, you’re done.

Is the payoff worth the risk in your case? If you’re already a tinkerer and have hardware you can afford to experiment with, then great. If you don’t want to risk your hardware, then stop now.

If you’re good to go, start with Google. I googled “Overclocking on Linux” and came up with LOTS of very interesting links worth reading. Lots of opinions out there.

Watch YouTubes about overclocking on Windows too. Overclocking happens really more at the BIOS level, so the OS is really more of a window to whatever is happening at that lower level of the machine. You can learn a lot from Windows TechTubers, such as JayzTwoCents, Pauls Hardware, Bitwit, LinusTechTips, and many, many others. There are tons of people out there doing this, so you are joining a huge group of people who have been overclocking for a long time.

Conclusion

If you’re the type of person who likes building computers or building things with software, you might also be the type of person who enjoys hot rodding their computer.

There’s some risk. But you could also drop a $500 CPU on the hardwood floor while you’re building a computer. Or you could accidentally spill your milkshake on your motherboard. If you have been willing to handle those risks, then overclocking might not be much more of a risk for you.

If you’ve ever run benchmarks on your system to see “how did I do?” then you might be a candidate for overclocking. Overclocking your computer is a great way to get higher scores on various benchmarks. It’s also a great way to measure your performance gains from overclocking. There are lots of benchmarks out there, and Google is your friend here.

Linux has tended to be a production type of platform, especially for servers and getting work done in the background. But if you’ve been using Linux as a desktop solution instead of Windows or Mac, then why change to Windows just for overclocking?

Finally, I have not mentioned specific software solutions for CPU-Z and HWMontor. That’s because your choices would probably not be my choices. Desktop Linux is full of different environments and window managers and toolkits and technology stacks. There isn’t one solution that fits everybody. If you’re already using Linux then you probably already know this. CPU-X may or may not work as a substitute for CPU-Z on Windows. You may well find that writing your own script that monitors /proc/cpuinfo works better than anything else. Sometimes Conky or GKrellM is enough for me to watch while I overclock. Your mileage may vary!

One thing I can tell you, I have been having lots of fun overclocking on Linux.

Migrating Antergos to Arch Linux

2019-11-01T15:23:07+00:00

Migrating Antergos Linux to Arch Linux

Since the Antergos project eneded, those of us who have loved Antergos have been sad.

And there was an Antergos forum post about future updates magically updating or migrating Antergos installations to Arch Linux installations. Well, that may not have happened so magically, and I guess no one ever really promised us Linux magic, especially with an Arch related distro, so we have to spin up our own magic I guess. Hopefully, this article can provide a bit of that.

I’m writing this on the first day of November, 2019, and the project ended in late May. So there have been updates that have helped to change existing Antergos installations into Almost Arch installations. But there are a few things that still need to happen. Here’s a recipe for converting your Antergos installation into a working Arch installation:

Logout of you desktop so you see your X display manager login. Then hit ALT-CTL-F2 and log into your account and become root:
```
su -
```
Change to multi-user (non-graphical) run level.
```
# systemctl start multi-user.target
```

Remove the pamac graphical package manager.

# kill -s SIGKILL $(pgrep pamac) && pacman -R pamac

Remove all antergos packages (if you want a pure Arch installation).
```
# pacman -Rddnus $(pacman -Qq | grep antergos)
```
Modify /etc/pacman.conf to remove references to Antergos repos. Just comment out all sections with [antergos] or [antergos-staging]

Change entries in /etc/os-release (which is a symlink /usr/lib/os-release) over to Arch values

 NAME="Arch Linux"
 PRETTY_NAME="Arch Linux"
 ID=archlinux
 ID_LIKE=archlinux
 ANSI_COLOR="0;36"
 HOME_URL="https://archlinux.org/"
 SUPPORT_URL="https://bbs.archlinux.org/"
 BUG_REPORT_URL="https://bugs.archlinux.org"

Update your system:
```
# pacman -Syyu
```
If when you run neofetch or screenfetch you still see Antergos, then you should re-install the lsb-release package
```
# pacman -S lsb-release
```
Make sure you have all required files for your desktop environment, including all Xorg and related files, video drivers, mesa, etc etc, which are now coming from Arch repos. May need to re-install from new repos. One of my machines had Cinnamon (desktop) antergos repos, so I had to remove all of those and get regular arch packages. Just to be safe, I re-installed all my xorg packages along with lightdm packages and my video drivers. This time I knew they all came from arch repos.
This article explains the process for Cinnamon. This particular article is good, but it is old (2014). Packages names can change a lot!
Change grub theme to an arch linux theme, if that’s what you want. Remove Antergos grub theme. (Should already be gone.) When you install grub from arch repos, it may not overwrite your existing antergos /etc/default/grub.cfg. If not, there should be another grub.cfg in /etc/default that has a different name, like grub.cfg.pacman or something. Look at them both and make sure the correct one is in /etc/default/grub.cfg. I would also install a grub theme, such as grub2-theme-archlinux.
```
# yay -S grub2-theme-archlinux
```
Then, run
```
# grub-mkconfig -o /boot/grub/grub.cfg
```
If there are no errors, you should be good for a reboot, almost.
Restart your graphical target run level.
```
# systemctl start graphical.target
```
If there are any missing X related packages, you can troubleshoot your errors and straighten them out.
When youre ready, reboot your computer, if you dare!

Leftovers

If you installed Antergos using LVM, you’ll still see Antergos in your LVM names. There are still some packages on your system, such as webkit-theme-antergos, and you can see some more using updatedb && locate antergos. But for all intents and purposes, you now should have an arch linux installation. Those remaining files weren’t removed when antergos packages were removed, but they can be removed manually I think without affecting the rest of the system. But I’m not sure. For now, they’re not bothering me. Maybe I’ll remove them later.

I’ve changed my icons around to reflect the change, in my start menu for example. Neofetch and Screenfetch now report an Arch installation. And really, there’s just very little left of Antergos in the installation. Linux distros are simply a collection of choices, and Antergos was mainly a collection of very nice styling decisions and special packages, and those packages are now replaced by Arch packages. And you’re free to keep the best choices for your indidualized Arch installation!