Stefan's Software Guides

Explanations, help, and useful links to software

(with also a little bit of a special reference to the Physics Department of the University of Innsbruck)

These pages give some documents and help with software and the use of the university computer infrastructure. The documents here are created to the best of my knowledge and tested on my available systems. They are also given as "as is". There is neither a claim for completeness nor a guarantee that it will work with every system. But the content surely gives the right "keywords" for another web-based search, in case it does not.

These documents help to shorten your own searches. But there is no information included, which could be found in different places anyway. So it is only a help to save time. But I do NOT see myself as an instructor in these things.
Therefore they are not "supported" - i.e. I cannot hold tutorials going beyond that. Requests for assistance beyond that will not be answered.

The Linux Command Line is probably the most powerful tool you should know. Many parts shown here are also implemented directly under MacOS, but even de PowerShell of Windows takes over more and more of it. The current versions are public and always available at https://linuxcommand.sourceforge.io/tlcl.php. Local copies are here. Especially in the new book (this is in addition to the first book - not meant as a new edition/replacement) chapters like the powerful awk command, which can take a lot of work out of rearranging data, is well elaborated.

The Linux Command Line (Version 29. Jan 2019)
Adventures in the Linux Command Line (Version 1. October 2021)

External access to Unix/LiNUX systems at the University of Innsbruck

Students of courses in Astro- and Particle Physics have further possibilities with their accounts at the computers in the user room 8/35 of the institute. The current explanations can be found at https://www.uibk.ac.at/astro/intranet/rechnerraum/ (account as member or student of LFUI required).
NOTE (Addendum 3/31/2022): At the moment the ZID allows again the VPN free access to zid-gpl.uibk.ac.at under each UNI account which one has activated for LiNUX (activation on https://orawww.uibk.ac.at/public_prod/owa/uvw$web$10.p001 - is only active the next day). So the jump goes via ssh -XC cNNNNN@zid-gpl.uibk.ac.at and from there ssh -XC myaccount@my_other_machine again. This is faster and more stable than using the VPN - but it is not clear if it will stay like this or if it will be switched off (again) someday.

WSL2 for Unix/Linux Programs below Windows 10 (> 20H1 after update April 2020) and Windows 11:
This allows the installation of LiNUX/Unix programs on your own Windows machine. But it also makes access to other systems under Unix/LiNUX all over the world much more "stress-free".

WSL2 replaces the need for nearly all student applications required under LiNUX for student classes and avoids complicated double boot solutions when using e.g. your Laptop temporarily with LiNUX only.
Details at the official Microsoft page https://docs.microsoft.com/en-us/windows/wsl/tutorials/gui-apps .
CAUTION (Addendum 30.3.2022): Windows 11 new installations will install a completely "naked" system. This does not even allow all apt install packages to get a stronger system. Therefore necessarily some steps after the installation:

Installing into your windows with the command wsl --install -d Ubuntu in the PowerShell (you need to reboot Windows after that step)
Update to a recent more complete version by wsl --update in the PowerShell
reboot your Linux (not the whole PC/Laptop) by wsl --shutdown
You can always do this last step after other installations, changes or problems within LiNUX. It reboots ONLY the LiNUX. Your Windows machine is not affected.
installing any software you need under your Ubuntu LiNUX (WSL2 is based on 20.04 LTS or 22.04 LTS = Long Time Support) using sudo apt install my_package
Works fine for all needs - e.g. gcc, g++, ddd, gdb, cfitsio but also eso-midas, grace, sextractor ...
With commands like scp and especially rsync you can build super backup programs/scripts/batches for your Windows part of the laptop, which copy only the changed areas and subparts of files (look for the UNIX command rsync on the net - you will never use anything else for backup afterward - especially nothing you have to pay for like Ashampoo, Synology C2, Aomei, ... but also top free tools like Duplicati no matter if you copy to a network disk/cloud -- as long as it knows ssh or sftp -- or to a local e.g. USB disk).

Software maintenance:

Since our university network is very open, not having updates means committing suicide, so to speak. But since a broken machine within the institute is often an easy stepping stone to all others, this is so to speak "extended suicide" since it drags all others with it. The colleagues concerned will not be enthusiastic.

Regular UPDATES:
(especially important for the browsers - but also kernel vulnerabilities are discovered from time to time)

First, do your (e.g. weekly) update within a version - e.g. in all RPM-based systems

dnf update

yum update

DEB-package based systems have the same with

sudo apt-get update

The automated system updates in nearly all package managers only do those defined as (very) critical. Thus you should schedule your update regularly (about weekly - e.g. together with your major backup).

Many of us do have NVIDIA kernel packages. The versions of the drivers offered at Rawhide or similar servers are not the best ones. Moreover, you have to remove the free version (e.g. nouveau) once after installing the system (updates and upgrades remember this decision):
https://docs.nvidia.com/ai-enterprise/deployment-guide-vmware/0.1.0/nouveau.html
https://linuxconfig.org/how-to-disable-blacklist-nouveau-nvidia-driver-on-ubuntu-20-04-focal-fossa-linux
https://discussion.fedoraproject.org/t/how-to-delete-nouveau-so-i-can-install-nvidia-driver/72774

I always NVIDIA use the Kernel drivers from the homepage https://www.nvidia.de/Download/index.aspx?lang=C
I download a package and keep it - kernel updates will require it again.
I use at the moment https://us.download.nvidia.com/XFree86/Linux-x86_64/530.41.03/NVIDIA-Linux-x86_64-530.41.03.run

Make this package executable
chmod a+x NVIDIA-Linux-x86_64-530.41.03.run

Then install the kernel header/devel packages of your distro.
yum -y install kernel-devel kernel-headers
or
sudo apt install linux-headers-$(uname -r)
Again this is required only once. Updates and upgrades will remember this.

move your system to text mode
sudo init 3
run the package sudo ./NVIDIA-Linux-x86_64-530.41.03.run and answer most questions with YES. Only the 32bit compatibility will fail for most of our modern PCs. The last question about an update of the X11 configuration can be answered with NO if you had a running version before. Do this step even if the first question says - you have already this version installed - if you have a new kernel.
test the driver by using in single user text mode
startx
If the last step was fine (even havng some trange desktop settings) start to graphical multi user mode again by
init 5
or reboot the computer

These last few steps (thus keep the package) have to be repeated every time your update or upgrade process loaded a new kernel version. As a new kernel version takes effect only after a reboot, you should do that NVIDIA package update only after a reboot - it will then most likely end up in a very low-resolution standard VGA mode after changing the kernel (before updating the NVIDIA kernel module).

Regular UPGRADES:

One should be aware, that many distros stop to support security updates once new releases are offered. Moreover, no application software updates come (e.g. browsers) at this channel now. This applies despite the UPDATE commands mentioned before.
The policy varies strongly from distro to distro.

Thus, to protect you and the others here, upgrade your LiNUX distro timely!

e.g. currently when writing these lines Fedora 36 is out of update already as Fedora 38 is out.
See: https://en.wikipedia.org/wiki/Fedora_Linux_release_history

CentOS had a long-term policy as a free spinoff of the stable versions of RedHat enterprise solutions. Thus CentOS 7 (released July 2014) is maintained until June 30, 2024. But then the policy changed to CentOS stream (floating upgrades even if not yet identified as stable packages in RedHat Enterprise). Thus CentOS 8 is already End of Life on December 31st, 2021 before CentOS 7 stopped.

Ubuntu is a bit better - but only with the LTS versions (normally the even year number with 04 and the LTS behind it like 18.04 LTS, 20.04 LTS, 22.04 LTS). The versions with odd year numbers (e.g. 23.04) but also all versions ending in *.10 are supported only for about 9 months.
But also with the LTS version keep in mind that the Extended Security Maintenance (ESM) period only covers very basic OS security updates of high-risk vulnerabilities and exploits - e.g. non referring to outdated browsers or user applications.
However, browsing the web is the largest risk factor for us as scientists on our office PCs!
Thus as of today - do not use 18.04 LTS anymore and start to move away from 20.04 LTS before the end of the year 2023 as it supports only the critical patches of the kernel then, but no application security.

However, it is often tricky to keep some not-so-smoothly (not to say badly) maintained science software then still running.
The hacks here are related to ESO software and the thus running Fedora/Redhat line (as ESO does not support officially the Debian type lines).

However, it is an instruction/example that will very often work out for various of those problems with other software as well. Thus take it as a hint how one makes such mixed version upgrades.

This is not a compendium claiming completeness. It is just a list of hacks that worked out for me.
I upgraded from Fedora 37 to 38 on June, 1st. However, the ESO support packages there are incoming often irregularly.

The update process of Fedora revisions:

Check if the revision you intend as a destination is supported at ESO by looking to
https://ftp.eso.org/pub/dfs/pipelines/repositories/stable/fedora/
if NOT - try to patch the file /etc/yum.repos.d/esorepo.repo at the line
baseurl=ftp://ftp.eso.org/pub/dfs/pipelines/repositories/stable/fedora/$releasever/$basearch
by replacing $releasever it with the number of the latest you find on the ESO FTP server.
run the upgrade process as described on the official Fedora pages.
https://docs.fedoraproject.org/en-US/quick-docs/dnf-system-upgrade/
Here never do more than 2 versions at once - never skip more than one version in between. This warning, however, normally never should apply ;-)
To force for the latter change in step 3 --releasever=38 to a number n+2 if you have a very outdated old version (before 36). In this case, you then have to undergo the process of this step 2 after the complete system upgrade again, until reaching the current version (By the way - Ubuntu allows upgrade only from one LTS to just the next LTS version !)
If you now encounter massive transaction test error messages with the ESO software (as I had when going for Fedora 38) you have to make additional steps:
remove ESO software if you use the ESO repro. An upgrade will not work including them directly
sudo dnf erase eso* --noautoremovewill do it perfectly for you. ESO (as of today, June 12, 2023) supports Fedora 37.
I received further incompatibility messages. That may use, depending on your previous configurations) manual interactions like:

Error: Transaction test error:
file /usr/share/texlive/doc.html from install of texlive-base-10:20220321-72.fc38.x86_64 conflicts with file from package texlive-texlive-docindex-doc-9:svn54903-39.fc34.noarch

dnf erase "texlive-texlive-docindex-doc*"
dnf erase cext
Then do the upgrade of the OS itself as described in section 3.
Now, try to install the ESO packages - you will encounter transaction errors. Note those packages and manually download a slightly older Version from the web (from https://rpmfind.net/linux/rpm2html/search.php ). In my case, I used Fedora 37 packages as the Fedora 38 packages were too new - it affected the GNU Scientific Library (2.6-7 instead of 2.7.1-4), python3-astropy (5.2-3 instead of 5.2.2-1), and the NASA CFITSIO (4.0.0-4 instead of 4.2.0-3). These marginally older versions do not harm functionalities for e.g. astro-python applications.

dnf install ~stefan/Downloads/cfitsio-4.0.0-4.fc37.x86_64.rpm --allowerasing --noautoremove
dnf install ~stefan/Downloads/gsl-2.6-7.fc37.x86_64.rpm --allowerasing --noautoremove dnf install ~stefan/Downloads/python3-astropy-5.2-3.fc38.x86_64.rpm --allowerasing -x cfitsio -x cpl
dnf install esopipe-fors* --allowerasing --noautoremove --releasever 37
dnf install esopipe-xshoo* --allowerasing --noautoremove --releasever 37
dnf install esopipe-visir* --allowerasing --noautoremove --releasever 37
Finally from now on take care not to overwrite those downgraded packages by the newest versions e.g. during the weekly updates. It will kill your hack. I need e.g. for my regular (weekly) update within the distro

dnf update -x gsl -x cfitsio -x python3-astropy

Debian-based package systems have the same upgrade processes and thus similar steps if you have to run older software.
e.g. upgrade via the same steps https://ubuntu.com/server/docs/upgrade-introduction - be aware - the 5 years mentioned there do only apply to the core OS - not to all the user application packages like browsers and LibreOffice. Even kernel updates are stopped earlier. e.g. the latest subversion (5) of 18.04.5 LTS stopped already in May 2023 - the extended support is also not free of charge; Ubuntu 22.10 (Kinetik Kudu) stopped end of July 2023.

Older, unsupported software is thus NO argument for keeping old OS versions!
Laziness endangers not only your work but also that of the whole institute!

My links to useful software especially for the field of physics and astrophysics:

Graphical elaborations (line graphs only) with the extension of a simple spreadsheet for manipulation of data and the possibility of a simple fit (unweighted minimization of distance squares) of arbitrary functions. The created files (especially those in EPS for LaTeX integration) are extremely compact compared to those of other tools. Also, the labels allow an almost LaTeX-like description for special characters, superscripts, subscripts, etc.

Grace Project: https://plasma-gate.weizmann.ac.il/Grace/
In most LiNUX Distributios it can be installed naturally with sudo apt install grace, yum install grace or dnf install grace .

Fitting data with MPFIT:

Originally from IDL: https://idlastro.gsfc.nasa.gov/ftp/pro/markwardt/mpfit.pro and documentation in https://www.l3harrisgeospatial.com/docs/mpfit.html. This is from the original FORTRAN versions of MINPAK.

Recommended version (still converted and maintained by the original author) in C: https://pages.physics.wisc.edu/~craigm/idl/cmpfit.html with many examples.

Several conversions (not checked/tested and verified by the original author) to Python: https://cars9.uchicago.edu/software/python/mpfit.html & https://github.com/segasai/astrolibpy/blob/master/mpfit/mpfit.py

MINPACK - a collection of many useful and very well-tested routines (in FORTRAN). The documentation is linked to
https://en.wikipedia.org/wiki/MINPACK . The download is best obtained from (https://www.netlib.org/minpack/ ).
ESO MIDAS - astronomical image processing (https://www.eso.org/sci/software/esomidas/ ). download the latest versions from https://ftp.eso.org/pub/midaspub/ . Current version is 22FEBpl1.0.
After download (first time also the calibration data tar files and the verification files ae required. Once you have installed them for upgrade it is sufficiont to load the version file (e.g. https://ftp.eso.org/pub/midaspub/22FEB/sources/22FEBpl1.0.tar.gz) untar all files in a setup directory (I use /usr/local/midas) , change to 22FEBpl1.0/install/unix, and run there ./autoconfig . The following compilation needs on my PC always about 4 minutes. Edit the file 22FEBpl1.0/monit/devices.sh to set your default BW and COLOR printer. Then make symbolic links in /usr/local/bin by

ln -s /usr/local/midas/22FEBpl1.0/system/unix/gomidas .
ln -s /usr/local/midas/22FEBpl1.0/system/unix/inmidas .
ln -s /usr/local/midas/22FEBpl1.0/system/unix/helpmidas .
ln -s /usr/local/midas/22FEBpl1.0/system/unix/drs .
To change the default behaviour of MIDAS you may create a private batch called $HOME/midwork/login.prg .
I for example set there the default file type to FITS (that is independent of Endianess and thus archieable) by
set/midas_system newfiles=fits,fits,tfits
and open some graphics windows for plotting by default.
Under current Ubuntu versions, the installation is even possible using sudo apt install eso-midas. Only the instructions have to be downloaded from ESO. However, the version is a bit behind (17FEBpl1.2). The original documentations are still in the somewhat bulky printer language postscript. Therefore here from my Adobe PDF converted versions with the capability of the full-text search in typical PDF browsers like xpdf or evince.

Volume A PDF & HTML : Command Language and Data I/O
Volume B PDF & HTML : Concepts and workflows for data reduction in imaging, long slit spectroscopy, Echelle spectroscopy, and crowded field photometry.
Volume C: individual commands - Since 1998 this part is only available directly in the program using the help or the create/gui help command within your MIDAS shell according to the version.
Extraction of sources in images: SExtractor https://www.astromatic.net/software/sextractor/ and https://sextractor.readthedocs.io/en/latest/Introduction.html . In all common LiNUX distributions, you can install it directly with sudo apt install sextractor, yum install sextractor or dnf install sextractor.
Finally, there is a very underestimated project under the much larger umbrella of the GNU consortium
(latest release April 29th 2023) providing GNU standard coded standalone programs being called from the shell
doing many of our operations - header analysis, arithmetic operations with images, arithmetic operations with FITS tables, statistics, generation of mock images and noise, ... (with I/O from/to FITS and ASCII files):
https://www.gnu.org/software/gnuastro/
The lifecycles of such standard C codes are certainly currently the longest ones - much better than astropy (short lifetime of 1-2 years without changes), IRAF (intermediate but now EoL - 1985-2012), and MIDAS (long since 1987 and still maintained)!
Even the Euclid project preparation papers by the consortium (e.g. Borlaff et al. 2022, A&A 657, A92
https://doi.org/10.1051/0004-6361/202141935) are using these GNU C codes already.