Shell tools/extensions you might want

Here are some tools and extensions that I use in my shell. They’re not necessarily related to data science/development, but useful nonetheless.

  • ShellCheck will find bugs in your shell scripts and make them pretty
  • DirEnv will load and unload environment variables depending on the current directory
  • tldr.sh will show you simplified man pages
  • Attercop will generate shell commands from natural language, using the OpenAI API (yup, GPT-3 to the rescue)
  • jdupes is powerful duplicate file finder and an enhanced fork of ‘fdupes’.

General

Speed up new terminals being slow

In my case it was __conda_setup, apparently this is the cause:

A TL;DR as this is currently the first hit on Google: the problem is that the NotImplementedError() raised in conda/activate.py takes a long time to pop up in some configurations.

This worked for me:

conda config --set auto_activate_base false

– via this GitHub issue

Exec a command over all files of a certain type (or that fulfill certain criteria)

# Find all files with a .txt extension and execute a echo on them
find . -name "*.txt" -exec echo {} \;

– via Baeldung

Remove double quotes characters surrounding text

Two approaches:

text="\"hello world\""

# 1. Using sed, this doesn't look nice but will work as expected and will only remove surrounding quotes
sed -e 's/^"//' -e 's/"$//' <<<"$text"

# 2. Using tr, this looks way nicer but will also remove quotes from the middle of the text
echo "$text" | tr -d '"'

# 3. Using cut which was recommended to me by Copilot and, surprisingly, it works
cut -d '"' -f2 <<<"$text"

– via Stack Overflow

Use and format current date in scripts

# Pretty month and day - "February_03"
echo "$(date '+%B_%d')"

# Standard date format - "2023-02-03"
echo "$(date '+%Y-%m-%d')"

– via Stack Overflow

Zsh

Enable AZ CLI autocomplete in Zsh

If you don’t have the Bash autocomplete script available at /usr/local/etc/bash_completion.d/az, make sure to download it from here.

Then include the following lines in your .zshrc, after sourcing oh-my-zsh.sh.

# Enable bash autocompletions in zsh
autoload -U +X bashcompinit && bashcompinit

# Enable the completions
source /usr/local/etc/bash_completion.d/az

– via Stack Overflow and MyDiemHo

Prevent issues with brackets in commands

If you’re using Zsh and you’re getting errors like zsh: no matches found: pip install fuzzy[optional], you can either prefix the entire command with noglob or surround the bit with the brackets with single quotes.

noglob pip install fuzzy[optional]

pip install 'fuzzy[optional]'

SSH

Download one or more files to your local machine via SSH

Use scp.

# Download single file
local> scp username@host.com:/path/to/dir/file local/path 

# Download directory recursively
local> scp -r username@host.com:/path/to/dir local/path

– via Stack Overflow

Create a script that runs over SSH

Basic script

ssh <your-server> << EOF
    echo "Hello world"
    echo "This is where your script should be"
EOF

– via Amit Chaudhary

Displaying the ssh command outputs and avoiding the “Pseudo-terminal will not be allocated because stdin is not a terminal.” warning

ssh <your-server> -tt << EOF
    echo "Hello world"
    echo "This is where your script should be"
EOF

– via Stack Overflow

Bonus: Setting up a ssh key for passwordless auth

ssh-keygen -t rsa -b 2048

ssh-copy-id id@server -i <path_to_your_new_key>

ssh id@server  -i <path_to_your_new_key>

I’m including the path to the new key since I’m using a dedicated, passwordless key for this specific instance. If you’re just setting up the default key (~/.ssh/id_rsa) then you won’t need the -i option.

– via Stack Overflow

Grepping a text file with Windows line endings

tr '\r' '\n' < info.csv | grep "42"

Load environment variables from .env file in Bash

 export $(cat .env | xargs)

– via @mihow

xargs: unterminated quote

Well, it probably contains a quote. Use | xargs -0, to let it know that the input is NULL-terminated.

– via Stack Overflow