Handy Linux hacks for Freshers at Work
Tags: linux, ubuntu, beginners, career
Published: 2020-02-09 11:01:00

**Are you a freshser** feeling lost in the world of servers, devboxes, debugging realtime systems etc?. We've been sailing in the same boat. Here are a few handy stuff that will be your lifesaver till you are a fresher. 
*We can all do a* `man` *now and understand what the command actually does, but here's a cheatsheet to get you started quick.*

**Returning complete Exceptions/Trace from logs using grep**
You can use the -B and -A to print lines before and after the match.
`grep -i -A 10 'some.java.Exception' my-server.log`
This will print 10 lines after the Exception, including the line with exception. similarly -A will print lines before the exception

**Print all files , with  ports, parent processes etc**
*list the files/processes:* `sudo lsof -i -P -n`
*[For Debugging] LISTENING to:* `sudo lsof -i -P -n | grep LISTEN `
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/pzuse7np7gxihoba8kd2.png)
Want more detail?: `ps aux`
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/1in6h1bdkfykhs76g01z.png)
*a:- This option prints the running processes from all users.
u:- This option shows user or owner column in output.
x:- This option prints the processes those have not been executed from the terminal.*
When you know the PID of the process, you can use `kill PID_OF_THAT_PROCESS` to kill it 

**Tracking/ Debugging Network connections**
`netstat pntl`
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/orj9ujuoil87jwuqahec.png)


**Digging deeper into networking | fixing 502s, 504s**
Did you get a Gateway Timeout or a Bad Gateway response? Do you know if the packet is even reaching that server ? Try to do a `tcpdump` to see what packets are going in and coming into your system. 
`tcpdump -nnvvS src 172.31.1.237 and dst port 80`
this command will track all packets coming to port 80 of your system from 172.31.1.237. If we do the same on 172.31.1.237 and use our system's public IP on that device, we can see the packets arriving to that system from our server.

If you want to analyse network dum on private cloud cluster you can generate a pcap file and analyse it on Wireshark

`sudo ip -s -s link show

sudo nstat `


**Use curl instead of POSTMAN for request/response debugging**
`curl -k -i -X POST your_url_of_concern.com -d"{your data in json or urlencoded text}" -H   -H 'set cookie: some_cookie_expected_in_request=someid'`

The -k is for avoiding checks for ssl, which might be unnecessary inside your own dev-box or organization's internal APIs.

the -i option will print the headers, cookies information, which is highly useful while debugging server failures, session tracking
 
**replace edit delete occurence of text in file**
 *this one replaces unix with linux in geekfile.txt*
`sed 's/unix/linux/' geekfile.txt`

**Allowing an app through firewall**
First list all your apps requiring the feature
`sudo ufw app list`
Now allow the app through firewall
`sudo ufw allow 'APP_NAME'`

**Transferring files to-from remote host over SCTP**
This can come in handy when you want to get some downloaded or config files from/to servers to run some stuff locally

*without keys:* `scp -r  source_host_full_dir_path destination_host_full_dir_path`
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/4mt2hl5n1lglv6aamor5.png)
In this example, we transfer s.py from local to 172.31.1.237 remote machine

*if using ssh key:* `scp -i your_ssh_key_path -r  source_host_full_dir_path destination_host_full_dir_path`
*The -r is for including subfolders and files inside that folder. you wont need that for single files*

If ever you get permission denied during transfer to or from a server, go to the source's directory and change ownership of the file to the current logged in user


**Trouble sending certificates in json body ?***
certificates are in PEM format and line breaks are a hassle to pass through in json format. If you have to pass the plaintext of a cert in json, better convert those linebreaks into '\n'

`awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' your-cert.pem`

**Shortcut links to long directory paths**
*Do you have very long directorypaths, which you frequently need to access from current directory? why not have a shortcut to that file in your current directory. A symlink will create a pointer to that actual file from the current directory. All changes will be reflected on the original link.*
`ln sourcefilepath destinationfilepath`
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ive91g08ynx96wk820je.png)

**Executing commands directly in remote box**
example : `ssh -t username@host 'pwd'`
*The -t is required if you are executing multiple commands or if you want the shell to be interactive. Not required if you are not doing any of that. However if you want to execute a shell script from your local to your remote machine , you have to scp file transfer the script to that machine and then execute it. credits to this *[link](https://www.unix.com/302889378-post2.html?s=28ba6d8f25831ed014498356dcd786fe).

**Follwing up .logs using tail**
`tail -10f /location/of/your/.log`
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ws7z7r66zeeswgxn6hn8.png)
*the "10" denotes that it will tail/print the last 10 lines, in that file. You can set it to whatever value you want. This will update in real-time as your log file updates.
You can use \*.log instead of specific filename to track all log files in that directory*


**Setting up ENVIRONMENT VARIABLES :**
`export variablename=dont_add_spaces_here_whateveryouwant`
![Example: export SERTVERURL=127.0.0.1:8000](https://dev-to-uploads.s3.amazonaws.com/i/hiidw0lhdaiswan4b3iv.png)
use `printenv variablename` to print them

**Comparing Floats in a shell script**
Yes, it's a pain to compare floating point numbers in shell. This is not an issue in bash though ( yes bash and sh are different, in case you didn't know that). Here is an example to do so

`num1=18.04
num2=18.04
OUTPUT=$(echo "$num1 == $num2" |bc -l)`
will produce output 1 since they are equal.
Now, to use this OUTPUT, you again have to compare it using 
`[ $OUTPUT -ne 0]` 
this checks if the value of OUTPUT is not equal to zero. 

**Debugging | Logging stdouts from shell scripts**
There might be occassions where you have to debug stuff inside a shell script, but there is no elegant way of logging inside of shell ... or is it ?

We can dynamically open, close, and copy stdout to achieve logging operations. Let’s redirect stdout (FD 1) to the log file:
`#! /bin/bash
script_log="/tmp/log_`date +%F`.log"
exec 1>>$script_log
echo "This will be written into log file rather than terminal.."
echo "This too.."`
[Learn more](https://www.baeldung.com/linux/exec-command-in-shell-script)

**commits between 2 git hashes for better gitblame**

`git log <older commit hash>..<newer commit hash> --oneline | cut -d " " -f 1
`

**easy git update between branches without rebase**
Is rebasing causing conflicts for you even though you know your diff is perfectly fine and conflicts have already been resolved in upcoming commits ? In this case forget rebasing and do patching. take a diff between your current branch and the branch you wanna update (say master)

`git diff origin/master --no-color > ~/Documents/your_patch_file.patch
`
now checkout to a fresh master branch and
`git apply ~/Documents/your_patch_file.patch`

you will have all the changes of your current branch, in this one now, without any conflicts.
<note that this hack is good to have in experimental/rough branches and not in your release branch where people actually audit git commit hashes>


**Local git branch messed up? fix to latest working remote branch**
`sudo git clean -f -d; sudo git reset HEAD —hard`

**remove untracked files from git commits**
`git status | grep 'modified:' | cut -d' ' -f4|xargs sudo git rm -f`

*if you want to untrack deleted files too:*
`git status | grep 'deleted:' | cut -d' ' -f5|xargs sudo git rm -f`

*that* `cut -d' ' -f_number` *means you are splitting the current line by spaces and picking up the string corresponding to your file. so that f_number should be the position your file occurs in. It might vary in your case*

**clean unused old libraries from .m2 repository**
cleans all files more than an hour of last access from m2. They will be accessed if you had built within the past hour 
```
find ~/.m2 -amin +60 -iname '*.pom' | while read pom; do parent=`dirname "$pom"`; rm -Rf "$parent"; done
```

**Using anaconda? Do you clear the unused files?**
if you are thinking where all my disk space is going, go and check your anaconda package directory. Mine was 17.6GB. its crazy. Anaconda does not automatically clean files after installation.
`conda clean -a` will remove all tarballs, and setup packages of already installed packages. My directory size is now 10GB.

**A few stuff for precautions:**
Here are a few things to avoid when you are using `sudo`.

1.NEVER sudo in the root folder. This will deadlock your system.
2.To avoid using sudo in every place else, all the time, use sudo -s. and from there onwards, you need not type sudo again.
3.To avoid sudo in some situations,( especially when you are just testing some stuff), maybe consider changing ownership of files in that folder to your name so you can liberally access them.

`chown -R your_username /foldername` *will change ownership of all files subfolders, in that directory to you*

**Learning regex and VIM**
This will save you hours of typing time and your reputation for faster prod-issue debugging, once you get used to it. I recommend the below places to learn these skills in the fastest way possible:
[Regexone](https://regexone.com/lesson/repeating_characters?)
[thoughtbot](https://www.youtube.com/watch?v=OnUiHLYZgaA&t=630s)

**Working with JAVA ? Read about tweaking JVM with these**
[Java Options](https://geekflare.com/important-jvm-options/)