6 often needed sed scripts for string processing

1. How to remove all empty lines in a file
  sed -e '/^$/d' filename.txt

2. How to remove the leading empty spaces
  (whitespace only) -> sed -e 's/^[ ]*//' /file/path/filename.txt
  (whitespace and tab \t) -> sed -e 's/^[ \t]*//' /file/path/filename.txt

3. How to add some string to starting of all lines of a file
  sed -e 's/^/string-to-add-at-starting/g' /file/path/filename.txt

4. How to add date to starting of all lines
  sed -e "s/^/$(date) /g" /file/path/filename.txt  - Note: The Double quotes.
  Before    ->    line in a file
  After    ->    Thu May  4 17:31:58 IST 2017 line in a file

5. How to add formatted date to starting of all lines
  sed -e "s/^/$(date '+%m\/%d\/%Y %H:%M:%S') /g" /file/path/filename.txt
  Example:
   Before    ->    line in a file
   After    ->    05/04/2017 17:30:50 line in a file

6. How to do multiple replace at one go
  sed -e 's/string-to-find1/string-to-replace1/g' -e 's/string-to-find2/string-to-replace2/g' /file/path/filename.txt

Some More scripts:
7. How to remove all contents after pattern match
  sed -e '/pattern/,$d' /path-to/file/filename.txt
  For Ex: To remove all contents after first empty line -> sed -e '/^$/,$d'


8. How to remove all contents before pattern match
  sed -e '0,/pattern/d' /path-to/file/filename.txt
  For Ex: To remove all contents before first empty line -> sed -e '0,/^$/d'

How to create printable version of a man page..?

Most of us are always feel boring to read man pages in terminal window. what if we have a printable version of a particular command's manpage.? Nice isn't it.. so, how we can create a printable version of a manpage...

Step 1: Issue the following command in terminal
$man -t > destination_file_path.ps

Ex:$man -t mkdir > ~/Desktop/mkdir_manpage_printableversion.ps

Now you got a file named " mkdir_manpage_printableversion.ps " in your Desktop :). you can now print and use it..

Note: .ps(Postscript) is the default file type used to print.  It can be opened with any PDF viewer and can be printed.

How to find filesystem type in linux machine?

There are many ways to find the filesystem of a partition in linux (provided the partition mounted)
Following are the command line tools run these commands as is in terminal.

1.  cat /etc/fstab

2.  cat /etc/mtab

3.  mount

4.  df -T

Note:  You should know the partition number (like /dev/sda2, /dev/sda4 etc.,) or the partition label (Like mypendrive, 8GB_drive etc.,) or the UUID number.

How to create a Linux repository for the intranet/local machines?

1. Copy the whole repository of a particular linux distro(BOSS Gnu/linux, ubunutu) into a harddisk(external/internal), either by debmirror command or request/contact to the repository maintainers.
 1.1 Copy the repository using debmirror
   $debmirror -a -d -s --method= -h --root= --no-source --progress --ignore-release-gpg 
   switch a - architecture, d - distro etc..  Note: For more information read man pages of debmirror

ex: 
$debmirror -a i386 -d savir -s main,main/debian-installer --method=http -h packages.bosslinux.in --root=boss --no-source --progress --ignore-release-gpg /home/test/repository


2. install apache2 web server using
 $sudo apt-get install apache2

3. Either copy the whole content distro copy "/home/test/repository" to "/var//www" directory, or give a link to /home/test/repository in /var/www

4. find the ip address of the machine(where you do all the above procedures. i.e machine in which you keep the local/intranet repository/server) using
 $sudo ifconfig

5. add the following line as first entry in the  /etc/apt/sources.list file of all the linux machines that connected to the intranet as follows
 deb http:///boss savir main contrib non-free

6. Try apt-get update from the client machines if it updates fine, everything is fine.  Your local intranet repository is setup successfully.

how to view all the uninstalled packages in a debian based linux?

dpkg - > is a debian package manager command
dpkg -l - > will list all the packages with their status installed/deinstalled/removed/half-installed etc

The first 2(actually 3) letters of the output of " dpkg -l " command are significant to us, because it tells the current state of the package.  if it is
ii  - installed
rc - removed but configuration files are kept
So, " $dpkg -l | grep ^rc "
will display all the package that are removed but configuration files are still kept.

Note: if you purged configuration files of a package during removal using
"apt-get remove --purge < package_name >" then those packages will not be displayed.

For more information about dpkg :
i) man dpkg
ii) dpkg -l - > Read the first 4 lines ;-)
iii) Read this http://linuxprograms.wordpress.com/2010/05/11/status-dpkg-list/

How to install a specific version of a debian package?

Some times the debian package version which is installed in your linux box (say foo-package-x.y.xy ) is different from what version you really want(say foo-package-y.x.yx).  Assume that both the versions are available in the repository.  So, how we install the exact version we needed..? Here comes the solution, from terminal issue the following command,

$sudo apt-get install foo-package=y.x.yx
Replace the foo-package with your packagename, y.x.yx with the version (name)number you need.  Now, the package with the given version will be installed.

If you want to see all the available versions of particular debian package in the repository.. do the following,
$sudo apt-get update 
The above command updates the package index of your machine with the repository.
$sudo apt-cache show <package-name>
The above command shows the details of package, where we find version details also.

Want to install linux debian packages without internet connection..? check this out

How to enable wireless in BOSS GNU/Linux?

 Some time wireless/wi-fi is not working in BOSS GNU/Linux[or any debian based linux like ubuntu etc].  The reason

1.  The exact firmware/driver for the wireless card, what you are using is not installed.  Because your driver may not be the popular one which is used by many laptop vendor/company/product/model.

2.  Due to the license under which the firmware/driver is released.

Note: If the license of the driver allow to distribute then by default the driver is added with the Linux Operating system.  Some drivers are free but not allowed to distribute, in this case the user manually should install that driver by his own. 

In both cases you should find and install the correct firmware.  In general the following firmwares are enough for get wi-fi access. (There are some special case, where your wi-fi card not from the vendors like b43, broadcom, intel.  If it is so, you need to get the .deb file from the vendor's site or from other distro's repository.)


Install the following firmwares from the repository using terminal(with root permission),
Applications -> Accessories -> Terminal

$sudo apt-get install firmware-b43-installer firmware-brcm80211firmware-iwlwifi firmware-linux-free

Now you can check whether these firmwares installed, by the command
$dpkg -l | grep firmware
the o/p of this command will be like the following

ii  firmware-b43-installer       4.150.10.5-4        Installer package for firmware for the b43 driver
ii  firmware-brcm80211         0.32~bpo60+1    Binary firmware for Broadcom 802.11 wireless cards
ii  firmware-iwlwifi                0.32~bpo60+1    Binary firmware for Intel Wireless 3945, 4965 and 5000-series cards
ii  firmware-linux-free            2.6.32-39            Binary firmware for various drivers in the Linux kernel

How to findout make of the wireless card:

1. After the system bootup, from termianl read the message in "/var/log/syslog" by

$sudo tail /var/log/syslog

2. $lspci

3. $lshw

4. $lsusb (if you are using wireless dongle for your PC.)

after the installation of these things in your machine(some times we need to restart) wireless thing comes into the effect.  now just click on the network icon (nm-applet) and now you can see the options like

wireless network

enable wireless

create new wireless network

connect to hidden wireless network and the menu itself shows you all the available wireless networks.

For intel wifi cards:
install firmware-iwlwifi

For broadcom lp-phy type wifi cards:
install firmware-b43-lpphy-installer

How to lock a linux user from logging into a machine?

In some scenarios its needed to lock a user to login into his own account.  This can be achieved by many ways.

1.  GUI way

 1.1 Go to System -> Administration -> Users and Groups

 1.2 Now select the particular user from the left side, then click on the advanced settings button

 1.3 Click on the Advanced tab then tick the check box with value "Disable Account"

 1.4 To revoke follow the 1.1 & 1.2 now you can see a "Enable account" button click and enable

2. Terminal way

1. As a root user issue the following command in the terminal

    $sudo passwd -l "username of the user to lock"

How to create a simple local apt repository?

This post is regarding debian based distros which supports .deb, so the procedures are not applicable for distros that uses .rpm like fedora, redhat etc..
Requirements:
build-essential
dpkg-scanpackages
gzip

How to do:
1. Create a directory named "repo" in your home directory.  say $mkdir /home/test/repo
2. Copy all the debs you have into the /home/test/repo
3. Open your terminal as root and do the following & change directory to /home/test/repo
    #mkdir /home/test/repo
     #cd /home/test/repo
     #dpkg-scanpackages . /dev/null | gzip -9c > ./Packages.gz

Here, '.' specifies -> present working directory
/dev/null specifies -> override file, as we not going to use override file we give /dev/null
gzip -9c>./Packages.gz -> Compresses the package details into the file Packages.gz
(Packages.gz will be created within the present working directory)
4. add the following line in your /etc/apt/sources.list file
deb file:/home/test/repo/ /
5. Update your system using apt-get update command
#apt-get update
6. Thats all now your /var/lib/apt/lists will be updated
Note: Remember now whenever you use "apt-cache search" "apt-get install" it only installs from debs inside /home/test/repo.

Lost your grub after windows installation.. How to update, reinstall the grub in (BOSS) Linux?

Scenario:
  Once windows(vista, 7, xp) installed after the linux installation then one will lost his grub and there is no way to select the OS during the system startup(Dual Boot).  By default windows installs its boot loader into the MBR and not care about the other OS in the same machine.
So how to get the old grub with the new Windows OS entry in it..? here follows the steps..

1. Insert your BOSS Linux DVD(live, install, utility all are in the same DVD) in the drive and reboot your
system.
2. Select the live option and boot your system in live mode.
3. Click and mount the Linux partition whose grub is going to be used.
4. Now copy the mount location in the addressbar(location: it is somewhat like /media/some_chars_numbers)
5. Open your "root terminal" Applications --@ Accessories --@ Root terminal and issue the following commands
6. #mount --bind /dev /media/some_chars_numbers/dev
7. #chroot /media/some_chars_numbers
8. #update-grub
This updates the grub with the newly installed Windows OS entry in it
9. #grub-install /dev/sda
This installs the grub in the MBR(Master Boot Record).

Thats all.. now reboot the system and remove the DVD.  Now system will loads with the new updated grub where you can see all the OS available in your system and you can select(dual/multi boot) the OS which you want to login.  That's all :)

No display and Input signal out of range error in Linux?

Some times after the installation of linux(Debian based) one will end-up with the terminal and no GUI.  So, how can we fix it? Some quick checks 

1. Restart the gdm service by following command

$sudo /etc/init.d/gdm restart

2. Start the x-server

$sudo startx

3. Even after this if you are not getting the GUI and or got the error message like "Input signal out or range" then you need to edit your xorg.conf file.  This file is at /etc/X11/xorg.conf

i  ) Open the /etc/X11/xorg.conf file in your favorite editor with root privilege

ii ) Go to the section "Screen"  -> "Display" -> "Modes" put only the value "1024x768" and remove the all other values.  (If you know the correct resolution of your monitor the adding it is not a problem.)

iii) save the changes then again start the x-server(as like in quick check 2)

Hope this will help you.

Note: This is worked out in EduBOSS-2.0 and works fine.  EduBOSS is a linux distro for the school, college students which is derived from BOSS GNU/Linux.

ls command is your friend.. :)

How to know the inode of a file of directory:
$ls -i <- displays the innode of the file& directory in current directory
Ex:
$ ls -i
320785 a 320800 abc 320786 b 320787 c
The numbers represents the inode

How to list files sorted with modification time:
$ls -lt <- displays with time sorted manner with latest first
Ex:
$ ls -lt
total 4
-rw-r--r-- 1 hanu hanu 0 Aug 26 11:54 c.odt
drwxr-xr-x 2 hanu hanu 4096 Aug 26 11:53 empty_dir
-rw-r--r-- 1 hanu hanu 0 Aug 26 11:53 a.txt
-rw-r--r-- 1 hanu hanu 0 Aug 26 11:53 b.png

Note: if you want to reverse the output of ls add option -r
Ex:
$ ls -ltr
total 4
-rw-r--r-- 1 hanu hanu 0 Aug 26 11:53 b.png
-rw-r--r-- 1 hanu hanu 0 Aug 26 11:53 a.txt
drwxr-xr-x 2 hanu hanu 4096 Aug 26 11:53 empty_dir
-rw-r--r-- 1 hanu hanu 0 Aug 26 11:54 c.odt

When a new package installed.., What are the files installed & where they are?

In Debian linux(which uses apt-get & dpkg for package mangement) we can find out what are the files
get installed when a new package installed into your system. There are two ways to do this.

GUI Way:
System -> Administration -> Synaptic Package Manager
1. Now you got a GUI window. In that window on left side select "status" button & then "Installed"
2. Now you will see all the installed packages in your system on the right side.
3. Select the package to which you want to get the information, then right click select "properties"
4. Now you will get a new window with tabs "common, dependencies, installed files.. etc". Click the installed files tab.

Now you can see all the files installed in your system regarding that particular package with the full path to those files. :)

CUI Way(Terminal):
Issue the following command in terminal
#dpkg -L "packagename" | more
Note: Replace with the package which you want without quotes. I am using the debian derived Indian linux distro "BOSS- Bharat Operating System Solutions"



This will be useful when you are not sure/know where the configuration, log files, command regarding particular package is.

How to remove continuous(unnecessary) occurance of a character?

In Debian Linux we have one superb command tool called 'tr' [Translate or delete characters]. This simple handy tool has may usages, here am going to explain how to remove sequence of repeated (same)character.

The date command will produce output as follows
Ex-1-scenario:
$date
Thu Jun 2 17:10:17 IST 2011
Note: There are 2 spaces between the Month(Jun) and the date(2) because the date is single digit.
Ex-2-scenario:
$date
Thu Jun 22 17:10:17 IST 2011

So, if i try to cut the date using cut command then with empty space( ) as delimiter. then the field count will change.
For example:
Ex-1-scenario:
In f1-Thu f2-Jun f3-( ) f4-2 f5-17:10:17 f6-IST f7-2011

Ex-2-scenario:
In f1-Thu f2-Jun f3-2 f4-17:10:17 f5-IST f6-2011

So, how can i split fields correctly regardless of the single or double digit dates. Here comes our tr tool. It will be done as follows
for both scenario-1, scenario-2
$date | tr -s ' '
Note: if you use tr -s 'y' it will remove the continues occarance of 'yyyyy' and put only one single 'y' there.
will output with single space( ) between fields. Happy scripting.. :)

For more information see the man page of tr, date commands. For online man pages of tr, date.

What is cron,crontab & how to use it?

Cron :
is a daemon process which checks the crontab every minute for checking is there any scheduled task to execute. After this checking it goes to sleep and waits for the next minute to wokeup and check the crontab. The cron daemon is one of the bootup process which starts running from the bootup of the system. If you want to check the cron daemon process, issue the following command in terminal

$ps -e | grep cron

Crontab(le) :
is a file which has the entries for scheduled tasks. The entries follows the following syntax.
M H DOM MOY DOW Task(command)
Where,
M -> Minute entry. Value range (0-59)
H -> Hour entry Value range (0-23)
DOM -> Day of Month Value range (1-31)
MOY -> Month of the Year Value range (1-12)
DOW -> Day of week Value range (0-6) where 0-Sunday, 1-Monday, ....

Example entry:1

Min Hour DOM MOY DOW Task

*   *    *   *   *   date  >> /tmp/every_minute.txt 

* <- means 'for all' The above entry will works on every minute and just appends the output of the command 'date' into the file '/tmp/every_minute.txt'. By opening the every_minute file in gedit you can see the output on every minute. If you want any other task to automate in the backend or chage the time when it needs to run configure the task entry as your need. Example entry:2
Min Hour DOM MOY DOW Task
30 22 * * * rm -r /tmp/*

The above entry will run everyday night 10.30(22.30) and removes everything from the /tmp/ Directory.

So how edit crontab:
In terminal type the following command
$crontab -e

Now the crontab file will opens in your default terminal editor(Mostly vi or vim). Now just add the Example 1 at the bottom of the file then save and exit.
Now open the /tmp/every_minute.txt file you can see it is updated each minute with the output of the date command.

For more information about cron, crontab
$man cron

$man 5 crontab


Location of files related to Cron :
Files in /var/spool/cron, /var/spool/anacron
Files in /etc/cron* ex. /etc/cront.d/, /etc/cront.weekly/
File /etc/crontab


Details Based on:
BOSS GNU/Linux (And most of the debian derived linux)

how to find the pattern within files?

Using grep we achieve this..

Say we have to find out the word "gdk" from all the '.c' files in a directory.. the command would be

$grep -i "pattern" files

ex: $grep -i "gtk" *.c          <===== Will search all .c files in the current directory.

ex: $grep -i "gtk" file1, file.23, file <== Will search only in the specified files.

how to get the information/details about the installed packages in debian linux?

$dpkg -s "package-name"

-s <- is the option for find out the status of the given gives information like

1. The package is installed or not, removed
2. size of the installed package
3. Architecture support
4. Dependency
5 Version and so..

How to fetch n number of lines before or after a pattern match..

Lets assume there is a file which has 1000 lines. Now i want to see 20 lines from the a pattern (say pattern is 'top secret'). How come..?

Step 1: open the file in command prompt using 'cat' command with -n option(this option shows the content of the file with the line numbers).

Ex: $cat -n ./secrets.txt

step 2: piping the output of the 'step 1' to grep command with -i option(this ignores the case of given pattern i.e case insensitive) and pattern.

Ex: $cat -n ./secrets.txt | grep -i 'top secret'

step 3: now your output gives a line number(say 654) with the give pattern match, so, now using the head & tail commands with the line numbers we get the required output.

Ex: $head -654 | tail -10

Thats all.. :)

Using "ls" and "find" command how to display only directories..?

Following commands list only directories in the current directory.

1. #ls -l | grep ^d

2. #ls -d */

3. #find -type d
Ex: #find /tmp -type d <-- This command shows all the directories with in the /tmp directory.

Want only files to be listed.. use 1. #ls -l | grep -v ^d

2. #find -type f
Ex: #find /tmp -type f <-- This command shows all regular files with in the /tmp directory.