File and Pipe Redirection

When running a process, 3 things are needed. An input, an output, and an error. File descriptors for each are:
– fd 0 – input (stdin)
– fd 1 – normal output (stdinout)
– fd 2 -abnormal (stderr)

Redirection is a function that can redirect standard streams [Wikipedia]. Three operators for redirection are:
– File redirect: <, >, >>
– Pipeline: |
– File descriptor redirection: >&

FILE REDIRECT OPERATORS
Each file descriptors can be redirected to/from files: from file to fd 0 (<),  from fd 1 to file (>),  from fd 2 to file (2>).

PIPELINE REDIRECT OPERATOR
Consider cmd1 | cmd2. The pipe operator takes data sent to stdout by cmd1 and sends it to the input file descriptor of cmd2.

I summarized some samples of file output redirection. Probably, you could understand them better.

echo “This is a sample” > sample
— inputs “This is a sample” to sample text file
— an example of stdin

rm nofile
— outputs rm: cannot remove `nofile’: No such file or directory to screen
— an example of stderr

cat good nofile
will produce:
– this is good (stdout)
– cat: nofile: No such file or directory (stderr)

Try this! Standard file output redirection

cat good nofile > out.txt
output: cat: nofile: No such file or directory
cat out.txt
output: this is good
cat good nofile 2> err.txt
output: cat: nofile: No such file or directory
cat err.txt
output: cat: nofile: No such file or directory

Try this! Output redirection – two at once

cat good nofile > out1.txt 2> err.txt

cat out.txt
output: this is good
cat err.txt
output: cat: nofile: No such file or directory

Try this! Redirecting stdout & stderr

cat good nofile > out-err.txt 2>&1
cat out-err.txt
output: this is good
cat: nofile: No such file or directory

2>&1 is called duplicating. File descriptor 2 is converted to descriptor 1. Note: duplicating is always from right to left.
> means save all file descriptor 1.

Try this! Duplicating before and after redirection

cat good nofile 2>&1 > out-err.txt
cat out-error.txt
output: this is good

cat good nofile > out-err.txt 2>&1
cat out-err.txt
output: this is good
cat: nofile: No such file or directory

Try this! Piping stdout to stdin

cat good nofile | sed -n p > out.txt
cat out.txt
output: this is good

Try this! Piping stdout & stderr to stdin

cat good nofile 2>&1 | sed -n p > out-err.txt
output: cat: nofile: No such file or directory
stdin only are allowed by sed

Try this! Piping stderr to stdin

cat good nofile 2>&1 > /dev/null | sed -n p > err.txt
output cat: nofile: No such file or directory

Basic File Management

Some of the tasks in managing files include creating, editing, deleting, and copying files. I summarized them below for some kind of reference. 🙂

C R E A T I N G   A   F I L E
syntax:
cat > filename
content
content
content

cat > todo_list
eat
rest
sleep
* use ctrl + D to denote the end of the line

syntax:
touch filename(s)

touch file1
touch file1 file2

D I S P L A Y I N G   F I L E   C O N T E N T S
syntax:
cat filename

cat todo_list

syntax:
less filename

less todo_list

F I N D I N G   F I L E S
syntax:
find directory criteria [-exec commandx{} \;]

find / -name "x*"
find / -user 1000
find / -type f -user 502 -exec rm -f{}\;

C R E A T I N G   D I R E C T O R Y
syntax:
mkdir directory

mkdir newDir
mkdir -p docs/programs/versions
- p: creates all subdirectories

C O P Y I N G   F I L E S
syntax:
cp oldfilename newfilename

cp old.pdf new.pdf

Most common options for cp
– d: do not follow symbolic link (when used with -R)
– f: force
– I:interactive, prompt before overwrite
– p: preserve file attributes
– R:recursively copy directories

cp -r /mydir/* dir2/: copy files from mydir
cp -r /mydir/* dir2/: copy files from mydir including mydir

M O V I N G   F I L E S
syntax:
mv oldfilename newfilename

mv old.pdf new.pdf

R E N A M I N G   F I L E S
syntax:
mv oldfilename newfilename

mv oldfilename.txt newfilename.txt

R E M O V I N G   F I L E S
syntax:
rm filename

rm todo_list

L I S T I N G   DI R E C T O R Y
syntax:
ls pathnames

ls ~: list contents of home directory
ls /:list contents of root directory
ls ../:list contents of parent directory
ls */:list contents of subdirectories
ls -d */:list directories in the current directory

Process Text Stream with Filters

Text filters are used to modify the output of a file. They are helpful when searching for a specific keyword  and viewing a different output format from a file.

cat command is used to display the contents of a file. There are helpful options in viewing the file contents. The most commonly used are:
–   n: number each line of output
–   b: number only non-blank output line
–   A: show carriage return

Another function of cat is it can be used as a rudimentary text editor. How? By using a redirect. A redirect is used with a greater than sign >. Here’s a sample:

cat > sample-file
I'm typing the content of the sample-file.

Use CTRL+D to to save the content and end the interactive input.

You can also use the tac command to read text from the last to the first line. Obviously it’s the opposite of cat.

When analyzing log files, head or tail commands are helpful. By default, both output 10 lines of  text.

head file: outputs the first 10 lines
head [-20] file: outputs the first 20 lines
tail [-20] file: outputs the last 20 lines
head [+25] file: list text starting at line 25
tail -f file: continuously read a file, for real-time monitoring.

The wc utility for word count. It counts the number of bytes, words, and lines in files.
Options for wc output are:
–   l: count number of lines
–   w: count number of words
–   c or m: count number of bytes or characters

Another utility is nl. It functions like the cat -n.

nl -ba file: number lines including blanks
nl -bt file: number lines with text

The expand utility is used to replace TABs with spaces while the unexpand command is used for reverse operation.

To view binary files, use the hexdump utility. od utility may also be used.

To split a file into a smaller file, use the split utility.

split -1 5 file: create files xaa to xae w/ 5 lines each
split -1 5 file name-: create files xname-aa to name-ae

To not display the consecutive identical lines, use the uniq utility.

To extract a range of characters or fields from each line of text, use the cut utility.

cut - c range1, range 2 file: to manipulate characters
cut -d delimeter -f fields --output-delimeter=" " file

paste utility concatenates two files next to each other.

paste file1 file2

join utility outputs match fields between two files.

join -1 field_num -2 field_num file1 file2

sort utility arrange text in alphabetical order. -n option is used for numerical sort.

fmt utility to format output lines. fmt options are:
–   w: number of characters per line
–   s: split long lines but do not refill
–   u: place one space between each word and two spaces at the end of the sentence.

pr utility to paginate a file.

tr utility to translate one set of character ot another.

tr 'character 'character' < file

sed utility to search and replace patterns in text

sed '/ search_pattern/ new_pattern' file: substitute a pattern
sed 's/search_pattern/ new_pattern/g' file: substitute a pattern
 s - substitute
 g - globally; force substitution
sed '/key/new_pattern/g' file: if string exists, substitute a pattern

options for sed
–   e: execute following command
–   f: read commands from file
–   n: do not print out unedited lines
commands of sed
d – delete an entire line
r – read a file and append to output
s – substitute
w – write output to file

Work with Command Line

Who provides a Command Line Interface to the OS?  It’s actually the shell, the program that manages the interaction between the system and the user. the shell can be accessed via virtual console, terminal attached to a serial line, remote access over tcp/ip, X terminal running on an X session. There are various type of shell. Some of them are the following:
– sh or Bourne Shell: original shell used in UNIX system
– bash or Bourne Again Shell: Standard GNU Shell
– csh or CShell:  resembles of C PL. Asked by programmers
– tcsh or Turbo C Shell: superset of CShell
– ksh or Kor Shell:  a superset of sh, for people w/ UNIX background.

Okay, now that you are introduced to shells. It’s time for some practice on command lines. I’ve listed some shell-related commands with their function. Try them out! 🙂

cat etc/shells: list known shells of Linux system
echo $SHELL: check SHELL you are using.
chsh: change the shell. It modifies etc/passwd
chsh -s shell: change to a new shell

Environment variables are a set of dynamic named values that can affect the way running processes will behave on a computer[Wikipedia]. Environment variables in Linux are set in .bashrc.

set PS1
PS1= format
export PS1=
echo $PS1
unset PS1
x=5
echo $x

echo command is used to view the content of the specified variable.
set PS1: set a value for the variable PS1
P1=value: assign the new value
export PS1 make the variable environmental
unset PS1: unset the value of PS1
x=5: assign value to x
$SHELL - environment variable
$x - user defined variable

All environment variable are in all-caps. To see all defined variables, use the env command. Use | grep to search a specific variable

env | grep SHELL

Bash prompts are of two types. The primary and the secondary.
The primary bash prompt is set by the environment variable PS1. The secondary prompt is set by variable PS2.

The bash prompt settings:
$ export PS1=[\u@\h \w]\$
[dhee@engrPC ~]$
–   \u username
–   \h host name
–   \$ $ for users and # for root
–   \w full path of current working directory, ~ for home directory
–   \W base name of the current working directory
–   \! history number of the current command

The PATH  lists directories the shell searches, for the commands the user may type without having to provide the full path. To add a directory to path, type PATH=”$PATH:directory“ or PATH=”directory:$PATH”. This modifies the environment file in the etc directory.You can also modify /etc/login.defs to change the PATH of super users.

Summary of Commands (Part I)

Commands explained below aren’t necessarily the commands first encountered when running the Linux OS. It just happen that these are the first set of commands I met. All of the commands are helpful once you started reading and exploring Linux.

which command: display the file location of the indicated commands

whereis file: locates source/binary and manuals sections for specified files.

file directory: display the type of file (txt, dll, exe, bin)

touch filename: create a file

free: display memory information (-m)

lsusb: check connected usb devices (-v, -vv)

lspci: check the controllers (lspci | less, lspci | grep search)

lsmod: list all modules of drivers

modinfo module: display module information

ps -uxa: see running processes

ldd file: print shared library dependencies

cat filename: look, modify, or combine a file

less filename: display output one screen at a time

grep string filename: display output that contain the entered string. Use braces to ignore case-sensitve property

Debian Package Management

Package management automate the process of installing, upgrading, configuring, and removing software packages.
dpkg is the basic tool for package management. dpkg does not handle dependencies. Here’s a list of of tasks using dpkg.

Package Management Task using dpkg

Install a package: dpgk -i package-file
Remove a package: dpkg -r package-name
Purge a package: dpkg -P package-name
Find out which files a package owns: dpkg -L package-name
Find out which package a file belongs to: dpkg -S filename
Extract information from package: dpkg -e package-file
List contents of package-file: dpkg -c package-file

configuration file for dpkg (dpkg.cfg) is stored in /etc/dpkg directory.
/var/lib/dpkg/ directory contains package management status and system files. The most important are:
alternatives – contains files that define and store configuration for the command alternatives on the system.
available – information about packages available to the system, retrieved from every specified sources.
status – information about packages installed on, or removed from the system.

You may also use apt tool, the ultimate package tool to manage packages. apt handles dependecies and it is one of the strengths of dpkg. apt provides users an easy way of installing and upgrading a system. Here’s a list of tasks using apt.

Package Management Task using apt

Choose which mirror to download from: apt-setup
Allows CD-ROM to be scanned for packages: apt-cdrom
Search all package description: apt-cache search package-name
Search a package full description: apt-cache show package-name
Install a package: apt-get install package-name
update list of packages: apt-get update
Upgrade any packages: apt-get upgrade
Upgrade entire distribution: apt-get dist-upgrade
Remove package: apt-get remove package-name

Configuration file for apt is stored in /etc/apt directory. apt.conf contains general options for APT, such as which release of Debian to install, which proxy settings to use, etc.
apt.conf.d is managed by the software that integrates with apt such as dpkg-reconfigure and apt-list changes.
apt-config is useful for troubleshooting apt.conf problems.

Linux Ubuntu Desktop

The good news? It is now easier to use a Linux OS on your machine. Unlike before, it is just used by geeks working with command lines. Thanks to the power of user interfaces, thanks to GNOME. Well, I won’t discuss much of GNOME here. But as an overview, it is the reason why Linux is now a user-friendly OS. Linux desktop have its similarities and differences with Windows. Linux and Windows do have same functionality. Things you do with Windows, can be done with Linux as well. Applications on Windows do have have a corresponding application in Linux. The difference? One observation is the position of the minimize/maximize/close button, start menu button, and the task bar icons.

Another difference is: in Linux, you can move to another workspace but in Windows, you can’t. What a great feature, isn’t it? So helpful to those who open numerous applications. Additional great features are: Ubuntu Software Center and Network Tools.

With Ubuntu Software Center, application are categorized. This feature provides an efficient way of searching installers. With network tools, networking guys would really appreciate this one. You can do network-related stuff in one window.

I wont put much details about Linux Desktop. If you’re interested, just explore it and eventually you’ll get used to it and discover another great feature. Those three features are some of the great features i love with Linux. The main disadvantage of Linux is the availability of some software and drivers but there are always alternative to those unavailable installers.

3 Ways to Partition a Disk in Linux

By using GParted (GNOME Partition Editor)
It’s actually the easiest way to partition a disk.

You can access it on a live cd by clicking Systems > Administrator > GParted before the installation process. Or download, install, then run it after installation.

By fdisk command
Type fdisk command in the command line.
syntax: fdisk [device]
e.g. : fdisk /dev/sda

The basic commands needed are:
p – print the new partition table
n – create a new partition
d – delete a partition
q – quit without saving changes
w – write the new partition and exit
It is important to enter the w command for changes to take effect.

By using cfdisk
Type cfdisk command in the command line.
syntax: cfdisk [device]
e.g. : cfdisk /dev/sda

The difference of cfdisk from fdisk is it has a user interface. The partition table is listed and command buttons are located on the bottom of the screen. Use the arrow keys to change between partitions and commands.

fdisk and cfdisk commands are root user commands. But it is recommended to execute this commands from a non-root user. The root user can access important files on your system and it is important to secure those files. It is actually a bad idea to log-in as a root user. To execute a root user command from another user, use the sudo command. The sudo allows temporary access to run commands that would not normally be executed due to file permission restrictions. As an example, type sudo fdisk /dev/sda or sudo cfdisk /dev/sda. More of sudo in the upcoming article. 🙂

Plan your Linux Partition

This may sound boring but it is important to take note  the Linux File System. I pasted a screenshot of the sub directories of the root directory in Linux. You may check http://tldp.org/LDP/intro-linux/html/intro-linux.html for details.

root-subdirectories

It is important to plan your Linux partition. Why partition? To achieve higher data security in case of disaster. You can partition your hard disk to a maximum of four partitions and you could even extend it. i386 systems can be sliced up to fifteen to sixty three partitions. For this to become possible, one of the primary partition should become an extended partition and logical partitions should be added to it. Sound confusing? It looks like this:

/dev/hda1 –> primary
/dev/hda2 –> extended
—   /dev/hda5 –> logical
—   /dev/hda6 –> logical
—   /dev/hda7 –> logical
—   /dev/hda.. –> logical

Note: Logical partition always starts from hda5.

Recommended partition:

Swap
This is the space on the hard drive that can be used as a virtual memory. Linux have a maximum of 2GB swap limit. The traditional swap space value is twice the physical RAM. You can increase or decrease it or enter any value you prefer. Or you may not use any swap partition at all, as long as your system have a higher physical RAM.

The root file system (/)
This is the top of the directory tree and it contains Linux and all Linux installation files. The size of the root partition will vary depending on what you plan to install.

/home
This is the directory where your data are stored. It is roughly equivalent to the “My Documents” folder on a MS Windows desktop.

Or another recommendation would be:
sda1   –>   /boot
sda5   –>   /
sda6   –>   /home
sda7   –>  /usr
sda8   –>   /var
sda9   –>   /tmp
sda10 –>  swap

On  servers, system data tends to be separated from user data. Here is the recommended partition for a server system:
– a partition with all data necessary to boot the machine
– a partition with configuration data and server programs
– one or more partitions containing the server data such as database tables, user mails, an ftp archive etc.
– a partition with user programs and applications
– one or more partitions for the user specific files (home directories) •
– one or more swap partitions (virtual memory)

It will still depend upon you on how you’ll partition a system. These are just recommendations. Once partitions are made, you can only add more. Changing sizes or properties of existing partitions is even possible but not advisable.

All About Linux

What’s the main difference of Linux from Windows?? It’s an open source! Anyone can download it, anyone can install it. Anyone can use it anytime without charge!

But are you not wondering why most of us still use Windows beside knowing its high price? Windows 7 Home Basic price range from Php3000 to Php4000 (I can buy an external hard disk for that!). Most people are into windows because one main reason is that they got used to it. Windows started to develop their OS from desktops to servers while Linux started from servers to desktops.

Linux started with command lines that’s why we knew that it’s an operating system for the geek. But the good news is anyone can use Linux OS because it already has its user interface. Thanks to GNOME, a graphical user interface that runs on top of a computer operating system. Unity is also another user interface used by Linux.

Linux have more than a hundred distribution. It’s because anyone can modify and develop it. You can use any distribution you want, you have a lot of choices to choose from. 😉 These distributions have one thing in common, they have all the same kernel, the central core of a computer operating system. Linux distributions can be Red-hat based or Debian based. The difference of Redhat and Debian are their config files and package management. Some Redhat based distributions are Fedora, CentOS, Mandrake while some Debian based distributions are Ubuntu and Mint.

I’m not in favor of Linux nor against Windows, In fact I’m still a Windows user. I just had the chance to study Linux and compare both OS. It’s true that there are some things that Linux can do but Windows cannot do like Linux built-in remote desktop, single action update to all installed software and a lot more.