Linux I/O redirection
Introduction
One of the powers of the Unix command line is the use of input/output redirection and pipes. In this session, we have covered the redirection of input, output and error streams.
stdin, stdout, and stderr
The bash shell has three basic streams; it takes input from stdin (stream 0), it sends output to stdout (stream 1) and it sends error messages to stderr (stream 2) .
The keyboard often serves as stdin, whereas stdout and stderr both go to the display. This can be confusing to new Linux users because there is no obvious way to recognize stdout from stderr. Experienced users know that separating output from errors can be very useful.
The next sections will explain how to redirect these streams.
output redirection
stdout ( > )
stdout can be redirected with a greater than sign. While scanning the line, the shell will see the > sign and will clear the file.
The > notation is, in fact, the abbreviation of 1> (stdout being referred to as stream 1).
datasoft @ datasoft-linux ~/test10$ echo It is summer today!
It is summer today!
datasoft @ datasoft-linux ~/test10$ echo It is summer today! > summer.txt
datasoft @ datasoft-linux ~/test10$ cat summer.txt
It is summer today!
datasoft @ datasoft-linux ~/test10$
Note that the bash shell effectively removes the redirection from the command line before argument 0 is executed. This means that in the case of this command:
datasoft @ datasoft-linux ~/test10$ echo Hei > greeting.txt
the shell only counts two arguments (echo = argument 0, hello = argument 1). The redirection is removed before the argument counting takes place.
output file is erased
While scanning the line, the shell will see the > sign and will clear the file! Since this happens before resolving argument 0, this means that even when the command fails, the file will have been cleared!
datasoft @ datasoft-linux ~/test10$ cat summer.txt
It is summer today!
datasoft @ datasoft-linux ~/test10$ zcho It is summer today! > summer.txt
No command 'zcho' found, did you mean:
Command 'echo' from package 'coreutils' (main)
zcho: command not found
datasoft @ datasoft-linux ~/test10$ cat summer.txt
datasoft @ datasoft-linux ~/test10$
noclobber
Erasing a file while using > can be prevented by setting the noclobber option.
datasoft @ datasoft-linux ~/test10$ cat summer.txt
datasoft @ datasoft-linux ~/test10$ set -o noclobber
datasoft @ datasoft-linux ~/test10$ echo It is cold today! > summer.txt
bash: summer.txt: cannot overwrite existing file
datasoft @ datasoft-linux ~/test10$ set +o noclobber
datasoft @ datasoft-linux ~/test10$
overruling noclobber
The noclobber can be overruled with >|.
datasoft @ datasoft-linux ~/test10$ set -o noclobber
datasoft @ datasoft-linux ~/test10$ echo It is summer today! > summer.txt
bash: summer.txt: cannot overwrite existing file
datasoft @ datasoft-linux ~/test10$ echo It is summer today! >|summer.txt
datasoft @ datasoft-linux ~/test10$ cat summer.txt
It is summer today!
datasoft @ datasoft-linux ~/test10$
append (>>)
Use >> to append output to a file.
datasoft @ datasoft-linux ~/test10$ echo It is summer today! > summer.txt
bash: summer.txt: cannot overwrite existing file
datasoft @ datasoft-linux ~/test10$ cat summer.txt
It is summer today!
datasoft @ datasoft-linux ~/test10$ echo Where is the hot summer ? >> summer.txt
datasoft @ datasoft-linux ~/test10$ cat summer.txt
It is summer today!
Where is the hot summer ?
datasoft @ datasoft-linux ~/test10$
error redirection
2> stderr
Redirecting stderr is done with 2>. This can be very useful to prevent error messages from cluttering your screen.
The screenshot below shows redirection of stdout to a file, and stderr to /dev/null. Writing 1> is the same as >.
datasoft @ datasoft-linux ~/test10$ find / > allfiles.txt 2> /dev/null
datasoft @ datasoft-linux ~/test10$
2>&1
To redirect both stdout and stderr to the same file, use 2>&1.
datasoft @ datasoft-linux ~/test10$ find / > allfiles_and_error.txt 2>&1
Note that the order of redirections is significant. For example, the command
ls > dirlist 2>&1
directs both standard output (file descriptor 1) and standard error (file descriptor 2) to the file dirlist, while the command
directs only the standard output to file dirlist, because the standard error made a copy of the standard output before the standard output was redirected to dirlist.
output redirection and pipes
By default, you cannot grep inside stderr when using pipes on the command line, because the only stdout is passed.
datasoft @ datasoft-linux ~/test10$ rm file35 file=10 | grep test
rm: cannot remove 'file35': No such file or directory
rm: cannot remove 'file=10': No such file or directory
datasoft @ datasoft-linux ~/test10$ rm file10 | grep.txt
With 2>&1 you can force stderr to go to stdout. This enables the next command in the pipe to act on both streams.
datasoft @ datasoft-linux ~/test10$ rm file35 file10 file101 2>&1 1>&2 | grep file35
rm: cannot remove 'file35': No such file or directory
datasoft @ datasoft-linux ~/test10$
You cannot use both 1>&2 and 2>&1 to switch stdout and stderr.
datasoft @ datasoft-linux /$ rm file35 file10 file101 2>&1 1>&2 | grep file35
rm: cannot remove 'file35': No such file or directory
datasoft @ datasoft-linux /$ echo file35 2>&1 1>&2 | sed 's/file35/FILE35/'
FILE35
You need a third stream to switch stdout and stderr after a pipe symbol.
datasoft @ datasoft-linux /$ echo file35 2>&1 1>&2 | sed 's/file35/FILE35/'
FILE35
datasoft @ datasoft-linux /$ echo file35 3>&1 1>&2 2>&3 | sed 's/file35/FILE35/' file35
file35
sed: can't read file35: No such file or directory
datasoft @ datasoft-linux /$
joining stdout and stderr
The &> construction will put both stdout and stderr in one stream (to a file).
datasoft @ datasoft-linux ~$ cd test10
datasoft @ datasoft-linux ~/test10$ rm file35 &> out_and_err
datasoft @ datasoft-linux ~/test10$ cat out_and_err
rm: cannot remove 'file35': No such file or directory
datasoft @ datasoft-linux ~/test10$ echo file35 &> out_and_err
bash: out_and_err: cannot overwrite existing file
datasoft @ datasoft-linux ~/test10$
input redirection
stdin(<)
Redirecting stdin is done with < (short for 0<).
datasoft @ datasoft-linux ~$ cat < mno.txt
What is your name?
My name is prasanta.
datasoft @ datasoft-linux ~$ tr 'onetw' 'onezz' < mno.txt
Whaz is your name?
My name is prasanza.
datasoft @ datasoft-linux ~$
<< here document
The here document (sometimes called here-is-document) is a way to append input until a certain sequence (usually EOF) is encountered. The EOF marker can be typed literally or can be called with Ctrl-D.
check the below code
datasoft @ datasoft-linux ~$ cat < mno.txt
> What is your name?
> My name is prasanta.
> EOF
bash: mno.txt: cannot overwrite existing file
datasoft @ datasoft-linux ~$ cat mno.txt
What is your name?
My name is prasanta.
datasoft @ datasoft-linux ~$ cat < mno.txt
> What's your name?
> prasanta
> ^C
datasoft @ datasoft-linux ~$ cat mno.txt
What is your name?
My name is prasanta.
datasoft @ datasoft-linux ~$
<<< here string
The here string can be used to directly pass strings to a command. The result is the same as using echo string | command (but you have one less process running).
See rfc 3548 for more information about base64.
datasoft @ datasoft-linux ~$ base64 <<< linux-training.be
bGludXgtdHJhaW5pbmcuYmUK
datasoft @ datasoft-linux ~$ base64 -d <<< bGludXgtdHJhaW5pbmcuYmUK
linux-training.be
datasoft @ datasoft-linux ~$
confusing redirection
The shell will scan the whole line before applying redirection. The following command line is very readable and is correct.
datasoft @ datasoft-linux ~$ cat mno.txt > summer.txt 2> wrong.txt
But this one is also correct, but less readable.
datasoft @ datasoft-linux ~$ 2> wrong.txt cat mno.txt > summer.txt
Even this will be understood perfectly by the shell.
datasoft @ datasoft-linux ~$ < mno.txt > summer.txt 2> wrong.txt
quick file clear
So what is the quickest way to clear a file ?
datasoft @ datasoft-linux ~$ >foo
And what is the quickest way to clear a file when the noclobber option is set ?
datasoft @ datasoft-linux ~$ > |bar
bash: syntax error near unexpected token `|'
Exercise, Practice and Solution:
1. Activate the noclobber shell option.
Code:
set -o noclobber
set -C
2. Verify that noclobber is active by repeating a ls on /etc/ with redirected output to a file.
Code:
ls /etc > etc.txt
ls /etc > etc.txt (should not work)
3. When listing all shell options, which character represents the noclobber option?
Code:
echo $- (noclobber is visible as C)
4. Deactivate the noclobber option.
Code:
set +o noclobber
5. Make sure you have two shells open on the same computer. Create an empty tailing.txt file. Then type tail -f tailing.txt. Use the second shell to append a line of text to that file. Verify that the first shell displays this line.
Code:
datasoft @ datasoft-linux ~$ > tailing.txt
datasoft @ datasoft-linux ~$ tail -f tailing.txt
hello
world
in the other shell:
datasoft @ datasoft-linux ~$ echo hello >> tailing.txt
datasoft @ datasoft-linux ~$ echo world >> tailing.txt
6. Create a file that contains the names of five people. Use cat and output redirection to create the file and use a here document to end the input.
Code:
datasoft @ datasoft-linux ~$ cat > tennis.txt << ace
> Justine Henin
> Venus Williams
> Serena Williams
> Martina Hingis
> Kim Clijsters
> ace
datasoft @ datasoft-linux ~$ cat tennis.txt
Justine Henin
Venus Williams
Serena Williams
Martina Hingis
Kim Clijsters
datasoft @ datasoft-linux ~$
Previous:
Linux file globbing
Next:
Linux - Filters
It will be nice if you may share this link in any developer community or anywhere else, from where other developers may find this content. Thanks.
https://www.w3resource.com/linux-system-administration/i-o-redirection.php
- Weekly Trends and Language Statistics
- Weekly Trends and Language Statistics