Using Wget with Proxy Setting

As we already knew that Wget is a super-useful utility to download pages and automate all
types of web related tasks. It works for HTTP as well as FTP URL's.
And this is  how to use wget through proxy server.

To get wget to use a proxy, you must set up an environment variable
before using wget. Type this at the command prompt / console:

For Windows:

set http_proxy=http://proxy.example.com:8080

For Linux/Unix:

export http_proxy="http://proxy.example.com:8080"

Replace proxy.example.com with your actual proxy server.

Replace 8080 with your actual proxy server port.

You can similarly use ftp_proxy to proxy ftp requests. An example on Linux would be:

export ftp_proxy="http://proxy.example.com:8080"

Then you should specify the following option in wget command line to turn the proxy behavior on:

–proxy=on

Alternatively you can use the following to turn it off:

–proxy=off

You can use –proxy-username="user name" –proxy-passwd="password" to set proxy user name and password where required.

Replace user name with your proxy server user name and password
with your proxy server password. Another alternative is to specify them
in http_proxy / ftp_proxy environment variable as follows:

export http_proxy="http://username:password@proxy.example.com:8080"

Happy using wget!

Curl Http Scripting

Online:  http://curl.haxx.se/docs/httpscripting.html
Date:    May 28, 2008
 
                The Art Of Scripting HTTP Requests Using Curl
                =============================================
 
 This document will assume that you're familiar with HTML and general
 networking.
 
 The possibility to write scripts is essential to make a good computer
 system. Unix' capability to be extended by shell scripts and various tools to
 run various automated commands and scripts is one reason why it has succeeded
 so well.
 
 The increasing amount of applications moving to the web has made "HTTP
 Scripting" more frequently requested and wanted. To be able to automatically
 extract information from the web, to fake users, to post or upload data to
 web servers are all important tasks today.
 
 Curl is a command line tool for doing all sorts of URL manipulations and
 transfers, but this particular document will focus on how to use it when
 doing HTTP requests for fun and profit. I'll assume that you know how to
 invoke 'curl --help' or 'curl --manual' to get basic information about it.
 
 Curl is not written to do everything for you. It makes the requests, it gets
 the data, it sends data and it retrieves the information. You probably need
 to glue everything together using some kind of script language or repeated
 manual invokes.
 
1. The HTTP Protocol
 
 HTTP is the protocol used to fetch data from web servers. It is a very simple
 protocol that is built upon TCP/IP. The protocol also allows information to
 get sent to the server from the client using a few different methods, as will
 be shown here.
 
 HTTP is plain ASCII text lines being sent by the client to a server to
 request a particular action, and then the server replies a few text lines
 before the actual requested content is sent to the client.
 
 Using curl's option -v will display what kind of commands curl sends to the
 server, as well as a few other informational texts. -v is the single most
 useful option when it comes to debug or even understand the curl<->server
 interaction.
 
2. URL
 
 The Uniform Resource Locator format is how you specify the address of a
 particular resource on the Internet. You know these, you've seen URLs like
 http://curl.haxx.se or https://yourbank.com a million times.
 
3. GET a page
 
 The simplest and most common request/operation made using HTTP is to get a
 URL. The URL could itself refer to a web page, an image or a file. The client
 issues a GET request to the server and receives the document it asked for.
 If you issue the command line
 
        curl http://curl.haxx.se
 
 you get a web page returned in your terminal window. The entire HTML document
 that that URL holds.
 
 All HTTP replies contain a set of headers that are normally hidden, use
 curl's -i option to display them as well as the rest of the document. You can
 also ask the remote server for ONLY the headers by using the -I option (which
 will make curl issue a HEAD request).
 
4. Forms
 
 Forms are the general way a web site can present a HTML page with fields for
 the user to enter data in, and then press some kind of 'OK' or 'submit'
 button to get that data sent to the server. The server then typically uses
 the posted data to decide how to act. Like using the entered words to search
 in a database, or to add the info in a bug track system, display the entered
 address on a map or using the info as a login-prompt verifying that the user
 is allowed to see what it is about to see.
 
 Of course there has to be some kind of program in the server end to receive
 the data you send. You cannot just invent something out of the air.
 
 4.1 GET
 
  A GET-form uses the method GET, as specified in HTML like:
 
        <form method="GET" action="junk.cgi">
          <input type=text name="birthyear">
          <input type=submit name=press value="OK">
        </form>
 
  In your favorite browser, this form will appear with a text box to fill in
  and a press-button labeled "OK". If you fill in '1905' and press the OK
  button, your browser will then create a new URL to get for you. The URL will
  get "junk.cgi?birthyear=1905&press=OK" appended to the path part of the
  previous URL.
 
  If the original form was seen on the page "www.hotmail.com/when/birth.html",
  the second page you'll get will become
  "www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK".
 
  Most search engines work this way.
 
  To make curl do the GET form post for you, just enter the expected created
  URL:
 
        curl "www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK"
 
 4.2 POST
 
  The GET method makes all input field names get displayed in the URL field of
  your browser. That's generally a good thing when you want to be able to
  bookmark that page with your given data, but it is an obvious disadvantage
  if you entered secret information in one of the fields or if there are a
  large amount of fields creating a very long and unreadable URL.
 
  The HTTP protocol then offers the POST method. This way the client sends the
  data separated from the URL and thus you won't see any of it in the URL
  address field.
 
  The form would look very similar to the previous one:
 
        <form method="POST" action="junk.cgi">
          <input type=text name="birthyear">
          <input type=submit name=press value=" OK ">
        </form>
 
  And to use curl to post this form with the same data filled in as before, we
  could do it like:
 
        curl -d "birthyear=1905&press=%20OK%20" www.hotmail.com/when/junk.cgi
 
  This kind of POST will use the Content-Type
  application/x-www-form-urlencoded and is the most widely used POST kind.
 
  The data you send to the server MUST already be properly encoded, curl will
  not do that for you. For example, if you want the data to contain a space,
  you need to replace that space with %20 etc. Failing to comply with this
  will most likely cause your data to be received wrongly and messed up.
 
  Recent curl versions can in fact url-encode POST data for you, like this:
 
        curl --data-urlencode "name=I am Daniel" www.example.com
 
 4.3 File Upload POST
 
  Back in late 1995 they defined an additional way to post data over HTTP. It
  is documented in the RFC 1867, why this method sometimes is referred to as
  RFC1867-posting.
 
  This method is mainly designed to better support file uploads. A form that
  allows a user to upload a file could be written like this in HTML:
 
    <form method="POST" enctype='multipart/form-data' action="upload.cgi">
      <input type=file name=upload>
      <input type=submit name=press value="OK">
    </form>
 
  This clearly shows that the Content-Type about to be sent is
  multipart/form-data.
 
  To post to a form like this with curl, you enter a command line like:
 
        curl -F upload=@localfilename -F press=OK [URL]
 
 4.4 Hidden Fields
 
  A very common way for HTML based application to pass state information
  between pages is to add hidden fields to the forms. Hidden fields are
  already filled in, they aren't displayed to the user and they get passed
  along just as all the other fields.
 
  A similar example form with one visible field, one hidden field and one
  submit button could look like:
 
    <form method="POST" action="foobar.cgi">
      <input type=text name="birthyear">
      <input type=hidden name="person" value="daniel">
      <input type=submit name="press" value="OK">
    </form>
 
  To post this with curl, you won't have to think about if the fields are
  hidden or not. To curl they're all the same:
 
        curl -d "birthyear=1905&press=OK&person=daniel" [URL]
 
 4.5 Figure Out What A POST Looks Like
 
  When you're about fill in a form and send to a server by using curl instead
  of a browser, you're of course very interested in sending a POST exactly the
  way your browser does.
 
  An easy way to get to see this, is to save the HTML page with the form on
  your local disk, modify the 'method' to a GET, and press the submit button
  (you could also change the action URL if you want to).
 
  You will then clearly see the data get appended to the URL, separated with a
  '?'-letter as GET forms are supposed to.
 
5. PUT
 
 The perhaps best way to upload data to a HTTP server is to use PUT. Then
 again, this of course requires that someone put a program or script on the
 server end that knows how to receive a HTTP PUT stream.
 
 Put a file to a HTTP server with curl:
 
        curl -T uploadfile www.uploadhttp.com/receive.cgi
 
6. HTTP Authentication
 
 HTTP Authentication is the ability to tell the server your username and
 password so that it can verify that you're allowed to do the request you're
 doing. The Basic authentication used in HTTP (which is the type curl uses by
 default) is *plain* *text* based, which means it sends username and password
 only slightly obfuscated, but still fully readable by anyone that sniffs on
 the network between you and the remote server.
 
 To tell curl to use a user and password for authentication:
 
        curl -u name:password www.secrets.com
 
 The site might require a different authentication method (check the headers
 returned by the server), and then --ntlm, --digest, --negotiate or even
 --anyauth might be options that suit you.
 Sometimes your HTTP access is only available through the use of a HTTP
 proxy. This seems to be especially common at various companies. A HTTP proxy
 may require its own user and password to allow the client to get through to
 the Internet. To specify those with curl, run something like:
 
        curl -U proxyuser:proxypassword curl.haxx.se
 
 If your proxy requires the authentication to be done using the NTLM method,
 use --proxy-ntlm, if it requires Digest use --proxy-digest.
 
 If you use any one these user+password options but leave out the password
 part, curl will prompt for the password interactively.
 
 Do note that when a program is run, its parameters might be possible to see
 when listing the running processes of the system. Thus, other users may be
 able to watch your passwords if you pass them as plain command line
 options. There are ways to circumvent this.
 
 It is worth noting that while this is how HTTP Authentication works, very
 many web sites will not use this concept when they provide logins etc. See
 the Web Login chapter further below for more details on that.
 
7. Referer
 
 A HTTP request may include a 'referer' field (yes it is misspelled), which
 can be used to tell from which URL the client got to this particular
 resource. Some programs/scripts check the referer field of requests to verify
 that this wasn't arriving from an external site or an unknown page. While
 this is a stupid way to check something so easily forged, many scripts still
 do it. Using curl, you can put anything you want in the referer-field and
 thus more easily be able to fool the server into serving your request.
 
 Use curl to set the referer field with:
 
        curl -e http://curl.haxx.se daniel.haxx.se
 
8. User Agent
 
 Very similar to the referer field, all HTTP requests may set the User-Agent
 field. It names what user agent (client) that is being used. Many
 applications use this information to decide how to display pages. Silly web
 programmers try to make different pages for users of different browsers to
 make them look the best possible for their particular browsers. They usually
 also do different kinds of javascript, vbscript etc.
 
 At times, you will see that getting a page with curl will not return the same
 page that you see when getting the page with your browser. Then you know it
 is time to set the User Agent field to fool the server into thinking you're
 one of those browsers.
 
 To make curl look like Internet Explorer on a Windows 2000 box:
 
        curl -A "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" [URL]
 
 Or why not look like you're using Netscape 4.73 on a Linux (PIII) box:
 
        curl -A "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL]
 
9. Redirects
 
 When a resource is requested from a server, the reply from the server may
 include a hint about where the browser should go next to find this page, or a
 new page keeping newly generated output. The header that tells the browser
 to redirect is Location:.
 
 Curl does not follow Location: headers by default, but will simply display
 such pages in the same manner it display all HTTP replies. It does however
 feature an option that will make it attempt to follow the Location: pointers.
 
 To tell curl to follow a Location:
        curl -L www.sitethatredirects.com
 
 If you use curl to POST to a site that immediately redirects you to another
 page, you can safely use -L and -d/-F together. Curl will only use POST in
 the first request, and then revert to GET in the following operations.
 
10. Cookies
 
 The way the web browsers do "client side state control" is by using
 cookies. Cookies are just names with associated contents. The cookies are
 sent to the client by the server. The server tells the client for what path
 and host name it wants the cookie sent back, and it also sends an expiration
 date and a few more properties.
 
 When a client communicates with a server with a name and path as previously
 specified in a received cookie, the client sends back the cookies and their
 contents to the server, unless of course they are expired.
 
 Many applications and servers use this method to connect a series of requests
 into a single logical session. To be able to use curl in such occasions, we
 must be able to record and send back cookies the way the web application
 expects them. The same way browsers deal with them.
 
 The simplest way to send a few cookies to the server when getting a page with
 curl is to add them on the command line like:
 
        curl -b "name=Daniel" www.cookiesite.com
 
 Cookies are sent as common HTTP headers. This is practical as it allows curl
 to record cookies simply by recording headers. Record cookies with curl by
 using the -D option like:
 
        curl -D headers_and_cookies www.cookiesite.com
 
 (Take note that the -c option described below is a better way to store
 cookies.)
 
 Curl has a full blown cookie parsing engine built-in that comes to use if you
 want to reconnect to a server and use cookies that were stored from a
 previous connection (or handicrafted manually to fool the server into
 believing you had a previous connection). To use previously stored cookies,
 you run curl like:
 
        curl -b stored_cookies_in_file www.cookiesite.com
 
 Curl's "cookie engine" gets enabled when you use the -b option. If you only
 want curl to understand received cookies, use -b with a file that doesn't
 exist. Example, if you want to let curl understand cookies from a page and
 follow a location (and thus possibly send back cookies it received), you can
 invoke it like:
 
        curl -b nada -L www.cookiesite.com
 
 Curl has the ability to read and write cookie files that use the same file
 format that Netscape and Mozilla do. It is a convenient way to share cookies
 between browsers and automatic scripts. The -b switch automatically detects
 if a given file is such a cookie file and parses it, and by using the
 -c/--cookie-jar option you'll make curl write a new cookie file at the end of
 an operation:
 
        curl -b cookies.txt -c newcookies.txt www.cookiesite.com
 
11. HTTPS
 
 There are a few ways to do secure HTTP transfers. The by far most common
 protocol for doing this is what is generally known as HTTPS, HTTP over
 SSL. SSL encrypts all the data that is sent and received over the network and
 thus makes it harder for attackers to spy on sensitive information.
 
 SSL (or TLS as the latest version of the standard is called) offers a
 truckload of advanced features to allow all those encryptions and key
 infrastructure mechanisms encrypted HTTP requires.
 
 Curl supports encrypted fetches thanks to the freely available OpenSSL
 libraries. To get a page from a HTTPS server, simply run curl like:
 
        curl https://that.secure.server.com
 
 11.1 Certificates
 
  In the HTTPS world, you use certificates to validate that you are the one
  you claim to be, as an addition to normal passwords. Curl supports client-
  side certificates. All certificates are locked with a pass phrase, which you
  need to enter before the certificate can be used by curl. The pass phrase
  can be specified on the command line or if not, entered interactively when
  curl queries for it. Use a certificate with curl on a HTTPS server like:
 
        curl -E mycert.pem https://that.secure.server.com
 
  curl also tries to verify that the server is who it claims to be, by
  verifying the server's certificate against a locally stored CA cert
  bundle. Failing the verification will cause curl to deny the connection. You
  must then use -k in case you want to tell curl to ignore that the server
  can't be verified.
 
  More about server certificate verification and ca cert bundles can be read
  in the SSLCERTS document, available online here:
 
        http://curl.haxx.se/docs/sslcerts.html
 
12. Custom Request Elements
 
 Doing fancy stuff, you may need to add or change elements of a single curl
 request.
 
 For example, you can change the POST request to a PROPFIND and send the data
 as "Content-Type: text/xml" (instead of the default Content-Type) like this:
 
        curl -d "<xml>" -H "Content-Type: text/xml" -X PROPFIND url.com
 
 You can delete a default header by providing one without content. Like you
 can ruin the request by chopping off the Host: header:
 
        curl -H "Host:" http://mysite.com
 
 You can add headers the same way. Your server may want a "Destination:"
 header, and you can add it:
 
        curl -H "Destination: http://moo.com/nowhere" http://url.com
 
13. Web Login
 
 While not strictly just HTTP related, it still cause a lot of people problems
 so here's the executive run-down of how the vast majority of all login forms
 work and how to login to them using curl.
 
 It can also be noted that to do this properly in an automated fashion, you
 will most certainly need to script things and do multiple curl invokes etc.
 
 First, servers mostly use cookies to track the logged-in status of the
 client, so you will need to capture the cookies you receive in the
 responses. Then, many sites also set a special cookie on the login page (to
 make sure you got there through their login page) so you should make a habit
 of first getting the login-form page to capture the cookies set there.
 
 Some web-based login systems features various amounts of javascript, and
 sometimes they use such code to set or modify cookie contents. Possibly they
 do that to prevent programmed logins, like this manual describes how to...
 Anyway, if reading the code isn't enough to let you repeat the behavior
 manually, capturing the HTTP requests done by your browers and analyzing the
 sent cookies is usually a working method to work out how to shortcut the
 javascript need.
 
 In the actual <form> tag for the login, lots of sites fill-in random/session
 or otherwise secretly generated hidden tags and you may need to first capture
 the HTML code for the login form and extract all the hidden fields to be able
 to do a proper login POST. Remember that the contents need to be URL encoded
 when sent in a normal POST.
 
 
14. Debug
 
 Many times when you run curl on a site, you'll notice that the site doesn't
 seem to respond the same way to your curl requests as it does to your
 browser's.
 
 Then you need to start making your curl requests more similar to your
 browser's requests:
 
 * Use the --trace-ascii option to store fully detailed logs of the requests
   for easier analyzing and better understanding
 
 * Make sure you check for and use cookies when needed (both reading with -b
   and writing with -c)
 
 * Set user-agent to one like a recent popular browser does
 
 * Set referer like it is set by the browser
 
 * If you use POST, make sure you send all the fields and in the same order as
   the browser does it. (See chapter 4.5 above)
 
 A very good helper to make sure you do this right, is the LiveHTTPHeader tool
 that lets you view all headers you send and receive with Mozilla/Firefox
 (even when using HTTPS).
 
 A more raw approach is to capture the HTTP traffic on the network with tools
 such as ethereal or tcpdump and check what headers that were sent and
 received by the browser. (HTTPS makes this technique inefficient.)
 
15. References
 
 RFC 2616 is a must to read if you want in-depth understanding of the HTTP
 protocol.
 
 RFC 2396 explains the URL syntax.
 
 RFC 2109 defines how cookies are supposed to work.
 
 RFC 1867 defines the HTTP post upload format.
 
 http://www.openssl.org is the home of the OpenSSL project
 
 http://curl.haxx.se is the home of the cURL project

Unix Grep Command Examples

First create the following demo_file that will be used in the examples

below to demonstrate grep command.<br>

<code> </code>

<pre>$ cat demo_file<br>THIS LINE IS THE 1ST UPPER CASE
LINE IN THIS FILE.<br>this line is the 1st lower case line in
this file.<br>This Line Has All Its First Character Of The Word
With Upper Case.<br><br>Two lines above this line is
empty.<br>And this is the last line.</pre>

<h3>1. Search for the given string in a single file</h3>

<p>The basic usage of grep command is to search for a specific string

in the specified file as shown below.</p>

<pre>Syntax:<br>grep "literal_string" filename</pre>

<p><code> </code></p>

<center>

</center>

<pre>$ grep "this" demo_file<br>this line is the 1st lower
case line in this file.<br>Two lines above this line is
empty.</pre>

<h3>2. Checking for the given string in multiple files.</h3>

<pre>Syntax:<br>grep "string" FILE_PATTERN</pre>

<p><code> </code><br>

This is also a basic usage of grep command. For this example, let us

copy the demo_file to demo_file1. The grep output will also include the

file name in front of the line that matched the specific pattern as

shown below. When the Linux shell sees the meta character, it does the

expansion and gives all the files as input to grep.</p>

<pre>$ cp demo_file demo_file1<br><br>$ grep "this"
demo_*<br>demo_file:this line is the 1st lower case line in this
file.<br>demo_file:Two lines above this line is
empty.<br>demo_file:And this is the last
line.<br>demo_file1:this line is the 1st lower case line in this
file.<br>demo_file1:Two lines above this line is
empty.<br>demo_file1:And this is the last line.</pre>

<h3>3. Case insensitive search using grep -i</h3>

<pre>Syntax:<br>grep -i "string" FILE</pre>

<p><code> </code><br>

This is also a basic usage of the grep. This searches for the given

string/pattern case insensitively. So it matches all the words such as

“the”, “THE” and “The” case insensitively as shown below.</p>

<pre>$ grep -i "the" demo_file<br>THIS LINE IS THE 1ST
UPPER CASE LINE IN THIS FILE.<br>this line is the 1st lower case
line in this file.<br>This Line Has All Its First Character Of
The Word With Upper Case.<br>And this is the last
line.</pre>

<h3>4. Match regular expression in files</h3>

<pre>Syntax:<br>grep "REGEX" filename</pre>

<p><code> </code><br>

This is a very powerful feature, if you can use use regular expression

effectively. In the following example, it searches for all the pattern

that starts with “lines” and ends with “empty” with anything

in-between. i.e To search “lines[anything in-between]empty” in the

demo_file.</p>

<pre>$ grep "lines.*empty" demo_file<br>Two lines above this line is empty.</pre>

<p>From documentation of grep: A regular expression may be followed by

one of several repetition operators:</p>

<ul>

<li>? The preceding item is optional and matched at most once.</li>

<li>* The preceding item will be matched zero or more times.</li>

<li>+ The preceding item will be matched one or more times.</li>

<li>{n} The preceding item is matched exactly n times.</li>

<li>{n,} The preceding item is matched n or more times.</li>

<li>{,m} The preceding item is matched at most m times.</li>

<li>{n,m} The preceding item is matched at least n times, but not

more than m times.</li>

</ul>

<h3>5. Checking for full words, not for sub-strings using grep -w</h3>

<p>If you want to search for a word, and to avoid it to match the

substrings use -w option. Just doing out a normal search will show out

all the lines.<br>

<code> </code><br>

The following example is the regular grep where it is searching for

“is”. When you search for “is”, without any option it will show out

“is”, “his”, “this” and everything which has the substring “is”.</p>

<pre>$ grep -i "is" demo_file<br>THIS LINE IS THE 1ST UPPER
CASE LINE IN THIS FILE.<br>this line is the 1st lower case line
in this file.<br>This Line Has All Its First Character Of The
Word With Upper Case.<br>Two lines above this line is
empty.<br>And this is the last line.</pre>

<p><code> </code><br>

The following example is the WORD grep where it is searching only for

the word “is”. Please note that this output does not contain the line

“This Line Has All Its First Character Of The Word With Upper Case”,

even though “is” is there in the “This”, as the following is looking

only for the word “is” and not for “this”.</p>

<pre>$ grep -iw "is" demo_file<br>THIS LINE IS THE 1ST
UPPER CASE LINE IN THIS FILE.<br>this line is the 1st lower case
line in this file.<br>Two lines above this line is
empty.<br>And this is the last line.</pre>

<h3>6. Displaying lines before/after/around the match using grep -A, -B

and -C</h3>

<p>When doing a grep on a huge file, it may be useful to see some lines

after the match. You might feel handy if grep can show you not only the

matching lines but also the lines after/before/around the match.</p>

<p><code> </code><br>

Please create the following demo_text file for this example.</p>

<pre>$ cat demo_text<br>4. Vim Word
Navigation<br><br>You may want to do several navigation in
relation to the words, such as:<br><br> * e - go to the end
of the current word.<br> * E - go to the end of the current
WORD.<br> * b - go to the previous (before) word.<br> * B -
go to the previous (before) WORD.<br> * w - go to the next
word.<br> * W - go to the next WORD.<br><br>WORD -
WORD consists of a sequence of non-blank characters, separated with
white space.<br>word - word consists of a sequence of letters,
digits and underscores.<br><br>Example to show the
difference between WORD and word<br><br> * 192.168.1.1 -
single WORD<br> * 192.168.1.1 - seven words.</pre>

<h4>6.1 Display N lines after match</h4>

<p>-A is the option which prints the specified N lines after the match

as shown below.</p>

<pre>Syntax:<br>grep -A "string" FILENAME</pre>

<p><code> </code><br>

The following example prints the matched line, along with the 3 lines

after it.</p>

<pre>$ grep -A 3 -i "example" demo_text<br>Example to show
the difference between WORD and word<br><br>* 192.168.1.1 -
single WORD<br>* 192.168.1.1 - seven words.</pre>

<h4>6.2 Display N lines before match</h4>

<p>-B is the option which prints the specified N lines before the match.</p>

<pre>Syntax:<br>grep -B &lt;N&gt; "string" FILENAME<br></pre>

<p><code> </code><br>

When you had option to show the N lines after match, you have the -B

option for the opposite.</p>

<pre>$ grep -B 2 "single WORD" demo_text<br>Example to show
the difference between WORD and word<br><br>* 192.168.1.1 -
single WORD</pre>

<h4>6.3 Display N lines around match</h4>

<p>-C is the option which prints the specified N lines before the

match. In some occasion you might want the match to be appeared with

the lines from both the side. This options shows N lines in both the

side(before &amp; after) of match.</p>

<pre>$ grep -C 2 "Example" demo_text<br>word - word
consists of a sequence of letters, digits and
underscores.<br><br>Example to show the difference between
WORD and word<br><br>* 192.168.1.1 - single WORD</pre>

<h3>7. Highlighting the search using GREP_OPTIONS</h3>

<p>As grep prints out lines from the file by the pattern / string you

had given, if you wanted it to highlight which part matches the line,

then you need to follow the following way.<br>

<code> </code><br>

When you do the following export you will get the highlighting of the

matched searches. In the following example, it will highlight all the

this when you set the GREP_OPTIONS environment variable as shown below.</p>

<pre>$ export GREP_OPTIONS='--color=auto'
GREP_COLOR='100;8'<br><br>$ grep this
demo_file<br><strong>this</strong> line is the 1st
lower case line in this file.<br>Two lines above
<strong>this</strong> line is empty.<br>And
<strong>this</strong> is the last line.</pre>

<h3>8. Searching in all files recursively using grep -r</h3>

<p>When you want to search in all the files under the current directory

and its sub directory. -r option is the one which you need to use. The

following example will look for the string “ramesh” in all the files in

the current directory and all it’s subdirectory.</p>

<pre>$ grep -r "ramesh" *</pre>

<h3>9. Invert match using grep -v</h3>

<p>You had different options to show the lines matched, to show the

lines before match, and to show the lines after match, and to highlight

match. So definitely You’d also want the option -v to do invert match.<br>

<code> </code><br>

When you want to display the lines which does not matches the given

string/pattern, use the option -v as shown below. This example will

display all the lines that did not match the word “go”.</p>

<pre>$ grep -v "go" demo_text<br>4. Vim Word
Navigation<br><br>You may want to do several navigation in
relation to the words, such as:<br><br>WORD - WORD consists
of a sequence of non-blank characters, separated with white
space.<br>word - word consists of a sequence of letters, digits
and underscores.<br><br>Example to show the difference
between WORD and word<br><br>* 192.168.1.1 - single
WORD<br>* 192.168.1.1 - seven words.</pre>

<h3>10. display the lines which does not matches all the given pattern.</h3>

<pre>Syntax:<br>grep -v -e "pattern" -e "pattern"</pre>

<p><code> </code></p>

<pre>$ cat
test-file.txt<br>a<br>b<br>c<br>d<br><br>$
grep -v -e "a" -e "b" -e "c" test-file.txt<br>d</pre>

<h3>11. Counting the number of matches using grep -c</h3>

<p>When you want to count that how many lines matches the given

pattern/string, then use the option -c.</p>

<pre>Syntax:<br>grep -c "pattern" filename</pre>

<p><code> </code></p>

<pre>$ grep -c "go" demo_text<br>6</pre>

<p><code> </code><br>

When you want do find out how many lines matches the pattern</p>

<pre>$ grep -c this demo_file<br>3</pre>

<p><code> </code><br>

When you want do find out how many lines that does not match the pattern</p>

<pre>$ grep -v -c this demo_file<br>4</pre>

<h3>12. Display only the file names which matches the given pattern

using grep -l</h3>

<p>If you want the grep to show out only the file names which matched

the given pattern, use the -l (lower-case L) option.<br>

<code> </code><br>

When you give multiple files to the grep as input, it displays the

names of file which contains the text that matches the pattern, will be

very handy when you try to find some notes in your whole directory

structure.</p>

<pre>$ grep -l this demo_*<br>demo_file<br>demo_file1</pre>

<h3>13. Show only the matched string</h3>

<p>By default grep will show the line which matches the given

pattern/string, but if you want the grep to show out only the matched

string of the pattern then use the -o option.<br>

<code> </code><br>

It might not be that much useful when you give the string straight

forward. But it becomes very useful when you give a regex pattern and

trying to see what it matches as</p>

<pre>$ grep -o "is.*line" demo_file<br>is line is the 1st
lower case line<br>is line<br>is is the last
line</pre>

<h3>14. Show the position of match in the line</h3>

<p>When you want grep to show the position where it matches the pattern

in the file, use the following options as</p>

<pre>Syntax:<br>grep -o -b "pattern" file</pre>

<p><code> </code></p>

<pre>$ cat
temp-file.txt<br>12345<br>12345<br><br>$ grep
-o -b "3" temp-file.txt<br>2:3<br>8:3</pre>

<p><code> </code><br>

<strong>Note:</strong> The output of the grep command above is not the

position in the line, it is byte offset of the whole file.</p>

<h3>15. Show line number while displaying the output using grep -n</h3>

<p>To show the line number of file with the line matched. It does

1-based line numbering for each file. Use -n option to utilize this

feature.</p>

<pre>$ grep -n "go" demo_text<br>5: * e - go to the end of
the current word.<br>6: * E - go to the end of the current
WORD.<br>7: * b - go to the previous (before) word.<br>8: *
B - go to the previous (before) WORD.<br>9: * w - go to the next
word.<br>10: * W - go to the next WORD.</pre>

<p><code> </code></p>

link source

Delecious....

when i look at it ..... slurp...
CHICKEN POT PIE SOUP OVER MASHED POTATOES

Karst Sulawesi

WOW.....

Go Ad-Free

To support free features you love, from time to time, we display
text ads on your blog to logged-out users. And you can make money from
your blog content. Any click on those ads will generate a revenue and 75% of the revenue from these ads will be credited to your account. All you need is Google AdSense account.

Become a Blogetery Supporter and you can earn 100% of the revenue or remove ads completely from your blogs for small fee.

AdSense ads already blended into your blogs, and your blog readers
will be more likely to read the ad, and subsequently click it. Plus, you can put ads into your sidebars using widget system.

Become a Blogetery Supporter today!

Did you know that Blogetery sometimes have ads on them, limited upload space and somewhat restricted functionality?

However, Blogetery Supporters have no ads, can earn 100% of AdSense revenue from your blog content, receive 1GB of upload space and get exclusive access to plugins alongside other extra functionality!

And Blogetery Supporters can easily use their blog on iPhones, other mobiles and through third party publishing devices like Windows Live Writer.