CodeSnippets: grep code

Never been to CodeSnippets before?

Snippets is a public source code repository. Easily build up your personal collection of code snippets, categorize them with tags / keywords, and share them with the world (or not, you can keep them private!)

Older Snippets »

10 total

Counting lines

to awk grep shell osx mac bash line unix count file sed directory lines wc by jvs on Sep 09, 2008

# create a test file
testfile="${HOME}/Desktop/testfile.txt"
jot -b 'sample text' 10 | cat -n > "$testfile"
printf "%s\n" >> "$testfile"                                 # add an empty line
printf "%s\n\r\n\r\n\r\r\n" "sample text" >> "$testfile"     # add lines with '\r\n' as line separators 
printf "%s" "sample text" >> "$testfile"                     # add a last line without a terminating '\n'       

open -e "$testfile"

function odcfile() {
/usr/bin/od -A n -c < "$@" | /usr/bin/sed -E -e 's/^[[:space:]]{11}//' \
       -e s/[[:space:]]{4}/$'\001'/g \
       -e 's/[[:space:]]+//g' | \
       /usr/bin/tr -d '\n' | /usr/bin/tr '\001' ' ' | \
       /usr/bin/sed -e s/\\\\n/$'\\\\\\n\\\n'/g 

return 0
}

odcfile "$testfile"



# wc -l
wc -l < "$testfile"     # returns the number of '\n' characters

# ed 
ed -s "$testfile" <<< '='     # all lines

# cat
cat -n "$testfile" | awk 'END {print $1}'   # all lines


# grep
grep -c $'\n' "$testfile"                 # all lines
grep -c '^.*$' "$testfile"                # all lines
#grep -c $'\000' "$testfile"
grep -c $'\r$' "$testfile"                # all lines ending with \r\n
grep -c '^[[:space:]]*$' "$testfile"      # all blank lines
grep -cv '^[[:space:]]*$' "$testfile"     # all non-blank lines


# sed
sed -n '$=' "$testfile"                          # all lines
sed -n /$'\r'$/= "$testfile" | wc -l             # all lines ending with \r\n
sed -n '/^[[:space:]]*$/=' "$testfile" | wc -l   # all blank lines
sed -n '/[^[:space:]]/=' "$testfile" | wc -l     # all non-blank lines


# awk
awk 'END {print NR}' "$testfile"                           # all lines
awk '{x++} END {print x}' "$testfile"                      # all lines
awk '/^.*$/ {++x} END {print x}' "$testfile"               # all lines
awk '/\r$/ {++x} END {print x}' "$testfile"                # all lines ending with \r\n
awk '/^[[:space:]]*$/ {++x} END {print x}' "$testfile"     # all blank lines
awk '/[^[:space:]]/ {++x} END {print x}' "$testfile"       # all non-blank lines


# nl
nl "$testfile" | awk 'END {print $1}'
nl -b a "$testfile" | awk 'END {print $1}'   # including empty lines


#---------------------------------------------


# counting the lines of files in a directory

DIR=/path/to/dir

find "$DIR" -type f -name "*.txt" -print0 | xargs -0 wc -l

find "$DIR" -type f -name "*.txt" -print0 | xargs -0 sed -n '$='

find "$DIR" -type f -name "*.txt" -print0 | xargs -0 awk 'END {print NR}'
find "$DIR" -type f -name "*.txt" -print0 | xargs -0 awk '{x++} END {print x}'
find "$DIR" -type f -name "*.txt" -print0 | xargs -0 awk '/\r$/ {++x} END {print x}'
find "$DIR" -type f -name "*.txt" -print0 | xargs -0 awk '/^[[:space:]]*$/ {++x} END {print x}'
find "$DIR" -type f -name "*.txt" -print0 | xargs -0 awk '/[^[:space:]]/ {++x} END {print x}'


# rather slow, but without the "Argument list too long" issue: /usr/sbin/sysctl kern.argmax
# cf. http://www.onlamp.com/pub/a/bsd/2002/03/14/FreeBSD_Basics.html

declare -i linecnt=0
while read -d $'\0' file; do
   #linecnt=$((${linecnt} + $(/usr/bin/sed -n '$=' "${file}")))
   let "linecnt += $(/usr/bin/sed -n '$=' "${file}")"   # cf. help let
done < <(/usr/bin/find "$DIR" -type f -name "*.txt" -print0)

echo $linecnt

pcregrep - UTF-8 aware grep replacement

to grep perl shell osx mac bash unix utf8 UTF-8 pcre pcregrep multiline heirloom project by jvs on Aug 24, 2008

# we first have to download, compile & install the PCRE library, cf. http://www.pcre.org/pcre.txt
# requirement: Xcode, http://developer.apple.com/tools/xcode/index.html

cd ~/Desktop
/usr/bin/curl -L -O http://downloads.sourceforge.net/pcre/pcre-7.7.tar.gz
/usr/bin/tar -xzf pcre-7.7.tar.gz
cd pcre-7.7
./configure --help
./configure --prefix=/usr/local --enable-utf8 --enable-unicode-properties
# for Intel Macs, see http://hivelogic.com/articles/2005/12/ruby_rails_lighttpd_mysql_tiger
#./configure --prefix=/usr/local --enable-utf8 --enable-unicode-properties CFLAGS=-O1
/usr/bin/make
/usr/bin/sudo /usr/bin/make install 


ls -l /usr/local/bin/pcregrep
stat -x /usr/local/bin/pcregrep

pcregrep --version
pcregrep --help
pcregrep --help | pcregrep -i 'utf-?8'
pcregrep --help | pcregrep -i multiline

man pcregrep
man pcrepattern
man pcretest
man perlretut

man pcregrep | less -p utf-8
man pcregrep | less -p multiline
man perlretut | less -p 'single line and multi'

open /usr/local/share/doc/pcre/html/pcregrep.html


# check if character set encoding of Terminal.app is set to UTF-8
if [[ "$(/usr/bin/defaults read com.apple.Terminal StringEncoding)" != "4" ]]; then 
   echo 'Terminal.app does not use UTF-8 character set encoding!'
   exit 1
fi


utf8str=$'caf\303\251'

printf $utf8str | /usr/bin/egrep -o '.'
printf $utf8str | /usr/local/bin/pcregrep -o '.'
printf $utf8str | /usr/local/bin/pcregrep -ou '.'     # UTF-8 aware
printf $utf8str | /usr/local/bin/pcregrep -ou 'f.$'

printf $utf8str | /usr/bin/egrep -o '.' | wc -l
printf $utf8str | /usr/local/bin/pcregrep -o '.' | wc -l
printf $utf8str | /usr/local/bin/pcregrep -ou '.' | wc -l     # UTF-8 aware


#---------------------------------------------


# cf. also The Heirloom Project, http://heirloom.sourceforge.net
# download & install from http://homepage.mac.com/stefan.tramm/iWiki/HeirloomNotes.html

# backup ~/.bash_login with time stamp in filename
/bin/cp -ip "${HOME}/.bash_login"{,".orig-$(/bin/date +%Y-%m-%d-%H.%M.%S)"}


# To use the Heirloom tools insert the following statements into your ~/.profile or ~/.bash_login:

/bin/cat >> "${HOME}/.bash_login" <<-'EOF'

# Heirloom userland
# http://homepage.mac.com/stefan.tramm/iWiki/HeirloomNotes.html

HEIRLOOM=/opt/heirloom
if [[ -d $HEIRLOOM ]]; then
  export HEIRLOOM
  PATH=$PATH:$HEIRLOOM/5bin
else
  unset HEIRLOOM
fi

EOF


# make Heirloom source the ~/.bash_login shell functions
/bin/ls -lo /opt/heirloom/etc/5.rc
/usr/bin/sudo /bin/cp -ip /opt/heirloom/etc/5.rc{,.orig}   # backup
/usr/bin/sudo /bin/chmod 766 /opt/heirloom/etc/5.rc

# ~/.bash_login should "source ~/.bashrc" and "bind -f ~/.inputrc"
/usr/bin/sudo echo 'source ~/.bash_login' >> /opt/heirloom/etc/5.rc

/usr/bin/sudo /usr/sbin/chown root:admin /opt/heirloom/etc/5.rc
/usr/bin/sudo /bin/chmod 644 /opt/heirloom/etc/5.rc
/bin/ls -lo /opt/heirloom/etc/5.rc

# delete the last (added) line
#/usr/bin/sudo /usr/bin/sed -i '' '$,$d' /opt/heirloom/etc/5.rc

/usr/bin/open -e /opt/heirloom/etc/5.rc


source ~/.bash_login


/usr/bin/open -e /opt/heirloom/README
/usr/bin/open /opt/heirloom/{,{etc/,5bin/}}      # open three directories in one go
/usr/bin/open /opt/heirloom/doc/{,doctools}
/usr/bin/open /opt/heirloom/doc/{,doctools/{,quickstart.pdf}}  
/usr/bin/open /opt/heirloom/doc/doctools/quickstart.pdf       # explore Heirloom troff

5 man intro | less -p 'Multibyte character encodings'
5 man sh | less
5 man tsort | less
5 whodo
5
man pgrep
pgrep sh
pgrep bash
man bfs | less  # bfs - big file scanner


/usr/bin/stat -x $HEIRLOOM/5bin/5
/usr/bin/stat -f '%N:  %HT%SY' $HEIRLOOM/5bin/5
/usr/bin/stat -f $'%N:  \e[1m%HT%SY\e[m' /opt/heirloom/bin/tsort
/usr/bin/stat -f $'%N:  \e[1;31m%HT\e[m%SY' /opt/heirloom/5bin/awk


/usr/bin/open http://heirloom.sourceforge.net/man/grep.1.html
5 man grep | less


5

utf8str=$'caf\303\251'
echo $utf8str

printf "${utf8str}\n" | /usr/local/bin/pcregrep -u 'f.$'
printf "${utf8str}\n" | /opt/heirloom/5bin/grep 'f.$'
printf "${utf8str}\n" | /opt/heirloom/5bin/posix/grep -E -e 'f.$'

Mass find-replace using sed

to find grep sed replace by jamiew on Mar 30, 2007

Replace some widespread nasty hardcoded strings. 'sed $file > $file' doesn't work so hot so we use a temp file, and also make a backup of the old file.

for i in $(find . -type f); do sed 's/oldstring/newstring/g' $i > $i-tmp; mv $i $i-backup; mv $i-tmp $i; done

Add all new files to Subversion

to awk grep svn subversion by Chrononaut on Mar 13, 2007 1 Comments

This solution is shell-agnostic, unlike http://textsnippets.com/posts/show/450, and cleaner than to do a --force. I recommend adding it to a alias in your shell, i.e "sall".

svn st | grep "^?" | awk -F "      " '{print $2}' | xargs svn add

Can be modified to do a svn delete on files you accidentally rm -rf'ed:

svn st | grep "^!" | awk -F "      " '{print $2}' | xargs svn delete

Add all new files to Subversion

to awk grep svn bash subversion by Seven.net.nz on May 04, 2006 4 Comments

Bash script to add all new files to Subversion

for i in `svn st | grep ? | awk -F "      " '{print $2}'`; do svn add $i; done

opening in a textmate project all files that match a pattern

to grep regex shell osx mac unix textmate by cch on Apr 20, 2006

This will pass all matched filenames to textmate, generating a project with all files flatly in the drawer (no dirs)

egrep -lr pattern * | grep -v .svn | xargs mate

Searching with grep & Spotlight's kMDItemDisplayName

to grep regex shell osx mac mdfind by jv on Mar 16, 2006


# search for a file or folder (path) name that contains the specified search string
mdfind 'kMDItemDisplayName == "*apple*"wc'
mdfind -0 'kMDItemDisplayName == "*apple*"wc' | xargs -0 basename
mdfind -0 'kMDItemDisplayName == "*apple*"wc' | xargs -0 basename | grep Apple
mdfind -0 -onlyin ~/Desktop 'kMDItemDisplayName == "*apple*"wc' | xargs -0 basename
mdfind '*==*' | grep -i apple
mdfind '*==*' | grep -i -E '/[^/]*apple[^/]*$'
mdfind '*==*' | grep -i -E '/[^/]*apple[^/]*$' | while read filename; do basename "$filename"; done



# the following is an example of bash process substitution
# http://wooledge.org:8000/BashFAQ (FAQ 20 & 24)
# http://wooledge.org:8000/ProcessSubstitution

# read matched file names into an array

unset -v ar i
dir="$HOME/Desktop"
mdfind_search="'*search_string*'wc"
mdfind_search="\"*search_string*\"wc"   # alternative

while read -d $'\0' filename; do 
   ar[i++]="$filename"
done < <(mdfind -0 -onlyin ${dir} "kMDItemDisplayName == ${mdfind_search}")

echo ${ar[*]}; echo; echo ${ar[0]}; echo ${ar[1]}; echo



# let mdfind & kMDItemDisplayName search for the fixed part of the search string, and grep for the variable one
# read the matched file names into an array

unset -v ar i
dir="$HOME/Desktop"
mdfind_search="'*search_string*'wc"
grep_search="[0-5]"

while read -d $'\0' filename; do
   #if [[ -n $(echo "$filename" | grep -Eos -m1 "$grep_search") ]]; then     # match entire path name with grep
   if [[ -n $(basename "$filename" | grep -Eos -m1 "$grep_search") ]]; then     # match last part of path name with grep
      #ar[i++]="$filename"
      ar[i++]="$(basename "$filename")"
   fi 
done < <(mdfind -0 -onlyin ${dir} "kMDItemDisplayName == ${mdfind_search}") 

echo ${ar[*]}; echo; echo ${ar[0]}; echo ${ar[1]}; echo



# To search only for files add: 'kMDItemKind != "Folder" && ...'

unset -v ar i
dir="$HOME/Desktop"
mdfind_search="'*search_string*'wc"
grep_search="[0-5]"

item1="'*text*'wc"
item2="'*document*'wc"
item3="'*source*'wc"

while read -d $'\0' filename; do
   #if [[ -n $(echo "$filename" | grep -Eos -m1 "$grep_search") ]]; then 
   if [[ -n $(basename "$filename" | grep -Eos -m1 "$grep_search") ]]; then 
      #ar[i++]="$filename"
      ar[i++]="$(basename "$filename")"
   fi 
done < <(mdfind -0 -onlyin ${dir} "kMDItemKind != 'Folder' && kMDItemDisplayName == ${mdfind_search}"); 
#done < <(mdfind -0 -onlyin ${dir} "( kMDItemKind != 'Folder' && ( kMDItemKind == ${item1} || kMDItemKind == ${item2} || kMDItemKind == ${item3}) ) && kMDItemDisplayName == ${mdfind_search}"); 

echo ${ar[*]}; echo; echo ${ar[0]}; echo ${ar[1]}; echo



# To search only for folders add: 'kMDItemKind == "Folder" && ...'

unset -v ar i
dir="$HOME/Desktop"
mdfind_search="'*search_string*'wc"
grep_search="[0-5]"

while read -d $'\0' filename; do
   #if [[ -n $(echo "$filename" | grep -Eos -m1 "$grep_search") ]]; then 
   if [[ -n $(basename "$filename" | grep -Eos -m1 "$grep_search") ]]; then 
      #ar[i++]="$filename"
      ar[i++]="$(basename "$filename")"
   fi 
done < <(mdfind -0 -onlyin ${dir} "kMDItemKind == 'Folder' && kMDItemDisplayName == ${mdfind_search}"); 

echo ${ar[*]}; echo; echo ${ar[0]}; echo ${ar[1]}; echo

grep-based phrase search with Spotlight

to grep regex shell osx mac mdfind by jv on Mar 10, 2006

Some examples:


# for mdfind "phrase* " and "phrase! " also work, but not "phrase ! "
mdfind -0 -onlyin ~ "phrase This is a" | xargs -0 grep -ails -E 'This is a phrase!'

mdfind -0 "It's just $8.99!" | xargs -0 grep -ails -E "It's just \\\$8\\.99\\!"

mdfind -0 'Fr\303\251d\303\251ric Chopin' | xargs -0 grep -ails -Z -E 'Fr.d.ric Chopin' | xargs -0 basename

Regex-based file/folder search with Spotlight

to grep regex shell osx mac mdfind by jv on Mar 06, 2006

To get more Spotlight search options use the commands mdimport -A and mdls file.txt in Terminal.app.


mdfind -onlyin ~ "*==*" | grep -i -E 'part.*of.*name'

mdfind -onlyin ~ 'kMDItemContentType == "text"wc' | grep -i -E '.*\.txt$'
#mdfind -onlyin ~ '*==* && kMDItemContentType == "text"wc' | grep -i -E '.*\.txt$'

mdfind -onlyin ~ 'kMDItemKind == "Folder"' | grep -i -E '.*photos.*[0-5]'

View processes, grep out by user and then kill all their PIDs

to awk ps sysadmin grep kill by jason on May 26, 2005

ps axu | grep user | kill -9 `awk �{print $2}�`

Older Snippets »

10 total

CodeSnippets > grep

Related Tags

Never been to CodeSnippets before?

On This Page:

Counting lines

pcregrep - UTF-8 aware grep replacement

Mass find-replace using sed

Add all new files to Subversion

Add all new files to Subversion

opening in a textmate project all files that match a pattern

Searching with grep & Spotlight's kMDItemDisplayName

grep-based phrase search with Spotlight

Regex-based file/folder search with Spotlight

View processes, grep out by user and then kill all their PIDs

On This Page:

Joyent

CodeSnippets > grep

Related Tags

Never been to CodeSnippets before?

On This Page:

On This Page: