Awk—APatternScanningandProcessingLanguage (SecondEdition) AlfredV.Aho BrianW.Kernighan PeterJ.Weinberger ABSTRACT Awkisaprogramminglanguagewhosebasicoperationistosearchasetof filesforpatterns,andtoperformspecifiedactionsuponlinesorfieldsoflines whichcontaininstancesofthosepatterns.Awkmakescertaindataselectionand transformationoperationseasytoexpress;forexample,theawkprogram length>72 printsallinputlineswhoselengthexceeds72characters;theprogram NF%2==0 printsalllineswithanevennumberoffields;andtheprogram {$1=log($1);print} replacesthefirstfieldofeachlinebyitslogarithm. Awkpatternsmayincludearbitrarybooleancombinationsofregularexpressionsandofrelationaloperatorsonstrings,numbers,fields,variables,andarray elements.Actionsmayincludethesamepattern-matchingconstructionsasin patterns,aswellasarithmeticandstringexpressionsandassignments,if-else, while,forstatements,andmultipleoutputstreams. Thisreportcontainsauser’sguide,adiscussionofthedesignandimplementationofawk,andsometimingstatistics. September1,1978 Awk—APatternScanningandProcessingLanguage (SecondEdition) AlfredV.Aho BrianW.Kernighan PeterJ.Weinberger 1.Introduction Awkisaprogramminglanguagedesignedto makemanycommoninformationretrievalandtext manipulationtaskseasytostateandtoperform. Thebasicoperationofawkistoscanaset ofinputlinesinorder,searchingforlineswhich matchanyofasetofpatternswhichtheuserhas specified.Foreachpattern,anactioncanbespecified;thisactionwillbeperformedoneachline thatmatchesthepattern. ReadersfamiliarwiththeUNIX◯rprogram grep1willrecognizetheapproach,althoughinawk thepatternsmaybemoregeneralthaningrep, andtheactionsallowedaremoreinvolvedthan merelyprintingthematchingline.Forexample, theawkprogram {print$3,$2} printsthethirdandsecondcolumnsofatablein thatorder.Theprogram $2∼/A|B|C/ printsallinputlineswithanA,B,orCinthesecondfield.Theprogram $1!=prev{print;prev=$1} printsalllinesinwhichthefirstfieldisdifferent fromthepreviousfirstfield. 1.1.Usage Thecommand awkprogram[files] executestheawkcommandsinthestringprogramonthesetofnamedfiles,oronthestandard inputiftherearenofiles.Thestatementscanalso beplacedinafilepfile,andexecutedbythecommand awk−fpfile[files] 1.2.ProgramStructure Anawkprogramisasequenceofstatements oftheform: pattern{action} pattern{action} ... Eachlineofinputismatchedagainsteachofthe patternsinturn.Foreachpatternthatmatches, theassociatedactionisexecuted.Whenallthe patternshavebeentested,thenextlineisfetched andthematchingstartsover. Eitherthepatternortheactionmaybeleft out,butnotboth.Ifthereisnoactionforapattern,thematchinglineissimplycopiedtotheoutput.(Thusalinewhichmatchesseveralpatterns canbeprintedseveraltimes.)Ifthereisnopatternforanaction,thentheactionisperformedfor everyinputline.Alinewhichmatchesnopattern isignored. Sincepatternsandactionsareboth optional,actionsmustbeenclosedinbracestodistinguishthemfrompatterns. 1.3.RecordsandFields Awkinputisdividedinto‘‘records’’terminatedbyarecordseparator.Thedefaultrecord separatorisanewline,sobydefaultawkprocesses itsinputalineatatime.ThenumberofthecurrentrecordisavailableinavariablenamedNR. Eachinputrecordisconsideredtobe dividedinto‘‘fields.’’Fieldsarenormallyseparatedbywhitespace—blanksortabs—butthe inputfieldseparatormaybechanged,asdescribed below.Fieldsarereferredtoas$1,$2,andso forth,where$1isthefirstfield,and$0isthe wholeinputrecorditself.Fieldsmaybeassigned to.Thenumberoffieldsinthecurrentrecordis availableinavariablenamedNF. ThevariablesFSandRSrefertotheinput fieldandrecordseparators;theymaybechanged atanytimetoanysinglecharacter.Theoptional command-lineargument−Fcmayalsobeusedto setFStothecharacterc. Iftherecordseparatorisempty,anempty inputlineistakenastherecordseparator,and -2-blanks,tabsandnewlinesaretreatedasfieldsepa-rators. ThevariableFILENAMEcontainsthe nameofthecurrentinputfile. 1.4.Printing Anactionmayhavenopattern,inwhich casetheactionisexecutedforalllines.Thesimplestactionistoprintsomeorallofarecord;this isaccomplishedbytheawkcommandprint.The awkprogram {print} printseachrecord,thuscopyingtheinputtothe outputintact.Moreusefulistoprintafieldor fieldsfromeachrecord.Forinstance, print$2,$1 printsthefirsttwofieldsinreverseorder.Items separatedbyacommaintheprintstatementwill beseparatedbythecurrentoutputfieldseparator whenoutput.Itemsnotseparatedbycommaswill beconcatenated,so print$1$2 runsthefirstandsecondfieldstogether. ThepredefinedvariablesNFandNRcan beused;forexample {printNR,NF,$0} printseachrecordprecededbytherecordnumber andthenumberoffields. Outputmaybedivertedtomultiplefiles; theprogram {print$1>"foo1";print$2>"foo2"} writesthefirstfield,$1,onthefilefoo1,andthe secondfieldonfilefoo2.The>>notationcan alsobeused: print$1>>"foo" appendstheoutputtothefilefoo.(Ineachcase, theoutputfilesarecreatedifnecessary.)Thefile namecanbeavariableorafieldaswellasaconstant;forexample, print$1>$2 usesthecontentsoffield2asafilename. Naturallythereisalimitonthenumberof outputfiles;currentlyitis10. Similarly,outputcanbepipedintoanother process(onUNIXonly);forinstance, print|"mailbwk" mailstheoutputtobwk.

ThevariablesOFSandORSmaybeused tochangethecurrentoutputfieldseparatorand outputrecordseparator.Theoutputrecordseparatorisappendedtotheoutputoftheprintstatement. Awkalsoprovidestheprintfstatementfor outputformatting: printfformatexpr,expr,... formatstheexpressionsinthelistaccordingtothe specificationinformatandprintsthem.For example, printf"%8.2f%10ld\n",$1,$2 prints$1asafloatingpointnumber8digitswide, withtwoafterthedecimalpoint,and$2asa 10-digitlongdecimalnumber,followedbyanew-line.Nooutputseparatorsareproducedautomati-cally;youmustaddthemyourself,asinthisexam-ple.Theversionofprintfisidenticaltothatused withC.2 2.Patterns Apatterninfrontofanactionactsasa selectorthatdetermineswhethertheactionisto beexecuted.Avarietyofexpressionsmaybeused aspatterns:regularexpressions,arithmeticrelationalexpressions,string-valuedexpressions,and arbitrarybooleancombinationsofthese. 2.1.BEGINandEND ThespecialpatternBEGINmatchesthe beginningoftheinput,beforethefirstrecordis read.ThepatternENDmatchestheendofthe input,afterthelastrecordhasbeenprocessed. BEGINandENDthusprovideawaytogain controlbeforeandafterprocessing,forinitializationandwrapup. Asanexample,thefieldseparatorcanbe settoacolonby BEGIN{FS=":"} ...restofprogram... Ortheinputlinesmaybecountedby END{printNR} IfBEGINispresent,itmustbethefirstpattern; ENDmustbethelastifused. 2.2.RegularExpressions Thesimplestregularexpressionisaliteral stringofcharactersenclosedinslashes,like /smith/ Thisisactuallyacompleteawkprogramwhich willprintalllineswhichcontainanyoccurrenceof thename‘‘smith’’.Ifalinecontains‘‘smith’’as -3-partofalargerword,itwillalsobeprinted,asin blacksmithing Awkregularexpressionsincludetheregular expressionformsfoundintheUNIXtexteditor ed3andgrep(withoutback-referencing).Inaddi-tion,awkallowsparenthesesforgrouping,|for alternatives,+for‘‘oneormore’’,and?for‘‘zero orone’’,allasinlex.Characterclassesmaybe abbreviated:[a−zA−Z0−9]isthesetofallletters anddigits.Asanexample,theawkprogram /[Aa]ho|[Ww]einberger|[Kk]ernighan/ willprintalllineswhichcontainanyofthenames ‘‘Aho,’’‘‘Weinberger’’or‘‘Kernighan,’’whether capitalizedornot. Regularexpressions(withtheextensions listedabove)mustbeenclosedinslashes,justasin edandsed.Withinaregularexpression,blanks andtheregularexpressionmetacharactersaresignificant.Toturnofthemagicmeaningofoneof theregularexpressioncharacters,precedeitwitha backslash.Anexampleisthepattern /\/.∗\// whichmatchesanystringofcharactersenclosedin slashes. Onecanalsospecifythatanyfieldorvariablematchesaregularexpression(ordoesnot matchit)withtheoperators∼and!∼.Theprogram $1∼/[jJ]ohn/ printsalllineswherethefirstfieldmatches‘‘john’’ or‘‘John.’’Noticethatthiswillalsomatch‘‘Johnson’’,‘‘St.Johnsbury’’,andsoon.Torestrictitto exactly[jJ]ohn,use $1∼/ˆ [jJ]ohn$/ Thecaretˆ referstothebeginningofalineor field;thedollarsign$referstotheend. 2.3.RelationalExpressions Anawkpatterncanbearelationalexpressioninvolvingtheusualrelationaloperators<, <=,==,!=,>=,and>.Anexampleis $2>$1+100 whichselectslineswherethesecondfieldisat least100greaterthanthefirstfield.Similarly, NF%2==0 printslineswithanevennumberoffields. Inrelationaltests,ifneitheroperandis numeric,astringcomparisonismade;otherwiseit isnumeric.Thus, $1>="s" selectslinesthatbeginwithans,t,u,etc.Inthe absenceofanyotherinformation,fieldsaretreated asstrings,sotheprogram $1>$2 willperformastringcomparison. 2.4.CombinationsofPatterns Apatterncanbeanybooleancombination ofpatterns,usingtheoperators||(or),&&(and), and!(not).Forexample, $1>="s"&&$1<"t"&&$1!="smith" selectslineswherethefirstfieldbeginswith‘‘s’’, butisnot‘‘smith’’.&&and||guaranteethat theiroperandswillbeevaluatedfromlefttoright; evaluationstopsassoonasthetruthorfalsehood isdetermined. 2.5.PatternRanges The‘‘pattern’’thatselectsanactionmay alsoconsistoftwopatternsseparatedbyacomma, asin pat1,pat2{...} Inthiscase,theactionisperformedforeachline betweenanoccurrenceofpat1andthenext occurrenceofpat2(inclusive).Forexample, /start/,/stop/ printsalllinesbetweenstartandstop,while NR==100,NR==200{...} doestheactionforlines100through200ofthe input. 3.Actions Anawkactionisasequenceofactionstatementsterminatedbynewlinesorsemicolons. Theseactionstatementscanbeusedtodoavarietyofbookkeepingandstringmanipulatingtasks. 3.1.Built-inFunctions Awkprovidesa‘‘length’’functiontocomputethelengthofastringofcharacters.Thisprogramprintseachrecord,precededbyitslength: {printlength,$0} lengthbyitselfisa‘‘pseudo-variable’’which yieldsthelengthofthecurrentrecord; length(argument)isafunctionwhichyieldsthe lengthofitsargument,asintheequivalent {printlength($0),$0} Theargumentmaybeanyexpression.

-4-Awkalsoprovidesthearithmeticfunctions sqrt,log,exp,andint,forsquareroot,basee logarithm,exponential,andintegerpartoftheir respectivearguments. Thenameofoneofthesebuilt-infunctions, withoutargumentorparentheses,standsforthe valueofthefunctiononthewholerecord.The program length<10||length>20 printslineswhoselengthislessthan10orgreater than20. Thefunctionsubstr(s,m,n)producesthe substringofsthatbeginsatpositionm(origin1) andisatmostncharacterslong.Ifnisomitted, thesubstringgoestotheendofs.Thefunction index(s1,s2)returnsthepositionwherethe strings2occursins1,orzeroifitdoesnot. Thefunctionsprintf(f,e1,e2,...)producesthevalueoftheexpressionse1,e2,etc.,in theprintfformatspecifiedbyf.Thus,forexample, x=sprintf("%8.2f%10ld",$1,$2) setsxtothestringproducedbyformattingthe valuesof$1and$2. 3.2.Variables,Expressions,andAssignments Awkvariablestakeonnumeric(floating point)orstringvaluesaccordingtocontext.For example,in x=1 xisclearlyanumber,whilein x="smith" itisclearlyastring.Stringsareconvertedto numbersandviceversawhenevercontextdemands it.Forinstance, x="3"+"4" assigns7tox.Stringswhichcannotbeinterpretedasnumbersinanumericalcontextwillgenerallyhavenumericvaluezero,butitisunwiseto countonthisbehavior. Bydefault,variables(otherthanbuilt-ins) areinitializedtothenullstring,whichhasnumericalvaluezero;thiseliminatestheneedformost BEGINsections.Forexample,thesumsofthe firsttwofieldscanbecomputedby {s1+=$1;s2+=$2} END{prints1,s2} Arithmeticisdoneinternallyinfloating point.Thearithmeticoperatorsare+,−,∗,/, and%(mod).TheCincrement++and decrement−−operatorsarealsoavailable,andso aretheassignmentoperators+=,−=,∗=,/=, and%=.Theseoperatorsmayallbeusedin expressions. 3.3.FieldVariables Fieldsinawkshareessentiallyallofthe propertiesofvariables—theymaybeusedin arithmeticorstringoperations,andmaybe assignedto.Thusonecanreplacethefirstfield withasequencenumberlikethis: {$1=NR;print} oraccumulatetwofieldsintoathird,likethis: {$1=$2+$3;print$0} orassignastringtoafield: {if($3>1000) $3="toobig" print } whichreplacesthethirdfieldby‘‘toobig’’whenit is,andinanycaseprintstherecord. Fieldreferencesmaybenumericalexpressions,asin {print$i,$(i+1),$(i+n)} Whetherafieldisdeemednumericorstring dependsoncontext;inambiguouscaseslike if($1==$2)... fieldsaretreatedasstrings. Eachinputlineissplitintofieldsautomaticallyasnecessary.Itisalsopossibletosplitany variableorstringintofields: n=split(s,array,sep) splitsthethestringsintoarray[1],...,array[n]. Thenumberofelementsfoundisreturned.Ifthe separgumentisprovided,itisusedasthefield separator;otherwiseFSisusedastheseparator. 3.4.StringConcatenation Stringsmaybeconcatenated.Forexample length($1$2$3) returnsthelengthofthefirstthreefields.Orina printstatement, print$1"is"$2 printsthetwofieldsseparatedby‘‘is’’.Variables andnumericexpressionsmayalsoappearinconcatenations.

-5-3.5.Arrays Arrayelementsarenotdeclared;theyspring intoexistencebybeingmentioned.Subscripts mayhaveanynon-nullvalue,includingnon-numericstrings.Asanexampleofaconventional numericsubscript,thestatement x[NR]=$0 assignsthecurrentinputrecordtotheNR-thele-mentofthearrayx.Infact,itispossibleinprin-ciple(thoughperhapsslow)toprocesstheentire inputinarandomorderwiththeawkprogram {x[NR]=$0} END{...program...} Thefirstactionmerelyrecordseachinputlinein thearrayx. Arrayelementsmaybenamedbynonnumericvalues,whichgivesawkacapability ratherliketheassociativememoryofSnobol tables.Supposetheinputcontainsfieldswithvalueslikeapple,orange,etc.Thentheprogram /apple/{x["apple"]++} /orange/{x["orange"]++} END{printx["apple"],x["orange"]} incrementscountsforthenamedarrayelements, andprintsthemattheendoftheinput. 3.6.Flow-of-ControlStatements Awkprovidesthebasicflow-of-controlstate-mentsif-else,while,for,andstatementgrouping withbraces,asinC.Weshowedtheifstatement insection3.3withoutdescribingit.Thecondition inparenthesesisevaluated;ifitistrue,thestatementfollowingtheifisdone.Theelsepartis optional. Thewhilestatementisexactlylikethatof C.Forexample,toprintallinputfieldsoneper line, i=1 while(i<=NF){ print$i ++i } TheforstatementisalsoexactlythatofC: for(i=1;i<=NF;i++) print$i doesthesamejobasthewhilestatementabove. Thereisanalternateformoftheforstatementwhichissuitedforaccessingtheelementsof anassociativearray: for(iinarray) statement doesstatementwithisetinturntoeachelement ofarray.Theelementsareaccessedinanapparentlyrandomorder.Chaoswillensueifiis altered,orifanynewelementsareaccessedduring theloop. Theexpressionintheconditionpartofan if,whileorforcanincluderelationaloperators like<,<=,>,>=,==(‘‘isequalto’’),and!= (‘‘notequalto’’);regularexpressionmatcheswith thematchoperators∼and!∼;thelogicaloperators||,&&,and!;andofcourseparenthesesfor grouping. Thebreakstatementcausesanimmediate exitfromanenclosingwhileorfor;thecontinue statementcausesthenextiterationtobegin. Thestatementnextcausesawktoskip immediatelytothenextrecordandbeginscanning thepatternsfromthetop.Thestatementexit causestheprogramtobehaveasiftheendofthe inputhadoccurred. Commentsmaybeplacedinawkprograms: theybeginwiththecharacter#andendwiththe endoftheline,asin printx,y#thisisacomment 4.Design TheUNIXsystemalreadyprovidesseveral programsthatoperatebypassinginputthrougha selectionmechanism.Grep,thefirstandsimplest, merelyprintsalllineswhichmatchasinglespecifiedpattern.Egrepprovidesmoregeneralpatterns,i.e.,regularexpressionsinfullgenerality; fgrepsearchesforasetofkeywordswithaparticularlyfastalgorithm.Sed4providesmostofthe editingfacilitiesoftheeditored,appliedtoa streamofinput.Noneoftheseprogramsprovides numericcapabilities,logicalrelations,orvariables. Lex5providesgeneralregularexpression recognitioncapabilities,and,byservingasaC programgenerator,isessentiallyopen-endedinits capabilities.Theuseoflex,however,requiresa knowledgeofCprogramming,andalexprogram mustbecompiledandloadedbeforeuse,which discouragesitsuseforone-shotapplications. Awkisanattempttofillinanotherpartof thematrixofpossibilities.Itprovidesgeneralregularexpressioncapabilitiesandanimplicit input/outputloop.Butitalsoprovidesconvenient numericprocessing,variables,moregeneralselection,andcontrolflowintheactions.Itdoesnot requirecompilationoraknowledgeofC.Finally, awkprovidesaconvenientwaytoaccessfields withinlines;itisuniqueinthisrespect.

-6-Awkalsotriestointegratestringsandnum-berscompletely,bytreatingallquantitiesasboth stringandnumeric,decidingwhichrepresentation isappropriateaslateaspossible.Inmostcases theusercansimplyignorethedifferences. Mostoftheeffortindevelopingawkwent intodecidingwhatawkshouldorshouldnotdo (forinstance,itdoesn’tdostringsubstitution)and whatthesyntaxshouldbe(noexplicitoperator forconcatenation)ratherthanonwritingor debuggingthecode.Wehavetriedtomakethe syntaxpowerfulbuteasytouseandwelladapted toscanningfiles.Forexample,theabsenceofdeclarationsandimplicitinitializations,whileprobablyabadideaforageneral-purposeprogramming language,isdesirableinalanguagethatismeant tobeusedfortinyprogramsthatmayevenbe composedonthecommandline. Inpractice,awkusageseemstofallintotwo broadcategories.Oneiswhatmightbecalled ‘‘reportgeneration’’—processinganinputto extractcounts,sums,sub-totals,etc.Thisalso includesthewritingoftrivialdatavalidationprograms,suchasverifyingthatafieldcontainsonly numericinformationorthatcertaindelimitersare properlybalanced.Thecombinationoftextual andnumericprocessingisinvaluablehere. Asecondareaofuseisasadatatransformer,convertingdatafromtheformproduced byoneprogramintothatexpectedbyanother. Thesimplestexamplesmerelyselectfields,perhapswithrearrangements. 5.Implementation Theactualimplementationofawkusesthe languagedevelopmenttoolsavailableontheUNIX operatingsystem.Thegrammarisspecifiedwith yacc;6thelexicalanalysisisdonebylex;theregularexpressionrecognizersaredeterministicfinite automataconstructeddirectlyfromtheexpressions.Anawkprogramistranslatedintoaparse treewhichisthendirectlyexecutedbyasimple interpreter. Awkwasdesignedforeaseofuserather thanprocessingspeed;thedelayedevaluationof variabletypesandthenecessitytobreakinput intofieldsmakeshighspeeddifficulttoachievein anycase.Nonetheless,theprogramhasnot proventobeunworkablyslow. TableIbelowshowstheexecution(user+ system)timeonaPDP-11/70oftheUNIXpro-gramswc,grep,egrep,fgrep,sed,lex,andawk onthefollowingsimpletasks: 1.countthenumberoflines. 2.printalllinescontaining‘‘doug’’.

3.printalllinescontaining‘‘doug’’,‘‘ken’’or ‘‘dmr’’. 4.printthethirdfieldofeachline. 5.printthethirdandsecondfieldsofeachline, inthatorder. 6.appendalllinescontaining‘‘doug’’,‘‘ken’’, and‘‘dmr’’tofiles‘‘jdoug’’,‘‘jken’’,and ‘‘jdmr’’,respectively. 7.printeachlineprefixedby‘‘line-number:’’. 8.sumthefourthcolumnofatable. Theprogramwcmerelycountswords,linesand charactersinitsinput;wehavealreadymentioned theothers.Inallcasestheinputwasafilecontaining10,000linesascreatedbythecommandls −l;eachlinehastheform −rw−rw−rw−1ava123Oct1517:05xxx Thetotallengthofthisinputis452,960characters.Timesforlexdonotincludecompileorload. Asmightbeexpected,awkisnotasfastas thespecializedtoolswc,sed,ortheprogramsin thegrepfamily,butisfasterthanthemoregeneraltoollex.Inallcases,thetaskswereaboutas easytoexpressasawkprogramsasprogramsin theseotherlanguages;tasksinvolvingfieldswere considerablyeasiertoexpressasawkprograms. Someofthetestprogramsareshowninawk,sed andlex. References 1. 2. 3. 4. 5. 6.

-7-Task Program12345678 wc8.6 grep11.713.1 egrep6.211.511.6 fgrep7.713.816.1 sed10.211.615.829.030.516.1 lex65.1150.1144.267.770.3104.081.792.8 awk15.025.629.933.338.946.471.431.1 TableI.ExecutionTimesofPrograms.(Timesareinsec.) Theprogramsforsomeofthesejobsare shownbelow.Thelexprogramsaregenerallytoo longtoshow. AWK: 1.END{printNR} 2./doug/ 3./ken|doug|dmr/ 4.{print$3} 5.{print$3,$2} 6./ken/{print>"jken"} /doug/{print>"jdoug"} /dmr/{print>"jdmr"} 7.{printNR":"$0} 8.{sum=sum+$4} END{printsum} SED: 1.$= 2./doug/p 3./doug/p /doug/d /ken/p /ken/d /dmr/p /dmr/d 4./[ˆ ]∗[]∗[ˆ ]∗[]∗\([ˆ ]∗\).∗/s//\1/p 5./[ˆ ]∗[]∗\([ˆ ]∗\)[]∗\([ˆ ]∗\).∗/s//\2\1/p 6./ken/wjken /doug/wjdoug /dmr/wjdmr LEX: 1.%{ inti; %} %% \ni++; .; %% yywrap(){ printf("%d\n",i); } 2.%% ˆ .∗doug.∗$printf("%s\n",yytext); .; \n;