UsingADBtoDebugtheUNIX†Kernel RevisedJanuary,1983 SamuelJ.Leffler WilliamN.Joy ComputerSystemsResearchGroup DepartmentofElectricalEngineeringandComputerScience UniversityofCalifornia,Berkeley Berkeley,California94720 (415)642-7780 ABSTRACT Thisdocumentdescribestheuseofextensionsmadetothe4.1bsdreleaseof theVAX*UNIXdebuggeradbforthepurposeofdebuggingtheUNIXkernel.It discussesthechangesmadetoallowstandardadbcommandstofunctionproperly withthekernelandintroducesthebasicsnecessaryforuserstowriteadbcommandscriptswhichmaybeusedtoaugmentthestandardadbcommandset.The examinationtechniquesdescribedheremaybeappliedtorunningsystems,aswell asthepost-mortemdumpsautomaticallycreatedbythesavecore(8)program afterasystemcrash.Thereaderisexpectedtohaveatleastapassingfamiliarity withthedebuggercommandlanguage. †UNIXisaTrademarkofBellLaboratories. *DECandVAXaretrademarksofDigitalEquipmentCorporation. 6April1993 UsingADBontheUNIXKernel Introduction 1.INTRODUCTION ModificationshavebeenmadetothestandardVAXUNIXdebuggeradbtosimplifyexaminationofpost-mortemdumpsautomaticallygeneratedfollowingasystemcrash.Thesechanges mayalsobeusedwhenexaminingUNIXinitsnormaloperation.Thisdocumentservesasan introductiontotheuseofthesefacilities,andshouldnotbeconstruedasadescriptionofhowto debugthekernel. 0.1.Invocation WhenexaminingtheUNIXkernelanewoption,−−k,shouldbeused,e.g. adb−k/vmunix/dev/mem ThisflagcausesadbtopartiallysimulatetheVAXvirtualmemoryhardwarewhenaccessingthe corefile.InadditiontheinternalstatemaintainedbythedebuggerisinitializedfromdatastructuresmaintainedbytheUNIXkernelexplicitlyfordebugging‡.Apost-mortemdumpmaybe examinedinasimilarfashion, adb−kvmunix.?vmcore.? wheretheappropriateversionofthesavedoperatingsystemimageandcoredumparesuppliedin placeof‘‘?’’. 0.2.EstablishingContext Duringinitializationadbattemptstoestablishthecontextofthe‘‘currentlyactiveprocess’’ byexaminingthevalueofthekernelvariablemasterpaddr.Thisvariablecontainsthevirtual addressoftheprocesscontextblockofthelastprocesswhichwassetexecutingbytheSwtchroutine.Masterpaddrnormallyprovidessufficientinformationtolocatethecurrentstackframe(via thestackpointersfoundinthecontextblock).BylocatingtheVAXprocesscontextblockforthe process,adbmaythenperformvirtualtophysicaladdresstranslationusingthatprocess’sin-core pagetables. Whenexaminingpost-mortemdumpslocatingthemostrecentstackframeofthe‘‘currently activeprocess’’isnontrivial.ThisisduetothedifferentwaysinwhichtheVAXmaysavestate afteranonrecoverableerror.Crashesmayormaynotbe‘‘clean’’(i.e.thetopoftheinterrupt stackcontainstheprocess’skernelmodestackpointerandprogramcounter);an‘‘unclean’’crash willoccur,forinstance,iftheinterruptstackoverflows.Thus,onemustmanuallytryoneoftwo possibletechniquestogetastacktracefromapost-mortemdump.Ifthecrashwascleanthecur-rentstackpointerispresentintherestartparameterblock,atscb−4(orrpb+1fc),andthecom-mand *(scb−4)$c willgenerateastacktraceallthewayfromthekerneltothetopoftheuserprocess’sstack(e.g. tothemainroutineintheuserprocesswhichwasrunning).Otherwise,onemustscanthrough theinterruptstacklookingforthestackframe.Thisisusuallyindicatedbyazerolongwordentry (theprocedurecallhandler)followedbyalongwordentrywithbit29set(indicatingthecall framewasgeneratedasaresultofa‘‘calls’’instruction). intstack/X ‡Ifthe−kflagisnotusedwheninvokingadbtheusermustexplicitlycalculatevirtualaddresses. Withthe−−koptionadbinterpretspagetablestoautomaticallyperformvirtualtophysicaladdress translation. 6April1993 UsingADBontheUNIXKernel-2- Introduction Oncethestackpointerhasbeenlocated,thecommand .$c willgenerateastacktrace.Analternatemethodmaybeusedwhenatraceofaparticularprocessisrequired,seesection2.3. 6April1993 UsingADBontheUNIXKernel-3- CommandScripts 2.ADBCOMMANDSCRIPTS 2.1.ExtendingtheFormattingFacilities Oncetheprocesscontexthasbeenestablished,thecompleteadbcommandsetisavailable forinterpretingdatastructures.Inaddition,anumberofadbscriptshavebeencreatedtosimplifythestructuredprintingofcommonlyreferencedkerneldatastructures.Thescriptsnormally resideinthedirectory/usr/lib/adb,andareinvokedwiththe‘‘$<’’operator.(Alatertablelists the‘‘standard’’scripts.) Asanexample,considerthefollowinglistingwhichcontainsadumpofafaultyprocess’s state(ourtypingisshownemboldened). %adb−−kvmunix.17vmcore.17 sbr8001d064slrd9c p0br800efa00p0lr34p1br7f8efe00p1lr1ffff2 *(intstack−−4)$c boot()from80004025 boot(0,4)from80004025 panic(80021185)from800057e2 soreceive(8017478c,0)from80007c90 read()from800098d7 syscall()from8000b6e2 Xsyscall(3,7fffe834,258)from80000f64 ?()fromc1c ?()from26a ?(0,7fffef18,7fffef1c)from1d3 ?()from2f 800021185/s icpreg+99:receive u$<u u: u:kspusp 7fffff9c7fffe59c r0r1r2r3 155c00800237d4800418003 r4r5r6r7 001109080041800 r8r9r10r11 80021244c7fffe5b480000000 apfppcpsl 7fffffe87fffffa48000b784d80004 p0brp0lrp1brp1lr 800efa0040000347f8efe001ffff2 szptcmap2sswap 2940003070 sigc1sigc2sigc3 1af03fbfa007f0240cbc6c u+78:arg0arg1arg2 37fffe834258 u+8c:segflgerroruidgidruidrgidprocp 004a4a80041800 u+d4:uaprv1rv2ubase 6April1993 UsingADBontheUNIXKernel-4- CommandScripts 7ffff078017fffe834 countoffcdirrdir 2581508003cf000 u+f4:pathname .netrc dirpdinoentrypdir 31395.netrc0 7ffff11c:ofiles 800408188004081880040818800406b0 800406d4800406ec00 0000 0000 0000 ofileflg 00000000 00000000 0000 7ffff180:sigs 0360c1360c 000aae 0000 0000 0000 1000 0000 0000 codear0prbaseprsize 08000000000 7ffff248:proffprscaleosyssepttyp 0000800288b4 7ffff258:ttyminttymaj 00 7ffff25e:xmagxtsizxdsizxbsiz 3c00000010000000108c0000a680000 xssizentlocrelflg 006c720000 7ffff27e:directory ogin startacflgfpflgcmsktsizdsiz 11688012016000060000 7ffff2a2:ssiz 80000 80041800$<proc 80041800:linkrlinkaddr 800237d40800efde0 8004180c:upripricpustattimeniceslpcursig 0730730450302302400 80041814:sigsiga0siga1flag 6April1993 UsingADBontheUNIXKernel-5- CommandScripts 080002458001 80041824:uidpgrppidppidpoipszpttsize 4bbbcbb021e 80041834:dsizessizerssizemaxrss 166143fffff 80041844:swrssswaddrwchantextp 00080044ee0 80041854:clktimp0brxlinkticks 0800efa008004172022 80041864:%cpundxidhashpptr +5.1369253545999527e−021c880041720 80044ee0$<text 80044ee0:daddr 7e2000 0000 0000 ptdaddrsizecaddriptr 3521e800418008003cfa0 rssizeswrsscountccountflagslptimpoip 1a0020204200 Thecauseofthecrashwasa‘‘panic’’(seethestacktrace)duetothe0argumentpassedthe soreceiveroutine.Themajorityofthedumpwasdonetoillustratetheuseoftwocommand scriptsusedtoformatkerneldatastructures.The‘‘u’’script,invokedbythecommand‘‘u$<u’’,is alengthyseriesofcommandswhichpretty-printstheuservector.Likewise,‘‘proc’’and‘‘text’’are scriptsusedtoformattheobviousdatastructures.Let’squicklyexaminethe‘‘text’’script(the scripthasbeenbrokenintoanumberoflinesforconveniencehere;inactualityitisasinglelineof text). ./"daddr"n12Xn\ "ptdaddr"16t"size"16t"caddr"16t"iptr"n4Xn\ "rssize"8t"swrss"8t"count"8t"ccount"8t"flag"8t"slptim"8t"poip"n2x4bx++n Thefirstlineproducesthelistofdiskblockaddressesassociatedwithaswappedouttextsegment.The‘‘n’’formatforcesanew-linecharacter,with12hexadecimalintegersprintedimmedi-atelyafter.Likewise,theremainingtwolinesofthecommandformattheremainderofthetext structure.Theexpression‘‘16t’’causesadbtotabtothenextcolumnwhichisamultipleof16. Thelasttwoplusoperatorsarepresenttoround‘‘.’’totheendofthetextstructure.Thisallows theusertoreinvoketheformatonconsecutivetextstructureswithouthavingtobeconcerned aboutproperalignmentof‘‘.’’. Themajorityofthescriptsprovidedareofthisnature.Whenpossible,theformatting scriptsprintadatastructurewithasingleformattoallowsubsequentreusewheninterrogating arraysofstructures.Thatis,thepreviousscriptcouldhavebeenwritten ./"daddr"n12Xn +/"ptdaddr"16t"size"16t"caddr"16t"iptr"n4Xn +/"rssize"8t"swrss"8t"count"8t"ccount"8t"flag"8t"slptim"8t"poip"n2x4bx++n butthenreuseoftheformatwouldhaveinvokedonlythelastlineoftheformat. 6April1993 UsingADBontheUNIXKernel-6- CommandScripts 2.2.TraversingDataStructures Theadbcommandlanguagecanbeusedtotraversecomplexdatastructures.Onesuch datastructure,alinkedlist,occursquiteofteninthekernel.Byusingadbvariablesandthenormalexpressionoperatorsitisasimplemattertoconstructascriptwhichchainsdownthelist printingeachelementalongtheway. Forinstance,thequeueofprocessesawaitingtimerevents,thecalloutqueue,isprintedwith thefollowingtwoscripts: callout: calltodo/"time"16t"arg"16t"func"12+ *+$<callout.next callout.next: ./Dpp *+>l ,#<l$< <l$<callout.next Thefirstlineofthescriptcalloutstartsthetraversalattheglobalsymbolcalltodoand printsasetofheadings.Itthenskipstheemptyportionofthestructureusedastheheadofthe queue.Thesecondlinetheninvokesthescriptcallout.nextmoving‘‘.’’tothetopofthequeue (‘‘*+’’performstheindirectionthroughthelinkentryofthestructureattheheadofthequeue). callout.nextprintsvaluesforeachcolumn,thenperformsaconditionaltestonthelinkto thenextentry.Thistestisperformedasfollows, *+>lPlacethevalueofthe‘‘link’’intheadbvariable‘‘<l’’. ,#<l$<Ifthevaluestoredin‘‘<l’’isnon-zero,thenthecurrentinputstream(i.e.thescript callout.next)isterminated.Otherwise,theexpression‘‘#<l’’willbezero,andthe ‘‘$<’’willbeignored.Thatis,thecombinationofthelogicalnegationoperator‘‘#’’, adbvariable‘‘<l’’,and‘‘$<’’operatorcreatesastatementoftheform, if(!link)exit; Theremaininglineofcallout.nextsimplyreappliesthescriptonthenextelementin thelinkedlist. Asamplecalloutdumpisshownbelow. 6April1993 UsingADBontheUNIXKernel-7- CommandScripts %adb−−k/vmunix/dev/mem sbr8001f864slrd9c p0br800efa00p0lr8ep1br7f8efe00p1lr1ffff2 $<callout calltodo: calltodo:timeargfunc 8004ecfc:260dzscan 8004ed0c:80upwatch 8004ed1c:00iptimeo 8004ed5c:00tcptimeo 8004ed6c:00rkwatch 8004ecfc:520dzscan 8004ed2c:68Syssize+70tmtimer 8004ed3c:29200memenable 2.3.SupplyingParameters Ifoneisclever,acommandscriptmayusetheaddressandcountportionsofanadbcommandasparameters.Anexampleofthisisthesetprocscriptusedtoswitchtothecontextofa processwithaknownprocess-id; 0t99$<setproc Thebodyofsetprocis .>4 *nproc>l *proc>f $<setproc.nxt whilesetproc.nxtis (*(<f+28))&0xffff="pid"X ,#((*(<f+28)&0xffff)-<4)$<setproc.done <l-1>l <f+70>f ,#<l$< $<setproc.nxt Theprocess-id,suppliedastheparameter,isstoredinthevariable‘‘<4’’,thenumberofprocesses isplacedin‘‘<l’’,andthebaseofthearrayofprocessstructuresin‘‘<f’’.setproc.nxtthenperformsalinearsearchthroughthearrayuntilitmatchestheprocess-idrequested,oruntilitruns outofprocessstructurestocheck.Thescriptsetproc.donesimplyestablishesthecontextofthe process,thenexits. 2.4.StandardScripts Thefollowingtablesummarizesthecommandscriptscurrentlyavailableinthedirectory /usr/lib/adb. 6April1993 UsingADBontheUNIXKernel-8- CommandScripts StandardCommandScripts NameUseDescription bufaddr$<bufformatblockI/Obuffer callout$<calloutprinttimerqueue clistaddr$<clistformatcharacterI/Olinkedlist dinoaddr$<dinoformatdirectoryinode diraddr$<dirformatdirectoryentry dirblkaddr$<dirblkscandirectoryentries fileaddr$<fileformatopenfilestructure fsaddr$<filsysformatin-coresuperblockstructure findprocpid$<findprocfindprocessbyprocessid hostsaddr$<hostsformatIMPhosttableentries hosttableaddr$<hosttableshowallIMPhosttableentries ifnetaddr$<ifnetformatnetworkinterfacestructure ifubaaddr$<ifubaformatUNIBUSresourcestructure inodeaddr$<inodeformatin-coreinodestructure inpcbaddr$<inpcbformatinternetprotocolcontrolblock iovecaddr$<iovecformatalistofiovstructures ipreassaddr$<ipreassformatanipreassemblyqueue mactaddr$<mactshow‘‘active’’listofmbuf’s mbstat$<mbstatshowmbufstatistics mbufaddr$<mbufshow‘‘next’’listofmbuf’s mbufsaddr$<mbufsshowanumberofmbuf’s mountaddr$<mountformatmountstructure pcbaddr$<pcbformatprocesscontextblock procaddr$<procformatprocesstableentry rawcbaddr$<rawcbformatarawprotocolcontrolblock rtentryaddr$<rtentryformataroutingtableentry setprocpid$<setprocswitchprocesscontexttopid socketaddr$<socketformatsocketstructure tcpcbaddr$<tcpcbformatTCPcontrolblock tcpipaddr$<tcpipformataTCP/IPpacketheader tcpreassaddr$<tcpreassshowaTCPreassemblyqueue textaddr$<textformattextstructure traceall$<traceallshowstacktraceforallprocesses ttyaddr$<ttyformatttystructure uaddr$<uformatuservector,includingpcb ubahdaddr$<ubahdformataUNIBUSheaderstructure 6April1993 UsingADBontheUNIXKernel-9- Summary 3.SUMMARY TheextensionsmadetoadbprovidebasicsupportfordebuggingtheUNIXkernelbyeliminatingtheneedforausertocarryoutvirtualtophysicaladdresstranslation.Acollectionof scriptshavebeenwrittentonicelyformatthemajorkerneldatastructuresandaidinswitching betweenprocesscontexts.Thishasbeencarriedoutwithonlyminimalchangestothedebugger. Moreworkisneededtoprovideenoughinformationforthedebuggertoautomaticallyestablishcontextafterasystemcrash.Thesystemcurrentlydoesnotalwayssaveenoughstateto allowthedebuggertoreliablylocatethestackframejustpriortoanexception. Moreworkisalsorequiredontheuserinterfacetoadb.Itappearstheinscrutableadbcommandlanguagehaslimitedwidespreaduseofmuchofthepowerofadb.Onepossibilityistoprovideamorecomprehensible‘‘adbfrontend’’,justasbc(1)isusedtofrontenddc(1). Finally,adbcouldbesignificantlyimprovedifitwereknowledgeableaboutaprogram’sdata structures.ThiswouldeliminatetheuseofnumericoffsetsintoCstructures. 6April1993