TheBerkeleyNetwork−ARetrospective EricSchmidt ComputerScienceDivision DepartmentofElectricalEngineeringandComputerScience UniversityofCalifornia,Berkeley Berkeley,California94720 ABSTRACT TheBerkeleyNetworkconnectsanumberofUNIX◯rmachinesontheBerkeleycampus.Itprovidesfacilitiesforfiletransfer,sendingandreadingmail,and remoteprinting.Operatinginabatchmode,networkrequestsaretransferred onebyonethroughaninter-connectednetworkuntiltheyreachtheirfinaldesti-nation. Thisdocumentdescribesthehistoryandgoalsofthisproject,thedesign decisionsfaced,discussesissuesinportablesoftwaredevelopmentinnetworks, anddiscussesthefutureofthisproject. Introduction Thisdocumentisintendedforreaderswithaninterestinnetworkingwhoarefamiliarwith twootherdocumentsaboutthenetwork,‘‘AnIntroductiontotheBerkeleyNetwork’’,andthe ‘‘NetworkSystemManual’’,bythisauthor.Itisnotnecessarytoreadthisdocumenttosetup andmaintainthenetwork,althoughsystemspersonswillbenefitiftheyarefamiliarwiththeconceptspresentedhere. Thesectionsarepresentedasfollows: Principals History OverallSystemDescription ProtocolExplanation Portability Security FuturePlans Summary Themostimportantsectionisthelast,whichdetailsasetofprinciplestheauthorhas learnedduringthisproject. Principals Thisprojectwasacollaborationofmanyindividuals.Dr.RobertS.Fabryparticipatedin theinitialdesignandhasexertedthestrongestinfluenceonpolishingthefinalproduct.BillJoy andEdGouldprovidedvaluabletechnicalexpertiseeverystepoftheway,primarilyaboutdevelopingsystemsprograms.ThesupportstaffoftheEECSDepartmentandtheComputerCenter (BobKridle,JeffSchreibman,VanceVaughan,RobynAllsman,andRickiBlau)wereinvolvedin settingupandadministeringthismulti-domainproject.Thelowest-levelconceptstheauthor -2-usedcamefromexperienceatXEROXPARC,primarilyfromdiscussionswithDavidBoggs,oneof theETHERNET†designers. History Thenetworkprojectcanbedividedintotwodistinctphases.Thefirst,fromJanuary1978 toMay1978(41/2months)involveddesigningandimplementingthenetworkfacilitiesnowavailable.Thesecond,fromOctober1978toMarch1979(5months),sawtheadditionofmanymore machinestothenetworkwithemphasisonportability,security,andminordesignchanges.The networkhasbeeninalmostcontinuousservicetouserssinceMay1,1978. FirstPhase AninitialdesignwasworkedoutwithDr.FabryinJanuary1978.Asuitableconnection wasmadebetweentheComputerCenter‘‘A’’and‘‘Q’’machines(thencalled‘‘D’’). A Q Developmentproceededontwofronts−asetofdaemonswerewrittentotransferfiles, usingUNIXpipesfordebugging.Thelowest-levelprotocolsweredesignedandimplementedofthe terminal-typeconnectionbetweenAandQ.Theseweredebuggedusingsimpleprogramstosend andreceivepackets,andapairofprogramstotransferafilefromonetotheother.Thiswasthe firstexperiencewithadistributedsoftwaredevelopment−theQmachineisaDECPDP-11/34 withnolineprinter,anon-standardtapedrive,andterminalaccessonlybytelephone,somost developmenthadtoproceedontheAmachine. Duringthisphasethegoalsoftheprojectincreasedinscope.Theimplementorandonly userhadtousethenetworktotransfernetworksourceandworkedoutsimplewaystoautomate this(the‘‘.netrc’’fileisanexample). Whenitappearedusablebymorethantheimplementor,theconnectionwaschangedtobe betweenAandtheComputerCenter‘‘C’’machine: A C Upuntilnow,thenetworkhadrequiredanaccountonbothmachines.Itbecameclearthis wasahandicapsincetheAmachinehadtoomanyaccountsandthepasswordfilewasimmense. Certain‘‘free’’commandswereallowed,withoutanaccount. TheCorymachinewassoonaddedtothenetwork: Cory A C †‘‘Ethernet:DistributedPacketSwitchingforLocalComputerNetworks’’,byRobertMetcalfeand DavidBoggs,CACM,July1976.

-3-Thisproducedmajorchangesinthedesign−initiallywehadassumednetworkuserswould haveaccountsonallmachines.Thiswasunreasonablystrictandasolution(kludge)wasworked outwherearequestwasexaminedandforwardedthroughthequeue(s)ontheintermediate machine.Thenetworkbecametable-driventoaccomplishtherouting,anddistributedsoftware developmentbecamemoredifficultbecauseoftheincreasednumberofmachines.Theimplementorquicklydiscoveredonlyonesolution:Alwayshaveadesignatedsourcemachineforallchanges. Tothisday,softwarechangesaremadeonlyonthis‘‘source’’machine,theothersareguaranteed tohavecopies.Thismakesremoteupdating(copyingnewversionsaroundthenetwork)possible. Documentationwaswrittenandsentofftoaboutfifteenfaculty,staff,andgraduatestudent users.AfewbugswerefixedandthesystemfrozenfromtheendofMaytoOctober.Endof Phase1. SecondPhase Whiletheimplementorwasawayatasummerjob,theconnectionbetweentheAandC machineswasswitchedtoreducetheloadingontheAmachine: Cory C A Unknowntotheimplementor,thenetworksourcewasmodifiedindivergentandincompatibleways,andthecommandsweremovedtoadifferentplace.Thesechangesinvalidatedcertain assumptionsaboutfullpathnamesandsomecommandssuchasinter-machinemailstoppedwork-ing. TheComputerCenterhadalsomadeabsolutelyincompatiblechangestosomesystemcalls. Thisbeganapathofsoftwaredivergencethatbecameverypainfulandisstillnotcompletely solved. Fortunately,theComputerCenterplacedRobynAllsmaninchargeofmaintainingthenetworkontheirmachines−tolightenthatroutinepartoftheloadfromtheimplementor. TheComputerCenteracquireda‘‘D’’machine,andtheEECSDivisionaDECVAX11/780, runninganexperimentalVersion7UNIXsystem.Theimplementordecidedtousethe(atthis point)unusedVAXtotodosoftwaredevelopmentandincorporatetheVersion7changesintothe networkcode.Bythistime,theprotocolswerestablewhichmadeitpossibletorunaVersion7 networkontheVAXconnectedtotheoldVersion6codeontheexistingnetwork,tofacilitate debugging.Becauseofimprovedterminalavailabilityandbettermachineresponse,manynew wayswereusedtodebugthenetwork−usingpipes,usingTTYlineswiredtogetherontheVAX andovertheusualmachine-to-machinelink.Afilewasadded(‘‘initfile’’)whichallowedquick reconfigurationofthedaemonswhensystemparameterswerechanged.Atemporaryconnection betweentheVAXandCmachineswasarranged.

-4-VAX Cory C A Thenetworkcodehadtobeabletorunonthreedifferenttypesofmachines−theVAX runningUNIXVersion7,theCorymachinerunningVersion6,andtheanomalousComputerCentermachines.Therewasnoconversionpackageavailableatthetime,†andtheoldnetworkcode hadnotusedanysystemheaderfiles,soafteragreatdealofexperimentation,conditionalcompilationwasusedasmuchaspossibleandaproceduralinterfacewasusedtoelidesystemdifferences. ThenewUNIXcommandmake(I)wasusedwitha‘‘makefile’’toorganizethisregime.The oldnetworkcodewasusedtobootstraptheDmachineontoanetworkrunningthenewnetwork codeexclusively. VAX Cory C A D Shortlythereafter,overtheChristmasbreak,theVAXandCoryconnectionswentdownfor securityreasons(discussedinthe‘‘Security’’section).Aftersevenweeks,theyreenteredthenetworkinanewconfiguration. †The‘‘retrofit’’library,byBillJoy,isnowavailable.

-5-Cory C D VAX A Shortlyafterthat,theyweredownagainbecauseofalightningstrikeforanotherweekbut havebeenoperationalsincethen.DuringthelasttimeperiodthenetworkwasmadelessBerkeley-specificandacopywasrunontheRandCorporationUNIXmachines.Documentationwas rewrittenandpreparedforrelease.Thenetworkqueueswereconvertedtosendshortest-jobfirst. Extensivemonitoringofsystemload,networkperformance,andnetworkusewasadded.Theformatofthenetqcommandwaschangedtosummarizemoreinformationonoutput.TheE machine,andthenlatertheSurveyResearchCenter(SRC)machine,wereadded. A Cory C D VAX E SRC Tuningwasstillimportant−seriousoverloadingproblemscausedbysluggishresponse stoppedthenetworkbetweenCoryandCforaweek.Networkparametersweretunedtohelp solvethisproblem.Thecomplexityofsoftwaredevelopmentandmaintenancebecametoogreat forunstructuredchanges.VersionsonallthemachinesexcepttheVAXwerefrozenforamonth atatime.Theprotocolswerealmostimmutable.Peopleweredelegatedresponsibilitytoobserve andstraightenout,ifnecessary,problemswiththenetqueues. Asthisiswritten,thesoftwareisstableandtheuser-documentationisfinishedandbeing sold,andthereishopeofaddingmoreUNIXmachinestothenetwork.

-6-OverallSystemDescription TheBerkeleynetworkoperatesinabatch/requestmode,andissimilarinconcepttoaline printerqueue.‘‘Requests’’arequeuedupatthesource,wheretheyaresentinshortestjobfirst orderthroughaninterconnectednetworkofmachinestotheirdestination.Ateachintermediate node,theyarequeuedasiftheywereoriginatedlocally. Thenetworkconsistsofasetofuser-executedcommands,aqueueofrequeststobesent, andacontinuously-runningprogramcalledadaemonwhichtransmitsrequestsinthequeueand listensforanyrequestbeingsenttoit.Therearemanynetworkcommands−onetosendmail, onetoreadmail,onetocopyfiles,etc.Theyallusethenetcommandtoaccessthenetwork. Thenetcommandtakesacommand,assortedparameters,withanyinputdata,andputsa requestinthequeue.Theserequestsarecomposedofaheader,thecommandtobeexecuted,and anydataforinputtothecommand.Theheadercontainsnetworkinformationsuchasthedestinationmachine,loginname,andpassword.ThisrequestisstoredinthequeueasanormalASCII file,ownedbytheinvoker.ThiswayUNIXcommandscanbeusedtoexaminethefile. Thedaemonexaminesthequeuetoseeifthereisanythingtosend.Ifso,itbeginssending toaremotedaemon,usingaprotocoltoestablishthisdialog,involvingretransmissionsandtimeouts.Theremotedaemonacceptstherequests,parsestheheaderinformation,andtakesanydata forthecommandandputsitinafile.Themainloopofthedaemonthenreturnstoawaiting state. Thecommandexecutionisdoneby‘forking’aseriesofprocesses.Oneoftheseistheuser’s loginshell,whichisgiventhecommandtoexecute.Anotherisaprocesswhichwaitsforthecommandtobeexecuted,thenexaminestheoutputofthecommand.Theoutputistypicallysent backtotheuser,viaanetcommand,executedbythedaemon. Inthereversetransmission,thecommandiscalled‘‘mwrite’’,anditisroutedandhandled exactlyasintheforwardmode,exceptnopasswordisrequired.The‘‘mwrite’’commandisexecutedontheoriginalmachinewithinputdatawhichisacopyoftheoutputoftheremotecommand.Theuseriseither‘‘written’’or‘‘mailed’’to,dependingoncertainoptions.If‘‘written’’, theuser’sscreenisfilledwiththeoutput†ofthecommandexecutedremotely,justasifhehad executeditlocally.Otherwise,itisinhismailbox,asmailfromtheremoteaccountheused.The user’sterminalmustbewrite-able(seethemesg(I)option),theoriginatingusermuststillbe loggedin,andhemustnothaveloggedoffandonagain. Theoutputfromthecommandisprecededbyalineofinformationlistingthecommand,the timeitwassent,andthetimeelapsed. OurdesignthentriestosimulatelocalUNIXcommandsasmuchaspossible.Withdefaults setcorrectly†theuserinprincipleneedonlyprecedethecommandheisexecutingwiththecommandnet. CopyingRemoteFiles Themostfrequentuseofthenetworkisfile-transferusingthenetcpcommand.Thenetcp commandisbasedonthecp(I)command.Itstwoargumentsareasourceanddestinationfile, optionallywithremotemachinenamesprepended: netcpfrom−fileto−file wherethenamesarelocalorremote.Since Cory:/usr/pascal/sh isafileontheCorymachine, †Standardoutputandstandarderrorfiles. †Witha‘‘.netrc’’file,seebelow.

-7- %netcpjunkCory:/usr/pascal/sh willtransferthefile‘‘junk’’tothenamedfileontheCorymachine‡. Thewaythetransferisaccomplisheddependsonthetypeoffilecopy: 1.Copylocalfiletoremotefile− Ontheremotemachineacat(I)commandisexecutedontheremotefilewiththelocalfile asstandardinput. 2.Copyremotefiletolocalfile− Acatcommandisexecutedontheremotemachinefromtheremotefiletostandardoutput. Thisstandardoutputissentbacktothelocalmachineintoaresponsefile,insteadofbeing writtenormailedtotheuser. 3.Copyremotefiletoanotherremotefile− Ifbothareonthesamemachine,acpcommandissent.Otherwise,anetcpcommandis senttotheremotemachinewherethefrom−fileexists,tocopythatfiletotheto−file’s machine. Thislastcaseisexperimental.Unfortunately,thesystemisstructuredonlytocarryone loginnameandpasswordtoaremotemachine.Sincethelastoptioninvolvesthreemachines,the secondremotemachineishandledimperfectlyatbest. SendingMail Themail(I)commandonthenetworkmachineshasbeenmodifiedtoexaminethenamesof therecipientsofaparticularmessage.Iftheirnameshavearemoteprefix,mailexecutesaninternalcommand‘‘sendmail’’,whichinturnexecutesanetcommand.Thisnetcommandsendsa mailcommandtotheremotemachine,withthemessageasinput.Sincetherecipientwouldlike toknowwhichmachinethemessagecamefrom,asimpleprogram‘‘mmail’’isexecutedtoinserta pseudo-headerindicatingtherealsourceofthemail.Thenetcommandlogsinasuser‘‘network’’, soremotemaildoesnotrequireanaccountonthedestinationmachine.Thisfacilityhasproven invaluable. ReadingMail Thenetmailcommandusesthenetcommandtosendacommandtoreadmailforaspecifieduseronaremotemachine.Sincetheexistingmailprogramsondifferentmachinesvaryin theiroptions,itwasdecidedtheonlythingthatwouldworkonbothUNIXVersion6and7systemswastocopythemailfromtheremotetothelocalmachine.Iftheusersubsequentlylogsin ontheremotemachine,hismailwillbethere,asifitwereunread.Aninternalprogram‘‘prmail’’ isusedtocopytheuser’smailbacktothelocalmachine.Itknowsthelocationofthemailboxes andtheuser’sname. ThemailprogramsatBerkeleyarebeingmodifiedtosearchadatabasetoseewhethera userwouldliketoreceiveallhismailonanothermachineandautomaticallyforwardit.Thiswill diminishtheneedforthenetmailcommand. Printingonremotelineprinters Thenetlprcommandtakesalistofargumentsasfilestobeprintedonaremotelineprinter. Unfortunately,therecanonlybeonestandardinputfilefortheremotecommand,soeachfileis sentasanetcommandexecutingthecommandlpr. OtherSystemComponents The‘‘.netrc’’file. Ausermustspecifydefaultsforfrequentlyrepeatedoptionsonaper-machinebasis.Thisis doneinafile‘‘.netrc’’intheuser’slogindirectory,andisfullydescribedinthe‘‘AnIntroduction ‡Formoreexamples,seethe‘‘AnIntroductiontotheBerkeleyNetwork’’document.

-8-totheBerkeleyNetwork’’. Thenetqcommand. Toseethenetworkqueue,theusermusttypethenetqcommand.Itliststhequeueforeach directly-connectedmachine,intheorderrequestswillbesent.Eachrequestislisted,oneperline, givingthelocalandremotemachines,thedestinationmachine,thetimesent,andthecommand tobeexecuted.Commandswhichareinternaltothenetworkarecalled‘‘responses’’inthenetq listing. Thenetrmcommand. Requestsmayberemovedfromthesendqueueusingthenetrmcommand.Sincetheoriginatorofthefileownsthequeuefile,asimpleuser-idchecksufficestovalidatepermission.Unfor-tunately,thisnotionbreaksdownforqueuefilesofrequestsonintermediatemachines.Onsucha machineanappropriateuserdoesnotexist,andthefilesareownedby‘‘root’’.Thereisanoption tonetrmtoremovealltheuser’squeuefiles,tomakenetrmeasiertouse. Thenetlogcommandandotherinformation. Anumberoflogfilesarekeptbythenetwork.Usersmayexecuteanetlogcommandwhich printsthelastfewentriesofalogofcommandssentandreceived.Alsolistedistheuser,the timeoftheentry,andthereturncodeofthecommand. Amoreunreadablelogfileis‘/usr/net/plogfile?’.Thislogfilehasalltheinformationofnetlog,inamorecrypticform,alongwithtraceinformationaboutneterrors.Thebeginningand endingofsendingandreceivingarenoted.Thiswaytheexactstateofthenetworkcanbedetermined. Hourlyanddailystatisticsinafile‘/usr/net/netstat?’.Thenumberofbytestransmitted, thenumberofcommands,andabreakdownoftheirtype,andsystemloadisrecorded.This informationisrecordedinbothhourlyanddailyformtotracktheperformanceofthenetwork underdifferentsystemloads. Everyhour,anetqcommandisexecutedandthenumberofqueueentriesisrecordedina file‘/usr/net/netqstats’.Thisgivesanestimateofthequeuelength. Finally,theloginnamesofeachlocaluserarerecordedinafile‘/usr/net/usernames’.Periodically,thesenamesaresortedandduplicatesremoved.Thisgivesacompletelistingofnetwork users,usefulforsendingnetwork-specificmailandforgeneralinterest. ProtocolExplanation Thenetworkusesthreedistinctlevelsofprotocol.Thehighestlevelofprotocol(the‘‘command’’protocol)referstotheorganizationoftheinformationsenttotheremotemachine.An intermediatelevelsplitssuchastreamintodistinctnumberedpacketswithasmallheaderineach packet.Thelowestlevelreferstotheappearanceofthesepacketsonthehardwareconnection. Atthepresent,thisisamodified6-bitASCIIcode.Eachoftheselayersisdistinct,andpresents theinterfacethroughprocedurecalls. TheCommandProtocol Eachmachinesendsarequestusingaprecisecommandprotocolinvolvingaheader,the command,andanyassociateddata. lengthheadercommand...data... Allbutthelengthfieldisformattedbythenetcommandbeforethefileisqueuedfortransmission.Thelengthisusedtodetectabnormallyshort,andpoorly-formed,requests.Theheader includesalltheinformationtorouteandverifytherequest.Itincludes -9-a)theoriginanddestinationmachines, b)theloginnamesonbothmachines, c)aversionstampforthiscommandprotocol, d)thetimesent, e)informationabouttheoriginatingterminal,and f)thepseudo-command. Thepseudo-commandisreadbynetq,andinsteadofprintingtheactualcommandbeing executed,printssomethingmoreappropriate.Allthecommandswhichusenet(e.g.netcp)use thepseudo-commandtolistthemselvesratherthanthecommandtheyareexecutingonthe remotemachine. InordertobeabletoprintthedataonanormalUNIXterminalfordebugging,thefields withintheheaderarevariable-lengthASCII,separatedbycolons(‘:’).Thisforcesthedaemonto parsetheheaderinformationandrequiresthatliteralcolons(e.g.inthecommandbeingsent)be properlyescapedwithintheprotocol,butIfeltthealternativesofusingbytecountsorfixedlengthfieldsweretoodifficulttodebug.Theshortestheaderisapproximately85bytes.Fortunately,thismeanstheshortestcommandwillfitintoasinglepacket.† ThePacketProtocol Theaboveinformationisconveyedtoeachmachineasastream.Thisisdoneusingsubroutinecallstoreadandwritedataofarbitrarylengthoverthelink.Thewriteprocedurebreaksthe informationintoasetofnumberedpackets,withalengthandexclusive-orcheck-suminaheader: lengthseqnumtypechksum...data... Thelength,typeandchecksumareonebyteeach,andthesequencenumberistwobytes. Sincethepacketsarevariablelengththechecksumisintheheaderratherthanattheendofthe packettoavoidtheextracomputationrequiredtoaccessit. Eachpacketistransmittedoveralinktoalistener.Normallyanacknowledgementpacketis sentback.Ifthereisanerror,nothingissentback,andthesendingendwillretransmitaftera certainnumberofseconds. Therearenowindowingorpiggybackacknowledgementsfortworeasons:1)thisschemeis verysimpletoimplementand2)theerrorrateifeachpacketwerenotacknowledgedwouldbe highbecauseofthehardwareinvolved.Ifthefuture,Ihopethatbothhardwareandkerneldevice driverswillallowthisimprovement. Theso-called‘‘rendezvous’’protocol,wherebytwodaemonsagreetocommunicate,isasim-ple‘‘contention’’scheme.Whenonedaemonwantstotransmit,itsendsaspecialpacket‘‘reset’’ tothepossiblereceiver,thentransmitshisfirstpacket.Normallyadaemonabletoreceivelistens fora‘‘reset’’.Ifitreceivesone,itentersasectionofcodedesignedtoreceiveaheadercommand, anddata,andultimatelywillexecuteit.Ifnot,afteraprescribedtimeintervalischeckstoseeif thereareanyrequeststosend.Shouldbothsendatonce,thecharactersmaybegarbled,orboth mayreceiveresetsatthesametime.Ineachcasetheybothwillretransmit.Eachhasarandomizingfactortobreakanytieswhichmightdevelop. Inretrospect,thisprotocolisveryprimitive.Nowthatthenetworkisinproductionuse,the extraacknowledgementsandseparate‘‘reset’’aretooexpensive.Aredesignwouldmodifythe protocoltotransmitmorethanonepacketbeforeacknowledgingit(ACKs),usenegativeACKsto indicateimmediatelythatanerrorhasoccurred,andeliminatetheseparate‘‘reset’’entirely. The‘‘rendezvous’’protocolconsumesafairamountoftimewhenbothdaemonschooseto sendpackets.Thealternativeofconstantlysendingstatuspacketswhenthedaemonswouldbe idlewasneverseriouslyconsideredbecauseitwasfeltthatthedaemonshouldhaveaslighta †(lessthan100bytes,seebelow).

-10-systemloadaspossible;itseemsnowthedaemonsarebusymostofthetimeandthustheinitial connectiontradeoffsshouldhavebeenstudiedmoreclosely. TheLow-LevelProtocol ThenetworktransmitsoverTTYlinesthroughterminalinterfacesandsystemdriverswhich behaveasifthecharacterscomingfromanothermachinearefromaterminal.Thismodewas chosenbecauseitisabsolutelythesimplest,cheapestinterconnectionschemepossible.Unfortunately,theUNIXterminaldriverscannotaccept8-bitbytesunlesstheyareinrawmode.Thiswas judgedtobeahighsystemload,sotheTTYlinesoperateincooked,thereverseofraw,mode.In thismodecertainbitcombinations,e.g.ASCIInewlineandescape,donottransmitthroughthe terminaldrivertotheuser’sprogrambutratherareinterpretedascontrolinformation. Aftermuchexperimentation,thefollowingtransmissionmethodwaschosen.Each3byte tripleisviewedas24bits,andbrokeninto46-bitgroups.Each6-bitnumberisintherange0-63, andisaddedtoaconstantrepresentingthelowestacceptablecharactercode(ablank)inthe ASCIIsequence.ThisissentasanASCIIcharactertoareceiverwhogathers4bytes,subtracts theincrement,andshiftsthe46-bitgroupsinto3bytes.Thisrepresentsa3to4expansionofall charactersoverthelink,ora33%lossofbandwidth. Inretrospect,thisexpansionhasaconsiderablecost.Themostscarceresourceinthenetworkistheactualhardwarespeedofthelinks.Thealternativeofusingrawmodewasneverseriouslyconsidered. Theimplementor’shopeisthatbetterhardwarewillmakebettermiddle-andlower-level protocolseasier.Untilthen,thedifficultiesofusingTTYlinesefficientlymakefurtherprotocol improvementsunlikelytoyieldbigincreasesinspeed. ANoteAboutFeaturesthisProtocolLacks InUNIXaprocessmayonlyreadorwriteoneI/Odeviceatatime.Adaemonapproach requiresasingleprocessreadingandwritingonalinktoanothermachine.Thisprocessmust decidewhowillreceivethispacket.Ijudged(correctly)thatthisdecisionwashardtoschedule usingUNIXpipesandsignals,andonlyallowonekindofcommunicationbetweendaemons.This alsomakesitalmostimpossibletoforwardpacketsthroughintermediatemachines.Thusintermediatemachinescopywholerequestsbeforesendingthemagain. Ifthedesignspecificationrequiredasimplepacket-orienteddriverwithinthesystem,the UNIXkernelcoulddecidewhichofseveralspecialfilesthiswasdestinedto,andallowmuchmore intermixingoftrafficthanbefore.Ididnotrealizetheimportanceofthisand,inretrospect, wouldhavechosentheotherofthetwoapproaches. Portability TheacquisitionofVAX/UNIX(Version7)andthedivergenceoftheComputerCenterand CoryHallVersion6systemsmadetheportabilityofthenetworksourcecodeimportant.Until then,thesourcecodeonallmachineswasidentical.Fortunately,theUNIXimplementorsencounteredthesesameproblemsanddevelopedanumberoffacilitiesthenetworknowuses. Sincemanysystemcallsusemachineandversiondependentdatafields,so-called‘‘include’’ filesareavailabletohidethesystemdifferencesandhelpstandardizethesysteminterface.The conditionalcompilationfeatureoftheClanguagewasusedtoselectwhichkindsofcodetogenerate,whenthe‘‘include’’fileswerenotsufficient.Roughly,thefollowingcommandincludedatthe beginningofeachCmodule: #include<whoami.h> woulddefinewhichsystem,byname,thecodewasrunon.Forexample,theabovedefines‘‘VAX’’ ontheVAXmachine,andthenlinessuchas -11- #ifdefVAX . . #endif controlthecodegenerated.Inthenetwork,sequencessuchasthisinturndefineothersequences, suchas #ifdefCORY #defineFUID #defineOLDTTY #definePASSWDF #endif definestheunusualfeaturesoftheCorymachine:thecombineduser-idandgroup-idreturnedby thegetuid()systemcall,thatitusestheold1-characterterminalnames,andthatithasasplit passwordfileforsecurityreasons.Eachofthesesymbols,e.g.‘‘FUID’’,istestedinordertocompilethecorrectcodeforthatfeature. Tohelpinisolatingthechanges,attemptsweremadetocreateaproceduralinterfacetohide machinesdifferences.Theseproceduresareallinonefile.Onlyoneortwocasesexistofconditionalsectionsofcodenotin‘‘mach.c’’or‘‘mach.h’’,itsheaderfile. Oneproblemthesefeaturesposeistestingchanges−theconditionalsectionshideerrorsin inapplicablecode.Aregimenwasadopted:TestingwasfirstdoneontheVAX(Version7),then, afteritwasstableforafewdays,movedtoCory,wheretypicallytherewassomeVersion6-depen-denterror,andafterthatwasfixedandstable,itwasmovedtotheComputerCentermachines. Thisnotionofa‘‘testing’’machineisveryimportant−theVAXalwayshasanup-to-datecopyof thenetworksource,eventhoughothermachinesmaylaginimprovements. Thereisnowa‘‘retrofitlibrary’’thatsimulatesmanyofthefeatures‘‘mach.c’’provides.It wasnotstableenoughwhenthenetworkwasconvertedtoVersion7,otherwiseIwouldhaveused it. Atthispoint,whentheentiresourceforthesoftwareforthenetworkistransferredbetween machinesonlythefirstfivelinesofthe‘‘makefile’’needbechanged. Security OverChristmasvacationof1978,a15-yearoldhighschoolstudentrepeatedlybrokeintothe ComputerCenterandCorymachines.Hewasabletousethenetworktogainaccesstoprivileged filesontheVAX,andthefearofprotection‘‘holes’’causedthestafftotakethenetworkdownfor sevenweeks. Thereweretwosecurityproblemsposedbythenetwork: 1.Thethreattothe‘‘root’’accountonanothersystem. 2.Thethreattoauser’sremoteaccounts. 1.Threatto‘‘root’’ Originallythenetworkwouldallowauserloggedinas‘‘root’’ononemachinetoexecuteany commandas‘‘root’’onanothernetworkmachine.Asfarasweknow,thisfeaturewasneverused tobreakintoasystem.However,thenetworkhasbeenchangedtopreventauserfromloggingin as‘‘root’’onanothermachine,regardlessofthepassword.Thischeckisperformedonthesending machine.Since‘‘root’’couldconceivablycircumventthischeckbyalteringthecommand,the receivingendofacommandcheckstheuser-idofallcommandsbeingexecuted.Ifitiszero(i.e. ‘‘root’’)onlyasetoffivecommandsisallowed,allneededbythenetworkinternally,andbelieved ‘‘safe’’.

-12-Webelievethismakesthenetwork‘‘safer’’thanmanylocalmachinefeaturessuchastheuse ofdial-uplines. 2.Threattouser’sremoteaccounts. Ifauserplacesremotepasswordsinhis‘‘.netrc’’file,anillicitsuperusercouldgetthepasswordtoalltheuser’sremoteaccounts.Eveniftheuserdoesnotcare,systemmanagersdislike thisbecausetheillicitsuperusercouldnowusealegalaccountonanothersystemtobreakintoit. Wehavenogoodsolutionstothis.Usersarenowwarnedofthisdangerinthedocumentation,anda‘‘.netrc’’filewithpasswordsmustbereadableonlybytheownerofthefile. Varioussolutionshavebeenproposed: a)Disallowpasswordsin‘‘.netrc’’files. Unfortunately,heavynetworkuserswouldhavetorepeatedlytypetheirpassword. b)Encryptthe‘‘.netrc’’file. Aprogramwouldhavetoexisttodecryptthefile.Asuperusercouldgetaccessto whateverkeytechniquethatprogramused,ifitwereonthelocalmachine.Apublickeyencryptionschemewouldmakethisoptionpossible.Wedecideditwastoomuch worktoimplementthis. c)Once-a-sessionpasswords. Inthisscheme,auserwouldregisterhispasswordwhenheloggedin,thenusethenetworkwithoutneedingtotypeinapassword.Whenheloggedoff,thepasswordwould beremoved.Wediscardedthisbecausewecouldnotguaranteethepasswordwould disappearunlesswewroteadaemon,whichitselfcouldbecompromised.Thebest solutionalongthislineusesthe‘‘alias’’featureoftheCshell.Eachnetcommandis aliasedwithitselfandthe−l,−poptions.Whentheuserlogsin,hesetsashellvariabletohispassword.Whenhelogsoff,hisshelldiesandthepasswordsareforgotten. Ibelievethecurrentalternativesaresufficientforaconscientioususertoprotecthimselfand stillhaveaneasy-to-usenetworkinterface. FuturePlans 1.Hardware Thenethassufferedwithlowspeedhardware.Short-termplansincludespeedingupthe currentterminalinterfacehardwarefrom1200Baudto9600Baudandwritingadriverforthe devicetobypasstheinternalUNIXcharacterqueues.Thisdriverwillimprovethereliabilityof transmissionanddecreasethecharacterinterruptoverhead.Thespeedupfrom1200Baudto9600 Baudmayoverloadthesystemsduetothenumberofhardwareinterruptsitcauses. Inthelongerterm,theEECSDepartmentisconsideringacquiringdirect-memory-access (DMA)devicessuchastheLogicalNetworkInterface(LNI)ortheDigitalCorp.DMC-11linksfor high-speedtransmission.Thesedevicesarecapableofover1-megabitspeeds,andwouldincrease thespeedofthecurrentnetworkbyfactorsofhundreds. 2.AddingMoreMachinestotheNetwork TheINGRESResearchgroupandvariousotherresearchunitswithintheEECSdepartment haveexpressedinterestinbeingaddedtothenetwork. 3.RemoteUseoftheTypesetterFacilities TheComputerCenterAmachinehasaGraphicsSystemsphototypesetterandtheVAX ResearchmachinehasaVersatec36’’dot-matrixplotterwithatroffsimulator.Softwarenow beingdebuggedwillallowremoteuseofthetypesetterbyrunningthetroffprogramlocallyand sendingthetypesetterdevicecodestotheremotemachine.

-13-ThiswilldistributethetypesettingloadandhelpoverloadingontheAandVAXmachines. Itwillalsoallowtheuseoftroffmacropackagesonlyavailableonsomemachines. 4.RemoteMailForwarding TheUNIXmailprogramswillbemodifiedtoforwardmailtoanotheraccountonanother machine,allowingauserwithaccountsonmanymachinestoreaditallononedesignated machine. SummaryPoints Theauthorwouldliketostressthesepointsabouthisexperience: 1.Successinbuildingnetworkingsoftwaredependsonhavingreadyaccesstothecorrecthardware.Theminimumistwoterminalsconnectedtotwousablemachineswithtwomagnetic tapedrivesorsomeotherexistingmeansofsoftwaretransfer. 2.Designinportabilityandsecurity.Morecarefulattentiontomachinedependenceandsecurityinthefirstphasewouldhavesavedmuchlaterre-programming. 3.Developgoodlocaldebuggingtechniques.The‘‘self-loop’’trickfornetworkdebugging dependsontheaccuracyofthatsimulation.UNIXpipes,forexample,werenotsufficientto simulateTTYlinesbecauseTTYlinesare7-bitswitharestrictedASCIIrange. 4.Encourageuserstousethesystem.Theirfeedbackisimportant.However,itisnecessary tohaveanunusednetworklinkfornewprotocoldevelopment,etc. 5.Thereisafinelinebetweensupportofanexistingnetworkandresearch.Inthebestofall possibleworlds,supportofresearch-developedsoftwarewouldbetheresponsibilityofthe systemsstaffforthemachinesitrunson.Thisisseldomthecase. 6.Theconceptoflayersofnetworkswasveryhelpfulinthisproject.Thereappeartobethese levels: userinterface queuesanddaemons commandprotocol packetprotocol transmissionprotocol teletypelines Theselevelsarequitedistinct.Ifanew,betternetworknotinvolvingqueuesisbuilt,the transferoffilescouldstillbebynetcp.Ifstate-of-the-artlinkhardwareisused,perhapsall ofthelevelsbelowthecommandprotocolcouldbediscarded. 7.Thechiefvirtueofthecurrentsystemisitsextremeflexibilityandlowstart-upcosts.No modificationstotheUNIXkernelarerequiredandalllocalfeaturesareconditionallyspecified inaheaderfile. 8.Networksarehardtobuildbecause -14-a)Theyinvolvemutually-cooperatingcopiesofsoftwareon(usually)differingcomputers. b)Manyoptionsarenotpracticalbecauseofcompatibilityconsiderations−newnetworksneeddriversinunchangeablesystems,andnewprotocolshavetoaccepttheold protocolsuntiltheoldprotocolsareextinct.