SPORADIC DIALOGUES by Eugene Volokh, VESOFT Q&A column published in INTERACT Magazine. Published in "Thoughts and Discourses on HP3000 Software", 3rd ed. Q: We do our system cleanup -- things like a VINIT >CONDense and such -- at night without operator assistance. However, some people sometimes leave their sessions logged on overnight, and sometimes jobs are started up in the evening that are still running when we want to do the cleanup. We'd like to be able to automatically abort all the jobs and sessions in the system (except, of course, the job doing the aborting). How can we do this? A: What we would really like to have -- what would make the problem trivial to solve -- is a command of the form :ABORTJOB @.@ Try finding that one in your reference manual. The sad fact, of course, is that no such a command exists. However, we do have two commands that each do about half of the task: * :SHOWJOB, which finds all the jobs in the system. * :ABORTJOB, which aborts a single job. The trick is to take the output of the :SHOWJOB command and feed it to the :ABORTJOB command, so all the jobs shown are aborted. Without much further ado, here's the job stream we want: !JOB ABORTALL,MANAGER.SYS;OUTCLASS=,1 !COMMENT by Vladimir Volokh of VESOFT, Inc. !FILE JOBLIST;REC=-80;NOCCTL;TEMP !SHOWJOB;*JOBLIST !EDITOR TEXT JOBLIST DELETE 1/3 DELETE LAST-6/LAST FIND FIRST WHILE FLAG DELETE "MANAGER.SYS" CHANGE 8/80,"",ALL CHANGE 1,":ABORTJOB ",ALL KEEP $NEWPASS,UNN USE $OLDPASS EXIT !EOJ A brief explanation: * First we do a :SHOWJOB of all the jobs in the system to a temporary disc file (note that this redirection of :SHOWJOB output to a disc file might not be documented -- it certainly isn't documented in :HELP, nor is it mentioned in some of the older manuals). * Then we enter editor and massage the file so that it contains only the lines pertaining to the actual jobs (all the lines except the first 3 and the last 7). Then, we exclude all the jobs signed on as MANAGER.SYS, since we don't want to abort our system cleanup job streams (like ABORTALL itself or the job that streamed it), and they are presumably signed on as MANAGER.SYS. Of course, you can easily change this to any other user name. * Now, we strip all the data from the lines except for the job number, and then we insert an :ABORTJOB in front of the number. The file now looks like: :ABORTJOB #S127 :ABORTJOB #S129 :ABORTJOB #J31 :ABORTJOB #S105 :ABORTJOB #J33 * Finally, we keep this as a disc file (in our case, $NEWPASS), and then we /USE this file, causing each of its commands to be executed as an EDITOR command. Since ":ABORTJOB" is a valid EDITOR command -- it just makes the EDITOR call the COMMAND intrinsic -- all these commands get executed! * Note that in order for this job stream to work properly, the :ABORTJOB command must be :ALLOWed to MANAGER.SYS. This can be done via the :ALLOW command or by use of the ALLOWME program in the contributed library (see the answer to the previous question for more information). And there's your answer. Fairly simple, no need to go into privileged mode, no need to even write a special program (other than the job stream itself). A perfect example of "MPE PROGRAMMING" -- performing complicated system programming tasks without having to write a special program in SPL or some such language. Incidentally, for more information on MPE Programming, you can read my "MPE Programming" article. Q: Sometimes we have a program file and aren't sure which version of source code it came from. Usually, we have a strong suspicion which one it is, so we tried to simply recompile the suspected source file and then use :FCOPY ;COMPARE to compare the compiled source with the program file. Unfortunately, FCOPY gave us hundreds of discrepancies; in fact, we found that the program file changed every time it was run! What can we do? A: As you pointed out, every time a program is run, various parts of it -- record 0, the Segment Transfer Tables (STTs) in each segment, etc. -- are changed. FCOPY is therefore a dead end, since it's virtually impossible to tell a legitimate discrepancy from a difference caused by the loader. However, there are some things you can do. First of all, you can of course :LISTF both program files -- if they are of different sizes, they're clearly substantively different. Of course, if they're of the same size, you can't be certain that they are actually the same. Beyond that, your best solution is to have some kind of version numbering system. If you can be sure to increment a version number variable in your source every time you make a change, and then print the version number whenever you run the program, you can very easily tell whether two program versions are the same just by looking at their version numbers. The only problem with this is that it's rather easy to forget to change the version number. If you think this will happen often, you could try some more automatic but less reliable approaches. For instance, you could simply compare the creation dates of the program file and the source file; since whenever you /KEEP a file in EDITOR, it's re-created, thus updating its creation date, it's a pretty good bet that if the two creation dates are equal, you're dealing with the same version. Of course, this approach is hardly foolproof -- either file might have been copied, thus changing its creation date, or two changes might have been made on the same day. Unfortunately, this is as precise as you're going to get without some kind of version numbering system in the source code. Q: One of my programs keeps aborting with a FSERR 74 ("No room left in stack segment for another file entry") on an FOPEN. True, I do have a procedure that allocates a very large array on the stack -- I run my program with ;MAXDATA=30000 -- but I don't do the FOPEN at the time I'm in the procedure, so at FOPEN time my stack can't be larger than about 15K. What am I doing wrong? A: To answer this question, I have to digress for a moment and discuss the structure of your stack. All of the variables and arrays you declare in your program -- local or global -- are put onto your stack, which is stored by MPE in a single data segment. Now, you don't view your stack as a data segment, since you don't need to call DMOVIN or DMOVOUT to access it; however, deep down inside the system does, and places on the stack a fundamental restriction common to all data segments -- no data segment may ever be more than 32K words long. Now, your stack is actually partitioned into several pieces, each piece pointed to by a machine register: High memory Z > --------------------------------------------- ^ | unused space | ^ S > --------------------------------------------- ^ | operands of machine instructions | ^ | and local variables of the currently | ^ | executing procedure | ^ Q > --------------------------------------------- ^ | local variables of other procedures | ^ | and stack markers indicating calls from | ^ | one procedure to another | ^ Qi > --------------------------------------------- positive | global variables | addresses DB > --------------------------------------------- | "DB negative area," accessible only by | negative | SPL procedures (like V/3000) -- usable | addresses | for global storage | v DL > --------------------------------------------- v | The Nether Regions, where mortals may | v | not stray and non-privileged accessors | v | are punished with bounds violations | v --------------------------------------------- v Low memory (The above picture reproduced with permission from "The Secrets of Systems Tables... Revealed!" by VESOFT. Said permission was not especially difficult to obtain.) Now your stack contains all of this stuff, from the bottom of the Nether Regions to your Z register. The Nether Regions (the DL negative area, also knows as the PCBX) is where the file system allocates miscellaneous information about your file; when you open enough files, the initial space allocated below DL is exhausted, and the system has to allocate some more. Now, if this would make the total size of the stack data segment larger than 32K, you get an FSERR 74. The problem is that the data segment size is not measured from the bottom of the PCBX to the S register, but rather to the Z register. Your S register points to the current top of your stack, so if you're using 15K, your S register value is 15K; however, your Z register points to the highest place that the S register has ever been! Thus, say your stack size starts out at 10K. Your S points to 10K, and your Z points to about 12K (since the system allocates about 2K of overhead between S and Z). Now, you call your procedure which allocates an 18K local variable; now your S is at 28K and Z is at 30K. When you exit the procedure, the S is reset to 10K, but the Z stays at 30K! This leaves you only 2K for the PCBX, even though you have 20K of unused space between S and Z. The solution? Well, I think that your best solution is simply to call the ZSIZE intrinsic, passing to it the parameter 0, before calling FOPEN. This call frees any space you may have between S and Z, thus leaving the maximum possible space for the file system to work with. But, you might say, if the file system allocates an extra 1K in the DL negative area, this will decrease the maximum Z value to 29K, and so when I next try to allocate that 18K array, I'll get a stack overflow! Right? Wrong. It turns out that the highest value I could get Z to take was 30848 (use this number for comparison, your mileage may vary with your MPE version). Thus, if you allocate enough space for S to be pushed up to 28K, Z will be set to 30848; however, if you push S all the way up to 30K, Z will be left at the same value of 30848 without any adverse effects. Thus, you can contract the S-Z area by calling "ZSIZE (0)", call FOPEN, have it allocate a couple of hundred extra words in the DL negative area, and still be able to push S and Z up by 18K with no problems! So, doing a "ZSIZE (0)" before an FOPEN will probably solve your problem (I do this in my MPEX program all the time). However, if it doesn't -- if you are really using every word of stack space and can't fit both your own data and the file system's file information into one 32K stack -- there is an alternative; you may run your program with ;NOCB, which causes most of the file system information to be allocated in a separate data segment. The reason why I didn't suggest this first was that this slightly slows down file system accesses; furthermore, if your program doesn't run without ;NOCB, you can bet that half the time the user will forget to specify ;NOCB and will get an error. I think that the ZSIZE call, if it works, is the cleaner solution. Q: What effect does virtual memory size have on the performance of my system? Can I configure it too large? Too small? A: None. No. Yes. What? You want to know more? I gave you the answers, didn't I? Oh, all right. If you have too little main memory, this will undoubtedly affect your system performance because there'll be a lot of swapping. However, let's pretend that there was no virtual memory -- that if you ran out of main memory, you'd simply get an error message (instead of getting an unused segment swapped out to disk). In that case, the size of main memory would actually have no effect on the performance of your system, if all you mean by performance is the speed at which the system runs. If you configure your main memory too small, you'll simply get an error message when you try to overflow it; however, whatever can run in the reduced memory will run as fast as it would in a configuration with a lot of memory. That's exactly what happens with virtual memory. If virtual memory gets full, you'll just get an error message; then, you can reconfigure your system to have more virtual memory, and that's that. If you want to check how much virtual memory you're actually using, you can run the contributed utility TUNER. So, as long as you don't run out of virtual memory, changing its size will not affect your system performance one iota. The only problem with making it too large is that you waste some disc space (which is fairly cheap anyway). The System Operation and Resource Management Manual suggests that you determine the amount of virtual memory you will need as follows: estimate the average number of concurrent users, the average stack size, the average number of buffered files open for every user, and the number of users who will be simultaneously running programs; then use these figures in conjunction with the values listed below to calculate the amount of disc space to reserve for virtual memory: * 32 sectors for each Command Interpreter stack * 4 sectors for every open buffered file * 16 sectors for the system area in the user's stack, plus 4 sectors for every 512 words in the DL/Z area of the stack * 40 sectors for each program being loaded in the system The amount of virtual memory calculated by the above is adequate for the overwhelming majority of systems. If you actually run out of it, just increase it. Q: We have a 2621P terminal as our system console. We'd like to let programs send special messages to the operator in which they turn the integral printer on, print the message, and then turn the integral printer off. However, when we just send the message using :TELLOP, all the escape sequences we use to manipulate the printer are stripped. What can we do? A: First, allow me to give you a bit of historical background. When the :TELL and :TELLOP commands were first built, they just took the message and printed it to the target terminal. In the days of the old dumb terminals, this was quite fine. Then, along comes the HP 264X series terminal with its potpourri of escape sequences. One of these escape sequences is "ESC d", which causes the terminal on which it is printed to send to the computer the current line of the display. So, some wise guy enters the following command: :TELL MANAGER.SYS B ALTACCT DEV;CAP=IA,BA,...,PM,SMd The message is printed on a terminal that's signed on as MANAGER.SYS, the "B" makes the terminal go to a new line, the :ALTACCT command is displayed, and the "d" tells the terminal to send the command to the computer. Simple and effective -- with one :TELL command, you can "enslave" any session and make it do your bidding. HP got wind of this, and realized that it had to do something about it. So, now the :TELL and :TELLOP commands (and PRINTOP intrinsic) strips out all escape sequences from the message except for a select few legal ones (e.g. "&d" which sets the terminal enhancements). Fortunately, what the security system taketh, privileged mode can give back. The :TELLOP command, after stripping the escape sequences, calls a system-internal, privileged, and VERY DANGEROUS IF YOU DON'T CALL IT RIGHT procedure called GENMSG. With PM, you can call GENMSG directly. Just compile the following procedure and add it to your system SL (or any SL with PM capability): $CONTROL NOSOURCE, USLINIT, SUBPROGRAM, SEGMENT=TELLOP'ESCAPE BEGIN PROCEDURE TELLOPESCAPE (BUFFER, LENGTH); BYTE ARRAY BUFFER; INTEGER LENGTH; OPTION PRIVILEGED; BEGIN << This procedure sends the message contained in BUFFER to the system console; BUFFER may contain escape sequences. LENGTH must be the length of the message in bytes, NO MORE THAN 255!!! GOD FORBID THAT YOU SHOULD PASS AN INVALID BUFFER ADDRESS OR AN INVALID LENGTH!!! >> BYTE ARRAY TEMP(0:255); INTEGER PROCEDURE GENMSG (NSET, NMSG, MASK, P1, P2, P3, P4, P5, DEST, REPLY, OFFSET, DSEG, CONTROL); VALUE NSET,NMSG,MASK,P1,P2,P3,P4,P5,DEST,REPLY,OFFSET,DSEG,CONTROL; INTEGER NSET, NMSG, DEST, DSEG; LOGICAL MASK, P1, P2, P3, P4, P5, REPLY, OFFSET, CONTROL; OPTION EXTERNAL, VARIABLE; MOVE TEMP:=BUFFER,(LENGTH); TEMP(LENGTH):=0; GENMSG (-1,@TEMP,<>,<>,<>,<>,<>,<>, 0<>); END; END. You can call this procedure from any program and pass to it the byte array containing the message to be printed (including any escape sequences you'd like) and the length of the message. Note, however, that if you make this procedure publicly available, people can pull the same stunt that made HP forbid sending escape sequences in the first place. If you're concerned about this, you might put checking code into the procedure, or put the procedure into a group or account SL rather than making it available to everyone by putting it into SL.PUB.SYS. Q: We would like to know how the CALENDAR and FMTCALENDAR intrinsics are going to cope with dates beyond 31 December 1999. We do not think it premature to seek clarification on this topic, because each user of the HP3000 system will have programs and procedures that compare dates and assume that the smaller number is the earlier date. With the CALENDAR intrinsic, bits 0 to 6 represent the year of the century and bits 7 to 15 represent the day of the year. For non-computer people, we usually describe this dating convention as 512 * (year of century) + (day of year). On 3 January 2000 if we want a SYSDUMP to include all files that have been accessed since 28 December 1999, so we enter 12/28/99 as the response to "ENTER DUMP DATE?". Now, how can the computer recognize that files accessed on 1, 2, or 3 January with an access date of %1, %2, or %3 are actually after year 99 day 362 (which is represented as 512 * 99 + 362 = %143552). Somehow, %1 will have to be recognized as subsequent to or effectively a larger number than %143552. To demonstrate the present inability of MPE to cope with this, users can on their next system restart enter a system date of 12/31/99 and a time of 23:58 and do a :SHOWTIME after each elapsed minute to see what happens when the century changes. We have submitted this problem on a Service Request and it has been assigned a number 4700094458 but we are not aware of any progress as it has not yet appeared on the System Status Bulletin. A: You raise a very interesting question that is rapidly becoming more and more relevant to system design. The actual CALENDAR intrinsic will not be the first thing to fail come the new century (new millennium!). There is room in there for up to 128 years, so HP should be OK all the way until 2027, as long as it recognizes that 12/31/03 is actually 31 December 2003, not 1903. FMTCALENDAR (and DATELINE, :SHOWTIME, etc.) will all be in a bit more trouble. If you're too impatient to see for yourself by resetting your system clock, I'll tell you that on 1 January 2000 00:00, the :SHOWTIME will read SUN, JAN 1, 19:0, 12:00 AM The ":" comes because it's the next ASCII character after "9"; the formatted algorithm uses a simple-minded approach which say that the decades digit is always ("0"+YEAR/10) and the years digit is ("0"+YEAR MOD 10). However, this will only hurt you if you rely on FMTCALENDAR or DATELINE in your program. After all, if all you care about is :SHOWTIMEs, when will you forget what decade you're in? The big problem is that ALMOST ALL USER PROGRAMS RELY ON 6-DIGIT DATES. It's not just that the HP can't handle the 21st century; chances are that your payroll, accounts payable, general ledger -- all your programs will believe that something done on 01/01/00 was long before 12/31/99. That is the thing people should start worrying about soon. The good news, of course, is that things aren't pressing just yet. It's unlike that the 3000 line will survive into the 21st century, and the few of you who might have one stashed in your attic will be laughed at for using "those old dinosaurs that couldn't even talk." The designers of the 3000 are probably not too worried about this problem right now. You -- the working programmers of the world -- should however start getting concerned, if not now, about 5 years from now. That's when programs that you write will begin to have a reasonable chance of surviving into the 21st century. At least, they'll be likely to at some time need to work with 21st century dates (as in "this loan is due 10 years from now in 2003"). You probably do not want to have your comparison routines fail and notify the debtor that his loan is 91 years overdue. Actually, there exists right now a small (but ever-growing) number of programs that at some time will have to concern them with this. I suggest that anyone designing new application systems -- and, after all, your system will probably outgrow the current computer you're on, migrating to Spectrum and god-knows-what-else -- plan for 8-digit dates (MM/DD/YYYY). This will waste 2 bytes per date, but in these days of 400 Mb discs, you can afford this relatively easily. Putting on my Nostradamus (or is it Cassandra?) hat, I predict that most people will not do this Until It's Too Late, even though far more weighty voices than mine will start suggesting this fairly soon. Then, about 1997 or so, they will See The Light, say Oh, My God, What Do We Do Now, and hire great batches of programmers to correct all of their old programs. This will rescue millions of programmers from dismal fates as grocery clerks, sanitation engineers, or data processing managers, and will be hailed far and wide as the Boom of '97. Congress, seeing the stimulating effect of this on the economy -- all those new jobs! -- will find this all A Very Good Thing, and will pass the National Date Standards Act of 1998, which will mandate that henceforth all dates must be stored in 5 digits, thus necessitating modification every 10 years. Senator Prescott of Massachusetts will propose an amendment that requires storage in 4 digits, but this will be voted down 64-29 (7 abstaining). See, they never listen to me. Oh, your Service Request? I'm sorry to say that even as I write this, it has inadvertently been filed in a small, cylindrical file in Building 43U, and will soon meet an ignoble fate at the hands of Rover, who frequents the garbage cans thereabouts. A future archaeologist will uncover shreds of it right next to the last three Service Requests that I've sent in, and will write a doctoral thesis. Any other questions? Q: Can you explain what happens when I run QUERY.PUB.SYS from inside SPOOK.PUB.SYS? No output goes to the screen, although QUERY seems to be accepting input, and writing stuff to a file it builds called QSOUT. In a similar vein, when I try to run a compiled BASIC program within SPOOK, it fails with a CHAIN ERROR. Can you explain? A: It is to the credit of the author of SPOOK that SPOOK has a RUN command in the first place. Note that until TDP, no other HP product (except perhaps BASIC) allowed one to run other programs within it (SPOOK, incidentally, started out as an SE-written program and only later became an HP-supported utility). Apparently, one of the reasons that SPOOK's author put the >RUN command in in the first place was to allow SPOOK to be run from within itself! Presumably, this could be useful for being able to look at two spool files at a time; frankly, I don't see the need for it, but all the signs point to the author's desire to be able to run SPOOK from itself. Try going into SPOOK and saying > RUN SPOOK.PUB.SYS A new copy of SPOOK will be launched, AND IT WILL PROMPT YOU NOT WITH "> ", BUT WITH ">(1)"! If you run SPOOK again from within the newly-created process, the newest process will prompt you with ">(2)", and so on. If you run SPOOK from itself 10 times, instead of getting the ">(10)" that you'd expect, you'll get ">(:)". Apparently, SPOOK's author decided that it was important to indicate which copy of SPOOK was which, so SPOOK always run its sons with ;PARM=49 (which is the ASCII equivalent of "1"). If a copy of SPOOK was already with a PARM=, it creates all its sons with a PARM= value 1 greater (i.e. 50, 51, etc.). The prompt is then set to ">(x)" where "x" is the ASCII character corresponding to the PARM= value. The bottom line is that when you run SPOOK in the default state (with ;PARM=0), it will run all its sons with PARM=49. This is usually OK, except for those programs -- like QUERY and BASIC -- that act differently when their ;PARM=0; in those cases, life gets difficult. Unfortunately, on SPOOK's >RUN, you can't specify a different ;PARM= value; you're pretty much stuck with what SPOOK gives you. You can, however, use one or more of the following workarounds: * Don't run QUERY or BASIC compiled programs from SPOOK. * Alternatively, if you're an MPEX customer, you can "hook" SPOOK to accept any line starting with a "%" as an MPEX command, which may be a %RUN (with arbitrary parameters), %PREP, UDC, or anything else. This kind of "hooking" can be done to any program, and MPEX users regularly hook EDITOR, TDP, QUAD, LISTDIR5, RJE, etc. to recognize MPEX commands. * In the case of QUERY, QUERY uses ;PARM= to indicate that output should be sent to the file called QSOUT. If you issue a file equation :FILE QSOUT=$STDLIST then even if QUERY is run with PARM=49, it'll still output stuff to $STDLIST (even though it think it's going to QSOUT). * Finally, you can :RUN SPOOK.PUB.SYS;PARM=-1 to start with. This will cause SPOOK to prompt you with ">()" (since ASCII character -1 is unprintable); all RUNs from within SPOOK will be done with a ;PARM= value one greater than the one which was passed to SPOOK -- in this case, -1+1, or 0! This way, you fool SPOOK into running all its sons with ;PARM=0, just like it should have in the first place. This, I think, can serve as a valuable lesson: BEWARE OF GALLOPING NIFTINESS! Before adding more and more baroque features to a program, consider that perhaps the simplest solution -- run everything with ;PARM=0, even if it means that if you run SPOOK from SPOOK, the prompt will still be "> " -- may be the easiest one to live with. Q: I'd like to let one of my COBOL subroutines save data from one call to another, but I don't want to use a huge chunk of working-storage or use an extra data segment. How can I do this? I've heard that the DL-DB area of the stack can be used for this purpose, but how can I access it from COBOL? A: There are two answers to your problem, the simple one and the complicated one. The simple one is to use "$CONTROL SUBPROGRAM" instead of "$CONTROL DYNAMIC" at the beginning of your procedure source file (see Appendix A of the COBOL Reference Manual). This makes all your variables "static", i.e. allocated relative to DB and thus retained from call to call, as opposed to "dynamic", which means that they're deallocated every time the procedure exits. However, since Connie Wright wants the Q&A column to be of decent length, I'll also give you the complicated answer. God knows, it might even come in handy if the $CONTROL SUBPROGRAM solution isn't sufficient (perhaps if you want to put the procedure in an SL, where "$CONTROL SUBPROGRAM" procedures aren't allowed). The stack of every process is laid out rather like this: ----------------------------------------------------------------- ... | global variables | subroutine-local variables | ... ----------------------------------------------------------------- <- DL DB Qi S Z -> Between DB and Qi (the initial location of the Q register), you'd keep your global variables; between Qi and S, the procedure-local variables. This requirement is enforced by the HP3000's architecture because every time a procedure is exited, all the local variables that were allocated for it above its Q register have their space deallocated. Next time the procedure is called, the local variables will be reallocated at new addresses, and will of course have new values. Everything below Qi -- including DB-Qi and DL-DB -- is not automatically changed by procedure calls and returns. The compilers allocate global variables between DB and Qi, so if you try to use that space, you'll interfere with them, but you can safely (fairly safely) put stuff between DL and DB, and unless you explicitly change it, it'll remain the same no matter how many procedures you call or exit. What IS the DL-DB area? It is just a part of the stack, containing memory locations belonging to your process (like DB-Qi or Qi-S). There are several rules, however, that you must follow to access this area: * The locations in this area have NEGATIVE addresses (since they're below DB and all stack addresses are DB-relative). This means that they can only be accessed directly using SPL, and you have to write special SPL interface procedures to access them from other languages. * By default, any program has less than 100 words of space available between DL and DB. If you want more (as apparently you do), you can run your program with ;DL=nnn to ensure that at least nnn words of DL-DB space will be available. Even better, you can use the DLSIZE intrinsic to dynamically expand and contract your DL-DB space: CALL INTRINSIC "DLSIZE" USING -1280, GIVING DLSIZE-GRANTED. I usually use DLSIZE instead of running my program with ;DL=, not just because it allows me to dynamically change the amount of space I have allocated, but also because it saves me embarrassment when I forget to :PREP my program with ;DL=. * Finally, although the system left the entire DL-DB area for "user applications", from MPE's point of view V/3000 and the like are also "user applications", and some of them are quite liberal with their DL-DB space usage. V/3000 is the big culprit, so I'd suggest that you avoid using the DL-DB area in programs that use V/3000. If you don't use V/3000, you should still stay clear of the area between about DL-16 and DL-1, which is used by SORT/MERGE, COBOL, and the FORTRAN formatter. So the point is that, once you've allocated enough space with ;DL= or DLSIZE, and have made sure that you're not colliding with anybody else (like V/3000), all you need to do is write a few small SPL procedures: $CONTROL NOSOURCE, SEGMENT=ADDR'WALDOES, SUBPROGRAM, USLINIT BEGIN PROCEDURE MOVEFROMADDR (BUFF, ADDR, LEN); VALUE ADDR; VALUE LEN; INTEGER ARRAY BUFF; INTEGER ADDR; INTEGER LEN; BEGIN INTEGER ARRAY RELATIVE'DB(*)=DB+0; MOVE BUFF:=RELATIVE'DB(ADDR),(LEN); END; PROCEDURE MOVETOADDR (ADDR, BUFF, LEN); VALUE ADDR; VALUE LEN; INTEGER ADDR; INTEGER ARRAY BUFF; INTEGER LEN; BEGIN INTEGER ARRAY RELATIVE'DB(*)=DB+0; MOVE RELATIVE'DB(ADDR):=BUFF,(LEN); END; END. Saying CALL "MOVETOADDR" USING -1023, MY-STUFF, 100. will move the first 100 words of MY-STUFF to the 100 locations starting with -1023; similarly, CALL "MOVEFROMADDR" USING MY-STUFF, -1023, 100. will move the 100 words starting with -1023 to MY-STUFF. In the above examples, "-1023" is the address of the DL-DB chunk you're using. It's a negative number like all DL-DB addresses, and it could either be a constant (if you always know that words -1023 through -924 are the ones that you'll use), or a variable. As you may have noticed, MOVETOADDR and MOVEFROMADDR move to and from any arbitrary address in the stack, with no concern as to whether it's between DL and DB or not. They are simply interfaces to SPL's arbitrary address handling capability; the COBOL program assigns the addresses, allocates the space, etc. Many such things that can "only be done in SPL" may easily be implemented in other languages by writing very simple SPL procedures that can be called from COBOL, FORTRAN, PASCAL, etc. programs. Q: How can I determine all the UDC files :SETCATALOG'd on my system? Also, can I group UDC files together to improve performance? A: Starting with T-MIT (I believe -- it may have been T-delta or something like that), HP allows a user with SM capability to say :SHOWCATALOG ;USER=@.@ to show all the UDCs files currently set on the system level. That means all the files set using ":SETCATALOG ;SYSTEM", something that any user could always do without SM using the :SHOWCATALOG command. How incredibly useful. What you want -- and quite reasonably, I might add -- can NOT easily be done with any HP command. It should be, but it can't. There's no justice in the world. Fortunately, this information is not so hard for you to get at yourself. All the information on who has what UDCs set is kept in COMMAND.PUB.SYS, an ordinary file with a fairly simple format (described, in its usual incredibly readable style, by the System Tables Manual in Chapter 15). Simply put, * Record 0 is a header. * Any other record with a 1 in its word #1 is a "user record", which contains the user and account for which a UDC file (or files) is set. The user name starts in word 2, the account name in 6; account-level UDCs have an "@" in the user field, and system-level UDCs have an "@" in both the user and account. * Word #0 of any "user record" contains the record number of the first "file record" for this user. Each file record's word #0 contains the record number of the next file record belonging to this user, until the user's last file record, which has a 0 in its word #0. Each file record contains the filename starting with word 2. So, all you have to do is read through COMMAND.PUB.SYS, finding each user record and traversing its linked list of file records. In fact, that's exactly what the following quickie SPL program does: $CONTROL NOSOURCE, USLINIT << )C( COPYWRONG 1985 BY VESOFT. NO RIGHTS RESERVED. >> BEGIN INTRINSIC FOPEN, FREADDIR, PRINT; INTEGER ARRAY REC(0:19); INTEGER FNUM; BYTE ARRAY FILE(0:35):="COMMAND.PUB.SYS "; DOUBLE RECNUM; FNUM:=FOPEN (FILE, 1); RECNUM:=1D; << 0th record is just a header >> FREADDIR (FNUM, REC, 20, RECNUM); WHILE = DO BEGIN IF REC(1)=1 THEN << user record >> BEGIN PRINT (REC(2), -16, 0); WHILE REC(0)<>0 DO BEGIN FREADDIR (FNUM, REC, 20, DOUBLE(REC(0))); PRINT (REC(2), -36, 0); END; PRINT (REC, 0, 0); << blank line >> END; RECNUM:=RECNUM+1D; FREADDIR (FNUM, REC, 20, RECNUM); END; END. For brevity's sake, I've omitted much of the prettifying, so all you'll see is the name of each user who has UDCs set, followed by the names of all his files, followed by a blank line. Clean it up as you will, and put it in some revered spot on your system (note: this printout will show the lockwords for any UDC files which were :SETCATALOGed filename/lockword). It's likely to come in handy. About performance: Run-time performance (i.e. how long it takes MPE to recognize and parse your UDC invocation) is affected solely by the number of UDCs (not UDC files) a particular user has set; in any event, UDC lookup and expansion are usually very fast. Performance at logon time can be rather poor because MPE must open and read every UDC file set up on the user's behalf. The order of entries in COMMAND.PUB.SYS doesn't matter; what does matter is the number of UDC files (very important, since there's an FOPEN for each one and FOPENs ain't cheap), the combined size of the UDC files, and the blocking factor of each. First, try merging several UDC files into one; after that, increase the blocking factor of each UDC file to minimize the number of I/Os necessary to read it. Q: I don't like the fact that during a system backup, you can't run most programs. True enough, if you :ALLOCATE a program before the backup, you'll be able to run it, but why is that necessary? It seems that the :RUN fails because it somehow has to update the program file -- why should it do that? :RUN looks like it ought to be a read-only operation. A: True enough, for the most part, :RUN need only read the program file -- get various control information (capabilities, maxdata, etc.) and bring the code segments into memory. However, for some very interesting optimization reasons, a little bit of information needs to be written to the file, too. The principle of HP memory management is that you can have more code and data than will fit into memory. Just because you have 2 Megabytes of memory doesn't mean that all the stacks, code segments, etc. must fit into 2 Meg; if memory is full and you still need more, HP's Virtual Memory Manager will make room by getting rid of a segment that's already in memory but hasn't been used in a long time. Then, if the segment you've just taken out of main memory is needed again later, it'll be brought in when needed and some other segment will be thrown out. Note that I said that an old segment would be "removed from memory". You've probably heard this as a segment being "swapped out" -- if you're going to take, for instance, a process's stack out of main memory, you can't just throw it out; you have to copy out to disk, so that when it's needed again, it can be brought back into main memory with exactly the same contents as it had when it was taken out of main memory. Whenever the space currently used by a data segment must be used by something else, the data segment must be written out to disk for future reference. Now, say that the least recently used segment in memory is not a data segment, but rather a code segment. It is a fundamental principle of HP's architecture that code segments are CONSTANT and unchangeable. If we need to re-use the space currently used by a code segment, we can just THROW IT OUT without writing its contents out to disk, since its contents are already stored on disk in the program file itself! To summarize, every segment that isn't currently kept in main memory must be stored somewhere on disk. * Data segments -- which can change -- have a special place on disk allotted to them (this is what the "virtual memory" you configure is for), and any time they're removed from main memory, they have to be saved in this place on disk. * Code segments, being constant, are more efficient: when you need to get them out of main memory, you don't have to write them out to disk, since their contents are already stored on disk in the program file. What does all this have to do with the question at hand? Consider for a moment what a code segment must contain: * First, it must have the actual machine code that belongs to the segment. This machine code, of course, never changes until you recompile the program. * Furthermore, it must also contain links to all the various other segments that are called from within it. Say an SL procedure called XYZ is called from the segment -- the segment must contain the segment number of the segment that contains XYZ and the location of XYZ in that segment. This is information that can not be deduced at compile time or :PREP time, but must be filled in at :RUN time (since that's when all SL procedure references are bound). So, the problem that HP faced is that each segment had to contain information (in a table called the STT, Segment Transfer Table) that was not known until the program was :RUN, and even then could change from one :RUN of the program to another! Now, the only thing that makes HP's efficient code segment handling plan work is that the code segment in memory must be EXACTLY like the code segment in the program file on disk, since the segment may later have to be read in from the program file. Thus, if the STT of every segment has to be initialized every time the program is :RUN (which it does) and if the STT is kept in the segment (which it is), the STT must be initialized in the program file as well in the in-memory code segment. So there's the explanation. Every time a program is loaded, the STTs of all the segments (and also some other things) must be initialized, and so the program file has to be updated. That's why the loader opens the program with read/write access, and when the file is being :STOREd, this kind of thing is forbidden. On the other hand, if the program is already being :RUN or has been :ALLOCATEd, then it's already been loaded, and the loader needn't re-update the STTs, and thus needn't open the file for read/write. Incidentally, have you ever wondered why program files must only have one extent? HP demands that the disk images of all code and data segments must be contiguous, so they could be read in one I/O. The only way to make sure that a code segment in a program file is contiguous is to make sure that the entire program file is contiguous, which means that it must have one extent. Q: What's the difference between $STDIN and $STDINX? A: Good question. An equally good question, incidentally, is: "WHY the difference between $STDIN and $STDINX?", but I'll get to that in a moment. One of the key points of the HP file system is that all sorts of files are accessed in pretty much the same way. There's no special intrinsic for writing to the line printer -- you just open a file on device class "LP"; similarly, the terminal is accessed using file system intrinsics, too, using the special files "$STDIN", "$STDINX", and "$STDLIST". Now, all disc, tape, etc. files have "end of file" conditions. A typical file read loop in your program might be: 10: READ FILE F IF END OF FILE ON F GOTO 20 process the record that was read from F GOTO 10 20: ... You read records from the file until you hit an end of file condition. Now, let's say that using a :FILE equations you've redirected file F to $STDIN -- instead of getting the data from disk, you want your program to read it from the terminal. You have to have some way of indicating an end of file condition that would look just like the end of a disk file. Now, the actual difference between $STDIN and $STDINX is only that: * On $STDIN, any record that starts with a ":" is an end of file. * On $STDINX, only records that start with ":EOD", ":EOF: ", or (in batch jobs) ":JOB", ":EOJ", or ":DATA" constitute an end of file. Based on this alone, it's pretty obvious that all on-line programs should read from $STDINX (which, of course, is NOT the default for FORTRAN, COBOL, etc.). After all, the last thing that you want to do is abort with a tombstone if a user accidentally enters a ":". In fact, I dare say that in a session environment, $STDIN is quite thoroughly useless, and $STDINX should have been the default and perhaps the only option available. If this is so, though, why does $STDIN even exist? Well, remember that in 1972, when the 3000 was first built, batch use was far more prominent than on-line use. A typical job stream then (as now) might look like this: !JOB JOE.SCHMOE !RUN FOOBAR 123 << all this stuff is being read from $STDIN >> 456 789 !RUN FROBOZZ XYZ123 << another program, also reading $STDIN >> !EOJ The Powers That Be decided that it would be nice if the "!RUN FROBOZZ" (which the :STREAM command translates into ":RUN FROBOZZ") terminated the previous program (if it was still running) as well as starting the next one. So, any command that started with a ":" would cause an EOF on $STDIN (if the previous program was still reading from $STDIN), and then would be executed as an ordinary MPE command. Similarly, if a :JOB, :EOJ, or :DATA was encountered in the input stream, an EOF would be triggered even on $STDINX. That way, if several job streams are submitted at once (in 1972, this was quite common when jobs were fed in using punch cards), it would never happen that one job's input requests would receive as input the commands of another job. So both the ":" termination of $STDIN and the :EOD/:EOF:/:JOB/:EOJ/ :DATA termination of $STDINX were done to make sure that MPE commands would not be inadvertently fed as input to a program that is expecting normal data. The distinction between $STDIN and $STDINX was invented because although it was felt that the ":" end-of-file condition would be more useful (which today is definitely not the case), some programs (e.g. EDITOR) needed to be able to accept lines starting with ":". So, I've told you the WHAT and the WHY. Now, all that's left is a bit of WARNING: * First of all, an end-of-file on $STDIN isn't just a temporary thing. If a program gets an EOF while reading $STDIN, all subsequent read requests by that program against $STDIN will also get an EOF. In other words, once a program gets an EOF on $STDIN, it won't be able to do any more $STDIN input; also, any program that gets an EOF on $STDINX won't be able to do any more $STDIN or $STDINX input. * Furthermore, if a program gets a $STDIN[X] EOF, ALL OTHER PROGRAMS IN THE PROCESS TREE WILL ALSO GET AN EOF. In other words, if you run SPOOK (which reads $STDIN) from within QEDIT or TDP (which read $STDINX) and enter a ":", SPOOK will terminate with an EOF. Whenever you run SPOOK again from the same execution of QEDIT/TDP, it will immediately get an EOF. To reset the EOF condition, you must get back to MPE and then re-run your program; OR, you may do a ":STREAM" command and immediately enter a ":" in response to the prompt. It turns out that :STREAM magically resets the EOF condition on $STDIN. * If you enter a ":EOF:" (followed by a space or carriage return), you get a "super-EOF" condition. Getting out to MPE will NOT fix it -- an end-of-file is caused on $STDIN and $STDINX of all processes, including MPE itself. As soon as you get out to MPE, you get automatically logged off; even hitting [BREAK] to get out to MPE will automatically abort the broken program and sign you off. * Finally, let me expose a couple of errors that have been floating around HP documentation: - In session mode, only ":EOD" and ":EOF:" cause an end-of-file on $STDINX. :JOB/:EOJ/:DATA/:BYE/:HELLO/etc. do NOT. - In job mode, ":EOD", ":EOF:", ":JOB", ":EOJ", and ":DATA" cause an end-of-file on $STDINX. :HELLO does NOT. - Some manuals say that if you read less than 4 characters from $STDINX, anything that starts with a ":" cause an EOF (since you can't enter an :EOD if less than 4 characters are being read). This is WRONG -- there's no way to enter an EOF if you're reading less than 4 characters from $STDINX. So, there it is, in all its dubious glory. If you've got a choice, always use $STDINX (at least in session mode). Q: Many of my programs need a lot of stack space for things like an internal sort or for V/3000 screen handling. What kind of performance penalty would I suffer if I ran them with a high maxdata -- 20,000 or 25,000 -- just to be safe? A: Your question brings up a very interesting point -- what does the ;MAXDATA= parameter do, after all? All the variables and arrays that your program uses are put in your "stack data segment". This is just a segment that's allocated on your behalf by the system; it can be up to 32640 (32768-128) words, of which some amount is always used by the system for things like open file information. Initially, the stack is made with room for all your global variables (which are allocated throughout the duration of the program) and some extra space for a small amount of local variables. As local variables are allocated (by a call to any procedure, of which SORT and V/3000 are only particularly conspicuous examples), the system checks to see if there's enough room for them. If not, it expands the stack to make room. This is where ;MAXDATA= comes in. If the expanded stack would be bigger than the size given as the MAXDATA, the program aborts with a stack overflow. There are two reasons for having a MAXDATA in the first place. One is that MAXDATA specifies the space allocated for the stack data segment in VIRTUAL MEMORY (not in main memory, but out on disk where the segment will go when it's swapped out). For all the tens of thousands of users out there who're running out of virtual memory, this is a good reason to keep MAXDATAs low. The other reason is that if you care about saving REAL MEMORY, specifying a low MAXDATA= can prohibit your stack from getting too big; when the stack overflow happens, you'll know that your program is a memory hog and maybe you'll be able to do something about it. What setting a low MAXDATA doesn't do is actually save you any real memory. If your program needs only 5,000 words, it'll use only 5,000 if its MAXDATA is 10,000 or 30,000; if your program needs 25,000 words, setting the MAXDATA lower than that will only make it abort. So, my advice is: If you get a stack overflow, push the MAXDATA way up -- to 20,000 or 30,000. If you really want to save memory, you have to manually optimize your program to minimize its memory usage. So, there's the answer to your immediate question; but, like many answers, it actually raises more questions than it solves. First of all: What's the easiest way of increasing a program file's MAXDATA? Well, you can re-:PREP it if you want to, but that requires the USL file to still be present and also takes quite a bit of time. Fortunately, a program's MAXDATA value is just one word in the program file's 0th record; various programs are available to change it, including some Contributed Library utilities (like MAXCAP) and VESOFT's MPEX/3000 %ALTFILE ;MAXDATA= command. If demanding more stack space dynamically increases stack size, does relinquishing space (say, by exiting a procedure that's allocated a lot of local variables) decrease stack size? No. If you want to actually decrease stack size, you'll have to call the ZSIZE system intrinsic, passing to it a stack size of 0 (CALL INTRINSIC "ZSIZE" USING 0). This'll cut down the actual memory usage of the program to only the amount of space that the program really needs. Watch out, though -- this operation may take more processor time than will be saved by the decreased memory use! OK, now that I've explained exactly what MAXDATA= does, what does the STACK= parameter do? Well, as I said, MPE initially allocates your stack data segment to be large enough to fit your global variables, but not much more. As local variables are allocated and the allocated stack size is exceeded, the stack will be dynamically expanded. Theoretically, a dynamic stack expansion can be a fairly inefficient thing (it might actually require a swap-out and swap-in of your stack). So, if you know that your program will use at least 10,000 words of stack space for most of its life, you can tell MPE to allocate that much space initially by specifying ;STACK=10000 on the :RUN or :PREP command. Practically, though, I don't think this is worth your time. Oh, yes, one minor confession: I lied a bit when I talked about V/3000 and SORT allocating local variables on the stack. SORT actually does stick all its stuff on top of the stack, above the Q register, but V/3000 puts it in the so-called "DB negative area", between DL and DB. This fact is actually quite irrelevant to our discussion; I only mention it so that those who know it won't think I'm trying to pull a fast one. Q: I've got a program that I run a lot in batch mode. It asks for several lines of input data, and then chugs along to produce a report. Sometimes, it'll find that some of the data is invalid, and then it terminates (without calling QUIT or anything -- just TERMINATE) immediately after seeing the incorrect piece of data. Unfortunately, MPE insists on reading the next piece of data (which is, of course, NOT a valid MPE command) as input for the command interpreter; MPE sees that it's an invalid command, and promptly aborts the job stream, which isn't what I want. In other words, if my job stream looks like !JOB FROBOZZ,USER.PROD !RUN MYPROG 01/01/86 A037 F999 127.44 !RUN MORESTUF !EOJ and MYPROG finds that the "A037" is an invalid input, MYPROG will then terminate, and MPE will try to execute "F999" as an MPE command. What can I do? I tried putting a "!CONTINUE" in front of the "!RUN", but it doesn't help. A: Your desire -- to have MPE throw out all the remaining MYPROG input until it sees a valid MPE command, like "!RUN MORESTUF" -- is perfectly legitimate. In fact, it's so legitimate that MPE actually does exactly what you want! But not all the time. The key distinction, amazing as it may seem, is whether the program being run HAS EVER FOPEN'ED $STDIN. In other words, has it ever called the FOPEN intrinsic with the appropriate filename or foptions that indicate either $STDIN or $STDINX? If it has, then when the program terminates, MPE will read lines from the job stream and throw them away until it finds one that starts with a ":" (the :STREAM facility translates the "!"s in your stream file into ":"s); then, it will start executing MPE commands from that point onward. On the other hand, if your program NEVER FOPENs $STDIN or $STDINX -- which means that it reads input using the READ or READX intrinsics -- then MPE will NOT throw away any invalid input, but will start trying to execute MPE commands starting with the first input line that wasn't read by the program (in your case, the "F999"). A good test of this is to run the following job stream: !JOB TESTER,USER.PROD;OUTCLASS=,1 !CONTINUE !FCOPY FROM=$STDIN;TO;SUBSET=0,1 TESTING ONE TWO THREE !SHOWTIME !CONTINUE !FCOPY TESTING ONE TWO THREE !SHOWTIME !EOJ The first run of FCOPY will read one record (that's what the ";SUBSET=0,1" is there for) from $STDIN and print it to $STDLIST. Then, FCOPY will terminate, since it was only asked to read one record; but, since the "FROM=$STDIN" made FCOPY do a FOPEN of $STDIN, the ONE, TWO, and THREE will be thrown away, and the job stream will continue executing starting with the :SHOWTIME. The second run of FCOPY reads TESTING as an FCOPY command; FCOPY sees that it's an invalid command, and will terminate (just like it does in the first case after reading one line). However, since in this run FCOPY has never FOPENed $STDIN (it usually reads its input by calling the READX intrinsic, which doesn't require an explicit $STDIN FOPEN), the "ONE", "TWO", and "THREE" won't be thrown away. MPE will read the "ONE", see that it's not a valid MPE command, and will abort the job stream. The "!CONTINUE" won't help, since it only affects the next command, which is "!FCOPY" -- since there is no "!CONTINUE" before the "ONE", an error reading the "ONE" will abort the stream. One thing you have to realize, by the way, is that some languages automatically FOPEN $STDIN or $STDINX on your behalf. FORTRAN, PASCAL, and BASIC, for instance, do -- I'm not sure about COBOL or RPG. SPL, on the other hand, doesn't, so if your program is written in SPL and you want to take advantage of this "throw away unread input" feature, you have to make sure that your program does the FOPEN itself. All it has to do is have a line like FOPEN (,%40); << foptions %40 means $STDIN >> at the beginning -- it can just throw away the result of the FOPEN and keep calling READ or READX. As long as the FOPEN is done, you'll have what you want. Q: Can you tell me what the following file codes are for: VREF, RJEPN, PCELL, PCCMP, RASTR, TEPES, TEPEL, SAMPL, MPEDL, TSR, TSD, DSTOR, TCODE, RCODE, ICODE, MDIST, MTEXT, VCSF, TTYPE, TVFC, NCONF, NTRAC, NLOG, and MIDAS. Perhaps some of them are created by products we don't have. A: Often, when they have nothing better to do, HP engineers invent new filecodes to intellectually stimulate and confuse the user community. This phenomenon, known technically to psychiatrists as "cybernetic coding release", is thought to be a beneficial thing, something that safely channels frustrations that would otherwise go towards things like putting unneeded SUDDENDEATHs into MPE code or purging random files from your system. I called up my friend Wilfred Harrison at HP -- an unimpeachable source who would never lead me astray -- and he told me the following: * VREF stands for View REFormat specification files, made by HP's REFSPEC utility. * RJEPN are RJE PuNch files, created by Remote Job Entry/3000. * ICODE, RCODE, and TCODE files are created by HP's RAPID/3000 system (INFORM, REPORT, and TRANSACT respectively), which may or may not be very fast on your system. * PCELL stands for Padded CELL, and denotes a file which contains various programs that have gone crazy and had to be locked up for their own good. VESOFT's MPEX/3000 is indispensable for handling these files, since with it you can say: %LISTF @.@.@(CODE="PCELL") and find all those screwed-up files before they break loose and do something nasty. * SAMPL files are SAMPLer/3000 (now known as APS/3000) log files. * MIDAS files turn to gold any computer system on which they're present. Just try: :BUILD POOF;CODE=MIDAS These are, unfortunately, the only ones I managed to find out about. As I hear of more, I'll be sure to let you know. Q: Once upon a time, I was using EDITOR and wanted to look at one of my files. I said /TEXT FFF and EDITOR replied *23*FAILURE TO OPEN TEXT FILE (0) END OF FILE (FSERR 0) Naturally, I was rather taken aback; FFF was supposed to be a program of a thousand-odd lines. I entered :LISTF FFF,2 and up comes ACCOUNT= VESOFT GROUP= DEV FILENAME CODE ------------LOGICAL RECORD----------- ----SPACE---- SIZE TYP EOF LIMIT R/B SECTORS #X MX FFF 80B FA 1076 1076 3 360 8 8 Most distressing indeed! Now, to add to my confusion, the operator at this point warmstarted the system, and when it came back up, my file was back! I /TEXTed it in using EDITOR, and there it was, all 1076 lines of it. Naturally, I'm glad to have my file back, but tell me: what's going on here? A: As best I can tell, you have fallen victim to the infamous INVISIBLE TEMPORARY FILE. Actually, it's not quite invisible, but the way HP designed the system, it's pretty hard to see if you're not looking for it. When you build a file, you have the choice of two DOMAINS in which to build it. You can create it as a PERMANENT file, which means it stays around forever (or at least until you purge it); most files, including your FFF program, are like that. You can also build a file as a TEMPORARY file, which means that the file is automatically deleted when you log off; also, different sessions can have temporary files with the same names without any problem (since each session can only access its own temporary files). Say your program needs to build a file that it wants to :STREAM; if it built a permanent file called, say, STRMFL.PUB.DEV, then it might conflict with the same file built at the same time by another user who's running this program. If it builds STRMFL as a temporary file, however, no such problem will arise. As an additional bonus, if the program doesn't delete the file itself, it'll automatically vanish (and have its space freed for future use) when the session logs off. So, both permanent and temporary files have their place, although most of the files you deal with are permanent files (after all, the computer's job is to store data more or less permanently). The problem that arises is: when you open a file called XYZ, does this mean the PERMANENT file XYZ or the TEMPORARY file XYZ? There may, after all, be both a permanent and a temporary file with the same name. Therein lies the root of your problem. MPE gives you three ways to open an already-existing file: * You can demand that MPE open the PERMANENT file with a given filename. This corresponds to the :FILE command option ;OLD, and to an FOPEN call with bits .(14:2) of FOPTIONS set to 1. * You can demand that MPE open the TEMPORARY file with a given filename. This corresponds to :FILE ;OLDTEMP and FOPTIONS.(14:2) = 2. * Or, you can tell MPE to try to open the TEMPORARY file with this name, or, if no such file exists, to open the PERMANENT file with this name. Now, most system programs, including EDITOR, FCOPY, and MPE's :STREAM command, use the third option -- try to open the temporary file first, and the try the permanent file. It's convenient for them to do it, since then the same /TEXT command or :STREAM command can work on both temporary and permanent files -- they don't need to have special keywords like /TEXT FFF,TEMP or :STREAM FFF,TEMP. Unfortunately, it can also be rather confusing, since the :LISTF command only prints permanent files and the :PURGE command by default purges only permanent files. So, that's my diagnosis of your troubles. You must have somehow gotten a TEMPORARY file called FFF, which didn't have any records in it. When you did a /TEXT FFF in EDITOR, EDITOR saw this temporary file, and gave you an error because it -- the temporary file -- was empty. However, :LISTF showed you the permanent file, which wasn't empty at all. Fortunately for you, the system went down and all your session's temporary files perished. When you signed back on, the temporary file wasn't there and the /TEXT once again accessed the permanent file. So, the lesson: be wary of the INVISIBLE TEMPORARY FILE. If any eerie things like this happen to you, run LISTEQ5 or do a :LISTFTEMP to see whether there are any temporary files "shadowing" your permanent files. (While you're at it, you might also check :LISTEQ to see whether there are any improper file equations for the file you're trying to open. But that is another story.) And, above all, remember: things are not always as they seem. Q: I've heard that HP's new Spectrum machine isn't a stack architecture machine like the 3000. If this is so, what's going to happen to all the variables and stuff that I keep in my stack? Will I still be able to write recursive procedures, which require a stack to operate? What will they use instead of a stack. A: Unfortunately, labels like "stack architecture", "fourth- generation language", "relational database", and such often obscure far more than they clarify. When HP's says that their new Spectrum (or Series 950, or Precision Architecture, or whatever they're calling it this week) is not a stack architecture machine, they mean something quite different from what one might guess. A stack is a data structure onto which you can push objects and then pop them in reverse order. In other words, it's not like a variable, storing into which overwrites the old value, or like an IMAGE master dataset, from which data can be retrieved by its key rather than by its order of insertion. A stack is useful for situations in which you want to do something, and, when you're done with it, return to the thing you were doing immediately before that. What are good applications for a stack? Well, one that was recognized very early was for saving RETURN STATE INFORMATION. Say that instruction 175 in subroutine A decides to call subroutine B. When subroutine B is done, you want to return back to instruction 176; but, how is subroutine B to know that's where you want to go? Early machines would save the return instruction address in a special register, and the "RETURN FROM SUBROUTINE" instruction would branch to the address stored in that register. But, there was only one register -- when subroutine B called subroutine C, the return address for the A-to-B call would be forgotten, and the return from subroutine B would fail. Other computers, like the HP 2100, were wiser -- they stored the return address of a subroutine IN A MEMORY LOCATION AT THE FRONT OF SUBROUTINE BEING RETURNED FROM. In other words, if instruction 175 in subroutine A would call subroutine B, whose first instruction is 305, the return address (176) would be saved at location 304. Then, when subroutine B called subroutine C, the C-to-B return address would be saved at the header of subroutine C, thus not overwriting the B-to-A return address! A return-from-subroutine instruction would just branch to the address stored in the header of the routine that's doing the return. However, the most general and the most convenient method for keeping track of return information has been found to be a STACK. Every subroutine call would push the return address (as well as other information, like the processor status word, various registers, etc.) onto the stack; every subroutine exit would pop an address from the stack and branch to the popped address. Clean, simple, powerful -- this way even RECURSIVE routines, in which the same routine can call itself many times in a loop, would work, as long as you had room on your stack. Another, related use for stack structures has historically been the keeping of subroutine parameters and subroutine-local variables. Note the similarity with return information -- as a new routine is called, space is allocated on the stack for its local data; when it exits, the local information should be deallocated. Furthermore, since subroutines are always exited in reverse of the order in which they were entered, the stack is a perfect structure for keeping this information. Finally, arithmetic expressions (like A*B+C*D) have often been evaluated using a stack. A would be pushed onto the stacked, then B, then the top two stack elements would be multiplied, yielding A*B; then, C and D would be pushed, the top two stack elements would again be multiplied (= C*D) and then the new top two stack elements (A*B and C*D) would be added to yield the procedure result. Just like on your HP calculator, expressions on the HP3000 are evaluated by pushing operands onto the stack and then using the operators to pop them and leave the results in their place. These three things are the primary uses of the data structure called a "stack" on the HP3000 and other machines. However, the overwhelming majority of both STACK-ARCHITECTURE and the opposite, REGISTER-ARCHITECTURE machines, use stacks for the first two purposes (subroutine return information and subroutine parameters and local variables). Stacks are perfectly tailored for these purposes, and everything from your PC to a VAX to HP's new architecture uses them. You'll still be able to do all the recursion you want, because return addresses will always be kept in a stack, as is needed for recursion; or, conversely, HP was forced to keep all the returned information in a stack architecture, since without it useful mechanisms such as recursion would be unavailable. The distinction between STACK- and REGISTER-ARCHITECTURE machines comes mainly in the third area: where the temporary data used in evaluating expressions is put. In the HP3000 it's pushed on the stack by some instructions (e.g. LOAD) and popped by others (e.g. ADD and STOR). In register machines (like most modern machines, such as the 8080, 68000, and the HP3000/950), the temporary data is put in special high-speed machine registers. In other words, instead of pushing A, B, C, and D on the stack in order to evaluate A*B+C*D, the 950 might load A into register 0, B into register 1, C into register 2, D into register 3, multiply registers 0 and 1 (leaving the result in 0), multiply registers 2 and 3 (leaving the result in 2), and then add registers 0 and 2 to yield the result of the expression. This is actually a VERY GOOD THING, since registers are usually much faster to access than a stack, and a large number of registers can substantially cut down on costly memory accesses. HP may or may not do some other things to use its registers better. For instance, it might put some procedure parameters or frequently used local variables into registers, again striving to minimize memory (non-register) accesses. Of course, on all computers there's a limited number of registers (usually from 8 to 32), so anything that won't fit in registers will be kept in a memory stack (including perhaps even the intermediate values of complicated expressions). The important thing, though, is that it's all done behind your back -- the REGISTER/STACK ARCHITECTURE distinction here is only relevant for performance considerations. So that's what'll happen to stack data structures in HP's new machines, and that's why you'll still have recursion, dynamic allocation of subroutine-local storage, and all the other things that are so good about stack architectures. What about the other stuff HP keeps in the stack, like global variables? Well, there you're the victim of an unfortunate semantic muddle in the HP3000. What HP calls a "stack", many other computers call your "data segment" or "data space". Global variables aren't really kept in a "stack", in that they're never pushed or popped. Rather, they're stored in a data segment that happens to also contain your return address/local variable stack. In the 950, you'll still have your main data segment (how could you not?); in fact, it might still have the return/local stack stored inside it. HP probably won't call it a stack, but that's only a name; the important thing is that it will support all the operations you've grown to know and love, and probably (cross your fingers) do them a lot faster, to boot. So, just as the moral of the last story was "don't believe everything you read", the moral of this one is "things are seldom as they seem." Or, perhaps, "put not your trust in buzzwords." Q: I hear that HP's new 3000 Series 950s aren't going to support SPL. What about all my programs and vendors' programs that are written in SPL? Will we have to re-write them all? What if we've lost the sources? A: HP has built a well-deserved reputation for compatibility, both on the source-code and object-code levels. You can write a program, in SPL or in any other language and have it run on anything from a Series 30 to a Series 70 without any recompilation. The 950 has a radically different instruction set (RISC = Radically different Instruction Set Computer...), so you wouldn't be able to feed your 3000 program files, written in any language, to the CPU, just like you couldn't feed VAX or IBM programs to the 950's CPU. However, in addition to providing FORTRAN, PASCAL, COBOL, BASIC, and RPG compilers that will generate code that the 950 CPU can handle, HP also allows you to run 3000 code that will be EMULATED by the 950. In other words, any code (except for some privileged operations) that the 3000 can execute DIRECTLY, the 950 can execute in EMULATION MODE, essentially a very fast (but sometimes not fast enough) interpreter. They don't call it a Really Intelligent Super-Computer for nothing. Consider, for a moment, the SPL compiler. Forget the aura of glory and mysticism that comes of our calling it a "compiler". It's just a program, a program that takes as input one data file (which happens to contain SPL source code) and generates as output another data file (which, it turns out, contains 3000 machine code in USL format). In fact, SPL doesn't even use privileged mode; it's just another HP3000 program, written in the HP3000 instruction set, that reads an input file and writes an output file. The point, you see, is just like you can run your 3000 application system on the 950 without recompiling it, so you can run the SPL compiler on the 950. And, the USL file that SPL.PUB.SYS outputs -- on the 3000 or the 950 -- will contain 3000 code, which can also be run on the 950. The only thing you have to realize is that both SPL.PUB.SYS and the program that it outputs will have to be run in Compatibility Mode; however, they can be run just as easily as any Compatibility Mode program written in COBOL, PASCAL, FORTRAN, or what have you. After all, it would be a Really Idiotic and Silly Computer that would single out only SPL-generated code for unfavorable treatment while allowing the same instructions generated by COBOL, PASCAL, or FORTRAN compilers to run just fine. Of course, since HP's SPL compiler hasn't been re-written to generate Native Mode code, its output can only be run in the relatively slower Compatibility Mode. If you want your SPL programs to run in Native Mode, you'll have to use SPLash!, the new third-party compiler that the people at Software Research Northwest (super-programmers like Steve Cooper, Jason Goertz, Wayne Holt, Stan Sieler, and Jacques Van Damme) are putting out -- write to them at: PO Box 16348 Seattle WA 98116 USA. If you BOTH want it to run in Native Mode AND don't want to use SPLash!, then you'll have to re-write your programs in some other language, preferably C or PASCAL (if you really want to have fun, write them in RPG). But, remember that this is essentially only a performance improvement; unless you use PM, your SPL programs will run intact in Compatibility Mode on the 950, with or without recompilation. I'd guess that for starters, most HP users, vendors, and HP itself will keep much of their SPL code in SPL and run it in Compatibility Mode, if need be fixing some of the spots that use PM. Then, over several years (depending on how important performance is and how good a job HP does in making its Emulation Mode fast), most of the code will migrate into Native Mode. But, I predict that ten years from now there'll still be code on Series 950's that runs in SPL, and nobody will be at all the worse for it. And that's a Rather Insightful Soothsayer's Cerebration, if I may say so myself. Q: I wanted to redirect the output of my program to a disc file, so I said: :RUN DAD;STDLIST=LFILE,NEW However, DAD creates several son processes, all of whose output I want to send to LFILE as well. What happened was that only one of the processes' output was put in LFILE, and the rest went into the bit bucket. What's up? A: What you've run into here is an artifact of the way the 3000 takes care of the ;STDLIST= parameter -- a restriction that makes the ";STDLIST=xxx,NEW" feature far less useful that one might think. Let's say that you have three processes -- DAD, KID1, and KID2. DAD is run with $STDLIST redirected to some file, and it then creates KID1 and KID2. When MPE opens DAD's $STDLIST file (MPE always opens a $STDIN file and a $STDLIST file on behalf of each process), it will open the file LFILE as a NEW file. Remember what happens when you open a file as NEW -- the system won't look for the file in the temporary or permanent file directory, and in fact won't even put the file in the directory until you close it. If you FCLOSE it mode 1, it'll save it as a permanent file; if you FCLOSE it mode 2 (as ;STDLIST=xxx,NEW does), it'll save it as a temporary file; if you FCLOSE it mode 0 or 4, it won't save it at all; but IN ANY CASE, WHEN YOU OPEN A FILE AS NEW, IT WON'T BE PUT IN THE DIRECTORY UNTIL YOU FCLOSE IT. So DAD's now running, with its $STDLIST redirected to the file LFILE -- a file that has a file label, has disc space, but DOESN'T have a directory entry (either permanent or temporary). Now, KID1 is created. MPE's smart enough to realize that KID1 should inherit DAD's $STDLIST, so when it creates KID1, it's essentially as if it was also run with ";STDLIST=LFILE,NEW". However, IT'S NOT THE SAME 'LFILE'! Since KID1 is also told to open LFILE as new, it'll create its own new file, also called LFILE, which won't cause any sort of conflict since DAD's LFILE hasn't been put into a directory yet. In fact, even if KID1 WANTED to open DAD's LFILE, it couldn't, since DAD's LFILE isn't in any directory. The same happens to KID2. It also opens a new file called LFILE -- instead of what you wanted, which is a single LFILE with all three processes' outputs, you get three LFILEes, one for each process. You see, the system took you at your word when you said that you wanted the $STDLIST to be a NEW file called LFILE, and opened each LFILE as a NEW file. So, as long as DAD, KID1, and KID2 are running, they're writing all their output to their own $STDLIST files. Now, say KID1 dies; at THIS point, its LFILE file is closed, and put into the temporary file directory. When KID2 and DAD die, and THEIR list files are closed, the system will also try to save them as temporary files, but since a temporary file called LFILE (from KID1) already exists, the saves will fail. Thus, when the entire process structure is done, the file LFILE will contain only KID1's output. So, that's what's causing your problem. The solution? Well, recall that the root of all your woes is that the system tried to open all the $STDLISTs as NEW files, which means that there was no chance for KID1 or KID2 to find DAD's $STDLIST. All you need to do is build LFILE as a temporary (or permanent) file BEFORE running DAD, and NOT specify the ",NEW" in the ;STDLIST= keyword. In other words, simply say :BUILD LFILE;REC=-250,,V,ASCII;CCTL :RUN DAD;STDLIST=LFILE to make LFILE a permanent file, or :BUILD LFILE;TEMP;REC=-250,,V,ASCII;CCTL :FILE LFILE,OLDTEMP :RUN DAD;STDLIST=*LFILE :RESET LFILE to make it a temporary file. Incidentally, this problem is NOT CONFINED TO PROGRAMS THAT DO PROCESS HANDLING! Exactly the same thing will happen if your program explicitly FOPENs $STDLIST -- if you then do I/O both to your normal $STDLIST (using, say, the PRINT intrinsic, or by calling the COMMAND intrinsic with a command like :SHOWTIME or :LISTF that outputs stuff to $STDLIST) and the FOPENed file, then the output to one of these files will get lost! This can be very bad for programs written in languages like FORTRAN, which always do their output using FWRITEs to an explicitly FOPENed $STDLIST file, rather than using the PRINT intrinsic to output to the system-opened $STDLIST. In conclusion, the ";STDLIST=xxx,NEW" option on the :RUN command isn't as good as it might seem. It'll only work right for you if the process does all its output to the system-opened $STDLIST, and doesn't run any sons or explicitly open $STDLIST itself. For safety's sake, I recommend that you always build the output file first and use ";STDLIST=xxx" without the ",NEW" option. Q: What are primary and secondary DB? I've heard a lot about them, and my SPL compiles all show "PRIMARY DB=%xxx; SECONDARY DB=%yyy" at the end, but I'm not sure what they are. I thought every process had only one DB register. A: The HP3000 is a 16-bit machine, with all (well, almost all) of its instructions encoded in 16 bits. For instance, a "LOAD DB+123" instruction -- which pushes the value at DB+123 onto the stack -- is represented as the 16-bit word %041173. Now, these 16 bits must contain a lot of information. They must indicate both the OPERATION to be performed (LOAD), and the PARAMETER on which it is to be executed (DB+123). What's more, since you might also say "LOAD Q+7" or "LOAD S-10", the PARAMETER must include the ADDRESSING MODE (e.g. DB+, Q+, or S-) and the ADDRESS VALUE (123). It is quite a challenge to compress all this information into a single 16-bit instruction. Consider the way the LOAD instruction is laid out (you can find this in your MPE V software pocket guide): bits 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 OPCODE(04) X I P ---------address--------- * The first four bits (04 octal, 0100 binary) indicate this to be a LOAD instruction; * Bit 4 indicates whether or not this LOAD uses an index register (used for array indexing); * Bit 5 indicates whether the LOAD should load the value at the given address or the value POINTED to by the value address (more about this later); * Bit 6 indicates whether it should load data from your data segment or your code segment; * And bits 7 through 15 contain further information about what address you want to load from. The address that you can load from is restricted to 9 bits! What's more, we still haven't determined whether this is a LOAD DB+ instruction (used to load global variables), a LOAD Q- (load a procedure parameter), LOAD Q+ (load a procedure local variable), or a LOAD S- (load an expression temporary value). All this must also be encoded in these 9 bits, while still leaving room for indicating exactly what address is to be loaded from! Thus, the HP3000 is not just restricted to a 16-bit address space; even the 64K bytes that you have in your stack can't be directly accessed because no instruction can access the entire address space. Back to the LOAD command, the full format of those 9 bits that contain the address is: * If P=0 (we're loading VARIABLES from our STACK), and bit 7=0, this is a LOAD DB+ instruction. The DB-relative address is given by bits 8 through 15, so we can directly load any value stored in DB+0 through DB+255. * If P=0, bit 7=1, and bit 8=0, this is a LOAD Q+ instruction. The Q-relative address is stored in bits 9 through 15, so we can load Q+0 through Q+127. * If P=0, bit 7=1, bit 8=1, and bit 9=0, this is a LOAD Q- instruction, with the address in bits 10 through 15, allowing us to get at Q-0 through Q-63. * If P=0, bit 7=1, bit 8=1, and bit 9=1, this is a LOAD S- instruction, the address is in bits 10 through 15, and we can access S-0 through S-63. * If P=1 (we're loading CONSTANTS from our CODE SEGMENT), then this is a LOAD P+ if bit 7=0 and a LOAD P- if bit 7=1. THIS is the key to the distinction between primary DB and secondary DB (as well as many other confusions and conundrums). All program global variables are allocated (at least in SPL) with DB-relative addresses (since DB always stays constant, unlike Q and S, which move around). If we allocated all our global variables contiguously from DB+0 up, we'd quickly run out of our 256 directly accessible words. What the compilers do instead is they allocate all SIMPLE VARIABLES (non-arrays) contiguously in the DB+ area; but, for every array, they allocate a 1-WORD POINTER CELL, which is initialized to point to the actual array data. Any access to an array will then go INDIRECTLY through the pointer cell, rather than specifying the array address directly in the LOAD (or STOR, etc.) instruction. So, if we have a program like $CONTROL NOSOURCE, USLINIT BEGIN INTEGER I; DOUBLE J; INTEGER K; INTEGER ARRAY L(0:9999); INTEGER M; ... END. then I will be put at DB+0 J at DB+1 and DB+2 (since it's still a simple variable) K at DB+3 a POINTER to L at DB+4 M at DB+5 When the program is run, the pointer to L will be initialized to the actual address of L (which now no longer has to be between DB+0 and DB+255, since it's accessed with a full 16-bit pointer), and code like "K:=L;" (where saying L without an index means getting the 0th element of L) will compile into LOAD DB+4,I << load the value POINTED to by DB+4 >> STOR DB+5 << store DIRECTLY into DB+5 >> Of course, fetching the 0th element of L will be slower than fetching, say, K, or M, because two memory addresses will be needed -- one to DB+4 and one to the location pointed to by DB+4. However, the quick, direct access mode would only allow us to access 256 words, not quite enough in today's world. To summarize, only locations from DB+0 through DB+255 can be accessed directly using a LOAD instruction, so compilers are careful to allocate these locations to simple variables and to array pointers. The area used by these variables and pointers is thus called PRIMARY DB. Array pointers point to array data, which can be anywhere in the stack, at any address, above or below 255. All this array data is called SECONDARY DB. PRIMARY DB can easily overflow if you have more than 256 words' worth of simple global variables and global array pointers; SECONDARY DB, on the other hand, can go all the way up to your maximum 30,000-odd words. There is only one DB register -- primary DB is accessed by DIRECT DB-relative addressing, whereas secondary DB is accessed by INDIRECT addressing, using primary DB values as pointers to the secondary DB data. Q: I'm streaming a job stream which has a long line inside it: !JOB ... !RUN MYPROG << data containing about 100 characters >> ... I built this file in EDITOR with LENGTH=132, RIGHT=132; the stream file looks great, with the full 100 characters in it. But, when I submit it, MYPROG sees only the first 80 characters! Help me! A: I'll bet you that, when you run SYSINFO (a really great program that you can usually find in PRV.TELESUP), and type )IO 10 << or whatever your :STREAMS device is configured at >> you'll see something like this: LDN DRT UN CH TY ST TYPE SPEED RECL DRIVER CLASSES 10 38 0 0 24 0 (7970) 40 HIOTAPE0 JOBTAPE See that RECL number? Its value is 40, and it means "record length". The record length of your :STREAMS device is configured at 40 words, or 80 bytes. Now the :STREAMS device is a mighty perverse sort of thing. If you can believe it, it is a "virtual device" emulating a tape drive! The practical effect of this, though, is that all the "input spool files" that the system builds -- these are the temporary disc files that contain all the input for any :STREAMed jobs -- will be written with a maximum record length of 80 bytes (40 words). The solution is fairly simple. Reconfigure your device 10 to have record size 128 words (256 bytes), or some such figure; then, that will become the new maximum :STREAM file line length. In "Configuring the MPE Streaming Facility", p. 5-19 of the "System Operation and Resource Management Reference Manual" (JUL 84), you are told to configure it as a TAPE DRIVER or a CARD READER, specifying parameters that "duplicate the values for an actual card reader or tape drive". Well, the recommended record with for a card reader is 40 words (see Table 7-12); for a tape driver, it's 128 words (Table 7-13). That's probably why you had a maximum record width of 40 words on your system in the first place -- whoever configured it chose the maximum record width value for a card reader or a tape drive. This, incidentally, should indicate WHY the :STREAMS device emulates a tape drive. Actually, it emulates a tape drive or a card reader; the latter, especially, was the way that you submitted ALL jobs to a computer in the bad old days of punch cards. As interactive devices (i.e. terminals) became more and more popular, people started wanting to submit jobs from already logged-on sessions. What the :STREAM command actually does is it "punches" a virtual card deck -- which is actually an input spool file -- which is then "read" by the :STREAMS device. Thus, to be precise, your mistake was a rather obvious one. You tried to punch 100 characters onto a punch cards that's only 80 characters wide! Haven't you ever used a REAL computer before? Q: I know that when you do a chained read on an IMAGE database, IMAGE can sometimes return to you an error 18 (BROKEN CHAIN) even though the database is structurally sound. I understand that this happens when somebody deletes the record that immediately follows the one I just read -- when I go to get the next record, IMAGE complains. However, recently I've been getting a very strange condition. I do a DBFIND and then some DBGET mode 5s; but, instead of getting me the records in the chain (which is what I want) or giving me an error 18 (which I can handle), it gets me records from an entirely different chain, with an entirely different key! What's happening? A: Each IMAGE chain is a DOUBLY-LINKED LIST. Every record in the chain points to the NEXT record in the chain and also to the PREVIOUS record in the chain. Say that you do a DBFIND with key "FOO" and then a DBGET mode 5. IMAGE knows: you've just gotten record number 17 -- the next record in the chain is number 27, and no previous record exists, since this is the first record in the chain. You process the record, and then go to DBGET the next one. But, hold on! While you were messing around with record 17, somebody else got to record 27 and deleted it. What's more, they've also DBPUT a new record, with key "BAR", that just happened to drop into the newly-vacated record number 27. Remember, any physical record in a detail dataset can end up having any key; it's simply a matter of what record slot was free at the time a DBPUT is done. In fact, it's quite likely that if you do a DBDELETE and then a DBPUT, the new record will be put into the same physical record slot that the just-deleted record occupied. Now, you do a DBGET mode 5. IMAGE says: "I know what the next record in the chain is -- it's record 27". Actually, that's what the next record in the chain WAS when the previous DBGET was done, but IMAGE doesn't realize it's changed. So, it gets record 27, unaware that the record now belongs to a different chain! Now, don't you start shouting "Eureka!" IMAGE's not THAT dumb. Sure, it gets record 27, BUT IT DOESN'T JUST RETURN IT TO YOU. First, it checks for precisely the circumstance that I just described -- that the record now belongs to a different chain. How can it do this? Well, it could save the current key value -- the one that we passed to DBFIND -- and then check every record it DBGETs (mode 5) to make sure that it has that key value. However, the key could be hundreds or even thousands of bytes long; IMAGE doesn't want to have to use all that memory, especially since there could be hundreds of users simultaneously accessing the database, each with his own key. What does IMAGE do? Well, when it gets record 27, it looks at the just-gotten record's PREVIOUS RECORD NUMBER. In a properly-structured doubly-linked list, the previous record number of the next record must point to the current record, right? Since record 27 was pointed to by the "next record number" field of record 17, the "previous record number" field of record 27 must point back to record 17. Now, record 17 is part of chain "FOO"; the only way record 27 can point back to record 17 is if it's part of the same chain. If record 27 has been deleted and a record with key "BAR" has been added in its place, then the new record 27 will point back to the previous entry in chain "BAR" -- not record 17. So, IMAGE checks the previous record number of the next record; if it doesn't point back to the current record, you get an IMAGE error 18. Not very nice, but certainly better than getting a record with the wrong key. (Note: An IMAGE error 18 can mean either a "true broken chain", which means that the links in the database are actually WRONG and the database is thus partially corrupted; or, it could mean -- as it does in this case -- a "temporary broken chain" condition, in which no real data corruption exists.) However, let's say that you read record 17 and then somebody else deletes BOTH records 17 AND 27, and then adds two more records BELONGING TO THE SAME CHAIN that just happen to fall back into records 17 and 27. Now, it's quite possible that the new record 27 points back to the new record 17 -- after all, they're on the same chain. This isn't the chain that the old record 17 was on, but IMAGE doesn't know this. All it knows is that the doubly-linked list is sound -- record 17 points to record 27, and record 27 points back to record 17. Since it doesn't check the keys, it doesn't detect the "broken chain" condition, so you get condition code 0 (all's well) and a record with key "BAR", even though you THOUGHT you were reading down the chain for key "FOO". If this strikes you as improbable, you're right. It's unlikely that, just as you're processing one record, somebody deletes both that record and the next one in the chain, and then adds new records with the same key that happen to fall into the same newly-vacated record slots. The very fact that it's improbable convinced the authors of IMAGE that they shouldn't bother checking for it; but one shouldn't confuse the IMPROBABLE with the IMPOSSIBLE. In fact, there's yet another circumstance in which a DBGET mode 5 could get you a record from the wrong chain. Say that you just did a DBFIND and are about to do your FIRST mode 5 DBGET. All that IMAGE knows is that the master record points to record 17, so that's the record it should get. But in the meantime (after the DBFIND but before the DBGET), somebody deleted record 17 and then added a new record (with a different key) that happened to fall into the same record 17. Now IMAGE goes to get record 17, and does its usual "previous record of next record must point to current record" check. But, in case of the first record in a chain, this check is somewhat different; the previous record pointer of the first record in a chain is always 0. When record 17 belonged to our "FOO" chain, this was true; however, it's still true even if record 17 is the first record of the "BAR" chain! Somebody deleted the old record and added the new record, but the previous record number is still 0, as it should be. There's no way (short of checking the key, which IMAGE doesn't do) for IMAGE to find out that it's now on the wrong chain. To summarize, there are two conditions in which IMAGE may return you the WRONG RECORD when you're doing a mode 5 DBGET: * If BOTH the current record and the next record in the chain have been deleted (between the previous DBGET and the current one), AND new records belonging to the same chain were put in their place. * If this is your first mode 5 DBGET after a DBFIND, and the first record in the chain has been deleted and a new record has been added in its place (as long as the new record is the first record of its own chain). In these cases, IMAGE won't signal a broken chain error; it'll just pretend that everything's OK, but return a record from the wrong chain. What's more, all the subsequent DBGET mode 5s will keep on following this erroneous chain. There are two possible solutions to this problem: * You might lock the dataset or at least the chain that you're traversing to make sure that nobody deletes any of its records while you're in the middle of reading it. It doesn't matter that you're only reading the chain, not modifying anything; somebody else's modifications might screw up your reads. * Or, you can keep track of the "current key" that you expect all the gotten records to have -- this is the same key you passed to the DBFIND -- and check each newly-gotten record to make sure that it has that key. If it doesn't, you should treat the situation the same way that you would a "broken chain" error -- either print an error message and abort (hoping that this problem doesn't happen very often) or going back to the beginning of the chain with a DBFIND and re-reading the entire chain from the beginning. This requires a great deal of extra logic (especially since you'll now be reading some records twice -- once on the first pass, and once on the second pass after the "broken chain" was detected). However, if you can't live with the DBLOCKs and aren't willing to just print an error message, you have to do this. Q: I'm running a program with its $STDIN redirected to a message file. Now, although my message file has variable length records (in fact, it seems that all message files have variable length records), the program seems to be getting all its input records with trailing blanks! What gives? A: Variable record length files are rather nifty things. I use them all the time for my job streams -- after all, a job stream is supposed to emulate a terminal, which is a variable-length input device. If you use a fixed record length file for a job stream, then all the programs that the stream runs will see their input with trailing blanks; some programs can't handle this. This is, of course, also the reason that you make your ;STDIN= file have variable length records -- if you're going to emulate the terminal, you might as well emulate it as closely as possible. Plus, of course, you save disc space, and as you noted, all message files must have variable length records anyway. So, variable record length files are good -- but why is the program getting all these trailing blanks? Well, this is caused by a little-known bit of behavior on the part of the file system, behavior that I think is quite inconsistent with the way things normally work. The FOPEN intrinsic has 13 parameters -- filename, foptions, aoptions, device, number of extents, etc. When you use FOPEN to create a new file, you might specify many of these parameters; you can tell the file system how many extents the file should have, what its filecode should be, whether it should have fixed-length records or variable-length records, etc. Now, say you're opening an already existing file. You might tell FOPEN its file size, its number of extents, its file code, its ASCII/BINARY flag, etc.; but what can the file system do with it? If the file already has filecode 123, 16 extents, and room for 1500 records, that's what it'll always have -- an FOPEN call can't change it. Now, there are still a lot of FOPEN parameters that make sense for an old file -- the access mode (INPUT/OUTPUT/APPEND/etc.), the number of buffers, etc. However, the file size, number of extents, number of user labels, blocking factor, etc. are all ignored when you're opening an old file. Normally, the record format (fixed, variable, or undefined) is one of those parameters that is ignored for already-existing files. If a file is built with fixed-length records, then opening it as a variable record length file won't change its fundamental structure; the file will remain a fixed record length file, and when read will always return records of the same length. Similarly, a variable record length file will always look like a variable record length file, even if it is opened as a fixed record length file (which, in fact, is the default FOPEN parameter setting). As you've probably guessed by now, message files are an exception. You :LISTF one, and you'll see it as a "VAM" or a "VBM" file (Variable Ascii/Binary Message file). But, when your program opens it, it will see it is as EITHER A FIXED RECORD LENGTH FILE OR AS A VARIABLE RECORD LENGTH FILE, DEPENDING ON THE SETTING GIVEN IN THE FOPEN CALL (which is bits .(8:2) of the "foptions" parameter). On disc, the message file has variable-length records -- but, if the program opens it with foptions.(8:2) = 0 (indicating fixed-length records), it will see all these records as fixed-length records, padded with trailing blanks if necessary. So, your program opens its $STDIN file (or has its $STDIN opened for it by MPE) with the default foptions, which indicate fixed-length records, and all subsequent reads of this file get fixed length records padded with blanks. All you need to do is override this with a :FILE equation, to wit :FILE INFILE=MSGFILE;REC=,,V :RUN MYPROG;STDIN=*INFILE Or, if you're running MYPROG from within a program (using the CREATEPROCESS intrinsic), you can specify the "REC=,,V" directly in the $STDIN file parameter (which can be a complete right-hand side of a file equation, not just a filename). And that's the story. Usually, the record format, the record size, the ASCII/BINARY flag, the file code, the number of extents, etc. are all ignored when opening an already existing file; however, when opening an already existing message file (whether it's the program's ;STDIN= or not), the record format (fixed vs. variable) is not ignored. Although the message file on disc remains variable record length, the program will see the file as fixed record length if that's the way it opened it. I suppose that the reason for this is the very fact that you're not allowed to :BUILD a message file with fixed length records; if you should happen to have a program that MUST have fixed length records for input, there has to be some way of viewing the message file as a fixed record length file. Q: When one of my programs reads its data file, it gets a rather bizarre error, as if the input data was incorrect. The data file looks OK, but a friend suggested that it might have some kind of "garbage characters" in it (like escapes or nulls). How can I find out if this is in fact the case? (I tried DISPLAY FUNCTIONS but it didn't show anything.) A: Well, DISPLAY FUNCTIONS shows many special characters but by no means all. In particular, it will not show: NULL (decimal 0) ENQ, also known as control-E (decimal 5) DEL (decimal 127) Also, any characters with the high-order (parity) bit set -- those with values 128-255 -- might be shown as either their parity-less equivalents (i.e. character 193 might be shown as A, which is a 65). Things are seldom what they seem. A file which seems to contain XYZ123 might actually have three null characters before the "X", a control-E between the "Z" and the "1", and the parity bit set in the "2" and the "3". Your program will see these characters, and will no doubt abort because of them; however, when you look at the file yourself using, say, EDITOR, you won't see them, even using DISPLAY FUNCTIONS. Your best bet is the FCOPY ;CHAR parameter. If you say :FCOPY FROM=DATAFILE;TO;CHAR then FCOPY will print all the records in DATAFILE, but replacing all garbage characters by "."s. If you say :FCOPY FROM=DATAFILE;TO;CHAR;OCTAL it will also display the data in each record in octal -- this way, you can see the exact values of each of the special characters (and also see which "."s are garbage characters and which are genuine dots!). You can even say :FCOPY FROM=DATAFILE;TO;CHAR;HEX which will display the data both as text (with dots replacing garbage characters) and in (surprise!) hexadecimal. Hex is actually better than octal for this, since in hex each byte is exactly two hex digits. Finally, if you don't want to use FCOPY and are fortunate (?) enough to have an HP 2645 terminal, you can hit the DISPLAY FUNCTIONS key and the CONTROL key simultaneously and enter so-called "MONITOR MODE" (some HP 2645s don't have this option, but most do). In this mode, all control characters, including NULLs, ENQs, and DELs will be displayed; however, characters with parity bit set will still be displayed as their parity-less equivalents, and every so often the terminal will hang up waiting for you to type a control-F (since in monitor mode even the ENQ/ACK protocol is displayed). On the plus side, hitting control-DISPLAY FUNCTIONS will actually make the DISPLAY FUNCTIONS light blink -- how high-tech! Q: Where are the HP system intrinsics kept? How are they different from, say, compiler library routines (like the mathematical routines such as SQRT, EXP, etc. or others like EXTIN', INEXT', and RAND)? How are they different from the "internal" routines like ATTACHIO and EXCHANGEDB? A: The actual machine code for all HP intrinsics is kept in the system SL -- SL.PUB.SYS. This, incidentally, is also where all the compiler library routines are kept, too, as are the internal MPE routines you mentioned. In fact, when we speak of the "operating system", we're really talking about SL.PUB.SYS (plus some other stand-alone PUB.SYS programs like LOAD, MEMLOG, and the I/O drivers). From MPE's point of view, then, FOPEN (an intrinsic), SQRT, and ATTACHIO are the same sort of thing -- system SL procedures that are called as external references of programs. If your program calls FOPEN, SQRT, and ATTACHIO, the loader will take care of these external references in exactly the same way -- it will find all three procedures in the system SL and link them to your program for the duration of its execution. The difference between FOPEN and ATTACHIO -- beyond, of course, the fact that they do different things -- is that you can say INTRINSIC FOPEN; << SPL >> SYSTEM INTRINSIC FOPEN << FORTRAN >> FUNCTION FOPEN: SHORTINT; INTRINSIC; << PASCAL >> CALL INTRINSIC "FOPEN" ... << COBOL II >> If you say this, then the compiler will simply KNOW WHAT THE NUMBER AND TYPE OF EACH PROCEDURE PARAMETER WILL BE. You can call ATTACHIO from SPL, FORTRAN, PASCAL, and COBOL, but the compiler won't know anything about the procedure parameters -- you'll have to define them yourselves (using an EXTERNAL declaration in SPL or PASCAL or by specifying the correct calling sequence in FORTRAN or COBOL), and woe to you if you define them incorrectly! The various compiler INTRINSIC constructs simply go to a file called SPLINTR.PUB.SYS and pick up the procedure definitions from there. SPLINTR.PUB.SYS (in its own rather cryptic way) defines, for instance, that FOPEN is OPTION VARIABLE and has 13 parameters, of which the first, fifth, and sixth are byte arrays passed by reference, the tenth is a double integer passed by value, and all the others are ordinary integer passed by value. This way, any compiler to which FOPEN is declared as an intrinsic will know how to generate the correct code to call it. Thus, if I were asked "what is an intrinsic?", I'd say "anything whose parameter layout is recorded in SPLINTR.PUB.SYS", which means the same as "anything which can be called using the INTRINSIC construct in SPL, FORTRAN, PASCAL, or COBOL II". Note that this includes more than what is documented in the System Intrinsics Manual -- it also includes all the IMAGE intrinsics, V/3000 intrinsics, graphics intrinsics, everything described in the Compiler Library Manual (including SQRT, EXTIN', INEXT', RAND, etc.), and other various and sundry routines. Thus, saying that something is an "intrinsic" is not really a reflection on where the procedure code resides (which is in SL.PUB.SYS, along with all the other non-intrinsics) or what the procedure does (except insofar as HP didn't choose to describe most of the more privileged system internal routines in the SPLINTR file). Intrinsics are just easier to call (from most languages) than non-intrinsics, since the language will "know" a lot about how the code to call the intrinsic needs to be constructed. Finally, to complicate matters further, SPL and PASCAL allow you to define your own "intrinsic files". Again, when you make your own procedure an "intrinsic", you aren't adding it to the operating system or even necessarily to the system SL (it could be in another USL, an RL, or a group or account SL) -- you're simply adding a description of its parameters to your own intrinsic file, thus making it simpler for your programs to call it. Q: How can I write a random number generator? I tried just calling the TIMER intrinsic and taking its results modulo some number, but the results didn't seem very random. Is there a random-number generator provided by HP? A: Writing computer games on company time, eh? Well, I can think of worse things to be doing. The HP Compiler Library Reference Manual (part number 30000-90028) is probably one of the least-read (and least-updated) manuals put out by HP. It describes exciting procedures like FMTINIT', FTNAUX', ADDCVRV', and such. However, it also describes: * A number of mathematical support routines (like cosine, sine, tangent, arctangent, logarithm, etc.); * Two rather nifty procedures called EXTIN' and INEXT', which convert strings to numbers and vice versa (including real numbers, in either exponential [1.234E-02] or fixed-point [.01234] format); * And, among other things, the HP random number generator. I guess the random number generator is considered part of the "compiler library" because compiled BASIC and FORTRAN programs might need it. In any case, it's provided by HP and even documented. There are two routines provided: * RAND1, which returns a real "seed" value. * RAND, which returns a real random number in the range 0.0 to just below 1.0 (i.e. RAND might return 0.0, but never 1.0). You pass it as a by-reference parameter the seed returned by RAND1; RAND modifies the seed on every call -- you should always pass the same seed variable to RAND. A reasonable SPL procedure that uses these routines might be: INTEGER PROCEDURE RANDOM (MAX); VALUE MAX; INTEGER MAX; BEGIN << Returns a random integer from 0 to MAX-1. >> OWN LOGICAL INITIALIZED:=FALSE; OWN REAL SEED; IF NOT INITIALIZED THEN BEGIN SEED:=RAND1; INITIALIZED:=TRUE; END; RANDOM:=INTEGER(FIXT(RAND(SEED)*REAL(MAX))); END; Note that the seed value passed to RAND completely determines the generated random number. Thus, if you wanted to, you could have the program input the seed value from the user rather than calling RAND1 to initialize it -- that way, you'll always be able to duplicate the sequence of random numbers generated by just specifying the same seed next time you run the program.