many-data.sas Create many output tables, no input data.Richard A. DeVenezia, 03 June 2003
This problem pertains to SAS version 8.2 with latest hotfixes. The problem has not (yet) manifested for me in preproduction release 9.0.
Consider a base table with N groups. You are required to split this table into N tables, each table containing only records from one group. (jump to: program-1, program-2)
I had been working on this problem recently and noticed a related post in SAS-L (https://listserv.uga.edu/cgi-bin/wa?A2=ind0306a&L=sas-l&D=1&O=D&P=2731)
My work involved quite a few groups (>250) and in some recent testing I noticed an unusual SAS ERROR: I had never seen before:
ERROR: Rename of temporary member for X.D0000217.DATA failed.
File may be found in d:\temp\foo.
and in some cases:
ERROR: ERROR: A lock is not available for X.D0000217.DATA, lock held by another process.
I am trying to understand the source of these error from the standpoint of my hardware or system configuration. The typically scenario is thus:
A run; generates the first error (the source cause as yet unknown) and a second run; generates the second error when trying to recreate the member for which the rename failed.
I also have a recollection (but possibly a mismemory) of a run; that worked and a second run; that generated the second error message. If true, this indicates a file was renamed, but the state of some internal element of the file was not properly set.
I have been unable to find the exact parameters needed to cause failure on my system, but the problem occurs when the data step is creating more than 200 data sets. Presume 300 tables are being created. On my system, the most tables appearing in the log before the error appears has been ~217. All the tables occurring after the one having the rename error are OK.
Presume we are using a DATA Step to create 300 same sized tables named Di, D0000001 to D0000300. An inspection of the library being written to while the data step in running indicates
Additionally, I have been able to produce the same type of error when simply creating a lot of tables, with no input data from a SET statement.
Some system info:
So far I've spouted plenty of woe and hot air, what am I asking for ?
Please submit some the following code to a SAS session on your system.
If you experience the rename error or want to report run results,
please contact me.
many-data.sas Create many output tables, no input data.
Try different parameter values and libname paths.
| %ManyData (outlib=, nTables=, nRowsPer=, nVars=) | ||||||
| P4-3.06Ghz | ||||||
| # Tables | Each table has | |||||
| # Rows | # Variables | real time | cpu time | outlib= | Rename error occurred at table | |
| 1000 | 100 | 30 | 16.67 | 4.56 | X -> D:\temp\saswork\_TD1168 | |
| 1000 | 100 | 50 | 18.18 | 4.70 | X -> D:\temp\saswork\_TD1168 | |
| 500 | 500 | 100 | 31.45 | 2.28 | X -> D:\temp\saswork\_TD1168 | |
| 500 | 500 | 100 | 54.92 | 2.62 | X -> C:\temp\foo | |
| 500 | 500 | 100 | 28.35 | 2.57 | X -> D:\temp\foo | 211 |
| 1000 | 1000 | 75 | 1:16.12 | 9.79 | X -> D:\temp\foo | 221 |
| P3-850, 384M ram PC-100, Adaptec 29160 in 32bit slot, Quantum Atlas 18g (M:), Win2K SP3 + Updates | ||||||
| 1000 | 1000 | 75 | 4:14.86 | 1:23.48 | X -> M:\temp\_saswork\_TD11000 | 238 |
| 1000 | 1000 | 75 | 2:30.63 | 40.27 | X -> M:\temp\_saswork\_TD1036 | 237 |
| 1000 | 50 | 75 | 48.29 | 11.84 | X -> M:\temp\_saswork\_TD896 | 237 |
| 1000 | 50 | 75 | 1:17.00 | 11.66 | X -> C:\temp\foo | |
| 850 | 1000 | 50 | 2:48.06 | 41.91 | X -> C:\temp\foo | 237 |
| 1000 | 50 | 75 | 41.61 | 12.24 | X -> D:\temp\foo | |
| 1000 | 50 | 75 | 58.02 | 10.78 | X -> E:\temp\foo | |
| 1000 | 50 | 75 | 1:34.49 | 19.71 | X -> F:\temp\foo | 237 |
| Your results can go here. | ||||||
This is what the log will look like if you get an error. The program also opens a window if an error occurred.
...
NOTE: The data set X.D0000208 has 500 observations and 101 variables.
NOTE: The data set X.D0000209 has 500 observations and 101 variables.
NOTE: The data set X.D0000210 has 500 observations and 101 variables.
NOTE: The data set X.D0000211 has 500 observations and 101 variables.
ERROR: Rename of temporary member for X.D0000211.DATA failed.
File may be found in d:\temp\foo.
NOTE: The data set X.D0000212 has 500 observations and 101 variables.
NOTE: The data set X.D0000213 has 500 observations and 101 variables.
...
split-data.sas More complicated program. Create one large table and split it into many smaller output tables.
Try different parameter values and libname paths. The nPerPass parameter will let you split data by outputting smaller amounts of tables at a time. Since repeated passes of the data are necessary when nPerPass < nGroups, the SASFILE statement is used to open the source data into a memory buffer.
| %MakeData (out=perm.BigPile, nGroups=1000, nRowsPer=500, nVars=50, force=1) | ||||||||
| %SplitData (data=perm.BigPile, outlib=, splitVar=groupId, nPerPass=) | ||||||||
| # Groups | # Tables per pass |
Each table has | ||||||
| # Rows | # Variables | real time | cpu time | outlib=libname X | Rename error occurred at table | |||
| 1000 | 1000 | 500 | 50 | 51.01 | 7.07 | d:\temp\saswork\_TD800 | 192 | |
| Your results can go here. | ||||||||
This is what the log will look like if you get an error. The program also opens a window if an error occurred.
...
NOTE: The data set X.D0000190 has 500 observations and 101 variables.
NOTE: The data set X.D0000191 has 500 observations and 101 variables.
NOTE: The data set X.D0000192 has 500 observations and 101 variables.
ERROR: Rename of temporary member for X.D0000192.DATA failed.
File may be found in d:\temp\saswork\_TD800.
NOTE: The data set X.D0000193 has 500 observations and 101 variables.
NOTE: The data set X.D0000194 has 500 observations and 101 variables.
...