SSMA bug watch

The SQL Server 2005 Developer Edition installed on my Windows XP SP3 laptop had crashed over the weekend, so after a fresh reinstall including SP2 with hotfixes up to version 3073, I decided to re-run my SQL Server Migration Assistant for Access (SSMA) to create a new conversion file for testing.

The application performance was extremely poor, so I tried running SQL Profiler to try to find the bottlenecks, but I was seeing reads in the range of 20-50 which seemed trivial, however the Audit Login and Audit Logout events were literally taking minutes instead of seconds. I had saved the trace to a table, so I copied some of the SQL statements into SSMS and they ran very quickly, each barely taking a second.

I did a bit of research on connection pooling, but unfortunately that .NET option is not available in either SQL Server 2005 or MS-Access 2003. So why was Access connecting & disconnecting for every single database access?

Then I looked into the Access app and discovered that the SSMA was using the "SQL Server" provider in the connection strings. I changed that to the "SQL Native Client" and voila! the app performance sped up by a factor of at least 10x. Overall, the performance is now marginally slower than when using the native Access tables. That's fine, because this database will ultimately be running on a real Windows 2003 server with four dual-core CPU's and mucho RAM, waaay more powerful than the puny Centrino in my laptop.

I also took a couple of miscellaneous Access files & converted those as a further test, and they all converted using the old "SQL Server" intended for SQL 2000.

I've always run the SSMA using the wizard to save time, but now I'll need to revisit that to see if there is an option to use the correct provider for SQL 2005.

Max Locks Per File

I have a replicated MS-Access database that is used to store archive data from the production system. This archive is Access 2002 format and is approx 140 MB in size.

The design master was recovered from a replica & saved to a local C: drive, and then a new replica was created & published to a network share. So far, so good.

However, when I attempted to sync the two files, I received an error that the MaxLocksPerFile setting in the registry was exceeded and the sync could not be done. The "Help" button said that the default was 9500 (decimal), and suggested that I use SetOption to change it, but SetOption() has no such option.

I found it curious that a brand-new replica would require the complete sync process even though nothing was changed in either design or data, but I digress...

In any event, I did find the key in the registry by that name, it is located at:

HKLM\SOFTWARE\Microsoft\Jet\4.0\Engines\Jet 4.0\MaxLocksPerFile

=dword:0000251c

I changed the setting to 12,000 and the progress meter showed two bars and then froze with the error message. With 15,000 it reached five bars; and finally 25,000 which worked. The progress meter displayed the typical appearance & the sync was successful after a few seconds.

I then went back to experiment with a lower setting of 20,000 to try to find the go/no-go level but the sync was nearly instantaneous, so I suppose that the act of moving the replica forced Access to run the sync process on the entire contents of the file.

Excel prompts to save when no changes made

I have a large Excel workbook that is a catch-all for budgeting, time analysis & other miscellaneous items. When I would open the file to look at something, not change anything but simply close it, Excel would prompt me to save changes?

I found this article that explains "volatile formulas" in Excel such as CELL(), NOW(), RAND(), TODAY() and a few others. Here is the full article:

"Save changes in " prompt even if no changes are made

In the database world, expressions like NOW() are called non-deterministic functions, because they don't have a single repeatable value - NOW() always returns the current date / time which of course changes every time you invoke it.

If your worksheet uses those functions, the only way to avoid the "save" prompt is to turn off automatic recalculation & check the option for recalculate before save. But that makes your worksheet "dead" in the sense that changing cell values does not recalc, which is one of the great things about Excel. You'd have to either press F9 or click the Save icon to recalc after every edit.

Apparently this affects all versions of Excel from 97 up thru 2007.

Sharing Linux folders for Windows

Sharing Windows folders for access by Ubuntu Linux is fairly straightforward using Samba (SMB) but going in the opposite direction had me stumped.

From Windows, I would go into My Network Places ~ Entire Network ~ Microsoft Windows Network ~ Workgroup and I could see the Linux machine. Then I would double-click on it and enter my Linux name and password, but it would not connect.

The solution is that you need to add a Samba user, and you can't do that from the Linux GUI. You have to open a Terminal window and add the Samba user like this, where username is your Linux user name:

sudo smbpasswd -a username

after which you are prompted for the password, which is your Linux password.

At that point, you can access your Linux shares from Windows just like you would any other network share.

Replicated Queries

MS-Access makes it very easy to change the SQL of a query simply by using a statement like

CurrentDb.QueryDefs("MyQuery").SQL = "SELECT * FROM MyTable"

there is no need to Dim or Set anything, simply plug in the SQL and you're ready to go.

However, I just discovered that this does not work in a replicated database. Or I should say, it works in the Design Master but it does not work in a Replica. I needed to apply some complex criteria, more complex than could be expressed simply by using a WHERE condition when opening a report. Testing the code in the DM worked just fine, but users received a message "cannot update...object is read-only" because Access considers this to be a design change & does not allow that in a Replica.

You can code things like changing visibility, font colors, enabling & disabling controls, but QueryDefs are off limits.

The challenge: I had an existing report that used a query for the detail section, and the report footer contained a subreport with a summary that used the same query as the body of the report. Opening the report with a filter or where argument would properly filter the detail section, but the subreport would display all of the data. I tried using the OnOpen or OnFormat event to apply a filter to the subreport but that did not work.

(EDIT) At this point, I tried to change the SQL for the QueryDef using the type of statement shown above, to filter the data for both the body of the report and the summary subreport; this worked in the DM but not the Replica. My original post has not been clear about that. (EDIT)

My solution was basically to create a staging table for the subreport, and flush-and-fill the table prior to opening the report.

In the DM I did this:

• Ran the old (bad) code to set up the SQL statement
• Switched the query to a make-table, to create the staging table
• Switched the query to an append query
• Copied the append SQL and used that to build up a dynamic SQL statement in VBA

The button OnClick would then delete all records from the staging table, run the dynamic SQL to populate that table, and then open the report.

Moving Days

{EDIT} Around about September 2008, I decided not to move articles off this blog after all. I might copy a few to the tips page at my Web site, but otherwise I'm going to keep this blog intact. {EDIT}

When I started this blog three years ago, I had intended to use it as a quick way to publish my technology findings both for the general Internet community, and also to use it as a knowledge base for myself.

As it turns out, the older articles wind up being buried in the archives, grouped by month, so many times I've had to guess the month & drill down from there. The built-in search is quite effective, but being able to scan down a complete list of topics is much faster.

For those reasons, I've been moving articles from blogger to my personal web site, where I use a classic ASP page and the FileSystemObject to present a scrollable list of all articles, sizes, dates etc. The current list can be found at http://www.wvmitchell.com/tips.

Another difficulty using the blogger format is that many html tags are scrubbed-out from posts, so any code samples will lose their indenting & formatting, making them hard to decipher. My workaround has been to present a link to the actual code on a separate web page, but I would prefer to be able to post a complete article intact with all the formatting.

The blogger search facilities work very well, and Google indexes the blogger posts, so much information can be found via a normal Google search, but my web site also includes a search feature so nothing will be lost.

In closing, this blog will eventually be shrunk down to include only tech comments and observations.

Batch Files

By all accounts, DOS is dead. But since cmd.exe is still part of Windows, you can still write DOS batch programs to automate certain tasks.

For example, I am working with an Access application that is rebuilt every month. The previous process was to have the user navigate to a shared drive and use Windows Explorer to copy a new version of the MDB file and then paste it to a specific folder on their C: drive.

The new process uses a simple batch file like this:

@ECHO OFF
CLS
ECHO.
ECHO *** XYZ file setup program ***
ECHO.
IF NOT EXIST C:\XYZ MD C:\XYZ

IF NOT EXIST C:\XYZ\XYZ.MDB GOTO NOFILE
ECHO.
ECHO You already have a copy of XYZ on your computer
ECHO ===============================================
ECHO.
ECHO  Do you want to replace your existing copy?
ECHO  ------------------------------------------
GOTO HASFILE

:NOFILE
ECHO *** COPYING FILE, PLEASE WAIT ***
COPY /Y \\MYSERVER\XYZ\XYZ.MDB C:\XYZ
GOTO END

:HASFILE
ECHO  Y = Yes, replace my old file
ECHO  N = No, keep my existing copy of the file
ECHO.
SET CHOICE=
SET /P CHOICE= Type the letter and press {ENTER}
IF /I '%CHOICE%'=='Y' GOTO NOFILE
ECHO.
ECHO  (Nothing was copied or replaced)
ECHO.
:END
PAUSE
EXIT

For a new user, the folder is created if it does not exist.
Then, if the file exists the user is prompted to either overwrite or keep their existing copy.

Without the CHOICE statement, DOS would ask the overwrite question as
Yes/No/All
and this avoids the confusing "All" option.

Removing Access Replication

Using replication with MS-Access adds numerous system fields and tables, and combined with the replication-tracking system, a replicated Access database can grow to an unwieldy size, especially if you have many users and/or there are frequent changes in the design master.

The customary solution is to rebuild the database, i.e. to create a new non-replicated file and then use that to create a new design master and the replicas.

Unfortunately, you cannot simply create a new empty file and import all your Access objects, because all the system fields in the tables will also be carried over via the import. You cannot simply go into design view & delete those replication fields because they are system fields.

The basic process is to create a new empty file, and then export all the Access objects except the tables, and then export the tables but without the replication fields.

Import the VBA module Unreplicate_Access into your design master, and when you execute the code it will create a new non-replicated version of your database. It will also set a number of startup properties such as hiding the DB Window, turning off shortcut keys and so on.

Once you have created the new file, simply copy it over your existing design master (after you make a backup copy, of course) and then click Tools ~ Replication ~ Create a Replica and follow the prompts.

This module is still a work in progress; it does not set any primary keys, nor does it copy any default values for the tables. This will appear in a future post. But there won't be any code to create relationships; Microsoft actually recommends against using RI in a replicated db due to the mechanics of replication.

SSMA, part 3

My previous posts about the SQL Migration Assistant for Access reported generally favorable results with this product which is used to migrate Access data into SQL Server. It does a remarkably better job than the old Access Upsizing Wizard which very often failed, especially when dealing with date fields. When SSMA has completed its work, the system is mostly operational.

However, there are two common Access design flaws that can impact the application - the first is relying on the behavior of Boolean fields to evaluate as -1 = True and 0 = False. Many Access developers will test for "-1" meaning True, but SQL Server returns "1" for bit fields so the comparison logic fails. The fix is simply to substitute "-1" with "True" (without the quotes) and the operation will succeed.

A second design flaw recently noted is that in Access you can subtract one date from another date, and then perform math on the result - for example

24 * ([date_end] - [date_start])

in Access will give you the number of hours between the two date/time values. However, once your data is moved into SQL Server, this expression will result in

Implicit conversion from data type smalldatetime to int is not allowed. Use the CONVERT function to run this query.

The solution is to use the DateDiff function which exists in both Access and SQL Server. In Access you would write

DateDiff("h", [date_start], [date_end])

and when converting this query to T-SQL for SQL Server you might use

DATEDIFF(hh, [date_start], [date_end])

Access Replication

Replication in Access provides a method of allowing remote users to work with a local copy of the database while disconnected from the network, and then synchronizing their changes when they do connect. This feature has been part of Access since the 90's but sadly, although Access 2007 supports it if you stay with the .mdb file format, if you choose to convert to the new .accdb format this feature goes away (along with WorkGroup Security).

The idea of replication is that you create a Design Master (DM) that adds additional hidden system fields which Access uses to track record changes that need to be replicated. You also create a Replica which has all the programming and data of the DM but without any design options, thus protecting the design from unauthorized changes. During the sync process, data is exchanged between the DM and replica, and design changes made to the DM are also propagated to the replica.

The replication model is sensitive to the drive, path, and file name; moving the DM will cause it to become a replica; copying the DM will result in creating another replica.

In addition to the fields added to each table, there are also a number of hidden system tables created when you replicate a database; the net result is that the DM and the replicas are considerably larger than the original non-replicated file. For example, a 27 MB file can grow to 44 MB when replicated. Once the users begin working with the file and syncing, that file can grow to 48-59 MB (these numbers are taken from an actual production system). For that reason, it is strongly recommended that you enable the Compact on Close option to keep the file as small as possible under the circumstances.

It is important to note that creating a replica will change AutoNumber fields from Increment to Random. This is necessary to prevent conflicts - for example, if user A added a new record, and user B added a new record, Increment would otherwise result in a key violation when attempting to sync. Therefore, if the app relies on a sequential series of new numbers, e.g. invoice numbers, you will need to develop a different strategy for that numbering system.

During the sync process, any conflicts will result in a hidden table being created, e.g. if a table tblOrders has sync conflicts then Access will create a tblOrders_Conflict table that holds the GUIDS for the "winning" and "losing" replica, table, and field.

It should be noted that simply deleting old records will not necessarily reduce the file size; Access maintains a system table, MSysTombstone, which retains a pointer to every deleted record. Unless you utilize Replication Manager (from the Office 2000 developer kit) or a third-party utility, the default retention period is 1000 days, so all those deleted records will be adding to the size of the file with little real benefit.

A common technique of dealing with the bloat of replication is to rebuild the master file periodically. To accomplish this, you first have all users sync & then delete their local copy, then you would need to create a new empty file, export all the Access objects into the new file, and also create new tables in the new file that do not have the replication fields & export the data into those new tables. (It is not possible to delete the replication fields from the tables because they are system fields.) The final steps would be to convert that new file to a design master & create a replica, and then distribute that replica to the database users.